Restarting of flink master does not change the jobId if one yarn
application.
To be simple, in a yarn application that runs a flink cluster, the job id
of a job does not change once the job is submitted.
You can even submit a flink application multiples times to that cluster (if
it is session
Thanks Till - in the case of restart of flink master - I believe the jobid
will be different. Thanks
On Thursday, May 28, 2020, 11:33:38 AM EDT, Till Rohrmann
wrote:
Hi,
Yarn won't resubmit the job. In case of a process failure where Yarn restarts
the Flink Master, the Master will
Hi,
Yarn won't resubmit the job. In case of a process failure where Yarn
restarts the Flink Master, the Master will recover the submitted jobs from
a persistent storage system.
Cheers,
Till
On Thu, May 28, 2020 at 4:05 PM M Singh wrote:
> Hi Till/Zhu/Yang: Thanks for your replies.
>
> So
Hi Till/Zhu/Yang: Thanks for your replies.
So just to clarify - the job id remains same if the job restarts have not been
exhausted. Does Yarn also resubmit the job in case of failures and if so, then
is the job id different.
ThanksOn Wednesday, May 27, 2020, 10:05:40 AM EDT, Till
Hi M,
Sorry I missed your message.
JobID will not change for a generated JobGraph. However, a new JobGraph
will be generated each time a job is submitted.
So that multiple submissions will have multiple JobGraphs. This is because
different submissions are considered as different jobs, as Till
Hi,
if you submit the same job multiple times, then it will get every time a
different JobID assigned. For Flink, different job submissions are
considered to be different jobs. Once a job has been submitted, it will
keep the same JobID which is important in order to retrieve the checkpoints
Hi Zhu Zhu:
I have another clafication - it looks like if I run the same app multiple times
- it's job id changes. So it looks like even though the graph is the same the
job id is not dependent on the job graph only since with different runs of the
same app it is not the same.
Please let me
Hi Zhu Zhu:
Just to clarify - from what I understand, EMR also has by default restart times
(I think it is 3). So if the EMR restarts the job - the job id is the same
since the job graph is the same.
Thanks for the clarification.
On Monday, May 25, 2020, 04:01:17 AM EDT, Yang Wang
wrote:
Just share some additional information.
When deploying Flink application on Yarn and it exhausted restart policy,
then
the whole application will failed. If you start another instance(Yarn
application),
even the high availability is configured, we could not recover from the
latest
checkpoint
Hi M,
Regarding your questions:
1. yes. The id is fixed once the job graph is generated.
2. yes
Regarding yarn mode:
1. the job id keeps the same because the job graph will be generated once
at client side and persist in DFS for reuse
2. yes if high availability is enabled
Thanks,
Zhu Zhu
M
10 matches
Mail list logo
