Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/3340
@StephanEwen , sorry for delay update of this feature.
This modification covers almost all the work on job manager side concerning
about the suggestions you mentioned before, including:
1. Once `JobManagerRunner` granting leadership, it will check the job
status by `RunningJobsRegistry`. If the job is already running, then the
`JobManager` will enter reconciling process and confirm leadership last.
To do so, we can confirm that the `TaskManager` will not report task status
before `JobManager` reconciles, and it is also reasonable for 'JobManager`
exposing its address to outside `TaskManager` after internal initialization. To
keep the behavior consistency, I modified the `EmbeddedLeaderService` to set
leader flag as true in the process of `grantLeadership` instead of
'confirmLeadership`.
2. 'ExecutionGraph' recovery based on task status report form
`TaskManager`, and I created the new execution attempt to replace the current
one for recovery, also the same case for `IntermediateResultPartition` to avoid
updating the final fields directly in previous way. And also integrating two
phases of recovery into one phase you mentioned before.
3. After duration time, the `ExecutionGraph` decides to enter `RUNNING` or
`FAILING` from `RECONCILING` based on reconciliation results.
Currently I have not considered the improvements for `LAZY_FROM_SOURCES`
schedule mode in this version, because it needs to remember how many vertexes
have already been scheduled before `JobManager` failure. Maybe we can improve
it in next version if needed.
I may supplement some extra tests and update the attached design doc after
you approve this realization, and the 'TaskManager' part will be submitted in
another separate PR which mainly includes:
1. `TaskManager` will not fail task directly after notified of `JobManager`
leader lost, and triggers a timer to later check whether reconnects to the new
job leader successfully.
2. If reconnection success, it will report the current internal task and
slot status that belongs to this job leader, otherwise it will fail the tasks
as the current behavior.
Wish your further response, and I will begin to consider resource issues of
FLIP-6, thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
