[
https://issues.apache.org/jira/browse/FLINK-24063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407945#comment-17407945
]
Aitozi commented on FLINK-24063:
--------------------------------
Hi [~wangyang0918] I truly mean the {{STOP_APPLICATION}} in the {{runCluster
not the startCluster.}}
The step 3 and 4 is point to a case not just one machine problem. It's more
like a cluster level network problem. So it will fail fast on the second node.
Then in current implementation, the application will be shut down.
I glanced over your PR，I think the idea of {{NotAllJobsFinishedException}} may
take effect. My original thought is that we should keep retrying {{runCluster}}
if the job not all reach global termianted.
> Reconsider the behavior of ClusterEntrypoint#startCluster failure handler
> -------------------------------------------------------------------------
>
> Key: FLINK-24063
> URL: https://issues.apache.org/jira/browse/FLINK-24063
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Reporter: Aitozi
> Priority: Minor
>
> If the job runCluster failed, it will trigger the STOP_APPLICATION behavior.
> But if we consider a case like that:
> # A job have running for a long time
> # Then the JobManager encounter a fatal error like the network problem,
> which may let the jobManager process down
> # Then a new process will be started by the resource framework like yarn or
> kubernetes. But it will failed at the ClusterEntrypoint#startCluster due to
> the same network problem.
> # Then the job turn into the FAILED status.
>
> This means a streaming job will no longer run due to some fatal error, this
> is somehow fragile. I think we should give some retry mechanism to prevent
> the job fast fail twice ,so that deal with some external error which may keep
> for a period of time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
