[
https://issues.apache.org/jira/browse/FLINK-24303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17417578#comment-17417578
]
Stephan Ewen edited comment on FLINK-24303 at 9/20/21, 11:01 AM:
-----------------------------------------------------------------
The issue is relatively easy to fix on the level of the {{SourceCoordinator}}.
Why not fix it for all {{OperatorCoordinators}} on the generic level? Meaning,
trigger failover when the start method of {{OperatorCoordinator}} produces an
exception? I was looking to do that initially, but the scheduler is not
prepared to handle global failures in cases where the starting procedure wasn't
completed.
I investigated whether re-instantiating the source enumerator when restoring
from a checkpoint has the same problem. It behaves already correctly, resulting
in a global failover rather than a JM crash. I am adding a test to guard this
behavior.
was (Author: stephanewen):
The issue is relatively easy to fix on the level of the {SourceCoordinator}}.
Why not fix it for all {{OperatorCoordinators}} on the generic level? Meaning,
trigger failover when the start method of {{OperatorCoordinator}} produces an
exception? I was looking to do that initially, but the scheduler is not
prepared to handle global failures in cases where the starting procedure wasn't
completed.
I investigated whether re-instantiating the source enumerator when restoring
from a checkpoint has the same problem. It behaves already correctly, resulting
in a global failover rather than a JM crash. I am adding a test to guard this
behavior.
> SourceCoordinator exception may fail Session Cluster
> ----------------------------------------------------
>
> Key: FLINK-24303
> URL: https://issues.apache.org/jira/browse/FLINK-24303
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
> Reporter: Seth Wiesman
> Assignee: Stephan Ewen
> Priority: Blocker
> Fix For: 1.14.0
>
>
> The SourceCoordinator currently forwards all exceptions from
> `Source#createEnumerator` up the stack triggering a JobMaster failover.
> However, JobMaster failover only works if HA is enabled[1]. If HA is not
> enabled the fatal error handler will simply exit the JM process killing the
> entire cluster. This is problematic in the case of a session cluster where
> there may be multiple jobs running. It also does not play well with external
> tooling that does not expect job failure to cause a full cluster failure.
>
> It would be preferable if failure to create an enumerator did not take down
> the entire cluster, but instead failed that particular job.
>
> [1]
> [https://github.com/apache/flink/blob/7f69331294ab2ab73f77b40a4320cdda53246afe/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L898-L903]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
