[
https://issues.apache.org/jira/browse/FLINK-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878018#comment-15878018
]
ASF GitHub Bot commented on FLINK-5830:
---------------------------------------
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/3360
Thanks for the clarification @zhijiangW. I know understand the problem that
we effectively introduce via `RpcEndpoint.runAsync` another message which might
get "lost" (e.g. due to OOM exception).
I agree with Stephan that it's hard to reason about the consistency of the
`AkkaRpcActor
s` internal state once we see an exception. The conservative approach would
probably be to let it terminate or calling `notifyFatalError` to handle it.
Related to this is also how we handle exceptions in the
`AkkaRpcActor.handleRpcInvocation`. There we catch all exception and simply
send them to the caller. I think in this method we should only send the
non-fatal exceptions back and terminate otherwise.
To follow a similar pattern for the `handleRunAsync` we could think about
returning a `Future` which we return when calling `RpcEndpoint.runAsync` which
will be completed with non fatal exceptions or if the `Runnable` has been
executed. And in case that we see a fatal exception we terminate or call
`notifyFatalError`. What do you think?
> OutOfMemoryError during notify final state in TaskExecutor may cause job stuck
> ------------------------------------------------------------------------------
>
> Key: FLINK-5830
> URL: https://issues.apache.org/jira/browse/FLINK-5830
> Project: Flink
> Issue Type: Bug
> Reporter: zhijiang
> Assignee: zhijiang
>
> The scenario is like this:
> {{JobMaster}} tries to cancel all the executions when process failed
> execution, and the task executor already acknowledge the cancel rpc message.
> When notify the final state in {{TaskExecutor}}, it causes OOM in
> {{AkkaRpcActor}} and this error is caught to log the info. The final state
> will not be sent any more.
> The {{JobMaster}} can not receive the final state and trigger the restart
> strategy.
> One solution is to catch the {{OutOfMemoryError}} and throw it, then it will
> cause to shut down the {{ActorSystem}} resulting in exiting the
> {{TaskExecutor}}. The {{JobMaster}} can be notified of {{TaskExecutor}}
> failure and fail all the tasks to trigger restart successfully.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
