[
https://issues.apache.org/jira/browse/FLINK-17947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-17947:
----------------------------------
Description:
Currently, it can happen that a REST handler sends a request to a leader
{{RpcEndpoint}} and before the {{RpcEndpoint}} has a chance to respond, it
might shut down (e.g. due to losing the leadership). In this case, the
{{ActorSystem}} will send an {{AskTimeoutException}} as the response with the
message {{Recipient Actor[akka://flink/user/rpc/dispatcher_1#-1875884516] had
already been terminated.}}. This exception will be treated as any other
exception and forwarded to the REST client. There it will be treated as a
normal timeout exception which causes the operation (e.g. requesting job
details) to fail.
I was wondering whether this case should not be handled slightly differently.
If the REST handler would respond with a {{SERVICE_UNAVAILABLE}} HTTP response
code, then the {{RestClusterClient}} would retry the operation. One could think
of it as if there wouldn't have been a leader available before. This is similar
to the situation when there is no current leader and we are waiting for the
leader election to finish. Alternatively, we could extend the
{{RestClusterClient.isConnectionProblemOrServiceUnavailable}} predicate to also
cover the case of special {{AskTimeoutExceptions}}.
cc [~chesnay]
was:
Currently, it can happen that a REST handler sends a request to a leader
{{RpcEndpoint}} and before the {{RpcEndpoint}} has a chance to respond, it
might shut down (e.g. due to losing the leadership). In this case, the
{{ActorSystem}} will send an {{AskTimeoutException}} as the response with the
message {{Recipient Actor[akka://flink/user/rpc/dispatcher_1#-1875884516] had
already been terminated.}}. This exception will be treated as any other
exception and forwarded to the REST client. There it will be treated as a
normal timeout exception which causes the operation (e.g. requesting job
details) to fail.
I was wondering whether this case should not be handled slightly differently.
If the REST handler would respond with a {{SERVICE_UNAVAILABLE}}} HTTP response
code, then the {{RestClusterClient}} would retry the operation. One could think
of it as if there wouldn't have been a leader available before. This is similar
to the situation when there is no current leader and we are waiting for the
leader election to finish. Alternatively, we could extend the
{{RestClusterClient.isConnectionProblemOrServiceUnavailable}} predicate to also
cover the case of special {{AskTimeoutExceptions}}.
cc [~chesnay]
> Retry REST requests if RpcEndpoint died before responding to request
> --------------------------------------------------------------------
>
> Key: FLINK-17947
> URL: https://issues.apache.org/jira/browse/FLINK-17947
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
> Affects Versions: 1.10.1, 1.11.0
> Reporter: Till Rohrmann
> Priority: Major
> Fix For: 1.12.0
>
>
> Currently, it can happen that a REST handler sends a request to a leader
> {{RpcEndpoint}} and before the {{RpcEndpoint}} has a chance to respond, it
> might shut down (e.g. due to losing the leadership). In this case, the
> {{ActorSystem}} will send an {{AskTimeoutException}} as the response with the
> message {{Recipient Actor[akka://flink/user/rpc/dispatcher_1#-1875884516] had
> already been terminated.}}. This exception will be treated as any other
> exception and forwarded to the REST client. There it will be treated as a
> normal timeout exception which causes the operation (e.g. requesting job
> details) to fail.
> I was wondering whether this case should not be handled slightly differently.
> If the REST handler would respond with a {{SERVICE_UNAVAILABLE}} HTTP
> response code, then the {{RestClusterClient}} would retry the operation. One
> could think of it as if there wouldn't have been a leader available before.
> This is similar to the situation when there is no current leader and we are
> waiting for the leader election to finish. Alternatively, we could extend the
> {{RestClusterClient.isConnectionProblemOrServiceUnavailable}} predicate to
> also cover the case of special {{AskTimeoutExceptions}}.
> cc [~chesnay]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
