Till Rohrmann created FLINK-17947:
-------------------------------------
Summary: Retry REST requests if RpcEndpoint died before responding
to request
Key: FLINK-17947
URL: https://issues.apache.org/jira/browse/FLINK-17947
Project: Flink
Issue Type: Improvement
Components: Runtime / REST
Affects Versions: 1.10.1, 1.11.0
Reporter: Till Rohrmann
Fix For: 1.12.0
Currently, it can happen that a REST handler sends a request to a leader
{{RpcEndpoint}} and before the {{RpcEndpoint}} has a chance to respond, it
might shut down (e.g. due to losing the leadership). In this case, the
{{ActorSystem}} will send an {{AskTimeoutException}} as the response with the
message {{Recipient Actor[akka://flink/user/rpc/dispatcher_1#-1875884516] had
already been terminated.}}. This exception will be treated as any other
exception and forwarded to the REST client. There it will be treated as a
normal timeout exception which causes the operation (e.g. requesting job
details) to fail.
I was wondering whether this case should not be handled slightly differently.
If the REST handler would respond with a {{SERVICE_UNAVAILABLE}}} HTTP response
code, then the {{RestClusterClient}} would retry the operation. One could think
of it as if there wouldn't have been a leader available before. This is similar
to the situation when there is no current leader and we are waiting for the
leader election to finish. Alternatively, we could extend the
{{RestClusterClient.isConnectionProblemOrServiceUnavailable}} predicate to also
cover the case of special {{AskTimeoutExceptions}}.
cc [~chesnay]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
