[
https://issues.apache.org/jira/browse/HBASE-28048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17759836#comment-17759836
]
Duo Zhang commented on HBASE-28048:
-----------------------------------
The key here is that, we are not safe to give up if we are not sure whether we
have already sent the procedure to the remote region server.
If we can make sure that the request does not reach the remote region server,
then we are safe to give up and try another region server.
You can see scheduleForRetry method for more details. We used to consider
connection exception in the past IIRC, for example, if we get a connection
refused exception, we can make sure that we have not sent the request to region
server. But in general, if we get a connection refused exception, usually this
means the region server is already dead, so soon we will go with the dead
server processing and solve the problem.
I think here, kerberos failure is the same with connection refused exception,
we can make sure that the region server has not received the rquest yet, so we
are safe to quit and try another region server.
> RSProcedureDispatcher to abort executing request after configurable retries
> ---------------------------------------------------------------------------
>
> Key: HBASE-28048
> URL: https://issues.apache.org/jira/browse/HBASE-28048
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0-alpha-4, 2.4.17, 2.5.5
> Reporter: Viraj Jasani
> Priority: Major
> Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1
>
>
> In a recent incident, we observed that RSProcedureDispatcher continues
> executing region open/close procedures with unbounded retries even in the
> presence of known failures like GSS initiate failure:
>
> {code:java}
> 2023-08-25 02:21:02,821 WARN [ispatcher-pool-40777]
> procedure.RSProcedureDispatcher - request to rs1,61020,1692930044498 failed
> due to java.io.IOException: Call to address=rs1:61020 failed on local
> exception: java.io.IOException:
> org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed, try=0, retrying... {code}
>
>
> If the remote execution results in IOException, the dispatcher attempts to
> schedule the procedure for further retries:
>
> {code:java}
> private boolean scheduleForRetry(IOException e) {
> LOG.debug("Request to {} failed, try={}", serverName,
> numberOfAttemptsSoFar, e);
> // Should we wait a little before retrying? If the server is starting
> it's yes.
> ...
> ...
> ...
> numberOfAttemptsSoFar++;
> // Add some backoff here as the attempts rise otherwise if a stuck
> condition, will fill logs
> // with failed attempts. None of our backoff classes -- RetryCounter or
> ClientBackoffPolicy
> // -- fit here nicely so just do something simple; increment by
> rsRpcRetryInterval millis *
> // retry^2 on each try
> // up to max of 10 seconds (don't want to back off too much in case of
> situation change).
> submitTask(this,
> Math.min(rsRpcRetryInterval * (this.numberOfAttemptsSoFar *
> this.numberOfAttemptsSoFar),
> 10 * 1000),
> TimeUnit.MILLISECONDS);
> return true;
> }
> {code}
>
>
> Even though we try to provide backoff while retrying, max wait time is 10s:
>
> {code:java}
> submitTask(this,
> Math.min(rsRpcRetryInterval * (this.numberOfAttemptsSoFar *
> this.numberOfAttemptsSoFar),
> 10 * 1000),
> TimeUnit.MILLISECONDS); {code}
>
>
> This results in endless loop of retries, until either the underlying issue is
> fixed (e.g. krb issue in this case) or regionserver is killed and the ongoing
> open/close region procedure (and perhaps entire SCP) for the affected
> regionserver is sidelined manually.
> {code:java}
> 2023-08-25 03:04:18,918 WARN [ispatcher-pool-41274]
> procedure.RSProcedureDispatcher - request to rs1,61020,1692930044498 failed
> due to java.io.IOException: Call to address=rs1:61020 failed on local
> exception: java.io.IOException:
> org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed, try=217, retrying...
> 2023-08-25 03:04:18,916 WARN [ispatcher-pool-41280]
> procedure.RSProcedureDispatcher - request to rs1,61020,1692930044498 failed
> due to java.io.IOException: Call to address=rs1:61020 failed on local
> exception: java.io.IOException:
> org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed, try=193, retrying...
> 2023-08-25 03:04:28,968 WARN [ispatcher-pool-41315]
> procedure.RSProcedureDispatcher - request to rs1,61020,1692930044498 failed
> due to java.io.IOException: Call to address=rs1:61020 failed on local
> exception: java.io.IOException:
> org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed, try=266, retrying...
> 2023-08-25 03:04:28,969 WARN [ispatcher-pool-41240]
> procedure.RSProcedureDispatcher - request to rs1,61020,1692930044498 failed
> due to java.io.IOException: Call to address=rs1:61020 failed on local
> exception: java.io.IOException:
> org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS
> initiate failed, try=266, retrying...{code}
>
> While external issues like "krb ticket expiry" requires operator
> intervention, it is not prudent to fill up the active handlers with endless
> retries while attempting to execute RPC on only single affected regionserver.
> This eventually leads to overall cluster state degradation, specifically in
> the event of multiple regionserver restarts resulting from any planned
> activities.
> One of the resolutions here would be:
> # Configure max retries as part of ExecuteProceduresRequest request (or it
> could be part of RemoteProcedureRequest)
> # This retry count should be used by RSProcedureDispatcher while scheduling
> request failures for further retries
> # After exhausting retries, mark the failure to the remote call, and bubble
> up the failure to parent procedure.
> If the series of above mentioned calls result into aborting active master, we
> should clearly log the FATAL/ERROR msg with the underlying root cause (e.g.
> GSS initiate failure in this case), which can help operator to either fix the
> krb ticket expiry or abort the regionserver, which would lead to SCP
> performing the heavy task of WAL splitting recoveries, however this would not
> prevent other procedures as well as active handlers from getting stuck
> executing remote calls without any conditional termination.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
