[
https://issues.apache.org/jira/browse/HBASE-8729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698628#comment-13698628
]
stack commented on HBASE-8729:
------------------------------
So we do the below:
{code}
+ boolean needReleaseLock = false;
+ if (!this.services.isInitialized()) {
+ // during master initialization, we could have multiple places splitting
a same wal
+ this.splitLogLock.lock();
+ needReleaseLock = true;
...
{code}
rather than ask if lock is 'locked' and unlock it if current thread matches the
lock 'owner'?
It is a little odd that the SSH adds a new executor. I suppose SSH puts itself
back on the executor queue if not read to proceed so maybe this ok.
I would be +1 on commit.
> distributedLogReplay may hang during chained region server failure
> ------------------------------------------------------------------
>
> Key: HBASE-8729
> URL: https://issues.apache.org/jira/browse/HBASE-8729
> Project: HBase
> Issue Type: Bug
> Components: MTTR
> Reporter: Jeffrey Zhong
> Assignee: Jeffrey Zhong
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 8729-v2.patch, hbase-8729.patch, hbase-8729-v3.patch,
> hbase-8729-v4.patch, hbase-8729-v5.patch
>
>
> In a test, half cluster(in terms of region servers) was down and some log
> replay had incurred chained RS failures(receiving RS of a log replay failed
> again).
> Since by default, we only allow 3 concurrent SSH handlers(controlled by
> {code}this.executorService.startExecutorService(ExecutorType.MASTER_SERVER_OPERATIONS,conf.getInt("hbase.master.executor.serverops.threads",
> 3));{code}).
> If all 3 SSH handlers are doing logReplay(blocking call) and one of receiving
> RS fails again then logReplay will hang because regions of the newly failed
> RS can't be re-assigned to another live RS(no ssh handler will be processed
> due to max threads setting) and existing log replay will keep routing replay
> traffic to the dead RS.
> The fix is to submit logReplay work into a separate type of executor queue in
> order not to block SSH region assignment so that logReplay can route traffic
> to a live RS after retries and move forward.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
