[
https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292646#comment-13292646
]
ramkrishna.s.vasudevan commented on HBASE-6060:
-----------------------------------------------
@Stack
Few comments,
First of all we need to do some more changes if we want to make the above
change in handleRegion
{code}
// Should see OPENING after we have asked it to OPEN or additional
// times after already being in state of OPENING
if (regionState == null ||
(!regionState.isPendingOpen() && !regionState.isOpening())) {
LOG.warn("Received OPENING for region " +
prettyPrintedRegionName +
" from server " + data.getOrigin() + " but region was in " +
" the state " + regionState + " and not " +
"in expected PENDING_OPEN or OPENING states");
{code}
Second is, you mean like the above change of making pending_open after the RPC
call along with the changes in the earlier patches like having the new items
lik 'ritintersection' and 'outstandingRegionPlans'?
Here there is one concern is, if before we update the state to PENDING_OPEN in
master side if the OpenRegionHandler has really transitioned the node to
OPENING then the inmemory state will also be changing to OPENING after we
correct the code above i have mentioned. Now in that case the OPENING state
will be rewritten to PENDING_OPEN? So we may need to add a check to see if
already a change has happened in the REgionState.
The problem with Rajesh's patch is
(6060_suggestion_toassign_rs_wentdown_beforerequest.patch:)
{code}
+ if ((region.getState() == RegionState.State.OFFLINE)
+ && (region.getState() == RegionState.State.PENDING_OPEN)) {
+ regionPlans.remove(region.getRegion());
+ }
{code}
What if the RS went down befor sending RPC. The SSH collected the regionPlan
but before he could remove the collected regionplan in the above code, if the
master completed the retry assignment and the OpenRegionHandler has changed to
OPENING and the state is now Opening, then we will again try to assign thro SSH.
The above problem can happen in the change that you mentioned also like moving
PENDING_OPEN after RPC.
So can we take a copy of RIT befor forming the RegionPlan and work based on
that? Will update the change that we are suggesting in some time.
> Regions's in OPENING state from failed regionservers takes a long time to
> recover
> ---------------------------------------------------------------------------------
>
> Key: HBASE-6060
> URL: https://issues.apache.org/jira/browse/HBASE-6060
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
> Reporter: Enis Soztutar
> Assignee: rajeshbabu
> Fix For: 0.96.0, 0.94.1, 0.92.3
>
> Attachments: 6060-94-v3.patch, 6060-94-v4.patch, 6060-94-v4_1.patch,
> 6060-94-v4_1.patch, 6060-trunk.patch, 6060-trunk.patch, 6060-trunk_2.patch,
> 6060-trunk_3.patch, 6060_alternative_suggestion.txt,
> 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch,
> 6060_suggestion_toassign_rs_wentdown_beforerequest.patch,
> HBASE-6060-92.patch, HBASE-6060-94.patch
>
>
> we have seen a pattern in tests, that the regions are stuck in OPENING state
> for a very long time when the region server who is opening the region fails.
> My understanding of the process:
>
> - master calls rs to open the region. If rs is offline, a new plan is
> generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in
> master memory, zk still shows OFFLINE). See HRegionServer.openRegion(),
> HMaster.assign()
> - RegionServer, starts opening a region, changes the state in znode. But
> that znode is not ephemeral. (see ZkAssign)
> - Rs transitions zk node from OFFLINE to OPENING. See
> OpenRegionHandler.process()
> - rs then opens the region, and changes znode from OPENING to OPENED
> - when rs is killed between OPENING and OPENED states, then zk shows OPENING
> state, and the master just waits for rs to change the region state, but since
> rs is down, that wont happen.
> - There is a AssignmentManager.TimeoutMonitor, which does exactly guard
> against these kind of conditions. It periodically checks (every 10 sec by
> default) the regions in transition to see whether they timedout
> (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min,
> which explains what you and I are seeing.
> - ServerShutdownHandler in Master does not reassign regions in OPENING
> state, although it handles other states.
> Lowering that threshold from the configuration is one option, but still I
> think we can do better.
> Will investigate more.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
