[
https://issues.apache.org/jira/browse/HBASE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658616#action_12658616
]
Jim Kellerman commented on HBASE-543:
-------------------------------------
Replicating problems:
HBASE-1046: Region assigned to two regionservers after split
Running a slightly modified version of TestSplit (in which there are
two region servers instead of one) under the debugger (with lease and
client timeouts jacked up), I set a breakpoint in
ServerManager.processSplitRegion. When that was hit, I put a
breakpoint in ServerManager.processRegionOpen just before the call to
RegionManager.setPending. I also cleared the breakpoint in
processSplitRegion but did not resume the thread, yet.
During the period in which the thread stopped in processSplitRegion
was stalled the meta scanner ran, assigning the split regions.
The region server(s) assigned the split regions opened them and
reported back, stopping in processRegionOpen between the point where
the split regions we no longer unassigned and before they became
pending. (This works because there are multiple threads that handle
the region server report messages).
Continuing the thread that had been stopped in processSplitRegion,
it now slid through the window I had created by pausing the region
open reports between removing the region from the unassigned list but
before they were marked pending, and the regions were marked
unassigned again.
RegionManager.setPending has no synchronization, but that doesn't
really matter because the synchronization needs to happen at the level
where multiple Map(s)/Set(s) are being updated (i.e., the region's
state is kept in serveral places opening the possibility for race
conditions).
RegionManager itself could not be synchronized without adding new
methods that performed the multiple operations while synchronized
on each of the structures being modified.
HBASE-1051: Regionserver attempting to open reassigned region but
master ignoring, thinks region is still closing.
Using a method similar to that used for HBASE-1046 (with the test
being TestRegionRebalancing), I was found that there is a similar
window in which a region is marked "no longer closing" in
ServerManager.processRegionClose and the region is marked as "no
longer unassigned". If the meta scanner runs between these two points,
it will mark the region as unassigned, it could then get assigned, and
then be marked "no longer unassigned".
As a result, if the region server reports that it has opened the
region, ServerManager.processRegion open would interpret that
as a duplicate assignment, and tell the region server that just opened
the region to "close without report".
Since ProcessRegionClose does not delete the info:server field (nor
should it lest even worse race conditions ensue), the master still
sees a valid server when it scans the meta and will not attempt to
reassign the region.
> A region's state is kept in several places in the master opening the
> possibility for race conditions
> ----------------------------------------------------------------------------------------------------
>
> Key: HBASE-543
> URL: https://issues.apache.org/jira/browse/HBASE-543
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.1.0, 0.1.1, 0.2.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.19.0
>
> Attachments: 543.patch, 543.patch, recent-changes.patch,
> regionstate.txt
>
>
> A region's state exists in multiple maps in the RegionManager:
> unassignedRegions, pendingRegions, regionsToClose, closingRegions,
> regionsToDelete, etc.
> One of these race conditions was found in HBASE-534.
> For HBase-0.1.x, we should just patch the holes we find.
> The ultimate solution (which requires a lot of changes in HMaster) should be
> applied to HBase trunk.
> Proposed solution:
> Create a class that encapsulates a region's state and provide synchronized
> access to the class that validates state changes.
> There should be a single structure that holds regions in these transitional
> states and it should be a synchronized collection of some kind.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
