[
https://issues.apache.org/jira/browse/HBASE-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Daniel Cryans updated HBASE-3789:
--------------------------------------
Attachment: HBASE-3789.patch
This first patch is a dirty work in progress.
First thing I changed is in ZKW I used a ConcurrentSkipListSet instead of a
HashSet which resulted in removing all the weird locking in ZKAssign.
Next up is AssignmentManager where I removed all the sync done in
nodeCreated/DataChanged/ChildrenChanged since it was already handled inside
handleRegion(). Also it is doing ZK operations under that sync so it's double
bad.
Third I changed RegionState so that the stamp is atomically modifiable since it
doesn't really matter that both the state and the stamp be changed by exactly
the same person, what you want in the end is progress. This was also the source
of a lot of locking in updateTimers.
I tested this patch under load while killing region servers (multiple at a
time), and then running the balancer. Didn't a single bug. Still needs more
testing and need to document the locking and see if there's any other
optimization that could be done.
At least, my profiling shows that the master is now only waiting on ZK.
> Cleanup the locking contention in the master
> --------------------------------------------
>
> Key: HBASE-3789
> URL: https://issues.apache.org/jira/browse/HBASE-3789
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.2
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBASE-3789.patch
>
>
> The new master uses a lot of synchronized blocks to be safe, but it only
> takes a few jstacks to see that there's multiple layers of lock contention
> when a bunch of regions are moving (like when the balancer runs). The main
> culprits are regionInTransition in AssignmentManager, ZKAssign that uses
> ZKW.getZNnodes (basically another set of region in transitions), and locking
> at the RegionState level.
> My understanding is that even tho we have multiple threads to handle regions
> in transition, everything is actually serialized. Most of the time, lock
> holders are talking to ZK or a region server, which can take a few
> milliseconds.
> A simple example is when AssignmentManager wants to update the timers for all
> the regions on a RS, it will usually be waiting on another thread that's
> holding the lock while talking to ZK.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
