[
https://issues.apache.org/jira/browse/HBASE-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-3894:
-------------------------------
Attachment: concurrentRowLocks.patch
Here's a patch I've been playing with.
In the existing code each time any lock is unlocked, all 20 threads trying to
acquire a lock wake up, contend for the monitor, and have to check for their
lock in that TreeSet (15 byte[] comparisons), whether or not their particular
row was unlocked.
This patch replaces the set with a concurrent hash map. In order to use it, we
must wrap the byte array in another object that gives it a hash identity based
on its contents rather than its instance. However, every row lock is already
creating a couple objects (the Integer lockId, as well as the tree node), so
the object creation overhead is worth it.
The patch also only awakens threads when their particular row is unlocked.
Some further considerations:
On release, should we throw an exception if the client attempts to release
a lock id that doesn't exist, or just log it?
Do we really need to generate lock ids? Can we trust HBase client
implementations to not allow arbitrary lock releases? Or if not, for locks that
are only acquired / released internally to the regionserver, we should be able
to trust that code to use the row key rather than need to generate another lock
id
When an HRegion is doing a miniBatch of thousands of rows, is it really
best to attempt to acquire thousands of locks and hold them all while doing the
write? This one is probably a separate JIRA.
This patch has not yet been tested, but I wanted to put it up for discussion
since other people are looking at similar issues.
> Thread contention over row locks set monitor
> --------------------------------------------
>
> Key: HBASE-3894
> URL: https://issues.apache.org/jira/browse/HBASE-3894
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.2
> Reporter: Dave Latham
> Priority: Blocker
> Fix For: 0.90.4
>
>
> HRegion maintains a set of row locks. Whenever any thread attempts to lock
> or release a row it needs to acquire the monitor on that set. We've been
> encountering cases with 30 handler threads all contending for that monitor,
> blocked progress on the region server. Clients timeout, and retry making it
> worse, and the region server stops responding to new clients almost entirely.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
