[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909617#comment-14909617
]
Yu Li commented on HBASE-14463:
-------------------------------
I assume the "lock" mentioned above means the "lockOnMap", right? Notice that
it's the lock on the map recording entries for each block offset in the
bucketcache, not the lock inside the entry, and we already use different
ReentrantReadWriteLocks for different entries.
Regarding the lockOnMap, yes you are right that read and write on the map will
block each other. However, since cache is a "one write many read" thing, I
think the read/write contention is limited. We could see this as a trade off: a
little bit more cost when read/write on same block happens but lots of
improvement for the parallel read which happens more frequently.
The lockOnMap is necessary to prevent the lock leak issue you and Anoop
mentioned previously. Actually I thought about removing the block-offset->entry
map by maintaining an array of locks for each offset during the buckets
instantiation, however, I found this design not that efficient since an initial
64k bucket might change to 128k bucket later if more 128k blocks in need. So
I'm afraid we have to take the trade-off here.
Performance of the exiting code is mentioned in my previous comments, so allow
me to simply quote here:
{quote}
Time to run TestBucketCache#testCacheMultiThreadedSingleKey with the new
implementation compared with old ones (also attached the JUnit screenshot):
w/ IdLock with blocksize=16384: 19.676s
w/ IdReadWriteLock-old with blocksize=16384: 0.172s
w/ IdReadWriteLock-new with blocksize=16384: 0.318s
{quote}
The current TestBucketCache#testCacheMultiThreadedSingleKey case only includes
read operation on single key, but after all the discussion here, I'd like to
change it adding some write operations, and make sure write less than read.
[~jingchengdu] and [~ikeda], feel free to let me know your thoughts. :-)
> Severe performance downgrade when parallel reading a single key from
> BucketCache
> --------------------------------------------------------------------------------
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.98.14, 1.1.2
> Reporter: Yu Li
> Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14463.patch, HBASE-14463_v2.patch,
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
