[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969211#comment-14969211
]
Yu Li commented on HBASE-14463:
-------------------------------
Checked multi get with --multiGet=100, and checked both 25 threads and 50
threads, results are as follows:
25threads:
{noformat}
w/o patch:
1. Min: 94303ms Max: 94853ms Avg: 94651ms
2. Min: 92547ms Max: 93140ms Avg: 92934ms
3. Min: 90562ms Max: 91374ms Avg: 91168ms
Average: 92917.67ms
w/ patch:
1. Min: 94026ms Max: 94631ms Avg: 94417ms
2. Min: 94086ms Max: 94850ms Avg: 94643ms
3. Min: 95041ms Max: 95585ms Avg: 95371ms
Average: 94810ms
with slow purge(lock pool size reach 5000):
1. Min: 94699ms Max: 95276ms Avg: 95031ms
2. Min: 97087ms Max: 98106ms Avg: 97871ms
3. Min: 93804ms Max: 94743ms Avg: 94519ms
Average: 95807ms
{noformat}
>From which we could see there's indeed a ~2% downgrade w/ patch, and slow
>purge (only purge lockPool when its size reaches 5000) didn't bring any
>improvement
However, with 50 threads:
{noformat}
w/o patch
1. Min: 195573ms Max: 199973ms Avg: 199016ms
2. Min: 191126ms Max: 194468ms Avg: 193592ms
3. Min: 195415ms Max: 198985ms Avg: 198133ms
Average: 196913.67
w/ patch:
1. Min: 199279ms Max: 202442ms Avg: 201607ms
2. Min: 191189ms Max: 194442ms Avg: 193626ms
3. Min: 190235ms Max: 193817ms Avg: 192996ms
Average: 196076.33ms
{noformat}
I got a better result w/ patch.
Now my suspicion is the different distribution of chosen keys makes different
result, so I'm planning to test the 2 cases (w/ and w/o patch) with exactly the
same randomly-generated keys and see what will happen
> Severe performance downgrade when parallel reading a single key from
> BucketCache
> --------------------------------------------------------------------------------
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.98.14, 1.1.2
> Reporter: Yu Li
> Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16
>
> Attachments: GC_with_WeakObjectPool.png, HBASE-14463.patch,
> HBASE-14463_v11.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch,
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch,
> TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
