[
https://issues.apache.org/jira/browse/HBASE-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated HBASE-26281:
-----------------------------
Description:
{{DBB}} got from BucketCache would be freed unexpectedly before RPC completed
for following two cases:
# Because of splitting or stream read，two or more read threads may try to cache
the same {{Block}} concurrently and one thread read all of the next block
header could replace the another one didn't(see HBASE-20447). When replacing
{{Block}}，the following {{BucketCache.WriterThread.putIntoBackingMap}} method
invokes {{BucketCache.blockEvicted}} in line 916,which free the {{BucketEntry}}
directly in line 544, not through the {{RefCount}}.
{code:java}
912 private void putIntoBackingMap(BlockCacheKey key, BucketEntry bucketEntry)
{
913 BucketEntry previousEntry = backingMap.put(key, bucketEntry);
914 if (previousEntry != null && previousEntry != bucketEntry) {
915 previousEntry.withWriteLock(offsetLock, () -> {
916 blockEvicted(key, previousEntry, false);
917 return null;
918 });
919 }
920 }
543 void blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean
decrementBlockNumber) {
544 bucketAllocator.freeBlock(bucketEntry.offset());
545 realCacheSize.add(-1 * bucketEntry.getLength());
546 blocksByHFile.remove(cacheKey);
547 if (decrementBlockNumber) {
548 this.blockNumber.decrement();
549 }
550 }
{code}
Freeing the {{BucketEntry}} directly may cause {{HFileBlock}} returned from
{{BucketCache.getBlock}} be freed unexpectedly before RPC completed.HBASE-26155
fixed this problem to some extent, but not completely. The
{{BucketCache.getBlock}} might be invoked just before
{{BucketCache.cacheBlockWithWaitInternal}} and after the
{{BucketEntry.isRpcRef}} checking, considering the following sequence:
1. Block1 was cached successfully,the {{RefCnt}} of Block1 is 1.
2.Thread1 caching the same {{BlockCacheKey}} with Block2 satisfied
{{BlockCacheUtil.shouldReplaceExistingCacheBlock}}, so Block2 would replace
Block1, but thread1 stopped before {{BucketCache.cacheBlockWithWaitInternal}}
3. Thread2 invoking {{BucketCache.getBlock}} with the same
{{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
4. Thread1 continues caching Block2, in
{{BucketCache.WriterThread.putIntoBackingMap}} , the old Block1 is freed
directly which {{RefCnt}} is 2. The Block1 is still used by Thread2 and the
content of Block1 would be overwitten after it is freed, which may cause a
serious error.
was:
{{DBB}} got from BucketCache would be freed unexpectedly before RPC completed
for following two cases:
# Because of splitting or stream read，two or more read threads may try to cache
the same block concurrently and one thread read all of the next block header
could replace the another one didn't(see HBASE-20447). When replacing
{{Block}}，the following {{BucketCache.WriterThread.putIntoBackingMap}} method
invokes {{BucketCache.blockEvicted}},which free
the {{BucketEntry}} directly in line 544, not through the {{RefCount}}
{code:java}
912 private void putIntoBackingMap(BlockCacheKey key, BucketEntry bucketEntry)
{
913 BucketEntry previousEntry = backingMap.put(key, bucketEntry);
914 if (previousEntry != null && previousEntry != bucketEntry) {
915 previousEntry.withWriteLock(offsetLock, () -> {
blockEvicted(key, previousEntry, false);
return null;
});
}
}
543 void blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean
decrementBlockNumber) {
544 bucketAllocator.freeBlock(bucketEntry.offset());
545 realCacheSize.add(-1 * bucketEntry.getLength());
546 blocksByHFile.remove(cacheKey);
547 if (decrementBlockNumber) {
548 this.blockNumber.decrement();
549 }
550 }
{code}
> DBB got from BucketCache would be freed unexpectedly before RPC completed
> -------------------------------------------------------------------------
>
> Key: HBASE-26281
> URL: https://issues.apache.org/jira/browse/HBASE-26281
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
> Affects Versions: 3.0.0-alpha-1, 2.4.6
> Reporter: chenglei
> Priority: Critical
>
> {{DBB}} got from BucketCache would be freed unexpectedly before RPC completed
> for following two cases:
> # Because of splitting or stream read，two or more read threads may try to
> cache the same {{Block}} concurrently and one thread read all of the next
> block header could replace the another one didn't(see HBASE-20447). When
> replacing {{Block}}，the following
> {{BucketCache.WriterThread.putIntoBackingMap}} method invokes
> {{BucketCache.blockEvicted}} in line 916,which free the {{BucketEntry}}
> directly in line 544, not through the {{RefCount}}.
> {code:java}
> 912 private void putIntoBackingMap(BlockCacheKey key, BucketEntry
> bucketEntry) {
> 913 BucketEntry previousEntry = backingMap.put(key, bucketEntry);
> 914 if (previousEntry != null && previousEntry != bucketEntry) {
> 915 previousEntry.withWriteLock(offsetLock, () -> {
> 916 blockEvicted(key, previousEntry, false);
> 917 return null;
> 918 });
> 919 }
> 920 }
> 543 void blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry,
> boolean decrementBlockNumber) {
> 544 bucketAllocator.freeBlock(bucketEntry.offset());
> 545 realCacheSize.add(-1 * bucketEntry.getLength());
> 546 blocksByHFile.remove(cacheKey);
> 547 if (decrementBlockNumber) {
> 548 this.blockNumber.decrement();
> 549 }
> 550 }
>
> {code}
> Freeing the {{BucketEntry}} directly may cause {{HFileBlock}} returned from
> {{BucketCache.getBlock}} be freed unexpectedly before RPC
> completed.HBASE-26155 fixed this problem to some extent, but not completely.
> The {{BucketCache.getBlock}} might be invoked just before
> {{BucketCache.cacheBlockWithWaitInternal}} and after the
> {{BucketEntry.isRpcRef}} checking, considering the following sequence:
> 1. Block1 was cached successfully,the {{RefCnt}} of Block1 is 1.
> 2.Thread1 caching the same {{BlockCacheKey}} with Block2 satisfied
> {{BlockCacheUtil.shouldReplaceExistingCacheBlock}}, so Block2 would replace
> Block1, but thread1 stopped before {{BucketCache.cacheBlockWithWaitInternal}}
> 3. Thread2 invoking {{BucketCache.getBlock}} with the same
> {{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
> 4. Thread1 continues caching Block2, in
> {{BucketCache.WriterThread.putIntoBackingMap}} , the old Block1 is freed
> directly which {{RefCnt}} is 2. The Block1 is still used by Thread2 and the
> content of Block1 would be overwitten after it is freed, which may cause a
> serious error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
