[
https://issues.apache.org/jira/browse/HBASE-26281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenglei updated HBASE-26281:
-----------------------------
Description:
{{DBB}} got from BucketCache would be freed unexpectedly before RPC completed
for following two cases:
* (1) Because of splitting or stream read，two or more read threads may try to
cache the same {{Block}} concurrently and one thread read all of the next block
header could replace the another one didn't(see HBASE-20447). When replacing
{{Block}}，the following {{BucketCache.WriterThread.putIntoBackingMap}} method
invokes {{BucketCache.blockEvicted}} in line 916,which free the {{BucketEntry}}
directly in line 544, not through the {{RefCount}}.
{code:java}
912 private void putIntoBackingMap(BlockCacheKey key, BucketEntry bucketEntry)
{
913 BucketEntry previousEntry = backingMap.put(key, bucketEntry);
914 if (previousEntry != null && previousEntry != bucketEntry) {
915 previousEntry.withWriteLock(offsetLock, () -> {
916 blockEvicted(key, previousEntry, false);
917 return null;
918 });
919 }
920 }
543 void blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean
decrementBlockNumber) {
544 bucketAllocator.freeBlock(bucketEntry.offset());
545 realCacheSize.add(-1 * bucketEntry.getLength());
546 blocksByHFile.remove(cacheKey);
547 if (decrementBlockNumber) {
548 this.blockNumber.decrement();
549 }
550 }
{code}
Freeing the {{BucketEntry}} directly may cause {{HFileBlock}} returned from
{{BucketCache.getBlock}} be freed unexpectedly before RPC completed.HBASE-26155
fixed this problem to some extent, but not completely. The
{{BucketCache.getBlock}} might be invoked just before
{{BucketCache.cacheBlockWithWaitInternal}} and after the
{{BucketEntry.isRpcRef}} checking, considering the following sequence:
1. Block1 was cached successfully,the {{RefCnt}} of Block1 is 1.
2. Thread1 caching the same {{BlockCacheKey}} with Block2 satisfied
{{BlockCacheUtil.shouldReplaceExistingCacheBlock}}, so Block2 would replace
Block1, but thread1 stopping before {{BucketCache.cacheBlockWithWaitInternal}}
3. Thread2 invoking {{BucketCache.getBlock}} with the same
{{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
4. Thread1 continues caching Block2, in
{{BucketCache.WriterThread.putIntoBackingMap}} , the old Block1 is freed
directly which {{RefCnt}} is 2. The Block1 is still used by Thread2 and the
content of Block1 would be overwitten after it is freed, which may cause a
serious error.
* (2) Evicting Block , caching Block and getting Block execute concurrently
considering the following sequence:
1. Thread1 caching Block1, stopping after
{{WriterThread.putIntoBackingMap}}, the {{RefCnt}} of Block1 is 1.
2. Thread2 invoking {{BucketCache.evictBlock}} with the same
{{BlockCacheKey}} , but stopping after
{{BucketCache.removeFromRamCache(BlockCacheKey)}}.
3. Thread3 invoking {{BucketCache.getBlock}} with the same
{{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
4. Thread1 continues caching block1,but finding that
{{BucketCache.RAMCache.remove}} returning false, so invoking
{{BucketCache.blockEvicted}} to free the the Block1 directly
which {{RefCnt}} is 2. The Block1 is still used by Thread3
was:
{{DBB}} got from BucketCache would be freed unexpectedly before RPC completed
for following two cases:
* 1. Because of splitting or stream read，two or more read threads may try to
cache the same {{Block}} concurrently and one thread read all of the next block
header could replace the another one didn't(see HBASE-20447). When replacing
{{Block}}，the following {{BucketCache.WriterThread.putIntoBackingMap}} method
invokes {{BucketCache.blockEvicted}} in line 916,which free the {{BucketEntry}}
directly in line 544, not through the {{RefCount}}.
{code:java}
912 private void putIntoBackingMap(BlockCacheKey key, BucketEntry bucketEntry)
{
913 BucketEntry previousEntry = backingMap.put(key, bucketEntry);
914 if (previousEntry != null && previousEntry != bucketEntry) {
915 previousEntry.withWriteLock(offsetLock, () -> {
916 blockEvicted(key, previousEntry, false);
917 return null;
918 });
919 }
920 }
543 void blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry, boolean
decrementBlockNumber) {
544 bucketAllocator.freeBlock(bucketEntry.offset());
545 realCacheSize.add(-1 * bucketEntry.getLength());
546 blocksByHFile.remove(cacheKey);
547 if (decrementBlockNumber) {
548 this.blockNumber.decrement();
549 }
550 }
{code}
Freeing the {{BucketEntry}} directly may cause {{HFileBlock}} returned from
{{BucketCache.getBlock}} be freed unexpectedly before RPC completed.HBASE-26155
fixed this problem to some extent, but not completely. The
{{BucketCache.getBlock}} might be invoked just before
{{BucketCache.cacheBlockWithWaitInternal}} and after the
{{BucketEntry.isRpcRef}} checking, considering the following sequence:
1. Block1 was cached successfully,the {{RefCnt}} of Block1 is 1.
2. Thread1 caching the same {{BlockCacheKey}} with Block2 satisfied
{{BlockCacheUtil.shouldReplaceExistingCacheBlock}}, so Block2 would replace
Block1, but thread1 stopping before {{BucketCache.cacheBlockWithWaitInternal}}
3. Thread2 invoking {{BucketCache.getBlock}} with the same
{{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
4.Thread1 continues caching Block2, in
{{BucketCache.WriterThread.putIntoBackingMap}} , the old Block1 is freed
directly which {{RefCnt}} is 2. The Block1 is still used by Thread2 and the
content of Block1 would be overwitten after it is freed, which may cause a
serious error.
* 2. Evicting Block , caching Block and getting Block execute concurrently
considering the following sequence:
1. Thread1 caching Block1, stopping after
{{WriterThread.putIntoBackingMap}}, the {{RefCnt}} of Block1 is 1.
2. Thread2 invoking {{BucketCache.evictBlock}} with the same
{{BlockCacheKey}} , but stopping after
{{BucketCache.removeFromRamCache(BlockCacheKey)}}.
3. Thread3 invoking {{BucketCache.getBlock}} with the same
{{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
4. Thread1 continues caching block1,but finding that
{{BucketCache.RAMCache.remove}} returning false, so invoking
{{BucketCache.blockEvicted}} to free the the Block1 directly which {{RefCnt}}
is 2. The Block1 is still used by Thread3
> DBB got from BucketCache would be freed unexpectedly before RPC completed
> -------------------------------------------------------------------------
>
> Key: HBASE-26281
> URL: https://issues.apache.org/jira/browse/HBASE-26281
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
> Affects Versions: 3.0.0-alpha-1, 2.4.6
> Reporter: chenglei
> Assignee: chenglei
> Priority: Critical
>
> {{DBB}} got from BucketCache would be freed unexpectedly before RPC completed
> for following two cases:
> * (1) Because of splitting or stream read，two or more read threads may try to
> cache the same {{Block}} concurrently and one thread read all of the next
> block header could replace the another one didn't(see HBASE-20447). When
> replacing {{Block}}，the following
> {{BucketCache.WriterThread.putIntoBackingMap}} method invokes
> {{BucketCache.blockEvicted}} in line 916,which free the {{BucketEntry}}
> directly in line 544, not through the {{RefCount}}.
> {code:java}
> 912 private void putIntoBackingMap(BlockCacheKey key, BucketEntry
> bucketEntry) {
> 913 BucketEntry previousEntry = backingMap.put(key, bucketEntry);
> 914 if (previousEntry != null && previousEntry != bucketEntry) {
> 915 previousEntry.withWriteLock(offsetLock, () -> {
> 916 blockEvicted(key, previousEntry, false);
> 917 return null;
> 918 });
> 919 }
> 920 }
> 543 void blockEvicted(BlockCacheKey cacheKey, BucketEntry bucketEntry,
> boolean decrementBlockNumber) {
> 544 bucketAllocator.freeBlock(bucketEntry.offset());
> 545 realCacheSize.add(-1 * bucketEntry.getLength());
> 546 blocksByHFile.remove(cacheKey);
> 547 if (decrementBlockNumber) {
> 548 this.blockNumber.decrement();
> 549 }
> 550 }
>
> {code}
> Freeing the {{BucketEntry}} directly may cause {{HFileBlock}} returned from
> {{BucketCache.getBlock}} be freed unexpectedly before RPC
> completed.HBASE-26155 fixed this problem to some extent, but not completely.
> The {{BucketCache.getBlock}} might be invoked just before
> {{BucketCache.cacheBlockWithWaitInternal}} and after the
> {{BucketEntry.isRpcRef}} checking, considering the following sequence:
> 1. Block1 was cached successfully,the {{RefCnt}} of Block1 is 1.
> 2. Thread1 caching the same {{BlockCacheKey}} with Block2 satisfied
> {{BlockCacheUtil.shouldReplaceExistingCacheBlock}}, so Block2 would replace
> Block1, but thread1 stopping before {{BucketCache.cacheBlockWithWaitInternal}}
> 3. Thread2 invoking {{BucketCache.getBlock}} with the same
> {{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
> 4. Thread1 continues caching Block2, in
> {{BucketCache.WriterThread.putIntoBackingMap}} , the old Block1 is freed
> directly which {{RefCnt}} is 2. The Block1 is still used by Thread2 and the
> content of Block1 would be overwitten after it is freed, which may cause a
> serious error.
> * (2) Evicting Block , caching Block and getting Block execute concurrently
> considering the following sequence:
> 1. Thread1 caching Block1, stopping after
> {{WriterThread.putIntoBackingMap}}, the {{RefCnt}} of Block1 is 1.
> 2. Thread2 invoking {{BucketCache.evictBlock}} with the same
> {{BlockCacheKey}} , but stopping after
> {{BucketCache.removeFromRamCache(BlockCacheKey)}}.
> 3. Thread3 invoking {{BucketCache.getBlock}} with the same
> {{BlockCacheKey}}, which returned Block1, the {{RefCnt}} of Block1 is 2.
> 4. Thread1 continues caching block1,but finding that
> {{BucketCache.RAMCache.remove}} returning false, so invoking
> {{BucketCache.blockEvicted}} to free the the Block1 directly
> which {{RefCnt}} is 2. The Block1 is still used by Thread3
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
