[ https://issues.apache.org/jira/browse/HBASE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840955#comment-16840955 ]
Zheng Hu commented on HBASE-22422: ---------------------------------- Another risk is here, LruBlockCache#evictBlock, we should move the previous.getBuffer().release() to the last line before return because once the release decrease the refCnt to zero then nobody can access the buf (such as victimHandler). {code} protected long evictBlock(LruCachedBlock block, boolean evictedByEvictionProcess) { LruCachedBlock previous = map.remove(block.getCacheKey()); if (previous == null) { return 0; } // Decrease the block's reference count, and if refCount is 0, then it'll auto-deallocate. previous.getBuffer().release(); updateSizeMetrics(block, true); long val = elements.decrementAndGet(); if (LOG.isTraceEnabled()) { long size = map.size(); assertCounterSanity(size, val); } if (block.getBuffer().getBlockType().isData()) { dataBlockElements.decrement(); } if (evictedByEvictionProcess) { // When the eviction of the block happened because of invalidation of HFiles, no need to // update the stats counter. stats.evicted(block.getCachedTime(), block.getCacheKey().isPrimary()); if (victimHandler != null) { victimHandler.cacheBlock(block.getCacheKey(), block.getBuffer()); } } return block.heapSize(); } {code} > Retain an ByteBuff with refCnt=0 when getBlock from LRUCache > ------------------------------------------------------------ > > Key: HBASE-22422 > URL: https://issues.apache.org/jira/browse/HBASE-22422 > Project: HBase > Issue Type: Sub-task > Components: BlockCache > Reporter: Zheng Hu > Assignee: Zheng Hu > Priority: Major > Attachments: LRUBlockCache-getBlock.png, > image-2019-05-15-12-00-03-641.png > > > After runing YCSB scan/get benchmark in our XiaoMi cluster, we found the get > QPS dropped from 25000/s to hunderds per second in a cluster with five > nodes. > After enable the debug log at YCSB client side, I found the following > stacktrace , see > https://issues.apache.org/jira/secure/attachment/12968745/image-2019-05-15-12-00-03-641.png > Need a patch to fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)