[
https://issues.apache.org/jira/browse/HBASE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844460#comment-16844460
]
Zheng Hu commented on HBASE-22422:
----------------------------------
I've tried to check all the code and made some patch to ensure the bug, but
seems it did not work. So I write a simple patch to dump all the release
caller's stack trace if any IllegalReferenceCountException happen when retain.
Let see what it will say.
> Retain an ByteBuff with refCnt=0 when getBlock from LRUCache
> ------------------------------------------------------------
>
> Key: HBASE-22422
> URL: https://issues.apache.org/jira/browse/HBASE-22422
> Project: HBase
> Issue Type: Sub-task
> Components: BlockCache
> Reporter: Zheng Hu
> Assignee: Zheng Hu
> Priority: Major
> Attachments: HBASE-22422.HBASE-21879.v1.patch,
> LRUBlockCache-getBlock.png, debug.patch,
> failed-to-check-positive-on-web-ui.png, image-2019-05-15-12-00-03-641.png
>
>
> After runing YCSB scan/get benchmark in our XiaoMi cluster, we found the get
> QPS dropped from 25000/s to hunderds per second in a cluster with five
> nodes.
> After enable the debug log at YCSB client side, I found the following
> stacktrace , see
> https://issues.apache.org/jira/secure/attachment/12968745/image-2019-05-15-12-00-03-641.png.
>
> After looking into the stractrace, I can ensure that the zero refCnt block is
> an intermedia index block, see [2] http://hbase.apache.org/images/hfilev2.png
> Need a patch to fix this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
