[
https://issues.apache.org/jira/browse/HBASE-30551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148301#comment-18148301
]
Ronald Macmaster commented on HBASE-30551:
------------------------------------------
thanks for the merge [~wchevreuil]. got another PR opened for branch-2 with
minor functional changes (patches the janitor thread to work with the previous
IdReadWriteLock class) https://github.com/apache/hbase/pull/8919
I don't think 2.5 cherry picks cleanly either, but this should cherry pick
cleanly for 2.6! let me know if you would like one more PR to help resolve the
minor merge conflict for 2.5
> BucketCache: Move offsetLock ObjectPool purge off the read path
> ---------------------------------------------------------------
>
> Key: HBASE-30551
> URL: https://issues.apache.org/jira/browse/HBASE-30551
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
> Affects Versions: 3.0.0, 2.5.16, 2.6.7
> Environment: Discovered on HBase 2.5.11, but also appears unpatched
> on the 2.6 + 3.0 + master branches
> Reporter: Ronald Macmaster
> Assignee: Ronald Macmaster
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 3.0.1
>
> Attachments: after-patch-1.png, after-patch-2.png, after-patch-3.png,
> before-patch-zgc-1.png, orig-2.5.patch, profile1.png, profile2.png
>
>
> While experimenting with JDK 21 and the new ZGC on the server side, we
> noticed a performance issue from the BucketCache. GC Threads can contend with
> the reader threads and significantly degrade P99 / Max Latencies.
> Every [{{offsetLock.getLock()}}
> call|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLockWithObjectPool.java#L76-L81]
> on the read path runs {{{}ObjectPool.purge(){}}}, which [attempts cleanup on
> the pool's stale-reference
> queue|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L113-L128].
> Most of these attempts fall through the {{{}tryLock(){}}}, but one unlucky
> read req thread gets saddled with a large amount of reference queue cleanup.
> This is a standard cleanup convention aligned with the JDK docs for
> [ReferenceQueues|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/package-summary.html].
> And it's a reasonable approach under old G1GC/CMS assumptions
> (stop-the-world JDK8-17).
> However, it breaks down under ZGC. The GC threads clean up soft refs
> concurrently while app threads continue to run, and they continuously add
> more stale refs to the queue that the reader has paused to clean up. [The
> cleanup while loop is
> infinite|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L116-L123],
> so reader threads from the BucketCache livelock until ZGC finishes. It
> manifests as significantly worse tail-end performance (P99 / Max Get Latency).
> {quote}While some programs will choose to dedicate a thread to removing
> reference objects from one or more queues and processing them, this is by no
> means necessary. A tactic that often works well is to examine a reference
> queue in the course of performing some other fairly-frequent action. For
> example, a hashtable that uses weak references to implement weak keys could
> poll its reference queue each time the table is accessed. This is how the
> WeakHashMap class works. Because the ReferenceQueue.poll method simply checks
> an internal data structure, this check will add little overhead to the
> hashtable access methods.
> {quote}
> We should consider moving the ObjectPool clean-up off the read path. The docs
> state a separate cleanup thread is also a valid alternative approach.
> See the async-profiler and grafana attachments below for more details.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)