Ronald Macmaster created HBASE-30551:
----------------------------------------

             Summary: BucketCache: Move offsetLock ObjectPool purge off the 
read path
                 Key: HBASE-30551
                 URL: https://issues.apache.org/jira/browse/HBASE-30551
             Project: HBase
          Issue Type: Improvement
          Components: BucketCache
    Affects Versions: 2.6.7, 2.5.16, 3.0.0
         Environment: Discovered on HBase 2.5.11, but also appears unpatched on 
the 2.6 + 3.0 + master branches
            Reporter: Ronald Macmaster
            Assignee: Ronald Macmaster
         Attachments: after-patch-1.png, after-patch-2.png, after-patch-3.png, 
before-patch-zgc-1.png, profile1.png, profile2.png

While experimenting with JDK 21 and the new ZGC on the server side, we noticed 
a performance issue from the BucketCache. GC Threads can contend with the 
reader threads and significantly degrade P99 / Max Latencies.

Every [{{offsetLock.getLock()}} 
call|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLockWithObjectPool.java#L76-L81]
 on the read path runs {{{}ObjectPool.purge(){}}}, which [attempts cleanup on 
the pool's stale-reference 
queue|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L113-L128].
 Most of these attempts fall through the {{{}tryLock(){}}}, but one unlucky 
read req thread gets saddled with a large amount of reference queue cleanup.

This is a standard cleanup convention aligned with the JDK docs for 
[ReferenceQueues|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/package-summary.html].
 And it's a reasonable approach under old G1GC/CMS assumptions (stop-the-world 
JDK8-17).

However, it breaks down under ZGC. The GC threads clean up soft refs 
concurrently while app threads continue to run, and they continuously add more 
stale refs to the queue that the reader has paused to clean up. [The cleanup 
while loop is 
infinite|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L116-L123],
 so reader threads from the BucketCache livelock until ZGC finishes. It 
manifests as significantly worse tail-end performance (P99 / Max Get Latency).
{quote}While some programs will choose to dedicate a thread to removing 
reference objects from one or more queues and processing them, this is by no 
means necessary. A tactic that often works well is to examine a reference queue 
in the course of performing some other fairly-frequent action. For example, a 
hashtable that uses weak references to implement weak keys could poll its 
reference queue each time the table is accessed. This is how the WeakHashMap 
class works. Because the ReferenceQueue.poll method simply checks an internal 
data structure, this check will add little overhead to the hashtable access 
methods.
{quote}
We should consider moving the ObjectPool clean-up off the read path. The docs 
state a separate cleanup thread is also a valid alternative approach.

See the async-profiler and grafana attachments below for more details.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to