Ronald Macmaster created HBASE-30551:
----------------------------------------
Summary: BucketCache: Move offsetLock ObjectPool purge off the
read path
Key: HBASE-30551
URL: https://issues.apache.org/jira/browse/HBASE-30551
Project: HBase
Issue Type: Improvement
Components: BucketCache
Affects Versions: 2.6.7, 2.5.16, 3.0.0
Environment: Discovered on HBase 2.5.11, but also appears unpatched on
the 2.6 + 3.0 + master branches
Reporter: Ronald Macmaster
Assignee: Ronald Macmaster
Attachments: after-patch-1.png, after-patch-2.png, after-patch-3.png,
before-patch-zgc-1.png, profile1.png, profile2.png
While experimenting with JDK 21 and the new ZGC on the server side, we noticed
a performance issue from the BucketCache. GC Threads can contend with the
reader threads and significantly degrade P99 / Max Latencies.
Every [{{offsetLock.getLock()}}
call|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLockWithObjectPool.java#L76-L81]
on the read path runs {{{}ObjectPool.purge(){}}}, which [attempts cleanup on
the pool's stale-reference
queue|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L113-L128].
Most of these attempts fall through the {{{}tryLock(){}}}, but one unlucky
read req thread gets saddled with a large amount of reference queue cleanup.
This is a standard cleanup convention aligned with the JDK docs for
[ReferenceQueues|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/package-summary.html].
And it's a reasonable approach under old G1GC/CMS assumptions (stop-the-world
JDK8-17).
However, it breaks down under ZGC. The GC threads clean up soft refs
concurrently while app threads continue to run, and they continuously add more
stale refs to the queue that the reader has paused to clean up. [The cleanup
while loop is
infinite|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L116-L123],
so reader threads from the BucketCache livelock until ZGC finishes. It
manifests as significantly worse tail-end performance (P99 / Max Get Latency).
{quote}While some programs will choose to dedicate a thread to removing
reference objects from one or more queues and processing them, this is by no
means necessary. A tactic that often works well is to examine a reference queue
in the course of performing some other fairly-frequent action. For example, a
hashtable that uses weak references to implement weak keys could poll its
reference queue each time the table is accessed. This is how the WeakHashMap
class works. Because the ReferenceQueue.poll method simply checks an internal
data structure, this check will add little overhead to the hashtable access
methods.
{quote}
We should consider moving the ObjectPool clean-up off the read path. The docs
state a separate cleanup thread is also a valid alternative approach.
See the async-profiler and grafana attachments below for more details.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)