Zhiwen Deng created HBASE-30560:
-----------------------------------
Summary: WAL: add a minimum sync interval (sync linger) to avoid
per-drain hflush on high-QPS small-write workloads
Key: HBASE-30560
URL: https://issues.apache.org/jira/browse/HBASE-30560
Project: HBase
Issue Type: Improvement
Components: wal
Affects Versions: 3.0.0
Reporter: Zhiwen Deng
h3. Current state
WAL sync is decided in {{AbstractFSWAL#appendAndSync()}} / {{#consume()}}, by
two triggers only:
# siize: {{(writer.getLength() - fleLengthAtLastSync) >= hbase.wal.batch.size}}
(default 64KB, a byte threshold rather than an entry count);
# empty buffer: the consumer has drained the ring buffer and a SyncFuture is
pending -> issue {{sync(writer)}} immediately and stop scheduling the consumer
(see "sync because buffer size limit" in {{appendAndSync()}}).
Between the two, {{appendAndSync()}} deliberately defers sync while more
appends may arrive, so batching works as long as writes keep coming. The
empty-buffer path has no grace period.
h3. Problem
The empty-buffer trigger makes the sync rate independent of QPS but tightly
coupled to the hflush round trip: as soon as the ring buffer drains, a sync is
issued, so sync/s approaches 1/hflush-RTT (~5000/s on local NVMe pipelines) no
matter how few bytes each sync carries.
With single-row 1KB puts this is the dominant cost: one sync per drain, each
covering a few KB. The single WAL consumer thread ends up spending essentially
all of its time inside sync, leaving no headroom, so throughput saturates and
p99 write latency becomes unstable under pipeline jitter. Tuning
{{hbase.wal.batch.size}} does not help here: it only decides "enough bytes ->
sync now", and never sees the "not enough bytes but buffer empty" case.
h3. Proposed change
Add an opt-in minimum interval between two WAL syncs, default 0 (disabled,
current behaviour unchanged):
{{hbase.regionserver.wal.sync.linger.ms}}
When a sync would fire only because the ring buffer is empty, and the previous
sync finished less than N ms ago, postpone it. Size trigger, forceSync
requests, a pending roll and shutdown always bypass the linger.
Default 0 keeps behaviour byte-identical to today. The setting is a
latency/throughput trade: write p99 can grow by up to the linger value and the
SYNC_WAL durability window widens by the same amount, so it should stay opt-in
for latency-sensitive workloads.
h3. Result
With {{hbase.regionserver.wal.sync.linger.ms}} = 1, on a single RegionServer
(NVMe, 3 replicas), 100% single-row puts with one 1KB cell per RPC:
* throughput: 160,000 - 170,000 ops/s, single WAL;
* write p99: under 10 ms;
the sync rate is now bounded by the window instead of by hflush RTT, so the
consumer thread keeps headroom and stops being the limiting factor at high QPS.
I'm not sure whether this is the right approach for the WAL sync path - any
feedback would be welcome. If the direction looks reasonable, I'd be happy to
prepare and submit a patch for it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)