Caideyipi opened a new pull request, #18402:
URL: https://github.com/apache/iotdb/pull/18402
## Description
This PR removes two coupled bottlenecks in consensus subscription
prefetching:
1. Reuse the subscription WAL iterator across prefetch rounds instead of
reopening it at the current search index on every round.
2. Let late/duplicate ACK handling share the queue read lock with
prefetching instead of waiting for the queue write lock.
### Bottleneck shown by the flame graphs
The four CPU/wall profiles show one server-side bottleneck and its
client-side amplification.
On the DataNode CPU profile:
- Consensus subscription prefetch accounts for about 79.3% of CPU.
- The WAL input path under DataInputStream.readFully accounts for about
60.4%.
- LZ4_decompress_safe self time is about 32.3%.
- openReaderAtIndex is about 33.5% cumulative and skipEntries is about 16.2%.
- Actual low-level disk reads are only a few percent.
This is repeated replay work rather than raw disk throughput.
tryCatchUpFromWAL() previously reset the iterator before every batch. A
preceding hasNext() could already have parsed and cached nextReady, but the
next round closed that iterator, discarded the cached request, reopened the WAL
at nextExpectedSearchIndex, and repeated reader lookup, skipping, reads, and
LZ4 decompression.
On the DataNode wall profile, the same expensive prefetch round holds the
queue read lock. A late/missing ACK then waits for the queue write lock for
roughly 86% of its RPC wall time.
On the consumer wall profile, the auto-commit RPC waits for that ACK
response while holding the consumer SynchronizedHandler monitor. A concurrent
business thread calling commitSync waits for the same monitor for roughly 48%
of consumer wall time. Thus enabling auto-commit together with manual
commitSync amplifies the server stall, but the server-side WAL replay and ACK
lock contention are the root causes addressed here.
The consumer CPU profile also shows Tablet.deserialize at about 49% and
Tablet.readValuesFromBuffer at about 40%. This PR does not change tablet
deserialization; the wall profiles show that the larger end-to-end stall is the
ACK/WAL contention chain above.
### Design
- Keep ProgressWALIterator and its hasNext() cache alive across normal WAL
catch-up rounds. refresh() discovers newly visible WAL files without rewinding
already consumed data.
- Request an explicit deferred iterator realignment when the realtime
pending path advances the local cursor independently. Seek, gap recovery,
memory retry, close, and lifecycle transitions retain their explicit reset
behavior.
- While a reset is pending, scheduling reports immediate work without
calling hasNext() on the stale iterator. The next prefetch round applies the
reset under the queue lock.
- Handle late/missing ACKs under the queue read lock. The touched queue/map
indexes are concurrent containers, event cleanup is idempotently protected by
the event monitor, and commit-state updates are serialized by the commit-state
monitor. The read lock still fences seek/close and other lifecycle transitions
that require the queue write lock.
### Performance test
ProgressWALIteratorTest#testIteratorReusePerformance is disabled by default
and can be enabled manually:
~~~bash
mvn test -pl iotdb-core/datanode \
-Dtest=ProgressWALIteratorTest#testIteratorReusePerformance \
-Diotdb.test.subscription.performance=true
~~~
Optional parameters:
- -Diotdb.test.subscription.performance.entries=<count>
- -Diotdb.test.subscription.performance.batch-size=<count>
With 4,096 entries and batches of 64 on the same generated WAL:
~~~text
reopen=647.872 ms, reuse=18.445 ms, speedup=35.12x
~~~
A final repeat after recompilation produced:
~~~text
reopen=606.675 ms, reuse=16.032 ms, speedup=37.84x
~~~
The benchmark isolates the repeated reopen/skip/decompress behavior from
consumer RPC and tablet deserialization.
### Tests
- ConsensusPrefetchingQueueTest#testLateAckDoesNotWaitForPrefetchReadLock
- ConsensusPrefetchingQueueTest#testWalCatchUpReusesIteratorAcrossRounds
-
ConsensusPrefetchingQueueTest#testPendingCursorAdvanceDefersWalIteratorRealignment
- Full ConsensusPrefetchingQueueTest and ProgressWALIteratorTest: 31 tests,
0 failures, 0 errors, 1 performance test skipped by default
- Spotless and Checkstyle passed
<hr>
This PR has:
- [x] been self-reviewed.
- [x] concurrent read
- [x] concurrent write
- [x] concurrent read and write
- [x] added comments explaining the why and the intent of the code wherever
it would not be obvious for an unfamiliar reader.
- [x] added unit tests to cover the new code paths.
<hr>
##### Key changed/added classes
- ConsensusPrefetchingQueue
- ConsensusPrefetchingQueueTest
- ProgressWALIteratorTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
