Caideyipi commented on code in PR #18402:
URL: https://github.com/apache/iotdb/pull/18402#discussion_r3719269525
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -1597,9 +1597,20 @@ private boolean isBeforeLocalCursor(final
IndexedConsensusRequest request) {
return hasLocalSearchIndex(request) && request.getSearchIndex() <
nextExpectedSearchIndex.get();
}
- private void advanceLocalCursorIfPresent(final IndexedConsensusRequest
request) {
+ private boolean advanceLocalCursorIfPresent(final IndexedConsensusRequest
request) {
if (hasLocalSearchIndex(request)) {
nextExpectedSearchIndex.set(request.getSearchIndex() + 1);
+ return true;
+ }
+ return false;
+ }
+
+ private void advanceLocalCursorFromPendingIfPresent(
+ final IndexedConsensusRequest request, final long
expectedSeekGeneration) {
+ if (advanceLocalCursorIfPresent(request)) {
+ // The pending path advances independently of the WAL iterator. Defer
realignment until the
+ // next round so the current pending batch can finish without repeatedly
reopening the WAL.
+ requestSubscriptionWalReset(nextExpectedSearchIndex.get(),
expectedSeekGeneration);
Review Comment:
Could we clear or update this deferred reset when fillGapFromWAL realigns
the iterator? For example, with pending requests [1, 5], processing 1 records a
reset target of 2. Processing 5 then fills the WAL gap from 2 and may advance
both the iterator and nextExpectedSearchIndex to 6. Since request 5 is now
before the local cursor, this method is not called again, so the pending target
remains 2. The next prefetch round applies that stale target and
rereads/skips/decompresses WAL entries 2 through 5, recreating the replay cost
this PR is intended to remove. More generally, applying this reset at the start
of every following round also reconstructs ProgressWALIterator (including
listing and sorting retained WAL files) once per steady-state pending batch
even when that round never needs WAL. Please consider consuming the marker only
immediately before entering the WAL path, and clearing/updating it when
fillGapFromWAL has already aligned the iterator. A regression test with pending
[1
, 5] and WAL [2..5] would cover the stale-target case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
