Caideyipi opened a new pull request, #18766:
URL: https://github.com/apache/iotdb/pull/18766

   ## Summary
   
   - Bound pending-entry WAL gap recovery by the configured subscription 
consensus per-round WAL entry budget.
   - Continue a large gap across later prefetch rounds instead of scanning it 
without an entry limit.
   - Add a regression test that verifies a two-entry budget advances the replay 
cursor across three rounds.
   
   ## Root cause
   
   After real-time pending delivery overflows, the next retained pending entry 
can be millions of search indexes ahead. The pending path called 
fillGapFromWAL, which passed Integer.MAX_VALUE to pumpFromSubscriptionWAL.
   
   For sparse partial topics, topic filtering happens after reading the region 
WAL. A single prefetch round could therefore scan millions of unrelated entries 
while holding the queue read lock. Queue close must acquire the write lock 
before detached cleanup starts, so unsubscribe can remain blocked and serialize 
later consumer-group procedures behind it.
   
   ## Field evidence
   
   The September 29, 2026 regression runs using build 48a60ca showed:
   
   - 27400: the full topic eventually consumed 8,622,154 of 8,622,154 rows, but 
P99 latency reached 1,133,871 ms and 3,661,849 real-time entries were dropped 
before WAL catch-up.
   - 27401: the partial topic consumed only 428 of 6,900 matching rows and 
logged 6,662,834 dropped real-time entries. DropSubscription, CreateConsumer, 
and DropConsumer procedures then remained stuck while subscription metadata 
synchronization was starved.
   - 27402: delivery stopped after 11,707,960 of 29,199,410 rows during 
concurrent DataNode internal RPC timeouts. That stop remains a separate 
investigation.
   
   ## Scope
   
   This change prevents pending-gap recovery from holding the queue lock for an 
unbounded WAL scan. It does not remove the region-level full WAL scan cost for 
sparse partial topics.
   
   ## Tests
   
   - mvn spotless:apply -pl iotdb-core/datanode
   - mvn -o test -pl iotdb-core/datanode 
-Dtest=ConsensusPrefetchingQueueTest#testPendingGapReplayHonorsPerRoundWalEntryLimit+testCloseDetachesBlockedPrefetchCleanup
   - Checkstyle: 0 violations
   - Spotless check: passed
   
   ## Baseline note
   
   
ConsensusPrefetchingQueueTest#testWalReplayCountsOnlyUnavailableSearchIndexes 
currently fails identically on unmodified origin/master with expected 2 but was 
0. The failure was reproduced with this commit stashed and is outside this 
change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to