Caideyipi opened a new pull request, #18805:
URL: https://github.com/apache/iotdb/pull/18805

   ## Description
   
   Resubmit #18741 after its revert in #18786, on current `master` and with 
#18785's linger-batch flush retained.
   
   ### Why a subscription can no longer find WAL
   
   The incident showed the local replay cursor expecting search index 57 while 
the next visible WAL request was 113. The older retention calculation used 
committed subscription progress alone. A different consumer or replica could 
advance that committed progress while this queue still needed an earlier local 
search index; WAL cleanup could then remove a segment before this queue 
replayed it. #18769 already changed the retention bound to the earlier of 
committed progress and this queue's local replay cursor. This PR restores the 
replay-side guard so any remaining missing range is detected instead of 
silently skipped.
   
   An apparent gap can also be temporary when the active WAL has not become 
readable yet. On the first gap, the queue rolls the active WAL and recreates 
its iterator at the expected index. If the range is still unavailable on retry, 
it keeps the cursor at the missing index, stops delivery and real-time 
admission, marks progress as stalled, and returns a critical error to the 
consumer.
   
   ### Seek and #18785 compatibility
   
   - Preserve #18785's flush of an existing linger batch before a pending WAL 
gap retry.
   - Bind gap validation to the seek generation. A result from before a seek is 
treated as stale; applying a seek clears the old gap and stalled state.
   - Resolve `seekToBeginning` to the earliest replayable WAL entry. For seeks 
to the tail, continue after the last existing search index. This prevents an 
intentional seek from being reported as a missing WAL range.
   - Keep the last unread local WAL file protected by the retention boundary 
already present on `master`.
   
   ### Verification
   
   - `ConsensusPrefetchingQueueTest`: 44 tests passed (`surefire:test 
-DforkCount=0`), including WAL gap retry, critical failure, seek recovery, and 
the at-end cursor boundary.
   - Spotless and Checkstyle passed for DataNode.
   - The full English reactor passed through DataNode and later modules, then 
stopped in `distribution`: its snapshot ZIP dependencies could not be 
downloaded from the internal Nexus repository (HTTP 500).
   - Chinese locale reactor test compilation through DataNode and its upstream 
modules passed (`mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale 
-DskipTests`).
   
   This PR has:
   - [x] been self-reviewed, including the seek/gap interaction.
   - [x] added comments for the WAL retention and seek boundaries.
   - [x] added or updated unit tests for the changed paths.
   
   ##### Key changed/added classes
   
   - `ConsensusPrefetchingQueue`: WAL replay continuity, seek generation 
handling, and seek cursor normalization.
   - `ConsensusPrefetchingQueueTest`: gap retry, stalled delivery, seek 
recovery, and tail cursor regression coverage.
   - `DataNodePipeMessages` (en/zh): localized gap retry and critical error 
messages.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to