Caideyipi opened a new pull request, #18844: URL: https://github.com/apache/iotdb/pull/18844
## Description Consensus subscriptions can release a newer writer's records before an older replicated record that lies beyond the current read batch. With the default 1024-record scan limit, 1024 records from writer 7 at physical time 2000 followed by writer 8 at physical time 1000 can produce `7@2000 -> 8@1000`, even though both applied safe frontiers have reached 3000. A writer blocked by a safe frontier can also keep immediately rescheduling without delivering data. Pair each applied safe physical time with its covered writer sequence, and use that frontier only after the subscription reader consumes the sequence. Merge raw writer heads by `(physicalTime, nodeId, localSeq)` before converting Tablets, so the example delivers `8@1000 -> 7@2000`. Persistent WAL readers retain one reserved raw head per writer and share a per-round scan budget. This also allows ordering to progress with memory for only one Tablet, releases pending/WAL duplicates, and closes the additional readers on seek or deactivation. Confirmed corrupt WAL skips are shared between readers and reported once. When an idle writer's frontier blocks delivery, retry after 100 ms instead of spinning. Actual unfinished scans can still immediately reschedule, and the existing batch linger deadline remains effective. Independent writer readers require additional WAL scan passes for multiple writers; retained heads and per-round scans stay bounded. ## Validation - Added regressions for barrier scheduling, applied/read sequence separation, default pending batch boundaries, real V3 WAL boundaries, one-Tablet memory, delayed WAL visibility, duplicate suppression, corrupt-file coordination, deactivation, and repeated seek. - Reproduced the scheduling and default-batch ordering failures against the original `origin/master` queue. - `mvn test -pl iotdb-core/consensus,iotdb-core/datanode -Dtest=WriterSafeFrontierTrackerTest,ConsensusPrefetchingQueueWriterOrderingTest,ConsensusPrefetchingQueueDataNodeMemoryTest,ConsensusPrefetchingQueueSeekTest,ConsensusPrefetchingQueueWalBackpressureTest,ConsensusPrefetchingQueueTest,ProgressWALIteratorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false`: passed, 92 tests in total, 91 passed and one existing opt-in performance benchmark skipped. - `mvn test-compile -DskipTests -Dmaven.build.cache.enabled=false -Ddevelocity.off=true`: full English reactor passed. - `mvn test-compile -DskipTests -P with-zh-locale -Dmaven.build.cache.enabled=false -Ddevelocity.off=true`: full Chinese reactor passed. - Spotless and `git diff --check`: passed. For local build setup, installed the server, CLI, and ConfigNode ZIP artifacts needed by distribution. That prerequisite packaging run used `-Drat.skip=true` because RAT rejects five existing generated Go files in the client submodule. Both reactor compiles and targeted tests passed without that skip flag. This PR has: - [x] been self-reviewed, including concurrent admission, lifecycle cleanup, and seek fencing. - [x] added comments explaining the ordering and memory decisions. - [x] added unit tests and updated existing tests for the changed behavior. Key changed classes: `ConsensusPrefetchingQueue`, `ProgressWALIterator`, and `WriterSafeFrontierTracker`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
