Caideyipi opened a new pull request, #18844:
URL: https://github.com/apache/iotdb/pull/18844

   ## Description
   
   Consensus subscriptions can release a newer writer's records before an older 
replicated record that lies beyond the current read batch. With the default 
1024-record scan limit, 1024 records from writer 7 at physical time 2000 
followed by writer 8 at physical time 1000 can produce `7@2000 -> 8@1000`, even 
though both applied safe frontiers have reached 3000. A writer blocked by a 
safe frontier can also keep immediately rescheduling without delivering data.
   
   Pair each applied safe physical time with its covered writer sequence, and 
use that frontier only after the subscription reader consumes the sequence. 
Merge raw writer heads by `(physicalTime, nodeId, localSeq)` before converting 
Tablets, so the example delivers `8@1000 -> 7@2000`. Persistent WAL readers 
retain one reserved raw head per writer and share a per-round scan budget. This 
also allows ordering to progress with memory for only one Tablet, releases 
pending/WAL duplicates, and closes the additional readers on seek or 
deactivation. Confirmed corrupt WAL skips are shared between readers and 
reported once.
   
   When an idle writer's frontier blocks delivery, retry after 100 ms instead 
of spinning. Actual unfinished scans can still immediately reschedule, and the 
existing batch linger deadline remains effective. Independent writer readers 
require additional WAL scan passes for multiple writers; retained heads and 
per-round scans stay bounded.
   
   ## Validation
   
   - Added regressions for barrier scheduling, applied/read sequence 
separation, default pending batch boundaries, real V3 WAL boundaries, 
one-Tablet memory, delayed WAL visibility, duplicate suppression, corrupt-file 
coordination, deactivation, and repeated seek.
   - Reproduced the scheduling and default-batch ordering failures against the 
original `origin/master` queue.
   - `mvn test -pl iotdb-core/consensus,iotdb-core/datanode 
-Dtest=WriterSafeFrontierTrackerTest,ConsensusPrefetchingQueueWriterOrderingTest,ConsensusPrefetchingQueueDataNodeMemoryTest,ConsensusPrefetchingQueueSeekTest,ConsensusPrefetchingQueueWalBackpressureTest,ConsensusPrefetchingQueueTest,ProgressWALIteratorTest
 -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false`: passed, 92 
tests in total, 91 passed and one existing opt-in performance benchmark skipped.
   - `mvn test-compile -DskipTests -Dmaven.build.cache.enabled=false 
-Ddevelocity.off=true`: full English reactor passed.
   - `mvn test-compile -DskipTests -P with-zh-locale 
-Dmaven.build.cache.enabled=false -Ddevelocity.off=true`: full Chinese reactor 
passed.
   - Spotless and `git diff --check`: passed.
   
   For local build setup, installed the server, CLI, and ConfigNode ZIP 
artifacts needed by distribution. That prerequisite packaging run used 
`-Drat.skip=true` because RAT rejects five existing generated Go files in the 
client submodule. Both reactor compiles and targeted tests passed without that 
skip flag.
   
   This PR has:
   
   - [x] been self-reviewed, including concurrent admission, lifecycle cleanup, 
and seek fencing.
   - [x] added comments explaining the ordering and memory decisions.
   - [x] added unit tests and updated existing tests for the changed behavior.
   
   Key changed classes: `ConsensusPrefetchingQueue`, `ProgressWALIterator`, and 
`WriterSafeFrontierTracker`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to