Caideyipi opened a new pull request, #18866:
URL: https://github.com/apache/iotdb/pull/18866

   ## Description
   
   Delayed manual ACKs currently keep every delivered consensus tablet batch 
materialized, so a consumer can exhaust the broker's tablet budget before it 
finishes a transaction, batch, or checkpoint. This change releases a delivered 
batch's tablets and response cache after serialization once every original 
member and WAL seal is readable, while retaining its commit context and a 
compact delivery record. Releasing the payload does not advance committed 
progress.
   
   ### Delivery and replay
   
   - Record exact batch members using delta/zigzag varints, together with the 
original batch shape, memory charge, and frozen column projection. Redelivery 
preserves sparse membership and replicated writer records without a local 
search index.
   - On NACK, timeout, or disconnect, reserve the original materialization 
budget and rebuild the batch through an independent WAL iterator. Incomplete 
WAL keeps the original payload; failed replay keeps the delivery record and 
returns the reserved memory for a later retry.
   - Fence serialization, payload replacement, cleanup, and late delivery 
callbacks with the event monitor and a response identity for each poll attempt. 
Seek and close clear the delivery indexes.
   - Protect WAL needed by live delivery records, remove those records after 
ACK, and refresh the deletion boundary. Restart recovery continues from 
committed writer positions.
   
   ### Resource tradeoffs
   
   WAL validation and redelivery add disk reads and materialization work. 
Delivery metadata remains proportional to the number of outstanding batches; 
member coordinates are compressed and records retain neither tablets nor 
serialized payloads. The existing single-entry size limit still applies.
   
   ### Validation
   
   - English full-reactor main and test-source compilation: `mvn -nsu 
test-compile -DskipTests`.
   - Chinese full-reactor main and test-source compilation: `mvn -nsu 
test-compile -P with-zh-locale -DskipTests`.
   - Focused subscription/WAL regression: **157 tests, 0 failures, 0 errors, 1 
existing skipped test**, covering delivery, conversion, ACK state, 
initialization, seek, memory admission, WAL backpressure, the receiver, and the 
WAL iterator.
   - All nine new delivery tests pass, including consuming ten batches with a 
two-batch memory budget before committing, out-of-order/late ACKs, redelivery 
after reclamation, incomplete/unreadable WAL, sparse/replicated members, 
restart recovery, seek reset, and serialization/cleanup fencing.
   - Spotless and Checkstyle pass during compilation. English/Chinese message 
keys and format placeholders match; `git diff --check` passes.
   
   <hr>
   
   This PR has:
   
   - [x] been self-reviewed for concurrent read, concurrent write, and 
concurrent read/write.
   - [x] added comments explaining the lifecycle and WAL retention decisions.
   - [x] added unit tests covering new delivery and replay paths.
   
   ##### Key changed classes
   
   `ConsensusPrefetchingQueue`, `ConsensusLogToTabletConverter`, 
`SubscriptionEvent`, `SubscriptionEventSingleResponse`, 
`CachedSubscriptionPollResponse`, and `SubscriptionReceiverV1`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to