Caideyipi opened a new pull request, #18870:
URL: https://github.com/apache/iotdb/pull/18870

   ## Description
   
   When several DataRegions share a subscription memory budget, fixed-depth 
read-ahead can exhaust that budget and reject realtime requests even when the 
pending queue has room. A prefetch round also removes its entire pending batch 
before processing, so memory pressure or delivery capacity can discard the 
unprocessed suffix and force it through WAL replay.
   
   Keep unprocessed requests in the bounded pending queue, including their 
consensus memory reservations, and remove them only after successful 
materialization or deduplication. Subscription memory pressure now pauses 
materialization while realtime admission remains governed by pending capacity, 
consensus request memory, and existing lifecycle/writer-backlog checks. ACK 
releases memory and wakes prefetch to resume ordered delivery.
   
   Bound speculative read-ahead by half of each active queue's current memory 
quota as well as the configured event capacity. Account for queued and 
in-flight Tablet memory, allow an empty delivery queue to make progress with a 
larger entry, and retain the shared memory manager's hard allocation limits. 
Count the current delivery batch toward event capacity and publish reserved 
batches when prefetch pauses. Already materialized writer buffers are drained 
before converting more requests.
   
   ## Validation
   
   - Two new four-region regressions fail against the original implementation: 
realtime admission with exhausted node memory, and bounded read-ahead followed 
by ordered ACK/resume cycles.
   - All 75 focused subscription queue and memory tests pass (JDK 17, Windows), 
including initialization, seek, real WAL gaps/retries, memory accounting, and 
ACK recovery.
   - Full-reactor `test-compile -DskipTests` passes for both English and `-P 
with-zh-locale` (53 modules each, captured exit codes 0).
   - Added coverage for preserving a pending suffix at the configured event 
capacity, pending memory cleanup, and resuming without WAL replay. Existing WAL 
gap/retry and seek coverage is included in the focused test run.
   
   The original throughput workload and its 400 ms P99 target still need to be 
rerun in the benchmark environment.
   
   ## WAL latency scope
   
   This change avoids unnecessary WAL fallback after transient materialization 
pressure. It does not remove all possible slow WAL operations: an iterator can 
read an entire file's metadata or scan multiple physical entries before 
returning a request, damaged metadata can trigger prefix recovery, and an 
exhausted subscription reader can synchronously wait for WAL rotation. The 
`waiting ...ms for WAL gap` log reports cumulative gap recovery time, rather 
than the latency of one physical read.
   
   This PR has:
   
   - [x] been self-reviewed for concurrent read/write and lifecycle cleanup.
   - [x] added comments and Javadocs explaining request retention and the 
read-ahead memory target.
   - [x] added or updated unit tests for the changed behavior.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to