suxiaogang223 opened a new issue, #1386:
URL: https://github.com/apache/paimon-rust/issues/1386

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-rust/issues) and found nothing 
similar.
   
   Searched issues and PRs for batch_buffer, retained batch, sort merge memory, 
and merge memory. #911 concerns broader resource governance; no directly 
matching issue was found for this Reader retention path as of November 12, 2026.
   
   ### Please describe the bug
   
   Source inspection of main `f91fdb8ab034887547a5ee2f26c1979e30250558` 
identifies a batch-retention path in `sort_merge_stream`:
   
   - A consumed input batch is added to `batch_buffer`.
   - A surviving key adds an entry to `output_indices`.
   - `MergeResult::Omit` adds no output.
   - The buffer is compacted when output is empty, or after a full output batch 
has been materialized.
   
   When output is nonempty but below the configured batch size, a subsequent 
long sequence of different keys whose winners are retracts does not meet either 
cleanup condition. Consumed source batches can remain retained until the 
pending output is finally emitted or the input ends.
   
   A focused regression can exercise the merge reader with multiple input 
streams, one early surviving key, then many distinct keys whose final state is 
deleted. Keep the total number of surviving keys below batch_size and split 
inputs across many small RecordBatches. The output should contain the surviving 
row, while retention of unrelated consumed batches should not grow with the 
number of deleted keys.
   
   This report records a static code-path finding. No new runtime reproduction, 
allocation count, or RSS measurement has been run, and no OOM frequency is 
claimed.
   
   ### Solution
   
   Reclaim consumed batches independently of filling a complete output batch, 
while preserving batches still referenced by cursors and pending output. 
Partial-output materialization or reference-aware reclamation are possible 
directions; the final algorithm needs review.
   
   Use deterministic retained-buffer or ownership observations in a regression 
rather than an RSS-only threshold. Cover output order, row values, all-retract 
input, and pending surviving rows.
   
   One key with arbitrarily many versions is a separate merge-state boundary 
and is outside this narrowly scoped report. This issue does not claim to 
establish a bound on every allocation in the Reader.
   
   Reference: [sort_merge output and cleanup 
conditions](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/crates/paimon/src/table/sort_merge.rs#L1742).
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to