suxiaogang223 opened a new issue, #1386: URL: https://github.com/apache/paimon-rust/issues/1386
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. Searched issues and PRs for batch_buffer, retained batch, sort merge memory, and merge memory. #911 concerns broader resource governance; no directly matching issue was found for this Reader retention path as of November 12, 2026. ### Please describe the bug Source inspection of main `f91fdb8ab034887547a5ee2f26c1979e30250558` identifies a batch-retention path in `sort_merge_stream`: - A consumed input batch is added to `batch_buffer`. - A surviving key adds an entry to `output_indices`. - `MergeResult::Omit` adds no output. - The buffer is compacted when output is empty, or after a full output batch has been materialized. When output is nonempty but below the configured batch size, a subsequent long sequence of different keys whose winners are retracts does not meet either cleanup condition. Consumed source batches can remain retained until the pending output is finally emitted or the input ends. A focused regression can exercise the merge reader with multiple input streams, one early surviving key, then many distinct keys whose final state is deleted. Keep the total number of surviving keys below batch_size and split inputs across many small RecordBatches. The output should contain the surviving row, while retention of unrelated consumed batches should not grow with the number of deleted keys. This report records a static code-path finding. No new runtime reproduction, allocation count, or RSS measurement has been run, and no OOM frequency is claimed. ### Solution Reclaim consumed batches independently of filling a complete output batch, while preserving batches still referenced by cursors and pending output. Partial-output materialization or reference-aware reclamation are possible directions; the final algorithm needs review. Use deterministic retained-buffer or ownership observations in a regression rather than an RSS-only threshold. Cover output order, row values, all-retract input, and pending surviving rows. One key with arbitrarily many versions is a separate merge-state boundary and is outside this narrowly scoped report. This issue does not claim to establish a bound on every allocation in the Reader. Reference: [sort_merge output and cleanup conditions](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/crates/paimon/src/table/sort_merge.rs#L1742). ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
