suxiaogang223 opened a new pull request, #1390: URL: https://github.com/apache/paimon-rust/pull/1390
### Purpose Linked issue: close #1386 A partially filled output batch keeps `output_indices` nonempty. If many later keys are omitted because their winning rows are retracts, consumed source batches remain in `batch_buffer` until the output fills or input ends. Reclaim unrelated batches while keeping both cursor and pending-output references valid. ### Brief change log - Mark pending output batches as live during buffer compaction and remap their indices alongside cursor indices. - Compact when retained batches exceed twice the upper bound of cursor/pending-row references, amortizing cleanup without changing output batch sizing. - Add a lazy two-stream regression with three surviving keys and many deleted keys. Observe source-array ownership through Weak references and verify output keys, values, order, batch count, and eventual release. ### Tests - The new regression fails before the fix with 128 observed retained source arrays; after the fix it passes the bound of at most 16. - `cargo +1.95.0 test --locked -p paimon --lib table::sort_merge::tests`: 42 passed. - `cargo +1.95.0 test --locked -p paimon --lib`: 4,860 passed, 6 ignored. - Targeted Clippy passed with `-D warnings -A clippy::nonminimal_bool`. Strict Clippy's six existing nonminimal-bool errors were reproduced on unmodified main; no source lint suppressions were added. - `cargo +1.95.0 fmt --all -- --check` and `git diff --check`: passed. Local Cargo builds used debug info disabled and incremental compilation disabled. The regression measures retained source-array ownership, not RSS. This change does not establish a byte budget for all merge state or address a single key with arbitrarily many versions. ### API and Format No public API or storage-format changes. Pending output retains its original rows and order; output batch sizing is unchanged. ### Documentation Update the buffer ownership and compaction comments beside the implementation. AI assistance: Codex helped develop the implementation, regression, and this description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
