suxiaogang223 opened a new pull request, #1390:
URL: https://github.com/apache/paimon-rust/pull/1390

   ### Purpose
   
   Linked issue: close #1386
   
   A partially filled output batch keeps `output_indices` nonempty. If many 
later keys are omitted because their winning rows are retracts, consumed source 
batches remain in `batch_buffer` until the output fills or input ends. Reclaim 
unrelated batches while keeping both cursor and pending-output references valid.
   
   ### Brief change log
   
   - Mark pending output batches as live during buffer compaction and remap 
their indices alongside cursor indices.
   - Compact when retained batches exceed twice the upper bound of 
cursor/pending-row references, amortizing cleanup without changing output batch 
sizing.
   - Add a lazy two-stream regression with three surviving keys and many 
deleted keys. Observe source-array ownership through Weak references and verify 
output keys, values, order, batch count, and eventual release.
   
   ### Tests
   
   - The new regression fails before the fix with 128 observed retained source 
arrays; after the fix it passes the bound of at most 16.
   - `cargo +1.95.0 test --locked -p paimon --lib table::sort_merge::tests`: 42 
passed.
   - `cargo +1.95.0 test --locked -p paimon --lib`: 4,860 passed, 6 ignored.
   - Targeted Clippy passed with `-D warnings -A clippy::nonminimal_bool`. 
Strict Clippy's six existing nonminimal-bool errors were reproduced on 
unmodified main; no source lint suppressions were added.
   - `cargo +1.95.0 fmt --all -- --check` and `git diff --check`: passed.
   
   Local Cargo builds used debug info disabled and incremental compilation 
disabled. The regression measures retained source-array ownership, not RSS. 
This change does not establish a byte budget for all merge state or address a 
single key with arbitrarily many versions.
   
   ### API and Format
   
   No public API or storage-format changes. Pending output retains its original 
rows and order; output batch sizing is unchanged.
   
   ### Documentation
   
   Update the buffer ownership and compaction comments beside the 
implementation.
   
   AI assistance: Codex helped develop the implementation, regression, and this 
description.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to