shyjsarah opened a new pull request, #680: URL: https://github.com/apache/paimon-rust/pull/680
### Purpose Linked issue: close #679 `$referenced_files_size` used `try_collect` after `buffer_unordered`, so every completed per-snapshot `ScopeFileSet` remained alive until the whole scope finished. Long snapshot histories therefore multiplied retained hash-map state even when snapshots referenced the same files. In a synthetic collector benchmark with a fixed 500-file result, peak RSS grew from 34.3 MB at 10 snapshots to 211.0 MB at 1000 snapshots. Incremental merging kept the 1000-snapshot peak at 32.5 MB with identical output. ### Brief change log - Merge completed snapshot and tag file sets incrementally instead of collecting all results first. - Process sorted branches one at a time, merging each branch into `total` and retaining only its compact summary. - Preserve snapshot concurrency, tag ordering, branch output ordering, deduplication, missing-file handling, and error propagation. - Add a drop-probe regression test that fails if completed scope results are retained before the next stream item is polled. - Extend branch-tag coverage to verify stable branch ordering. ### Tests - `cargo fmt --all -- --check` - `cargo clippy -p paimon -p paimon-datafusion --all-targets -- -D warnings` - `cargo test -p paimon --lib` - `cargo test -p paimon referenced_files -- --nocapture` - `cargo test -p paimon-datafusion system_tables` The full `cargo test -p paimon` run passed all 2154 library tests. Some REST mock-server integration tests failed intermittently with local HTTP `IncompleteMessage` errors; the same failure reproduced on an unmodified `apache/main` worktree. ### API and Format No public API or storage format changes. Normal Paimon table scans are not affected; this only changes the `$referenced_files_size` collection path. ### Documentation No documentation changes are required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
