shyjsarah opened a new issue, #679: URL: https://github.com/apache/paimon-rust/issues/679
### Describe the bug The `$referenced_files_size` system table retains a complete `ScopeFileSet` for every snapshot before merging the results. `buffer_unordered(SNAPSHOT_CONCURRENCY)` bounds active futures, but the following `try_collect` keeps all completed per-snapshot hash maps alive until the entire scope has been scanned. For tables with long snapshot histories and substantial overlap, peak memory therefore grows with the number of snapshots even when the final deduplicated result cardinality is constant. Additional `try_join_all` calls for branches, tags, manifests, and extra-file metadata can further amplify transient memory and duplicate manifest reads on concurrent cache misses. A synthetic collector test with 500 shared referenced files produced the following fresh-process peak RSS: | Snapshots | Current | Incremental fold | |---:|---:|---:| | 10 | 34.3 MB | 32.2 MB | | 100 | 51.2 MB | 32.8 MB | | 1000 | 211.0 MB | 32.5 MB | The output counts and sizes were identical. ### Expected behavior Auxiliary memory should be proportional to the final unique referenced-file state plus bounded in-flight work, rather than multiplying with snapshot count and nested concurrency. ### Proposed fix - Merge completed snapshot and tag results incrementally instead of collecting all `ScopeFileSet`s. - Process manifest and extra-file metadata work with explicit bounded concurrency. - Ensure concurrent requests for the same manifest share one in-flight load. - Preserve existing result rows, deduplication semantics, missing-file behavior, and public APIs. This issue is limited to `$referenced_files_size`; `$physical_files_size`, normal table scans, and filter pushdown are out of scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
