jerry-024 opened a new pull request, #628: URL: https://github.com/apache/paimon-rust/pull/628
## What changed - replace unbounded row-range manifest collection with a memory-budgeted external spill planner - expand manifest candidates from exact entry overlap groups instead of transitive manifest envelopes - add `scan.manifest-planner-memory` (64 MiB default) and scan trace counters - add regressions for envelope holes, cross-manifest providers, ADD/DELETE, unknown ranges, and spill runs ## Why A non-empty global-index result could expand 63 overlapping manifest envelopes into one component, then materialize roughly 19 million `ManifestEntry` values before exact pruning. The resulting memory use exceeded an 8 GiB pod. ## Impact Worst-case planning may still scan all manifests, but compact metadata spills to temporary runs and full manifest entries are materialized only for selected exact groups. Existing Data Evolution provider-group and fail-open semantics are preserved. ## Validation - `cargo test -p paimon --lib` (2010 passed, 1 ignored) - `cargo clippy -p paimon --lib -- -D warnings` - `cargo check -p paimon --all-features` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
