XiaoHongbo-Hope opened a new pull request, #666: URL: https://github.com/apache/paimon-rust/pull/666
### Purpose Linked issue: close #665 Reduce peak memory during DataFusion scan planning for tables with many manifest entries and data files. The current path deep-copies immutable `DataSplit` metadata when transferring a plan to the physical scan and again in asynchronous readers. It also retains all per-manifest result vectors before flattening them. ### Brief change log - Store immutable `DataSplit` partition, path, data-file, deletion-file, and row-range metadata in `Arc`-backed storage so necessary split clones are shallow. - Add `Plan::into_splits` and consume plans in the DataFusion, full-text, hybrid, and variant scan paths instead of cloning their splits. - Fold buffered manifest results into the aggregate incrementally instead of collecting every result vector first. - Reuse the manifest-entry allocation when applying ADD/DELETE netting. - Add tests proving cloned splits share their metadata allocation and consuming a plan preserves metadata ownership. ### Tests - `cargo fmt --all -- --check` - `cargo test -p paimon --all-targets --features fulltext,vortex` (2186 passed, 1 ignored; all additional test targets passed) - `cargo test -p paimon table::source::tests --lib` (51 passed) - `cargo test -p paimon table::table_scan::tests --lib` (80 passed) - `cargo test -p paimon-datafusion --lib` (329 passed; 7 Spark-fixture tests could not run locally because the generated warehouse is unavailable, as documented in CONTRIBUTING.md) - `cargo clippy --locked -p paimon -p paimon-datafusion --all-targets --features paimon/fulltext,paimon/vortex -- -D warnings` ### API and Format No storage format or serialized split format changes. `DataSplit` getters keep the same signatures, and JSON plus Java binary golden compatibility tests pass. `Plan::into_splits` is an additive API. Total planning memory still grows with the number of live planned files. This change removes multiplicative copies and intermediate containers; production-scale deployment validation is still needed to quantify the reduction on very large tables. ### Documentation No documentation update is required. ### AI assistance This change was prepared with Codex assistance. The memory ownership paths, compatibility constraints, tests, and remaining production-validation assumption are documented above for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
