XiaoHongbo-Hope opened a new issue, #665: URL: https://github.com/apache/paimon-rust/issues/665
## Bug For scans with many manifest entries and data files, the DataFusion planning path can retain multiple full copies of immutable split metadata: - Table scan planning builds a Vec<DataSplit> containing DataFileMeta values. - PaimonScanBuilder copies the plan with splits().to_vec(), which deep-clones the partition, bucket path, data files, deletion files, and row ranges. - Asynchronous readers clone splits again before execution. - Manifest reads collect every per-manifest result Vec before flattening them, and manifest ADD/DELETE merging allocates another output Vec. The resulting peak memory is a multiple of the planned metadata size and can exhaust a query worker during planning, before it returns any rows. ## Expected behavior Planning should transfer ownership where possible and share immutable split metadata across necessary clones. Manifest results should be merged incrementally and reuse their allocation without changing scan semantics. ## Compatibility requirements - Preserve the public DataSplit getters. - Preserve JSON and Java binary split serialization. - Preserve manifest merge and scan behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
