XiaoHongbo-Hope commented on PR #666: URL: https://github.com/apache/paimon-rust/pull/666#issuecomment-5173877724
Local synthetic A/B validation against the same community `main` base: - Workload: one split containing 120,000 representative `DataFileMeta` values (512-byte embedded payload plus key/value stats), then clone the split 6 times. - `main`: RSS grew from 256,304 KiB after construction to 1,763,744 KiB after cloning; clone step took 2,119 ms. - This PR: RSS remained 256,344 KiB before and after cloning; clone step completed below the 1 ms measurement resolution. Process max RSS was 306,912 KiB. This confirms that the split-copy portion of the planning memory amplification is eliminated locally. It does not replace production-scale validation of the complete manifest/planning path. Also addressed review suggestions in `35cc755`: - Added `#[must_use]` to `Plan::into_splits`. - Documented that manifest memory is bounded by the aggregate plus up to 64 in-flight buffered reads. Revalidated 51 source tests, 80 table-scan tests, formatting, and strict clippy for `paimon` plus `paimon-datafusion`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
