JingsongLi opened a new pull request, #547: URL: https://github.com/apache/paimon-rust/pull/547
## Summary Propagate DataFusion's configured session batch size into Paimon's direct file readers. This aligns Paimon Parquet scan batch boundaries with native DataFusion Parquet scans instead of falling back to the smaller format default. ## Changes - add an optional preferred batch size to `TableRead` while preserving existing defaults for other callers - pass the preference through Paimon and format-table reads into `DataFileReader` - apply the preference to normal, incremental/raw, and audit direct-read construction paths - read the batch size from `TaskContext` when executing `PaimonTableScan` - add an end-to-end regression asserting a five-row Parquet scan produces batches of `[2, 2, 1]` when configured with batch size two ## Testing - [x] Regression test fails with `[5]` before the implementation and passes with `[2, 2, 1]` afterward - [x] `cargo test -p paimon-datafusion physical_plan::scan::tests::test_execute_uses_datafusion_batch_size -- --exact` - [x] `cargo test -p paimon --test audit_log_table_test` - [x] `cargo test -p paimon --lib` — 1664 passed, 1 ignored - [x] `cargo check -p paimon -p paimon-datafusion` - [x] `cargo clippy -p paimon -p paimon-datafusion --all-targets -- -D warnings` - [x] `cargo fmt --all -- --check` ## Notes This follows #543 and keeps the next performance changes independent. Runtime filters, pruning-only predicates, manifest statistics, benchmark follow-ups, and internal plans/designs are intentionally excluded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
