sundapeng opened a new pull request, #644: URL: https://github.com/apache/paimon-rust/pull/644
### Purpose Linked issue: close #643 A format table has no manifest, so scan planning does not know how many rows a data file holds. `FormatTableScan::apply_limit_pushdown` kept only the first `limit` splits anyway, which reads the number of files as if it were a number of rows. An empty data file is enough to break it, and an empty data file is ordinary output: an engine that writes one file per task leaves one behind whenever a task matched nothing. A table of three rows whose first file is empty answers `SELECT id FROM t LIMIT 1` with zero rows, and reports success. ### Brief change log - Keep every split for a positive limit and let execution stop once it has enough rows. A limit of zero still needs no split. This is what Java does. `FormatTableScan.FormatTableScanPlan.splits` returns an empty list only for a non-positive limit, with the same reason in a comment. paimon-rust's own Paimon-table path is already stricter than the format path was: `PaimonTableScan::apply_limit_pushdown` skips a split whose merged row count is unknown rather than counting it as if it held enough rows. ### Tests `crates/integrations/datafusion/tests/format_table_statistics.rs`: `test_format_table_limit_is_not_answered_from_a_file_count` writes an empty file followed by a three-row file and asserts `LIMIT 1` returns a row. It fails on main with `left: 0, right: 1`. ``` cargo fmt --all -- --check cargo clippy --locked -p paimon -p paimon-datafusion --all-targets -- -D warnings cargo test --locked -p paimon --lib cargo test --locked -p paimon-datafusion --lib --test format_table_statistics --test count_pushdown --test scan_pruning_trace --test read_tables ``` ### API and Format No API or storage format change. ### Documentation None needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
