sundapeng opened a new issue, #643: URL: https://github.com/apache/paimon-rust/issues/643
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Paimon Rust version main (34449a7) ### Compute Engine DataFusion ### Minimal reproduce step A format table with two parquet files, the first one empty: ```rust write_parquet(&table_dir.join("part-0.parquet"), &[]); // 0 rows write_parquet(&table_dir.join("part-1.parquet"), &[1, 2, 3]); // 3 rows ``` ```sql SELECT id FROM paimon.test_db.events LIMIT 1; ``` ### What doesn't meet your expectations? The query returns 0 rows and reports success. The table has 3 rows, so `LIMIT 1` must return 1. `FormatTableScan::apply_limit_pushdown` keeps only the first `limit` splits: https://github.com/apache/paimon-rust/blob/34449a7/crates/paimon/src/table/format_table_scan.rs#L275-L284 A format table has no manifest, so scan planning does not know how many rows a data file holds. Keeping `limit` files reads a number of files as if it were a number of rows. An empty data file is enough to break it, and an empty data file is ordinary output: an engine that writes one file per task leaves one behind whenever a task matched nothing. Java does not do this. `FormatTableScan.FormatTableScanPlan.splits` drops splits only for a non-positive limit, and says why: ```java // Keep all splits for a positive limit because FormatDataSplit has no row count. if (limit != null && limit <= 0) { return new ArrayList<>(); } ``` paimon-rust's own Paimon-table path is also stricter than this: `PaimonTableScan::apply_limit_pushdown` skips a split whose merged row count is unknown, instead of counting it as if it held enough rows. ### Anything else? The same shape can under-read without an empty file whenever a filter is pushed down exactly, because the first `limit` files may hold no matching row. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
