sundapeng opened a new pull request, #624: URL: https://github.com/apache/paimon-rust/pull/624
### Purpose A format table has no manifest, so planning cannot know how many rows a data file holds. `format_table_scan` recorded that as `row_count: 0` — and 0 is a number a reader is entitled to trust: - `DataSplit::raw_merged_row_count` returned `Some(0)` for a raw-convertible split with no deletion files; - DataFusion turned that into `Precision::Exact(0)` for the whole scan; - the `aggregate_statistics` rule then rewrote `SELECT COUNT(*)` into the constant `0` and never opened a file. `SELECT *` was unaffected because it reads the files, which is why this went unnoticed: the same table answered 0 rows and returned rows. The same 0 also silenced column statistics. `ColumnStatsAccumulator::add_file` skips a file whose `row_count` is 0, so every format-table file was treated as empty and the accumulator kept its default `null_count = 0`, handed out as `Precision::Exact` — an exact answer derived from files nobody read. ### Approach Make "unknown" expressible instead of overloading a legal value: - `DataFileMeta::ROW_COUNT_UNKNOWN` is `-1`, with `row_count_known()` to test for it; - `DataSplit::row_counts_known()` reports whether a whole split is known; - `merged_row_count()` returns `None` as soon as one file is unknown; - `row_count()` sums only known files and is documented as a lower bound rather than a total; - `format_table_scan` fills the sentinel instead of `0`. Any producer that cannot determine a row count can adopt the same convention — nothing here is specific to format tables. A file that genuinely holds zero rows still reports `0`, and `Precision::Exact(0)` for it stays correct. ### Tests `crates/integrations/datafusion/tests/format_table_statistics.rs`: | case | pins | |---|---| | `test_format_table_count_star_matches_scanned_rows` | `COUNT(*)` matches the rows a scan returns — `0` instead of `5` before this change | | `test_empty_format_table_counts_zero` | an empty table still counts `0` | | `test_format_table_with_empty_file_counts_zero` | a table holding an empty file still counts `0` exactly | | `test_partitioned_format_table_count_with_partition_predicate` | the predicate path was already correct: a pushed-down predicate makes the statistics inexact and the rule does not fire | `cargo test -p paimon --lib source::` stays green (96 cases). ### Note `DataSplit::serialize` writes `row_count` into the v8 wire format as-is, so a consumer that reads splits planned by this crate needs to understand the sentinel. If splits are only produced and consumed within this crate that is a non-issue; flagging it in case a cross-language path exists that I am not aware of. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
