mbutrovich opened a new pull request, #114: URL: https://github.com/apache/datafusion-iceberg/pull/114
`IcebergDataSource` reported unknown statistics, so DataFusion's join selection had nothing to compare for Iceberg scans. The source now reports a row count from the scanned snapshot's [`total-records`](https://github.com/apache/iceberg/blob/bb8348b18fa21832048b5fb6873841ba1a8da57e/format/spec.md?plain=1#L2500) summary field, adjusted for position deletes and the scan's fetch. It also reports a byte size when every projected column is fixed-width. A table with no current snapshot reports exactly zero rows. Row counts stay inexact even when a snapshot has no delete files, so DataFusion doesn't answer `COUNT(*)` from the summary. That would mean trusting writer-maintained summary fields for query results, which deserves its own discussion. The exact zero changes one plan in `binary_predicate_pushdown.slt`. That `EXPLAIN` runs on an empty table, and DataFusion only adds a round-robin repartition when an exact row count exceeds `batch_size` ([`enforce_distribution.rs`](https://github.com/apache/datafusion/blob/500744d5c25499055e3b2ca46b022bc425989203/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L978-L987)). So the `RepartitionExec` above the scan is gone. The idea comes from @JanKaul's [`datafusion_iceberg`](https://github.com/JanKaul/iceberg-rust/blob/5ab0716076ff0a4accdd61e116026f89dcbf6991/datafusion_iceberg/src/table/mod.rs#L441-L474), which reports `total-records` as an inexact row count. Thanks! ### Tests - Unit tests in `crates/datafusion/src/read/scan.rs` call `partition_statistics` on tables built in memory with chosen snapshot summaries. - `crates/datafusion/tests/statistics.rs` checks the statistics in `EXPLAIN` output for the taxi fixture through the catalog provider. # AI usage disclosure I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
