mbutrovich opened a new pull request, #114:
URL: https://github.com/apache/datafusion-iceberg/pull/114

   `IcebergDataSource` reported unknown statistics, so DataFusion's join 
selection had nothing to compare for Iceberg scans. The source now reports a 
row count from the scanned snapshot's 
[`total-records`](https://github.com/apache/iceberg/blob/bb8348b18fa21832048b5fb6873841ba1a8da57e/format/spec.md?plain=1#L2500)
 summary field, adjusted for position deletes and the scan's fetch. It also 
reports a byte size when every projected column is fixed-width. A table with no 
current snapshot reports exactly zero rows.
   
   Row counts stay inexact even when a snapshot has no delete files, so 
DataFusion doesn't answer `COUNT(*)` from the summary. That would mean trusting 
writer-maintained summary fields for query results, which deserves its own 
discussion.
   
   The exact zero changes one plan in `binary_predicate_pushdown.slt`. That 
`EXPLAIN` runs on an empty table, and DataFusion only adds a round-robin 
repartition when an exact row count exceeds `batch_size` 
([`enforce_distribution.rs`](https://github.com/apache/datafusion/blob/500744d5c25499055e3b2ca46b022bc425989203/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L978-L987)).
 So the `RepartitionExec` above the scan is gone.
   
   The idea comes from @JanKaul's 
[`datafusion_iceberg`](https://github.com/JanKaul/iceberg-rust/blob/5ab0716076ff0a4accdd61e116026f89dcbf6991/datafusion_iceberg/src/table/mod.rs#L441-L474),
 which reports `total-records` as an inexact row count. Thanks!
   
   ### Tests
   
   - Unit tests in `crates/datafusion/src/read/scan.rs` call 
`partition_statistics` on tables built in memory with chosen snapshot summaries.
   - `crates/datafusion/tests/statistics.rs` checks the statistics in `EXPLAIN` 
output for the taxi fixture through the catalog provider.
   
   # AI usage disclosure
   
   I wrote this PR with help from Claude but understand and support the core 
approach, implementation, and test coverage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to