toutane opened a new pull request, #2671:
URL: https://github.com/apache/iceberg-rust/pull/2671
## Which issue does this PR close?
- Closes #2220
- Related to
- #1604
- #128
- #2298
## What changes are included in this PR?
This PR adds an opt-in eager scan planning path for the DataFusion
integration.
When `iceberg.enable_eager_scan_planning` is enabled,
`IcebergTableProvider::scan()` plans `FileScanTask`s during physical planning,
groups them across DataFusion `target_partitions`, and exposes the resulting
output partition count as `UnknownPartitioning(N)`. Each `execute(partition)`
call then reads only the task group assigned to that output partition through
`ArrowReaderBuilder`.
The default behavior is unchanged: eager planning is disabled by default, so
scans keep the existing lazy single-partition planning path unless explicitly
enabled.
This is a narrower split from #2298, focused on file-level eager planning
and output partition count reporting. Some of the broader design discussions
and review feedback happened in #2298; this PR keeps only the first scoped step
and intentionally does not include hash partitioning, size-aware bin-packing,
or row-group/sub-file planning.
#### Trade-offs:
- Eager planning does catalog/metadata work during `TableProvider::scan()`,
so it is kept opt-in
- Task grouping is round-robin and count-based, not size-aware. This is
simple and deterministic, but can be imbalanced when file sizes vary
- Parallelism is at the `FileScanTask` level only. A table with one large
file will not benefit from this change
- The scan reports `UnknownPartitioning(N)`, not hash partitioning. This
exposes the number of output partitions without claiming stronger partitioning
semantics
#### Follow-up work:
- Add size-aware task bin-packing using `file_size_in_bytes` (#128)
- Add sub-file / row-group level parallelism as the longer-term direction
(#1604)
- Cache planned tasks for repeated scans with the same snapshot, projection,
filter, and target partition count
- Investigate stronger output partitioning declarations for safe
identity-partitioned cases
- Add benchmarks for multi-file tables and skewed file-size distributions
## Are these changes tested?
Yes. Integration tests cover:
- eager scan planning disabled by default
- enabling eager scan planning through `iceberg.enable_eager_scan_planning`
- exposing multiple scan output partitions when eager planning is enabled
- preserving query results between single-partition and multi-partition scans
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
