anuragmantri opened a new issue, #3126:
URL: https://github.com/apache/iceberg-rust/issues/3126
### What's the feature are you trying to implement?
Today, `IcebergTableScan::compute_properties` always builds
`EquivalenceProperties` with no ordering, so DataFusion has no way to know a
scan's output is already sorted. DataFusion's optimizer decides whether to
insert a sort by checking what each node's `PlanProperties::output_ordering`
already claims. A sort-merge join, an ORDER BY, or a MERGE all require their
input in a specific order; if the node feeding them already claims that order,
the physical optimizer can skip the SortExec (or SortPreservingMergeExec) it
would otherwise insert above the scan.
This epic tracks the iceberg-rust side, split into two parts that land
independently:
### Part 1: Per-file sort order on FileScanTask
FileScanTask carries no sort-order information today. A file's
sort_order_id is per-file (`DataFile.sort_order_id`), not table-wide — a
table's `default_sort_order()` describes what new writes should conform to, not
what any specific existing file actually is sorted by. Resolve each manifest
entry's sort_order_id against the table's known sort orders and carry the
result on FileScanTask, threaded through PlanContext -> ManifestFileContext ->
ManifestEntryContext
### Part 2: DataFusion integration
Depends on apache/iceberg-rust#2671 (eager scan planning) merging first.
`PlanProperties::output_ordering` is one value for the whole
IcebergTableScan node, not one per partition, so an ordering can only be
claimed when every file across every one of #2671's task groups agrees on the
same sort order (via Part 1's per-file data). When they do, the claim must be
backed by a real per-partition merge in execute() (e.g. DataFusion's
`StreamingMergeBuilder`), since two files agreeing on `sort_order_id` doesn't
mean their key ranges don't overlap. When even one file disagrees or has no
resolvable sort order, fall back to today's behavior exactly.
Related
* apache/iceberg#16430 — Java SupportsReportOrdering
* apache/datafusion-comet#4367 — Comet outputOrdering forwarding
* apache/iceberg-rust#2671 — eager scan planning (Part 2's dependency)
### Willingness to contribute
I can contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
