ykisana opened a new pull request, #59387:
URL: https://github.com/apache/spark/pull/59387
### What changes were proposed in this pull request?
`V2ScanPartitioningAndOrdering` resolves a scan's `SupportsReportOrdering`
ordering against the full `DataSourceV2Relation`. For a nested sort key such as
`s.y`, the result is `GetStructField(s, 1)` over the unpruned type `s:
struct<x, y>`. When nested schema pruning narrows the scan output to `s:
struct<y>`, the attribute keeps its `ExprId` but changes type, and `y` moves to
ordinal 0. `DataSourceV2ScanExecBase.outputOrdering` checks the ordering
against the output only by `ExprId`, so the stale expression is kept and never
matches a required ordering on the pruned `s`.
This PR binds the reported ordering to the scan output in
`DataSourceV2ScanExecBase` before `outputOrdering` uses it. It uses
`ProjectionOverSchema`, which rebinds the attribute to its pruned type and
recomputes each field ordinal by name, the same way
`DataSourceV2ScanRelation.outputBoundPushedFilters` remaps pushed filters. A
sort order on a nested field that the scan no longer reads (`FIELD_NOT_FOUND`)
is treated like a sort order on a pruned column and ends the kept prefix. The
bound ordering is a `lazy val`, like `outputPartitioning`, because the planner
asks for it many times.
The logical `DataSourceV2ScanRelation.ordering` is unchanged and stays in
relation space, which `PlanMerger` relies on to compare two scans of the same
table.
### Why are the changes needed?
Follow-up to SPARK-59948, which removed the redundant sort only when the
scan reads the whole struct. With `s struct<x: bigint, y: bigint>` and a
reported ordering on `s.y`:
```sql
SELECT id, s.y FROM t SORT BY s.y
```
plans a redundant sort above a scan that already reports that ordering:
```
Sort [_extract_y#23L ASC NULLS FIRST], false, 0
+- Project [id#13L, s#14.y AS y#15L, s#14.y AS _extract_y#23L]
+- BatchScan testcat.ns.nested_items[id#13L, s#14] ...
```
Found by @peter-toth while reviewing #59199.
### Does this PR introduce _any_ user-facing change?
No. Query results are unchanged. For DSv2 sources that report an ordering on
a nested field, the physical plan no longer contains a redundant `SortExec`
when nested schema pruning narrows the struct.
### How was this patch tested?
Added a test to `KeyGroupedPartitioningSuite`. A table with `s: struct<x,
y>` reports an ordering on `s.y`.
- `SELECT id, s.y ... SORT BY s.y` has no `SortExec` and returns the correct
answer. Without the fix, the plan contains the `SortExec` above.
- `SELECT id, s.x ... SORT BY s.x`, where pruning drops `y`, still plans a
sort and returns the correct answer.
Also ran `KeyGroupedPartitioningSuite`, `MergeSubplansSuite`,
`WriteDistributionAndOrderingSuite` and `DataSourceV2Suite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Fix written by and all changes reviewed by @ykisana. Tests, comments, pr
desc by Claude Opus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
