yhuang-db opened a new pull request, #59974:
URL: https://github.com/apache/spark/pull/59974
### What changes were proposed in this pull request?
In `PushDownUtils.pruneColumns`, when nested schema pruning is enabled, only
call `SchemaPruning.pruneSchema` if at least one requested root field is a
nested field (`rootFields.exists(!_.derivedFromAtt)`). This is the same check
that the v1 `SchemaPruning` rule does:
```scala
// If requestedRootFields includes a nested field, continue. Otherwise,
// return op
if (requestedRootFields.exists { root: RootField => !root.derivedFromAtt }) {
```
Otherwise, the needed top-level columns (`neededOutput.toStructType`) are
pushed directly, which is the same schema that the path without nested schema
pruning pushes.
### Why are the changes needed?
For parity between DSv2 and v1. When every requested root field is a
top-level attribute, `pruneSchema` keeps each requested column as a whole, so
after filtering by the needed column names the result equals the needed output
schema. The call is then pure overhead: it merges all requested root fields
into one struct, looks up a match in it for every field of the relation schema,
and sorts the result by the relation schema. This cost grows with the table
width times the number of requested columns, which adds up for wide tables.
### Does this PR introduce _any_ user-facing change?
No. The required schema pushed to the data source is the same.
### How was this patch tested?
Added a test to `PushDownUtilsSuite` that checks the required schema pushed
to a `SupportsPushDownRequiredColumns` scan builder when only top-level columns
are requested and when a nested field is requested. The pushed schema does not
change, so this test guards the behavior rather than failing without this
change.
```
build/sbt 'sql/testOnly *PushDownUtilsSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac (Claude Code, Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
