xiaoh1024 opened a new pull request, #59791: URL: https://github.com/apache/spark/pull/59791
### What changes were proposed in this pull request? Extend `NestedColumnAliasing` to extract nested fields across adjacent `Project` nodes when `CollapseProject` keeps both projections, replacing unreferenced parent struct outputs in the lower `Project` with the extracted field aliases. Extraction is limited to direct nested field accesses and reuses `CollapseProject.canCollapseExpressions` for lower-project aliases, so array/map lookups stay at their original evaluation sites and non-deterministic or expensive struct expressions are not duplicated. This builds on closed PR #40098 by @ulysses-you with the additional eligibility checks above. ### Why are the changes needed? In queries such as `SELECT employer.id, employer.id + 1 AS x, x + 1 AS y FROM contacts`, lateral column alias resolution introduces two adjacent `Project` nodes and `CollapseProject` retains them to avoid duplicating `employer.id + 1`. Because `NestedColumnAliasing` does not handle `Project(_, child: Project)`, the lower `Project` outputs the whole `employer` struct and prevents `SchemaPruning` from pruning unused nested fields at the scan. ### Does this PR introduce _any_ user-facing change? Yes. Queries with adjacent `Project` nodes that access only a subset of nested fields now prune unused nested fields from the scan. Query results are unchanged. ### How was this patch tested? Added optimizer plan tests in `NestedColumnAliasingSuite` covering passthrough structs, nested aliases, whole-struct retention, expensive and non-deterministic struct expressions, and conditional array indexing. Added Parquet and ORC scan pruning tests in `SchemaPruningSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
