AnishMahto opened a new pull request, #59192: URL: https://github.com/apache/spark/pull/59192
### What changes were proposed in this pull request? Within any expression, there can be multiple nested struct read subexpressions (ex. read `a.b.c`, `a.b`, and `a.d`). All of these are independent candidates for column aliasing, and `NestedColumnAliasing.getAttributeToExtractValues` is responsible for dropping read expressions that are nested anywhere beneath another read. Ex. instead of aliasing `a.b.c` and `a.b` independently, we should only alias `a.b` and then reuse that alias as `some_alias.c` But `NestedColumnAliasing` achieves this by iterating over every nested read subexpression, and comparing it against every other read subexpression for semantic equality on its ancestor. This is quadratic with respect to the number of nested field read subexpressions, which can be noticeable for plans constructed against large (wide/deep) tables, especially generated ones. Two behavior-preserving optimizations are proposed: 1. Build an `ExpressionSet` of the reads once and look up each parent node in it. This is equivalent to `semanticEquals`. `ExpressionSet` stores only deterministic expressions in canonical form and canonicalizes the queried expression, so a match requires both expressions to be deterministic with equal canonical forms. 2. Plans often repeat the same read many times, so deduplicate with `.distinct` before filtering rather than after, to avoid repeated work. The result is the same, because removing exact duplicates doesn't change which reads are semantically present. For typical small plans, the overhead is negligible - the previous `semanticEquals` comparisons already canonicalized every read, so the only added work is building a small hash set. ### Why are the changes needed? Improve nested field read aliasing performance. ### Does this PR introduce _any_ user-facing change? No. This is a pure optimization that should not introduce any behavioral change. ### How was this patch tested? No new logic is added, but added tests to bring coverage for existing untested contracts (non-deterministic but identically canonicalized plans are still aliased separately, and deterministic plans that are canonically the same are indeed considered for shared-aliasing). ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Claude Opus 5.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
