zsxwing opened a new pull request, #58007: URL: https://github.com/apache/spark/pull/58007
### What changes were proposed in this pull request? Update `SelectedField` to pass metadata to `StructField` objects when pruning columns. ### Why are the changes needed? When SelectedField prunes nested columns, it doesn't copy column metadata at the following two places: https://github.com/apache/spark/blob/042ad7d0c4ac1c4d3e9fdeb48e2695fdeb861135/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SelectedField.scala#L88 https://github.com/apache/spark/blob/042ad7d0c4ac1c4d3e9fdeb48e2695fdeb861135/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SelectedField.scala#L112 This can cause column metadata to be dropped and downstream cannot see the column metadata. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added two new unit test to verify `SelectedField` does keep metadata. ### Was this patch authored or co-authored using generative AI tooling? Yep. Generated by Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
