allisonwang-db opened a new pull request #30659: URL: https://github.com/apache/spark/pull/30659
### What changes were proposed in this pull request? This PR fixes a bug introduced by the rule `RemoveRedundantProjects` that incorrectly removes necessary projects below a union. This fix also changes the rule `RemoveRedundantProjects` from directly passing the ordering requirements from the parent to always require column orders unless otherwise specified. ### Why are the changes needed? UnionExec requires its children's columns to have the same order in order to merge the columns. Currently, the rule can pass through the ordering requirements from its parent and incorrectly remove the necessary project nodes below a union operation. An earlier PR also fixed a similar issue for GenerateExec (SPARK-32861). In order to prevent similar issues, the rule has been changed to always require column ordering except for a few specific nodes that can pass through the requirement. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Add unit tests ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
