cloud-fan commented on pull request #29485: URL: https://github.com/apache/spark/pull/29485#issuecomment-680673887
It's a bit frustrating to see the issue in `RemoveNoopOperators` and `RemoveRedundantProjects`. It should be a valid assumption that attributes with the same exprId are the same attribute, but Spark doesn't guarantee it. Maybe we should go with the other direction: create new attributes when necessary, and rewrite the parent nodes. We need to follow `ResolveReferences.rewritePlan`, which uses a custom way to traverse the plan tree, instead of the normal `transform`. I'll check other places that create `Alias` with existing exprId, @maropu can you update this PR to go with the safer direction? Sorry for the back and forth! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
