cloud-fan commented on PR #38434: URL: https://github.com/apache/spark/pull/38434#issuecomment-1308715889
I think this needs a bit more design. Partitioning is a physical property it's very weird to "pushdown" it at the logical phase. I think what we really need is tracking the requirement when doing top-down planning. e.g. when we planning a sort merge join, we should track the requirement (partitioned and ordered by join keys) when planning the join children. This is also an idea from the volcano optimizer and is a widely adopted technology. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
