andygrove commented on PR #6459: URL: https://github.com/apache/datafusion-comet/pull/6459#issuecomment-6015273159
9d54ba50f covers the case that the review of #5634 raised. The rule no longer leaves a stage alone because it holds an `AQEShuffleReadExec`. Below a Cartesian product, Spark's rule coalesces a shuffle on the other side first, and the read it leaves there kept the union's shuffled branch at every partition. The rule now hides each read behind a leaf while Spark's rule runs over the whole stage, and puts it back afterwards, so Spark's rule leaves alone the shuffles it would coalesce together with a read one and coalesces the rest. The ancestor sizing and the `UnknownPartitioning` guard are unchanged. `AQE coalesces the shuffle partitions of a union in a stage Spark coalesced in part` covers it. It fails against the previous revision, whose plan has one coalesced read where Spark's has two. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
