snmvaughan commented on issue #7441: URL: https://github.com/apache/datafusion-comet/issues/7441#issuecomment-6932129825
Thanks for the report and the suggestion. Having `CometIcebergNativeScanExec` report the replaced `BatchScanExec`'s partitioning (as #7424 does for `SinglePartition`) is needed for an aggregate above the join, but it cannot stop the join's own exchanges. The DPP rule wraps the already converted build side in a new `AdaptiveSparkPlanExec`, whose constructor runs `EnsureRequirements` again. Since #7231 the Comet sort-merge and shuffled-hash joins declare the original join's `ClusteredDistribution`, and `EnsureRequirements` only treats key-grouped children as co-partitioned when the parent is Spark's own `SortMergeJoinExec` or `ShuffledHashJoinExec` (on 4.1 also over Spark's own `KeyGroupedPartitionedScan` leaves). A Comet join is neither, so each side gets a hash exchange whatever the scan reports, and AQE has no logical link for it. What works is for a Comet join whose two sides are scans Spark planned as key-grouped, with no shuffle below either side, to declare no required distri bution, as it did before #7231. Joins with a shuffle below keep the #7231 behavior. The scan change stays for the aggregate case. #7662 makes both changes and adds a test for each shape. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
