snmvaughan commented on issue #7441:
URL: 
https://github.com/apache/datafusion-comet/issues/7441#issuecomment-6932129825

   Thanks for the report and the suggestion. Having 
`CometIcebergNativeScanExec` report the replaced `BatchScanExec`'s partitioning 
(as #7424 does for `SinglePartition`) is needed for an aggregate above the 
join, but it cannot stop the join's own exchanges. The DPP rule wraps the 
already converted build side in a new `AdaptiveSparkPlanExec`, whose 
constructor runs `EnsureRequirements` again. Since #7231 the Comet sort-merge 
and shuffled-hash joins declare the original join's `ClusteredDistribution`, 
and `EnsureRequirements` only treats key-grouped children as co-partitioned 
when the parent is Spark's own `SortMergeJoinExec` or `ShuffledHashJoinExec` 
(on 4.1 also over Spark's own `KeyGroupedPartitionedScan` leaves). A Comet join 
is neither, so each side gets a hash exchange whatever the scan reports, and 
AQE has no logical link for it. What works is for a Comet join whose two sides 
are scans Spark planned as key-grouped, with no shuffle below either side, to 
declare no required distri
 bution, as it did before #7231. Joins with a shuffle below keep the #7231 
behavior. The scan change stays for the aggregate case. #7662 makes both 
changes and adds a test for each shape.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to