snmvaughan opened a new pull request, #7662:
URL: https://github.com/apache/datafusion-comet/pull/7662

   ## Which issue does this PR close?
   
   Closes #7441.
   
   ## Rationale for this change
   
   With AQE on, a query fails with an `AssertionError` in 
`AdaptiveSparkPlanExec.setLogicalLinkForNewQueryStage` when the build side of a 
DPP broadcast contains a storage-partitioned join over native Iceberg scans.
   
   Comet builds the DPP subquery's plan by wrapping the build side it already 
converted in a new `AdaptiveSparkPlanExec`, and that constructor runs 
`EnsureRequirements` over it again. Since #7231 the Comet joins, aggregates and 
windows declare Spark's required child distribution, and `EnsureRequirements` 
puts an exchange under each side of the join. AQE has no logical link for those 
exchanges.
   
   The issue suggests having the native scan report the key-grouped 
partitioning of the scan it replaces, as #7424 does for `SinglePartition`. That 
is not enough for a join. In Spark 3.5 through 4.2, `EnsureRequirements` only 
treats two key-grouped children as co-partitioned when the parent is Spark's 
own `SortMergeJoinExec` or `ShuffledHashJoinExec` (`checkKeyGroupCompatible`), 
and in 4.1 every leaf below them must also be a `KeyGroupedPartitionedScan` 
(`canPushDownSPJParamsToScan`). A `CometSortMergeJoinExec` over 
`CometIcebergNativeScanExec`s is neither, so it finds no usable shuffle spec 
and shuffles both sides whatever the scans report. With only the scan change, 
the new join test fails in the same way.
   
   An aggregate has one child, so `EnsureRequirements` only asks whether that 
child's partitioning satisfies the distribution. An aggregate above a 
storage-partitioned join does need the scans to report their partitioning. 
Without it the final aggregate gets an exchange between itself and the partial 
aggregate, inside a native block that was already serialized, and the native 
side fails with `Column index 1 is out of bound`.
   
   ## What changes are included in this PR?
   
   - `CometSortMergeJoinExec` and `CometHashJoinExec` declare no required child 
distribution when both sides are storage-partitioned, meaning every leaf is a 
key-grouped scan and there is no shuffle above it. No shuffle sits below such a 
join for AQE to split, so the requirement from #7231 protects nothing there. 
Every other join still returns the original operator's requirement. 
`CometExec.isStoragePartitioned` checks the join's current children, because 
conversion works bottom-up and `originalPlan.children` are already Comet nodes. 
It uses `DataSourceV2ScanExecBase.keyGroupedPartitioning`, which every 
supported Spark version declares, so it needs no shim even though Spark 4.2 
renamed `KeyGroupedPartitioning` to `KeyedPartitioning`.
   - `CometIcebergNativeScanExec.outputPartitioning` reports whatever the 
`BatchScanExec` it replaces reports, and `UnknownPartitioning(0)` when that is 
unknown. This generalizes #7424. Reading it resolves no DPP subquery, because 
in Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 `BatchScanExec.outputPartitioning` is 
built from `inputPartitions` and only `inputRDD` uses `filteredPartitions`.
   - Two tests in `CometIcebergNativeSuite`. One is the reproduction from the 
issue, with an assertion that neither the plan nor the subquery's plan has an 
exchange. The other puts an aggregate above the storage-partitioned join.
   
   ## How are these changes tested?
   
   On Spark 4.1:
   
   - The join test fails with only the scan change, and passes with the join 
change.
   - The aggregate test fails without the scan change, with `Column index 1 is 
out of bound`, and passes with it.
   - The 17 other `AQE DPP` tests in `CometIcebergNativeSuite` pass.
   
   The change compiles on Spark 3.5 with Scala 2.12, and its main sources 
compile on Spark 4.2. I have not run the tests on Spark 3.5 or 4.0.
   
   A join over Comet's non-native Iceberg reader is not covered. 
`CometExecRule` wraps `CometBatchScanExec` in a `CometScanWrapper` leaf, which 
`isStoragePartitioned` does not unwrap, so that join keeps the requirement from 
#7231. The tests also use only `CometSortMergeJoinExec`, through the `MERGE` 
hint. `CometHashJoinExec` shares the trait but has no test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to