snmvaughan opened a new pull request, #7662: URL: https://github.com/apache/datafusion-comet/pull/7662
## Which issue does this PR close? Closes #7441. ## Rationale for this change With AQE on, a query fails with an `AssertionError` in `AdaptiveSparkPlanExec.setLogicalLinkForNewQueryStage` when the build side of a DPP broadcast contains a storage-partitioned join over native Iceberg scans. Comet builds the DPP subquery's plan by wrapping the build side it already converted in a new `AdaptiveSparkPlanExec`, and that constructor runs `EnsureRequirements` over it again. Since #7231 the Comet joins, aggregates and windows declare Spark's required child distribution, and `EnsureRequirements` puts an exchange under each side of the join. AQE has no logical link for those exchanges. The issue suggests having the native scan report the key-grouped partitioning of the scan it replaces, as #7424 does for `SinglePartition`. That is not enough for a join. In Spark 3.5 through 4.2, `EnsureRequirements` only treats two key-grouped children as co-partitioned when the parent is Spark's own `SortMergeJoinExec` or `ShuffledHashJoinExec` (`checkKeyGroupCompatible`), and in 4.1 every leaf below them must also be a `KeyGroupedPartitionedScan` (`canPushDownSPJParamsToScan`). A `CometSortMergeJoinExec` over `CometIcebergNativeScanExec`s is neither, so it finds no usable shuffle spec and shuffles both sides whatever the scans report. With only the scan change, the new join test fails in the same way. An aggregate has one child, so `EnsureRequirements` only asks whether that child's partitioning satisfies the distribution. An aggregate above a storage-partitioned join does need the scans to report their partitioning. Without it the final aggregate gets an exchange between itself and the partial aggregate, inside a native block that was already serialized, and the native side fails with `Column index 1 is out of bound`. ## What changes are included in this PR? - `CometSortMergeJoinExec` and `CometHashJoinExec` declare no required child distribution when both sides are storage-partitioned, meaning every leaf is a key-grouped scan and there is no shuffle above it. No shuffle sits below such a join for AQE to split, so the requirement from #7231 protects nothing there. Every other join still returns the original operator's requirement. `CometExec.isStoragePartitioned` checks the join's current children, because conversion works bottom-up and `originalPlan.children` are already Comet nodes. It uses `DataSourceV2ScanExecBase.keyGroupedPartitioning`, which every supported Spark version declares, so it needs no shim even though Spark 4.2 renamed `KeyGroupedPartitioning` to `KeyedPartitioning`. - `CometIcebergNativeScanExec.outputPartitioning` reports whatever the `BatchScanExec` it replaces reports, and `UnknownPartitioning(0)` when that is unknown. This generalizes #7424. Reading it resolves no DPP subquery, because in Spark 3.5.9, 4.0.4, 4.1.3 and 4.2.0 `BatchScanExec.outputPartitioning` is built from `inputPartitions` and only `inputRDD` uses `filteredPartitions`. - Two tests in `CometIcebergNativeSuite`. One is the reproduction from the issue, with an assertion that neither the plan nor the subquery's plan has an exchange. The other puts an aggregate above the storage-partitioned join. ## How are these changes tested? On Spark 4.1: - The join test fails with only the scan change, and passes with the join change. - The aggregate test fails without the scan change, with `Column index 1 is out of bound`, and passes with it. - The 17 other `AQE DPP` tests in `CometIcebergNativeSuite` pass. The change compiles on Spark 3.5 with Scala 2.12, and its main sources compile on Spark 4.2. I have not run the tests on Spark 3.5 or 4.0. A join over Comet's non-native Iceberg reader is not covered. `CometExecRule` wraps `CometBatchScanExec` in a `CometScanWrapper` leaf, which `isStoragePartitioned` does not unwrap, so that join keeps the requirement from #7231. The tests also use only `CometSortMergeJoinExec`, through the `MERGE` hint. `CometHashJoinExec` shares the trait but has no test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
