dwsmith1983 opened a new pull request, #7624: URL: https://github.com/apache/datafusion-comet/pull/7624
## Which issue does this PR close? Backport of #6268 to `branch-1.1`. Part of #6133. #7623 must merge first: one of the tests here needs it. This PR and #7338 (the `branch-1.1` backport of #6270, the other half of #6133) don't depend on each other; whichever merges second should take `branch-1.1` and rerun its suites. `branch-1.2` already has #6268. `branch-1.0` has the same code; no backport is planned there. ## Rationale for this change AQE reads `outputPartitioning` when `CoalesceShufflePartitions` changes a stage, before the DPP placeholders are converted. On `branch-1.1`, a non-bucketed `CometNativeScanExec` answers with its split count, which runs the placeholder and fails with `SubqueryAdaptiveBroadcastExec does not support the execute() code path`. `CometIcebergNativeScanExec` fails the same way. ## What changes are included in this PR? Cherry-pick of 695ca3ab7e6edefffffd6e073069561e66086c50 with `-x`. - A non-bucketed native scan and the Iceberg scan report `UnknownPartitioning(0)`. A bucketed scan keeps its bucket layout. - `CometNativeExec` sizes their native RDD from the plan data at execution. Adaptations: the `CometTaskMetricsSuite` change is dropped, because the test it edits is not on this branch, and two suites import what the new tests use, which `main` already had. ## How are these changes tested? The source PR's three tests. In a Linux container on Spark 4.1, `CometExecSuite` and `CometIcebergNativeSuite` pass apart from the bucketed test, plus the scan, join and DPP suites. On Spark 3.5 the AQE DPP tests in both suites pass apart from the same test. With `spark/src/main` reverted, both shuffle coalescing tests fail with the errors above. The bucketed test fails on `branch-1.1` with or without this change, on an empty bucket's scan schema, which #7623 fixes. With both applied, all three tests pass on Spark 4.1 and 3.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
