andygrove opened a new pull request, #7510: URL: https://github.com/apache/datafusion-comet/pull/7510
## Which issue does this PR close? Backport of #7231 to `branch-1.2`. Closes #7208 on this branch. ## Rationale for this change With AQE on, `REBALANCE(k)` followed by `GROUP BY k` returns duplicate groups when the data is skewed, and a `PARTITION BY k` window restarts `row_number()` inside a key (#7208, a correctness bug). Comet's aggregate, window and join operators declared no `requiredChildDistribution`, so Spark's `OptimizeSkewInRebalancePartitions` could split a skewed partition under them. `branch-1.2` has the same operators and the same gap. See #7231 for the full analysis. ## What changes are included in this PR? Cherry-pick of ce3e665e84 (`git cherry-pick -x`) with no conflicts and no adaptations. The diff against the source commit is identical apart from line numbers and blob hashes. It adds `CometDelegatesRequiredChildDistribution` and mixes it into the Comet aggregate, window, window group limit and join operators. Other release branches: `branch-1.2` is the newest release branch, so there is no newer one to port to. Whether `branch-1.1` needs a backport is not part of this PR. ## How are these changes tested? On this branch, with JDK 17 and a native library built from `branch-1.2`: the new `CometExecSuite` test (`AQE does not split a skewed rebalance partition under operators that need clustering`) passes. I confirmed it fails with the four main-source files reverted to `branch-1.2`'s version, so the branch has the bug and the test catches it. I did not rerun the wider Comet suites that the source PR ran. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
