LuciferYang opened a new pull request, #59208: URL: https://github.com/apache/spark/pull/59208
### What changes were proposed in this pull request? This backports SPARK-59122 (#58419, in master and branch-4.x; #59151 for branch-4.3) to branch-4.2. It is cherry-picked from the branch-4.3 commit. The change makes `UnionExec` decide once, right after `EnsureRequirements`, whether it is a plain concatenation, and freeze that decision plus the codegen confs in a `TreeNodeTag`, so the fusion gate, `metrics`, `unionRDDs` and the copy of the union inside the codegen shell can no longer give different answers depending on when each is read. See #58419 for the full design. The conflicts against branch-4.2 all come from code the branch does not have. None of them changes the fix: - branch-4.2 does not have SPARK-57881 or SPARK-58317, so `UnionExec.outputPartitioning` is still the SPARK-52921 body: remap every child's partitioning to the first child's attributes, require them all to match, and lift the match to the union's output. The backport keeps that body under the name `rawPartitioning` and makes the same SPARK-59122 edit as on branch-4.3: the `spark.sql.unionOutputPartitioning` check moves out of it and into the stamped decision. Doc comment sentences about `KeyedPartitioning` and about intersecting candidates are dropped or reworded, since neither exists here. - The AQE query-stage preparation list on branch-4.2 ends with `OptimizeSkewedJoin`, so the trailing union barrier is appended after it. The snapshot and the stamp around `ensureRequirements` sit exactly where they sit on master. - `applyPhysicalRules` on branch-4.2 takes the plan change logger as `Some((planChangeLogger, name))`. The post-planner-strategy call keeps that argument. - `UnionCodegenSuite` still extends `QueryTest` here, which it keeps, and `AdaptiveQueryExecSuite` does not have the SPARK-57996 test the hunk was anchored next to. One test needed a change. `spark.sql.codegen.wholeStage.union.enabled` defaults to false on branch-4.2 (true on branch-4.3 and master), and "the codegen conf is recorded for a prep rule after the one that added it" in `SparkSessionExtensionSuite` inherited the default. Its observer reads the codegen gate with that conf flipped off, so the conf has to start on for a recorded value to differ from a live one. The test now sets it on its session. ### Why are the changes needed? SPARK-59122 is a Blocker correctness fix, and branch-4.2 hits the wrong-result case on default settings, because `spark.sql.unionOutputPartitioning` defaults to true there. The repartition-union-aggregate query from #58419, with the conf turned off after `executedPlan` was first read, returns ten rows of count 4 on branch-4.2 instead of five rows of count 8. The codegen crash needs union fusion, which is off by default on branch-4.2. With `spark.sql.codegen.wholeStage.union.enabled=true`, the cached-aggregate query from #58419 fails on branch-4.2 with `java.util.NoSuchElementException: key not found: numOutputRows`. ### Does this PR introduce _any_ user-facing change? Yes, the same change as #58419. The wrong-result query above now returns the correct groups, and with union fusion enabled the cached-aggregate query returns rows instead of failing. The three confs the decision reads no longer reach a plan that has already been prepared, only plans prepared afterwards. ### How was this patch tested? The tests come with the cherry-pick, unchanged from #58419 apart from the `SparkSessionExtensionSuite` case described above. The AQE rule-order case asserts positions relative to `EnsureRequirements` and to the end of the list, so it holds on the shorter branch-4.2 list without editing. The two queries from #58419 were also run by hand against branch-4.2 with and without this change. Without it they give the results quoted above; with it, five rows of count 8, and 20 rows from the cached-aggregate query with fusion on and off. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
