LuciferYang commented on PR #58419: URL: https://github.com/apache/spark/pull/58419#issuecomment-5810737230
Pushed the two AQE cases the test thread asked for, and the preparation-time caveat on `spark.sql.unionOutputPartitioning`. On validation reading a fresh union: which of the two reads is the permissive one decides what is at stake. With the conf on at validation and off in the snapshot, nothing above the union depends on its partitioning, because preparation saw it plain and `EnsureRequirements` put an exchange wherever a parent required a distribution; a rewrite that `ValidateRequirements` keeps cannot remove that exchange. With the conf off at validation and on in the snapshot, validation sees `UnknownPartitioning`, reads the parent's requirement as unmet, and `optimizeQueryStage` falls back to the plan before the injected rule. That costs a rewrite, not a layout the union will not execute. On sequential columnar rules: there is no seam to list a pass into. The injected rules share one `ApplyColumnarRulesAndInsertTransitions`, which is a single rule from the outside, so recording between two of them means changing that rule. The hook is also not where exchanges are planned: under AQE it runs after the stage is created, without AQE after `EnsureRequirements`, so nothing downstream would reconcile an exchange added or dropped there. What a read between two columnar rules can still move is what the codegen gate answers, and the barrier for that lands before `CollapseCodegenStages`, so the gate and the copy in the shell agree either way. That is the boundary written on `SnapshotUnionPreparationConf`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
