XiDuo You created SPARK-61209:
---------------------------------
Summary: Snapshot the confs that physical output partitioning
depends on once per query preparation
Key: SPARK-61209
URL: https://issues.apache.org/jira/browse/SPARK-61209
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
Some physical operators derive their outputPartitioning from a conf that is
read when the node is built or when the getter is called, not once per query
preparation. A plan prepared under one value can then execute nodes rebuilt
under another, for example the copies AQE makes with withNewChildren when it
creates query stages. A parent that was planned against the old answer (a
partitioning-aware UnionExec, an aggregate without an exchange) then runs over
children that report a different partitioning.
Known shapes:
- spark.sql.optimizer.expressionProjectionCandidateLimit
(AliasAwareOutputExpression.aliasCandidateLimit), with AQE on: a union of JOIN
USING legs (inner, or FULL OUTER after SPARK-60904), each leg's alias Project
rebuilt at stage creation, then a GROUP BY k above the union. Preparing,
setting the limit to 0 and collecting returns duplicated groups (19 rows vs 28
in the SPARK-60904 suite data).
- spark.sql.execution.broadcastHashJoin.outputPartitioningExpandLimit, read on
BroadcastHashJoinExec.outputPartitioning, with AQE off and on.
UnionExec already records its confs once per preparation
(SnapshotUnionPreparationConf). A shared per-preparation snapshot of the confs
that physical output partitioning depends on would cover these.
Raised in https://github.com/apache/spark/pull/60118#discussion_r4500201123.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
