[
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-61211:
------------------------------
Affects Version/s: 4.4.0
> UnionExec reports a wrong output partitioning when a child's partitioning
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61211
> URL: https://issues.apache.org/jira/browse/SPARK-61211
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.4.0
> Reporter: XiDuo You
> Priority: Major
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto
> the union's output by zipping child.output with union.output into an
> AttributeMap. Two cases leave a child attribute that does not map correctly,
> and the union can then advertise a layout its rows do not have.
> EnsureRequirements omits a necessary exchange, and window, aggregate or join
> results are wrong. No error is raised.
> *Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the
> first column carrying an ExprId, while AttributeMap keeps the last pair for a
> repeated key. If the child output repeats an ExprId, the child's partitioning
> is bound to its first occurrence but mapped from its last. If the union
> output repeats an ExprId, a child attribute can map to a position a parent
> never binds. Duplicate ExprIds reach a union only with
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
> "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
> "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
> "false")
> spark.conf.set("spark.sql.codegen.wholeStage", "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
> row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1) // expected 1000, actual 4582
> {code}
> *Case 2: an attribute the child does not output.* An unmapped attribute is
> left unchanged and can equal another child's real output attribute. Object
> operators such as MapGroupsExec and SerializeFromObjectExec pass their
> child's partitioning through while outputting only the object column. This
> needs no non-default conf, since spark.sql.unionOutputPartitioning has
> defaulted to true since 4.1.0.
> {code:scala}
> spark.conf.set("spark.sql.adaptive.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
> Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
> .collect().count(_.getInt(1) == 1) // expected 11, actual 19
> {code}
> The union reports hash(value#K) over a child whose only column is value#V, so
> the window's exchange is omitted.
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0
> with default true, and every release since then has the same AttributeMap
> mapping. All three reproductions above give the same wrong results on
> branch-4.1 (4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with
> AQE on there; case 1 is wrong with AQE on and off. 4.0 does not have this
> feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]