[
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-61211:
------------------------------
Affects Version/s: (was: 4.4.0)
Description:
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto
the union's output by zipping child.output with union.output into an
AttributeMap. Two cases leave a child attribute that does not map correctly,
and the union can then advertise a layout its rows do not have.
EnsureRequirements omits a necessary exchange, and window, aggregate or join
results are wrong. No error is raised.
*Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the
first column carrying an ExprId, while AttributeMap keeps the last pair for a
repeated key. If the child output repeats an ExprId, the child's partitioning
is bound to its first occurrence but mapped from its last. If the union output
repeats an ExprId, a child attribute can map to a position a parent never
binds. Duplicate ExprIds reach a union only with
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
"false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
"false")
spark.conf.set("spark.sql.codegen.wholeStage", "false")
val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1) // expected 1000, actual 4582
{code}
*Case 2: an attribute the child does not output.* An unmapped attribute is left
unchanged and can equal another child's real output attribute. Object operators
such as MapGroupsExec and SerializeFromObjectExec pass their child's
partitioning through while outputting only the object column. This needs no
non-default conf, since spark.sql.unionOutputPartitioning has defaulted to true
since 4.1.0.
{code:scala}
spark.conf.set("spark.sql.adaptive.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
.collect().count(_.getInt(1) == 1) // expected 11, actual 19
{code}
The union reports hash(value#K) over a child whose only column is value#V, so
the window's exchange is omitted.
*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with
default true, and every release since then has the same AttributeMap mapping.
All three reproductions above give the same wrong results on branch-4.1
(4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with AQE on there;
case 1 is wrong with AQE on and off. 4.0 does not have this feature.
Disabling spark.sql.unionOutputPartitioning avoids both cases.
Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741
was:
UnionExec.prepareOutputPartitioning can report an incorrect hash layout when a
child output repeats an exprId. Runtime BindReferences selects the first
occurrence, while AttributeMap(child.output.zip(union.output)) retains the last
occurrence. A union can consequently advertise clustering on a column whose
rows were actually shuffled by another column, causing EnsureRequirements to
omit a necessary exchange.
This is independent of SPARK-60379's shuffle push-down. Reviewers reproduced
the existing bug on merge base 352ac9bf228 with push-down off, and with AQE
both on and off. Disabling spark.sql.unionOutputPartitioning restores the
expected result. The union output partitioning mapping dates back to
SPARK-52921 and shipped in 4.1.0; the two analyzer confs this shape needs were
also added in 4.1.0, so 4.1.0 is the earliest affected release.
Reported reproduction (Scala):
{code:scala}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window
import spark.implicits._
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
"false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.unionOutputPartitioning", "true")
// Test harness controls; avoid resolver dual-run validation and codegen
compilation of duplicates.
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
"false")
spark.conf.set("spark.sql.codegen.wholeStage", "false")
val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1)
{code}
Expected: 1000. Reported actual: 4582. Collect all columns before counting to
retain the triggering plan shape.
The duplicate second child reports hash(p), physically bound to its first
column. Its mapping into the outer union reports hash(c), corresponding to the
second column. Both outer children then appear to report hash(c), although
their rows are not co-located by c.
Investigate a conservative refusal to propagate partitioning from ambiguous
child outputs, or a mapping aligned with first-occurrence runtime binding, with
coverage for both mapping directions and all supported partitioning types. Keep
this general UnionExec fix separate from the narrower push-down eligibility
guards in SPARK-60379.
Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741
Summary: UnionExec reports a wrong output partitioning when a
child's partitioning references an attribute that does not map to the union
output (was: Avoid incorrect union partitioning with duplicate child
expression IDs)
> UnionExec reports a wrong output partitioning when a child's partitioning
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61211
> URL: https://issues.apache.org/jira/browse/SPARK-61211
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Reporter: XiDuo You
> Priority: Major
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto
> the union's output by zipping child.output with union.output into an
> AttributeMap. Two cases leave a child attribute that does not map correctly,
> and the union can then advertise a layout its rows do not have.
> EnsureRequirements omits a necessary exchange, and window, aggregate or join
> results are wrong. No error is raised.
> *Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the
> first column carrying an ExprId, while AttributeMap keeps the last pair for a
> repeated key. If the child output repeats an ExprId, the child's partitioning
> is bound to its first occurrence but mapped from its last. If the union
> output repeats an ExprId, a child attribute can map to a position a parent
> never binds. Duplicate ExprIds reach a union only with
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
> "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
> "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
> "false")
> spark.conf.set("spark.sql.codegen.wholeStage", "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
> row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1) // expected 1000, actual 4582
> {code}
> *Case 2: an attribute the child does not output.* An unmapped attribute is
> left unchanged and can equal another child's real output attribute. Object
> operators such as MapGroupsExec and SerializeFromObjectExec pass their
> child's partitioning through while outputting only the object column. This
> needs no non-default conf, since spark.sql.unionOutputPartitioning has
> defaulted to true since 4.1.0.
> {code:scala}
> spark.conf.set("spark.sql.adaptive.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
> Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
> .collect().count(_.getInt(1) == 1) // expected 11, actual 19
> {code}
> The union reports hash(value#K) over a child whose only column is value#V, so
> the window's exchange is omitted.
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0
> with default true, and every release since then has the same AttributeMap
> mapping. All three reproductions above give the same wrong results on
> branch-4.1 (4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with
> AQE on there; case 1 is wrong with AQE on and off. 4.0 does not have this
> feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]