[
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You resolved SPARK-61211.
-------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 60271
[https://github.com/apache/spark/pull/60271]
> UnionExec reports a wrong output partitioning when a child's partitioning
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61211
> URL: https://issues.apache.org/jira/browse/SPARK-61211
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.4.0
> Reporter: XiDuo You
> Assignee: XiDuo You
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto
> the union's output by zipping child.output with union.output into an
> AttributeMap. When a child attribute does not map correctly, the union can
> advertise a layout its rows do not have. EnsureRequirements then omits a
> necessary exchange, and window, aggregate or join results are wrong. No error
> is raised.
> *Case 1 (default configs): a partitioning references an attribute the child
> does not output.* An unmapped attribute was left unchanged and can equal
> another child's real output attribute. A column-pruned InMemoryTableScanExec
> reports the cached plan's partitioning over columns it no longer outputs, and
> MapGroupsExec and SerializeFromObjectExec pass the grouping key's
> partitioning through while outputting only the mapped value.
> {code:scala}
> val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
> val c = kv.keys.toDF().repartition($"_1").cache()
> c.count()
> c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
> // expected 10, actual 38 (AQE on and off)
> {code}
> {code:scala}
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
> Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
> .collect().count(_.getInt(1) == 1) // expected 11, actual 19
> {code}
> With AQE on and partition coalescing on, the second query's few rows are
> coalesced into one partition, which hides the wrong result.
> *Case 2 (non-default configs): duplicate ExprIds.* Runtime binding
> (AttributeSeq.indexOf) picks the first column carrying an ExprId, while
> AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach
> a union only with
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
> "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
> "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
> "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
> row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1) // expected 1000, actual 4582
> {code}
> Case 2 is fixed by SPARK-61356.
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0
> with default true, and every release since then has the same AttributeMap
> mapping. The mapGroups and duplicate-ExprId reproductions give the same wrong
> results on branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition
> coalescing is off. 4.0 does not have this feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741
> https://github.com/apache/spark/pull/60271#discussion_r4522188035
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]