Peter Toth created SPARK-61379:
----------------------------------
Summary: RemoveRedundantAliases leaves a Union unresolved when a
later child aliases one column twice under its own name
Key: SPARK-61379
URL: https://issues.apache.org/jira/browse/SPARK-61379
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
With the default configs, this query fails in the optimizer:
{code:scala}
val a = spark.range(100).selectExpr("id % 7 k", "id % 10 v").repartition($"v")
val b = spark.range(50).selectExpr("id % 10 x").repartition($"x")
.selectExpr("x AS x", "x AS x")
a.union(b).collect()
{code}
{code}
org.apache.spark.SparkException: [PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAliases in batch
Operator Optimization before Inferring Filters generated an invalid plan: The
plan was previously resolved and now became unresolved. SQLSTATE: XXKD0
{code}
The second child of the union is {{Project [x#4 AS x#5, x#4 AS x#6]}}. Both
aliases look redundant to {{RemoveRedundantAliases}}, since each has the name
of the attribute it aliases. In a union child other than the first one,
{{shouldRemoveAliasesUnderUnion}} (SPARK-53308) keeps an alias only if its
attribute is already in the project's output. {{x#4}} is not, so both aliases
are removed and the child becomes {{Project [x#4, x#4]}}.
With {{spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved}} on
(the default, SPARK-52686), a union child with duplicate attributes leaves the
{{Union}} unresolved. The lightweight plan change validation, which is on by
default, then fails the query.
The first child is not affected, because its output is excluded from the
removal.
A fix is to also keep an alias when removing it would repeat an attribute in
the project list, for example when two aliases share the same attribute.
Reproduced on master at 4686647b3a1.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]