Peter Toth created SPARK-61379:
----------------------------------

             Summary: RemoveRedundantAliases leaves a Union unresolved when a 
later child aliases one column twice under its own name
                 Key: SPARK-61379
                 URL: https://issues.apache.org/jira/browse/SPARK-61379
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Peter Toth


With the default configs, this query fails in the optimizer:

{code:scala}
val a = spark.range(100).selectExpr("id % 7 k", "id % 10 v").repartition($"v")
val b = spark.range(50).selectExpr("id % 10 x").repartition($"x")
  .selectExpr("x AS x", "x AS x")
a.union(b).collect()
{code}

{code}
org.apache.spark.SparkException: [PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule 
org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAliases in batch 
Operator Optimization before Inferring Filters generated an invalid plan: The 
plan was previously resolved and now became unresolved. SQLSTATE: XXKD0
{code}

The second child of the union is {{Project [x#4 AS x#5, x#4 AS x#6]}}. Both 
aliases look redundant to {{RemoveRedundantAliases}}, since each has the name 
of the attribute it aliases. In a union child other than the first one, 
{{shouldRemoveAliasesUnderUnion}} (SPARK-53308) keeps an alias only if its 
attribute is already in the project's output. {{x#4}} is not, so both aliases 
are removed and the child becomes {{Project [x#4, x#4]}}.

With {{spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved}} on 
(the default, SPARK-52686), a union child with duplicate attributes leaves the 
{{Union}} unresolved. The lightweight plan change validation, which is on by 
default, then fails the query.

The first child is not affected, because its output is excluded from the 
removal.

A fix is to also keep an alias when removing it would repeat an attribute in 
the project list, for example when two aliases share the same attribute.

Reproduced on master at 4686647b3a1.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to