Yaoxuan Wu created SPARK-61768:
----------------------------------

             Summary: DataFrame API: a column referenced inside a lambda 
function resolves to a same-named alias of the same select list instead of the 
input column
                 Key: SPARK-61768
                 URL: https://issues.apache.org/jira/browse/SPARK-61768
             Project: Spark
          Issue Type: Bug
          Components: Connect, PySpark, SQL
    Affects Versions: 4.1.1, 4.2.0, 4.0.0
            Reporter: Yaoxuan Wu


In the DataFrame API, a column referenced inside a lambda function resolves to 
a same-named alias defined earlier in the same select list (implicit lateral 
column alias), not to the input column. Outside a lambda, the same reference 
resolves to the input column, and the equivalent SQL query is correct. The 
result is wrong when the types match; when they don't, the query fails analysis 
instead.

{code:python}
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()
df = spark.sql("SELECT 1 AS a, 100 AS b, array(1, 2) AS arr")
df.createOrReplaceTempView("t")

spark.sql("SELECT b AS a, transform(arr, x -> x + a) AS c FROM t").collect()
# [Row(a=100, c=[2, 3])]                      correct: a is the input column (1)

df.select(F.col("b").alias("a"), F.transform("arr", lambda x: x + 
F.col("a")).alias("c")).collect()
# [Row(a=100, c=[101, 102])]                  expected c=[2, 3]

df.withColumns({"a": F.col("b"), "c": F.transform("arr", lambda x: x + 
F.col("a"))}).collect()
# [Row(a=100, b=100, arr=[1, 2], c=[101, 102])]   expected c=[2, 3]

df.withColumns({"a": F.col("b"), "c": F.col("a") + 0}).collect()
# [Row(a=100, b=100, arr=[1, 2], c=1)]        correct outside a lambda

m = spark.sql("SELECT map(1, 10) AS m, 7 AS b")
m.withColumns({"m": F.col("b"),
               "c": F.map_filter("m", lambda k, v: 
F.array_contains(F.map_keys(F.col("m")), k))}).collect()
# [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve 
"map_keys(lateralAliasReference(m))" ...
{code}

Same on 3.5.8, 4.0.0, 4.1.1 and 4.2.0 (classic), and on 4.2.0 with Spark 
Connect. Lateral column aliases should apply only to names the input does not 
have, and {{withColumns}} is documented to evaluate every expression against 
the input DataFrame.

Analyzed plan of the {{select}} above: the lambda body refers to the new alias 
{{a#6}}, not the input {{a#0}}:
{noformat}
Project [a#6, transform(arr#2, lambdafunction((lambda x_1#8 + a#6), lambda 
x_1#8, false)) AS c#7]
+- Project [a#0, b#1, arr#2, b#1 AS a#6]
   +- Project [1 AS a#0, 100 AS b#1, array(1, 2) AS arr#2]
{noformat}

Cause (master d3f4a73a, ColumnResolutionHelper.scala):
1. {{resolveExpression}} skips unbound lambda functions ({{case f: 
LambdaFunction if !f.bound => f}}). An input column used inside a lambda body 
therefore stays an {{UnresolvedAttribute}} until {{ResolveLambdaVariables}} 
binds the lambda.
2. In the same {{ResolveReferences}} pass, {{resolveLateralColumnAlias}} walks 
the whole select list with {{transformUp}} and does not skip lambdas. It 
matches that {{UnresolvedAttribute}} to the earlier alias {{b AS a}} and wraps 
it in a {{LateralColumnAliasReference}}, before the reference has been tried 
against the input.
3. SQL escapes because the parser builds every name in a lambda body as an 
{{UnresolvedNamedLambdaVariable}}: the parsed plan has {{lambda 'a}}, which 
{{ResolveLambdaVariables}} later resolves against the input. The Column API 
builds an ordinary {{UnresolvedAttribute}} ({{'a}}).

A possible fix is to make {{resolveLateralColumnAlias}} skip unbound 
{{LambdaFunction}} bodies, as {{resolveExpression}} does. A lateral alias 
inside a lambda would then only be used if the name doesn't resolve against the 
input once the lambda is bound.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to