Yaoxuan Wu created SPARK-61768:
----------------------------------
Summary: DataFrame API: a column referenced inside a lambda
function resolves to a same-named alias of the same select list instead of the
input column
Key: SPARK-61768
URL: https://issues.apache.org/jira/browse/SPARK-61768
Project: Spark
Issue Type: Bug
Components: Connect, PySpark, SQL
Affects Versions: 4.1.1, 4.2.0, 4.0.0
Reporter: Yaoxuan Wu
In the DataFrame API, a column referenced inside a lambda function resolves to
a same-named alias defined earlier in the same select list (implicit lateral
column alias), not to the input column. Outside a lambda, the same reference
resolves to the input column, and the equivalent SQL query is correct. The
result is wrong when the types match; when they don't, the query fails analysis
instead.
{code:python}
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.getOrCreate()
df = spark.sql("SELECT 1 AS a, 100 AS b, array(1, 2) AS arr")
df.createOrReplaceTempView("t")
spark.sql("SELECT b AS a, transform(arr, x -> x + a) AS c FROM t").collect()
# [Row(a=100, c=[2, 3])] correct: a is the input column (1)
df.select(F.col("b").alias("a"), F.transform("arr", lambda x: x +
F.col("a")).alias("c")).collect()
# [Row(a=100, c=[101, 102])] expected c=[2, 3]
df.withColumns({"a": F.col("b"), "c": F.transform("arr", lambda x: x +
F.col("a"))}).collect()
# [Row(a=100, b=100, arr=[1, 2], c=[101, 102])] expected c=[2, 3]
df.withColumns({"a": F.col("b"), "c": F.col("a") + 0}).collect()
# [Row(a=100, b=100, arr=[1, 2], c=1)] correct outside a lambda
m = spark.sql("SELECT map(1, 10) AS m, 7 AS b")
m.withColumns({"m": F.col("b"),
"c": F.map_filter("m", lambda k, v:
F.array_contains(F.map_keys(F.col("m")), k))}).collect()
# [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve
"map_keys(lateralAliasReference(m))" ...
{code}
Same on 3.5.8, 4.0.0, 4.1.1 and 4.2.0 (classic), and on 4.2.0 with Spark
Connect. Lateral column aliases should apply only to names the input does not
have, and {{withColumns}} is documented to evaluate every expression against
the input DataFrame.
Analyzed plan of the {{select}} above: the lambda body refers to the new alias
{{a#6}}, not the input {{a#0}}:
{noformat}
Project [a#6, transform(arr#2, lambdafunction((lambda x_1#8 + a#6), lambda
x_1#8, false)) AS c#7]
+- Project [a#0, b#1, arr#2, b#1 AS a#6]
+- Project [1 AS a#0, 100 AS b#1, array(1, 2) AS arr#2]
{noformat}
Cause (master d3f4a73a, ColumnResolutionHelper.scala):
1. {{resolveExpression}} skips unbound lambda functions ({{case f:
LambdaFunction if !f.bound => f}}). An input column used inside a lambda body
therefore stays an {{UnresolvedAttribute}} until {{ResolveLambdaVariables}}
binds the lambda.
2. In the same {{ResolveReferences}} pass, {{resolveLateralColumnAlias}} walks
the whole select list with {{transformUp}} and does not skip lambdas. It
matches that {{UnresolvedAttribute}} to the earlier alias {{b AS a}} and wraps
it in a {{LateralColumnAliasReference}}, before the reference has been tried
against the input.
3. SQL escapes because the parser builds every name in a lambda body as an
{{UnresolvedNamedLambdaVariable}}: the parsed plan has {{lambda 'a}}, which
{{ResolveLambdaVariables}} later resolves against the input. The Column API
builds an ordinary {{UnresolvedAttribute}} ({{'a}}).
A possible fix is to make {{resolveLateralColumnAlias}} skip unbound
{{LambdaFunction}} bodies, as {{resolveExpression}} does. A lateral alias
inside a lambda would then only be used if the name doesn't resolve against the
input once the lambda is bound.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]