[
https://issues.apache.org/jira/browse/SPARK-61768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61768:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> DataFrame API: a column referenced inside a lambda function resolves to a
> same-named alias of the same select list instead of the input column
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61768
> URL: https://issues.apache.org/jira/browse/SPARK-61768
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark, SQL
> Affects Versions: 4.0.0, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> In the DataFrame API, a column referenced inside a lambda function resolves
> to a same-named alias defined earlier in the same select list (implicit
> lateral column alias), not to the input column. Outside a lambda, the same
> reference resolves to the input column, and the equivalent SQL query is
> correct. The result is wrong when the types match; when they don't, the query
> fails analysis instead.
> {code:python}
> from pyspark.sql import SparkSession, functions as F
> spark = SparkSession.builder.getOrCreate()
> df = spark.sql("SELECT 1 AS a, 100 AS b, array(1, 2) AS arr")
> df.createOrReplaceTempView("t")
> spark.sql("SELECT b AS a, transform(arr, x -> x + a) AS c FROM t").collect()
> # [Row(a=100, c=[2, 3])] correct: a is the input column
> (1)
> df.select(F.col("b").alias("a"), F.transform("arr", lambda x: x +
> F.col("a")).alias("c")).collect()
> # [Row(a=100, c=[101, 102])] expected c=[2, 3]
> df.withColumns({"a": F.col("b"), "c": F.transform("arr", lambda x: x +
> F.col("a"))}).collect()
> # [Row(a=100, b=100, arr=[1, 2], c=[101, 102])] expected c=[2, 3]
> df.withColumns({"a": F.col("b"), "c": F.col("a") + 0}).collect()
> # [Row(a=100, b=100, arr=[1, 2], c=1)] correct outside a lambda
> m = spark.sql("SELECT map(1, 10) AS m, 7 AS b")
> m.withColumns({"m": F.col("b"),
> "c": F.map_filter("m", lambda k, v:
> F.array_contains(F.map_keys(F.col("m")), k))}).collect()
> # [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve
> "map_keys(lateralAliasReference(m))" ...
> {code}
> Same on 3.5.8, 4.0.0, 4.1.1 and 4.2.0 (classic), and on 4.2.0 with Spark
> Connect. Lateral column aliases should apply only to names the input does not
> have, and {{withColumns}} is documented to evaluate every expression against
> the input DataFrame.
> Analyzed plan of the {{select}} above: the lambda body refers to the new
> alias {{a#6}}, not the input {{a#0}}:
> {noformat}
> Project [a#6, transform(arr#2, lambdafunction((lambda x_1#8 + a#6), lambda
> x_1#8, false)) AS c#7]
> +- Project [a#0, b#1, arr#2, b#1 AS a#6]
> +- Project [1 AS a#0, 100 AS b#1, array(1, 2) AS arr#2]
> {noformat}
> Cause (master d3f4a73a, ColumnResolutionHelper.scala):
> 1. {{resolveExpression}} skips unbound lambda functions ({{case f:
> LambdaFunction if !f.bound => f}}). An input column used inside a lambda body
> therefore stays an {{UnresolvedAttribute}} until {{ResolveLambdaVariables}}
> binds the lambda.
> 2. In the same {{ResolveReferences}} pass, {{resolveLateralColumnAlias}}
> walks the whole select list with {{transformUp}} and does not skip lambdas.
> It matches that {{UnresolvedAttribute}} to the earlier alias {{b AS a}} and
> wraps it in a {{LateralColumnAliasReference}}, before the reference has been
> tried against the input.
> 3. SQL escapes because the parser builds every name in a lambda body as an
> {{UnresolvedNamedLambdaVariable}}: the parsed plan has {{lambda 'a}}, which
> {{ResolveLambdaVariables}} later resolves against the input. The Column API
> builds an ordinary {{UnresolvedAttribute}} ({{'a}}).
> A possible fix is to make {{resolveLateralColumnAlias}} skip unbound
> {{LambdaFunction}} bodies, as {{resolveExpression}} does. A lateral alias
> inside a lambda would then only be used if the name doesn't resolve against
> the input once the lambda is bound.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]