dongjoon-hyun opened a new pull request, #59749:
URL: https://github.com/apache/spark/pull/59749
### What changes were proposed in this pull request?
This PR aims to make `HigherOrderFunction.canonicalized` renumber only the
lambda variables that are bound in the higher-order function, i.e. the
arguments of its lambda functions, including those of nested higher-order
functions. A captured lambda variable, which is bound by an enclosing lambda
function, now keeps its own `exprId`.
As before, when the enclosing higher-order function is canonicalized, a
nested higher-order function skips the variables that the enclosing one has
already renamed.
### Why are the changes needed?
Since SPARK-35580, `HigherOrderFunction.canonicalized` renames every
`NamedLambdaVariable` in its lambda functions by position, including the
captured ones. As a result, `filter(ys, y -> y > x1)` and `filter(ys, y -> y >
x2)` have the same canonical form when `x1` and `x2` have the same data type
and nullability, and `semanticEquals` returns `true` for them. The optimizer
rules that use `semanticEquals`, such as `SimplifyConditionals`,
`BooleanSimplification`, and `SimplifyBinaryComparison`, then simplify lambda
function bodies incorrectly.
```sql
CREATE TABLE t USING parquet AS
SELECT array(1, 5) AS xs1, array(3, 2) AS xs2, array(1, 2, 3, 4, 5) AS ys;
SELECT zip_with(xs1, xs2, (x1, x2) ->
IF(x1 > x2, filter(ys, y -> y > x1), filter(ys, y -> y > x2))) FROM t;
```
| Lambda function body of `zip_with(xs1, xs2, (x1, x2) -> ...)` | Expected |
Apache Spark 4.0.4, 4.1.3, 4.2.0, 4.3.0 |
| --- | --- | --- |
| `IF(x1 > x2, filter(ys, y -> y > x1), filter(ys, y -> y > x2))` |
`[[4,5],[]]` | `[[2,3,4,5],[]]` |
| `CASE WHEN x1 > x2 THEN filter(ys, y -> y > x1) ELSE filter(ys, y -> y >
x2) END` | `[[4,5],[]]` | `[[4,5],[3,4,5]]` |
| `exists(ys, y -> y > x1) OR exists(ys, y -> y > x2)` | `[true,true]` |
`[true,false]` |
| `filter(ys, y -> y > x1) <=> filter(ys, y -> y > x2)` | `[false,false]` |
`[true,true]` |
For example, `SimplifyConditionals` replaces the whole `IF` with its first
branch.
```
Project [zip_with(xs1#3, xs2#4, lambdafunction(filter(ys#5,
lambdafunction((lambda y#8 > lambda x1#6), lambda y#8, false)), lambda x1#6,
lambda x2#7, false)) AS ...]
+- Relation spark_catalog.default.t[xs1#3,xs2#4,ys#5] parquet
```
A `LocalRelation` source such as `VALUES` hides this bug because
`ConvertToLocalRelation` evaluates the query before these rules run.
In addition, `ExtractPythonUDFFromLambda`, which exists only in the
unreleased `master` and `branch-4.x`, identifies a Python UDF call in a lambda
function by its canonical form. So, it merges two calls that capture different
lambda variables into one.
```python
from pyspark.sql import functions as sf
@sf.udf("int")
def f(xs):
return len(xs)
df = spark.range(1).selectExpr(
"array(1, 2) AS l", "array(10, 20) AS r", "array(0, 5, 15) AS c")
df.select(sf.zip_with("l", "r", lambda a, b:
f(sf.filter("c", lambda y: y < a)) + f(sf.filter("c", lambda y: y <
b)))).collect()
# [3, 4] is expected, but [2, 2] is returned.
```
### Does this PR introduce _any_ user-facing change?
Yes. This fixes the wrong query results that have existed since Apache Spark
3.2.0 ([v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0),
2021-10-06). Two higher-order functions that capture different lambda variables
are no longer semantically equal, so the optimizer does not treat them as the
same expression anymore. This also fixes the Python UDF example above in the
unreleased `master` and `branch-4.x`.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `HigherOrderFunctionsSuite`: `semanticEquals` returns `false` for
higher-order functions that capture different lambda variables, and `true` for
identical ones that differ only in the IDs of their own lambda variables,
including nested higher-order functions.
- `DataFrameFunctionsSuite`: the four queries above on a Parquet table.
Without the fix, both new test cases fail. I also ran the Python example
above manually. It returns `[2, 2]` without the fix and `[3, 4]` with the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
