This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 4f14839c2e65 [SPARK-60509][SQL][PYTHON] Fix Python UDFs in nested
lambdas iterating another UDF's result
4f14839c2e65 is described below
commit 4f14839c2e658cc904c42954bfd1675ce370398d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 09:37:41 2026 -0700
[SPARK-60509][SQL][PYTHON] Fix Python UDFs in nested lambdas iterating
another UDF's result
### What changes were proposed in this pull request?
This PR aims to fix `ExtractPythonUDFs` to chain two element-wise Python
UDFs only when they have the same `elementwiseNestingDepth`.
SPARK-58728 (#57940) lets `ExtractPythonUDFFromLambda` lift a Python UDF
out of a nested higher-order function lambda into an element-wise UDF over the
whole array. `elementwiseNestingDepth` is the number of array levels that the
Python worker flattens before it calls the function. When a UDF in a nested
lambda iterates over the result of another UDF, the two lifted UDFs have
different depths.
```
transform(values, x -> transform(f(x), y -> g(y)))
f: depth 1 over `values`, because `f(x)` is the argument of the inner
`transform`
g: depth 2 over the result of `f`, because `g` is in the inner lambda
```
Previously, `ExtractPythonUDFs.shouldExtractUDFExpressionTree` chained `g`
with its child `f` because they have the same eval type. However, the Python
worker evaluates a chain as one composed function, `g(f(...))`, and
`ArrowEvalPythonExec` sends only the depth of the outermost UDF of each chain
(`udfs.map(_.elementwiseNestingDepth)`). So, the worker flattened the input of
`f` two levels instead of one.
With this PR, `g` is evaluated by a separate `ArrowEvalPython` over the
result column of `f`. This also avoids evaluating `f` again inside the chain.
### Why are the changes needed?
This is a correctness issue. Depending on the input, the query fails or
returns wrong results silently.
For example, a natural query like `transform(docs, d ->
transform(tokenize(d), t -> normalize(t)))` fails.
```python
from pyspark.sql import functions as sf
from pyspark.sql.functions import udf
df = spark.createDataFrame([(["Hello World", "Foo"],)], "docs
array<string>")
tokenize = udf(lambda d: d.split(" "), "array<string>")
normalize = udf(lambda t: t.lower(), "string")
df.select(
sf.transform("docs", lambda d: sf.transform(tokenize(d), lambda t:
normalize(t))).alias("r")
).collect()
```
**BEFORE**
```
pyspark.errors.exceptions.captured.PythonException: An exception was thrown
from the Python worker:
Traceback (most recent call last):
...
File ".../pyspark/eval_handlers/_elementwise.py", line 83, in
_elementwise_input_schema
assert isinstance(data_type, ArrayType)
AssertionError
```
**AFTER**
```
[Row(r=[['hello', 'world'], ['foo']])]
```
The same query fails in the same way with `udf(..., useArrow=False)`,
`pandas_udf`, and `arrow_udf`.
When the outer array element is itself an array, `f` receives the leaf
values instead of each element, so the query can return wrong results silently.
```python
df = spark.createDataFrame([([[1, 2], [3]],)], "values array<array<int>>")
repeat = udf(lambda a: a + a, "array<int>")
plus_one = udf(lambda y: y + 1, "int")
df.select(
sf.transform("values", lambda row: sf.transform(repeat(row), lambda y:
plus_one(y))).alias("r")
).collect()
```
**BEFORE**
```
[Row(r=[[3, 5, None, None], [7, None]])]
```
**AFTER**
```
[Row(r=[[2, 3, 2, 3], [4, 4]])]
```
### Does this PR introduce _any_ user-facing change?
No. This fixes a bug in SPARK-58728, which is not released yet. It exists
only in `master` and `branch-4.x`.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `ExtractPythonUDFFromLambdaSuite` checks that `f` and `g` are evaluated
by two `ArrowEvalPython` operators, and that UDFs of the same depth are still
chained.
- `test_udf_in_higher_order_function.py` checks the result of the first
example with `udf`, `pandas_udf`, and `arrow_udf`, and the result of the second
example with `udf`. Its Spark Connect parity suite runs the same tests.
Without this fix, the new Scala test fails with `List(List(("f", 1), ("g",
2))) did not equal List(List(("g", 2)), List(("f", 1)))`, and the new Python
tests fail with the `AssertionError` and the wrong results above.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59751 from dongjoon-hyun/SPARK-60509.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit b2a1dfdb6452b124e8cf9aee04363e2aa7380b5e)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../sql/tests/test_udf_in_higher_order_function.py | 74 ++++++++++++++++++++++
.../sql/execution/python/ExtractPythonUDFs.scala | 4 ++
.../python/ExtractPythonUDFFromLambdaSuite.scala | 26 ++++++++
3 files changed, 104 insertions(+)
diff --git a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
index 6ac4ba9df73e..e06fc733a760 100644
--- a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
+++ b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
@@ -636,6 +636,80 @@ class UDFInHigherOrderFunctionTestsMixin:
native,
)
+ def test_udf_in_nested_lambda_over_udf_result(self):
+ # SPARK-60509: in `transform(docs, d -> transform(tokenize(d), t ->
normalize(t)))`,
+ # `tokenize(d)` is the inner function's argument, so it lifts one
array level deep over
+ # `docs`, while `normalize` lifts two levels deep over `tokenize`'s
result. The two must
+ # not be chained into one worker function, which would flatten `docs`
two levels.
+ # Compared against the equivalent native expression for the
row-at-a-time, pandas and
+ # Arrow flavors.
+ import pandas as pd
+ import pyarrow as pa
+ import pyarrow.compute as pc
+
+ from pyspark.sql.functions import arrow_udf, pandas_udf
+
+ df = self.spark.createDataFrame(
+ [(["Hello World", "Foo"],), ([],), (None,), ([None, "A b"],)],
"docs array<string>"
+ )
+
+ @pandas_udf(ArrayType(StringType()))
+ def tokenize_pandas(s: pd.Series) -> pd.Series:
+ return s.str.split(" ")
+
+ @pandas_udf(StringType())
+ def normalize_pandas(s: pd.Series) -> pd.Series:
+ return s.str.lower()
+
+ @arrow_udf(ArrayType(StringType()))
+ def tokenize_arrow(a: pa.Array) -> pa.Array:
+ return pc.split_pattern(a, " ")
+
+ @arrow_udf(StringType())
+ def normalize_arrow(a: pa.Array) -> pa.Array:
+ return pc.utf8_lower(a)
+
+ native = df.select(
+ sf.transform(
+ "docs", lambda d: sf.transform(sf.split(d, " "), lambda t:
sf.lower(t))
+ ).alias("r")
+ )
+ for tokenize, normalize in (
+ (
+ udf(lambda d: None if d is None else d.split(" "),
ArrayType(StringType())),
+ udf(lambda t: t.lower(), StringType()),
+ ),
+ (tokenize_pandas, normalize_pandas),
+ (tokenize_arrow, normalize_arrow),
+ ):
+ assertDataFrameEqual(
+ df.select(
+ sf.transform(
+ "docs", lambda d: sf.transform(tokenize(d), lambda t:
normalize(t))
+ ).alias("r")
+ ),
+ native,
+ )
+
+ def test_udf_in_nested_lambda_over_udf_result_of_array_element(self):
+ # SPARK-60509: as above, but the outer element is itself an array. A
chained call would
+ # pass the leaves to `repeat` instead of each row; `a + a` also runs
on an int, so the
+ # query would silently return wrong values rather than fail.
+ df = self.spark.createDataFrame(
+ [([[1, 2], [3]],), ([[], None],), (None,)], "values
array<array<int>>"
+ )
+ repeat = udf(lambda a: None if a is None else a + a,
ArrayType(IntegerType()))
+ plus_one = udf(lambda y: y + 1, IntegerType())
+
+ assertDataFrameEqual(
+ df.select(
+ sf.transform(
+ "values", lambda row: sf.transform(repeat(row), lambda y:
plus_one(y))
+ ).alias("r")
+ ),
+ [([[2, 3, 2, 3], [4, 4]],), ([[], None],), (None,)],
+ )
+
def test_nested_lambda_with_nondeterministic_inner_argument_fails(self):
# The inner iterated argument is nondeterministic, so the rewrite -
which references it more
# than once - would evaluate it independently and misalign the
results. It must keep failing
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
index 052949dd6146..a784718084f2 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
@@ -188,6 +188,9 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with
Logging {
* - PythonUDF (bar)
* - PythonUDF (baz)
* - if the eval types of the UDF expressions in the chain differ, return
false.
+ * - if the element-wise nesting depths of the UDF expressions in the chain
differ, return false.
+ * The worker runs a chain as one function under the outermost UDF's
depth. In
+ * transform(arr, x -> transform(f(x), y -> g(y))), g lifts to depth 2 and
f to depth 1.
* - if a UDF has more than one child, e.g. foo(bar(), baz()), return false
* If we return false here, the expectation is that the recursive calls of
* collectEvaluableUDFsFromExpressions will then visit the children and
extract them first to
@@ -204,6 +207,7 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with
Logging {
PythonUDF.chainsInWorker(e) &&
correctEvalType(e, pythonUDFArrowFallbackOnUDT) ==
correctEvalType(child, pythonUDFArrowFallbackOnUDT) &&
+ e.elementwiseNestingDepth == child.elementwiseNestingDepth &&
shouldExtractUDFExpressionTree(child, pythonUDFArrowFallbackOnUDT)
// Python UDF can't be evaluated directly in JVM
case children => !children.exists(hasScalarPythonUDF)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
index b59d26a1f9ea..44852ce8ec8a 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
@@ -277,6 +277,32 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest
with SharedSparkSession
assert(liftedUDFs(df).size == 1)
}
+ test("SPARK-60509: lifted UDFs of different nesting depths are not chained")
{
+ // `transform(values, x -> transform(f(x), y -> g(y)))`: `f(x)` is the
inner function's
+ // argument, so it lifts to depth 1 over `values`, while `g` lifts onto
`f(x)` and then again
+ // onto `values`, ending at depth 2 over `f`'s lifted result. A chain
`g(f(...))` runs in the
+ // worker as one function under the outer UDF's depth, so `g` must instead
be evaluated by a
+ // later operator over `f`'s result column.
+ val f = UserDefinedPythonFunction("f", new DummyUDF,
ArrayType(IntegerType),
+ PythonEvalType.SQL_BATCHED_UDF, udfDeterministic = true)
+ val g = UserDefinedPythonFunction("g", new DummyUDF, IntegerType,
+ PythonEvalType.SQL_BATCHED_UDF, udfDeterministic = true)
+ val df = arrayDF.select(transform(col("values"), x => transform(f(x), y =>
g(y))).as("r"))
+ assert(udfsInsideLambda(df).isEmpty)
+ // `collect` is pre-order, so the upper operator comes first.
+ val evals = df.queryExecution.optimizedPlan.collect { case a:
ArrowEvalPython => a }
+ assert(evals.map(_.udfs.map(u => u.name -> u.elementwiseNestingDepth)) ==
+ Seq(Seq("g" -> 2), Seq("f" -> 1)))
+ assert(evals.head.udfs.head.children == evals.last.resultAttrs)
+
+ // Lifted UDFs of the same depth are still chained, here both at depth 2.
+ val sameDepth = Seq(Seq(Seq(1, 2), Seq(3))).toDF("values")
+ .select(transform(col("values"), row => transform(row, x =>
g(g(x)))).as("r"))
+ val lifted = liftedUDFs(sameDepth)
+ assert(lifted.map(_.elementwiseNestingDepth).toSet == Set(2))
+ assert(lifted.exists(_.children.exists(_.isInstanceOf[PythonUDF])))
+ }
+
test("a lambda with no Python UDF is left unchanged") {
val df = arrayDF.select(transform(col("values"), x => x + lit(1)).as("r"))
val analyzed = df.queryExecution.analyzed
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]