This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new b2a1dfdb6452 [SPARK-60509][SQL][PYTHON] Fix Python UDFs in nested 
lambdas iterating another UDF's result
b2a1dfdb6452 is described below

commit b2a1dfdb6452b124e8cf9aee04363e2aa7380b5e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 09:37:41 2026 -0700

    [SPARK-60509][SQL][PYTHON] Fix Python UDFs in nested lambdas iterating 
another UDF's result
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `ExtractPythonUDFs` to chain two element-wise Python 
UDFs only when they have the same `elementwiseNestingDepth`.
    
    SPARK-58728 (#57940) lets `ExtractPythonUDFFromLambda` lift a Python UDF 
out of a nested higher-order function lambda into an element-wise UDF over the 
whole array. `elementwiseNestingDepth` is the number of array levels that the 
Python worker flattens before it calls the function. When a UDF in a nested 
lambda iterates over the result of another UDF, the two lifted UDFs have 
different depths.
    
    ```
    transform(values, x -> transform(f(x), y -> g(y)))
    
    f: depth 1 over `values`, because `f(x)` is the argument of the inner 
`transform`
    g: depth 2 over the result of `f`, because `g` is in the inner lambda
    ```
    
    Previously, `ExtractPythonUDFs.shouldExtractUDFExpressionTree` chained `g` 
with its child `f` because they have the same eval type. However, the Python 
worker evaluates a chain as one composed function, `g(f(...))`, and 
`ArrowEvalPythonExec` sends only the depth of the outermost UDF of each chain 
(`udfs.map(_.elementwiseNestingDepth)`). So, the worker flattened the input of 
`f` two levels instead of one.
    
    With this PR, `g` is evaluated by a separate `ArrowEvalPython` over the 
result column of `f`. This also avoids evaluating `f` again inside the chain.
    
    ### Why are the changes needed?
    
    This is a correctness issue. Depending on the input, the query fails or 
returns wrong results silently.
    
    For example, a natural query like `transform(docs, d -> 
transform(tokenize(d), t -> normalize(t)))` fails.
    
    ```python
    from pyspark.sql import functions as sf
    from pyspark.sql.functions import udf
    
    df = spark.createDataFrame([(["Hello World", "Foo"],)], "docs 
array<string>")
    tokenize = udf(lambda d: d.split(" "), "array<string>")
    normalize = udf(lambda t: t.lower(), "string")
    df.select(
        sf.transform("docs", lambda d: sf.transform(tokenize(d), lambda t: 
normalize(t))).alias("r")
    ).collect()
    ```
    
    **BEFORE**
    
    ```
    pyspark.errors.exceptions.captured.PythonException: An exception was thrown 
from the Python worker:
    Traceback (most recent call last):
      ...
      File ".../pyspark/eval_handlers/_elementwise.py", line 83, in 
_elementwise_input_schema
        assert isinstance(data_type, ArrayType)
    AssertionError
    ```
    
    **AFTER**
    
    ```
    [Row(r=[['hello', 'world'], ['foo']])]
    ```
    
    The same query fails in the same way with `udf(..., useArrow=False)`, 
`pandas_udf`, and `arrow_udf`.
    
    When the outer array element is itself an array, `f` receives the leaf 
values instead of each element, so the query can return wrong results silently.
    
    ```python
    df = spark.createDataFrame([([[1, 2], [3]],)], "values array<array<int>>")
    repeat = udf(lambda a: a + a, "array<int>")
    plus_one = udf(lambda y: y + 1, "int")
    df.select(
        sf.transform("values", lambda row: sf.transform(repeat(row), lambda y: 
plus_one(y))).alias("r")
    ).collect()
    ```
    
    **BEFORE**
    
    ```
    [Row(r=[[3, 5, None, None], [7, None]])]
    ```
    
    **AFTER**
    
    ```
    [Row(r=[[2, 3, 2, 3], [4, 4]])]
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This fixes a bug in SPARK-58728, which is not released yet. It exists 
only in `master` and `branch-4.x`.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases.
    
    - `ExtractPythonUDFFromLambdaSuite` checks that `f` and `g` are evaluated 
by two `ArrowEvalPython` operators, and that UDFs of the same depth are still 
chained.
    - `test_udf_in_higher_order_function.py` checks the result of the first 
example with `udf`, `pandas_udf`, and `arrow_udf`, and the result of the second 
example with `udf`. Its Spark Connect parity suite runs the same tests.
    
    Without this fix, the new Scala test fails with `List(List(("f", 1), ("g", 
2))) did not equal List(List(("g", 2)), List(("f", 1)))`, and the new Python 
tests fail with the `AssertionError` and the wrong results above.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59751 from dongjoon-hyun/SPARK-60509.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../sql/tests/test_udf_in_higher_order_function.py | 74 ++++++++++++++++++++++
 .../sql/execution/python/ExtractPythonUDFs.scala   |  4 ++
 .../python/ExtractPythonUDFFromLambdaSuite.scala   | 26 ++++++++
 3 files changed, 104 insertions(+)

diff --git a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py 
b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
index 6ac4ba9df73e..e06fc733a760 100644
--- a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
+++ b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
@@ -636,6 +636,80 @@ class UDFInHigherOrderFunctionTestsMixin:
                 native,
             )
 
+    def test_udf_in_nested_lambda_over_udf_result(self):
+        # SPARK-60509: in `transform(docs, d -> transform(tokenize(d), t -> 
normalize(t)))`,
+        # `tokenize(d)` is the inner function's argument, so it lifts one 
array level deep over
+        # `docs`, while `normalize` lifts two levels deep over `tokenize`'s 
result. The two must
+        # not be chained into one worker function, which would flatten `docs` 
two levels.
+        # Compared against the equivalent native expression for the 
row-at-a-time, pandas and
+        # Arrow flavors.
+        import pandas as pd
+        import pyarrow as pa
+        import pyarrow.compute as pc
+
+        from pyspark.sql.functions import arrow_udf, pandas_udf
+
+        df = self.spark.createDataFrame(
+            [(["Hello World", "Foo"],), ([],), (None,), ([None, "A b"],)], 
"docs array<string>"
+        )
+
+        @pandas_udf(ArrayType(StringType()))
+        def tokenize_pandas(s: pd.Series) -> pd.Series:
+            return s.str.split(" ")
+
+        @pandas_udf(StringType())
+        def normalize_pandas(s: pd.Series) -> pd.Series:
+            return s.str.lower()
+
+        @arrow_udf(ArrayType(StringType()))
+        def tokenize_arrow(a: pa.Array) -> pa.Array:
+            return pc.split_pattern(a, " ")
+
+        @arrow_udf(StringType())
+        def normalize_arrow(a: pa.Array) -> pa.Array:
+            return pc.utf8_lower(a)
+
+        native = df.select(
+            sf.transform(
+                "docs", lambda d: sf.transform(sf.split(d, " "), lambda t: 
sf.lower(t))
+            ).alias("r")
+        )
+        for tokenize, normalize in (
+            (
+                udf(lambda d: None if d is None else d.split(" "), 
ArrayType(StringType())),
+                udf(lambda t: t.lower(), StringType()),
+            ),
+            (tokenize_pandas, normalize_pandas),
+            (tokenize_arrow, normalize_arrow),
+        ):
+            assertDataFrameEqual(
+                df.select(
+                    sf.transform(
+                        "docs", lambda d: sf.transform(tokenize(d), lambda t: 
normalize(t))
+                    ).alias("r")
+                ),
+                native,
+            )
+
+    def test_udf_in_nested_lambda_over_udf_result_of_array_element(self):
+        # SPARK-60509: as above, but the outer element is itself an array. A 
chained call would
+        # pass the leaves to `repeat` instead of each row; `a + a` also runs 
on an int, so the
+        # query would silently return wrong values rather than fail.
+        df = self.spark.createDataFrame(
+            [([[1, 2], [3]],), ([[], None],), (None,)], "values 
array<array<int>>"
+        )
+        repeat = udf(lambda a: None if a is None else a + a, 
ArrayType(IntegerType()))
+        plus_one = udf(lambda y: y + 1, IntegerType())
+
+        assertDataFrameEqual(
+            df.select(
+                sf.transform(
+                    "values", lambda row: sf.transform(repeat(row), lambda y: 
plus_one(y))
+                ).alias("r")
+            ),
+            [([[2, 3, 2, 3], [4, 4]],), ([[], None],), (None,)],
+        )
+
     def test_nested_lambda_with_nondeterministic_inner_argument_fails(self):
         # The inner iterated argument is nondeterministic, so the rewrite - 
which references it more
         # than once - would evaluate it independently and misalign the 
results. It must keep failing
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
index 052949dd6146..a784718084f2 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
@@ -188,6 +188,9 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with 
Logging {
    *         - PythonUDF (bar)
    *           - PythonUDF (baz)
    * - if the eval types of the UDF expressions in the chain differ, return 
false.
+   * - if the element-wise nesting depths of the UDF expressions in the chain 
differ, return false.
+   *   The worker runs a chain as one function under the outermost UDF's 
depth. In
+   *   transform(arr, x -> transform(f(x), y -> g(y))), g lifts to depth 2 and 
f to depth 1.
    * - if a UDF has more than one child, e.g. foo(bar(), baz()), return false
    * If we return false here, the expectation is that the recursive calls of
    * collectEvaluableUDFsFromExpressions will then visit the children and 
extract them first to
@@ -204,6 +207,7 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with 
Logging {
         PythonUDF.chainsInWorker(e) &&
           correctEvalType(e, pythonUDFArrowFallbackOnUDT) ==
           correctEvalType(child, pythonUDFArrowFallbackOnUDT) &&
+          e.elementwiseNestingDepth == child.elementwiseNestingDepth &&
           shouldExtractUDFExpressionTree(child, pythonUDFArrowFallbackOnUDT)
       // Python UDF can't be evaluated directly in JVM
       case children => !children.exists(hasScalarPythonUDF)
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
index b59d26a1f9ea..44852ce8ec8a 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
@@ -277,6 +277,32 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest 
with SharedSparkSession
     assert(liftedUDFs(df).size == 1)
   }
 
+  test("SPARK-60509: lifted UDFs of different nesting depths are not chained") 
{
+    // `transform(values, x -> transform(f(x), y -> g(y)))`: `f(x)` is the 
inner function's
+    // argument, so it lifts to depth 1 over `values`, while `g` lifts onto 
`f(x)` and then again
+    // onto `values`, ending at depth 2 over `f`'s lifted result. A chain 
`g(f(...))` runs in the
+    // worker as one function under the outer UDF's depth, so `g` must instead 
be evaluated by a
+    // later operator over `f`'s result column.
+    val f = UserDefinedPythonFunction("f", new DummyUDF, 
ArrayType(IntegerType),
+      PythonEvalType.SQL_BATCHED_UDF, udfDeterministic = true)
+    val g = UserDefinedPythonFunction("g", new DummyUDF, IntegerType,
+      PythonEvalType.SQL_BATCHED_UDF, udfDeterministic = true)
+    val df = arrayDF.select(transform(col("values"), x => transform(f(x), y => 
g(y))).as("r"))
+    assert(udfsInsideLambda(df).isEmpty)
+    // `collect` is pre-order, so the upper operator comes first.
+    val evals = df.queryExecution.optimizedPlan.collect { case a: 
ArrowEvalPython => a }
+    assert(evals.map(_.udfs.map(u => u.name -> u.elementwiseNestingDepth)) ==
+      Seq(Seq("g" -> 2), Seq("f" -> 1)))
+    assert(evals.head.udfs.head.children == evals.last.resultAttrs)
+
+    // Lifted UDFs of the same depth are still chained, here both at depth 2.
+    val sameDepth = Seq(Seq(Seq(1, 2), Seq(3))).toDF("values")
+      .select(transform(col("values"), row => transform(row, x => 
g(g(x)))).as("r"))
+    val lifted = liftedUDFs(sameDepth)
+    assert(lifted.map(_.elementwiseNestingDepth).toSet == Set(2))
+    assert(lifted.exists(_.children.exists(_.isInstanceOf[PythonUDF])))
+  }
+
   test("a lambda with no Python UDF is left unchanged") {
     val df = arrayDF.select(transform(col("values"), x => x + lit(1)).as("r"))
     val analyzed = df.queryExecution.analyzed


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to