This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 88145e6b48e7 [SPARK-27052][SQL][PYTHON][FOLLOWUP] Fix issues in 
lifting Python UDFs out of higher-order function lambdas
88145e6b48e7 is described below

commit 88145e6b48e7a371e39581ae0cde2d2e22ad03ab
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Oct 23 13:35:12 2026 +0900

    [SPARK-27052][SQL][PYTHON][FOLLOWUP] Fix issues in lifting Python UDFs out 
of higher-order function lambdas
    
    ### What changes were proposed in this pull request?
    
    This PR fixes four issues in `ExtractPythonUDFFromLambda`, the optimizer 
rule that lifts Python UDFs out of higher-order function lambdas (SPARK-27052):
    
    1. **Deduplication ignored argument determinism.** Whether two UDF calls 
are the same call is now decided by one helper, 
`PythonUDF.canonicalizeDeterministic`. It checks `deterministic` instead of the 
UDF's own `udfDeterministic` flag, and `deterministic` also covers the 
arguments. This rule uses it for its call keys and for the signature that lets 
identical calls share one lifted UDF, and `ExtractPythonUDFs` uses it to 
deduplicate the calls it extracts. So calls that this rule keeps  [...]
    2. **A key-form `array_sort` comparator read a different draw on each 
side.** In `array_sort(arr, (a, b) -> k(a) - k(b))`, the lifted `k` is each 
element's sort key. In such a comparator, calls that lift to the same function 
over the same array now share one lifted UDF even when they are 
nondeterministic, so each element has one key. Outside comparators, 
nondeterministic calls stay distinct.
    3. **Nested lifted calls were substituted bottom-up.** `overArray` replaces 
each already-lifted call inside a composite argument, such as `g(x)` in `f(g(x) 
+ 1)`, with a variable that stands for that call's array result. It now does 
this top-down. It also zips only the arrays the argument reads: the argument of 
`f(g(x) + 1)` is now a `transform` over `g`'s result alone, instead of over 
`arrays_zip(values, g(values))`.
    4. **The rule ran too late.** `ExtractPythonUDFFromLambda` is now its own 
rule. It runs first in the "Extract UDFs" batch and is listed in 
`nonExcludableRules`. Before, `ExtractPythonUDFs` invoked it internally, after 
`ExtractPythonUDFFromJoinCondition`, `ExtractPythonUDFFromAggregate` and 
`ExtractGroupingPythonUDFFromAggregate` had already run.
    
    ### Why are the changes needed?
    
    **1. Wrong results.** A deterministic UDF over a nondeterministic argument 
was keyed by its canonical form. Canonicalization drops every `PythonUDF`'s 
`resultId`, so the two inner draws below could no longer be told apart, and 
both sides read one lifted call:
    
    ```python
    draw = udf(lambda x: uuid.uuid4().hex, "string").asNondeterministic()
    keep = udf(lambda s: s, "string")
    
    df.select(F.transform("values", lambda x: keep(draw(x)) == keep(draw(x))))
    # before: [true, true, true]
    # after:  [false, false, false]
    ```
    
    Without the higher-order function, the same comparison already returns 
`false`, and so does `draw(x) == draw(x)` inside the lambda. The same applies 
to `F.transform("values", lambda x: (lambda r: keep(r) == keep(r))(draw(x)))`. 
PySpark runs the Python lambda once while building the plan, so `r` is not 
bound to one draw per element. It is inlined as two `draw` calls, and the 
result changes from `true` to `false`, in both classic and Connect, as it 
already was without the higher-order f [...]
    
    **2. Sort failure.** A comparator has to compare an element by the same key 
whichever side of the comparison it is on. With one draw per call, a 
nondeterministic key UDF gave each element one key as `a` and another as `b`, 
and in the usual `when` form a different key in each branch. The comparator 
then did not define an order:
    
    ```python
    key = udf(lambda x: random.randint(0, 1_000_000), 
"int").asNondeterministic()
    df.select(F.array_sort(
        "values",
        lambda a, b: F.when(key(a) < key(b), -1).when(key(a) > key(b), 
1).otherwise(0)))
    # before, for an array of 1000 elements:
    #   java.lang.IllegalArgumentException: Comparison method violates its 
general contract!
    # after: the elements in a random order
    ```
    
    On master, `keep(draw(a)) - keep(draw(b))` happened to share one lifted 
call, through the canonicalization that fix 1 removes, while a bare `draw(a) - 
draw(b)` never did. Both now share one draw per element. A comparator whose 
single call takes both elements, `(a, b) -> udf(a, b)`, is unchanged: it has no 
per-element key to share, so a nondeterministic `udf` still gets one draw per 
ordered pair.
    
    **3. Runtime failure, or double evaluation.** For `f(g(h(x)) + 1)`, 
bottom-up substitution replaced `h(x)` first. The rewritten `g(...)` then no 
longer matched the lifted result of `g(h(x))`, and a raw `g` was left inside 
the lambda generated for `f`'s argument:
    
    - If that lambda could be rewritten again, `g` was lifted a second time and 
ran twice per element.
    - If it could not, the `PythonUDF` stayed inside the lambda and the query 
failed. This happens, for example, when `h` is nondeterministic, because a 
lambda that iterates a nondeterministic array is not rewritable:
    
    ```python
    same = udf(lambda x: x, "int").asNondeterministic()
    df.select(F.transform("values", lambda x: times_ten(plus_one(same(x)) + 1)))
    # before: [INTERNAL_ERROR] Cannot generate code for expression:
    #         <lambda>(transform(arrays_zip(...), 
lambdafunction((<lambda>(lambda z.c1) + 1), ...)))
    # after:  [30, 40, 50] for values [1, 2, 3]
    ```
    
    Zipping in arrays that the argument does not read only cost an extra struct 
per element, or more with `zip_with` or an index parameter.
    
    **4. UDFs placed in the wrong operator.** The rules that ran before the 
lift decide where a Python UDF is evaluated based on the columns it references. 
Inside a lambda, a UDF references only the lambda's variables, so those rules 
did not see it. It was lifted afterwards, and `ExtractPythonUDFs` then placed 
it on its own:
    
    ```sql
    SELECT transform(collect_list(id), x -> plus_one(x)) FROM range(3)
    -- before: [INTERNAL_ERROR] Cannot evaluate expression: collect_list(...)
    --         (the lifted UDF, with collect_list(id) as its argument, was 
evaluated below the Aggregate)
    -- after:  [1, 2, 3]
    ```
    
    ```python
    df.groupBy("values").agg(F.transform("values", lambda x: plus_one(x)))
    # before: [INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND] Could not find pythonUDF0 in 
[values]
    
    left.join(right, (left.k == right.k) & F.exists(left["values"], lambda x: 
plus_one(x) > 3),
              "left_outer")
    # before: [UNSUPPORTED_FEATURE.PYTHON_UDF_IN_ON_CLAUSE] ... Python UDF in 
the ON clause of a
    #         LEFT OUTER JOIN
    ```
    
    When it is lifted first, each of these is an ordinary UDF over a whole 
array. Those rules already handle that the same way as a plain UDF: it is 
evaluated above the aggregate, or on the one join side it reads. The ordering 
is now also explicit in `SparkOptimizer`, instead of a comment explaining that 
`ExtractPythonUDFs` does the lifting internally.
    
    This also moves a lambda UDF that reads one side of an inner join. In `l.k 
= r.k AND exists(l.values, x -> f(x) > r.t)` the conjunct reads both sides, so 
it stays in the join condition. Before, `f(x)` referenced only its lambda 
variable, so `ExtractPythonUDFFromJoinCondition` moved the conjunct into a 
`Filter` above the join, and `f` ran only on joined rows. Now the lifted `f` 
reads only `l.values`, and `ExtractPythonUDFs` evaluates it below the join on 
every row of the left side. Thi [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    No for released versions. SPARK-27052 is only in `master` and `branch-4.x` 
and not released yet. On those branches, the queries above now return correct 
results, or run instead of failing, and a lambda UDF that reads one side of an 
inner join is placed as described above. Plans without a Python UDF in a lambda 
are unaffected: the lambda rule does not change them, and the other rules run 
in the same order as before.
    
    ### How was this patch tested?
    
    New tests in `ExtractPythonUDFFromLambdaSuite`, each of which fails on 
master:
    - "a deterministic UDF over a nondeterministic argument is not deduplicated"
    - "a key-form array_sort comparator shares a nondeterministic call between 
its sides"
    - "a lifted call two levels deep inside a composite argument is substituted 
whole"
    - "a composite argument over a nested call zips only the arrays it reads"
    - "a UDF in a lambda over an aggregate's result is evaluated above the 
aggregate"
    - "a UDF in a lambda over one side of an outer join condition is evaluated 
on that side"
    - "a UDF in a lambda over one side of an inner join condition is evaluated 
on that side"
    
    New tests in `pyspark.sql.tests.test_udf_in_higher_order_function` and its 
Connect parity suite:
    - `test_deterministic_udf_over_nondeterministic_udf_calls_are_distinct`, 
`test_array_sort_with_nondeterministic_udf_key`, 
`test_nested_udf_two_levels_inside_composite_argument`, 
`test_udf_in_lambda_over_aggregate_result` and 
`test_udf_in_lambda_in_outer_join_condition` fail on master.
    - `test_udf_in_lambda_in_inner_join_condition`, and the index case added to 
`test_nested_udf_inside_composite_argument`, check the results of the new plan 
shapes.
    
    Also ran all `org.apache.spark.sql.execution.python.*` suites, the classic 
and Connect `test_udf_in_higher_order_function` suites, 
`pyspark.sql.tests.test_udf` and 
`pyspark.sql.tests.arrow.test_arrow_python_udf`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59701 from HyukjinKwon/SPARK-27052-followup-hof-udf-fixes.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
    (cherry picked from commit b19dd54a05291bb5eaa012e7622289edf6807768)
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 .../sql/tests/test_udf_in_higher_order_function.py | 141 +++++++++++++++++++++
 .../spark/sql/catalyst/expressions/PythonUDF.scala |  21 ++-
 .../spark/sql/execution/SparkOptimizer.scala       |  13 +-
 .../python/ExtractPythonUDFFromLambda.scala        | 117 +++++++++--------
 .../sql/execution/python/ExtractPythonUDFs.scala   |  20 +--
 .../python/ExtractPythonUDFFromLambdaSuite.scala   | 136 ++++++++++++++++++--
 6 files changed, 362 insertions(+), 86 deletions(-)

diff --git a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py 
b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
index b3479f4a2660..6ac4ba9df73e 100644
--- a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
+++ b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
@@ -152,6 +152,36 @@ class UDFInHigherOrderFunctionTestsMixin:
             df.select(sf.transform("values", lambda x: 
times_ten(-plus_one(x))).alias("r")),
             df.select(sf.transform("values", lambda x: (-(x + 1)) * 
10).alias("r")),
         )
+        # Reads the index and the inner result, but not the element itself.
+        assertDataFrameEqual(
+            df.select(sf.transform("values", lambda x, i: 
times_ten(plus_one(x) + i)).alias("r")),
+            df.select(sf.transform("values", lambda x, i: ((x + 1) + i) * 
10).alias("r")),
+        )
+
+    def test_nested_udf_two_levels_inside_composite_argument(self):
+        # `f(g(h(x)) + 1)`: the composite argument of `f` holds the lifted 
call `g(h(x))`, which
+        # itself holds the lifted call `h(x)`. `g(h(x))` must be substituted 
whole; a raw `g` left
+        # behind could not be lifted again over a nondeterministic `h` and 
failed at runtime.
+        import random
+
+        df = self.spark.createDataFrame([([1, 2, 3],)], "values array<int>")
+        plus_one = udf(lambda x: x + 1, IntegerType())
+        times_ten = udf(lambda x: x * 10, IntegerType())
+        # Nondeterministic, but returns its input unchanged so the expected 
result is known.
+        same = udf(lambda x: x + 0 * random.randint(0, 1), 
IntegerType()).asNondeterministic()
+
+        assertDataFrameEqual(
+            df.select(
+                sf.transform("values", lambda x: 
times_ten(plus_one(plus_one(x)) + 1)).alias("r")
+            ),
+            df.select(sf.transform("values", lambda x: ((x + 2) + 1) * 
10).alias("r")),
+        )
+        assertDataFrameEqual(
+            df.select(
+                sf.transform("values", lambda x: times_ten(plus_one(same(x)) + 
1)).alias("r")
+            ),
+            df.select(sf.transform("values", lambda x: ((x + 1) + 1) * 
10).alias("r")),
+        )
 
     def test_udf_with_outer_column_argument(self):
         # A non-element argument must be broadcast to every element of its row.
@@ -250,6 +280,38 @@ class UDFInHigherOrderFunctionTestsMixin:
             [([3, 2, 1],), ([],), (None,)],
         )
 
+    def test_array_sort_with_nondeterministic_udf_key(self):
+        # A per-element key is computed once per element, even by a 
nondeterministic UDF, so that
+        # an element has the same key on either side of a comparison. With a 
draw per call, the
+        # comparator would be inconsistent and the sort could fail with 
"Comparison method
+        # violates its general contract!".
+        import random
+
+        # Nondeterministic, but returns its input negated so the expected 
order is known.
+        negate = udf(lambda x: -x + 0 * random.randint(0, 1), 
IntegerType()).asNondeterministic()
+        keep = udf(lambda x: x, IntegerType())
+        df = self.spark.createDataFrame([([3, 1, 2],)], "values array<int>")
+        assertDataFrameEqual(
+            df.select(
+                sf.array_sort("values", lambda a, b: keep(negate(a)) - 
keep(negate(b))).alias("r")
+            ),
+            [([3, 2, 1],)],
+        )
+
+        # A random key: any order is valid, but the sort must succeed and keep 
every element.
+        key = udf(lambda x: random.randint(0, 1_000_000), 
IntegerType()).asNondeterministic()
+        values = list(range(1000))
+        df = self.spark.createDataFrame([(values,)], "values array<int>")
+        row = df.select(
+            sf.array_sort(
+                "values",
+                lambda a, b: sf.when(key(a) < key(b), sf.lit(-1))
+                .when(key(a) > key(b), sf.lit(1))
+                .otherwise(sf.lit(0)),
+            ).alias("r")
+        ).collect()[0]
+        self.assertEqual(sorted(row["r"]), values)
+
     def test_array_sort_pairwise_comparator(self):
         # One UDF call receiving both elements has no per-element key, so the 
UDF is precomputed
         # over every ordered pair and the comparator indexes that matrix. 
Assert an actual
@@ -354,6 +416,32 @@ class UDFInHigherOrderFunctionTestsMixin:
         # If the two calls were deduplicated, every element would be exactly 0.
         self.assertTrue(any(v != 0 for v in row["r"]), row["r"])
 
+    def 
test_deterministic_udf_over_nondeterministic_udf_calls_are_distinct(self):
+        # `keep` is deterministic, but each `keep(draw(x))` wraps its own 
independent draw, so the
+        # two sides must not be collapsed into one call compared with itself. 
The same holds when
+        # the calls are re-lifted out of a nested lambda.
+        import uuid
+
+        draw = udf(lambda x: uuid.uuid4().hex, 
StringType()).asNondeterministic()
+        keep = udf(lambda s: s, StringType())
+
+        df = self.spark.createDataFrame([([1, 2, 3],)], "values array<int>")
+        assertDataFrameEqual(
+            df.select(sf.transform("values", lambda x: keep(draw(x)) == 
keep(draw(x))).alias("r")),
+            [([False, False, False],)],
+        )
+
+        nested = self.spark.createDataFrame([([[1, 2], [3]],)], "values 
array<array<int>>")
+        assertDataFrameEqual(
+            nested.select(
+                sf.transform(
+                    "values",
+                    lambda a: sf.transform(a, lambda x: keep(draw(x)) == 
keep(draw(x))),
+                ).alias("r")
+            ),
+            [([[False, False], [False]],)],
+        )
+
     def test_fused_transforms_with_different_lengths(self):
         # SPARK-27052: two transforms over arrays of different per-row lengths 
and null layouts get
         # fused by ExtractPythonUDFs into one element-wise batch. Each UDF 
must be re-nested by its
@@ -714,6 +802,59 @@ class UDFInHigherOrderFunctionTestsMixin:
         # (1+1)+(2+1) + (3+1) = 9
         assertDataFrameEqual(grouped, [(9,)])
 
+    def test_udf_in_lambda_over_aggregate_result(self):
+        # Once lifted out of the lambda, the UDF reads the aggregate's result, 
so it must be
+        # evaluated above the aggregate. Below it, neither an aggregate 
function nor the grouped
+        # value of a grouping key exists, and the query failed at runtime.
+        df = self.spark.createDataFrame([(1, 1), (1, 2), (2, 3)], "k int, v 
int")
+        plus_one = udf(lambda x: x + 1, IntegerType())
+
+        assertDataFrameEqual(
+            df.groupBy("k").agg(
+                sf.sort_array(sf.transform(sf.collect_list("v"), lambda x: 
plus_one(x))).alias("r")
+            ),
+            [(1, [2, 3]), (2, [4])],
+        )
+        assertDataFrameEqual(
+            df.groupBy("k").agg(
+                sf.transform(sf.array(sf.sum("v")), lambda x: 
plus_one(x)).alias("r")
+            ),
+            [(1, [4]), (2, [4])],
+        )
+
+        arrays = self.spark.createDataFrame([([1, 2],), ([1, 2],), ([3],)], 
"values array<int>")
+        assertDataFrameEqual(
+            arrays.groupBy("values").agg(sf.transform("values", lambda x: 
plus_one(x)).alias("r")),
+            [([1, 2], [2, 3]), ([3], [4])],
+        )
+
+    def test_udf_in_lambda_in_outer_join_condition(self):
+        # A left outer join keeps a predicate over only its left side in the 
join condition. Once
+        # lifted, the UDF reads only the left side and is evaluated there, 
like a plain UDF.
+        left = self.spark.createDataFrame([(1, [1, 2]), (2, [3])], "k int, 
values array<int>")
+        right = self.spark.createDataFrame([(1, "a"), (2, "b")], "k int, v 
string")
+        plus_one = udf(lambda x: x + 1, IntegerType())
+
+        joined = left.join(
+            right,
+            (left.k == right.k) & sf.exists(left["values"], lambda x: 
plus_one(x) > 3),
+            "left_outer",
+        ).select(left.k, right.v)
+        assertDataFrameEqual(joined, [(1, None), (2, "b")])
+
+    def test_udf_in_lambda_in_inner_join_condition(self):
+        # The predicate reads both sides, so it stays in the join condition. 
Once lifted, the UDF
+        # reads only the left side and is evaluated there, as a plain one-side 
UDF is.
+        left = self.spark.createDataFrame([(1, [1, 2]), (2, [3])], "k int, 
values array<int>")
+        right = self.spark.createDataFrame([(1, 3), (2, 3), (2, 5)], "k int, t 
int")
+        plus_one = udf(lambda x: x + 1, IntegerType())
+
+        joined = left.join(
+            right,
+            (left.k == right.k) & sf.exists(left["values"], lambda x: 
plus_one(x) > right.t),
+        ).select(left.k, right.t)
+        assertDataFrameEqual(joined, [(2, 3)])
+
     def test_mixed_with_plain_python_udf(self):
         df = self.spark.createDataFrame([([1, 2],)], "values array<int>")
         plus_one = udf(lambda x: x + 1, IntegerType())
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
index 84b43f59631e..045301ff505e 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
@@ -70,6 +70,23 @@ object PythonUDF {
       e.asInstanceOf[PythonUDF].evalType != 
PythonEvalType.SQL_SCALAR_ARROW_INPROCESS_UDF
   }
 
+  /**
+   * The key that decides whether two Python UDF calls are the same call, 
evaluated once. A
+   * deterministic call is keyed by its canonical form, so identical calls 
share one evaluation. A
+   * nondeterministic call is keyed by itself: in `f(x) + f(x)` each call may 
return a different
+   * value, and only the calls' `resultId`s, which `canonicalized` drops, keep 
them apart.
+   *
+   * A call is deterministic only if its arguments are too (`deterministic`, 
not
+   * `udfDeterministic`). In `f(r(x)) = f(r(x))` with a nondeterministic `r`, 
each `f` call wraps
+   * its own draw of `r`; canonicalizing the `f` calls would drop the inner 
calls' `resultId`s as
+   * well and merge both sides into one call, compared with itself.
+   *
+   * Shared by `ExtractPythonUDFs` and `ExtractPythonUDFFromLambda`, so that 
calls the latter keeps
+   * apart when lifting them out of a lambda are not merged by the former 
afterwards.
+   */
+  def canonicalizeDeterministic(udf: PythonUDF): PythonUDF =
+    if (udf.deterministic) udf.canonicalized.asInstanceOf[PythonUDF] else udf
+
   /**
    * Whether `e` is a Python UDF that can be lifted out of a higher-order 
function's lambda by
    * `ExtractPythonUDFFromLambda`, which applies it to the whole array outside 
the lambda.
@@ -181,8 +198,8 @@ object PythonUDF {
    *     each lifted UDF's argument, the `map_keys`/`map_values` desugar, the 
pairwise `array_sort`
    *     path), and nondeterministic expressions are not 
subexpression-eliminated, so the copies
    *     would evaluate independently and disagree - keeping the results 
misaligned. (This is
-   *     distinct from a nondeterministic UDF *call*, which 
`ExtractPythonUDFFromLambda.liftKey`
-   *     keeps distinct but well-defined.)
+   *     distinct from a nondeterministic UDF *call*, which 
[[canonicalizeDeterministic]] keeps
+   *     distinct but well-defined.)
    *   - a HOF (at any level) whose shape the rewrite does not model (see 
[[isRewritableShape]]).
    */
   def canRewritePythonUDFInLambda(hof: HigherOrderFunction): Boolean = {
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
index 11db680fefb8..650ba5eefdb2 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
@@ -28,7 +28,7 @@ import 
org.apache.spark.sql.execution.datasources.v2.{GroupBasedRowLevelOperatio
 import 
org.apache.spark.sql.execution.dynamicpruning.{CleanupDynamicPruningFilters, 
PartitionPruning, RowLevelOperationRuntimeGroupFiltering}
 import 
org.apache.spark.sql.execution.externalUDF.{ExtractExternalUDFFromWindow, 
PlanExternalUDFs}
 import org.apache.spark.sql.execution.planmerging.MergeSubplans
-import 
org.apache.spark.sql.execution.python.{ExtractGroupingPythonUDFFromAggregate, 
ExtractPythonUDFFromAggregate, ExtractPythonUDFs, ExtractPythonUDTFs}
+import 
org.apache.spark.sql.execution.python.{ExtractGroupingPythonUDFFromAggregate, 
ExtractPythonUDFFromAggregate, ExtractPythonUDFFromLambda, ExtractPythonUDFs, 
ExtractPythonUDTFs}
 
 class SparkOptimizer(
     catalogManager: CatalogManager,
@@ -87,6 +87,10 @@ class SparkOptimizer(
       PruneFilters),
     postHocOptimizationBatches,
     Batch("Extract UDFs", Once,
+      // Lift Python UDFs out of higher-order function lambdas first. A UDF in 
a lambda reads the
+      // lambda's variables, so the rules below can place it by the columns it 
really reads (e.g.
+      // above an aggregate it is applied to) only once it is an ordinary UDF 
over a whole array.
+      ExtractPythonUDFFromLambda,
       ExtractPythonUDFFromJoinCondition,
       // Expose window results as attributes before creating external UDF 
evaluation nodes.
       ExtractExternalUDFFromWindow,
@@ -94,8 +98,6 @@ class SparkOptimizer(
       ExtractPythonUDFFromAggregate,
       // This must be executed after `ExtractPythonUDFFromAggregate` and 
before `ExtractPythonUDFs`.
       ExtractGroupingPythonUDFFromAggregate,
-      // `ExtractPythonUDFs` first lifts Python UDFs out of higher-order 
function lambdas
-      // (via `ExtractPythonUDFFromLambda`) and then extracts them as ordinary 
top-level UDFs.
       ExtractPythonUDFs,
       ExtractPythonUDTFs,
       // The eval-python node may be between Project/Filter and the scan node, 
which breaks
@@ -123,13 +125,14 @@ class SparkOptimizer(
 
   override def nonExcludableRules: Seq[String] = super.nonExcludableRules ++
     Seq(
+      // A plan with a Python UDF in a higher-order function lambda passes 
analysis only because
+      // this rule lifts the UDF out of the lambda.
+      ExtractPythonUDFFromLambda.ruleName,
       ExtractPythonUDFFromJoinCondition.ruleName,
       ExtractPythonUDFFromAggregate.ruleName,
       ExtractGroupingPythonUDFFromAggregate.ruleName,
       ExtractExternalUDFFromWindow.ruleName,
       PlanExternalUDFs.ruleName,
-      // Non-excludable: a plan with a Python UDF in a higher-order function 
lambda only works
-      // because `ExtractPythonUDFs` lifts it out (via 
`ExtractPythonUDFFromLambda`).
       ExtractPythonUDFs.ruleName,
       GroupBasedRowLevelOperationScanPlanning.ruleName,
       V2ScanRelationPushDown.ruleName,
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
index dfaa4732ca9d..1609bc0c2990 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.execution.python
 
 import org.apache.spark.api.python.PythonEvalType
 import org.apache.spark.sql.catalyst.expressions._
+import 
org.apache.spark.sql.catalyst.expressions.PythonUDF.canonicalizeDeterministic
 import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.TreePattern._
@@ -55,10 +56,16 @@ import org.apache.spark.sql.types.{ArrayType, IntegerType, 
MapType}
  * them uniformly with no per-argument metadata. With the result now an 
ordinary column, arithmetic,
  * `when`, casts, the element index, multiple UDFs and nested calls `f(g(x))` 
all just work.
  *
- * Runs before [[ExtractPythonUDFs]]. Handles all ten single-lambda functions: 
`transform`,
- * `filter`, `exists`, `forall`, `zip_with`, `array_sort`, and the four map 
functions (desugared to
- * `map_keys`/`map_values` arrays and rebuilt with `map_from_arrays`). 
`array_sort` precomputes a
- * per-element key, or, when one call takes both elements, the UDF over the 
cross product of pairs.
+ * Runs first in the optimizer's UDF extraction batch, before every rule that 
decides where a
+ * Python UDF is evaluated from the columns it reads 
(`ExtractPythonUDFFromJoinCondition`,
+ * [[ExtractPythonUDFFromAggregate]], [[ExtractPythonUDFs]], ...): inside a 
lambda a UDF reads the
+ * lambda's variables, so only its lifted form tells them, e.g., that it 
applies to an aggregate's
+ * result and must be evaluated above the aggregate. Handles all ten 
single-lambda functions:
+ * `transform`, `filter`, `exists`, `forall`, `zip_with`, `array_sort`, and 
the four map functions
+ * (desugared to `map_keys`/`map_values` arrays and rebuilt with 
`map_from_arrays`). `array_sort`
+ * precomputes a per-element key - one per element even for a nondeterministic 
UDF, so that an
+ * element has the same key on either side of a comparison - or, when one call 
takes both elements,
+ * the UDF over the cross product of pairs.
  * A UDF in a *nested* lambda, `transform(arr, i -> transform(i, x -> f(x)))`, 
is handled too: the
  * whole nest is rewritten root-first (see `apply` / `rewriteNest`), lifting 
the UDF out one lambda
  * level at a time so it ends up applied to the fully flattened leaves.
@@ -154,6 +161,8 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan] 
{
    * no per-element key. Precomputes the UDF over every ordered pair - an n x 
n matrix with
    * `udf(arr[i], arr[j])` at (i, j) - and the comparator reads it by the two 
elements' positions,
    * so no Python runs while sorting. Costs O(n^2) calls and memory vs. O(n) 
for a per-element key.
+   * A nondeterministic UDF is drawn once per ordered pair, so `udf(x, y)` and 
`udf(y, x)` need not
+   * agree: with no per-element key to share, such a comparator does not 
define an order.
    */
   private def rewritePairwiseComparator(sort: ArraySort): Expression = {
     val ArraySort(argument, function, allowNull) = sort
@@ -361,8 +370,6 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan] 
{
       indexVar: Option[NamedLambdaVariable],
       alsoBind: Seq[NamedLambdaVariable] = Nil): Carrier = {
     val LambdaFunction(body, _, _) = function
-    val lambdaExprIds =
-      (elementVars ++ indexVar.toSeq ++ alsoBind).map(_.exprId).toSet
 
     // Collect the UDF calls to lift. Innermost first, so that a nested call 
like `f(g(x))` has
     // `g` lifted before `f`, letting `f`'s array UDF consume `g`'s array 
result.
@@ -398,13 +405,13 @@ object ExtractPythonUDFFromLambda extends 
Rule[LogicalPlan] {
         indexVar.map(_.exprId -> indexArray.get).toMap ++
         alsoBind.map(_.exprId -> alignedArguments.head).toMap
 
-    // One lifted array UDF per distinct call. `arrayResults` maps each 
original call (by `liftKey`)
-    // to the array holding its per-element results, so a nested call 
`f(g(x))` and the carrier
-    // lookups can find it. Deterministic calls that lift to the *same* 
function over the *same*
-    // array arguments share one lifted UDF: a key-form comparator's `udf(a)` 
and `udf(b)`
-    // canonicalize differently (`a` != `b`) but both read the whole array, so 
without this the
-    // Python function would run 2n times instead of n. Nondeterministic calls 
stay distinct (their
-    // signature is the lifted node itself, carrying a distinct `resultId`), 
matching `liftKey`.
+    // One lifted array UDF per distinct call. `arrayResults` maps each 
original call (by
+    // `canonicalizeDeterministic`) to the array holding its per-element 
results, so a nested call
+    // `f(g(x))` and the carrier lookups can find it. Deterministic calls that 
lift to the *same*
+    // function over the *same* array arguments share one lifted UDF: a 
key-form comparator's
+    // `udf(a)` and `udf(b)` canonicalize differently (`a` != `b`) but both 
read the whole array, so
+    // without this the Python function would run 2n times instead of n. 
Nondeterministic calls stay
+    // distinct, except in a key-form comparator (see `signature` below).
     var arrayResults = Map.empty[Expression, Expression]
     val distinctLifted = scala.collection.mutable.ArrayBuffer.empty[PythonUDF]
     val ordinalBySignature = 
scala.collection.mutable.HashMap.empty[Expression, Int]
@@ -417,9 +424,9 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan] 
{
       val arrayArgs = udf.children.map {
         case NamedArgumentExpression(key, value) =>
           NamedArgumentExpression(
-            key, overArray(value, alignedArguments.head, arrayOfVar, 
lambdaExprIds, arrayResults))
+            key, overArray(value, alignedArguments.head, arrayOfVar, 
arrayResults))
         case child =>
-          overArray(child, alignedArguments.head, arrayOfVar, lambdaExprIds, 
arrayResults)
+          overArray(child, alignedArguments.head, arrayOfVar, arrayResults)
       }
       // Each lift wraps the arguments in exactly one more `array` level. 
Lifting a base UDF gives
       // depth 1; re-lifting an already-lifted element-wise UDF (a UDF from a 
*nested* lambda,
@@ -442,14 +449,23 @@ object ExtractPythonUDFFromLambda extends 
Rule[LogicalPlan] {
         PythonUDF.liftedElementwiseEvalType(udf.evalType),
         udf.udfDeterministic,
         elementwiseNestingDepth = newDepth)
-      val signature: Expression = if (udf.udfDeterministic) 
lifted.canonicalized else lifted
+      // Calls share one lifted UDF when their lifted forms have the same key; 
`overArray` keeps
+      // each argument's determinism, so `lifted` is deterministic exactly 
when `udf` is. A key-form
+      // comparator (the one caller passing `alsoBind`) shares a 
nondeterministic call too: its
+      // lifted result is each element's sort key, and an element must have 
one key whichever side
+      // of a comparison it is on. With a draw per call, `(a, b) -> r(a) - 
r(b)` would not be
+      // antisymmetric, and the sort could fail with "Comparison method 
violates its general
+      // contract!".
+      val signature =
+        if (alsoBind.nonEmpty) lifted.canonicalized else 
canonicalizeDeterministic(lifted)
       val ordinal = ordinalBySignature.getOrElseUpdate(signature, {
         val o = distinctLifted.length
         distinctLifted += lifted
         o
       })
-      arrayResults += (liftKey(udf) -> distinctLifted(ordinal))
-      udfFieldByKey += (liftKey(udf) -> ordinal)
+      val key = canonicalizeDeterministic(udf)
+      arrayResults += (key -> distinctLifted(ordinal))
+      udfFieldByKey += (key -> ordinal)
     }
     val liftedArrays = distinctLifted.toSeq
 
@@ -490,8 +506,8 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan] 
{
     }
 
     val rewrittenBody = body.transformDown {
-      case udf: PythonUDF if udfFieldByKey.contains(liftKey(udf)) =>
-        val ordinal = udfFieldByKey(liftKey(udf))
+      case udf: PythonUDF if 
udfFieldByKey.contains(canonicalizeDeterministic(udf)) =>
+        val ordinal = udfFieldByKey(canonicalizeDeterministic(udf))
         // A UDF over a comparator's right-hand element must read that 
element's key, so the
         // struct field is read through whichever bound variable the call's 
own arguments used.
         val side = udf.collectFirst {
@@ -541,28 +557,12 @@ object ExtractPythonUDFFromLambda extends 
Rule[LogicalPlan] {
     }
     visit(body)
     // Deduplicate identical calls so the same UDF is evaluated once per 
array. Nondeterministic
-    // calls are kept distinct (see `liftKey`).
+    // calls are kept distinct (see `canonicalizeDeterministic`).
     val seen = scala.collection.mutable.LinkedHashMap.empty[Expression, 
PythonUDF]
-    collected.result().foreach(udf => seen.getOrElseUpdate(liftKey(udf), udf))
+    collected.result().foreach(udf => 
seen.getOrElseUpdate(canonicalizeDeterministic(udf), udf))
     seen.values.toSeq
   }
 
-  private def readsLambdaVariable(e: Expression, lambdaExprIds: Set[ExprId]): 
Boolean =
-    e.exists {
-      case v: NamedLambdaVariable => lambdaExprIds.contains(v.exprId)
-      case _ => false
-    }
-
-  /**
-   * The key that decides whether two UDF calls are "the same call" for 
lifting. A deterministic
-   * call is deduplicated by canonical form, so an identical call is evaluated 
once. A
-   * nondeterministic call must stay distinct - `transform(arr, x -> f(x) + 
f(x))` calls `f` twice
-   * and each call may return a different value - so it is keyed by its own 
`resultId`-bearing node
-   * (`canonicalized` erases `resultId`, which would collapse the two calls 
into one).
-   */
-  private def liftKey(udf: PythonUDF): Expression =
-    if (udf.udfDeterministic) udf.canonicalized else udf
-
   /**
    * Turns a UDF argument expression, written in terms of single elements, 
into the equivalent
    * `array<T>` aligned with the iterated array, so the worker can flatten 
every argument uniformly.
@@ -585,28 +585,36 @@ object ExtractPythonUDFFromLambda extends 
Rule[LogicalPlan] {
       child: Expression,
       firstArgument: Expression,
       arrayOfVar: Map[ExprId, Expression],
-      lambdaExprIds: Set[ExprId],
       arrayResults: Map[Expression, Expression]): Expression = child match {
     case v: NamedLambdaVariable if arrayOfVar.contains(v.exprId) => 
arrayOfVar(v.exprId)
-    case udf: PythonUDF if arrayResults.contains(liftKey(udf)) =>
-      arrayResults(liftKey(udf))
+    case udf: PythonUDF if 
arrayResults.contains(canonicalizeDeterministic(udf)) =>
+      arrayResults(canonicalizeDeterministic(udf))
     case e =>
       // Replace each already-lifted nested UDF call with a synthetic variable 
standing for that
       // call's aligned array result, then fold those arrays into the 
variable-to-array map so the
       // logic below treats them exactly like element variables. This keeps a 
lifted UDF buried in
       // a composite argument (`f(g(x) + 1)`) from being left as a raw UDF 
inside a lambda.
+      // Top-down, so the outermost lifted call is replaced whole: bottom-up 
would first replace
+      // a lifted call nested in its arguments (`h(x)` in `f(g(h(x)) + 1)`), 
after which `g(...)`
+      // no longer matches its own lifted result and would be left behind as a 
raw UDF.
       val nestedVars = 
scala.collection.mutable.LinkedHashMap.empty[Expression, NamedLambdaVariable]
-      val expr = e.transformUp {
-        case u: PythonUDF if arrayResults.contains(liftKey(u)) =>
-          nestedVars.getOrElseUpdate(liftKey(u), {
-            val arrType = 
arrayResults(liftKey(u)).dataType.asInstanceOf[ArrayType]
+      val expr = e.transformDown {
+        case u: PythonUDF if 
arrayResults.contains(canonicalizeDeterministic(u)) =>
+          val key = canonicalizeDeterministic(u)
+          nestedVars.getOrElseUpdate(key, {
+            val arrType = arrayResults(key).dataType.asInstanceOf[ArrayType]
             NamedLambdaVariable("g", arrType.elementType, arrType.containsNull)
           })
       }
-      val fullArrayOf = arrayOfVar ++
-        nestedVars.map { case (key, v) => v.exprId -> arrayResults(key) }
-
-      if (!readsLambdaVariable(expr, fullArrayOf.keySet)) {
+      // The arrays of just the variables `expr` reads: zipping in one it does 
not read, such as the
+      // element array for `f(g(x) + 1)`, which reads only `g`'s result, would 
build a struct per
+      // element for nothing.
+      val readIds = expr.collect { case v: NamedLambdaVariable => v.exprId 
}.toSet
+      val arrayOfRead =
+        (arrayOfVar ++ nestedVars.map { case (key, v) => v.exprId -> 
arrayResults(key) })
+          .filter { case (id, _) => readIds.contains(id) }
+
+      if (arrayOfRead.isEmpty) {
         // Independent of the element (an outer column or constant): repeat it 
into an array aligned
         // with the iterated array, so every UDF argument is a single-level 
array the worker
         // flattens the same way. `transform(arr, _ -> e)` keeps the value 
constant, matching shape.
@@ -616,23 +624,24 @@ object ExtractPythonUDFFromLambda extends 
Rule[LogicalPlan] {
       } else {
         // An expression over the element(s) and/or nested results, e.g. 
`udf(x * 2)` or
         // `f(g(x) + 1)`. Compute it for every element with a native 
`transform`, which stays inside
-        // the JVM. Multi-argument shapes need the values side by side, so zip 
them first.
-        val arrays = fullArrayOf.values.toSeq.distinct
+        // the JVM. An expression reading several arrays needs their values 
side by side, so zip
+        // them first.
+        val arrays = arrayOfRead.values.toSeq.distinct
         if (arrays.length == 1) {
           val arr = arrays.head
           val elemType = arr.dataType.asInstanceOf[ArrayType]
           val v = NamedLambdaVariable("x", elemType.elementType, 
elemType.containsNull)
           val substituted = expr.transformUp {
-            case old: NamedLambdaVariable if fullArrayOf.contains(old.exprId) 
=> v
+            case old: NamedLambdaVariable if arrayOfRead.contains(old.exprId) 
=> v
           }
           ArrayTransform(arr, LambdaFunction(substituted, Seq(v)))
         } else {
-          // Zip every array the expression may read, then project the fields 
it needs.
+          // Zip the arrays the expression reads, then project the fields it 
needs.
           val names = arrays.indices.map(i => s"$carrierElementPrefix$i")
           val zipped = ArraysZip(arrays, names.map(Literal(_)))
           val structType = zipped.dataType.asInstanceOf[ArrayType].elementType
           val v = NamedLambdaVariable("z", structType, nullable = false)
-          val ordinalOf = fullArrayOf.map { case (id, arr) => id -> 
arrays.indexOf(arr) }
+          val ordinalOf = arrayOfRead.map { case (id, arr) => id -> 
arrays.indexOf(arr) }
           val substituted = expr.transformUp {
             case old: NamedLambdaVariable if ordinalOf.contains(old.exprId) =>
               GetStructField(v, ordinalOf(old.exprId), 
Some(names(ordinalOf(old.exprId))))
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
index 3ad83a74c056..052949dd6146 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
@@ -25,7 +25,8 @@ import org.apache.spark.api.python.PythonEvalType
 import org.apache.spark.internal.Logging
 import org.apache.spark.internal.LogKeys.REASON
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.PythonUDF.{correctEvalType, 
isScalarPythonUDF}
+import 
org.apache.spark.sql.catalyst.expressions.PythonUDF.{canonicalizeDeterministic,
+  correctEvalType, isScalarPythonUDF}
 import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
 import org.apache.spark.sql.catalyst.plans.logical._
 import org.apache.spark.sql.catalyst.rules.Rule
@@ -272,14 +273,7 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with 
Logging {
     expressions.flatMap(collectEvaluableUDFs)
   }
 
-  def apply(plan: LogicalPlan): LogicalPlan = applyInternal(
-    // Lift Python UDFs out of higher-order function lambdas first, turning 
them into ordinary
-    // top-level UDFs this rule can then extract. Running it here rather than 
as a separate batch
-    // entry makes the ordering structural: `CheckAnalysis` accepts a 
lambda-UDF plan only because
-    // this rewrite will run, so the two must not be separable by rule 
reordering or exclusion.
-    ExtractPythonUDFFromLambda(plan))
-
-  private def applyInternal(plan: LogicalPlan): LogicalPlan = plan match {
+  def apply(plan: LogicalPlan): LogicalPlan = plan match {
     // SPARK-26293: A subquery will be rewritten into join later, and will go 
through this rule
     // eventually. Here we skip subquery, as Python UDF only needs to be 
extracted once.
     case s: Subquery if s.correlated => plan
@@ -298,14 +292,6 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with 
Logging {
     }
   }
 
-  private def canonicalizeDeterministic(u: PythonUDF) = {
-    if (u.deterministic) {
-      u.canonicalized.asInstanceOf[PythonUDF]
-    } else {
-      u
-    }
-  }
-
   /**
    * Extract all the PythonUDFs from the current operator and evaluate them 
before the operator.
    */
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
index 09d982026006..46596bef42b8 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
@@ -19,11 +19,11 @@ package org.apache.spark.sql.execution.python
 
 import org.apache.spark.api.python.PythonEvalType
 import org.apache.spark.sql.{AnalysisException, QueryTest}
-import org.apache.spark.sql.catalyst.expressions.{LambdaFunction, Literal, 
NamedArgumentExpression,
-  PythonUDF}
-import org.apache.spark.sql.catalyst.plans.logical.ArrowEvalPython
-import org.apache.spark.sql.functions.{array_sort, col, forall, lit, 
map_filter, map_zip_with,
-  transform, transform_keys, transform_values, when, zip_with}
+import org.apache.spark.sql.catalyst.expressions.{ArraysZip, LambdaFunction, 
Literal,
+  NamedArgumentExpression, PythonUDF}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, 
ArrowEvalPython, Join}
+import org.apache.spark.sql.functions.{array_sort, col, collect_list, forall, 
lit, map_filter,
+  map_zip_with, transform, transform_keys, transform_values, when, zip_with}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.{ExamplePointUDT, SharedSparkSession}
 import org.apache.spark.sql.types.{ArrayType, IntegerType, MapType, StringType}
@@ -144,6 +144,42 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest 
with SharedSparkSession
     assert(liftedUDFs(negated).size == 2)
   }
 
+  test("a lifted call two levels deep inside a composite argument is 
substituted whole") {
+    // `f(g(h(x)) + 1)`: `f`'s composite argument holds the lifted call 
`g(h(x))`, which itself
+    // holds the lifted call `h(x)`. Substituting bottom-up would replace 
`h(x)` first, so `g(...)`
+    // no longer matches its lifted result and a raw `g` is left inside the 
generated lambda. That
+    // `g` is lifted a second time when it can be, and stays inside the lambda 
when it cannot (its
+    // lambda then iterates the nondeterministic result of `h`).
+    def nested(inner: UserDefinedPythonFunction): 
org.apache.spark.sql.DataFrame =
+      arrayDF.select(transform(col("values"), x =>
+        pythonUDF(pythonUDF(inner(x)).cast("int") + lit(1))).as("r"))
+
+    val deterministic = nested(pythonUDF)
+    assert(udfsInsideLambda(deterministic).isEmpty)
+    // One lifted UDF per call: `h`, `g` and `f`.
+    assert(liftedUDFs(deterministic).size == 3)
+
+    assert(udfsInsideLambda(nested(nondeterministicUDF)).isEmpty)
+  }
+
+  test("a composite argument over a nested call zips only the arrays it 
reads") {
+    // `f(g(x) + 1)` reads only `g`'s result, not the element, so its argument 
is a `transform` over
+    // that result alone: zipping in the element array as well would build a 
struct per element for
+    // nothing. With the index, `f(g(x) + i)` zips just the index and `g`'s 
result.
+    def zippedArrayCounts(df: org.apache.spark.sql.DataFrame): Seq[Int] =
+      liftedUDFs(df).flatMap(_.collect { case z: ArraysZip => z.children.size 
})
+
+    val plusOne = arrayDF.select(
+      transform(col("values"), x => pythonUDF(pythonUDF(x).cast("int") + 
lit(1))).as("r"))
+    assert(udfsInsideLambda(plusOne).isEmpty)
+    assert(zippedArrayCounts(plusOne).isEmpty)
+
+    val plusIndex = arrayDF.select(
+      transform(col("values"), (x, i) => pythonUDF(pythonUDF(x).cast("int") + 
i)).as("r"))
+    assert(udfsInsideLambda(plusIndex).isEmpty)
+    assert(zippedArrayCounts(plusIndex) == Seq(2))
+  }
+
   test("identical nondeterministic calls are lifted distinctly, not 
deduplicated") {
     // Deduplicating `f(x)` with `f(x)` would collapse two independent draws 
into one; a
     // nondeterministic UDF must keep each call, so both are lifted.
@@ -153,6 +189,18 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest 
with SharedSparkSession
     assert(liftedUDFs(df).size == 2)
   }
 
+  test("a deterministic UDF over a nondeterministic argument is not 
deduplicated") {
+    // `f(r(x)) === f(r(x))`: `f` itself is deterministic, but each side wraps 
its own draw of the
+    // nondeterministic `r`, so the two `f` calls are distinct. Keying them by 
canonical form would
+    // erase the inner calls' `resultId`s and lift one `f` call for both 
sides, comparing a single
+    // draw with itself.
+    val df = arrayDF.select(transform(col("values"), x =>
+      pythonUDF(nondeterministicUDF(x)) === 
pythonUDF(nondeterministicUDF(x))).as("r"))
+    assert(udfsInsideLambda(df).isEmpty)
+    // Two inner draws, and one outer call over each of them.
+    assert(liftedUDFs(df).size == 4)
+  }
+
   test("transform_values whose result type equals the key type replaces 
values, not keys") {
     // SPARK-27052: dispatch is by concrete function, not result type. For 
map<string, string>,
     // transform_values' lambda also returns string, which must not be treated 
as new keys.
@@ -283,6 +331,27 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest 
with SharedSparkSession
     assert(liftedUDFs(df).size == 1)
   }
 
+  test("a key-form array_sort comparator shares a nondeterministic call 
between its sides") {
+    // An element must have one sort key whichever side of a comparison it is 
on, so `k(a)` and
+    // `k(b)` share one lifted call even when `k` is nondeterministic: one 
draw per element. A draw
+    // per side would make the comparator inconsistent, and the sort could 
fail with "Comparison
+    // method violates its general contract!". The same holds for a 
deterministic UDF over a
+    // nondeterministic argument, `f(r(a))` and `f(r(b))`.
+    val bare = arrayDF.select(
+      array_sort(col("values"), (a, b) =>
+        when(nondeterministicUDF(a) === nondeterministicUDF(b), 
lit(0)).otherwise(lit(1))).as("r"))
+    assert(udfsInsideLambda(bare).isEmpty)
+    assert(liftedUDFs(bare).size == 1)
+
+    val wrapped = arrayDF.select(
+      array_sort(col("values"), (a, b) =>
+        when(pythonUDF(nondeterministicUDF(a)) === 
pythonUDF(nondeterministicUDF(b)), lit(0))
+          .otherwise(lit(1))).as("r"))
+    assert(udfsInsideLambda(wrapped).isEmpty)
+    // One draw per element, and one outer call over it.
+    assert(liftedUDFs(wrapped).size == 2)
+  }
+
   test("a vectorized scalar UDF inside a lambda is lifted to its element-wise 
eval type") {
     // Each vectorized scalar flavor lifts to its own element-wise eval type 
so the worker keeps
     // that flavor's batching contract (pandas Series, Arrow Array, or an 
iterator of batches).
@@ -404,9 +473,9 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest 
with SharedSparkSession
   }
 
   test("the rewrite cannot be disabled via excludedRules") {
-    // The lambda rewrite is driven by `ExtractPythonUDFs` (not a standalone 
batch rule), and that
-    // rule is non-excludable, so a plan that only works because of the 
rewrite cannot be broken by
-    // excludedRules. Excluding either name must leave the UDF lifted out of 
the lambda.
+    // The lambda rewrite is a non-excludable rule, so a plan that only works 
because of the rewrite
+    // cannot be broken by excludedRules. Excluding it (or 
`ExtractPythonUDFs`, which extracts the
+    // lifted UDF) must leave the UDF lifted out of the lambda.
     withSQLConf(
       SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
         Seq(ExtractPythonUDFFromLambda.ruleName, 
ExtractPythonUDFs.ruleName).mkString(",")) {
@@ -415,6 +484,57 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest 
with SharedSparkSession
     }
   }
 
+  test("a UDF in a lambda over an aggregate's result is evaluated above the 
aggregate") {
+    // `transform(collect_list(id), x -> f(x))`: once lifted, the UDF reads 
the aggregate's result,
+    // so it must be lifted before `ExtractPythonUDFFromAggregate` runs, which 
then evaluates it
+    // above the `Aggregate`. Lifted later, it would be evaluated below the 
aggregate, where neither
+    // an aggregate function nor (for a lambda over a grouping key) the 
grouped value exists.
+    val overAggregate = spark.range(3).groupBy()
+      .agg(transform(collect_list(col("id")), x => pythonUDF(x)).as("r"))
+    val overGroupingKey = Seq(Seq(1, 2), 
Seq(3)).toDF("values").groupBy("values")
+      .agg(transform(col("values"), x => pythonUDF(x)).as("r"))
+    Seq(overAggregate, overGroupingKey).foreach { df =>
+      assert(udfsInsideLambda(df).isEmpty)
+      val evals = df.queryExecution.optimizedPlan.collect { case a: 
ArrowEvalPython => a }
+      assert(evals.size == 1)
+      assert(evals.head.child.exists(_.isInstanceOf[Aggregate]))
+    }
+  }
+
+  test("a UDF in a lambda over one side of an outer join condition is 
evaluated on that side") {
+    // A left outer join keeps a predicate over only its left side in the join 
condition. Once
+    // lifted, the UDF reads only left-side columns and is evaluated below the 
join on that side,
+    // like a plain UDF. While still inside the lambda it read neither side, 
so it was rejected as a
+    // Python UDF in the ON clause of an outer join.
+    val left = Seq((1, Seq(1, 2))).toDF("k", "values")
+    val right = Seq((1, "a")).toDF("k", "v")
+    val df = left.join(
+      right,
+      left("k") === right("k") &&
+        org.apache.spark.sql.functions.exists(left("values"), x => 
pythonUDF(x)),
+      "left_outer")
+    val join = df.queryExecution.optimizedPlan.collectFirst { case j: Join => 
j }.get
+    assert(join.left.exists(_.isInstanceOf[ArrowEvalPython]))
+    assert(!join.right.exists(_.isInstanceOf[ArrowEvalPython]))
+  }
+
+  test("a UDF in a lambda over one side of an inner join condition is 
evaluated on that side") {
+    // The conjunct reads both sides, so it stays in the join condition, but 
once lifted the UDF
+    // itself reads only the left side and is evaluated below the join on that 
side, as a plain
+    // one-side UDF in a join condition is. While still inside the lambda it 
read neither side, so
+    // the whole conjunct was pulled out into a `Filter` above the join.
+    val left = Seq((1, Seq(1, 2))).toDF("k", "values")
+    val right = Seq((1, true)).toDF("k", "flag")
+    val df = left.join(
+      right,
+      left("k") === right("k") &&
+        org.apache.spark.sql.functions.exists(left("values"), x => 
pythonUDF(x) === right("flag")),
+      "inner")
+    val join = df.queryExecution.optimizedPlan.collectFirst { case j: Join => 
j }.get
+    assert(join.left.exists(_.isInstanceOf[ArrowEvalPython]))
+    assert(!join.right.exists(_.isInstanceOf[ArrowEvalPython]))
+  }
+
   test("a UDF over only constants is still lifted per element") {
     // SPARK-27052: `transform(arr, x -> udf(lit(10)))` does not read the 
element, but it is still
     // lifted per element (`overArray` repeats the constant into an aligned 
array) so it keeps the


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to