This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 88145e6b48e7 [SPARK-27052][SQL][PYTHON][FOLLOWUP] Fix issues in
lifting Python UDFs out of higher-order function lambdas
88145e6b48e7 is described below
commit 88145e6b48e7a371e39581ae0cde2d2e22ad03ab
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Oct 23 13:35:12 2026 +0900
[SPARK-27052][SQL][PYTHON][FOLLOWUP] Fix issues in lifting Python UDFs out
of higher-order function lambdas
### What changes were proposed in this pull request?
This PR fixes four issues in `ExtractPythonUDFFromLambda`, the optimizer
rule that lifts Python UDFs out of higher-order function lambdas (SPARK-27052):
1. **Deduplication ignored argument determinism.** Whether two UDF calls
are the same call is now decided by one helper,
`PythonUDF.canonicalizeDeterministic`. It checks `deterministic` instead of the
UDF's own `udfDeterministic` flag, and `deterministic` also covers the
arguments. This rule uses it for its call keys and for the signature that lets
identical calls share one lifted UDF, and `ExtractPythonUDFs` uses it to
deduplicate the calls it extracts. So calls that this rule keeps [...]
2. **A key-form `array_sort` comparator read a different draw on each
side.** In `array_sort(arr, (a, b) -> k(a) - k(b))`, the lifted `k` is each
element's sort key. In such a comparator, calls that lift to the same function
over the same array now share one lifted UDF even when they are
nondeterministic, so each element has one key. Outside comparators,
nondeterministic calls stay distinct.
3. **Nested lifted calls were substituted bottom-up.** `overArray` replaces
each already-lifted call inside a composite argument, such as `g(x)` in `f(g(x)
+ 1)`, with a variable that stands for that call's array result. It now does
this top-down. It also zips only the arrays the argument reads: the argument of
`f(g(x) + 1)` is now a `transform` over `g`'s result alone, instead of over
`arrays_zip(values, g(values))`.
4. **The rule ran too late.** `ExtractPythonUDFFromLambda` is now its own
rule. It runs first in the "Extract UDFs" batch and is listed in
`nonExcludableRules`. Before, `ExtractPythonUDFs` invoked it internally, after
`ExtractPythonUDFFromJoinCondition`, `ExtractPythonUDFFromAggregate` and
`ExtractGroupingPythonUDFFromAggregate` had already run.
### Why are the changes needed?
**1. Wrong results.** A deterministic UDF over a nondeterministic argument
was keyed by its canonical form. Canonicalization drops every `PythonUDF`'s
`resultId`, so the two inner draws below could no longer be told apart, and
both sides read one lifted call:
```python
draw = udf(lambda x: uuid.uuid4().hex, "string").asNondeterministic()
keep = udf(lambda s: s, "string")
df.select(F.transform("values", lambda x: keep(draw(x)) == keep(draw(x))))
# before: [true, true, true]
# after: [false, false, false]
```
Without the higher-order function, the same comparison already returns
`false`, and so does `draw(x) == draw(x)` inside the lambda. The same applies
to `F.transform("values", lambda x: (lambda r: keep(r) == keep(r))(draw(x)))`.
PySpark runs the Python lambda once while building the plan, so `r` is not
bound to one draw per element. It is inlined as two `draw` calls, and the
result changes from `true` to `false`, in both classic and Connect, as it
already was without the higher-order f [...]
**2. Sort failure.** A comparator has to compare an element by the same key
whichever side of the comparison it is on. With one draw per call, a
nondeterministic key UDF gave each element one key as `a` and another as `b`,
and in the usual `when` form a different key in each branch. The comparator
then did not define an order:
```python
key = udf(lambda x: random.randint(0, 1_000_000),
"int").asNondeterministic()
df.select(F.array_sort(
"values",
lambda a, b: F.when(key(a) < key(b), -1).when(key(a) > key(b),
1).otherwise(0)))
# before, for an array of 1000 elements:
# java.lang.IllegalArgumentException: Comparison method violates its
general contract!
# after: the elements in a random order
```
On master, `keep(draw(a)) - keep(draw(b))` happened to share one lifted
call, through the canonicalization that fix 1 removes, while a bare `draw(a) -
draw(b)` never did. Both now share one draw per element. A comparator whose
single call takes both elements, `(a, b) -> udf(a, b)`, is unchanged: it has no
per-element key to share, so a nondeterministic `udf` still gets one draw per
ordered pair.
**3. Runtime failure, or double evaluation.** For `f(g(h(x)) + 1)`,
bottom-up substitution replaced `h(x)` first. The rewritten `g(...)` then no
longer matched the lifted result of `g(h(x))`, and a raw `g` was left inside
the lambda generated for `f`'s argument:
- If that lambda could be rewritten again, `g` was lifted a second time and
ran twice per element.
- If it could not, the `PythonUDF` stayed inside the lambda and the query
failed. This happens, for example, when `h` is nondeterministic, because a
lambda that iterates a nondeterministic array is not rewritable:
```python
same = udf(lambda x: x, "int").asNondeterministic()
df.select(F.transform("values", lambda x: times_ten(plus_one(same(x)) + 1)))
# before: [INTERNAL_ERROR] Cannot generate code for expression:
# <lambda>(transform(arrays_zip(...),
lambdafunction((<lambda>(lambda z.c1) + 1), ...)))
# after: [30, 40, 50] for values [1, 2, 3]
```
Zipping in arrays that the argument does not read only cost an extra struct
per element, or more with `zip_with` or an index parameter.
**4. UDFs placed in the wrong operator.** The rules that ran before the
lift decide where a Python UDF is evaluated based on the columns it references.
Inside a lambda, a UDF references only the lambda's variables, so those rules
did not see it. It was lifted afterwards, and `ExtractPythonUDFs` then placed
it on its own:
```sql
SELECT transform(collect_list(id), x -> plus_one(x)) FROM range(3)
-- before: [INTERNAL_ERROR] Cannot evaluate expression: collect_list(...)
-- (the lifted UDF, with collect_list(id) as its argument, was
evaluated below the Aggregate)
-- after: [1, 2, 3]
```
```python
df.groupBy("values").agg(F.transform("values", lambda x: plus_one(x)))
# before: [INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND] Could not find pythonUDF0 in
[values]
left.join(right, (left.k == right.k) & F.exists(left["values"], lambda x:
plus_one(x) > 3),
"left_outer")
# before: [UNSUPPORTED_FEATURE.PYTHON_UDF_IN_ON_CLAUSE] ... Python UDF in
the ON clause of a
# LEFT OUTER JOIN
```
When it is lifted first, each of these is an ordinary UDF over a whole
array. Those rules already handle that the same way as a plain UDF: it is
evaluated above the aggregate, or on the one join side it reads. The ordering
is now also explicit in `SparkOptimizer`, instead of a comment explaining that
`ExtractPythonUDFs` does the lifting internally.
This also moves a lambda UDF that reads one side of an inner join. In `l.k
= r.k AND exists(l.values, x -> f(x) > r.t)` the conjunct reads both sides, so
it stays in the join condition. Before, `f(x)` referenced only its lambda
variable, so `ExtractPythonUDFFromJoinCondition` moved the conjunct into a
`Filter` above the join, and `f` ran only on joined rows. Now the lifted `f`
reads only `l.values`, and `ExtractPythonUDFs` evaluates it below the join on
every row of the left side. Thi [...]
### Does this PR introduce _any_ user-facing change?
No for released versions. SPARK-27052 is only in `master` and `branch-4.x`
and not released yet. On those branches, the queries above now return correct
results, or run instead of failing, and a lambda UDF that reads one side of an
inner join is placed as described above. Plans without a Python UDF in a lambda
are unaffected: the lambda rule does not change them, and the other rules run
in the same order as before.
### How was this patch tested?
New tests in `ExtractPythonUDFFromLambdaSuite`, each of which fails on
master:
- "a deterministic UDF over a nondeterministic argument is not deduplicated"
- "a key-form array_sort comparator shares a nondeterministic call between
its sides"
- "a lifted call two levels deep inside a composite argument is substituted
whole"
- "a composite argument over a nested call zips only the arrays it reads"
- "a UDF in a lambda over an aggregate's result is evaluated above the
aggregate"
- "a UDF in a lambda over one side of an outer join condition is evaluated
on that side"
- "a UDF in a lambda over one side of an inner join condition is evaluated
on that side"
New tests in `pyspark.sql.tests.test_udf_in_higher_order_function` and its
Connect parity suite:
- `test_deterministic_udf_over_nondeterministic_udf_calls_are_distinct`,
`test_array_sort_with_nondeterministic_udf_key`,
`test_nested_udf_two_levels_inside_composite_argument`,
`test_udf_in_lambda_over_aggregate_result` and
`test_udf_in_lambda_in_outer_join_condition` fail on master.
- `test_udf_in_lambda_in_inner_join_condition`, and the index case added to
`test_nested_udf_inside_composite_argument`, check the results of the new plan
shapes.
Also ran all `org.apache.spark.sql.execution.python.*` suites, the classic
and Connect `test_udf_in_higher_order_function` suites,
`pyspark.sql.tests.test_udf` and
`pyspark.sql.tests.arrow.test_arrow_python_udf`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59701 from HyukjinKwon/SPARK-27052-followup-hof-udf-fixes.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit b19dd54a05291bb5eaa012e7622289edf6807768)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../sql/tests/test_udf_in_higher_order_function.py | 141 +++++++++++++++++++++
.../spark/sql/catalyst/expressions/PythonUDF.scala | 21 ++-
.../spark/sql/execution/SparkOptimizer.scala | 13 +-
.../python/ExtractPythonUDFFromLambda.scala | 117 +++++++++--------
.../sql/execution/python/ExtractPythonUDFs.scala | 20 +--
.../python/ExtractPythonUDFFromLambdaSuite.scala | 136 ++++++++++++++++++--
6 files changed, 362 insertions(+), 86 deletions(-)
diff --git a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
index b3479f4a2660..6ac4ba9df73e 100644
--- a/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
+++ b/python/pyspark/sql/tests/test_udf_in_higher_order_function.py
@@ -152,6 +152,36 @@ class UDFInHigherOrderFunctionTestsMixin:
df.select(sf.transform("values", lambda x:
times_ten(-plus_one(x))).alias("r")),
df.select(sf.transform("values", lambda x: (-(x + 1)) *
10).alias("r")),
)
+ # Reads the index and the inner result, but not the element itself.
+ assertDataFrameEqual(
+ df.select(sf.transform("values", lambda x, i:
times_ten(plus_one(x) + i)).alias("r")),
+ df.select(sf.transform("values", lambda x, i: ((x + 1) + i) *
10).alias("r")),
+ )
+
+ def test_nested_udf_two_levels_inside_composite_argument(self):
+ # `f(g(h(x)) + 1)`: the composite argument of `f` holds the lifted
call `g(h(x))`, which
+ # itself holds the lifted call `h(x)`. `g(h(x))` must be substituted
whole; a raw `g` left
+ # behind could not be lifted again over a nondeterministic `h` and
failed at runtime.
+ import random
+
+ df = self.spark.createDataFrame([([1, 2, 3],)], "values array<int>")
+ plus_one = udf(lambda x: x + 1, IntegerType())
+ times_ten = udf(lambda x: x * 10, IntegerType())
+ # Nondeterministic, but returns its input unchanged so the expected
result is known.
+ same = udf(lambda x: x + 0 * random.randint(0, 1),
IntegerType()).asNondeterministic()
+
+ assertDataFrameEqual(
+ df.select(
+ sf.transform("values", lambda x:
times_ten(plus_one(plus_one(x)) + 1)).alias("r")
+ ),
+ df.select(sf.transform("values", lambda x: ((x + 2) + 1) *
10).alias("r")),
+ )
+ assertDataFrameEqual(
+ df.select(
+ sf.transform("values", lambda x: times_ten(plus_one(same(x)) +
1)).alias("r")
+ ),
+ df.select(sf.transform("values", lambda x: ((x + 1) + 1) *
10).alias("r")),
+ )
def test_udf_with_outer_column_argument(self):
# A non-element argument must be broadcast to every element of its row.
@@ -250,6 +280,38 @@ class UDFInHigherOrderFunctionTestsMixin:
[([3, 2, 1],), ([],), (None,)],
)
+ def test_array_sort_with_nondeterministic_udf_key(self):
+ # A per-element key is computed once per element, even by a
nondeterministic UDF, so that
+ # an element has the same key on either side of a comparison. With a
draw per call, the
+ # comparator would be inconsistent and the sort could fail with
"Comparison method
+ # violates its general contract!".
+ import random
+
+ # Nondeterministic, but returns its input negated so the expected
order is known.
+ negate = udf(lambda x: -x + 0 * random.randint(0, 1),
IntegerType()).asNondeterministic()
+ keep = udf(lambda x: x, IntegerType())
+ df = self.spark.createDataFrame([([3, 1, 2],)], "values array<int>")
+ assertDataFrameEqual(
+ df.select(
+ sf.array_sort("values", lambda a, b: keep(negate(a)) -
keep(negate(b))).alias("r")
+ ),
+ [([3, 2, 1],)],
+ )
+
+ # A random key: any order is valid, but the sort must succeed and keep
every element.
+ key = udf(lambda x: random.randint(0, 1_000_000),
IntegerType()).asNondeterministic()
+ values = list(range(1000))
+ df = self.spark.createDataFrame([(values,)], "values array<int>")
+ row = df.select(
+ sf.array_sort(
+ "values",
+ lambda a, b: sf.when(key(a) < key(b), sf.lit(-1))
+ .when(key(a) > key(b), sf.lit(1))
+ .otherwise(sf.lit(0)),
+ ).alias("r")
+ ).collect()[0]
+ self.assertEqual(sorted(row["r"]), values)
+
def test_array_sort_pairwise_comparator(self):
# One UDF call receiving both elements has no per-element key, so the
UDF is precomputed
# over every ordered pair and the comparator indexes that matrix.
Assert an actual
@@ -354,6 +416,32 @@ class UDFInHigherOrderFunctionTestsMixin:
# If the two calls were deduplicated, every element would be exactly 0.
self.assertTrue(any(v != 0 for v in row["r"]), row["r"])
+ def
test_deterministic_udf_over_nondeterministic_udf_calls_are_distinct(self):
+ # `keep` is deterministic, but each `keep(draw(x))` wraps its own
independent draw, so the
+ # two sides must not be collapsed into one call compared with itself.
The same holds when
+ # the calls are re-lifted out of a nested lambda.
+ import uuid
+
+ draw = udf(lambda x: uuid.uuid4().hex,
StringType()).asNondeterministic()
+ keep = udf(lambda s: s, StringType())
+
+ df = self.spark.createDataFrame([([1, 2, 3],)], "values array<int>")
+ assertDataFrameEqual(
+ df.select(sf.transform("values", lambda x: keep(draw(x)) ==
keep(draw(x))).alias("r")),
+ [([False, False, False],)],
+ )
+
+ nested = self.spark.createDataFrame([([[1, 2], [3]],)], "values
array<array<int>>")
+ assertDataFrameEqual(
+ nested.select(
+ sf.transform(
+ "values",
+ lambda a: sf.transform(a, lambda x: keep(draw(x)) ==
keep(draw(x))),
+ ).alias("r")
+ ),
+ [([[False, False], [False]],)],
+ )
+
def test_fused_transforms_with_different_lengths(self):
# SPARK-27052: two transforms over arrays of different per-row lengths
and null layouts get
# fused by ExtractPythonUDFs into one element-wise batch. Each UDF
must be re-nested by its
@@ -714,6 +802,59 @@ class UDFInHigherOrderFunctionTestsMixin:
# (1+1)+(2+1) + (3+1) = 9
assertDataFrameEqual(grouped, [(9,)])
+ def test_udf_in_lambda_over_aggregate_result(self):
+ # Once lifted out of the lambda, the UDF reads the aggregate's result,
so it must be
+ # evaluated above the aggregate. Below it, neither an aggregate
function nor the grouped
+ # value of a grouping key exists, and the query failed at runtime.
+ df = self.spark.createDataFrame([(1, 1), (1, 2), (2, 3)], "k int, v
int")
+ plus_one = udf(lambda x: x + 1, IntegerType())
+
+ assertDataFrameEqual(
+ df.groupBy("k").agg(
+ sf.sort_array(sf.transform(sf.collect_list("v"), lambda x:
plus_one(x))).alias("r")
+ ),
+ [(1, [2, 3]), (2, [4])],
+ )
+ assertDataFrameEqual(
+ df.groupBy("k").agg(
+ sf.transform(sf.array(sf.sum("v")), lambda x:
plus_one(x)).alias("r")
+ ),
+ [(1, [4]), (2, [4])],
+ )
+
+ arrays = self.spark.createDataFrame([([1, 2],), ([1, 2],), ([3],)],
"values array<int>")
+ assertDataFrameEqual(
+ arrays.groupBy("values").agg(sf.transform("values", lambda x:
plus_one(x)).alias("r")),
+ [([1, 2], [2, 3]), ([3], [4])],
+ )
+
+ def test_udf_in_lambda_in_outer_join_condition(self):
+ # A left outer join keeps a predicate over only its left side in the
join condition. Once
+ # lifted, the UDF reads only the left side and is evaluated there,
like a plain UDF.
+ left = self.spark.createDataFrame([(1, [1, 2]), (2, [3])], "k int,
values array<int>")
+ right = self.spark.createDataFrame([(1, "a"), (2, "b")], "k int, v
string")
+ plus_one = udf(lambda x: x + 1, IntegerType())
+
+ joined = left.join(
+ right,
+ (left.k == right.k) & sf.exists(left["values"], lambda x:
plus_one(x) > 3),
+ "left_outer",
+ ).select(left.k, right.v)
+ assertDataFrameEqual(joined, [(1, None), (2, "b")])
+
+ def test_udf_in_lambda_in_inner_join_condition(self):
+ # The predicate reads both sides, so it stays in the join condition.
Once lifted, the UDF
+ # reads only the left side and is evaluated there, as a plain one-side
UDF is.
+ left = self.spark.createDataFrame([(1, [1, 2]), (2, [3])], "k int,
values array<int>")
+ right = self.spark.createDataFrame([(1, 3), (2, 3), (2, 5)], "k int, t
int")
+ plus_one = udf(lambda x: x + 1, IntegerType())
+
+ joined = left.join(
+ right,
+ (left.k == right.k) & sf.exists(left["values"], lambda x:
plus_one(x) > right.t),
+ ).select(left.k, right.t)
+ assertDataFrameEqual(joined, [(2, 3)])
+
def test_mixed_with_plain_python_udf(self):
df = self.spark.createDataFrame([([1, 2],)], "values array<int>")
plus_one = udf(lambda x: x + 1, IntegerType())
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
index 84b43f59631e..045301ff505e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala
@@ -70,6 +70,23 @@ object PythonUDF {
e.asInstanceOf[PythonUDF].evalType !=
PythonEvalType.SQL_SCALAR_ARROW_INPROCESS_UDF
}
+ /**
+ * The key that decides whether two Python UDF calls are the same call,
evaluated once. A
+ * deterministic call is keyed by its canonical form, so identical calls
share one evaluation. A
+ * nondeterministic call is keyed by itself: in `f(x) + f(x)` each call may
return a different
+ * value, and only the calls' `resultId`s, which `canonicalized` drops, keep
them apart.
+ *
+ * A call is deterministic only if its arguments are too (`deterministic`,
not
+ * `udfDeterministic`). In `f(r(x)) = f(r(x))` with a nondeterministic `r`,
each `f` call wraps
+ * its own draw of `r`; canonicalizing the `f` calls would drop the inner
calls' `resultId`s as
+ * well and merge both sides into one call, compared with itself.
+ *
+ * Shared by `ExtractPythonUDFs` and `ExtractPythonUDFFromLambda`, so that
calls the latter keeps
+ * apart when lifting them out of a lambda are not merged by the former
afterwards.
+ */
+ def canonicalizeDeterministic(udf: PythonUDF): PythonUDF =
+ if (udf.deterministic) udf.canonicalized.asInstanceOf[PythonUDF] else udf
+
/**
* Whether `e` is a Python UDF that can be lifted out of a higher-order
function's lambda by
* `ExtractPythonUDFFromLambda`, which applies it to the whole array outside
the lambda.
@@ -181,8 +198,8 @@ object PythonUDF {
* each lifted UDF's argument, the `map_keys`/`map_values` desugar, the
pairwise `array_sort`
* path), and nondeterministic expressions are not
subexpression-eliminated, so the copies
* would evaluate independently and disagree - keeping the results
misaligned. (This is
- * distinct from a nondeterministic UDF *call*, which
`ExtractPythonUDFFromLambda.liftKey`
- * keeps distinct but well-defined.)
+ * distinct from a nondeterministic UDF *call*, which
[[canonicalizeDeterministic]] keeps
+ * distinct but well-defined.)
* - a HOF (at any level) whose shape the rewrite does not model (see
[[isRewritableShape]]).
*/
def canRewritePythonUDFInLambda(hof: HigherOrderFunction): Boolean = {
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
index 11db680fefb8..650ba5eefdb2 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
@@ -28,7 +28,7 @@ import
org.apache.spark.sql.execution.datasources.v2.{GroupBasedRowLevelOperatio
import
org.apache.spark.sql.execution.dynamicpruning.{CleanupDynamicPruningFilters,
PartitionPruning, RowLevelOperationRuntimeGroupFiltering}
import
org.apache.spark.sql.execution.externalUDF.{ExtractExternalUDFFromWindow,
PlanExternalUDFs}
import org.apache.spark.sql.execution.planmerging.MergeSubplans
-import
org.apache.spark.sql.execution.python.{ExtractGroupingPythonUDFFromAggregate,
ExtractPythonUDFFromAggregate, ExtractPythonUDFs, ExtractPythonUDTFs}
+import
org.apache.spark.sql.execution.python.{ExtractGroupingPythonUDFFromAggregate,
ExtractPythonUDFFromAggregate, ExtractPythonUDFFromLambda, ExtractPythonUDFs,
ExtractPythonUDTFs}
class SparkOptimizer(
catalogManager: CatalogManager,
@@ -87,6 +87,10 @@ class SparkOptimizer(
PruneFilters),
postHocOptimizationBatches,
Batch("Extract UDFs", Once,
+ // Lift Python UDFs out of higher-order function lambdas first. A UDF in
a lambda reads the
+ // lambda's variables, so the rules below can place it by the columns it
really reads (e.g.
+ // above an aggregate it is applied to) only once it is an ordinary UDF
over a whole array.
+ ExtractPythonUDFFromLambda,
ExtractPythonUDFFromJoinCondition,
// Expose window results as attributes before creating external UDF
evaluation nodes.
ExtractExternalUDFFromWindow,
@@ -94,8 +98,6 @@ class SparkOptimizer(
ExtractPythonUDFFromAggregate,
// This must be executed after `ExtractPythonUDFFromAggregate` and
before `ExtractPythonUDFs`.
ExtractGroupingPythonUDFFromAggregate,
- // `ExtractPythonUDFs` first lifts Python UDFs out of higher-order
function lambdas
- // (via `ExtractPythonUDFFromLambda`) and then extracts them as ordinary
top-level UDFs.
ExtractPythonUDFs,
ExtractPythonUDTFs,
// The eval-python node may be between Project/Filter and the scan node,
which breaks
@@ -123,13 +125,14 @@ class SparkOptimizer(
override def nonExcludableRules: Seq[String] = super.nonExcludableRules ++
Seq(
+ // A plan with a Python UDF in a higher-order function lambda passes
analysis only because
+ // this rule lifts the UDF out of the lambda.
+ ExtractPythonUDFFromLambda.ruleName,
ExtractPythonUDFFromJoinCondition.ruleName,
ExtractPythonUDFFromAggregate.ruleName,
ExtractGroupingPythonUDFFromAggregate.ruleName,
ExtractExternalUDFFromWindow.ruleName,
PlanExternalUDFs.ruleName,
- // Non-excludable: a plan with a Python UDF in a higher-order function
lambda only works
- // because `ExtractPythonUDFs` lifts it out (via
`ExtractPythonUDFFromLambda`).
ExtractPythonUDFs.ruleName,
GroupBasedRowLevelOperationScanPlanning.ruleName,
V2ScanRelationPushDown.ruleName,
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
index dfaa4732ca9d..1609bc0c2990 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.execution.python
import org.apache.spark.api.python.PythonEvalType
import org.apache.spark.sql.catalyst.expressions._
+import
org.apache.spark.sql.catalyst.expressions.PythonUDF.canonicalizeDeterministic
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.TreePattern._
@@ -55,10 +56,16 @@ import org.apache.spark.sql.types.{ArrayType, IntegerType,
MapType}
* them uniformly with no per-argument metadata. With the result now an
ordinary column, arithmetic,
* `when`, casts, the element index, multiple UDFs and nested calls `f(g(x))`
all just work.
*
- * Runs before [[ExtractPythonUDFs]]. Handles all ten single-lambda functions:
`transform`,
- * `filter`, `exists`, `forall`, `zip_with`, `array_sort`, and the four map
functions (desugared to
- * `map_keys`/`map_values` arrays and rebuilt with `map_from_arrays`).
`array_sort` precomputes a
- * per-element key, or, when one call takes both elements, the UDF over the
cross product of pairs.
+ * Runs first in the optimizer's UDF extraction batch, before every rule that
decides where a
+ * Python UDF is evaluated from the columns it reads
(`ExtractPythonUDFFromJoinCondition`,
+ * [[ExtractPythonUDFFromAggregate]], [[ExtractPythonUDFs]], ...): inside a
lambda a UDF reads the
+ * lambda's variables, so only its lifted form tells them, e.g., that it
applies to an aggregate's
+ * result and must be evaluated above the aggregate. Handles all ten
single-lambda functions:
+ * `transform`, `filter`, `exists`, `forall`, `zip_with`, `array_sort`, and
the four map functions
+ * (desugared to `map_keys`/`map_values` arrays and rebuilt with
`map_from_arrays`). `array_sort`
+ * precomputes a per-element key - one per element even for a nondeterministic
UDF, so that an
+ * element has the same key on either side of a comparison - or, when one call
takes both elements,
+ * the UDF over the cross product of pairs.
* A UDF in a *nested* lambda, `transform(arr, i -> transform(i, x -> f(x)))`,
is handled too: the
* whole nest is rewritten root-first (see `apply` / `rewriteNest`), lifting
the UDF out one lambda
* level at a time so it ends up applied to the fully flattened leaves.
@@ -154,6 +161,8 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan]
{
* no per-element key. Precomputes the UDF over every ordered pair - an n x
n matrix with
* `udf(arr[i], arr[j])` at (i, j) - and the comparator reads it by the two
elements' positions,
* so no Python runs while sorting. Costs O(n^2) calls and memory vs. O(n)
for a per-element key.
+ * A nondeterministic UDF is drawn once per ordered pair, so `udf(x, y)` and
`udf(y, x)` need not
+ * agree: with no per-element key to share, such a comparator does not
define an order.
*/
private def rewritePairwiseComparator(sort: ArraySort): Expression = {
val ArraySort(argument, function, allowNull) = sort
@@ -361,8 +370,6 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan]
{
indexVar: Option[NamedLambdaVariable],
alsoBind: Seq[NamedLambdaVariable] = Nil): Carrier = {
val LambdaFunction(body, _, _) = function
- val lambdaExprIds =
- (elementVars ++ indexVar.toSeq ++ alsoBind).map(_.exprId).toSet
// Collect the UDF calls to lift. Innermost first, so that a nested call
like `f(g(x))` has
// `g` lifted before `f`, letting `f`'s array UDF consume `g`'s array
result.
@@ -398,13 +405,13 @@ object ExtractPythonUDFFromLambda extends
Rule[LogicalPlan] {
indexVar.map(_.exprId -> indexArray.get).toMap ++
alsoBind.map(_.exprId -> alignedArguments.head).toMap
- // One lifted array UDF per distinct call. `arrayResults` maps each
original call (by `liftKey`)
- // to the array holding its per-element results, so a nested call
`f(g(x))` and the carrier
- // lookups can find it. Deterministic calls that lift to the *same*
function over the *same*
- // array arguments share one lifted UDF: a key-form comparator's `udf(a)`
and `udf(b)`
- // canonicalize differently (`a` != `b`) but both read the whole array, so
without this the
- // Python function would run 2n times instead of n. Nondeterministic calls
stay distinct (their
- // signature is the lifted node itself, carrying a distinct `resultId`),
matching `liftKey`.
+ // One lifted array UDF per distinct call. `arrayResults` maps each
original call (by
+ // `canonicalizeDeterministic`) to the array holding its per-element
results, so a nested call
+ // `f(g(x))` and the carrier lookups can find it. Deterministic calls that
lift to the *same*
+ // function over the *same* array arguments share one lifted UDF: a
key-form comparator's
+ // `udf(a)` and `udf(b)` canonicalize differently (`a` != `b`) but both
read the whole array, so
+ // without this the Python function would run 2n times instead of n.
Nondeterministic calls stay
+ // distinct, except in a key-form comparator (see `signature` below).
var arrayResults = Map.empty[Expression, Expression]
val distinctLifted = scala.collection.mutable.ArrayBuffer.empty[PythonUDF]
val ordinalBySignature =
scala.collection.mutable.HashMap.empty[Expression, Int]
@@ -417,9 +424,9 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan]
{
val arrayArgs = udf.children.map {
case NamedArgumentExpression(key, value) =>
NamedArgumentExpression(
- key, overArray(value, alignedArguments.head, arrayOfVar,
lambdaExprIds, arrayResults))
+ key, overArray(value, alignedArguments.head, arrayOfVar,
arrayResults))
case child =>
- overArray(child, alignedArguments.head, arrayOfVar, lambdaExprIds,
arrayResults)
+ overArray(child, alignedArguments.head, arrayOfVar, arrayResults)
}
// Each lift wraps the arguments in exactly one more `array` level.
Lifting a base UDF gives
// depth 1; re-lifting an already-lifted element-wise UDF (a UDF from a
*nested* lambda,
@@ -442,14 +449,23 @@ object ExtractPythonUDFFromLambda extends
Rule[LogicalPlan] {
PythonUDF.liftedElementwiseEvalType(udf.evalType),
udf.udfDeterministic,
elementwiseNestingDepth = newDepth)
- val signature: Expression = if (udf.udfDeterministic)
lifted.canonicalized else lifted
+ // Calls share one lifted UDF when their lifted forms have the same key;
`overArray` keeps
+ // each argument's determinism, so `lifted` is deterministic exactly
when `udf` is. A key-form
+ // comparator (the one caller passing `alsoBind`) shares a
nondeterministic call too: its
+ // lifted result is each element's sort key, and an element must have
one key whichever side
+ // of a comparison it is on. With a draw per call, `(a, b) -> r(a) -
r(b)` would not be
+ // antisymmetric, and the sort could fail with "Comparison method
violates its general
+ // contract!".
+ val signature =
+ if (alsoBind.nonEmpty) lifted.canonicalized else
canonicalizeDeterministic(lifted)
val ordinal = ordinalBySignature.getOrElseUpdate(signature, {
val o = distinctLifted.length
distinctLifted += lifted
o
})
- arrayResults += (liftKey(udf) -> distinctLifted(ordinal))
- udfFieldByKey += (liftKey(udf) -> ordinal)
+ val key = canonicalizeDeterministic(udf)
+ arrayResults += (key -> distinctLifted(ordinal))
+ udfFieldByKey += (key -> ordinal)
}
val liftedArrays = distinctLifted.toSeq
@@ -490,8 +506,8 @@ object ExtractPythonUDFFromLambda extends Rule[LogicalPlan]
{
}
val rewrittenBody = body.transformDown {
- case udf: PythonUDF if udfFieldByKey.contains(liftKey(udf)) =>
- val ordinal = udfFieldByKey(liftKey(udf))
+ case udf: PythonUDF if
udfFieldByKey.contains(canonicalizeDeterministic(udf)) =>
+ val ordinal = udfFieldByKey(canonicalizeDeterministic(udf))
// A UDF over a comparator's right-hand element must read that
element's key, so the
// struct field is read through whichever bound variable the call's
own arguments used.
val side = udf.collectFirst {
@@ -541,28 +557,12 @@ object ExtractPythonUDFFromLambda extends
Rule[LogicalPlan] {
}
visit(body)
// Deduplicate identical calls so the same UDF is evaluated once per
array. Nondeterministic
- // calls are kept distinct (see `liftKey`).
+ // calls are kept distinct (see `canonicalizeDeterministic`).
val seen = scala.collection.mutable.LinkedHashMap.empty[Expression,
PythonUDF]
- collected.result().foreach(udf => seen.getOrElseUpdate(liftKey(udf), udf))
+ collected.result().foreach(udf =>
seen.getOrElseUpdate(canonicalizeDeterministic(udf), udf))
seen.values.toSeq
}
- private def readsLambdaVariable(e: Expression, lambdaExprIds: Set[ExprId]):
Boolean =
- e.exists {
- case v: NamedLambdaVariable => lambdaExprIds.contains(v.exprId)
- case _ => false
- }
-
- /**
- * The key that decides whether two UDF calls are "the same call" for
lifting. A deterministic
- * call is deduplicated by canonical form, so an identical call is evaluated
once. A
- * nondeterministic call must stay distinct - `transform(arr, x -> f(x) +
f(x))` calls `f` twice
- * and each call may return a different value - so it is keyed by its own
`resultId`-bearing node
- * (`canonicalized` erases `resultId`, which would collapse the two calls
into one).
- */
- private def liftKey(udf: PythonUDF): Expression =
- if (udf.udfDeterministic) udf.canonicalized else udf
-
/**
* Turns a UDF argument expression, written in terms of single elements,
into the equivalent
* `array<T>` aligned with the iterated array, so the worker can flatten
every argument uniformly.
@@ -585,28 +585,36 @@ object ExtractPythonUDFFromLambda extends
Rule[LogicalPlan] {
child: Expression,
firstArgument: Expression,
arrayOfVar: Map[ExprId, Expression],
- lambdaExprIds: Set[ExprId],
arrayResults: Map[Expression, Expression]): Expression = child match {
case v: NamedLambdaVariable if arrayOfVar.contains(v.exprId) =>
arrayOfVar(v.exprId)
- case udf: PythonUDF if arrayResults.contains(liftKey(udf)) =>
- arrayResults(liftKey(udf))
+ case udf: PythonUDF if
arrayResults.contains(canonicalizeDeterministic(udf)) =>
+ arrayResults(canonicalizeDeterministic(udf))
case e =>
// Replace each already-lifted nested UDF call with a synthetic variable
standing for that
// call's aligned array result, then fold those arrays into the
variable-to-array map so the
// logic below treats them exactly like element variables. This keeps a
lifted UDF buried in
// a composite argument (`f(g(x) + 1)`) from being left as a raw UDF
inside a lambda.
+ // Top-down, so the outermost lifted call is replaced whole: bottom-up
would first replace
+ // a lifted call nested in its arguments (`h(x)` in `f(g(h(x)) + 1)`),
after which `g(...)`
+ // no longer matches its own lifted result and would be left behind as a
raw UDF.
val nestedVars =
scala.collection.mutable.LinkedHashMap.empty[Expression, NamedLambdaVariable]
- val expr = e.transformUp {
- case u: PythonUDF if arrayResults.contains(liftKey(u)) =>
- nestedVars.getOrElseUpdate(liftKey(u), {
- val arrType =
arrayResults(liftKey(u)).dataType.asInstanceOf[ArrayType]
+ val expr = e.transformDown {
+ case u: PythonUDF if
arrayResults.contains(canonicalizeDeterministic(u)) =>
+ val key = canonicalizeDeterministic(u)
+ nestedVars.getOrElseUpdate(key, {
+ val arrType = arrayResults(key).dataType.asInstanceOf[ArrayType]
NamedLambdaVariable("g", arrType.elementType, arrType.containsNull)
})
}
- val fullArrayOf = arrayOfVar ++
- nestedVars.map { case (key, v) => v.exprId -> arrayResults(key) }
-
- if (!readsLambdaVariable(expr, fullArrayOf.keySet)) {
+ // The arrays of just the variables `expr` reads: zipping in one it does
not read, such as the
+ // element array for `f(g(x) + 1)`, which reads only `g`'s result, would
build a struct per
+ // element for nothing.
+ val readIds = expr.collect { case v: NamedLambdaVariable => v.exprId
}.toSet
+ val arrayOfRead =
+ (arrayOfVar ++ nestedVars.map { case (key, v) => v.exprId ->
arrayResults(key) })
+ .filter { case (id, _) => readIds.contains(id) }
+
+ if (arrayOfRead.isEmpty) {
// Independent of the element (an outer column or constant): repeat it
into an array aligned
// with the iterated array, so every UDF argument is a single-level
array the worker
// flattens the same way. `transform(arr, _ -> e)` keeps the value
constant, matching shape.
@@ -616,23 +624,24 @@ object ExtractPythonUDFFromLambda extends
Rule[LogicalPlan] {
} else {
// An expression over the element(s) and/or nested results, e.g.
`udf(x * 2)` or
// `f(g(x) + 1)`. Compute it for every element with a native
`transform`, which stays inside
- // the JVM. Multi-argument shapes need the values side by side, so zip
them first.
- val arrays = fullArrayOf.values.toSeq.distinct
+ // the JVM. An expression reading several arrays needs their values
side by side, so zip
+ // them first.
+ val arrays = arrayOfRead.values.toSeq.distinct
if (arrays.length == 1) {
val arr = arrays.head
val elemType = arr.dataType.asInstanceOf[ArrayType]
val v = NamedLambdaVariable("x", elemType.elementType,
elemType.containsNull)
val substituted = expr.transformUp {
- case old: NamedLambdaVariable if fullArrayOf.contains(old.exprId)
=> v
+ case old: NamedLambdaVariable if arrayOfRead.contains(old.exprId)
=> v
}
ArrayTransform(arr, LambdaFunction(substituted, Seq(v)))
} else {
- // Zip every array the expression may read, then project the fields
it needs.
+ // Zip the arrays the expression reads, then project the fields it
needs.
val names = arrays.indices.map(i => s"$carrierElementPrefix$i")
val zipped = ArraysZip(arrays, names.map(Literal(_)))
val structType = zipped.dataType.asInstanceOf[ArrayType].elementType
val v = NamedLambdaVariable("z", structType, nullable = false)
- val ordinalOf = fullArrayOf.map { case (id, arr) => id ->
arrays.indexOf(arr) }
+ val ordinalOf = arrayOfRead.map { case (id, arr) => id ->
arrays.indexOf(arr) }
val substituted = expr.transformUp {
case old: NamedLambdaVariable if ordinalOf.contains(old.exprId) =>
GetStructField(v, ordinalOf(old.exprId),
Some(names(ordinalOf(old.exprId))))
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
index 3ad83a74c056..052949dd6146 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala
@@ -25,7 +25,8 @@ import org.apache.spark.api.python.PythonEvalType
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys.REASON
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.PythonUDF.{correctEvalType,
isScalarPythonUDF}
+import
org.apache.spark.sql.catalyst.expressions.PythonUDF.{canonicalizeDeterministic,
+ correctEvalType, isScalarPythonUDF}
import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
@@ -272,14 +273,7 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with
Logging {
expressions.flatMap(collectEvaluableUDFs)
}
- def apply(plan: LogicalPlan): LogicalPlan = applyInternal(
- // Lift Python UDFs out of higher-order function lambdas first, turning
them into ordinary
- // top-level UDFs this rule can then extract. Running it here rather than
as a separate batch
- // entry makes the ordering structural: `CheckAnalysis` accepts a
lambda-UDF plan only because
- // this rewrite will run, so the two must not be separable by rule
reordering or exclusion.
- ExtractPythonUDFFromLambda(plan))
-
- private def applyInternal(plan: LogicalPlan): LogicalPlan = plan match {
+ def apply(plan: LogicalPlan): LogicalPlan = plan match {
// SPARK-26293: A subquery will be rewritten into join later, and will go
through this rule
// eventually. Here we skip subquery, as Python UDF only needs to be
extracted once.
case s: Subquery if s.correlated => plan
@@ -298,14 +292,6 @@ object ExtractPythonUDFs extends Rule[LogicalPlan] with
Logging {
}
}
- private def canonicalizeDeterministic(u: PythonUDF) = {
- if (u.deterministic) {
- u.canonicalized.asInstanceOf[PythonUDF]
- } else {
- u
- }
- }
-
/**
* Extract all the PythonUDFs from the current operator and evaluate them
before the operator.
*/
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
index 09d982026006..46596bef42b8 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambdaSuite.scala
@@ -19,11 +19,11 @@ package org.apache.spark.sql.execution.python
import org.apache.spark.api.python.PythonEvalType
import org.apache.spark.sql.{AnalysisException, QueryTest}
-import org.apache.spark.sql.catalyst.expressions.{LambdaFunction, Literal,
NamedArgumentExpression,
- PythonUDF}
-import org.apache.spark.sql.catalyst.plans.logical.ArrowEvalPython
-import org.apache.spark.sql.functions.{array_sort, col, forall, lit,
map_filter, map_zip_with,
- transform, transform_keys, transform_values, when, zip_with}
+import org.apache.spark.sql.catalyst.expressions.{ArraysZip, LambdaFunction,
Literal,
+ NamedArgumentExpression, PythonUDF}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate,
ArrowEvalPython, Join}
+import org.apache.spark.sql.functions.{array_sort, col, collect_list, forall,
lit, map_filter,
+ map_zip_with, transform, transform_keys, transform_values, when, zip_with}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.{ExamplePointUDT, SharedSparkSession}
import org.apache.spark.sql.types.{ArrayType, IntegerType, MapType, StringType}
@@ -144,6 +144,42 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest
with SharedSparkSession
assert(liftedUDFs(negated).size == 2)
}
+ test("a lifted call two levels deep inside a composite argument is
substituted whole") {
+ // `f(g(h(x)) + 1)`: `f`'s composite argument holds the lifted call
`g(h(x))`, which itself
+ // holds the lifted call `h(x)`. Substituting bottom-up would replace
`h(x)` first, so `g(...)`
+ // no longer matches its lifted result and a raw `g` is left inside the
generated lambda. That
+ // `g` is lifted a second time when it can be, and stays inside the lambda
when it cannot (its
+ // lambda then iterates the nondeterministic result of `h`).
+ def nested(inner: UserDefinedPythonFunction):
org.apache.spark.sql.DataFrame =
+ arrayDF.select(transform(col("values"), x =>
+ pythonUDF(pythonUDF(inner(x)).cast("int") + lit(1))).as("r"))
+
+ val deterministic = nested(pythonUDF)
+ assert(udfsInsideLambda(deterministic).isEmpty)
+ // One lifted UDF per call: `h`, `g` and `f`.
+ assert(liftedUDFs(deterministic).size == 3)
+
+ assert(udfsInsideLambda(nested(nondeterministicUDF)).isEmpty)
+ }
+
+ test("a composite argument over a nested call zips only the arrays it
reads") {
+ // `f(g(x) + 1)` reads only `g`'s result, not the element, so its argument
is a `transform` over
+ // that result alone: zipping in the element array as well would build a
struct per element for
+ // nothing. With the index, `f(g(x) + i)` zips just the index and `g`'s
result.
+ def zippedArrayCounts(df: org.apache.spark.sql.DataFrame): Seq[Int] =
+ liftedUDFs(df).flatMap(_.collect { case z: ArraysZip => z.children.size
})
+
+ val plusOne = arrayDF.select(
+ transform(col("values"), x => pythonUDF(pythonUDF(x).cast("int") +
lit(1))).as("r"))
+ assert(udfsInsideLambda(plusOne).isEmpty)
+ assert(zippedArrayCounts(plusOne).isEmpty)
+
+ val plusIndex = arrayDF.select(
+ transform(col("values"), (x, i) => pythonUDF(pythonUDF(x).cast("int") +
i)).as("r"))
+ assert(udfsInsideLambda(plusIndex).isEmpty)
+ assert(zippedArrayCounts(plusIndex) == Seq(2))
+ }
+
test("identical nondeterministic calls are lifted distinctly, not
deduplicated") {
// Deduplicating `f(x)` with `f(x)` would collapse two independent draws
into one; a
// nondeterministic UDF must keep each call, so both are lifted.
@@ -153,6 +189,18 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest
with SharedSparkSession
assert(liftedUDFs(df).size == 2)
}
+ test("a deterministic UDF over a nondeterministic argument is not
deduplicated") {
+ // `f(r(x)) === f(r(x))`: `f` itself is deterministic, but each side wraps
its own draw of the
+ // nondeterministic `r`, so the two `f` calls are distinct. Keying them by
canonical form would
+ // erase the inner calls' `resultId`s and lift one `f` call for both
sides, comparing a single
+ // draw with itself.
+ val df = arrayDF.select(transform(col("values"), x =>
+ pythonUDF(nondeterministicUDF(x)) ===
pythonUDF(nondeterministicUDF(x))).as("r"))
+ assert(udfsInsideLambda(df).isEmpty)
+ // Two inner draws, and one outer call over each of them.
+ assert(liftedUDFs(df).size == 4)
+ }
+
test("transform_values whose result type equals the key type replaces
values, not keys") {
// SPARK-27052: dispatch is by concrete function, not result type. For
map<string, string>,
// transform_values' lambda also returns string, which must not be treated
as new keys.
@@ -283,6 +331,27 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest
with SharedSparkSession
assert(liftedUDFs(df).size == 1)
}
+ test("a key-form array_sort comparator shares a nondeterministic call
between its sides") {
+ // An element must have one sort key whichever side of a comparison it is
on, so `k(a)` and
+ // `k(b)` share one lifted call even when `k` is nondeterministic: one
draw per element. A draw
+ // per side would make the comparator inconsistent, and the sort could
fail with "Comparison
+ // method violates its general contract!". The same holds for a
deterministic UDF over a
+ // nondeterministic argument, `f(r(a))` and `f(r(b))`.
+ val bare = arrayDF.select(
+ array_sort(col("values"), (a, b) =>
+ when(nondeterministicUDF(a) === nondeterministicUDF(b),
lit(0)).otherwise(lit(1))).as("r"))
+ assert(udfsInsideLambda(bare).isEmpty)
+ assert(liftedUDFs(bare).size == 1)
+
+ val wrapped = arrayDF.select(
+ array_sort(col("values"), (a, b) =>
+ when(pythonUDF(nondeterministicUDF(a)) ===
pythonUDF(nondeterministicUDF(b)), lit(0))
+ .otherwise(lit(1))).as("r"))
+ assert(udfsInsideLambda(wrapped).isEmpty)
+ // One draw per element, and one outer call over it.
+ assert(liftedUDFs(wrapped).size == 2)
+ }
+
test("a vectorized scalar UDF inside a lambda is lifted to its element-wise
eval type") {
// Each vectorized scalar flavor lifts to its own element-wise eval type
so the worker keeps
// that flavor's batching contract (pandas Series, Arrow Array, or an
iterator of batches).
@@ -404,9 +473,9 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest
with SharedSparkSession
}
test("the rewrite cannot be disabled via excludedRules") {
- // The lambda rewrite is driven by `ExtractPythonUDFs` (not a standalone
batch rule), and that
- // rule is non-excludable, so a plan that only works because of the
rewrite cannot be broken by
- // excludedRules. Excluding either name must leave the UDF lifted out of
the lambda.
+ // The lambda rewrite is a non-excludable rule, so a plan that only works
because of the rewrite
+ // cannot be broken by excludedRules. Excluding it (or
`ExtractPythonUDFs`, which extracts the
+ // lifted UDF) must leave the UDF lifted out of the lambda.
withSQLConf(
SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
Seq(ExtractPythonUDFFromLambda.ruleName,
ExtractPythonUDFs.ruleName).mkString(",")) {
@@ -415,6 +484,57 @@ class ExtractPythonUDFFromLambdaSuite extends QueryTest
with SharedSparkSession
}
}
+ test("a UDF in a lambda over an aggregate's result is evaluated above the
aggregate") {
+ // `transform(collect_list(id), x -> f(x))`: once lifted, the UDF reads
the aggregate's result,
+ // so it must be lifted before `ExtractPythonUDFFromAggregate` runs, which
then evaluates it
+ // above the `Aggregate`. Lifted later, it would be evaluated below the
aggregate, where neither
+ // an aggregate function nor (for a lambda over a grouping key) the
grouped value exists.
+ val overAggregate = spark.range(3).groupBy()
+ .agg(transform(collect_list(col("id")), x => pythonUDF(x)).as("r"))
+ val overGroupingKey = Seq(Seq(1, 2),
Seq(3)).toDF("values").groupBy("values")
+ .agg(transform(col("values"), x => pythonUDF(x)).as("r"))
+ Seq(overAggregate, overGroupingKey).foreach { df =>
+ assert(udfsInsideLambda(df).isEmpty)
+ val evals = df.queryExecution.optimizedPlan.collect { case a:
ArrowEvalPython => a }
+ assert(evals.size == 1)
+ assert(evals.head.child.exists(_.isInstanceOf[Aggregate]))
+ }
+ }
+
+ test("a UDF in a lambda over one side of an outer join condition is
evaluated on that side") {
+ // A left outer join keeps a predicate over only its left side in the join
condition. Once
+ // lifted, the UDF reads only left-side columns and is evaluated below the
join on that side,
+ // like a plain UDF. While still inside the lambda it read neither side,
so it was rejected as a
+ // Python UDF in the ON clause of an outer join.
+ val left = Seq((1, Seq(1, 2))).toDF("k", "values")
+ val right = Seq((1, "a")).toDF("k", "v")
+ val df = left.join(
+ right,
+ left("k") === right("k") &&
+ org.apache.spark.sql.functions.exists(left("values"), x =>
pythonUDF(x)),
+ "left_outer")
+ val join = df.queryExecution.optimizedPlan.collectFirst { case j: Join =>
j }.get
+ assert(join.left.exists(_.isInstanceOf[ArrowEvalPython]))
+ assert(!join.right.exists(_.isInstanceOf[ArrowEvalPython]))
+ }
+
+ test("a UDF in a lambda over one side of an inner join condition is
evaluated on that side") {
+ // The conjunct reads both sides, so it stays in the join condition, but
once lifted the UDF
+ // itself reads only the left side and is evaluated below the join on that
side, as a plain
+ // one-side UDF in a join condition is. While still inside the lambda it
read neither side, so
+ // the whole conjunct was pulled out into a `Filter` above the join.
+ val left = Seq((1, Seq(1, 2))).toDF("k", "values")
+ val right = Seq((1, true)).toDF("k", "flag")
+ val df = left.join(
+ right,
+ left("k") === right("k") &&
+ org.apache.spark.sql.functions.exists(left("values"), x =>
pythonUDF(x) === right("flag")),
+ "inner")
+ val join = df.queryExecution.optimizedPlan.collectFirst { case j: Join =>
j }.get
+ assert(join.left.exists(_.isInstanceOf[ArrowEvalPython]))
+ assert(!join.right.exists(_.isInstanceOf[ArrowEvalPython]))
+ }
+
test("a UDF over only constants is still lifted per element") {
// SPARK-27052: `transform(arr, x -> udf(lit(10)))` does not read the
element, but it is still
// lifted per element (`overArray` repeats the constant into an aligned
array) so it keeps the
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]