This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 22ed7311 [SPARK-61142][CONNECT][PYTHON] Make the functions of
pyspark.sql.functions those of pyspark.sql.functions.builtin, which pickle by
reference, in the Python client as PySpark does
22ed7311 is described below
commit 22ed73117942633455659f58abc398e10494a773
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:44:18 2026 +0900
[SPARK-61142][CONNECT][PYTHON] Make the functions of pyspark.sql.functions
those of pyspark.sql.functions.builtin, which pickle by reference, in the
Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61142](https://issues.apache.org/jira/browse/SPARK-61142): PySpark
defines the functions of `pyspark.sql.functions` in
`pyspark.sql.functions.builtin`, all but `pandas_udf` and `arrow_udf` (of
`pyspark.sql.pandas.functions`). So `pickle` and `cloudpickle` pickle them by
reference, by that module and name.
The client's functions now have the same `__module__`,
`pyspark.sql.functions.builtin`, the synthetic module that already mirrors
`pyspark.sql.functions`:
- the 518 generated ones, which were of `functions_generated`
- those of `functions.py`, such as `col` and `lit`, which were of
`pyspark.sql.functions`
- `udf`, `udtf` and `arrow_udtf`, which were of `pyspark.sql.udf` and
`pyspark.sql.udtf`
Only a function that the `builtin` module gives back by its name is
changed, so each one pickles by reference. On the server, the Python worker
imports PySpark's function of that name.
### Why are the changes needed?
`functions_generated` is not importable. So `cloudpickle` pickled the
generated functions by value, and failed on the Rust functions that they call:
```
_pickle.PicklingError: Can't pickle <built-in function
pyfunc_invoke_function>: import of module 'functions' failed
```
A UDF, a UDTF or a `foreachBatch` function that captured one of them, such
as `array_sort`, could not be sent. Apache Spark's
`test_parity_pandas_transform_with_state_state_variable.py` failed on this
client: its `foreachBatch` functions capture `array_sort` and `col`
(`STREAMING_CONNECT_SERIALIZATION_ERROR`).
### Does this PR introduce _any_ user-facing change?
Yes:
- UDFs, UDTFs and `foreachBatch` functions that capture functions of
`pyspark.sql.functions` can be sent, as with PySpark 4.3.0.
- The `__module__` of those functions is `pyspark.sql.functions.builtin`,
as PySpark's.
### How was this patch tested?
- A new offline test checks:
- the module of every function of `pyspark.sql.functions.__all__`, and of
the deprecated `countDistinct`
- that `pickle` gives back each function itself
- that `cloudpickle` pickles a function that captures some by their
names, without any Rust function
It passes with official PySpark 4.3.0, and fails without this change.
- A new live test runs a UDF that captures `array_sort`, whose worker
imports it. It fails without this change (`PicklingError`).
- The modules of all the functions of `pyspark.sql.functions`, also those
that `__all__` leaves out, are those of official PySpark 4.3.0, and all of them
pickle.
- Apache Spark's
`test_parity_pandas_transform_with_state_state_variable.py` passes on this
client against a local server: 19 tests pass and 1 is skipped. Before this
change, 3 failed.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2202 passed, 8 skipped), and the
live drop-in and Arrow suites (185 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #335 from HyukjinKwon/SPARK-61142-functions-pickle.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/functions.py | 18 ++++++++++++++++++
python/tests/test_dropin_e2e.py | 14 ++++++++++++++
python/tests/test_dropin_offline.py | 33 +++++++++++++++++++++++++++++++++
3 files changed, 65 insertions(+)
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index c6334b08..bfc2b298 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -1333,3 +1333,21 @@ for _bname, _bobj in list(globals().items()):
if not _bname.startswith("_") and not isinstance(_bobj, _types.ModuleType):
setattr(builtin, _bname, _bobj)
_sys.modules["pyspark.sql.functions.builtin"] = builtin
+
+# As PySpark's, the functions are of the module
``pyspark.sql.functions.builtin``, where
+# PySpark defines all of them but pandas_udf and arrow_udf (of
pyspark.sql.pandas.functions).
+# So pickle, and cloudpickle, pickle them by reference, by that name: a UDF or
a foreachBatch
+# function that captures one, such as ``array_sort``, pickles, and the Python
worker of the
+# server imports PySpark's. Of the module of the generated functions, which is
not importable,
+# cloudpickle would pickle them by value, and fail on the Rust functions that
they call.
+for _bname in dir(builtin):
+ _bobj = getattr(builtin, _bname)
+ if (
+ not _bname.startswith("_")
+ and _inspect.isfunction(_bobj)
+ and _bobj.__module__
+ in ("functions_generated", __name__, "pyspark.sql.udf",
"pyspark.sql.udtf")
+ and getattr(builtin, _bobj.__qualname__, None) is _bobj
+ ):
+ _bobj.__module__ = builtin.__name__
+del _bname, _bobj
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index cb17b2ef..ffe952ef 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1065,6 +1065,20 @@ def
test_catalog_and_stream_start_errors_carry_their_operation_id(session):
assert isinstance(error.value.operation_id, str) and
error.value.operation_id
+def test_udfs_that_capture_functions_pickle_them_by_name_as_pyspark(session):
+ # As PySpark's, the functions are of pyspark.sql.functions.builtin: a UDF
that captures one
+ # pickles it by its name, of which the Python worker of the server imports
PySpark's.
+ sort = F.array_sort
+
+ @F.udf("string")
+ def name_of(value):
+ return f"{sort.__module__}.{sort.__name__}"
+
+ assert session.range(1).select(name_of("id")).first()[0] == (
+ "pyspark.sql.functions.builtin.array_sort"
+ )
+
+
def test_create_dataframe_from_empty_arrow_table(session):
empty = pa.table({"a": pa.array([], pa.int64()), "b": pa.array([],
pa.string())})
df = session.createDataFrame(empty, "a long, b string")
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 91098843..00de03f0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9364,6 +9364,39 @@ def
test_numpy_scalars_of_other_types_are_unsupported_as_pyspark(mock_connect):
# ---------------------------------------------------------------------------
4.3.0 gaps
+def
test_functions_are_of_the_builtin_module_and_pickle_by_reference_as_pyspark():
+ # As PySpark's, the functions are of pyspark.sql.functions.builtin, but
pandas_udf and
+ # arrow_udf, so they pickle by reference, by that name, which PySpark's
workers import.
+ import inspect
+ from pyspark.serializers import CloudPickleSerializer
+ from pyspark.sql import functions as F
+ import pyspark.sql.functions.builtin as builtin
+
+ functions = [getattr(F, name) for name in F.__all__ if
inspect.isfunction(getattr(F, name))]
+ assert len(functions) > 500
+ for function in functions:
+ expected = (
+ "pyspark.sql.pandas.functions"
+ if function.__name__ in ("pandas_udf", "arrow_udf")
+ else "pyspark.sql.functions.builtin"
+ )
+ assert function.__module__ == expected, function.__name__
+ assert pickle.loads(pickle.dumps(function)) is function
+ assert F.countDistinct.__module__ == "pyspark.sql.functions.builtin"
+ assert builtin.array_sort is F.array_sort
+
+ # A function that captures some, as a UDF or a foreachBatch function does,
pickles them by
+ # their names: before, cloudpickle pickled them by value, and failed.
+ sort, col, udf = F.array_sort, F.col, F.udf
+
+ def captures(df):
+ return sort, col, udf, F.abs
+
+ payload = CloudPickleSerializer().dumps(captures)
+ assert b"pyspark.sql.functions.builtin" in payload and b"pyfunc_" not in
payload
+ assert CloudPickleSerializer().loads(payload)(None) == (sort, col, udf,
F.abs)
+
+
def test_timestamp_nanos_types_roundtrip_and_pickle():
# Spark's TimestampNTZNanosType/TimestampLTZNanosType (precision 7..9)
materialize as
# their own types, not as TimeType / a microsecond TimestampType.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]