This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 22ed7311 [SPARK-61142][CONNECT][PYTHON] Make the functions of 
pyspark.sql.functions those of pyspark.sql.functions.builtin, which pickle by 
reference, in the Python client as PySpark does
22ed7311 is described below

commit 22ed73117942633455659f58abc398e10494a773
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:44:18 2026 +0900

    [SPARK-61142][CONNECT][PYTHON] Make the functions of pyspark.sql.functions 
those of pyspark.sql.functions.builtin, which pickle by reference, in the 
Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61142](https://issues.apache.org/jira/browse/SPARK-61142): PySpark 
defines the functions of `pyspark.sql.functions` in 
`pyspark.sql.functions.builtin`, all but `pandas_udf` and `arrow_udf` (of 
`pyspark.sql.pandas.functions`). So `pickle` and `cloudpickle` pickle them by 
reference, by that module and name.
    
    The client's functions now have the same `__module__`, 
`pyspark.sql.functions.builtin`, the synthetic module that already mirrors 
`pyspark.sql.functions`:
    - the 518 generated ones, which were of `functions_generated`
    - those of `functions.py`, such as `col` and `lit`, which were of 
`pyspark.sql.functions`
    - `udf`, `udtf` and `arrow_udtf`, which were of `pyspark.sql.udf` and 
`pyspark.sql.udtf`
    
    Only a function that the `builtin` module gives back by its name is 
changed, so each one pickles by reference. On the server, the Python worker 
imports PySpark's function of that name.
    
    ### Why are the changes needed?
    
    `functions_generated` is not importable. So `cloudpickle` pickled the 
generated functions by value, and failed on the Rust functions that they call:
    
    ```
    _pickle.PicklingError: Can't pickle <built-in function 
pyfunc_invoke_function>: import of module 'functions' failed
    ```
    
    A UDF, a UDTF or a `foreachBatch` function that captured one of them, such 
as `array_sort`, could not be sent. Apache Spark's 
`test_parity_pandas_transform_with_state_state_variable.py` failed on this 
client: its `foreachBatch` functions capture `array_sort` and `col` 
(`STREAMING_CONNECT_SERIALIZATION_ERROR`).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes:
    - UDFs, UDTFs and `foreachBatch` functions that capture functions of 
`pyspark.sql.functions` can be sent, as with PySpark 4.3.0.
    - The `__module__` of those functions is `pyspark.sql.functions.builtin`, 
as PySpark's.
    
    ### How was this patch tested?
    
    - A new offline test checks:
      - the module of every function of `pyspark.sql.functions.__all__`, and of 
the deprecated `countDistinct`
      - that `pickle` gives back each function itself
      - that `cloudpickle` pickles a function that captures some by their 
names, without any Rust function
    
      It passes with official PySpark 4.3.0, and fails without this change.
    - A new live test runs a UDF that captures `array_sort`, whose worker 
imports it. It fails without this change (`PicklingError`).
    - The modules of all the functions of `pyspark.sql.functions`, also those 
that `__all__` leaves out, are those of official PySpark 4.3.0, and all of them 
pickle.
    - Apache Spark's 
`test_parity_pandas_transform_with_state_state_variable.py` passes on this 
client against a local server: 19 tests pass and 1 is skipped. Before this 
change, 3 failed.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2202 passed, 8 skipped), and the 
live drop-in and Arrow suites (185 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #335 from HyukjinKwon/SPARK-61142-functions-pickle.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/functions.py     | 18 ++++++++++++++++++
 python/tests/test_dropin_e2e.py     | 14 ++++++++++++++
 python/tests/test_dropin_offline.py | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 65 insertions(+)

diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index c6334b08..bfc2b298 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -1333,3 +1333,21 @@ for _bname, _bobj in list(globals().items()):
     if not _bname.startswith("_") and not isinstance(_bobj, _types.ModuleType):
         setattr(builtin, _bname, _bobj)
 _sys.modules["pyspark.sql.functions.builtin"] = builtin
+
+# As PySpark's, the functions are of the module 
``pyspark.sql.functions.builtin``, where
+# PySpark defines all of them but pandas_udf and arrow_udf (of 
pyspark.sql.pandas.functions).
+# So pickle, and cloudpickle, pickle them by reference, by that name: a UDF or 
a foreachBatch
+# function that captures one, such as ``array_sort``, pickles, and the Python 
worker of the
+# server imports PySpark's. Of the module of the generated functions, which is 
not importable,
+# cloudpickle would pickle them by value, and fail on the Rust functions that 
they call.
+for _bname in dir(builtin):
+    _bobj = getattr(builtin, _bname)
+    if (
+        not _bname.startswith("_")
+        and _inspect.isfunction(_bobj)
+        and _bobj.__module__
+        in ("functions_generated", __name__, "pyspark.sql.udf", 
"pyspark.sql.udtf")
+        and getattr(builtin, _bobj.__qualname__, None) is _bobj
+    ):
+        _bobj.__module__ = builtin.__name__
+del _bname, _bobj
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index cb17b2ef..ffe952ef 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1065,6 +1065,20 @@ def 
test_catalog_and_stream_start_errors_carry_their_operation_id(session):
         assert isinstance(error.value.operation_id, str) and 
error.value.operation_id
 
 
+def test_udfs_that_capture_functions_pickle_them_by_name_as_pyspark(session):
+    # As PySpark's, the functions are of pyspark.sql.functions.builtin: a UDF 
that captures one
+    # pickles it by its name, of which the Python worker of the server imports 
PySpark's.
+    sort = F.array_sort
+
+    @F.udf("string")
+    def name_of(value):
+        return f"{sort.__module__}.{sort.__name__}"
+
+    assert session.range(1).select(name_of("id")).first()[0] == (
+        "pyspark.sql.functions.builtin.array_sort"
+    )
+
+
 def test_create_dataframe_from_empty_arrow_table(session):
     empty = pa.table({"a": pa.array([], pa.int64()), "b": pa.array([], 
pa.string())})
     df = session.createDataFrame(empty, "a long, b string")
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 91098843..00de03f0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9364,6 +9364,39 @@ def 
test_numpy_scalars_of_other_types_are_unsupported_as_pyspark(mock_connect):
 # --------------------------------------------------------------------------- 
4.3.0 gaps
 
 
+def 
test_functions_are_of_the_builtin_module_and_pickle_by_reference_as_pyspark():
+    # As PySpark's, the functions are of pyspark.sql.functions.builtin, but 
pandas_udf and
+    # arrow_udf, so they pickle by reference, by that name, which PySpark's 
workers import.
+    import inspect
+    from pyspark.serializers import CloudPickleSerializer
+    from pyspark.sql import functions as F
+    import pyspark.sql.functions.builtin as builtin
+
+    functions = [getattr(F, name) for name in F.__all__ if 
inspect.isfunction(getattr(F, name))]
+    assert len(functions) > 500
+    for function in functions:
+        expected = (
+            "pyspark.sql.pandas.functions"
+            if function.__name__ in ("pandas_udf", "arrow_udf")
+            else "pyspark.sql.functions.builtin"
+        )
+        assert function.__module__ == expected, function.__name__
+        assert pickle.loads(pickle.dumps(function)) is function
+    assert F.countDistinct.__module__ == "pyspark.sql.functions.builtin"
+    assert builtin.array_sort is F.array_sort
+
+    # A function that captures some, as a UDF or a foreachBatch function does, 
pickles them by
+    # their names: before, cloudpickle pickled them by value, and failed.
+    sort, col, udf = F.array_sort, F.col, F.udf
+
+    def captures(df):
+        return sort, col, udf, F.abs
+
+    payload = CloudPickleSerializer().dumps(captures)
+    assert b"pyspark.sql.functions.builtin" in payload and b"pyfunc_" not in 
payload
+    assert CloudPickleSerializer().loads(payload)(None) == (sort, col, udf, 
F.abs)
+
+
 def test_timestamp_nanos_types_roundtrip_and_pickle():
     # Spark's TimestampNTZNanosType/TimestampLTZNanosType (precision 7..9) 
materialize as
     # their own types, not as TimeType / a microsecond TimestampType.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to