This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new d1d01e3d [SPARK-61164][CONNECT][PYTHON] Make AnalyzeArgument, 
PartitioningColumn, OrderingColumn and SelectedColumn the frozen dataclasses of 
PySpark in the Python client
d1d01e3d is described below

commit d1d01e3d75ff5b39687584a245707b04f1e7f1d1
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 15:06:41 2026 +0900

    [SPARK-61164][CONNECT][PYTHON] Make AnalyzeArgument, PartitioningColumn, 
OrderingColumn and SelectedColumn the frozen dataclasses of PySpark in the 
Python client
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61164](https://issues.apache.org/jira/browse/SPARK-61164): PySpark's 
`AnalyzeArgument`, `PartitioningColumn`, `OrderingColumn` and `SelectedColumn`, 
of the `analyze` method of a polymorphic UDTF, are frozen dataclasses of 
`pyspark.sql.udtf`.
    
    The client's are now PySpark 4.3.0's, verbatim, with their docstrings, as 
`AnalyzeResult` already is (SPARK-61089). So:
    - a UDTF handler that captures them pickles with them, by reference
    - the Python worker of the server, which runs `analyze`, unpickles them as 
PySpark's
    - they have the fields, defaults, equality, hashing, `repr` and 
`FrozenInstanceError` of dataclasses
    
    The classes of the Rust extension that these were, and the Rust 
`AnalyzeResult` that SPARK-61089 left, are removed: nothing used them any more. 
`SkipRestOfInputTableException` stays the exception of the extension.
    
    ### Why are the changes needed?
    
    The classes of the extension could not be pickled. A handler whose 
`analyze` returns an `AnalyzeResult` defined outside of it failed to serialize:
    
    ```
    PySparkPicklingError: [UDTF_SERIALIZATION_ERROR] Cannot serialize the UDTF 
'TestUDTF' ...
    TypeError: cannot pickle 'pyspark.sql.udtf.PartitioningColumn' object
    ```
    
    Apache Spark's `test_udtf_with_scalar_analyze_returning_wrong_result` and 
`test_udtf_with_table_analyze_returning_wrong_result` of `test_parity_udtf.py` 
failed on this client, in two variants each.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The four classes are PySpark 4.3.0's frozen dataclasses, and handlers 
that capture them can be sent.
    
    ### How was this patch tested?
    
    - A new offline test checks:
      - that the four classes are frozen dataclasses of `pyspark.sql.udtf`, of 
PySpark's fields and defaults
      - their equality, hashing, `repr` and `FrozenInstanceError`
      - the pickling of a function that captures an `AnalyzeResult` of them
    
      It passes with official PySpark 4.3.0, and fails without this change.
    - A new live test runs a UDTF whose `analyze` returns a captured 
`AnalyzeResult` of a `PartitioningColumn`, and gets the `AnalysisException` of 
`partitionBy` from the worker, as Apache Spark's test does. It passes with 
official PySpark 4.3.0, and fails without this change.
    - Apache Spark's `test_parity_udtf.py` and 
`arrow/test_parity_arrow_udtf.py` pass on this client against a local server: 
256 and 47 tests pass. Before this change, 4 tests of `test_parity_udtf.py` 
failed.
    - The lib tests of both crates (597 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2219 passed, 8 skipped), and the 
live drop-in and Arrow suites (202 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #355 from HyukjinKwon/SPARK-61164-udtf-dataclasses.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/lib.rs          |   5 --
 crates/pyspark-rs/src/udtf_analyze.rs | 126 ++--------------------------------
 python/pyspark/sql/udtf.py            |  97 ++++++++++++++++++++++++--
 python/tests/test_dropin_e2e.py       |  21 ++++++
 python/tests/test_dropin_offline.py   |  39 +++++++++++
 5 files changed, 157 insertions(+), 131 deletions(-)

diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 08159103..840843af 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -223,11 +223,6 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> 
PyResult<()> {
     // Register TVF + Observation
     m.add_class::<tvf::PyTableValuedFunction>()?;
     m.add_class::<tablearg::PyTableArg>()?;
-    m.add_class::<udtf_analyze::PyAnalyzeArgument>()?;
-    m.add_class::<udtf_analyze::PyPartitioningColumn>()?;
-    m.add_class::<udtf_analyze::PyOrderingColumn>()?;
-    m.add_class::<udtf_analyze::PySelectedColumn>()?;
-    m.add_class::<udtf_analyze::PyAnalyzeResult>()?;
     m.add(
         "SkipRestOfInputTableException",
         m.py()
diff --git a/crates/pyspark-rs/src/udtf_analyze.rs 
b/crates/pyspark-rs/src/udtf_analyze.rs
index ac0863d5..625193a9 100644
--- a/crates/pyspark-rs/src/udtf_analyze.rs
+++ b/crates/pyspark-rs/src/udtf_analyze.rs
@@ -1,10 +1,8 @@
-//! PyO3-backed data holders for polymorphic UDTF `analyze`, mirroring
-//! 
`pyspark.sql.udtf.{AnalyzeArgument,PartitioningColumn,OrderingColumn,SelectedColumn,
-//! AnalyzeResult}` and `SkipRestOfInputTableException`. They carry the analyze
-//! specification (constructed in the user's `analyze` staticmethod, 
cloudpickled to the
-//! server worker), so the fields are stored as opaque Python objects where 
typed.
-
-use pyo3::prelude::*;
+//! `pyspark.sql.udtf.SkipRestOfInputTableException`. The data classes of the 
`analyze` of a
+//! polymorphic UDTF (`AnalyzeArgument`, `PartitioningColumn`, 
`OrderingColumn`,
+//! `SelectedColumn` and `AnalyzeResult`) are PySpark's dataclasses, of the 
shim
+//! `pyspark.sql.udtf`: a handler that captures them pickles with them, and 
the Python worker
+//! of the server unpickles them as PySpark's.
 
 // The module argument sets `__module__`; use the reference path (upstream 
defines this
 // in pyspark/sql/udtf.py) rather than the extension's own name.
@@ -13,117 +11,3 @@ pyo3::create_exception!(
     SkipRestOfInputTableException,
     pyo3::exceptions::PyException
 );
-
-/// `pyspark.sql.udtf.AnalyzeArgument`.
-#[pyclass(name = "AnalyzeArgument", module = "pyspark.sql.udtf", get_all)]
-pub struct PyAnalyzeArgument {
-    pub dataType: Py<PyAny>,
-    pub value: Py<PyAny>,
-    pub isTable: bool,
-    pub isConstantExpression: bool,
-}
-#[pymethods]
-impl PyAnalyzeArgument {
-    #[new]
-    #[allow(non_snake_case)]
-    fn new(
-        dataType: Py<PyAny>,
-        value: Py<PyAny>,
-        isTable: bool,
-        isConstantExpression: bool,
-    ) -> Self {
-        PyAnalyzeArgument {
-            dataType,
-            value,
-            isTable,
-            isConstantExpression,
-        }
-    }
-}
-
-/// `pyspark.sql.udtf.PartitioningColumn` (frozen).
-#[pyclass(
-    name = "PartitioningColumn",
-    module = "pyspark.sql.udtf",
-    get_all,
-    frozen
-)]
-pub struct PyPartitioningColumn {
-    pub name: String,
-}
-#[pymethods]
-impl PyPartitioningColumn {
-    #[new]
-    fn new(name: String) -> Self {
-        PyPartitioningColumn { name }
-    }
-}
-
-/// `pyspark.sql.udtf.OrderingColumn` (frozen).
-#[pyclass(name = "OrderingColumn", module = "pyspark.sql.udtf", get_all, 
frozen)]
-pub struct PyOrderingColumn {
-    pub name: String,
-    pub ascending: bool,
-    pub overrideNullsFirst: Option<bool>,
-}
-#[pymethods]
-impl PyOrderingColumn {
-    #[new]
-    #[pyo3(signature = (name, ascending=true, overrideNullsFirst=None))]
-    #[allow(non_snake_case)]
-    fn new(name: String, ascending: bool, overrideNullsFirst: Option<bool>) -> 
Self {
-        PyOrderingColumn {
-            name,
-            ascending,
-            overrideNullsFirst,
-        }
-    }
-}
-
-/// `pyspark.sql.udtf.SelectedColumn` (frozen).
-#[pyclass(name = "SelectedColumn", module = "pyspark.sql.udtf", get_all, 
frozen)]
-pub struct PySelectedColumn {
-    pub name: String,
-    pub alias: String,
-}
-#[pymethods]
-impl PySelectedColumn {
-    #[new]
-    #[pyo3(signature = (name, alias=String::new()))]
-    fn new(name: String, alias: String) -> Self {
-        PySelectedColumn { name, alias }
-    }
-}
-
-/// `pyspark.sql.udtf.AnalyzeResult` (mutable dataclass).
-#[pyclass(name = "AnalyzeResult", module = "pyspark.sql.udtf", get_all, 
set_all)]
-pub struct PyAnalyzeResult {
-    pub schema: Py<PyAny>,
-    pub withSinglePartition: bool,
-    pub partitionBy: Py<PyAny>,
-    pub orderBy: Py<PyAny>,
-    pub select: Py<PyAny>,
-}
-#[pymethods]
-impl PyAnalyzeResult {
-    #[new]
-    #[pyo3(signature = (schema, withSinglePartition=false, partitionBy=None, 
orderBy=None, select=None))]
-    #[allow(non_snake_case)]
-    fn new(
-        py: Python<'_>,
-        schema: Py<PyAny>,
-        withSinglePartition: bool,
-        partitionBy: Option<Py<PyAny>>,
-        orderBy: Option<Py<PyAny>>,
-        select: Option<Py<PyAny>>,
-    ) -> PyResult<Self> {
-        let empty = || pyo3::types::PyTuple::empty(py).into_any().unbind();
-        Ok(PyAnalyzeResult {
-            schema,
-            withSinglePartition,
-            partitionBy: partitionBy.unwrap_or_else(empty),
-            orderBy: orderBy.unwrap_or_else(empty),
-            select: select.unwrap_or_else(empty),
-        })
-    }
-}
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 476f1a47..904cc3d3 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -20,10 +20,7 @@ User-defined table function (UDTF) support for the Spark 
Connect client.
 Mirrors pyspark.sql.connect.udtf and pyspark.sql.functions.udtf.
 """
 
-from pyspark._pyspark import (
-    AnalyzeArgument, PartitioningColumn, OrderingColumn, SelectedColumn,
-    SkipRestOfInputTableException,
-)
+from pyspark._pyspark import SkipRestOfInputTableException
 
 import inspect
 import pickle
@@ -33,7 +30,7 @@ from typing import Any, Optional, Sequence, Type
 
 from pyspark.errors import PySparkAttributeError, PySparkPicklingError, 
PySparkTypeError
 from pyspark.serializers import CloudPickleSerializer
-from pyspark.sql.types import StructType
+from pyspark.sql.types import DataType, StructType
 
 __all__ = [
     "AnalyzeArgument",
@@ -51,6 +48,96 @@ SQL_ARROW_TABLE_UDF = 301
 SQL_ARROW_UDTF = 302
 
 
+# Verbatim from PySpark (pyspark.sql.udtf): the frozen dataclasses of the 
analyze method of a
+# UDTF, which a handler that the client pickles may capture, and which the 
Python worker of
+# the server unpickles as PySpark's of the same names.
+@dataclass(frozen=True)
+class AnalyzeArgument:
+    """
+    The argument for Python UDTF's analyze static method.
+
+    Parameters
+    ----------
+    dataType : :class:`DataType`
+        The argument's data type
+    value : any, optional
+        The calculated value if the argument is foldable; otherwise None
+    isTable : bool
+        If True, the argument is a table argument.
+    isConstantExpression : bool
+        If True, the argument is a constant-foldable scalar expression. Then 
the 'value' field
+        contains None if the argument is a NULL literal, or a non-None value 
if the argument is a
+        non-NULL literal. In this way, we can distinguish between a literal 
NULL argument and other
+        types of arguments such as complex expression trees or table arguments 
where the 'value'
+        field is always None.
+    """
+
+    dataType: DataType
+    value: Optional[Any]
+    isTable: bool
+    isConstantExpression: bool
+
+
+@dataclass(frozen=True)
+class PartitioningColumn:
+    """
+    Represents an expression that the UDTF is specifying for Catalyst to 
partition the input table
+    by. This can be either the name of a single column from the input table 
(such as "columnA"), or
+    a SQL expression based on the column names of the input table (such as 
"columnA + columnB").
+
+    Parameters
+    ----------
+    name : str
+        The contents of the partitioning column name or expression represented 
as a SQL string.
+    """
+
+    name: str
+
+
+@dataclass(frozen=True)
+class OrderingColumn:
+    """
+    Represents an expression that the UDTF is specifying for Catalyst to order 
the input partition
+    by. This can be either the name of a single column from the input table 
(such as "columnA"),
+    or a SQL expression based on the column names of the input table (such as 
"columnA + columnB").
+
+    Parameters
+    ----------
+    name : str
+        The contents of the ordering column name or expression represented as 
a SQL string.
+    ascending : bool, default True
+        This is if this expression specifies an ascending sorting order.
+    overrideNullsFirst : str, optional
+        If this is None, use the default behavior to sort NULL values first 
when sorting in
+        ascending order, or last when sorting in descending order. Otherwise, 
if this is
+        True or False, we override the default behavior accordingly.
+    """
+
+    name: str
+    ascending: bool = True
+    overrideNullsFirst: Optional[bool] = None
+
+
+@dataclass(frozen=True)
+class SelectedColumn:
+    """
+    Represents an expression that the UDTF is specifying for Catalyst to 
evaluate against the
+    columns in the input TABLE argument. The UDTF then receives one input 
column for each expression
+    in the list, in the order they are listed.
+
+    Parameters
+    ----------
+    name : str
+        The contents of the selected column name or expression represented as 
a SQL string.
+    alias : str, default ''
+        If non-empty, this is the alias for the column or expression as 
visible from the UDTF's
+        'eval' method. This is required if the expression is not a simple 
column reference.
+    """
+
+    name: str
+    alias: str = ""
+
+
 @dataclass
 class AnalyzeResult:
     """The result of a Python UDTF's ``analyze`` static method.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index f3f383e8..61d746fc 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -545,6 +545,27 @@ def 
test_udtf_passes_custom_analyze_result_to_handler(session):
     assert Echo(F.lit("kept")).collect() == [Row(buffer="kept")]
 
 
+def test_udtf_handlers_that_capture_analysis_columns_run_as_pyspark(session):
+    # As PySpark's test_udtf_with_scalar_analyze_returning_wrong_result: a 
handler that captures
+    # the result of its analysis, of partitioning columns, pickles, and the 
Python worker of the
+    # server, which unpickles them as PySpark's, raises its error of them.
+    from pyspark.sql.udtf import AnalyzeResult, PartitioningColumn
+
+    result = AnalyzeResult(StructType().add("a", StringType()), 
partitionBy=[PartitioningColumn("a")])
+
+    @F.udtf
+    class Partitioned:
+        @staticmethod
+        def analyze(**kwargs):
+            return result
+
+        def eval(self, **kwargs):
+            yield (1,)
+
+    with pytest.raises(AnalysisException, match="partitionBy"):
+        Partitioned().collect()
+
+
 def test_arrow_udtf_takes_and_returns_arrow_data_as_pyspark(session):
     # As PySpark's arrow_udtf(), a PyArrow-native UDTF: eval() takes a pyarrow 
array of the
     # argument, and returns pyarrow tables.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index c83fdb00..05be95c1 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2790,6 +2790,45 @@ def 
test_udtf_analyze_result_subclass_preserves_custom_state():
     assert list(handler(analysis).eval(None)) == [("kept",)]
 
 
+def test_udtf_analysis_classes_are_the_frozen_dataclasses_of_pyspark():
+    # As PySpark's: the arguments and the columns of the analysis of a UDTF 
are frozen dataclasses,
+    # which a handler that captures them pickles with, by reference, of which 
the Python worker
+    # of the server unpickles PySpark's.
+    import dataclasses
+    from pyspark.serializers import CloudPickleSerializer
+    from pyspark.sql.udtf import (
+        AnalyzeArgument, AnalyzeResult, OrderingColumn, PartitioningColumn, 
SelectedColumn,
+    )
+
+    missing = dataclasses.MISSING
+    for cls, fields in [
+        (AnalyzeArgument, [("dataType", missing), ("value", missing), 
("isTable", missing),
+                           ("isConstantExpression", missing)]),
+        (PartitioningColumn, [("name", missing)]),
+        (OrderingColumn, [("name", missing), ("ascending", True), 
("overrideNullsFirst", None)]),
+        (SelectedColumn, [("name", missing), ("alias", "")]),
+    ]:
+        assert dataclasses.is_dataclass(cls) and 
cls.__dataclass_params__.frozen, cls
+        assert cls.__module__ == "pyspark.sql.udtf"
+        assert [(f.name, f.default) for f in dataclasses.fields(cls)] == fields
+    ordering = OrderingColumn("a", ascending=False)
+    with pytest.raises(dataclasses.FrozenInstanceError):
+        ordering.name = "b"
+    assert ordering == OrderingColumn("a", False) and hash(ordering) == 
hash(OrderingColumn("a", False))
+    assert repr(ordering) == "OrderingColumn(name='a', ascending=False, 
overrideNullsFirst=None)"
+    result = AnalyzeResult(
+        T.StructType().add("a", T.StringType()), 
partitionBy=[PartitioningColumn("a")],
+        orderBy=[ordering], select=[SelectedColumn("a", alias="b")],
+    )
+
+    def analyze():
+        return result
+
+    serializer = CloudPickleSerializer()
+    assert serializer.loads(serializer.dumps(analyze))() == result
+    assert serializer.loads(serializer.dumps(PartitioningColumn("a"))) == 
PartitioningColumn("a")
+
+
 def 
test_variant_spatial_and_user_defined_values_are_sent_as_pyspark(mock_connect):
     # As PySpark's LocalDataToArrowConversion: a VariantVal is the struct of 
its value and
     # metadata, a Geometry or a Geography that of its srid and WKB, whose 
metadata tags the


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to