This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new d1d01e3d [SPARK-61164][CONNECT][PYTHON] Make AnalyzeArgument,
PartitioningColumn, OrderingColumn and SelectedColumn the frozen dataclasses of
PySpark in the Python client
d1d01e3d is described below
commit d1d01e3d75ff5b39687584a245707b04f1e7f1d1
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 15:06:41 2026 +0900
[SPARK-61164][CONNECT][PYTHON] Make AnalyzeArgument, PartitioningColumn,
OrderingColumn and SelectedColumn the frozen dataclasses of PySpark in the
Python client
### What changes were proposed in this pull request?
[SPARK-61164](https://issues.apache.org/jira/browse/SPARK-61164): PySpark's
`AnalyzeArgument`, `PartitioningColumn`, `OrderingColumn` and `SelectedColumn`,
of the `analyze` method of a polymorphic UDTF, are frozen dataclasses of
`pyspark.sql.udtf`.
The client's are now PySpark 4.3.0's, verbatim, with their docstrings, as
`AnalyzeResult` already is (SPARK-61089). So:
- a UDTF handler that captures them pickles with them, by reference
- the Python worker of the server, which runs `analyze`, unpickles them as
PySpark's
- they have the fields, defaults, equality, hashing, `repr` and
`FrozenInstanceError` of dataclasses
The classes of the Rust extension that these were, and the Rust
`AnalyzeResult` that SPARK-61089 left, are removed: nothing used them any more.
`SkipRestOfInputTableException` stays the exception of the extension.
### Why are the changes needed?
The classes of the extension could not be pickled. A handler whose
`analyze` returns an `AnalyzeResult` defined outside of it failed to serialize:
```
PySparkPicklingError: [UDTF_SERIALIZATION_ERROR] Cannot serialize the UDTF
'TestUDTF' ...
TypeError: cannot pickle 'pyspark.sql.udtf.PartitioningColumn' object
```
Apache Spark's `test_udtf_with_scalar_analyze_returning_wrong_result` and
`test_udtf_with_table_analyze_returning_wrong_result` of `test_parity_udtf.py`
failed on this client, in two variants each.
### Does this PR introduce _any_ user-facing change?
Yes. The four classes are PySpark 4.3.0's frozen dataclasses, and handlers
that capture them can be sent.
### How was this patch tested?
- A new offline test checks:
- that the four classes are frozen dataclasses of `pyspark.sql.udtf`, of
PySpark's fields and defaults
- their equality, hashing, `repr` and `FrozenInstanceError`
- the pickling of a function that captures an `AnalyzeResult` of them
It passes with official PySpark 4.3.0, and fails without this change.
- A new live test runs a UDTF whose `analyze` returns a captured
`AnalyzeResult` of a `PartitioningColumn`, and gets the `AnalysisException` of
`partitionBy` from the worker, as Apache Spark's test does. It passes with
official PySpark 4.3.0, and fails without this change.
- Apache Spark's `test_parity_udtf.py` and
`arrow/test_parity_arrow_udtf.py` pass on this client against a local server:
256 and 47 tests pass. Before this change, 4 tests of `test_parity_udtf.py`
failed.
- The lib tests of both crates (597 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2219 passed, 8 skipped), and the
live drop-in and Arrow suites (202 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #355 from HyukjinKwon/SPARK-61164-udtf-dataclasses.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/lib.rs | 5 --
crates/pyspark-rs/src/udtf_analyze.rs | 126 ++--------------------------------
python/pyspark/sql/udtf.py | 97 ++++++++++++++++++++++++--
python/tests/test_dropin_e2e.py | 21 ++++++
python/tests/test_dropin_offline.py | 39 +++++++++++
5 files changed, 157 insertions(+), 131 deletions(-)
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 08159103..840843af 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -223,11 +223,6 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
// Register TVF + Observation
m.add_class::<tvf::PyTableValuedFunction>()?;
m.add_class::<tablearg::PyTableArg>()?;
- m.add_class::<udtf_analyze::PyAnalyzeArgument>()?;
- m.add_class::<udtf_analyze::PyPartitioningColumn>()?;
- m.add_class::<udtf_analyze::PyOrderingColumn>()?;
- m.add_class::<udtf_analyze::PySelectedColumn>()?;
- m.add_class::<udtf_analyze::PyAnalyzeResult>()?;
m.add(
"SkipRestOfInputTableException",
m.py()
diff --git a/crates/pyspark-rs/src/udtf_analyze.rs
b/crates/pyspark-rs/src/udtf_analyze.rs
index ac0863d5..625193a9 100644
--- a/crates/pyspark-rs/src/udtf_analyze.rs
+++ b/crates/pyspark-rs/src/udtf_analyze.rs
@@ -1,10 +1,8 @@
-//! PyO3-backed data holders for polymorphic UDTF `analyze`, mirroring
-//!
`pyspark.sql.udtf.{AnalyzeArgument,PartitioningColumn,OrderingColumn,SelectedColumn,
-//! AnalyzeResult}` and `SkipRestOfInputTableException`. They carry the analyze
-//! specification (constructed in the user's `analyze` staticmethod,
cloudpickled to the
-//! server worker), so the fields are stored as opaque Python objects where
typed.
-
-use pyo3::prelude::*;
+//! `pyspark.sql.udtf.SkipRestOfInputTableException`. The data classes of the
`analyze` of a
+//! polymorphic UDTF (`AnalyzeArgument`, `PartitioningColumn`,
`OrderingColumn`,
+//! `SelectedColumn` and `AnalyzeResult`) are PySpark's dataclasses, of the
shim
+//! `pyspark.sql.udtf`: a handler that captures them pickles with them, and
the Python worker
+//! of the server unpickles them as PySpark's.
// The module argument sets `__module__`; use the reference path (upstream
defines this
// in pyspark/sql/udtf.py) rather than the extension's own name.
@@ -13,117 +11,3 @@ pyo3::create_exception!(
SkipRestOfInputTableException,
pyo3::exceptions::PyException
);
-
-/// `pyspark.sql.udtf.AnalyzeArgument`.
-#[pyclass(name = "AnalyzeArgument", module = "pyspark.sql.udtf", get_all)]
-pub struct PyAnalyzeArgument {
- pub dataType: Py<PyAny>,
- pub value: Py<PyAny>,
- pub isTable: bool,
- pub isConstantExpression: bool,
-}
-#[pymethods]
-impl PyAnalyzeArgument {
- #[new]
- #[allow(non_snake_case)]
- fn new(
- dataType: Py<PyAny>,
- value: Py<PyAny>,
- isTable: bool,
- isConstantExpression: bool,
- ) -> Self {
- PyAnalyzeArgument {
- dataType,
- value,
- isTable,
- isConstantExpression,
- }
- }
-}
-
-/// `pyspark.sql.udtf.PartitioningColumn` (frozen).
-#[pyclass(
- name = "PartitioningColumn",
- module = "pyspark.sql.udtf",
- get_all,
- frozen
-)]
-pub struct PyPartitioningColumn {
- pub name: String,
-}
-#[pymethods]
-impl PyPartitioningColumn {
- #[new]
- fn new(name: String) -> Self {
- PyPartitioningColumn { name }
- }
-}
-
-/// `pyspark.sql.udtf.OrderingColumn` (frozen).
-#[pyclass(name = "OrderingColumn", module = "pyspark.sql.udtf", get_all,
frozen)]
-pub struct PyOrderingColumn {
- pub name: String,
- pub ascending: bool,
- pub overrideNullsFirst: Option<bool>,
-}
-#[pymethods]
-impl PyOrderingColumn {
- #[new]
- #[pyo3(signature = (name, ascending=true, overrideNullsFirst=None))]
- #[allow(non_snake_case)]
- fn new(name: String, ascending: bool, overrideNullsFirst: Option<bool>) ->
Self {
- PyOrderingColumn {
- name,
- ascending,
- overrideNullsFirst,
- }
- }
-}
-
-/// `pyspark.sql.udtf.SelectedColumn` (frozen).
-#[pyclass(name = "SelectedColumn", module = "pyspark.sql.udtf", get_all,
frozen)]
-pub struct PySelectedColumn {
- pub name: String,
- pub alias: String,
-}
-#[pymethods]
-impl PySelectedColumn {
- #[new]
- #[pyo3(signature = (name, alias=String::new()))]
- fn new(name: String, alias: String) -> Self {
- PySelectedColumn { name, alias }
- }
-}
-
-/// `pyspark.sql.udtf.AnalyzeResult` (mutable dataclass).
-#[pyclass(name = "AnalyzeResult", module = "pyspark.sql.udtf", get_all,
set_all)]
-pub struct PyAnalyzeResult {
- pub schema: Py<PyAny>,
- pub withSinglePartition: bool,
- pub partitionBy: Py<PyAny>,
- pub orderBy: Py<PyAny>,
- pub select: Py<PyAny>,
-}
-#[pymethods]
-impl PyAnalyzeResult {
- #[new]
- #[pyo3(signature = (schema, withSinglePartition=false, partitionBy=None,
orderBy=None, select=None))]
- #[allow(non_snake_case)]
- fn new(
- py: Python<'_>,
- schema: Py<PyAny>,
- withSinglePartition: bool,
- partitionBy: Option<Py<PyAny>>,
- orderBy: Option<Py<PyAny>>,
- select: Option<Py<PyAny>>,
- ) -> PyResult<Self> {
- let empty = || pyo3::types::PyTuple::empty(py).into_any().unbind();
- Ok(PyAnalyzeResult {
- schema,
- withSinglePartition,
- partitionBy: partitionBy.unwrap_or_else(empty),
- orderBy: orderBy.unwrap_or_else(empty),
- select: select.unwrap_or_else(empty),
- })
- }
-}
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 476f1a47..904cc3d3 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -20,10 +20,7 @@ User-defined table function (UDTF) support for the Spark
Connect client.
Mirrors pyspark.sql.connect.udtf and pyspark.sql.functions.udtf.
"""
-from pyspark._pyspark import (
- AnalyzeArgument, PartitioningColumn, OrderingColumn, SelectedColumn,
- SkipRestOfInputTableException,
-)
+from pyspark._pyspark import SkipRestOfInputTableException
import inspect
import pickle
@@ -33,7 +30,7 @@ from typing import Any, Optional, Sequence, Type
from pyspark.errors import PySparkAttributeError, PySparkPicklingError,
PySparkTypeError
from pyspark.serializers import CloudPickleSerializer
-from pyspark.sql.types import StructType
+from pyspark.sql.types import DataType, StructType
__all__ = [
"AnalyzeArgument",
@@ -51,6 +48,96 @@ SQL_ARROW_TABLE_UDF = 301
SQL_ARROW_UDTF = 302
+# Verbatim from PySpark (pyspark.sql.udtf): the frozen dataclasses of the
analyze method of a
+# UDTF, which a handler that the client pickles may capture, and which the
Python worker of
+# the server unpickles as PySpark's of the same names.
+@dataclass(frozen=True)
+class AnalyzeArgument:
+ """
+ The argument for Python UDTF's analyze static method.
+
+ Parameters
+ ----------
+ dataType : :class:`DataType`
+ The argument's data type
+ value : any, optional
+ The calculated value if the argument is foldable; otherwise None
+ isTable : bool
+ If True, the argument is a table argument.
+ isConstantExpression : bool
+ If True, the argument is a constant-foldable scalar expression. Then
the 'value' field
+ contains None if the argument is a NULL literal, or a non-None value
if the argument is a
+ non-NULL literal. In this way, we can distinguish between a literal
NULL argument and other
+ types of arguments such as complex expression trees or table arguments
where the 'value'
+ field is always None.
+ """
+
+ dataType: DataType
+ value: Optional[Any]
+ isTable: bool
+ isConstantExpression: bool
+
+
+@dataclass(frozen=True)
+class PartitioningColumn:
+ """
+ Represents an expression that the UDTF is specifying for Catalyst to
partition the input table
+ by. This can be either the name of a single column from the input table
(such as "columnA"), or
+ a SQL expression based on the column names of the input table (such as
"columnA + columnB").
+
+ Parameters
+ ----------
+ name : str
+ The contents of the partitioning column name or expression represented
as a SQL string.
+ """
+
+ name: str
+
+
+@dataclass(frozen=True)
+class OrderingColumn:
+ """
+ Represents an expression that the UDTF is specifying for Catalyst to order
the input partition
+ by. This can be either the name of a single column from the input table
(such as "columnA"),
+ or a SQL expression based on the column names of the input table (such as
"columnA + columnB").
+
+ Parameters
+ ----------
+ name : str
+ The contents of the ordering column name or expression represented as
a SQL string.
+ ascending : bool, default True
+ This is if this expression specifies an ascending sorting order.
+ overrideNullsFirst : str, optional
+ If this is None, use the default behavior to sort NULL values first
when sorting in
+ ascending order, or last when sorting in descending order. Otherwise,
if this is
+ True or False, we override the default behavior accordingly.
+ """
+
+ name: str
+ ascending: bool = True
+ overrideNullsFirst: Optional[bool] = None
+
+
+@dataclass(frozen=True)
+class SelectedColumn:
+ """
+ Represents an expression that the UDTF is specifying for Catalyst to
evaluate against the
+ columns in the input TABLE argument. The UDTF then receives one input
column for each expression
+ in the list, in the order they are listed.
+
+ Parameters
+ ----------
+ name : str
+ The contents of the selected column name or expression represented as
a SQL string.
+ alias : str, default ''
+ If non-empty, this is the alias for the column or expression as
visible from the UDTF's
+ 'eval' method. This is required if the expression is not a simple
column reference.
+ """
+
+ name: str
+ alias: str = ""
+
+
@dataclass
class AnalyzeResult:
"""The result of a Python UDTF's ``analyze`` static method.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index f3f383e8..61d746fc 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -545,6 +545,27 @@ def
test_udtf_passes_custom_analyze_result_to_handler(session):
assert Echo(F.lit("kept")).collect() == [Row(buffer="kept")]
+def test_udtf_handlers_that_capture_analysis_columns_run_as_pyspark(session):
+ # As PySpark's test_udtf_with_scalar_analyze_returning_wrong_result: a
handler that captures
+ # the result of its analysis, of partitioning columns, pickles, and the
Python worker of the
+ # server, which unpickles them as PySpark's, raises its error of them.
+ from pyspark.sql.udtf import AnalyzeResult, PartitioningColumn
+
+ result = AnalyzeResult(StructType().add("a", StringType()),
partitionBy=[PartitioningColumn("a")])
+
+ @F.udtf
+ class Partitioned:
+ @staticmethod
+ def analyze(**kwargs):
+ return result
+
+ def eval(self, **kwargs):
+ yield (1,)
+
+ with pytest.raises(AnalysisException, match="partitionBy"):
+ Partitioned().collect()
+
+
def test_arrow_udtf_takes_and_returns_arrow_data_as_pyspark(session):
# As PySpark's arrow_udtf(), a PyArrow-native UDTF: eval() takes a pyarrow
array of the
# argument, and returns pyarrow tables.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index c83fdb00..05be95c1 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2790,6 +2790,45 @@ def
test_udtf_analyze_result_subclass_preserves_custom_state():
assert list(handler(analysis).eval(None)) == [("kept",)]
+def test_udtf_analysis_classes_are_the_frozen_dataclasses_of_pyspark():
+ # As PySpark's: the arguments and the columns of the analysis of a UDTF
are frozen dataclasses,
+ # which a handler that captures them pickles with, by reference, of which
the Python worker
+ # of the server unpickles PySpark's.
+ import dataclasses
+ from pyspark.serializers import CloudPickleSerializer
+ from pyspark.sql.udtf import (
+ AnalyzeArgument, AnalyzeResult, OrderingColumn, PartitioningColumn,
SelectedColumn,
+ )
+
+ missing = dataclasses.MISSING
+ for cls, fields in [
+ (AnalyzeArgument, [("dataType", missing), ("value", missing),
("isTable", missing),
+ ("isConstantExpression", missing)]),
+ (PartitioningColumn, [("name", missing)]),
+ (OrderingColumn, [("name", missing), ("ascending", True),
("overrideNullsFirst", None)]),
+ (SelectedColumn, [("name", missing), ("alias", "")]),
+ ]:
+ assert dataclasses.is_dataclass(cls) and
cls.__dataclass_params__.frozen, cls
+ assert cls.__module__ == "pyspark.sql.udtf"
+ assert [(f.name, f.default) for f in dataclasses.fields(cls)] == fields
+ ordering = OrderingColumn("a", ascending=False)
+ with pytest.raises(dataclasses.FrozenInstanceError):
+ ordering.name = "b"
+ assert ordering == OrderingColumn("a", False) and hash(ordering) ==
hash(OrderingColumn("a", False))
+ assert repr(ordering) == "OrderingColumn(name='a', ascending=False,
overrideNullsFirst=None)"
+ result = AnalyzeResult(
+ T.StructType().add("a", T.StringType()),
partitionBy=[PartitioningColumn("a")],
+ orderBy=[ordering], select=[SelectedColumn("a", alias="b")],
+ )
+
+ def analyze():
+ return result
+
+ serializer = CloudPickleSerializer()
+ assert serializer.loads(serializer.dumps(analyze))() == result
+ assert serializer.loads(serializer.dumps(PartitioningColumn("a"))) ==
PartitioningColumn("a")
+
+
def
test_variant_spatial_and_user_defined_values_are_sent_as_pyspark(mock_connect):
# As PySpark's LocalDataToArrowConversion: a VariantVal is the struct of
its value and
# metadata, a Geometry or a Geography that of its srid and WKB, whose
metadata tags the
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]