This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 829c8ea  [SPARK-60678][CONNECT] Support Python UDTF DDL string return 
schemas
829c8ea is described below

commit 829c8ead5526a5e3f2cc6398860bdc59ace9195f
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 16:03:57 2026 +0900

    [SPARK-60678][CONNECT] Support Python UDTF DDL string return schemas
    
    ### What changes were proposed in this pull request?
    
    Python UDTFs now work against a Spark Connect server, with a DDL string or 
a `StructType` as the return type, as in PySpark:
    
    - The PyO3 binding keeps a string return type as `DataType::Unparsed`, so 
the server parses and validates the DDL when the query runs, as PySpark Connect 
does. Typed schemas go through the existing Python-to-Rust type converter. 
Before, the binding took only a `DataType`, so a string raised a `TypeError`.
    - `PythonUDTF.command` now holds only the cloudpickled handler class, as in 
PySpark Connect; the return schema is sent separately in 
`PythonUDTF.return_type`. Before, the command was the tuple `(handler, 
returnType)`.
    
    ```python
    class Echo:
        def eval(self, value):
            yield (value,)
    
    udtf(Echo, returnType="value BIGINT")(lit(1)).collect()
    ```
    
    | Return type | PySpark 4.3.0 and now | Before |
    | --- | --- | --- |
    | `"value BIGINT"` | `[Row(value=1)]` | `TypeError: argument 'return_type': 
'str' object is not an instance of 'DataType'` |
    | `StructType([StructField("value", LongType())])` | `[Row(value=1)]` | 
`PythonException`: "Invalid UDTF handler type. Expected a class (type 'type'), 
but got an instance of tuple." from the worker |
    | `"invalid DDL !!!"` | `ParseException` (`PARSE_SYNTAX_ERROR`) when the 
query runs | `TypeError` when called |
    
    ### Why are the changes needed?
    
    [SPARK-60678](https://issues.apache.org/jira/browse/SPARK-60678): Python 
UDTFs did not run at all. A string return type failed in the client, and the 
server's Python worker rejected the tuple command for any return type.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `udtf` with a DDL string or a `StructType` return type returns its 
rows, as in PySpark, instead of failing. An invalid DDL string fails when the 
query runs, as in PySpark, instead of when the UDTF is called.
    
    ### How was this patch tested?
    
    - Offline tests check the direct and decorator forms with valid, nested, 
malformed and scalar DDL, a `StructType` (with metadata), and an omitted schema 
for a handler with `analyze`. Against a recording mock gRPC server, they check 
the unparsed DDL, the typed schema, the evaluation type and the handler-only 
command that reach it.
    - A Rust test checks the serialized return schema of the inline UDTF 
relation.
    - A new live test runs UDTFs with DDL and `StructType` return types, and an 
invalid DDL, against a Spark 4.3.0 server. It also passes with official PySpark 
4.3.0 and fails without this change.
    - Against the current master, merged into this branch, and a local Spark 
4.3.0 server: the lib and every integration test of `apache-spark-connect` and 
`apache-spark-connect-core`, e2e included (772 passed), the build of all tests 
with `wasm-udf`, the offline Python suites (722 passed, 9 skipped), the live 
drop-in and Arrow suites (77 passed) and the live cases (10 passed). `cargo fmt 
--all --check` passes.
    
    Not in this change: PySpark keeps a string return type as an 
`UnparsedDataType` in `UserDefinedTableFunction.returnType`, which this keeps 
as the string, and PySpark validates the handler class when the UDTF is created 
(`INVALID_UDTF_NO_EVAL` and others).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the merge with master and the review 
follow-ups by Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #119 from dbtsai/codex/spark-60678-udtf-ddl.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Co-authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/functions.rs                 |  17 +++-
 .../spark-connect/tests/coverage_plan_map_udtf.rs  |  45 +++++++++
 python/pyspark/sql/udtf.py                         |   5 +-
 python/tests/test_dropin_e2e.py                    |  20 ++++
 python/tests/test_dropin_offline.py                | 112 +++++++++++++++++++++
 5 files changed, 194 insertions(+), 5 deletions(-)

diff --git a/crates/pyspark-rs/src/functions.rs 
b/crates/pyspark-rs/src/functions.rs
index d02da81..6f24de8 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -9,7 +9,6 @@ use 
spark_connect::udf::{CommonInlineUserDefinedFunctionExpression, PythonUDFPay
 use crate::column::PyColumn;
 use crate::dataframe::PyDataFrame;
 use crate::session::PySparkSession;
-use crate::types::PyDataType;
 
 // Include the auto-generated dispatch function
 mod dispatch {
@@ -78,7 +77,7 @@ fn pyfunc_make_udf(
 fn pyfunc_make_udtf(
     session: PyRef<'_, PySparkSession>,
     name: String,
-    return_type: Option<PyRef<'_, PyDataType>>,
+    return_type: Option<&Bound<'_, PyAny>>,
     eval_type: i32,
     command_bytes: Vec<u8>,
     python_ver: String,
@@ -88,7 +87,19 @@ fn pyfunc_make_udtf(
     for arg in args {
         col_args.push(to_column(&arg)?);
     }
-    let rt = return_type.map(|d| d.inner.clone());
+    // Connect leaves DDL parsing and validation to the server. In particular,
+    // do not use py_to_data_type's eager DDL parser for string return types.
+    let rt = return_type
+        .map(|value| {
+            if let Ok(ddl) = value.extract::<String>() {
+                Ok(spark_connect::types::DataType::Unparsed {
+                    data_type_string: ddl,
+                })
+            } else {
+                crate::types::py_to_data_type(value)
+            }
+        })
+        .transpose()?;
     let df = session.session.tvf().udtf(
         &name,
         col_args,
diff --git a/crates/spark-connect/tests/coverage_plan_map_udtf.rs 
b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
index 431ef2e..e452eea 100644
--- a/crates/spark-connect/tests/coverage_plan_map_udtf.rs
+++ b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
@@ -135,3 +135,48 @@ fn 
common_inline_udtf_to_proto_with_and_without_return_type() {
     };
     assert!(no_rt.to_proto().rel_type.is_some());
 }
+
+#[test]
+fn common_inline_udtf_preserves_return_schema() {
+    use spark_connect_proto::{common_inline_user_defined_table_function, 
relation};
+
+    // DDL must survive unchanged, including malformed/scalar DDL: Connect
+    // delegates parsing and validation to the server. Struct schemas stay 
typed.
+    for return_type in [
+        Some(DataType::Unparsed {
+            data_type_string: "value ARRAY<STRUCT<x: INT>>".to_string(),
+        }),
+        Some(DataType::Unparsed {
+            data_type_string: "invalid DDL !!!".to_string(),
+        }),
+        Some(DataType::Unparsed {
+            data_type_string: "INT".to_string(),
+        }),
+        Some(DataType::from_ddl("value BIGINT").unwrap()),
+        None,
+    ] {
+        let expected = return_type.as_ref().map(DataType::to_proto);
+        let plan = LogicalPlan::CommonInlineUdtf {
+            function_name: "echo".to_string(),
+            deterministic: false,
+            arguments: vec![col_expr("x")],
+            return_type,
+            eval_type: 300,
+            command: vec![1, 2, 3],
+            python_ver: "3.12".to_string(),
+        };
+        let 
Some(relation::RelType::CommonInlineUserDefinedTableFunction(call)) =
+            plan.to_proto().rel_type
+        else {
+            panic!("expected an inline UDTF relation");
+        };
+        let 
Some(common_inline_user_defined_table_function::Function::PythonUdtf(function)) 
=
+            call.function
+        else {
+            panic!("expected a Python UDTF");
+        };
+        assert_eq!(function.return_type, expected);
+        assert_eq!(function.eval_type, 300);
+        assert_eq!(function.command.as_ref(), &[1, 2, 3]);
+    }
+}
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index c62fa88..152fc71 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -59,8 +59,9 @@ class UserDefinedTableFunction:
         )
         self.deterministic = deterministic
         self.python_ver = f"{sys.version_info.major}.{sys.version_info.minor}"
-        # Cloudpickle the handler class + declared return type for the worker.
-        self.command = CloudPickleSerializer().dumps((func, returnType))
+        # Connect sends the schema separately in PythonUDTF.return_type. The
+        # worker command contains only the handler class, unlike scalar UDFs.
+        self.command = CloudPickleSerializer().dumps(func)
 
     def _active_session(self):
         from pyspark.sql import SparkSession
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 0fbb97b..6868abb 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -107,6 +107,26 @@ def 
test_map_in_arrow_and_pandas_with_a_struct_type_built_with_add(session, sche
     assert [tuple(r) for r in rows] == [(0, 0), (1, 2), (2, 4)]
 
 
+def test_udtf_takes_a_ddl_or_struct_return_type_as_pyspark(session):
+    from pyspark.errors.exceptions.connect import ParseException
+
+    class Echo:
+        def eval(self, value):
+            yield (value,)
+            yield (value + 1,)
+
+    struct = StructType([StructField("value", LongType())])
+    for return_type in ["value BIGINT", "value: bigint", struct]:
+        for fn in (F.udtf(Echo, returnType=return_type), 
F.udtf(returnType=return_type)(Echo)):
+            assert fn(F.lit(1)).collect() == [Row(value=1), Row(value=2)]
+
+    # The server parses the DDL, so an invalid one fails when the query runs.
+    df = F.udtf(Echo, returnType="invalid DDL !!!")(F.lit(1))
+    with pytest.raises(ParseException) as error:
+        df.collect()
+    assert error.value.getCondition() == "PARSE_SYNTAX_ERROR"
+
+
 def test_map_functions_take_a_ddl_schema(session):
     df = session.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0)], ["k", "v"])
     schema = "k long, v double"
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index f7e4110..7cb7536 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1138,6 +1138,118 @@ def test_udtf_call_without_session_raises():
         T2(1)
 
 
+def test_udtf_return_type_connect_validation_and_command():
+    # A Connect DDL string (including invalid DDL) is validated by the server,
+    # not when decorating the class or constructing the DataFrame. Isolate the
+    # active session so the other offline tests still start without a session.
+    import subprocess
+    import sys
+
+    subprocess.run([sys.executable, "-c", """
+from pyspark.sql import SparkSession, DataFrame, functions as F, types as T
+from pyspark.serializers import CloudPickleSerializer
+
+spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
+class Echo:
+    def eval(self, value):
+        yield (value,)
+
+schema = T.StructType([T.StructField("value", T.LongType())])
+for return_type in ["value BIGINT", "STRUCT<value: BIGINT>",
+                    "value ARRAY<STRUCT<x: INT>>", "invalid DDL !!!", "INT", 
schema]:
+    for decorate in [False, True]:
+        fn = (F.udtf(returnType=return_type)(Echo) if decorate else
+              F.udtf(Echo, returnType=return_type))
+        assert isinstance(fn(F.lit(1)), DataFrame)
+        handler = CloudPickleSerializer().loads(fn.command)
+        assert isinstance(handler, type)
+        assert list(handler().eval(7)) == [(7,)]
+        assert fn.returnType == return_type
+
+# Dynamic schemas omit return_type and leave analysis to the Python handler.
+class Dynamic(Echo):
+    @staticmethod
+    def analyze(value):
+        pass
+assert isinstance(F.udtf(Dynamic)(F.lit(1)), DataFrame)
+"""], check=True)
+
+
+def test_udtf_python_binding_preserves_wire_schema_and_command():
+    # Exercise the Python -> PyO3 -> protobuf seam against the recording mock,
+    # including DDL that the client must send without parsing. Keep the active
+    # session isolated, as in the construction-only test above.
+    import subprocess
+    import sys
+    from pathlib import Path
+
+    pytest.importorskip("grpc")
+    subprocess.run([sys.executable, "-c", """
+import json
+import sys
+sys.path.insert(0, sys.argv[1])
+from _mock_connect_server import mock_connect_session, parse_message
+from pyspark.sql import functions as F, types as T
+from pyspark.sql.udtf import AnalyzeResult
+from pyspark.serializers import CloudPickleSerializer
+
+class Echo:
+    def eval(self, value):
+        yield (value,)
+
+class Dynamic(Echo):
+    @staticmethod
+    def analyze(value):
+        return AnalyzeResult(T.StructType([T.StructField("value", 
T.LongType())]))
+
+schema = T.StructType([T.StructField("value", T.LongType(), False, {"comment": 
"kept"})])
+with mock_connect_session() as (spark, service):
+    for return_type in ["value BIGINT", "STRUCT<value: BIGINT>",
+                        "value ARRAY<STRUCT<x: INT>>", "  `v à` BIGINT  ",
+                        "", "invalid DDL !!!", "INT", schema, None]:
+        for decorate in [False, True]:
+            handler = Dynamic if return_type is None else Echo
+            fn = (F.udtf(returnType=return_type)(handler) if decorate else
+                  F.udtf(handler, returnType=return_type))
+            calls = service.calls
+            df = fn(F.lit(1))
+            assert service.calls == calls
+            assert df.columns == ["id"]
+            assert service.calls == calls + 1
+
+            # AnalyzePlanRequest.schema=4 -> plan=1 -> root=1 -> 
inline_udtf=38.
+            call = parse_message(service.analyze_requests[-1])
+            for field in (4, 1, 1, 38):
+                call = parse_message(call[field][0])
+            assert call[1] == [handler.__name__.encode()]
+            # CommonInlineUserDefinedTableFunction.python_udtf=4.
+            udtf = parse_message(call[4][0])
+            if return_type is None:
+                assert 1 not in udtf
+            else:
+                sent_type = parse_message(udtf[1][0])
+                if isinstance(return_type, str):
+                    assert set(sent_type) == {24}  # DataType.unparsed
+                    unparsed = parse_message(sent_type[24][0])
+                    assert unparsed.get(1, [b""])[0].decode() == return_type
+                else:
+                    assert set(sent_type) == {21}  # DataType.struct
+                    fields = parse_message(sent_type[21][0])[1]
+                    assert len(fields) == 1
+                    field = parse_message(fields[0])
+                    assert field[1] == [b"value"]
+                    assert parse_message(field[2][0]) == {7: [b""]}  # LongType
+                    assert field.get(3, [0]) == [0]  # nullable=False
+                    assert json.loads(field[4][0]) == {"comment": "kept"}
+            assert udtf[2] == [300]
+            command_handler = CloudPickleSerializer().loads(udtf[3][0])
+            assert isinstance(command_handler, type)
+            assert command_handler.__name__ == handler.__name__
+            assert list(command_handler().eval(7)) == [(7,)]
+            assert udtf[4] == [fn.python_ver.encode()]
+""", str(Path(__file__).resolve().parent)], check=True)
+
+
 def test_spark_conf_stub():
     from pyspark import SparkConf
     c = SparkConf().set("a", "1").setAll({"b": "2", "c": "3"})


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to