This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 829c8ea [SPARK-60678][CONNECT] Support Python UDTF DDL string return
schemas
829c8ea is described below
commit 829c8ead5526a5e3f2cc6398860bdc59ace9195f
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 16:03:57 2026 +0900
[SPARK-60678][CONNECT] Support Python UDTF DDL string return schemas
### What changes were proposed in this pull request?
Python UDTFs now work against a Spark Connect server, with a DDL string or
a `StructType` as the return type, as in PySpark:
- The PyO3 binding keeps a string return type as `DataType::Unparsed`, so
the server parses and validates the DDL when the query runs, as PySpark Connect
does. Typed schemas go through the existing Python-to-Rust type converter.
Before, the binding took only a `DataType`, so a string raised a `TypeError`.
- `PythonUDTF.command` now holds only the cloudpickled handler class, as in
PySpark Connect; the return schema is sent separately in
`PythonUDTF.return_type`. Before, the command was the tuple `(handler,
returnType)`.
```python
class Echo:
def eval(self, value):
yield (value,)
udtf(Echo, returnType="value BIGINT")(lit(1)).collect()
```
| Return type | PySpark 4.3.0 and now | Before |
| --- | --- | --- |
| `"value BIGINT"` | `[Row(value=1)]` | `TypeError: argument 'return_type':
'str' object is not an instance of 'DataType'` |
| `StructType([StructField("value", LongType())])` | `[Row(value=1)]` |
`PythonException`: "Invalid UDTF handler type. Expected a class (type 'type'),
but got an instance of tuple." from the worker |
| `"invalid DDL !!!"` | `ParseException` (`PARSE_SYNTAX_ERROR`) when the
query runs | `TypeError` when called |
### Why are the changes needed?
[SPARK-60678](https://issues.apache.org/jira/browse/SPARK-60678): Python
UDTFs did not run at all. A string return type failed in the client, and the
server's Python worker rejected the tuple command for any return type.
### Does this PR introduce _any_ user-facing change?
Yes. `udtf` with a DDL string or a `StructType` return type returns its
rows, as in PySpark, instead of failing. An invalid DDL string fails when the
query runs, as in PySpark, instead of when the UDTF is called.
### How was this patch tested?
- Offline tests check the direct and decorator forms with valid, nested,
malformed and scalar DDL, a `StructType` (with metadata), and an omitted schema
for a handler with `analyze`. Against a recording mock gRPC server, they check
the unparsed DDL, the typed schema, the evaluation type and the handler-only
command that reach it.
- A Rust test checks the serialized return schema of the inline UDTF
relation.
- A new live test runs UDTFs with DDL and `StructType` return types, and an
invalid DDL, against a Spark 4.3.0 server. It also passes with official PySpark
4.3.0 and fails without this change.
- Against the current master, merged into this branch, and a local Spark
4.3.0 server: the lib and every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (772 passed), the build of all tests
with `wasm-udf`, the offline Python suites (722 passed, 9 skipped), the live
drop-in and Arrow suites (77 passed) and the live cases (10 passed). `cargo fmt
--all --check` passes.
Not in this change: PySpark keeps a string return type as an
`UnparsedDataType` in `UserDefinedTableFunction.returnType`, which this keeps
as the string, and PySpark validates the handler class when the UDTF is created
(`INVALID_UDTF_NO_EVAL` and others).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master and the review
follow-ups by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #119 from dbtsai/codex/spark-60678-udtf-ddl.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Co-authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/functions.rs | 17 +++-
.../spark-connect/tests/coverage_plan_map_udtf.rs | 45 +++++++++
python/pyspark/sql/udtf.py | 5 +-
python/tests/test_dropin_e2e.py | 20 ++++
python/tests/test_dropin_offline.py | 112 +++++++++++++++++++++
5 files changed, 194 insertions(+), 5 deletions(-)
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index d02da81..6f24de8 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -9,7 +9,6 @@ use
spark_connect::udf::{CommonInlineUserDefinedFunctionExpression, PythonUDFPay
use crate::column::PyColumn;
use crate::dataframe::PyDataFrame;
use crate::session::PySparkSession;
-use crate::types::PyDataType;
// Include the auto-generated dispatch function
mod dispatch {
@@ -78,7 +77,7 @@ fn pyfunc_make_udf(
fn pyfunc_make_udtf(
session: PyRef<'_, PySparkSession>,
name: String,
- return_type: Option<PyRef<'_, PyDataType>>,
+ return_type: Option<&Bound<'_, PyAny>>,
eval_type: i32,
command_bytes: Vec<u8>,
python_ver: String,
@@ -88,7 +87,19 @@ fn pyfunc_make_udtf(
for arg in args {
col_args.push(to_column(&arg)?);
}
- let rt = return_type.map(|d| d.inner.clone());
+ // Connect leaves DDL parsing and validation to the server. In particular,
+ // do not use py_to_data_type's eager DDL parser for string return types.
+ let rt = return_type
+ .map(|value| {
+ if let Ok(ddl) = value.extract::<String>() {
+ Ok(spark_connect::types::DataType::Unparsed {
+ data_type_string: ddl,
+ })
+ } else {
+ crate::types::py_to_data_type(value)
+ }
+ })
+ .transpose()?;
let df = session.session.tvf().udtf(
&name,
col_args,
diff --git a/crates/spark-connect/tests/coverage_plan_map_udtf.rs
b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
index 431ef2e..e452eea 100644
--- a/crates/spark-connect/tests/coverage_plan_map_udtf.rs
+++ b/crates/spark-connect/tests/coverage_plan_map_udtf.rs
@@ -135,3 +135,48 @@ fn
common_inline_udtf_to_proto_with_and_without_return_type() {
};
assert!(no_rt.to_proto().rel_type.is_some());
}
+
+#[test]
+fn common_inline_udtf_preserves_return_schema() {
+ use spark_connect_proto::{common_inline_user_defined_table_function,
relation};
+
+ // DDL must survive unchanged, including malformed/scalar DDL: Connect
+ // delegates parsing and validation to the server. Struct schemas stay
typed.
+ for return_type in [
+ Some(DataType::Unparsed {
+ data_type_string: "value ARRAY<STRUCT<x: INT>>".to_string(),
+ }),
+ Some(DataType::Unparsed {
+ data_type_string: "invalid DDL !!!".to_string(),
+ }),
+ Some(DataType::Unparsed {
+ data_type_string: "INT".to_string(),
+ }),
+ Some(DataType::from_ddl("value BIGINT").unwrap()),
+ None,
+ ] {
+ let expected = return_type.as_ref().map(DataType::to_proto);
+ let plan = LogicalPlan::CommonInlineUdtf {
+ function_name: "echo".to_string(),
+ deterministic: false,
+ arguments: vec![col_expr("x")],
+ return_type,
+ eval_type: 300,
+ command: vec![1, 2, 3],
+ python_ver: "3.12".to_string(),
+ };
+ let
Some(relation::RelType::CommonInlineUserDefinedTableFunction(call)) =
+ plan.to_proto().rel_type
+ else {
+ panic!("expected an inline UDTF relation");
+ };
+ let
Some(common_inline_user_defined_table_function::Function::PythonUdtf(function))
=
+ call.function
+ else {
+ panic!("expected a Python UDTF");
+ };
+ assert_eq!(function.return_type, expected);
+ assert_eq!(function.eval_type, 300);
+ assert_eq!(function.command.as_ref(), &[1, 2, 3]);
+ }
+}
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index c62fa88..152fc71 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -59,8 +59,9 @@ class UserDefinedTableFunction:
)
self.deterministic = deterministic
self.python_ver = f"{sys.version_info.major}.{sys.version_info.minor}"
- # Cloudpickle the handler class + declared return type for the worker.
- self.command = CloudPickleSerializer().dumps((func, returnType))
+ # Connect sends the schema separately in PythonUDTF.return_type. The
+ # worker command contains only the handler class, unlike scalar UDFs.
+ self.command = CloudPickleSerializer().dumps(func)
def _active_session(self):
from pyspark.sql import SparkSession
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 0fbb97b..6868abb 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -107,6 +107,26 @@ def
test_map_in_arrow_and_pandas_with_a_struct_type_built_with_add(session, sche
assert [tuple(r) for r in rows] == [(0, 0), (1, 2), (2, 4)]
+def test_udtf_takes_a_ddl_or_struct_return_type_as_pyspark(session):
+ from pyspark.errors.exceptions.connect import ParseException
+
+ class Echo:
+ def eval(self, value):
+ yield (value,)
+ yield (value + 1,)
+
+ struct = StructType([StructField("value", LongType())])
+ for return_type in ["value BIGINT", "value: bigint", struct]:
+ for fn in (F.udtf(Echo, returnType=return_type),
F.udtf(returnType=return_type)(Echo)):
+ assert fn(F.lit(1)).collect() == [Row(value=1), Row(value=2)]
+
+ # The server parses the DDL, so an invalid one fails when the query runs.
+ df = F.udtf(Echo, returnType="invalid DDL !!!")(F.lit(1))
+ with pytest.raises(ParseException) as error:
+ df.collect()
+ assert error.value.getCondition() == "PARSE_SYNTAX_ERROR"
+
+
def test_map_functions_take_a_ddl_schema(session):
df = session.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0)], ["k", "v"])
schema = "k long, v double"
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index f7e4110..7cb7536 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1138,6 +1138,118 @@ def test_udtf_call_without_session_raises():
T2(1)
+def test_udtf_return_type_connect_validation_and_command():
+ # A Connect DDL string (including invalid DDL) is validated by the server,
+ # not when decorating the class or constructing the DataFrame. Isolate the
+ # active session so the other offline tests still start without a session.
+ import subprocess
+ import sys
+
+ subprocess.run([sys.executable, "-c", """
+from pyspark.sql import SparkSession, DataFrame, functions as F, types as T
+from pyspark.serializers import CloudPickleSerializer
+
+spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
+class Echo:
+ def eval(self, value):
+ yield (value,)
+
+schema = T.StructType([T.StructField("value", T.LongType())])
+for return_type in ["value BIGINT", "STRUCT<value: BIGINT>",
+ "value ARRAY<STRUCT<x: INT>>", "invalid DDL !!!", "INT",
schema]:
+ for decorate in [False, True]:
+ fn = (F.udtf(returnType=return_type)(Echo) if decorate else
+ F.udtf(Echo, returnType=return_type))
+ assert isinstance(fn(F.lit(1)), DataFrame)
+ handler = CloudPickleSerializer().loads(fn.command)
+ assert isinstance(handler, type)
+ assert list(handler().eval(7)) == [(7,)]
+ assert fn.returnType == return_type
+
+# Dynamic schemas omit return_type and leave analysis to the Python handler.
+class Dynamic(Echo):
+ @staticmethod
+ def analyze(value):
+ pass
+assert isinstance(F.udtf(Dynamic)(F.lit(1)), DataFrame)
+"""], check=True)
+
+
+def test_udtf_python_binding_preserves_wire_schema_and_command():
+ # Exercise the Python -> PyO3 -> protobuf seam against the recording mock,
+ # including DDL that the client must send without parsing. Keep the active
+ # session isolated, as in the construction-only test above.
+ import subprocess
+ import sys
+ from pathlib import Path
+
+ pytest.importorskip("grpc")
+ subprocess.run([sys.executable, "-c", """
+import json
+import sys
+sys.path.insert(0, sys.argv[1])
+from _mock_connect_server import mock_connect_session, parse_message
+from pyspark.sql import functions as F, types as T
+from pyspark.sql.udtf import AnalyzeResult
+from pyspark.serializers import CloudPickleSerializer
+
+class Echo:
+ def eval(self, value):
+ yield (value,)
+
+class Dynamic(Echo):
+ @staticmethod
+ def analyze(value):
+ return AnalyzeResult(T.StructType([T.StructField("value",
T.LongType())]))
+
+schema = T.StructType([T.StructField("value", T.LongType(), False, {"comment":
"kept"})])
+with mock_connect_session() as (spark, service):
+ for return_type in ["value BIGINT", "STRUCT<value: BIGINT>",
+ "value ARRAY<STRUCT<x: INT>>", " `v à` BIGINT ",
+ "", "invalid DDL !!!", "INT", schema, None]:
+ for decorate in [False, True]:
+ handler = Dynamic if return_type is None else Echo
+ fn = (F.udtf(returnType=return_type)(handler) if decorate else
+ F.udtf(handler, returnType=return_type))
+ calls = service.calls
+ df = fn(F.lit(1))
+ assert service.calls == calls
+ assert df.columns == ["id"]
+ assert service.calls == calls + 1
+
+ # AnalyzePlanRequest.schema=4 -> plan=1 -> root=1 ->
inline_udtf=38.
+ call = parse_message(service.analyze_requests[-1])
+ for field in (4, 1, 1, 38):
+ call = parse_message(call[field][0])
+ assert call[1] == [handler.__name__.encode()]
+ # CommonInlineUserDefinedTableFunction.python_udtf=4.
+ udtf = parse_message(call[4][0])
+ if return_type is None:
+ assert 1 not in udtf
+ else:
+ sent_type = parse_message(udtf[1][0])
+ if isinstance(return_type, str):
+ assert set(sent_type) == {24} # DataType.unparsed
+ unparsed = parse_message(sent_type[24][0])
+ assert unparsed.get(1, [b""])[0].decode() == return_type
+ else:
+ assert set(sent_type) == {21} # DataType.struct
+ fields = parse_message(sent_type[21][0])[1]
+ assert len(fields) == 1
+ field = parse_message(fields[0])
+ assert field[1] == [b"value"]
+ assert parse_message(field[2][0]) == {7: [b""]} # LongType
+ assert field.get(3, [0]) == [0] # nullable=False
+ assert json.loads(field[4][0]) == {"comment": "kept"}
+ assert udtf[2] == [300]
+ command_handler = CloudPickleSerializer().loads(udtf[3][0])
+ assert isinstance(command_handler, type)
+ assert command_handler.__name__ == handler.__name__
+ assert list(command_handler().eval(7)) == [(7,)]
+ assert udtf[4] == [fn.python_ver.encode()]
+""", str(Path(__file__).resolve().parent)], check=True)
+
+
def test_spark_conf_stub():
from pyspark import SparkConf
c = SparkConf().set("a", "1").setAll({"b": "2", "c": "3"})
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]