This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new aa09124c [SPARK-61152][CONNECT][PYTHON] Raise 
INVALID_RETURN_TYPE_FOR_PANDAS_UDF for a non-struct return type of mapInPandas, 
applyInPandas and the other map functions, in the Python client as PySpark does
aa09124c is described below

commit aa09124c97011d073fb429070dc21e276ba63703
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:46:25 2026 +0900

    [SPARK-61152][CONNECT][PYTHON] Raise INVALID_RETURN_TYPE_FOR_PANDAS_UDF for 
a non-struct return type of mapInPandas, applyInPandas and the other map 
functions, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61152](https://issues.apache.org/jira/browse/SPARK-61152): Raise 
`INVALID_RETURN_TYPE_FOR_PANDAS_UDF` for a non-struct return type of 
`mapInPandas`, `applyInPandas` and the other map functions, in the Python 
client as PySpark does.
    
    PySpark builds the UDF of `mapInPandas`, `mapInArrow`, `applyInPandas`, 
`applyInArrow`, `applyInPandasWithState` and the cogrouped `applyInPandas` and 
`applyInArrow` through `UserDefinedFunction`, which checks the return type for 
the eval type. A return type that is not a struct raises `PySparkTypeError` 
`INVALID_RETURN_TYPE_FOR_PANDAS_UDF`, or `INVALID_RETURN_TYPE_FOR_ARROW_UDF` 
for the Arrow functions, at the call, and the plan is never sent.
    
    The drop-in already has this check verbatim in `pyspark/sql/udf.py`. 
`build_map_udf` now calls `UserDefinedFunction._check_return_type` with the 
resolved return type and the eval type, right after the schema is resolved, as 
the official client does. The check is a no-op for the transform-with-state 
eval types, which build with `build_map_udf` too.
    
    ### Why are the changes needed?
    
    `df.mapInPandas(f, "int")`, `df.mapInPandas(f, IntegerType())`, 
`df.mapInArrow(f, "long")`, `df.groupBy("id").applyInPandas(g, "int")` and 
`df.groupBy("id").applyInArrow(g, LongType())` were sent to the server. The 
server failed with `UnknownException` (`ClassCastException`), or 
`IllegalArgumentException: The returnType of the udf must be a struct`. PySpark 
raises `PySparkTypeError` `INVALID_RETURN_TYPE_FOR_PANDAS_UDF` (for example 
`Pandas UDF should return StructType for SQL_MAP_PAND [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A map or group function with a non-struct return type raises the 
PySpark `PySparkTypeError` on the client when the call builds the plan, with 
the same error class, message and `return_type` as PySpark, instead of a server 
error.
    
    ### How was this patch tested?
    
    - `test_map_and_group_functions_reject_a_non_struct_return_type_as_pyspark` 
(offline, mock server): the nine call forms (mapInPandas with a type and a DDL 
string, mapInArrow, applyInPandas with a DDL string and an ArrayType, 
applyInArrow, the two cogrouped applies, applyInPandasWithState) raise the 
expected condition and `return_type` at the call, and no request is sent. The 
same calls give the same condition, message and return type with official 
PySpark 4.3.0.
    - `test_map_functions_reject_a_non_struct_return_type_before_the_server` 
(live): mapInPandas and applyInArrow with a non-struct type raise on the 
client, and a struct schema still runs.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and 
the live tests of the map, grouped, cogrouped and stateful functions (22 
passed) against a local Spark 4.3.0 server; rustfmt passes, and clippy reports 
no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #341 from HyukjinKwon/SPARK-61152-map-functions-struct-return-type.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs  | 12 ++++++++++++
 python/tests/test_dropin_e2e.py     | 16 ++++++++++++++++
 python/tests/test_dropin_offline.py | 38 +++++++++++++++++++++++++++++++++++++
 3 files changed, 66 insertions(+)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 26a96332..756b2f97 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -3958,6 +3958,18 @@ pub(crate) fn build_map_udf(
 ) -> PyResult<CommonInlineUserDefinedFunctionExpression> {
     let name = udf_name(func)?;
     let return_type = resolve_datatype(py, session, schema)?;
+    // As PySpark's `MapPartitions`, `GroupMap` and the other map nodes do 
when they build the
+    // UDF, the return type is checked for the eval type: a map or group 
function returns a
+    // struct, so another type raises `INVALID_RETURN_TYPE_FOR_PANDAS_UDF` 
here, not at the action.
+    py.import("pyspark.sql.udf")?
+        .getattr("UserDefinedFunction")?
+        .call_method1(
+            "_check_return_type",
+            (
+                crate::types::data_type_to_py(py, &return_type)?,
+                eval_type_id,
+            ),
+        )?;
     // The worker reads the pickled type as a `DataType`, so a DDL string is 
pickled as the
     // type it parses to, as the reference client pickles the type of 
`_parse_ddl`.
     let pickled_type = if schema.is_instance_of::<pyo3::types::PyString>() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ffe952ef..16348e4c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1784,6 +1784,22 @@ def 
test_apply_in_pandas_with_state_runs_as_pyspark(session, tmp_path):
         session.conf.unset("spark.sql.shuffle.partitions")
 
 
+def 
test_map_functions_reject_a_non_struct_return_type_before_the_server(session):
+    # As PySpark's, a map or group function whose return type is not a struct 
raises on the
+    # client when it is built, so the server never sees the plan.
+    from pyspark.errors import PySparkTypeError
+
+    df = session.range(2)
+    with pytest.raises(PySparkTypeError) as e:
+        df.mapInPandas(lambda frames: frames, "long")
+    assert e.value.getCondition() == "INVALID_RETURN_TYPE_FOR_PANDAS_UDF"
+    with pytest.raises(PySparkTypeError) as e:
+        df.groupBy("id").applyInArrow(lambda table: table, "long")
+    assert e.value.getCondition() == "INVALID_RETURN_TYPE_FOR_ARROW_UDF"
+    assert e.value.getMessageParameters()["return_type"] == "LongType()"
+    assert df.mapInPandas(lambda frames: frames, "id long").collect() == 
[Row(id=0), Row(id=1)]
+
+
 def 
test_variant_spatial_and_user_defined_values_round_trip_as_pyspark(session):
     # As PySpark's: createDataFrame sends a VariantVal, a Geometry, a 
Geography and a value of
     # a user-defined type, and collects them as they are.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 00de03f0..8ff09063 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8271,6 +8271,44 @@ def 
test_schema_cache_cold_children_keep_independent_caches(mock_connect):
     assert df.columns == ["id"]
 
 
+def 
test_map_and_group_functions_reject_a_non_struct_return_type_as_pyspark(mock_connect):
+    # As PySpark's MapPartitions, GroupMap, CoGroupMap and 
ApplyInPandasWithState, which check
+    # the return type of their UDF when they are built: a type that is not a 
struct raises
+    # INVALID_RETURN_TYPE_FOR_PANDAS_UDF, or INVALID_RETURN_TYPE_FOR_ARROW_UDF 
for an Arrow
+    # function, at the call, before the server sees the plan.
+    from pyspark.errors import PySparkTypeError
+
+    spark, service = mock_connect
+    df = spark.range(1)
+    grouped = df.groupBy("id")
+    cogrouped = grouped.cogroup(df.groupBy("id"))
+    pandas_error = "INVALID_RETURN_TYPE_FOR_PANDAS_UDF"
+    arrow_error = "INVALID_RETURN_TYPE_FOR_ARROW_UDF"
+    cases = [
+        (lambda: df.mapInPandas(lambda it: it, T.IntegerType()), pandas_error, 
"IntegerType()"),
+        (lambda: df.mapInPandas(lambda it: it, "int"), pandas_error, 
"IntegerType()"),
+        (lambda: df.mapInArrow(lambda it: it, "long"), pandas_error, 
"LongType()"),
+        (lambda: grouped.applyInPandas(lambda pdf: pdf, "int"), pandas_error, 
"IntegerType()"),
+        (lambda: grouped.applyInPandas(lambda pdf: pdf, 
T.ArrayType(T.IntegerType())),
+         pandas_error, "ArrayType(IntegerType(), True)"),
+        (lambda: grouped.applyInArrow(lambda table: table, T.LongType()), 
arrow_error, "LongType()"),
+        (lambda: cogrouped.applyInPandas(lambda left, right: left, "int"),
+         pandas_error, "IntegerType()"),
+        (lambda: cogrouped.applyInArrow(lambda left, right: left, "long"),
+         arrow_error, "LongType()"),
+        (lambda: grouped.applyInPandasWithState(
+            lambda key, pdfs, state: iter([]), T.IntegerType(),
+            T.StructType([T.StructField("c", T.LongType())]), "append", 
"NoTimeout"),
+         pandas_error, "IntegerType()"),
+    ]
+    for call, condition, return_type in cases:
+        with pytest.raises(PySparkTypeError) as e:
+            call()
+        assert e.value.getCondition() == condition
+        assert e.value.getMessageParameters()["return_type"] == return_type
+    assert service.execute_requests == []
+
+
 @pytest.mark.parametrize("transform", [
     pytest.param(lambda df: df.selectExpr("id AS renamed"), id="select"),
     pytest.param(lambda df: df.zipWithIndex(), id="zipWithIndex"),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to