This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new aa09124c [SPARK-61152][CONNECT][PYTHON] Raise
INVALID_RETURN_TYPE_FOR_PANDAS_UDF for a non-struct return type of mapInPandas,
applyInPandas and the other map functions, in the Python client as PySpark does
aa09124c is described below
commit aa09124c97011d073fb429070dc21e276ba63703
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:46:25 2026 +0900
[SPARK-61152][CONNECT][PYTHON] Raise INVALID_RETURN_TYPE_FOR_PANDAS_UDF for
a non-struct return type of mapInPandas, applyInPandas and the other map
functions, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61152](https://issues.apache.org/jira/browse/SPARK-61152): Raise
`INVALID_RETURN_TYPE_FOR_PANDAS_UDF` for a non-struct return type of
`mapInPandas`, `applyInPandas` and the other map functions, in the Python
client as PySpark does.
PySpark builds the UDF of `mapInPandas`, `mapInArrow`, `applyInPandas`,
`applyInArrow`, `applyInPandasWithState` and the cogrouped `applyInPandas` and
`applyInArrow` through `UserDefinedFunction`, which checks the return type for
the eval type. A return type that is not a struct raises `PySparkTypeError`
`INVALID_RETURN_TYPE_FOR_PANDAS_UDF`, or `INVALID_RETURN_TYPE_FOR_ARROW_UDF`
for the Arrow functions, at the call, and the plan is never sent.
The drop-in already has this check verbatim in `pyspark/sql/udf.py`.
`build_map_udf` now calls `UserDefinedFunction._check_return_type` with the
resolved return type and the eval type, right after the schema is resolved, as
the official client does. The check is a no-op for the transform-with-state
eval types, which build with `build_map_udf` too.
### Why are the changes needed?
`df.mapInPandas(f, "int")`, `df.mapInPandas(f, IntegerType())`,
`df.mapInArrow(f, "long")`, `df.groupBy("id").applyInPandas(g, "int")` and
`df.groupBy("id").applyInArrow(g, LongType())` were sent to the server. The
server failed with `UnknownException` (`ClassCastException`), or
`IllegalArgumentException: The returnType of the udf must be a struct`. PySpark
raises `PySparkTypeError` `INVALID_RETURN_TYPE_FOR_PANDAS_UDF` (for example
`Pandas UDF should return StructType for SQL_MAP_PAND [...]
### Does this PR introduce _any_ user-facing change?
Yes. A map or group function with a non-struct return type raises the
PySpark `PySparkTypeError` on the client when the call builds the plan, with
the same error class, message and `return_type` as PySpark, instead of a server
error.
### How was this patch tested?
- `test_map_and_group_functions_reject_a_non_struct_return_type_as_pyspark`
(offline, mock server): the nine call forms (mapInPandas with a type and a DDL
string, mapInArrow, applyInPandas with a DDL string and an ArrayType,
applyInArrow, the two cogrouped applies, applyInPandasWithState) raise the
expected condition and `return_type` at the call, and no request is sent. The
same calls give the same condition, message and return type with official
PySpark 4.3.0.
- `test_map_functions_reject_a_non_struct_return_type_before_the_server`
(live): mapInPandas and applyInArrow with a non-struct type raise on the
client, and a struct schema still runs.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and
the live tests of the map, grouped, cogrouped and stateful functions (22
passed) against a local Spark 4.3.0 server; rustfmt passes, and clippy reports
no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #341 from HyukjinKwon/SPARK-61152-map-functions-struct-return-type.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 12 ++++++++++++
python/tests/test_dropin_e2e.py | 16 ++++++++++++++++
python/tests/test_dropin_offline.py | 38 +++++++++++++++++++++++++++++++++++++
3 files changed, 66 insertions(+)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 26a96332..756b2f97 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -3958,6 +3958,18 @@ pub(crate) fn build_map_udf(
) -> PyResult<CommonInlineUserDefinedFunctionExpression> {
let name = udf_name(func)?;
let return_type = resolve_datatype(py, session, schema)?;
+ // As PySpark's `MapPartitions`, `GroupMap` and the other map nodes do
when they build the
+ // UDF, the return type is checked for the eval type: a map or group
function returns a
+ // struct, so another type raises `INVALID_RETURN_TYPE_FOR_PANDAS_UDF`
here, not at the action.
+ py.import("pyspark.sql.udf")?
+ .getattr("UserDefinedFunction")?
+ .call_method1(
+ "_check_return_type",
+ (
+ crate::types::data_type_to_py(py, &return_type)?,
+ eval_type_id,
+ ),
+ )?;
// The worker reads the pickled type as a `DataType`, so a DDL string is
pickled as the
// type it parses to, as the reference client pickles the type of
`_parse_ddl`.
let pickled_type = if schema.is_instance_of::<pyo3::types::PyString>() {
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ffe952ef..16348e4c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1784,6 +1784,22 @@ def
test_apply_in_pandas_with_state_runs_as_pyspark(session, tmp_path):
session.conf.unset("spark.sql.shuffle.partitions")
+def
test_map_functions_reject_a_non_struct_return_type_before_the_server(session):
+ # As PySpark's, a map or group function whose return type is not a struct
raises on the
+ # client when it is built, so the server never sees the plan.
+ from pyspark.errors import PySparkTypeError
+
+ df = session.range(2)
+ with pytest.raises(PySparkTypeError) as e:
+ df.mapInPandas(lambda frames: frames, "long")
+ assert e.value.getCondition() == "INVALID_RETURN_TYPE_FOR_PANDAS_UDF"
+ with pytest.raises(PySparkTypeError) as e:
+ df.groupBy("id").applyInArrow(lambda table: table, "long")
+ assert e.value.getCondition() == "INVALID_RETURN_TYPE_FOR_ARROW_UDF"
+ assert e.value.getMessageParameters()["return_type"] == "LongType()"
+ assert df.mapInPandas(lambda frames: frames, "id long").collect() ==
[Row(id=0), Row(id=1)]
+
+
def
test_variant_spatial_and_user_defined_values_round_trip_as_pyspark(session):
# As PySpark's: createDataFrame sends a VariantVal, a Geometry, a
Geography and a value of
# a user-defined type, and collects them as they are.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 00de03f0..8ff09063 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -8271,6 +8271,44 @@ def
test_schema_cache_cold_children_keep_independent_caches(mock_connect):
assert df.columns == ["id"]
+def
test_map_and_group_functions_reject_a_non_struct_return_type_as_pyspark(mock_connect):
+ # As PySpark's MapPartitions, GroupMap, CoGroupMap and
ApplyInPandasWithState, which check
+ # the return type of their UDF when they are built: a type that is not a
struct raises
+ # INVALID_RETURN_TYPE_FOR_PANDAS_UDF, or INVALID_RETURN_TYPE_FOR_ARROW_UDF
for an Arrow
+ # function, at the call, before the server sees the plan.
+ from pyspark.errors import PySparkTypeError
+
+ spark, service = mock_connect
+ df = spark.range(1)
+ grouped = df.groupBy("id")
+ cogrouped = grouped.cogroup(df.groupBy("id"))
+ pandas_error = "INVALID_RETURN_TYPE_FOR_PANDAS_UDF"
+ arrow_error = "INVALID_RETURN_TYPE_FOR_ARROW_UDF"
+ cases = [
+ (lambda: df.mapInPandas(lambda it: it, T.IntegerType()), pandas_error,
"IntegerType()"),
+ (lambda: df.mapInPandas(lambda it: it, "int"), pandas_error,
"IntegerType()"),
+ (lambda: df.mapInArrow(lambda it: it, "long"), pandas_error,
"LongType()"),
+ (lambda: grouped.applyInPandas(lambda pdf: pdf, "int"), pandas_error,
"IntegerType()"),
+ (lambda: grouped.applyInPandas(lambda pdf: pdf,
T.ArrayType(T.IntegerType())),
+ pandas_error, "ArrayType(IntegerType(), True)"),
+ (lambda: grouped.applyInArrow(lambda table: table, T.LongType()),
arrow_error, "LongType()"),
+ (lambda: cogrouped.applyInPandas(lambda left, right: left, "int"),
+ pandas_error, "IntegerType()"),
+ (lambda: cogrouped.applyInArrow(lambda left, right: left, "long"),
+ arrow_error, "LongType()"),
+ (lambda: grouped.applyInPandasWithState(
+ lambda key, pdfs, state: iter([]), T.IntegerType(),
+ T.StructType([T.StructField("c", T.LongType())]), "append",
"NoTimeout"),
+ pandas_error, "IntegerType()"),
+ ]
+ for call, condition, return_type in cases:
+ with pytest.raises(PySparkTypeError) as e:
+ call()
+ assert e.value.getCondition() == condition
+ assert e.value.getMessageParameters()["return_type"] == return_type
+ assert service.execute_requests == []
+
+
@pytest.mark.parametrize("transform", [
pytest.param(lambda df: df.selectExpr("id AS renamed"), id="select"),
pytest.param(lambda df: df.zipWithIndex(), id="zipWithIndex"),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]