This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0b40b84 [SPARK-60974][CONNECT][PYTHON] Accept a DDL string schema in
mapInPandas, applyInPandas and the other map functions as PySpark does
0b40b84 is described below
commit 0b40b840a02ff0b3250e5b316d19db19c8e3361b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:08:52 2026 +0900
[SPARK-60974][CONNECT][PYTHON] Accept a DDL string schema in mapInPandas,
applyInPandas and the other map functions as PySpark does
### What changes were proposed in this pull request?
[SPARK-60974](https://issues.apache.org/jira/browse/SPARK-60974): a DDL
string schema of the map functions is now pickled with the function as the type
it parses to, instead of as the string. PySpark does the same: it pickles the
type of `session._parse_ddl`.
This covers:
- `mapInPandas` and `mapInArrow`;
- the grouped and cogrouped `applyInPandas` and `applyInArrow`;
- the output type of `applyInPandasWithState`, `transformWithStateInPandas`
and `transformWithState`.
These all build their function in one helper, `build_map_udf`, which now
pickles the parsed type when the schema is a string. A `DataType` schema is
pickled as before. The schema in the plan was already parsed.
### Why are the changes needed?
The Python worker reads the pickled type as a `DataType`, so a DDL string
broke these functions:
```python
df.groupBy("k").applyInPandas(lambda pdf: pdf, "k long, v double").collect()
```
This raised `PythonException` with `AssertionError: dataType k long, v
double should be an instance of <class 'pyspark.sql.types.DataType'>`, where
PySpark 4.3.0 returns the rows.
- With a DDL string, five of the six batch functions failed. `applyInArrow`
raised `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`, and `mapInArrow` worked.
- A streaming `transformWithStateInPandas` failed the same way.
### Does this PR introduce _any_ user-facing change?
Yes, these functions take a DDL string schema as in PySpark 4.3.0.
### How was this patch tested?
- Against a local Spark 4.3.0 server, the six batch functions with a DDL
string schema return the same rows as official PySpark 4.3.0, and before this
change five of them failed. A streaming `transformWithStateInPandas` with a DDL
output type also gives the same rows as official PySpark, and failed before.
- A new offline test unpickles the command that each of the six batch
functions sends, and checks its type.
- New live tests check the six batch functions, and a
`transformWithStateInPandas` query with a DDL output type. They also pass with
official PySpark 4.3.0, and on a server started with
`SPARK_SKIP_CONNECT_COMPAT_TESTS`, as CI's is.
- The lib tests and every integration test, e2e included, against the local
server (643 passed), the build of all tests with `wasm-udf`, the offline Python
suites (619 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (59 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
`applyInPandasWithState` takes the same path, but the server rejects its
plan for another reason, which
[SPARK-60976](https://issues.apache.org/jira/browse/SPARK-60976) fixes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #180 from HyukjinKwon/SPARK-60974-map-ddl.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 9 +++++-
python/tests/test_dropin_e2e.py | 61 +++++++++++++++++++++++++++++++++++++
python/tests/test_dropin_offline.py | 30 ++++++++++++++++++
3 files changed, 99 insertions(+), 1 deletion(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index f491d23..faa8b58 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2464,7 +2464,14 @@ pub(crate) fn build_map_udf(
input_cols: &[String],
) -> PyResult<CommonInlineUserDefinedFunctionExpression> {
let return_type = resolve_datatype(schema)?;
- let tup = PyTuple::new(py, [func.clone(), schema.clone()])?;
+ // The worker reads the pickled type as a `DataType`, so a DDL string is
pickled as the
+ // type it parses to, as the reference client pickles the type of
`_parse_ddl`.
+ let pickled_type = if schema.is_instance_of::<pyo3::types::PyString>() {
+ crate::types::data_type_to_py(py, &return_type)?.into_bound(py)
+ } else {
+ schema.clone()
+ };
+ let tup = PyTuple::new(py, [func.clone(), pickled_type])?;
let command = py_cloudpickle(py, tup.as_any())?;
let payload = PythonUDFPayload::new(return_type, eval_type_id, command,
py_version(py));
// The worker reconstructs the input pandas/Arrow DataFrame from the UDF's
argument
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 15c5714..20e3131 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -107,6 +107,67 @@ def
test_map_in_arrow_and_pandas_with_a_struct_type_built_with_add(session, sche
assert [tuple(r) for r in rows] == [(0, 0), (1, 2), (2, 4)]
+def test_map_functions_take_a_ddl_schema(session):
+ df = session.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0)], ["k", "v"])
+ schema = "k long, v double"
+ grouped = df.groupBy("k")
+ for result in (
+ df.mapInPandas(lambda frames: frames, schema),
+ df.mapInArrow(lambda batches: batches, schema),
+ grouped.applyInPandas(lambda pdf: pdf, schema),
+ grouped.applyInArrow(lambda table: table, schema),
+ grouped.cogroup(grouped).applyInPandas(lambda left, right: left,
schema),
+ grouped.cogroup(grouped).applyInArrow(lambda left, right: left,
schema),
+ ):
+ assert sorted(tuple(r) for r in result.collect()) == [(1, 1.0), (1,
2.0), (2, 3.0)]
+
+
+def test_transform_with_state_in_pandas_takes_a_ddl_output_type(session,
tmp_path):
+ import pandas as pd
+ from pyspark.sql.streaming.stateful_processor import StatefulProcessor
+
+ class Counter(StatefulProcessor):
+ def init(self, handle):
+ self.count = handle.getValueState("count", "c long")
+
+ def handleInputRows(self, key, rows, timerValues):
+ total = sum(len(pdf) for pdf in rows)
+ self.count.update((total,))
+ yield pd.DataFrame({"k": [key[0]], "c": [total]})
+
+ def close(self):
+ pass
+
+ confs = {
+ "spark.sql.streaming.stateStore.providerClass":
+
"org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider",
+ # A state store per shuffle partition; the default 200 take half a
minute.
+ "spark.sql.shuffle.partitions": "2",
+ }
+ for key, value in confs.items():
+ session.conf.set(key, value)
+ try:
+ source = str(tmp_path / "source")
+ session.createDataFrame([(1,), (1,), (2,)], "k
long").write.parquet(source)
+ query = (
+ session.readStream.schema("k long").parquet(source).groupBy("k")
+ .transformWithStateInPandas(Counter(), "k long, c long", "Update",
"None")
+
.writeStream.format("memory").queryName("tws_ddl_output").outputMode("update")
+ .option("checkpointLocation", str(tmp_path / "checkpoint"))
+ .trigger(availableNow=True).start()
+ )
+ try:
+ query.awaitTermination(120)
+ assert query.exception() is None
+ rows = session.table("tws_ddl_output").collect()
+ assert sorted(tuple(r) for r in rows) == [(1, 2), (2, 1)]
+ finally:
+ query.stop()
+ finally:
+ for key in confs:
+ session.conf.unset(key)
+
+
def _arrow_table():
return pa.table(
{
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index deabc98..1d6e393 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4639,6 +4639,36 @@ def test_grpc_status_codes():
assert _status_code(99) is StatusCode.UNKNOWN
+def _sent_udf_command(request):
+ """The pickled command of the function in a captured ExecutePlanRequest:
plan=3 ->
+ root=1 -> map_partitions=28 (func=2), group_map=31 (func=3) or
co_group_map=32
+ (func=5) -> python_udf=4 -> command=3."""
+ root = parse_message(parse_message(parse_message(request)[3][0])[1][0])
+ for relation, func in ((28, 2), (31, 3), (32, 5)):
+ if relation in root:
+ udf = parse_message(parse_message(root[relation][0])[func][0])
+ return parse_message(udf[4][0])[3][0]
+ raise AssertionError(f"no map function in the relation fields
{sorted(root)}")
+
+
[email protected]("build", [
+ lambda df: df.mapInPandas(lambda frames: frames, "id long"),
+ lambda df: df.mapInArrow(lambda batches: batches, "id long"),
+ lambda df: df.groupBy("id").applyInPandas(lambda pdf: pdf, "id long"),
+ lambda df: df.groupBy("id").applyInArrow(lambda table: table, "id long"),
+ lambda df: df.groupBy("id").cogroup(df.groupBy("id")).applyInPandas(lambda
a, b: a, "id long"),
+ lambda df: df.groupBy("id").cogroup(df.groupBy("id")).applyInArrow(lambda
a, b: a, "id long"),
+])
+def test_map_functions_pickle_a_ddl_schema_as_its_type(mock_connect, build):
+ import pickle
+
+ spark, service = mock_connect
+ build(spark.range(3)).collect()
+ # The worker reads the pickled type as a DataType, so a DDL string is
pickled parsed.
+ _, return_type =
pickle.loads(_sent_udf_command(service.execute_requests[-1]))
+ assert return_type == T.StructType([T.StructField("id", T.LongType())])
+
+
def _sent_define_flow(request):
"""The DefineFlow in a captured ExecutePlanRequest: plan=3 -> command=2 ->
pipeline_command=19 -> define_flow=3."""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]