This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 0b40b84  [SPARK-60974][CONNECT][PYTHON] Accept a DDL string schema in 
mapInPandas, applyInPandas and the other map functions as PySpark does
0b40b84 is described below

commit 0b40b840a02ff0b3250e5b316d19db19c8e3361b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:08:52 2026 +0900

    [SPARK-60974][CONNECT][PYTHON] Accept a DDL string schema in mapInPandas, 
applyInPandas and the other map functions as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60974](https://issues.apache.org/jira/browse/SPARK-60974): a DDL 
string schema of the map functions is now pickled with the function as the type 
it parses to, instead of as the string. PySpark does the same: it pickles the 
type of `session._parse_ddl`.
    
    This covers:
    
    - `mapInPandas` and `mapInArrow`;
    - the grouped and cogrouped `applyInPandas` and `applyInArrow`;
    - the output type of `applyInPandasWithState`, `transformWithStateInPandas` 
and `transformWithState`.
    
    These all build their function in one helper, `build_map_udf`, which now 
pickles the parsed type when the schema is a string. A `DataType` schema is 
pickled as before. The schema in the plan was already parsed.
    
    ### Why are the changes needed?
    
    The Python worker reads the pickled type as a `DataType`, so a DDL string 
broke these functions:
    
    ```python
    df.groupBy("k").applyInPandas(lambda pdf: pdf, "k long, v double").collect()
    ```
    
    This raised `PythonException` with `AssertionError: dataType k long, v 
double should be an instance of <class 'pyspark.sql.types.DataType'>`, where 
PySpark 4.3.0 returns the rows.
    
    - With a DDL string, five of the six batch functions failed. `applyInArrow` 
raised `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`, and `mapInArrow` worked.
    - A streaming `transformWithStateInPandas` failed the same way.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these functions take a DDL string schema as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, the six batch functions with a DDL 
string schema return the same rows as official PySpark 4.3.0, and before this 
change five of them failed. A streaming `transformWithStateInPandas` with a DDL 
output type also gives the same rows as official PySpark, and failed before.
    - A new offline test unpickles the command that each of the six batch 
functions sends, and checks its type.
    - New live tests check the six batch functions, and a 
`transformWithStateInPandas` query with a DDL output type. They also pass with 
official PySpark 4.3.0, and on a server started with 
`SPARK_SKIP_CONNECT_COMPAT_TESTS`, as CI's is.
    - The lib tests and every integration test, e2e included, against the local 
server (643 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (619 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (59 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    `applyInPandasWithState` takes the same path, but the server rejects its 
plan for another reason, which 
[SPARK-60976](https://issues.apache.org/jira/browse/SPARK-60976) fixes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #180 from HyukjinKwon/SPARK-60974-map-ddl.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs  |  9 +++++-
 python/tests/test_dropin_e2e.py     | 61 +++++++++++++++++++++++++++++++++++++
 python/tests/test_dropin_offline.py | 30 ++++++++++++++++++
 3 files changed, 99 insertions(+), 1 deletion(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index f491d23..faa8b58 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2464,7 +2464,14 @@ pub(crate) fn build_map_udf(
     input_cols: &[String],
 ) -> PyResult<CommonInlineUserDefinedFunctionExpression> {
     let return_type = resolve_datatype(schema)?;
-    let tup = PyTuple::new(py, [func.clone(), schema.clone()])?;
+    // The worker reads the pickled type as a `DataType`, so a DDL string is 
pickled as the
+    // type it parses to, as the reference client pickles the type of 
`_parse_ddl`.
+    let pickled_type = if schema.is_instance_of::<pyo3::types::PyString>() {
+        crate::types::data_type_to_py(py, &return_type)?.into_bound(py)
+    } else {
+        schema.clone()
+    };
+    let tup = PyTuple::new(py, [func.clone(), pickled_type])?;
     let command = py_cloudpickle(py, tup.as_any())?;
     let payload = PythonUDFPayload::new(return_type, eval_type_id, command, 
py_version(py));
     // The worker reconstructs the input pandas/Arrow DataFrame from the UDF's 
argument
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 15c5714..20e3131 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -107,6 +107,67 @@ def 
test_map_in_arrow_and_pandas_with_a_struct_type_built_with_add(session, sche
     assert [tuple(r) for r in rows] == [(0, 0), (1, 2), (2, 4)]
 
 
+def test_map_functions_take_a_ddl_schema(session):
+    df = session.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0)], ["k", "v"])
+    schema = "k long, v double"
+    grouped = df.groupBy("k")
+    for result in (
+        df.mapInPandas(lambda frames: frames, schema),
+        df.mapInArrow(lambda batches: batches, schema),
+        grouped.applyInPandas(lambda pdf: pdf, schema),
+        grouped.applyInArrow(lambda table: table, schema),
+        grouped.cogroup(grouped).applyInPandas(lambda left, right: left, 
schema),
+        grouped.cogroup(grouped).applyInArrow(lambda left, right: left, 
schema),
+    ):
+        assert sorted(tuple(r) for r in result.collect()) == [(1, 1.0), (1, 
2.0), (2, 3.0)]
+
+
+def test_transform_with_state_in_pandas_takes_a_ddl_output_type(session, 
tmp_path):
+    import pandas as pd
+    from pyspark.sql.streaming.stateful_processor import StatefulProcessor
+
+    class Counter(StatefulProcessor):
+        def init(self, handle):
+            self.count = handle.getValueState("count", "c long")
+
+        def handleInputRows(self, key, rows, timerValues):
+            total = sum(len(pdf) for pdf in rows)
+            self.count.update((total,))
+            yield pd.DataFrame({"k": [key[0]], "c": [total]})
+
+        def close(self):
+            pass
+
+    confs = {
+        "spark.sql.streaming.stateStore.providerClass":
+            
"org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider",
+        # A state store per shuffle partition; the default 200 take half a 
minute.
+        "spark.sql.shuffle.partitions": "2",
+    }
+    for key, value in confs.items():
+        session.conf.set(key, value)
+    try:
+        source = str(tmp_path / "source")
+        session.createDataFrame([(1,), (1,), (2,)], "k 
long").write.parquet(source)
+        query = (
+            session.readStream.schema("k long").parquet(source).groupBy("k")
+            .transformWithStateInPandas(Counter(), "k long, c long", "Update", 
"None")
+            
.writeStream.format("memory").queryName("tws_ddl_output").outputMode("update")
+            .option("checkpointLocation", str(tmp_path / "checkpoint"))
+            .trigger(availableNow=True).start()
+        )
+        try:
+            query.awaitTermination(120)
+            assert query.exception() is None
+            rows = session.table("tws_ddl_output").collect()
+            assert sorted(tuple(r) for r in rows) == [(1, 2), (2, 1)]
+        finally:
+            query.stop()
+    finally:
+        for key in confs:
+            session.conf.unset(key)
+
+
 def _arrow_table():
     return pa.table(
         {
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index deabc98..1d6e393 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4639,6 +4639,36 @@ def test_grpc_status_codes():
     assert _status_code(99) is StatusCode.UNKNOWN
 
 
+def _sent_udf_command(request):
+    """The pickled command of the function in a captured ExecutePlanRequest: 
plan=3 ->
+    root=1 -> map_partitions=28 (func=2), group_map=31 (func=3) or 
co_group_map=32
+    (func=5) -> python_udf=4 -> command=3."""
+    root = parse_message(parse_message(parse_message(request)[3][0])[1][0])
+    for relation, func in ((28, 2), (31, 3), (32, 5)):
+        if relation in root:
+            udf = parse_message(parse_message(root[relation][0])[func][0])
+            return parse_message(udf[4][0])[3][0]
+    raise AssertionError(f"no map function in the relation fields 
{sorted(root)}")
+
+
[email protected]("build", [
+    lambda df: df.mapInPandas(lambda frames: frames, "id long"),
+    lambda df: df.mapInArrow(lambda batches: batches, "id long"),
+    lambda df: df.groupBy("id").applyInPandas(lambda pdf: pdf, "id long"),
+    lambda df: df.groupBy("id").applyInArrow(lambda table: table, "id long"),
+    lambda df: df.groupBy("id").cogroup(df.groupBy("id")).applyInPandas(lambda 
a, b: a, "id long"),
+    lambda df: df.groupBy("id").cogroup(df.groupBy("id")).applyInArrow(lambda 
a, b: a, "id long"),
+])
+def test_map_functions_pickle_a_ddl_schema_as_its_type(mock_connect, build):
+    import pickle
+
+    spark, service = mock_connect
+    build(spark.range(3)).collect()
+    # The worker reads the pickled type as a DataType, so a DDL string is 
pickled parsed.
+    _, return_type = 
pickle.loads(_sent_udf_command(service.execute_requests[-1]))
+    assert return_type == T.StructType([T.StructField("id", T.LongType())])
+
+
 def _sent_define_flow(request):
     """The DefineFlow in a captured ExecutePlanRequest: plan=3 -> command=2 ->
     pipeline_command=19 -> define_flow=3."""


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to