This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 9591a2e  [SPARK-61107][CONNECT][PYTHON] Cast the table of toArrow to 
the Arrow schema of the DataFrame in the Python client as PySpark does
9591a2e is described below

commit 9591a2e52faeba71ac9a1e15789b82ff895a49ca
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 19:28:00 2026 +0900

    [SPARK-61107][CONNECT][PYTHON] Cast the table of toArrow to the Arrow 
schema of the DataFrame in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61107](https://issues.apache.org/jira/browse/SPARK-61107): 
`DataFrame.toArrow` now casts the table to the Arrow schema of the DataFrame, 
as PySpark Connect does.
    
    As PySpark Connect's `toArrow`, the table of the result is cast to the 
Arrow schema of the DataFrame's schema, `to_arrow_schema(schema, 
error_on_duplicated_field_names_in_struct=True, timezone="UTC")` (vendored). So:
    - The table has the Arrow types of the Spark types. Strings and binaries 
are `string` and `binary`, even with 
`spark.sql.execution.arrow.useLargeVarTypes`. Timestamps are of UTC, whatever 
the session time zone.
    - The fields of nested structs have the names of the schema.
    - The field metadata of spatial types is PySpark's.
    - A type that PySpark does not convert to Arrow raises 
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`. These are year-month intervals, 
calendar intervals, char and varchar.
    - Duplicate names of nested fields raise 
`DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT`.
    - An array or map of `NullType` that is not nullable raises PyArrow's "A 
null type field may not be non-nullable".
    - A result without batches is the empty table of the schema.
    
    The schema is the one that the server sends with the result 
(`ExecutePlanResponse.schema`), which is that of the analysis of the query. 
PySpark analyzes the query before it runs it, so for an unsupported type it 
raises without running the query; the client raises after. The data from a 
server that sends no schema is returned as it is. The Arrow handoff tests' mock 
service is one such server: it implements no analysis.
    
    ### Why are the changes needed?
    
    The client returned the Arrow data of the server as it was. That data 
differs from PySpark's table:
    - `large_string` and `large_binary` columns with `useLargeVarTypes`
    - timestamps of the session time zone
    - `a_0`, `a_1` for duplicate nested names
    - `month_interval` columns that PyArrow cannot read (`to_pylist()` raises 
`KeyError: 21`)
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `toArrow` returns the table, or raises the errors, of PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test checks:
      - the types of a string and a timestamp, with large var types and a 
session time zone of America/Los_Angeles
      - the errors of year-month and calendar intervals
      - the error of duplicate nested names
      It passes with official PySpark 4.3.0 and fails without this change.
    - A new offline test checks, against a mock Connect server:
      - the cast of `large_string` and `large_binary` data to the schema sent 
with the result
      - that a year-month interval raises
      - that data without a schema is as it is
    - On a local server, 252 cases of `toArrow` and `toPandas` of a value of 
each type, with and without `useLargeVarTypes` and `structHandlingMode`, give 
the same results as official PySpark 4.3.0.
    - The lib tests of both crates (589 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2056 passed, 12 skipped), and the 
live drop-in and Arrow suites (160 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #307 from HyukjinKwon/SPARK-61107-to-arrow-cast.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs  | 44 +++++++++++++++++++++++++++++++------
 python/tests/test_dropin_e2e.py     | 29 ++++++++++++++++++++++++
 python/tests/test_dropin_offline.py | 30 +++++++++++++++++++++++++
 3 files changed, 96 insertions(+), 7 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index b0b96ee..aad41c0 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1986,16 +1986,46 @@ impl PyDataFrame {
 
     /// Collect the DataFrame into a `pyarrow.Table`.
     ///
-    /// Mirrors `pyspark.sql.DataFrame.toArrow()`: the result is decoded from 
the
-    /// server's Arrow IPC stream and handed to `pyarrow` through arrow-rs's
-    /// reader conversion, sharing compatible decoded Arrow value buffers. 
Bitmap alignment
-    /// and schemas requiring the IPC compatibility fallback can still copy.
+    /// Mirrors PySpark Connect's `toArrow()`: the table of the result, cast 
to the Arrow schema of
+    /// its schema, `to_arrow_schema(schema, 
error_on_duplicated_field_names_in_struct=True,
+    /// timezone="UTC")`, which raises for a type that PySpark does not 
convert to Arrow: so of
+    /// the Arrow types of the Spark types, and of the names of the schema for 
the fields of
+    /// nested structs. The schema is the one that the server sends with the 
result, of the
+    /// analysis of the query, which PySpark asks for before it runs the 
query; the data of a
+    /// server that sends none is as it is.
+    ///
+    /// The result is decoded from the server's Arrow IPC stream and handed to 
`pyarrow` through
+    /// arrow-rs's reader conversion, sharing compatible decoded Arrow value 
buffers. Bitmap
+    /// alignment and schemas requiring the IPC compatibility fallback can 
still copy.
     #[pyo3(name = "toArrow")]
     fn to_arrow(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
-        let batches = py
-            .detach(|| self.dataframe.collect_record_batches())
+        let (batches, schema) = py
+            .detach(|| {
+                self.dataframe
+                    .collect_record_batches_with_schema()
+                    .map_err(Box::new)
+            })
             .to_pyerr()?;
-        crate::pyarrow_export::record_batches_to_pyarrow(py, batches)
+        let table = crate::pyarrow_export::record_batches_to_pyarrow(py, 
batches)?;
+        let Some(schema) = schema else {
+            return Ok(table);
+        };
+        let kwargs = PyDict::new(py);
+        kwargs.set_item("error_on_duplicated_field_names_in_struct", true)?;
+        kwargs.set_item("timezone", "UTC")?;
+        let arrow_schema = py
+            .import("pyspark.sql.pandas.types")?
+            .getattr("to_arrow_schema")?
+            .call(
+                (crate::types::data_type_to_py(py, &schema)?,),
+                Some(&kwargs),
+            )?;
+        let table = table.bind(py);
+        // A result without batches has no columns: the empty table of the 
schema.
+        if table.getattr("num_columns")?.extract::<usize>()? == 0 && 
arrow_schema.len()? > 0 {
+            return Ok(arrow_schema.call_method0("empty_table")?.unbind());
+        }
+        Ok(table.call_method1("cast", (arrow_schema,))?.unbind())
     }
 
     /// Collect the DataFrame into a pandas DataFrame.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 060e0cc..1123a2c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2056,6 +2056,35 @@ def 
test_column_alias_names_every_column_and_keeps_its_metadata(session):
     assert df.schema.fields[0].metadata == {"k": 1, "n": [1, 2], "s": "v"}
 
 
+def test_to_arrow_casts_to_the_arrow_schema_of_the_schema_as_pyspark(session):
+    # As PySpark Connect's: the Arrow types of the Spark types, of UTC 
timestamps and of strings
+    # even of large var types, and the errors of the types that PySpark does 
not convert to
+    # Arrow and of duplicate names of nested fields.
+    import pyarrow as pa
+    from pyspark.errors import PySparkTypeError, UnsupportedOperationException
+
+    confs = {"spark.sql.execution.arrow.useLargeVarTypes": "true",
+             "spark.sql.session.timeZone": "America/Los_Angeles"}
+    previous = {key: session.conf.get(key) for key in confs}
+    for key, value in confs.items():
+        session.conf.set(key, value)
+    try:
+        table = session.sql("SELECT 'x' AS s, TIMESTAMP'2020-01-01 00:00:00' 
AS t").toArrow()
+    finally:
+        for key, value in previous.items():
+            session.conf.set(key, value)
+    assert table.schema.field("s").type == pa.string()
+    assert table.schema.field("t").type == pa.timestamp("us", tz="UTC")
+    for expr, data_type in [("INTERVAL '1-2' YEAR TO MONTH", 
"YearMonthIntervalType(0, 1)"),
+                            ("make_interval(1)", "CalendarIntervalType()")]:
+        with pytest.raises(PySparkTypeError) as e:
+            session.sql(f"SELECT {expr} AS v").toArrow()
+        assert e.value.getCondition() == 
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+        assert e.value.getMessageParameters() == {"data_type": data_type}
+    with pytest.raises(UnsupportedOperationException) as e:
+        session.sql("SELECT named_struct('a', 1, 'a', 2) AS v").toArrow()
+    assert e.value.getCondition() == "DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT"
+
 def test_to_pandas_converts_the_columns_as_pyspark(session):
     # As official PySpark 4.3.0's SparkConnectClient.to_pandas: each column by 
its Spark type,
     # in the session time zone, with the metrics of the execution in attrs.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 70dad79..9de9f77 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7952,6 +7952,36 @@ def test_arrow_input_roundtrip_live_server():
         spark.stop()
 
 
+def 
test_to_arrow_casts_to_the_arrow_schema_of_the_schema_as_pyspark(mock_connect):
+    # As PySpark Connect's: toArrow casts the table of the result to the Arrow 
schema of its
+    # schema, of string and binary even of large var types, which raises for a 
type that
+    # PySpark does not convert to Arrow. The data of a server that sends no 
schema is as it is.
+    import pyarrow as pa
+    from pyspark.errors import PySparkTypeError
+
+    spark, service = mock_connect
+    table = pa.table({"s": pa.array(["x"], pa.large_string()),
+                      "b": pa.array([b"y"], pa.large_binary())})
+    sink = pa.BufferOutputStream()
+    with pa.ipc.new_stream(sink, table.schema) as writer:
+        writer.write_table(table)
+    # ExecutePlanResponse.arrow_batch=2: ArrowBatch.row_count=1, data=2.
+    batch = message_field(2, varint_field(1, 1) + message_field(2, 
sink.getvalue().to_pybytes()))
+
+    service.execute_payload = batch
+    assert spark.range(1).toArrow().schema == table.schema
+    # ExecutePlanResponse.schema=7.
+    service.execute_payload = message_field(7, ddl_data_type("s string, b 
binary")) + batch
+    result = spark.range(1).toArrow()
+    assert result.schema == pa.schema([("s", pa.string()), ("b", pa.binary())])
+    assert result.to_pylist() == [{"s": "x", "b": b"y"}]
+    service.execute_payload = message_field(7, ddl_data_type("v interval year 
to month")) + batch
+    with pytest.raises(PySparkTypeError) as e:
+        spark.range(1).toArrow()
+    assert e.value.getCondition() == 
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+    assert e.value.getMessageParameters() == {"data_type": 
"YearMonthIntervalType(0, 1)"}
+
+
 @pytest.mark.parametrize("kind", ["slice", "dictionary", "decimal", "null", 
"struct", "nul"])
 def test_arrow_input_types_live_server(kind):
     import os


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to