This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 9591a2e [SPARK-61107][CONNECT][PYTHON] Cast the table of toArrow to
the Arrow schema of the DataFrame in the Python client as PySpark does
9591a2e is described below
commit 9591a2e52faeba71ac9a1e15789b82ff895a49ca
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 19:28:00 2026 +0900
[SPARK-61107][CONNECT][PYTHON] Cast the table of toArrow to the Arrow
schema of the DataFrame in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61107](https://issues.apache.org/jira/browse/SPARK-61107):
`DataFrame.toArrow` now casts the table to the Arrow schema of the DataFrame,
as PySpark Connect does.
As PySpark Connect's `toArrow`, the table of the result is cast to the
Arrow schema of the DataFrame's schema, `to_arrow_schema(schema,
error_on_duplicated_field_names_in_struct=True, timezone="UTC")` (vendored). So:
- The table has the Arrow types of the Spark types. Strings and binaries
are `string` and `binary`, even with
`spark.sql.execution.arrow.useLargeVarTypes`. Timestamps are of UTC, whatever
the session time zone.
- The fields of nested structs have the names of the schema.
- The field metadata of spatial types is PySpark's.
- A type that PySpark does not convert to Arrow raises
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`. These are year-month intervals,
calendar intervals, char and varchar.
- Duplicate names of nested fields raise
`DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT`.
- An array or map of `NullType` that is not nullable raises PyArrow's "A
null type field may not be non-nullable".
- A result without batches is the empty table of the schema.
The schema is the one that the server sends with the result
(`ExecutePlanResponse.schema`), which is that of the analysis of the query.
PySpark analyzes the query before it runs it, so for an unsupported type it
raises without running the query; the client raises after. The data from a
server that sends no schema is returned as it is. The Arrow handoff tests' mock
service is one such server: it implements no analysis.
### Why are the changes needed?
The client returned the Arrow data of the server as it was. That data
differs from PySpark's table:
- `large_string` and `large_binary` columns with `useLargeVarTypes`
- timestamps of the session time zone
- `a_0`, `a_1` for duplicate nested names
- `month_interval` columns that PyArrow cannot read (`to_pylist()` raises
`KeyError: 21`)
### Does this PR introduce _any_ user-facing change?
Yes. `toArrow` returns the table, or raises the errors, of PySpark 4.3.0.
### How was this patch tested?
- A new live test checks:
- the types of a string and a timestamp, with large var types and a
session time zone of America/Los_Angeles
- the errors of year-month and calendar intervals
- the error of duplicate nested names
It passes with official PySpark 4.3.0 and fails without this change.
- A new offline test checks, against a mock Connect server:
- the cast of `large_string` and `large_binary` data to the schema sent
with the result
- that a year-month interval raises
- that data without a schema is as it is
- On a local server, 252 cases of `toArrow` and `toPandas` of a value of
each type, with and without `useLargeVarTypes` and `structHandlingMode`, give
the same results as official PySpark 4.3.0.
- The lib tests of both crates (589 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2056 passed, 12 skipped), and the
live drop-in and Arrow suites (160 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #307 from HyukjinKwon/SPARK-61107-to-arrow-cast.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 44 +++++++++++++++++++++++++++++++------
python/tests/test_dropin_e2e.py | 29 ++++++++++++++++++++++++
python/tests/test_dropin_offline.py | 30 +++++++++++++++++++++++++
3 files changed, 96 insertions(+), 7 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index b0b96ee..aad41c0 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1986,16 +1986,46 @@ impl PyDataFrame {
/// Collect the DataFrame into a `pyarrow.Table`.
///
- /// Mirrors `pyspark.sql.DataFrame.toArrow()`: the result is decoded from
the
- /// server's Arrow IPC stream and handed to `pyarrow` through arrow-rs's
- /// reader conversion, sharing compatible decoded Arrow value buffers.
Bitmap alignment
- /// and schemas requiring the IPC compatibility fallback can still copy.
+ /// Mirrors PySpark Connect's `toArrow()`: the table of the result, cast
to the Arrow schema of
+ /// its schema, `to_arrow_schema(schema,
error_on_duplicated_field_names_in_struct=True,
+ /// timezone="UTC")`, which raises for a type that PySpark does not
convert to Arrow: so of
+ /// the Arrow types of the Spark types, and of the names of the schema for
the fields of
+ /// nested structs. The schema is the one that the server sends with the
result, of the
+ /// analysis of the query, which PySpark asks for before it runs the
query; the data of a
+ /// server that sends none is as it is.
+ ///
+ /// The result is decoded from the server's Arrow IPC stream and handed to
`pyarrow` through
+ /// arrow-rs's reader conversion, sharing compatible decoded Arrow value
buffers. Bitmap
+ /// alignment and schemas requiring the IPC compatibility fallback can
still copy.
#[pyo3(name = "toArrow")]
fn to_arrow(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
- let batches = py
- .detach(|| self.dataframe.collect_record_batches())
+ let (batches, schema) = py
+ .detach(|| {
+ self.dataframe
+ .collect_record_batches_with_schema()
+ .map_err(Box::new)
+ })
.to_pyerr()?;
- crate::pyarrow_export::record_batches_to_pyarrow(py, batches)
+ let table = crate::pyarrow_export::record_batches_to_pyarrow(py,
batches)?;
+ let Some(schema) = schema else {
+ return Ok(table);
+ };
+ let kwargs = PyDict::new(py);
+ kwargs.set_item("error_on_duplicated_field_names_in_struct", true)?;
+ kwargs.set_item("timezone", "UTC")?;
+ let arrow_schema = py
+ .import("pyspark.sql.pandas.types")?
+ .getattr("to_arrow_schema")?
+ .call(
+ (crate::types::data_type_to_py(py, &schema)?,),
+ Some(&kwargs),
+ )?;
+ let table = table.bind(py);
+ // A result without batches has no columns: the empty table of the
schema.
+ if table.getattr("num_columns")?.extract::<usize>()? == 0 &&
arrow_schema.len()? > 0 {
+ return Ok(arrow_schema.call_method0("empty_table")?.unbind());
+ }
+ Ok(table.call_method1("cast", (arrow_schema,))?.unbind())
}
/// Collect the DataFrame into a pandas DataFrame.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 060e0cc..1123a2c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2056,6 +2056,35 @@ def
test_column_alias_names_every_column_and_keeps_its_metadata(session):
assert df.schema.fields[0].metadata == {"k": 1, "n": [1, 2], "s": "v"}
+def test_to_arrow_casts_to_the_arrow_schema_of_the_schema_as_pyspark(session):
+ # As PySpark Connect's: the Arrow types of the Spark types, of UTC
timestamps and of strings
+ # even of large var types, and the errors of the types that PySpark does
not convert to
+ # Arrow and of duplicate names of nested fields.
+ import pyarrow as pa
+ from pyspark.errors import PySparkTypeError, UnsupportedOperationException
+
+ confs = {"spark.sql.execution.arrow.useLargeVarTypes": "true",
+ "spark.sql.session.timeZone": "America/Los_Angeles"}
+ previous = {key: session.conf.get(key) for key in confs}
+ for key, value in confs.items():
+ session.conf.set(key, value)
+ try:
+ table = session.sql("SELECT 'x' AS s, TIMESTAMP'2020-01-01 00:00:00'
AS t").toArrow()
+ finally:
+ for key, value in previous.items():
+ session.conf.set(key, value)
+ assert table.schema.field("s").type == pa.string()
+ assert table.schema.field("t").type == pa.timestamp("us", tz="UTC")
+ for expr, data_type in [("INTERVAL '1-2' YEAR TO MONTH",
"YearMonthIntervalType(0, 1)"),
+ ("make_interval(1)", "CalendarIntervalType()")]:
+ with pytest.raises(PySparkTypeError) as e:
+ session.sql(f"SELECT {expr} AS v").toArrow()
+ assert e.value.getCondition() ==
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+ assert e.value.getMessageParameters() == {"data_type": data_type}
+ with pytest.raises(UnsupportedOperationException) as e:
+ session.sql("SELECT named_struct('a', 1, 'a', 2) AS v").toArrow()
+ assert e.value.getCondition() == "DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT"
+
def test_to_pandas_converts_the_columns_as_pyspark(session):
# As official PySpark 4.3.0's SparkConnectClient.to_pandas: each column by
its Spark type,
# in the session time zone, with the metrics of the execution in attrs.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 70dad79..9de9f77 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7952,6 +7952,36 @@ def test_arrow_input_roundtrip_live_server():
spark.stop()
+def
test_to_arrow_casts_to_the_arrow_schema_of_the_schema_as_pyspark(mock_connect):
+ # As PySpark Connect's: toArrow casts the table of the result to the Arrow
schema of its
+ # schema, of string and binary even of large var types, which raises for a
type that
+ # PySpark does not convert to Arrow. The data of a server that sends no
schema is as it is.
+ import pyarrow as pa
+ from pyspark.errors import PySparkTypeError
+
+ spark, service = mock_connect
+ table = pa.table({"s": pa.array(["x"], pa.large_string()),
+ "b": pa.array([b"y"], pa.large_binary())})
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, table.schema) as writer:
+ writer.write_table(table)
+ # ExecutePlanResponse.arrow_batch=2: ArrowBatch.row_count=1, data=2.
+ batch = message_field(2, varint_field(1, 1) + message_field(2,
sink.getvalue().to_pybytes()))
+
+ service.execute_payload = batch
+ assert spark.range(1).toArrow().schema == table.schema
+ # ExecutePlanResponse.schema=7.
+ service.execute_payload = message_field(7, ddl_data_type("s string, b
binary")) + batch
+ result = spark.range(1).toArrow()
+ assert result.schema == pa.schema([("s", pa.string()), ("b", pa.binary())])
+ assert result.to_pylist() == [{"s": "x", "b": b"y"}]
+ service.execute_payload = message_field(7, ddl_data_type("v interval year
to month")) + batch
+ with pytest.raises(PySparkTypeError) as e:
+ spark.range(1).toArrow()
+ assert e.value.getCondition() ==
"UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION"
+ assert e.value.getMessageParameters() == {"data_type":
"YearMonthIntervalType(0, 1)"}
+
+
@pytest.mark.parametrize("kind", ["slice", "dictionary", "decimal", "null",
"struct", "nul"])
def test_arrow_input_types_live_server(kind):
import os
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]