This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new d30024d [SPARK-61113][CONNECT][PYTHON] Add
DataFrame.__arrow_c_stream__, the Arrow C stream of a DataFrame, in the Python
client as PySpark does
d30024d is described below
commit d30024d97d79fee3b54b992a0379c4ddc7f4caef
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 21:31:56 2026 +0900
[SPARK-61113][CONNECT][PYTHON] Add DataFrame.__arrow_c_stream__, the Arrow
C stream of a DataFrame, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61113](https://issues.apache.org/jira/browse/SPARK-61113):
`DataFrame.__arrow_c_stream__` is added, as PySpark's.
PySpark's DataFrame implements the stream of the Arrow PyCapsule interface.
`__arrow_c_stream__(requested_schema=None)` returns an Arrow C stream of the
rows of the DataFrame, of the Arrow schema of its schema.
`pyspark.sql.interchange.SparkArrowCStreamer` reads one partition at a time
through `mapInArrow` and `toLocalIterator`, so the client never holds all the
rows at once. A requested schema is passed to the stream, which casts to it.
`pyspark.sql.interchange` is vendored byte for byte from Apache Spark
v4.3.0 (and added to the drift check). `DataFrame.__arrow_c_stream__` returns
its stream.
### Why are the changes needed?
The client had neither `DataFrame.__arrow_c_stream__` nor
`pyspark.sql.interchange`. Libraries that read Arrow streams could not take a
Spark DataFrame: `pa.RecordBatchReader.from_stream(df)` raised `TypeError:
Expected an object implementing the Arrow PyCapsule Protocol for streams`.
### Does this PR introduce _any_ user-facing change?
Yes. `pa.RecordBatchReader.from_stream(df)`, Polars and DuckDB read a
DataFrame directly, as with PySpark 4.3.0.
### How was this patch tested?
- A new live test reads the stream of a DataFrame of three partitions, with
nested columns, through `pa.RecordBatchReader.from_stream`. It checks:
- the schema, which is that of `toArrow`
- one batch per partition
- the rows
- the stream of a requested schema, cast to it
- the stream of an empty DataFrame
It passes with official PySpark 4.3.0 and fails without this change.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2105 passed, 8 skipped), and the
live drop-in and Arrow suites (165 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #310 from HyukjinKwon/SPARK-61113-arrow-c-stream.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 18 ++++++++
python/.coveragerc | 1 +
python/pyspark/sql/interchange.py | 89 ++++++++++++++++++++++++++++++++++++++
python/tests/test_arrow_e2e.py | 25 +++++++++++
scripts/check_vendored_upstream.sh | 3 ++
5 files changed, 136 insertions(+)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index c7f771c..e7178b9 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2074,6 +2074,24 @@ impl PyDataFrame {
Ok(table.call_method1("cast", (arrow_schema,))?.unbind())
}
+ /// The Arrow C stream of the DataFrame, of the Arrow PyCapsule interface,
as PySpark's
+ /// `__arrow_c_stream__`: `pyspark.sql.interchange.SparkArrowCStreamer`,
which reads the rows of
+ /// one partition at a time, of the Arrow schema of the schema, and
`requested_schema`, a
+ /// PyCapsule of a schema, if any, which the stream is cast to as far as
Arrow can.
+ #[pyo3(signature = (requested_schema=None))]
+ fn __arrow_c_stream__(
+ slf: &Bound<'_, Self>,
+ requested_schema: Option<&Bound<'_, PyAny>>,
+ ) -> PyResult<Py<PyAny>> {
+ let py = slf.py();
+ Ok(py
+ .import("pyspark.sql.interchange")?
+ .getattr("SparkArrowCStreamer")?
+ .call1((slf,))?
+ .call_method1("__arrow_c_stream__", (requested_schema,))?
+ .unbind())
+ }
+
/// Collect the DataFrame into a pandas DataFrame.
///
/// Mirrors `pyspark.sql.DataFrame.toPandas()`, as
`SparkConnectClient.to_pandas` converts
diff --git a/python/.coveragerc b/python/.coveragerc
index a64a46c..9cb8189 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -43,6 +43,7 @@ omit =
*/pyspark/sql/datasource_internal.py
*/pyspark/sql/profiler.py
*/pyspark/sql/metrics.py
+ */pyspark/sql/interchange.py
*/pyspark/serializers.py
*/pyspark/loose_version.py
*/pyspark/_globals.py
diff --git a/python/pyspark/sql/interchange.py
b/python/pyspark/sql/interchange.py
new file mode 100644
index 0000000..141d9f3
--- /dev/null
+++ b/python/pyspark/sql/interchange.py
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import Iterator, Optional
+
+import pyarrow as pa
+
+import pyspark.sql
+from pyspark.sql.types import StructType, StructField, BinaryType
+from pyspark.sql.pandas.types import to_arrow_schema
+
+
+def _get_arrow_array_partition_stream(df: pyspark.sql.DataFrame) ->
Iterator[pa.RecordBatch]:
+ """Return all the partitions as Arrow arrays in an Iterator."""
+ # We will be using mapInArrow to convert each partition to Arrow
RecordBatches.
+ # The return type of the function will be a single binary column containing
+ # the serialized RecordBatch in Arrow IPC format.
+ binary_schema = StructType([StructField("arrow_ipc_bytes", BinaryType(),
nullable=False)])
+
+ def batch_to_bytes_iter(batch_iter: Iterator[pa.RecordBatch]) ->
Iterator[pa.RecordBatch]:
+ """
+ A generator function that converts RecordBatches to serialized Arrow
IPC format.
+
+ Spark sends each partition as an iterator of RecordBatches. In order
to return
+ the entire partition as a stream of Arrow RecordBatches, we need to
serialize
+ each RecordBatch to Arrow IPC format and yield it as a single binary
blob.
+ """
+ # The size of the batch can be controlled by the Spark config
+ # `spark.sql.execution.arrow.maxRecordsPerBatch`.
+ for arrow_batch in batch_iter:
+ # We create an in-memory byte stream to hold the serialized batch
+ sink = pa.BufferOutputStream()
+ # Write the batch to the stream using Arrow IPC format
+ with pa.ipc.new_stream(sink, arrow_batch.schema) as writer:
+ writer.write_batch(arrow_batch)
+ buf = sink.getvalue()
+ # The second buffer contains the offsets we are manually creating.
+ offset_buf = pa.array([0, len(buf)], type=pa.int32()).buffers()[1]
+ null_bitmap = None
+ # Wrap the bytes in a new 1-row, 1-column RecordBatch to satisfy
mapInArrow return
+ # signature. This serializes the whole batch into a single pyarrow
serialized cell.
+ storage_arr = pa.Array.from_buffers(
+ type=pa.binary(), length=1, buffers=[null_bitmap, offset_buf,
buf]
+ )
+ yield pa.RecordBatch.from_arrays([storage_arr],
names=["arrow_ipc_bytes"])
+
+ # Convert all partitions to Arrow RecordBatches and map to binary blobs.
+ byte_df = df.mapInArrow(batch_to_bytes_iter, binary_schema)
+ # A row is actually a batch of data in Arrow IPC format. Fetch the batches
one by one.
+ for row in byte_df.toLocalIterator():
+ with pa.ipc.open_stream(row.arrow_ipc_bytes) as reader:
+ for batch in reader:
+ # Each batch corresponds to a chunk of data in the partition.
+ yield batch
+
+
+class SparkArrowCStreamer:
+ """
+ A class that implements that __arrow_c_stream__ protocol for Spark
partitions.
+
+ This class is implemented in a way that allows consumers to consume each
partition
+ one at a time without materializing all partitions at once on the driver
side.
+ """
+
+ def __init__(self, df: pyspark.sql.DataFrame):
+ self._df = df
+ self._schema = to_arrow_schema(df.schema)
+
+ def __arrow_c_stream__(self, requested_schema: Optional[object] = None) ->
object:
+ """
+ Return the Arrow C stream for the dataframe partitions.
+ """
+ reader: pa.RecordBatchReader = pa.RecordBatchReader.from_batches(
+ self._schema, _get_arrow_array_partition_stream(self._df)
+ )
+ return reader.__arrow_c_stream__(requested_schema=requested_schema)
diff --git a/python/tests/test_arrow_e2e.py b/python/tests/test_arrow_e2e.py
index eeec709..ffa1e70 100644
--- a/python/tests/test_arrow_e2e.py
+++ b/python/tests/test_arrow_e2e.py
@@ -67,6 +67,31 @@ def test_to_arrow_empty_result(session):
assert table.num_rows == 0
+def test_arrow_c_stream_reads_the_partitions_as_pyspark(session):
+ # As official PySpark 4.3.0's DataFrame.__arrow_c_stream__: an Arrow C
stream of the rows,
+ # of the Arrow schema of the DataFrame, of a batch of each partition, which
+ # pa.RecordBatchReader.from_stream reads.
+ df = session.range(0, 6, 1, 3).selectExpr(
+ "id", "CAST(id AS STRING) AS s", "array(id, id + 1) AS a",
"named_struct('x', id) AS n"
+ )
+ expected = df.toArrow()
+ reader = pa.RecordBatchReader.from_stream(df)
+ assert reader.schema == expected.schema
+ batches = list(reader)
+ assert len(batches) == 3
+ assert pa.Table.from_batches(batches, expected.schema).to_pydict() ==
expected.to_pydict()
+ assert type(df.__arrow_c_stream__()).__name__ == "PyCapsule"
+
+ # The stream of a requested schema is cast to it.
+ requested = pa.schema([("id", pa.int32()), ("s", pa.large_string())])
+ table = pa.RecordBatchReader.from_stream(df.select("id", "s"),
schema=requested).read_all()
+ assert table.schema == requested
+ assert table.to_pydict() == {"id": list(range(6)), "s": [str(i) for i in
range(6)]}
+
+ empty = pa.RecordBatchReader.from_stream(session.range(0)).read_all()
+ assert empty.num_rows == 0 and empty.schema == pa.schema([pa.field("id",
pa.int64(), False)])
+
+
def test_create_dataframe_arrow_cached_upload(session):
"""Materialize cached Arrow input larger than 1 MiB on a real server."""
import numpy as np
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index 87a6af4..ecef653 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -58,6 +58,9 @@ VENDORED_FILES=(
"conf.py"
"taskcontext.py"
+ # The Arrow C stream of a DataFrame, of DataFrame.__arrow_c_stream__.
+ "sql/interchange.py"
+
# Logging
"logger/__init__.py"
"logger/logger.py"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]