This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new d30024d  [SPARK-61113][CONNECT][PYTHON] Add 
DataFrame.__arrow_c_stream__, the Arrow C stream of a DataFrame, in the Python 
client as PySpark does
d30024d is described below

commit d30024d97d79fee3b54b992a0379c4ddc7f4caef
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 21:31:56 2026 +0900

    [SPARK-61113][CONNECT][PYTHON] Add DataFrame.__arrow_c_stream__, the Arrow 
C stream of a DataFrame, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61113](https://issues.apache.org/jira/browse/SPARK-61113): 
`DataFrame.__arrow_c_stream__` is added, as PySpark's.
    
    PySpark's DataFrame implements the stream of the Arrow PyCapsule interface. 
`__arrow_c_stream__(requested_schema=None)` returns an Arrow C stream of the 
rows of the DataFrame, of the Arrow schema of its schema. 
`pyspark.sql.interchange.SparkArrowCStreamer` reads one partition at a time 
through `mapInArrow` and `toLocalIterator`, so the client never holds all the 
rows at once. A requested schema is passed to the stream, which casts to it.
    
    `pyspark.sql.interchange` is vendored byte for byte from Apache Spark 
v4.3.0 (and added to the drift check). `DataFrame.__arrow_c_stream__` returns 
its stream.
    
    ### Why are the changes needed?
    
    The client had neither `DataFrame.__arrow_c_stream__` nor 
`pyspark.sql.interchange`. Libraries that read Arrow streams could not take a 
Spark DataFrame: `pa.RecordBatchReader.from_stream(df)` raised `TypeError: 
Expected an object implementing the Arrow PyCapsule Protocol for streams`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `pa.RecordBatchReader.from_stream(df)`, Polars and DuckDB read a 
DataFrame directly, as with PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test reads the stream of a DataFrame of three partitions, with 
nested columns, through `pa.RecordBatchReader.from_stream`. It checks:
      - the schema, which is that of `toArrow`
      - one batch per partition
      - the rows
      - the stream of a requested schema, cast to it
      - the stream of an empty DataFrame
    
      It passes with official PySpark 4.3.0 and fails without this change.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2105 passed, 8 skipped), and the 
live drop-in and Arrow suites (165 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #310 from HyukjinKwon/SPARK-61113-arrow-c-stream.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs | 18 ++++++++
 python/.coveragerc                 |  1 +
 python/pyspark/sql/interchange.py  | 89 ++++++++++++++++++++++++++++++++++++++
 python/tests/test_arrow_e2e.py     | 25 +++++++++++
 scripts/check_vendored_upstream.sh |  3 ++
 5 files changed, 136 insertions(+)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index c7f771c..e7178b9 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2074,6 +2074,24 @@ impl PyDataFrame {
         Ok(table.call_method1("cast", (arrow_schema,))?.unbind())
     }
 
+    /// The Arrow C stream of the DataFrame, of the Arrow PyCapsule interface, 
as PySpark's
+    /// `__arrow_c_stream__`: `pyspark.sql.interchange.SparkArrowCStreamer`, 
which reads the rows of
+    /// one partition at a time, of the Arrow schema of the schema, and 
`requested_schema`, a
+    /// PyCapsule of a schema, if any, which the stream is cast to as far as 
Arrow can.
+    #[pyo3(signature = (requested_schema=None))]
+    fn __arrow_c_stream__(
+        slf: &Bound<'_, Self>,
+        requested_schema: Option<&Bound<'_, PyAny>>,
+    ) -> PyResult<Py<PyAny>> {
+        let py = slf.py();
+        Ok(py
+            .import("pyspark.sql.interchange")?
+            .getattr("SparkArrowCStreamer")?
+            .call1((slf,))?
+            .call_method1("__arrow_c_stream__", (requested_schema,))?
+            .unbind())
+    }
+
     /// Collect the DataFrame into a pandas DataFrame.
     ///
     /// Mirrors `pyspark.sql.DataFrame.toPandas()`, as 
`SparkConnectClient.to_pandas` converts
diff --git a/python/.coveragerc b/python/.coveragerc
index a64a46c..9cb8189 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -43,6 +43,7 @@ omit =
     */pyspark/sql/datasource_internal.py
     */pyspark/sql/profiler.py
     */pyspark/sql/metrics.py
+    */pyspark/sql/interchange.py
     */pyspark/serializers.py
     */pyspark/loose_version.py
     */pyspark/_globals.py
diff --git a/python/pyspark/sql/interchange.py 
b/python/pyspark/sql/interchange.py
new file mode 100644
index 0000000..141d9f3
--- /dev/null
+++ b/python/pyspark/sql/interchange.py
@@ -0,0 +1,89 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import Iterator, Optional
+
+import pyarrow as pa
+
+import pyspark.sql
+from pyspark.sql.types import StructType, StructField, BinaryType
+from pyspark.sql.pandas.types import to_arrow_schema
+
+
+def _get_arrow_array_partition_stream(df: pyspark.sql.DataFrame) -> 
Iterator[pa.RecordBatch]:
+    """Return all the partitions as Arrow arrays in an Iterator."""
+    # We will be using mapInArrow to convert each partition to Arrow 
RecordBatches.
+    # The return type of the function will be a single binary column containing
+    # the serialized RecordBatch in Arrow IPC format.
+    binary_schema = StructType([StructField("arrow_ipc_bytes", BinaryType(), 
nullable=False)])
+
+    def batch_to_bytes_iter(batch_iter: Iterator[pa.RecordBatch]) -> 
Iterator[pa.RecordBatch]:
+        """
+        A generator function that converts RecordBatches to serialized Arrow 
IPC format.
+
+        Spark sends each partition as an iterator of RecordBatches. In order 
to return
+        the entire partition as a stream of Arrow RecordBatches, we need to 
serialize
+        each RecordBatch to Arrow IPC format and yield it as a single binary 
blob.
+        """
+        # The size of the batch can be controlled by the Spark config
+        # `spark.sql.execution.arrow.maxRecordsPerBatch`.
+        for arrow_batch in batch_iter:
+            # We create an in-memory byte stream to hold the serialized batch
+            sink = pa.BufferOutputStream()
+            # Write the batch to the stream using Arrow IPC format
+            with pa.ipc.new_stream(sink, arrow_batch.schema) as writer:
+                writer.write_batch(arrow_batch)
+            buf = sink.getvalue()
+            # The second buffer contains the offsets we are manually creating.
+            offset_buf = pa.array([0, len(buf)], type=pa.int32()).buffers()[1]
+            null_bitmap = None
+            # Wrap the bytes in a new 1-row, 1-column RecordBatch to satisfy 
mapInArrow return
+            # signature. This serializes the whole batch into a single pyarrow 
serialized cell.
+            storage_arr = pa.Array.from_buffers(
+                type=pa.binary(), length=1, buffers=[null_bitmap, offset_buf, 
buf]
+            )
+            yield pa.RecordBatch.from_arrays([storage_arr], 
names=["arrow_ipc_bytes"])
+
+    # Convert all partitions to Arrow RecordBatches and map to binary blobs.
+    byte_df = df.mapInArrow(batch_to_bytes_iter, binary_schema)
+    # A row is actually a batch of data in Arrow IPC format. Fetch the batches 
one by one.
+    for row in byte_df.toLocalIterator():
+        with pa.ipc.open_stream(row.arrow_ipc_bytes) as reader:
+            for batch in reader:
+                # Each batch corresponds to a chunk of data in the partition.
+                yield batch
+
+
+class SparkArrowCStreamer:
+    """
+    A class that implements that __arrow_c_stream__ protocol for Spark 
partitions.
+
+    This class is implemented in a way that allows consumers to consume each 
partition
+    one at a time without materializing all partitions at once on the driver 
side.
+    """
+
+    def __init__(self, df: pyspark.sql.DataFrame):
+        self._df = df
+        self._schema = to_arrow_schema(df.schema)
+
+    def __arrow_c_stream__(self, requested_schema: Optional[object] = None) -> 
object:
+        """
+        Return the Arrow C stream for the dataframe partitions.
+        """
+        reader: pa.RecordBatchReader = pa.RecordBatchReader.from_batches(
+            self._schema, _get_arrow_array_partition_stream(self._df)
+        )
+        return reader.__arrow_c_stream__(requested_schema=requested_schema)
diff --git a/python/tests/test_arrow_e2e.py b/python/tests/test_arrow_e2e.py
index eeec709..ffa1e70 100644
--- a/python/tests/test_arrow_e2e.py
+++ b/python/tests/test_arrow_e2e.py
@@ -67,6 +67,31 @@ def test_to_arrow_empty_result(session):
     assert table.num_rows == 0
 
 
+def test_arrow_c_stream_reads_the_partitions_as_pyspark(session):
+    # As official PySpark 4.3.0's DataFrame.__arrow_c_stream__: an Arrow C 
stream of the rows,
+    # of the Arrow schema of the DataFrame, of a batch of each partition, which
+    # pa.RecordBatchReader.from_stream reads.
+    df = session.range(0, 6, 1, 3).selectExpr(
+        "id", "CAST(id AS STRING) AS s", "array(id, id + 1) AS a", 
"named_struct('x', id) AS n"
+    )
+    expected = df.toArrow()
+    reader = pa.RecordBatchReader.from_stream(df)
+    assert reader.schema == expected.schema
+    batches = list(reader)
+    assert len(batches) == 3
+    assert pa.Table.from_batches(batches, expected.schema).to_pydict() == 
expected.to_pydict()
+    assert type(df.__arrow_c_stream__()).__name__ == "PyCapsule"
+
+    # The stream of a requested schema is cast to it.
+    requested = pa.schema([("id", pa.int32()), ("s", pa.large_string())])
+    table = pa.RecordBatchReader.from_stream(df.select("id", "s"), 
schema=requested).read_all()
+    assert table.schema == requested
+    assert table.to_pydict() == {"id": list(range(6)), "s": [str(i) for i in 
range(6)]}
+
+    empty = pa.RecordBatchReader.from_stream(session.range(0)).read_all()
+    assert empty.num_rows == 0 and empty.schema == pa.schema([pa.field("id", 
pa.int64(), False)])
+
+
 def test_create_dataframe_arrow_cached_upload(session):
     """Materialize cached Arrow input larger than 1 MiB on a real server."""
     import numpy as np
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index 87a6af4..ecef653 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -58,6 +58,9 @@ VENDORED_FILES=(
   "conf.py"
   "taskcontext.py"
 
+  # The Arrow C stream of a DataFrame, of DataFrame.__arrow_c_stream__.
+  "sql/interchange.py"
+
   # Logging
   "logger/__init__.py"
   "logger/logger.py"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to