This is an automated email from the ASF dual-hosted git repository.
dbtsai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new d9defae [SPARK-60586][CONNECT] Export Arrow tables through the Arrow
C Stream Interface
d9defae is described below
commit d9defae6f691671d8bad4927f58a7283cd0a0b50
Author: DB Tsai <[email protected]>
AuthorDate: Thu Oct 29 16:18:31 2026 -0700
[SPARK-60586][CONNECT] Export Arrow tables through the Arrow C Stream
Interface
### What changes were proposed in this pull request?
Under [SPARK-60586](https://issues.apache.org/jira/browse/SPARK-60586),
replace the extra Rust Arrow → IPC bytes → PyArrow round trip in native
`DataFrame.toArrow()` with arrow-rs's built-in export of one `Box<dyn
RecordBatchReader + Send>`:
```rust
let reader: Box<dyn RecordBatchReader + Send> =
Box::new(RecordBatchIterator::new(batches.into_iter().map(Ok), schema));
Ok(reader.into_pyarrow(py)?.call_method0("read_all")?.unbind())
```
The helper is `crates/pyspark-rs/src/pyarrow_export.rs`. Arrow owns the FFI
callbacks, ownership transfer, and release handling; the client adds no custom
unsafe code. The separate converter crate and custom callback adapter are gone.
Before exporting, validate batch schema consistency and retain an IPC
fallback for field names or timezones containing NUL. The standalone PyArrow
schema preflight was removed: its claimed older-version abort rationale was
unproven. Arrow 58.3.0 itself uses `CString::new(...).unwrap()` when exporting
[formats](https://github.com/apache/arrow-rs/blob/58.3.0/arrow-schema/src/ffi.rs#L148)
and
[names](https://github.com/apache/arrow-rs/blob/58.3.0/arrow-schema/src/ffi.rs#L174),
so the exp [...]
PyO3 remains **0.28.3**, Arrow **58.3.0**; enabling Arrow's `pyarrow`
feature adds `arrow-pyarrow 58.3.0`. The contract covers ordinary Spark
results: unsorted maps and non-dictionary-encoded fields. Row/scalar
conversion, UDF workers, and Python-to-Rust imports are unchanged. #103 and
#117 are alternative implementations, not dependencies.
### Why are the changes needed?
Already-decoded Arrow batches previously had to be serialized into another
IPC file, copied into Python bytes, and decoded again. One upstream reader now
consumes the entire result without a Python import call per batch. All decoded
batches are still collected into a Rust vector first; this does not provide
network streaming or bounded-memory collection.
### Does this PR introduce _any_ user-facing change?
`toArrow()` shares compatible decoded value/offset buffers with PyArrow.
`toPandas()` uses the same handoff, but pandas conversion may copy. Network IPC
decoding, descriptors, Python rows, sliced validity-bitmap realignment, and the
compatibility fallback are outside the buffer-sharing claim.
**Intentional behavior change:** inconsistent batch schemas now raise
`ValueError`, including differences in nullability or field/schema metadata.
Previously batches were written under the first batch's schema. Validation
happens before the IPC fallback. Spark normally produces batches from one
result schema.
Values, metadata, batch boundaries, and zero-column row counts are covered
by public-behavior tests. An empty batch vector retains master's empty-schema
behavior; retaining the typed schema when no batches arrive is a separate
follow-up. NUL-containing field names/timezones use the already-collected
batches for IPC without another RPC.
### How was this patch tested?
Current tested source: **`e60aa53`**, based on upstream master `d884e34`.
Rebuilt locally on macOS arm64 with Rust 1.99.1 and optimized release settings:
- **299 passed, 1 skipped** on Python 3.12 / PyArrow 26.0.0.
- **299 passed, 1 skipped** on Python 3.9 / PyArrow 10.0.0, after removing
the schema preflight. Each run includes **26 handoff/fixture cases** plus the
offline suite.
- **24 fixture-dependent handoff cases skip locally with an actionable
message when `protoc` is unavailable; CI fails instead of skipping. Two
additional tests cover both missing-protoc outcomes.** With `protoc`, the mock
uses descriptors compiled from the repository's protocol instead of
hand-encoded field numbers.
- Tests cover values, nested types, metadata, chunks, empty/zero-column
results, `toPandas()`, NUL fallback, schema mismatch, and buffers remaining
usable after their session/table is dropped. Post-deletion value checks
establish retained-buffer lifetime; redundant address-equality assertions were
removed. **These checks do not independently prove Rust-to-Python buffer
identity.** The private FFI mirrors, callback hooks, schema monkeypatch, and
subprocess-isolated boundary spies were [...]
- The normal-handoff regression guard makes `pyarrow.ipc.open_file` fail;
it passes on this head and fails against master `d884e34`. NUL cases explicitly
verify the IPC fallback, including list-of-struct and map-value-struct names.
Schema-mismatch cases now include nullability.
- **392 core Rust unit tests passed**, all integrations compiled with and
without `wasm-udf`, **3 native live-server smoke tests passed**, and official
collection parity passed (**15 passed, 1 skipped**) on the review-fix source.
- Formatting, Python lint, and whitespace checks pass.
The preceding source `d89c01b` also passed **392 core Rust unit tests**,
compiled all core integration tests with and without `wasm-udf`, passed **3
native `toArrow()` live-server smoke tests**, and passed official collection
parity through the Rust transport (**15 passed, 1 skipped**). Collection parity
required `spark.sql.timeType.enabled=true` on the local Spark 4.2 server. The
current review follow-up strengthens regression coverage and documents a
matched master RSS control; it m [...]
**CI scope:** the coverage workflow invokes `test_arrow_handoff.py` and
installs `protobuf-compiler`. Follow-up `e0eb002` adds a native smoke step in
the Connect parity job of `build_python_connect.yml`, using the already-built
extension and running `python -m pytest tests/test_arrow_e2e.py -q` with
`SPARK_REMOTE` and the local package on `PYTHONPATH`. It runs after the
official suite and before the server restarts in local-cluster mode, so the
official suite still begins with a prist [...]
#### Historical benchmark evidence — not measured on the current head
Using [pyspark-connect-benchmark at
`1bbdec9`](https://github.com/dbtsai/pyspark-connect-benchmark/tree/1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4),
the earlier upstream-reader candidate `a9ee289` was compared with master
`d884e34`. That candidate still had the standalone schema preflight and the
older test suite. The PR has since been rebased and updated; these timings are
**historical results, not current-head measurements**.
The matched optimized builds used Rust 1.99.1, `opt-level=3`, `abi3-py39`,
PyO3 0.28.3, Arrow 58.3.0, Python 3.12.14 and PyArrow 26.0.0 on an Apple M3
Max. **240/240 samples passed** across 10 workloads, 6 fresh processes per
client, and 2 counterbalanced rounds. Validation checked shape and first/last
values per column, not every cell.
Representative historical medians, **master → earlier candidate**:
| Workload | Round | Wall time (ms) | Sampled peak RSS (MiB) |
|---|---|---:|---:|
| 38 MiB numeric `toArrow()` | Forward | 24.843 → 14.255 | 268.94 → 146.92 |
| 38 MiB numeric `toArrow()` | Reversed | 27.990 → 14.005 | 268.90 → 146.92
|
| 128 MiB numeric `toArrow()` | Forward | 63.207 → 30.491 | 628.53 → 235.24
|
| 128 MiB numeric `toArrow()` | Reversed | 63.300 → 31.891 | 629.55 →
236.22 |
| 38 MiB numeric `toPandas()` | Forward | 30.023 → 15.494 | 308.38 → 186.51
|
| 38 MiB numeric `toPandas()` | Reversed | 29.995 → 15.163 | 308.52 →
186.50 |
These are shared-desktop measurements of client receive/conversion against
canned localhost RPC responses, not Spark execution. RSS sampling can miss
short peaks; timings and RSS do not establish buffer identity.
**RSS control added during review.** The identical 28-call, 128 MiB probe
was rerun in fresh processes against rebuilt master `d884e34` and this C Stream
revision, using the same environment. After four warmup calls, master rose from
**1129.23 to 3976.06 MiB**, while C Stream rose from **418.81 to 569.47 MiB**.
All results passed validation. Growth is therefore not unique to this PR; the
control grew more in this single run. Both mock server and client still share
one process, so this [...]
<details>
<summary>RSS control methodology and reproducer</summary>
This is a local diagnostic for PR #108, not a performance benchmark or
proof of
leak freedom. Both the mock gRPC server and the native client run in one
Python
process, so the measurement includes their allocations and allocator
retention.
On 2026-10-29, the same 28-call probe was run in fresh processes on master
`d884e34` (IPC handoff) and the review revision of #108 based on `fe79d54`
(C Stream handoff; production changes since that head are comments only).
Both extensions were rebuilt with the same release Cargo configuration and
Rust
1.99.1, then loaded using the same Python 3.12 / PyArrow 26 environment on
macOS
arm64. Master ran first. These are single runs on a shared desktop.
Each call receives 128 one-MiB numeric batches from the same mock service,
validates the 16,777,216-row table's endpoints, deletes the table, forces
Python
GC, and samples process RSS. The first four calls are warmup. All calls
passed
validation.
| Handoff | First measured RSS (MiB) | Last measured RSS (MiB) | Measured
minimum–maximum (MiB) |
| --- | ---: | ---: | ---: |
| Master IPC | 1129.23 | 3976.06 | 1129.23–3976.06 |
| PR C Stream | 418.81 | 569.47 | 418.81–569.47 |
Growth is not unique to the C Stream change; the master control grew much
more
in this run. This does not identify the source of retention, isolate client
allocations, or establish a general memory advantage. The earlier C Stream
run
reported 416.27 → 481.66 MiB; variation between runs remains unexplained. A
claim
of flat post-GC RSS is not supported.
To reproduce, run the following from the PR checkout with its tests
available,
setting `PYTHONPATH` to `<PR checkout>/python`, the directory containing
`pyspark`.
For the master control, build master and set `PYTHONPATH` to `<master
checkout>/python`;
keep this test harness and all other environment settings the same. `PROTOC`
and `PROTOC_INCLUDE` must identify an installed protocol compiler and
includes.
The baseline IPC implementation does not include the new test file, so the
working directory must remain the PR checkout for both runs.
```python
import gc
import json
import runpy
import tempfile
from pathlib import Path
import numpy as np
import psutil
import pyarrow as pa
namespace = runpy.run_path("python/tests/test_arrow_handoff.py")
class TempFactory:
def mktemp(self, name):
return Path(tempfile.mkdtemp(prefix=name))
protocol = namespace["protocol"].__wrapped__(TempFactory())
batch = pa.record_batch(
[pa.array(np.arange(131072, dtype=np.int64))], names=["value"]
)
process = psutil.Process()
samples = []
with namespace["_dataframe"]([batch] * 128, protocol=protocol) as (frame,
session):
for iteration in range(28):
table = frame.toArrow()
assert table.num_rows == 16777216
assert table.column(0)[0].as_py() == 0
assert table.column(0)[-1].as_py() == 131071
del table
gc.collect()
samples.append(process.memory_info().rss / 1048576)
print(json.dumps({"warmup": 4, "post_gc_rss_mib": samples}))
```
</details>
#### Remaining validation and follow-ups
- [x] Run a matched master control for the repeated-result RSS probe.
- [ ] Isolate the source of retention and investigate the historical
high-RSS tail.
- [ ] Rerun comparative benchmarks on the current head.
- [ ] Full official parity suite and minimum-Rust build. Existing workflows
use stable Rust; `pyspark-rs` declares 1.85 while the workspace still declares
1.82. Workspace MSRV cleanup is separate.
- [ ] Review CI and outstanding baseline lint/API gaps. Workspace Clippy
fails in unchanged core/binding code; the existing `SparkSession.Hook` gap
remains. Local checks are not a green-CI claim.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (application version unavailable).
This pull request and its description were written by Isaac.
Closes #108 from dbtsai/codex/arrow-c-stream.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: DB Tsai <[email protected]>
---
.github/workflows/build_python_connect.yml | 7 +
.github/workflows/coverage.yml | 4 +-
Cargo.lock | 14 +
crates/pyspark-rs/Cargo.toml | 4 +-
crates/pyspark-rs/src/dataframe.rs | 18 +-
crates/pyspark-rs/src/lib.rs | 1 +
crates/pyspark-rs/src/pyarrow_export.rs | 108 ++++++++
dev/design/architecture.md | 30 ++-
docs/architecture.md | 9 +
python/tests/test_arrow_e2e.py | 67 +++++
python/tests/test_arrow_handoff.py | 397 +++++++++++++++++++++++++++++
11 files changed, 643 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index 993eccf..b520e28 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -213,6 +213,13 @@ jobs:
--exclude
"test_artifact.py,test_artifact_localcluster.py,test_parity_resources.py" \
2>&1 | tee /tmp/parity_ours.log
+ - name: Native Arrow handoff smoke tests (against the live server)
+ working-directory: python
+ env:
+ SPARK_REMOTE: sc://localhost:15002
+ PYTHONPATH: ${{ github.workspace }}/python
+ run: python -m pytest tests/test_arrow_e2e.py -q
+
- name: Restart Connect server in local-cluster mode
# Resource profiles and artifact distribution across executors can
only be
# exercised against a real multi-executor cluster, so - mirroring
Apache's
diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
index 7f017ad..5afc7c0 100644
--- a/.github/workflows/coverage.yml
+++ b/.github/workflows/coverage.yml
@@ -226,7 +226,9 @@ jobs:
- name: Measure Python coverage with offline test suite
run: |
- cd python && PYTHONPATH=$(pwd) python -m coverage run
--rcfile=.coveragerc -m pytest tests/test_dropin_offline.py
+ cd python
+ PYTHONPATH=$(pwd) python -m coverage run --rcfile=.coveragerc -m
pytest \
+ tests/test_dropin_offline.py tests/test_arrow_handoff.py
PYTHONPATH=$(pwd) python -m coverage json --rcfile=.coveragerc -o
coverage-python.json
- name: Parse Python coverage and create badge JSON
diff --git a/Cargo.lock b/Cargo.lock
index 39d506d..29e7d3f 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -168,6 +168,7 @@ dependencies = [
"arrow-ipc",
"arrow-json",
"arrow-ord",
+ "arrow-pyarrow",
"arrow-row",
"arrow-schema",
"arrow-select",
@@ -322,6 +323,18 @@ dependencies = [
"arrow-select",
]
+[[package]]
+name = "arrow-pyarrow"
+version = "58.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d29abdf672a81c1aeb57fd2661457f9918964d49aed0e9f18932535f2a9e49ce"
+dependencies = [
+ "arrow-array",
+ "arrow-data",
+ "arrow-schema",
+ "pyo3",
+]
+
[[package]]
name = "arrow-row"
version = "58.3.0"
@@ -341,6 +354,7 @@ version = "58.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f633dbfdf39c039ada1bf9e34c694816eb71fbb7dc78f613993b7245e078a1ed"
dependencies = [
+ "bitflags",
"serde_core",
"serde_json",
]
diff --git a/crates/pyspark-rs/Cargo.toml b/crates/pyspark-rs/Cargo.toml
index 16148d4..2417169 100644
--- a/crates/pyspark-rs/Cargo.toml
+++ b/crates/pyspark-rs/Cargo.toml
@@ -2,6 +2,8 @@
name = "pyspark-rs"
version.workspace = true
edition.workspace = true
+# Arrow 58.3.0, including arrow-pyarrow, requires Rust 1.85.
+rust-version = "1.85"
license.workspace = true
description = "PyO3 bindings exposing the Rust Spark Connect client as a
drop-in pyspark package"
# The Python extension (cdylib) is distributed as a PyPI wheel, not on
crates.io.
@@ -23,7 +25,7 @@ spark-connect = { package = "apache-spark-connect", path =
"../spark-connect" }
spark-connect-core = { package = "apache-spark-connect-core", path =
"../spark-connect-core" }
spark-connect-proto = { package = "apache-spark-connect-proto", path =
"../spark-connect-proto" }
pyo3.workspace = true
-arrow.workspace = true
+arrow = { workspace = true, features = ["pyarrow"] }
tokio.workspace = true
prost.workspace = true
tonic.workspace = true
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 7101aad..5ae82b0 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1166,19 +1166,15 @@ impl PyDataFrame {
/// Collect the DataFrame into a `pyarrow.Table`.
///
/// Mirrors `pyspark.sql.DataFrame.toArrow()`: the result is decoded from
the
- /// server's Arrow IPC stream and handed to `pyarrow` (no row-by-row copy).
+ /// server's Arrow IPC stream and handed to `pyarrow` through arrow-rs's
+ /// reader conversion, sharing compatible decoded Arrow value buffers.
Bitmap alignment
+ /// and schemas requiring the IPC compatibility fallback can still copy.
#[pyo3(name = "toArrow")]
fn to_arrow(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
- use pyo3::types::PyBytes;
- // `DataFrame::to_arrow` produces Arrow IPC *file* bytes (FileWriter).
- let bytes = py.detach(|| self.dataframe.to_arrow()).to_pyerr()?;
- let pa = py.import("pyarrow")?;
- let py_bytes = PyBytes::new(py, &bytes);
- let table = pa
- .getattr("ipc")?
- .call_method1("open_file", (py_bytes,))?
- .call_method0("read_all")?;
- Ok(table.unbind())
+ let batches = py
+ .detach(|| self.dataframe.collect_record_batches())
+ .to_pyerr()?;
+ crate::pyarrow_export::record_batches_to_pyarrow(py, batches)
}
/// Collect the DataFrame into a pandas DataFrame.
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 43a850f..fec1a7d 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -37,6 +37,7 @@ mod ml;
mod observation;
mod pipelines;
mod profiler;
+mod pyarrow_export;
mod readwriter;
mod resource;
mod row;
diff --git a/crates/pyspark-rs/src/pyarrow_export.rs
b/crates/pyspark-rs/src/pyarrow_export.rs
new file mode 100644
index 0000000..0a9418d
--- /dev/null
+++ b/crates/pyspark-rs/src/pyarrow_export.rs
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Export decoded Spark results using arrow-rs's PyArrow reader conversion.
+
+use std::sync::Arc;
+
+use arrow::datatypes::{DataType, Field, Schema};
+use arrow::ipc::writer::FileWriter;
+use arrow::pyarrow::IntoPyArrow;
+use arrow::record_batch::{RecordBatch, RecordBatchIterator, RecordBatchReader};
+use pyo3::exceptions::{PyRuntimeError, PyValueError};
+use pyo3::prelude::*;
+use pyo3::types::PyBytes;
+
+/// Share compatible value and offset buffers with PyArrow in one reader
import.
+///
+/// Batches have already been collected; this does not stream the network
response.
+/// Spark result maps are unsorted and result fields are not
dictionary-encoded.
+/// An empty vector retains the existing empty-schema behavior.
+pub(crate) fn record_batches_to_pyarrow(
+ py: Python<'_>,
+ batches: Vec<RecordBatch>,
+) -> PyResult<Py<PyAny>> {
+ let schema = batches
+ .first()
+ .map(|batch| batch.schema())
+ .unwrap_or_else(|| Arc::new(Schema::empty()));
+ // RecordBatchIterator does not validate schemas. Check before either path,
+ // including metadata, so later batches are not relabeled with the first
schema.
+ if batches.iter().any(|batch| batch.schema_ref() != &schema) {
+ return Err(PyValueError::new_err(
+ "Arrow record batches have inconsistent schemas",
+ ));
+ }
+ if schema.fields().iter().any(|field| field_needs_ipc(field)) {
+ // Arrow 58.3 panics on NUL-containing C strings inside the C stream
+ // `get_schema` callback. Unwinding across that extern "C" boundary
+ // aborts Python instead of raising an exception. Check before export,
+ // then reuse the collected batches for IPC without another request.
+ // CString::new(...).unwrap() in FFI_ArrowSchema::try_new / with_name:
+ //
https://github.com/apache/arrow-rs/blob/58.3.0/arrow-schema/src/ffi.rs#L148
+ //
https://github.com/apache/arrow-rs/blob/58.3.0/arrow-schema/src/ffi.rs#L174
+ let bytes = py
+ .detach(|| -> arrow::error::Result<Vec<u8>> {
+ let mut bytes = Vec::new();
+ let mut writer = FileWriter::try_new(&mut bytes, &schema)?;
+ for batch in &batches {
+ writer.write(batch)?;
+ }
+ writer.finish()?;
+ drop(writer);
+ Ok(bytes)
+ })
+ .map_err(|error| PyRuntimeError::new_err(error.to_string()))?;
+ return Ok(py
+ .import("pyarrow")?
+ .getattr("ipc")?
+ .call_method1("open_file", (PyBytes::new(py, &bytes),))?
+ .call_method0("read_all")?
+ .unbind());
+ }
+
+ let reader: Box<dyn RecordBatchReader + Send> =
Box::new(RecordBatchIterator::new(
+ batches.into_iter().map(Ok),
+ schema,
+ ));
+ Ok(reader.into_pyarrow(py)?.call_method0("read_all")?.unbind())
+}
+
+fn field_needs_ipc(field: &Field) -> bool {
+ field.name().contains('\0') || data_type_needs_ipc(field.data_type())
+}
+
+fn data_type_needs_ipc(data_type: &DataType) -> bool {
+ match data_type {
+ DataType::Timestamp(_, Some(timezone)) => timezone.contains('\0'),
+ DataType::List(field)
+ | DataType::LargeList(field)
+ | DataType::ListView(field)
+ | DataType::LargeListView(field)
+ | DataType::FixedSizeList(field, _)
+ | DataType::Map(field, _) => field_needs_ipc(field),
+ DataType::Struct(fields) => fields.iter().any(|field|
field_needs_ipc(field)),
+ DataType::Union(fields, _) => fields.iter().any(|(_, field)|
field_needs_ipc(field)),
+ DataType::RunEndEncoded(run_ends, values) => {
+ field_needs_ipc(run_ends) || field_needs_ipc(values)
+ }
+ DataType::Dictionary(keys, values) => {
+ data_type_needs_ipc(keys) || data_type_needs_ipc(values)
+ }
+ _ => false,
+ }
+}
diff --git a/dev/design/architecture.md b/dev/design/architecture.md
index 8730376..ead757c 100644
--- a/dev/design/architecture.md
+++ b/dev/design/architecture.md
@@ -67,9 +67,33 @@ Server-streaming RPCs (`ExecutePlan`) are consumed into
Arrow record batches.
## Data path
-Results arrive as Arrow IPC streams. `arrow-rs` decodes them; hand-off to
Python
-uses the **Arrow C Data Interface** (zero-copy to `pyarrow`), which is also how
-we build `toPandas()` / `createDataFrame(pandas)` without re-serializing.
+Results arrive as Arrow IPC streams. `arrow-rs` decodes them; `toArrow()`
collects
+all batches into a Rust vector. `pyspark-rs` uses arrow-rs's `IntoPyArrow` for
+one boxed `RecordBatchReader`, importing it as a PyArrow
+`RecordBatchReader`, and calls `read_all()`. PyArrow pulls batches through
native
+callbacks that do not call Python, avoiding a per-batch Python import loop.
+This is an in-process buffer handoff, not network streaming or bounded-memory
+result collection.
+
+Compatible Arrow value and offset buffers are shared with PyArrow; sliced
+validity bitmaps may need realignment. The bridge validates batch schema
+consistency, including metadata. Schemas with embedded
+NUL field names or timezones use an IPC compatibility fallback, without another
+RPC. Spark results use unsorted maps and are not dictionary-encoded; this path
+does not repair generic Arrow ordering flags. `toPandas()` then calls
+PyArrow's pandas conversion, which may copy.
+
+Scalar input conversion and row materialization stay in `pyspark-rs`.
`collect()`
+still uses the native client's owned rows; `createDataFrame(pandas)` still
+materializes Python rows. Neither path is claimed to be zero-copy.
+
+The Arrow handoff has been tested locally with PyArrow 10.0.0 (Python 3.9)
+and 26.0.0 (Python 3.12). CI installs the latest PyArrow; it does not pin or
+separately test the declared PyArrow 10 minimum. The native live-server smoke
+tests in `python/tests/test_arrow_e2e.py` run in the Connect parity CI job
+against its live server and built native extension. Official parity tests
+exercise the Rust transport with official PySpark; the separate native smoke
+step covers the drop-in `toArrow()` path.
## UDF strategy (full compatibility)
diff --git a/docs/architecture.md b/docs/architecture.md
index f7fdb92..17559ed 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -102,6 +102,15 @@ triggers `ExecutePlan`. If the response stream breaks
mid-flight, the core's
reattachable-execute iterator resumes from the last response it saw, so long
results survive transient disconnects.
+### Python Arrow table handoff
+
+`toArrow()` collects decoded batches and uses arrow-rs's `IntoPyArrow` for one
+boxed `RecordBatchReader`. Compatible value and offset buffers are shared;
+network decoding, sliced validity bitmaps, and pandas conversion may still
copy.
+Batch schema consistency is validated before export. Field names or timezones
+containing NUL use IPC over the collected batches, without another RPC. Empty
+batch vectors retain the empty-schema behavior.
+
## Two front-ends, one core
=== "Rust (native)"
diff --git a/python/tests/test_arrow_e2e.py b/python/tests/test_arrow_e2e.py
new file mode 100644
index 0000000..dc76866
--- /dev/null
+++ b/python/tests/test_arrow_e2e.py
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Native toArrow smoke tests against a live Spark Connect server."""
+
+import os
+from datetime import date, datetime, timezone
+
+import pyarrow as pa
+import pytest
+from pyspark.sql import SparkSession
+
+
[email protected]
+def session():
+ remote = os.environ.get("SPARK_REMOTE")
+ if not remote:
+ pytest.skip("SPARK_REMOTE is required for live-server tests")
+ session = SparkSession.builder.remote(remote).getOrCreate()
+ try:
+ yield session
+ finally:
+ session.stop()
+
+
+def test_to_arrow_nested_null_binary_and_temporal_values(session):
+ table = session.sql("""
+ SELECT array(1, CAST(NULL AS INT), 3) AS items,
+ named_struct('name', 'spark', 'value', CAST(NULL AS BIGINT)) AS
nested,
+ X'00FF' AS binary_value, DATE '2020-01-02' AS day,
+ TIMESTAMP '2020-01-02 03:04:05+00:00' AS instant
+ """).toArrow()
+ assert isinstance(table, pa.Table)
+ values = table.to_pylist()
+ assert len(values) == 1
+ assert values[0] == {
+ "items": [1, None, 3],
+ "nested": {"name": "spark", "value": None},
+ "binary_value": b"\x00\xff",
+ "day": date(2020, 1, 2),
+ "instant": datetime(2020, 1, 2, 3, 4, 5, tzinfo=timezone.utc),
+ }
+
+
+def test_to_arrow_numeric_range(session):
+ table = session.range(10000).toArrow()
+ assert table.num_rows == 10000
+ assert table.column("id").to_pylist() == list(range(10000))
+
+
+def test_to_arrow_empty_result(session):
+ table = session.range(0).toArrow()
+ assert isinstance(table, pa.Table)
+ assert table.num_rows == 0
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
new file mode 100644
index 0000000..214a09a
--- /dev/null
+++ b/python/tests/test_arrow_handoff.py
@@ -0,0 +1,397 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Check public Arrow handoff behavior against a deterministic mock RPC
service.
+
+Cases cover Spark-output values, schemas, chunks, and lifetimes through public
+APIs. Server-side Python addresses cannot prove Rust-to-Python buffer sharing.
+"""
+
+import gc
+import os
+import shutil
+import subprocess
+from concurrent.futures import ThreadPoolExecutor
+from contextlib import contextmanager
+from datetime import datetime, timezone
+from decimal import Decimal
+from pathlib import Path
+
+import grpc
+import pandas as pd
+import pyarrow as pa
+import pytest
+from google.protobuf import descriptor_pb2, descriptor_pool, message_factory
+from pyspark.sql import SparkSession
+
+
[email protected](scope="module")
+def protocol(tmp_path_factory):
+ """Compile the repository protocol instead of hard-coding protobuf field
numbers."""
+ protoc = shutil.which(os.environ.get("PROTOC", "protoc"))
+ if protoc is None:
+ message = "Arrow handoff tests require protoc; install it or set
PROTOC"
+ if os.environ.get("CI"):
+ pytest.fail(message)
+ pytest.skip(message)
+ proto_root = Path(__file__).resolve().parents[2] /
"crates/spark-connect-proto/proto"
+ descriptor = tmp_path_factory.mktemp("connect-proto") / "connect.pb"
+ command = [protoc, f"-I{proto_root}"]
+ if include := os.environ.get("PROTOC_INCLUDE"):
+ command.append(f"-I{include}")
+ subprocess.run(
+ command
+ + [
+ "--include_imports",
+ f"--descriptor_set_out={descriptor}",
+ str(proto_root / "spark/connect/base.proto"),
+ ],
+ check=True,
+ )
+ pool = descriptor_pool.DescriptorPool()
+ for file in
descriptor_pb2.FileDescriptorSet.FromString(descriptor.read_bytes()).file:
+ pool.Add(file)
+
+ def message(name, **fields):
+ return
message_factory.GetMessageClass(pool.FindMessageTypeByName(f"spark.connect.{name}"))(
+ **fields
+ )
+
+ return message
+
+
[email protected]("in_ci", [False, True])
+def test_missing_protoc_fails_in_ci_and_skips_locally(monkeypatch,
tmp_path_factory, in_ci):
+ monkeypatch.setattr(shutil, "which", lambda _: None)
+ if in_ci:
+ monkeypatch.setenv("CI", "true")
+ else:
+ monkeypatch.delenv("CI", raising=False)
+ error = pytest.fail.Exception if in_ci else pytest.skip.Exception
+ with pytest.raises(error, match="require protoc"):
+ protocol.__wrapped__(tmp_path_factory)
+
+
+@contextmanager
+def _dataframe(batches, *, protocol, payloads=None, calls=None):
+ """Serve Arrow results using the repository's protobuf message
definitions."""
+ if payloads is None:
+ payloads = []
+ for batch in batches:
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, batch.schema) as writer:
+ writer.write_batch(batch)
+ payloads.append((batch.num_rows, sink.getvalue().to_pybytes()))
+ server_id = "00000000-0000-0000-0000-000000000001"
+
+ def execute(request, context):
+ if calls is not None:
+ calls["execute"] = calls.get("execute", 0) + 1
+ request = protocol("ExecutePlanRequest").FromString(request)
+ header = dict(
+ session_id=request.session_id,
+ operation_id=request.operation_id,
+ server_side_session_id=server_id,
+ )
+ for index, (rows, payload) in enumerate(payloads):
+ yield protocol(
+ "ExecutePlanResponse",
+ **header,
+ response_id=str(index),
+ arrow_batch=dict(row_count=rows, data=payload),
+ ).SerializeToString()
+ yield protocol(
+ "ExecutePlanResponse", **header, response_id="complete",
result_complete={}
+ ).SerializeToString()
+
+ def release_execute(request, context):
+ request = protocol("ReleaseExecuteRequest").FromString(request)
+ return protocol(
+ "ReleaseExecuteResponse",
+ session_id=request.session_id,
+ operation_id=request.operation_id,
+ server_side_session_id=server_id,
+ ).SerializeToString()
+
+ def release_session(request, context):
+ request = protocol("ReleaseSessionRequest").FromString(request)
+ return protocol(
+ "ReleaseSessionResponse",
+ session_id=request.session_id,
+ server_side_session_id=server_id,
+ ).SerializeToString()
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ server = grpc.server(executor)
+ server.add_generic_rpc_handlers(
+ [
+ grpc.method_handlers_generic_handler(
+ "spark.connect.SparkConnectService",
+ {
+ "ExecutePlan":
grpc.unary_stream_rpc_method_handler(execute),
+ "ReleaseExecute":
grpc.unary_unary_rpc_method_handler(release_execute),
+ "ReleaseSession":
grpc.unary_unary_rpc_method_handler(release_session),
+ },
+ )
+ ]
+ )
+ port = server.add_insecure_port("127.0.0.1:0")
+ assert port != 0
+ server.start()
+ session = None
+ try:
+ session =
SparkSession.builder.remote(f"sc://127.0.0.1:{port}").getOrCreate()
+ yield session.range(1), session
+ finally:
+ try:
+ if session is not None:
+ session.stop()
+ finally:
+ server.stop(0).wait(timeout=5)
+
+
+def _mixed_batch():
+ nested_type = pa.struct(
+ [
+ pa.field("items", pa.list_(pa.int64()), metadata={b"nested":
b"metadata"}),
+ pa.field("labels", pa.map_(pa.string(), pa.int64())),
+ ]
+ )
+ arrays = [
+ pa.array([1, None, -3, 4], type=pa.int64()),
+ pa.array([True, None, False, True]),
+ pa.array(["alpha", None, "", "omega"]),
+ pa.array([b"a", None, b"", b"xyz"]),
+ pa.array(
+ [Decimal("1.23"), None, Decimal("-4.56"), Decimal("0.00")],
type=pa.decimal128(12, 2)
+ ),
+ pa.array(
+ [
+ datetime(2020, 1, 1, tzinfo=timezone.utc),
+ None,
+ datetime(2020, 1, 2, tzinfo=timezone.utc),
+ datetime(2020, 1, 3, tzinfo=timezone.utc),
+ ],
+ type=pa.timestamp("us", tz="UTC"),
+ ),
+ pa.array(
+ [
+ {"items": [1, None], "labels": [("a", 1)]},
+ None,
+ {"items": [], "labels": []},
+ {"items": None, "labels": [("b", None)]},
+ ],
+ type=nested_type,
+ ),
+ pa.nulls(4),
+ ]
+ fields = [
+ pa.field(f"c{index}", array.type, metadata={b"field":
str(index).encode()})
+ for index, array in enumerate(arrays)
+ ]
+ schema = pa.schema(fields, metadata={b"schema": b"preserved"})
+ return pa.RecordBatch.from_arrays(arrays, schema=schema)
+
+
+def test_handoff_preserves_values_metadata_and_chunks(protocol):
+ batch = _mixed_batch()
+ batches = [batch.slice(1, 2), batch.slice(0, 1), batch.slice(3, 1)]
+ expected = pa.Table.from_batches(batches)
+ calls = {}
+ with _dataframe(protocol=protocol, batches=batches, calls=calls) as
(frame, _):
+ actual = frame.toArrow()
+ assert actual.equals(expected, check_metadata=True)
+ for column in actual.columns:
+ assert [len(chunk) for chunk in column.chunks] == [2, 1, 1]
+ assert calls == {"execute": 1}
+
+
+def test_normal_handoff_does_not_reopen_ipc(protocol, monkeypatch):
+ def unexpected_ipc(*args, **kwargs):
+ pytest.fail("Normal toArrow handoff must use the C Stream importer")
+
+ monkeypatch.setattr(pa.ipc, "open_file", unexpected_ipc)
+ batch = _mixed_batch()
+ with _dataframe(protocol=protocol, batches=[batch]) as (frame, _):
+ assert frame.toArrow().equals(pa.Table.from_batches([batch]),
check_metadata=True)
+
+
+def test_handoff_buffers_outlive_dataframe_session_and_table(protocol):
+ batch = _mixed_batch()
+ expected = batch.to_pydict()
+ calls = {}
+
+ def collect():
+ with _dataframe(protocol=protocol, batches=[batch], calls=calls) as
(frame, session):
+ table = frame.toArrow()
+ session.stop()
+ return table
+
+ table = collect()
+ gc.collect()
+ assert table.to_pydict() == expected
+ numeric = table.column(0).chunk(0).slice(1, 3)
+ strings = table.column(2).chunk(0).slice(1, 3)
+ del table
+ gc.collect()
+ assert numeric.to_pylist() == [None, -3, 4]
+ assert strings.to_pylist() == [None, "", "omega"]
+ assert calls == {"execute": 1}
+
+
[email protected]("kind", ["no-batches", "empty-typed", "zero-columns"])
+def test_handoff_empty_and_zero_column_results(protocol, kind):
+ if kind == "no-batches":
+ batches = []
+ expected = pa.table({})
+ elif kind == "empty-typed":
+ batches = [_mixed_batch().slice(0, 0)]
+ expected = pa.Table.from_batches(batches)
+ else:
+ structs = pa.array([{}] * 5, type=pa.struct([]))
+ batch = pa.RecordBatch.from_struct_array(structs)
+ batch = batch.replace_schema_metadata({b"zero-column": b"preserved"})
+ batches = [batch.slice(0, 3), batch.slice(3, 2)]
+ expected = pa.Table.from_batches(batches)
+ assert expected.num_rows == 5
+ calls = {}
+ with _dataframe(protocol=protocol, batches=batches, calls=calls) as
(frame, _):
+ actual = frame.toArrow()
+ assert actual.num_rows == expected.num_rows
+ assert actual.num_columns == expected.num_columns
+ assert actual.equals(expected, check_metadata=True)
+ assert calls == {"execute": 1}
+
+
+def test_to_pandas_uses_the_same_handoff(protocol):
+ batch = _mixed_batch()
+ indices = [0, 2, 4, 5]
+ schema = pa.schema(
+ [batch.schema.field(index) for index in indices],
metadata=batch.schema.metadata
+ )
+ arrays = [batch.column(index) for index in indices]
+ selected = pa.RecordBatch.from_arrays(arrays, schema=schema)
+ batches = [selected.slice(0, 2), selected.slice(2, 2)]
+ expected = pa.Table.from_batches(batches).to_pandas()
+ calls = {}
+ with _dataframe(protocol=protocol, batches=batches, calls=calls) as
(frame, _):
+ actual = frame.toPandas()
+ pd.testing.assert_frame_equal(actual, expected)
+ assert calls == {"execute": 1}
+
+
[email protected](
+ "kind", ["field", "nested-field", "list-struct", "map-value-struct",
"timezone"]
+)
+def test_nul_schema_strings_fall_back_without_executing_again(protocol, kind,
monkeypatch):
+ open_file = pa.ipc.open_file
+ ipc_calls = []
+
+ def record_ipc(*args, **kwargs):
+ ipc_calls.append(True)
+ return open_file(*args, **kwargs)
+
+ monkeypatch.setattr(pa.ipc, "open_file", record_ipc)
+ if kind == "field":
+ name = "bad\x00name"
+ values = pa.array([1, None, -3, 4])
+ elif kind == "nested-field":
+ name = "nested"
+ data_type = pa.struct([pa.field("bad\x00name", pa.int64())])
+ values = pa.array(
+ [{"bad\x00name": 1}, None, {"bad\x00name": None}, {"bad\x00name":
4}],
+ type=data_type,
+ )
+ elif kind == "list-struct":
+ name = "items"
+ data_type = pa.list_(pa.struct([pa.field("bad\x00name", pa.int64())]))
+ values = pa.array([[{"bad\x00name": 1}], None, [], [{"bad\x00name":
None}]], type=data_type)
+ elif kind == "map-value-struct":
+ name = "mapping"
+ data_type = pa.map_(pa.string(), pa.struct([pa.field("bad\x00name",
pa.int64())]))
+ values = pa.array(
+ [[("a", {"bad\x00name": 1})], None, [], [("b", {"bad\x00name":
None})]], type=data_type
+ )
+ else:
+ name = "timestamp"
+ values = pa.array([0, None, 2, 3], type=pa.timestamp("us",
tz="UTC\x00suffix"))
+ schema = pa.schema(
+ [pa.field(name, values.type, metadata={b"field": b"preserved"})],
+ metadata={b"schema": b"preserved"},
+ )
+ batch = pa.RecordBatch.from_arrays([values], schema=schema)
+ batches = [batch.slice(0, 2), batch.slice(2, 2)]
+ expected = pa.Table.from_batches(batches)
+ calls = {}
+ with _dataframe(protocol=protocol, batches=batches, calls=calls) as
(frame, _):
+ actual = frame.toArrow()
+ # Compare raw Arrow data, not Python datetimes: the deliberately unusual
+ # timezone string is preserved but is not a Python timezone identifier.
+ assert actual.equals(expected, check_metadata=True)
+ assert [len(chunk) for chunk in actual.column(0).chunks] == [2, 2]
+ assert ipc_calls == [True]
+ assert calls == {"execute": 1}
+
+
+def test_nul_schema_and_field_metadata_are_preserved(protocol):
+ field = pa.field("value", pa.int64(), metadata={b"field\x00key":
b"field\x00value"})
+ schema = pa.schema([field], metadata={b"schema\x00key":
b"schema\x00value"})
+ batch = pa.RecordBatch.from_arrays([pa.array([1, None, 3])], schema=schema)
+ calls = {}
+ with _dataframe(protocol=protocol, batches=[batch], calls=calls) as
(frame, _):
+ actual = frame.toArrow()
+ assert actual.equals(pa.Table.from_batches([batch]), check_metadata=True)
+ assert calls == {"execute": 1}
+
+
[email protected](
+ "difference", ["type", "name", "schema-metadata", "field-metadata",
"nullability"]
+)
[email protected]("nul_name", [False, True])
+def test_mismatched_batch_schemas_are_rejected_before_fallback(protocol,
difference, nul_name):
+ name = "bad\x00name" if nul_name else "value"
+ first_field = pa.field(name, pa.int64(), metadata={b"field": b"original"})
+ first_schema = pa.schema([first_field], metadata={b"schema": b"original"})
+ second_values = pa.array(["text", "other"]) if difference == "type" else
pa.array([3, 4])
+ second_name = name + "-different" if difference == "name" else name
+ field_metadata = {b"field": b"different"} if difference ==
"field-metadata" else None
+ schema_metadata = {b"schema": b"different"} if difference ==
"schema-metadata" else None
+ second_field = pa.field(
+ second_name,
+ second_values.type,
+ nullable=difference != "nullability",
+ metadata=field_metadata or first_field.metadata,
+ )
+ second_schema = pa.schema([second_field], metadata=schema_metadata or
first_schema.metadata)
+ batches = [
+ pa.RecordBatch.from_arrays([pa.array([1, 2])], schema=first_schema),
+ pa.RecordBatch.from_arrays([second_values], schema=second_schema),
+ ]
+ calls = {}
+ with _dataframe(protocol=protocol, batches=batches, calls=calls) as
(frame, _):
+ with pytest.raises(ValueError, match="inconsistent schemas"):
+ frame.toArrow()
+ assert calls == {"execute": 1}
+
+
+def test_malformed_ipc_is_reported(protocol):
+ calls = {}
+ with _dataframe(
+ protocol=protocol, batches=[], payloads=[(1, b"not arrow ipc")],
calls=calls
+ ) as (frame, _):
+ with pytest.raises(RuntimeError, match="Arrow"):
+ frame.toArrow()
+ assert calls == {"execute": 1}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]