DB Tsai created SPARK-60586:
-------------------------------
Summary: Extract a shared Python and Arrow converter for Spark
Connect Rust
Key: SPARK-60586
URL: https://issues.apache.org/jira/browse/SPARK-60586
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
h2. Summary
Extract a standalone spark-python-arrow Rust crate for Python-facing Spark
clients and workers, and adopt it in pyspark-rs. This provides one reusable,
versioned implementation of Arrow/PyArrow interoperability and Python
scalar/UDF value conversions without adding Python dependencies to the Rust
client core.
Draft implementation: https://github.com/apache/spark-connect-rust/pull/103
h2. Scope
* Share compatible, already-decoded Arrow value and offset buffers through the
Arrow C Data Interface instead of reserializing them to IPC for the Python
client's toArrow() handoff.
* Provide reusable schema, type, array, and record-batch import/export with
explicit ownership and lifetime contracts.
* Share scalar materialization and UDF input/result conversion policy while
retaining caller-owned Python classes and fallback callbacks.
* Keep transport, worker execution, session state, and downstream dependency
migration outside this crate.
h2. Zero-copy boundary
Zero-copy describes the compatible Rust Arrow/PyArrow buffer handoff, not the
entire pipeline. Network/IPC decoding, schema and Python object allocation,
scalar/row materialization, UDF result construction, and pandas conversion may
allocate or copy. Nullable slices can require validity-bitmap realignment.
Embedded NUL field names or timezones use an IPC compatibility fallback for
table export.
h2. Acceptance criteria
* Verify shared value-buffer addresses and correct release/lifetime behavior,
including sliced arrays, nested fields, empty results, and zero-column batches.
* Preserve sorted-map flags on export and ordered-dictionary flags on import,
including nested fields. Review of the current draft identified losses in these
two cases.
* Preserve dictionary-value extension identity, or reject unsupported inputs
explicitly instead of silently lowering them to their storage type.
* Preserve scalar/UDF conversion semantics, fallback behavior, mutable-result
handling, and conversion-error ordering.
* Document copying paths and tested dependency versions accurately.
h2. Validation status
At draft head 234353b5e0350587d736ecdd21e1c90e97c0fd7f, 32 standalone converter
tests and 17 client handoff tests pass locally with PyO3 0.28.3 and PyArrow 26.
The three schema-preservation cases above were reproduced separately and still
need fixes and regression coverage. Full official parity, PyO3 0.29, and
downstream worker integration remain unverified; this issue does not claim
those migrations or validations are complete.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
