[
https://issues.apache.org/jira/browse/SPARK-60586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60586:
-----------------------------------
Labels: pull-request-available (was: )
> Extract a shared Python and Arrow converter for Spark Connect Rust
> ------------------------------------------------------------------
>
> Key: SPARK-60586
> URL: https://issues.apache.org/jira/browse/SPARK-60586
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> h2. Summary
> Extract a standalone spark-python-arrow Rust crate for Python-facing Spark
> clients and workers, and adopt it in pyspark-rs. This provides one reusable,
> versioned implementation of Arrow/PyArrow interoperability and Python
> scalar/UDF value conversions without adding Python dependencies to the Rust
> client core.
> Draft implementation: https://github.com/apache/spark-connect-rust/pull/103
> h2. Scope
> * Share compatible, already-decoded Arrow value and offset buffers through
> the Arrow C Data Interface instead of reserializing them to IPC for the
> Python client's toArrow() handoff.
> * Provide reusable schema, type, array, and record-batch import/export with
> explicit ownership and lifetime contracts.
> * Share scalar materialization and UDF input/result conversion policy while
> retaining caller-owned Python classes and fallback callbacks.
> * Keep transport, worker execution, session state, and downstream dependency
> migration outside this crate.
> h2. Zero-copy boundary
> Zero-copy describes the compatible Rust Arrow/PyArrow buffer handoff, not the
> entire pipeline. Network/IPC decoding, schema and Python object allocation,
> scalar/row materialization, UDF result construction, and pandas conversion
> may allocate or copy. Nullable slices can require validity-bitmap
> realignment. Embedded NUL field names or timezones use an IPC compatibility
> fallback for table export.
> h2. Acceptance criteria
> * Verify shared value-buffer addresses and correct release/lifetime behavior,
> including sliced arrays, nested fields, empty results, and zero-column
> batches.
> * Preserve sorted-map flags on export and ordered-dictionary flags on import,
> including nested fields. Review of the current draft identified losses in
> these two cases.
> * Preserve dictionary-value extension identity, or reject unsupported inputs
> explicitly instead of silently lowering them to their storage type.
> * Preserve scalar/UDF conversion semantics, fallback behavior, mutable-result
> handling, and conversion-error ordering.
> * Document copying paths and tested dependency versions accurately.
> h2. Validation status
> At draft head 234353b5e0350587d736ecdd21e1c90e97c0fd7f, 32 standalone
> converter tests and 17 client handoff tests pass locally with PyO3 0.28.3 and
> PyArrow 26. The three schema-preservation cases above were reproduced
> separately and still need fixes and regression coverage. Full official
> parity, PyO3 0.29, and downstream worker integration remain unverified; this
> issue does not claim those migrations or validations are complete.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
