[
https://issues.apache.org/jira/browse/SPARK-61100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61100:
------------------------------------
Assignee: DB Tsai
> Preserve tuple compatibility for Python Row objects
> ---------------------------------------------------
>
> Key: SPARK-61100
> URL: https://issues.apache.org/jira/browse/SPARK-61100
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Official PySpark Row is a tuple subclass. The Rust-backed Python Row was a
> native extension class, so isinstance(Row(a=1), tuple) was false and
> tuple-aware serializers such as json.dumps could not encode it. Its
> __fields__ accessor also returned a fresh list, losing in-place mutations.
> These behaviors were initially observed on public apache/spark-connect-rust
> master 83d74462ec41012993176cf3af99017576688eb8.
> The proposed implementation in
> https://github.com/apache/spark-connect-rust/pull/383 embeds the PySpark
> 4.3.0 Row implementation and uses one public tuple-based Row identity for
> constructed, collected and nested rows. It preserves tuple operations,
> equality/hash, pickle and mutable field metadata. Collection and iteration
> create eager Rows, with field-name sharing per collected result, per
> top-level iterator Arrow batch and per nested schema path. Local-input and
> UDT conversions use the same public Row identity.
> Regression coverage exercises tuple-aware serialization, field-name mutation
> and sharing, nested inputs, Arrow batches, datatype conversion and
> UDT/spatial results. The inherited customized-pandas named-Row conversion
> behavior remains. The branch includes the NumPy prerequisite in SPARK-61102
> and binary configuration/iterator lifecycle work in SPARK-61099.
> Materialization costs were measured against the final binary prerequisite.
> The measured CPU and retained-memory regressions are accepted as an
> engineering tradeoff for tuple compatibility and are disclosed in the
> validation comment. This issue retains connect-rust-4.3.0 release tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
