[
https://issues.apache.org/jira/browse/SPARK-60585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60585.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 105
[https://github.com/apache/spark-connect-rust/pull/105]
> Share immutable field names across decoded rows in the Rust Spark Connect
> client
> --------------------------------------------------------------------------------
>
> Key: SPARK-60585
> URL: https://issues.apache.org/jira/browse/SPARK-60585
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Share immutable field names across top-level Arrow-decoded rows and nested
> struct values.
> Implementation and benchmark table:
> https://github.com/apache/spark-connect-rust/pull/105
> SPARK-60588 is closed as a duplicate of SPARK-60585. Nested struct sharing is
> included in PR #105; implementation and remaining validation are tracked in
> SPARK-60585.
> https://issues.apache.org/jira/browse/SPARK-60585
> https://github.com/apache/spark-connect-rust/pull/105
> Apple M3 Max/macOS arm64; Apache master `a85e474` → combined PR head
> `834402b` (top-level and nested sharing; no incremental nested-only
> comparison). The [benchmark
> harness](https://github.com/dbtsai/pyspark-connect-benchmark) uses canned
> Arrow/gRPC fixtures, 23 frozen cases, 30 fresh processes per case/client and
> three rotating clients. screen: 2,070 recorded, 2,010 successful, 60 failed,
> 0 missing; confirm: 2,070 recorded, 2,010 successful, 60 failed, 0 missing.
> **Overall frozen verdict: rejected.** Arrow `Duration(Microsecond)` is
> unsupported in both Rust builds (30 failed mixed-record samples each per
> run); timing guards also remain uncertain. No accepted overall CPU or memory
> improvement is claimed. Independent full confirmation is included.
> Rust API break: `Value::Struct` now holds public `StructValue` instead of
> `Vec<(String, Value)>`, sharing names while owning values. Migrate
> construction to `Value::Struct(Row::new(fields, values).into())` and use its
> field/value accessors. Tested Python Row behavior is unchanged. Measured
> arm64 `Row` and `Value` both remain 48 bytes.
> 412 Rust checks passed; all core integration targets compiled with and
> without `wasm-udf`. The 108-class Python API snapshot and 11 behavior/fixture
> comparisons pass, as do formatting and core Clippy. Local workspace Clippy
> lacks the wasm standard library; local live-server parity was unavailable.
> Exact-head CI: 12 checks passed; Connect parity is still running; docs
> deployment was skipped.
> Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
