DB Tsai created SPARK-60588:
-------------------------------
Summary: Share nested struct field names in the Rust Spark Connect
client
Key: SPARK-60588
URL: https://issues.apache.org/jira/browse/SPARK-60588
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Arrow decoding still copies every nested struct field name for every row
because Value::Struct stores Vec<(String, Value)>. A wide struct column
therefore retains repeated name allocations even after PR #105 shares top-level
Row names.
Investigate sharing immutable nested struct names while preserving field order,
duplicate-name lookup, null handling, nested value ownership, equality/Debug
behavior and the Rust/Python interfaces. Keep locally constructed values and
Arrow-decoded values behaviorally equivalent.
Validate with independent correctness and round-trip tests plus repeated
fresh-process comparisons against the exact current base. Include wide struct
columns, nested arrays/maps of structs and fully consumed output scans; measure
CPU, wall time and process RSS, and guard unrelated inputs/return paths. Do not
infer nested savings from top-level row measurements.
Follow-up to SPARK-60585 and
https://github.com/apache/spark-connect-rust/pull/105 . This work is outside
that PR's scope.
Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
