DB Tsai created SPARK-61212:
-------------------------------
Summary: Normalize Arrow map child names in the Rust Spark Connect
local input API
Key: SPARK-61212
URL: https://issues.apache.org/jira/browse/SPARK-61212
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
The public Rust SparkSession::create_dataframe_from_arrow API serializes the
default Arrow Rust MapBuilder child names (keys and values). Spark Java
ArrowColumnVector expects key and value, so analyzing or collecting a nonempty
local map relation fails with a NullPointerException in
ArrowColumnVector.MapAccessor. A zero-row relation also retains the
incompatible IPC field names. Maps nested in lists or structs have the same
mismatch.
Normalize Arrow map entry/key/value names to entries/key/value on the public
Arrow-import boundary for inline and chunked uploads, including empty input.
Preserve schema metadata, nullability, ordering flags and slices, and share the
original value/offset/validity buffers during normalization. Keep raw
byte-based LocalRelation construction transparent.
Rust regression tests cover IPC names, nested containers, metadata and buffer
sharing, original schema mismatch validation, and live inline/chunked round
trips against Spark. The live tests fail on unpatched upstream
925dc876f63f1f779fe176f62575a0aa96cc6886 and pass with the normalization patch.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
