[
https://issues.apache.org/jira/browse/SPARK-61212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61212:
-----------------------------------
Labels: pull-request-available (was: )
> Normalize Arrow map child names in the Rust Spark Connect local input API
> -------------------------------------------------------------------------
>
> Key: SPARK-61212
> URL: https://issues.apache.org/jira/browse/SPARK-61212
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The public Rust SparkSession::create_dataframe_from_arrow API serializes the
> default Arrow Rust MapBuilder child names (keys and values). Spark Java
> ArrowColumnVector expects key and value, so analyzing or collecting a
> nonempty local map relation fails with a NullPointerException in
> ArrowColumnVector.MapAccessor. A zero-row relation also retains the
> incompatible IPC field names. Maps nested in lists or structs have the same
> mismatch.
> Normalize Arrow map entry/key/value names to entries/key/value on the public
> Arrow-import boundary for inline and chunked uploads, including empty input.
> Preserve schema metadata, nullability, ordering flags and slices, and share
> the original value/offset/validity buffers during normalization. Keep raw
> byte-based LocalRelation construction transparent.
> Rust regression tests cover IPC names, nested containers, metadata and buffer
> sharing, original schema mismatch validation, and live inline/chunked round
> trips against Spark. The live tests fail on unpatched upstream
> 925dc876f63f1f779fe176f62575a0aa96cc6886 and pass with the normalization
> patch.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
