dbtsai opened a new pull request, #114: URL: https://github.com/apache/spark-connect-rust/pull/114
### What changes were proposed in this pull request? Local Row-to-Arrow encoding currently clones each input column and nested child Values before building Arrow arrays. Borrow these Values through `&[&Value]`, including null placeholders and struct name/position lookup. Build string/char/varchar map keys directly from borrowed key text; retain the existing validation path for non-string key declarations, including empty maps. Numeric conversions, duplicate-name lookup, nullability, schema validation and error traversal order remain unchanged. Arrow output buffers and IPC serialization still allocate and copy; this is not zero-copy Arrow construction. **Depends on #109**, pinned to `ee0582c37f888a66edaf147345b4ccc46f51f626`. That PR is still open, so this PR currently includes its commits. The incremental change for SPARK-60640 is the final commit; rebase onto master after #109 merges. ### Why are the changes needed? [SPARK-60640](https://issues.apache.org/jira/browse/SPARK-60640): input Rows retain the source payloads for the duration of encoding, so the temporary deep copies are redundant. Python field-name sharing and conversion-policy changes remain separate work. ### Does this PR introduce any user-facing change? No API or intended conversion behavior change. The Python API snapshot remains identical across 108 exported classes. ### How was this patch tested? - 394 client library tests and the complete offline client Rust suite passed. Server-gated tests did not execute against a live Spark server. - 95 core library tests passed. - All client integration targets compiled in release mode, with and without `wasm-udf`; formatting and diff checks passed. - Fresh baseline and candidate release wheels built with the same toolchain and dependency cache. All 11 independent Python fixture comparisons passed, including five nested input cases that inspect actual LocalRelation IPC. Three Python width-error schema paths also passed. - New complete IPC regressions cover nested strings/binary, Unicode, embedded NULs, empty values, nulls, arrays/maps/structs, and decoding after source Rows are dropped. Expectations are reconstructed independently to avoid keeping shared source StructValues alive. Existing IPC regressions cover duplicate-name lookup and positional fallback. New error-order checks cover parent-before-child and map-key-before-value validation. - Workspace Clippy was attempted but is blocked by the missing `wasm32-unknown-unknown` target. Client all-target Clippy passes with the existing `clippy::approx_constant` errors allowed; existing warnings remain. The unmodified approximate constants are in other files. ### Remaining validation (draft) - Fresh CPU, wall-time and RSS comparisons for string/binary-heavy, nested and numeric guard inputs. Another benchmark is actively running in the workspace, so no performance measurements or performance acceptance are claimed here. - Live Spark parity / official Connect suite. No usable local Spark server distribution is available. - Rebase after #109 merges. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
