[
https://issues.apache.org/jira/browse/SPARK-61214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-61214:
----------------------------
Fix Version/s: connect-rust-4.3.0
> Match named Row inference in customized pandas input for Spark Connect Rust
> ---------------------------------------------------------------------------
>
> Key: SPARK-61214
> URL: https://issues.apache.org/jira/browse/SPARK-61214
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Customized pandas DataFrames on the native normalization path currently infer
> named Row values as structs, while PySpark Connect treats inferred Arrow Row
> values positionally. For example, an inferred object-column Row(a=1, b=2)
> becomes struct<a:bigint,b:bigint> instead of array<bigint> containing [1, 2].
> Mixed-value conversion errors can also differ.
> Adapt Row-containing inferred columns before whole-column Arrow conversion,
> preserving arrays, names, nullability, timestamps, large-width types,
> dictionary inference and conversion-safety settings. Explicit struct schemas,
> ordinary local rows and the fallback without PyArrow retain their existing
> behavior. Document remaining normalization-hook and conversion limits.
> Validation: 2,682 offline Python passes including125requiredcases;8general
> pandas live passes;39paired JVM cases match (24schema/collection
> successes,15expected conversionerrors). Rust library/compile/fmt/build checks
> pass. Official rawtransport:2586passes641skips across87ordinarymodules
> plus15passes14skips in4cluster/resource modules, no unexpectedfailures.
> Advisory Clippy exits101 on125unchanged-master errors; no changed-source
> diagnostic. Independent fullpatch source review is complete. No global
> pandas/performance parity claimed.
> Generated-by: OpenAI Codex
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
