DB Tsai created SPARK-61214:
-------------------------------
Summary: Match named Row inference in customized pandas input for
Spark Connect Rust
Key: SPARK-61214
URL: https://issues.apache.org/jira/browse/SPARK-61214
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Customized pandas DataFrames on the native normalization path currently infer
named Row values as structs, while PySpark Connect treats inferred Arrow Row
values positionally. For example, an inferred object-column Row(a=1, b=2)
becomes struct<a:bigint,b:bigint> instead of array<bigint> containing [1, 2].
Mixed-value conversion errors can also differ.
Adapt Row-containing inferred columns before whole-column Arrow conversion,
preserving arrays, names, nullability, timestamps, large-width types,
dictionary inference and conversion-safety settings. Explicit struct schemas,
ordinary local rows and the fallback without PyArrow retain their existing
behavior. Document remaining normalization-hook and conversion limits.
Validation: 2,682 offline Python passes including125requiredcases;8general
pandas live passes;39paired JVM cases match (24schema/collection
successes,15expected conversionerrors). Rust library/compile/fmt/build checks
pass. Official rawtransport:2586passes641skips across87ordinarymodules
plus15passes14skips in4cluster/resource modules, no unexpectedfailures.
Advisory Clippy exits101 on125unchanged-master errors; no changed-source
diagnostic. Independent fullpatch source review is complete. No global
pandas/performance parity claimed.
Generated-by: OpenAI Codex
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
