DB Tsai created SPARK-60727:
-------------------------------
Summary: Use Arrow columns for supported pandas float64 input in
Spark Connect Rust
Key: SPARK-60727
URL: https://issues.apache.org/jira/browse/SPARK-60727
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
The existing pandas signed-int64 columnar path avoids object-row and Rust Row
materialization, but homogeneous NumPy float64 DataFrames still take that
general path. Evaluate a narrow exact pandas.DataFrame plus explicit native
StructType of DoubleType fields path. Preserve NaN-to-null behavior,
infinities, signed zero, field names, metadata, nullability, and
unsupported/custom input fallback. Exclude nullable Float64 extension dtype
initially. Verify native schema descriptors and public converter aliases. The
frozen campaign catalog has no homogeneous float64 input workload, so use a
clearly labelled external supplement with unchanged frozen harness, separate
immutable baseline/candidate environments, documented hashes, and full received
IPC schema/values/null-mask validators. Measure 10k/100k/1M rows and cold/warm
behavior; publish only with a meaningful repeatable gain and no material
fallback regression. This is distinct from SPARK-60676 signed-int64 pandas
input, SPARK-60686 timestamps, SPARK-60716 dates, SPARK-60720 strings, and
generic numeric dispatch. No JVM required.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
