[
https://issues.apache.org/jira/browse/SPARK-60727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60727.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 207
[https://github.com/apache/spark-connect-rust/pull/207]
> Use Arrow columns for supported pandas float64 input in Spark Connect Rust
> --------------------------------------------------------------------------
>
> Key: SPARK-60727
> URL: https://issues.apache.org/jira/browse/SPARK-60727
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The existing pandas signed-int64 columnar path avoids object-row and Rust Row
> materialization, but homogeneous NumPy float64 DataFrames still take that
> general path. Evaluate a narrow exact pandas.DataFrame plus explicit native
> StructType of DoubleType fields path. Preserve NaN-to-null behavior,
> infinities, signed zero, field names, metadata, nullability, and
> unsupported/custom input fallback. Exclude nullable Float64 extension dtype
> initially. Verify native schema descriptors and public converter aliases. The
> frozen campaign catalog has no homogeneous float64 input workload, so use a
> clearly labelled external supplement with unchanged frozen harness, separate
> immutable baseline/candidate environments, documented hashes, and full
> received IPC schema/values/null-mask validators. Measure 10k/100k/1M rows and
> cold/warm behavior; publish only with a meaningful repeatable gain and no
> material fallback regression. This is distinct from SPARK-60676 signed-int64
> pandas input, SPARK-60686 timestamps, SPARK-60716 dates, SPARK-60720 strings,
> and generic numeric dispatch. No JVM required.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
