[
https://issues.apache.org/jira/browse/SPARK-60716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60716:
-----------------------------------
Labels: pull-request-available (was: )
> Use Arrow columns for supported pandas date input in Spark Connect Rust
> -----------------------------------------------------------------------
>
> Key: SPARK-60716
> URL: https://issues.apache.org/jira/browse/SPARK-60716
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The frozen combined client benchmark measures 10000x3 pandas date input
> at4.10ms versus Python1.73ms after PR128 exact-date scalar optimization.
> Evaluate a conservative columnar path for ordinary pandas DataFrames with
> object columns containing only exact built-in datetime.date or None, explicit
> all-DateType schemas and supported names. Build Arrow date32 columns and
> reuse the existing importer rather than materializing pandas rows/Rust Row
> values. Subclasses, datetime/Timestamp, mixed values, other null
> representations, inference/DDL and unsupported schemas retain fallback.
> Distinct from SPARK-60685/PR128 scalar detection and SPARK-60676/PR121
> integer /SPARK-60686/PR130 timestamp columnar paths. Publish only meaningful
> incremental benefit against identical PR128/PR134-containing frozen baseline,
> unchanged harness, independent every-cell outbound IPC/schema/nullability
> validation, date boundary and subclass/error guards. No JVM needed.
> User-authorized extension hard cutoff2026-10-29T17:33:31Z.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
