DB Tsai created SPARK-60716:
-------------------------------
Summary: Use Arrow columns for supported pandas date input in
Spark Connect Rust
Key: SPARK-60716
URL: https://issues.apache.org/jira/browse/SPARK-60716
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
The frozen combined client benchmark measures 10000x3 pandas date input
at4.10ms versus Python1.73ms after PR128 exact-date scalar optimization.
Evaluate a conservative columnar path for ordinary pandas DataFrames with
object columns containing only exact built-in datetime.date or None, explicit
all-DateType schemas and supported names. Build Arrow date32 columns and reuse
the existing importer rather than materializing pandas rows/Rust Row values.
Subclasses, datetime/Timestamp, mixed values, other null representations,
inference/DDL and unsupported schemas retain fallback. Distinct from
SPARK-60685/PR128 scalar detection and SPARK-60676/PR121 integer
/SPARK-60686/PR130 timestamp columnar paths. Publish only meaningful
incremental benefit against identical PR128/PR134-containing frozen baseline,
unchanged harness, independent every-cell outbound IPC/schema/nullability
validation, date boundary and subclass/error guards. No JVM needed.
User-authorized extension hard cutoff2026-10-29T17:33:31Z.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
