DB Tsai created SPARK-60720:
-------------------------------
Summary: Use Arrow columns for supported pandas string input in
Spark Connect Rust
Key: SPARK-60720
URL: https://issues.apache.org/jira/browse/SPARK-60720
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
Frozen combined10000x3 pandas string workload remains3.986ms vsPython1.823ms
after SPARK-60684/PR127 exact scalar string optimization. Evaluate conservative
columnar explicitStringType pandas object columns with exact str and
permittedNone cells, authentic native schema/type descriptors, ordinary
DataFrame and measured small-frame threshold. Build Arrow string columns and
reuse importer instead of pandas row materialization/Rust scalar conversion.
Subclasses, mutated descriptors/rebindings, inferred/DDL/mixed/extension dtypes
and unsupportedcollation/schema cases retain fallback. Lone-surrogate Arrow
conversion failures must return to old row path before importer, preserving
existing error behavior/order. Validate every IPCcell/schema/names,
Unicode/NUL/empty/all-null/nonnullable, metadata/collation/large-var-types and
fallbackhooks. Benchmark identicalPR127-containing frozenbaseline/candidate
with unchanged harness fivefreshsamples and full-stack supplement; publish only
meaningful repeatablegain. Distinct from
PR121int64/PR130timestamp/SPARK60716date. User-authorized revised
cutoff2026-10-29T18:19:36Z; noJVM/liveSpark needed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
