DB Tsai created SPARK-60699:
-------------------------------
Summary: Build Arrow columns directly for exact integer tuple
input in Spark Connect Rust
Key: SPARK-60699
URL: https://issues.apache.org/jira/browse/SPARK-60699
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
The frozen combined client benchmark input-scale-tuples-1000000x3
remains140.09ms vsPython107.57ms after borrowed-value encoding/shared field
names/exact builtin dispatch. Current Python binding materializes a Vec<Value>
per tuple then a million Row objects before columnar Arrow construction.
Evaluate a conservative direct Arrow-column path limited to exact built-in
tuple rows and exact int/None cells with an explicit compatible LongType struct
schema, avoiding intermediate per-row allocations. Preserve
schema/name/nullability/error behavior, integer overflow, empty inputs,
mixed/subclass fallback and offline transport semantics. Require frozen paired
baseline/candidate benchmarks with identical existing input optimizations on
both sides and independent all-cell IPC/schema validators. Publish only with
meaningful measured improvement and no material guard regression. This is
distinct from PR111 exact-number detection, PR113 name sharing and PR114
borrowing existing Value rows; it removes intermediate row materialization only
for the supported narrow input. No JVM.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
