DB Tsai created SPARK-60728:
-------------------------------
Summary: Build Arrow lists directly for explicit integer array
tuple input in Spark Connect Rust
Key: SPARK-60728
URL: https://issues.apache.org/jira/browse/SPARK-60728
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
The unchanged frozen input-type-tuples-array-10000x3 workload takes about4.17ms
in the Rust client versus2.61ms in the Python reference. The general path
builds Python scalar arrays into nested Value::List and Row objects before
flattening them into Arrow columns. Evaluate direct Arrow List<Int64> column
construction for exact built-in outer list and tuple rows with an explicit
native StructType of ArrayType(LongType) fields, exact list/tuple cells, and
exact int64/None values. Preserve null list versus empty list versus null
elements, schema names/metadata/nullability and containsNull, overflow,
wrong-width/error ordering, late unsupported cells, and custom/subclass
fallback. Reuse the direct-column seam from SPARK-60699 without widening
generic scalar conversion. Compare unchanged frozen workload in separate
immutable baseline/candidate environments using full received IPC validators,
including fallback and guard overhead. This differs from SPARK-60687 generic
exact list/tuple scalar dispatch and SPARK-60699 scalar LongType tuple columns.
Publish only a meaningful repeatable gain. No JVM required.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
