DB Tsai created SPARK-60687:
-------------------------------
Summary: Avoid numeric probes for exact list and tuple scalar
arrays in Spark Connect Rust
Key: SPARK-60687
URL: https://issues.apache.org/jira/browse/SPARK-60687
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
Frozen nested array tuple input10000x3 takes39.98ms Rust vs2.93ms Python;
pandas array42.07ms vs6.30ms. py_to_value performs Decimal/numeric probes
before recognizing built-in sequence arrays, repeated at every nesting level.
Investigate exact list/tuple dispatch before numeric probes with recursive
conversion, preserving subclasses and custom numeric protocols on fallback,
row-as-struct behavior, binary distinctions, nesting/nulls and errors. Separate
from PR111 exact numbers, PR113 row field names, PR114 borrowed encoding.
Compare on a baseline containing PR111 too to isolate incremental
array-container benefit; publish only if worthwhile.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
