DB Tsai created SPARK-60717:
-------------------------------
Summary: Skip redundant normalization for canonical explicit int64
Arrow input in Spark Connect Rust
Key: SPARK-60717
URL: https://issues.apache.org/jira/browse/SPARK-60717
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
After SPARK-60698/PR134 configuration batching, frozen integrated-v4 cached
input remains6.131ms vsPython2.249ms at30000x3 and11.979ms vs6.708ms at60000x3.
CPU-locked cold profiles attribute9-10ms instrumented construction to owned
_arrow_input helper imports including7-8ms error hierarchy. Evaluate bypassing
normalization only for already-canonical explicit Arrow int64 tables: exact
supported StructType, nullable LongType fields, identical names/int64 types, no
schema or field metadata. Lazy-import general-path converters and errors within
owned helper; do not edit vendored modules. Preserve fresh configuration reads,
artifact uploads/counters and fallback casts/nullability/metadata/errors.
Identical normalized schema, every cell and IPC bytes verified in feasibility
for actual30k/60k inputs and nullable extrema; implementation performance not
yet measured. Validate chunked/sliced/null input, rebinding/subclasses,
metadata/nullability/cast fallbacks, canonical errors and cold import
avoidance. Compare frozen baseline/candidate with identicalPR134 and unchanged
harness5freshsamples, plus warm/10k/1M guards. Publish only repeatable
meaningful gain. Distinct from PR121 integer pandas, PR126 SQLfunction import,
PR134 config, PR135 tuple conversion. User extension cutoff2026-10-29T17:33:31Z.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
