[
https://issues.apache.org/jira/browse/SPARK-60717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60717:
-----------------------------------
Labels: pull-request-available (was: )
> Skip redundant normalization for canonical explicit int64 Arrow input in
> Spark Connect Rust
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-60717
> URL: https://issues.apache.org/jira/browse/SPARK-60717
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> After SPARK-60698/PR134 configuration batching, frozen integrated-v4 cached
> input remains6.131ms vsPython2.249ms at30000x3 and11.979ms vs6.708ms
> at60000x3. CPU-locked cold profiles attribute9-10ms instrumented construction
> to owned _arrow_input helper imports including7-8ms error hierarchy. Evaluate
> bypassing normalization only for already-canonical explicit Arrow int64
> tables: exact supported StructType, nullable LongType fields, identical
> names/int64 types, no schema or field metadata. Lazy-import general-path
> converters and errors within owned helper; do not edit vendored modules.
> Preserve fresh configuration reads, artifact uploads/counters and fallback
> casts/nullability/metadata/errors. Identical normalized schema, every cell
> and IPC bytes verified in feasibility for actual30k/60k inputs and nullable
> extrema; implementation performance not yet measured. Validate
> chunked/sliced/null input, rebinding/subclasses, metadata/nullability/cast
> fallbacks, canonical errors and cold import avoidance. Compare frozen
> baseline/candidate with identicalPR134 and unchanged harness5freshsamples,
> plus warm/10k/1M guards. Publish only repeatable meaningful gain. Distinct
> from PR121 integer pandas, PR126 SQLfunction import, PR134 config, PR135
> tuple conversion. User extension cutoff2026-10-29T17:33:31Z.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
