DB Tsai created SPARK-60917:
-------------------------------
Summary: Reduce cached artifact upload copies and accelerate
SHA-256 in Spark Connect Rust
Key: SPARK-60917
URL: https://issues.apache.org/jira/browse/SPARK-60917
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Following SPARK-60675, cached Arrow input upload has two avoidable client
costs: sha2 0.10 uses software SHA-256 on aarch64 without its asm feature, and
cache artifact framing copies serialized IPC buffers and eagerly materializes
all request frames.
Enable sha2 assembly acceleration on non-Windows aarch64 while retaining
runtime CPU detection/software fallback. Move cache blobs into shared Bytes and
yield the existing 32 KiB artifact frames lazily. Keep hashes, CRCs, cache
lookup/deduplication, acknowledgements and generic file-artifact framing
unchanged.
Local macOS arm64 measurements on 136 MB of Arrow input: isolated SHA hashing
improved from about 302 to 53 ms. In a separate five-sample framing comparison
with accelerated SHA in both native clients, median createDataFrame time
improved from 404.25 to 341.43 ms and sampled extra peak RSS fell from 288.59
to 135.81 MiB. Spark 4.2.0 local[2], 3 GiB driver, Python 3.12.14 and PyArrow
26; input preparation and count/eight-column-sum validation were outside the
timed interval. These are local creation measurements, not end-to-end query or
remote-network claims.
This issue covers SHA acceleration and lazy shared-buffer framing only.
Serialization/upload pipelining is excluded. SPARK-60898 tracks the independent
stable-plan-ID/server-plan-cache problem.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
