This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new d526bed [SPARK-60917][CONNECT] Reduce cached upload copies and
accelerate SHA-256
d526bed is described below
commit d526bed5fdae5db31fd6a9fdddbf68c63444000b
Author: DB Tsai <[email protected]>
AuthorDate: Thu Nov 5 18:36:26 2026 +0900
[SPARK-60917][CONNECT] Reduce cached upload copies and accelerate SHA-256
### What changes were proposed in this pull request?
[SPARK-60917](https://issues.apache.org/jira/browse/SPARK-60917) reduces
the client cost of uploading cached Arrow input. Enable the `sha2` assembly
feature on non-Windows aarch64 targets, retaining CPU feature detection and
software fallback. Move serialized cache blobs into shared `Bytes` and
construct upload frames as the transport consumes them.
The cache path retains the existing 32 KiB framing, small-artifact
batching, hash names, duplicate ordering, cache checks, CRC validation and
acknowledgement checks. The generic file-artifact builder keeps its existing
behavior.
### Why are the changes needed?
`sha2` 0.10 uses software SHA-256 on aarch64 without the assembly feature.
The cache upload path also clones in-memory IPC data when opening artifact
readers and eagerly copies all request payloads before starting the RPC. Shared
buffers and lazy framing remove those extra payload allocations.
In a local five-sample comparison with accelerated SHA in both native
clients, 136 MB Arrow input had median extra peak client RSS of **288.59 →
135.81 MiB** and median `createDataFrame` time of **404.43 → 341.43 ms**. A
separate isolated hashing probe improved from approximately **302 → 53 ms**
after enabling ARM acceleration. These experiments used macOS arm64, Spark
4.2.0 `local[2]` with a 3 GiB driver, Python 3.12.14 and PyArrow 26.
Input/session preparation and count/eight-column [...]
### Does this PR introduce _any_ user-facing change?
Lower temporary memory use during cached Arrow input upload, with faster
SHA-256 hashing on supported ARM CPUs. No public API change.
### How was this patch tested?
- Added transport unit tests for framing equivalence and boundaries, shared
payload storage, and cleanup when the stream is dropped.
- 102 transport and 410 Rust API release unit tests passed. All Rust client
integration targets compiled with and without `wasm-udf`; formatting and
whitespace checks passed.
- 420 offline/mock Python tests passed, including large cached uploads,
deduplication, CRC failures and missing acknowledgements; eight live-server
cases were skipped in that offline invocation.
- Against a checksum-verified Spark **4.3.0** distribution: 10 native Arrow
end-to-end tests, 34 live Arrow-input cases, and 33 official
creation/collection parity tests passed; two official cases were skipped. The
official gate was scoped to those two affected test modules.
- API introspection against official PySpark 4.3.0 found only the
intentionally absent `Column.to_plan` among the twelve inspected Connect
classes.
- Workspace Clippy still fails on existing lints in unchanged
API/Python/test files. Upstream master also fails; no Clippy errors point into
this PR's changed files, and all source files containing errors are
byte-identical to upstream.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (application version unavailable).
This pull request and its description were written by Isaac.
Closes #155 from dbtsai/codex/cache-upload-memory.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
Cargo.lock | 10 ++
crates/spark-connect-core/Cargo.toml | 4 +
crates/spark-connect-core/src/artifact.rs | 233 ++++++++++++++++++++++++++++++
crates/spark-connect-core/src/client.rs | 27 ++--
4 files changed, 261 insertions(+), 13 deletions(-)
diff --git a/Cargo.lock b/Cargo.lock
index 6238cb9..cd25ca0 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -3934,6 +3934,16 @@ dependencies = [
"cfg-if",
"cpufeatures 0.2.17",
"digest",
+ "sha2-asm",
+]
+
+[[package]]
+name = "sha2-asm"
+version = "0.6.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b845214d6175804686b2bd482bcffe96651bb2d1200742b712003504a2dac1ab"
+dependencies = [
+ "cc",
]
[[package]]
diff --git a/crates/spark-connect-core/Cargo.toml
b/crates/spark-connect-core/Cargo.toml
index 6cdf0e0..a8b61d0 100644
--- a/crates/spark-connect-core/Cargo.toml
+++ b/crates/spark-connect-core/Cargo.toml
@@ -28,5 +28,9 @@ prost-types.workspace = true
crc32fast = "1"
sha2 = "0.10"
+[target.'cfg(all(target_arch = "aarch64", not(target_os =
"windows")))'.dependencies]
+# sha2 0.10 requires this feature for ARM hardware acceleration; sha2-asm
excludes Windows.
+sha2 = { version = "0.10", features = ["asm"] }
+
[dev-dependencies]
tempfile = "3"
diff --git a/crates/spark-connect-core/src/artifact.rs
b/crates/spark-connect-core/src/artifact.rs
index 20affd2..4838cc6 100644
--- a/crates/spark-connect-core/src/artifact.rs
+++ b/crates/spark-connect-core/src/artifact.rs
@@ -94,6 +94,74 @@ impl NamedArtifact {
}
}
+/// Frame cached in-memory artifacts as they are consumed by the upload stream.
+/// Payload slices share the original allocation instead of copying each chunk.
+pub(crate) fn cache_artifact_requests(
+ session_id: String,
+ artifacts: Vec<(String, Bytes)>,
+) -> impl Iterator<Item = AddArtifactsRequest> + Send + 'static {
+ let mut artifacts = artifacts.into_iter().peekable();
+ let mut remaining_chunks = None::<Bytes>;
+
+ std::iter::from_fn(move || {
+ let payload = if let Some(mut data) = remaining_chunks.take() {
+ let chunk_data = data.split_to(CHUNK_SIZE.min(data.len()));
+ if !data.is_empty() {
+ remaining_chunks = Some(data);
+ }
+
add_artifacts_request::Payload::Chunk(add_artifacts_request::ArtifactChunk {
+ crc: compute_crc32(&chunk_data) as i64,
+ data: chunk_data,
+ })
+ } else {
+ let (_, next_data) = artifacts.peek()?;
+ if next_data.len() > CHUNK_SIZE {
+ let (name, mut data) = artifacts.next().unwrap();
+ let total_bytes = data.len() as i64;
+ let num_chunks = data.len().div_ceil(CHUNK_SIZE) as i64;
+ let chunk_data = data.split_to(CHUNK_SIZE);
+ remaining_chunks = Some(data);
+ add_artifacts_request::Payload::BeginChunk(
+ add_artifacts_request::BeginChunkedArtifact {
+ name,
+ total_bytes,
+ num_chunks,
+ initial_chunk:
Some(add_artifacts_request::ArtifactChunk {
+ crc: compute_crc32(&chunk_data) as i64,
+ data: chunk_data,
+ }),
+ },
+ )
+ } else {
+ let mut batch_artifacts = Vec::new();
+ let mut batch_size = 0;
+ while let Some((_, data)) = artifacts.peek() {
+ if data.len() > CHUNK_SIZE || batch_size + data.len() >
CHUNK_SIZE {
+ break;
+ }
+ let (name, data) = artifacts.next().unwrap();
+ batch_size += data.len();
+
batch_artifacts.push(add_artifacts_request::SingleChunkArtifact {
+ name,
+ data: Some(add_artifacts_request::ArtifactChunk {
+ crc: compute_crc32(&data) as i64,
+ data,
+ }),
+ });
+ }
+
add_artifacts_request::Payload::Batch(add_artifacts_request::Batch {
+ artifacts: batch_artifacts,
+ })
+ }
+ };
+ Some(AddArtifactsRequest {
+ session_id: session_id.clone(),
+ payload: Some(payload),
+ ..Default::default()
+ })
+ })
+}
+
/// Build a stream of AddArtifactsRequest messages for artifacts.
/// Small artifacts (<= CHUNK_SIZE) are batched together.
/// Large artifacts (> CHUNK_SIZE) are sent as chunked requests.
@@ -344,4 +412,169 @@ mod tests {
// Multiple small artifacts should be batched into one request
assert_eq!(requests.len(), 1);
}
+
+ #[test]
+ fn test_cache_requests_match_existing_framing() {
+ let sizes = [
+ 0,
+ 1,
+ CHUNK_SIZE - 1,
+ CHUNK_SIZE,
+ CHUNK_SIZE + 1,
+ CHUNK_SIZE * 2,
+ CHUNK_SIZE * 2 + 1,
+ CHUNK_SIZE / 2,
+ CHUNK_SIZE / 2 + 1,
+ 0,
+ ];
+ let artifacts: Vec<_> = sizes
+ .into_iter()
+ .enumerate()
+ .map(|(index, size)| {
+ (
+ format!("cache/artifact-{index}"),
+ Bytes::from(
+ (0..size)
+ .map(|offset| (offset + index) as u8)
+ .collect::<Vec<_>>(),
+ ),
+ )
+ })
+ .collect();
+ let eager_artifacts = artifacts
+ .iter()
+ .map(|(name, data)| {
+ NamedArtifact::new(name.clone(),
Box::new(InMemoryArtifact::new(data.to_vec())))
+ })
+ .collect();
+ let expected =
+ build_artifact_request_stream("session".to_string(),
eager_artifacts).unwrap();
+ let actual: Vec<_> = cache_artifact_requests("session".to_string(),
artifacts).collect();
+
+ assert_eq!(actual, expected);
+ assert!(cache_artifact_requests("session".to_string(), vec![])
+ .next()
+ .is_none());
+ }
+
+ #[test]
+ fn test_cache_requests_batch_boundaries() {
+ let artifacts = [0, CHUNK_SIZE, 0, 1, CHUNK_SIZE - 1, 1]
+ .into_iter()
+ .enumerate()
+ .map(|(index, size)| (index.to_string(), Bytes::from(vec![1;
size])))
+ .collect();
+ let requests: Vec<_> = cache_artifact_requests("session".to_string(),
artifacts).collect();
+ let batches: Vec<_> = requests
+ .into_iter()
+ .map(|request| {
+ let Some(add_artifacts_request::Payload::Batch(batch)) =
request.payload else {
+ panic!("small artifacts must use batch requests");
+ };
+ batch
+ })
+ .collect();
+
+ assert_eq!(batches.len(), 3);
+ assert_eq!(
+ batches[0]
+ .artifacts
+ .iter()
+ .map(|artifact| artifact.name.as_str())
+ .collect::<Vec<_>>(),
+ ["0", "1", "2"]
+ );
+ assert_eq!(batches[0].artifacts[0].data.as_ref().unwrap().crc, 0);
+ assert_eq!(batches[0].artifacts[2].data.as_ref().unwrap().crc, 0);
+ assert_eq!(batches[1].artifacts.len(), 2);
+ assert_eq!(batches[2].artifacts.len(), 1);
+ }
+
+ #[test]
+ fn test_cache_requests_share_payload_storage() {
+ let small = Bytes::from(vec![1; 7]);
+ let large = Bytes::from(vec![2; CHUNK_SIZE * 2 + 5]);
+ let next_large = Bytes::from(vec![3; CHUNK_SIZE + 1]);
+ let artifacts = vec![
+ ("small".to_string(), small.clone()),
+ ("large".to_string(), large.clone()),
+ ("next-large".to_string(), next_large.clone()),
+ ];
+ let requests: Vec<_> = cache_artifact_requests("session".to_string(),
artifacts).collect();
+ let chunks: Vec<_> = requests
+ .iter()
+ .flat_map(|request| match request.payload.as_ref().unwrap() {
+ add_artifacts_request::Payload::Batch(batch) => batch
+ .artifacts
+ .iter()
+ .map(|artifact| artifact.data.as_ref().unwrap())
+ .collect::<Vec<_>>(),
+ add_artifacts_request::Payload::BeginChunk(begin) => {
+ vec![begin.initial_chunk.as_ref().unwrap()]
+ }
+ add_artifacts_request::Payload::Chunk(chunk) => vec![chunk],
+ })
+ .collect();
+ let expected = [
+ (small.as_ptr(), small.len()),
+ (large.as_ptr(), CHUNK_SIZE),
+ (large.as_ptr().wrapping_add(CHUNK_SIZE), CHUNK_SIZE),
+ (large.as_ptr().wrapping_add(CHUNK_SIZE * 2), 5),
+ (next_large.as_ptr(), CHUNK_SIZE),
+ (next_large.as_ptr().wrapping_add(CHUNK_SIZE), 1),
+ ];
+
+ assert_eq!(chunks.len(), expected.len());
+ for (chunk, (pointer, len)) in chunks.into_iter().zip(expected) {
+ assert_eq!(chunk.data.as_ptr(), pointer);
+ assert_eq!(chunk.data.len(), len);
+ }
+ }
+
+ #[test]
+ fn test_cache_requests_release_storage_on_drop() {
+ use std::sync::atomic::{AtomicUsize, Ordering};
+ use std::sync::Arc;
+
+ struct TrackedData {
+ data: Vec<u8>,
+ drops: Arc<AtomicUsize>,
+ }
+
+ impl AsRef<[u8]> for TrackedData {
+ fn as_ref(&self) -> &[u8] {
+ &self.data
+ }
+ }
+
+ impl Drop for TrackedData {
+ fn drop(&mut self) {
+ self.drops.fetch_add(1, Ordering::SeqCst);
+ }
+ }
+
+ let first_drops = Arc::new(AtomicUsize::new(0));
+ let second_drops = Arc::new(AtomicUsize::new(0));
+ let first = Bytes::from_owner(TrackedData {
+ data: vec![1; CHUNK_SIZE + 1],
+ drops: first_drops.clone(),
+ });
+ let second = Bytes::from_owner(TrackedData {
+ data: vec![2; CHUNK_SIZE + 1],
+ drops: second_drops.clone(),
+ });
+ let mut requests = cache_artifact_requests(
+ "session".to_string(),
+ vec![("first".to_string(), first), ("second".to_string(), second)],
+ );
+ let first_request = requests.next().unwrap();
+
+ assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+ assert_eq!(second_drops.load(Ordering::SeqCst), 0);
+ drop(requests);
+ assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+ assert_eq!(second_drops.load(Ordering::SeqCst), 1);
+ drop(first_request);
+ assert_eq!(first_drops.load(Ordering::SeqCst), 1);
+ }
}
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 8534be3..0c8c88b 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -18,7 +18,9 @@ use spark_connect_proto::{
ReleaseExecuteRequest, ReleaseExecuteResponse, ReleaseSessionRequest,
UserContext,
};
-use crate::artifact::{build_artifact_request_stream, FileArtifact,
NamedArtifact};
+use crate::artifact::{
+ build_artifact_request_stream, cache_artifact_requests, FileArtifact,
NamedArtifact,
+};
use crate::channel::{ChannelBuilder, GRPC_MAX_MESSAGE_LENGTH_DEFAULT};
use crate::error::{Result, SparkError};
use crate::reattach::ExecutePlanResponseReattachableIterator;
@@ -744,7 +746,8 @@ impl SparkConnectClient {
// Build request stream with proper chunking/batching
let requests = build_artifact_request_stream(self.session_id.clone(),
artifacts)?;
- self.upload_artifact_requests(requests, None).await
+ self.upload_artifact_requests(futures::stream::iter(requests), None)
+ .await
}
/// Upload a single local file as an artifact under an explicit artifact
name
@@ -758,13 +761,14 @@ impl SparkConnectClient {
Box::new(FileArtifact::new(local_path)),
)];
let requests = build_artifact_request_stream(self.session_id.clone(),
artifacts)?;
- self.upload_artifact_requests(requests, None).await
+ self.upload_artifact_requests(futures::stream::iter(requests), None)
+ .await
}
/// Upload session-scoped cache blobs, returning SHA-256 hashes in input
order.
/// Existing artifacts and duplicate blobs are uploaded only once.
pub async fn cache_artifacts(&self, blobs: Vec<Vec<u8>>) ->
Result<Vec<String>> {
- use crate::artifact::InMemoryArtifact;
+ use bytes::Bytes;
use sha2::{Digest, Sha256};
use std::collections::HashSet;
@@ -789,18 +793,15 @@ impl SparkConnectClient {
let mut artifacts = Vec::new();
for (blob, name) in blobs.into_iter().zip(names) {
if !cached.get(&name).copied().unwrap_or(false) &&
seen.insert(name.clone()) {
- artifacts.push(NamedArtifact::new(
- name,
- Box::new(InMemoryArtifact::new(blob)),
- ));
+ artifacts.push((name, Bytes::from(blob)));
}
}
if artifacts.is_empty() {
return Ok(hashes);
}
- let expected: HashSet<String> = artifacts.iter().map(|a|
a.name.clone()).collect();
- let requests = build_artifact_request_stream(self.session_id.clone(),
artifacts)?;
- self.upload_artifact_requests(requests, Some(&expected))
+ let expected: HashSet<String> = artifacts.iter().map(|(name, _)|
name.clone()).collect();
+ let requests = cache_artifact_requests(self.session_id.clone(),
artifacts);
+ self.upload_artifact_requests(futures::stream::iter(requests),
Some(&expected))
.await?;
Ok(hashes)
}
@@ -809,10 +810,10 @@ impl SparkConnectClient {
/// Cache uploads also require an acknowledgement for every requested
artifact.
async fn upload_artifact_requests(
&self,
- requests: Vec<spark_connect_proto::AddArtifactsRequest>,
+ requests: impl futures::Stream<Item =
spark_connect_proto::AddArtifactsRequest> + Send + 'static,
expected_cache_names: Option<&std::collections::HashSet<String>>,
) -> Result<()> {
- let mut req = Request::new(futures::stream::iter(requests));
+ let mut req = Request::new(requests);
self._attach_metadata(&mut req);
let response = self
.stub
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]