This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new d526bed  [SPARK-60917][CONNECT] Reduce cached upload copies and 
accelerate SHA-256
d526bed is described below

commit d526bed5fdae5db31fd6a9fdddbf68c63444000b
Author: DB Tsai <[email protected]>
AuthorDate: Thu Nov 5 18:36:26 2026 +0900

    [SPARK-60917][CONNECT] Reduce cached upload copies and accelerate SHA-256
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60917](https://issues.apache.org/jira/browse/SPARK-60917) reduces 
the client cost of uploading cached Arrow input. Enable the `sha2` assembly 
feature on non-Windows aarch64 targets, retaining CPU feature detection and 
software fallback. Move serialized cache blobs into shared `Bytes` and 
construct upload frames as the transport consumes them.
    
    The cache path retains the existing 32 KiB framing, small-artifact 
batching, hash names, duplicate ordering, cache checks, CRC validation and 
acknowledgement checks. The generic file-artifact builder keeps its existing 
behavior.
    
    ### Why are the changes needed?
    
    `sha2` 0.10 uses software SHA-256 on aarch64 without the assembly feature. 
The cache upload path also clones in-memory IPC data when opening artifact 
readers and eagerly copies all request payloads before starting the RPC. Shared 
buffers and lazy framing remove those extra payload allocations.
    
    In a local five-sample comparison with accelerated SHA in both native 
clients, 136 MB Arrow input had median extra peak client RSS of **288.59 → 
135.81 MiB** and median `createDataFrame` time of **404.43 → 341.43 ms**. A 
separate isolated hashing probe improved from approximately **302 → 53 ms** 
after enabling ARM acceleration. These experiments used macOS arm64, Spark 
4.2.0 `local[2]` with a 3 GiB driver, Python 3.12.14 and PyArrow 26. 
Input/session preparation and count/eight-column [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Lower temporary memory use during cached Arrow input upload, with faster 
SHA-256 hashing on supported ARM CPUs. No public API change.
    
    ### How was this patch tested?
    
    - Added transport unit tests for framing equivalence and boundaries, shared 
payload storage, and cleanup when the stream is dropped.
    - 102 transport and 410 Rust API release unit tests passed. All Rust client 
integration targets compiled with and without `wasm-udf`; formatting and 
whitespace checks passed.
    - 420 offline/mock Python tests passed, including large cached uploads, 
deduplication, CRC failures and missing acknowledgements; eight live-server 
cases were skipped in that offline invocation.
    - Against a checksum-verified Spark **4.3.0** distribution: 10 native Arrow 
end-to-end tests, 34 live Arrow-input cases, and 33 official 
creation/collection parity tests passed; two official cases were skipped. The 
official gate was scoped to those two affected test modules.
    - API introspection against official PySpark 4.3.0 found only the 
intentionally absent `Column.to_plan` among the twelve inspected Connect 
classes.
    - Workspace Clippy still fails on existing lints in unchanged 
API/Python/test files. Upstream master also fails; no Clippy errors point into 
this PR's changed files, and all source files containing errors are 
byte-identical to upstream.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (application version unavailable).
    
    This pull request and its description were written by Isaac.
    
    Closes #155 from dbtsai/codex/cache-upload-memory.
    
    Authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 Cargo.lock                                |  10 ++
 crates/spark-connect-core/Cargo.toml      |   4 +
 crates/spark-connect-core/src/artifact.rs | 233 ++++++++++++++++++++++++++++++
 crates/spark-connect-core/src/client.rs   |  27 ++--
 4 files changed, 261 insertions(+), 13 deletions(-)

diff --git a/Cargo.lock b/Cargo.lock
index 6238cb9..cd25ca0 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -3934,6 +3934,16 @@ dependencies = [
  "cfg-if",
  "cpufeatures 0.2.17",
  "digest",
+ "sha2-asm",
+]
+
+[[package]]
+name = "sha2-asm"
+version = "0.6.4"
+source = "registry+https://github.com/rust-lang/crates.io-index";
+checksum = "b845214d6175804686b2bd482bcffe96651bb2d1200742b712003504a2dac1ab"
+dependencies = [
+ "cc",
 ]
 
 [[package]]
diff --git a/crates/spark-connect-core/Cargo.toml 
b/crates/spark-connect-core/Cargo.toml
index 6cdf0e0..a8b61d0 100644
--- a/crates/spark-connect-core/Cargo.toml
+++ b/crates/spark-connect-core/Cargo.toml
@@ -28,5 +28,9 @@ prost-types.workspace = true
 crc32fast = "1"
 sha2 = "0.10"
 
+[target.'cfg(all(target_arch = "aarch64", not(target_os = 
"windows")))'.dependencies]
+# sha2 0.10 requires this feature for ARM hardware acceleration; sha2-asm 
excludes Windows.
+sha2 = { version = "0.10", features = ["asm"] }
+
 [dev-dependencies]
 tempfile = "3"
diff --git a/crates/spark-connect-core/src/artifact.rs 
b/crates/spark-connect-core/src/artifact.rs
index 20affd2..4838cc6 100644
--- a/crates/spark-connect-core/src/artifact.rs
+++ b/crates/spark-connect-core/src/artifact.rs
@@ -94,6 +94,74 @@ impl NamedArtifact {
     }
 }
 
+/// Frame cached in-memory artifacts as they are consumed by the upload stream.
+/// Payload slices share the original allocation instead of copying each chunk.
+pub(crate) fn cache_artifact_requests(
+    session_id: String,
+    artifacts: Vec<(String, Bytes)>,
+) -> impl Iterator<Item = AddArtifactsRequest> + Send + 'static {
+    let mut artifacts = artifacts.into_iter().peekable();
+    let mut remaining_chunks = None::<Bytes>;
+
+    std::iter::from_fn(move || {
+        let payload = if let Some(mut data) = remaining_chunks.take() {
+            let chunk_data = data.split_to(CHUNK_SIZE.min(data.len()));
+            if !data.is_empty() {
+                remaining_chunks = Some(data);
+            }
+            
add_artifacts_request::Payload::Chunk(add_artifacts_request::ArtifactChunk {
+                crc: compute_crc32(&chunk_data) as i64,
+                data: chunk_data,
+            })
+        } else {
+            let (_, next_data) = artifacts.peek()?;
+            if next_data.len() > CHUNK_SIZE {
+                let (name, mut data) = artifacts.next().unwrap();
+                let total_bytes = data.len() as i64;
+                let num_chunks = data.len().div_ceil(CHUNK_SIZE) as i64;
+                let chunk_data = data.split_to(CHUNK_SIZE);
+                remaining_chunks = Some(data);
+                add_artifacts_request::Payload::BeginChunk(
+                    add_artifacts_request::BeginChunkedArtifact {
+                        name,
+                        total_bytes,
+                        num_chunks,
+                        initial_chunk: 
Some(add_artifacts_request::ArtifactChunk {
+                            crc: compute_crc32(&chunk_data) as i64,
+                            data: chunk_data,
+                        }),
+                    },
+                )
+            } else {
+                let mut batch_artifacts = Vec::new();
+                let mut batch_size = 0;
+                while let Some((_, data)) = artifacts.peek() {
+                    if data.len() > CHUNK_SIZE || batch_size + data.len() > 
CHUNK_SIZE {
+                        break;
+                    }
+                    let (name, data) = artifacts.next().unwrap();
+                    batch_size += data.len();
+                    
batch_artifacts.push(add_artifacts_request::SingleChunkArtifact {
+                        name,
+                        data: Some(add_artifacts_request::ArtifactChunk {
+                            crc: compute_crc32(&data) as i64,
+                            data,
+                        }),
+                    });
+                }
+                
add_artifacts_request::Payload::Batch(add_artifacts_request::Batch {
+                    artifacts: batch_artifacts,
+                })
+            }
+        };
+        Some(AddArtifactsRequest {
+            session_id: session_id.clone(),
+            payload: Some(payload),
+            ..Default::default()
+        })
+    })
+}
+
 /// Build a stream of AddArtifactsRequest messages for artifacts.
 /// Small artifacts (<= CHUNK_SIZE) are batched together.
 /// Large artifacts (> CHUNK_SIZE) are sent as chunked requests.
@@ -344,4 +412,169 @@ mod tests {
         // Multiple small artifacts should be batched into one request
         assert_eq!(requests.len(), 1);
     }
+
+    #[test]
+    fn test_cache_requests_match_existing_framing() {
+        let sizes = [
+            0,
+            1,
+            CHUNK_SIZE - 1,
+            CHUNK_SIZE,
+            CHUNK_SIZE + 1,
+            CHUNK_SIZE * 2,
+            CHUNK_SIZE * 2 + 1,
+            CHUNK_SIZE / 2,
+            CHUNK_SIZE / 2 + 1,
+            0,
+        ];
+        let artifacts: Vec<_> = sizes
+            .into_iter()
+            .enumerate()
+            .map(|(index, size)| {
+                (
+                    format!("cache/artifact-{index}"),
+                    Bytes::from(
+                        (0..size)
+                            .map(|offset| (offset + index) as u8)
+                            .collect::<Vec<_>>(),
+                    ),
+                )
+            })
+            .collect();
+        let eager_artifacts = artifacts
+            .iter()
+            .map(|(name, data)| {
+                NamedArtifact::new(name.clone(), 
Box::new(InMemoryArtifact::new(data.to_vec())))
+            })
+            .collect();
+        let expected =
+            build_artifact_request_stream("session".to_string(), 
eager_artifacts).unwrap();
+        let actual: Vec<_> = cache_artifact_requests("session".to_string(), 
artifacts).collect();
+
+        assert_eq!(actual, expected);
+        assert!(cache_artifact_requests("session".to_string(), vec![])
+            .next()
+            .is_none());
+    }
+
+    #[test]
+    fn test_cache_requests_batch_boundaries() {
+        let artifacts = [0, CHUNK_SIZE, 0, 1, CHUNK_SIZE - 1, 1]
+            .into_iter()
+            .enumerate()
+            .map(|(index, size)| (index.to_string(), Bytes::from(vec![1; 
size])))
+            .collect();
+        let requests: Vec<_> = cache_artifact_requests("session".to_string(), 
artifacts).collect();
+        let batches: Vec<_> = requests
+            .into_iter()
+            .map(|request| {
+                let Some(add_artifacts_request::Payload::Batch(batch)) = 
request.payload else {
+                    panic!("small artifacts must use batch requests");
+                };
+                batch
+            })
+            .collect();
+
+        assert_eq!(batches.len(), 3);
+        assert_eq!(
+            batches[0]
+                .artifacts
+                .iter()
+                .map(|artifact| artifact.name.as_str())
+                .collect::<Vec<_>>(),
+            ["0", "1", "2"]
+        );
+        assert_eq!(batches[0].artifacts[0].data.as_ref().unwrap().crc, 0);
+        assert_eq!(batches[0].artifacts[2].data.as_ref().unwrap().crc, 0);
+        assert_eq!(batches[1].artifacts.len(), 2);
+        assert_eq!(batches[2].artifacts.len(), 1);
+    }
+
+    #[test]
+    fn test_cache_requests_share_payload_storage() {
+        let small = Bytes::from(vec![1; 7]);
+        let large = Bytes::from(vec![2; CHUNK_SIZE * 2 + 5]);
+        let next_large = Bytes::from(vec![3; CHUNK_SIZE + 1]);
+        let artifacts = vec![
+            ("small".to_string(), small.clone()),
+            ("large".to_string(), large.clone()),
+            ("next-large".to_string(), next_large.clone()),
+        ];
+        let requests: Vec<_> = cache_artifact_requests("session".to_string(), 
artifacts).collect();
+        let chunks: Vec<_> = requests
+            .iter()
+            .flat_map(|request| match request.payload.as_ref().unwrap() {
+                add_artifacts_request::Payload::Batch(batch) => batch
+                    .artifacts
+                    .iter()
+                    .map(|artifact| artifact.data.as_ref().unwrap())
+                    .collect::<Vec<_>>(),
+                add_artifacts_request::Payload::BeginChunk(begin) => {
+                    vec![begin.initial_chunk.as_ref().unwrap()]
+                }
+                add_artifacts_request::Payload::Chunk(chunk) => vec![chunk],
+            })
+            .collect();
+        let expected = [
+            (small.as_ptr(), small.len()),
+            (large.as_ptr(), CHUNK_SIZE),
+            (large.as_ptr().wrapping_add(CHUNK_SIZE), CHUNK_SIZE),
+            (large.as_ptr().wrapping_add(CHUNK_SIZE * 2), 5),
+            (next_large.as_ptr(), CHUNK_SIZE),
+            (next_large.as_ptr().wrapping_add(CHUNK_SIZE), 1),
+        ];
+
+        assert_eq!(chunks.len(), expected.len());
+        for (chunk, (pointer, len)) in chunks.into_iter().zip(expected) {
+            assert_eq!(chunk.data.as_ptr(), pointer);
+            assert_eq!(chunk.data.len(), len);
+        }
+    }
+
+    #[test]
+    fn test_cache_requests_release_storage_on_drop() {
+        use std::sync::atomic::{AtomicUsize, Ordering};
+        use std::sync::Arc;
+
+        struct TrackedData {
+            data: Vec<u8>,
+            drops: Arc<AtomicUsize>,
+        }
+
+        impl AsRef<[u8]> for TrackedData {
+            fn as_ref(&self) -> &[u8] {
+                &self.data
+            }
+        }
+
+        impl Drop for TrackedData {
+            fn drop(&mut self) {
+                self.drops.fetch_add(1, Ordering::SeqCst);
+            }
+        }
+
+        let first_drops = Arc::new(AtomicUsize::new(0));
+        let second_drops = Arc::new(AtomicUsize::new(0));
+        let first = Bytes::from_owner(TrackedData {
+            data: vec![1; CHUNK_SIZE + 1],
+            drops: first_drops.clone(),
+        });
+        let second = Bytes::from_owner(TrackedData {
+            data: vec![2; CHUNK_SIZE + 1],
+            drops: second_drops.clone(),
+        });
+        let mut requests = cache_artifact_requests(
+            "session".to_string(),
+            vec![("first".to_string(), first), ("second".to_string(), second)],
+        );
+        let first_request = requests.next().unwrap();
+
+        assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+        assert_eq!(second_drops.load(Ordering::SeqCst), 0);
+        drop(requests);
+        assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+        assert_eq!(second_drops.load(Ordering::SeqCst), 1);
+        drop(first_request);
+        assert_eq!(first_drops.load(Ordering::SeqCst), 1);
+    }
 }
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 8534be3..0c8c88b 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -18,7 +18,9 @@ use spark_connect_proto::{
     ReleaseExecuteRequest, ReleaseExecuteResponse, ReleaseSessionRequest, 
UserContext,
 };
 
-use crate::artifact::{build_artifact_request_stream, FileArtifact, 
NamedArtifact};
+use crate::artifact::{
+    build_artifact_request_stream, cache_artifact_requests, FileArtifact, 
NamedArtifact,
+};
 use crate::channel::{ChannelBuilder, GRPC_MAX_MESSAGE_LENGTH_DEFAULT};
 use crate::error::{Result, SparkError};
 use crate::reattach::ExecutePlanResponseReattachableIterator;
@@ -744,7 +746,8 @@ impl SparkConnectClient {
 
         // Build request stream with proper chunking/batching
         let requests = build_artifact_request_stream(self.session_id.clone(), 
artifacts)?;
-        self.upload_artifact_requests(requests, None).await
+        self.upload_artifact_requests(futures::stream::iter(requests), None)
+            .await
     }
 
     /// Upload a single local file as an artifact under an explicit artifact 
name
@@ -758,13 +761,14 @@ impl SparkConnectClient {
             Box::new(FileArtifact::new(local_path)),
         )];
         let requests = build_artifact_request_stream(self.session_id.clone(), 
artifacts)?;
-        self.upload_artifact_requests(requests, None).await
+        self.upload_artifact_requests(futures::stream::iter(requests), None)
+            .await
     }
 
     /// Upload session-scoped cache blobs, returning SHA-256 hashes in input 
order.
     /// Existing artifacts and duplicate blobs are uploaded only once.
     pub async fn cache_artifacts(&self, blobs: Vec<Vec<u8>>) -> 
Result<Vec<String>> {
-        use crate::artifact::InMemoryArtifact;
+        use bytes::Bytes;
         use sha2::{Digest, Sha256};
         use std::collections::HashSet;
 
@@ -789,18 +793,15 @@ impl SparkConnectClient {
         let mut artifacts = Vec::new();
         for (blob, name) in blobs.into_iter().zip(names) {
             if !cached.get(&name).copied().unwrap_or(false) && 
seen.insert(name.clone()) {
-                artifacts.push(NamedArtifact::new(
-                    name,
-                    Box::new(InMemoryArtifact::new(blob)),
-                ));
+                artifacts.push((name, Bytes::from(blob)));
             }
         }
         if artifacts.is_empty() {
             return Ok(hashes);
         }
-        let expected: HashSet<String> = artifacts.iter().map(|a| 
a.name.clone()).collect();
-        let requests = build_artifact_request_stream(self.session_id.clone(), 
artifacts)?;
-        self.upload_artifact_requests(requests, Some(&expected))
+        let expected: HashSet<String> = artifacts.iter().map(|(name, _)| 
name.clone()).collect();
+        let requests = cache_artifact_requests(self.session_id.clone(), 
artifacts);
+        self.upload_artifact_requests(futures::stream::iter(requests), 
Some(&expected))
             .await?;
         Ok(hashes)
     }
@@ -809,10 +810,10 @@ impl SparkConnectClient {
     /// Cache uploads also require an acknowledgement for every requested 
artifact.
     async fn upload_artifact_requests(
         &self,
-        requests: Vec<spark_connect_proto::AddArtifactsRequest>,
+        requests: impl futures::Stream<Item = 
spark_connect_proto::AddArtifactsRequest> + Send + 'static,
         expected_cache_names: Option<&std::collections::HashSet<String>>,
     ) -> Result<()> {
-        let mut req = Request::new(futures::stream::iter(requests));
+        let mut req = Request::new(requests);
         self._attach_metadata(&mut req);
         let response = self
             .stub


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to