This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 4d6a59e  [SPARK-60942][CONNECT] Fix artifact attribution and async 
hashing
4d6a59e is described below

commit 4d6a59ea26b02494948a7c6568e7afd310ece984
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 07:06:59 2026 +0900

    [SPARK-60942][CONNECT] Fix artifact attribution and async hashing
    
    ### What changes were proposed in this pull request?
    
    Populate `user_context` and `client_type` on every artifact upload frame 
and include the same client type on artifact status requests. A shared 
`artifact_user_context()` helper keeps upload and status identity construction 
consistent. Hash small cache batches inline (up to 1 MiB and 128 blobs) and 
move larger batches to a single Tokio blocking task, returning the original 
buffers for lazy upload framing. Encode each SHA-256 digest into one 
preallocated string and preserve panic propag [...]
    
    Tracks [SPARK-60942](https://issues.apache.org/jira/browse/SPARK-60942), 
following #155. The upload replay contract is documented on master by merged PR 
#163.
    
    Request attribution changes in this PR are **artifact-only**: 
`AddArtifacts` frames and `ArtifactStatus`. `client_type` population for core 
`ExecutePlan`, `Config`, `AnalyzePlan`, `Interrupt` and `ReleaseSession` 
remains separate work. #140 currently adds broader typed `UserContext` 
handling; it does not add the missing `client_type` fields.
    
    **Merge coordination with #140:** both PRs change `client.rs` and artifact 
attribution. Whichever merges second must rebase and reconcile the overlap, 
retaining the shared upload helper so cache, file and named-file frames receive 
the same context. #140 is already conflicting with master at the time of this 
update.
    
    ### Why are the changes needed?
    
    A configured user ID currently reaches gRPC metadata and artifact-status 
bodies, but upload bodies omit it. Servers using protobuf identity can 
associate these requests with different users. Synchronous hashing inside 
`cache_artifacts` can also occupy an async worker for a large input; moving the 
owned buffers to the blocking pool lets the caller yield without copying 
payloads. Small batches avoid the thread hop, while the blob-count cap also 
bounds inline work for empty or tiny blobs.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Cache, file and named-file uploads carry the configured user identity 
and client agent in every protobuf frame. Hashing yields to the async executor 
for batches above either inline limit. Hash names, ordering, deduplication, 
framing and upload validation remain compatible.
    
    Both 
[PySpark](https://github.com/apache/spark/blob/98bc0212c94/python/pyspark/sql/connect/client/artifact.py#L175)
 and 
[Scala](https://github.com/apache/spark/blob/98bc0212c94/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala#L169)
 send a `UserContext` on artifact requests. The `client_type` field follows 
Scala; PySpark omits it. Without a configured user ID, this client sends a 
present, empty `UserContext`, matching PySpark and existing arti [...]
    
    Shared slices retain their source allocation until the last slice is 
dropped. An already-started blocking hash retains its blob buffers until 
hashing finishes, even if its caller is cancelled. This change preserves shared 
buffers and does not add upload retries.
    
    Session-ID consolidation in the request builders and configurable 
`user_name` remain follow-ups. User context and client type strings are cloned 
into each protobuf frame; at 32 KiB payload chunks that envelope cost is small, 
and the payload buffers remain shared.
    
    ### How was this patch tested?
    
    - Six wire regression cases cover cached blobs, ordinary files and named 
files, with default and configured users. They inspect every 
Batch/BeginChunk/Chunk and cache-status envelope and verify payload integrity. 
The gRPC user-agent must equal `client_type` or start with `client_type` 
followed by a space, allowing transport suffixes without depending on Tonic or 
Spark/OS/Python formatting. All 58 Arrow mock tests passed, including these six 
attribution cases.
    - Tracked-buffer lifecycle tests verify completed blobs release storage 
while later blobs remain queued, and retained frames keep their source valid 
through iterator exhaustion or cancellation.
    - Deterministic single-thread executor tests occupy the blocking pool to 
distinguish inline completion from offloading at the byte and blob-count 
boundaries. They check known hashes, input order, duplicates and original 
buffer pointers/capacities; task-error tests verify panic propagation and 
cancellation errors. An empty-input test uses lazy connection setup and needs 
no server.
    - 115 core unit tests, 414 API unit tests and 545 offline/mock Python tests 
passed (8 existing skips). All integration targets compile with and without 
`wasm-udf`; formatting passes.
    - Against an isolated Spark 4.3 server: 10 native Arrow e2e tests, 34 
Arrow-input tests and the official creation/collection parity modules passed. 
API introspection found only the intentional `Column.to_plan` difference.
    - Advisory workspace Clippy still reports existing errors in unchanged 
files; all reported error files were verified byte-identical to master, with no 
errors in the changed core crate.
    
    Hash-only comparison on local ARM macOS using the same accelerated SHA 
dependency (five alternating samples, excluding RPCs): a 256-byte batch changed 
from 7.44 to 0.51 µs and 32 KiB from 21.90 to 13.85 µs. A 1 MiB inline batch 
took 0.40 ms. Inputs above the cutoff still offload; large-input results varied 
by a few percent. These measurements cover hashing and scheduling, not 
end-to-end uploads.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex
    
    This pull request and its description were written by Isaac.
    
    Closes #164 from dbtsai/codex/artifact-upload-context.
    
    Authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/artifact.rs |  45 +++++-
 crates/spark-connect-core/src/client.rs   | 224 +++++++++++++++++++++++++++---
 python/tests/test_arrow_handoff.py        | 126 ++++++++++++++++-
 3 files changed, 369 insertions(+), 26 deletions(-)

diff --git a/crates/spark-connect-core/src/artifact.rs 
b/crates/spark-connect-core/src/artifact.rs
index 4838cc6..ace2bba 100644
--- a/crates/spark-connect-core/src/artifact.rs
+++ b/crates/spark-connect-core/src/artifact.rs
@@ -96,6 +96,7 @@ impl NamedArtifact {
 
 /// Frame cached in-memory artifacts as they are consumed by the upload stream.
 /// Payload slices share the original allocation instead of copying each chunk.
+/// The allocation is freed when its last slice is dropped, not chunk by chunk.
 pub(crate) fn cache_artifact_requests(
     session_id: String,
     artifacts: Vec<(String, Bytes)>,
@@ -533,6 +534,15 @@ mod tests {
 
     #[test]
     fn test_cache_requests_release_storage_on_drop() {
+        assert_cache_requests_release_storage(false);
+    }
+
+    #[test]
+    fn test_cache_requests_release_storage_on_exhaustion() {
+        assert_cache_requests_release_storage(true);
+    }
+
+    fn assert_cache_requests_release_storage(exhaust_stream: bool) {
         use std::sync::atomic::{AtomicUsize, Ordering};
         use std::sync::Arc;
 
@@ -555,6 +565,7 @@ mod tests {
 
         let first_drops = Arc::new(AtomicUsize::new(0));
         let second_drops = Arc::new(AtomicUsize::new(0));
+        let third_drops = Arc::new(AtomicUsize::new(0));
         let first = Bytes::from_owner(TrackedData {
             data: vec![1; CHUNK_SIZE + 1],
             drops: first_drops.clone(),
@@ -563,18 +574,42 @@ mod tests {
             data: vec![2; CHUNK_SIZE + 1],
             drops: second_drops.clone(),
         });
+        let third = Bytes::from_owner(TrackedData {
+            data: vec![3; CHUNK_SIZE + 1],
+            drops: third_drops.clone(),
+        });
         let mut requests = cache_artifact_requests(
             "session".to_string(),
-            vec![("first".to_string(), first), ("second".to_string(), second)],
+            vec![
+                ("first".to_string(), first),
+                ("second".to_string(), second),
+                ("third".to_string(), third),
+            ],
         );
-        let first_request = requests.next().unwrap();
-
+        // Each CHUNK_SIZE + 1 blob emits BeginChunk followed by one Chunk.
+        drop(requests.next().unwrap());
+        let first_final_request = requests.next().unwrap();
         assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+        drop(first_final_request);
+        // A consumed blob is released while later blobs remain queued.
+        assert_eq!(first_drops.load(Ordering::SeqCst), 1);
         assert_eq!(second_drops.load(Ordering::SeqCst), 0);
+
+        let second_initial_request = requests.next().unwrap();
+        if exhaust_stream {
+            // Consume the second blob's Chunk and both frames of the third 
blob.
+            drop(requests.next().unwrap());
+            drop(requests.next().unwrap());
+            drop(requests.next().unwrap());
+            assert!(requests.next().is_none());
+        }
         drop(requests);
-        assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+        // An earlier frame keeps its allocation alive after either early drop
+        // or complete exhaustion of the iterator.
+        assert_eq!(second_drops.load(Ordering::SeqCst), 0);
+        assert_eq!(third_drops.load(Ordering::SeqCst), 1);
+        drop(second_initial_request);
         assert_eq!(second_drops.load(Ordering::SeqCst), 1);
-        drop(first_request);
         assert_eq!(first_drops.load(Ordering::SeqCst), 1);
     }
 }
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index f0e03df..1e49a3f 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -52,6 +52,53 @@ async fn wait_ready(grpc: &mut tonic::client::Grpc<Channel>) 
-> Result<()> {
     }
 }
 
+/// Avoid a blocking-pool round trip for small artifact batches.
+const ARTIFACT_HASH_INLINE_MAX_BYTES: usize = 1024 * 1024;
+/// Bound per-blob digest and encoding work even when every blob is empty.
+const ARTIFACT_HASH_INLINE_MAX_BLOBS: usize = 128;
+
+/// Hash up to 128 blobs totaling at most 1 MiB inline; offload larger batches.
+/// Returns the original buffers for upload framing. If the caller is dropped 
after
+/// blocking hashing starts, the task retains the blob buffers until it 
finishes.
+async fn hash_artifact_blobs(blobs: Vec<Vec<u8>>) -> Result<(Vec<Vec<u8>>, 
Vec<String>)> {
+    use sha2::{Digest, Sha256};
+    use std::fmt::Write;
+
+    let hash_inline = blobs.len() <= ARTIFACT_HASH_INLINE_MAX_BLOBS
+        && blobs
+            .iter()
+            .try_fold(ARTIFACT_HASH_INLINE_MAX_BYTES, |remaining, blob| {
+                remaining.checked_sub(blob.len())
+            })
+            .is_some();
+    let hash = move || {
+        let hashes = blobs
+            .iter()
+            .map(|blob| {
+                let mut hash = String::with_capacity(64);
+                for byte in Sha256::digest(blob) {
+                    write!(&mut hash, "{byte:02x}").expect("writing to a 
String cannot fail");
+                }
+                hash
+            })
+            .collect();
+        (blobs, hashes)
+    };
+    if hash_inline {
+        return Ok(hash());
+    }
+    tokio::task::spawn_blocking(hash)
+        .await
+        .map_err(artifact_hashing_task_error)
+}
+
+fn artifact_hashing_task_error(error: tokio::task::JoinError) -> SparkError {
+    if error.is_panic() {
+        std::panic::resume_unwind(error.into_panic());
+    }
+    SparkError::connect_msg(format!("Artifact hashing task failed: {error}"))
+}
+
 /// A Spark Connect client for communicating with a remote Spark server.
 ///
 /// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient`.
@@ -849,23 +896,14 @@ impl SparkConnectClient {
     /// Existing artifacts and duplicate blobs are uploaded only once.
     pub async fn cache_artifacts(&self, blobs: Vec<Vec<u8>>) -> 
Result<Vec<String>> {
         use bytes::Bytes;
-        use sha2::{Digest, Sha256};
         use std::collections::HashSet;
 
-        let hashes: Vec<String> = blobs
-            .iter()
-            .map(|blob| {
-                Sha256::digest(blob)
-                    .iter()
-                    .map(|byte| format!("{byte:02x}"))
-                    .collect::<String>()
-            })
-            .collect();
+        if blobs.is_empty() {
+            return Ok(Vec::new());
+        }
+        let (blobs, hashes) = hash_artifact_blobs(blobs).await?;
         let names: Vec<String> = hashes.iter().map(|hash| 
format!("cache/{hash}")).collect();
         let unique: HashSet<&str> = names.iter().map(String::as_str).collect();
-        if unique.is_empty() {
-            return Ok(hashes);
-        }
         let cached = self
             .artifact_status(&unique.into_iter().collect::<Vec<_>>())
             .await?;
@@ -886,7 +924,18 @@ impl SparkConnectClient {
         Ok(hashes)
     }
 
+    /// PySpark and Scala ArtifactManager both send a UserContext even when no
+    /// user ID is configured.
+    fn artifact_user_context(&self) -> UserContext {
+        UserContext {
+            user_id: self.user_id.clone().unwrap_or_default(),
+            ..Default::default()
+        }
+    }
+
     /// Send artifact frames and validate the response.
+    /// Stamps user_context and client_type on every frame; all upload paths 
must
+    /// go through here.
     /// Cache uploads also require an acknowledgement for every requested 
artifact.
     /// The stream is consumed once; retries must rebuild the frames and 
restart
     /// AddArtifacts rather than resume a partially consumed stream.
@@ -895,6 +944,15 @@ impl SparkConnectClient {
         requests: impl futures::Stream<Item = 
spark_connect_proto::AddArtifactsRequest> + Send + 'static,
         expected_cache_names: Option<&std::collections::HashSet<String>>,
     ) -> Result<()> {
+        use futures::StreamExt;
+
+        let user_context = self.artifact_user_context();
+        let client_type = self.user_agent.clone();
+        let requests = requests.map(move |mut request| {
+            request.user_context = Some(user_context.clone());
+            request.client_type = Some(client_type.clone());
+            request
+        });
         let mut req = Request::new(requests);
         self._attach_metadata(&mut req);
         let response = match self.stub.clone().add_artifacts(req).await {
@@ -932,10 +990,8 @@ impl SparkConnectClient {
     ) -> Result<std::collections::HashMap<String, bool>> {
         let mut request = ArtifactStatusesRequest::default();
         request.session_id = self.session_id.clone();
-        request.user_context = Some(UserContext {
-            user_id: self.user_id.clone().unwrap_or_default(),
-            ..Default::default()
-        });
+        request.user_context = Some(self.artifact_user_context());
+        request.client_type = Some(self.user_agent.clone());
         request.names = names.iter().map(|n| n.to_string()).collect();
 
         let mut req = Request::new(request);
@@ -1169,6 +1225,140 @@ impl Drop for ReattachableResponseStream {
 mod tests {
     use super::*;
 
+    fn hash_with_busy_blocking_pool(blobs: Vec<Vec<u8>>) -> (bool, 
Vec<String>) {
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .max_blocking_threads(1)
+            .build()
+            .unwrap();
+        runtime.block_on(async {
+            let (started, ready) = tokio::sync::oneshot::channel();
+            let (release, wait) = std::sync::mpsc::channel();
+            let blocker = tokio::task::spawn_blocking(move || {
+                started.send(()).unwrap();
+                let _ = wait.recv();
+            });
+            ready.await.unwrap();
+
+            let allocations: Vec<_> = blobs
+                .iter()
+                .map(|blob| (blob.as_ptr(), blob.capacity()))
+                .collect();
+            let mut hashing = std::pin::pin!(hash_artifact_blobs(blobs));
+            // Occupying the only blocking thread makes Pending deterministic 
and
+            // catches a regression that hashes large batches inline on the 
first poll.
+            let first_poll = futures::poll!(&mut hashing);
+            let pending = first_poll.is_pending();
+            // Release the blocking pool before asserting so failures cannot 
hang shutdown.
+            release.send(()).unwrap();
+            blocker.await.unwrap();
+            let (blobs, hashes) = match first_poll {
+                std::task::Poll::Ready(result) => result.unwrap(),
+                std::task::Poll::Pending => hashing.await.unwrap(),
+            };
+            for (blob, allocation) in blobs.iter().zip(allocations) {
+                assert_eq!((blob.as_ptr(), blob.capacity()), allocation);
+            }
+            (pending, hashes)
+        })
+    }
+
+    #[test]
+    fn test_small_artifact_hashing_stays_inline_and_preserves_buffers() {
+        let (pending, hashes) =
+            hash_with_busy_blocking_pool(vec![Vec::new(), b"abc".to_vec(), 
b"abc".to_vec()]);
+        assert!(
+            !pending,
+            "small artifact batches must not wait for a blocking thread"
+        );
+        assert_eq!(
+            hashes,
+            [
+                
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+                
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
+                
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
+            ]
+        );
+    }
+
+    #[test]
+    fn test_artifact_hashing_inline_size_boundary() {
+        let (pending, hashes) =
+            hash_with_busy_blocking_pool(vec![vec![b'a'; 
ARTIFACT_HASH_INLINE_MAX_BYTES]]);
+        assert!(!pending, "the inline size limit is inclusive");
+        assert_eq!(
+            hashes,
+            
["9bc1b2a288b26af7257a36277ae3816a7d4f16e89c1e7e77d0a5c48bad62b360"]
+        );
+    }
+
+    #[test]
+    fn test_large_artifact_hashing_yields_and_preserves_buffers() {
+        // Each blob fits the inline limit, but their combined size exceeds it 
by one.
+        let (pending, hashes) = hash_with_busy_blocking_pool(vec![
+            vec![b'a'; ARTIFACT_HASH_INLINE_MAX_BYTES / 2],
+            vec![b'a'; ARTIFACT_HASH_INLINE_MAX_BYTES / 2 + 1],
+        ]);
+        assert!(
+            pending,
+            "large batches must yield while the blocking pool is busy"
+        );
+        assert_eq!(
+            hashes,
+            [
+                
"85a84a75886e8a526dbec4e16e3375faa307b4aead79c9ed3264c0477a6f6eba",
+                
"8d666ffa0196841cce7c504d43bf27e311775220d2490a23a2f984a43d901015",
+            ]
+        );
+    }
+
+    #[test]
+    fn test_artifact_hashing_inline_blob_count_boundary() {
+        for count in [
+            ARTIFACT_HASH_INLINE_MAX_BLOBS,
+            ARTIFACT_HASH_INLINE_MAX_BLOBS + 1,
+        ] {
+            let (pending, hashes) = 
hash_with_busy_blocking_pool(vec![Vec::new(); count]);
+            assert_eq!(
+                pending,
+                count > ARTIFACT_HASH_INLINE_MAX_BLOBS,
+                "blob count: {count}"
+            );
+            assert_eq!(
+                hashes,
+                
vec!["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; count]
+            );
+        }
+    }
+
+    #[tokio::test]
+    async fn test_artifact_hashing_resumes_task_panics() {
+        let error = tokio::task::spawn_blocking(|| panic!("hashing panic"))
+            .await
+            .unwrap_err();
+        let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+            artifact_hashing_task_error(error)
+        }))
+        .unwrap_err();
+        assert_eq!(panic.downcast_ref::<&str>(), Some(&"hashing panic"));
+    }
+
+    #[tokio::test]
+    async fn test_artifact_hashing_reports_task_cancellation() {
+        let task = tokio::spawn(std::future::pending::<()>());
+        task.abort();
+        let error = artifact_hashing_task_error(task.await.unwrap_err());
+        assert!(error.to_string().contains("Artifact hashing task failed:"));
+    }
+
+    #[tokio::test]
+    async fn test_empty_cache_artifacts_needs_no_server() {
+        let builder = ChannelBuilder::parse("sc://localhost").unwrap();
+        // connect() is lazy; the empty upload must return without dialing a 
server.
+        let client = SparkConnectClient::connect(&builder).await.unwrap();
+        assert!(client.cache_artifacts(Vec::new()).await.unwrap().is_empty());
+    }
+
     #[tokio::test]
     async fn test_session_id_generation() {
         let builder = ChannelBuilder::parse("sc://localhost").unwrap();
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index f38e3e5..a379eb8 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -29,6 +29,7 @@ from contextlib import contextmanager
 from datetime import datetime, timezone
 from decimal import Decimal
 from pathlib import Path
+from urllib.parse import quote
 
 import grpc
 import pandas as pd
@@ -398,7 +399,15 @@ def test_malformed_ipc_is_reported(protocol):
 
 
 @contextmanager
-def _arrow_input_service(protocol, *, upload_error=None, **overrides):
+def _arrow_input_service(
+    protocol,
+    *,
+    upload_error=None,
+    user_id=None,
+    user_agent=None,
+    allow_file_artifacts=False,
+    **overrides,
+):
     """Resolve cache artifacts at a mock service with a 4 MiB receive limit."""
     import hashlib
     import zlib
@@ -415,9 +424,23 @@ def _arrow_input_service(protocol, *, upload_error=None, 
**overrides):
     }
     config.update(overrides)
     state = {"cache": {}, "uploads": 0, "relations": [], "message_sizes": [],
-             "config_requests": []}
+             "config_requests": [], "upload_contexts": [], "status_contexts": 
[]}
     server_id = "00000000-0000-0000-0000-000000000001"
 
+    def attribution(request, context):
+        grpc_user_agent = dict(context.invocation_metadata()).get("user-agent")
+        assert grpc_user_agent, "artifact RPC is missing a non-empty 
user-agent header"
+        # Keep only scalar envelope fields; do not retain another payload copy.
+        return {
+            "session_id": request.session_id,
+            "has_user_context": request.HasField("user_context"),
+            "user_id": request.user_context.user_id,
+            "user_name": request.user_context.user_name,
+            "has_client_type": request.HasField("client_type"),
+            "client_type": request.client_type,
+            "grpc_user_agent": grpc_user_agent,
+        }
+
     def configure(raw, context):
         request = protocol("ConfigRequest").FromString(raw)
         operation = request.operation.WhichOneof("op_type")
@@ -430,6 +453,7 @@ def _arrow_input_service(protocol, *, upload_error=None, 
**overrides):
 
     def statuses(raw, context):
         request = protocol("ArtifactStatusesRequest").FromString(raw)
+        state["status_contexts"].append(attribution(request, context))
         return protocol("ArtifactStatusesResponse", 
session_id=request.session_id,
                         statuses={name: dict(exists=name in state["cache"])
                                   for name in 
request.names}).SerializeToString()
@@ -440,7 +464,8 @@ def _arrow_input_service(protocol, *, upload_error=None, 
**overrides):
         session_id = None
 
         def save(name, data):
-            assert name == "cache/" + hashlib.sha256(data).hexdigest()
+            if not (allow_file_artifacts and name.startswith(("files/", 
"forward_to_fs/"))):
+                assert name == "cache/" + hashlib.sha256(data).hexdigest()
             assert name not in state["cache"], "cached artifacts should not be 
uploaded again"
             state["cache"][name] = bytes(data)
             summaries.append(dict(name=name, is_crc_successful=True))
@@ -452,6 +477,9 @@ def _arrow_input_service(protocol, *, upload_error=None, 
**overrides):
         for raw in requests:
             state["message_sizes"].append(len(raw))
             request = protocol("AddArtifactsRequest").FromString(raw)
+            state["upload_contexts"].append({
+                **attribution(request, context), "payload": 
request.WhichOneof("payload")
+            })
             session_id = request.session_id
             if request.HasField("batch"):
                 for artifact in request.batch.artifacts:
@@ -545,7 +573,11 @@ def _arrow_input_service(protocol, *, upload_error=None, 
**overrides):
             })])
         port = server.add_insecure_port("127.0.0.1:0")
         server.start()
-        session = 
SparkSession.builder.remote(f"sc://127.0.0.1:{port}").getOrCreate()
+        remote = f"sc://127.0.0.1:{port}/"
+        for key, value in [("user_id", user_id), ("user_agent", user_agent)]:
+            if value is not None:
+                remote += f";{key}={quote(value, safe='')}"
+        session = SparkSession.builder.remote(remote).getOrCreate()
         try:
             yield session, state
         finally:
@@ -553,6 +585,92 @@ def _arrow_input_service(protocol, *, upload_error=None, 
**overrides):
             server.stop(0).wait(timeout=5)
 
 
+def _assert_artifact_contexts(contexts, *, session_id, user_id, user_agent):
+    assert contexts
+    client_types = {context["client_type"] for context in contexts}
+    assert len(client_types) == 1
+    client_type = client_types.pop()
+    assert client_type
+    assert client_type.startswith(user_agent)
+    for context in contexts:
+        grpc_user_agent = context["grpc_user_agent"]
+        assert grpc_user_agent == client_type or 
grpc_user_agent.startswith(client_type + " ")
+        assert context["session_id"] == session_id
+        # PySpark's ArtifactManager sends a present context even without a 
user ID.
+        assert context["has_user_context"]
+        assert context["user_id"] == user_id
+        assert context["user_name"] == ""
+        assert context["has_client_type"]
+
+
[email protected]("user_id,user_agent", [
+    (None, None),
+    ("[email protected]", "artifact wire test"),
+], ids=["default-user", "configured-user"])
+def test_cached_arrow_upload_attribution_on_every_frame(protocol, monkeypatch, 
user_id, user_agent):
+    from pyspark.sql import types as T
+
+    # Exercise ChannelBuilder's default prefix when no URL or environment 
value is set.
+    monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+    table = pa.table({"id": range(10_000)})
+    schema = T.StructType([T.StructField("id", T.LongType())])
+    with _arrow_input_service(protocol, user_id=user_id, 
user_agent=user_agent, **{
+        "spark.sql.session.localRelationCacheThreshold": "0",
+    }) as (spark, state):
+        # The explicit schema is batched; the IPC data spans begin/chunk 
frames.
+        spark.createDataFrame(table, schema)
+        assert {context["payload"] for context in state["upload_contexts"]} == 
{
+            "batch", "begin_chunk", "chunk"
+        }
+        assert state["status_contexts"]
+        _assert_artifact_contexts(
+            state["upload_contexts"] + state["status_contexts"],
+            session_id=spark.session_id,
+            user_id=user_id or "",
+            user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
+        )
+
+
[email protected]("user_id,user_agent", [
+    (None, None),
+    ("[email protected]", "file wire test"),
+], ids=["default-user", "configured-user"])
[email protected]("upload", ["file", "named"])
+def test_file_upload_attribution_on_every_frame(
+    protocol, monkeypatch, tmp_path, user_id, user_agent, upload
+):
+    monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+    small, large = tmp_path / "small.txt", tmp_path / "large.txt"
+    small.write_bytes(b"small artifact")
+    chunk_size = 32 * 1024  # Mirrors CHUNK_SIZE in 
crates/spark-connect-core/src/artifact.rs.
+    large_size = chunk_size * 3 + 1  # BeginChunk followed by three Chunk 
frames.
+    large.write_bytes(bytes(index % 256 for index in range(large_size)))
+    with _arrow_input_service(
+        protocol, user_id=user_id, user_agent=user_agent, 
allow_file_artifacts=True
+    ) as (spark, state):
+        if upload == "file":
+            spark.addArtifacts(str(small), str(large))
+            artifact_names = ["files/small.txt", "files/large.txt"]
+        else:
+            destinations = ["/wire/small.txt", "/wire/large.txt"]
+            for path, destination in zip([small, large], destinations):
+                spark.copyFromLocalToFs(str(path), destination)
+            artifact_names = ["forward_to_fs/" + destination for destination 
in destinations]
+        assert {context["payload"] for context in state["upload_contexts"]} == 
{
+            "batch", "begin_chunk", "chunk"
+        }
+        assert state["cache"][artifact_names[0]] == small.read_bytes()
+        assert state["cache"][artifact_names[1]] == large.read_bytes()
+        assert state["uploads"] == (1 if upload == "file" else 2)
+        assert not state["status_contexts"]
+        _assert_artifact_contexts(
+            state["upload_contexts"],
+            session_id=spark.session_id,
+            user_id=user_id or "",
+            user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
+        )
+
+
 @pytest.mark.parametrize("cached", [False, True], ids=["inline", "cached"])
 @pytest.mark.parametrize("explicit", [False, True], ids=["inferred", 
"explicit"])
 def test_arrow_input_schema_handoff(protocol, cached, explicit):


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to