This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 4d6a59e [SPARK-60942][CONNECT] Fix artifact attribution and async
hashing
4d6a59e is described below
commit 4d6a59ea26b02494948a7c6568e7afd310ece984
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 07:06:59 2026 +0900
[SPARK-60942][CONNECT] Fix artifact attribution and async hashing
### What changes were proposed in this pull request?
Populate `user_context` and `client_type` on every artifact upload frame
and include the same client type on artifact status requests. A shared
`artifact_user_context()` helper keeps upload and status identity construction
consistent. Hash small cache batches inline (up to 1 MiB and 128 blobs) and
move larger batches to a single Tokio blocking task, returning the original
buffers for lazy upload framing. Encode each SHA-256 digest into one
preallocated string and preserve panic propag [...]
Tracks [SPARK-60942](https://issues.apache.org/jira/browse/SPARK-60942),
following #155. The upload replay contract is documented on master by merged PR
#163.
Request attribution changes in this PR are **artifact-only**:
`AddArtifacts` frames and `ArtifactStatus`. `client_type` population for core
`ExecutePlan`, `Config`, `AnalyzePlan`, `Interrupt` and `ReleaseSession`
remains separate work. #140 currently adds broader typed `UserContext`
handling; it does not add the missing `client_type` fields.
**Merge coordination with #140:** both PRs change `client.rs` and artifact
attribution. Whichever merges second must rebase and reconcile the overlap,
retaining the shared upload helper so cache, file and named-file frames receive
the same context. #140 is already conflicting with master at the time of this
update.
### Why are the changes needed?
A configured user ID currently reaches gRPC metadata and artifact-status
bodies, but upload bodies omit it. Servers using protobuf identity can
associate these requests with different users. Synchronous hashing inside
`cache_artifacts` can also occupy an async worker for a large input; moving the
owned buffers to the blocking pool lets the caller yield without copying
payloads. Small batches avoid the thread hop, while the blob-count cap also
bounds inline work for empty or tiny blobs.
### Does this PR introduce _any_ user-facing change?
Yes. Cache, file and named-file uploads carry the configured user identity
and client agent in every protobuf frame. Hashing yields to the async executor
for batches above either inline limit. Hash names, ordering, deduplication,
framing and upload validation remain compatible.
Both
[PySpark](https://github.com/apache/spark/blob/98bc0212c94/python/pyspark/sql/connect/client/artifact.py#L175)
and
[Scala](https://github.com/apache/spark/blob/98bc0212c94/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala#L169)
send a `UserContext` on artifact requests. The `client_type` field follows
Scala; PySpark omits it. Without a configured user ID, this client sends a
present, empty `UserContext`, matching PySpark and existing arti [...]
Shared slices retain their source allocation until the last slice is
dropped. An already-started blocking hash retains its blob buffers until
hashing finishes, even if its caller is cancelled. This change preserves shared
buffers and does not add upload retries.
Session-ID consolidation in the request builders and configurable
`user_name` remain follow-ups. User context and client type strings are cloned
into each protobuf frame; at 32 KiB payload chunks that envelope cost is small,
and the payload buffers remain shared.
### How was this patch tested?
- Six wire regression cases cover cached blobs, ordinary files and named
files, with default and configured users. They inspect every
Batch/BeginChunk/Chunk and cache-status envelope and verify payload integrity.
The gRPC user-agent must equal `client_type` or start with `client_type`
followed by a space, allowing transport suffixes without depending on Tonic or
Spark/OS/Python formatting. All 58 Arrow mock tests passed, including these six
attribution cases.
- Tracked-buffer lifecycle tests verify completed blobs release storage
while later blobs remain queued, and retained frames keep their source valid
through iterator exhaustion or cancellation.
- Deterministic single-thread executor tests occupy the blocking pool to
distinguish inline completion from offloading at the byte and blob-count
boundaries. They check known hashes, input order, duplicates and original
buffer pointers/capacities; task-error tests verify panic propagation and
cancellation errors. An empty-input test uses lazy connection setup and needs
no server.
- 115 core unit tests, 414 API unit tests and 545 offline/mock Python tests
passed (8 existing skips). All integration targets compile with and without
`wasm-udf`; formatting passes.
- Against an isolated Spark 4.3 server: 10 native Arrow e2e tests, 34
Arrow-input tests and the official creation/collection parity modules passed.
API introspection found only the intentional `Column.to_plan` difference.
- Advisory workspace Clippy still reports existing errors in unchanged
files; all reported error files were verified byte-identical to master, with no
errors in the changed core crate.
Hash-only comparison on local ARM macOS using the same accelerated SHA
dependency (five alternating samples, excluding RPCs): a 256-byte batch changed
from 7.44 to 0.51 µs and 32 KiB from 21.90 to 13.85 µs. A 1 MiB inline batch
took 0.40 ms. Inputs above the cutoff still offload; large-input results varied
by a few percent. These measurements cover hashing and scheduling, not
end-to-end uploads.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
This pull request and its description were written by Isaac.
Closes #164 from dbtsai/codex/artifact-upload-context.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/artifact.rs | 45 +++++-
crates/spark-connect-core/src/client.rs | 224 +++++++++++++++++++++++++++---
python/tests/test_arrow_handoff.py | 126 ++++++++++++++++-
3 files changed, 369 insertions(+), 26 deletions(-)
diff --git a/crates/spark-connect-core/src/artifact.rs
b/crates/spark-connect-core/src/artifact.rs
index 4838cc6..ace2bba 100644
--- a/crates/spark-connect-core/src/artifact.rs
+++ b/crates/spark-connect-core/src/artifact.rs
@@ -96,6 +96,7 @@ impl NamedArtifact {
/// Frame cached in-memory artifacts as they are consumed by the upload stream.
/// Payload slices share the original allocation instead of copying each chunk.
+/// The allocation is freed when its last slice is dropped, not chunk by chunk.
pub(crate) fn cache_artifact_requests(
session_id: String,
artifacts: Vec<(String, Bytes)>,
@@ -533,6 +534,15 @@ mod tests {
#[test]
fn test_cache_requests_release_storage_on_drop() {
+ assert_cache_requests_release_storage(false);
+ }
+
+ #[test]
+ fn test_cache_requests_release_storage_on_exhaustion() {
+ assert_cache_requests_release_storage(true);
+ }
+
+ fn assert_cache_requests_release_storage(exhaust_stream: bool) {
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
@@ -555,6 +565,7 @@ mod tests {
let first_drops = Arc::new(AtomicUsize::new(0));
let second_drops = Arc::new(AtomicUsize::new(0));
+ let third_drops = Arc::new(AtomicUsize::new(0));
let first = Bytes::from_owner(TrackedData {
data: vec![1; CHUNK_SIZE + 1],
drops: first_drops.clone(),
@@ -563,18 +574,42 @@ mod tests {
data: vec![2; CHUNK_SIZE + 1],
drops: second_drops.clone(),
});
+ let third = Bytes::from_owner(TrackedData {
+ data: vec![3; CHUNK_SIZE + 1],
+ drops: third_drops.clone(),
+ });
let mut requests = cache_artifact_requests(
"session".to_string(),
- vec![("first".to_string(), first), ("second".to_string(), second)],
+ vec![
+ ("first".to_string(), first),
+ ("second".to_string(), second),
+ ("third".to_string(), third),
+ ],
);
- let first_request = requests.next().unwrap();
-
+ // Each CHUNK_SIZE + 1 blob emits BeginChunk followed by one Chunk.
+ drop(requests.next().unwrap());
+ let first_final_request = requests.next().unwrap();
assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+ drop(first_final_request);
+ // A consumed blob is released while later blobs remain queued.
+ assert_eq!(first_drops.load(Ordering::SeqCst), 1);
assert_eq!(second_drops.load(Ordering::SeqCst), 0);
+
+ let second_initial_request = requests.next().unwrap();
+ if exhaust_stream {
+ // Consume the second blob's Chunk and both frames of the third
blob.
+ drop(requests.next().unwrap());
+ drop(requests.next().unwrap());
+ drop(requests.next().unwrap());
+ assert!(requests.next().is_none());
+ }
drop(requests);
- assert_eq!(first_drops.load(Ordering::SeqCst), 0);
+ // An earlier frame keeps its allocation alive after either early drop
+ // or complete exhaustion of the iterator.
+ assert_eq!(second_drops.load(Ordering::SeqCst), 0);
+ assert_eq!(third_drops.load(Ordering::SeqCst), 1);
+ drop(second_initial_request);
assert_eq!(second_drops.load(Ordering::SeqCst), 1);
- drop(first_request);
assert_eq!(first_drops.load(Ordering::SeqCst), 1);
}
}
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index f0e03df..1e49a3f 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -52,6 +52,53 @@ async fn wait_ready(grpc: &mut tonic::client::Grpc<Channel>)
-> Result<()> {
}
}
+/// Avoid a blocking-pool round trip for small artifact batches.
+const ARTIFACT_HASH_INLINE_MAX_BYTES: usize = 1024 * 1024;
+/// Bound per-blob digest and encoding work even when every blob is empty.
+const ARTIFACT_HASH_INLINE_MAX_BLOBS: usize = 128;
+
+/// Hash up to 128 blobs totaling at most 1 MiB inline; offload larger batches.
+/// Returns the original buffers for upload framing. If the caller is dropped
after
+/// blocking hashing starts, the task retains the blob buffers until it
finishes.
+async fn hash_artifact_blobs(blobs: Vec<Vec<u8>>) -> Result<(Vec<Vec<u8>>,
Vec<String>)> {
+ use sha2::{Digest, Sha256};
+ use std::fmt::Write;
+
+ let hash_inline = blobs.len() <= ARTIFACT_HASH_INLINE_MAX_BLOBS
+ && blobs
+ .iter()
+ .try_fold(ARTIFACT_HASH_INLINE_MAX_BYTES, |remaining, blob| {
+ remaining.checked_sub(blob.len())
+ })
+ .is_some();
+ let hash = move || {
+ let hashes = blobs
+ .iter()
+ .map(|blob| {
+ let mut hash = String::with_capacity(64);
+ for byte in Sha256::digest(blob) {
+ write!(&mut hash, "{byte:02x}").expect("writing to a
String cannot fail");
+ }
+ hash
+ })
+ .collect();
+ (blobs, hashes)
+ };
+ if hash_inline {
+ return Ok(hash());
+ }
+ tokio::task::spawn_blocking(hash)
+ .await
+ .map_err(artifact_hashing_task_error)
+}
+
+fn artifact_hashing_task_error(error: tokio::task::JoinError) -> SparkError {
+ if error.is_panic() {
+ std::panic::resume_unwind(error.into_panic());
+ }
+ SparkError::connect_msg(format!("Artifact hashing task failed: {error}"))
+}
+
/// A Spark Connect client for communicating with a remote Spark server.
///
/// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient`.
@@ -849,23 +896,14 @@ impl SparkConnectClient {
/// Existing artifacts and duplicate blobs are uploaded only once.
pub async fn cache_artifacts(&self, blobs: Vec<Vec<u8>>) ->
Result<Vec<String>> {
use bytes::Bytes;
- use sha2::{Digest, Sha256};
use std::collections::HashSet;
- let hashes: Vec<String> = blobs
- .iter()
- .map(|blob| {
- Sha256::digest(blob)
- .iter()
- .map(|byte| format!("{byte:02x}"))
- .collect::<String>()
- })
- .collect();
+ if blobs.is_empty() {
+ return Ok(Vec::new());
+ }
+ let (blobs, hashes) = hash_artifact_blobs(blobs).await?;
let names: Vec<String> = hashes.iter().map(|hash|
format!("cache/{hash}")).collect();
let unique: HashSet<&str> = names.iter().map(String::as_str).collect();
- if unique.is_empty() {
- return Ok(hashes);
- }
let cached = self
.artifact_status(&unique.into_iter().collect::<Vec<_>>())
.await?;
@@ -886,7 +924,18 @@ impl SparkConnectClient {
Ok(hashes)
}
+ /// PySpark and Scala ArtifactManager both send a UserContext even when no
+ /// user ID is configured.
+ fn artifact_user_context(&self) -> UserContext {
+ UserContext {
+ user_id: self.user_id.clone().unwrap_or_default(),
+ ..Default::default()
+ }
+ }
+
/// Send artifact frames and validate the response.
+ /// Stamps user_context and client_type on every frame; all upload paths
must
+ /// go through here.
/// Cache uploads also require an acknowledgement for every requested
artifact.
/// The stream is consumed once; retries must rebuild the frames and
restart
/// AddArtifacts rather than resume a partially consumed stream.
@@ -895,6 +944,15 @@ impl SparkConnectClient {
requests: impl futures::Stream<Item =
spark_connect_proto::AddArtifactsRequest> + Send + 'static,
expected_cache_names: Option<&std::collections::HashSet<String>>,
) -> Result<()> {
+ use futures::StreamExt;
+
+ let user_context = self.artifact_user_context();
+ let client_type = self.user_agent.clone();
+ let requests = requests.map(move |mut request| {
+ request.user_context = Some(user_context.clone());
+ request.client_type = Some(client_type.clone());
+ request
+ });
let mut req = Request::new(requests);
self._attach_metadata(&mut req);
let response = match self.stub.clone().add_artifacts(req).await {
@@ -932,10 +990,8 @@ impl SparkConnectClient {
) -> Result<std::collections::HashMap<String, bool>> {
let mut request = ArtifactStatusesRequest::default();
request.session_id = self.session_id.clone();
- request.user_context = Some(UserContext {
- user_id: self.user_id.clone().unwrap_or_default(),
- ..Default::default()
- });
+ request.user_context = Some(self.artifact_user_context());
+ request.client_type = Some(self.user_agent.clone());
request.names = names.iter().map(|n| n.to_string()).collect();
let mut req = Request::new(request);
@@ -1169,6 +1225,140 @@ impl Drop for ReattachableResponseStream {
mod tests {
use super::*;
+ fn hash_with_busy_blocking_pool(blobs: Vec<Vec<u8>>) -> (bool,
Vec<String>) {
+ let runtime = tokio::runtime::Builder::new_current_thread()
+ .enable_all()
+ .max_blocking_threads(1)
+ .build()
+ .unwrap();
+ runtime.block_on(async {
+ let (started, ready) = tokio::sync::oneshot::channel();
+ let (release, wait) = std::sync::mpsc::channel();
+ let blocker = tokio::task::spawn_blocking(move || {
+ started.send(()).unwrap();
+ let _ = wait.recv();
+ });
+ ready.await.unwrap();
+
+ let allocations: Vec<_> = blobs
+ .iter()
+ .map(|blob| (blob.as_ptr(), blob.capacity()))
+ .collect();
+ let mut hashing = std::pin::pin!(hash_artifact_blobs(blobs));
+ // Occupying the only blocking thread makes Pending deterministic
and
+ // catches a regression that hashes large batches inline on the
first poll.
+ let first_poll = futures::poll!(&mut hashing);
+ let pending = first_poll.is_pending();
+ // Release the blocking pool before asserting so failures cannot
hang shutdown.
+ release.send(()).unwrap();
+ blocker.await.unwrap();
+ let (blobs, hashes) = match first_poll {
+ std::task::Poll::Ready(result) => result.unwrap(),
+ std::task::Poll::Pending => hashing.await.unwrap(),
+ };
+ for (blob, allocation) in blobs.iter().zip(allocations) {
+ assert_eq!((blob.as_ptr(), blob.capacity()), allocation);
+ }
+ (pending, hashes)
+ })
+ }
+
+ #[test]
+ fn test_small_artifact_hashing_stays_inline_and_preserves_buffers() {
+ let (pending, hashes) =
+ hash_with_busy_blocking_pool(vec![Vec::new(), b"abc".to_vec(),
b"abc".to_vec()]);
+ assert!(
+ !pending,
+ "small artifact batches must not wait for a blocking thread"
+ );
+ assert_eq!(
+ hashes,
+ [
+
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
+
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
+ ]
+ );
+ }
+
+ #[test]
+ fn test_artifact_hashing_inline_size_boundary() {
+ let (pending, hashes) =
+ hash_with_busy_blocking_pool(vec![vec![b'a';
ARTIFACT_HASH_INLINE_MAX_BYTES]]);
+ assert!(!pending, "the inline size limit is inclusive");
+ assert_eq!(
+ hashes,
+
["9bc1b2a288b26af7257a36277ae3816a7d4f16e89c1e7e77d0a5c48bad62b360"]
+ );
+ }
+
+ #[test]
+ fn test_large_artifact_hashing_yields_and_preserves_buffers() {
+ // Each blob fits the inline limit, but their combined size exceeds it
by one.
+ let (pending, hashes) = hash_with_busy_blocking_pool(vec![
+ vec![b'a'; ARTIFACT_HASH_INLINE_MAX_BYTES / 2],
+ vec![b'a'; ARTIFACT_HASH_INLINE_MAX_BYTES / 2 + 1],
+ ]);
+ assert!(
+ pending,
+ "large batches must yield while the blocking pool is busy"
+ );
+ assert_eq!(
+ hashes,
+ [
+
"85a84a75886e8a526dbec4e16e3375faa307b4aead79c9ed3264c0477a6f6eba",
+
"8d666ffa0196841cce7c504d43bf27e311775220d2490a23a2f984a43d901015",
+ ]
+ );
+ }
+
+ #[test]
+ fn test_artifact_hashing_inline_blob_count_boundary() {
+ for count in [
+ ARTIFACT_HASH_INLINE_MAX_BLOBS,
+ ARTIFACT_HASH_INLINE_MAX_BLOBS + 1,
+ ] {
+ let (pending, hashes) =
hash_with_busy_blocking_pool(vec![Vec::new(); count]);
+ assert_eq!(
+ pending,
+ count > ARTIFACT_HASH_INLINE_MAX_BLOBS,
+ "blob count: {count}"
+ );
+ assert_eq!(
+ hashes,
+
vec!["e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; count]
+ );
+ }
+ }
+
+ #[tokio::test]
+ async fn test_artifact_hashing_resumes_task_panics() {
+ let error = tokio::task::spawn_blocking(|| panic!("hashing panic"))
+ .await
+ .unwrap_err();
+ let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+ artifact_hashing_task_error(error)
+ }))
+ .unwrap_err();
+ assert_eq!(panic.downcast_ref::<&str>(), Some(&"hashing panic"));
+ }
+
+ #[tokio::test]
+ async fn test_artifact_hashing_reports_task_cancellation() {
+ let task = tokio::spawn(std::future::pending::<()>());
+ task.abort();
+ let error = artifact_hashing_task_error(task.await.unwrap_err());
+ assert!(error.to_string().contains("Artifact hashing task failed:"));
+ }
+
+ #[tokio::test]
+ async fn test_empty_cache_artifacts_needs_no_server() {
+ let builder = ChannelBuilder::parse("sc://localhost").unwrap();
+ // connect() is lazy; the empty upload must return without dialing a
server.
+ let client = SparkConnectClient::connect(&builder).await.unwrap();
+ assert!(client.cache_artifacts(Vec::new()).await.unwrap().is_empty());
+ }
+
#[tokio::test]
async fn test_session_id_generation() {
let builder = ChannelBuilder::parse("sc://localhost").unwrap();
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index f38e3e5..a379eb8 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -29,6 +29,7 @@ from contextlib import contextmanager
from datetime import datetime, timezone
from decimal import Decimal
from pathlib import Path
+from urllib.parse import quote
import grpc
import pandas as pd
@@ -398,7 +399,15 @@ def test_malformed_ipc_is_reported(protocol):
@contextmanager
-def _arrow_input_service(protocol, *, upload_error=None, **overrides):
+def _arrow_input_service(
+ protocol,
+ *,
+ upload_error=None,
+ user_id=None,
+ user_agent=None,
+ allow_file_artifacts=False,
+ **overrides,
+):
"""Resolve cache artifacts at a mock service with a 4 MiB receive limit."""
import hashlib
import zlib
@@ -415,9 +424,23 @@ def _arrow_input_service(protocol, *, upload_error=None,
**overrides):
}
config.update(overrides)
state = {"cache": {}, "uploads": 0, "relations": [], "message_sizes": [],
- "config_requests": []}
+ "config_requests": [], "upload_contexts": [], "status_contexts":
[]}
server_id = "00000000-0000-0000-0000-000000000001"
+ def attribution(request, context):
+ grpc_user_agent = dict(context.invocation_metadata()).get("user-agent")
+ assert grpc_user_agent, "artifact RPC is missing a non-empty
user-agent header"
+ # Keep only scalar envelope fields; do not retain another payload copy.
+ return {
+ "session_id": request.session_id,
+ "has_user_context": request.HasField("user_context"),
+ "user_id": request.user_context.user_id,
+ "user_name": request.user_context.user_name,
+ "has_client_type": request.HasField("client_type"),
+ "client_type": request.client_type,
+ "grpc_user_agent": grpc_user_agent,
+ }
+
def configure(raw, context):
request = protocol("ConfigRequest").FromString(raw)
operation = request.operation.WhichOneof("op_type")
@@ -430,6 +453,7 @@ def _arrow_input_service(protocol, *, upload_error=None,
**overrides):
def statuses(raw, context):
request = protocol("ArtifactStatusesRequest").FromString(raw)
+ state["status_contexts"].append(attribution(request, context))
return protocol("ArtifactStatusesResponse",
session_id=request.session_id,
statuses={name: dict(exists=name in state["cache"])
for name in
request.names}).SerializeToString()
@@ -440,7 +464,8 @@ def _arrow_input_service(protocol, *, upload_error=None,
**overrides):
session_id = None
def save(name, data):
- assert name == "cache/" + hashlib.sha256(data).hexdigest()
+ if not (allow_file_artifacts and name.startswith(("files/",
"forward_to_fs/"))):
+ assert name == "cache/" + hashlib.sha256(data).hexdigest()
assert name not in state["cache"], "cached artifacts should not be
uploaded again"
state["cache"][name] = bytes(data)
summaries.append(dict(name=name, is_crc_successful=True))
@@ -452,6 +477,9 @@ def _arrow_input_service(protocol, *, upload_error=None,
**overrides):
for raw in requests:
state["message_sizes"].append(len(raw))
request = protocol("AddArtifactsRequest").FromString(raw)
+ state["upload_contexts"].append({
+ **attribution(request, context), "payload":
request.WhichOneof("payload")
+ })
session_id = request.session_id
if request.HasField("batch"):
for artifact in request.batch.artifacts:
@@ -545,7 +573,11 @@ def _arrow_input_service(protocol, *, upload_error=None,
**overrides):
})])
port = server.add_insecure_port("127.0.0.1:0")
server.start()
- session =
SparkSession.builder.remote(f"sc://127.0.0.1:{port}").getOrCreate()
+ remote = f"sc://127.0.0.1:{port}/"
+ for key, value in [("user_id", user_id), ("user_agent", user_agent)]:
+ if value is not None:
+ remote += f";{key}={quote(value, safe='')}"
+ session = SparkSession.builder.remote(remote).getOrCreate()
try:
yield session, state
finally:
@@ -553,6 +585,92 @@ def _arrow_input_service(protocol, *, upload_error=None,
**overrides):
server.stop(0).wait(timeout=5)
+def _assert_artifact_contexts(contexts, *, session_id, user_id, user_agent):
+ assert contexts
+ client_types = {context["client_type"] for context in contexts}
+ assert len(client_types) == 1
+ client_type = client_types.pop()
+ assert client_type
+ assert client_type.startswith(user_agent)
+ for context in contexts:
+ grpc_user_agent = context["grpc_user_agent"]
+ assert grpc_user_agent == client_type or
grpc_user_agent.startswith(client_type + " ")
+ assert context["session_id"] == session_id
+ # PySpark's ArtifactManager sends a present context even without a
user ID.
+ assert context["has_user_context"]
+ assert context["user_id"] == user_id
+ assert context["user_name"] == ""
+ assert context["has_client_type"]
+
+
[email protected]("user_id,user_agent", [
+ (None, None),
+ ("[email protected]", "artifact wire test"),
+], ids=["default-user", "configured-user"])
+def test_cached_arrow_upload_attribution_on_every_frame(protocol, monkeypatch,
user_id, user_agent):
+ from pyspark.sql import types as T
+
+ # Exercise ChannelBuilder's default prefix when no URL or environment
value is set.
+ monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+ table = pa.table({"id": range(10_000)})
+ schema = T.StructType([T.StructField("id", T.LongType())])
+ with _arrow_input_service(protocol, user_id=user_id,
user_agent=user_agent, **{
+ "spark.sql.session.localRelationCacheThreshold": "0",
+ }) as (spark, state):
+ # The explicit schema is batched; the IPC data spans begin/chunk
frames.
+ spark.createDataFrame(table, schema)
+ assert {context["payload"] for context in state["upload_contexts"]} ==
{
+ "batch", "begin_chunk", "chunk"
+ }
+ assert state["status_contexts"]
+ _assert_artifact_contexts(
+ state["upload_contexts"] + state["status_contexts"],
+ session_id=spark.session_id,
+ user_id=user_id or "",
+ user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
+ )
+
+
[email protected]("user_id,user_agent", [
+ (None, None),
+ ("[email protected]", "file wire test"),
+], ids=["default-user", "configured-user"])
[email protected]("upload", ["file", "named"])
+def test_file_upload_attribution_on_every_frame(
+ protocol, monkeypatch, tmp_path, user_id, user_agent, upload
+):
+ monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+ small, large = tmp_path / "small.txt", tmp_path / "large.txt"
+ small.write_bytes(b"small artifact")
+ chunk_size = 32 * 1024 # Mirrors CHUNK_SIZE in
crates/spark-connect-core/src/artifact.rs.
+ large_size = chunk_size * 3 + 1 # BeginChunk followed by three Chunk
frames.
+ large.write_bytes(bytes(index % 256 for index in range(large_size)))
+ with _arrow_input_service(
+ protocol, user_id=user_id, user_agent=user_agent,
allow_file_artifacts=True
+ ) as (spark, state):
+ if upload == "file":
+ spark.addArtifacts(str(small), str(large))
+ artifact_names = ["files/small.txt", "files/large.txt"]
+ else:
+ destinations = ["/wire/small.txt", "/wire/large.txt"]
+ for path, destination in zip([small, large], destinations):
+ spark.copyFromLocalToFs(str(path), destination)
+ artifact_names = ["forward_to_fs/" + destination for destination
in destinations]
+ assert {context["payload"] for context in state["upload_contexts"]} ==
{
+ "batch", "begin_chunk", "chunk"
+ }
+ assert state["cache"][artifact_names[0]] == small.read_bytes()
+ assert state["cache"][artifact_names[1]] == large.read_bytes()
+ assert state["uploads"] == (1 if upload == "file" else 2)
+ assert not state["status_contexts"]
+ _assert_artifact_contexts(
+ state["upload_contexts"],
+ session_id=spark.session_id,
+ user_id=user_id or "",
+ user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
+ )
+
+
@pytest.mark.parametrize("cached", [False, True], ids=["inline", "cached"])
@pytest.mark.parametrize("explicit", [False, True], ids=["inferred",
"explicit"])
def test_arrow_input_schema_handoff(protocol, cached, explicit):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]