DB Tsai created SPARK-60942:
-------------------------------
Summary: Fix artifact request attribution and async hashing in
Spark Connect Rust
Key: SPARK-60942
URL: https://issues.apache.org/jira/browse/SPARK-60942
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Spark Connect Rust artifact upload frames are built with the default protobuf
user_context and client_type, even when ChannelBuilder has a configured user ID
and agent. gRPC headers carry these values, but servers that use protobuf
request identity cannot associate uploads with the same user as artifact status
and query requests. Every Batch, BeginChunk and Chunk frame needs the
configured context. ArtifactStatusesRequest should carry the same client_type.
cache_artifacts also computes SHA-256 for all IPC blobs synchronously before
its first await. A large input can occupy an async worker for tens of
milliseconds even with accelerated SHA. Hash owned blobs in one Tokio blocking
task, then return the original allocations for lazy upload framing. Preserve
hash order, duplicate handling, cache checks, CRC and acknowledgements.
The shared Bytes slices should continue to avoid copying. Their source
allocation is retained until its last slice is dropped; document this expected
lifetime. The one-shot stream replay contract is documented separately in PR
#163.
Regression coverage should inspect every upload frame and status request for
configured/default users, cover cache/file/named uploads, and deterministically
prove hashing yields while retaining the original payload buffers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
