L. C. Hsieh created SPARK-61425:
-----------------------------------
Summary: Frame artifact uploads in one place and reject a chunked
artifact that changed size during its upload
Key: SPARK-61425
URL: https://issues.apache.org/jira/browse/SPARK-61425
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
In spark-connect-rust, crates/spark-connect-core/src/artifact.rs frames
artifact uploads in three places: build_artifact_request_stream (public,
eager), cache_artifact_requests (in-memory cache artifacts) and ArtifactFrames
(the lazy frames of a file upload). Each has its own state machine for the same
policy: batch small artifacts up to CHUNK_SIZE, and send a larger one as a
BeginChunk followed by Chunks. They produce the same requests today, but their
conditions are written differently and nothing keeps them the same.
A chunked artifact whose data is not of the size it declared is also not
detected. The size is taken before the data is read, and total_bytes and
num_chunks of the BeginChunk come from it. If the file is truncated in between
and nothing can be read, the frames end before any BeginChunk: the server
receives an empty stream, answers with success, and the upload returns Ok with
nothing uploaded. If it is shorter or longer, the frames contradict their
BeginChunk and no error is raised on the client.
Proposed: frame artifacts in one place, without changing the requests on the
wire, and make a chunked artifact whose data ends before or goes on past its
declared size an error that names the artifact and both sizes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]