Venkata Sai Akhil Gudesa created SPARK-53329:
------------------------------------------------
Summary: Process all sent-over artifacts rather than failing early
for modification attempts
Key: SPARK-53329
URL: https://issues.apache.org/jira/browse/SPARK-53329
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.1.0, 4.0.1
Reporter: Venkata Sai Akhil Gudesa
When a user sends multiple artifacts with the `addArtifacts` API, we process
each artifact one at a time on the server-side.
If the server detects the user attempting to modify an artifact (by overwriting
an existing artifact of the same path with a different byte sequence), an
exception is immediately thrown and artifact addition process is terminated.
This can be a bit wasteful as the server discards all other artifacts sent over
the wire regardless of their own status. Thus, an improvement can be made to
process all artifacts, catch any exceptions and rethrow them at the end.
This can potentially save bandwidth by avoiding multiple transfers. Further,
retry logic is unimpacted as we currently do NOT throw an error when attempting
to "overwrite" an existing file with the same bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
