chromafish opened a new pull request, #863: URL: https://github.com/apache/arrow-rs-object-store/pull/863
Closes #289 This adds support for S3 and `InMemory` only. GCS, Azure and `LocalFileSystem` return `NotImplemented`. Azure's Put Block List accepts `If-None-Match: *`, so it could be implemented, but I'm not very familiar with Azure Blob Storage. I can take a shot at it if it's wanted. # Rationale for this change Multipart uploads are currently always unconditional overwrites. S3 supports `If-None-Match: *` on `CompleteMultipartUpload`, and the S3 client already sends it internally for `S3CopyIfNotExists::Multipart`. It just isn't exposed through the public multipart API. # What changes are included in this PR? - `PutMultipartOptions` gains a `mode` field, allowing multipart uploads to use `PutMode::Create`. - `AmazonS3::put_multipart_opts` resolves the mode the same way `put_opts` does. On completion, it uses the same error handling as `put_opts`, which I extracted into a `map_create_error` function. - `MultipartStore::create_multipart_opts` returns `NotSupported` for any mode other than `Overwrite` (see limitations below). - `complete_multipart` no longer marks `Create` completions as idempotent, so they aren't retried on timeout, matching `put_opts` with `Create`. This is to avoid the scenario where the first attempt succeeded but its response was lost, so the retry would fail its `If-None-Match` and report the caller's own write as `AlreadyExists`. This also applies to the existing `copy_if_not_exists` with `S3CopyIfNotExists::Multipart`. **Limitations** - `mode` only takes effect through `ObjectStore::put_multipart_opts`. `MultipartStore` uploads are stateless between `create_multipart_opts` and `complete_multipart`, and on S3 the condition is a header on `CompleteMultipartUpload`, so there's nowhere to carry the mode to completion. Supporting it would need `MultipartStore::complete_multipart` to accept the mode, e.g. through a new `complete_multipart_opts` method. I've left that out since it adds a new API surface to the trait, but I can follow up with this if you'd like. - `PutMode::Update` isn't supported for multipart uploads. # Are there any user-facing changes? Yes. `PutMultipartOptions` has a new public field, so code that builds it with a struct literal and no `..Default::default()` will stop compiling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
