behnam-oneschema opened a new issue, #875: URL: https://github.com/apache/arrow-rs-object-store/issues/875
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** When `MicrosoftAzure` uploads data, nothing in the request lets Blob Storage validate the bytes it received. If the payload is corrupted between the client and the service, the service has no way to notice before it commits the blob. - `put` → Put Blob (`AzureClient::put_blob`) sends no `Content-MD5` or `x-ms-content-crc64`. - `put_multipart` → Put Block (`AzureClient::put_block`) sends no per-block checksum, and Put Block List (`AzureClient::put_block_list`) sends no `x-ms-blob-content-md5`. Blob Storage already supports server-side validation for both operations: - [Put Blob](https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob): "When this header is specified, the storage service checks the hash that has arrived against the one that was sent. If the two hashes don't match, the operation fails with error code 400 (Bad Request)." This refers to `Content-MD5`; `x-ms-content-crc64` is also accepted. - [Put Block](https://learn.microsoft.com/en-us/rest/api/storageservices/put-block): `Content-MD5` / `x-ms-content-crc64` are validated per block. - [Put Block List](https://learn.microsoft.com/en-us/rest/api/storageservices/put-block-list): `x-ms-blob-content-md5` is stored as the blob's MD5 but not validated, since the blocks were already validated. Storing it is still useful, because later full reads return it as `Content-MD5` (or `x-ms-blob-content-md5` for ranged reads), so readers can verify downloads. This is the Azure counterpart of #611 / #633 (S3, `aws_checksum_algorithm`) and #874 (GCS CRC32C). **Describe the solution you'd like** Add a `MicrosoftAzureBuilder` option, off by default, mirroring `aws_checksum_algorithm`, e.g. `azure_checksum_algorithm` / `AzureConfigKey::Checksum`, initially supporting `md5`. When it is set: - `put` computes MD5 over the `PutPayload` and sends it as `Content-MD5` on Put Blob. - `put_part` computes MD5 per part and sends `Content-MD5` on Put Block, so every block is validated before it is staged. - Optionally, `complete` sends the whole-object MD5 as `x-ms-blob-content-md5` on Put Block List, so the stored blob has a checksum for readers. This needs the parts' digests combined in order. MD5 can't be combined from per-part digests, so this either needs an incremental hasher across `put_part` calls or can be left out of a first version. `md-5` is already an optional dependency (enabled today by `aws-base`), so this only needs `azure-base` to enable it too; no new crate. **Describe alternatives you've considered** - #280 proposes a caller-supplied `Checksum` attribute (MD5 via `Content-MD5`) on `PutOptions`. It would cover single `put` on Azure, but not per-block validation in `put_multipart`, and it requires every caller (e.g. Lance) to compute and pass the checksum. A builder option makes validation automatic for all writes, which is how #633 solved it for S3. The two are complementary. - `x-ms-content-crc64` would be cheaper than MD5, but Azure's CRC64 variant would need to be confirmed against an existing CRC implementation first. MD5 works today. **Additional context** We hit this through Lance (`az://` datasets), which writes through `object_store`. We added MD5 validation to our own direct Azure SDK paths, but Lance writes remain unvalidated until `object_store` supports it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
