behnam-oneschema opened a new issue, #874: URL: https://github.com/apache/arrow-rs-object-store/issues/874
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** We'd like Google Cloud Storage to validate the integrity of objects written through `object_store`, the same way S3 can today. The AWS client supports this through `AmazonS3Builder::with_checksum_algorithm` / `aws_checksum_algorithm`. It sends `x-amz-checksum-sha256` or, since #633 (closes #611), `x-amz-checksum-crc64nvme` on single and part uploads. The GCP client sends no checksum on any write. `GoogleCloudStorageClient::put` and `put_part` (`src/gcp/client.rs`) send no `x-goog-hash` or `Content-MD5` header, so a body corrupted in transit or by a buggy intermediary is committed as-is. GCS recommends CRC32C for this (see [Data validation and change detection](https://cloud.google.com/storage/docs/hashes-etags) and the [end-to-end checksums announcement](https://cloud.google.com/blog/products/storage-data-transfer/enabling-end-to-end-checksums-in-cloud-storage)). #280 proposes a cross-provider MD5 `Checksum` attribute, but MD5 doesn't cover GCS multipart uploads. The GCS docs say: "Only CRC32C checksums can be used to verify the integrity of XML API multipart uploads. MD5 checksums aren't supported." The GCP client uses XML API multipart uploads for `put_multipart`, so large writes need CRC32C to be validated at all. **Describe the solution you'd like** Mirror the AWS checksum option in the GCP client: - Add a checksum option to `GoogleCloudStorageBuilder`, e.g. `with_checksum_algorithm(GcpChecksum::Crc32c)` plus a `google_checksum_algorithm` config key. It's off by default, as `aws_checksum_algorithm` is. - When enabled, compute CRC32C over the payload and send `x-goog-hash: crc32c=<base64 big-endian>` on: - `put` (XML API single-request `PUT /OBJECT`, [docs](https://cloud.google.com/storage/docs/xml-api/put-object-upload)) - `put_part` (`PUT /OBJECT?partNumber=N&uploadId=…`, [docs](https://cloud.google.com/storage/docs/xml-api/put-object-multipart): "you should specify a Content-MD5 header or a x-goog-hash header in the request so that Cloud Storage can check the uploaded data against the provided value") - GCS then rejects a mismatched upload with 400 before committing the object or part. `crc-fast` (added by #633, behind `aws-base`) already provides hardware-accelerated CRC-32/ISCSI (`crc32_iscsi`), which is CRC32C, so this needs no new dependency, only enabling `crc-fast` for the `gcp` feature too. #611 motivated CRC64NVME on S3 by the 25-30% upload overhead of SHA-256; CRC32C is likewise cheap enough to leave on for large uploads. **Describe alternatives you've considered** - **#280 (MD5 via `Content-MD5`)**: it works for GCS single-request uploads, but GCS doesn't accept MD5 for XML multipart parts, so large uploads stay unvalidated. It's complementary rather than a substitute. - **Validating after the write**: compare a locally computed CRC32C with the `x-goog-hash` response header, or HEAD the object afterwards. This only detects corruption after the object has already been committed or overwritten, and costs an extra round trip. - **Using `google-cloud-storage` (the official Rust SDK) directly**: that's what we do for our own direct GCS writes. It sends CRC32C when you call `precompute_checksums()`. But libraries built on `object_store`, such as Lance, DataFusion and Delta, can't switch clients. **Additional context** - Related: #611 / #633 (CRC64NVME for S3), #280 (MD5 attribute, open), #475 (S3 checksum configuration). - We hit this on Lance datasets stored on GCS (`gs://` URIs). Lance writes through `object_store`, so those writes are the only GCS writes in our stack that can't send a checksum. - Happy to help with a PR if the builder-option approach sounds right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
