>From Ritik Raj <[email protected]>:
Ritik Raj has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21291?usp=email )
Change subject: [ASTERIXDB-3768][CLOUD] Set explicit CRC32C checksum on S3
DeleteObjects
......................................................................
[ASTERIXDB-3768][CLOUD] Set explicit CRC32C checksum on S3 DeleteObjects
- user model changes: no
- storage format changes: no
- interface changes: no
When the S3 checksum behavior resolves to when_required, the SDK no
longer adds an automatic integrity header (neither a flexible checksum
nor the legacy Content-MD5) to DeleteObjects, which S3-compatible stores
such as OCI Object Storage reject. Request an explicit CRC32C checksum
on DeleteObjects in that case.
Ext-ref: MB-72035
Change-Id: I49f6c6f1c44243f7e69f1ec55d5534a14019dcdf
---
M
asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java
1 file changed, 11 insertions(+), 2 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/91/21291/1
diff --git
a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java
b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java
index e22579d..edd43d4 100644
---
a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java
+++
b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java
@@ -57,6 +57,7 @@
import org.apache.hyracks.api.io.FileReference;
import org.apache.hyracks.api.util.IoUtil;
import org.apache.hyracks.cloud.io.ICloudProperties;
+import org.apache.hyracks.cloud.io.S3ChecksumBehavior;
import org.apache.hyracks.control.nc.io.IOManager;
import org.apache.hyracks.util.annotations.ThreadSafe;
import org.apache.logging.log4j.LogManager;
@@ -76,6 +77,7 @@
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3ClientBuilder;
+import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm;
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
import software.amazon.awssdk.services.s3.model.Delete;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
@@ -280,8 +282,15 @@
}
Delete delete =
Delete.builder().objects(objectIdentifiers).build();
- DeleteObjectsRequest deleteReq =
DeleteObjectsRequest.builder().bucket(bucket).delete(delete).build();
- DeleteObjectsResponse deleteObjectsResponse =
s3Client.deleteObjects(deleteReq);
+ DeleteObjectsRequest.Builder deleteReqBuilder =
+
DeleteObjectsRequest.builder().bucket(bucket).delete(delete);
+ if (config.getChecksumBehavior() ==
S3ChecksumBehavior.WHEN_REQUIRED) {
+ // DeleteObjects needs a request-body integrity header;
when_required suppresses the SDK's flexible
+ // checksum and it no longer adds Content-MD5, so request an
explicit checksum for S3-compatible stores
+ //
https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
+ deleteReqBuilder.checksumAlgorithm(ChecksumAlgorithm.CRC32_C);
+ }
+ DeleteObjectsResponse deleteObjectsResponse =
s3Client.deleteObjects(deleteReqBuilder.build());
if (deleteObjectsResponse.hasErrors()) {
List<S3Error> deleteErrors = deleteObjectsResponse.errors();
for (S3Error s3Error : deleteErrors) {
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21291?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: lumina
Gerrit-Change-Id: I49f6c6f1c44243f7e69f1ec55d5534a14019dcdf
Gerrit-Change-Number: 21291
Gerrit-PatchSet: 1
Gerrit-Owner: Ritik Raj <[email protected]>
