>From Ritik Raj <[email protected]>: Ritik Raj has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21291?usp=email )
Change subject: [ASTERIXDB-3768][CLOUD] Set explicit CRC32C checksum on S3 DeleteObjects ...................................................................... [ASTERIXDB-3768][CLOUD] Set explicit CRC32C checksum on S3 DeleteObjects - user model changes: no - storage format changes: no - interface changes: no When the S3 checksum behavior resolves to when_required, the SDK no longer adds an automatic integrity header (neither a flexible checksum nor the legacy Content-MD5) to DeleteObjects, which S3-compatible stores such as OCI Object Storage reject. Request an explicit CRC32C checksum on DeleteObjects in that case. Ext-ref: MB-72035 Change-Id: I49f6c6f1c44243f7e69f1ec55d5534a14019dcdf Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21291 Tested-by: Jenkins <[email protected]> Integration-Tests: Jenkins <[email protected]> Reviewed-by: Ritik Raj <[email protected]> Tested-by: Ritik Raj <[email protected]> Reviewed-by: Michael Blow <[email protected]> --- M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java 1 file changed, 11 insertions(+), 2 deletions(-) Approvals: Ritik Raj: Looks good to me, but someone else must approve; Verified Jenkins: Verified; Verified Anon. E. Moose #1000171: Michael Blow: Looks good to me, approved diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java index e22579d..edd43d4 100644 --- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java +++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3CloudClient.java @@ -57,6 +57,7 @@ import org.apache.hyracks.api.io.FileReference; import org.apache.hyracks.api.util.IoUtil; import org.apache.hyracks.cloud.io.ICloudProperties; +import org.apache.hyracks.cloud.io.S3ChecksumBehavior; import org.apache.hyracks.control.nc.io.IOManager; import org.apache.hyracks.util.annotations.ThreadSafe; import org.apache.logging.log4j.LogManager; @@ -76,6 +77,7 @@ import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.S3ClientBuilder; +import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm; import software.amazon.awssdk.services.s3.model.CopyObjectRequest; import software.amazon.awssdk.services.s3.model.Delete; import software.amazon.awssdk.services.s3.model.DeleteObjectRequest; @@ -280,8 +282,15 @@ } Delete delete = Delete.builder().objects(objectIdentifiers).build(); - DeleteObjectsRequest deleteReq = DeleteObjectsRequest.builder().bucket(bucket).delete(delete).build(); - DeleteObjectsResponse deleteObjectsResponse = s3Client.deleteObjects(deleteReq); + DeleteObjectsRequest.Builder deleteReqBuilder = + DeleteObjectsRequest.builder().bucket(bucket).delete(delete); + if (config.getChecksumBehavior() == S3ChecksumBehavior.WHEN_REQUIRED) { + // DeleteObjects needs a request-body integrity header; when_required suppresses the SDK's flexible + // checksum and it no longer adds Content-MD5, so request an explicit checksum for S3-compatible stores + // https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html + deleteReqBuilder.checksumAlgorithm(ChecksumAlgorithm.CRC32_C); + } + DeleteObjectsResponse deleteObjectsResponse = s3Client.deleteObjects(deleteReqBuilder.build()); if (deleteObjectsResponse.hasErrors()) { List<S3Error> deleteErrors = deleteObjectsResponse.errors(); for (S3Error s3Error : deleteErrors) { -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21291?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: asterixdb Gerrit-Branch: lumina Gerrit-Change-Id: I49f6c6f1c44243f7e69f1ec55d5534a14019dcdf Gerrit-Change-Number: 21291 Gerrit-PatchSet: 2 Gerrit-Owner: Ritik Raj <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Michael Blow <[email protected]> Gerrit-Reviewer: Ritik Raj <[email protected]>
