>From Michael Blow <[email protected]>: Michael Blow has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21646?usp=email )
Change subject: [NO ISSUE][MISC] Replace io.findify S3 mock with adobe's ...................................................................... [NO ISSUE][MISC] Replace io.findify S3 mock with adobe's - com.adobe.testing:s3mock 5.2.0, replacing io.findify:s3mock_2.12 0.2.6 and the com.typesafe.akka:akka-http-core_2.12 10.1.15 that was pinned against it io.findify's mock has had no release since 2019, and it mishandles the upload framing the sdk has used since 2.30: the put appears to succeed and the object reads back empty. That is why the test client asked for checksums only where the operation required them; the replacement handles them, so the request is dropped again. Two behaviours differ from the old mock, both of them the real service's: - a bucket with content cannot be deleted, so the bucket is now emptied before it is dropped - the message for a missing bucket ends in a period The mock is a spring boot application, and its starter brings logback and log4j-to-slf4j along; both fight the log4j2 binding the tests run with, and spring refuses to start when it finds logback behind another binding, so spring-boot-starter-logging is excluded. Scala and akka leave the test classpath entirely, and with them the akka-http pin: 10.2 onwards is BUSL-1.1 rather than Apache-2.0, so that dependency could never be advanced. AwsS3ExternalDatasetTest passes. Ext-ref: MB-73268 Co-Authored-By: Claude Opus 5 <[email protected]> Change-Id: I974678973e4fd50dcfc3adfedff3614d5d60c635 --- M asterixdb/asterix-app/pom.xml M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/aws/AwsS3ExternalDatasetTest.java M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml M asterixdb/pom.xml 4 files changed, 32 insertions(+), 34 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/46/21646/1 diff --git a/asterixdb/asterix-app/pom.xml b/asterixdb/asterix-app/pom.xml index f760373..c83ca2d 100644 --- a/asterixdb/asterix-app/pom.xml +++ b/asterixdb/asterix-app/pom.xml @@ -892,14 +892,8 @@ </dependency> <!-- Mock for AWS S3 --> <dependency> - <groupId>io.findify</groupId> - <artifactId>s3mock_2.12</artifactId> - <scope>test</scope> - </dependency> - <!-- Needed for the s3 mock --> - <dependency> - <groupId>com.typesafe.akka</groupId> - <artifactId>akka-http-core_2.12</artifactId> + <groupId>com.adobe.testing</groupId> + <artifactId>s3mock</artifactId> <scope>test</scope> </dependency> <!-- Azure --> diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/aws/AwsS3ExternalDatasetTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/aws/AwsS3ExternalDatasetTest.java index 330d27b..738e19f 100644 --- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/aws/AwsS3ExternalDatasetTest.java +++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/aws/AwsS3ExternalDatasetTest.java @@ -62,16 +62,17 @@ import org.junit.runners.Parameterized; import org.junit.runners.Parameterized.Parameters; -import io.findify.s3mock.S3Mock; +import com.adobe.testing.s3mock.S3MockApplication; + import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider; -import software.amazon.awssdk.core.checksums.RequestChecksumCalculation; -import software.amazon.awssdk.core.checksums.ResponseChecksumValidation; import software.amazon.awssdk.core.sync.RequestBody; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.S3ClientBuilder; import software.amazon.awssdk.services.s3.model.CreateBucketRequest; import software.amazon.awssdk.services.s3.model.DeleteBucketRequest; +import software.amazon.awssdk.services.s3.model.DeleteObjectRequest; +import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; import software.amazon.awssdk.services.s3.model.NoSuchBucketException; import software.amazon.awssdk.services.s3.model.PutObjectRequest; @@ -110,7 +111,7 @@ private static final DeleteBucketRequest.Builder DELETE_BUCKET_BUILDER = DeleteBucketRequest.builder(); private static final PutObjectRequest.Builder PUT_OBJECT_BUILDER = PutObjectRequest.builder(); - private static S3Mock s3MockServer; + private static S3MockApplication s3MockServer; private static S3Client client; protected TestCaseContext tcCtx; @@ -151,7 +152,7 @@ client.close(); } if (s3MockServer != null) { - s3MockServer.shutdown(); + s3MockServer.stop(); } LOGGER.info("S3 mock down and client shut down successfully"); } @@ -193,8 +194,12 @@ private static void startAwsS3MockServer() { // Starting S3 mock server to be used instead of real S3 server LOGGER.info("Starting S3 mock server"); - s3MockServer = new S3Mock.Builder().withPort(MOCK_SERVER_PORT).withInMemoryBackend().build(); - s3MockServer.start(); + // start() mutates the map it is handed, so it cannot be an immutable one + Map<String, Object> mockProperties = new HashMap<>(); + mockProperties.put(S3MockApplication.PROP_HTTP_PORT, MOCK_SERVER_PORT); + mockProperties.put(S3MockApplication.PROP_HTTPS_PORT, S3MockApplication.RANDOM_PORT); + mockProperties.put(S3MockApplication.PROP_SILENT, true); + s3MockServer = S3MockApplication.start(mockProperties); LOGGER.info("S3 mock server started successfully"); // Create a client and add some files to the S3 mock server @@ -202,11 +207,7 @@ S3ClientBuilder builder = S3Client.builder(); URI endpoint = URI.create(MOCK_SERVER_HOSTNAME); // endpoint pointing to S3 mock server builder.region(Region.of(MOCK_SERVER_REGION)).credentialsProvider(AnonymousCredentialsProvider.create()) - .endpointOverride(endpoint) - // the mock server does not understand the aws-chunked framing used when the sdk computes - // upload checksums (its default since 2.30); objects uploaded that way read back empty - .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED) - .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED); + .endpointOverride(endpoint); client = builder.build(); client.createBucket(CreateBucketRequest.builder().bucket(PLAYGROUND_CONTAINER).build()); client.createBucket(CreateBucketRequest.builder().bucket(FIXED_DATA_CONTAINER).build()); @@ -298,6 +299,10 @@ LOGGER.info("Dropping bucket " + bucketName); try { + // the bucket has to be emptied first, a bucket with content cannot be deleted + client.listObjectsV2Paginator(ListObjectsV2Request.builder().bucket(bucketName).build()).contents() + .forEach(object -> client + .deleteObject(DeleteObjectRequest.builder().bucket(bucketName).key(object.key()).build())); client.deleteBucket(DELETE_BUCKET_BUILDER.bucket(bucketName).build()); } catch (NoSuchBucketException e) { // ignore diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml index 723c118..859a7e5 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml +++ b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml @@ -268,7 +268,7 @@ <compilation-unit name="common/bucket-does-not-exist"> <placeholder name="adapter" value="S3" /> <output-dir compare="Text">common/bucket-does-not-exist</output-dir> - <expected-error>External source error. software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404</expected-error> + <expected-error>External source error. software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist. (Service: S3, Status Code: 404</expected-error> </compilation-unit> </test-case> <test-case FilePath="external-dataset" check-warnings="true"> diff --git a/asterixdb/pom.xml b/asterixdb/pom.xml index a6c92e1..52c32b2 100644 --- a/asterixdb/pom.xml +++ b/asterixdb/pom.xml @@ -94,7 +94,7 @@ <jacoco.version>0.7.6.201602180812</jacoco.version> <log4j.version>2.25.5</log4j.version> <awsjavasdk.version>2.54.7</awsjavasdk.version> - <s3mock.version>0.2.6</s3mock.version> + <s3mock.version>5.2.0</s3mock.version> <parquet.version>1.17.1</parquet.version> <!-- NOTICE: please update transitives from parquet below on any change --> <hadoop-awsjavasdk.version>1.12.797</hadoop-awsjavasdk.version> @@ -1596,19 +1596,18 @@ </dependency> <!-- Mock for AWS S3 --> <dependency> - <groupId>io.findify</groupId> - <artifactId>s3mock_2.12</artifactId> + <groupId>com.adobe.testing</groupId> + <artifactId>s3mock</artifactId> <version>${s3mock.version}</version> - </dependency> - <!-- Needed for the s3 mock --> - <dependency> - <groupId>com.typesafe.akka</groupId> - <artifactId>akka-http-core_2.12</artifactId> - <!-- Must stay on the same akka-http line as the akka-http_2.12 that s3mock pulls in - (10.5.x makes S3Mock.bind() return null). Never advance past 10.1.x: 10.2+ is - BUSL-1.1 rather than Apache-2.0. --> - <!-- @pinned-with: ${s3mock.version} --> - <version>10.1.15</version> + <exclusions> + <!-- the mock is a spring boot application, and its starter brings logback and + log4j-to-slf4j along; both fight the log4j2 binding these tests run with, + and spring refuses to start when it finds logback behind another binding --> + <exclusion> + <groupId>org.springframework.boot</groupId> + <artifactId>spring-boot-starter-logging</artifactId> + </exclusion> + </exclusions> </dependency> <!-- Azure Blob Storage start --> <dependency> -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21646?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I974678973e4fd50dcfc3adfedff3614d5d60c635 Gerrit-Change-Number: 21646 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Blow <[email protected]>
