This is an automated email from the ASF dual-hosted git repository.
cdeppisch pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-jbang-examples.git
The following commit(s) were added to refs/heads/main by this push:
new ac7e3b0 Add automated test for AWS S3 event based example
ac7e3b0 is described below
commit ac7e3b075da056e217d7ee9c4f6b38944f79c712
Author: Christoph Deppisch <[email protected]>
AuthorDate: Fri Jan 10 16:23:41 2025 +0100
Add automated test for AWS S3 event based example
---
.github/workflows/build.yml | 5 +
aws-s3-event-based/README.adoc | 32 +++++-
aws-s3-event-based/application.properties | 21 ++++
aws-s3-event-based/aws-s3-cdc-log.camel.yaml | 14 +--
aws-s3-event-based/test/amazonS3Client.groovy | 116 +++++++++++++++++++++
.../test/application.test.properties | 30 ++++++
.../test/aws-s3-cdc-log.camel.it.yaml | 48 +++++++++
aws-s3-event-based/test/jbang.properties | 8 ++
mqtt/test/jbang.properties | 4 +-
openapi/client/test/jbang.properties | 3 +-
openapi/server/test/jbang.properties | 4 +-
11 files changed, 272 insertions(+), 13 deletions(-)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 3b6c9c9..bc4af51 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -42,6 +42,7 @@ jobs:
- name: Run Tests
env:
CITRUS_CAMEL_JBANG_DUMP_INTEGRATION_OUTPUT: "true"
+ CITRUS_TESTCONTAINERS_LOCALSTACK_IMAGE_NAME:
"mirror.gcr.io/localstack/localstack"
run: |
echo "Install JBang via SDKMAN"
@@ -74,6 +75,10 @@ jobs:
pushd openapi/client/test
jbang citrus@citrusframework/citrus run petstore-client.camel.it.yaml
popd
+
+ pushd aws-s3-event-based/test
+ jbang citrus@citrusframework/citrus run aws-s3-cdc-log.camel.it.yaml
+ popd
- uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874
# v4.4.0
if: always()
with:
diff --git a/aws-s3-event-based/README.adoc b/aws-s3-event-based/README.adoc
index c4e2c3e..de4ecd6 100644
--- a/aws-s3-event-based/README.adoc
+++ b/aws-s3-event-based/README.adoc
@@ -89,7 +89,7 @@ Then you can run this example using:
[source,shell]
----
-$ camel run aws-s3-cdc-log.camel.yaml
+$ camel run application.properties aws-s3-cdc-log.camel.yaml
----
Or run it even shorter:
@@ -105,7 +105,7 @@ You can enable the developer console via `--console` flag
as show:
[source,shell]
----
-$ camel run aws-s3-cdc-log.camel.yaml --console
+$ camel run application.properties aws-s3-cdc-log.camel.yaml --console
----
Then you can browse: http://localhost:8080/q/dev to introspect the running
Camel Application.
@@ -147,7 +147,7 @@ aws s3api delete-object --bucket cdc-s3-bucket --key
example-file.txt
== Cleanup AWS S3 bucket, SQS Queue and EventBridge Rule through Terraform
-You'll need to cleanup everything from AWS console or CLI.
+You'll need to clean up everything from AWS console or CLI.
If you used terraform it will be enough to run terraform destroy
@@ -156,7 +156,7 @@ If you used terraform it will be enough to run terraform
destroy
cd terraform/
----
-At this point you should be to run the destroy
+At this point you should be to run the destroy operation with terraform.
[source,shell]
----
@@ -167,6 +167,30 @@ You'll need to specify the same var used for terraform
apply.
At the end the AWS environment on your account will be clean.
+== Integration testing
+
+The example provides an automated integration test
(`aws-s3-cdc-log.camel.it.yaml`) that you can run with the
https://citrusframework.org/[Citrus] test framework.
+Please make sure to install Citrus as a JBang application (see
link:../../install-citrus.adoc[Citrus installation guide]).
+
+Navigate to the test folder.
+
+[source,shell]
+----
+cd test/
+----
+
+You can run the test with:
+
+[source,shell]
+----
+citrus run aws-s3-cdc-log.camel.it.yaml
+----
+
+The test prepares the complete infrastructure and starts the Camel route
automatically via JBang.
+The Citrus test starts a Localstack Testcontainers instance to simulate the
AWS S3, SQS and EVENT_BRIDGE services.
+Once the AWS services are configured the test loads the Camel route and pushes
some data to the S3 bucket.
+As a result you will see the S3 data being consumed by the event driven Camel
application through SQS and EVENT_BRIDGE notifications.
+
== Help and contributions
If you hit any problem using Camel or have some feedback, then please
diff --git a/aws-s3-event-based/application.properties
b/aws-s3-event-based/application.properties
new file mode 100644
index 0000000..0797a10
--- /dev/null
+++ b/aws-s3-event-based/application.properties
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+aws.queueNameOrArn=sqs-eventbridge-test-123
+aws.secretKey=<secret_key>
+aws.accessKey=<access_key>
+aws.region=eu-west-1
diff --git a/aws-s3-event-based/aws-s3-cdc-log.camel.yaml
b/aws-s3-event-based/aws-s3-cdc-log.camel.yaml
index 0e4640e..e7605f4 100644
--- a/aws-s3-event-based/aws-s3-cdc-log.camel.yaml
+++ b/aws-s3-event-based/aws-s3-cdc-log.camel.yaml
@@ -19,13 +19,15 @@
from:
uri: "kamelet:aws-s3-event-based-source"
parameters:
- queueNameOrArn: "sqs-eventbridge-test-123"
- accessKey: <access_key>
- secretKey: <secret_key>
- region: "eu-west-1"
+ queueNameOrArn: "{{aws.queueNameOrArn}}"
+ accessKey: "{{aws.accessKey}}"
+ secretKey: "{{aws.secretKey}}"
+ region: "{{aws.region}}"
+ overrideEndpoint: "{{aws.overrideEndpoint:false}}"
+ uriEndpointOverride: "{{aws.uriEndpointOverride:none}}"
getObject: true
steps:
- - to:
+ - to:
uri: "kamelet:log-sink"
parameters:
- showStreams: true
+ showStreams: true
diff --git a/aws-s3-event-based/test/amazonS3Client.groovy
b/aws-s3-event-based/test/amazonS3Client.groovy
new file mode 100644
index 0000000..0f9b6ee
--- /dev/null
+++ b/aws-s3-event-based/test/amazonS3Client.groovy
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.util.Collections;
+
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider
+import software.amazon.awssdk.regions.Region
+import software.amazon.awssdk.services.eventbridge.EventBridgeClient
+import software.amazon.awssdk.services.eventbridge.model.Target
+import software.amazon.awssdk.services.eventbridge.model.PutRuleResponse
+import software.amazon.awssdk.services.s3.S3Client
+import software.amazon.awssdk.services.s3.model.EventBridgeConfiguration
+import software.amazon.awssdk.services.sqs.SqsClient
+import software.amazon.awssdk.services.sqs.model.CreateQueueResponse
+import software.amazon.awssdk.services.sqs.model.QueueAttributeName
+
+// Creates an AWS S3 client that is used in the test to push data to the
bucket.
+// Also creates and prepares the bucket as well as the SQS queue and
EVENT_BRIDGE notifications.
+
+S3Client s3 = S3Client
+ .builder()
+
.endpointOverride(URI.create('${CITRUS_TESTCONTAINERS_LOCALSTACK_S3_URL}'))
+ .credentialsProvider(StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ '${CITRUS_TESTCONTAINERS_LOCALSTACK_ACCESS_KEY}',
+ '${CITRUS_TESTCONTAINERS_LOCALSTACK_SECRET_KEY}')
+ ))
+ .forcePathStyle(true)
+ .region(Region.of('${CITRUS_TESTCONTAINERS_LOCALSTACK_REGION}'))
+ .build()
+
+SqsClient sqsClient = SqsClient
+ .builder()
+
.endpointOverride(URI.create('${CITRUS_TESTCONTAINERS_LOCALSTACK_SQS_URL}'))
+ .credentialsProvider(StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ '${CITRUS_TESTCONTAINERS_LOCALSTACK_ACCESS_KEY}',
+ '${CITRUS_TESTCONTAINERS_LOCALSTACK_SECRET_KEY}')
+ ))
+ .region(Region.of('${CITRUS_TESTCONTAINERS_LOCALSTACK_REGION}'))
+ .build()
+
+EventBridgeClient eventBridgeClient = EventBridgeClient
+ .builder()
+
.endpointOverride(URI.create('${CITRUS_TESTCONTAINERS_LOCALSTACK_EVENTBRIDGE_URL}'))
+ .credentialsProvider(StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(
+ '${CITRUS_TESTCONTAINERS_LOCALSTACK_ACCESS_KEY}',
+ '${CITRUS_TESTCONTAINERS_LOCALSTACK_SECRET_KEY}')
+ ))
+ .region(Region.of('${CITRUS_TESTCONTAINERS_LOCALSTACK_REGION}'))
+ .build()
+
+// Create bucket
+s3.createBucket(b -> b.bucket('${aws.bucketNameOrArn}'))
+
+// Enable EventBridge notification on the bucket
+s3.putBucketNotificationConfiguration(b ->
b.bucket('${aws.bucketNameOrArn}').notificationConfiguration(nb ->
nb.eventBridgeConfiguration(EventBridgeConfiguration.builder().build())));
+
+eventBridgeClient.createEventBus(b -> b.name("s3-events-cdc"))
+
+// Add an EventBridge rule on the bucket
+PutRuleResponse putRuleResponse = eventBridgeClient.putRule(b ->
b.name("s3-events-cdc").eventPattern('''
+{
+ "source": ["aws.s3"],
+ "detail": {
+ "bucket": {
+ "name": [ "${aws.bucketNameOrArn}" ]
+ }
+ }
+}
+'''))
+
+CreateQueueResponse createQueueResponse = sqsClient.createQueue(s ->
s.queueName('${aws.queueNameOrArn}'))
+
+// Modify access policy for the queue just created
+String queueUrl = createQueueResponse.queueUrl()
+String queueArn =
'arn:aws:sqs:${aws.region}:000000000000:${aws.queueNameOrArn}'
+
+sqsClient.setQueueAttributes(b ->
b.queueUrl(queueUrl).attributes(Collections.singletonMap(QueueAttributeName.POLICY,
'{' +
+ '"Version": "2012-10-17",' +
+ '"Id": "' + queueArn + '/SQSDefaultPolicy",' +
+ '"Statement":' +
+ '[{ "Sid": "EventsToMyQueue", ' +
+ '"Effect": "Allow", ' +
+ '"Principal": {' +
+ '"Service": "events.amazonaws.com"}, ' +
+ '"Action": "sqs:SendMessage", ' +
+ '"Resource": "' + queueArn + '", ' +
+ '"Condition": {' +
+ '"ArnEquals": {' +
+ '"aws:SourceArn": "' + putRuleResponse.ruleArn() + '"' +
+ '}' +
+ '}' +
+ '}]' +
+'}')))
+
+// Add a target for EventBridge Rule which will be the SQS Queue just created
+eventBridgeClient.putTargets(b ->
b.rule('s3-events-cdc').targets(Target.builder().id("sqs-sub").arn(queueArn).build()))
+
+return s3
diff --git a/aws-s3-event-based/test/application.test.properties
b/aws-s3-event-based/test/application.test.properties
new file mode 100644
index 0000000..80170ba
--- /dev/null
+++ b/aws-s3-event-based/test/application.test.properties
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+aws.queueNameOrArn=${aws.queueNameOrArn}
+aws.secretKey=${CITRUS_TESTCONTAINERS_LOCALSTACK_SECRET_KEY}
+aws.accessKey=${CITRUS_TESTCONTAINERS_LOCALSTACK_ACCESS_KEY}
+aws.region=${CITRUS_TESTCONTAINERS_LOCALSTACK_REGION}
+aws.overrideEndpoint=true
+aws.uriEndpointOverride=${CITRUS_TESTCONTAINERS_LOCALSTACK_S3_URL}
+
+# AWS S3 component
+aws.bucketNameOrArn=${aws.bucketNameOrArn}
+camel.component.aws2-s3.forcePathStyle=true
+camel.component.aws2-s3.overrideEndpoint=true
+camel.component.aws2-s3.uriEndpointOverride=${CITRUS_TESTCONTAINERS_LOCALSTACK_S3_URL}
+
diff --git a/aws-s3-event-based/test/aws-s3-cdc-log.camel.it.yaml
b/aws-s3-event-based/test/aws-s3-cdc-log.camel.it.yaml
new file mode 100644
index 0000000..43391e8
--- /dev/null
+++ b/aws-s3-event-based/test/aws-s3-cdc-log.camel.it.yaml
@@ -0,0 +1,48 @@
+name: aws-s3-cdc-log-test
+description: Sample test in YAML
+variables:
+ - name: "aws.bucketNameOrArn"
+ value: cdc-s3-bucket
+ - name: "aws.queueNameOrArn"
+ value: cdc-sqs-eventbridge
+ - name: "aws.s3.key"
+ value: hello.txt
+ - name: "aws.s3.message"
+ value: "Hello Camel!"
+actions:
+ - testcontainers:
+ start:
+ localstack:
+ services:
+ - "S3"
+ - "SQS"
+ - "EVENT_BRIDGE"
+ - camel:
+ startContext: { }
+ - camel:
+ createComponent:
+ name: amazonS3Client
+ file: amazonS3Client.groovy
+ - camel:
+ jbang:
+ run:
+ integration:
+ name: "aws-s3-cdc-log"
+ file: "../aws-s3-cdc-log.camel.yaml"
+ systemProperties:
+ file: "application.test.properties"
+ - send:
+ endpoint: |
+ camel:aws2-s3:${aws.bucketNameOrArn}?amazonS3Client=#amazonS3Client
+ message:
+ body:
+ data: "${aws.s3.message}"
+ headers:
+ - name: "CamelAwsS3Key"
+ value: "${aws.s3.key}"
+ - camel:
+ jbang:
+ verify:
+ integration: "aws-s3-cdc-log"
+ logMessage: |
+ Body: ${aws.s3.message}
diff --git a/aws-s3-event-based/test/jbang.properties
b/aws-s3-event-based/test/jbang.properties
new file mode 100644
index 0000000..d4f7d94
--- /dev/null
+++ b/aws-s3-event-based/test/jbang.properties
@@ -0,0 +1,8 @@
+# Declare required additional dependencies
+run.deps=org.citrusframework:citrus-camel:4.5.0,\
+org.citrusframework:citrus-testcontainers:4.5.0,\
+org.apache.camel:camel-endpointdsl:4.9.0,\
+org.apache.camel:camel-aws2-s3:4.9.0,\
+org.apache.camel:camel-aws2-sqs:4.9.0,\
+org.apache.camel:camel-aws2-eventbridge:4.9.0,\
+software.amazon.awssdk:sdk-core:2.29.35
diff --git a/mqtt/test/jbang.properties b/mqtt/test/jbang.properties
index 25fa91e..beacda5 100644
--- a/mqtt/test/jbang.properties
+++ b/mqtt/test/jbang.properties
@@ -1,2 +1,4 @@
# Declare required additional dependencies
-run.deps=org.citrusframework:citrus-camel:4.5.0,org.citrusframework:citrus-testcontainers:4.5.0,org.apache.camel:camel-paho-mqtt5:4.9.0
+run.deps=org.citrusframework:citrus-camel:4.5.0,\
+org.citrusframework:citrus-testcontainers:4.5.0,\
+org.apache.camel:camel-paho-mqtt5:4.9.0
diff --git a/openapi/client/test/jbang.properties
b/openapi/client/test/jbang.properties
index 894640e..28ce761 100644
--- a/openapi/client/test/jbang.properties
+++ b/openapi/client/test/jbang.properties
@@ -1,2 +1,3 @@
# Declare required additional dependencies
-run.deps=org.citrusframework:citrus-camel:4.5.0,org.citrusframework:citrus-openapi:4.5.0
+run.deps=org.citrusframework:citrus-camel:4.5.0,\
+org.citrusframework:citrus-openapi:4.5.0
diff --git a/openapi/server/test/jbang.properties
b/openapi/server/test/jbang.properties
index 8ff5693..f591c1f 100644
--- a/openapi/server/test/jbang.properties
+++ b/openapi/server/test/jbang.properties
@@ -1,2 +1,4 @@
# Declare required additional dependencies
-run.deps=org.citrusframework:citrus-camel:4.5.0,org.citrusframework:citrus-openapi:4.5.0,io.vertx:vertx-core:4.5.11
+run.deps=org.citrusframework:citrus-camel:4.5.0,\
+org.citrusframework:citrus-openapi:4.5.0,\
+io.vertx:vertx-core:4.5.11
