This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-jbang-examples.git
The following commit(s) were added to refs/heads/main by this push:
new a8c1256 Add AWS S3 Event Based Example (#4)
a8c1256 is described below
commit a8c1256056c9c2a73cbdf1fbb4c7aea23e49b87b
Author: Andrea Cosentino <[email protected]>
AuthorDate: Wed Dec 4 15:45:46 2024 +0100
Add AWS S3 Event Based Example (#4)
Signed-off-by: Andrea Cosentino <[email protected]>
---
aws-s3-event-based/README.adoc | 196 +++++++++++++++++++++++++++
aws-s3-event-based/aws-s3-cdc-log.camel.yaml | 31 +++++
aws-s3-event-based/example-file.txt | 1 +
aws-s3-event-based/policy-queue.json | 3 +
aws-s3-event-based/terraform/main.tf | 127 +++++++++++++++++
5 files changed, 358 insertions(+)
diff --git a/aws-s3-event-based/README.adoc b/aws-s3-event-based/README.adoc
new file mode 100644
index 0000000..fb81418
--- /dev/null
+++ b/aws-s3-event-based/README.adoc
@@ -0,0 +1,196 @@
+== AWS S3 CDC Example
+
+In this sample you'll use the AWS S3 CDC Source Kamelet.
+
+Through the usage of EventBridge and SQS Services you'll be able to consume
events from specific bucket.
+
+=== Install JBang
+
+First install JBang according to https://www.jbang.dev
+
+When JBang is installed then you should be able to run from a shell:
+
+[source,sh]
+----
+$ jbang --version
+----
+
+This will output the version of JBang.
+
+To run this example you can either install Camel on JBang via:
+
+[source,sh]
+----
+$ jbang app install camel@apache/camel
+----
+
+Which allows to run CamelJBang with `camel` as shown below.
+
+=== Setup the AWS S3 bucket, SQS Queue and EventBridge Rule
+
+You'll need a fully working AWS CLI locally.
+
+Create a bucket on AWS on a particular region
+
+[source,sh]
+----
+aws s3api create-bucket --bucket cdc-s3-bucket --region eu-west-1
--create-bucket-configuration LocationConstraint=eu-west-1
+----
+
+Enable EventBridge notification on the bucket
+
+[source,sh]
+----
+aws s3api put-bucket-notification-configuration --bucket cdc-s3-bucket
--region eu-west-1 --notification-configuration '{ "EventBridgeConfiguration":
{} }'
+----
+
+Add an EventBridge rule on the bucket
+
+[source,sh]
+----
+aws events put-rule --name "s3-events-cdc" --event-pattern
"{\"source\":[\"aws.s3\"], \"detail\": {\"bucket\": { \"name\":
[\"cdc-s3-bucket\"]}}}" --region=eu-west-1
+----
+
+Create an SQS Queue where we're going to receive the notification
+
+[source,sh]
+----
+aws sqs create-queue --queue-name test-queue-3 --region eu-west-1
+----
+
+Modify Access Policy for the queue just created. Don't forget to edit the
policy-queue.json file (by adding the correct rule_arn and queue_arn), before
submitting the command.
+
+[source,sh]
+----
+aws sqs set-queue-attributes --queue-url <just_created_queue_arn> --attributes
file://policy-queue.json
+----
+
+Add a target for EventBridge Rule which will be the SQS Queue just created
+
+[source,sh]
+----
+aws events put-targets --rule s3-events-cdc --targets
"Id"="sqs-sub","Arn"="<just_created_queue_arn>" --region eu-west-1
+----
+
+=== Setup the AWS S3 bucket, SQS Queue and EventBridge Rule through Terraform
+
+If you are in a hurry you can also try this example by running the Terraform
configuration provided in Terraform folder.
+
+[source,sh]
+----
+cd terraform/
+----
+
+and then run
+
+[source,sh]
+----
+terraform init
+----
+
+At this point you should be to run the configuration
+
+[source,sh]
+----
+terraform apply -var="s3_bucket_name=s3-eventbridge-test-123"
-var="sqs_queue_name=sqs-eventbridge-test-123"
+----
+
+You can specify whatever bucket name or SQS name you want.
+
+At the end the AWS environment on your account will be completed, and you
could go ahead with the example.
+
+Don't forget to specify the correct sqs queue name in the yaml file and adding
correct credentials for AWS.
+
+=== How to run
+
+Then you can run this example using:
+
+[source,sh]
+----
+$ camel run aws-s3-cdc-log.camel.yaml
+----
+
+Or run it even shorter:
+
+[source,sh]
+----
+$ camel run *
+----
+
+=== Developer Web Console
+
+You can enable the developer console via `--console` flag as show:
+
+[source,sh]
+----
+$ camel run aws-s3-cdc-log.camel.yaml --console
+----
+
+Then you can browse: http://localhost:8080/q/dev to introspect the running
Camel Application.
+
+=== Create and delete an object
+
+While the integration is running you can run the following commands:
+
+[source,sh]
+----
+aws s3api put-object --bucket cdc-s3-bucket --key example-file-uploaded.txt
--body example-file.txt --region eu-west-1
+{
+ "ETag": "\"2b6153cbc11ea06a6d0b502a64b77e3c\""
+}
+aws s3api delete-object --bucket cdc-s3-bucket --key example-file.txt
+----
+
+[source,sh]
+----
+022-11-02 15:13:03.747 INFO 120300 --- [ main]
he.camel.cli.connector.LocalCliConnector : Camel CLI enabled (local)
+2022-11-02 15:13:08.570 INFO 120300 --- [ main]
e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.21.0-SNAPSHOT
(aws-s3-cdc-log) is starting
+2022-11-02 15:13:08.726 INFO 120300 --- [ main]
org.apache.camel.main.BaseMainSupport : Property-placeholders summary
+2022-11-02 15:13:08.727 INFO 120300 --- [ main]
org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml]
queueNameOrArn=test-queue-3
+2022-11-02 15:13:08.727 INFO 120300 --- [ main]
org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml]
accessKey=xxxxxx
+2022-11-02 15:13:08.727 INFO 120300 --- [ main]
org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml]
region=eu-west-1
+2022-11-02 15:13:08.727 INFO 120300 --- [ main]
org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml]
secretKey=xxxxxx
+2022-11-02 15:13:08.727 INFO 120300 --- [ main]
org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml]
getObject=true
+2022-11-02 15:13:08.727 INFO 120300 --- [ main]
org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml]
showStreams=true
+2022-11-02 15:13:08.741 INFO 120300 --- [ main]
e.camel.impl.engine.AbstractCamelContext : Routes startup (started:3)
+2022-11-02 15:13:08.741 INFO 120300 --- [ main]
e.camel.impl.engine.AbstractCamelContext : Started route1
(kamelet://aws-s3-cdc-source)
+2022-11-02 15:13:08.741 INFO 120300 --- [ main]
e.camel.impl.engine.AbstractCamelContext : Started aws-s3-cdc-source-1
(aws2-sqs://test-queue-3)
+2022-11-02 15:13:08.741 INFO 120300 --- [ main]
e.camel.impl.engine.AbstractCamelContext : Started log-sink-2
(kamelet://source)
+2022-11-02 15:13:08.742 INFO 120300 --- [ main]
e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.21.0-SNAPSHOT
(aws-s3-cdc-log) started in 4s272ms (build:106ms init:3s995ms start:171ms
JVM-uptime:6s)
+
+
+2022-11-02 15:13:30.977 INFO 120300 --- [://test-queue-3] info
: Exchange[ExchangePattern: InOnly, BodyType:
org.apache.camel.converter.stream.InputStreamCache, Body: Text example]
+2022-11-02 15:13:41.250 INFO 120300 --- [://test-queue-3] info
: Exchange[ExchangePattern: InOnly, BodyType:
com.fasterxml.jackson.databind.node.ObjectNode, Body: { "version" : "0", "id"
: "d54290df-2ad9-31ff-308b-8331fee7344a", "detail-type" : "Object Deleted",
"source" : "aws.s3", "account" : "xxxx", "time" : "2022-11-02T14:13:37Z",
"region" : "eu-west-1", "resources" : [ "arn:aws:s3:::cdc-s3-bucket" ],
"detail" : { "version" : "0", " [...]
+----
+
+=== Cleanup AWS S3 bucket, SQS Queue and EventBridge Rule through Terraform
+
+You'll need to cleanup everything from AWS console or CLI.
+
+If you used terraform it will be enough to run terraform destroy
+
+[source,sh]
+----
+cd terraform/
+----
+
+At this point you should be to run the destroy
+
+[source,sh]
+----
+terraform destroy -var="s3_bucket_name=s3-eventbridge-test"
-var="sqs_queue_name=sqs-eventbridge-test"
+----
+
+You'll need to specify the same var used for terraform apply.
+
+At the end the AWS environment on your account will be clean.
+
+=== Help and contributions
+
+If you hit any problem using Camel or have some feedback, then please
+https://camel.apache.org/community/support/[let us know].
+
+We also love contributors, so
+https://camel.apache.org/community/contributing/[get involved] :-)
+
+The Camel riders!
diff --git a/aws-s3-event-based/aws-s3-cdc-log.camel.yaml
b/aws-s3-event-based/aws-s3-cdc-log.camel.yaml
new file mode 100644
index 0000000..0e4640e
--- /dev/null
+++ b/aws-s3-event-based/aws-s3-cdc-log.camel.yaml
@@ -0,0 +1,31 @@
+## ---------------------------------------------------------------------------
+## Licensed to the Apache Software Foundation (ASF) under one or more
+## contributor license agreements. See the NOTICE file distributed with
+## this work for additional information regarding copyright ownership.
+## The ASF licenses this file to You under the Apache License, Version 2.0
+## (the "License"); you may not use this file except in compliance with
+## the License. You may obtain a copy of the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+## ---------------------------------------------------------------------------
+
+- route:
+ from:
+ uri: "kamelet:aws-s3-event-based-source"
+ parameters:
+ queueNameOrArn: "sqs-eventbridge-test-123"
+ accessKey: <access_key>
+ secretKey: <secret_key>
+ region: "eu-west-1"
+ getObject: true
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+ parameters:
+ showStreams: true
diff --git a/aws-s3-event-based/example-file.txt
b/aws-s3-event-based/example-file.txt
new file mode 100644
index 0000000..105c45e
--- /dev/null
+++ b/aws-s3-event-based/example-file.txt
@@ -0,0 +1 @@
+Text example
diff --git a/aws-s3-event-based/policy-queue.json
b/aws-s3-event-based/policy-queue.json
new file mode 100644
index 0000000..b6d0964
--- /dev/null
+++ b/aws-s3-event-based/policy-queue.json
@@ -0,0 +1,3 @@
+{
+ "Policy":
"{\"Version\":\"2012-10-17\",\"Id\":\"<queue_arn>/SQSDefaultPolicy\",\"Statement\":[{\"Sid\":
\"EventsToMyQueue\", \"Effect\": \"Allow\", \"Principal\": {\"Service\":
\"events.amazonaws.com\"}, \"Action\": \"sqs:SendMessage\", \"Resource\":
\"<queue_arn>\", \"Condition\": {\"ArnEquals\": {\"aws:SourceArn\":
\"<rule_arn>\"}}}]}"
+}
diff --git a/aws-s3-event-based/terraform/main.tf
b/aws-s3-event-based/terraform/main.tf
new file mode 100644
index 0000000..4fc7da1
--- /dev/null
+++ b/aws-s3-event-based/terraform/main.tf
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+terraform {
+ required_providers {
+ aws = {
+ source = "hashicorp/aws"
+ version = "~> 3.27"
+ }
+ }
+
+ required_version = ">= 0.14.9"
+}
+
+provider "aws" {
+ profile = "default"
+ region = "eu-west-1"
+}
+
+variable "s3_bucket_name" {
+ type = string
+}
+
+variable "sqs_queue_name" {
+ type = string
+}
+
+
+data "aws_caller_identity" "current" {}
+
+# Create a new S3 bucket
+resource "aws_s3_bucket" "MyS3Bucket" {
+ bucket = var.s3_bucket_name
+ force_destroy = true
+}
+
+# Send notifications to EventBridge for all events in the bucket
+resource "aws_s3_bucket_notification" "MyS3BucketNotification" {
+ bucket = aws_s3_bucket.MyS3Bucket.id
+ eventbridge = true
+}
+
+# Create an EventBridge rule
+resource "aws_cloudwatch_event_rule" "MyEventRule" {
+ description = "Object create events on bucket
s3://${aws_s3_bucket.MyS3Bucket.id}"
+ event_pattern = <<EOF
+{
+ "source": [
+ "aws.s3"
+ ],
+ "detail": {
+ "bucket": {
+ "name": ["${aws_s3_bucket.MyS3Bucket.id}"]
+ }
+ }
+}
+EOF
+}
+
+# Set the SNS topic as a target of the EventBridge rule
+resource "aws_cloudwatch_event_target" "MyEventRuleTarget" {
+ rule = aws_cloudwatch_event_rule.MyEventRule.name
+ arn = aws_sqs_queue.sqs-queue.arn
+}
+
+# Create a new SQS queue
+resource "aws_sqs_queue" "sqs-queue" {
+ name = var.sqs_queue_name
+}
+
+# Allow EventBridge to publish to the SQS queue
+resource "aws_sqs_queue_policy" "MySQSQueuePolicy" {
+ queue_url = aws_sqs_queue.sqs-queue.id
+ policy = <<POLICY
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Sid": "AWSEventsPermission",
+ "Effect": "Allow",
+ "Principal": {
+ "Service": "events.amazonaws.com"
+ },
+ "Action": "sqs:SendMessage",
+ "Resource": "${aws_sqs_queue.sqs-queue.arn}",
+ "Condition": {
+ "ArnEquals": {
+ "aws:SourceArn": "${aws_cloudwatch_event_rule.MyEventRule.arn}"
+ }
+ }
+ }
+ ]
+}
+POLICY
+}
+
+# Display the EventBridge rule, S3 bucket and SQS queue
+output "EventBridge-Rule-Name" {
+ value = aws_cloudwatch_event_rule.MyEventRule.name
+ description = "The EventBridge Rule Name"
+}
+output "S3-Bucket" {
+ value = aws_s3_bucket.MyS3Bucket.id
+ description = "The S3 Bucket"
+}
+output "SQS-Queue-Name" {
+ value = aws_sqs_queue.sqs-queue.name
+ description = "The SQS Queue Name"
+}
+output "SQS-Queue-ARN" {
+ value = aws_sqs_queue.sqs-queue.arn
+ description = "The SQS Queue Arn"
+}
