This is an automated email from the ASF dual-hosted git repository. acosentino pushed a commit to branch issue-3 in repository https://gitbox.apache.org/repos/asf/camel-jbang-examples.git
commit fe14e9e3e72816db22f25e84a6bd752ef238dd49 Author: Andrea Cosentino <[email protected]> AuthorDate: Wed Dec 4 15:43:29 2024 +0100 Add AWS S3 Event Based Example Signed-off-by: Andrea Cosentino <[email protected]> --- aws-s3-event-based/README.adoc | 196 +++++++++++++++++++++++++++ aws-s3-event-based/aws-s3-cdc-log.camel.yaml | 31 +++++ aws-s3-event-based/example-file.txt | 1 + aws-s3-event-based/policy-queue.json | 3 + aws-s3-event-based/terraform/main.tf | 127 +++++++++++++++++ 5 files changed, 358 insertions(+) diff --git a/aws-s3-event-based/README.adoc b/aws-s3-event-based/README.adoc new file mode 100644 index 0000000..fb81418 --- /dev/null +++ b/aws-s3-event-based/README.adoc @@ -0,0 +1,196 @@ +== AWS S3 CDC Example + +In this sample you'll use the AWS S3 CDC Source Kamelet. + +Through the usage of EventBridge and SQS Services you'll be able to consume events from specific bucket. + +=== Install JBang + +First install JBang according to https://www.jbang.dev + +When JBang is installed then you should be able to run from a shell: + +[source,sh] +---- +$ jbang --version +---- + +This will output the version of JBang. + +To run this example you can either install Camel on JBang via: + +[source,sh] +---- +$ jbang app install camel@apache/camel +---- + +Which allows to run CamelJBang with `camel` as shown below. + +=== Setup the AWS S3 bucket, SQS Queue and EventBridge Rule + +You'll need a fully working AWS CLI locally. + +Create a bucket on AWS on a particular region + +[source,sh] +---- +aws s3api create-bucket --bucket cdc-s3-bucket --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1 +---- + +Enable EventBridge notification on the bucket + +[source,sh] +---- +aws s3api put-bucket-notification-configuration --bucket cdc-s3-bucket --region eu-west-1 --notification-configuration '{ "EventBridgeConfiguration": {} }' +---- + +Add an EventBridge rule on the bucket + +[source,sh] +---- +aws events put-rule --name "s3-events-cdc" --event-pattern "{\"source\":[\"aws.s3\"], \"detail\": {\"bucket\": { \"name\": [\"cdc-s3-bucket\"]}}}" --region=eu-west-1 +---- + +Create an SQS Queue where we're going to receive the notification + +[source,sh] +---- +aws sqs create-queue --queue-name test-queue-3 --region eu-west-1 +---- + +Modify Access Policy for the queue just created. Don't forget to edit the policy-queue.json file (by adding the correct rule_arn and queue_arn), before submitting the command. + +[source,sh] +---- +aws sqs set-queue-attributes --queue-url <just_created_queue_arn> --attributes file://policy-queue.json +---- + +Add a target for EventBridge Rule which will be the SQS Queue just created + +[source,sh] +---- +aws events put-targets --rule s3-events-cdc --targets "Id"="sqs-sub","Arn"="<just_created_queue_arn>" --region eu-west-1 +---- + +=== Setup the AWS S3 bucket, SQS Queue and EventBridge Rule through Terraform + +If you are in a hurry you can also try this example by running the Terraform configuration provided in Terraform folder. + +[source,sh] +---- +cd terraform/ +---- + +and then run + +[source,sh] +---- +terraform init +---- + +At this point you should be to run the configuration + +[source,sh] +---- +terraform apply -var="s3_bucket_name=s3-eventbridge-test-123" -var="sqs_queue_name=sqs-eventbridge-test-123" +---- + +You can specify whatever bucket name or SQS name you want. + +At the end the AWS environment on your account will be completed, and you could go ahead with the example. + +Don't forget to specify the correct sqs queue name in the yaml file and adding correct credentials for AWS. + +=== How to run + +Then you can run this example using: + +[source,sh] +---- +$ camel run aws-s3-cdc-log.camel.yaml +---- + +Or run it even shorter: + +[source,sh] +---- +$ camel run * +---- + +=== Developer Web Console + +You can enable the developer console via `--console` flag as show: + +[source,sh] +---- +$ camel run aws-s3-cdc-log.camel.yaml --console +---- + +Then you can browse: http://localhost:8080/q/dev to introspect the running Camel Application. + +=== Create and delete an object + +While the integration is running you can run the following commands: + +[source,sh] +---- +aws s3api put-object --bucket cdc-s3-bucket --key example-file-uploaded.txt --body example-file.txt --region eu-west-1 +{ + "ETag": "\"2b6153cbc11ea06a6d0b502a64b77e3c\"" +} +aws s3api delete-object --bucket cdc-s3-bucket --key example-file.txt +---- + +[source,sh] +---- +022-11-02 15:13:03.747 INFO 120300 --- [ main] he.camel.cli.connector.LocalCliConnector : Camel CLI enabled (local) +2022-11-02 15:13:08.570 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.21.0-SNAPSHOT (aws-s3-cdc-log) is starting +2022-11-02 15:13:08.726 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : Property-placeholders summary +2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] queueNameOrArn=test-queue-3 +2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] accessKey=xxxxxx +2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] region=eu-west-1 +2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] secretKey=xxxxxx +2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [s-s3-cdc-source.kamelet.yaml] getObject=true +2022-11-02 15:13:08.727 INFO 120300 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showStreams=true +2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes startup (started:3) +2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started route1 (kamelet://aws-s3-cdc-source) +2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started aws-s3-cdc-source-1 (aws2-sqs://test-queue-3) +2022-11-02 15:13:08.741 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started log-sink-2 (kamelet://source) +2022-11-02 15:13:08.742 INFO 120300 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.21.0-SNAPSHOT (aws-s3-cdc-log) started in 4s272ms (build:106ms init:3s995ms start:171ms JVM-uptime:6s) + + +2022-11-02 15:13:30.977 INFO 120300 --- [://test-queue-3] info : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: Text example] +2022-11-02 15:13:41.250 INFO 120300 --- [://test-queue-3] info : Exchange[ExchangePattern: InOnly, BodyType: com.fasterxml.jackson.databind.node.ObjectNode, Body: { "version" : "0", "id" : "d54290df-2ad9-31ff-308b-8331fee7344a", "detail-type" : "Object Deleted", "source" : "aws.s3", "account" : "xxxx", "time" : "2022-11-02T14:13:37Z", "region" : "eu-west-1", "resources" : [ "arn:aws:s3:::cdc-s3-bucket" ], "detail" : { "version" : "0", " [...] +---- + +=== Cleanup AWS S3 bucket, SQS Queue and EventBridge Rule through Terraform + +You'll need to cleanup everything from AWS console or CLI. + +If you used terraform it will be enough to run terraform destroy + +[source,sh] +---- +cd terraform/ +---- + +At this point you should be to run the destroy + +[source,sh] +---- +terraform destroy -var="s3_bucket_name=s3-eventbridge-test" -var="sqs_queue_name=sqs-eventbridge-test" +---- + +You'll need to specify the same var used for terraform apply. + +At the end the AWS environment on your account will be clean. + +=== Help and contributions + +If you hit any problem using Camel or have some feedback, then please +https://camel.apache.org/community/support/[let us know]. + +We also love contributors, so +https://camel.apache.org/community/contributing/[get involved] :-) + +The Camel riders! diff --git a/aws-s3-event-based/aws-s3-cdc-log.camel.yaml b/aws-s3-event-based/aws-s3-cdc-log.camel.yaml new file mode 100644 index 0000000..0e4640e --- /dev/null +++ b/aws-s3-event-based/aws-s3-cdc-log.camel.yaml @@ -0,0 +1,31 @@ +## --------------------------------------------------------------------------- +## Licensed to the Apache Software Foundation (ASF) under one or more +## contributor license agreements. See the NOTICE file distributed with +## this work for additional information regarding copyright ownership. +## The ASF licenses this file to You under the Apache License, Version 2.0 +## (the "License"); you may not use this file except in compliance with +## the License. You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## --------------------------------------------------------------------------- + +- route: + from: + uri: "kamelet:aws-s3-event-based-source" + parameters: + queueNameOrArn: "sqs-eventbridge-test-123" + accessKey: <access_key> + secretKey: <secret_key> + region: "eu-west-1" + getObject: true + steps: + - to: + uri: "kamelet:log-sink" + parameters: + showStreams: true diff --git a/aws-s3-event-based/example-file.txt b/aws-s3-event-based/example-file.txt new file mode 100644 index 0000000..105c45e --- /dev/null +++ b/aws-s3-event-based/example-file.txt @@ -0,0 +1 @@ +Text example diff --git a/aws-s3-event-based/policy-queue.json b/aws-s3-event-based/policy-queue.json new file mode 100644 index 0000000..b6d0964 --- /dev/null +++ b/aws-s3-event-based/policy-queue.json @@ -0,0 +1,3 @@ +{ + "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"<queue_arn>/SQSDefaultPolicy\",\"Statement\":[{\"Sid\": \"EventsToMyQueue\", \"Effect\": \"Allow\", \"Principal\": {\"Service\": \"events.amazonaws.com\"}, \"Action\": \"sqs:SendMessage\", \"Resource\": \"<queue_arn>\", \"Condition\": {\"ArnEquals\": {\"aws:SourceArn\": \"<rule_arn>\"}}}]}" +} diff --git a/aws-s3-event-based/terraform/main.tf b/aws-s3-event-based/terraform/main.tf new file mode 100644 index 0000000..4fc7da1 --- /dev/null +++ b/aws-s3-event-based/terraform/main.tf @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 3.27" + } + } + + required_version = ">= 0.14.9" +} + +provider "aws" { + profile = "default" + region = "eu-west-1" +} + +variable "s3_bucket_name" { + type = string +} + +variable "sqs_queue_name" { + type = string +} + + +data "aws_caller_identity" "current" {} + +# Create a new S3 bucket +resource "aws_s3_bucket" "MyS3Bucket" { + bucket = var.s3_bucket_name + force_destroy = true +} + +# Send notifications to EventBridge for all events in the bucket +resource "aws_s3_bucket_notification" "MyS3BucketNotification" { + bucket = aws_s3_bucket.MyS3Bucket.id + eventbridge = true +} + +# Create an EventBridge rule +resource "aws_cloudwatch_event_rule" "MyEventRule" { + description = "Object create events on bucket s3://${aws_s3_bucket.MyS3Bucket.id}" + event_pattern = <<EOF +{ + "source": [ + "aws.s3" + ], + "detail": { + "bucket": { + "name": ["${aws_s3_bucket.MyS3Bucket.id}"] + } + } +} +EOF +} + +# Set the SNS topic as a target of the EventBridge rule +resource "aws_cloudwatch_event_target" "MyEventRuleTarget" { + rule = aws_cloudwatch_event_rule.MyEventRule.name + arn = aws_sqs_queue.sqs-queue.arn +} + +# Create a new SQS queue +resource "aws_sqs_queue" "sqs-queue" { + name = var.sqs_queue_name +} + +# Allow EventBridge to publish to the SQS queue +resource "aws_sqs_queue_policy" "MySQSQueuePolicy" { + queue_url = aws_sqs_queue.sqs-queue.id + policy = <<POLICY +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "AWSEventsPermission", + "Effect": "Allow", + "Principal": { + "Service": "events.amazonaws.com" + }, + "Action": "sqs:SendMessage", + "Resource": "${aws_sqs_queue.sqs-queue.arn}", + "Condition": { + "ArnEquals": { + "aws:SourceArn": "${aws_cloudwatch_event_rule.MyEventRule.arn}" + } + } + } + ] +} +POLICY +} + +# Display the EventBridge rule, S3 bucket and SQS queue +output "EventBridge-Rule-Name" { + value = aws_cloudwatch_event_rule.MyEventRule.name + description = "The EventBridge Rule Name" +} +output "S3-Bucket" { + value = aws_s3_bucket.MyS3Bucket.id + description = "The S3 Bucket" +} +output "SQS-Queue-Name" { + value = aws_sqs_queue.sqs-queue.name + description = "The SQS Queue Name" +} +output "SQS-Queue-ARN" { + value = aws_sqs_queue.sqs-queue.arn + description = "The SQS Queue Arn" +}
