This is an automated email from the ASF dual-hosted git repository. acosentino pushed a commit to branch ibm-cos-examples in repository https://gitbox.apache.org/repos/asf/camel-jbang-examples.git
commit 08c73da9711158ef7acbc4f0d22c5d464b5495ba Author: Andrea Cosentino <[email protected]> AuthorDate: Fri Oct 31 11:07:29 2025 +0100 Added IBM COS Examples Signed-off-by: Andrea Cosentino <[email protected]> --- ibm/ibm-cos/README.adoc | 270 ++++++++++++++++++++++++++++++++ ibm/ibm-cos/application.properties | 39 +++++ ibm/ibm-cos/ibm-cos-sink.kamelet.yaml | 134 ++++++++++++++++ ibm/ibm-cos/ibm-cos-source.kamelet.yaml | 177 +++++++++++++++++++++ ibm/ibm-cos/ibm-cos-to-log.camel.yaml | 39 +++++ ibm/ibm-cos/kafka-to-ibm-cos.camel.yaml | 42 +++++ 6 files changed, 701 insertions(+) diff --git a/ibm/ibm-cos/README.adoc b/ibm/ibm-cos/README.adoc new file mode 100644 index 0000000..eb03f12 --- /dev/null +++ b/ibm/ibm-cos/README.adoc @@ -0,0 +1,270 @@ += IBM Cloud Object Storage Examples + +This directory contains examples demonstrating the IBM COS Kamelets for Apache Camel. + +== Available Examples + +=== 1. Kafka to IBM COS Sink (kafka-to-ibm-cos.camel.yaml) + +This example uses the IBM COS Sink Kamelet to upload data to IBM Cloud Object Storage. + +Messages consumed from a Kafka topic are uploaded to an IBM COS bucket. + +=== 2. IBM COS Source to Log (ibm-cos-to-log.camel.yaml) + +This example uses the IBM COS Source Kamelet to consume data from IBM Cloud Object Storage. + +The example polls an IBM COS bucket for new objects and logs their content and metadata. + +== Prerequisites + +You must have: +- A valid IBM Cloud account and an IBM Cloud Object Storage service instance +- A running Kafka broker (for the Kafka to IBM COS sink example) + +== Setup Kafka (for Kafka to IBM COS example) + +You need a running Kafka broker. The easiest way is to use Camel JBang's built-in infrastructure support. + +=== Start Kafka with JBang + +Start a local Kafka broker using Camel JBang: + +[source,shell] +---- +$ camel infra run kafka +---- + +This command will start a local Kafka broker on port 9092 using Docker Compose in the background. + +The Kafka broker will automatically create topics on first use, so you don't need to manually create the `ibm-cos-topic` topic. + +== Setup IBM Cloud Object Storage + +=== Create an IBM Cloud Object Storage Instance + +1. Log in to the https://cloud.ibm.com/[IBM Cloud Console] +2. Navigate to the https://cloud.ibm.com/catalog/services/cloud-object-storage[IBM Cloud Object Storage] service in the catalog +3. Create a new instance with a name of your choice +4. Once created, note the **Service Instance ID (CRN)** + +=== Create a Bucket + +1. In your COS instance, go to "Buckets" and click "Create bucket" +2. Choose a bucket name (e.g., `my-camel-bucket`) +3. Select a resiliency (Regional, Cross-region, or Single Data Center) +4. Select a location (e.g., `us-south`) +5. Click "Create bucket" + +=== Upload Test Files (for IBM COS Source example) + +If you want to test the IBM COS Source to Log example, upload some test files to your bucket: + +1. Navigate to your bucket in the IBM Cloud console +2. Click "Upload" and select one or more files +3. Alternatively, use the IBM Cloud CLI: + +[source,shell] +---- +ibmcloud cos upload --bucket my-camel-bucket --key test-file.txt --file /path/to/local/file.txt +---- + +=== Create API Credentials + +1. In your COS instance, navigate to "Service credentials" +2. Click "New credential" +3. Provide a name for the credential +4. Ensure the role includes at least "Writer" permissions +5. Click "Add" +6. View the created credentials and note the **apikey** value + +=== Get the Endpoint URL + +IBM COS provides different endpoint URLs based on the bucket's location and access type (public, private, or direct). + +For example: +- Public endpoint (US South): `https://s3.us-south.cloud-object-storage.appdomain.cloud` +- Public endpoint (EU GB): `https://s3.eu-gb.cloud-object-storage.appdomain.cloud` + +For a complete list of endpoints, see the https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints[IBM COS Endpoints documentation]. + +== Configuration + +Update the `application.properties` file with your Kafka and IBM Cloud Object Storage credentials: + +[source,properties] +---- +# Kafka Configuration +kafka.topic=ibm-cos-topic +kafka.bootstrapServers=localhost:9092 +kafka.consumerGroup=kafka-ibm-cos-group +kafka.autoOffsetReset=earliest + +# IBM Cloud Object Storage Configuration +ibm.bucketName=my-camel-bucket +ibm.apiKey=<your-ibm-cloud-api-key> +ibm.serviceInstanceId=<your-service-instance-id> +ibm.endpointUrl=https://s3.us-south.cloud-object-storage.appdomain.cloud +ibm.location=us-south +ibm.autoCreateBucket=false +---- + +Replace: +- `<your-ibm-cloud-api-key>` with your IBM Cloud API key +- `<your-service-instance-id>` with your COS Service Instance ID (CRN) +- Update the endpoint URL and location to match your bucket's region + +Kafka Configuration: +- `kafka.topic`: The Kafka topic to consume messages from +- `kafka.bootstrapServers`: Comma-separated list of Kafka broker URLs +- `kafka.consumerGroup`: Consumer group ID for the Kafka consumer +- `kafka.autoOffsetReset`: What to do when there is no initial offset (earliest or latest) + +IBM COS Source Configuration (for ibm-cos-to-log.camel.yaml): +- `ibm.deleteAfterRead`: Delete objects after consuming them (default: true) +- `ibm.moveAfterRead`: Move objects to another bucket instead of deleting (default: false) +- `ibm.prefix`: Only consume objects with keys starting with this prefix (optional) +- `ibm.delay`: Delay in milliseconds between polling (default: 5000) +- `ibm.maxMessagesPerPoll`: Maximum number of objects to consume per poll (default: 10) + +== Quick Start + +Here's a quick walkthrough to run the Kafka to IBM COS example: + +1. Start Kafka infrastructure: ++ +[source,shell] +---- +$ camel infra run kafka +---- + +2. Run the Camel integration: ++ +[source,shell] +---- +$ jbang -Dcamel.jbang.version=4.16.0-SNAPSHOT camel@apache/camel run --properties=application.properties kafka-to-ibm-cos.camel.yaml ibm-cos-sink.kamelet.yaml +---- + +3. In another terminal, send test messages: ++ +[source,shell] +---- +$ docker exec -it camel-infra-kafka-kafka-1 /opt/kafka/bin/kafka-console-producer.sh \ + --topic ibm-cos-topic \ + --bootstrap-server localhost:9092 +---- ++ +Then type your messages and press Enter after each one. + +4. When done, stop Kafka: ++ +[source,shell] +---- +$ camel infra stop kafka +---- + +=== Running the IBM COS Source to Log Example + +You can run the source example using: + +[source,shell] +---- +$ jbang -Dcamel.jbang.version=4.16.0-SNAPSHOT camel@apache/camel run --properties=application.properties ibm-cos-to-log.camel.yaml ibm-cos-source.kamelet.yaml +---- + +=== Running Both Examples + +Note: Running both examples simultaneously (using `camel run *`) is not recommended as they will interfere with each other (source consuming what sink produces). Run them separately or configure them to use different buckets. + +== Developer Web Console + +You can enable the developer console via `--console` flag as shown: + +[source,shell] +---- +$ camel run * --console --deps=camel:kamelet +---- + +Then you can browse: http://localhost:8080/q/dev to introspect the running Camel application. + +== What happens + +=== Kafka to IBM COS Sink Example + +The Kafka source consumes messages from the configured Kafka topic. + +Each message is uploaded to your IBM COS bucket with a filename like `kafka-20241029-143025-123.txt` (timestamp-based with milliseconds). + +You should see output similar to: + +[source,shell] +---- +2024-10-29 14:30:25.123 INFO 12345 --- [kafka-consumer] route1 : Uploading to IBM COS: kafka-20241029-143025-123.txt with body: Message from Kafka +2024-10-29 14:30:26.456 INFO 12345 --- [kafka-consumer] route1 : Successfully uploaded file to IBM COS bucket +---- + +You can verify the uploaded files in the IBM Cloud console by navigating to your bucket. + +==== Send Test Messages to Kafka + +To test the example, send some messages to the Kafka topic: + +[source,shell] +---- +$ docker exec -it camel-infra-kafka-kafka-1 /opt/kafka/bin/kafka-console-producer.sh \ + --topic ibm-cos-topic \ + --bootstrap-server localhost:9092 +---- + +This will open an interactive console where you can type messages. Each line you type and press Enter will be sent as a message to Kafka: + +[source,text] +---- +> Hello from Kafka! +> First message +> Second message +> Third message +---- + +Press Ctrl+C to exit the producer console. + +Each message will be consumed by the Camel route and uploaded to IBM COS. + +=== IBM COS Source to Log Example + +The IBM COS source polls the bucket every 5 seconds (configurable via `delay` parameter) and retrieves up to 10 objects per poll (configurable via `maxMessagesPerPoll`). + +For each object found: +1. The object is downloaded from IBM COS +2. Metadata headers are set (bucket name, key, content type, ETag) +3. The object content and headers are logged +4. By default, the object is deleted from IBM COS (configurable via `deleteAfterRead`) + +You should see output similar to: + +[source,shell] +---- +2024-10-29 14:30:25.123 INFO 12345 --- [ibm-cos-source] route1 : Received file from IBM COS - Key: test-file.txt, Bucket: my-camel-bucket, ContentType: text/plain +2024-10-29 14:30:25.456 INFO 12345 --- [ibm-cos-source] info : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: Hello from IBM Cloud Object Storage!] +---- + +== Cleanup + +When you're done with the example, you can stop the Kafka infrastructure: + +[source,shell] +---- +$ camel infra stop kafka +---- + +This will stop and remove the Kafka broker. + +== Help and contributions + +If you hit any problem using Camel or have some feedback, then please +https://camel.apache.org/community/support/[let us know]. + +We also love contributors, so +https://camel.apache.org/community/contributing/[get involved] :-) + +The Camel riders! diff --git a/ibm/ibm-cos/application.properties b/ibm/ibm-cos/application.properties new file mode 100644 index 0000000..ae7b6d7 --- /dev/null +++ b/ibm/ibm-cos/application.properties @@ -0,0 +1,39 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Kafka Configuration +kafka.topic=ibm-cos-topic +kafka.bootstrapServers=localhost:9092 +kafka.consumerGroup=kafka-ibm-cos-group +kafka.autoOffsetReset=earliest + +# IBM Cloud Object Storage Configuration +ibm.bucketName=my-camel-bucket +ibm.apiKey=<apikey> +ibm.serviceInstanceId=<serviceinstance> +ibm.endpointUrl=https://s3.eu-de.cloud-object-storage.appdomain.cloud +ibm.location=eu-de + +# Sink Configuration +ibm.autoCreateBucket=false + +# Source Configuration +ibm.deleteAfterRead=true +ibm.moveAfterRead=false +ibm.prefix= +ibm.delay=5000 +ibm.maxMessagesPerPoll=10 diff --git a/ibm/ibm-cos/ibm-cos-sink.kamelet.yaml b/ibm/ibm-cos/ibm-cos-sink.kamelet.yaml new file mode 100644 index 0000000..a5a30f2 --- /dev/null +++ b/ibm/ibm-cos/ibm-cos-sink.kamelet.yaml @@ -0,0 +1,134 @@ +# --------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# --------------------------------------------------------------------------- + +apiVersion: camel.apache.org/v1 +kind: Kamelet +metadata: + name: ibm-cos-sink + annotations: + camel.apache.org/kamelet.support.level: "Preview" + camel.apache.org/catalog.version: "4.16.0-SNAPSHOT" + camel.apache.org/kamelet.icon: "data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjUwMCIgaGVpZ2h0PSIyNTAwIiB2aWV3Qm94PSIwIDAgMjU2IDI1NiIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiBwcmVzZXJ2ZUFzcGVjdFJhdGlvPSJ4TWlkWU1pZCI+PHBhdGggZD0iTTAgMGgyNTZ2MjU2SDB6IiBmaWxsPSIjMDUzMzYyIi8+PHBhdGggZD0iTTE4Ny43MzIgMTI1LjkxOGMtLjAzNi0xLjM1Ny0uMDktMi43MTQtLjE4LTQuMDcyLS4wMzYtLjUyMi0uMDkxLTEuMDQ1LS4xMjgtMS41NjctLjA1NC0uNjEzLS4xMDgtMS4yMjYtLjE4LTEuODM5LS4wNzItLjc3LS4xOC0xLjU0LS4yODgtMi4zMWEzOS43NjYgMzkuNz [...] + camel.apache.org/provider: "Apache Software Foundation" + camel.apache.org/kamelet.group: "IBM Cloud Object Storage" + camel.apache.org/kamelet.namespace: "IBM" + labels: + camel.apache.org/kamelet.type: "sink" +spec: + definition: + title: "IBM Cloud Object Storage Sink" + description: Upload data to an IBM Cloud Object Storage Bucket. + required: + - bucketName + - apiKey + - serviceInstanceId + - endpointUrl + type: object + properties: + bucketName: + title: Bucket Name + description: The IBM COS Bucket name. + type: string + apiKey: + title: API Key + description: IBM Cloud API Key for authentication. + type: string + format: password + x-descriptors: + - urn:camel:group:credentials + serviceInstanceId: + title: Service Instance ID + description: IBM COS Service Instance ID (CRN). + type: string + format: password + x-descriptors: + - urn:camel:group:credentials + endpointUrl: + title: Endpoint URL + description: IBM COS Endpoint URL (e.g., https://s3.us-south.cloud-object-storage.appdomain.cloud). + type: string + example: "https://s3.us-south.cloud-object-storage.appdomain.cloud"; + location: + title: Location + description: IBM COS Location/Region (e.g., us-south, eu-gb). + type: string + example: "us-south" + autoCreateBucket: + title: Autocreate Bucket + description: Specifies to automatically create the IBM COS bucket. + type: boolean + default: false + keyName: + title: Key Name + description: The key name for saving an element in the bucket. + type: string + multiPartUpload: + title: Multi-Part Upload + description: Use multi-part upload for large files. + type: boolean + default: false + partSize: + title: Part Size + description: Part size for multi-part uploads (default 25MB). + type: integer + default: 26214400 + storageClass: + title: Storage Class + description: The storage class to use when storing objects (e.g., STANDARD, VAULT, COLD, FLEX). + type: string + dependencies: + - "camel:core" + - "camel:ibm-cos" + - "camel:kamelet" + template: + from: + uri: "kamelet:source" + steps: + - choice: + precondition: true + when: + - simple: '${propertiesExist:!keyName}' + steps: + - choice: + when: + - simple: "${header[file]}" + steps: + - setHeader: + name: CamelIbmCosKey + simple: "${header[file]}" + - simple: "${header[ce-file]}" + steps: + - setHeader: + name: CamelIbmCosKey + simple: "${header[ce-file]}" + otherwise: + steps: + - setHeader: + name: CamelIbmCosKey + simple: "${exchangeId}" + - to: + uri: "ibm-cos:{{bucketName}}" + parameters: + apiKey: "{{apiKey}}" + serviceInstanceId: "{{serviceInstanceId}}" + endpointUrl: "{{endpointUrl}}" + location: "{{?location}}" + autoCreateBucket: "{{autoCreateBucket}}" + keyName: "{{?keyName}}" + multiPartUpload: "{{multiPartUpload}}" + partSize: "{{partSize}}" + storageClass: "{{?storageClass}}" diff --git a/ibm/ibm-cos/ibm-cos-source.kamelet.yaml b/ibm/ibm-cos/ibm-cos-source.kamelet.yaml new file mode 100644 index 0000000..24e24c6 --- /dev/null +++ b/ibm/ibm-cos/ibm-cos-source.kamelet.yaml @@ -0,0 +1,177 @@ +# --------------------------------------------------------------------------- +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# --------------------------------------------------------------------------- + +apiVersion: camel.apache.org/v1 +kind: Kamelet +metadata: + name: ibm-cos-source + annotations: + camel.apache.org/kamelet.support.level: "Preview" + camel.apache.org/catalog.version: "4.16.0-SNAPSHOT" + camel.apache.org/kamelet.icon: "data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjUwMCIgaGVpZ2h0PSIyNTAwIiB2aWV3Qm94PSIwIDAgMjU2IDI1NiIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiBwcmVzZXJ2ZUFzcGVjdFJhdGlvPSJ4TWlkWU1pZCI+PHBhdGggZD0iTTAgMGgyNTZ2MjU2SDB6IiBmaWxsPSIjMDUzMzYyIi8+PHBhdGggZD0iTTE4Ny43MzIgMTI1LjkxOGMtLjAzNi0xLjM1Ny0uMDktMi43MTQtLjE4LTQuMDcyLS4wMzYtLjUyMi0uMDkxLTEuMDQ1LS4xMjgtMS41NjctLjA1NC0uNjEzLS4xMDgtMS4yMjYtLjE4LTEuODM5LS4wNzItLjc3LS4xOC0xLjU0LS4yODgtMi4zMWEzOS43NjYgMzkuNz [...] + camel.apache.org/provider: "Apache Software Foundation" + camel.apache.org/kamelet.group: "IBM Cloud Object Storage" + camel.apache.org/kamelet.namespace: "IBM" + labels: + camel.apache.org/kamelet.type: "source" +spec: + definition: + title: "IBM Cloud Object Storage Source" + description: Receive data from an IBM Cloud Object Storage Bucket. + required: + - bucketName + - apiKey + - serviceInstanceId + - endpointUrl + type: object + properties: + bucketName: + title: Bucket Name + description: The IBM COS Bucket name. + type: string + apiKey: + title: API Key + description: IBM Cloud API Key for authentication. + type: string + format: password + x-descriptors: + - urn:camel:group:credentials + serviceInstanceId: + title: Service Instance ID + description: IBM COS Service Instance ID (CRN). + type: string + format: password + x-descriptors: + - urn:camel:group:credentials + endpointUrl: + title: Endpoint URL + description: IBM COS Endpoint URL (e.g., https://s3.us-south.cloud-object-storage.appdomain.cloud). + type: string + example: "https://s3.us-south.cloud-object-storage.appdomain.cloud"; + location: + title: Location + description: IBM COS Location/Region (e.g., us-south, eu-gb). + type: string + example: "us-south" + deleteAfterRead: + title: Auto-delete Objects + description: Specifies to delete objects after consuming them. + type: boolean + default: true + moveAfterRead: + title: Move Objects After Read + description: Move objects from IBM COS bucket to a different bucket after they have been retrieved. + type: boolean + default: false + destinationBucket: + title: Destination Bucket + description: Define the destination bucket where an object must be moved when moveAfterRead is set to true. + type: string + destinationBucketPrefix: + title: Destination Bucket Prefix + description: Define the destination bucket prefix to use when an object must be moved, and moveAfterRead is set to true. + type: string + destinationBucketSuffix: + title: Destination Bucket Suffix + description: Define the destination bucket suffix to use when an object must be moved, and moveAfterRead is set to true. + type: string + autoCreateBucket: + title: Autocreate Bucket + description: Specifies to automatically create the IBM COS bucket. + type: boolean + default: false + prefix: + title: Prefix + description: The IBM COS bucket prefix to consider while searching. + type: string + example: 'folder/' + delimiter: + title: Delimiter + description: The delimiter to use for listing objects. + type: string + includeBody: + title: Include Body + description: If true, the object body is included in the exchange. If false, only headers are set. + type: boolean + default: true + includeFolders: + title: Include Folders + description: Include folders/directories when listing objects. + type: boolean + default: true + delay: + title: Delay + description: The number of milliseconds before the next poll of the selected bucket. + type: integer + default: 500 + maxMessagesPerPoll: + title: Max Messages Per Poll + description: Gets the maximum number of messages as a limit to poll at each polling. The default value is 10. Use 0 or a negative number to set it as unlimited. + type: integer + default: 10 + dataTypes: + out: + default: binary + headers: + CamelIbmCosBucketName: + title: IBM COS Bucket Name + description: The bucket name which has been used to retrieve objects + type: string + CamelIbmCosKey: + title: IBM COS Key + description: The key under which the retrieved object is stored. + type: string + CamelIbmCosContentType: + title: Content Type + description: The content type of the retrieved object. + default: application/octet-stream + type: string + CamelIbmCosETag: + title: ETag Value + description: The hex encoded 128-bit MD5 digest of the associated object according to RFC 1864. + type: string + types: + binary: + format: "application-octet-stream" + description: Default binary representation of the IBM COS object retrieved from the bucket. + mediaType: application/octet-stream + dependencies: + - "camel:core" + - "camel:ibm-cos" + - "camel:kamelet" + template: + from: + uri: "ibm-cos:{{bucketName}}" + parameters: + apiKey: "{{apiKey}}" + serviceInstanceId: "{{serviceInstanceId}}" + endpointUrl: "{{endpointUrl}}" + location: "{{?location}}" + deleteAfterRead: "{{deleteAfterRead}}" + moveAfterRead: "{{moveAfterRead}}" + destinationBucket: "{{?destinationBucket}}" + destinationBucketPrefix: "{{?destinationBucketPrefix}}" + destinationBucketSuffix: "{{?destinationBucketSuffix}}" + autoCreateBucket: "{{autoCreateBucket}}" + prefix: "{{?prefix}}" + delimiter: "{{?delimiter}}" + includeBody: "{{includeBody}}" + includeFolders: "{{includeFolders}}" + delay: "{{delay}}" + maxMessagesPerPoll: "{{maxMessagesPerPoll}}" + steps: + - to: "kamelet:sink" diff --git a/ibm/ibm-cos/ibm-cos-to-log.camel.yaml b/ibm/ibm-cos/ibm-cos-to-log.camel.yaml new file mode 100644 index 0000000..0ebe2f6 --- /dev/null +++ b/ibm/ibm-cos/ibm-cos-to-log.camel.yaml @@ -0,0 +1,39 @@ +## --------------------------------------------------------------------------- +## Licensed to the Apache Software Foundation (ASF) under one or more +## contributor license agreements. See the NOTICE file distributed with +## this work for additional information regarding copyright ownership. +## The ASF licenses this file to You under the Apache License, Version 2.0 +## (the "License"); you may not use this file except in compliance with +## the License. You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## --------------------------------------------------------------------------- + +- route: + from: + uri: "kamelet:ibm-cos-source" + parameters: + bucketName: "{{ibm.bucketName}}" + apiKey: "{{ibm.apiKey}}" + serviceInstanceId: "{{ibm.serviceInstanceId}}" + endpointUrl: "{{ibm.endpointUrl}}" + location: "{{ibm.location}}" + deleteAfterRead: "{{ibm.deleteAfterRead:true}}" + moveAfterRead: "{{ibm.moveAfterRead:false}}" + prefix: "{{ibm.prefix:}}" + delay: "{{ibm.delay:5000}}" + maxMessagesPerPoll: "{{ibm.maxMessagesPerPoll:10}}" + steps: + - log: + message: "Received file from IBM COS - Key: ${header.CamelIbmCosKey}, Bucket: ${header.CamelIbmCosBucketName}, ContentType: ${header.CamelIbmCosContentType}" + - to: + uri: "kamelet:log-sink" + parameters: + showStreams: true + showHeaders: true diff --git a/ibm/ibm-cos/kafka-to-ibm-cos.camel.yaml b/ibm/ibm-cos/kafka-to-ibm-cos.camel.yaml new file mode 100644 index 0000000..46d0eca --- /dev/null +++ b/ibm/ibm-cos/kafka-to-ibm-cos.camel.yaml @@ -0,0 +1,42 @@ +## --------------------------------------------------------------------------- +## Licensed to the Apache Software Foundation (ASF) under one or more +## contributor license agreements. See the NOTICE file distributed with +## this work for additional information regarding copyright ownership. +## The ASF licenses this file to You under the Apache License, Version 2.0 +## (the "License"); you may not use this file except in compliance with +## the License. You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## --------------------------------------------------------------------------- + +- route: + from: + uri: "kamelet:kafka-not-secured-source" + parameters: + topic: "{{kafka.topic}}" + bootstrapServers: "{{kafka.bootstrapServers}}" + consumerGroup: "{{kafka.consumerGroup:kafka-ibm-cos-group}}" + autoOffsetReset: "{{kafka.autoOffsetReset:earliest}}" + steps: + - setHeader: + name: CamelIbmCosKey + simple: "kafka-${date:now:yyyyMMdd-HHmmss-SSS}.txt" + - log: + message: "Uploading to IBM COS: ${header.CamelIbmCosKey} with body: ${body}" + - to: + uri: "kamelet:ibm-cos-sink" + parameters: + bucketName: "{{ibm.bucketName}}" + apiKey: "{{ibm.apiKey}}" + serviceInstanceId: "{{ibm.serviceInstanceId}}" + endpointUrl: "{{ibm.endpointUrl}}" + location: "{{ibm.location}}" + autoCreateBucket: "{{ibm.autoCreateBucket:false}}" + - log: + message: "Successfully uploaded file to IBM COS bucket"
