ajithcnambiar opened a new issue, #1570:
URL: https://github.com/apache/camel-kafka-connector/issues/1570
**Usecase**:
I'm trying out the S3 source connector. The S3 bucket will be periodically
updated, and I want the new files to be sourced to the Kafka topic, without
duplicates, without deleting from the existing S3 bucket, and without moving to
a new bucket.
**Test**
My test was with the deleteAfterRead `false` and with idempotency enabled
(with the Kafka type repository), with the below configuration:
**Configuration**:
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: aws-s3-source-connector
namespace: kafka
labels:
strimzi.io/cluster: kafka-connect
spec:
class:
org.apache.camel.kafkaconnector.awss3source.CamelAwss3sourceSourceConnector
tasksMax: 1
config:
camel.kamelet.aws-s3-source.accessKey: <access-key>
camel.kamelet.aws-s3-source.secretKey: <secret-key>
camel.kamelet.aws-s3-source.region: <region>
camel.kamelet.aws-s3-source.deleteAfterRead: false
camel.kamelet.aws-s3-source.bucketNameOrArn: arn:aws:s3:::<bucket-name>
camel.idempotency.enabled: true
camel.idempotency.repository.type: kafka
camel.idempotency.expression.type: header
camel.idempotency.expression.header: CamelAwsS3Key
camel.idempotency.kafka.topic: idem-topic
camel.idempotency.kafka.bootstrap.servers: <kafka-servers>:9092
camel.idempotency.kafka.poll.duration.ms: 150
topics: bucket-topic
```
**Other info**:
- My reference was [this old
thread](https://github.com/apache/camel-kafka-connector/issues/311) and [the
idempotency blog](https://camel.apache.org/blog/2020/12/CKC-idempotency-070/)
to achieve the intended functionality.
- Could it be because `maxMessagesPerPoll` is 10? But then there seems to be
no configuration property to set this for S3 source connector? 🤔
**Versions tested**
camel-aws2-s3-kafka-connector 0.11.5
camel-aws-s3-source-kafka-connector 3.20.6
camel-aws-s3-source-kafka-connector 4.0.0
**Question**
Please let me know if the intended functionality can be realized. And if so,
what am I missing? Kindly advise 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
