Yes, the approach without url is correct
Autoclosebody will close s3object after consuming the payload, so you want
to get something it must be equal to false, while using the converter.
My suggestion by the way is using the camel-aws2-s3 connector, based on ask
v2
Il sab 26 dic 2020, 15:53 Arundhati Bhende <arundhati.bhe...@prudential.com>
ha scritto:
> This is the full configuration that I have used. I am been experimenting
> with changing the value.converter between StringConverter and
> S3ObjectConverter - otherwise everything is same.
>
>
>
> DATA=$( cat << EOF
> {
> "connector.class":
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> "key.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "value.converter":
> "org.apache.kafka.connect.storage.StringConverter",
> "camel.source.maxPollDuration": "10000",
> "topics": "TEST-S3-SOURCE-DZONE-POC",
> "camel.source.path.bucketNameOrArn": " push-json-poc",
> "camel.component.aws-s3.region": "US_EAST_1",
> "tasks.max": "1",
> "camel.source.endpoint.useIAMCredentials": "true",
> "camel.source.endpoint.autocloseBody": "true"
> }
> EOF
> )
> # "value.converter":
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
>
>
> This is official one from the link in the other email.
> name=CamelAWSS3SourceConnector
>
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> key.converter=org.apache.kafka.connect.storage.StringConverter
>
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> camel.source.maxPollDuration=10000
>
> topics=mytopic
>
> camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
>
> camel.component.aws-s3.access-key=xxxx
> camel.component.aws-s3.secret-key=yyyy
> camel.component.aws-s3.region=EU_WEST_1
>
>
>
>
> I can see two differences -
> 1. How the AWS credentials are set-up
> 2. In my configuration
> I am using camel.source.path.bucketNameOrArn instead of
> camel.source.url
> Because I was getting error that bucketNameOrArn must be defined
>
> In my configuration - autocloseBody is set true to whereas in the
> default configuration it is set to false
>
> How does autocloseBody work? What impact does it have when
> setting it to true vs. false?
>
>
> Thank you
>
>
> -----Original Message-----
> From: Andrea Cosentino <anco...@gmail.com>
> Sent: Saturday, December 26, 2020 3:59 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> Can you show your full configuration?
>
> Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > I am trying to test The Camel S3 Source connector using basic example
> > from this link
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2fe91ef852ff491ba73608d8a97c923a%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445700040770003%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=84qab73sSiJnvgsTZGVcraOjp8tEMuLmNyeFOrYHZbM%3D&reserved=0
> .
> > Below is the file I am uploading to S3.
> >
> > testfile.txt
> > add one line
> > add another line
> > test the connector
> >
> >
> > After the connector picks up and processes the file, If I use this for
> > value converter
> >
> >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > ectConverter
> >
> > When I run the consumer, it shows the message processed but the output
> > shows blank.
> >
> > If I use the String converter as value converter
> >
> > org.apache.kafka.connect.storage.StringConverter
> >
> > I get the output in the form of
> >
> > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> >
> > From the example in the above link, the author doesn't seem to have to
> > do any actions to see the string.
> >
> > So, why would the S3ObjectConverter show a blank output for me? I can
> > understand the String representation for the object really shows the
> > memory address.
> >
> > What do I need to look at to fix the output.
> >
> > Also, is it possible to change the converter to JSON or Avro converter
> > in the configuration file or I need to add either different properties
> > to the configuration or do some processing afterwards?
> >
> > Thank You
> >
> >
>
