In order to overcome - "There is no splitting support actually. So it will
be something like 1 file
- 1 message."
Is there a way - like using transformer ( I have never used a transformer, but
wondering if it will help here ) - to split the message into multiple messages?
If it is possible, is there an example I could look at to define how I can
use it?
-----Original Message-----
From: Andrea Cosentino <anco...@gmail.com>
Sent: Saturday, December 26, 2020 10:19 AM
To: users@camel.apache.org
Subject: Re: Camel S3 Source Connector - value.converter
You can use any of the dataformat provided by camel or converters from kafka
connect. So even avro or json.
There is no splitting support actually. So it will be something like 1 file
- 1 message.
Il sab 26 dic 2020, 16:08 Arundhati Bhende <arundhati.bhe...@prudential.com> ha
scritto:
> Thank you.
>
> I will test with setting the autocloseBody to false. In that case,
> will I be able to use any valid converter ? What I mean is currently
> I am testing with a simple text file and hence was trying with
> StringConverter, but in reality, those maybe JSON on Avro formatted
> messages in the files, so will I be able to use those formats as converters?
>
> Other part of the question - after I get a simple example working - a
> single file will contain multiple records, one record per line - do I
> need to set-up any other property to convert each line to a separate
> message to the topic?
>
> One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> must be at version 2, right? Because, currently it is at v1 and do not
> know, if we have immediate option of moving to v2
>
>
>
> On 12/26/20, 10:00 AM, "Andrea Cosentino" <anco...@gmail.com> wrote:
>
> Yes, the approach without url is correct
> Autoclosebody will close s3object after consuming the payload, so
> you want
> to get something it must be equal to false, while using the converter.
>
> My suggestion by the way is using the camel-aws2-s3 connector,
> based on ask
> v2
>
> Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> arundhati.bhe...@prudential.com>
> ha scritto:
>
> > This is the full configuration that I have used. I am been
> experimenting
> > with changing the value.converter between StringConverter and
> > S3ObjectConverter - otherwise everything is same.
> >
> >
> >
> > DATA=$( cat << EOF
> > {
> > "connector.class":
> > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > "key.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "value.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "camel.source.maxPollDuration": "10000",
> > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > "camel.component.aws-s3.region": "US_EAST_1",
> > "tasks.max": "1",
> > "camel.source.endpoint.useIAMCredentials": "true",
> > "camel.source.endpoint.autocloseBody": "true"
> > }
> > EOF
> > )
> > # "value.converter":
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> >
> >
> > This is official one from the link in the other email.
> > name=CamelAWSS3SourceConnector
> >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > key.converter=org.apache.kafka.connect.storage.StringConverter
> >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > camel.source.maxPollDuration=10000
> >
> > topics=mytopic
> >
> > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> >
> > camel.component.aws-s3.access-key=xxxx
> > camel.component.aws-s3.secret-key=yyyy
> > camel.component.aws-s3.region=EU_WEST_1
> >
> >
> >
> >
> > I can see two differences -
> > 1. How the AWS credentials are set-up
> > 2. In my configuration
> > I am using camel.source.path.bucketNameOrArn instead of
> > camel.source.url
> > Because I was getting error that bucketNameOrArn must be
> defined
> >
> > In my configuration - autocloseBody is set true to whereas
> in the
> > default configuration it is set to false
> >
> > How does autocloseBody work? What impact does it have when
> > setting it to true vs. false?
> >
> >
> > Thank you
> >
> >
> > -----Original Message-----
> > From: Andrea Cosentino <anco...@gmail.com>
> > Sent: Saturday, December 26, 2020 3:59 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > Can you show your full configuration?
> >
> > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > arundhati.bhe...@prudential.com> ha scritto:
> >
> > > I am trying to test The Camel S3 Source connector using basic
> example
> > > from this link
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C885aea82555a4e4e1b2d08d8a9b1aa84%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637445927846361197%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yr9urZgFayjVnAF72kDV87GPr1TMKzPQSj%2BrKyvD0wo%3D&reserved=0
> > .
> > > Below is the file I am uploading to S3.
> > >
> > > testfile.txt
> > > add one line
> > > add another line
> > > test the connector
> > >
> > >
> > > After the connector picks up and processes the file, If I use
> this for
> > > value converter
> > >
> > >
> > >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > ectConverter
> > >
> > > When I run the consumer, it shows the message processed but
> the output
> > > shows blank.
> > >
> > > If I use the String converter as value converter
> > >
> > > org.apache.kafka.connect.storage.StringConverter
> > >
> > > I get the output in the form of
> > >
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > >
> > > From the example in the above link, the author doesn't seem to
> have to
> > > do any actions to see the string.
> > >
> > > So, why would the S3ObjectConverter show a blank output for
> me? I can
> > > understand the String representation for the object really
> shows the
> > > memory address.
> > >
> > > What do I need to look at to fix the output.
> > >
> > > Also, is it possible to change the converter to JSON or Avro
> converter
> > > in the configuration file or I need to add either different
> properties
> > > to the configuration or do some processing afterwards?
> > >
> > > Thank You
> > >
> > >
> >
>
>
