Currently I think so.
Il giorno dom 27 dic 2020 alle ore 17:32 Arundhati Bhende <
arundhati.bhe...@prudential.com> ha scritto:
> Thanks. So my only option is the second one I mentioned earlier - write
> an application to read 1 message from this topic and do processing on that
> message and create individual messages in a different topic?
>
> On 12/27/20, 11:13 AM, "Andrea Cosentino" <anco...@gmail.com> wrote:
>
> I don't think it is feasible to do that at transformers level, we need
> to
> introduce that at camel level eventually.
>
> Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > In order to overcome - "There is no splitting support actually.
> So it
> > will be something like 1 file
> > - 1 message."
> >
> > Is there a way - like using transformer ( I have never used a
> transformer,
> > but wondering if it will help here ) - to split the message into
> multiple
> > messages? If it is possible, is there an example I could look at to
> define
> > how I can use it?
> >
> > -----Original Message-----
> > From: Andrea Cosentino <anco...@gmail.com>
> > Sent: Saturday, December 26, 2020 10:19 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > You can use any of the dataformat provided by camel or converters
> from
> > kafka connect. So even avro or json.
> >
> > There is no splitting support actually. So it will be something like
> 1 file
> > - 1 message.
> >
> >
> >
> > Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> > arundhati.bhe...@prudential.com> ha scritto:
> >
> > > Thank you.
> > >
> > > I will test with setting the autocloseBody to false. In that case,
> > > will I be able to use any valid converter ? What I mean is
> currently
> > > I am testing with a simple text file and hence was trying with
> > > StringConverter, but in reality, those maybe JSON on Avro
> formatted
> > > messages in the files, so will I be able to use those formats as
> > converters?
> > >
> > > Other part of the question - after I get a simple example working
> - a
> > > single file will contain multiple records, one record per line -
> do I
> > > need to set-up any other property to convert each line to a
> separate
> > > message to the topic?
> > >
> > > One doubt / question - to use the camel-aws2-s3 connector, our aws
> set-up
> > > must be at version 2, right? Because, currently it is at v1 and
> do not
> > > know, if we have immediate option of moving to v2
> > >
> > >
> > >
> > > On 12/26/20, 10:00 AM, "Andrea Cosentino" <anco...@gmail.com>
> wrote:
> > >
> > > Yes, the approach without url is correct
> > > Autoclosebody will close s3object after consuming the
> payload, so
> > > you want
> > > to get something it must be equal to false, while using the
> > converter.
> > >
> > > My suggestion by the way is using the camel-aws2-s3 connector,
> > > based on ask
> > > v2
> > >
> > > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > > arundhati.bhe...@prudential.com>
> > > ha scritto:
> > >
> > > > This is the full configuration that I have used. I am been
> > > experimenting
> > > > with changing the value.converter between StringConverter and
> > > > S3ObjectConverter - otherwise everything is same.
> > > >
> > > >
> > > >
> > > > DATA=$( cat << EOF
> > > > {
> > > > "connector.class":
> > > >
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > > "key.converter":
> > > > "org.apache.kafka.connect.storage.StringConverter",
> > > > "value.converter":
> > > > "org.apache.kafka.connect.storage.StringConverter",
> > > > "camel.source.maxPollDuration": "10000",
> > > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > > "camel.source.path.bucketNameOrArn": "
> push-json-poc",
> > > > "camel.component.aws-s3.region": "US_EAST_1",
> > > > "tasks.max": "1",
> > > > "camel.source.endpoint.useIAMCredentials": "true",
> > > > "camel.source.endpoint.autocloseBody": "true"
> > > > }
> > > > EOF
> > > > )
> > > > # "value.converter":
> > > >
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > > >
> > > >
> > > > This is official one from the link in the other email.
> > > > name=CamelAWSS3SourceConnector
> > > >
> > > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > >
> key.converter=org.apache.kafka.connect.storage.StringConverter
> > > >
> > > >
> > >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > > >
> > > > camel.source.maxPollDuration=10000
> > > >
> > > > topics=mytopic
> > > >
> > > >
> camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > > >
> > > > camel.component.aws-s3.access-key=xxxx
> > > > camel.component.aws-s3.secret-key=yyyy
> > > > camel.component.aws-s3.region=EU_WEST_1
> > > >
> > > >
> > > >
> > > >
> > > > I can see two differences -
> > > > 1. How the AWS credentials are set-up
> > > > 2. In my configuration
> > > > I am using camel.source.path.bucketNameOrArn instead
> of
> > > > camel.source.url
> > > > Because I was getting error that bucketNameOrArn
> must be
> > > defined
> > > >
> > > > In my configuration - autocloseBody is set true to
> whereas
> > > in the
> > > > default configuration it is set to false
> > > >
> > > > How does autocloseBody work? What impact does it
> have when
> > > > setting it to true vs. false?
> > > >
> > > >
> > > > Thank you
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Andrea Cosentino <anco...@gmail.com>
> > > > Sent: Saturday, December 26, 2020 3:59 AM
> > > > To: users@camel.apache.org
> > > > Subject: Re: Camel S3 Source Connector - value.converter
> > > >
> > > > Can you show your full configuration?
> > > >
> > > > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > > > arundhati.bhe...@prudential.com> ha scritto:
> > > >
> > > > > I am trying to test The Camel S3 Source connector using
> basic
> > > example
> > > > > from this link
> > > > >
> > > >
> > >
> >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2e13a12f628e4014a5ab08d8aa824c14%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637446823884910393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UQd42qkez%2Fz0NblRLpqFlDru8Nj5duluzGQ8i3uVe%2Bg%3D&reserved=0
> > > > .
> > > > > Below is the file I am uploading to S3.
> > > > >
> > > > > testfile.txt
> > > > > add one line
> > > > > add another line
> > > > > test the connector
> > > > >
> > > > >
> > > > > After the connector picks up and processes the file, If I
> use
> > > this for
> > > > > value converter
> > > > >
> > > > >
> > > > >
> > >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > > > ectConverter
> > > > >
> > > > > When I run the consumer, it shows the message processed but
> > > the output
> > > > > shows blank.
> > > > >
> > > > > If I use the String converter as value converter
> > > > >
> > > > > org.apache.kafka.connect.storage.StringConverter
> > > > >
> > > > > I get the output in the form of
> > > > >
> > > > >
> com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > > >
> com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > > > >
> > > > > From the example in the above link, the author doesn't
> seem to
> > > have to
> > > > > do any actions to see the string.
> > > > >
> > > > > So, why would the S3ObjectConverter show a blank output for
> > > me? I can
> > > > > understand the String representation for the object really
> > > shows the
> > > > > memory address.
> > > > >
> > > > > What do I need to look at to fix the output.
> > > > >
> > > > > Also, is it possible to change the converter to JSON or
> Avro
> > > converter
> > > > > in the configuration file or I need to add either different
> > > properties
> > > > > to the configuration or do some processing afterwards?
> > > > >
> > > > > Thank You
> > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>
