Thanks. So my only option is the second one I mentioned earlier - write an
application to read 1 message from this topic and do processing on that message
and create individual messages in a different topic?
On 12/27/20, 11:13 AM, "Andrea Cosentino" <anco...@gmail.com> wrote:
I don't think it is feasible to do that at transformers level, we need to
introduce that at camel level eventually.
Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
arundhati.bhe...@prudential.com> ha scritto:
> In order to overcome - "There is no splitting support actually. So it
> will be something like 1 file
> - 1 message."
>
> Is there a way - like using transformer ( I have never used a transformer,
> but wondering if it will help here ) - to split the message into multiple
> messages? If it is possible, is there an example I could look at to
define
> how I can use it?
>
> -----Original Message-----
> From: Andrea Cosentino <anco...@gmail.com>
> Sent: Saturday, December 26, 2020 10:19 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> You can use any of the dataformat provided by camel or converters from
> kafka connect. So even avro or json.
>
> There is no splitting support actually. So it will be something like 1
file
> - 1 message.
>
>
>
> Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > Thank you.
> >
> > I will test with setting the autocloseBody to false. In that case,
> > will I be able to use any valid converter ? What I mean is currently
> > I am testing with a simple text file and hence was trying with
> > StringConverter, but in reality, those maybe JSON on Avro formatted
> > messages in the files, so will I be able to use those formats as
> converters?
> >
> > Other part of the question - after I get a simple example working - a
> > single file will contain multiple records, one record per line - do I
> > need to set-up any other property to convert each line to a separate
> > message to the topic?
> >
> > One doubt / question - to use the camel-aws2-s3 connector, our aws
set-up
> > must be at version 2, right? Because, currently it is at v1 and do not
> > know, if we have immediate option of moving to v2
> >
> >
> >
> > On 12/26/20, 10:00 AM, "Andrea Cosentino" <anco...@gmail.com> wrote:
> >
> > Yes, the approach without url is correct
> > Autoclosebody will close s3object after consuming the payload, so
> > you want
> > to get something it must be equal to false, while using the
> converter.
> >
> > My suggestion by the way is using the camel-aws2-s3 connector,
> > based on ask
> > v2
> >
> > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > arundhati.bhe...@prudential.com>
> > ha scritto:
> >
> > > This is the full configuration that I have used. I am been
> > experimenting
> > > with changing the value.converter between StringConverter and
> > > S3ObjectConverter - otherwise everything is same.
> > >
> > >
> > >
> > > DATA=$( cat << EOF
> > > {
> > > "connector.class":
> > > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > "key.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "value.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "camel.source.maxPollDuration": "10000",
> > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > > "camel.component.aws-s3.region": "US_EAST_1",
> > > "tasks.max": "1",
> > > "camel.source.endpoint.useIAMCredentials": "true",
> > > "camel.source.endpoint.autocloseBody": "true"
> > > }
> > > EOF
> > > )
> > > # "value.converter":
> > >
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > >
> > >
> > > This is official one from the link in the other email.
> > > name=CamelAWSS3SourceConnector
> > >
> > >
> >
>
connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > key.converter=org.apache.kafka.connect.storage.StringConverter
> > >
> > >
> >
>
value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > camel.source.maxPollDuration=10000
> > >
> > > topics=mytopic
> > >
> > >
camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > >
> > > camel.component.aws-s3.access-key=xxxx
> > > camel.component.aws-s3.secret-key=yyyy
> > > camel.component.aws-s3.region=EU_WEST_1
> > >
> > >
> > >
> > >
> > > I can see two differences -
> > > 1. How the AWS credentials are set-up
> > > 2. In my configuration
> > > I am using camel.source.path.bucketNameOrArn instead of
> > > camel.source.url
> > > Because I was getting error that bucketNameOrArn must be
> > defined
> > >
> > > In my configuration - autocloseBody is set true to whereas
> > in the
> > > default configuration it is set to false
> > >
> > > How does autocloseBody work? What impact does it have
when
> > > setting it to true vs. false?
> > >
> > >
> > > Thank you
> > >
> > >
> > > -----Original Message-----
> > > From: Andrea Cosentino <anco...@gmail.com>
> > > Sent: Saturday, December 26, 2020 3:59 AM
> > > To: users@camel.apache.org
> > > Subject: Re: Camel S3 Source Connector - value.converter
> > >
> > > Can you show your full configuration?
> > >
> > > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > > arundhati.bhe...@prudential.com> ha scritto:
> > >
> > > > I am trying to test The Camel S3 Source connector using basic
> > example
> > > > from this link
> > > >
> > >
> >
>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdzone.com%2Farticles%2Freading-aws-s3-file-content-to-kafka-topic&data=04%7C01%7Carundhati.bhende%40prudential.com%7C2e13a12f628e4014a5ab08d8aa824c14%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637446823884910393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UQd42qkez%2Fz0NblRLpqFlDru8Nj5duluzGQ8i3uVe%2Bg%3D&reserved=0
> > > .
> > > > Below is the file I am uploading to S3.
> > > >
> > > > testfile.txt
> > > > add one line
> > > > add another line
> > > > test the connector
> > > >
> > > >
> > > > After the connector picks up and processes the file, If I use
> > this for
> > > > value converter
> > > >
> > > >
> > > >
> > value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3Obj
> > > > ectConverter
> > > >
> > > > When I run the consumer, it shows the message processed but
> > the output
> > > > shows blank.
> > > >
> > > > If I use the String converter as value converter
> > > >
> > > > org.apache.kafka.connect.storage.StringConverter
> > > >
> > > > I get the output in the form of
> > > >
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@522e135a
> > > > com.amazonaws.services.s3.model.S3ObjectInputStream@6c6cf103
> > > >
> > > > From the example in the above link, the author doesn't seem to
> > have to
> > > > do any actions to see the string.
> > > >
> > > > So, why would the S3ObjectConverter show a blank output for
> > me? I can
> > > > understand the String representation for the object really
> > shows the
> > > > memory address.
> > > >
> > > > What do I need to look at to fix the output.
> > > >
> > > > Also, is it possible to change the converter to JSON or Avro
> > converter
> > > > in the configuration file or I need to add either different
> > properties
> > > > to the configuration or do some processing afterwards?
> > > >
> > > > Thank You
> > > >
> > > >
> > >
> >
> >
>
