RE: https://camel.apache.org/components/latest/eips/setHeader-eip.html

[Vikas Jaiswal]

Hi just an update on this...documentation is correct it takes 'name' and 
not 'headerName' as suggested by IDE, even though 'name' shows error in IDE.. 
it works...I am using eclipse. Has it got to do with some eclipse settings?

Regards,
Vikas

From: Vikas Jaiswal
Sent: 24 December 2020 19:49
To: 'users@camel.apache.org'
Subject: https://camel.apache.org/components/latest/eips/setHeader-eip.html

Hi,
Set Header doesn't seem to work for me. In the documentation name option is 
given but in actual in takes headerName , but it also errors out. I have tired 
with version 3.6.0 & 3.7.0

Warm Regards,

Vikas Jaiswal
Lead Consultant
[cid:image001.png@01D67243.124E3960]
T: +91 2042006145 |M: +91 9371626126 |Skype: vikasdnjaiswal
W: www.saksoft.com  |  
www.acuma.co.uk  |  
www.360logica.com |  www.dreamorbit.com
[Description: 
cid:image003.jpg@01D2806E.41A0F0B0][Description:
 cid:image004.jpg@01D2806E.41A0F0B0] 
[Description: cid:image005.jpg@01D2806E.41A0F0B0] 
  [Description: 
cid:image006.jpg@01D2806E.41A0F0B0] 
  [Description: 
cid:image007.jpg@01D2806E.41A0F0B0] 

Innovation   Customer focus   Openness   Respect   Enterprising

Re: Camel S3 Source Connector - value.converter

[Andrea Cosentino]

Currently I think so.

Il giorno dom 27 dic 2020 alle ore 17:32 Arundhati Bhende <
arundhati.bhe...@prudential.com> ha scritto:

> Thanks.  So my only option is the second one I mentioned earlier - write
> an application to read 1 message from this topic and do processing on that
> message and create individual messages in a different topic?
>
> On 12/27/20, 11:13 AM, "Andrea Cosentino"  wrote:
>
> I don't think it is feasible to do that at transformers level, we need
> to
> introduce that at camel level eventually.
>
> Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > In order to overcome -"There is no splitting support actually.
> So it
> > will be something like 1 file
> > - 1 message."
> >
> > Is there a way - like using transformer ( I have never used a
> transformer,
> > but wondering if it will help here ) - to split the message into
> multiple
> > messages?  If it is possible, is there an example I could look at to
> define
> > how  I can use it?
> >
> > -Original Message-
> > From: Andrea Cosentino 
> > Sent: Saturday, December 26, 2020 10:19 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > You can use any of the dataformat provided by camel or converters
> from
> > kafka connect. So even avro or json.
> >
> > There is no splitting support actually. So it will be something like
> 1 file
> > - 1 message.
> >
> >
> >
> > Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> > arundhati.bhe...@prudential.com> ha scritto:
> >
> > > Thank you.
> > >
> > > I will test with setting the autocloseBody to false.  In that case,
> > > will I be able to use any valid converter ?  What I mean is
> currently
> > > I am testing with a simple text file and hence was trying with
> > > StringConverter,  but in reality, those maybe JSON on Avro
> formatted
> > > messages in the files, so will I be able to use those formats as
> > converters?
> > >
> > > Other part of the question - after I get a simple example working
>  - a
> > > single file will contain multiple records, one record per line -
> do I
> > > need to set-up any other property to convert each line to a
> separate
> > > message to the topic?
> > >
> > > One doubt / question - to use the camel-aws2-s3 connector, our aws
> set-up
> > > must be at version 2, right?   Because, currently it is at v1 and
> do not
> > > know, if we have immediate option of moving to v2
> > >
> > >
> > >
> > > On 12/26/20, 10:00 AM, "Andrea Cosentino" 
> wrote:
> > >
> > > Yes, the approach without url is correct
> > >  Autoclosebody will close s3object after consuming the
> payload, so
> > > you want
> > > to get something it must be equal to false, while using the
> > converter.
> > >
> > > My suggestion by the way is using the camel-aws2-s3 connector,
> > > based on ask
> > > v2
> > >
> > > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > > arundhati.bhe...@prudential.com>
> > > ha scritto:
> > >
> > > > This is the full configuration that I have used.  I am been
> > > experimenting
> > > > with changing the value.converter between StringConverter and
> > > > S3ObjectConverter - otherwise everything is same.
> > > >
> > > >
> > > >
> > > > DATA=$( cat << EOF
> > > > {
> > > > "connector.class":
> > > >
> "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > > "key.converter":
> > > > "org.apache.kafka.connect.storage.StringConverter",
> > > > "value.converter":
> > > > "org.apache.kafka.connect.storage.StringConverter",
> > > > "camel.source.maxPollDuration": "1",
> > > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > > "camel.source.path.bucketNameOrArn": "
> push-json-poc",
> > > > "camel.component.aws-s3.region": "US_EAST_1",
> > > > "tasks.max": "1",
> > > > "camel.source.endpoint.useIAMCredentials": "true",
> > > > "camel.source.endpoint.autocloseBody": "true"
> > > > }
> > > > EOF
> > > > )
> > > > #"value.converter":
> > > >
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > > >
> > > >
> > > > This is official one from the link in the other email.
> > > > name=CamelAWSS3SourceConnector
> > > >
> > > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > >
> 

Re: Camel S3 Source Connector - value.converter

[Arundhati Bhende]

Thanks.  So my only option is the second one I mentioned earlier - write an 
application to read 1 message from this topic and do processing on that message 
and create individual messages in a different topic?

On 12/27/20, 11:13 AM, "Andrea Cosentino"  wrote:

I don't think it is feasible to do that at transformers level, we need to
introduce that at camel level eventually.

Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
arundhati.bhe...@prudential.com> ha scritto:

> In order to overcome -"There is no splitting support actually. So it
> will be something like 1 file
> - 1 message."
>
> Is there a way - like using transformer ( I have never used a transformer,
> but wondering if it will help here ) - to split the message into multiple
> messages?  If it is possible, is there an example I could look at to 
define
> how  I can use it?
>
> -Original Message-
> From: Andrea Cosentino 
> Sent: Saturday, December 26, 2020 10:19 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> You can use any of the dataformat provided by camel or converters from
> kafka connect. So even avro or json.
>
> There is no splitting support actually. So it will be something like 1 
file
> - 1 message.
>
>
>
> Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > Thank you.
> >
> > I will test with setting the autocloseBody to false.  In that case,
> > will I be able to use any valid converter ?  What I mean is currently
> > I am testing with a simple text file and hence was trying with
> > StringConverter,  but in reality, those maybe JSON on Avro formatted
> > messages in the files, so will I be able to use those formats as
> converters?
> >
> > Other part of the question - after I get a simple example working   - a
> > single file will contain multiple records, one record per line - do I
> > need to set-up any other property to convert each line to a separate
> > message to the topic?
> >
> > One doubt / question - to use the camel-aws2-s3 connector, our aws 
set-up
> > must be at version 2, right?   Because, currently it is at v1 and do not
> > know, if we have immediate option of moving to v2
> >
> >
> >
> > On 12/26/20, 10:00 AM, "Andrea Cosentino"  wrote:
> >
> > Yes, the approach without url is correct
> >  Autoclosebody will close s3object after consuming the payload, so
> > you want
> > to get something it must be equal to false, while using the
> converter.
> >
> > My suggestion by the way is using the camel-aws2-s3 connector,
> > based on ask
> > v2
> >
> > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > arundhati.bhe...@prudential.com>
> > ha scritto:
> >
> > > This is the full configuration that I have used.  I am been
> > experimenting
> > > with changing the value.converter between StringConverter and
> > > S3ObjectConverter - otherwise everything is same.
> > >
> > >
> > >
> > > DATA=$( cat << EOF
> > > {
> > > "connector.class":
> > > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > "key.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "value.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "camel.source.maxPollDuration": "1",
> > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > > "camel.component.aws-s3.region": "US_EAST_1",
> > > "tasks.max": "1",
> > > "camel.source.endpoint.useIAMCredentials": "true",
> > > "camel.source.endpoint.autocloseBody": "true"
> > > }
> > > EOF
> > > )
> > > #"value.converter":
> > >
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > >
> > >
> > > This is official one from the link in the other email.
> > > name=CamelAWSS3SourceConnector
> > >
> > >
> >
> 
connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > key.converter=org.apache.kafka.connect.storage.StringConverter
> > >
> > >
> >
> 
value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > camel.source.maxPollDuration=1
> > >
> > > topics=mytopic
> > >
> > > 
camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > >
> > > 

Re: Correction needed in Documentation for Camel Source Connector

[Andrea Cosentino]

I don't thikn  it can be done through transforms.

It must be supported on camel level eventually

Il giorno dom 27 dic 2020 alle ore 16:58 Arundhati Bhende <
arundhati.bhe...@prudential.com> ha scritto:

> Sure.
>
> Do you have any suggestion on to how can I split the lines in the file as
> single records in adding to the topic or do I have to write a separate app
> that adds reads the single message and splits it as the multiple messages
> to yet another topic
>
> If you have any suggestions, please let me know.
>
> One other place this is repeated is on the RedHat site's copy of the same
> document.  Not sure how to inform them.   Hopefully fixing it on Camel site
> will get propagated there.
>
> Thank you
>
> -Original Message-
> From: Andrea Cosentino 
> Sent: Sunday, December 27, 2020 10:48 AM
> To: users@camel.apache.org
> Subject: Re: Correction needed in Documentation for Camel Source Connector
>
> Thanks for spotting.
>
> Il dom 27 dic 2020, 16:39 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > I am not yet familiar with other capabilities of a connector, have
> > just started learning about it.
> >
> > In regards to the comment  :  You're wrong. The aggregation strategy
> > can be used in sink too.
> >
> > I am not saying it cannot be used in the sink,  what I am saying is -
> > The official page for Source Connector has Sink connector wording at
> > the bottom - which I think should have been source.
> >
> > This is what is on the documentation page which I am saying the word
> > sink is there by mistake, as well as two lines above this on the page
> >
> > The camel-aws-s3 sink connector has no aggregation strategies out of
> > the box.
> >
> >
> > -Original Message-
> > From: Andrea Cosentino 
> > Sent: Sunday, December 27, 2020 10:33 AM
> > To: users@camel.apache.org
> > Subject: Re: Correction needed in Documentation for Camel Source
> > Connector
> >
> > You're wrong. The aggregation strategy can be used in sink too.
> >
> > Il dom 27 dic 2020, 16:23 Arundhati Bhende <
> > arundhati.bhe...@prudential.com> ha scritto:
> >
> > > Hi,
> > >
> > > I do not know if this is the correct list for this, but I wanted to
> > > bring it to notice of the team that maintains the documentation.
> > >
> > > In the page -  "camel-aws-s3-kafka-connector source configuration"
> > >  at the link
> > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca
> > > me
> > > l.apache.org%2Fcamel-kafka-connector%2Flatest%2Fconnectors%2Fcamel-a
> > > ws
> > > -s3-kafka-source-connector.htmldata=04%7C01%7Carundhati.bhende%
> > > 40
> > > prudential.com%7Cec4cbc71190f49a7c11a08d8aa7cc7a0%7Cd8fde2f593924260
> > > 8a
> > > 030ad01f4746e9%7C0%7C0%7C637446800196356811%7CUnknown%7CTWFpbGZsb3d8
> > > ey
> > > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> > > 00
> > > 0sdata=USwkv0KWNDm%2FJdPEfkQOxmOYwD%2BX9jB0cS7hFYxI7c8%3Dr
> > > es
> > > erved=0  the wording at the bottom needs to be changed ( Seems to be
> > > incorrect copy /paste from Sink connector )
> > >
> > >
> > > Below is the documentation where the word sink needs to be replaced
> > > with source
> > >
> > > The camel-aws-s3 sink connector supports 1 converters out of the
> > > box, which are listed below.
> > > org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > The camel-aws-s3 sink connector supports 1 transforms out of the
> > > box, which are listed below.
> > > org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransform
> > > s
> > >
> > > The camel-aws-s3 sink connector has no aggregation strategies out of
> > > the box.
> > >
> > > Thank you
> > >
> > >
> >
>

Re: Camel S3 Source Connector - value.converter

[Andrea Cosentino]

I don't think it is feasible to do that at transformers level, we need to
introduce that at camel level eventually.

Il giorno dom 27 dic 2020 alle ore 15:36 Arundhati Bhende <
arundhati.bhe...@prudential.com> ha scritto:

> In order to overcome -"There is no splitting support actually. So it
> will be something like 1 file
> - 1 message."
>
> Is there a way - like using transformer ( I have never used a transformer,
> but wondering if it will help here ) - to split the message into multiple
> messages?  If it is possible, is there an example I could look at to define
> how  I can use it?
>
> -Original Message-
> From: Andrea Cosentino 
> Sent: Saturday, December 26, 2020 10:19 AM
> To: users@camel.apache.org
> Subject: Re: Camel S3 Source Connector - value.converter
>
> You can use any of the dataformat provided by camel or converters from
> kafka connect. So even avro or json.
>
> There is no splitting support actually. So it will be something like 1 file
> - 1 message.
>
>
>
> Il sab 26 dic 2020, 16:08 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > Thank you.
> >
> > I will test with setting the autocloseBody to false.  In that case,
> > will I be able to use any valid converter ?  What I mean is currently
> > I am testing with a simple text file and hence was trying with
> > StringConverter,  but in reality, those maybe JSON on Avro formatted
> > messages in the files, so will I be able to use those formats as
> converters?
> >
> > Other part of the question - after I get a simple example working   - a
> > single file will contain multiple records, one record per line - do I
> > need to set-up any other property to convert each line to a separate
> > message to the topic?
> >
> > One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> > must be at version 2, right?   Because, currently it is at v1 and do not
> > know, if we have immediate option of moving to v2
> >
> >
> >
> > On 12/26/20, 10:00 AM, "Andrea Cosentino"  wrote:
> >
> > Yes, the approach without url is correct
> >  Autoclosebody will close s3object after consuming the payload, so
> > you want
> > to get something it must be equal to false, while using the
> converter.
> >
> > My suggestion by the way is using the camel-aws2-s3 connector,
> > based on ask
> > v2
> >
> > Il sab 26 dic 2020, 15:53 Arundhati Bhende <
> > arundhati.bhe...@prudential.com>
> > ha scritto:
> >
> > > This is the full configuration that I have used.  I am been
> > experimenting
> > > with changing the value.converter between StringConverter and
> > > S3ObjectConverter - otherwise everything is same.
> > >
> > >
> > >
> > > DATA=$( cat << EOF
> > > {
> > > "connector.class":
> > > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > > "key.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "value.converter":
> > > "org.apache.kafka.connect.storage.StringConverter",
> > > "camel.source.maxPollDuration": "1",
> > > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > > "camel.component.aws-s3.region": "US_EAST_1",
> > > "tasks.max": "1",
> > > "camel.source.endpoint.useIAMCredentials": "true",
> > > "camel.source.endpoint.autocloseBody": "true"
> > > }
> > > EOF
> > > )
> > > #"value.converter":
> > >
> "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> > >
> > >
> > > This is official one from the link in the other email.
> > > name=CamelAWSS3SourceConnector
> > >
> > >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > > key.converter=org.apache.kafka.connect.storage.StringConverter
> > >
> > >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> > >
> > > camel.source.maxPollDuration=1
> > >
> > > topics=mytopic
> > >
> > > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> > >
> > > camel.component.aws-s3.access-key=
> > > camel.component.aws-s3.secret-key=
> > > camel.component.aws-s3.region=EU_WEST_1
> > >
> > >
> > >
> > >
> > > I can see two differences -
> > > 1.  How the AWS credentials are set-up
> > > 2.  In my configuration
> > > I am using camel.source.path.bucketNameOrArn instead of
> > > camel.source.url
> > > Because I was getting error that bucketNameOrArn must be
> > defined
> > >
> > > In my configuration - autocloseBody is set true to whereas
> > in the
> > > default configuration it is set to false
> > >
> > > How does autocloseBody work?  What impact does it have when
> > > setting 

RE: Correction needed in Documentation for Camel Source Connector

[Arundhati Bhende]

Sure.  

Do you have any suggestion on to how can I split the lines in the file as 
single records in adding to the topic or do I have to write a separate app that 
adds reads the single message and splits it as the multiple messages to yet 
another topic 

If you have any suggestions, please let me know.

One other place this is repeated is on the RedHat site's copy of the same 
document.  Not sure how to inform them.   Hopefully fixing it on Camel site 
will get propagated there.

Thank you

-Original Message-
From: Andrea Cosentino  
Sent: Sunday, December 27, 2020 10:48 AM
To: users@camel.apache.org
Subject: Re: Correction needed in Documentation for Camel Source Connector

Thanks for spotting.

Il dom 27 dic 2020, 16:39 Arundhati Bhende  ha 
scritto:

> I am not yet familiar with other capabilities of a connector, have 
> just started learning about it.
>
> In regards to the comment  :  You're wrong. The aggregation strategy 
> can be used in sink too.
>
> I am not saying it cannot be used in the sink,  what I am saying is -  
> The official page for Source Connector has Sink connector wording at 
> the bottom - which I think should have been source.
>
> This is what is on the documentation page which I am saying the word 
> sink is there by mistake, as well as two lines above this on the page
>
> The camel-aws-s3 sink connector has no aggregation strategies out of 
> the box.
>
>
> -Original Message-
> From: Andrea Cosentino 
> Sent: Sunday, December 27, 2020 10:33 AM
> To: users@camel.apache.org
> Subject: Re: Correction needed in Documentation for Camel Source 
> Connector
>
> You're wrong. The aggregation strategy can be used in sink too.
>
> Il dom 27 dic 2020, 16:23 Arundhati Bhende < 
> arundhati.bhe...@prudential.com> ha scritto:
>
> > Hi,
> >
> > I do not know if this is the correct list for this, but I wanted to 
> > bring it to notice of the team that maintains the documentation.
> >
> > In the page -  "camel-aws-s3-kafka-connector source configuration"
> >  at the link
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca
> > me 
> > l.apache.org%2Fcamel-kafka-connector%2Flatest%2Fconnectors%2Fcamel-a
> > ws
> > -s3-kafka-source-connector.htmldata=04%7C01%7Carundhati.bhende%
> > 40 
> > prudential.com%7Cec4cbc71190f49a7c11a08d8aa7cc7a0%7Cd8fde2f593924260
> > 8a 
> > 030ad01f4746e9%7C0%7C0%7C637446800196356811%7CUnknown%7CTWFpbGZsb3d8
> > ey
> > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1
> > 00 
> > 0sdata=USwkv0KWNDm%2FJdPEfkQOxmOYwD%2BX9jB0cS7hFYxI7c8%3Dr
> > es
> > erved=0  the wording at the bottom needs to be changed ( Seems to be 
> > incorrect copy /paste from Sink connector )
> >
> >
> > Below is the documentation where the word sink needs to be replaced 
> > with source
> >
> > The camel-aws-s3 sink connector supports 1 converters out of the 
> > box, which are listed below.
> > org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > The camel-aws-s3 sink connector supports 1 transforms out of the 
> > box, which are listed below.
> > org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransform
> > s
> >
> > The camel-aws-s3 sink connector has no aggregation strategies out of 
> > the box.
> >
> > Thank you
> >
> >
>

Re: Correction needed in Documentation for Camel Source Connector

[Andrea Cosentino]

Thanks for spotting.

Il dom 27 dic 2020, 16:39 Arundhati Bhende 
ha scritto:

> I am not yet familiar with other capabilities of a connector, have just
> started learning about it.
>
> In regards to the comment  :  You're wrong. The aggregation strategy can
> be used in sink too.
>
> I am not saying it cannot be used in the sink,  what I am saying is -
>  The official page for Source Connector has Sink connector wording at the
> bottom - which I think should have been source.
>
> This is what is on the documentation page which I am saying the word sink
> is there by mistake, as well as two lines above this on the page
>
> The camel-aws-s3 sink connector has no aggregation strategies out of the
> box.
>
>
> -Original Message-
> From: Andrea Cosentino 
> Sent: Sunday, December 27, 2020 10:33 AM
> To: users@camel.apache.org
> Subject: Re: Correction needed in Documentation for Camel Source Connector
>
> You're wrong. The aggregation strategy can be used in sink too.
>
> Il dom 27 dic 2020, 16:23 Arundhati Bhende <
> arundhati.bhe...@prudential.com> ha scritto:
>
> > Hi,
> >
> > I do not know if this is the correct list for this, but I wanted to
> > bring it to notice of the team that maintains the documentation.
> >
> > In the page -  "camel-aws-s3-kafka-connector source configuration"
> >  at the link
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcame
> > l.apache.org%2Fcamel-kafka-connector%2Flatest%2Fconnectors%2Fcamel-aws
> > -s3-kafka-source-connector.htmldata=04%7C01%7Carundhati.bhende%40
> > prudential.com%7Cec4cbc71190f49a7c11a08d8aa7cc7a0%7Cd8fde2f5939242608a
> > 030ad01f4746e9%7C0%7C0%7C637446800196356811%7CUnknown%7CTWFpbGZsb3d8ey
> > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C100
> > 0sdata=USwkv0KWNDm%2FJdPEfkQOxmOYwD%2BX9jB0cS7hFYxI7c8%3Dres
> > erved=0  the wording at the bottom needs to be changed ( Seems to be
> > incorrect copy /paste from Sink connector )
> >
> >
> > Below is the documentation where the word sink needs to be replaced
> > with source
> >
> > The camel-aws-s3 sink connector supports 1 converters out of the box,
> > which are listed below.
> > org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > The camel-aws-s3 sink connector supports 1 transforms out of the box,
> > which are listed below.
> > org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransforms
> >
> > The camel-aws-s3 sink connector has no aggregation strategies out of
> > the box.
> >
> > Thank you
> >
> >
>

RE: Correction needed in Documentation for Camel Source Connector

[Arundhati Bhende]

I am not yet familiar with other capabilities of a connector, have just started 
learning about it.

In regards to the comment  :  You're wrong. The aggregation strategy can be 
used in sink too.

I am not saying it cannot be used in the sink,  what I am saying is -   The 
official page for Source Connector has Sink connector wording at the bottom - 
which I think should have been source.

This is what is on the documentation page which I am saying the word sink is 
there by mistake, as well as two lines above this on the page

The camel-aws-s3 sink connector has no aggregation strategies out of the box.


-Original Message-
From: Andrea Cosentino  
Sent: Sunday, December 27, 2020 10:33 AM
To: users@camel.apache.org
Subject: Re: Correction needed in Documentation for Camel Source Connector

You're wrong. The aggregation strategy can be used in sink too.

Il dom 27 dic 2020, 16:23 Arundhati Bhende  ha 
scritto:

> Hi,
>
> I do not know if this is the correct list for this, but I wanted to 
> bring it to notice of the team that maintains the documentation.
>
> In the page -  "camel-aws-s3-kafka-connector source configuration"
>  at the link
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcame
> l.apache.org%2Fcamel-kafka-connector%2Flatest%2Fconnectors%2Fcamel-aws
> -s3-kafka-source-connector.htmldata=04%7C01%7Carundhati.bhende%40
> prudential.com%7Cec4cbc71190f49a7c11a08d8aa7cc7a0%7Cd8fde2f5939242608a
> 030ad01f4746e9%7C0%7C0%7C637446800196356811%7CUnknown%7CTWFpbGZsb3d8ey
> JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C100
> 0sdata=USwkv0KWNDm%2FJdPEfkQOxmOYwD%2BX9jB0cS7hFYxI7c8%3Dres
> erved=0  the wording at the bottom needs to be changed ( Seems to be 
> incorrect copy /paste from Sink connector )
>
>
> Below is the documentation where the word sink needs to be replaced 
> with source
>
> The camel-aws-s3 sink connector supports 1 converters out of the box, 
> which are listed below.
> org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> The camel-aws-s3 sink connector supports 1 transforms out of the box, 
> which are listed below.
> org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransforms
>
> The camel-aws-s3 sink connector has no aggregation strategies out of 
> the box.
>
> Thank you
>
>

Re: Correction needed in Documentation for Camel Source Connector

[Andrea Cosentino]

You're wrong. The aggregation strategy can be used in sink too.

Il dom 27 dic 2020, 16:23 Arundhati Bhende 
ha scritto:

> Hi,
>
> I do not know if this is the correct list for this, but I wanted to bring
> it to notice of the team that maintains the documentation.
>
> In the page –  “camel-aws-s3-kafka-connector source configuration”
>  at the link
> https://camel.apache.org/camel-kafka-connector/latest/connectors/camel-aws-s3-kafka-source-connector.html
>  the wording at the bottom needs to be changed ( Seems to be incorrect copy
> /paste from Sink connector )
>
>
> Below is the documentation where the word sink needs to be replaced with
> source
>
> The camel-aws-s3 sink connector supports 1 converters out of the box,
> which are listed below.
> org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
>
> The camel-aws-s3 sink connector supports 1 transforms out of the box,
> which are listed below.
> org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransforms
>
> The camel-aws-s3 sink connector has no aggregation strategies out of the
> box.
>
> Thank you
>
>

Correction needed in Documentation for Camel Source Connector

[Arundhati Bhende]

Hi,

I do not know if this is the correct list for this, but I wanted to bring it to 
notice of the team that maintains the documentation.

In the page –  “camel-aws-s3-kafka-connector source configuration” at   
  the link   
https://camel.apache.org/camel-kafka-connector/latest/connectors/camel-aws-s3-kafka-source-connector.html
   the wording at the bottom needs to be changed ( Seems to be incorrect copy 
/paste from Sink connector )


Below is the documentation where the word sink needs to be replaced with source

The camel-aws-s3 sink connector supports 1 converters out of the box, which are 
listed below.
org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter

The camel-aws-s3 sink connector supports 1 transforms out of the box, which are 
listed below.
org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransforms

The camel-aws-s3 sink connector has no aggregation strategies out of the box.

Thank you

RE: Camel S3 Source Connector - value.converter

[Arundhati Bhende]

In order to overcome -"There is no splitting support actually. So it will 
be something like 1 file
- 1 message."

Is there a way - like using transformer ( I have never used a transformer, but 
wondering if it will help here ) - to split the message into multiple messages? 
 If it is possible, is there an example I could look at to define how  I can 
use it?

-Original Message-
From: Andrea Cosentino  
Sent: Saturday, December 26, 2020 10:19 AM
To: users@camel.apache.org
Subject: Re: Camel S3 Source Connector - value.converter

You can use any of the dataformat provided by camel or converters from kafka 
connect. So even avro or json.

There is no splitting support actually. So it will be something like 1 file
- 1 message.



Il sab 26 dic 2020, 16:08 Arundhati Bhende  ha 
scritto:

> Thank you.
>
> I will test with setting the autocloseBody to false.  In that case, 
> will I be able to use any valid converter ?  What I mean is currently 
> I am testing with a simple text file and hence was trying with 
> StringConverter,  but in reality, those maybe JSON on Avro formatted 
> messages in the files, so will I be able to use those formats as converters?
>
> Other part of the question - after I get a simple example working   - a
> single file will contain multiple records, one record per line - do I 
> need to set-up any other property to convert each line to a separate 
> message to the topic?
>
> One doubt / question - to use the camel-aws2-s3 connector, our aws set-up
> must be at version 2, right?   Because, currently it is at v1 and do not
> know, if we have immediate option of moving to v2
>
>
>
> On 12/26/20, 10:00 AM, "Andrea Cosentino"  wrote:
>
> Yes, the approach without url is correct
>  Autoclosebody will close s3object after consuming the payload, so 
> you want
> to get something it must be equal to false, while using the converter.
>
> My suggestion by the way is using the camel-aws2-s3 connector, 
> based on ask
> v2
>
> Il sab 26 dic 2020, 15:53 Arundhati Bhende < 
> arundhati.bhe...@prudential.com>
> ha scritto:
>
> > This is the full configuration that I have used.  I am been 
> experimenting
> > with changing the value.converter between StringConverter and
> > S3ObjectConverter - otherwise everything is same.
> >
> >
> >
> > DATA=$( cat << EOF
> > {
> > "connector.class":
> > "org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector",
> > "key.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "value.converter":
> > "org.apache.kafka.connect.storage.StringConverter",
> > "camel.source.maxPollDuration": "1",
> > "topics": "TEST-S3-SOURCE-DZONE-POC",
> > "camel.source.path.bucketNameOrArn": " push-json-poc",
> > "camel.component.aws-s3.region": "US_EAST_1",
> > "tasks.max": "1",
> > "camel.source.endpoint.useIAMCredentials": "true",
> > "camel.source.endpoint.autocloseBody": "true"
> > }
> > EOF
> > )
> > #"value.converter":
> > "org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter",
> >
> >
> > This is official one from the link in the other email.
> > name=CamelAWSS3SourceConnector
> >
> >
> connector.class=org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector
> > key.converter=org.apache.kafka.connect.storage.StringConverter
> >
> >
> value.converter=org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter
> >
> > camel.source.maxPollDuration=1
> >
> > topics=mytopic
> >
> > camel.source.url=aws-s3://camel-kafka-connector?autocloseBody=false
> >
> > camel.component.aws-s3.access-key=
> > camel.component.aws-s3.secret-key=
> > camel.component.aws-s3.region=EU_WEST_1
> >
> >
> >
> >
> > I can see two differences -
> > 1.  How the AWS credentials are set-up
> > 2.  In my configuration
> > I am using camel.source.path.bucketNameOrArn instead of
> > camel.source.url
> > Because I was getting error that bucketNameOrArn must be
> defined
> >
> > In my configuration - autocloseBody is set true to whereas
> in the
> > default configuration it is set to false
> >
> > How does autocloseBody work?  What impact does it have when
> > setting it to true vs. false?
> >
> >
> > Thank you
> >
> >
> > -Original Message-
> > From: Andrea Cosentino 
> > Sent: Saturday, December 26, 2020 3:59 AM
> > To: users@camel.apache.org
> > Subject: Re: Camel S3 Source Connector - value.converter
> >
> > Can you show your full configuration?
> >
> > Il sab 26 dic 2020, 07:40 Arundhati Bhende <
> > arundhati.bhe...@prudential.com> ha scritto:
> >
> > > I am trying to 

Re: https://camel.apache.org/components/latest/eips/setHeader-eip.html

[Claus Ibsen]

Hi

For Camel 3 its  wrote:

> Hi,
>
> Set Header doesn’t seem to work for me. In the documentation name
> option is given but in actual in takes headerName , but it also errors out.
> I have tired with version 3.6.0 & 3.7.0
>
>
>
> Warm Regards,
>
>
>
> *Vikas Jaiswal*
>
> Lead Consultant
>
> [image: cid:image001.png@01D67243.124E3960]
>
> *T: *+91 2042006145 |*M:* +91 9371626126 |*Skype*: vikasdnjaiswal
> *W:* www.saksoft.com  |  www.acuma.co.uk  |  www.360logica.com |
> www.dreamorbit.com
>
> [image: Description: cid:image003.jpg@01D2806E.41A0F0B0]
> [image: Description:
> cid:image004.jpg@01D2806E.41A0F0B0]  [image:
> Description: cid:image005.jpg@01D2806E.41A0F0B0]
>  [image:
> Description: cid:image006.jpg@01D2806E.41A0F0B0]
>  [image:
> Description: cid:image007.jpg@01D2806E.41A0F0B0]
> 
>
> *I*nnovation   *C*ustomer focus   *O*penness   *R*espect   *E*nterprising
>
>
>
>
>


-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2