Re: Camel S3SourceConnector and Effect of Topic cleanup.policy "compact" vs. "delete"
aws-s3 connector - not aws2-s3. On 1/4/21, 5:19 PM, "Andrea Cosentino" wrote: Is this with aws2-s3 connector or aws2-s3? Il lun 4 gen 2021, 23:05 Arundhati Bhende ha scritto: > Hi, I am testing the connector with different cleanup policies for the > Topic. > > If the topic cleanup.policy is set to "delete", the connector works > correctly and I am able to access the message in the topic > > If the topic cleanup.policy is set to "compact", the connect Task fails > with the below error. > > I am trying to find out why this happens. Can someone please explain? > > trace: org.apache.kafka.connect.errors.ConnectException: > OffsetStorageWriter is already flushing > at > org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:111) > at > org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:438) > at > org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:257) > at > org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) > at > org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Thank you > > >
Re: Camel S3SourceConnector and Effect of Topic cleanup.policy "compact" vs. "delete"
Is this with aws2-s3 connector or aws2-s3? Il lun 4 gen 2021, 23:05 Arundhati Bhende ha scritto: > Hi, I am testing the connector with different cleanup policies for the > Topic. > > If the topic cleanup.policy is set to "delete", the connector works > correctly and I am able to access the message in the topic > > If the topic cleanup.policy is set to "compact", the connect Task fails > with the below error. > > I am trying to find out why this happens. Can someone please explain? > > trace: org.apache.kafka.connect.errors.ConnectException: > OffsetStorageWriter is already flushing > at > org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:111) > at > org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:438) > at > org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:257) > at > org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) > at > org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Thank you > > >
Camel S3SourceConnector and Effect of Topic cleanup.policy "compact" vs. "delete"
Hi, I am testing the connector with different cleanup policies for the Topic. If the topic cleanup.policy is set to "delete", the connector works correctly and I am able to access the message in the topic If the topic cleanup.policy is set to "compact", the connect Task fails with the below error. I am trying to find out why this happens. Can someone please explain? trace: org.apache.kafka.connect.errors.ConnectException: OffsetStorageWriter is already flushing at org.apache.kafka.connect.storage.OffsetStorageWriter.beginFlush(OffsetStorageWriter.java:111) at org.apache.kafka.connect.runtime.WorkerSourceTask.commitOffsets(WorkerSourceTask.java:438) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:257) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thank you
Re: Correction needed in Documentation for Camel Source Connector
Thanks. I subscribed to that thread, so I should get any updates to it, Let me know, if you need me to provide any other information. On 1/4/21, 9:37 AM, "Andrea Cosentino" wrote: I created this one https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcamel-kafka-connector%2Fissues%2F822data=04%7C01%7Carundhati.bhende%40prudential.com%7Cc236ba08ba744eb2471c08d8b0be322b%7Cd8fde2f5939242608a030ad01f4746e9%7C0%7C0%7C637453678366921507%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=fIrqZtIrwrlMpZVC6tbIjTpH%2BX93xGa97L1opofnQoU%3Dreserved=0 Il giorno lun 4 gen 2021 alle ore 15:18 Arundhati Bhende < arundhati.bhe...@prudential.com> ha scritto: > Thanks Andrea. > > Do I make a feature / enhancement request for this through some mechanism > or will it be taken up by the development team as and when they can fit it > into their plan? > > Thanks > > On 12/27/20, 11:12 AM, "Andrea Cosentino" wrote: > > I don't thikn it can be done through transforms. > > It must be supported on camel level eventually > > Il giorno dom 27 dic 2020 alle ore 16:58 Arundhati Bhende < > arundhati.bhe...@prudential.com> ha scritto: > > > Sure. > > > > Do you have any suggestion on to how can I split the lines in the > file as > > single records in adding to the topic or do I have to write a > separate app > > that adds reads the single message and splits it as the multiple > messages > > to yet another topic > > > > If you have any suggestions, please let me know. > > > > One other place this is repeated is on the RedHat site's copy of the > same > > document. Not sure how to inform them. Hopefully fixing it on > Camel site > > will get propagated there. > > > > Thank you > > > > -Original Message- > > From: Andrea Cosentino > > Sent: Sunday, December 27, 2020 10:48 AM > > To: users@camel.apache.org > > Subject: Re: Correction needed in Documentation for Camel Source > Connector > > > > Thanks for spotting. > > > > Il dom 27 dic 2020, 16:39 Arundhati Bhende < > > arundhati.bhe...@prudential.com> ha scritto: > > > > > I am not yet familiar with other capabilities of a connector, have > > > just started learning about it. > > > > > > In regards to the comment : You're wrong. The aggregation > strategy > > > can be used in sink too. > > > > > > I am not saying it cannot be used in the sink, what I am saying > is - > > > The official page for Source Connector has Sink connector wording > at > > > the bottom - which I think should have been source. > > > > > > This is what is on the documentation page which I am saying the > word > > > sink is there by mistake, as well as two lines above this on the > page > > > > > > The camel-aws-s3 sink connector has no aggregation strategies out > of > > > the box. > > > > > > > > > -Original Message- > > > From: Andrea Cosentino > > > Sent: Sunday, December 27, 2020 10:33 AM > > > To: users@camel.apache.org > > > Subject: Re: Correction needed in Documentation for Camel Source > > > Connector > > > > > > You're wrong. The aggregation strategy can be used in sink too. > > > > > > Il dom 27 dic 2020, 16:23 Arundhati Bhende < > > > arundhati.bhe...@prudential.com> ha scritto: > > > > > > > Hi, > > > > > > > > I do not know if this is the correct list for this, but I wanted > to > > > > bring it to notice of the team that maintains the documentation. > > > > > > > > In the page - "camel-aws-s3-kafka-connector source > configuration" > > > > at the link > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca > > > > me > > > > l.apache.org > %2Fcamel-kafka-connector%2Flatest%2Fconnectors%2Fcamel-a > > > > ws > > > > > -s3-kafka-source-connector.htmldata=04%7C01%7Carundhati.bhende% > > > > 40 > > > > prudential.com > %7Cec4cbc71190f49a7c11a08d8aa7cc7a0%7Cd8fde2f593924260 > > > > 8a > > > > > 030ad01f4746e9%7C0%7C0%7C637446800196356811%7CUnknown%7CTWFpbGZsb3d8 > > > > ey > > > > > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1 > > > > 00 > > > > > 0sdata=USwkv0KWNDm%2FJdPEfkQOxmOYwD%2BX9jB0cS7hFYxI7c8%3Dr > > > > es > > > > erved=0 the wording at
Re: Correction needed in Documentation for Camel Source Connector
I created this one https://github.com/apache/camel-kafka-connector/issues/822 Il giorno lun 4 gen 2021 alle ore 15:18 Arundhati Bhende < arundhati.bhe...@prudential.com> ha scritto: > Thanks Andrea. > > Do I make a feature / enhancement request for this through some mechanism > or will it be taken up by the development team as and when they can fit it > into their plan? > > Thanks > > On 12/27/20, 11:12 AM, "Andrea Cosentino" wrote: > > I don't thikn it can be done through transforms. > > It must be supported on camel level eventually > > Il giorno dom 27 dic 2020 alle ore 16:58 Arundhati Bhende < > arundhati.bhe...@prudential.com> ha scritto: > > > Sure. > > > > Do you have any suggestion on to how can I split the lines in the > file as > > single records in adding to the topic or do I have to write a > separate app > > that adds reads the single message and splits it as the multiple > messages > > to yet another topic > > > > If you have any suggestions, please let me know. > > > > One other place this is repeated is on the RedHat site's copy of the > same > > document. Not sure how to inform them. Hopefully fixing it on > Camel site > > will get propagated there. > > > > Thank you > > > > -Original Message- > > From: Andrea Cosentino > > Sent: Sunday, December 27, 2020 10:48 AM > > To: users@camel.apache.org > > Subject: Re: Correction needed in Documentation for Camel Source > Connector > > > > Thanks for spotting. > > > > Il dom 27 dic 2020, 16:39 Arundhati Bhende < > > arundhati.bhe...@prudential.com> ha scritto: > > > > > I am not yet familiar with other capabilities of a connector, have > > > just started learning about it. > > > > > > In regards to the comment : You're wrong. The aggregation > strategy > > > can be used in sink too. > > > > > > I am not saying it cannot be used in the sink, what I am saying > is - > > > The official page for Source Connector has Sink connector wording > at > > > the bottom - which I think should have been source. > > > > > > This is what is on the documentation page which I am saying the > word > > > sink is there by mistake, as well as two lines above this on the > page > > > > > > The camel-aws-s3 sink connector has no aggregation strategies out > of > > > the box. > > > > > > > > > -Original Message- > > > From: Andrea Cosentino > > > Sent: Sunday, December 27, 2020 10:33 AM > > > To: users@camel.apache.org > > > Subject: Re: Correction needed in Documentation for Camel Source > > > Connector > > > > > > You're wrong. The aggregation strategy can be used in sink too. > > > > > > Il dom 27 dic 2020, 16:23 Arundhati Bhende < > > > arundhati.bhe...@prudential.com> ha scritto: > > > > > > > Hi, > > > > > > > > I do not know if this is the correct list for this, but I wanted > to > > > > bring it to notice of the team that maintains the documentation. > > > > > > > > In the page - "camel-aws-s3-kafka-connector source > configuration" > > > > at the link > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca > > > > me > > > > l.apache.org > %2Fcamel-kafka-connector%2Flatest%2Fconnectors%2Fcamel-a > > > > ws > > > > > -s3-kafka-source-connector.htmldata=04%7C01%7Carundhati.bhende% > > > > 40 > > > > prudential.com > %7Cec4cbc71190f49a7c11a08d8aa7cc7a0%7Cd8fde2f593924260 > > > > 8a > > > > > 030ad01f4746e9%7C0%7C0%7C637446800196356811%7CUnknown%7CTWFpbGZsb3d8 > > > > ey > > > > > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1 > > > > 00 > > > > > 0sdata=USwkv0KWNDm%2FJdPEfkQOxmOYwD%2BX9jB0cS7hFYxI7c8%3Dr > > > > es > > > > erved=0 the wording at the bottom needs to be changed ( Seems > to be > > > > incorrect copy /paste from Sink connector ) > > > > > > > > > > > > Below is the documentation where the word sink needs to be > replaced > > > > with source > > > > > > > > The camel-aws-s3 sink connector supports 1 converters out of the > > > > box, which are listed below. > > > > > org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter > > > > > > > > The camel-aws-s3 sink connector supports 1 transforms out of the > > > > box, which are listed below. > > > > > org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransform > > > > s > > > > > > > > The camel-aws-s3 sink connector has no aggregation strategies > out of > > > > the box. > > > > > > > > Thank you > > > > > > > > > > > > > > >
Re: Correction needed in Documentation for Camel Source Connector
Thanks Andrea. Do I make a feature / enhancement request for this through some mechanism or will it be taken up by the development team as and when they can fit it into their plan? Thanks On 12/27/20, 11:12 AM, "Andrea Cosentino" wrote: I don't thikn it can be done through transforms. It must be supported on camel level eventually Il giorno dom 27 dic 2020 alle ore 16:58 Arundhati Bhende < arundhati.bhe...@prudential.com> ha scritto: > Sure. > > Do you have any suggestion on to how can I split the lines in the file as > single records in adding to the topic or do I have to write a separate app > that adds reads the single message and splits it as the multiple messages > to yet another topic > > If you have any suggestions, please let me know. > > One other place this is repeated is on the RedHat site's copy of the same > document. Not sure how to inform them. Hopefully fixing it on Camel site > will get propagated there. > > Thank you > > -Original Message- > From: Andrea Cosentino > Sent: Sunday, December 27, 2020 10:48 AM > To: users@camel.apache.org > Subject: Re: Correction needed in Documentation for Camel Source Connector > > Thanks for spotting. > > Il dom 27 dic 2020, 16:39 Arundhati Bhende < > arundhati.bhe...@prudential.com> ha scritto: > > > I am not yet familiar with other capabilities of a connector, have > > just started learning about it. > > > > In regards to the comment : You're wrong. The aggregation strategy > > can be used in sink too. > > > > I am not saying it cannot be used in the sink, what I am saying is - > > The official page for Source Connector has Sink connector wording at > > the bottom - which I think should have been source. > > > > This is what is on the documentation page which I am saying the word > > sink is there by mistake, as well as two lines above this on the page > > > > The camel-aws-s3 sink connector has no aggregation strategies out of > > the box. > > > > > > -Original Message- > > From: Andrea Cosentino > > Sent: Sunday, December 27, 2020 10:33 AM > > To: users@camel.apache.org > > Subject: Re: Correction needed in Documentation for Camel Source > > Connector > > > > You're wrong. The aggregation strategy can be used in sink too. > > > > Il dom 27 dic 2020, 16:23 Arundhati Bhende < > > arundhati.bhe...@prudential.com> ha scritto: > > > > > Hi, > > > > > > I do not know if this is the correct list for this, but I wanted to > > > bring it to notice of the team that maintains the documentation. > > > > > > In the page - "camel-aws-s3-kafka-connector source configuration" > > > at the link > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fca > > > me > > > l.apache.org%2Fcamel-kafka-connector%2Flatest%2Fconnectors%2Fcamel-a > > > ws > > > -s3-kafka-source-connector.htmldata=04%7C01%7Carundhati.bhende% > > > 40 > > > prudential.com%7Cec4cbc71190f49a7c11a08d8aa7cc7a0%7Cd8fde2f593924260 > > > 8a > > > 030ad01f4746e9%7C0%7C0%7C637446800196356811%7CUnknown%7CTWFpbGZsb3d8 > > > ey > > > JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1 > > > 00 > > > 0sdata=USwkv0KWNDm%2FJdPEfkQOxmOYwD%2BX9jB0cS7hFYxI7c8%3Dr > > > es > > > erved=0 the wording at the bottom needs to be changed ( Seems to be > > > incorrect copy /paste from Sink connector ) > > > > > > > > > Below is the documentation where the word sink needs to be replaced > > > with source > > > > > > The camel-aws-s3 sink connector supports 1 converters out of the > > > box, which are listed below. > > > org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter > > > > > > The camel-aws-s3 sink connector supports 1 transforms out of the > > > box, which are listed below. > > > org.apache.camel.kafkaconnector.awss3.transformers.S3ObjectTransform > > > s > > > > > > The camel-aws-s3 sink connector has no aggregation strategies out of > > > the box. > > > > > > Thank you > > > > > > > > >
