Re: using Camel with CloudKarafka
Hello Andrea, Thanks for the callout. I had this correctly set up in my route that writes to Kafka but i had missed it on my read from Kafka flow as well as ended up having two lines of Yaml merged into one due to manual editing. Both work now. Here is my updated route: - route: id: route-09c3 nodePrefixId: route-229 from: id: from-f775 uri: kafka parameters: topic: 4422e1r5-default brokers: moped.srvs.cloudkafka.com:9094 saslJaasConfig: >- org.apache.kafka.common.security.scram.ScramLoginModule required username='user' password='password'; securityProtocol: SASL_SSL saslMechanism: SCRAM-SHA-256 clientId: kafkaconsumer1 autoOffsetReset: latest consumersCount: '10' groupId: kafkaconsumergroup1 steps: - log: id: log-1987 message: ${body} ચિરાગ/चिराग/Chirag -- Sent from My Gmail Account On Fri, Mar 1, 2024 at 12:43 AM Andrea Cosentino wrote: > Hi, > > Have a look at how we implement the Kafka SCRAM source Kamelet. > > > https://github.com/apache/camel-kamelets/blob/main/kamelets/kafka-scram-source.kamelet.yaml > > As far as I see, you need to set also saslMechanism and securityProtocol > > Cheers. > > Il giorno ven 1 mar 2024 alle ore 02:20 Chirag > ha scritto: > > > Hi, > > > > I am working on a Proof of concept with CloudKarafka (Kafka hosting > > provider) and camel. > > > > I am using camel-jbang 4.4.0 > > > > Route to write to Kafka@CloudKarafka works fine. > > But trying to read from Kafka seems to run into challenge. > > > > here is my route: > > - route: > > id: route-09c3 > > nodePrefixId: route-229 > > from: > > id: from-f775 > > uri: kafka > > parameters: > > topic: 4422e1r5-default > > brokers: moped.srvs.cloudkafka.com:9094 > > saslJaasConfig: >- > > org.apache.kafka.common.security.scram.ScramLoginModule > > required username='user' > > password='password' securityProtocol: SASL_SSL > > steps: > > - log: > > id: log-1987 > > message: ${body} > > > > Here are logs: > > he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1 > > he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5 > > he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232 > > ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy > > l.component.kafka.KafkaFetchRecords : Subscribing > 4422e1r5-default-Thread > > 0 to topic 4422e1r5-default > > afka.clients.consumer.KafkaConsumer : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s): > > 4422e1r5-default > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > > API_VERSIONS request with correlation id 1 due to node -1 being > > disconnected (elapsed time since creation: 401ms, elapsed time since > send: > > 401ms, request timeout: 3ms) > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > > API_VERSIONS request with correlation id 2 due to node -1 being > > disconnected (elapsed time since creation: 216ms, elapsed time since > send: > > 216ms, request timeout: 3ms) > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > > API_VERSIONS request with correlation id 3 due to node -1 being > > disconnected (elapsed time since creation:
Re: using Camel with CloudKarafka
That is unfortunate. CloudKarafka free tier allowed to get a kafka instance in about two minutes on the web simplifying prototyping. On Fri, Mar 1, 2024, 01:05 Claus Ibsen wrote: > Hi > > Just a caution that the company sent out this EOL announcement > https://www.cloudkarafka.com/blog/end-of-life-announcement.html > > On Fri, Mar 1, 2024 at 2:20 AM Chirag wrote: > > > Hi, > > > > I am working on a Proof of concept with CloudKarafka (Kafka hosting > > provider) and camel. > > > > I am using camel-jbang 4.4.0 > > > > Route to write to Kafka@CloudKarafka works fine. > > But trying to read from Kafka seems to run into challenge. > > > > here is my route: > > - route: > > id: route-09c3 > > nodePrefixId: route-229 > > from: > > id: from-f775 > > uri: kafka > > parameters: > > topic: 4422e1r5-default > > brokers: moped.srvs.cloudkafka.com:9094 > > saslJaasConfig: >- > > org.apache.kafka.common.security.scram.ScramLoginModule > > required username='user' > > password='password' securityProtocol: SASL_SSL > > steps: > > - log: > > id: log-1987 > > message: ${body} > > > > Here are logs: > > he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1 > > he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5 > > he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232 > > ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy > > l.component.kafka.KafkaFetchRecords : Subscribing > 4422e1r5-default-Thread > > 0 to topic 4422e1r5-default > > afka.clients.consumer.KafkaConsumer : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s): > > 4422e1r5-default > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > > API_VERSIONS request with correlation id 1 due to node -1 being > > disconnected (elapsed time since creation: 401ms, elapsed time since > send: > > 401ms, request timeout: 3ms) > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > > API_VERSIONS request with correlation id 2 due to node -1 being > > disconnected (elapsed time since creation: 216ms, elapsed time since > send: > > 216ms, request timeout: 3ms) > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > > .apache.kafka.clients.NetworkClient : [Consumer > > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > > API_VERSIONS request with correlation id 3 due to node -1 being > > disconnected (elapsed time since creation: 313ms, elapsed time since > send: > > 313ms, request timeout: 3ms) > > > > Does this require any other type of param to read successfully? > > > > ચિરાગ/चिराग/Chirag > > -- > > Sent from My Gmail Account > > > > > -- > Claus Ibsen > - > @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >
Re: using Camel with CloudKarafka
Hi Just a caution that the company sent out this EOL announcement https://www.cloudkarafka.com/blog/end-of-life-announcement.html On Fri, Mar 1, 2024 at 2:20 AM Chirag wrote: > Hi, > > I am working on a Proof of concept with CloudKarafka (Kafka hosting > provider) and camel. > > I am using camel-jbang 4.4.0 > > Route to write to Kafka@CloudKarafka works fine. > But trying to read from Kafka seems to run into challenge. > > here is my route: > - route: > id: route-09c3 > nodePrefixId: route-229 > from: > id: from-f775 > uri: kafka > parameters: > topic: 4422e1r5-default > brokers: moped.srvs.cloudkafka.com:9094 > saslJaasConfig: >- > org.apache.kafka.common.security.scram.ScramLoginModule > required username='user' > password='password' securityProtocol: SASL_SSL > steps: > - log: > id: log-1987 > message: ${body} > > Here are logs: > he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1 > he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5 > he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232 > ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy > l.component.kafka.KafkaFetchRecords : Subscribing 4422e1r5-default-Thread > 0 to topic 4422e1r5-default > afka.clients.consumer.KafkaConsumer : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s): > 4422e1r5-default > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > API_VERSIONS request with correlation id 1 due to node -1 being > disconnected (elapsed time since creation: 401ms, elapsed time since send: > 401ms, request timeout: 3ms) > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > API_VERSIONS request with correlation id 2 due to node -1 being > disconnected (elapsed time since creation: 216ms, elapsed time since send: > 216ms, request timeout: 3ms) > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > API_VERSIONS request with correlation id 3 due to node -1 being > disconnected (elapsed time since creation: 313ms, elapsed time since send: > 313ms, request timeout: 3ms) > > Does this require any other type of param to read successfully? > > ચિરાગ/चिराग/Chirag > -- > Sent from My Gmail Account > -- Claus Ibsen - @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
Re: using Camel with CloudKarafka
Hi, Have a look at how we implement the Kafka SCRAM source Kamelet. https://github.com/apache/camel-kamelets/blob/main/kamelets/kafka-scram-source.kamelet.yaml As far as I see, you need to set also saslMechanism and securityProtocol Cheers. Il giorno ven 1 mar 2024 alle ore 02:20 Chirag ha scritto: > Hi, > > I am working on a Proof of concept with CloudKarafka (Kafka hosting > provider) and camel. > > I am using camel-jbang 4.4.0 > > Route to write to Kafka@CloudKarafka works fine. > But trying to read from Kafka seems to run into challenge. > > here is my route: > - route: > id: route-09c3 > nodePrefixId: route-229 > from: > id: from-f775 > uri: kafka > parameters: > topic: 4422e1r5-default > brokers: moped.srvs.cloudkafka.com:9094 > saslJaasConfig: >- > org.apache.kafka.common.security.scram.ScramLoginModule > required username='user' > password='password' securityProtocol: SASL_SSL > steps: > - log: > id: log-1987 > message: ${body} > > Here are logs: > he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1 > he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5 > he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232 > ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy > l.component.kafka.KafkaFetchRecords : Subscribing 4422e1r5-default-Thread > 0 to topic 4422e1r5-default > afka.clients.consumer.KafkaConsumer : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s): > 4422e1r5-default > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > API_VERSIONS request with correlation id 1 due to node -1 being > disconnected (elapsed time since creation: 401ms, elapsed time since send: > 401ms, request timeout: 3ms) > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > API_VERSIONS request with correlation id 2 due to node -1 being > disconnected (elapsed time since creation: 216ms, elapsed time since send: > 216ms, request timeout: 3ms) > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker > moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. > .apache.kafka.clients.NetworkClient : [Consumer > clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, > groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight > API_VERSIONS request with correlation id 3 due to node -1 being > disconnected (elapsed time since creation: 313ms, elapsed time since send: > 313ms, request timeout: 3ms) > > Does this require any other type of param to read successfully? > > ચિરાગ/चिराग/Chirag > -- > Sent from My Gmail Account >
using Camel with CloudKarafka
Hi, I am working on a Proof of concept with CloudKarafka (Kafka hosting provider) and camel. I am using camel-jbang 4.4.0 Route to write to Kafka@CloudKarafka works fine. But trying to read from Kafka seems to run into challenge. here is my route: - route: id: route-09c3 nodePrefixId: route-229 from: id: from-f775 uri: kafka parameters: topic: 4422e1r5-default brokers: moped.srvs.cloudkafka.com:9094 saslJaasConfig: >- org.apache.kafka.common.security.scram.ScramLoginModule required username='user' password='password' securityProtocol: SASL_SSL steps: - log: id: log-1987 message: ${body} Here are logs: he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1 he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5 he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232 ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy l.component.kafka.KafkaFetchRecords : Subscribing 4422e1r5-default-Thread 0 to topic 4422e1r5-default afka.clients.consumer.KafkaConsumer : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s): 4422e1r5-default .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight API_VERSIONS request with correlation id 1 due to node -1 being disconnected (elapsed time since creation: 401ms, elapsed time since send: 401ms, request timeout: 3ms) .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight API_VERSIONS request with correlation id 2 due to node -1 being disconnected (elapsed time since creation: 216ms, elapsed time since send: 216ms, request timeout: 3ms) .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected. .apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1, groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight API_VERSIONS request with correlation id 3 due to node -1 being disconnected (elapsed time since creation: 313ms, elapsed time since send: 313ms, request timeout: 3ms) Does this require any other type of param to read successfully? ચિરાગ/चिराग/Chirag -- Sent from My Gmail Account