Re: using Camel with CloudKarafka
Hi
Just a caution that the company sent out this EOL announcement
https://www.cloudkarafka.com/blog/end-of-life-announcement.html
On Fri, Mar 1, 2024 at 2:20 AM Chirag wrote:
> Hi,
>
> I am working on a Proof of concept with CloudKarafka (Kafka hosting
> provider) and camel.
>
> I am using camel-jbang 4.4.0
>
> Route to write to Kafka@CloudKarafka works fine.
> But trying to read from Kafka seems to run into challenge.
>
> here is my route:
> - route:
> id: route-09c3
> nodePrefixId: route-229
> from:
> id: from-f775
> uri: kafka
> parameters:
> topic: 4422e1r5-default
> brokers: moped.srvs.cloudkafka.com:9094
> saslJaasConfig: >-
> org.apache.kafka.common.security.scram.ScramLoginModule
> required username='user'
> password='password' securityProtocol: SASL_SSL
> steps:
> - log:
> id: log-1987
> message: ${body}
>
> Here are logs:
> he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1
> he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5
> he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232
> ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy
> l.component.kafka.KafkaFetchRecords : Subscribing 4422e1r5-default-Thread
> 0 to topic 4422e1r5-default
> afka.clients.consumer.KafkaConsumer : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s):
> 4422e1r5-default
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
> API_VERSIONS request with correlation id 1 due to node -1 being
> disconnected (elapsed time since creation: 401ms, elapsed time since send:
> 401ms, request timeout: 3ms)
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker
> moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
> API_VERSIONS request with correlation id 2 due to node -1 being
> disconnected (elapsed time since creation: 216ms, elapsed time since send:
> 216ms, request timeout: 3ms)
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker
> moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
> API_VERSIONS request with correlation id 3 due to node -1 being
> disconnected (elapsed time since creation: 313ms, elapsed time since send:
> 313ms, request timeout: 3ms)
>
> Does this require any other type of param to read successfully?
>
> ચિરાગ/चिराग/Chirag
> --
> Sent from My Gmail Account
>
--
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: using Camel with CloudKarafka
Hi,
Have a look at how we implement the Kafka SCRAM source Kamelet.
https://github.com/apache/camel-kamelets/blob/main/kamelets/kafka-scram-source.kamelet.yaml
As far as I see, you need to set also saslMechanism and securityProtocol
Cheers.
Il giorno ven 1 mar 2024 alle ore 02:20 Chirag
ha scritto:
> Hi,
>
> I am working on a Proof of concept with CloudKarafka (Kafka hosting
> provider) and camel.
>
> I am using camel-jbang 4.4.0
>
> Route to write to Kafka@CloudKarafka works fine.
> But trying to read from Kafka seems to run into challenge.
>
> here is my route:
> - route:
> id: route-09c3
> nodePrefixId: route-229
> from:
> id: from-f775
> uri: kafka
> parameters:
> topic: 4422e1r5-default
> brokers: moped.srvs.cloudkafka.com:9094
> saslJaasConfig: >-
> org.apache.kafka.common.security.scram.ScramLoginModule
> required username='user'
> password='password' securityProtocol: SASL_SSL
> steps:
> - log:
> id: log-1987
> message: ${body}
>
> Here are logs:
> he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1
> he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5
> he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232
> ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy
> l.component.kafka.KafkaFetchRecords : Subscribing 4422e1r5-default-Thread
> 0 to topic 4422e1r5-default
> afka.clients.consumer.KafkaConsumer : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s):
> 4422e1r5-default
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
> API_VERSIONS request with correlation id 1 due to node -1 being
> disconnected (elapsed time since creation: 401ms, elapsed time since send:
> 401ms, request timeout: 3ms)
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker
> moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
> API_VERSIONS request with correlation id 2 due to node -1 being
> disconnected (elapsed time since creation: 216ms, elapsed time since send:
> 216ms, request timeout: 3ms)
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker
> moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
> .apache.kafka.clients.NetworkClient : [Consumer
> clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
> groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
> API_VERSIONS request with correlation id 3 due to node -1 being
> disconnected (elapsed time since creation: 313ms, elapsed time since send:
> 313ms, request timeout: 3ms)
>
> Does this require any other type of param to read successfully?
>
> ચિરાગ/चिराग/Chirag
> --
> Sent from My Gmail Account
>
using Camel with CloudKarafka
Hi,
I am working on a Proof of concept with CloudKarafka (Kafka hosting
provider) and camel.
I am using camel-jbang 4.4.0
Route to write to Kafka@CloudKarafka works fine.
But trying to read from Kafka seems to run into challenge.
here is my route:
- route:
id: route-09c3
nodePrefixId: route-229
from:
id: from-f775
uri: kafka
parameters:
topic: 4422e1r5-default
brokers: moped.srvs.cloudkafka.com:9094
saslJaasConfig: >-
org.apache.kafka.common.security.scram.ScramLoginModule
required username='user'
password='password' securityProtocol: SASL_SSL
steps:
- log:
id: log-1987
message: ${body}
Here are logs:
he.kafka.common.utils.AppInfoParser : Kafka version: 3.6.1
he.kafka.common.utils.AppInfoParser : Kafka commitId: 5e3c2b738d253ff5
he.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1709255355232
ort.classic.AssignmentAdapterHelper : Using NO-OP resume strategy
l.component.kafka.KafkaFetchRecords : Subscribing 4422e1r5-default-Thread
0 to topic 4422e1r5-default
afka.clients.consumer.KafkaConsumer : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Subscribed to topic(s):
4422e1r5-default
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
API_VERSIONS request with correlation id 1 due to node -1 being
disconnected (elapsed time since creation: 401ms, elapsed time since send:
401ms, request timeout: 3ms)
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker
moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
API_VERSIONS request with correlation id 2 due to node -1 being
disconnected (elapsed time since creation: 216ms, elapsed time since send:
216ms, request timeout: 3ms)
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Bootstrap broker
moped.srvs.cloudkafka.com:9094 (id: -1 rack: null) disconnected
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Node -1 disconnected.
.apache.kafka.clients.NetworkClient : [Consumer
clientId=consumer-12da9c71-2e7d-4726-935f-cf60c349fee3-1,
groupId=12da9c71-2e7d-4726-935f-cf60c349fee3] Cancelled in-flight
API_VERSIONS request with correlation id 3 due to node -1 being
disconnected (elapsed time since creation: 313ms, elapsed time since send:
313ms, request timeout: 3ms)
Does this require any other type of param to read successfully?
ચિરાગ/चिराग/Chirag
--
Sent from My Gmail Account
Re: Any way to have "Named/Keyed Endpoints" in YAML / Karavan?
Hello Mikael I believe that component configuration in application.properties should work, so don't need to duplicate in YAML: camel.component.jira.jira-url=URL camel.component.jira.password=PASSWORD Marat On Thu, Feb 29, 2024 at 9:00 AM Mikael Koskinen wrote: > Hi, > > Property placeholders allow us to reuse single things like Jira username > and password. But is there a concept in Camel that works with Camel YAML & > Karavan where we could reuse "named" endpoint definitions? > > So the idea is that I could (hopefully in application.properties) define > something like: > > myjira.username = test > myjira.password = 123456 > myjira.jiraUrl = https://comp.atlassian.com/jira > > And then in my route use this "named/keyed" definition when adding the Jira > component. So something like: > > - route: > from: > uri: jira > key: "myjira" > parameters: > type: NEWISSUES > steps: > - to: > uri: kamelet:http-sink > parameters: > url: https://webhooksite > method: POST > > The idea is that I could apply all the properties defined in the > application.properties to the endpoint instead of having to add the > properties one by one. > > I went through the documentation and found something that is almost there: > > https://camel.apache.org/manual/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringurisusingendpointwithbeanpropertystyle > > But then I checked the YAML schema and I don't think it supports these > Endpoint-definitions. And if it does, I'm not sure if the Karavan designer > could keep up with it. > > So I'm not sure if something like this already exists. Or if I should use > Beans? Or is this a new feature idea. Thanks for any tips in advance. > > Best regards, > Mikael >
Any way to have "Named/Keyed Endpoints" in YAML / Karavan?
Hi, Property placeholders allow us to reuse single things like Jira username and password. But is there a concept in Camel that works with Camel YAML & Karavan where we could reuse "named" endpoint definitions? So the idea is that I could (hopefully in application.properties) define something like: myjira.username = test myjira.password = 123456 myjira.jiraUrl = https://comp.atlassian.com/jira And then in my route use this "named/keyed" definition when adding the Jira component. So something like: - route: from: uri: jira key: "myjira" parameters: type: NEWISSUES steps: - to: uri: kamelet:http-sink parameters: url: https://webhooksite method: POST The idea is that I could apply all the properties defined in the application.properties to the endpoint instead of having to add the properties one by one. I went through the documentation and found something that is almost there: https://camel.apache.org/manual/faq/how-do-i-configure-endpoints.html#HowdoIconfigureendpoints-Configuringurisusingendpointwithbeanpropertystyle But then I checked the YAML schema and I don't think it supports these Endpoint-definitions. And if it does, I'm not sure if the Karavan designer could keep up with it. So I'm not sure if something like this already exists. Or if I should use Beans? Or is this a new feature idea. Thanks for any tips in advance. Best regards, Mikael
Re: REST DSL: Request param validation possibilities & existing option "allowableValues" not working
Hi
Ad 1)
https://issues.apache.org/jira/browse/CAMEL-20492
Ad 2)
Whats the use for this this, can you provide more detail and some example
Ad 3)
You can write a custom processor / ben / interceptor and do your own
validation - no need for pojo and all those crazy validation annotations.
On Wed, Feb 28, 2024 at 11:59 PM Ja Li wrote:
> Dear devs and users,
>
> I just stumbled upon the REST DSL validation part:
> https://camel.apache.org/manual/rest-dsl.html#_client_request_validation
>
> 1) In case I use the "camel-restdsl-openapi-plugin" (Maven plugin) to
> generate REST DSL routes from an openapi specification, e.g.:
> ...
>
> testParam:
>
> name: testparam
>
> in: query
>
> required: true
>
> schema:
>
> type: integer
>
> format: int32
>
> enum: [0, 1, 2, 3, 4]
>
> ...
>
> it will generate the following Camel Route parameter for that:
>
> .param()
>
> .name("testparam")
>
> .type(RestParamType.query)
>
> .dataType("integer")
>
> .allowableValues("0,1,2,3,4")
>
> .required(true)
>
> .endParam()
>
>
> If I now enable the validation via
> "restConfiguration().component("jetty").clientRequestValidation(true);"
>
> I can still pass non-allowed values to the REST endpoint, e.g.
> localhost:8080/rest?testparam=6 and will get an HTTP 200.
>
>
> The documentation so far is only mentioning the "required" property (and
> this one is working) but as the ".allowableValues("0,1,2,3,4")" property
> obviously seems to already have been implemented I'm wondering why it is
> not validated then.
> My request is to add validation for this already existing option.
> It seems similar to the request years ago which has been answers by Claus:
> https://stackoverflow.com/a/50496017/5446400 and he points to the used
> HTTP
> component which should be responsible to actually fulfill the desired
> validation option defined in the route.
> I am currently using platform-http but also tried Jetty - it makes no
> difference.
>
>
> 2) Also can we maybe add support for the JSON property "allowEmptyValue"?
>
>
> 3) One last question which could also fit the user mailing list but as we
> are in the right context right now:
> Do you know any alternatives for validation of request parameters?
> Of course I could put them also into a json, send them to the REST endpoint
> and use a lot of validation annotations on its java data model (provided by
> libs as Jackson etc.) but we are using a HTTP GET endpoint and the RFC 7231
> standards do not allow GET requests with json bodies.
> So it seems as Camel doesn't provide full validation possibilities we have
> to manually check the incoming HTTP request parameter for validity?
>
> I am grateful for any tips!
>
> Kind regards
> Jacob
>
--
Claus Ibsen
-
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
