Fwd: Camel always sending MQRFH2 headers to target WebsphereMQ queue

[Wang Yan]

@Willem.Jiang [via Camel] 

Thanks for the reply, I am using openJRE11

BTW , is the cast exception due to java version? My assumption is it should
be related to IBM libs
according to code, during runtime, the session is expected
as MQQueueSession instance but it is  MQSession instance instead .

MQQueueSession wmqSession = (MQQueueSession) session

-- Forwarded message -
From: Willem.Jiang [via Camel] 
Date: Sat, Oct 19, 2019 at 3:31 AM
Subject: Re: Camel always sending MQRFH2 headers to target WebsphereMQ queue
To: W.Y 


Which version of JDK are you using?
>From the stack trace, it looks like you cannot cast the
com.ibm.mq.jms.MQSession to com.ibm.mq.jms.MQQueueSession.
I think you need to check the client code for more information.

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Fri, Oct 18, 2019 at 11:51 PM Wang Yan <[hidden email]
> wrote:

>
> I have problem , camel is always sending MQRFH2 headers to target
> WebsphereMQ queue
>
> I used below code try to solve the problem but i got
> Setup of JMS message listener invoker failed for destination
>
> - trying to recover. Cause: class com.ibm.mq.jms.MQSession cannot be cast
> to class com.ibm.mq.jms.MQQueueSession (com.ibm.mq.jms.MQSession and
> com.ibm.mq.jms.MQQueueSession are in unnamed module of loader
> org.springframework.boot.loader.LaunchedURLClassLoader @5056dfcb)
>
>
> IBM library i used is
>  // https://mvnrepository.com/artifact/com.ibm.mq/com.ibm.mq.allclient
> implementation 'com.ibm.mq:com.ibm.mq.allclient:9.1.1.0'
>
> What could be the problem? Did I used to wrong libs? any suggestions or
> feedback are more than welcome!
>
>
> JmsComponent wmq = new JmsComponent(connectionFactory);
> wmq.setDestinationResolver(new DestinationResolver() { public Destination
> resolveDestinationName(Session session, String destinationName, boolean
> pubSubDomain) throws JMSException { MQQueueSession wmqSession =
> (MQQueueSession) session; return wmqSession.createQueue("queue:///" +
> destinationName + "?targetClient=1"); } });


--
If you reply to this email, your message will be added to the discussion
below:
https://camel.465427.n5.nabble.com/Camel-always-sending-MQRFH2-headers-to-target-WebsphereMQ-queue-tp5845716p5845741.html
To unsubscribe from Camel always sending MQRFH2 headers to target
WebsphereMQ queue, click here

.
NAML

Re: Camel always sending MQRFH2 headers to target WebsphereMQ queue

[Willem Jiang]

Which version of JDK are you using?
>From the stack trace, it looks like you cannot cast the
com.ibm.mq.jms.MQSession to com.ibm.mq.jms.MQQueueSession.
I think you need to check the client code for more information.

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Fri, Oct 18, 2019 at 11:51 PM Wang Yan  wrote:
>
> I have problem , camel is always sending MQRFH2 headers to target
> WebsphereMQ queue
>
> I used below code try to solve the problem but i got
> Setup of JMS message listener invoker failed for destination
>
> - trying to recover. Cause: class com.ibm.mq.jms.MQSession cannot be cast
> to class com.ibm.mq.jms.MQQueueSession (com.ibm.mq.jms.MQSession and
> com.ibm.mq.jms.MQQueueSession are in unnamed module of loader
> org.springframework.boot.loader.LaunchedURLClassLoader @5056dfcb)
>
>
> IBM library i used is
>  // https://mvnrepository.com/artifact/com.ibm.mq/com.ibm.mq.allclient
> implementation 'com.ibm.mq:com.ibm.mq.allclient:9.1.1.0'
>
> What could be the problem? Did I used to wrong libs? any suggestions or
> feedback are more than welcome!
>
>
> JmsComponent wmq = new JmsComponent(connectionFactory);
> wmq.setDestinationResolver(new DestinationResolver() { public Destination
> resolveDestinationName(Session session, String destinationName, boolean
> pubSubDomain) throws JMSException { MQQueueSession wmqSession =
> (MQQueueSession) session; return wmqSession.createQueue("queue:///" +
> destinationName + "?targetClient=1"); } });

Camel always sending MQRFH2 headers to target WebsphereMQ queue

[Wang Yan]

I have problem , camel is always sending MQRFH2 headers to target
WebsphereMQ queue

I used below code try to solve the problem but i got
Setup of JMS message listener invoker failed for destination

- trying to recover. Cause: class com.ibm.mq.jms.MQSession cannot be cast
to class com.ibm.mq.jms.MQQueueSession (com.ibm.mq.jms.MQSession and
com.ibm.mq.jms.MQQueueSession are in unnamed module of loader
org.springframework.boot.loader.LaunchedURLClassLoader @5056dfcb)


IBM library i used is
 // https://mvnrepository.com/artifact/com.ibm.mq/com.ibm.mq.allclient
implementation 'com.ibm.mq:com.ibm.mq.allclient:9.1.1.0'

What could be the problem? Did I used to wrong libs? any suggestions or
feedback are more than welcome!


JmsComponent wmq = new JmsComponent(connectionFactory);
wmq.setDestinationResolver(new DestinationResolver() { public Destination
resolveDestinationName(Session session, String destinationName, boolean
pubSubDomain) throws JMSException { MQQueueSession wmqSession =
(MQQueueSession) session; return wmqSession.createQueue("queue:///" +
destinationName + "?targetClient=1"); } });

Re: Problems connecting to OPC UA server

[Claus Ibsen]

Hi

Maybe you can debug the code, as it appears that the node id is null,
maybe debugging can help diagnose the problem?

On Fri, Oct 18, 2019 at 2:31 PM  wrote:
>
> Hi,
>
> I am working on a project where I receive data via OPC UA. I process the data 
> to get a unique OID for SNMP. Afterwards I send the OID to a PRTG Monitoring 
> System.
> Right now, I send data with the Prosys OPC UA simulation server and try to 
> connect with camel. The connection to the server works but I get a Java 
> NullPointerException on class Subscription Manager.
>
> The error message looks like that:
> java.lang.NullPointerException: null
> at 
> org.apache.camel.component.milo.client.internal.SubscriptionManager$Connected.lookupNamespace(SubscriptionManager.java:287)
>  ~[camel-milo-2.24.2.jar:2.24.2]
> at 
> org.apache.camel.component.milo.client.internal.SubscriptionManager$Connected.putSubscriptions(SubscriptionManager.java:168)
>  ~[camel-milo-2.24.2.jar:2.24.2]
> ...
>
> My route looks like that:
> from("milo-client:opc.tcp://localhost:4048/example?method=RAW(ns=5;s=MyLevel)=None")
> .bean("NotrufsaulenTransformation", "transformData")
> .to("snmp://127.0.0.1:161?protocol=udp=POLL");
>
> Is there anything wrong with my route or the connection to the OPC UA server?
>
> I am using Apache Camel version 2.24.2 with Spring Boot Framework 2.1.7.
>
>
> Thank you very much in advance for your help!
>
> Best regards,
> Stefan Lederer
>


-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Problems connecting to OPC UA server

[Lederer.Stefan]

Hi,

I am working on a project where I receive data via OPC UA. I process the data 
to get a unique OID for SNMP. Afterwards I send the OID to a PRTG Monitoring 
System.
Right now, I send data with the Prosys OPC UA simulation server and try to 
connect with camel. The connection to the server works but I get a Java 
NullPointerException on class Subscription Manager.

The error message looks like that:
java.lang.NullPointerException: null
at 
org.apache.camel.component.milo.client.internal.SubscriptionManager$Connected.lookupNamespace(SubscriptionManager.java:287)
 ~[camel-milo-2.24.2.jar:2.24.2]
at 
org.apache.camel.component.milo.client.internal.SubscriptionManager$Connected.putSubscriptions(SubscriptionManager.java:168)
 ~[camel-milo-2.24.2.jar:2.24.2]
...

My route looks like that:
from("milo-client:opc.tcp://localhost:4048/example?method=RAW(ns=5;s=MyLevel)=None")
.bean("NotrufsaulenTransformation", "transformData")
.to("snmp://127.0.0.1:161?protocol=udp=POLL");

Is there anything wrong with my route or the connection to the OPC UA server?

I am using Apache Camel version 2.24.2 with Spring Boot Framework 2.1.7.


Thank you very much in advance for your help!

Best regards,
Stefan Lederer

AW: BridgeErrorHandler on JMS Endpoint does not work as expected

[Daniel.Novak]

Hi Claus,


thanks a lot. I will try the workaround with the MapJmsMessage as soon as I 
have time.


Best regards,

Daniel Novak


Von: Claus Ibsen 
Gesendet: Freitag, 18. Oktober 2019 13:36:02
An: users@camel.apache.org
Betreff: Re: BridgeErrorHandler on JMS Endpoint does not work as expected

Hi

I created a ticket to improve camel-jms
https://issues.apache.org/jira/browse/CAMEL-14083



On Thu, Oct 17, 2019 at 5:35 PM  wrote:

> Hi Claus,
>
>
> this poison message mechanism is yet not configured but I will configure
> it and then this poisonous message will be treated like you said. That way
> every other possible poisonous situation will also work and the
> communication will be more stable so I like the poison message mechanism
> for that sort of problem.
>
>
> But I'm also looking forward to move on to Camel 3.0 
>
>
> Best regards,
>
> Daniel Novak
>
> 
> Von: Claus Ibsen 
> Gesendet: Donnerstag, 17. Oktober 2019 17:13:26
> An: users@camel.apache.org
> Betreff: Re: BridgeErrorHandler on JMS Endpoint does not work as expected
>
> Hi
>
> Ah okay, but if you are using JMS transactions and the message cannot
> be processed successfully and its rolled back. Then the JMS broker
> usually have a configuration to move the message to a DQL after N
> unsuccessful attempts (eg its deemed as a poison message)
>
> Dont you have that with the message that cannot be read as a string
> and you get that "german" error with something about charset decoding
> error?
>
> If the data is a bytes then you can configure Camel to treat the
> message as bytes via jmsMessageType=bytes
> Then maybe there is no charset decoding going on if reading the message
> body
>
> On Thu, Oct 17, 2019 at 4:27 PM  wrote:
> >
> > Hi Claus,
> >
> >
> > thanks a lot for the input.
> >
> > I tried the convertBodyTo method but it seems that this method uses the
> same API which causes the root problem, so that didn't help. But because of
> your idea I tried a custom made processor
> >
> >
> > public void process(Exchange exchange) throws Exception {
> > try {
> > exchange.getIn().getBody(String.class);
> > } catch( Exception ex) {
> > LOG.error("That's no good");
> > exchange.getIn().setBody("");
> > }
> > }
> >
> >
> > which kind of "solves" the problem with the endless loop because the
> exception is handled that way.
> >
> > But because you mentioned that the code in the next camel version is
> changed I don't prefer to implement workarounds in my route which I
> probably have to remove with the next version.
> >
> >
> > So I came to the conclusion that I will configure a "poison message"
> mechanism on MQ itself.
> >
> > This makes sense anyways because there might be other scenarios were an
> endless loop might occur.
> >
> >
> > Nevertheless, thx a lot,
> > Best regards
> > Daniel Novak
> >
> > 
> > Von: Claus Ibsen 
> > Gesendet: Donnerstag, 17. Oktober 2019 14:27:38
> > An: users@camel.apache.org
> > Betreff: Re: BridgeErrorHandler on JMS Endpoint does not work as expected
> >
> > Hi
> >
> > Okay yeah it smell as if there is a little bug in the bean component.
> > However as a workaround you can try to not remove those JMS headers
> > before calling the bean, and/or convert the payload into a string or
> > byte[] via convertBodyTo.
> >
> > In Camel 3 that BeanInvocation check is removed as BeanInovication is
> removed.
> >
> > I created a JIRA about the bug in the bean component
> > https://issues.apache.org/jira/browse/CAMEL-14078
> >
> > On Thu, Oct 17, 2019 at 10:44 AM  wrote:
> > >
> > > Hi Claus,
> > >
> > >
> > > I think the problem with the JMS Consumer is a bug in Camel.
> > >
> > > As described in CiA2 as soon as I am in the route the configured
> errorHandler and onException definition should be triggered as soon as an
> exception occurs.
> > >
> > >
> > > My route is the following:
> > >
> > >
> > > public void configure() {
> > >
>  
> errorHandler(deadLetterChannel(DLCRoute.ENDPOINT_DLC).onPrepareFailure(getErrorProcessor()));
> > >
> > > onException(Exception.class)
> > >
>  
> .maximumRedeliveries(0).logExhaustedMessageHistory(false).logExhausted(false);
> > >
> > > from(ENDPOINT_QUELLE).routeId(ROUTE_ALIAS)
> > > .removeHeaders("*", "JMS*") // Entfernt alle nicht notwendigen
> JMS-Header
> > > .bean(bridgeerrorhandlertestRouteEnricher, "addMetaInfo")
> > > .log(INFO, log, "Ende Route: ${routeId}");
> > > }
> > >
> > > As you can see the first step in the route is to remove unnecessary
> headers.
> > >
> > > This step is executed successfully in the
> RemoveHeadersProcessor#process method.
> > >
> > > Which means that I am already in the route.
> > >
> > >
> > > After that successfully processed step in my route I try to call a
> bean method.
> > >
> > > Regardless of what the method tries to do the method
> 

Re: BridgeErrorHandler on JMS Endpoint does not work as expected

[Claus Ibsen]

Hi

I created a ticket to improve camel-jms
https://issues.apache.org/jira/browse/CAMEL-14083



On Thu, Oct 17, 2019 at 5:35 PM  wrote:

> Hi Claus,
>
>
> this poison message mechanism is yet not configured but I will configure
> it and then this poisonous message will be treated like you said. That way
> every other possible poisonous situation will also work and the
> communication will be more stable so I like the poison message mechanism
> for that sort of problem.
>
>
> But I'm also looking forward to move on to Camel 3.0 
>
>
> Best regards,
>
> Daniel Novak
>
> 
> Von: Claus Ibsen 
> Gesendet: Donnerstag, 17. Oktober 2019 17:13:26
> An: users@camel.apache.org
> Betreff: Re: BridgeErrorHandler on JMS Endpoint does not work as expected
>
> Hi
>
> Ah okay, but if you are using JMS transactions and the message cannot
> be processed successfully and its rolled back. Then the JMS broker
> usually have a configuration to move the message to a DQL after N
> unsuccessful attempts (eg its deemed as a poison message)
>
> Dont you have that with the message that cannot be read as a string
> and you get that "german" error with something about charset decoding
> error?
>
> If the data is a bytes then you can configure Camel to treat the
> message as bytes via jmsMessageType=bytes
> Then maybe there is no charset decoding going on if reading the message
> body
>
> On Thu, Oct 17, 2019 at 4:27 PM  wrote:
> >
> > Hi Claus,
> >
> >
> > thanks a lot for the input.
> >
> > I tried the convertBodyTo method but it seems that this method uses the
> same API which causes the root problem, so that didn't help. But because of
> your idea I tried a custom made processor
> >
> >
> > public void process(Exchange exchange) throws Exception {
> > try {
> > exchange.getIn().getBody(String.class);
> > } catch( Exception ex) {
> > LOG.error("That's no good");
> > exchange.getIn().setBody("");
> > }
> > }
> >
> >
> > which kind of "solves" the problem with the endless loop because the
> exception is handled that way.
> >
> > But because you mentioned that the code in the next camel version is
> changed I don't prefer to implement workarounds in my route which I
> probably have to remove with the next version.
> >
> >
> > So I came to the conclusion that I will configure a "poison message"
> mechanism on MQ itself.
> >
> > This makes sense anyways because there might be other scenarios were an
> endless loop might occur.
> >
> >
> > Nevertheless, thx a lot,
> > Best regards
> > Daniel Novak
> >
> > 
> > Von: Claus Ibsen 
> > Gesendet: Donnerstag, 17. Oktober 2019 14:27:38
> > An: users@camel.apache.org
> > Betreff: Re: BridgeErrorHandler on JMS Endpoint does not work as expected
> >
> > Hi
> >
> > Okay yeah it smell as if there is a little bug in the bean component.
> > However as a workaround you can try to not remove those JMS headers
> > before calling the bean, and/or convert the payload into a string or
> > byte[] via convertBodyTo.
> >
> > In Camel 3 that BeanInvocation check is removed as BeanInovication is
> removed.
> >
> > I created a JIRA about the bug in the bean component
> > https://issues.apache.org/jira/browse/CAMEL-14078
> >
> > On Thu, Oct 17, 2019 at 10:44 AM  wrote:
> > >
> > > Hi Claus,
> > >
> > >
> > > I think the problem with the JMS Consumer is a bug in Camel.
> > >
> > > As described in CiA2 as soon as I am in the route the configured
> errorHandler and onException definition should be triggered as soon as an
> exception occurs.
> > >
> > >
> > > My route is the following:
> > >
> > >
> > > public void configure() {
> > >
>  
> errorHandler(deadLetterChannel(DLCRoute.ENDPOINT_DLC).onPrepareFailure(getErrorProcessor()));
> > >
> > > onException(Exception.class)
> > >
>  
> .maximumRedeliveries(0).logExhaustedMessageHistory(false).logExhausted(false);
> > >
> > > from(ENDPOINT_QUELLE).routeId(ROUTE_ALIAS)
> > > .removeHeaders("*", "JMS*") // Entfernt alle nicht notwendigen
> JMS-Header
> > > .bean(bridgeerrorhandlertestRouteEnricher, "addMetaInfo")
> > > .log(INFO, log, "Ende Route: ${routeId}");
> > > }
> > >
> > > As you can see the first step in the route is to remove unnecessary
> headers.
> > >
> > > This step is executed successfully in the
> RemoveHeadersProcessor#process method.
> > >
> > > Which means that I am already in the route.
> > >
> > >
> > > After that successfully processed step in my route I try to call a
> bean method.
> > >
> > > Regardless of what the method tries to do the method
> org.apache.camel.component.bean.AbstractBeanProcessor#process(org.apache.camel.Exchange,
> org.apache.camel.AsyncCallback) is called.
> > >
> > >
> > > This method has the following line which is causing the exception
> (line 132):
> > >
> > > if (in.getBody() instanceof BeanInvocation) {
> > >
> > >
> > > This is because at this point it is tried to extract the body from the

Re: Current best way to consume paged REST api / Web service

[Tom Coudyzer]

Hi,

Would it be possible to describe or give a sample how to aggregate/enrich a
paged REST API where the page interval is part of the JSON payload that is
part of each HTTP request?

Thank you..

On Wed, Oct 16, 2019 at 4:55 PM Jean-Baptiste Onofré 
wrote:

> Hi Tom,
>
> You can also use a poll enricher with a aggregation.
>
> All depends about the trigger to call the webservice.
>
> Regards
> JB
>
> On 16/10/2019 16:52, Tom Coudyzer wrote:
> > Hi,
> >
> > I just started on learning Apache Camel, so please forgive me my basic
> > questions.
> >
> > I'm having a webservice called via POST having a payload with paging
> > information in it.
> >
> > What is the best way currently in Camel to produce an output which
> contains
> > the consolidated values from the calls.
> >
> > I was able to come up with a solution by using a loopDoWhile with a
> > predicate. I manipulate the paging via a header value in the Exchange.
> >
> > Are there better ways to solve this? Enrich? Aggregator?
> >
> > Thanks for educating me on this !
> >
> > Regards,
> > /Tom
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: batch http requests

[Jean-Baptiste Onofré]

Let me check if I have something close to what you want to do.

Regards
JB

On 18/10/2019 09:24, arshad dar wrote:
> thank you for the reply. can you please provide a small example with
> aggregators ?
> 
> On Fri, Oct 18, 2019 at 12:06 PM Jean-Baptiste Onofré 
> wrote:
> 
>> Hi,
>>
>> you can use a aggregator to gather multiple reads and do a single HTTP
>> request.
>> Another option is to use a Processor.
>>
>> Regards
>> JB
>>
>> On 18/10/2019 08:27, arshad dar wrote:
>>> in the routes below i am reading from file and sending http requests, one
>>> record for each http requests. what i want to do send multiple records in
>>> http request as an array.
>>> can any one help! thanks.
>>>
>>>
>>> from(fileURI)
>>>
>>> .to(dataformatURL)
>>>
>>> .split(body())
>>>
>>> .streaming()
>>>
>>> .parallelProcessing()
>>>
>>>  .marshal().json(JsonLibrary.Jackson)
>>>
>>>.to("direct:api");
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>from("direct:api")
>>>
>>>   .streamCaching()
>>>
>>>   .onException(Exception.class)
>>>
>>>   .handled(true)
>>>
>>>   .to(errorFormat)
>>>
>>>   .end()
>>>
>>>   .loadBalance()
>>>
>>>   .failover(maxAttempts, false, true)
>>>
>>> .to(hostsArray)
>>>
>>> end()
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: batch http requests

[arshad dar]

thank you for the reply. can you please provide a small example with
aggregators ?

On Fri, Oct 18, 2019 at 12:06 PM Jean-Baptiste Onofré 
wrote:

> Hi,
>
> you can use a aggregator to gather multiple reads and do a single HTTP
> request.
> Another option is to use a Processor.
>
> Regards
> JB
>
> On 18/10/2019 08:27, arshad dar wrote:
> > in the routes below i am reading from file and sending http requests, one
> > record for each http requests. what i want to do send multiple records in
> > http request as an array.
> > can any one help! thanks.
> >
> >
> > from(fileURI)
> >
> > .to(dataformatURL)
> >
> > .split(body())
> >
> > .streaming()
> >
> > .parallelProcessing()
> >
> >  .marshal().json(JsonLibrary.Jackson)
> >
> >.to("direct:api");
> >
> >
> >
> >
> >
> >
> >
> >from("direct:api")
> >
> >   .streamCaching()
> >
> >   .onException(Exception.class)
> >
> >   .handled(true)
> >
> >   .to(errorFormat)
> >
> >   .end()
> >
> >   .loadBalance()
> >
> >   .failover(maxAttempts, false, true)
> >
> > .to(hostsArray)
> >
> > end()
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>


-- 
Arshid

Re: Configuring custom Kafka producer/consumer properties on Camel Kafka component/endpoint?

[Omar Al-Safi]

Hi Erik,

To be honest I will keep confluent discussion separate as I personally
don't 100% agree with you and I am afraid this will need entirely separate
thread :P.

Regardimg the maven plugin, this will solve different problem, the problem
of maintaining the configurations with more recent versions.

Yes as Claus mentioned, please create a ticket for it in order to allow the
component to set custom configurations.


Regards,
Omar

On Fri, Oct 18, 2019, 08:59 Erik Boye  wrote:

> Åpen informasjon / Public information
>
> Hi,
>
> Sorry for being offline on this matter
>
> Claus, please do not get political on this.  Confluent Kafka is also
> open source - at least the stuff we are talking about here. And I am pretty
> sure Confluent is one of, if not the, main contributor to Apache Kafka.
> Enterprise open source is the "new way", and you cannot blame others for
> doing the same as Red Hat has been doing successfully for many years! Camel
> should, in my option, provide the extensibility mechanisms to fulfill Camel
> users requirements - even if you consider them to be "proprietary"
> And no, I have absolutely no personal relationship to Confluent. I was
> also a bit surprised when our Kafka Ops tells me I have to use a "schema
> registry" to produce Kafka messages. But I have got used to it, and I even
> like the idea of having some API/schema on the Kafka data streams
>
> A Maven plugin would not solve my original issues - even if it was in
> place when I needed to configure some Confluent specific Kafka properties.
> Or did I misunderstood something? I think we really need a way for users to
> set (additional) Kafka consumer/producer properties that are not known to
> Camel. And other "complex" Camel components do have that. That is my main
> point of view. Do you want me to create an issue for this?
>
> Erik
>
> -Opprinnelig melding-
> Fra: Claus Ibsen 
> Sendt: søndag 13. oktober 2019 18:19
> Til: users@camel.apache.org
> Emne: [SUSPECT URL !]Re: Configuring custom Kafka producer/consumer
> properties on Camel Kafka component/endpoint?
>
> Hi Omar
>
> Yeah the maven plugin idea is really good.
> But maybe we need both.
>
> The maven plugin will only get you the options of the kafka version
> that was used for that release.
> With having a way for end users to set extra options then they could
> work against newer Kafka versions without Camel have to upgrade.
> And maybe also to proprietary options on Confluent.
>
>
> On Sun, Oct 13, 2019 at 2:31 PM Omar Al-Safi  wrote:
> >
> > Hi Claus,
> >
> > I also thought about this, however since Kafka producers/consumers
> options
> > increase and some of them may gets deprecated, I was thinking we can
> have a
> > maven plugin that generate these configurations for us without the need
> to
> > do it manually, it will generate using the configDef (e.g:
> >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fblob%2Ftrunk%2Fclients%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkafka%2Fclients%2Fconsumer%2FConsumerConfig.java%23L555data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830sdata=S%2BJyTAx8u7V0jow9oCELojh9af5dsgDtxYjRzRz89Lg%3Dreserved=0
> ).
> > I did this for DBZ as I built a maven plugin that scrap all these config
> > and generate the camel configurations, you can take a look at my progress
> > here:
> >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fomarsmak%2Fcamel%2Fblob%2FCAMEL-14009%2Fcomponents%2Fcamel-debezium%2Fcamel-debezium-maven-plugin%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fcamel%2Fmaven%2FTestMojo.java%23L24data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830sdata=hTRBEetZ7zBt%2BoS9fSfbQV94G0KRIudY0WlkDeo50Fs%3Dreserved=0
> .
> > Please take a look and share your thoughts about it.
> >
> > Regards,
> > Omar
> >
> > On Sun, 13 Oct 2019 at 14:12, Claus Ibsen  wrote:
> >
> > > Hi
> > >
> > > Yeah this has come up a few times that Kafka is adding new options, or
> > > more particularly Confluent adds propriety options that they dont give
> > > back to the community.
> > >
> > > Can you create a JIRA ticket.
> > >
> > > Then we can look at adding a way to add custom options. You are of
> > > course welcome to try to implement this. We love contributions.
> > >
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcamel%2Fblob%2Fmaster%2FCONTRIBUTING.mddata=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819sdata=jPZhXs6oPXeBZKVQn54EQhA2kTqGLQHm5CqUmCDaXo8%3Dreserved=0
> > >
> > > On Tue, Oct 8, 2019 at 9:57 AM Erik Boye 
> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Is there any way to configure custom Kafka producer/consumer
> properties
> > > on the Camel Kafka 

Re: Configuring custom Kafka producer/consumer properties on Camel Kafka component/endpoint?

[Erik Boye]

Åpen informasjon / Public information

Hi,

Sorry for being offline on this matter

Claus, please do not get political on this.  Confluent Kafka is also open 
source - at least the stuff we are talking about here. And I am pretty sure 
Confluent is one of, if not the, main contributor to Apache Kafka. Enterprise 
open source is the "new way", and you cannot blame others for doing the same as 
Red Hat has been doing successfully for many years! Camel should, in my option, 
provide the extensibility mechanisms to fulfill Camel users requirements - even 
if you consider them to be "proprietary"
And no, I have absolutely no personal relationship to Confluent. I was also a 
bit surprised when our Kafka Ops tells me I have to use a "schema registry" to 
produce Kafka messages. But I have got used to it, and I even like the idea of 
having some API/schema on the Kafka data streams

A Maven plugin would not solve my original issues - even if it was in place 
when I needed to configure some Confluent specific Kafka properties. Or did I 
misunderstood something? I think we really need a way for users to set 
(additional) Kafka consumer/producer properties that are not known to Camel. 
And other "complex" Camel components do have that. That is my main point of 
view. Do you want me to create an issue for this?

Erik

-Opprinnelig melding-
Fra: Claus Ibsen 
Sendt: søndag 13. oktober 2019 18:19
Til: users@camel.apache.org
Emne: [SUSPECT URL !]Re: Configuring custom Kafka producer/consumer properties 
on Camel Kafka component/endpoint?

Hi Omar

Yeah the maven plugin idea is really good.
But maybe we need both.

The maven plugin will only get you the options of the kafka version
that was used for that release.
With having a way for end users to set extra options then they could
work against newer Kafka versions without Camel have to upgrade.
And maybe also to proprietary options on Confluent.


On Sun, Oct 13, 2019 at 2:31 PM Omar Al-Safi  wrote:
>
> Hi Claus,
>
> I also thought about this, however since Kafka producers/consumers options
> increase and some of them may gets deprecated, I was thinking we can have a
> maven plugin that generate these configurations for us without the need to
> do it manually, it will generate using the configDef (e.g:
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fblob%2Ftrunk%2Fclients%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkafka%2Fclients%2Fconsumer%2FConsumerConfig.java%23L555data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830sdata=S%2BJyTAx8u7V0jow9oCELojh9af5dsgDtxYjRzRz89Lg%3Dreserved=0).
> I did this for DBZ as I built a maven plugin that scrap all these config
> and generate the camel configurations, you can take a look at my progress
> here:
> https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fomarsmak%2Fcamel%2Fblob%2FCAMEL-14009%2Fcomponents%2Fcamel-debezium%2Fcamel-debezium-maven-plugin%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fcamel%2Fmaven%2FTestMojo.java%23L24data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830sdata=hTRBEetZ7zBt%2BoS9fSfbQV94G0KRIudY0WlkDeo50Fs%3Dreserved=0.
> Please take a look and share your thoughts about it.
>
> Regards,
> Omar
>
> On Sun, 13 Oct 2019 at 14:12, Claus Ibsen  wrote:
>
> > Hi
> >
> > Yeah this has come up a few times that Kafka is adding new options, or
> > more particularly Confluent adds propriety options that they dont give
> > back to the community.
> >
> > Can you create a JIRA ticket.
> >
> > Then we can look at adding a way to add custom options. You are of
> > course welcome to try to implement this. We love contributions.
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcamel%2Fblob%2Fmaster%2FCONTRIBUTING.mddata=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819sdata=jPZhXs6oPXeBZKVQn54EQhA2kTqGLQHm5CqUmCDaXo8%3Dreserved=0
> >
> > On Tue, Oct 8, 2019 at 9:57 AM Erik Boye  wrote:
> > >
> > > Hi,
> > >
> > > Is there any way to configure custom Kafka producer/consumer properties
> > on the Camel Kafka component/endpoint (without "breaking in")?
> > > I specifically want to configure the Confluent Kafka property with key
> > 'value.subject.name.strategy':
> > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.confluent.io%2Fcurrent%2Fschema-registry%2Fserializer-formatter.htmldata=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819sdata=XyX9RJL%2Fp2CMp%2Fg0enZ7dTrRQ5oalOIcLuy6nWxr%2BcA%3Dreserved=0
> > >
> > > Other Camel components (i.e Camel ActiveMQ; destination.consumer) are
> > made extensible by providing users a way to set "additional 

Re: batch http requests

[Jean-Baptiste Onofré]

Hi,

you can use a aggregator to gather multiple reads and do a single HTTP
request.
Another option is to use a Processor.

Regards
JB

On 18/10/2019 08:27, arshad dar wrote:
> in the routes below i am reading from file and sending http requests, one
> record for each http requests. what i want to do send multiple records in
> http request as an array.
> can any one help! thanks.
> 
> 
> from(fileURI)
> 
> .to(dataformatURL)
> 
> .split(body())
> 
> .streaming()
> 
> .parallelProcessing()
> 
>  .marshal().json(JsonLibrary.Jackson)
> 
>.to("direct:api");
> 
> 
> 
> 
> 
> 
> 
>from("direct:api")
> 
>   .streamCaching()
> 
>   .onException(Exception.class)
> 
>   .handled(true)
> 
>   .to(errorFormat)
> 
>   .end()
> 
>   .loadBalance()
> 
>   .failover(maxAttempts, false, true)
> 
> .to(hostsArray)
> 
> end()
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

batch http requests

[arshad dar]

in the routes below i am reading from file and sending http requests, one
record for each http requests. what i want to do send multiple records in
http request as an array.
can any one help! thanks.


from(fileURI)

.to(dataformatURL)

.split(body())

.streaming()

.parallelProcessing()

 .marshal().json(JsonLibrary.Jackson)

   .to("direct:api");







   from("direct:api")

  .streamCaching()

  .onException(Exception.class)

  .handled(true)

  .to(errorFormat)

  .end()

  .loadBalance()

  .failover(maxAttempts, false, true)

.to(hostsArray)

end()
-- 
Arshid