Help Needed: Splitter and Parallel Processing

[Luiz Eduardo Valmont]

Hi, friends!

According to instructions found on this link (splitter)
 and this other link (bean)
, I wrote this (simple?) route:

  *public* *void* configure() *throws* Exception {
> from( "quartz2://instancesProcessor?cron=" + quartz )
>   .routeId( "InstancesProcessor" )
> .process( loadUnprocessedInstancesProcessor )
> .split()
>   .body()
>   .log( LoggingLevel.WARN, "Body type is:
> ${body.getClass().getName()}" )
>   .bean( instanceHandler, "handle" )
>   .log( LoggingLevel.WARN, "Handled one of:
> ${body.getClass().getName()}" )
> .end()
>   .end()
> .end()
> ;
>

And this is the general output:

2019-08-06 04:04:20.281 TRACE fcdc964e5ee8 --- [aultQuartzScheduler-Orthanc
> Overseer_Worker-1] .r.LoadUnprocessedInstancesProcessorImpl : Found 33
> unprocessed instances
> 2019-08-06 04:04:20.373  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>   : Body type is: model.Instance
> 2019-08-06 04:04:20.507  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler : Tagged instance [...]
> 2019-08-06 04:04:20.518 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl: Saving Instance [...]
> 2019-08-06 04:04:20.559 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.608  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>   : Handled one of: model.Instance
> 2019-08-06 04:04:20.610  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>   : Body type is: model.Instance
> 2019-08-06 04:04:20.617  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler : Tagged instance [...]
> 2019-08-06 04:04:20.621 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl: Saving Instance [...]
> 2019-08-06 04:04:20.625 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.750  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>   : Handled one of: [..]
> 2019-08-06 04:04:20.751  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>   : Body type is: model.Instance
> 2019-08-06 04:04:30.017 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed instances
> 2019-08-06 04:04:30.020  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2] InstancesProcessor
>   : Body type is: model.Instance
> 2019-08-06 04:04:40.020 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed instances
> 2019-08-06 04:04:40.027  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3] InstancesProcessor
>   : Body type is: model.Instance
>

It basically process 2 instances out of 33 available. And that's it. For
the following interations, it merely logs a single Body type. Since it
executes at least once, I reckon that "type-wise" I'm ok, else nothing
would work. I suppose I'm running out of consumers because of some other
route. But it still beats me because, as the logging shows, the
instanceHandler bean does not hang.

Question: what can be wrong/faulty in my routing setup? I wanted to execute
the instanceHandler in parallel, like X at a time. But one step at a time.

Any help or guidance or guess are mostly welcome. Even wild guesses.

Thanks in advance!

Luiz Eduardo Guida Valmont

(21) 2220-6223
(21) 98663-1337

ALIS - Soluções em Engenharia & Sistemas

Re: Camel + Spring Boot as a backend?

[Luiz Eduardo Valmont]

I use Spring Boot, Camel and Quartz on a symbiotic-ish application. There's
REST endpoints as well.

On Tue, 16 Apr 2019, 11:54 Michael Joyner,  wrote:

> I assume using Spring MVC for the front end. I think that you would be
> fine. Someone else will probably chime in from the project and confirm.
>
> On Tue, Apr 16, 2019 at 7:29 AM Ron Cecchini 
> wrote:
>
> > Hi, all.  If/when anyone has any time, I was hoping to get a few quick
> > opinions.
> > (and I do mean be brief; I don't want anyone wasting time on this.)
> >
> > *** Could Camel + Spring Boot *alone* be used to implement the Java
> > portion of
> > *** a simple backend for a low-throughput, non-realtime system that
> > doesn't need to scale?
> >
> > Backstory: I was thinking about the web site for my sons' little league
> (I
> > didn't create it),
> > and what I might do if I were to redo it totally from scratch without
> > using Wordpress, etc.
> >
> > This quickly morphed (away from baseball and) into what it would take to
> > implement what
> > basically amounts to a simple inventory tracking system of sorts.
> >
> > And the picture I had in mind is something like.  Imagine you are an
> > online business with:
> >
> > - 100 customers or offices  (whatever - call them "sites")
> >
> > - Each "site" is going to place at most, on average, 1 order a day
> >   where an "order" might be to ship goods to another site, request goods
> > from another site,
> >   or order goods from a vendor  (whatever - the details of the "order"
> > don't matter)
> >
> > - Each site now needs to be able to track the progress of its order
> (where
> > its good are)
> >
> > Basically, something like a poor man's Amazon or USPS/UPS/FedEx tracking
> > system.
> >
> > Again, the system doesn't need to scale because there will never be more
> > than 100-200 sites.
> > Sites will spend the majority of their time inactive; i.e. not placing
> > orders.
> > Each order is a simple movement of goods (shipping or
> procuring/receiving).
> > There are no real time demands to know exactly where goods are.  (A daily
> > update would suffice.)
> >
> > Since the concurrency needs seem to be negligible, I don't see a need for
> > a JBoss app server,
> > or a distributed server farm, etc.  I feel like Camel will already handle
> > whatever concurrency
> > issues that may arise, and its ability to seamlessly integrate with so
> > many others means alot of
> > the work is already done for me.
> >
> > What are your thoughts on trying to implement the backend of this simple
> > inventory system with
> > a pretty simple Spring Boot + Camel + RDMS application, hosted on a beefy
> > server and not running
> > in an app server or Docker?
> >
> > Thanks.
> >
> > Ron
> >
>
>
> --
> Sincerely,
> Michael Joyner
>

Re: Getting Error Bad HTTP Error

[Luiz Eduardo Valmont]

Hi Ayush!

In the error message, is the last tic part of the header? What is exactly
the Content-Type header? Can you repeat the request over the command line
or postman interface and send us the verbose result?

Thanks!

On Thu, 28 Feb 2019, 13:25 Ayush Dixit, 
wrote:

> Hi,
>
> We are getting an exception from client side when we are returning a
> response it is getting the below error we have used the camel restlet. We
> are not able to get the exact reason of this error. Kind help us to
> identify the exact cause of this error.
>
> Getting below error at server side
>
>
> ERROR MSM: ht_error: 0: -->
> + Bad HTTP header encountered, token = Content-type: application/xml;
> charset=UT
> + F-8'
>
>
> Please find below the code
> public class OrderResponseProcessor implements Processor {
>
> /** The Constant LOGGER. */
> private static final Logger  LOGGER = LoggerFactory
> .getLogger(OrderResponseProcessor.class.getName());
> private static final Map successMap;
>
> static {
> successMap =
> IcomsCustomErrorHandling.getErrorMap(TranslatorConstants.SUCCESS_KEY.value());
> }
>
> /*
>  * (non-Javadoc)
>  *
>  * @see org.apache.camel.Processor#process(org.apache.camel.Exchange)
>  */
> @Override
> public void process(Exchange exchange) throws HZNException {
> try {
> final String orderRes = exchange.getIn().getBody(String.class);
> final OrderResponse orderResponse = (OrderResponse)
> XmlUtil.getInstance().toObject(orderRes,
> OrderResponse.class);
> String response = "";
> if (orderResponse != null) {
> if ("OK".equalsIgnoreCase(orderResponse.getStatus())) {
> response = getSuccessMsg(orderResponse.getOrderId());
> } else {
> String orderId = orderResponse.getOrderId();
> String extId =
> exchange.getIn().getHeader(TranslatorConstants.EXT_KEY.value(),
> String.class);
> String ordExtMsg =
> String.format(TranslatorConstants.ORD_EXT_MSG.value(), extId, orderId);
> if
> (orderResponse.getError().get(0).getMessage().equalsIgnoreCase(ordExtMsg)) {
> response = getSuccessMsg(orderId);
> } else {
> throw new HZNException("",
> orderResponse.getError().get(0).getMessage());
>}
> }
> }
> exchange.getIn().setBody(response);
> } catch (Exception e) {
> LOGGER.error("Exception in OrderResponseProcessor :- {}", e);
> final Map errorMap = IcomsCustomErrorHandling
> .getErrorMap(TranslatorConstants.FAIL_KEY.value());
> final String errorCode =
> IcomsCustomErrorHandling.getValue(TranslatorConstants.ERR_CD.value(),
> errorMap);
> final String errorMsg =
> IcomsCustomErrorHandling.getValue(TranslatorConstants.ERR_MSG.value(),
> errorMap);
> throw new HZNException(errorCode, errorMsg + e.getMessage());
> }
>
> }
>
> public final String getSuccessMsg(String orderId) throws HZNException {
> String successMsg = CommonUtil.INSTANCE.getIIPrspnsResponse(
>
> IcomsCustomErrorHandling.getValue(TranslatorConstants.ERR_CD.value(),
> successMap),
>
> IcomsCustomErrorHandling.getValue(TranslatorConstants.ERR_MSG.value(),
> successMap) + orderId);
> return successMsg;
> }
>
> }
>
>
> Many Thanks
> Ayush
>

Re: Mocking Beans

[Luiz Eduardo Valmont]

I confess I'm not familiar with @BeanInject, but if it's similar in purpose
as @Autowired from Spring, then I'd suggest using constructor injection.
That way you can create your object with your mocked bean.

Personally I do it like that in unit testing.

HTH

On Fri, 18 Jan 2019, 13:42 Bradford Fisher  Does Camel have some built in functionality to replace a Bean within a
> Route while maintaining the method called on the bean?
>
> For instance, given the RouteBuilder below, I would like to replace myBean
> with myMockBean during testing. I've considered using AdviceWith, but that
> seams to replace the entire Bean Node rather than just the instance of the
> class used by the Bean Node.
>
> public class MyRouteBuilder extends RouteBuilder
> {
> @BeanInject
> private MyBean myBean;
>
> @Override
> public void configure() throws Exception
> {
> from("direct:myEndpoint")
> .bean(myBean, "myMethod")
> .id("myBeanId");
> }
> }
>

Re: HTTP response is not being chunked

[Luiz Eduardo Valmont]

Hi Tim!

Sorry about the delay. I was summoned into a meeting that took a while.
Plus my PC is in the process of upgrading from Debian 9 to Debian
10/testing.

On the important matter.

There are certainly differences when it comes to headers. But I suspect the
culprit is the accept-encoding. I don't think it's the Camel headers.

I believe because it has to compress the content, it has to read everything
beforehand. Try adding "-H 'Accept-Encoding: gzip'" (double quotes
excluded) to your curl command and see if it changes the actual behaviour.

If it does, then you'll have to configure your client accordingly.

In any case, you can emulate the Async request in full by configuring the
headers with curl's "-H" parameter.

HTH


On Wed, 16 Jan 2019, 15:17 Tim Dudgeon  Hi Luiz,
>
> here are the request headers. For Apache HTTPClient:
>
>accept -> chemical/x-mdl-sdfile
>accept-encoding -> gzip
>breadcrumbId -> ID-6667e3dab68a-38877-1547658276076-0-15
>CamelHttpMethod -> POST
>CamelHttpPath ->
>CamelHttpQuery -> null
>CamelHttpServletRequest ->
> org.apache.catalina.connector.RequestFacade@7551baa3
>CamelHttpServletResponse ->
> org.apache.catalina.connector.ResponseFacade@60e05546
>CamelHttpUri ->
> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelHttpUrl ->
>
> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelServletContextPath -> /v1/converters/dataset_to_sdf
>connection -> Keep-Alive
>content-encoding -> gzip
>Content-Type -> application/x-squonk-dataset-molecule+json
>host -> localhost:8092
>transfer-encoding -> chunked
>user-agent -> Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_191)
>
> For curl:
>
>accept -> chemical/x-mdl-sdfile
>breadcrumbId -> ID-09b7601451a2-34027-1547658868812-0-15
>CamelHttpMethod -> POST
>CamelHttpPath ->
>CamelHttpQuery -> null
>CamelHttpServletRequest ->
> org.apache.catalina.connector.RequestFacade@3af9ee39
>CamelHttpServletResponse ->
> org.apache.catalina.connector.ResponseFacade@1f33d20b
>CamelHttpUri ->
> /chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelHttpUrl ->
>
> http://localhost:8092/chem-services-cdk-basic/rest/v1/converters/dataset_to_sdf
>CamelServletContextPath -> /v1/converters/dataset_to_sdf
>content-encoding -> gzip
>content-length -> 19397182
>Content-Type -> application/x-squonk-dataset-molecule+json
>expect -> 100-continue
>host -> localhost:8092
>user-agent -> curl/7.55.1
>
>
> On 16/01/2019 16:55, Luiz Eduardo Valmont wrote:
> > Hi, Tim!
> >
> > I'd suggest trying to intercept / dump the HttpAsyncClient request as
> well
> > as the curl one. A simple "nc -l -p " in an *NIX OS will do.
> >
> > My guess is that the perceived difference lies in the request headers.
> > Might be in the Accept.
> >
> > Can you post both requests in full, body excluded?
> >
> > Thanks!
> >
> > On Wed, 16 Jan 2019, 14:46 Tim Dudgeon  >
> >> I'm hitting a strange problem with a route that is using the servlet
> >> component to send a large stream of data in its response.
> >> This is part of a route that uses the REST DSL which is set up like
> this:
> >>
> >> restConfiguration().component("servlet").host("0.0.0.0");
> >>
> >> The route sets the response body as an PipedInputStream and then (in a
> >> separate thread) writes to that InputStream and eventually closes it.
> >> The expectation is that as the response size is unknown the data will be
> >> chunked and response returned to the client immediately and the client
> >> will start to consume the data immediately.
> >>
> >> When using curl as the client it seems to be working correctly, and I
> >> see a 'Transfer-Encoding: chunked' header in the response.
> >>
> >> However my real client is Apache HttpAsyncClient and with that what I'm
> >> seeing is that the response is not returned until all the data is
> >> written, and instead of a 'Transfer-Encoding: chunked' header I see a
> >> 'Content-Length: 19196336' header suggesting that Camel has waited for
> >> the entire content to be written so that it can set the Content-Length.
> >> I tried manually setting the 'Transfer-Encoding: chunked' in the
> >> response message but it did not appear and instead I saw the
> >> Content-Length header.
> >> I've got HttpAsyncClient working in a test example to prove that it can
> >> received the response asynchronously, so I think (but can't be sure)
> >> that the problem is on the server (Camel) side.
> >>
> >> What can explain this different behaviour from the same service when
> >> using different clients?
> >>
> >>
>

Re: HTTP response is not being chunked

[Luiz Eduardo Valmont]

Hi, Tim!

I'd suggest trying to intercept / dump the HttpAsyncClient request as well
as the curl one. A simple "nc -l -p " in an *NIX OS will do.

My guess is that the perceived difference lies in the request headers.
Might be in the Accept.

Can you post both requests in full, body excluded?

Thanks!

On Wed, 16 Jan 2019, 14:46 Tim Dudgeon  I'm hitting a strange problem with a route that is using the servlet
> component to send a large stream of data in its response.
> This is part of a route that uses the REST DSL which is set up like this:
>
> restConfiguration().component("servlet").host("0.0.0.0");
>
> The route sets the response body as an PipedInputStream and then (in a
> separate thread) writes to that InputStream and eventually closes it.
> The expectation is that as the response size is unknown the data will be
> chunked and response returned to the client immediately and the client
> will start to consume the data immediately.
>
> When using curl as the client it seems to be working correctly, and I
> see a 'Transfer-Encoding: chunked' header in the response.
>
> However my real client is Apache HttpAsyncClient and with that what I'm
> seeing is that the response is not returned until all the data is
> written, and instead of a 'Transfer-Encoding: chunked' header I see a
> 'Content-Length: 19196336' header suggesting that Camel has waited for
> the entire content to be written so that it can set the Content-Length.
> I tried manually setting the 'Transfer-Encoding: chunked' in the
> response message but it did not appear and instead I saw the
> Content-Length header.
> I've got HttpAsyncClient working in a test example to prove that it can
> received the response asynchronously, so I think (but can't be sure)
> that the problem is on the server (Camel) side.
>
> What can explain this different behaviour from the same service when
> using different clients?
>
>