Re: How would you parallelize this?

[Giovanni Condello]

Of course, I've just hit send and realized the option is actually called 
concurrentConsumers

________________________________
From: Giovanni Condello <giovanni.conde...@coderit.it>
Sent: Friday, February 12, 2021 8:33:08 AM
To: users@camel.apache.org <users@camel.apache.org>
Subject: Re: How would you parallelize this?

Hi Ron,

Not sure I understood the question correctly, but if you simply want to make 
the calls to the ws in parallel you can use the parallel consumers options of 
the SEDA endpoint.

Btw, you should probably double check and configure the SEDA queue size and 
rejection policy (in case the SEDA is full) as well in this scenario.

Giovanni
________________________________
From: Ron Cecchini <roncecch...@comcast.net>
Sent: Friday, February 12, 2021 8:24:08 AM
To: users@camel.apache.org <users@camel.apache.org>
Subject: How would you parallelize this?

Hi, everyone.

Say you have 2 vectors of integers, V1 and V2, of size M and N, resp.

The core of the processing is to pair each element of V1 with each element V2 
and call a web service with the 2 integers.

So, M x N calls to a web service.

I have routes that look like the following pseudo-Camel:

.method("myBean", "getV1")
.split(body()).streaming()
    .setHeader("X", body())
    .method("myBean", "getV2")
    .split(body()).streaming()
        .setHeader("Y", body())
        .to("seda:call-web-service")

.from("seda:call-web-service")
    .toD("web-service?param1=X&param2=Y")

The M x N calls are completely independent of each other, but they are 
request-response and we do need to handle the web service's response.

The above processes things synchronously, and each call to "web-service" takes 
about 0.5 seconds.

Originally, I was processing small, 10 x 10 jobs, which took < minute, and was 
tolerable.

But now the jobs are on the order of 50 x 50, which are taking about 20 minutes 
- and that's a no go.

I'm in an OpenShift environment and can scale up "web-service" to as many pods 
as I'd like.

But that doesn't do anything unless I can get rid of the synchronous processing 
on the Camel side.

So what's a good strategy for breaking this up and parallelizing the calls?

Can I leave it all in Camel routes, and use .threads()?

Or do I have to put this in client API code and make calls to 
asyncCallbackRequestBody()?

Thank you.