Jawad Ahmad created CAMEL-16696:
-----------------------------------
Summary: Enrich and Aggregate Causing OutOfMemory
Key: CAMEL-16696
URL: https://issues.apache.org/jira/browse/CAMEL-16696
Project: Camel
Issue Type: Bug
Components: came-core
Affects Versions: 3.10.0
Reporter: Jawad Ahmad
I have a route that receives the bulk amount of data and then processes data
individually from 3rd Party API using Enrich Pattern. After Enriching I am then
aggregating the responses into a batch of 100 and then sending them to our
system. This causes the camel to allocate memory like crazy, stuck into the
loop and eventually going out of memory. I tried with Sync routes as well but
it gets failed.
Here is the route configuration :
{code:java}
from("direct:sync-likes-from-sf-reviews")
.id("direct:sync-likes-from-sf-reviews")
.process(this::convertReviewPayload)
.split(body())
.to("direct:enrich-and-fetch-likes")
.end();
from("direct:enrich-and-fetch-likes")
.id("direct-enrich-and-fetch-likes")
.enrich("direct:fetch-like-by-sf-review", new
ReviewLikesAggregation(), true)
.aggregate(constant(true), new GroupedBodyAggregationStrategy())
.completionSize(100)
.to("direct:upsert-collected-likes")
.end();
from("direct:upsert-collected-likes")
.id("direct:upsert-collected-likes")
.multicast()
.to("direct:upsert-consumers-batch").to("direct:upsert-likes-batch")
.end()
.end();
from("direct:upsert-consumers-batch?synchronous=false")
.to("direct:upsert-consumers-batch")
.process(this.trustpilotLikeConsumersMapper)
.process(this::setupConsumerHeaders)
.to("direct:upsert-collections")
.end();
from("direct:upsert-likes-batch?synchronous=false")
.to("direct:upsert-likes-batch")
.process(this.trustPilotLikesMapper)
.process(this::setupLikesHeaders)
.to("direct:upsert-collections")
.end();
{code}
Logs Loop :
2021-06-08 12:41:18.568 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.568 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
2021-06-08 12:41:18.569 DEBUG 11896 --- [ent@65b17373-56]
o.apache.camel.processor.SendProcessor : >>>> direct://upsert-consumers-batch
Exchange[]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
