Dan Brown created CAMEL-8221:
--------------------------------
Summary: No backpressure in aws-sqs Consumer
Key: CAMEL-8221
URL: https://issues.apache.org/jira/browse/CAMEL-8221
Project: Camel
Issue Type: Bug
Components: camel-aws
Affects Versions: 2.14.1
Reporter: Dan Brown
We're using a camel Consumer with an aws-sqs endpoint and running into issues
with large queues and slow consumers. The consumer jvm downloads messages very
fast from sqs, onto the jvm heap, even though the camel receive method
processes each message very slowly (and Ack's when done, with autoAck = false).
The result of this is that the consumer jvm continually fills its heap with an
unbounded queue of incoming messages and eventually throws OOME.
To avoid this failure, we're looking for a way to enable backpressure when
using aws-sqs—e.g. an on-heap bounded queue between the component fetching from
sqs and the user-defined actor processing the messages—but I don't see anything
relevant in the config:
- http://camel.apache.org/aws-sqs.html
Looking at the code, I see that SqsConsumer subtypes
ScheduledBatchPollingConsumer, which subtypes ScheduledPollConsumer. To get
backpressure, should it use something like EventDrivenPollingConsumer
somewhere, which uses a blocking queue to avoid the heap blowup above?
-
https://github.com/apache/camel/blob/camel-2.14.1/components/camel-aws/src/main/java/org/apache/camel/component/aws/sqs/SqsConsumer.java#L52-L57
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledBatchPollingConsumer.java#L31-L34
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledPollConsumer.java#L40-L43
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/EventDrivenPollingConsumer.java#L35-L42
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
