[
https://issues.apache.org/jira/browse/CAMEL-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322314#comment-14322314
]
Steven Marcus edited comment on CAMEL-8221 at 2/17/15 11:51 PM:
----------------------------------------------------------------
I think blocking exchanges is Camel's implementation of back-pressure for
polling consumers.
Have you tried using a seda component to block additional polls until in-flight
exchanges complete?
I only have experience doing this with the swf component:
from("aws-swf://activity").
to("seda:seda?
blockWhenFull=true&size=2&concurrentConsumers=2&waitForTaskToComplete=Always&timeout=0")
SQS is a ScheduledBatchPollingConsumer so you will need to set
maxMessagesPerPoll.
A value of 1 is probably right if you have "slow" consumers.
I don't know if there are any other implications of
ScheduledBatchPollingConsumer vs. a simple PollingConsumer so what I say here
may not be effective with SQS. ymmv... GL!
was (Author: srnm):
I think blocking exchanges is Camel's implementation of back-pressure for
polling consumers.
Have you tried using a seda component to block additional polls until in-flight
exchanges complete?
I only have experience doing this with the swf component:
{{from("aws-swf://activity").
to("seda:seda?
blockWhenFull=true&size=2&concurrentConsumers=2&waitForTaskToComplete=Always&timeout=0")}}
SQS is a ScheduledBatchPollingConsumer so you will need to set
maxMessagesPerPoll.
A value of 1 is probably right if you have "slow" consumers.
I don't know if there are any other implications of
ScheduledBatchPollingConsumer vs. a simple PollingConsumer so what I say here
may not be effective with SQS. ymmv... GL!
> No backpressure in aws-sqs Consumer
> -----------------------------------
>
> Key: CAMEL-8221
> URL: https://issues.apache.org/jira/browse/CAMEL-8221
> Project: Camel
> Issue Type: Improvement
> Components: camel-aws
> Affects Versions: 2.14.1
> Reporter: Dan Brown
> Fix For: Future
>
>
> We're using a camel Consumer with an aws-sqs endpoint and running into issues
> with large queues and slow consumers. The consumer jvm downloads messages
> very fast from sqs, onto the jvm heap, even though the camel receive method
> processes each message very slowly (and Ack's when done, with autoAck =
> false). The result of this is that the consumer jvm continually fills its
> heap with an unbounded queue of incoming messages and eventually throws OOME.
> To avoid this failure, we're looking for a way to enable backpressure when
> using aws-sqs—e.g. an on-heap bounded queue between the component fetching
> from sqs and the user-defined actor processing the messages—but I don't see
> anything relevant in the config:
> - http://camel.apache.org/aws-sqs.html
> Looking at the code, I see that SqsConsumer subtypes
> ScheduledBatchPollingConsumer, which subtypes ScheduledPollConsumer. To get
> backpressure, should it use something like EventDrivenPollingConsumer
> somewhere, which uses a blocking queue to avoid the heap blowup above?
> -
> https://github.com/apache/camel/blob/camel-2.14.1/components/camel-aws/src/main/java/org/apache/camel/component/aws/sqs/SqsConsumer.java#L52-L57
> -
> https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledBatchPollingConsumer.java#L31-L34
> -
> https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledPollConsumer.java#L40-L43
> -
> https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/EventDrivenPollingConsumer.java#L35-L42
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
