[
https://issues.apache.org/jira/browse/CAMEL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247265#comment-14247265
]
Christian Posta commented on CAMEL-5113:
----------------------------------------
Patch here:
https://github.com/christian-posta/camel/commit/43ad71e254107c8d9783018826936fe1050fce90
Would be great if someone on core camel team could review it ([~davsclaus]),
would be even better if someone could try it out. What this does is allow you
to set a "concurrentConsumers" property that allows you to run multiple threads
of the aws-sqs consumers.
We have deployed this patch in a top e-retailer and processed many many
millions of transactions during cyber Monday and it performed with ZERO flaws,
while draining a very deep queue very fast... I know, vague, but cannot post
the numbers :)
Please give me feedback before I commit it.
> Parallel and fault tolerant message processing for SQS endpoints.
> -----------------------------------------------------------------
>
> Key: CAMEL-5113
> URL: https://issues.apache.org/jira/browse/CAMEL-5113
> Project: Camel
> Issue Type: Improvement
> Components: camel-aws
> Reporter: Daniel Carleton
> Assignee: Christian Posta
> Fix For: 2.15.0
>
>
> I'm using Camel to implement parallel processing of jobs in an SQS queue, and
> ran into a few issues with the current implementation of the SQS component:
> # SqsConsumer uses a blocking/synchronous processor, which prevents parallel
> processing of multiple messages from a single endpoint.
> # Having a maxMessagesPerPoll other than one doesn't seem to make sense,
> because any messages not being actively processed should be left back in the
> queue for other consumers to have a chance with.
> # Rollback processing doesn't go back to SQS and set the visibility timeout
> to zero, which prevents immediate retries.
> I propose the following solutions to these problems:
> # Use an asynchronous processor in SqsConsumer by way of getAsyncProcessor().
> (Messages in SQS aren't guaranteed to be FIFO anyway, so there should be no
> issue with order of processing.)
> # Replace maxMessagesPerPoll with maxInFlightMessages. Put a semaphore in
> SqsConsumer to control the maximum number of in flight messages, and when
> polling SQS always set the number of available permits as the maximum number
> of messages to retrieve.
> # In the onFailure callback for an exchange set the visibility timeout in SQS
> to zero via ChangeMessageVisibility.
> How does this sound? I'm working on a patch. This is my first work on
> Camel, so if you see any problems with my approach let me know!
> Thanks,
> - Dan
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
