[
https://issues.apache.org/jira/browse/CAMEL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247838#comment-14247838
]
Claus Ibsen commented on CAMEL-5113:
------------------------------------
[~ceposta] looks fine. I guess even a concurrent scheduled thread pool works
fine since you tested this in real life ;)
Though would have though a boos and work pool approach would be able to react
faster. For example with scheduling and the workers are constantly processing
messages. Then when they are done they reschedule, eg sleep for 1/2 sec etc
before being ready for next task. Where as a boos worker pool approach the boos
is always working and putting new messages in the backlog queue for the worker
theads. The drawback is the thread context-switch between boss/worker.
But lets apply this patch - its a simple solution to this, and has been battle
tested and works fast as you say.
> Parallel and fault tolerant message processing for SQS endpoints.
> -----------------------------------------------------------------
>
> Key: CAMEL-5113
> URL: https://issues.apache.org/jira/browse/CAMEL-5113
> Project: Camel
> Issue Type: Improvement
> Components: camel-aws
> Reporter: Daniel Carleton
> Assignee: Christian Posta
> Fix For: 2.15.0
>
>
> I'm using Camel to implement parallel processing of jobs in an SQS queue, and
> ran into a few issues with the current implementation of the SQS component:
> # SqsConsumer uses a blocking/synchronous processor, which prevents parallel
> processing of multiple messages from a single endpoint.
> # Having a maxMessagesPerPoll other than one doesn't seem to make sense,
> because any messages not being actively processed should be left back in the
> queue for other consumers to have a chance with.
> # Rollback processing doesn't go back to SQS and set the visibility timeout
> to zero, which prevents immediate retries.
> I propose the following solutions to these problems:
> # Use an asynchronous processor in SqsConsumer by way of getAsyncProcessor().
> (Messages in SQS aren't guaranteed to be FIFO anyway, so there should be no
> issue with order of processing.)
> # Replace maxMessagesPerPoll with maxInFlightMessages. Put a semaphore in
> SqsConsumer to control the maximum number of in flight messages, and when
> polling SQS always set the number of available permits as the maximum number
> of messages to retrieve.
> # In the onFailure callback for an exchange set the visibility timeout in SQS
> to zero via ChangeMessageVisibility.
> How does this sound? I'm working on a patch. This is my first work on
> Camel, so if you see any problems with my approach let me know!
> Thanks,
> - Dan
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
