[
https://issues.apache.org/jira/browse/CAMEL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238728#comment-13238728
]
Daniel Carleton commented on CAMEL-5113:
----------------------------------------
Thanks for the feedback, Claus.
Ok, sounds like I should apply #1 and #3, change the default for
maxMessagesPerPoll to 1, and use ThrottlingInflightRoutePolicy to control
concurrency.
SQS does support grabbing multiple messages per request, so I suppose a
BatchConsumer makes some sense. If there's only a single Camel context
consuming from the SQS queue, then it doesn't make a difference. However if
you have multiple processes polling the SQS queue, then each one should only
reserve as many messages as it can process at a time. Otherwise messages will
get queued and processing latency will increase.
> Parallel and fault tolerant message processing for SQS endpoints.
> -----------------------------------------------------------------
>
> Key: CAMEL-5113
> URL: https://issues.apache.org/jira/browse/CAMEL-5113
> Project: Camel
> Issue Type: Improvement
> Components: camel-aws
> Reporter: Daniel Carleton
>
> I'm using Camel to implement parallel processing of jobs in an SQS queue, and
> ran into a few issues with the current implementation of the SQS component:
> # SqsConsumer uses a blocking/synchronous processor, which prevents parallel
> processing of multiple messages from a single endpoint.
> # Having a maxMessagesPerPoll other than one doesn't seem to make sense,
> because any messages not being actively processed should be left back in the
> queue for other consumers to have a chance with.
> # Rollback processing doesn't go back to SQS and set the visibility timeout
> to zero, which prevents immediate retries.
> I propose the following solutions to these problems:
> # Use an asynchronous processor in SqsConsumer by way of getAsyncProcessor().
> (Messages in SQS aren't guaranteed to be FIFO anyway, so there should be no
> issue with order of processing.)
> # Replace maxMessagesPerPoll with maxInFlightMessages. Put a semaphore in
> SqsConsumer to control the maximum number of in flight messages, and when
> polling SQS always set the number of available permits as the maximum number
> of messages to retrieve.
> # In the onFailure callback for an exchange set the visibility timeout in SQS
> to zero via ChangeMessageVisibility.
> How does this sound? I'm working on a patch. This is my first work on
> Camel, so if you see any problems with my approach let me know!
> Thanks,
> - Dan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
