Claus Ibsen created CAMEL-27183:
-----------------------------------
Summary: camel-azure-storage-queue - extend message visibility
while it is processed
Key: CAMEL-27183
URL: https://issues.apache.org/jira/browse/CAMEL-27183
Project: Camel
Issue Type: Improvement
Components: camel-azure
Reporter: Claus Ibsen
h3. Problem
The consumer receives messages with {{visibilityTimeout}} (30 s by default on
the service) and deletes each one when its exchange completes. Nothing extends
the visibility while the exchange is in flight. When processing takes longer
than the visibility timeout:
* the message becomes visible again and is received again: by another consumer,
or by the same one, since {{QueueConsumer.poll}} runs every {{delay}}
regardless of exchanges still in flight from the previous poll when the route
continues asynchronously;
* it is processed a second time, in parallel with the first;
* the delete of the first exchange then fails with {{PopReceiptMismatch}},
because receiving the message again issued a new pop receipt. {{processCommit}}
hands it to the exception handler ("Error occurred during deleting message.
This exception is ignored.") and the message is consumed again later.
h3. How to reproduce
{noformat}
from("azure-storage-queue:account/myqueue?visibilityTimeout=PT3S")
.delay(8000) // asynchronous by default: does not hold up the consumer
.log("${header.CamelAzureStorageQueueMessageId}
dequeueCount=${header.CamelAzureStorageQueueDequeueCount}");
{noformat}
One message sent: it is received again every 3 s while the first exchange is
still processing, and the deletes fail with {{PopReceiptMismatch}}. With a step
that holds up the consumer ({{.delay(8000).syncDelayed()}}), the same message
is received once and deleted - but a second consumer on the same queue would
still receive it again after 3 s.
Seen while adding {{visibilityTimeout}} to the Kamelet
{{azure-storage-queue-source}}: its description now has to warn that a message
still being processed after the visibility timeout can be received again.
h3. Where
{{components/camel-azure/camel-azure-storage-queue/src/main/java/org/apache/camel/component/azure/storage/queue/QueueConsumer.java}}
- {{poll}} / {{processBatch}} / {{processCommit}}.
h3. Suggested fix
Add a consumer option like {{extendMessageVisibility}} of camel-aws2-sqs: while
an exchange is in flight, a scheduled task calls {{updateMessage}} with the
visibility timeout (shortly before it ends) for its message, and keeps the new
pop receipt so that {{processCommit}} deletes with the latest one
({{QueueClientWrapper.updateMessage}} already returns it). The task stops when
the exchange completes or fails.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)