Claus Ibsen created CAMEL-27183:
-----------------------------------

             Summary: camel-azure-storage-queue - extend message visibility 
while it is processed
                 Key: CAMEL-27183
                 URL: https://issues.apache.org/jira/browse/CAMEL-27183
             Project: Camel
          Issue Type: Improvement
          Components: camel-azure
            Reporter: Claus Ibsen


h3. Problem
The consumer receives messages with {{visibilityTimeout}} (30 s by default on 
the service) and deletes each one when its exchange completes. Nothing extends 
the visibility while the exchange is in flight. When processing takes longer 
than the visibility timeout:
* the message becomes visible again and is received again: by another consumer, 
or by the same one, since {{QueueConsumer.poll}} runs every {{delay}} 
regardless of exchanges still in flight from the previous poll when the route 
continues asynchronously;
* it is processed a second time, in parallel with the first;
* the delete of the first exchange then fails with {{PopReceiptMismatch}}, 
because receiving the message again issued a new pop receipt. {{processCommit}} 
hands it to the exception handler ("Error occurred during deleting message. 
This exception is ignored.") and the message is consumed again later.

h3. How to reproduce
{noformat}
from("azure-storage-queue:account/myqueue?visibilityTimeout=PT3S")
    .delay(8000)   // asynchronous by default: does not hold up the consumer
    .log("${header.CamelAzureStorageQueueMessageId} 
dequeueCount=${header.CamelAzureStorageQueueDequeueCount}");
{noformat}
One message sent: it is received again every 3 s while the first exchange is 
still processing, and the deletes fail with {{PopReceiptMismatch}}. With a step 
that holds up the consumer ({{.delay(8000).syncDelayed()}}), the same message 
is received once and deleted - but a second consumer on the same queue would 
still receive it again after 3 s.

Seen while adding {{visibilityTimeout}} to the Kamelet 
{{azure-storage-queue-source}}: its description now has to warn that a message 
still being processed after the visibility timeout can be received again.

h3. Where
{{components/camel-azure/camel-azure-storage-queue/src/main/java/org/apache/camel/component/azure/storage/queue/QueueConsumer.java}}
 - {{poll}} / {{processBatch}} / {{processCommit}}.

h3. Suggested fix
Add a consumer option like {{extendMessageVisibility}} of camel-aws2-sqs: while 
an exchange is in flight, a scheduled task calls {{updateMessage}} with the 
visibility timeout (shortly before it ends) for its message, and keeps the new 
pop receipt so that {{processCommit}} deletes with the latest one 
({{QueueClientWrapper.updateMessage}} already returns it). The task stops when 
the exchange completes or fails.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to