Claus Ibsen created CAMEL-27295:
-----------------------------------

             Summary: camel-azure-storage-blob - get each blob just before 
routing it
                 Key: CAMEL-27295
                 URL: https://issues.apache.org/jira/browse/CAMEL-27295
             Project: Camel
          Issue Type: Improvement
          Components: camel-azure
            Reporter: Claus Ibsen
             Fix For: 4.24.0


Follow-up of CAMEL-26655 (PR https://github.com/apache/camel/pull/30481).

h3. Where CAMEL-26655 leaves it
PR #30481 adds {{maxMessagesPerPoll}} and lists the container page by page, but 
it still gets every blob of the poll (opens the stream, or downloads to 
{{fileDir}}) before the first exchange is routed. That keeps 
{{CamelBatchSize}}, {{CamelBatchIndex}} and {{CamelBatchComplete}} exact. With 
{{maxMessagesPerPoll}} set, this is bounded by the limit. With the default 
{{maxMessagesPerPoll=0}}, every listed blob is still opened up front: the cost 
measured in CAMEL-26655 (48 MiB buffered before the first exchange for 12 blobs 
of 5 MiB).

h3. Proposed change
* Get each blob just before its exchange is routed, fetching one blob ahead, as 
the S3 consumer does since CAMEL-26662 and minio since CAMEL-17100.
* Keep {{maxMessagesPerPoll}} counting blobs that were got, not listed ones 
(the CAMEL-25770 rule: blobs that cannot be got, are in progress or are gone do 
not count).
* Keep the CAMEL-25770 rule that a poll fails only when no listed blob could be 
got.
* Batch properties: with one blob fetched ahead, {{CamelBatchIndex}} and 
{{CamelBatchComplete}} stay exact (an exchange is the last one when there is no 
next successful get). Only {{CamelBatchSize}} is no longer exact, since the 
number of successful gets is not known when the first exchange is routed.

h3. Docs and upgrade guide
* Document the batch properties, and that the {{completionFromBatchConsumer}} 
option of the Aggregate EIP does not fit this consumer ({{AggregateProcessor}} 
counts exchanges against {{CamelBatchSize}}).
* Add an entry to the 4.24 upgrade guide: blobs are read at routing time 
instead of at poll time, and {{CamelBatchSize}} changes meaning.

_Claude Code on behalf of Claus Ibsen_



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to