Claus Ibsen created CAMEL-27295:
-----------------------------------
Summary: camel-azure-storage-blob - get each blob just before
routing it
Key: CAMEL-27295
URL: https://issues.apache.org/jira/browse/CAMEL-27295
Project: Camel
Issue Type: Improvement
Components: camel-azure
Reporter: Claus Ibsen
Fix For: 4.24.0
Follow-up of CAMEL-26655 (PR https://github.com/apache/camel/pull/30481).
h3. Where CAMEL-26655 leaves it
PR #30481 adds {{maxMessagesPerPoll}} and lists the container page by page, but
it still gets every blob of the poll (opens the stream, or downloads to
{{fileDir}}) before the first exchange is routed. That keeps
{{CamelBatchSize}}, {{CamelBatchIndex}} and {{CamelBatchComplete}} exact. With
{{maxMessagesPerPoll}} set, this is bounded by the limit. With the default
{{maxMessagesPerPoll=0}}, every listed blob is still opened up front: the cost
measured in CAMEL-26655 (48 MiB buffered before the first exchange for 12 blobs
of 5 MiB).
h3. Proposed change
* Get each blob just before its exchange is routed, fetching one blob ahead, as
the S3 consumer does since CAMEL-26662 and minio since CAMEL-17100.
* Keep {{maxMessagesPerPoll}} counting blobs that were got, not listed ones
(the CAMEL-25770 rule: blobs that cannot be got, are in progress or are gone do
not count).
* Keep the CAMEL-25770 rule that a poll fails only when no listed blob could be
got.
* Batch properties: with one blob fetched ahead, {{CamelBatchIndex}} and
{{CamelBatchComplete}} stay exact (an exchange is the last one when there is no
next successful get). Only {{CamelBatchSize}} is no longer exact, since the
number of successful gets is not known when the first exchange is routed.
h3. Docs and upgrade guide
* Document the batch properties, and that the {{completionFromBatchConsumer}}
option of the Aggregate EIP does not fit this consumer ({{AggregateProcessor}}
counts exchanges against {{CamelBatchSize}}).
* Add an entry to the 4.24 upgrade guide: blobs are read at routing time
instead of at poll time, and {{CamelBatchSize}} changes meaning.
_Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)