[
https://issues.apache.org/jira/browse/CAMEL-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramu reassigned CAMEL-27295:
----------------------------
Assignee: Ramu
> camel-azure-storage-blob - get each blob just before routing it
> ---------------------------------------------------------------
>
> Key: CAMEL-27295
> URL: https://issues.apache.org/jira/browse/CAMEL-27295
> Project: Camel
> Issue Type: Improvement
> Components: camel-azure
> Reporter: Claus Ibsen
> Assignee: Ramu
> Priority: Major
> Fix For: 4.25.0
>
>
> Follow-up of CAMEL-26655 (PR https://github.com/apache/camel/pull/30481).
> h3. Where CAMEL-26655 leaves it
> PR #30481 adds {{maxMessagesPerPoll}} and lists the container page by page,
> but it still gets every blob of the poll (opens the stream, or downloads to
> {{fileDir}}) before the first exchange is routed. That keeps
> {{CamelBatchSize}}, {{CamelBatchIndex}} and {{CamelBatchComplete}} exact.
> With {{maxMessagesPerPoll}} set, this is bounded by the limit. With the
> default {{maxMessagesPerPoll=0}}, every listed blob is still opened up front:
> the cost measured in CAMEL-26655 (48 MiB buffered before the first exchange
> for 12 blobs of 5 MiB).
> h3. Proposed change
> * Get each blob just before its exchange is routed, fetching one blob ahead,
> as the S3 consumer does since CAMEL-26662 and minio since CAMEL-17100.
> * Keep {{maxMessagesPerPoll}} counting blobs that were got, not listed ones
> (the CAMEL-25770 rule: blobs that cannot be got, are in progress or are gone
> do not count).
> * Keep the CAMEL-25770 rule that a poll fails only when no listed blob could
> be got.
> * Batch properties: with one blob fetched ahead, {{CamelBatchIndex}} and
> {{CamelBatchComplete}} stay exact (an exchange is the last one when there is
> no next successful get). Only {{CamelBatchSize}} is no longer exact, since
> the number of successful gets is not known when the first exchange is routed.
> h3. Docs and upgrade guide
> * Document the batch properties, and that the {{completionFromBatchConsumer}}
> option of the Aggregate EIP does not fit this consumer
> ({{AggregateProcessor}} counts exchanges against {{CamelBatchSize}}).
> * Add an entry to the 4.24 upgrade guide: blobs are read at routing time
> instead of at poll time, and {{CamelBatchSize}} changes meaning.
> _Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)