ramu11 opened a new pull request, #30481:
URL: https://github.com/apache/camel/pull/30481

   ## Description
   
   This PR addresses 
[[CAMEL-26655](https://issues.apache.org/jira/browse/CAMEL-26655)](https://issues.apache.org/jira/browse/CAMEL-26655).
   
   The Azure Storage Blob consumer currently retrieves all blobs in a container 
during each poll before routing them. This can result in unnecessary Azure 
reads and downloads, especially when the container contains many blobs.
   
   This change introduces `maxMessagesPerPoll` and paginated blob listing.
   
   ### Changes
   
   - Add the `maxMessagesPerPoll` consumer option. The default is `0` 
(unlimited).
   - Iterate through Azure blob listing pages rather than eagerly materializing 
the entire listing.
   - Count successfully retrieved blobs toward the configured poll limit. Blobs 
skipped because they are in progress, no longer exist, or cannot be retrieved 
do not count.
   - Retain listing progress across polls and wrap around after reaching the 
end, so retained blobs do not cause later pages to be starved.
   - Preserve Camel batch metadata (`CamelBatchSize`, `CamelBatchIndex`, and 
`CamelBatchComplete`).
   
   ### Batch processing semantics
   
   The consumer prepares the complete batch before routing its first exchange. 
This bounded-prefetch approach preserves the exact Camel Batch Consumer 
metadata.
   
   It is not strict just-in-time retrieval immediately before each route 
invocation. Strict JIT retrieval and exact batch-size metadata cannot both be 
guaranteed when the successful retrieval count is not known in advance. This 
trade-off is documented and is open for maintainer feedback.
   
   ### Tests
   
   Verified locally:
   
   - 18 targeted unit tests passed.
   - 7 `BlobConsumerIT` integration tests passed using Azurite.
   - Total: 25 tests, 0 failures, 0 errors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to