I found a solution for this issue. When I use Azure Api directly and
operate with Page all works. It is useful when your azure blob storage
contains a lot of blobs (in my case ~250k).
from("direct:list")
.routeId("listBlobs")
.process(exchange -> {
BlobServiceClient blobServiceClient = new
BlobServiceClientBuilder()
.endpoint(String
.format("https://%s.blob.core.windows.net";,
AzureBlobConfigHelper.ACCOUNT))
.credential(new
StorageSharedKeyCredential(AzureBlobConfigHelper.ACCOUNT,
AzureBlobConfigHelper.ACCESS_KEY))
.buildClient();
BlobContainerClient containerClient =
blobServiceClient.getBlobContainerClient(AzureBlobConfigHelper.BLOB_CONTAINER_NAME);
ListBlobsOptions blobsOptions = new
ListBlobsOptions().setMaxResultsPerPage(5);
Duration duration = Duration.of(10, ChronoUnit.SECONDS);
Iterator<BlobItem>
iterator = containerClient.listBlobs(blobsOptions,
duration).iterator();
exchange.getIn().setBody(iterator);
})
.loopDoWhile(exchange -> {
Iterator<BlobItem> iterator =
(Iterator<BlobItem>) exchange.getIn().getBody(Iterator.class);
return iterator.hasNext();
})
.process(exchange -> {
Iterator<BlobItem> iterator =
(Iterator<BlobItem>) exchange.getIn().getBody(Iterator.class);
exchange.getIn().setBody(iterator.next());
})
.process(exchange -> {
BlobItem item = exchange.getIn().getBody(BlobItem.class);
log.info(item);
})
.end();
On Tue, 26 Jan 2021 at 13:57, Omar Al-Safi <[email protected]> wrote:
> Hi,
>
> Actually you are doing nothing wrong here. Apparently when you set
> maxResultsPerPage, in Azure client, that means it will return pageable
> list, now in every pagelist it will return the max blobs that you set
> earlier in `maxResultsPerPage` which is IMO pretty weird from Azure's side.
> Now in the Camel component side, we just collect all that list of blobs in
> the pages into one list and return that in the body and hence the reason
> why you see no effects. I hope Azure will improve the API in the future to
> allow alternative options as max results regardless of the pageable list.
>
> Regards,
> Omar
>
> On Mon, Jan 25, 2021 at 8:04 PM Mark Andreev <[email protected]>
> wrote:
>
> > Hi, I use Spring 2.4.2 with camel-azure-storage-blob 3.7.0.
> >
> > When I set BlobConstants.LIST_BLOB_OPTIONS as "new
> > ListBlobsOptions().setMaxResultsPerPage(5)" driver returns all entities
> > from blob.
> >
> > from("direct:list")
> > .routeId("listBlobs")
> > .setHeader(BlobConstants.LIST_BLOB_OPTIONS)
> >
> > .constant(new ListBlobsOptions().setMaxResultsPerPage(5))
> > .to(
> > String.format(
> > "
> >
> >
> azure-storage-blob://%s/%s?accessKey=RAW(%s)&operation=listBlobs&synchronous=true
> > ",
> > ACCOUNT,
> > BLOB_CONTAINER_NAME,
> > ACCESS_KEY
> > )
> > )
> > .process(exchange -> {
> > int itemsCount = exchange.getIn().getBody(List.class).size();
> > log.info(String.format("Items count = '%d'", itemsCount));
> > });
> >
> > How should I use this API to fetch only 5 items?
> >
> > Full example:
> >
> >
> https://github.com/mrk-andreev/example-camel-azure-blob-pagination-bug/blob/master/src/main/java/name/mrkandreev/camel/MyApplication.java
> > --
> > Best regards,
> > Mark Andreev
> >
>
--
Best regards,
Mark Andreev
