[
https://issues.apache.org/jira/browse/NIFI-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Kelly updated NIFI-14318:
------------------------------
Description: PutAzureBlobStorage_v12 currently uses the Azure Blob Storage
default block size of 4MB. Azure Blob Storage has a maximum number of blocks
of 50,000 per blob. Therefore, the largest file that PutAzureBlobStorage_v12
can currently upload is 4MB * 50,000 = 200GB. The service supports blocks up
to 4,000MiB for a total of 4,000MiB * 50,000 = 190.7TiB per blob, about 1,000x
larger than NiFi can currently upload. The block size should be calculated
dynamically – or at least be configurable – to allow uploading files larger
than 200GB. (was: NIFI-11987 added a block size limit of 4MB. Azure Blob
Storage has a maximum number of blocks of 50,000 per blob. Therefore, the
largest file that PutAzureBlobStorage_v12 can currently upload is 4MB * 50,000
= 200GB. The service supports blocks up to 100MB for a total of 100MB * 50,000
= 5TB per blob. The block size should be calculated dynamically – or at least
be configurable – to allow uploading files larger than 200GB.)
> PutAzureBlobStorage_v12 cannot upload files larger than 200GB
> -------------------------------------------------------------
>
> Key: NIFI-14318
> URL: https://issues.apache.org/jira/browse/NIFI-14318
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.28.1, 2.2.0
> Reporter: Paul Kelly
> Assignee: Paul Kelly
> Priority: Major
> Labels: azure, azureblob
>
> PutAzureBlobStorage_v12 currently uses the Azure Blob Storage default block
> size of 4MB. Azure Blob Storage has a maximum number of blocks of 50,000 per
> blob. Therefore, the largest file that PutAzureBlobStorage_v12 can currently
> upload is 4MB * 50,000 = 200GB. The service supports blocks up to 4,000MiB
> for a total of 4,000MiB * 50,000 = 190.7TiB per blob, about 1,000x larger
> than NiFi can currently upload. The block size should be calculated
> dynamically – or at least be configurable – to allow uploading files larger
> than 200GB.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
