[
https://issues.apache.org/jira/browse/NIFI-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333499#comment-17333499
]
ASF subversion and git services commented on NIFI-7886:
-------------------------------------------------------
Commit 0ed35345245da057400a7d875e47f57bc2d3794d in nifi's branch
refs/heads/main from Paul Kelly
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0ed3534 ]
NIFI-7886 Added read by range to Azure, GCS, S3 fetch processors
This closes #4576
Co-authored-by: Joey Frazee <[email protected]>
Signed-off-by: Joey Frazee <[email protected]>
> FetchAzureBlobStorage, FetchS3Object, and FetchGCSObject processors should be
> able to fetch ranges
> --------------------------------------------------------------------------------------------------
>
> Key: NIFI-7886
> URL: https://issues.apache.org/jira/browse/NIFI-7886
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.12.0
> Reporter: Paul Kelly
> Assignee: Paul Kelly
> Priority: Minor
> Labels: azureblob, gcs, s3
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Azure Blob Storage, AWS S3, and Google Cloud Storage all support retrieving
> byte ranges of stored objects. Current versions of NiFi processors for these
> services do not support fetching by byte range.
> Allowing to fetch by range would allow multiple enhancements:
> * Parallelized downloads
> ** Faster speeds if the bandwidth delay product of the connection is lower
> than the available bandwidth
> ** Load distribution over a cluster
> * Cost savings
> ** If the file is large and only part of the file is needed, the desired
> part of the file can be downloaded, saving bandwidth costs by not retrieving
> unnecessary bytes
> ** Download failures would only need to retry the failed segment, rather
> than the full file
> * Download extremely large files
> ** Ability to download files that are larger than the available content repo
> by downloading a segment and moving it off to a system with more capacity
> before downloading another segment
>
> Some of these enhancements would require an upstream processor to generate
> multiple flow files, each covering a different part of the overall range.
> Something like this:
> ListS3 -> ExecuteGroovyScript (to split into multiple flow files with
> different range attributes) -> FetchS3Object.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
