Re: FetchS3Object fails to fetch small files from S3
I was able to replicate this on Minio too and have a fix for it. The change is trivial but it needs more testing. -joey > On Aug 17, 2021, at 8:22 AM, Vibhath Ileperuma > wrote: > > > Hi Joey, > > It happens only for 0B files. > Sorry for the confusion caused. > > Vibhath.
Re: FetchS3Object fails to fetch small files from S3
Hi Joey, It happens only for 0B files. Sorry for the confusion caused. Vibhath.
Re: FetchS3Object fails to fetch small files from S3
Arpad, the validator on range length should prevent that from happening if
there’s no EL. Not set or >= 1 should be the only allowed values.
Vibhath, does this happen with small, non-zero length files?
FYI, opened https://issues.apache.org/jira/browse/NIFI-9055
-joey
> On Aug 17, 2021, at 4:41 AM, Arpad Boda wrote:
>
>
> final Long rangeLength = (context.getProperty(RANGE_LENGTH).isSet() ?
> context.getProperty(RANGE_LENGTH).evaluateAttributeExpressions(flowFile).asDataSize(DataUnit.B).longValue()
> : null);
>
> if (rangeLength != null) {
> request.setRange(rangeStart, rangeStart + rangeLength - 1);
> }
>
> I think the problem is that rangeLength can be 0 when it's not null, so the
> request becomes invalid.
>
>
>
>> On Tue, Aug 17, 2021 at 1:19 PM Vibhath Ileperuma
>> wrote:
>> Hi Joey and Denes,
>>
>> I'm also getting this error for 0 byte files.
>>
>> Thank You.
>>
>>> On Tue, Aug 17, 2021 at 12:25 PM Denes Arvay wrote:
>>> Joey,
>>>
>>> I was able to reproduce the issue with a 0 byte file.
>>> Reverting the commit you were referring to ([1]) fixes the problem.
>>>
>>> Denes
>>>
>>> [1]
>>> https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d
>>>
On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee wrote:
Vibhath, there was a change to allow reading by ranges that I’m wondering
if is causing this.
When you say small, exactly how small are the files?
-joey
>> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma
>> wrote:
>>
>
> Hi All,
>
> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The
> 'FetchS3Object' processor throws the following error only for small
> files. I tried to fetch the same set of files using Nifi 1.13.2. There
> was no issue with 1.13.2. Can you please let me know a way to overcome
> this issue.
>
> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to
> retrieve S3 Object for
> StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1629123317469-1,
> container=default, section=1], offset=10164,
> length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1];
> routing to failure: The requested range is not satisfiable (Service:
> Amazon S3; Status Code: 416; Error Code: InvalidRange; Request ID:
> NDCXP1G52A96HJ2R; S3 Extended Request ID:
> KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=;
> Proxy: null)
>
> Thank You.
> Vibhath
>
Re: FetchS3Object fails to fetch small files from S3
final Long rangeLength = (context.getProperty(RANGE_LENGTH).isSet() ?
context.getProperty(RANGE_LENGTH).evaluateAttributeExpressions(flowFile).asDataSize(DataUnit.B).longValue()
: null);
if (rangeLength != null) {
request.setRange(rangeStart, rangeStart + rangeLength - 1);
}
I think the problem is that rangeLength can be 0 when it's not null, so the
request becomes invalid.
On Tue, Aug 17, 2021 at 1:19 PM Vibhath Ileperuma <
vibhatharunapr...@gmail.com> wrote:
> Hi Joey and Denes,
>
> I'm also getting this error for 0 byte files.
>
> Thank You.
>
> On Tue, Aug 17, 2021 at 12:25 PM Denes Arvay wrote:
>
>> Joey,
>>
>> I was able to reproduce the issue with a 0 byte file.
>> Reverting the commit you were referring to ([1]) fixes the problem.
>>
>> Denes
>>
>> [1]
>> https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d
>>
>> On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee
>> wrote:
>>
>>> Vibhath, there was a change to allow reading by ranges that I’m
>>> wondering if is causing this.
>>>
>>> When you say small, exactly how small are the files?
>>>
>>> -joey
>>>
>>> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma <
>>> vibhatharunapr...@gmail.com> wrote:
>>>
>>>
>>> Hi All,
>>>
>>> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The
>>> 'FetchS3Object' processor throws the following error only for small files.
>>> I tried to fetch the same set of files using Nifi 1.13.2. There was no
>>> issue with 1.13.2. Can you please let me know a way to overcome this issue.
>>>
>>> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to
>>> retrieve S3 Object for
>>> StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim
>>> [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default,
>>> section=1], offset=10164,
>>> length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1];
>>> routing to failure: The requested range is not satisfiable (Service: Amazon
>>> S3; Status Code: 416; Error Code: InvalidRange; Request ID:
>>> NDCXP1G52A96HJ2R; S3 Extended Request ID:
>>> KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=;
>>> Proxy: null)
>>>
>>> Thank You.
>>> Vibhath
>>>
>>>
>>>
Re: FetchS3Object fails to fetch small files from S3
Hi Joey and Denes, I'm also getting this error for 0 byte files. Thank You. On Tue, Aug 17, 2021 at 12:25 PM Denes Arvay wrote: > Joey, > > I was able to reproduce the issue with a 0 byte file. > Reverting the commit you were referring to ([1]) fixes the problem. > > Denes > > [1] > https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d > > On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee > wrote: > >> Vibhath, there was a change to allow reading by ranges that I’m wondering >> if is causing this. >> >> When you say small, exactly how small are the files? >> >> -joey >> >> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma < >> vibhatharunapr...@gmail.com> wrote: >> >> >> Hi All, >> >> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The >> 'FetchS3Object' processor throws the following error only for small files. >> I tried to fetch the same set of files using Nifi 1.13.2. There was no >> issue with 1.13.2. Can you please let me know a way to overcome this issue. >> >> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to >> retrieve S3 Object for >> StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim >> [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default, >> section=1], offset=10164, >> length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1]; >> routing to failure: The requested range is not satisfiable (Service: Amazon >> S3; Status Code: 416; Error Code: InvalidRange; Request ID: >> NDCXP1G52A96HJ2R; S3 Extended Request ID: >> KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=; >> Proxy: null) >> >> Thank You. >> Vibhath >> >> >>
Re: FetchS3Object fails to fetch small files from S3
Joey, I was able to reproduce the issue with a 0 byte file. Reverting the commit you were referring to ([1]) fixes the problem. Denes [1] https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee wrote: > Vibhath, there was a change to allow reading by ranges that I’m wondering > if is causing this. > > When you say small, exactly how small are the files? > > -joey > > On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma < > vibhatharunapr...@gmail.com> wrote: > > > Hi All, > > I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The > 'FetchS3Object' processor throws the following error only for small files. > I tried to fetch the same set of files using Nifi 1.13.2. There was no > issue with 1.13.2. Can you please let me know a way to overcome this issue. > > FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to retrieve > S3 Object for > StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default, > section=1], offset=10164, > length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1]; > routing to failure: The requested range is not satisfiable (Service: Amazon > S3; Status Code: 416; Error Code: InvalidRange; Request ID: > NDCXP1G52A96HJ2R; S3 Extended Request ID: > KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=; > Proxy: null) > > Thank You. > Vibhath > > >
Re: FetchS3Object fails to fetch small files from S3
Vibhath, there was a change to allow reading by ranges that I’m wondering if is causing this. When you say small, exactly how small are the files? -joey > On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma > wrote: > > > Hi All, > > I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The > 'FetchS3Object' processor throws the following error only for small files. I > tried to fetch the same set of files using Nifi 1.13.2. There was no issue > with 1.13.2. Can you please let me know a way to overcome this issue. > > FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to retrieve S3 > Object for > StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default, > section=1], offset=10164, > length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1]; routing > to failure: The requested range is not satisfiable (Service: Amazon S3; > Status Code: 416; Error Code: InvalidRange; Request ID: NDCXP1G52A96HJ2R; S3 > Extended Request ID: > KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=; > Proxy: null) > > Thank You. > Vibhath >
