Re: FetchS3Object fails to fetch small files from S3

2021-08-17 Thread Denes Arvay
Joey, I was able to reproduce the issue with a 0 byte file. Reverting the commit you were referring to ([1]) fixes the problem. Denes [1] https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee wrote: > Vibhath, there was a

Re: FetchS3Object fails to fetch small files from S3

2021-08-17 Thread Joey Frazee
Vibhath, there was a change to allow reading by ranges that I’m wondering if is causing this. When you say small, exactly how small are the files? -joey > On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma > wrote: > >  > Hi All, > > I'm using Nifi 1.14.0 to fetch a set of files from a S3

Re: Re: Disfunctional cluster with version 1.13.2

2021-08-17 Thread Ryan Hendrickson
We've rolled back to 1.11.4 in a couple scenarios. We have also setup a cron job to restart 1.13.2 nodes once a day. Ryan On Tue, Aug 17, 2021 at 1:25 AM Axel Schwarz wrote: > Hey Ryan, > > that sounds awefully familiar. What we successfully battled so far is the > load balancing problem. >

Re: Re: Disfunctional cluster with version 1.13.2

2021-08-17 Thread Joe Witt
If restarting is helping you then generally it should be easily found. Is mem usage spiking? Are thread dumps revealing? On Tue, Aug 17, 2021 at 6:12 AM Ryan Hendrickson < ryan.andrew.hendrick...@gmail.com> wrote: > We've rolled back to 1.11.4 in a couple scenarios. We have also setup a >

Re: Disfunctional cluster with version 1.13.2

2021-08-17 Thread Mark Payne
Ryan, Have you filed Jiras for these? If so, can you share the links? Thanks -Mark On Aug 16, 2021, at 9:32 PM, Ryan Hendrickson mailto:ryan.andrew.hendrick...@gmail.com>> wrote: Axel, We've had significant issues with 1.13.2 in a Cluster as well. We're working on a test config...

Re: FetchS3Object fails to fetch small files from S3

2021-08-17 Thread Vibhath Ileperuma
Hi Joey and Denes, I'm also getting this error for 0 byte files. Thank You. On Tue, Aug 17, 2021 at 12:25 PM Denes Arvay wrote: > Joey, > > I was able to reproduce the issue with a 0 byte file. > Reverting the commit you were referring to ([1]) fixes the problem. > > Denes > > [1] >

Re: FetchS3Object fails to fetch small files from S3

2021-08-17 Thread Arpad Boda
final Long rangeLength = (context.getProperty(RANGE_LENGTH).isSet() ? context.getProperty(RANGE_LENGTH).evaluateAttributeExpressions(flowFile).asDataSize(DataUnit.B).longValue() : null); if (rangeLength != null) { request.setRange(rangeStart, rangeStart + rangeLength - 1); } I think the

Re: FetchS3Object fails to fetch small files from S3

2021-08-17 Thread Joey Frazee
Arpad, the validator on range length should prevent that from happening if there’s no EL. Not set or >= 1 should be the only allowed values. Vibhath, does this happen with small, non-zero length files? FYI, opened https://issues.apache.org/jira/browse/NIFI-9055 -joey > On Aug 17, 2021, at

Re: FetchS3Object fails to fetch small files from S3

2021-08-17 Thread Vibhath Ileperuma
Hi Joey, It happens only for 0B files. Sorry for the confusion caused. Vibhath.

Re: Disfunctional cluster with version 1.13.2

2021-08-17 Thread Mark Payne
Axel, So a few things to consider here. The NullPointer that you’re showing there in the logs was fixed in 1.14.0 [1]. Secondly, is a question of why the ZooKeeper client is disconnecting. Given that you’re running an embedded zookeeper server, it could be for any number of reasons - low java

Re: Disfunctional cluster with version 1.13.2

2021-08-17 Thread Ryan Hendrickson
Hi Mark, We just submitted this one: https://issues.apache.org/jira/browse/NIFI-9056 Another one will be on the way today/tomorrow for Relationship Load Balancing issues. Ryan On Tue, Aug 17, 2021 at 10:17 AM Mark Payne wrote: > Ryan, > > Have you filed Jiras for these? If so, can you

Re: FetchS3Object fails to fetch small files from S3

2021-08-17 Thread Joey Frazee
I was able to replicate this on Minio too and have a fix for it. The change is trivial but it needs more testing. -joey > On Aug 17, 2021, at 8:22 AM, Vibhath Ileperuma > wrote: > >  > Hi Joey, > > It happens only for 0B files. > Sorry for the confusion caused. > > Vibhath.

RE: Power BI Real Time REST API failure

2021-08-17 Thread Hendrik Ruijter
Sorry, setting the property "Ignore response's content" to true, as described in the commit, is test OK. Works as expected now! From: Hendrik Ruijter Sent: den 16 augusti 2021 14:56 To: users@nifi.apache.org Subject: Power BI Real Time REST API failure Hello, I use docker apache/nifi 1.13.2

Persistence in DistributedMapCacheServer

2021-08-17 Thread Ravneet Cheema
Hello , I am using DistributedMapCacheServer with setting of Persistent Directory. I can see contents of cache in the directory, but the contents do not survive nifi restart . Older values from cache should be available after restart, right ? I have attached template where I have configured