Can you show what is happening inside the first process group? Is there a SplitText processor with line count of 1?
On Fri, Aug 30, 2019 at 4:21 AM Jeremy Pemberton-Pigott < fuzzych...@gmail.com> wrote: > Hi Pierre, > > I'm using Nifi version 1.6.0. > > 04/03/2018 08:16:22 UTC > > Tagged nifi-1.6.0-RC3 > > From 7c0ee01 on branch NIFI-4995-RC3 > FlowFile expiration = 0 > Back pressure object threshold = 20000 > Back pressure data size threshold = 1GB > > The connection is just from the output port of 1 PG to the input port of > another PG. Inside the PG all the connections are using the same settings > between processors. > > Regards, > > Jeremy > > On Fri, Aug 30, 2019 at 4:14 PM Pierre Villard < > pierre.villard...@gmail.com> wrote: > >> Hi Jeremy, >> >> It seems very weird that you get 200M flow files in a relationship that >> should have backpressure set at 20k flow files. While backpressure is not a >> hard limit you should not get to such numbers. Can you give us more >> details? What version of NiFi are you using? What's the configuration of >> your relationship between your two process groups? >> >> Thanks, >> Pierre >> >> Le ven. 30 août 2019 à 07:46, Jeremy Pemberton-Pigott < >> fuzzych...@gmail.com> a écrit : >> >>> Hi, >>> >>> I have a 3 node Nifi 1.6.0 cluster. It ran out of disk space when there >>> was a log jam of flow files (from slow HBase lookups). My queue is >>> configured for 20,000 but 1 node has over 206 million flow files stuck in >>> the queue. I managed to clear up some disk space to get things going again >>> but it seems that after a few mins of processing all the processors in the >>> Log Parser process group will stop processing and show zero in/out. >>> >>> Is this a bug fixed in a later version? >>> >>> Each time I have to tear down the Docker containers running Nifi and >>> restart it to process a few 10,000s and repeat every few mins. Any idea >>> what I should do to keep it processing the data (nifi-app.log doesn't show >>> my anything unusual about the stop or delay) until the 1 node can clear the >>> backlog? >>> >>> [image: image.png] >>> >>> Regards, >>> >>> Jeremy >>> >> -- Sent from Gmail Mobile