Hi Jeremy, It seems very weird that you get 200M flow files in a relationship that should have backpressure set at 20k flow files. While backpressure is not a hard limit you should not get to such numbers. Can you give us more details? What version of NiFi are you using? What's the configuration of your relationship between your two process groups?
Thanks, Pierre Le ven. 30 août 2019 à 07:46, Jeremy Pemberton-Pigott <[email protected]> a écrit : > Hi, > > I have a 3 node Nifi 1.6.0 cluster. It ran out of disk space when there > was a log jam of flow files (from slow HBase lookups). My queue is > configured for 20,000 but 1 node has over 206 million flow files stuck in > the queue. I managed to clear up some disk space to get things going again > but it seems that after a few mins of processing all the processors in the > Log Parser process group will stop processing and show zero in/out. > > Is this a bug fixed in a later version? > > Each time I have to tear down the Docker containers running Nifi and > restart it to process a few 10,000s and repeat every few mins. Any idea > what I should do to keep it processing the data (nifi-app.log doesn't show > my anything unusual about the stop or delay) until the 1 node can clear the > backlog? > > [image: image.png] > > Regards, > > Jeremy >
