Hi all, I have a question that some of you must have tackled already. On a NiFi cluster (still 1.9 at the moment) that is normally very stable, the users sometimes trigger peak loads that cause disconnections or other issues either in NiFi itself or the external zookeeper cluster. The clear example I found is a ListHDFS processor that maintains large amount of state (many millions of files) being cleared and refilled, but I suspect it may just keep adding more and more to the state.
So far, we’ve increased Zookeeper initLimit and SyncLimit and did some NiFi timeout tuning, but it’s hard to figure out a sensible value when the reported times are normally nowhere near the limits. The users also keep finding bigger data loads which they repeatedly process through NiFi into some processing applications. Deleting the files is also not an option because of the re-processing It seems to me that increasing timeouts from several seconds to what’s going to be minutes must impact some other aspect of Zookeeper. Is there a flow design or tuning strategy that avoids large changes to the state in a short time like this? Are Zookeeper timeouts of 60+ secs actually usual? Met vriendelijke groet, Isha Lamboo Data Engineer +31 (0)6 20 50 15 91 [cid:[email protected]] [email protected]<mailto:[email protected]> Edisonbaan 15 3439 MN Nieuwegein www.virtualsciences.nl<http://www.virtualsciences.nl/> www.conclusion.nl<http://www.conclusion.nl/> Bekijk hier de algemene voorwaarden van Conclusion<http://www.conclusion.nl/kleine-lettertjes/algemene-voorwaarden> [conclusion_logo]
