Hi all, I’m hoping to get some perspective from people that have NiFi with a large number of Remote Process Groups.
I’m supporting a NiFi 1.12.1 (yes, I know) cluster of 3 nodes that has about 5k processors and load-balancing still done the pre-1.8 way, with RPGs looping back to the local cluster. There are 500+ RPGs with only about 30 actually going to other NiFi clusters. We’re having several problems: * input ports getting stuck when the RPG is set to HTTP protocol and connections get killed by the Jetty DoS filter after 30 secs. The standard is RAW, but sometimes a HTTP RPG still gets deployed. * Intermittent errors like EoF, connection closed etc on HTTP connections * The cluster being unable to sync changes made to the flow resulting in disconnected nodes and sometimes uninheritable flow exceptions. My idea is that the RPGs should be replaced by load-balanced connection and/or local ports, but developer resources are scarce, so I want to either make a business case or tune NiFi performance if 500 RPGs should not cause problems normally. So is this a known issue or particular to my case? How can I identify/solve performance bottlenecks with RPGs? Kind regards, Isha Lamboo
