Joe G I do recall there were some fixes and improvements related to clustering performance/thread pooling/ as it relates to site to site. I dont recall precisely which version they went into but i'd strongly recommend trying the latest release if you're able.
Thanks On Fri, Aug 10, 2018 at 4:13 PM Martijn Dekkers <[email protected]> wrote: > > Whats the OS you are running on? What kind of systems? Memory stats, network > stats, JVM stats etc. How much data coming through? > > On 10 August 2018 at 16:06, Joe Gresock <[email protected]> wrote: >> >> Any nifi developers on this list that have any suggestions? >> >> On Wed, Aug 8, 2018 at 7:38 AM Joe Gresock <[email protected]> wrote: >>> >>> I am running a 7-node NiFi 1.6.0 cluster that performs fairly well when >>> it's simply processing its own data (putting records in Elasticsearch, >>> MongoDB, running transforms, etc.). However, when we add receiving >>> Site-to-Site traffic to the mix, the CPU spikes to the point that the nodes >>> can't talk to each other, resulting in the inability to view or modify the >>> flow in the console. >>> >>> I have tried some basic things to mitigate this: >>> - Requested that the sending party use a comma-separated list of all 7 of >>> our nodes in their Remote Process Group that points to our cluster, in >>> hopes that that will help balance the requests >>> - Requested that the sending party use some of the batching settings on the >>> Remote Port (i.e., Count = 20, Size = 100 MB, Duration = 10 sec) >>> - Reduced the thread count on our Input Port to 2 >>> >>> Are there any known nifi.properties that can be set to help mitigate this >>> problem? Again, it only seems to be a problem when we are both receiving >>> site-to-site traffic and doing our normal processing, but taking each of >>> those activities in isolation seems to be okay. >>> >>> Thanks, >>> Joe >> >> >> >> -- >> I know what it is to be in need, and I know what it is to have plenty. I >> have learned the secret of being content in any and every situation, whether >> well fed or hungry, whether living in plenty or in want. I can do all this >> through him who gives me strength. -Philippians 4:12-13 > >
