Thank you Joe. We do one day intend to upgrade but are bound by enterprise options available to us to 1.7 for the near-term. So, do I understand your explanation correctly: behavior exhibited through the first 4000 flowfiles as past performance may not represent future results. It will do what it does, and I may find that node1 does get loaded as I work through flowfiles in steady state. Again, thanks.
On Wed, Jul 10, 2019 at 8:41 AM Joe Witt <[email protected]> wrote: > James > > For distributing work across the cluster the load balanced connection > capability in NiFi 1.8 and beyond is the right answer - purpose built for > the job. I'd strongly recommend upgrading to avoid use of s2s for this > scenario and instead use load balanced connections. When using load > balanced or s2s you will want to observe the behavior at typical sustained > scale. Using either strategy all nodes will have an opportunity to receive > data. However, backpressure/loading, configuration, and other factors > could mean that periodically a given node is not receiving data. We have > seen a lot of folks become confused about s2s behaviors for cases like this > so I think you'll find load balanced connections are much better for this. > > Thanks > Joe > > > On Wed, Jul 10, 2019 at 8:34 AM James McMahon <[email protected]> > wrote: > >> We are on 1.7.1.g and have just recently established our first clustered >> configuration. Using Pierre Villard's article from Feb 2017 ( >> https://pierrevillard.com/2017/02/23/listfetch-pattern-and-remote-process-group-in-apache-nifi/ >> ) and a few other related technical articles to flesh out some details, we >> have gotten a ListFile / FetchFile to distribute load using Remote Process >> Group - almost. >> >> Downstream of the FetchFile running on all nodes I connect to a Monitor >> Activity processor simply to examine the flowfiles that result from the >> fetch, in that following queue. In that queue one can look at the info for >> each flowfile and find what appears to be the node on which the flowfile >> was processed by field Node Address. >> >> I have four nodes in my cluster - one primary, three not primary. I can >> see in the queue listing that three flowfiles share common Position values. >> Three have Position 1, three have Position 2, etc etc etc in a pattern that >> repeats throughout the entire queue. Within each Position group, the >> flowfiles have been distributed to node2, node3, and node4 - *but none >> at all to node1*. >> >> What would cause such behavior? How can I get my files to distribute >> across all four nodes? >> I should mention: >> 1. all four node URLS are in the RPG URL configuration parameter, >> delimited by commas. >> 2. node1 is currently assigned by my external Zookeeper as my Primary, >> and is where the ListFile processor executes. >> 3. all four nodes are granted access for "retrieve site-to-site details" >> in my Hamburger Menu, Access Policies. >> 4. all four nodes are granted access for "receive data via site-to-site" >> in the Access Policies for the RPG Input Port. >> >> My concern is that I am leaving nearly 25% of my available cluster >> capacity unused. >> >
