Hey Jens, in Issue Nifi-8643 you wrote the last comment with the exactly same behaviour as we're experiencing now. 2 of 3 nodes were load balancing. How did you get the third node to participate in load balancing? An update to 1.14.0 does not change anything for us.
https://issues.apache.org/jira/browse/NIFI-8643?focusedCommentId=17361418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17361418 --- Ursprüngliche Nachricht --- Von: "Jens M. Kofoed" <[email protected]> Datum: 28.07.2021 12:07:50 An: [email protected], Axel Schwarz <[email protected]> Betreff: Re: Re: No Load Balancing since 1.13.2 > hi > > I can see that you have configured nifi.cluster.load.balance.address=0.0.0.0 > > Have your tried to set the correct ip adress? > node1: nifi.cluster.load.balance.address=192.168.1.10 > node2: nifi.cluster.load.balance.address=192.168.1.11 > node3: nifi.cluster.load.balance.address=192.168.1.12 > > regards > Jens M. Kofoed > > Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <[email protected]>: > > > > Just tried Java 11. But still does not work. Nothing changed. :( > > > > --- Ursprüngliche Nachricht --- > > Von: Jorge Machado <[email protected]> > > Datum: 27.07.2021 13:08:55 > > An: [email protected], Axel Schwarz <[email protected]> > > > Betreff: Re: No Load Balancing since 1.13.2 > > > > > Did you tried java 11 ? I have a client running a similar setup > to yours > > > but with a lower nigh version and it works fine. Maybe it is worth > to try > > > it. > > > > > > > > > > On 27. Jul 2021, at 12:42, Axel Schwarz <[email protected]> > > > > wrote: > > > > > > > > I did indeed, but I updated from u161 to u291, as this was > the newest > > > version at that time, because I thought it could help. > > > > So the issue started under u161. But I just saw that u301 > is out. I > > > will try this as well. > > > > --- Ursprüngliche Nachricht --- > > > > Von: Pierre Villard <[email protected]> > > > > Datum: 27.07.2021 10:18:38 > > > > An: [email protected], Axel Schwarz <[email protected]> > > > > > > > > Betreff: Re: No Load Balancing since 1.13.2 > > > > > > > > Hi, > > > > > > > > I believe the minor u291 is known to have issues (for some > of its early > > > builds). Did you upgrade the Java version recently? > > > > > > > > Thanks, > > > > Pierre > > > > > > > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <[email protected] > > > > <mailto:[email protected]>> a écrit : > > > > Dear Community, > > > > > > > > we're running a secured 3 node Nifi Cluster on Java 8_u291 > and Debian > > > 7 and experiencing > > > > problems with load balancing since version 1.13.2. > > > > > > > > I'm fully aware of Issue Nifi-8643 and tested alot around > this, but > > > gotta say, that this > > > > is not our problem. Mainly because the balance port never > binds to > > localhost, > > > but also because I > > > > implemented all workarounds under version 1.13.2 and even > tried version > > > 1.14.0 by now, > > > > but load blancing still does not work. > > > > What we experience is best described as "the primary > node balances > > > with itself"... > > > > > > > > So what it does is, opening the balancing connections to its > own IP > > > instead of the IPs > > > > of the other two nodes. And the other two nodes don't open > balancing > > > connections at all. > > > > > > > > When executing "ss | grep 6342" on the primary node, > this > > > is what it looks like: > > > > > > > > [root@nifiHost1 conf]# ss | grep 6342 > > > > tcp ESTAB 0 0 192.168.1.10:51380 < > > http://192.168.1.10:51380/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:51376 < > > http://192.168.1.10:51376/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:51378 < > > http://192.168.1.10:51378/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:51370 < > > http://192.168.1.10:51370/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:51372 < > > http://192.168.1.10:51372/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51376 <http://192.168.1.10:51376/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:51374 < > > http://192.168.1.10:51374/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51374 <http://192.168.1.10:51374/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:51366 < > > http://192.168.1.10:51366/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51370 <http://192.168.1.10:51370/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51366 <http://192.168.1.10:51366/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:51368 < > > http://192.168.1.10:51368/> > > > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51372 <http://192.168.1.10:51372/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51378 <http://192.168.1.10:51378/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51368 <http://192.168.1.10:51368/> > > > > > > > > tcp ESTAB 0 0 192.168.1.10:6342 < > > http://192.168.1.10:6342/> > > > 192.168.1.10:51380 <http://192.168.1.10:51380/> > > > > > > > > > Executing it on the other non primary nodes, just returns > absolutely > > > nothing. > > > > > > > > Netstat show the following on each server: > > > > > > > > [root@nifiHost1 conf]# netstat -tulpn > > > > Active Internet connections (only servers) > > > > Proto Recv-Q Send-Q Local Address Foreign Address > > > > State PID/Program name > > > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > 0.0.0.0:* LISTEN 10352/java > > > > > > > > [root@nifiHost2 conf]# netstat -tulpn > > > > Active Internet connections (only servers) > > > > Proto Recv-Q Send-Q Local Address Foreign Address > > > > State PID/Program name > > > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/> > > > > 0.0.0.0:* LISTEN 31562/java > > > > > > > > [root@nifiHost3 conf]# netstat -tulpn > > > > Active Internet connections (only servers) > > > > Proto Recv-Q Send-Q Local Address Foreign Address > > > > State PID/Program name > > > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/> > > > > 0.0.0.0:* LISTEN 31685/java > > > > > > > > And here is what our load balancing properties look like: > > > > > > > > > # cluster load balancing properties # > > > > nifi.cluster.load.balance.host=nifiHost1.contoso.com < > > > http://nifihost1.contoso.com/> > > > > > > > nifi.cluster.load.balance.address=0.0.0.0 > > > > nifi.cluster.load.balance.port=6342 > > > > nifi.cluster.load.balance.connections.per.node=4 > > > > nifi.cluster.load.balance.max.thread.count=8 > > > > nifi.cluster.load.balance.comms.timeout=30 sec > > > > > > > > When running Nifi in version 1.12.1 on the exact same setup > in the > > exact > > > same environment, load balancing is working absolutely fine. > > > > There was a time when load balancing even worked in version > 1.13.2. > > > But I'm not able to reproduce this and it just stopped > > > > working one day after some restart, without changing any property > or > > > whatsoever. > > > > > > > > If any more information would be helpful please let me know > and I'll > > > try to provide it as fast as possible. > > > > > > > > > > > > > > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail > > > > > > > > > * Unbegrenzt Speicherplatz > > > > * Eigenes Online-Büro > > > > * 24h besten Mailempfang > > > > * Spamschutz, Adressbuch > > > > > > > > > > > > > > > > > > > > Versendet mit Emailn.de <https://www.emailn.de/> - Freemail > > > > > > > > > > > > * Unbegrenzt Speicherplatz > > > > * Eigenes Online-Büro > > > > * 24h besten Mailempfang > > > > * Spamschutz, Adressbuch > > > > > > > > > > > > > > > > >
