Ok, so we played with these parameters again and got different results almost each time. Once we actually had a functioning balancing cluster, but just once. This was with Version 1.14.0 and .host set to 0.0.0.0. In this state NifiHost1 (Primary Node) had 8 connecctions established. 4 to each other host. And the other hosts each had 4 connections to NifiHost1. Right after a restart it stopped working again and the primary node only had connections to itself.
The third result we saw was, that NifiHost1 (Primary Node) had 4 connections open to itself, NifiHost2 had 4 connections open to NifiHost1 and NifiHost3 had no connection at all. The result in the flow then was, that the cluster load balanced between NifiHost1 and NifiHost2, but not like in the right order (1. NifiHost1, 2. NifiHost2, 3. NifiHost3 and so on) but with every third flowfile to NifiHost2 and the rest to NifiHost1 (so it load balanced with itself indeed) We can reproduce these outcomes randomly with Version 1.13.2 AND 1.14.0. It's really hard to obtain conclusions out of that behaviour and I get a little desperate by now :( --- Ursprüngliche Nachricht --- Von: "Axel Schwarz" <axelkop...@emailn.de> Datum: 28.07.2021 13:10:11 An: users@nifi.apache.org Betreff: Re: Re: Re: No Load Balancing since 1.13.2 > I've tried every single combination of values for > nifi.cluster.load.balance.host > and nifi.cluster.load.balance.address. > But again, the problem described in Nifi-8643 is not what happens to us. > The port always binds to the correct IP and NOT to localhost. > As seen in the netstat-excerpts from my original post. Instead the primary > node actually opens the desired count of connections for load balancing, > but ALL to itself... > > --- Ursprüngliche Nachricht --- > Von: Joe Gresock <jgres...@gmail.com> > Datum: 28.07.2021 12:09:42 > An: users@nifi.apache.org > Betreff: Re: Re: No Load Balancing since 1.13.2 > > > Also, make sure to use nifi.cluster.load.balance.host instead of .address > > > > > if you're still running 1.14.0. > > > > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> > > > > > Virus-free. > > www.avast.com > > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > > > On Wed, Jul 28, 2021 at 6:04 AM Jens M. Kofoed <jmkofoed....@gmail.com> > > > > > wrote: > > > > > hi > > > > > > I can see that you have configured > > > nifi.cluster.load.balance.address=0.0.0.0 > > > Have your tried to set the correct ip adress? > > > node1: nifi.cluster.load.balance.address=192.168.1.10 > > > node2: nifi.cluster.load.balance.address=192.168.1.11 > > > node3: nifi.cluster.load.balance.address=192.168.1.12 > > > > > > regards > > > Jens M. Kofoed > > > > > > Den ons. 28. jul. 2021 kl. 11.17 skrev Axel Schwarz <axelkop...@emailn.de > > > > > > >: > > > > > >> Just tried Java 11. But still does not work. Nothing changed. > :( > > > > >> > > >> --- Ursprüngliche Nachricht --- > > >> Von: Jorge Machado <jom...@me.com> > > >> Datum: 27.07.2021 13:08:55 > > >> An: users@nifi.apache.org, Axel Schwarz <axelkop...@emailn.de> > > > > > >> Betreff: Re: No Load Balancing since 1.13.2 > > >> > > >> > Did you tried java 11 ? I have a client running a similar > setup > > to yours > > >> > but with a lower nigh version and it works fine. Maybe > it is > > worth to > > >> try > > >> > it. > > >> > > > >> > > > >> > > On 27. Jul 2021, at 12:42, Axel Schwarz <axelkop...@emailn.de> > > > > > >> > wrote: > > >> > > > > >> > > I did indeed, but I updated from u161 to u291, as > this > > was the newest > > >> > version at that time, because I thought it could help. > > > >> > > So the issue started under u161. But I just saw that > u301 > > is out. I > > >> > will try this as well. > > >> > > --- Ursprüngliche Nachricht --- > > >> > > Von: Pierre Villard <pierre.villard...@gmail.com> > > > > > >> > > Datum: 27.07.2021 10:18:38 > > >> > > An: users@nifi.apache.org, Axel Schwarz <axelkop...@emailn.de> > > > > > >> > > > >> > > Betreff: Re: No Load Balancing since 1.13.2 > > >> > > > > >> > > Hi, > > >> > > > > >> > > I believe the minor u291 is known to have issues > (for > > some of its > > >> early > > >> > builds). Did you upgrade the Java version recently? > > >> > > > > >> > > Thanks, > > >> > > Pierre > > >> > > > > >> > > Le mar. 27 juil. 2021 à 08:07, Axel Schwarz <axelkop...@emailn.de > > > > > >> > <mailto:axelkop...@emailn.de>> a écrit : > > >> > > Dear Community, > > >> > > > > >> > > we're running a secured 3 node Nifi Cluster on Java > 8_u291 > > and Debian > > >> > 7 and experiencing > > >> > > problems with load balancing since version 1.13.2. > > > >> > > > > >> > > I'm fully aware of Issue Nifi-8643 and tested alot > around > > this, but > > >> > gotta say, that this > > >> > > is not our problem. Mainly because the balance port > never > > binds to > > >> localhost, > > >> > but also because I > > >> > > implemented all workarounds under version 1.13.2 > and even > > tried > > >> version > > >> > 1.14.0 by now, > > >> > > but load blancing still does not work. > > >> > > What we experience is best described as "the > primary > > node balances > > >> > with itself"... > > >> > > > > >> > > So what it does is, opening the balancing connections > > > to its own IP > > >> > instead of the IPs > > >> > > of the other two nodes. And the other two nodes don't > > > open balancing > > >> > connections at all. > > >> > > > > >> > > When executing "ss | grep 6342" on the > primary > > node, this > > >> > is what it looks like: > > >> > > > > >> > > [root@nifiHost1 conf]# ss | grep 6342 > > >> > > tcp ESTAB 0 0 192.168.1.10:51380 > < > > > > >> http://192.168.1.10:51380/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:51376 > < > > > > >> http://192.168.1.10:51376/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:51378 > < > > > > >> http://192.168.1.10:51378/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:51370 > < > > > > >> http://192.168.1.10:51370/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:51372 > < > > > > >> http://192.168.1.10:51372/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51376 <http://192.168.1.10:51376/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:51374 > < > > > > >> http://192.168.1.10:51374/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51374 <http://192.168.1.10:51374/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:51366 > < > > > > >> http://192.168.1.10:51366/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51370 <http://192.168.1.10:51370/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51366 <http://192.168.1.10:51366/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:51368 > < > > > > >> http://192.168.1.10:51368/> > > >> > 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51372 <http://192.168.1.10:51372/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51378 <http://192.168.1.10:51378/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51368 <http://192.168.1.10:51368/> > > > > > >> > > > >> > > tcp ESTAB 0 0 192.168.1.10:6342 > < > > > > >> http://192.168.1.10:6342/> > > >> > 192.168.1.10:51380 <http://192.168.1.10:51380/> > > > > > >> > > > > >> > > Executing it on the other non primary nodes, just > returns > > absolutely > > >> > nothing. > > >> > > > > >> > > Netstat show the following on each server: > > >> > > > > >> > > [root@nifiHost1 conf]# netstat -tulpn > > >> > > Active Internet connections (only servers) > > >> > > Proto Recv-Q Send-Q Local Address Foreign > Address > > > > >> > State PID/Program name > > >> > > tcp 0 0 192.168.1.10:6342 <http://192.168.1.10:6342/> > > > > > >> > 0.0.0.0:* LISTEN 10352/java > > > >> > > > > >> > > [root@nifiHost2 conf]# netstat -tulpn > > >> > > Active Internet connections (only servers) > > >> > > Proto Recv-Q Send-Q Local Address Foreign > Address > > > > >> > State PID/Program name > > >> > > tcp 0 0 192.168.1.11:6342 <http://192.168.1.11:6342/> > > > > > >> > 0.0.0.0:* LISTEN 31562/java > > > >> > > > > >> > > [root@nifiHost3 conf]# netstat -tulpn > > >> > > Active Internet connections (only servers) > > >> > > Proto Recv-Q Send-Q Local Address Foreign > Address > > > > >> > State PID/Program name > > >> > > tcp 0 0 192.168.1.12:6342 <http://192.168.1.12:6342/> > > > > > >> > 0.0.0.0:* LISTEN 31685/java > > > >> > > > > >> > > And here is what our load balancing properties look > like: > > > > >> > > > > >> > > # cluster load balancing properties # > > >> > > nifi.cluster.load.balance.host=nifiHost1.contoso.com > < > > > > >> http://nifihost1.contoso.com/> > > >> > > > >> > > nifi.cluster.load.balance.address=0.0.0.0 > > >> > > nifi.cluster.load.balance.port=6342 > > >> > > nifi.cluster.load.balance.connections.per.node=4 > > > >> > > nifi.cluster.load.balance.max.thread.count=8 > > >> > > nifi.cluster.load.balance.comms.timeout=30 sec > > >> > > > > >> > > When running Nifi in version 1.12.1 on the exact > same > > setup in the > > >> exact > > >> > same environment, load balancing is working absolutely > fine. > > > > >> > > There was a time when load balancing even worked > in version > > 1.13.2. > > >> > But I'm not able to reproduce this and it just stopped > > > >> > > working one day after some restart, without changing > any > > property or > > >> > whatsoever. > > >> > > > > >> > > If any more information would be helpful please let > me > > know and I'll > > >> > try to provide it as fast as possible. > > >> > > > > >> > > > > >> > > > > >> > > Versendet mit Emailn.de <https://www.emailn.de/> > > > - Freemail > > >> > > > > >> > > * Unbegrenzt Speicherplatz > > >> > > * Eigenes Online-Büro > > >> > > * 24h besten Mailempfang > > >> > > * Spamschutz, Adressbuch > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > Versendet mit Emailn.de <https://www.emailn.de/> > > > - Freemail > > >> > > > >> > > > > >> > > * Unbegrenzt Speicherplatz > > >> > > * Eigenes Online-Büro > > >> > > * 24h besten Mailempfang > > >> > > * Spamschutz, Adressbuch > > >> > > > > >> > > > >> > > > >> > > >> > > >> > > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> > > > > > Virus-free. > > www.avast.com > > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > > > >
