No Load Balancing since 1.13.2

[Axel Schwarz]

Dear Community,

we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7 and experiencing
problems with load balancing since version 1.13.2.

I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta say, that this
is not our problem. Mainly because the balance port never binds to localhost, but also because I
implemented all workarounds under version 1.13.2 and even tried version 1.14.0 by now,
but load blancing still does not work.
What we experience is best described as "the primary node balances with itself"...

So what it does is, opening the balancing connections to its own IP instead of the IPs
of the other two nodes. And the other two nodes don't open balancing connections at all.

When executing "ss | grep 6342" on the primary node, this is what it looks like:

[root@nifiHost1 conf]# ss | grep 6342
tcp    ESTAB      0      0      192.168.1.10:51380                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:51376                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:51378                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:51370                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:51372                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51376                
tcp    ESTAB      0      0      192.168.1.10:51374                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51374                
tcp    ESTAB      0      0      192.168.1.10:51366                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51370                
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51366                
tcp    ESTAB      0      0      192.168.1.10:51368                192.168.1.10:6342                 
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51372                
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51378                
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51368                
tcp    ESTAB      0      0      192.168.1.10:6342                 192.168.1.10:51380

Executing it on the other non primary nodes, just returns absolutely nothing.

Netstat show the following on each server:

[root@nifiHost1 conf]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.1.10:6342          0.0.0.0:*               LISTEN      10352/java

[root@nifiHost2 conf]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.1.11:6342          0.0.0.0:*               LISTEN      31562/java

[root@nifiHost3 conf]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.1.12:6342          0.0.0.0:*               LISTEN      31685/java

And here is what our load balancing properties look like:

# cluster load balancing properties #
nifi.cluster.load.balance.host=nifiHost1.contoso.com
nifi.cluster.load.balance.address=0.0.0.0
nifi.cluster.load.balance.port=6342
nifi.cluster.load.balance.connections.per.node=4
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.comms.timeout=30 sec

When running Nifi in version 1.12.1 on the exact same setup in the exact same environment, load balancing is working absolutely fine.
There was a time when load balancing even worked in version 1.13.2. But I'm not able to reproduce this and it just stopped
working one day after some restart, without changing any property or whatsoever.

If any more information would be helpful please let me know and I'll try to provide it as fast as possible.

---

Versendet mit Emailn.de - Freemail

* Unbegrenzt Speicherplatz
* Eigenes Online-Büro
* 24h besten Mailempfang
* Spamschutz, Adressbuch