Hi Mark Thanks for the reply. The “funny” part is that it worked before and I haven’t made any change to the configuration. And I have deleted the state folder at all nodes and it doesn’t work.
It is the same configuration as 3 other clusters except hostname/up adresses and certificates off course. All clusters are running 1.16.1. At the 3 other clusters don’t have any issues (yet). It all started after node 1 went down do to memory problem. After that they just don’t like me anymore 🤔 Tomorrow I will give it a chance and if it doesn’t work I will reinstall the cluster from scratch. I would like to be able to fix it. If the same situations happened with one of the productions clusters. Hopefully all flows are in the registry. Kind regards Jens > Den 19. maj 2022 kl. 15.21 skrev Mark Payne <[email protected]>: > > Jens, > > So that would tell us that the hostname or the port is wrong, or that NiFi is > not running/listening for load balanced connections. I would recommend the > following: > > - Check nifi.properties on all nodes to make sure that the > nifi.cluster.load.balance.host property is set to the node’s hostname. > - You could try deleting the state/ directory on the nodes and restarting. > Perhaps somehow it’s got an old value cached and is trying to communicate > with the wrong port? > > Thanks > -Mark > > >> On May 19, 2022, at 7:42 AM, Jens M. Kofoed <[email protected]> wrote: >> >> Hi >> >> I have a 3 node test cluster running v. 1.16.1. Which has been working fine, >> with no errors. But i doesn't do much, since it is my test cluster. >> But now I am struggling with load balance connection refuse between nodes. >> Both node 2 and 3 are refusing load balancing connections, even after >> stopping the cluster. I have deleted all files at node 2 and 3, but they >> still refuse connection. >> I have not made any change to the configuration, but I have created a new >> flow which got node 1 to die. >> The new flow is using an ExecuteStreamCommand using stdin/stdout to a shell >> command to manipulate with some data (1 GB files). I set the "Concurrent >> Tasks" to high so node 1 ran out of memory and stopped. It continues to stop >> if I tried to start it again. So I deleted the flow.gz file and the run, >> state and work folders and started the node. Now node 1 was running again. >> But after this node 2 and 3 are refusing load balancing connections. >> I can't see what this flow have to do with this, but I have now tried to >> remove all files at node 2 and 3 to get clean nodes. But they still doesn't >> work >> >> Does any one have an idea how to debug further? >> >> From the log: >> 2022-05-19 13:22:46,268 ERROR [Load-Balanced Client Thread-2] >> o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to >> nifi-n03:8443 for load balancing >> java.net.ConnectException: Connection refused >> at sun.nio.ch.Net.connect0(Native Method) >> at sun.nio.ch.Net.connect(Net.java:482) >> at sun.nio.ch.Net.connect(Net.java:474) >> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647) >> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:107) >> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) >> at >> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.createChannel(NioAsyncLoadBalanceClient.java:497) >> at >> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.establishConnection(NioAsyncLoadBalanceClient.java:440) >> at >> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:234) >> at >> org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) >> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> >> kind regards >> Jens >
