Hi all, we have recently tried to update zookeeper from 3.4.13 to 3.6.2 on our production storm cluster without success. Therefore, we are now tinkering with a test cluster, including just two nodes.
The two nodes have the following setup: *OS* Ubuntu 20.04.1 LTS *Zookeeper *3.6.2 *Storm *2.2.0 *Java *OpenJDK Runtime Environment (build 14.0.2+12-Ubuntu-120.04) *Python* 2.7.18 *Python2* 2.7.18 *Python3* 3.8.5 We are following this guide *to the letter*: http://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html First problem here: when following the guide and setting the mandatory parameters, you will end up with a configuration where the UI does not work. This is because according to the defaults.yaml, *both ui.port and storm.exhibitor.port are set to 8080*. This is where I get my information in regards to the default values from: https://github.com/apache/storm/blob/master/conf/defaults.yaml Left unconfigured, the UI will not start because it cannot bind to port 8080. Instead, you will see the 404 page from jetty. It took me quite a while to figure this, so I would be glad the person in charge could amend the docs to reflect the necessity to set one of the two to something different. Next problem, which I am not able to solve, but from searching the web I understand many storm users are facing similar problems, is the dreaded *NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts* Some examples: https://stackoverflow.com/questions/36742451/apache-storm-could-not-find-leader-nimbus-from-seed-hosts https://stackoverflow.com/questions/58155057/how-to-fix-could-not-find-leader-nimbus-from-seed-hosts-192-168-23-165 https://stackoverflow.com/questions/37480155/nimbusleadernotfoundexception-in-apache-storm I have double checked the ports are accessible on both nodes and that everything else is running as expected. Looking at the outputs from nimbus and supervisor, there is no error message (stderr is empty) and there is no useful logging in stdout, only the java command which starts the two processes. Upon accessing the UI in the browser, the process will load for a while, then, both nimbus and supervisor will crash/be killed/? and the error message appears. I have run the processes in supervision with supervisord and also have tried to start them via bash, to get more meaningful logging, no luck here. It is almost impossible to debug without more meaningful logging. My best guess is it has to do with that we Maybe I am missing something in the host configurations, but I am at a loss here so I would appreciate any pointers to where to look next. Attaching storm.yaml and hosts configurations below. Regards Jonas *storm.yaml* storm.zookeeper.servers: - "stormtest1.example.com" - "stormtest2.example.com" nimbus.seeds: ["stormtest1.example.com", "stormtest2.example.com"] storm.local.dir: "/storm" storm.zookeeper.root: "/zookeeper" supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703 ui.port: 8888 */etc/hosts (node 1)* 127.0.1.1 storm-1 127.0.0.1 localhost # Test Storm xxx.xxx.xxx.xxx stormtest1.example.com storm-test-1 xxx.xxx.xxx.xxx stormtest2.example.com storm-test-2 */etc/hosts (node 2)* 127.0.1.1 storm-2 127.0.0.1 localhost # Test Storm xxx.xxx.xxx.xxx stormtest1.example.com storm-test-1 xxx.xxx.xxx.xxx stormtest2.example.com storm-test-2
