Somehow, my 5-node Solr cluster got into a state in which each node reported it was going to be the leader and then failed to create /overseer_elect/leader since it was already there. This generated tons of logs. It's hard to say what other detrimental effect it had; I ended up stopping the cluster (leaving zk running) then bringing the cluster back up). The problem was absent then. This is Solr 6.5.1. Maybe I should I just file a JIRA issue?
2017-05-02T15:44:02.000Z null-bop-solr-2: 2017-05-02 15:44:02.434 INFO (zkCallback-4-thread-1-processing-n:bop-solr-2:8983_solr) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader bop-solr-2:8983_solr 2017-05-02T15:44:02.000Z null-bop-solr-2: 2017-05-02 15:44:02.447 ERROR (zkCallback-4-thread-1-processing-n:bop-solr-2:8983_solr) [ ] o.a.s.c.LeaderElector node exists 2017-05-02T15:44:02.000Z null-bop-solr-2: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /overseer_elect/leader 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:526) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:523) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:466) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:453) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:741) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268) 2017-05-02T15:44:02.000Z null-bop-solr-2: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 2017-05-02T15:44:02.000Z null-bop-solr-2: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2017-05-02T15:44:02.000Z null-bop-solr-2: at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) 2017-05-02T15:44:02.000Z null-bop-solr-2: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2017-05-02T15:44:02.000Z null-bop-solr-2: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2017-05-02T15:44:02.000Z null-bop-solr-2: at java.lang.Thread.run(Thread.java:745)
