Re: So Many Zookeeper Warnings--There Must Be a Problem

2019-01-04 Thread Erick Erickson
How brave are you? ;) I'll defer to Scott on the internals of ZK and why it might be necessary to delete the ZK data dirs, but what happens if you just correct your configuration and drive on? If that doesn't work here's something to try Shut down your Solr instances, then. - bin/solr zk

Re: So Many Zookeeper Warnings--There Must Be a Problem

2019-01-04 Thread Shawn Heisey
On 1/4/2019 5:24 AM, Joe Lerner wrote: server #1 = myid#1 server #2 = myid#2 server #3 = myid#2 My plan would be to do the following, while users are still online (it's a big [bad] deal if we need to take search offline): 1. Take zk #3 down. 2. Fix zk #3 by deleting the contents of the zk data

Re: So Many Zookeeper Warnings--There Must Be a Problem

2019-01-04 Thread Joe Lerner
wrt, "You'll probably have to delete the contents of the zk data directory and rebuild your collections." Rebuild my *SOLR* collections? That's easy enough for us. If this is how we're incorrectly configured now: server #1 = myid#1 server #2 = myid#2 server #3 = myid#2 My plan would be to do t

Re: So Many Zookeeper Warnings--There Must Be a Problem

2019-01-03 Thread Scott Stults
Good! Hopefully that's your smoking gun. The port settings are fine, but since you're deploying to separate servers you don't need different ports in the "server.x=" section. This section of the docs explains it better: http://zookeeper.apache.org/doc/r3.4.7/zookeeperAdmin.html#sc_zkMulitServerSe

Re: So Many Zookeeper Warnings--There Must Be a Problem

2019-01-03 Thread Joe Lerner
Hi Scott, First, we are definitely mis-onfigured for the myid thing. Basically two of them were identifying as ID #2, and they are the two ZK's claiming to be the leader. Definitely something to straighten out! Our 3 lines in zoo.cfg look correct. Except they look like this: clientPort:2181 ser

Re: So Many Zookeeper Warnings--There Must Be a Problem

2019-01-03 Thread Scott Stults
Hi Joe, Yeah, two leaders is definitely a problem. I'd fix that before wading through the error logs. Check out zoo.cfg on each server. You should have three lines at the end similar to this: server.1=host1:2181:2281 server.2=host2:2182:2282 server.3=host3:2183:2283 (substitute "host*" with the

So Many Zookeeper Warnings--There Must Be a Problem

2019-01-03 Thread Joe Lerner
Hi, We have a simple architecture: 2 SOLR Cloud servers (on servers #1 and #2), and 3 zookeeper instances (on servers #1, #2, and #3). Things work fine (although we had a couple of brief unexplained outages), but: One worrisome thing is that when I status zookeeper on #1 and #2, I get Mode=Leader