Deleting data on a zookeeper hiccup does sound bad if it's really solr's fault. Can you work up a set of steps to reproduce? Something like install solr, index tech products example, shut down solr, perform some editing to zk, start solr, observe data gone (but with lots of details about exact configurations/commands/edits etc)?
"some sort of split brain" is nebulous and nobody will know if they've solved your problem unless that can be quantified and the problem replicated. -Gus On Tue, Apr 9, 2019 at 1:37 PM Koen De Groote <koen.degro...@limecraft.com> wrote: > Hello, > > I recently ran in to the following scenario: > > Solr, version 7.5, in a docker container, running as cloud, with an > external zookeeper ensemble of 3 zookeepers. Instructions were followed to > make a root first, this was set correctly, as could be seen by the solr > logs outputting the connect info. > > root command is: "bin/solr zk mkroot /solr -z <addrress>" > > For a yet undetermined reason, the zookeeper ensemble had some kind of > split-brain occur. At a later point, Solr was restarted and then suddenly > all its directories were gone. > > By which I mean: the directories containing the configuration and the data. > The stopwords, the schema, the solr config, the "shard1_replica_n2" > directories, those directories. > > Those were gone without a trace. > > As far as I can tell, solr started, asked zookeeper for its config, > zookeeper returned an empty config and consequently "made it so". > > I am by no means very knowledgeable about solr internals. Can anyone chime > in as to what happened here and how to prevent it? Is more info needed? > > Ideally, if something like this were to happen, I'd like for either solr to > not delete folders or if that's not possible, add some kind of pre-startup > check that stops solr from going any further if things go wrong. > > Regards, > Koen > -- http://www.the111shift.com
