Attached to this mail is a tar.gz with instructions to reproduce. It contains 3 text files with commands and comments. Be sure to check the actual commands before executing. This was tested on a Ubuntu 18.04 VM, with docker installed on it.
The order of execution is: - zookeeper instructions.txt - solr instructions.txt - after setup.txt The behaviour was reproduced several times and it was found that it was the solr/java process calling for the deletion of the files and directories. The basic steps are: set up zookeeper, set up solr root, set up solr. Create dummy collection with example data. Stop the containers. Delete the zookeeper 'version-2' folder. Recreate zookeeper container. Redo the mkroot, recreate solr container. At this point, solr will start complaining about the cores after a bit and then the data folders will be deleted. Hope this is clear and complete. Regard, Koen On Wed, Apr 10, 2019 at 3:20 PM Gus Heck <gus.h...@gmail.com> wrote: > Deleting data on a zookeeper hiccup does sound bad if it's really solr's > fault. Can you work up a set of steps to reproduce? Something like install > solr, index tech products example, shut down solr, perform some editing to > zk, start solr, observe data gone (but with lots of details about exact > configurations/commands/edits etc)? > > "some sort of split brain" is nebulous and nobody will know if they've > solved your problem unless that can be quantified and the problem > replicated. > > -Gus > > On Tue, Apr 9, 2019 at 1:37 PM Koen De Groote <koen.degro...@limecraft.com > > > wrote: > > > Hello, > > > > I recently ran in to the following scenario: > > > > Solr, version 7.5, in a docker container, running as cloud, with an > > external zookeeper ensemble of 3 zookeepers. Instructions were followed > to > > make a root first, this was set correctly, as could be seen by the solr > > logs outputting the connect info. > > > > root command is: "bin/solr zk mkroot /solr -z <addrress>" > > > > For a yet undetermined reason, the zookeeper ensemble had some kind of > > split-brain occur. At a later point, Solr was restarted and then suddenly > > all its directories were gone. > > > > By which I mean: the directories containing the configuration and the > data. > > The stopwords, the schema, the solr config, the "shard1_replica_n2" > > directories, those directories. > > > > Those were gone without a trace. > > > > As far as I can tell, solr started, asked zookeeper for its config, > > zookeeper returned an empty config and consequently "made it so". > > > > I am by no means very knowledgeable about solr internals. Can anyone > chime > > in as to what happened here and how to prevent it? Is more info needed? > > > > Ideally, if something like this were to happen, I'd like for either solr > to > > not delete folders or if that's not possible, add some kind of > pre-startup > > check that stops solr from going any further if things go wrong. > > > > Regards, > > Koen > > > > > -- > http://www.the111shift.com >
instructions.tar.gz
Description: application/gzip
