What's most disturbing here is that the zookeeper info disappeared for certain shards. This simply should not be happening just because you lost a server or two. The info should still be in Zookeeper for the shards in question. All the replicas will be down of course, but they'll still be there.
So to be clear we're talking about the same thing, your Zookeeper has a collections>>my_collection>>state.json ZNode. In that state.json you have information for all the shards, and you're saying that you had only 16 when there used to be 20, right? (note older Solr versions would have it all in /clusterstate.json). If that's the case I'd be really curious how that happened, because there's no reason Solr should have done that. Best, Erick On Fri, Jun 10, 2016 at 12:35 PM, Henrik Brautaset Aronsen <hen...@synth.no> wrote: > On Fri, Jun 10, 2016 at 6:18 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > >> Well, how brave do you want to be ;)? > > > Hi Erick, thanks for your reply! > > >> There's no great magic to the >> Zookeeper nodes here. If you do everything just right you could create >> one manually. By that I mean you could "hand edit" the znode with the >> Zookeeper commands, you'd have to dig for the exact commands.... > > > So, if I create (or edit) the correct entries in ZK, Solr should just pick > that up and behave accordingly? I thought I had to do this through the > Solr API. I think I'll experiment some more with this. > > >> You _may_ be able to use the ADDREPLICA command, assuming that the shard >> information is still in the ZK node. I haven't tried this however. >> > > The shard information is gone from zookeeper (I guess that's what you mean > by ZK node?), and I can't specify hash ranges through the ADDREPLICA > command. > > >> All that said, if the node is somehow permanently gone, you have to >> re-index anyway to get the data back so recreating the collection >> would be less fooling around. >> > > I'm not really interested in the data, since all data in the collection has > a TTL of 30 minutes. > > I ended up re-creating the collection even though it gave me a couple of > minutes downtime. If this happens again, it would be awesome if I could > manually create the shards with the specified hash ranges. > > Cheers, > Henrik
