Re: Adding replica to a shard with only down replicas

2020-02-14 Thread Erick Erickson
Glad it worked out, I like to avoid heroics whenever possible ;)… It can take quite some time for solr to finally and for good give up, waiting 10-15 minutes for something to change seems like an eternity. What’s happening here is the node attempts to recover but fails for some reason. So it

Re: Adding replica to a shard with only down replicas

2020-02-14 Thread tedsolr
Overnight the replicas with a state of "down" changed to "recovery_failed". Nothing I did. So I brought down both nodes, then started one and waited 5 min. A leader was born then I started the other node. So luckily no heroics were needed. I'll remember your advice about creating a parallel

Re: Adding replica to a shard with only down replicas

2020-02-14 Thread Erick Erickson
Yes, you can manually manipulate the data in Zookeeper, but as you say that’s a “heroic” option. But even if it’s totally messed up, you’re no worse off. You can use bin/solr zk… to copy individual znodes up and down, or there are various tools to let you do the same if you have them. It’s also

Re: Adding replica to a shard with only down replicas

2020-02-14 Thread lstusr 5u93n4
Actually I should clarify: we stop solr on one of the nodes, wait for the other node to become the leader, and then start solr back up on the one that was stopped. On Fri, 14 Feb 2020 at 09:41, lstusr 5u93n4 wrote: > We've seen this type of deadlock pretty often. Our recourse is to restart >

Re: Adding replica to a shard with only down replicas

2020-02-14 Thread lstusr 5u93n4
We've seen this type of deadlock pretty often. Our recourse is to restart solr on only one of the nodes, this seems to force the leader election to take place and it soon stars rebuilding. Let me know if you try that and it works... Wouldn't mind another validation point that this happens to

Re: Adding replica to a shard with only down replicas

2020-02-14 Thread tedsolr
Yes I did Erick, and that didn't do it. What about manual manipulation of the zookeeper data? Rather than telling the customer they need to rebuild from scratch, I'd prefer to attempt some last minute heroics. -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Adding replica to a shard with only down replicas

2020-02-13 Thread Erick Erickson
Adding a new replica won’t do you much good. Since there’s no leader, it won’t (well, shouldn’t) sync the index. Did you try the collections API FORCELEADER? It was put in as a last resort for this kind of situation. Best, Erick > On Feb 13, 2020, at 3:22 PM, tedsolr wrote: > > Solr 5.5.4. I