Re: Solr cloud - poweroff procedure
“Take backups” is whatever you need for your environment. In AWS, we snapshot the EBS volumes, and so on. Backing up the Solr install and home directories would be good. There are some core.properties files in there that seem to be useful. Honestly, I don’t have a complete handle on the details of naming and properties for cores in Solr Cloud. For the Zookeeper ensemble, remember that each host has a different value in the myid file. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 31, 2018, at 5:17 AM, lstusr 5u93n4 wrote: > > Hi, > > Yes, zookeeper is external, and yes, we'll definitely wait until after solr > has stopped to bring it down. > > Thanks for the tip about disabling `autoAddReplicas`, we definitely don't > want the shards moving around during the process. > > Wunder, your point 3 mentions "take backups". Given that our data is on > hdfs (not co-located with the solr servers) and backed up separately, what > else would you recommend backing up? The contents of the `solr.home.home` > folder seem like good candidates... anything else? Let's say one of the > servers gets dropped during the move, is it sufficient to restore the > contents of `solr.home.home` onto a new server with the same > hostname/solrVersion/zookeeperConfig and bring it up in the same way as the > others? > > Thanks all, > > Kyle > > > On Wed, 31 Oct 2018 at 05:22, Shalin Shekhar Mangar > wrote: > >> In case you are using a recent Solr 7.x version with collections that have >> autoAddReplicas=true, you should disable the auto add replicas feature >> before powering off so that Solr does not decide to move replicas around >> because nodes have been lost. See >> >> https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-auto-add-replicas.html#using-cluster-property-to-enable-autoaddreplicas >> >> On Wed, Oct 31, 2018 at 3:27 AM lstusr 5u93n4 wrote: >> >>> Hi All, >>> >>> We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and >>> the data director on hdfs. It has 950 million documents in the index, >>> occupying 700GB of disk space. >>> >>> We need to completely power off the system to move it. >>> >>> Are there any actions we should take on shutdown to help the process? >>> Anyhing we should expect on power on? >>> >>> Thanks, >>> >>> Kyle >>> >> >> >> -- >> Regards, >> Shalin Shekhar Mangar. >>
Re: Solr cloud - poweroff procedure
Hi, Yes, zookeeper is external, and yes, we'll definitely wait until after solr has stopped to bring it down. Thanks for the tip about disabling `autoAddReplicas`, we definitely don't want the shards moving around during the process. Wunder, your point 3 mentions "take backups". Given that our data is on hdfs (not co-located with the solr servers) and backed up separately, what else would you recommend backing up? The contents of the `solr.home.home` folder seem like good candidates... anything else? Let's say one of the servers gets dropped during the move, is it sufficient to restore the contents of `solr.home.home` onto a new server with the same hostname/solrVersion/zookeeperConfig and bring it up in the same way as the others? Thanks all, Kyle On Wed, 31 Oct 2018 at 05:22, Shalin Shekhar Mangar wrote: > In case you are using a recent Solr 7.x version with collections that have > autoAddReplicas=true, you should disable the auto add replicas feature > before powering off so that Solr does not decide to move replicas around > because nodes have been lost. See > > https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-auto-add-replicas.html#using-cluster-property-to-enable-autoaddreplicas > > On Wed, Oct 31, 2018 at 3:27 AM lstusr 5u93n4 wrote: > > > Hi All, > > > > We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and > > the data director on hdfs. It has 950 million documents in the index, > > occupying 700GB of disk space. > > > > We need to completely power off the system to move it. > > > > Are there any actions we should take on shutdown to help the process? > > Anyhing we should expect on power on? > > > > Thanks, > > > > Kyle > > > > > -- > Regards, > Shalin Shekhar Mangar. >
Re: Solr cloud - poweroff procedure
In case you are using a recent Solr 7.x version with collections that have autoAddReplicas=true, you should disable the auto add replicas feature before powering off so that Solr does not decide to move replicas around because nodes have been lost. See https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-auto-add-replicas.html#using-cluster-property-to-enable-autoaddreplicas On Wed, Oct 31, 2018 at 3:27 AM lstusr 5u93n4 wrote: > Hi All, > > We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and > the data director on hdfs. It has 950 million documents in the index, > occupying 700GB of disk space. > > We need to completely power off the system to move it. > > Are there any actions we should take on shutdown to help the process? > Anyhing we should expect on power on? > > Thanks, > > Kyle > -- Regards, Shalin Shekhar Mangar.
Re: Solr cloud - poweroff procedure
I agree. 1. Shut down each Solr server process using the “bin/solr” script. 2. Shut down the Zookeeper ensemble. 3. Take backups. 4. Shut down the OS. Do that in reverse to get going. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 30, 2018, at 8:03 PM, Erick Erickson wrote: > > bin/solr stop > > As long as you don't kill it with extreme prejudice (i.e. kill -9 or > pull the plug) it should be fine. Assuming you're running ZooKeeper > in an external ensemble, I'd certainly stop those after all the Solr > instances were stopped. > > Powering the nodes up is irrelevant to Solr, the bin/solr start bit > will actually start the Solr instance. And no, there's nothing special > that needs to happen there, it's just like any other time the script > starts Solr. You'll have to have the ZooKeeper instances running for > Solr to come up of course. > > Best, > Erick > On Tue, Oct 30, 2018 at 2:57 PM lstusr 5u93n4 wrote: >> >> Hi All, >> >> We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and >> the data director on hdfs. It has 950 million documents in the index, >> occupying 700GB of disk space. >> >> We need to completely power off the system to move it. >> >> Are there any actions we should take on shutdown to help the process? >> Anyhing we should expect on power on? >> >> Thanks, >> >> Kyle
Re: Solr cloud - poweroff procedure
bin/solr stop As long as you don't kill it with extreme prejudice (i.e. kill -9 or pull the plug) it should be fine. Assuming you're running ZooKeeper in an external ensemble, I'd certainly stop those after all the Solr instances were stopped. Powering the nodes up is irrelevant to Solr, the bin/solr start bit will actually start the Solr instance. And no, there's nothing special that needs to happen there, it's just like any other time the script starts Solr. You'll have to have the ZooKeeper instances running for Solr to come up of course. Best, Erick On Tue, Oct 30, 2018 at 2:57 PM lstusr 5u93n4 wrote: > > Hi All, > > We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and > the data director on hdfs. It has 950 million documents in the index, > occupying 700GB of disk space. > > We need to completely power off the system to move it. > > Are there any actions we should take on shutdown to help the process? > Anyhing we should expect on power on? > > Thanks, > > Kyle
