Re: Solr node going to recovering state during heavy reindexing
Thanks Shawn for the insight. WIll try your recommendations . Gopal On Mon, Apr 27, 2015 at 9:46 PM, Rajesh Hazari rajeshhaz...@gmail.com wrote: thanks, i am sure that we have missed this command line property, this gives me more information on how to use latest solr scripts more effectively. *Thanks,* *Rajesh**.* On Mon, Apr 27, 2015 at 12:04 PM, Shawn Heisey apa...@elyograg.org wrote: On 4/27/2015 9:15 AM, Gopal Jee wrote: We have a 26 node solr cloud cluster. During heavy re-indexing, some of nodes go into recovering state. as per current config, soft commit is set to 15 minute and hard commit to 30 sec. Moreover, zkClientTimeout is set to 30 sec in solr nodes. Please advise. The most common reason for this is general performance issues that make some operations take longer than the zkClientTimeout. My first suspect would be long garbage collection pauses. This assumes you're not using a very recent version (4.10.x or 5.x) with the new bin/solr script, and your java commandline does not have any garbage collection tuning. The bin/solr script does a lot of GC tuning. The second suspect would be that you don't have enough RAM left for your operating system to cache your index effectively. It's possible to have both of these problems happening. These problems, and a few others, are outlined here: http://wiki.apache.org/solr/SolrPerformanceProblems Thanks, Shawn
Solr node going to recovering state during heavy reindexing
We have a 26 node solr cloud cluster. During heavy re-indexing, some of nodes go into recovering state. as per current config, soft commit is set to 15 minute and hard commit to 30 sec. Moreover, zkClientTimeout is set to 30 sec in solr nodes. Please advise. Thanks Gopal
Re: Solr node going to recovering state during heavy reindexing
On 4/27/2015 9:15 AM, Gopal Jee wrote: We have a 26 node solr cloud cluster. During heavy re-indexing, some of nodes go into recovering state. as per current config, soft commit is set to 15 minute and hard commit to 30 sec. Moreover, zkClientTimeout is set to 30 sec in solr nodes. Please advise. The most common reason for this is general performance issues that make some operations take longer than the zkClientTimeout. My first suspect would be long garbage collection pauses. This assumes you're not using a very recent version (4.10.x or 5.x) with the new bin/solr script, and your java commandline does not have any garbage collection tuning. The bin/solr script does a lot of GC tuning. The second suspect would be that you don't have enough RAM left for your operating system to cache your index effectively. It's possible to have both of these problems happening. These problems, and a few others, are outlined here: http://wiki.apache.org/solr/SolrPerformanceProblems Thanks, Shawn
Re: Solr node going to recovering state during heavy reindexing
our production solr nodes were having similar issue with 4 nodes everything is normal, but when we try to increase the replicas (nodes) to 10 most of then went to recovery. our config params : nodes : 20 (replica in each node) soft commit is 6 sec hard commit is 5 min indexing scheduled time : every 3 mins around 5k of documents. Now we are back on 4 nodes in prod, which is working our for this season, but we may be hitting this case once again in near future where we want to expand. I have been going through the blog which suggest soft commit and hard commit for near real time search instances, may be you can also have a look. http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ *Rajesh.* On Mon, Apr 27, 2015 at 11:15 AM, Gopal Jee gopal@myntra.com wrote: We have a 26 node solr cloud cluster. During heavy re-indexing, some of nodes go into recovering state. as per current config, soft commit is set to 15 minute and hard commit to 30 sec. Moreover, zkClientTimeout is set to 30 sec in solr nodes. Please advise. Thanks Gopal
Re: Solr node going to recovering state during heavy reindexing
thanks, i am sure that we have missed this command line property, this gives me more information on how to use latest solr scripts more effectively. *Thanks,* *Rajesh**.* On Mon, Apr 27, 2015 at 12:04 PM, Shawn Heisey apa...@elyograg.org wrote: On 4/27/2015 9:15 AM, Gopal Jee wrote: We have a 26 node solr cloud cluster. During heavy re-indexing, some of nodes go into recovering state. as per current config, soft commit is set to 15 minute and hard commit to 30 sec. Moreover, zkClientTimeout is set to 30 sec in solr nodes. Please advise. The most common reason for this is general performance issues that make some operations take longer than the zkClientTimeout. My first suspect would be long garbage collection pauses. This assumes you're not using a very recent version (4.10.x or 5.x) with the new bin/solr script, and your java commandline does not have any garbage collection tuning. The bin/solr script does a lot of GC tuning. The second suspect would be that you don't have enough RAM left for your operating system to cache your index effectively. It's possible to have both of these problems happening. These problems, and a few others, are outlined here: http://wiki.apache.org/solr/SolrPerformanceProblems Thanks, Shawn
