[
https://issues.apache.org/jira/browse/SOLR-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shawn Heisey closed SOLR-7030.
------------------------------
Resolution: Invalid
> replicas goes in recovery mode right after update
> -------------------------------------------------
>
> Key: SOLR-7030
> URL: https://issues.apache.org/jira/browse/SOLR-7030
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10.1
> Environment: Linux 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12
> 00:41:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Vijay Sekhri
>
> We have a cluster of solr cloud server with 10 shards and 4 replicas in each
> shard in our stress environment. In our prod environment we will have 10
> shards and 15 replicas in each shard. Our current commit settings are as
> follows
> <autoSoftCommit>
> <maxDocs>500000</maxDocs>
> <maxTime>180000</maxTime>
> </autoSoftCommit>
> <autoCommit>
> <maxDocs>2000000</maxDocs>
> <maxTime>180000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> We indexed roughly 90 Million docs. We have two different ways to index
> documents
> a) Full indexing. It takes 4 hours to index 90 Million docs and the rate
> of docs coming to the searcher is around 6000 per second
> b) Incremental indexing. It takes an hour to indexed delta changes.
> Roughly there are 3 million changes and rate of docs coming to the searchers
> is 2500 per second
> We have two collections search1 and search2. When we do full indexing , we do
> it in search2 collection while search1 is serving live traffic. After it
> finishes we swap the collection using aliases so that the search2 collection
> serves live traffic while search1 becomes available for next full indexing
> run.
> When we do incremental indexing we do it in the search1 collection which is
> serving live traffic.
> All our searchers have 12 GB of RAM available and have quad core Intel(R)
> Xeon(R) CPU X5570 @ 2.93GHz
> We have observed the following issue when we trigger indexing .
> In about 10 minutes after we trigger indexing on 14 parallel hosts, the
> replicas goes in to recovery mode. This happens to all the shards . In about
> 20 minutes more and more replicas start going into recovery mode. After about
> half an hour all replicas except the leader are in recovery mode. We cannot
> throttle the indexing load as that will increase our overall indexing time.
> So to overcome this issue, we remove all the replicas before we trigger the
> indexing and then add them back after the indexing finishes.
> We observe the same behavior of replicas going into recovery when we do
> incremental indexing. We cannot remove replicas during our incremental
> indexing because it is also serving live traffic. We tried to throttle our
> indexing speed , however the cluster still goes into recovery .
> If we leave the cluster as it , when the indexing finishes , it eventually
> recovers after a while. As it is serving live traffic we cannot have these
> replicas go into recovery mode because it degrades the search performance
> also , our tests have shown.
> We have tried different commit settings like below
> a) No auto soft commit, no auto hard commit and a commit triggered at the
> end of indexing
> b) No auto soft commit, yes auto hard commit and a commit in the end of
> indexing
> c) Yes auto soft commit , no auto hard commit
> d) Yes auto soft commit , yes auto hard commit
> e) Different frequency setting for commits for above
> Unfortunately all the above yields the same behavior . The replicas still
> goes in recovery
> We have increased the zookeeper timeout from 30 seconds to 5 minutes and the
> problem persists.
> Is there any setting that would fix this issue ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
