[
https://issues.apache.org/jira/browse/SOLR-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14291194#comment-14291194
]
Shawn Heisey commented on SOLR-7030:
------------------------------------
This should have been brought up on the mailing list, or possibly IRC, not the
bug tracker.
https://wiki.apache.org/solr/IRCChannels
http://lucene.apache.org/solr/resources.html#community
Based on the indexing rates you have mentioned, I'm fairly sure that the
maxTime value you have for autoCommit and autoSoftCommit is what will be used
-- you'll never reach half a million or two million new documents in three
minutes. Since this means they effectively have the same config, the
recommendation would be to remove autoSoftCommit and just add openSearcher=true
to autoCommit. If you want to use both, they should have different configs so
that they aren't happening at the same time.
We really need to move discussion to the mailing list, but there are some
initial questions:
Is that 12GB of RAM the total amount of RAM on each server? If so, how much of
that 12GB is dedicated to the Java heap and other programs on the system, and
what is the total size of all Solr indexes on each machine?
I will close this issue, if discussion does turn up a bug, then we can re-open.
> replicas goes in recovery mode right after update
> -------------------------------------------------
>
> Key: SOLR-7030
> URL: https://issues.apache.org/jira/browse/SOLR-7030
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10.1
> Environment: Linux 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12
> 00:41:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Vijay Sekhri
>
> We have a cluster of solr cloud server with 10 shards and 4 replicas in each
> shard in our stress environment. In our prod environment we will have 10
> shards and 15 replicas in each shard. Our current commit settings are as
> follows
> <autoSoftCommit>
> <maxDocs>500000</maxDocs>
> <maxTime>180000</maxTime>
> </autoSoftCommit>
> <autoCommit>
> <maxDocs>2000000</maxDocs>
> <maxTime>180000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> We indexed roughly 90 Million docs. We have two different ways to index
> documents
> a) Full indexing. It takes 4 hours to index 90 Million docs and the rate
> of docs coming to the searcher is around 6000 per second
> b) Incremental indexing. It takes an hour to indexed delta changes.
> Roughly there are 3 million changes and rate of docs coming to the searchers
> is 2500 per second
> We have two collections search1 and search2. When we do full indexing , we do
> it in search2 collection while search1 is serving live traffic. After it
> finishes we swap the collection using aliases so that the search2 collection
> serves live traffic while search1 becomes available for next full indexing
> run.
> When we do incremental indexing we do it in the search1 collection which is
> serving live traffic.
> All our searchers have 12 GB of RAM available and have quad core Intel(R)
> Xeon(R) CPU X5570 @ 2.93GHz
> We have observed the following issue when we trigger indexing .
> In about 10 minutes after we trigger indexing on 14 parallel hosts, the
> replicas goes in to recovery mode. This happens to all the shards . In about
> 20 minutes more and more replicas start going into recovery mode. After about
> half an hour all replicas except the leader are in recovery mode. We cannot
> throttle the indexing load as that will increase our overall indexing time.
> So to overcome this issue, we remove all the replicas before we trigger the
> indexing and then add them back after the indexing finishes.
> We observe the same behavior of replicas going into recovery when we do
> incremental indexing. We cannot remove replicas during our incremental
> indexing because it is also serving live traffic. We tried to throttle our
> indexing speed , however the cluster still goes into recovery .
> If we leave the cluster as it , when the indexing finishes , it eventually
> recovers after a while. As it is serving live traffic we cannot have these
> replicas go into recovery mode because it degrades the search performance
> also , our tests have shown.
> We have tried different commit settings like below
> a) No auto soft commit, no auto hard commit and a commit triggered at the
> end of indexing
> b) No auto soft commit, yes auto hard commit and a commit in the end of
> indexing
> c) Yes auto soft commit , no auto hard commit
> d) Yes auto soft commit , yes auto hard commit
> e) Different frequency setting for commits for above
> Unfortunately all the above yields the same behavior . The replicas still
> goes in recovery
> We have increased the zookeeper timeout from 30 seconds to 5 minutes and the
> problem persists.
> Is there any setting that would fix this issue ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
