Thanks for your answer Mark. I think I'll try to update to 4.2. I'll keep you
updated.
Anyway, I'd not say that the full index is replicated, I've been monitoring
the replication process in the Solr admin console and there I see that
usually not more than 50-100 files are transferrend, the total size is
rarely greater than 50MB. Is this info trustable?
Victor
Mark Miller-3 wrote
> Okay - yes, 4.0 is a better choice for replication than 4.1.
>
> It almost sounds like you may be replicating the full index rather than
> just changes or something. 4.0 had a couple issues as well - a couple
> things that were discovered while writing stronger tests for 4.2.
>
> 4.2 is spreading onto mirrors now.
>
> - Mark
>
> On Mar 11, 2013, at 2:00 PM, Victor Ruiz <
> bik1979@
> > wrote:
>
>> no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was
>> an
>> issue with the replication, so I decided not to try it for now
>>
>>
>> Mark Miller-3 wrote
>>> Are you using Solr 4.1?
>>>
>>> - Mark
>>>
>>> On Mar 11, 2013, at 1:53 PM, Victor Ruiz <
>>
>>> bik1979@
>>
>>> > wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I have a problem with Solr replication. I have 2 solr servers (Solr
>>>> 4.0.0) 1
>>>> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
>>>> every server, there are 2 independent instances of solr running (I
>>>> tried
>>>> also multicore config, but having independent instances has for me
>>>> better
>>>> performance), every instance having a differente collection. So, we
>>>> have
>>>> 2
>>>> masters in server 1, and 2 slaves in server 2.
>>>>
>>>> Index size is currently (for the biggest collection) around 17 million
>>>> documents, with a total size near 12 GB. The files transferred every
>>>> replication cycle are typically not more than 100, with a total size
>>>> not
>>>> bigger than 50MB. The other collection is not that big, just around 1
>>>> million docs and not bigger than 2 GB and not a high update ratio. The
>>>> big
>>>> collection has a load around 200 queries per second (MoreLikeThis,
>>>> RealTimeGetHandler , TermVectorComponent mainly), and for the small one
>>>> it
>>>> is below 50 queries per second
>>>>
>>>> Replication has been working for long time with any problem, but in the
>>>> last
>>>> weeks the replication cycles started to take long and long time for the
>>>> big
>>>> collection, even more than 2 minutes, some times even more. During that
>>>> time, slaves are so overloaded, that many queries are timing out,
>>>> despite
>>>> the timeout in my clients is 30 seconds.
>>>>
>>>> The servers are in same LAN, gigabit ethernet, so the broadband should
>>>> not
>>>> be the bottleneck.
>>>>
>>>> Since the index is receiving frequents updates and deletes (update
>>>> handler
>>>> receives more than 200 request per second for the big collection, but
>>>> not
>>>> more than 5 per second for the small one), I tried to use the
>>>> maxCommitsToKeep attribute, to ensure that no file was deleted during
>>>> replication, but it has no effect.
>>>>
>>>> My solrconfig.xml in the big collection is like that:
>>>>
>>>> <?xml version="1.0" encoding="UTF-8" ?>
>>>>
>>>>
>>>
> <config>
>>>>
>>>>
>>>
> <luceneMatchVersion>
>>> LUCENE_40
>>>
> </luceneMatchVersion>
>>>>
>>>>
>>>
> <directoryFactory name="DirectoryFactory"
>>
>>>
>>>
>>> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>>>>
>>>>
>>>>
>>>
> <indexConfig>
>>>>
>>>
> <mergeFactor>
>>> 3
>>>
> </mergeFactor>
>>>>
>>>>
>>>
> <deletionPolicy class="solr.SolrDeletionPolicy">
>>>>
>>>>
>>>
> <str name="maxCommitsToKeep">
>>> 10
>>>
> </str>
>>>>
>>>
> <str name="maxOptimizedCommitsToKeep">
>>> 1
>>>
> </str>
>>>>
>>>>
>>>
> <str name="maxCommitAge">
>>> 6HOUR
>>>
> </str>
>>>>
>>>>
>>>
> </deletionPolicy>
>>>>
>>>>
>>>
> </indexConfig>
>>>>
>>>>
>>>
> <jmx/>
>>>>
>>>>
>>>
> <updateHandler class="solr.DirectUpdateHandler2">
>>>>
>>>>
>>>
> <autoCommit>
>>>>
>>>
> <maxDocs>
>>> 2000
>>>
> </maxDocs>
>>>>
>>>
> <maxTime>
>>> 30000
>>>
> </maxTime>
>>>>
>>>
> </autoCommit>
>>>>
>>>>
>>>
> <autoSoftCommit>
>>>>
>>>
> <maxTime>
>>> 500
>>>
> </maxTime>
>>>>
>>>
> </autoSoftCommit>
>>>>
>>>>
>>>
> <updateLog>
>>>>
>>>
> <str name="dir">
>>> ${solr.data.dir:}
>>>
> </str>
>>>>
>>>
> </updateLog>
>>>>
>>>>
>>>
> </updateHandler>
>>>>
>>>>
>>>
> <query>
>>>>
>>>
> <maxBooleanClauses>
>>> 2048
>>>
> </maxBooleanClauses>
>>>>
>>>>
>>>
> <filterCache
>>
>>>
>>> class="solr.FastLRUCache"
>>>> size="2048"
>>>> initialSize="1024"
>>>> autowarmCount="1024"/>
>>>>
>>>>
>>>
> <queryResultCache
>>
>>>
>>> class="solr.LRUCache"
>>>> size="2048"
>>>> initialSize="1024"
>>>> autowarmCount="1024"/>
>>>>
>>>>
>>>>
>>>
> <documentCache
>>
>>>
>>> class="solr.LRUCache"
>>>> size="2048"
>>>> initialSize="1024"
>>>> autowarmCount="1024"/>
>>>>
>>>>
>>>
> <enableLazyFieldLoading>
>>> true
>>>
> </enableLazyFieldLoading>
>>>>
>>>>
>>>
> <queryResultWindowSize>
>>> 50
>>>
> </queryResultWindowSize>
>>>>
>>>>
>>>
> <queryResultMaxDocsCached>
>>> 50
>>>
> </queryResultMaxDocsCached>
>>>>
>>>>
>>>
> <listener event="newSearcher" class="solr.QuerySenderListener">
>>>>
>>>
> <arr name="queries">
>>>>
>>>
> <lst>
>>>>
>>>
> <str name="q">
>>> *:*
>>>
> </str>
>>>>
>>>
> <str name="fq">
>>> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
>>>
> </str>
>>>>
>>>
> <str name="rows">
>>> 1000
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> </arr>
>>>>
>>>
> </listener>
>>>>
>>>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
>>>>
>>>
> <arr name="queries">
>>>>
>>>
> <lst>
>>>>
>>>
> <str name="q">
>>> *:*
>>>
> </str>
>>>>
>>>
> <str name="fq">
>>> date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
>>>
> </str>
>>>>
>>>
> <str name="rows">
>>> 1000
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> </arr>
>>>>
>>>
> </listener>
>>>>
>>>>
>>>
> <useColdSearcher>
>>> true
>>>
> </useColdSearcher>
>>>>
>>>>
>>>
> <maxWarmingSearchers>
>>> 4
>>>
> </maxWarmingSearchers>
>>>>
>>>
> </query>
>>>>
>>>>
>>>>
>>>>
>>>
> <requestHandler name="/replication" class="solr.ReplicationHandler">
>>>>
>>>
> <lst name="master">
>>>>
>>>
> <str name="enable">
>>> ${enable.master:false}
>>>
> </str>
>>>>
>>>
> <str name="replicateAfter">
>>> commit
>>>
> </str>
>>>>
>>>
> <str name="replicateAfter">
>>> startup
>>>
> </str>
>>>>
>>>
> <str name="backupAfter">
>>> startup
>>>
> </str>
>>>>
>>>
> <str
>>
>>>
>>>
>>> name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> <lst name="slave">
>>>>
>>>
> <str name="enable">
>>> ${enable.slave:false}
>>>
> </str>
>>>>
>>>
> <str
>>
>>>
>>> name="masterUrl">http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}
>>>
> </str>
>>>>
>>>
> <str name="pollInterval">
>>> 05:00
>>>
> </str>
>>>>
>>>
> <str name="httpBasicAuthUser">
>>> ${MASTER_HTTP_USER}
>>>
> </str>
>>>>
>>>
> <str name="httpBasicAuthPassword">
>>> ${MASTER_HTTP_PWD}
>>>
> </str>
>>>>
>>>
> </lst>
>>>>
>>>
> </requestHandler>
>>>>
>>>>
>>>
> <admin>
>>>>
>>>
> <defaultQuery>
>>> *:*
>>>
> </defaultQuery>
>>>>
>>>
> </admin>
>>>>
>>>>
>>>
> </config>
>>>>
>>>> Poll interval is now set to 5 min, I tried to reduce it to 2 min and to
>>>> increase it up to 10, with no effect, the replication is always taking
>>>> so
>>>> long., even with a poll time of 2 minutes, when there are only a few
>>>> megas
>>>> to replicate.
>>>>
>>>> Any idea suggestion about what could be the problem?
>>>>
>>>> Thanks in advance,
>>>> Victor
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046392.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046428.html
Sent from the Solr - User mailing list archive at Nabble.com.
