Re: Very high load after replicating

2010-12-13 Thread Mark

Markus,

My configuration is as follows...

filterCache
  class=solr.FastLRUCache
  size=16384
  initialSize=16384
  autowarmCount=16384/

queryResultCache
  class=solr.LRUCache
  size=16384
  initialSize=16384
  autowarmCount=16384/

documentCache
  class=solr.LRUCache
  size=16384
  initialSize=16384
  autowarmCount=16384/
...
useColdSearcherfalse/useColdSearcher
maxWarmingSearchers2/maxWarmingSearchers
...
useCompoundFilefalse/useCompoundFile
ramBufferSizeMB64/ramBufferSizeMB
mergeFactor10/mergeFactor
unlockOnStartupfalse/unlockOnStartup
reopenReaderstrue/reopenReaders

No cache warming queries and our machines have 8g of memory in them with 
about 5120m of ram dedicated to so Solr. When our index is around 10-11g 
in size everything runs smoothly. At around 20g+ it just falls apart.


Can you (or anyone) provide some suggestions? Thanks


On 12/12/10 1:11 PM, Markus Jelsma wrote:

There can be numerous explanations such as your configuration (cache warm
queries, merge factor, replication events etc) but also I/O having trouble
flushing everything to disk. It could also be a memory problem, the OS might
start swapping if you allocate too much RAM to the JVM leaving little for the
OS to work with.

You need to provide more details.


After replicating an index of around 20g my slaves experience very high
load (50+!!)

Is there anything I can do to alleviate this problem?  Would solr cloud
be of any help?

thanks


Very high load after replicating

2010-12-12 Thread Mark
After replicating an index of around 20g my slaves experience very high 
load (50+!!)


Is there anything I can do to alleviate this problem?  Would solr cloud 
be of any help?


thanks


Re: Very high load after replicating

2010-12-12 Thread Markus Jelsma
There can be numerous explanations such as your configuration (cache warm 
queries, merge factor, replication events etc) but also I/O having trouble 
flushing everything to disk. It could also be a memory problem, the OS might 
start swapping if you allocate too much RAM to the JVM leaving little for the 
OS to work with.

You need to provide more details.

 After replicating an index of around 20g my slaves experience very high
 load (50+!!)
 
 Is there anything I can do to alleviate this problem?  Would solr cloud
 be of any help?
 
 thanks