1. I am using Oracle JVM
user@host:~$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

2. I will try out jHiccup and your GC settings.

3. Yes, I am running ZK instances in an ensemble. I didn't know I need to
pass all the instances of ZK to a single solr node. I will try it out right
now. This means if you have a large cluster say of 50 solr nodes and 10 ZK
nodes then I will need to pass all the 10 nodes to -DzkHost of the 50 solr
processes? What is the reasoning behind this?

Thanks,
-Utkarsh




On Tue, Apr 8, 2014 at 5:37 PM, Shawn Heisey <s...@elyograg.org> wrote:

> On 4/8/2014 6:00 PM, Utkarsh Sengar wrote:
> > Lots of questions indeed :)
> >
> > 1. Total virtual machines: 3
> > 2. Replication factor: 0 (don't have any replicas yet)
> > 3. Each machine has 1 shard which has 20GB of data. So data for a
> > collection is spread across 3 machines totalling to 60GB
> > 4. Start solr:
> > java -Xmx10000m
> >        -javaagent:newrelic/newrelic.jar
> >        -Dsolr.clustering.enabled=true
> >        -Dsolr.solr.home=multicore
> >        -Djetty.class.path=lib/ext/* "
> >        -Dbootstrap_conf=true
> >        -DnumShards=3
> >        -DzkHost=localhost:2181 -jar start.jar"
> > 5. Yes, all machines have 24GB RAM and 9GB heap. Separate process of ZK
> is
> > running on these machine.
> > 6. top screenshot: http://i.imgur.com/g6w9Bim.png
>
> A followup question:  What vendor and version of JVM are you running?
> Excellent choices include very recent Java 6 releases from Oracle,
> Oracle Java 7u25, and whatever OpenJDK version corresponds to Oracle
> 7u25.  Good choices include most version of Oracle Java 7, Oracle Java
> 6, and OpenJDK7.  The latest versions of Oracle Java 7 (from 7u40 to
> 7u51) have known bugs that affect Solr.
>
> OpenJDK6 and commercial java versions from non-Oracle vendors like IBM
> are very bad choices, because they have known serious bugs.  I don't
> know much about the Zing JVM, but it is probably a good choice.  If you
> are running Zing, then what I'm saying below about GC pauses will not
> apply.
>
> Solr 4.8 will require Java 7, so if you plan to upgrade that far, be
> sure you're not using Java 6 at all.
>
> One possible problem that I always investigate first is whether or not
> there's enough RAM to cache the index effectively.  The 14GB of RAM in
> your disk cache is not a perfect setup for a 20GB index, but it should
> be plenty.  The fact that you still have 4GB of RAM free on your top
> screenshot is further evidence that you do have plenty of disk cache.
> No need to pursue that any further.
>
> Garbage collection pauses are however a likely problem here.  I have
> some personal experience with this problem.  Because you're using the
> default collector and have 7GB heap allocated, I can almost guarantee
> that this is a problem, even if New Relic isn't showing it.  A program
> called jHiccup *will* show the problem.
>
> http://www.azulsystems.com/jHiccup
>
> These are my GC settings.  They work very well and are not specific to a
> certain heap size, although I am sure that the config can be improved:
>
> http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
>
> Regarding zookeeper:  Are you running all three of your ZK instances in
> a redundant ensemble, where the config on each of them knows about all
> of them?  You should definitely be doing this.  If you are, then your
> zkHost parameter for Solr needs to reflect that:
>
> -DzkHost=host1:2181,host2:2181,host3:2181
>
> Using only localhost:2181 could cause problems, and they could look like
> the problems you are seeing.
>
> Thanks,
> Shawn
>
>


-- 
Thanks,
-Utkarsh

Reply via email to