On 11/28/2012 9:06 AM, Annette Newton wrote:
We are seeing strange gc behaviour after running solr cloud under
quite heavy insert load for a period of time. The old generation
becomes full and no amount of garbage collection will free up the
memory. I have attached a memory profile, as you can see it gets
progressively worse as the day goes on to the point where we are
always doing full garbage collections all the time. The only way I
have found to resolve this issue is to reload the core, then
subsequent garbage collections reclaim the used space, that’s what
happened at 3pm on the Memory profile. All the nodes eventually
display the same behaviour.
We have multiple threads running adding batches of upto 100 documents
at a time. I have also attached our Schema and Config.
We are running 4 shards each with a single replica, have a 3 node
zookeeper setup and the 8 solr boxes instances are aws High-Memory
Double Extra Large with 34.2 GB Memory, 4 Virtual cores.
Looking at that jconsole graph, I am not seeing a problem. It looks
fairly normal to me, especially for heavy indexing.
Solr has been up for at least 29 hours, just based on the graph I can
see, which may not reflect the full JVM uptime - click on VM Summary to
see that. In that time, you've only spent a total of two minutes in
garbage collection, which does not seem problematic to me. Also, your
Old Gen is not full - it's only using 1.2GB out of 5.5GB available. A
full GC (ConcurrentMarkSweep) would only be automatically triggered when
the Old Gen reaches that 5.5GB mark.
Are you seeing actual performance problems, or are you just concerned
about what you see when you watch the memory?
General note about memory and Solr: I have had very good luck with the
following java memory options. Each machine handles three 22GB index
shards on one JVM.
-Xms4096M
-Xmx8192M
-XX:NewRatio=1
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
Thanks,
Shawn
