On 8/27/2013 11:56 AM, vsilgalis wrote:
We have a 2 shard SOLRCloud implementation with 6 servers in production. We
have allocated 24GB to each server and are using JVM max memory settings of
-Xmx14336 on each of the servers. We are using the same embedded jetty that
SOLR comes with. The JVM side of things looks like what I'd expect from
java, climbs until ~13GB and then GC occurs and we are back down to ~4GB
used.
However, on the original leaders of the cluster we currently are seeing
~22GB resident memory usage for Jetty/SOLR and the machines have begun using
swap, which is something I am concerned with. So the question is should I
expect jetty/SOLR to cleanup after itself in terms of memory usage (outside
of the JVM)? Is there something I'm missing to make things more efficient?
Anything else I should be looking at?
I see something interesting in my own production setup. Here are a
couple of screenshots:
https://www.dropbox.com/s/zacp4n3gu8wb9ab/idxb1-top-sorted-mem.png
https://www.dropbox.com/s/70e12m9iunqunzz/idxb1-solr-dashboard.png
The first is a 'top' output sorted by memory usage (pressing shift-M).
I have a 6GB heap, which you can see in the other screenshot, but top
shows that Solr's resident memory usage is 16GB. Of that, it says that
11GB is shared ... but I can't seem to figure out what it's shared *with*.
I think that there's some fibbing going on in the OS. If you add up the
"cached" number and the "free" numbers, you get 54302836k. Since the
machine only has 64GB of RAM, and if we trust the numbers we added, it's
not possible for the java process to actually have the 16GB resident
memory that it claims to have. Something around 6.5 to 7GB is much more
realistic, and that's pretty close to the resident minus shared value.
You can also see that the java process has 54GB of virtual memory, and
this is approximately equal to the total index size (on this machine)
plus the java heap size, plus a little extra for Java itself to operate.
It's my opinion that when Solr is using an MMAP-based directory (the
default), the java process shows some incorrect information about how
much memory it's using.
How much swap is being used on your system, and do you see the OS
actively using it - swapping pages in and out on a regular basis? I
suspect that the amount of swap used is not very much, and that vmstat
would show that there is very little active swapping occurring. If I'm
wrong, then you might be having a real problem.
The default sysctl setting for vm.swappiness on Linux is 60, which means
it is very likely to swap out a few unused memory pages even when there
is no actual memory pressure. On the system that produced the
screenshots, I have set vm.swappiness to 1 in sysctl.conf, and my swap
usage never goes above 0k. My dev server even stays at 0k swap used,
and it has a 7GB java heap, only 16GB of total RAM, and more than twice
as much total index size.
Thanks,
Shawn
