On 2/21/2018 7:58 PM, Susheel Kumar wrote:
Below output for prod machine based on the steps you described.  Please
take a look.  The solr searches are returning fine and no issue with
performance but since last 4 months swap space started going up. After
restart, it comes down to zero and then few weeks, it utilization reaches
to 40-50% and thus requires restart of solr process.


I bet that if you run this command, it will show you a value of 60:

cat /proc/sys/vm/swappiness

This makes the OS very aggressive about using swap, even when there is absolutely no need for it to do so.

If you type the following series of commands, it should fix the problem and prevent it from happening again until you reboot the system:

echo "0" > /proc/sys/vm/swappiness
swapoff -a
swapon -a

Note that when the swapoff command runs, it will force the OS to read all the swapped data back into memory.  It will take several minutes for this to occur, because it must read nearly a gigabyte of data and figure out how to put it back in memory. Both of the command outputs you included say that there is over 20GB of free memory.  So I do not anticipate the system having problems from running these commands.  It will slow the machine down temporarily, though -- so only do it during a quiet time for your Solr install.

To make this setting survive a reboot, find the sysctl.conf file somewhere in your /etc directory and add this line to it:

vm.swappiness = 0

This setting does not completely disable swap.  If the system finds itself with real memory pressure and actually does NEED to use swap, it still will ... it just won't swap anything out before it's actually required.

I do not think the behavior you are seeing is actually causing problems, based on your system load and CPU usage.  But what I've shared should fix it for you.

Thanks,
Shawn

Reply via email to