In a recent blog entry ("The MySQL “swap insanity” problem and the
effects of the NUMA architecture"
http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/),
Jeremy Cole describes a particular but common problem with large memory
installations of MySql on multi-core multi-cpu 64bit NUMA machines,
where debilitating swapping of large amounts of memory occurs even
when there is no (direct) indication of a need to swap.Without getting into the details (it involves how Linux assigns memory to the different nodes (each multi-core CPU is viewed as a 'node' in the Linux NUMA view)), the offered partial solution is to start MySql using the numactl[1] program, like: numactl --interleave all mysql I was wondering if any of the SOLR people have used this when starting up Apache (or whatever servlet engine you use for your SOLR) to reduce unnecessary swap. You probably want to be monitoring the NUMA memory hit statistics found here, with and without the numactl, while testing this: /sys/devices/system/node/node*/numastat -- Note that numactl has a number of other interesting and useful features. One that I have used is the --cpubind which restricts the number of CPUs that an application can run on. There are times when this can improve performance, such as when you have 2 demanding applications running: by assigning one to half of the CPUs and the other to the other half of the CPUs, you _can_ have improved performance due to better locality, cache hits, etc. It takes some tuning and experimentation. YMWV -Glen http://zzzoot.blogspot.com/ [1]http://linuxmanpages.com/man8/numactl.8.php -- -
