I'm a bit fuzzy, but IIRC there was a bug in zone reclaim mode that would
cause it to have a hard hang with significant tmpfs, even when a numa node
wasn't even close to full. This was causing nodes to not even be able to
complete boot.
I think when we noticed it not having a hard hang anymore, we let it go as
it pleased. Does performance increase on the fly if zone_reclaim_mode is
set to 0 after the fact?
From: Stuart Barkley <[email protected]>
To: [email protected]
Date: 09/17/2014 01:31 PM
Subject: [xcat-user] /proc/sys/vm/zone_reclaim_mode and xCAT
Background which may not be too relevant:
We are seeing some severe performance issues on our diskless systems
with an application doing mmap reads of large files on GPFS. The I/O
pattern is multiple sequential reads of a single file across several
nodes. The file is 5-10 times the size of ram on the nodes (we have
tried different physical memory sizes). We currently have the jobs
locked to a single job per node. These are IBM dx360 M2 systems with
8 cores (dual CPU), 24G or 96G of RAM and Infiniband.
Current status:
We have now tracked this down to 'pgscand/s' in the 'sar -B' output
going outrageous (13M pages scanned per second to try to find
something to drop).
This further appears to be related to /proc/sys/vm/zone_reclaim_mode
being set to 1 on our systems despite the documentation we find at
<https://www.kernel.org/doc/Documentation/sysctl/vm.txt> (and other
places) that indicates that the default value should be 0.
<
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/6.3_Technical_Notes/kernel_issues.html
>
indicates that Redhat 6.1 had an incorrect value of 1 and that RH6.2
and RH6.3 corrected that defect. We are running CentOS 6.5.
Lastly, I see some strangeness in xCAT regarding this setting and was
hoping someone here could supply some background. I see the following
lines in xCAT files:
% grep zone_reclaim_mode xCAT-server/share/xcat/netboot/rh/genimage
echo 0 > /proc/sys/vm/zone_reclaim_mode #Avoid kernel bug
% grep zone_reclaim_mode
xCAT-server/share/xcat/netboot/rh/dracut/xcatroot
#echo 0 > /proc/sys/vm/zone_reclaim_mode #Avoid kernel bug
The second one ends up in our stateless boot image and does nothing
since it is just a comment.
Based upon the git version of the imported svn history: The first has
been around nearly forever (a one line specific addition in 2009).
And the second line was in the original version of the file as part of
the original dracut support. Both appear to have been originally
authored by Jarrod.
Questions:
Was the original change a response to the Redhat bug probably in 6.0
or 6.1?
Was there any ispecific reason the change was partially removed as
part of the dracut support?
Any thoughts on why we might still be seeing this set to 1 on our
diskless systems? So far I have not found any references to this
variable in our startup other than the commented out xCAT one.
Thanks,
Stuart
--
I've never been lost; I was once bewildered for three days, but never lost!
-- Daniel Boone
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
