Background which may not be too relevant: We are seeing some severe performance issues on our diskless systems with an application doing mmap reads of large files on GPFS. The I/O pattern is multiple sequential reads of a single file across several nodes. The file is 5-10 times the size of ram on the nodes (we have tried different physical memory sizes). We currently have the jobs locked to a single job per node. These are IBM dx360 M2 systems with 8 cores (dual CPU), 24G or 96G of RAM and Infiniband.
Current status: We have now tracked this down to 'pgscand/s' in the 'sar -B' output going outrageous (13M pages scanned per second to try to find something to drop). This further appears to be related to /proc/sys/vm/zone_reclaim_mode being set to 1 on our systems despite the documentation we find at <https://www.kernel.org/doc/Documentation/sysctl/vm.txt> (and other places) that indicates that the default value should be 0. <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/6.3_Technical_Notes/kernel_issues.html> indicates that Redhat 6.1 had an incorrect value of 1 and that RH6.2 and RH6.3 corrected that defect. We are running CentOS 6.5. Lastly, I see some strangeness in xCAT regarding this setting and was hoping someone here could supply some background. I see the following lines in xCAT files: % grep zone_reclaim_mode xCAT-server/share/xcat/netboot/rh/genimage echo 0 > /proc/sys/vm/zone_reclaim_mode #Avoid kernel bug % grep zone_reclaim_mode xCAT-server/share/xcat/netboot/rh/dracut/xcatroot #echo 0 > /proc/sys/vm/zone_reclaim_mode #Avoid kernel bug The second one ends up in our stateless boot image and does nothing since it is just a comment. Based upon the git version of the imported svn history: The first has been around nearly forever (a one line specific addition in 2009). And the second line was in the original version of the file as part of the original dracut support. Both appear to have been originally authored by Jarrod. Questions: Was the original change a response to the Redhat bug probably in 6.0 or 6.1? Was there any ispecific reason the change was partially removed as part of the dracut support? Any thoughts on why we might still be seeing this set to 1 on our diskless systems? So far I have not found any references to this variable in our startup other than the commented out xCAT one. Thanks, Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone ------------------------------------------------------------------------------ Want excitement? Manually upgrade your production database. When you want reliability, choose Perforce Perforce version control. Predictably reliable. http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk _______________________________________________ xCAT-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/xcat-user
