Chad, You could use the HealthCheckProgram and friends (and likely some simple scripting) in your slurm configuration to do this. Might fit your need.
- Jared ARCC From: Chad Cropper [mailto:[email protected]] Sent: Thursday, March 30, 2017 10:20 AM To: slurm-dev <[email protected]> Subject: [slurm-dev] RE: Suggestions on node memory cleaning Yes, I have seen these. Thanks. But the issue is I only want this to run when the node is empty. Our workload is very serial and most job sonly use 1,2,4 cores. So on a large node of 32 cores, we would have many jobs active. I only ever want this to run when no jobs exist on the node. My current best plan is to write python to check for jobs on a node, if empty set to drain, then ssh as root and run the command. From: John Hearns <[email protected]<mailto:[email protected]>> Reply-To: slurm-dev <[email protected]<mailto:[email protected]>> Date: Thursday, March 30, 2017 at 11:13 AM To: slurm-dev <[email protected]<mailto:[email protected]>> Subject: [slurm-dev] RE: Suggestions on node memory cleaning I think this thread has the answer http://askubuntu.com/questions/609226/freeing-page-cache-using-echo-3-proc-sys-vm-drop-caches-doesnt-work echo 3 | sudo tee /proc/sys/vm/drop_caches From: John Hearns [mailto:[email protected]] Sent: 30 March 2017 17:11 To: slurm-dev <[email protected]<mailto:[email protected]>> Subject: [slurm-dev] RE: Suggestions on node memory cleaning Aha, follow this thread http://www.beowulf.org/pipermail/beowulf/2013-April/031407.html From: John Hearns [mailto:[email protected]] Sent: 30 March 2017 17:07 To: slurm-dev <[email protected]<mailto:[email protected]>> Subject: [slurm-dev] RE: Suggestions on node memory cleaning Chad, I did rather a lot of work on that issue in my last job, with PBSPro actually. We wanted a PBSPro prolog script which would flush the caches using a echo 3 > /proc/sys/vm/drop_caches before the job is run. So far so good. The wrinkle I found with PBSPro is that the prolog is run under the user id. You need to be root to run that command – so I put it in /etc/sudoers and did : sudo echo 3 > /proc/sys/vm/drop_caches As I remember that doesn’t work either due to some weirdness of bash redirects, but you can get it to work. From: Chad Cropper [mailto:[email protected]] Sent: 30 March 2017 16:53 To: slurm-dev <[email protected]<mailto:[email protected]>> Subject: [slurm-dev] Suggestions on node memory cleaning We would like to clean our memory buffers/cache on a regular basis without rebooting nodes. We can easily do this manually with “sync; echo 3 > /proc/sys/vm/drop_caches“. Is anyone else out there doing anything like this? Does SLURM offer anything builtin for running this when it sees an empty node? Outside of rotating nodes into a drain state for maintenance and then running this command, I have yet to see any other option. Any suggestions are greatly appreciated. -Chad Cropper ________________________________ *** The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copies of it. If you have any questions concerning this message, please contact the sender. *** Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP ________________________________ *** The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copies of it. If you have any questions concerning this message, please contact the sender. ***
