The problem here will be knowing how much free memory you need from the jobs which are already running.
In theory you could do this by adding free memory as a load_threshold (see man qconf). This would then allow low memory to put the node into an alarm state (like you get if the load average is too high), and stop further jobs going to it. You'd need to write a load sensor to get a suitable value for this from each node so you could decide when the memory state was too high. The problem is that the current memory usage may not reflect the maximum usage that the jobs you already have running will achieve. Limiting submissions based on current usage still, in our experience, caused the nodes to run out of memory regularly, causing jobs to get killed. In the end our solution was to have strict hard limits (h_vmem) on memory and to define h_vmem as a consumable complex. To make life easier for our users though we used a job submission verifier to add a default allocation of 1GB to any job which didn't ask for any memory. This covers all of the small jobs. For larger jobs we simply tell people to ask for more than they need if they're only doing something once, or if they have a bunch of jobs to run then run one with too much memory allocated and then use qacct to look at the actual max usage so they know what they should ask for next time. We had some teething troubles with this for a few weeks after it was introduced, but it's all been working smoothly for a long time now. Simon. From: <users-boun...@gridengine.org<mailto:users-boun...@gridengine.org>> on behalf of Ben Daniel Pere <ben.p...@gmail.com<mailto:ben.p...@gmail.com>> Date: Wednesday, 1 June 2016 at 14:44 To: "users@gridengine.org<mailto:users@gridengine.org>" <users@gridengine.org<mailto:users@gridengine.org>> Subject: [gridengine users] How to set a minimum free memory limit for any task submission on SGE? Hi all, I'm trying to limit SGE from submittion jobs to any node which has less than 10% memory free, regarrdless of queue, user or anything else - just add a rule / complex / resource quote that will make SGE not submit tasks to machines with less than 10% memory free (or 20GB if number must be absolute). We've got several queues and each can run tasks with various memory demands and we're not asking users to provide how much memory they need and there's no way to start doing it, people will give wrong estimations and we'll either have dying tasks or idle cluster due to exaggerated estimations and it really depends on the size of the dataset being worked on which often can't be predetermined. is there a way to do it? thanks!!! The Babraham Institute, Babraham Research Campus, Cambridge CB22 3AT Registered Charity No. 1053902. The information transmitted in this email is directed only to the addressee. If you received this in error, please contact the sender and delete this email from your system. The contents of this e-mail are the views of the sender and do not necessarily represent the views of the Babraham Institute. Full conditions at: www.babraham.ac.uk<http://www.babraham.ac.uk/terms>
_______________________________________________ users mailing list users@gridengine.org https://gridengine.org/mailman/listinfo/users
