What I might suggest in that kind of case, and have done in the past, is at the 8 hour back point, instead of stopping the queue, instead to
restrict it to a single job, and then issue the qstop later so you'd have three crontab entries, one to start the nightly queue a 9pm and a qmgr command to allow, e.g. 16 jobs in that queue one to restrict the queue to a single jobs using qmgr at, e.g. 10pm one to stop the queue at 4am. Then, at the worst, you may have one 8 hour job that runs from 4am to noon You could also use a job filter to renice any job that is submitted to the nightly queue to a nice of 19, which would limit its effect on other jobs. Maui does have some code for suspending and/or checkpointing a job but I have never used it. Jim Coyle Research Computing Group 217 Durham Center http://jjc.public.iastate.edu/ Iowa State Univ. Ames Iowa 50011 ________________________________ From: Dr. Henrik Schulz <[email protected]> Sent: Monday, March 24, 2014 8:16 AM To: Coyle, James J [ITACD] Cc: [email protected] Subject: Re: [Mauiusers] queue for nightly builds Dear Jim, thanks a lot for Your help! Your idea is simple and working well. I already configured my scheduler this way and from a general point of view it is exactly what I need. But there is a drawback: we have a time frame in the nightly queue of 8 hours length, but we have to limit the walltime of the jobs to 4 hours to be sure that outside the time frame there is no nightly job running. It would be better to be able to have a walltime of 8 hours (this is what the users want). But the longer the walltime the earlier I have to turn off the queue. If it would be possible with a reservation for the nightly queue, the scheduler could use its backfill mechanism and the user would be able to submit also very short jobs (like test jobs). The problem now is that between 1 a.m. and 5 a.m. no job is getting started, even if the walltime is 10 minutes. Thanks again! Henrik Am 21.03.2014 um 15:42 schrieb Coyle, James J [ITACD] <[email protected]<mailto:[email protected]>>: Henrik, I just use cron and qmgr commands for this. Good luck with your cluster. - Jim James Coyle, PhD High Performance Computing Group 217 Durham Center Iowa State Univ. phone: (515)-294-2099 Ames, Iowa 50011 web: http://jjc.public.iastate.edu/ Details: --- Create the queue nightly_queue lets suppose you have a max walltime limit of 4 hours on it. Use crontab –e to start the queue at 9:01 pm via: 01 21 * * * /root/start_nightly >> /var/log/nightyQ 2>&1 and stop at 1am (to allow the jobs 4 hours to finish by 5:01am the next day.) 01 01 * * * /root/stop_nightly >> /var/log/nightyQ 2>&1 Where /root/start_nightly might just be /usr/local/bin/qstart nightly_queue plus any other messages you want going into /var/log/nightlyQ such as perhaps a time stamp and the nightly jobs in that queue (e.g. /usr/local/bin/qstat -a nightly_queue ) and similarly for /root/start_nightly with a qstop command. The logging is so that you know that your commands ran and what jobs were waiting before you started the queue and after you stopped the queue in case you need to debug a situation. You could also just start the queue and limit it during the daytime to one node, and the change it via qmgr to allow more nodes during the night and then restrict it again at 1am. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Dr. Henrik Schulz Sent: Friday, March 21, 2014 8:18 AM To: [email protected]<mailto:[email protected]> Subject: [Mauiusers] queue for nightly builds Dear all, one of my users asked for a queue to run jobs for nightly builds/tests. Is this possible? What we have: - a number of nodes (for example node001-node100) - a queue defined to start jobs on all of these nodes (for example 'default_queue') What the user wants: - an additional queue (let´s say 'nightly_queue'), which starts jobs only during the night (let´s say 9:00 p.m. until 5:00 a.m. on Mon-Thu) and during the weekend (Fri, 9:00 p.m. until Mon, 5:00 a.m.). In between the jobs should just stay queued. - maybe also a lower priority for this queue compared to the default_queue Is this somehow possible with maui 3.3.1 (running on top of torque 4.1.7)? I tried to use standing reservations for this, but these reservations prevent jobs in the default_queue from starting. Any help will be appreciated. Best regards, Henrik -- Dr. Henrik Schulz Zentralabteilung Technischer Service Abteilung Informationstechnologie Helmholtz-Zentrum Dresden-Rossendorf Tel: +49 (0351) 260 3268 [email protected]<mailto:[email protected]> http://www.hzdr.de<http://www.hzdr.de/> Vorstand: Prof. Dr. Dr. h. c. Roland Sauerbrey, Prof. Dr. Dr. h. c. Peter Joehnk Vereinsregister: VR 1693 beim Amtsgericht Dresden
_______________________________________________ mauiusers mailing list [email protected] http://www.supercluster.org/mailman/listinfo/mauiusers
