Hi Henrik, I had a similar problem recently. Since "consumable resources" didn't work for me, I used standing reservations to "split the nodes":
For the GPU-related processors (=cores): # # maui.cfg # SRCFG[gpu1] HOSTLIST=node01 SRCFG[gpu1] PERIOD=INFINITY SRCFG[gpu1] TASKCOUNT=1 SRCFG[gpu1] RESOURCES=PROCS:2 SRCFG[gpu1] CLASSLIST=gpu ... I wrote one such stanza for each node, in order to make sure that the allocation goes to the right nodes. Then for the cpu queue, I wrote a "complementary reservation", e.g. for node0[1-4]: # # maui.cfg # SRCFG[cpux] HOSTLIST=node0[1-4] SRCFG[cpux] PERIOD=INFINITY SRCFG[cpux] TASKCOUNT=4 SRCFG[cpux] RESOURCES=PROCS:6 SRCFG[cpux] CLASSLIST=cpu If you have other nodes for queue "cpu" without any GPUs, you should prevent the gpu queue from sending jobs elsewhere. Give the gpu nodes a "node property" like GPU (in server_priv/nodes) and restrict the queue (in qmgr) accordingly: set queue gpu resources_default.neednodes = GPU For your last remark (more CPUs for gpu jobs if available), I can't see a simple solution. The jobs, however, may simply try to use more CPUs than requested from the batch system - PBS does not really force a job to use the requested processors only. Hope that helps. Regards, Burkhard Bunk. ---------------------------------------------------------------------- [email protected] Physics Institute, Humboldt University fax: ++49-30 2093 7628 Newtonstr. 15 phone: ++49-30 2093 7980 12489 Berlin, Germany ---------------------------------------------------------------------- On Thu, 11 Nov 2010, Dr. Henrik Schulz wrote: > Dear all, > > in our cluster we have several nodes containing 2 GPUs and 8 CPU-cores. Since > most of the jobs, which utilize the graphics cards, only need 1 CPU-core per > GPU-job, we have some free CPU-cores while all GPUs of the node are used. > > Is it possible (maybe using standing reservations) to have two queues for > that: > > - a queue "gpu" which provides 2 CPU-cores per node > - a queue "cpu" which provides the other 6 CPU-cores of the node. > > The most important fact is that it must not be possible to allocate more than > 6 CPU-cores in the queue "cpu" even if there is no GPU-job running on that > node. The other way round it would be nice to be able to use more than 2 > CPU-cores in the queue "gpu" if some of the other 6 cores are available. > > Do You have any ideas to configure these two queues? > > Thanks in advance! > Henrik > > _______________________________________________ > mauiusers mailing list > [email protected] > http://www.supercluster.org/mailman/listinfo/mauiusers > _______________________________________________ mauiusers mailing list [email protected] http://www.supercluster.org/mailman/listinfo/mauiusers
