Hi Gareth, > > > ie. What are you trying to achieve? - Why do you want to 'split CPUs on > > each nodes between different queues'? > > > > each node has GPU. Jobs which use GPU use only one CPU in node. > > So I wish to make special queue for this kind of jobs and still use other > > CPUs for jobs which need only CPUs. > > Of course we can directly ask the nodes in batch script but in this case > > idea of batch system on cluster with hundred nodes will be lost. > >
thanx for your answer. > We have a cluster with a configuration that may help you. We have nodes with > two sockets each populated with quad-core cpus and two gpus per node. With > one gpu per node your setup should be simpler. you are right with one gpu per node and only single jobs it is not difficult to configure maui (in my case). > If you are using maui or moab as a scheduler, you can define one 'gpu' > consumable generic resource per node. Job should request the resource > gres-gpu if they intend to use a gpu - and only one such task per node will > be scheduled at a time. (we also have a gpu group setup to make sure only > jobs requesting gpus can access them, but you don't really need that if you > trust your users). I use for this NODECFG parameter. But I will try gres possibilities but I am not sure that this really can help. > This will not prevent non-gpu jobs from using all the cpus and preventing gpu > jobs. You could do this with a standing reservation (SR) of one cpu per > node. You would need an ACL for this and maybe here is where a queue would > come in. To simplify the setup you could have a 'gpu' queue with a default > gres=gpu (not sure how to do this! But at least you could do it with a qsub > filter) and have the SR ACL include the gpu queue/class. You could also > allow shorter non-gpu jobs in the SR so that there was the potential to run > whole-node non-gpu jobs as long as they are short (or on the weekend or from > VIPs...). We have not done this so far (gpu SRs) and hope not to need it. Yesterday I played with SR. It also works with single jobs, not with multi-cpu/multi-gpu jobs and don't protect from occupation of an one CPU per node for gpu jobs. > Our setup breaks down if you want hybrid multi-cpu/multi-gpu jobs but we can > cope with the case if whole nodes are allocated (no need to request gpus in > that case - there can be no conflict). In my config: NODECFG[DEFAULT] MAXPROC[CLASS:ONE]=1 MAXPROC[CLASS:TWO]=7 MAXNODE[CLASS:ONE]=1 MAXJOB[CLASS:ONE]=1 it is perfectly works with single non-gpu jobs and single-gpu/multi-gpu jobs. but if I submit non-gpu multi-job at first on several nodes or only on one node than gpu-jobs start to run on one node and want to occupy 2 CPUs per nodes. Why I don't understand. Maybe maui guru can explain this situation. Thanx in advanced. _______________________________________________ mauiusers mailing list [email protected] http://www.supercluster.org/mailman/listinfo/mauiusers
