> > ie. What are you trying to achieve? - Why do you want to 'split CPUs on > each nodes between different queues'? > > each node has GPU. Jobs which use GPU use only one CPU in node. > So I wish to make special queue for this kind of jobs and still use other > CPUs for jobs which need only CPUs. > Of course we can directly ask the nodes in batch script but in this case > idea of batch system on cluster with hundred nodes will be lost. > > Thanx, > Egor.
Hi Egor, We have a cluster with a configuration that may help you. We have nodes with two sockets each populated with quad-core cpus and two gpus per node. With one gpu per node your setup should be simpler. If you are using maui or moab as a scheduler, you can define one 'gpu' consumable generic resource per node. Job should request the resource gres=gpu if they intend to use a gpu - and only one such task per node will be scheduled at a time. (we also have a gpu group setup to make sure only jobs requesting gpus can access them, but you don't really need that if you trust your users). This will not prevent non-gpu jobs from using all the cpus and preventing gpu jobs. You could do this with a standing reservation (SR) of one cpu per node. You would need an ACL for this and maybe here is where a queue would come in. To simplify the setup you could have a 'gpu' queue with a default gres=gpu (not sure how to do this! But at least you could do it with a qsub filter) and have the SR ACL include the gpu queue/class. You could also allow shorter non-gpu jobs in the SR so that there was the potential to run whole-node non-gpu jobs as long as they are short (or on the weekend or from VIPs...). We have not done this so far (gpu SRs) and hope not to need it. Our setup breaks down if you want hybrid multi-cpu/multi-gpu jobs but we can cope with the case if whole nodes are allocated (no need to request gpus in that case - there can be no conflict). Others probably have their own alternative strategies. Regards, Gareth _______________________________________________ mauiusers mailing list [email protected] http://www.supercluster.org/mailman/listinfo/mauiusers
