Hi - our reasons are generally about security and consistency. Our system is a commercial multi-tenanted machine (mostly running SGE rather than slurm, but hopefully not for too much longer) - although cgroups will help, we've had issues with jobs on shared nodes saturating IO or network and affecting other jobs. If a user is paying for core hours, they won't be happy if a single core job on one of 'their' nodes slowed their parallel job down.
I'd echo Marcin's comment: *a user could only hurt themselves when running new buggy code.* -- *Nathan Harper* // IT Systems Architect *e: * [email protected] // *t: * 0117 906 1104 // *m: * 07875 510891 // *w: * www.cfms.org.uk <http://www.cfms.org.uk%22> // [image: Linkedin grey icon scaled] <http://uk.linkedin.com/pub/nathan-harper/21/696/b81> CFMS Services Ltd // Bristol & Bath Science Park // Dirac Crescent // Emersons Green // Bristol // BS16 7FR [image: 4.2 CFMS_Artwork_RGB] <http://www.cfms.org.uk> ------------------------------ CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd CFMS Services Ltd registered office // Victoria House // 51 Victoria Street // Bristol // BS1 6AD On 24 November 2014 at 15:58, Marcin Sliwowski <[email protected]> wrote: > > Hello Moe, > > Correct me if I'm wrong, the task/cgroup plugin permits setting CPU/mem > job resource limits via the cgroup.conf but mostly builds off the values > passed in through --cpu_bind and --mem_bind options that users must set. > > Our problem is that user's don't often know their own resource needs. Then > the jobs of the serial job submitters tend to bring to a silent halt the > MPI jobs of other users that happen to share a node. > > I am hoping to get the most out of the cluster and I think I could do that > a little better if I had the option of employing a single user/multi job > node access policy. That way a user could only hurt themselves when running > new buggy code. > > Thanks > > Marcin Sliwowski | SysAdmin@RENCI | 919-445-0479 > > > On 11/21/2014 11:12 AM, [email protected] wrote: > >> >> Slurm does not provide this functionality today, but it would probably be >> simple to add. >> >> Linux cgroup (and the corresponding Slurm cgroup plugins) can bind each >> job to its own set of CPUs, memory, and device files (e.g. GPUs) to avoid >> jobs interfering with each other. Do you really need more than that (i.e. >> prevent one user from seeing another user's processors)? >> >> >> Quoting Nathan Harper <[email protected]>: >> >> although it's not something we have working, this is something I'd be >>> keen >>> to have. User exclusivity on a node is something that is important to >>> the >>> way that we use our clusters, so we currently use 'linear'. Most of our >>> jobs are parallel, so allocating a whole node isn't terrible wasteful, >>> but >>> it would be nice to use con_res, then let users choose if they want to >>> share jobs, but only with themselves. >>> >>> >>> -- >>> *Nathan Harper* // IT Systems Architect >>> >>> *e: * [email protected] // *t: * 0117 906 1104 // *m: * 07875 >>> 510891 // >>> *w: * www.cfms.org.uk <http://www.cfms.org.uk%22> // [image: Linkedin >>> grey >>> icon scaled] <http://uk.linkedin.com/pub/nathan-harper/21/696/b81> >>> CFMS Services Ltd // Bristol & Bath Science Park // Dirac Crescent // >>> Emersons >>> Green // Bristol // BS16 7FR >>> >>> [image: 4.2 CFMS_Artwork_RGB] <http://www.cfms.org.uk> >>> >>> ------------------------------ >>> CFMS Services Ltd is registered in England and Wales No 05742022 - a >>> subsidiary of CFMS Ltd >>> CFMS Services Ltd registered office // Victoria House // 51 Victoria >>> Street >>> // Bristol // BS1 6AD >>> >>> On 19 November 2014 21:36, Marcin Sliwowski <[email protected]> wrote: >>> >>> >>>> Was wondering if any of SLURM's internal schedulers have the equivalent >>>> setting as Maui's "nodeaccesspolicy singleuser". >>>> >>>> In Maui this means that multiple jobs owned by the same user may land >>>> on a >>>> single node. >>>> >>>> I am using the cons_res selection plugin with shed/backfill for >>>> SchedulerType and --exclusive only allows a single job on any given >>>> node. >>>> Without --exclusive jobs from many users can land on a single node. >>>> >>>> Thanks >>>> >>>> -- >>>> Marcin Sliwowski | SysAdmin@RENCI | 919-445-0479 >>>> >>>> >> >>
