Re: [slurm-users] GPU / cgroup challenges

Paul, Having recently set this up, this was my test, when you make a single GPU request from inside an interactive run (salloc ... --gres=gpu:1 srun --pty bash) request you should only see the GPU assigned to you via 'nvidia-smi' When gres is unset you should see nvidia-smi No devices were

Re: [slurm-users] GPU / cgroup challenges

Chris, Thanks for the correction there, that /dev/nvidia* isn’t needed in [cgroup_allowed_devices_file.conf] for constraining GPU devices. -Kevin From: slurm-users on behalf of "R. Paul Wiegand" Reply-To: "p...@tesseract.org"

Re: [slurm-users] Getting a runtime percentage of % allocated on cluster

Just to help 'sreport cluster utilization' is close to what I am looking for. -Kevin From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Kevin Manalo <kman...@jhu.edu> Sent: Monday, February 5, 2018 4:28 PM To: slurm-users@lists

Re: [slurm-users] Getting a runtime percentage of % allocated on cluster

[slurm-users] Getting a runtime percentage of % allocated on cluster On 06/02/18 09:09, Kevin Manalo wrote: > Just to help 'sreport cluster utilization' is close to what I am looking > for. Does this help? # sreport -t percent cluster utilization cheers, Chris

Re: [slurm-users] Jobs escaping cgroup device controls after some amount of time.

I’m asking in the hopes that others will chime in (I’m curious why this is happening) Could you share your related slurm.conf cgroup options cgroup.conf cgroup_allowed_devices_file.conf TaskPlugin ProctrackType JobAcctGatherType -Kevin PS Looking for similar style jobs, We have >1 day gpu

Re: [slurm-users] Jobs escaping cgroup device controls after some amount of time.

Shawn, Just to give you a compare and contrast: We have for related entries slurm.conf JobAcctGatherType=jobacct_gather/linux # will migrate to cgroup eventually JobAcctGatherFrequency=30 ProctrackType=proctrack/cgroup TaskPlugin=task/affinity,task/cgroup cgroup_allowed_devices_file.conf:

Re: [slurm-users] Lua Job Submit - Setting Features/Constraints

desc.features == nil then job_desc.features = "special" else job_desc.features = job_desc.features .. ",special" end Bill On 12/19/2018 09:27 AM, Kevin Manalo wrote: > Is it

Re: [slurm-users] Lua Job Submit - Setting Features/Constraints

/www.nersc.gov> dmjacob...@lbl.gov<mailto:dmjacob...@lbl.gov> - __o -- _ '\<,_ --(_)/ (_)__ On Wed, Dec 19, 2018 at 5:49 AM Kevin Manalo mailto:kman...@jhu.edu>> wrote: All, Does anyone have an example of setting features (if no

[slurm-users] Lua Job Submit - Setting Features/Constraints

All, Does anyone have an example of setting features (if not set) in the Lua job submission scripts? job_desc.features There was a discussion here, but it appears to be for the case where it is checked and rejected https://groups.google.com/d/topic/slurm-users/C-oYERITK9c/discussion -Kevin