[slurm-users] can't create memory group (cgroup)

Hi everyone, I'm getting this error lately for everyone's jobs, which results in memory not being constrained via the cgroups plugin. slurmstepd: error: task/cgroup: unable to add task[pid=21681] to memory cg '(null)' slurmstepd: error: jobacct_gather/cgroup: unable to instanciate user 3691 m

Re: [slurm-users] can't create memory group (cgroup)

Thanks everyone for your responses. It looks like the two suggestions were: 1. add "cgroup_enable=memory swapaccount=1" to the kernel command by adding it to /etc/default/grub in the GRUB_CMDLIND_LINUX variable 2. Add ConstrainKmemSpace=no in cgroup.conf >From this information I think option 2

Re: [slurm-users] can't create memory group (cgroup)

allows. -Original Message- From: slurm-users [mailto:slurm-users-boun...@lists.schedmd.com] On Behalf Of Mike Cammilleri Sent: Monday, September 10, 2018 9:49 AM To: Slurm User Community List Subject: Re: [slurm-users] can't create memory group (cgroup) Thanks everyone for

Re: [slurm-users] Seff error with Slurm-18.08.1

e failed perl: error: cannot create accounting_storage context for accounting_storage/slurmdbd Job not found. Mike Cammilleri Systems Administrator Department of Statistics | UW-Madison 1300 University Ave | Room 1280 608-263-6673 | mi...@stat.wisc.edu F

Re: [slurm-users] Seff error with Slurm-18.08.1

uster perhaps the fix will be available then. Mike Cammilleri Systems Administrator Department of Statistics | UW-Madison 1300 University Ave | Room 1280 608-263-6673 | mi...@stat.wisc.edu From: slurm-users on behalf of Chris Samuel Sent: Tuesday, Novem

[slurm-users] Limit concurrent gpu resources

that they submit the jobs as an array and limit with %7, but maybe there's a more elegant solution using the config. Any tips appreciated. Mike Cammilleri Systems Administrator Department of Statistics | UW-Madison 1300 University Ave | Room 1280 608-263-6673 | mi...@stat.wisc.edu

[slurm-users] detectCores() mess

Hi, We have allowed some courses to use our slurm cluster for teaching purposes, which of course leads to all kinds of exciting experiments - not always the most clever programming but it certainly teaches me where we need tighten up configurations. The default method of thinking for many stud

Re: [slurm-users] detectCores() mess

..@lists.schedmd.com] On Behalf Of Chris Samuel Sent: Friday, December 8, 2017 6:46 PM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] detectCores() mess On 9/12/17 4:54 am, Mike Cammilleri wrote: > I thought cgroups (which we are using) would prevent some of this > beh

[slurm-users] Are these threads actually unused?

I posted a question similar to this a couple months ago regarding CPU utilization which we figured out - sometimes too many threads on one cpu creates high CPU load, and thus slower compute time because things are waiting. A more proper allocation should be set in the submit script (e.g. --cpu

Re: [slurm-users] Are these threads actually unused?

tasks. -Original Message- From: slurm-users [mailto:slurm-users-boun...@lists.schedmd.com] On Behalf Of Mike Cammilleri Sent: Tuesday, February 13, 2018 10:31 AM To: slurm-users@lists.schedmd.com Subject: [slurm-users] Are these threads actually unused? I posted a question similar to this