[slurm-dev] weird integration bewteen Spank and "--export"

[Manuel Rodríguez Pascual]

Hi all,

I am experiencing a strange behavior on a plugin that I created. i
don't know if this is what I should expect, so maybe you can provide
me with some insights.

I want to have some stuff done in slurm_spank_task_init, but ONLY if
sbatch/srun was executed with "--myplugin". To do so, I created a
global variable "activate_plugin_var" in my myplugin.c, which is set
in an auxiliary function "set_ativate_plugin_var". This funcion is
registered as a spank_option, so when the user uses "--myplugin" it is
called and this global variable activate_plugin_var  gets a value.
Then, in slurm_spank_task_init, I check the value of the variable and
if it is set, then I do whatever.

So far this works fine. I don't know however if this is the most
elegant solution though. Probably the best idea would do whatever in
the auxiliary set_ativate_plugin_var function  instead of employing
this flag. The problem is that I want to access the values of the
spank_t object received in slurm_spank_task_init and with the
registered function you don't have them, so that's why I came with
this approach. The main problem I see is that the state is not saved
"forever", so the value of any global variable is not always what
you'd expect.   If there is any better solution, I am open to
suggestions.

The weird situation is however that this funcionality is affected by
"--export" flag.

- if I don't use the export, it works as expected.
- if I use export with "export=ALL", it is OK too.
- if I use export with "export=NONE" or "export=myVar=myValue", then:
-- set_ativate_plugin_var is called
-- slurm_spank_task_init is called
-- slurm_spank_task_init does NOT have activate_plugin_var with the
correct value.

This causes that my plugin does not detect that the user used the
flag, and it does nothing. I would say that this is an inconsistency,
as the behavior of the plugin should not depend on an unrelated flag.

Reading the manual I saw that with the "export" options that cause an
error in my plugin "the --get-user-env option will implicitly be set".
I thought that this might be the cause of the problem, but additional
tests on that flag proved me wrong. I am now a bit lost, so any help
would be welcome.

I guess my question would be,

- have you noticed a similar behavior? Is it expected, or in fact an
inconsistency?
- if you have coded a similar spank plugin, have you got any
suggestion for its architecture?
- have you got any solution to my problem, maybe through a different
approach or whatever?


thanks,


Manuel

[slurm-dev] How to find the srun associated with jobid.stepid?

[Bob Moench]

Hi,

Does anyone know how to discover the hostid and pid of srun for
a given jobid.stepid? I see that sstat can provide the hostid,
but it doesn't seem to deliver the stepid. I need this in a
third party scenario so environment variables don't help me.

So far hunting through man pages and googling have only gotten
me the hostid.

Thanks,
Bob

--
Bob Moench (rwm); PE Debugger Development; 605-9034; 354-7895; SP 161221

[slurm-dev] Re: slurmctld not pinging at regular interval

[Lachlan Musicman]

On 17 February 2017 at 05:34, Allan Streib  wrote:

>
> I finally got some time to debug this a bit more. I used scontrol to
> turn on debug logging. I'm seeing messages like this:
>
>   [2017-02-16T13:18:52.599] debug:  Spawning ping agent for
> j-[001-050],t-[019-052]
>   [2017-02-16T13:18:52.599] debug:  Spawning registration agent for
> j-[101-128],r-[001-004],t-[001-018] 50 hosts
>   [2017-02-16T13:18:52.621] agent/is_node_resp: node:t-019 rpc:1008 :
> Can't find an address, check slurm.conf
>
> t-019 is one of my nodes that's frequently "down" according to slurm but
> really isn't. What is that "Can't find an address" about? DNS lookups
> seem to be working fine in a shell on the same machine.
>



I can't remember where I saw it and I can't see it in the docs any more,
but for whatever reason, I was under the impression that the machines in
the cluster required all hosts to be in the hostfile both as FQDN and
shortname?

That's how we deploy. DNS is working, but cluster members are listed in
/etc/hosts file too.

cheers
L.

--
The most dangerous phrase in the language is, "We've always done it this
way."

- Grace Hopper



> Sometimes I see larger groups of nodes showing the same message, but
> they quickly become responsive again:
>
>   [2017-02-16T13:27:12.540] debug:  Spawning ping agent for
> j-[051-079,094,097-100],t-[036-064]
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-036 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-037 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-038 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-039 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-040 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-042 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-041 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-043 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-044 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-045 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-046 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-047 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-048 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-049 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-050 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-051 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-052 rpc:1008 :
> Can't find an address, check slurm.conf
>   [2017-02-16T13:27:13.541] error: Nodes t-[036-052] not responding
>
> [...]
>
>   [2017-02-16T13:28:52.659] debug:  Spawning ping agent for
> j-[001-050,101-128],r-[001-004],t-[001-052]
>   [2017-02-16T13:28:52.694] Node t-036 now responding
>   [2017-02-16T13:28:52.694] Node t-044 now responding
>   [2017-02-16T13:28:52.695] Node t-042 now responding
>   [2017-02-16T13:28:52.695] Node t-047 now responding
>   [2017-02-16T13:28:52.695] Node t-037 now responding
>   [2017-02-16T13:28:52.695] Node t-046 now responding
>   [2017-02-16T13:28:52.695] Node t-049 now responding
>   [2017-02-16T13:28:52.695] Node t-045 now responding
>   [2017-02-16T13:28:52.695] Node t-052 now responding
>   [2017-02-16T13:28:52.695] Node t-039 now responding
>   [2017-02-16T13:28:52.695] Node t-050 now responding
>   [2017-02-16T13:28:52.695] Node t-038 now responding
>   [2017-02-16T13:28:52.695] Node t-048 now responding
>   [2017-02-16T13:28:52.695] Node t-043 now responding
>   [2017-02-16T13:28:52.695] Node t-040 now responding
>   [2017-02-16T13:28:52.695] Node t-041 now responding
>   [2017-02-16T13:28:52.695] Node t-051 now responding
>
> All the "t-nnn" nodes are on one cluster and these are the only nodes
> showing this problem.
>
> Thanks,
>
> Allan
>
> Allan Streib  writes:
>
> > They are all running ntpd and clocks are in sync.
> >
> > In this slurmctld there are a total of 226 nodes, in several different
> > partitions. The cluster of 64 is the only one where I see this
> > happening. Unless that number of nodes is pushing the limit for a single
> > slurmctld (which I doubt) I'd be inclined to think it's more likely a
> > network 

[slurm-dev] RE: Configuring slurm accounts

[Lachlan Musicman]

On 17 February 2017 at 03:02, Baker D.J.  wrote:

> Hello,
>
>
>
> Thank you for the reply. There are two accounts on this cluster. What I
> was primarily trying to do was define a default QOS with a partition. My
> idea was to use sacctmgr to create an association between the partition,
> the QOS and a group of users. In other words to place this group of users
> in the same account – with access to the gpu partition and a specific QOS.
> This does appear to work, however perhaps I don’t fully understand the
> logic. So, when I submit using…
>
>
>
> sbatch -A gpuusers -p gpus slurm.serial  -- that works as expected and the
> correct QOS is selected
>
>
>
> sbatch -A gpuusers slurm.serial  -- that fails since the cluster assumes
> that the default partition is called batch (and submission to batch are not
> allowed for users in “gpuusers”). So that does make sense, however I had
> hoped that the command..
>
>
>
> sacctmgr add user=djb1 account=gpuusers Partition=gpus
>
>
>
> would assign the “gpus” as the default partition for users in the
> “gpuusers” group. However, that idea doesn’t appear to work.. Does anyone
> please have any comments? Am I going about this in the wrong way?
>
>
>
> Best regards,
>
> David
>
>
>



 Remember: everything happens at the Association level. That tuple of
cluster, account, user and (optional) partition.

An Association can have a default QOS, and users can belong to multiple
associations differentiated by partition.

So if you create an Association with Account=gpuusers and partition=gpus
and DefaultQOS= (note: you can also give an association CPUlimits or TRES
limits or...) and then add users to that association, the users can "use"
that association by sending jobs to that partition.

Does that make sense? I can re articulate more eloquently after my second
coffee.

cheers
L.


--
The most dangerous phrase in the language is, "We've always done it this
way."

- Grace Hopper

[slurm-dev] Re: slurmctld not pinging at regular interval

[Allan Streib]

I finally got some time to debug this a bit more. I used scontrol to
turn on debug logging. I'm seeing messages like this:

  [2017-02-16T13:18:52.599] debug:  Spawning ping agent for 
j-[001-050],t-[019-052]
  [2017-02-16T13:18:52.599] debug:  Spawning registration agent for 
j-[101-128],r-[001-004],t-[001-018] 50 hosts
  [2017-02-16T13:18:52.621] agent/is_node_resp: node:t-019 rpc:1008 : Can't 
find an address, check slurm.conf

t-019 is one of my nodes that's frequently "down" according to slurm but
really isn't. What is that "Can't find an address" about? DNS lookups
seem to be working fine in a shell on the same machine.

Sometimes I see larger groups of nodes showing the same message, but
they quickly become responsive again:

  [2017-02-16T13:27:12.540] debug:  Spawning ping agent for 
j-[051-079,094,097-100],t-[036-064]
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-036 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-037 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-038 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-039 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-040 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-042 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-041 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-043 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-044 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-045 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-046 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-047 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-048 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-049 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-050 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-051 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:12.554] agent/is_node_resp: node:t-052 rpc:1008 : Can't 
find an address, check slurm.conf
  [2017-02-16T13:27:13.541] error: Nodes t-[036-052] not responding

[...]

  [2017-02-16T13:28:52.659] debug:  Spawning ping agent for 
j-[001-050,101-128],r-[001-004],t-[001-052]
  [2017-02-16T13:28:52.694] Node t-036 now responding
  [2017-02-16T13:28:52.694] Node t-044 now responding
  [2017-02-16T13:28:52.695] Node t-042 now responding
  [2017-02-16T13:28:52.695] Node t-047 now responding
  [2017-02-16T13:28:52.695] Node t-037 now responding
  [2017-02-16T13:28:52.695] Node t-046 now responding
  [2017-02-16T13:28:52.695] Node t-049 now responding
  [2017-02-16T13:28:52.695] Node t-045 now responding
  [2017-02-16T13:28:52.695] Node t-052 now responding
  [2017-02-16T13:28:52.695] Node t-039 now responding
  [2017-02-16T13:28:52.695] Node t-050 now responding
  [2017-02-16T13:28:52.695] Node t-038 now responding
  [2017-02-16T13:28:52.695] Node t-048 now responding
  [2017-02-16T13:28:52.695] Node t-043 now responding
  [2017-02-16T13:28:52.695] Node t-040 now responding
  [2017-02-16T13:28:52.695] Node t-041 now responding
  [2017-02-16T13:28:52.695] Node t-051 now responding

All the "t-nnn" nodes are on one cluster and these are the only nodes
showing this problem.

Thanks,

Allan

Allan Streib  writes:

> They are all running ntpd and clocks are in sync.
>
> In this slurmctld there are a total of 226 nodes, in several different
> partitions. The cluster of 64 is the only one where I see this
> happening. Unless that number of nodes is pushing the limit for a single
> slurmctld (which I doubt) I'd be inclined to think it's more likely a
> network issue but in that case I'd expect wireshark to show an attempt
> by slurmctld to contact the node and then no response. What I'm actually
> seeing is no traffic either way for these nodes, on the same interval as
> the others.
>
> Allan
>
> Lachlan Musicman  writes:
>
>> Check they are all in the same time or ntpd against the same server. I
>> found that the nodes that kept going down had the time out of sync.
>>
>> Cheers
>> L.
>>
>> --
>> The most dangerous phrase in the language is, "We've always done it this 
>> way."
>>
>> - Grace Hopper
>>
>> On 25 January 2017 at 05:49, Allan Streib  wrote:
>>
>> I have a 

[slurm-dev] RE: Slurm with sssd - limits help please

[John Hearns]

Thankyou!
In this case this option in slurm.conf did the trick:

PropagateResourceLimitsExcept=MEMLOCK

I

From: Guy Coates [mailto:guy.coa...@gmail.com]
Sent: 16 February 2017 17:05
To: slurm-dev 
Subject: [slurm-dev] RE: Slurm with sssd - limits help please

You should double check the pam config too:

https://slurm.schedmd.com/faq.html#pam

Thanks,

Guy

On 16 February 2017 at 15:17, John Hearns 
> wrote:
My answer is here maybe?

https://slurm.schedmd.com/faq.html#memlock

RTFM ?

From: John Hearns [mailto:john.hea...@xma.co.uk]
Sent: 16 February 2017 15:13
To: slurm-dev >
Subject: [slurm-dev] Slurm with sssd - limits help please

Looks like there are others out there using slurm with sssd authentication, 
based on a quick mailing list search.

Forgive me if I have not understood something here.
On the cluster I am configuring at the moment, looking at the slurm daemon on 
the compute nodes
it has the max locked memory limit set to unlimited.
/etc/security/limits.conf has soft and hard max locked memory set to unlimited 
for all users.

I tried to run an MPI job and got the familiar warning that there was not 
enough locked memory available..
Looked at /etc/security/limits.conf as I have been bitten by that one before…

I log in using an srun and sure enough:

max locked memory   (kbytes, -l) 64

It looks to me like the limits are being inherited from the /usr/sbin/sssd 
daemon
In return this is started by systemd….
(cue dark clouds, the howling of a wolf and the sound of thunder off-stage)
At the moment I am looking at how to increase limits with systemd

Please, someone tell me I’m an idiot and there is an easy way to do this.
I weep though – is system really losing us the dependable and well known ways 
of doing things with limits.conf ??
Any views or opinions presented in this email are solely those of the author 
and do not necessarily represent those of the company. Employees of XMA Ltd are 
expressly required not to make defamatory statements and not to infringe or 
authorise any infringement of copyright or any other legal right by email 
communications. Any such communication is contrary to company policy and 
outside the scope of the employment of the individual concerned. The company 
will not accept any liability in respect of such communication, and the 
employee responsible will be personally liable for any damages or other 
liability arising. XMA Limited is registered in England and Wales (registered 
no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, 
Wilford, Nottingham, NG11 7EP
Any views or opinions presented in this email are solely those of the author 
and do not necessarily represent those of the company. Employees of XMA Ltd are 
expressly required not to make defamatory statements and not to infringe or 
authorise any infringement of copyright or any other legal right by email 
communications. Any such communication is contrary to company policy and 
outside the scope of the employment of the individual concerned. The company 
will not accept any liability in respect of such communication, and the 
employee responsible will be personally liable for any damages or other 
liability arising. XMA Limited is registered in England and Wales (registered 
no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, 
Wilford, Nottingham, NG11 7EP



Any views or opinions presented in this email are solely those of the author 
and do not necessarily represent those of the company. Employees of XMA Ltd are 
expressly required not to make defamatory statements and not to infringe or 
authorise any infringement of copyright or any other legal right by email 
communications. Any such communication is contrary to company policy and 
outside the scope of the employment of the individual concerned. The company 
will not accept any liability in respect of such communication, and the 
employee responsible will be personally liable for any damages or other 
liability arising. XMA Limited is registered in England and Wales (registered 
no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, 
Wilford, Nottingham, NG11 7EP

[slurm-dev] RE: Slurm with sssd - limits help please

[Guy Coates]

You should double check the pam config too:

https://slurm.schedmd.com/faq.html#pam

Thanks,

Guy

On 16 February 2017 at 15:17, John Hearns  wrote:

> My answer is here maybe?
>
>
>
> https://slurm.schedmd.com/faq.html#memlock
>
>
>
> RTFM ?
>
>
>
> *From:* John Hearns [mailto:john.hea...@xma.co.uk]
> *Sent:* 16 February 2017 15:13
> *To:* slurm-dev 
> *Subject:* [slurm-dev] Slurm with sssd - limits help please
>
>
>
> Looks like there are others out there using slurm with sssd
> authentication, based on a quick mailing list search.
>
>
>
> Forgive me if I have not understood something here.
>
> On the cluster I am configuring at the moment, looking at the slurm daemon
> on the compute nodes
>
> it has the max locked memory limit set to unlimited.
>
> /etc/security/limits.conf has soft and hard max locked memory set to
> unlimited for all users.
>
>
>
> I tried to run an MPI job and got the familiar warning that there was not
> enough locked memory available..
>
> Looked at /etc/security/limits.conf as I have been bitten by that one
> before…
>
>
>
> I log in using an srun and sure enough:
>
>
>
> max locked memory   (kbytes, -l) 64
>
>
>
> It looks to me like the limits are being inherited from the /usr/sbin/sssd
> daemon
>
> In return this is started by systemd….
>
> (cue dark clouds, the howling of a wolf and the sound of thunder off-stage)
>
> At the moment I am looking at how to increase limits with systemd
>
>
>
> Please, someone tell me I’m an idiot and there is an easy way to do this.
>
> I weep though – is system really losing us the dependable and well known
> ways of doing things with limits.conf ??
>
> Any views or opinions presented in this email are solely those of the
> author and do not necessarily represent those of the company. Employees of
> XMA Ltd are expressly required not to make defamatory statements and not to
> infringe or authorise any infringement of copyright or any other legal
> right by email communications. Any such communication is contrary to
> company policy and outside the scope of the employment of the individual
> concerned. The company will not accept any liability in respect of such
> communication, and the employee responsible will be personally liable for
> any damages or other liability arising. XMA Limited is registered in
> England and Wales (registered no. 2051703). Registered Office: Wilford
> Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP
> Any views or opinions presented in this email are solely those of the
> author and do not necessarily represent those of the company. Employees of
> XMA Ltd are expressly required not to make defamatory statements and not to
> infringe or authorise any infringement of copyright or any other legal
> right by email communications. Any such communication is contrary to
> company policy and outside the scope of the employment of the individual
> concerned. The company will not accept any liability in respect of such
> communication, and the employee responsible will be personally liable for
> any damages or other liability arising. XMA Limited is registered in
> England and Wales (registered no. 2051703). Registered Office: Wilford
> Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP
>

[slurm-dev] RE: Configuring slurm accounts

[Baker D . J .]

Hello,

Thank you for the reply. There are two accounts on this cluster. What I was 
primarily trying to do was define a default QOS with a partition. My idea was 
to use sacctmgr to create an association between the partition, the QOS and a 
group of users. In other words to place this group of users in the same account 
– with access to the gpu partition and a specific QOS. This does appear to 
work, however perhaps I don’t fully understand the logic. So, when I submit 
using…

sbatch -A gpuusers -p gpus slurm.serial  -- that works as expected and the 
correct QOS is selected

sbatch -A gpuusers slurm.serial  -- that fails since the cluster assumes that 
the default partition is called batch (and submission to batch are not allowed 
for users in “gpuusers”). So that does make sense, however I had hoped that the 
command..

sacctmgr add user=djb1 account=gpuusers Partition=gpus

would assign the “gpus” as the default partition for users in the “gpuusers” 
group. However, that idea doesn’t appear to work.. Does anyone please have any 
comments? Am I going about this in the wrong way?

Best regards,
David


From: Lachlan Musicman [mailto:data...@gmail.com]
Sent: Wednesday, February 15, 2017 9:57 PM
To: slurm-dev 
Subject: [slurm-dev] RE: Configuring slurm accounts

If you are only in one account, you don't need to list it.
What version of slurm are you using? Someone else mentioned needing to restart 
slurmctld to users to stick. Which is not something I've experienced, but try 
that maybe?
I am presuming that your slurm.conf is set up correctly for accounts?
I tend to use user= rather than user 
sacctmgr add user=djb1 account=gpuusers Partition=gpus

Do you see your user in:
sacctmgr list associations

?
cheers
L.

--
The most dangerous phrase in the language is, "We've always done it this way."

- Grace Hopper

On 16 February 2017 at 01:41, Baker D.J. 
> wrote:
PS.. please note the typo in my sbatch command.. it should be….

sbatch -A gpuusers slurm.serial

David

From: Baker D.J.
Sent: Wednesday, February 15, 2017 2:32 PM
To: slurm-dev >
Subject: Configuring slurm accounts

Hello,

I’m attempting to configure a slurm account and I suspect I’m missing something 
fundamental. As a test I’ve created a partition called “gpus”, a QOS called 
“fast” and an account called “gpuusers”. I’m attempting to ensure that the 
member of the account, gpuuser, only have access to the partition, gpus, with 
QOS fast. So this is what I tried..

sacctmgr -i add qos name=fast …
sacctmgr create account gpuusers DefaultQOS=fast
sacctmgr modify account gpuusers set qoslevel=fast
sacctmgr add user djb1 account=gpuusers Partition=gpus

I’m I correct in thinking that those associations  limit members of the account 
to the partition gpus with QOS=fast or have I made a fundamental error? When I 
try to submit a job as djb1, for example “sbatch –q gpuusers slurm.serial”, I 
find….

sbatch: error: Batch job submission failed: Invalid account or 
account/partition combination specified

Any help with this would be appreciated, please.

Best regards,
David

[slurm-dev] TRESBillingWeights Node billing

[Tomislav Subic]

Hi everyone,

I have a question regarding the TRESBillingWeights option and the 
calculation of used resources. I get how it works, and for things like 
CPU and memory the behavior is expected, as described in the docs. In 
our setup we would like to track Node-hours for each user.


I configured the partition like this: 
TRESBillingWeights="CPU=0.0,Node=1.0,Mem=0.0G"


I expected that for a 1 minute job on the whole node, there would be 
added 1 TRES to the users used resources, but instead 28 is added, which 
is the number of cores on the node. Is this how it is intended to be, or 
is it a bug? The docs don't mention anything about how the node option 
is billed.


Thanks

--
Tomislav Šubić
Arctur d.o.o.

[slurm-dev] RE: Slurm with sssd - limits help please

[John Hearns]

My answer is here maybe?

https://slurm.schedmd.com/faq.html#memlock

RTFM ?

From: John Hearns [mailto:john.hea...@xma.co.uk]
Sent: 16 February 2017 15:13
To: slurm-dev 
Subject: [slurm-dev] Slurm with sssd - limits help please

Looks like there are others out there using slurm with sssd authentication, 
based on a quick mailing list search.

Forgive me if I have not understood something here.
On the cluster I am configuring at the moment, looking at the slurm daemon on 
the compute nodes
it has the max locked memory limit set to unlimited.
/etc/security/limits.conf has soft and hard max locked memory set to unlimited 
for all users.

I tried to run an MPI job and got the familiar warning that there was not 
enough locked memory available..
Looked at /etc/security/limits.conf as I have been bitten by that one before…

I log in using an srun and sure enough:

max locked memory   (kbytes, -l) 64

It looks to me like the limits are being inherited from the /usr/sbin/sssd 
daemon
In return this is started by systemd….
(cue dark clouds, the howling of a wolf and the sound of thunder off-stage)
At the moment I am looking at how to increase limits with systemd

Please, someone tell me I’m an idiot and there is an easy way to do this.
I weep though – is system really losing us the dependable and well known ways 
of doing things with limits.conf ??
Any views or opinions presented in this email are solely those of the author 
and do not necessarily represent those of the company. Employees of XMA Ltd are 
expressly required not to make defamatory statements and not to infringe or 
authorise any infringement of copyright or any other legal right by email 
communications. Any such communication is contrary to company policy and 
outside the scope of the employment of the individual concerned. The company 
will not accept any liability in respect of such communication, and the 
employee responsible will be personally liable for any damages or other 
liability arising. XMA Limited is registered in England and Wales (registered 
no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, 
Wilford, Nottingham, NG11 7EP
Any views or opinions presented in this email are solely those of the author 
and do not necessarily represent those of the company. Employees of XMA Ltd are 
expressly required not to make defamatory statements and not to infringe or 
authorise any infringement of copyright or any other legal right by email 
communications. Any such communication is contrary to company policy and 
outside the scope of the employment of the individual concerned. The company 
will not accept any liability in respect of such communication, and the 
employee responsible will be personally liable for any damages or other 
liability arising. XMA Limited is registered in England and Wales (registered 
no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, 
Wilford, Nottingham, NG11 7EP

[slurm-dev] Slurm with sssd - limits help please

[John Hearns]

Looks like there are others out there using slurm with sssd authentication, 
based on a quick mailing list search.

Forgive me if I have not understood something here.
On the cluster I am configuring at the moment, looking at the slurm daemon on 
the compute nodes
it has the max locked memory limit set to unlimited.
/etc/security/limits.conf has soft and hard max locked memory set to unlimited 
for all users.

I tried to run an MPI job and got the familiar warning that there was not 
enough locked memory available..
Looked at /etc/security/limits.conf as I have been bitten by that one before...

I log in using an srun and sure enough:

max locked memory   (kbytes, -l) 64

It looks to me like the limits are being inherited from the /usr/sbin/sssd 
daemon
In return this is started by systemd
(cue dark clouds, the howling of a wolf and the sound of thunder off-stage)
At the moment I am looking at how to increase limits with systemd

Please, someone tell me I'm an idiot and there is an easy way to do this.
I weep though - is system really losing us the dependable and well known ways 
of doing things with limits.conf ??
Any views or opinions presented in this email are solely those of the author 
and do not necessarily represent those of the company. Employees of XMA Ltd are 
expressly required not to make defamatory statements and not to infringe or 
authorise any infringement of copyright or any other legal right by email 
communications. Any such communication is contrary to company policy and 
outside the scope of the employment of the individual concerned. The company 
will not accept any liability in respect of such communication, and the 
employee responsible will be personally liable for any damages or other 
liability arising. XMA Limited is registered in England and Wales (registered 
no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, 
Wilford, Nottingham, NG11 7EP

[slurm-dev] RE: A little bit help from my slurm-friends

[Marcin Stolarek]

You can also have a submit plugin that will put job in multiple partition
if non specified. This should reduce the drawback of multiple partitions.

However I  think that with features and topology plugin you should be able
to aviod multiple partitions setup.

cheers
Marcin

2017-01-17 9:49 GMT+01:00 David WALTER :

>
> Thanks Paul for your response and your advices.
>
> That's actually the reason why they asked me to set 3 and now 4
> partitions. As we have now 4 different generation of nodes with significant
> differences of hardware (not the same CPU, not the same amount of RAM) we
> thought that it was a good solution.
>
> I will test with people to adjust the solution with the needs of the many.
>
> Thanks again
>
> --
> David WALTER
> The computer guy
> david.wal...@ens.fr
> 01/44/32/27/94
>
> INSERM U960
> Laboratoire de Neurosciences Cognitives
> Ecole Normale Supérieure
> 29, rue d'Ulm
> 75005 Paris
>
> -Message d'origine-
> De : Paul Edmon [mailto:ped...@cfa.harvard.edu]
> Envoyé : lundi 16 janvier 2017 16:37
> À : slurm-dev
> Objet : [slurm-dev] RE: A little bit help from my slurm-friends
>
>
> I agree having multiple partitions will decrease efficiency of the
> scheduler.  That said if you have to do it, you have to do it.  Using the
> features is a good way to go if people need specific ones.  I could see
> having multiple partitions so you can charge differently for each
> generation of hardware, as run times will invariably be different.
> Still if that isn't a concern just have a single queue.
>
> For multifactor I would turn on fairshare and age.  JobSize really isn't
> useful unless you have people running multicore jobs and you want to
> prioritize, or deprioritize those.
>
> If you end up in a multipartition scenario then I recommend having a
> backfill queue that underlies all the partitions and setting up REQUEUE on
> that partition.  That way people can farm idle cycles.  This is especially
> good for people who are hardware agnostic and don't really care when their
> jobs get done but rather just have a ton to do that can be interrupted at
> any moment. That's what we do here and we have 110 partitions.  Our
> backfill queue does a pretty good job up picking up the idle cores but
> still there is structural inefficiencies with that many partitions so we
> never get above about 70% usage of our hardware.
>
> So just keep that in mind when you are setting things up.  More partitions
> means more structural inefficiency but it does give you other benefits such
> as isolating hardware for specific use.  It really depends on what you
> need.  I highly recommend experimenting to figure out what fits you and
> your users best.
>
> -Paul Edmon-
>
> On 1/16/2017 10:16 AM, Loris Bennett wrote:
> > David WALTER  writes:
> >
> >> Dear Loris,
> >>
> >> Thanks for your response !
> >>
> >> I'm going to look on this features in slurm.conf.  I only configured
> >> the CPUs, Sockets per node. Do you have any example or link to
> >> explain me how it's working and what can I use ?
> > It's not very complicated.  A feature is just a label, so if you had
> > some nodes with Intel processors and some with AMD, you could attach
> > the features, e.g.
> >
> > NodeName=node[001,002] Procs=12 Sockets=2 CoresPerSocket=6
> > ThreadsPerCore=1 RealMemory=42000 State=unknown Feature=intel
> > NodeName=node[003,004] Procs=12 Sockets=2 CoresPerSocket=6
> > ThreadsPerCore=1 RealMemory=42000 State=unknown Feature=amd
> >
> > Users then just request the required CPU type in their batch scripts
> > as a constraint, e.g:
> >
> > #SBATCH --constraint="intel"
> >
> >> My goal is to respond to people needs and launch their jobs as fast
> >> as possible without losing time when one partition is idle whereas
> >> the others are fully loaded.
> > The easiest way to avoid the problem you describe is to just have one
> > partition.  If you have multiple partitions, the users have to
> > understand what the differences are so that they can choose sensibly.
> >
> >> That's why I thought the fair share factor was the best solution
> > Fairshare won't really help you with the problem that one partition
> > might be full while another is empty.  It will just affect the
> > ordering of jobs in the full partition, although the weight of the
> > partition term in the priority expression can affect the relative
> > attractiveness of the partitions.
> >
> > In general, however, I would suggest you start with a simple set-up.
> > You can always add to it later to address specific issues as they arise.
> > For instance, you could start with one partition and two QOS: one for
> > normal jobs and one for test jobs.  The latter could have a higher
> > priority, but only a short maximum run-time and possibly a low maximum
> > number of jobs per user.
> >
> > Cheers,
> >
> > Loris
> >
>

[slurm-dev] Re: Standard suspend/resume scripts?

[Marcin Stolarek]

I don't think that there is something good for everyone. It depends on the
way you manage the cluster. If you have diskless/stateless nodes it can be
also good to force power off with ipmitool.

2017-02-16 8:19 GMT+01:00 Loris Bennett :

>
> Lachlan Musicman  writes:
>
> > Re: [slurm-dev] Standard suspend/resume scripts?
> >
> > If you are looking to suspend and resume jobs, use scontrol:
> >
> > scontrol suspend 
> > scontrol resume 
> >
> > https://slurm.schedmd.com/scontrol.html
> >
> > The docs you are pointing to look more like taking nodes offline in
> times of low usage?
>
> Yes, because that's what I'm interested in ;-)
>
> Cheers,
>
> Loris
>
> --
> Dr. Loris Bennett (Mr.)
> ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
>