We manage about 800 accounts in ours, so an order of magnitude smaller.
We use the lua plugin at submission time. In it the lau script checks
if the user has an account in slurm (the script itself keeps a cache so
as not to pound the database with this query). If not it will make one
with the proper associations. We key off the users primary group id for
their default association. Thus the users account is created when they
submit their first job.
This should spread out the load, unless of course you have all your
users submitting simultaneously.
-Paul Edmon-
On 12/08/2016 09:05 PM, Tuo Chen Peng wrote:
Hello,
Is there a guideline / best practice for
(1) maintain huge number of user accounts in slurm?
(2) adding thousands of user in batch?
We are maintaining a slurm cluster with 10k+ user accounts.
But recently we need to add thousands more of users and noticed that
adding new users could sometimes cause slurmctld to hang or ignore
user requests for several minutes
(users are currently added one-by-one by using sacctmgr add user)
By enabling verbose logging in slurmctld (‘debug5’) I can see
slurmctld does 2 things when a new user is added
(1) walk the account/user tree
(2) recalculate normalized shares for every other user
I’m guessing these steps takes time, and with usual scheduling work
they hanged slurmctld in our case.
This might be ok if we just need to add a handful of user, but we do
need to add thousands of user to slurm
So a few minutes of hang in slurm for each user add means we can only
add 1 user every 10 minutes to avoid blocking user from accessing slurm
And therefore will take 1 day to add 120 users, and 8+ days to add
1000 users for example
Which works, but seems very inefficient
And it doesn’t seem right, because with more users added, the
normalized shares calculation will only take more time.
So, does anyone know what might be the efficient way to add huge
number of users to slurm and maintain them?
TuoChen Peng
------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s)
and may contain confidential information. Any unauthorized review,
use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and
destroy all copies of the original message.
------------------------------------------------------------------------
