Re: [slurm-users] SLURM upgrade from 20.11.3 to 20.11.9 misidentification of job steps

2022-05-19 Thread John DeSantis 

Luke,


We ran into a similar issue a while ago (not sure what versions were involved 
though). Can't guarantee it's relevant, but have a look adding:

JobAcctGatherParams=NoShared,UsePSS


Thanks for your response!

I did adjust the values to include those listed above.  Unfortunately, the 
problem remained. Long story short, I upgraded to the latest stable version 
this morning and the issue appears resolved.

Thanks!
John DeSantis


On 5/19/22 05:41, Luke Sudbery wrote:

We ran into a similar issue a while ago (not sure what versions were involved 
though). Can't guarantee it's relevant, but have a look adding:

JobAcctGatherParams=NoShared,UsePSS

https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D9010%23c16data=05%7C01%7Cdesantis%40usf.edu%7Ce6f55a62938f4533f18108da397c162e%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C637885502376090312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=Ke2a48S60O32Oxi2K2yBshQFkmjkqFU3eXOkRCf1WVM%3Dreserved=0

By default multiple processes using shared memory get counted multiple times - 
but it's not actually using that much RAM.

Cheers,

Luke

--
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road

Please note I don’t work on Monday.

-Original Message-
From: slurm-users  On Behalf Of John 
DeSantis
Sent: 18 May 2022 15:39
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] SLURM upgrade from 20.11.3 to 20.11.9 
misidentification of job steps

Hello,

It also appears that random jobs are being identified as using too much memory, 
despite being well within limits.

For example, a job is running that requested 2048 MB per CPU and all processes 
are within the limit.  But, the job is identified as being over limit when it 
isn't.  Please see below:


[2022-05-18T10:26:45.020] Job 7724384 exceeded memory limit (13140>2056), 
cancelling it
[2022-05-18T10:26:53.736] Job 7724384 exceeded memory limit (13140>2056), 
cancelling it



PID PSR   RSSVSZ ELAPSED COMMAND
  16893   1 844172 1337540 15:53 vasp
  16885   0 842844 1337424 15:53 vasp
  16892  14 841216 1337652 15:53 vasp
  16888   6 840848 1337540 15:53 vasp
  16896   7 840756 1337536 15:53 vasp
  16891  12 840664 1337544 15:53 vasp
  16898  11 840556 1337540 15:53 vasp
  16900  15 840512 1337656 15:53 vasp
  16890  10 840500 1337540 15:53 vasp
  16889   8 840468 1337536 15:53 vasp
  16895   5 840416 1337544 15:53 vasp
  16886   2 840356 1337540 15:53 vasp
  16887   4 840348 1337540 15:53 vasp
  16897   9 840288 1337544 15:53 vasp
  16894   3 840128 1337536 15:53 vasp
  16899  13 839952 1337544 15:53 vasp



ps -U USER -o pid,psr,rss,vsz,etime,args h --sort=-rss|awk '{rss+=$3} END 
{printf("%.2f\n",rss/1024)}'
13140.52


This job isn't hasn't exceeded its memory request on a per CPU basis, yet it 
has been identified as having done so.  Unfortunately, this will cause concern 
with users because their output logs indicate that the job exceeded its memory 
limit.

I ran a few test jobs requesting 1K of memory and they were all terminated and 
identified correctly, so this new behavior is random and frequent.

The commonality between both properly and improperly misidentified jobs is that 
they are not terminated and their output files log the message from slurmstepd 
that the memory limit has been exceeded.

Are any other users on the 20.11.9 branch experiencing this issue?  Are any 
users on the 21.08.8 branch experiencing this issue?  We use 
JobAcctGatherParams=OverMemoryKill in our environment to monitor and kill jobs 
when the physical memory limit has been exceeded.

Thank you,
John DeSantis

On 5/18/22 09:45, John DeSantis wrote:

Hello,

Due to the recent CVE posted by Tim, we did upgrade from SLURM 20.11.3 to 
20.11.9.

Today, I received a ticket from a user with their output files populated with the 
"slurmstepd: error: Exceeded job memory limit" message.  But, the jobs are 
still running and it seems that the controller is misidentifying the job and/or step ID.  
Please see below.

# slurmd log


[2022-05-18T09:33:31.279] Job 7733409 exceeded memory limit (7973>5120), 
cancelling it
[2022-05-18T09:33:31.291] debug:  _rpc_job_notify, uid = 65536, JobId=7733409
[2022-05-18T09:33:31.291] [7733409.0] debug:  Handling REQUEST_STEP_UID
[2022-05-18T09:33:31.300] send notification to StepId=7733409.batch
[2022-05-18T09:33:31.300] [7733409.batch] debug:  Handling REQUEST_JOB_NOTIFY
[2022-05-18T09:33:31.302] [7733409.batch] error: Exceeded job memory limit


# controller log


[2022-05-18T09:33:31.293] debug2: Processing RPC: REQUEST_CANCEL_JOB_STEP from 
UID=0
[2022-05-18T09:33:31.293] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.77334

Re: [slurm-users] SLURM upgrade from 20.11.3 to 20.11.9 misidentification of job steps

2022-05-19 Thread Luke Sudbery 
We ran into a similar issue a while ago (not sure what versions were involved 
though). Can't guarantee it's relevant, but have a look adding:

JobAcctGatherParams=NoShared,UsePSS

https://bugs.schedmd.com/show_bug.cgi?id=9010#c16

By default multiple processes using shared memory get counted multiple times - 
but it's not actually using that much RAM.

Cheers,

Luke

-- 
Luke Sudbery
Principal Engineer (HPC and Storage).
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road

Please note I don’t work on Monday.

-Original Message-
From: slurm-users  On Behalf Of John 
DeSantis
Sent: 18 May 2022 15:39
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] SLURM upgrade from 20.11.3 to 20.11.9 
misidentification of job steps

Hello,

It also appears that random jobs are being identified as using too much memory, 
despite being well within limits.

For example, a job is running that requested 2048 MB per CPU and all processes 
are within the limit.  But, the job is identified as being over limit when it 
isn't.  Please see below:

> [2022-05-18T10:26:45.020] Job 7724384 exceeded memory limit (13140>2056), 
> cancelling it
> [2022-05-18T10:26:53.736] Job 7724384 exceeded memory limit (13140>2056), 
> cancelling it

>PID PSR   RSSVSZ ELAPSED COMMAND
>  16893   1 844172 1337540 15:53 vasp
>  16885   0 842844 1337424 15:53 vasp
>  16892  14 841216 1337652 15:53 vasp
>  16888   6 840848 1337540 15:53 vasp
>  16896   7 840756 1337536 15:53 vasp
>  16891  12 840664 1337544 15:53 vasp
>  16898  11 840556 1337540 15:53 vasp
>  16900  15 840512 1337656 15:53 vasp
>  16890  10 840500 1337540 15:53 vasp
>  16889   8 840468 1337536 15:53 vasp
>  16895   5 840416 1337544 15:53 vasp
>  16886   2 840356 1337540 15:53 vasp
>  16887   4 840348 1337540 15:53 vasp
>  16897   9 840288 1337544 15:53 vasp
>  16894   3 840128 1337536 15:53 vasp
>  16899  13 839952 1337544 15:53 vasp

> ps -U USER -o pid,psr,rss,vsz,etime,args h --sort=-rss|awk '{rss+=$3} END 
> {printf("%.2f\n",rss/1024)}'
> 13140.52

This job isn't hasn't exceeded its memory request on a per CPU basis, yet it 
has been identified as having done so.  Unfortunately, this will cause concern 
with users because their output logs indicate that the job exceeded its memory 
limit.

I ran a few test jobs requesting 1K of memory and they were all terminated and 
identified correctly, so this new behavior is random and frequent.

The commonality between both properly and improperly misidentified jobs is that 
they are not terminated and their output files log the message from slurmstepd 
that the memory limit has been exceeded.

Are any other users on the 20.11.9 branch experiencing this issue?  Are any 
users on the 21.08.8 branch experiencing this issue?  We use 
JobAcctGatherParams=OverMemoryKill in our environment to monitor and kill jobs 
when the physical memory limit has been exceeded.

Thank you,
John DeSantis

On 5/18/22 09:45, John DeSantis wrote:
> Hello,
> 
> Due to the recent CVE posted by Tim, we did upgrade from SLURM 20.11.3 to 
> 20.11.9.
> 
> Today, I received a ticket from a user with their output files populated with 
> the "slurmstepd: error: Exceeded job memory limit" message.  But, the jobs 
> are still running and it seems that the controller is misidentifying the job 
> and/or step ID.  Please see below.
> 
> # slurmd log
> 
>> [2022-05-18T09:33:31.279] Job 7733409 exceeded memory limit (7973>5120), 
>> cancelling it
>> [2022-05-18T09:33:31.291] debug:  _rpc_job_notify, uid = 65536, JobId=7733409
>> [2022-05-18T09:33:31.291] [7733409.0] debug:  Handling REQUEST_STEP_UID
>> [2022-05-18T09:33:31.300] send notification to StepId=7733409.batch
>> [2022-05-18T09:33:31.300] [7733409.batch] debug:  Handling REQUEST_JOB_NOTIFY
>> [2022-05-18T09:33:31.302] [7733409.batch] error: Exceeded job memory limit
> 
> # controller log
> 
>> [2022-05-18T09:33:31.293] debug2: Processing RPC: REQUEST_CANCEL_JOB_STEP 
>> from UID=0
>> [2022-05-18T09:33:31.293] STEPS: Processing RPC details: 
>> REQUEST_CANCEL_JOB_STEP StepId=4367416.7733409+0
>> [2022-05-18T09:33:31.293] kill_job_step: invalid JobId=4367416
>> [2022-05-18T09:33:31.293] debug2: slurm_send_timeout: Socket no longer there
> 
> A restart of the controller doesn't help either, as there are a log of 
> misidentified jobs (truncated):
> 
>> [2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
>> REQUEST_CANCEL_JOB_STEP StepId=4367416.7731668+0
>> [2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
>> REQUEST_CANCEL_JOB_STEP StepId=4367416.7731684+0
>> [2022-05-18T09:

Re: [slurm-users] SLURM upgrade from 20.11.3 to 20.11.9 misidentification of job steps

2022-05-18 Thread John DeSantis 

Hello,

It also appears that random jobs are being identified as using too much memory, 
despite being well within limits.

For example, a job is running that requested 2048 MB per CPU and all processes 
are within the limit.  But, the job is identified as being over limit when it 
isn't.  Please see below:


[2022-05-18T10:26:45.020] Job 7724384 exceeded memory limit (13140>2056), 
cancelling it
[2022-05-18T10:26:53.736] Job 7724384 exceeded memory limit (13140>2056), 
cancelling it



   PID PSR   RSSVSZ ELAPSED COMMAND
 16893   1 844172 1337540 15:53 vasp
 16885   0 842844 1337424 15:53 vasp
 16892  14 841216 1337652 15:53 vasp
 16888   6 840848 1337540 15:53 vasp
 16896   7 840756 1337536 15:53 vasp
 16891  12 840664 1337544 15:53 vasp
 16898  11 840556 1337540 15:53 vasp
 16900  15 840512 1337656 15:53 vasp
 16890  10 840500 1337540 15:53 vasp
 16889   8 840468 1337536 15:53 vasp
 16895   5 840416 1337544 15:53 vasp
 16886   2 840356 1337540 15:53 vasp
 16887   4 840348 1337540 15:53 vasp
 16897   9 840288 1337544 15:53 vasp
 16894   3 840128 1337536 15:53 vasp
 16899  13 839952 1337544 15:53 vasp



ps -U USER -o pid,psr,rss,vsz,etime,args h --sort=-rss|awk '{rss+=$3} END 
{printf("%.2f\n",rss/1024)}'
13140.52


This job isn't hasn't exceeded its memory request on a per CPU basis, yet it 
has been identified as having done so.  Unfortunately, this will cause concern 
with users because their output logs indicate that the job exceeded its memory 
limit.

I ran a few test jobs requesting 1K of memory and they were all terminated and 
identified correctly, so this new behavior is random and frequent.

The commonality between both properly and improperly misidentified jobs is that 
they are not terminated and their output files log the message from slurmstepd 
that the memory limit has been exceeded.

Are any other users on the 20.11.9 branch experiencing this issue?  Are any 
users on the 21.08.8 branch experiencing this issue?  We use 
JobAcctGatherParams=OverMemoryKill in our environment to monitor and kill jobs 
when the physical memory limit has been exceeded.

Thank you,
John DeSantis

On 5/18/22 09:45, John DeSantis wrote:

Hello,

Due to the recent CVE posted by Tim, we did upgrade from SLURM 20.11.3 to 
20.11.9.

Today, I received a ticket from a user with their output files populated with the 
"slurmstepd: error: Exceeded job memory limit" message.  But, the jobs are 
still running and it seems that the controller is misidentifying the job and/or step ID.  
Please see below.

# slurmd log


[2022-05-18T09:33:31.279] Job 7733409 exceeded memory limit (7973>5120), 
cancelling it
[2022-05-18T09:33:31.291] debug:  _rpc_job_notify, uid = 65536, JobId=7733409
[2022-05-18T09:33:31.291] [7733409.0] debug:  Handling REQUEST_STEP_UID
[2022-05-18T09:33:31.300] send notification to StepId=7733409.batch
[2022-05-18T09:33:31.300] [7733409.batch] debug:  Handling REQUEST_JOB_NOTIFY
[2022-05-18T09:33:31.302] [7733409.batch] error: Exceeded job memory limit


# controller log


[2022-05-18T09:33:31.293] debug2: Processing RPC: REQUEST_CANCEL_JOB_STEP from 
UID=0
[2022-05-18T09:33:31.293] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7733409+0
[2022-05-18T09:33:31.293] kill_job_step: invalid JobId=4367416
[2022-05-18T09:33:31.293] debug2: slurm_send_timeout: Socket no longer there


A restart of the controller doesn't help either, as there are a log of 
misidentified jobs (truncated):


[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731668+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731684+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731625+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731634+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731629+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724380+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724380+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731632+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724375+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731650+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7728855+0
[2022-05-18T09:41:27.130] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731681+0
[2022-05-18T09:41:27.130] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731651+0
[2022-05-18T09:41:27.131] STEPS: Processing RPC details:

[slurm-users] SLURM upgrade from 20.11.3 to 20.11.9 misidentification of job steps

2022-05-18 Thread John DeSantis 

Hello,

Due to the recent CVE posted by Tim, we did upgrade from SLURM 20.11.3 to 
20.11.9.

Today, I received a ticket from a user with their output files populated with the 
"slurmstepd: error: Exceeded job memory limit" message.  But, the jobs are 
still running and it seems that the controller is misidentifying the job and/or step ID.  
Please see below.

# slurmd log


[2022-05-18T09:33:31.279] Job 7733409 exceeded memory limit (7973>5120), 
cancelling it
[2022-05-18T09:33:31.291] debug:  _rpc_job_notify, uid = 65536, JobId=7733409
[2022-05-18T09:33:31.291] [7733409.0] debug:  Handling REQUEST_STEP_UID
[2022-05-18T09:33:31.300] send notification to StepId=7733409.batch
[2022-05-18T09:33:31.300] [7733409.batch] debug:  Handling REQUEST_JOB_NOTIFY
[2022-05-18T09:33:31.302] [7733409.batch] error: Exceeded job memory limit


# controller log


[2022-05-18T09:33:31.293] debug2: Processing RPC: REQUEST_CANCEL_JOB_STEP from 
UID=0
[2022-05-18T09:33:31.293] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7733409+0
[2022-05-18T09:33:31.293] kill_job_step: invalid JobId=4367416
[2022-05-18T09:33:31.293] debug2: slurm_send_timeout: Socket no longer there


A restart of the controller doesn't help either, as there are a log of 
misidentified jobs (truncated):


[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731668+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731684+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731625+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731634+0
[2022-05-18T09:41:27.128] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731629+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724380+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724380+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731632+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724375+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731650+0
[2022-05-18T09:41:27.129] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7728855+0
[2022-05-18T09:41:27.130] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731681+0
[2022-05-18T09:41:27.130] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7731651+0
[2022-05-18T09:41:27.131] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724380+0
[2022-05-18T09:41:27.131] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7728855+0
[2022-05-18T09:41:27.133] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724378+0
[2022-05-18T09:41:27.133] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724380+0
[2022-05-18T09:41:27.134] STEPS: Processing RPC details: 
REQUEST_CANCEL_JOB_STEP StepId=4367416.7724378+0


These jobs were started post upgrade, too.

Has anyone else seen this?

Thank you,
John DeSantis


OpenPGP_signature
Description: OpenPGP digital signature

[slurm-users] Slurm upgrade to 20.11.3, slurmdbd still trying to start old version 20.02.3

2021-03-03 Thread Robert Kudyba 
Slurmdbd has an issue and from the logs is still trying to load the old
version:
[2021-01-22T14:17:18.430] MySQL server version is: 5.5.68-MariaDB
[2021-01-22T14:17:18.433] error: Database settings not recommended values:
innodb_buffer_pool_size innodb_log_file_size innodb_lock_wait_timeout
[2021-01-22T14:17:18.528] Accounting storage MYSQL plugin loaded
[2021-01-22T14:17:18.529] error: chdir(/var/log): Permission denied
[2021-01-22T14:17:18.529] chdir to /var/tmp

*[2021-01-22T14:17:18.531] slurmdbd version 20.02.3
started[2021-01-22T14:56:40.334] error: g_slurm_auth_unpack: remote
plugin_id 144 not found*
[2021-01-22T14:56:40.334] error: slurm_unpack_received_msg: Invalid
Protocol Version 9216 from uid=-1 from problem connection: Socket operation
on non-socket
[2021-01-22T14:56:40.334]* error: slurm_unpack_received_msg: Incompatible
versions of client and server code*
[2021-01-22T14:56:40.345] error: CONN:7 Failed to unpack SLURM_PERSIST_INIT
message
[2021-03-03T09:49:57.607] Terminate signal (SIGINT or SIGTERM) received
[2021-03-03T09:49:57.610] Unable to remove pidfile '/var/run/slurmdbd.pid':
Permission denied

But I know it's updated:
rpm -qa|grep slurmdbd
slurm20-slurmdbd-20.11.3-mybuild.x86_64

And the pid file is not there:
ls -l /var/run/slurmdbd.pid
ls: cannot access /var/run/slurmdbd.pid: No such file or directory

And on the service file:
cat /usr/lib/systemd/system/slurmdbd.service
[Unit]
RequiresMountsFor=/cm/shared
Description=Slurm DBD accounting daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurmdbd.conf

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/slurmdbd
*ExecStart=/cm/shared/apps/slurm/20.11.3/sbin/slurmdbd -D $SLURMDBD_OPTIONS*
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536

I reinstalled the slurmdbd file that is local:
Dependencies Resolved

==
 PackageArch
  Version Repository
   Size
==
Reinstalling:
 slurm20-slurmdbd   x86_64
  20.11.3-mybuild
/slurm20-slurmdbd-20.11.3-mybuild.x86_64   2.3 M

Transaction Summary
==
Reinstall  1 Package

Total size: 2.3 M
Installed size: 2.3 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : slurm20-slurmdbd-20.11.3-mybuild.x86_64

   1/1
  Verifying  : slurm20-slurmdbd-20.11.3-mybuild.x86_64

   1/1

Installed:
  slurm20-slurmdbd.x86_64 0:20.11.3-mybuild

What did I miss? In the upgrade page
 I see this:
The libslurm.so version is increased every major release. So things like
MPI libraries with Slurm integration should be recompiled. Sometimes it
works to just symlink the old .so name(s) to the new one, but this has no
guarantee of working.

So I have this:
locate libslurm.so
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36.0.0

Is there some other place the old version is being referenced?

Re: [slurm-users] Slurm Upgrade Philosophy?

2020-12-24 Thread Chris Samuel 

On 24/12/20 6:24 am, Paul Edmon wrote:

We then have a test cluster that we install the release on a run a few 
test jobs to make sure things are working, usually MPI jobs as they tend 
to hit most of the features of the scheduler.


One thing I meant to mention last night was that we use Reframe from 
CSCS as the test framework for our systems, our user support folks 
maintain our local tests as they're best placed to understand the user 
requirements that need coverage and we feed in our system facing 
requirements to them so they can add tests for that side too.


https://reframe-hpc.readthedocs.io/

All the best,
Chris
--
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

Re: [slurm-users] Slurm Upgrade Philosophy?

2020-12-24 Thread Paul Edmon 
We are the same way, though we tend to keep pace with minor releases.  
We typically wait until the .1 release of a new major release before 
considering upgrade so that many of the bugs are worked out.  We then 
have a test cluster that we install the release on a run a few test jobs 
to make sure things are working, usually MPI jobs as they tend to hit 
most of the features of the scheduler.


We also like to stay current with releases as there are new features we 
want, or features we didn't know we wanted but our users find and start 
using.  So our general methodology is to upgrade to the latest minor 
release at our next monthly maintenance.  For major releases we will 
upgrade at our next monthly maintenance after the .1 release is out 
unless there is a show stopping bug that we run into in our own 
testing.  At which point we file a bug with SchedMD and get a patch.


-Paul Edmon-

On 12/24/2020 1:57 AM, Chris Samuel wrote:

On Friday, 18 December 2020 10:10:19 AM PST Jason Simms wrote:


Thanks to several helpful members on this list, I think I have a much better
handle on how to upgrade Slurm. Now my question is, do most of you upgrade
with each major release?

We do, though not immediately and not without a degree of testing on our test
systems.  One of the big reasons for us upgrading is that we've usually paid
for features in Slurm for our needs (for example in 20.11 that includes
scrontab so users won't be tied to favourite login nodes, as well as  the
experimental RPC queue code due to the large numbers of RPCs our systems need
to cope with).

I also keep an eye out for discussions of what other sites find with new
releases too, so I'm following the current concerns about 20.11 and the change
in behaviour for job steps that do (expanding NVIDIA's example slightly):

#SBATCH --exclusive
#SBATCH -N2
srun --ntasks-per-node=1 python multi_node_launch.py

which (if I'm reading the bugs correctly) fails in 20.11 as that srun no
longer gets all the allocated resources, instead just gets the default of
--cpus-per-task=1 instead, which also affects things like mpirun in OpenMPI
built with Slurm support (as it effectively calls "srun orted" and that "orted"
launches the MPI ranks, so in 20.11 it only has access to a single core for
them all to fight over).  Again - if I'm interpreting the bugs correctly!

I don't currently have a test system that's free to try 20.11 on, but
hopefully early in the new year I'll be able to test this out to see how much
of an impact this is going to have and how we will manage it.

https://bugs.schedmd.com/show_bug.cgi?id=10383
https://bugs.schedmd.com/show_bug.cgi?id=10489

All the best,
Chris

Re: [slurm-users] Slurm Upgrade Philosophy?

2020-12-23 Thread Chris Samuel 
On Friday, 18 December 2020 10:10:19 AM PST Jason Simms wrote:

> Thanks to several helpful members on this list, I think I have a much better
> handle on how to upgrade Slurm. Now my question is, do most of you upgrade
> with each major release?

We do, though not immediately and not without a degree of testing on our test 
systems.  One of the big reasons for us upgrading is that we've usually paid 
for features in Slurm for our needs (for example in 20.11 that includes 
scrontab so users won't be tied to favourite login nodes, as well as  the 
experimental RPC queue code due to the large numbers of RPCs our systems need 
to cope with).

I also keep an eye out for discussions of what other sites find with new 
releases too, so I'm following the current concerns about 20.11 and the change 
in behaviour for job steps that do (expanding NVIDIA's example slightly):

#SBATCH --exclusive
#SBATCH -N2
srun --ntasks-per-node=1 python multi_node_launch.py

which (if I'm reading the bugs correctly) fails in 20.11 as that srun no 
longer gets all the allocated resources, instead just gets the default of
--cpus-per-task=1 instead, which also affects things like mpirun in OpenMPI 
built with Slurm support (as it effectively calls "srun orted" and that "orted" 
launches the MPI ranks, so in 20.11 it only has access to a single core for 
them all to fight over).  Again - if I'm interpreting the bugs correctly!

I don't currently have a test system that's free to try 20.11 on, but 
hopefully early in the new year I'll be able to test this out to see how much 
of an impact this is going to have and how we will manage it.

https://bugs.schedmd.com/show_bug.cgi?id=10383
https://bugs.schedmd.com/show_bug.cgi?id=10489

All the best,
Chris
-- 
  Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

Re: [slurm-users] Slurm Upgrade Philosophy?

2020-12-18 Thread Alex Chekholko 
Hi Jason,

Ultimately each site decides how/why to do it; in my case I tend to do big
"forklift upgrades", so I'm running 18.08 on the current cluster and will
go to latest SLURM for my next cluster build.  But you may have good
reasons to upgrade slurm more often on your existing cluster.  I don't use
any of the advanced features.

Regards,
Alex


On Fri, Dec 18, 2020 at 10:13 AM Jason Simms  wrote:

> Hello all,
>
> Thanks to several helpful members on this list, I think I have a much
> better handle on how to upgrade Slurm. Now my question is, do most of you
> upgrade with each major release?
>
> I recognize that, normally, if something is working well, then don't
> upgrade it! In our case, we're running 20.02, and it seems to be working
> well for us. The notes for 20.11 don't indicate any "must have" features
> for our use cases, but I'm still new to Slurm, so maybe there is a hidden
> benefit I can't immediately see.
>
> Given that, I would normally not consider upgrading. But as I understand
> it, you cannot upgrade more than two major releases back, so if I skip this
> one, I'd have to upgrade to (presumably) 21.08, or else I'd have to "double
> upgrade" if, e.g., I wanted to go from 20.02 to 22.05.
>
> To prevent that, do most people try to stay within the most recent two
> versions? Or do you go as long as you possibly can with your existing
> version, upgrading only if you absolutely must?
>
> Warmest regards,
> Jason
>
> --
> *Jason L. Simms, Ph.D., M.P.H.*
> Manager of Research and High-Performance Computing
> XSEDE Campus Champion
> Lafayette College
> Information Technology Services
> 710 Sullivan Rd | Easton, PA 18042
> Office: 112 Skillman Library
> p: (610) 330-5632
>

[slurm-users] Slurm Upgrade Philosophy?

2020-12-18 Thread Jason Simms 
Hello all,

Thanks to several helpful members on this list, I think I have a much
better handle on how to upgrade Slurm. Now my question is, do most of you
upgrade with each major release?

I recognize that, normally, if something is working well, then don't
upgrade it! In our case, we're running 20.02, and it seems to be working
well for us. The notes for 20.11 don't indicate any "must have" features
for our use cases, but I'm still new to Slurm, so maybe there is a hidden
benefit I can't immediately see.

Given that, I would normally not consider upgrading. But as I understand
it, you cannot upgrade more than two major releases back, so if I skip this
one, I'd have to upgrade to (presumably) 21.08, or else I'd have to "double
upgrade" if, e.g., I wanted to go from 20.02 to 22.05.

To prevent that, do most people try to stay within the most recent two
versions? Or do you go as long as you possibly can with your existing
version, upgrading only if you absolutely must?

Warmest regards,
Jason

-- 
*Jason L. Simms, Ph.D., M.P.H.*
Manager of Research and High-Performance Computing
XSEDE Campus Champion
Lafayette College
Information Technology Services
710 Sullivan Rd | Easton, PA 18042
Office: 112 Skillman Library
p: (610) 330-5632

Re: [slurm-users] Slurm Upgrade

2020-11-04 Thread Ole Holm Nielsen 

On 11/5/20 7:14 AM, navin srivastava wrote:

Thank you all for the response.

but my question here is

I have already built a new server slurm 20.2 with the latest DB. my 
question is,  shall i do a mysqldump into this server from existing server 
running with version slurm version 17.11.8 and then i will upgrade all 
client with 20.x followed by 18.x and 19.x  or i can uninstall the slurm 
17.11.8 and install 20.2 on all compute nodes.


Please read 
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm 
explaining how you can upgrade step by step to newer versions.


/Ole

Re: [slurm-users] Slurm Upgrade

2020-11-04 Thread Christopher Samuel 

Hi Navin,

On 11/4/20 10:14 pm, navin srivastava wrote:

I have already built a new server slurm 20.2 with the latest DB. my 
question is,  shall i do a mysqldump into this server from existing 
server running with version slurm version 17.11.8


This won't work - you must upgrade your 17.11 database to 19.05.x first, 
then you can upgrade from 19.05.x to 20.02.


All the best,
Chris
--
  Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

Re: [slurm-users] Slurm Upgrade

2020-11-04 Thread navin srivastava 
Thank you all for the response.

but my question here is

I have already built a new server slurm 20.2 with the latest DB. my
question is,  shall i do a mysqldump into this server from existing server
running with version slurm version 17.11.8 and then i will upgrade all
client with 20.x followed by 18.x and 19.x  or i can uninstall the slurm
17.11.8 and install 20.2 on all compute nodes.

Regards
Navin.









On Tue, Nov 3, 2020 at 12:31 PM Ole Holm Nielsen 
wrote:

> On 11/2/20 2:25 PM, navin srivastava wrote:
> > Currently we are running slurm version 17.11.x and wanted to move to
> 20.x.
> >
> > We are building the New server with Slurm 20.2 version and planning to
> > upgrade the client nodes from 17.x to 20.x.
> >
> > wanted to check if we can upgrade the Client from 17.x to 20.x directly
> or
> > we need to go through 17.x to 18.x and 19.x then 20.x
>
> I have described the Slurm upgrade process in my Wiki page:
> https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm
>
> It's based upon experiences and Slurm documentation and seems to work
> correctly.
>
> /Ole
>
>

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Ole Holm Nielsen 

On 11/2/20 2:25 PM, navin srivastava wrote:

Currently we are running slurm version 17.11.x and wanted to move to 20.x.

We are building the New server with Slurm 20.2 version and planning to 
upgrade the client nodes from 17.x to 20.x.


wanted to check if we can upgrade the Client from 17.x to 20.x directly or 
we need to go through 17.x to 18.x and 19.x then 20.x


I have described the Slurm upgrade process in my Wiki page:
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

It's based upon experiences and Slurm documentation and seems to work 
correctly.


/Ole

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Paul Edmon 
We have hit this when we naively ran using the service and it timed out 
and borked the database.  Fortunately we had a backup to go back to.  
Since then we have run it straight from the command line.  Like yours 
our production DB is now 23 GB for 6 months worth of data so major 
schema updates take roughly 1-2 hours for us.


-Paul Edmon-

On 11/2/2020 11:15 AM, Chris Samuel wrote:

On 11/2/20 7:31 am, Paul Edmon wrote:

e. Run slurmdbd -Dv to do the database upgrade. Depending on the 
upgrade this can take a while because of database schema changes.


I'd like to emphasis the importance of doing the DB upgrade in this 
way, do not use systemctl for this as if systemd runs out of patience 
waiting for slurmdbd to finish the migration and start up it can kill 
it part way through the migration.


Fortunately not something I've run into myself, but as our mysqldump 
of our production DB is approaching 100GB now it's not something we 
want to run into!


All the best,
Chris

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Chris Samuel 

On 11/2/20 7:31 am, Paul Edmon wrote:

e. Run slurmdbd -Dv to do the database upgrade. Depending on the 
upgrade this can take a while because of database schema changes.


I'd like to emphasis the importance of doing the DB upgrade in this way, 
do not use systemctl for this as if systemd runs out of patience waiting 
for slurmdbd to finish the migration and start up it can kill it part 
way through the migration.


Fortunately not something I've run into myself, but as our mysqldump of 
our production DB is approaching 100GB now it's not something we want to 
run into!


All the best,
Chris
--
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Paul Edmon 
We haven't really had MPI ugliness with the latest versions. Plus we've 
been rolling our own PMIx and building against that which seems to have 
solved most of the cross compatibility issues.


-Paul Edmon-

On 11/2/2020 10:38 AM, Fulcomer, Samuel wrote:
Our strategy is a bit simpler. We're migrating compute nodes to a new 
cluster running 20.x. This isn't an upgrade. We'll keep the old 
slurmdbd running for at least enough time to suck the remaining 
accounting data into XDMoD.


The old cluster will keep running jobs until there are no more to run. 
We'll drain and move nodes to the new cluster as we start seeing more 
and more idle nodes in the old cluster. This avoids MPI ugliness and 
we move directly to 20.x.




On Mon, Nov 2, 2020 at 9:28 AM Paul Edmon > wrote:


In general  I would follow this:

https://slurm.schedmd.com/quickstart_admin.html#upgrade


Namely:

Almost every new major release of Slurm (e.g. 19.05.x to 20.02.x)
involves changes to the state files with new data structures, new
options, etc. Slurm permits upgrades to a new major release from
the past two major releases, which happen every nine months (e.g.
18.08.x or 19.05.x to 20.02.x) without loss of jobs or other state
information. State information from older versions will not be
recognized and will be discarded, resulting in loss of all running
and pending jobs. State files are *not* recognized when
downgrading (e.g. from 19.05.x to 18.08.x) and will be discarded,
resulting in loss of all running and pending jobs. For this
reason, creating backup copies of state files (as described below)
can be of value. Therefore when upgrading Slurm (more precisely,
the slurmctld daemon), saving the /StateSaveLocation/ (as defined
in /slurm.conf/) directory contents with all state information is
recommended. If you need to downgrade, restoring that directory's
contents will let you recover the jobs. Jobs submitted under the
new version will not be in those state files, but it can let you
recover most jobs. An exception to this is that jobs may be lost
when installing new pre-release versions (e.g. 20.02.0-pre1 to
20.02.0-pre2). Developers will try to note these cases in the NEWS
file. Contents of major releases are also described in the
RELEASE_NOTES file.

So I wouldn't go directly to 20.x, instead I would go from 17.x to
19.x and then to 20.x

-Paul Edmon-

On 11/2/2020 8:55 AM, Fulcomer, Samuel wrote:

We're doing something similar. We're continuing to run production
on 17.x and have set up a new server/cluster  running 20.x for
testing and MPI app rebuilds.

Our plan had been to add recently purchased nodes to the new
cluster, and at some point turn off submission on the old cluster
and switch everyone to submission on the new cluster (new
login/submission hosts). That way previously submitted MPI apps
would continue to run properly. As the old cluster partitions
started to clear out we'd mark ranges of nodes to drain and move
them to the new cluster.

We've since decided to wait until January, when we've scheduled
some downtime. The process will remain the same wrt moving nodes
from the old cluster to the new, _except_ that everything will be
drained, so we can move big blocks of nodes and avoid slurm.conf
Partition line ugliness.

We're starting with a fresh database to get rid of the bug
induced corruption that prevents GPUs from being fenced with cgroups.

regards,
s

On Mon, Nov 2, 2020 at 8:28 AM navin srivastava
mailto:navin.alt...@gmail.com>> wrote:

Dear All,

Currently we are running slurm version 17.11.x and wanted to
move to 20.x.

We are building the New server with Slurm 20.2 version and
planning to upgrade the client nodes from 17.x to 20.x.

wanted to check if we can upgrade the Client from 17.x to
20.x directly or we need to go through 17.x to 18.x and 19.x
then 20.x

Regards
Navin.

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Fulcomer, Samuel 
Our strategy is a bit simpler. We're migrating compute nodes to a new
cluster running 20.x. This isn't an upgrade. We'll keep the old slurmdbd
running for at least enough time to suck the remaining accounting data into
XDMoD.

The old cluster will keep running jobs until there are no more to run.
We'll drain and move nodes to the new cluster as we start seeing more and
more idle nodes in the old cluster. This avoids MPI ugliness and we move
directly to 20.x.



On Mon, Nov 2, 2020 at 9:28 AM Paul Edmon  wrote:

> In general  I would follow this:
>
> https://slurm.schedmd.com/quickstart_admin.html#upgrade
>
> Namely:
>
> Almost every new major release of Slurm (e.g. 19.05.x to 20.02.x) involves
> changes to the state files with new data structures, new options, etc.
> Slurm permits upgrades to a new major release from the past two major
> releases, which happen every nine months (e.g. 18.08.x or 19.05.x to
> 20.02.x) without loss of jobs or other state information. State information
> from older versions will not be recognized and will be discarded, resulting
> in loss of all running and pending jobs. State files are *not* recognized
> when downgrading (e.g. from 19.05.x to 18.08.x) and will be discarded,
> resulting in loss of all running and pending jobs. For this reason,
> creating backup copies of state files (as described below) can be of value.
> Therefore when upgrading Slurm (more precisely, the slurmctld daemon),
> saving the *StateSaveLocation* (as defined in *slurm.conf*) directory
> contents with all state information is recommended. If you need to
> downgrade, restoring that directory's contents will let you recover the
> jobs. Jobs submitted under the new version will not be in those state
> files, but it can let you recover most jobs. An exception to this is that
> jobs may be lost when installing new pre-release versions (e.g.
> 20.02.0-pre1 to 20.02.0-pre2). Developers will try to note these cases in
> the NEWS file. Contents of major releases are also described in the
> RELEASE_NOTES file.
>
> So I wouldn't go directly to 20.x, instead I would go from 17.x to 19.x
> and then to 20.x
>
> -Paul Edmon-
> On 11/2/2020 8:55 AM, Fulcomer, Samuel wrote:
>
> We're doing something similar. We're continuing to run production on 17.x
> and have set up a new server/cluster  running 20.x for testing and MPI app
> rebuilds.
>
> Our plan had been to add recently purchased nodes to the new cluster, and
> at some point turn off submission on the old cluster and switch everyone
> to  submission on the new cluster (new login/submission hosts). That way
> previously submitted MPI apps would continue to run properly. As the old
> cluster partitions started to clear out we'd mark ranges of nodes to drain
> and move them to the new cluster.
>
> We've since decided to wait until January, when we've scheduled some
> downtime. The process will remain the same wrt moving nodes from the old
> cluster to the new, _except_ that everything will be drained, so we can
> move big blocks of nodes and avoid slurm.conf Partition line ugliness.
>
> We're starting with a fresh database to get rid of the bug
> induced corruption that prevents GPUs from being fenced with cgroups.
>
> regards,
> s
>
> On Mon, Nov 2, 2020 at 8:28 AM navin srivastava 
> wrote:
>
>> Dear All,
>>
>> Currently we are running slurm version 17.11.x and wanted to move to 20.x.
>>
>> We are building the New server with Slurm 20.2 version and planning to
>> upgrade the client nodes from 17.x to 20.x.
>>
>> wanted to check if we can upgrade the Client from 17.x to 20.x directly
>> or we need to go through 17.x to 18.x and 19.x then 20.x
>>
>> Regards
>> Navin.
>>
>>
>>
>>

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Paul Edmon 
We don't follow the recommended procedure here but rather build RPMs and 
upgrade using those.  We haven't and any issues.  Here is our procedure:


1. Build rpms from source using a version of the slurm.spec file that we 
maintain. It's the version SchedMD provides but modified with some 
specific stuff for our env and to disable automatic restarts on upgrade 
which can cause problems especially for upgrading the slurm database.


2. We test the upgrade on our test cluster using the following sequence.

a. Pause all jobs and stop all scheduling.

b. Stop slurmctld and slurmdbd.

c. Backup spool and the database.

d. Upgrade slurm rpms (note that you need to make sure that the upgrade 
will not automatically restart the dbd or the ctld else you may end up 
in a world of hurt)


e. Run slurmdbd -Dv to do the database upgrade. Depending on the 
upgrade this can take a while because of database schema changes.


f. Restart slurmdbd using the service

g. Upgrading slurm rpms across the cluster using salt.

h. Global restart of slurmd and slurmctld.

3. If that all looks good we rinse and repeat on our production cluster.

The rpms have worked fine for us.  The main hitch is the automatic 
restart on upgrade, which I do not recommend.  You should neuter that 
portion of the provided spec file, especially for the slurmdbd upgrades.


We generally prefer the RPM method as it is the normal method for 
interaction with the OS and works well with Puppet.


-Paul Edmon-

On 11/2/2020 10:13 AM, Jason Simms wrote:

Hello all,

I am going to reveal the degree of my inexperience here, but am I 
perhaps the only one who thinks that Slurm's upgrade procedure is too 
complex? Or, at least maybe not explained in enough detail?


I'm running a CentOS 8 cluster, and to me, I should be able simply to 
update the Slurm package and any of its dependencies, and that's it. 
When I looked at the notes from the recent Slurm Users' Group meeting, 
however, I see that while that mode is technically supported, it is 
not recommended, and instead one should always rebuild from source. 
Really?


So, ok, regardless whether that's the case, the upgrade notes linked 
to in the prior post don't, in my opinion, go into enough detail. It 
tells you broadly what to do, but not necessarily how to do it. I'd 
welcome example commands for each step (understanding that changes 
might be needed to account for local configurations). There are no 
examples in that section, for example, addressing recompiling from source.


Now, I suspect a chorus of "if you don't understand it well enough, 
you shouldn't be managing it." OK. Perhaps that's fair enough. But I 
came into this role via a non-traditional route and am constantly 
trying to improve my admin skills, and I may not have the complete 
mastery of all aspects quite yet. But I would also say that 
documentation should be clear and complete, and not written solely for 
experts. To be honest, I've had to go to lots of documentation 
external to SchedMD to see good examples of actually working with 
Slurm, or even ask the helpful people on this group. And I firmly 
believe that if there is a packaged version of your software - as 
there is for Slurm - that should be the default, fully-working way to 
upgrade.


Warmest regards,
Jason

On Mon, Nov 2, 2020 at 9:28 AM Paul Edmon > wrote:


In general  I would follow this:

https://slurm.schedmd.com/quickstart_admin.html#upgrade


Namely:

Almost every new major release of Slurm (e.g. 19.05.x to 20.02.x)
involves changes to the state files with new data structures, new
options, etc. Slurm permits upgrades to a new major release from
the past two major releases, which happen every nine months (e.g.
18.08.x or 19.05.x to 20.02.x) without loss of jobs or other state
information. State information from older versions will not be
recognized and will be discarded, resulting in loss of all running
and pending jobs. State files are *not* recognized when
downgrading (e.g. from 19.05.x to 18.08.x) and will be discarded,
resulting in loss of all running and pending jobs. For this
reason, creating backup copies of state files (as described below)
can be of value. Therefore when upgrading Slurm (more precisely,
the slurmctld daemon), saving the /StateSaveLocation/ (as defined
in /slurm.conf/) directory contents with all state information is
recommended. If you need to downgrade, restoring that directory's
contents will let you recover the jobs. Jobs submitted under the
new version will not be in those state files, but it can let you
recover most jobs. An exception to this is that jobs may be lost
when installing new pre-release versions (e.g. 20.02.0-pre1 to
20.02.0-pre2). Developers will try to note these cases in the NEWS
file. Contents of major releases are also described in

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Paul Edmon 
We don't follow the recommended procedure here but rather build RPMs and 
upgrade using those.  We haven't and any issues.  Here is our procedure:


1. Build rpms from source using a version of the slurm.spec file that we 
maintain. It's the version SchedMD provides but modified with some 
specific stuff for our env and to disable automatic restarts on upgrade 
which can cause problems especially for upgrading the slurm database.


2. We test the upgrade on our test cluster using the following sequence.

a. Pause all jobs and stop all scheduling.

b. Stop slurmctld and slurmdbd.

c. Backup spool and the database.

d. Upgrade slurm rpms (note that you need to make sure that the upgrade 
will not automatically restart the dbd or the ctld else you may end up 
in a world of hurt)


e. Run slurmdbd -Dv to do the database upgrade. Depending on the 
upgrade this can take a while because of database schema changes.


f. Restart slurmdbd using the service

g. Upgrading slurm rpms across the cluster using salt.

h. Global restart of slurmd and slurmctld.

3. If that all looks good we rinse and repeat on our production cluster.

The rpms have worked fine for us.  The main hitch is the automatic 
restart on upgrade, which I do not recommend.  You should neuter that 
portion of the provided spec file, especially for the slurmdbd upgrades.


We generally prefer the RPM method as it is the normal method for 
interaction with the OS and works well with Puppet.


-Paul Edmon-

On 11/2/2020 10:13 AM, Jason Simms wrote:

Hello all,

I am going to reveal the degree of my inexperience here, but am I 
perhaps the only one who thinks that Slurm's upgrade procedure is too 
complex? Or, at least maybe not explained in enough detail?


I'm running a CentOS 8 cluster, and to me, I should be able simply to 
update the Slurm package and any of its dependencies, and that's it. 
When I looked at the notes from the recent Slurm Users' Group meeting, 
however, I see that while that mode is technically supported, it is 
not recommended, and instead one should always rebuild from source. 
Really?


So, ok, regardless whether that's the case, the upgrade notes linked 
to in the prior post don't, in my opinion, go into enough detail. It 
tells you broadly what to do, but not necessarily how to do it. I'd 
welcome example commands for each step (understanding that changes 
might be needed to account for local configurations). There are no 
examples in that section, for example, addressing recompiling from source.


Now, I suspect a chorus of "if you don't understand it well enough, 
you shouldn't be managing it." OK. Perhaps that's fair enough. But I 
came into this role via a non-traditional route and am constantly 
trying to improve my admin skills, and I may not have the complete 
mastery of all aspects quite yet. But I would also say that 
documentation should be clear and complete, and not written solely for 
experts. To be honest, I've had to go to lots of documentation 
external to SchedMD to see good examples of actually working with 
Slurm, or even ask the helpful people on this group. And I firmly 
believe that if there is a packaged version of your software - as 
there is for Slurm - that should be the default, fully-working way to 
upgrade.


Warmest regards,
Jason

On Mon, Nov 2, 2020 at 9:28 AM Paul Edmon > wrote:


In general  I would follow this:

https://slurm.schedmd.com/quickstart_admin.html#upgrade


Namely:

Almost every new major release of Slurm (e.g. 19.05.x to 20.02.x)
involves changes to the state files with new data structures, new
options, etc. Slurm permits upgrades to a new major release from
the past two major releases, which happen every nine months (e.g.
18.08.x or 19.05.x to 20.02.x) without loss of jobs or other state
information. State information from older versions will not be
recognized and will be discarded, resulting in loss of all running
and pending jobs. State files are *not* recognized when
downgrading (e.g. from 19.05.x to 18.08.x) and will be discarded,
resulting in loss of all running and pending jobs. For this
reason, creating backup copies of state files (as described below)
can be of value. Therefore when upgrading Slurm (more precisely,
the slurmctld daemon), saving the /StateSaveLocation/ (as defined
in /slurm.conf/) directory contents with all state information is
recommended. If you need to downgrade, restoring that directory's
contents will let you recover the jobs. Jobs submitted under the
new version will not be in those state files, but it can let you
recover most jobs. An exception to this is that jobs may be lost
when installing new pre-release versions (e.g. 20.02.0-pre1 to
20.02.0-pre2). Developers will try to note these cases in the NEWS
file. Contents of major releases are also described in

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Paul Edmon 

In general  I would follow this:

https://slurm.schedmd.com/quickstart_admin.html#upgrade

Namely:

Almost every new major release of Slurm (e.g. 19.05.x to 20.02.x) 
involves changes to the state files with new data structures, new 
options, etc. Slurm permits upgrades to a new major release from the 
past two major releases, which happen every nine months (e.g. 18.08.x or 
19.05.x to 20.02.x) without loss of jobs or other state information. 
State information from older versions will not be recognized and will be 
discarded, resulting in loss of all running and pending jobs. State 
files are *not* recognized when downgrading (e.g. from 19.05.x to 
18.08.x) and will be discarded, resulting in loss of all running and 
pending jobs. For this reason, creating backup copies of state files (as 
described below) can be of value. Therefore when upgrading Slurm (more 
precisely, the slurmctld daemon), saving the /StateSaveLocation/ (as 
defined in /slurm.conf/) directory contents with all state information 
is recommended. If you need to downgrade, restoring that directory's 
contents will let you recover the jobs. Jobs submitted under the new 
version will not be in those state files, but it can let you recover 
most jobs. An exception to this is that jobs may be lost when installing 
new pre-release versions (e.g. 20.02.0-pre1 to 20.02.0-pre2). Developers 
will try to note these cases in the NEWS file. Contents of major 
releases are also described in the RELEASE_NOTES file.


So I wouldn't go directly to 20.x, instead I would go from 17.x to 19.x 
and then to 20.x


-Paul Edmon-

On 11/2/2020 8:55 AM, Fulcomer, Samuel wrote:
We're doing something similar. We're continuing to run production on 
17.x and have set up a new server/cluster running 20.x for testing and 
MPI app rebuilds.


Our plan had been to add recently purchased nodes to the new cluster, 
and at some point turn off submission on the old cluster and switch 
everyone to  submission on the new cluster (new login/submission 
hosts). That way previously submitted MPI apps would continue to run 
properly. As the old cluster partitions started to clear out we'd mark 
ranges of nodes to drain and move them to the new cluster.


We've since decided to wait until January, when we've scheduled some 
downtime. The process will remain the same wrt moving nodes from the 
old cluster to the new, _except_ that everything will be drained, so 
we can move big blocks of nodes and avoid slurm.conf Partition line 
ugliness.


We're starting with a fresh database to get rid of the bug 
induced corruption that prevents GPUs from being fenced with cgroups.


regards,
s

On Mon, Nov 2, 2020 at 8:28 AM navin srivastava 
mailto:navin.alt...@gmail.com>> wrote:


Dear All,

Currently we are running slurm version 17.11.x and wanted to move
to 20.x.

We are building the New server with Slurm 20.2 version and
planning to upgrade the client nodes from 17.x to 20.x.

wanted to check if we can upgrade the Client from 17.x to 20.x
directly or we need to go through 17.x to 18.x and 19.x then 20.x

Regards
Navin.

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Fulcomer, Samuel 
We're doing something similar. We're continuing to run production on 17.x
and have set up a new server/cluster  running 20.x for testing and MPI app
rebuilds.

Our plan had been to add recently purchased nodes to the new cluster, and
at some point turn off submission on the old cluster and switch everyone
to  submission on the new cluster (new login/submission hosts). That way
previously submitted MPI apps would continue to run properly. As the old
cluster partitions started to clear out we'd mark ranges of nodes to drain
and move them to the new cluster.

We've since decided to wait until January, when we've scheduled some
downtime. The process will remain the same wrt moving nodes from the old
cluster to the new, _except_ that everything will be drained, so we can
move big blocks of nodes and avoid slurm.conf Partition line ugliness.

We're starting with a fresh database to get rid of the bug
induced corruption that prevents GPUs from being fenced with cgroups.

regards,
s

On Mon, Nov 2, 2020 at 8:28 AM navin srivastava 
wrote:

> Dear All,
>
> Currently we are running slurm version 17.11.x and wanted to move to 20.x.
>
> We are building the New server with Slurm 20.2 version and planning to
> upgrade the client nodes from 17.x to 20.x.
>
> wanted to check if we can upgrade the Client from 17.x to 20.x directly or
> we need to go through 17.x to 18.x and 19.x then 20.x
>
> Regards
> Navin.
>
>
>
>

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Christopher J Cawley 
Depending on how large the database is,
the database backend upgrades can take
while.

Chris

Christopher J. Cawley

Systems Engineer/Linux Engineer, Information Technology Services

223 Aquia Building, Ffx, MSN: 1B5

George Mason University


Phone: (703) 993-6397

Email: ccawl...@gmu.edu

​


From: slurm-users  on behalf of 
Christopher J Cawley 
Sent: Monday, November 2, 2020 8:33 AM
To: Slurm User Community List 
Subject: Re: [slurm-users] Slurm Upgrade

I do not think so.

In any case, make sure that you stop services
and make a backup of the database.

Chris

Christopher J. Cawley

Systems Engineer/Linux Engineer, Information Technology Services

223 Aquia Building, Ffx, MSN: 1B5

George Mason University


Phone: (703) 993-6397

Email: ccawl...@gmu.edu

​


From: slurm-users  on behalf of navin 
srivastava 
Sent: Monday, November 2, 2020 8:25 AM
To: Slurm User Community List 
Subject: [slurm-users] Slurm Upgrade

Dear All,

Currently we are running slurm version 17.11.x and wanted to move to 20.x.

We are building the New server with Slurm 20.2 version and planning to upgrade 
the client nodes from 17.x to 20.x.

wanted to check if we can upgrade the Client from 17.x to 20.x directly or we 
need to go through 17.x to 18.x and 19.x then 20.x

Regards
Navin.

Re: [slurm-users] Slurm Upgrade

2020-11-02 Thread Christopher J Cawley 
I do not think so.

In any case, make sure that you stop services
and make a backup of the database.

Chris

Christopher J. Cawley

Systems Engineer/Linux Engineer, Information Technology Services

223 Aquia Building, Ffx, MSN: 1B5

George Mason University


Phone: (703) 993-6397

Email: ccawl...@gmu.edu

​


From: slurm-users  on behalf of navin 
srivastava 
Sent: Monday, November 2, 2020 8:25 AM
To: Slurm User Community List 
Subject: [slurm-users] Slurm Upgrade

Dear All,

Currently we are running slurm version 17.11.x and wanted to move to 20.x.

We are building the New server with Slurm 20.2 version and planning to upgrade 
the client nodes from 17.x to 20.x.

wanted to check if we can upgrade the Client from 17.x to 20.x directly or we 
need to go through 17.x to 18.x and 19.x then 20.x

Regards
Navin.

[slurm-users] Slurm Upgrade

2020-11-02 Thread navin srivastava 
Dear All,

Currently we are running slurm version 17.11.x and wanted to move to 20.x.

We are building the New server with Slurm 20.2 version and planning to
upgrade the client nodes from 17.x to 20.x.

wanted to check if we can upgrade the Client from 17.x to 20.x directly or
we need to go through 17.x to 18.x and 19.x then 20.x

Regards
Navin.

Re: [slurm-users] Slurm Upgrade from 17.02

2020-02-20 Thread Steven Senator (slurm-dev-list) 
When upgrading to 18.08 it is prudent to add following lines into your
/etc/my.cnf as per
  https://slurm.schedmd.com/accounting.html
  https://slurm.schedmd.com/SLUG19/High_Throughput_Computing.pdf (slide #6)

[mysqld]
innodb_buffer_pool_size=1G
innodb_log_file_size=64M
innodb_lock_wait_timeout=900

If the node on which mysql is running has sufficient memory you may
want to increase the innodb_buffer_pool_size beyond 1G. That's just
the minimum threshold below which slurm complains. We use 8G, for
example, because it fits our churn rate for {job arrival, job dispatch
to run state} in RAM and our nodes enough RAM to accommodate an 8G
cache. (references on tuning below)

When you reset this, you will also need to remove the previous innodb
caches, which are probably in /var/lib/mysql. When we did this we
removed and recreated the slurm_acct_db, although that was partially
motivated by the fact that this coincided with an OS and database
patch upgrade and a major accounting and allocation cycle.
  0. Stop slurmctld, slurmdbd.
  1. Create a dump of your database. (mysqldump ...)
  2. Verify that the dump is complete and valid.
  3. Remove the slurm_acct_db. (mysql -e "drop database slurm_acct_db;")
  3. Stop your mysql instance cleanly.
  4. Check the logs. Verify that the mysql instance was stopped cleanly.
  5.  rm /var/lib/mysql/ib_logfile? /var/lib/ibdata1
  6. Put the new lines as above into /etc/my.cnf with the log file
sized appropriately.
  7. Start mysql.
  8. Verify it started cleanly.
  9. Restart the slurm dbd manually, possibly in non-daemon mode.
(slurmdbd -D -vv)
  10. sacctmgr create cluster 

If you want to restore the data back into the data base, do it
*before* step 9 so that the schema conversion can be performed. I like
using mutiple "-vv" so that I can see some of the messages as that
conversion process proceeds.

Some references on mysql innodb_buffer_pool_size tuning:
  
https://scalegrid.io/blog/calculating-innodb-buffer-pool-size-for-your-mysql-server/
  https://mariadb.com/kb/en/innodb-system-variables/#innodb_buffer_pool_size
  https://mariadb.com/kb/en/innodb-buffer-pool/
  https://www.percona.com/blog/2015/06/02/80-ram-tune-innodb_buffer_pool_size/
  https://dev.mysql.com/doc/refman/5.7/en/innodb-buffer-pool-resize.html

Hope this helps,
 -Steve Senator

On Wed, Feb 19, 2020 at 7:12 AM Ricardo Gregorio
 wrote:
>
> hi all,
>
>
>
> I am putting together an upgrade plan for slurm on our HPC. We are currently 
> running old version 17.02.11. Would you guys advise us upgrading to 18.08 or 
> 19.05?
>
>
>
> I understand we will have to also upgrade the version of mariadb from 5.5 to 
> 10.X and pay attention to 'long db upgrade from 17.02 to 18.X or 19.X' and 
> 'bug 6796' amongst other things.
>
>
>
> We would appreciate your comments/recommendations
>
>
>
> Regards,
>
> Ricardo Gregorio
>
> Research and Systems Administrator
>
> Operations ITS
>
>
>
>
>
>
> Rothamsted Research is a company limited by guarantee, registered in England 
> at Harpenden, Hertfordshire, AL5 2JQ under the registration number 2393175 
> and a not for profit charity number 802038.

Re: [slurm-users] Slurm Upgrade from 17.02

2020-02-20 Thread Ricardo Gregorio 
Thank you Ole/Chris/Marcus. Your input was much appreciated

Ole, I was(am) basing my upgrade plan using the documentation found on the link 
you had sent me. In fact your wiki is always my first stop when 
learning/tshooting SLURM issues, even before SLURM docs pages. Excellent work, 
well done.

Regards,
Ricardo Gregorio


-Original Message-
From: slurm-users  On Behalf Of Ole Holm 
Nielsen
Sent: 19 February 2020 14:41
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] Slurm Upgrade from 17.02

On 2/19/20 3:10 PM, Ricardo Gregorio wrote:
> I am putting together an upgrade plan for slurm on our HPC. We are
> currently running old version 17.02.11. Would you guys advise us
> upgrading to 18.08 or 19.05?

You should be able to upgrade 2 Slurm major versions in one step.  The
18.08 version is just about to become unsupported since 20.02 will be released 
shortly.  We use 19.05.5.

I have collected a number of upgrading details in my Slurm Wiki page:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.fysik.dtu.dk%2Fniflheim%2FSlurm_installation%23upgrading-slurmdata=01%7C01%7Cricardo.gregorio%40rothamsted.ac.uk%7C5fe28607ff8d455f5d9c08d7b54a06f6%7Cb688362589414342b0e37b8cc8392f64%7C1sdata=zQfmqJcyEEp%2BvC2WxHR1eKWIu4F%2Ftbms344YlwW0Bs0%3Dreserved=0

You really, really want to perform a dry-run Slurm database upgrade on a test 
machine before doing the real upgrade!  See
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.fysik.dtu.dk%2Fniflheim%2FSlurm_installation%23make-a-dry-run-database-upgradedata=01%7C01%7Cricardo.gregorio%40rothamsted.ac.uk%7C5fe28607ff8d455f5d9c08d7b54a06f6%7Cb688362589414342b0e37b8cc8392f64%7C1sdata=WWqmE7erGoSEJ9cMQ1o%2FOgXsI8kqK7YQ8zztSr9JpIg%3Dreserved=0

> I understand we will have to also upgrade the version of mariadb from
> 5.5 to 10.X and pay attention to 'long db upgrade from 17.02 to 18.X or 19.X'
> and 'bug 6796' amongst other things.

We use the default MariaDB 5.5 in CentOS 7.7.  Upgrading to MariaDB 10 seems to 
have quite a number of unresolved installation issues, so I would skip that for 
now.  Se s

> We would appreciate your comments/recommendations

Slurm 19.05 works great for us.  We're happy with our SchedMD support contract.

/Ole


Rothamsted Research is a company limited by guarantee, registered in England at 
Harpenden, Hertfordshire, AL5 2JQ under the registration number 2393175 and a 
not for profit charity number 802038.

Re: [slurm-users] Slurm Upgrade from 17.02

2020-02-19 Thread Marcus Wagner 

Hi Ricardo,

If I remember right, you can only upgrade two versions further. So you 
WILL have to upgrade to 18.08, even if you want to use 19.05 or the 
coming 20.02


17.02 -> 17.11 -> 18.08 -> 19.05 -> 20.02
^  ^
|  |
|- you are here    |- "farthest jump" to a newer version in one step.


As SchedMD introduced constres in 19.05, consres will become depercated 
in future versions. The way you order GPUs is more consistent in the new 
version. So, I would upgrade to 19.05. Still you will have in a first 
step to upgrade to 18.08 though.



Best
Marcus


On 2/19/20 3:10 PM, Ricardo Gregorio wrote:


hi all,

I am putting together an upgrade plan for slurm on our HPC. We are 
currently running old version 17.02.11. Would you guys advise us 
upgrading to 18.08 or 19.05?


I understand we will have to also upgrade the version of mariadb from 
5.5 to 10.X and pay attention to 'long db upgrade from 17.02 to 18.X 
or 19.X' and 'bug 6796' amongst other things.


We would appreciate your comments/recommendations

Regards,

*Ricardo Gregorio*

Research and Systems Administrator

Operations ITS


Rothamsted Research is a company limited by guarantee, registered in 
England at Harpenden, Hertfordshire, AL5 2JQ under the registration 
number 2393175 and a not for profit charity number 802038. 


--
Marcus Wagner, Dipl.-Inf.

IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
wag...@itc.rwth-aachen.de
www.itc.rwth-aachen.de

Re: [slurm-users] Slurm Upgrade from 17.02

2020-02-19 Thread Chris Samuel 

On 19/2/20 6:10 am, Ricardo Gregorio wrote:

I am putting together an upgrade plan for slurm on our HPC. We are 
currently running old version 17.02.11. Would you guys advise us 
upgrading to 18.08 or 19.05?


Slurm versions only support upgrading from 2 major versions back, so you 
could only upgrade from 17.02 to 17.11 or 18.08.  I'd suggest going 
straight to 18.08.


Remember you have to upgrade slurmdbd first, then upgrade slurmctld and 
then finally the slurmd's.


Also, as Ole points out, 20.02 is due out soon at which point 18.08 gets 
retired from support, so you'd probably want to jump to 19.05 from 18.08.


Don't forget to take backups first!  We do a mysqldump of the whole 
accounting DB and rsync backups of our state directories before an upgrade.


Best of luck!
Chris
--
 Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

Re: [slurm-users] Slurm Upgrade from 17.02

2020-02-19 Thread Ole Holm Nielsen 

On 2/19/20 3:10 PM, Ricardo Gregorio wrote:
I am putting together an upgrade plan for slurm on our HPC. We are 
currently running old version 17.02.11. Would you guys advise us upgrading 
to 18.08 or 19.05?


You should be able to upgrade 2 Slurm major versions in one step.  The 
18.08 version is just about to become unsupported since 20.02 will be 
released shortly.  We use 19.05.5.


I have collected a number of upgrading details in my Slurm Wiki page:
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

You really, really want to perform a dry-run Slurm database upgrade on a 
test machine before doing the real upgrade!  See

https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#make-a-dry-run-database-upgrade

I understand we will have to also upgrade the version of mariadb from 5.5 
to 10.X and pay attention to 'long db upgrade from 17.02 to 18.X or 19.X' 
and 'bug 6796' amongst other things.


We use the default MariaDB 5.5 in CentOS 7.7.  Upgrading to MariaDB 10 
seems to have quite a number of unresolved installation issues, so I would 
skip that for now.  Se s



We would appreciate your comments/recommendations


Slurm 19.05 works great for us.  We're happy with our SchedMD support 
contract.


/Ole

[slurm-users] Slurm Upgrade from 17.02

2020-02-19 Thread Ricardo Gregorio 
hi all,

I am putting together an upgrade plan for slurm on our HPC. We are currently 
running old version 17.02.11. Would you guys advise us upgrading to 18.08 or 
19.05?

I understand we will have to also upgrade the version of mariadb from 5.5 to 
10.X and pay attention to 'long db upgrade from 17.02 to 18.X or 19.X' and 'bug 
6796' amongst other things.

We would appreciate your comments/recommendations

Regards,
Ricardo Gregorio
Research and Systems Administrator
Operations ITS



Rothamsted Research is a company limited by guarantee, registered in England at 
Harpenden, Hertfordshire, AL5 2JQ under the registration number 2393175 and a 
not for profit charity number 802038.

[slurm-users] slurm upgrade from slurm-17.11.12-1 to slurm-19.05.2-1

2019-09-23 Thread Costin 
Hi all,

After the upgrade of our cluster from slurm 17.11.12 to 19.05.2 we started
noticing that jobs above ~ 10 nodes start failing with:
[2019-09-23T16:51:34.310] debug:  Checking credential with 640 bytes of sig
data
[2019-09-23T16:51:34.311] error: Credential signature check: Credential
data size mismatch
[2019-09-23T16:51:34.311] error: _convert_job_mem: slurm_cred_verify
failed: Invalid job credential

Any idea what might be wrong?

Regards,
Costin