DeSantis
Sent: 18 May 2022 15:39
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] SLURM upgrade from 20.11.3 to 20.11.9
misidentification of job steps
Hello,
It also appears that random jobs are being identified as using too much memory,
despite being well within limits.
For example
.
-Original Message-
From: slurm-users On Behalf Of John
DeSantis
Sent: 18 May 2022 15:39
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] SLURM upgrade from 20.11.3 to 20.11.9
misidentification of job steps
Hello,
It also appears that random jobs are being identified as using too much
Hello,
It also appears that random jobs are being identified as using too much memory,
despite being well within limits.
For example, a job is running that requested 2048 MB per CPU and all processes
are within the limit. But, the job is identified as being over limit when it
isn't. Please
On 24/12/20 6:24 am, Paul Edmon wrote:
We then have a test cluster that we install the release on a run a few
test jobs to make sure things are working, usually MPI jobs as they tend
to hit most of the features of the scheduler.
One thing I meant to mention last night was that we use Reframe
We are the same way, though we tend to keep pace with minor releases.
We typically wait until the .1 release of a new major release before
considering upgrade so that many of the bugs are worked out. We then
have a test cluster that we install the release on a run a few test jobs
to make
On Friday, 18 December 2020 10:10:19 AM PST Jason Simms wrote:
> Thanks to several helpful members on this list, I think I have a much better
> handle on how to upgrade Slurm. Now my question is, do most of you upgrade
> with each major release?
We do, though not immediately and not without a
Hi Jason,
Ultimately each site decides how/why to do it; in my case I tend to do big
"forklift upgrades", so I'm running 18.08 on the current cluster and will
go to latest SLURM for my next cluster build. But you may have good
reasons to upgrade slurm more often on your existing cluster. I
On 11/5/20 7:14 AM, navin srivastava wrote:
Thank you all for the response.
but my question here is
I have already built a new server slurm 20.2 with the latest DB. my
question is, shall i do a mysqldump into this server from existing server
running with version slurm version 17.11.8 and
Hi Navin,
On 11/4/20 10:14 pm, navin srivastava wrote:
I have already built a new server slurm 20.2 with the latest DB. my
question is, shall i do a mysqldump into this server from existing
server running with version slurm version 17.11.8
This won't work - you must upgrade your 17.11
Thank you all for the response.
but my question here is
I have already built a new server slurm 20.2 with the latest DB. my
question is, shall i do a mysqldump into this server from existing server
running with version slurm version 17.11.8 and then i will upgrade all
client with 20.x followed
On 11/2/20 2:25 PM, navin srivastava wrote:
Currently we are running slurm version 17.11.x and wanted to move to 20.x.
We are building the New server with Slurm 20.2 version and planning to
upgrade the client nodes from 17.x to 20.x.
wanted to check if we can upgrade the Client from 17.x to
We have hit this when we naively ran using the service and it timed out
and borked the database. Fortunately we had a backup to go back to.
Since then we have run it straight from the command line. Like yours
our production DB is now 23 GB for 6 months worth of data so major
schema updates
On 11/2/20 7:31 am, Paul Edmon wrote:
e. Run slurmdbd -Dv to do the database upgrade. Depending on the
upgrade this can take a while because of database schema changes.
I'd like to emphasis the importance of doing the DB upgrade in this way,
do not use systemctl for this as if systemd
We haven't really had MPI ugliness with the latest versions. Plus we've
been rolling our own PMIx and building against that which seems to have
solved most of the cross compatibility issues.
-Paul Edmon-
On 11/2/2020 10:38 AM, Fulcomer, Samuel wrote:
Our strategy is a bit simpler. We're
Our strategy is a bit simpler. We're migrating compute nodes to a new
cluster running 20.x. This isn't an upgrade. We'll keep the old slurmdbd
running for at least enough time to suck the remaining accounting data into
XDMoD.
The old cluster will keep running jobs until there are no more to run.
We don't follow the recommended procedure here but rather build RPMs and
upgrade using those. We haven't and any issues. Here is our procedure:
1. Build rpms from source using a version of the slurm.spec file that we
maintain. It's the version SchedMD provides but modified with some
We don't follow the recommended procedure here but rather build RPMs and
upgrade using those. We haven't and any issues. Here is our procedure:
1. Build rpms from source using a version of the slurm.spec file that we
maintain. It's the version SchedMD provides but modified with some
In general I would follow this:
https://slurm.schedmd.com/quickstart_admin.html#upgrade
Namely:
Almost every new major release of Slurm (e.g. 19.05.x to 20.02.x)
involves changes to the state files with new data structures, new
options, etc. Slurm permits upgrades to a new major release
We're doing something similar. We're continuing to run production on 17.x
and have set up a new server/cluster running 20.x for testing and MPI app
rebuilds.
Our plan had been to add recently purchased nodes to the new cluster, and
at some point turn off submission on the old cluster and switch
From: slurm-users on behalf of
Christopher J Cawley
Sent: Monday, November 2, 2020 8:33 AM
To: Slurm User Community List
Subject: Re: [slurm-users] Slurm Upgrade
I do not think so.
In any case, make sure that you stop services
and make a backup of the database
I do not think so.
In any case, make sure that you stop services
and make a backup of the database.
Chris
Christopher J. Cawley
Systems Engineer/Linux Engineer, Information Technology Services
223 Aquia Building, Ffx, MSN: 1B5
George Mason University
Phone: (703) 993-6397
Email:
When upgrading to 18.08 it is prudent to add following lines into your
/etc/my.cnf as per
https://slurm.schedmd.com/accounting.html
https://slurm.schedmd.com/SLUG19/High_Throughput_Computing.pdf (slide #6)
[mysqld]
innodb_buffer_pool_size=1G
innodb_log_file_size=64M
done.
Regards,
Ricardo Gregorio
-Original Message-
From: slurm-users On Behalf Of Ole Holm
Nielsen
Sent: 19 February 2020 14:41
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] Slurm Upgrade from 17.02
On 2/19/20 3:10 PM, Ricardo Gregorio wrote:
> I am putting toget
Hi Ricardo,
If I remember right, you can only upgrade two versions further. So you
WILL have to upgrade to 18.08, even if you want to use 19.05 or the
coming 20.02
17.02 -> 17.11 -> 18.08 -> 19.05 -> 20.02
^ ^
| |
|- you are here |- "farthest jump" to a
On 19/2/20 6:10 am, Ricardo Gregorio wrote:
I am putting together an upgrade plan for slurm on our HPC. We are
currently running old version 17.02.11. Would you guys advise us
upgrading to 18.08 or 19.05?
Slurm versions only support upgrading from 2 major versions back, so you
could only
On 2/19/20 3:10 PM, Ricardo Gregorio wrote:
I am putting together an upgrade plan for slurm on our HPC. We are
currently running old version 17.02.11. Would you guys advise us upgrading
to 18.08 or 19.05?
You should be able to upgrade 2 Slurm major versions in one step. The
18.08 version is
26 matches
Mail list logo