tributed across many nodes. Also see
the SelectParameters configuration parame-
ter CR_LLN to use the least loaded nodes in
every partition.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Just a short note about terminology. I believe "processor equivalents"
(PE) is a much used term for this. It is at least what Maui and Moab
uses, if I recall correctly. The "resource*time" would then be PE seconds
(or hours, or whatever).
--
Regards,
Bjørn-Helge Mevik, dr
that are printed on the user's stderr? If so,
how?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Thanks!
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
agnu
dejagnu-1.4.4-17.el6.noarch
# rpm -q check
check-0.9.8-1.1.el6.x86_64
Is there something else we are missing?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
writes:
> As far as I can tell, it has been this way (broken) forever. It will be fixed
> in 14.03.7.
Thx!
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
nfortunately, I don't know enough Expect (tcl?) to
suggest how to implement that.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
ient.
It would also be nice if the node-global destinations could be
configurable, instead of being hard-coded in the script (or at least be
set at the top of the script). For instance, on our system, the
node-global file systems are /work and /cluster, not /scratch and /home.
--
Regards,
B
COMPLETED 00:01:07 01:03.980 00:02.207 01:06.187
43COMPLETED 00:01:08 01:05.230 00:02.173 01:07.403
43.batch COMPLETED 00:01:08 01:05.230 00:02.173 01:07.403
i.e., time spent in subprocesses is reported.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department fo
like this?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Thanks for the tip!
We actually already have a setup where "srun
--ntasks=$SLURM_JOB_NUM_NODES /bin/true" is run at the start of every
job, so we're definitely going to look into this.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
_bitstr_bits(tmp_qos_bitstr) was still 26.
Any help in figuring out what goes wrong (or how to fix it :) is
appreciated!
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
the .../uid_NNN directories are
removed.
Does anyone know what these messages mean? Should we just ignore them?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
ng the test suite
for versions 14.03.8--14.03.10, we didn't upgrade to .10.)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
I second this wish. :)
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
clude CPU time of child processes.”
In my experience, that description might not be accurate. It seems also
child processes are included, as long as the job doesn't time out. Here
is an email I wrote about it last year:
From: Bjørn-Helge Mevik
Subject: [slurm-dev] UserCPU etc. for subprocess
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Thanks! I'll try to apply that patch.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
rds,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
pi itself?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Thanks!
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
.03.7, btw.)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Ok, thanks.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
accounting.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
.03.7. Perhaps this has changed in later versions?
Also, "nothing is ever easy": we want to account not CPU hours, but PE
(processor equivalents) hours.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Christopher Samuel writes:
> http://karaage.readthedocs.org/en/latest/introduction.html
Karaage looks interesting for managing projects and users. Can it
manage usage limits?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
atabase makes Gold quite slow, so we have had to add quite a lot of
error checking and handling in the prolog and epilog scripts.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
LIMIT ***
That usually means the job tried to run longer than its --time
specification.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
h IB1 or all with IB2. Search for "Matching OR" in
the sbatch man page for details. (We used this on our previous cluster,
which had two different IB networks.)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
thorough,
unfortunately, but according to
https://lists.fedoraproject.org/pipermail/mingw/2012-January/004421.html
the .la files are only needed in order to link against static libraries,
and since Slurm doesn't provide any static libraries, I guess it would
be safe for the slurm-devel rpm not to
we activated
checkpointing. When slurmcltd started, the checkpointing plugin
expected some extra data in the job states, which obviously wasn't
there, and slurmctld decided the data was invalid and killed all jobs.
(I don't know if this is still a problem.)
--
Regards,
Bjørn-Helge
Thanks. Nice to know!
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
with static libraries, which
slurm does _not_ install.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
s
enough. We have the partition because our lowpri jobs are allowed to
run on special nodes (like hugemem or accellerator nodes) that normal
jobs are not allowed to use.)
I hope this made sense to you. :)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
e data between several processes, the shared space will be
counted once for each process(!). Cgroups seems to count the shared
data only once. So if a process is killed by oom instead of by slurm,
it is probably not due to shared data.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Re
Felip Moll writes:
> I will try JobAcctGatherParams to NoShared.
>
> This is an example of job step being killed. It's being killed by oom, but
> it's invoked by cgroups:
Since the job was killed by the oom, NoShared will not help. It does
not affect cgroups.
--
Rega
"Wiegand, Paul" writes:
> This worked. Thank you Bjørn-Helge.
You're welcome! :)
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
n a
process needs more memory instead of killing the process. If I'm
correct, oom will _not_ kill a job due to cached data.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Test 14.10 in the test suite (of slurm 15.08.8, at least) uses
$sinfo -tidle -h -o%n
to find idle nodes. This only works if NodeHostname == NodeName on the
nodes. The following should work regardless of this:
$scontrol show hostnames \$($sinfo -tidle -h -o%N)
--
Regards,
Bjørn-Helge
obs just
before they time out.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
writes:
> We are looking for comments and feedback on this proposed behavior
[...]
> +#define HEALTH_RETRY_DELAY 10
Have you thought about using the health_check_interval instead? Or make
it a separate configurable option?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Re
tch empty-jobname.sm
sbatch: option requires an argument -- 'J'
Submitted batch job 14221261
$
A more consistent behaviour would have been nice. My suggestion is:
report error and fail to submit the job.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
provisioning tool?
- A locally developed solution?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
+1
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
I just noticed that as of 14.11.6, optimization is turned off (-O0) by
default when building slurm.
Is there any reason not to use --disable-debug when building slurm for a
production cluster?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
md appear to be
Just a note: I tried this (for a different reason), but found out it
didn't have any effect (gather the output to a log file and look at the
gcc lines). However, if I did -D '%with_cflags CFLAGS="-O0 -g3"' (i.e.,
removed the initial "_"), it had the
low" qos, and
to get that to work, we've found that we must put the accounts normal
limits on a qos, not on the account itself. Usually this means that we
have a qos for each account, and then a common "low" qos.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
;, which prints out resource usage. As long
as users remember to source the setup file, they get the usage
statistics in the bottom of their stdout file. Not very elegant, but it
works.
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
To me, this sounds like a job for a job submit plugin, for instance
job_submit.lua. That way you could reject the job before it gets
submitted into the queue.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
to generate
> my report. Does this approach make sense or are there better
> alternatives.
sacct can also give you the submit time, start time, end time and
elapsed time.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
There is a plugin under development, that will/might provide those
features. It was presented at SLUG 16:
http://slurm.schedmd.com/SLUG16/MCS.pdf
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
apart from that, this is my understanding too.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
#x27;d guess it
should have been possible to use scancel in PrologSlurmctld also in
15.08.12.
Does anyone know if this is an intentional change (and SchedMD just
forgot to update the docs) or a bug?
(I haven't found anything relevant in the NEWS file or on
bugs.schedmd.com.)
--
Regards,
B
access to more then one account, I can use 1000 cpus in each
account.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Jordan Willis writes:
>Thank you,
>Can you confirm that this will take an update from SLURM 14.11.15 to
>current?
I never ran 14.11, but in 14.03, you can use GrpCPUs=1000 instead of
GrpTRES=cpu=1000.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research
tell this Slurm. - OK, I can ask for 24
> cores and 64 GB in a node, but then I do not get the chance to run on 12
> cores/32 GB.
For the memory part, you could specify --mem=0. That will allocate all
of the memory on whichever node the job lands on. For the number of
cores, I don't know.
cpus it is allowed to use.)
This is on 15.08.12. YMMV.
--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Check out the thread on this list about a week ago, titled "Unrestricted
use of a node". (In short, --exclusive with --mem=0 or --mem-per-cpu=0
might be more or less what you want.)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
size" [1]
(JobAcctGatherParams=UsePss), which is cgroup uses (I believe), and
sounds like the best estimate to me.
[1] https://en.wikipedia.org/wiki/Proportional_set_size
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
omain config parameter was added in Slurm 17.02.
A different option would be to configure your sendmail to accept
domain-less mails (and perhaps add a default domain itself).
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
file names to a dot file in $SCRATCH)
- The Epilog copies any registered files back to the job submit dir (it
uses "su - $USER" when doing this).
- The epilog deletes the directory
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signa
ke
use Slurmdb qw(:all SLURMDB_ADD_USER);
$what = SLURMDB_ADD_USER();
just gives the error "SLURMDB_ADD_USER is not a valid Slurmdb macro"
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
u.se/~kent/python-hostlist/ by Kent Engström at NSC. It's
> simple to install this as an RPM package, see
> https://wiki.fysik.dtu.dk/niflheim/SLURM#expanding-host-lists
For the simple case you show, you could just use
$ scontrol show hostnames a[095,097-098]
a095
a097
a098
--
Regards,
e pids.
Not that I know of, but it should be possible to script.
> And how to parse the nodelist like "cn[11033,11069],gn[1103-1120]" ?
scontrol show hostnames cn[11033,11069],gn[1103-1120]
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
tup, the first option is preferrable; just putting it on the
queue and let it wait until it's turn. But of course, there are other
setups where the second option would be best. Could you perhaps make it
configurable, so a site can choose?
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department
ric usage. Then you can set
FairshareWeight to 0 and use the Grp*Mins parameters to set hard limits.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
dictable, both for the programmer and for the user.
It is by design, because people often need to give arguments or options
to their jobscript, e.g.,
sbatch --time=1-0:0:0 myjob.sh inputfile
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
ch), but looks like this assumption is wrong?
That is right, that is wrong. :)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
feature from the node, and then request themself to
be requeued.
Prior to submit the jobs, we add the "fixme" feature to all nodes
needing maintenance.
(In reality, our setup is a little mor complex, since it includes
reinstalling the os on the nodes, but the principle is the same
will be looking at
this feature again. Thanks for the tip! :)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
soon as the signal
arrives. I got bit by this behaviour trying to do exactly the same that
you did. :)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
rong with how my partitions are defined?
That sounds unlikely, IMO.
--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
of small,
distributed jobs running, and a long queue of pending jobs), I
personally wouldn't want schedmd to sacrifice that for making updates of
node lists easier. Especially since I haven't seen the problem JinSung
Kang reports. :)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for R
job for the lua job submit plugin
(job_submit.lua). It can check what users have specified, write out
custom errors or change the settings of jobs.
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
signature.asc
Description: PGP signature
Moe Jette writes:
> You findings are correct. I would like to leave the log file creation
> in the same place (as soon as possible) instead of moving after
> changing the user and group ID. The following patch creates the file
> at the same time, but immediately changes the file owner. Rat
uster for sensitive data, in which there should be
no (or as few as possible) possibilities for information to leak between
jobs.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
her a given
node has a reboot pending?
--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
egards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
We are in the process of setting up a new cluster. It is supposed to go
in production by the end of June, and we would prefer to have slurm 2.4
on it.
Do you have any plans/ideas for when 2.4 (or at least release candidates
for 2.4) will be out?
--
Regards,
Bjørn-Helge Mevik, dr. scient
Moe Jette writes:
> We hope to have a v2.4.0-rc1 in a couple of weeks and release 2.4 a
> few weeks later.
Very nice!
--
Cheers,
Bjørn-Helge Mevik
llocated all of the generic resources
that have allocated to the job."
Which one is correct? (I'm voting on man srun. :)
--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
Disk=1 Weight=1027
BootTime=2012-05-08T15:07:08 SlurmdStartTime=2012-05-25T10:30:10
(This is with 2.4.0-0.pre4.)
(We are planning to use cx-y instead of compute-x-y (the rocks default)
on our next cluster, to save some typing.)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computin
ThreadsPerCore=1 TmpDisk=0 Weight=666
BootTime=2012-06-13T16:20:49 SlurmdStartTime=2012-07-02T16:32:31
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
G when they want 10 GB. :)
Thanks for quick response!
--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
-03T17:11:02] [195.0] done with job
[2012-07-03T17:11:02] error: stat_jobacct for invalid job_id: 195
[2012-07-03T17:11:02] debug: _rpc_terminate_job, uid = 501
[2012-07-03T17:11:02] debug: task_slurmd_release_resources: 195
Is there something wrong here, or are we doing something wrong?
--
not turn of loggin alltogether)?
(Any other comments and suggestions about the task/cgroup use are
also welcome!)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
inated by the OOM killer. It is
> not perfect, but works 90% of the time. I can send it to you if
> you like.
Yes, I'd very much like that! Jobs killed by memory limit is quite
common on our cluster, and users get confused if there is no message
telling them why the job died.
Than
" lua plugin to the slurm-spank-plugins
> project on google code called oom-detect.lua. You can browse the
> code here:
Thanks!
--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
available.
Using srun within a fixed allocation limits your subjobs to the
resources in that allocation. (Now that slurm jobs can grow and shrink,
there might be a way to avoid this.)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
11T14:59:10 1:0 8
c11-13
(There is a node-failure in there, and the job failed when it finally
got to run long enough.) Apart from a short period around 21:00 the 10., less
than 7,000 of the ~ 10,000 cores were used.
--
Bjørn-Helge Mevik, dr. scient,
Research Computing Ser
ries once, but it
was a kludge. We ended up writing quite comples prolog and epilog
scripts that copied files back and forth, and had to put quite strict
limits on how and where to submit jobs.
Just my 2¢. :)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
t;) RAM.
Try filling the allocated memory with some values, and you will probably
see that after filling 4 GiB, the job is killed.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
Christopher Samuel writes:
> On 18/01/13 19:53, Bjørn-Helge Mevik wrote:
>
>> I don't know if this is the reason in your case, but note that cgroup
>> in slurm constrains_resident_ RAM, not_allocated_ ("virtual") RAM.
>
> Hmm, as a sysadmin that doesn&#x
here is no message about why in slurm-nnn.out. Then we get
an RT ticket, and have to grep in /var/log/messages.
--
Cheers,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
ill make backfill look at the whole queue eventually.
Interesting. We will take a look at this.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
ls the queue system until the job has finished.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
en
launch the main program, and then perhaps do some cleanup afterwards.
Thus one wouldn't want the job script itself to be run in parallell.
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Research Computing Services, University of Oslo
res:2, CUDA_VISIBLE_DEVICES gets the
value 0,1633906540
Is this correct? Are we doing something wrong?
(This is slurm 2.4.3, running on Rocks 6.0 based on CentOS 6.2.)
--
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
rds,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
Gary Brown writes:
> FYI, the value 1633906540 in hex is 61636f6c, which is ASCII "acol" and
> usually points to some kind of buffer overrun bug.
Thanks for the tip!
--
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo
1 - 100 of 128 matches
Mail list logo
