We have held some discussions on this subject and it isn't simple to
resolve. The best way to do this would probably be to establish
finer-grained locking so there can be more parallelism, say by locking
individual job records rather than the entire job list. That would
impact quite a few sub-systems, for example how we preserve job state.
If you could submit a smaller number of jobs that each have many job
steps, that could address your problem today (say submitting 1000 jobs
each with 1000 steps).
Moe Jette
Quoting Yuri D'Elia <wav...@thregr.org>:
Hi everyone. I'm trying to increase the number of jobs that can be
queued with SLURM. I'm submitting a lot of very small jobs (that
take around ~10 minutes) in batches of ~100k. I would like to be
able to queue around 500k to 1m jobs, if possible, but I'm having a
very hard time going beyond 100k with both 2.3.1 and 2.4.
To make a test, I've raised MaxJobCount to 200000, MessageTimeout to
60 and reduced MinJobAge to 60. Of course, SchedulerParameters has
already "defer" and I tried to set both max_job_bf and interval to
very low values (10 and 600 respectively).
After going beyond ~100k jobs, slurmctld becomes cpu-bound and
starts to timeout on any request. I noticed that just one cpu is
used: maybe there is a way to split the work on multiple cpus?
Is there any other feature that affects performance? I'm using
cons_res, multifactor priority along with accounting, but I would
gladly use a simpler scheduler and less features if I could go
beyond the current limit (which still looks pretty far below my
target).
Thanks again.
