Thanks David. The '%' separator works but requires users to use/add the separator on job submission.
Since one user doesn't know what others are doing we need a parameter that we can use without the users doing (or not doing) anything. Starting of hundreds or thousands of jobs during a single scheduler/backfill iteration is a scalability issue for us. Would it be possible to add a plugin somewhere that could enforce array task start limits without users doing anything to their job submissions? Trevor > On Sep 7, 2015, at 3:44 AM, David Bigagli <[email protected]> wrote: > > Would this work: > > A maximum number of simultaneously running > tasks from the job array may be specified using a "%" separator. > For example "--array=0-15%4" will limit the number of simultane- > ously running tasks from this job array to 4. The minimum index > value is 0. the maximum value is one less than the configura- > tion parameter MaxArraySize. > Thanks > > /David/Bigagli > [email protected] > =============================================================== > Slurm User Group Meeting, 15-16 September 2015, Washington D.C. > http://slurm.schedmd.com/slurm_ug_agenda.html > >> On 04 Sep 2015, at 20:43, Cooper, Trevor <[email protected]> wrote: >> >> >> The documentation[1] doesn't seem to indicate that there are any controls >> for limiting the number of job array tasks that can be started during each >> scheduler/backfill iteration. >> >> If a user submits multiple jobs with thousands of array tasks each I'd like >> to limit the number of array tasks started from each job to O(10) or O(100). >> >> The SchedulerParameters in slurm.conf[2], for example... >> >> $ scontrol show config | grep Sched >> FastSchedule = 0 >> SchedulerParameters = >> batch_sched_delay=30,defer,sched_max_job_start=2,bf_max_job_start=2 >> SchedulerPort = 7321 >> SchedulerRootFilter = 1 >> SchedulerTimeSlice = 30 sec >> SchedulerType = sched/backfill >> SlurmSchedLogFile = /var/log/slurm/slurmsched.log >> SlurmSchedLogLevel = 3 >> >> ...seem only to apply to the parent array job and do not control the >> starting of array tasks WITHIN the array job. >> >> Has anybody accomplished something like this successfully and, if so, are >> you willing/able to share your solution? >> >> -- Trevor >> >> [1] - http://slurm.schedmd.com/job_array.html >> [2] - http://slurm.schedmd.com/slurm.conf.html= >
