Just a followup comment on the --switches inline below: On Jan 4, 2013, at 10:48 AM, Moe Jette <[email protected]> wrote:
> > Andy, See inline below: > > Quoting Andy Wettstein <[email protected]>: > >> >> On Thu, Jan 03, 2013 at 03:50:05PM -0700, Ole Holm Nielsen wrote: >>> >>> On 03-01-2013 18:34, Moe Jette wrote: >>>> >>>> Slurm has an option --ntasks-per-node, so there are equivalents to >>>>> qsub -l nodes=8,ppn=4 >>>>> qsub -l nodes=4,ppn=8 >>>>> qsub -l nodes=2,ppn=16 >>>> like this >>>> sbatch -N8 --ntasks-per-node=4 >>>> sbatch -N4 --ntasks-per-node=8 >>>> sbatch -N2 --ntasks-per-node=16 >>>> >>>> You could use a job_submit plugin to map sbatch -n32 to _one_ of the >>>> above task distributions, but you would need to modify the Slurm code >>>> for it to use only one of these distributions. >>> >>> Moe, thanks for the hint! I heard your impressive presentation at the SC'12 >>> booth and wanted to investigate the option of switching our cluster >>> to Slurm. >>> >>> Do I understand you correctly that the node layout would have to be decided >>> manually with the Slurm scheduler? I.e., Slurm has no way of >>> automatically and >>> dynamically mapping jobs to whatever kinds of nodes are free (with different >>> numbers of cores per node), just like the situation with Torque/Maui? >> >> I don't know of a way to do this currently in slurm either, but I have >> kind of thought that it may be possible to use the topology/tree plugin >> for something like this. The topology/tree plugin allows you to group >> nodes by user defined switches. This is a completely arbitrary mapping, >> so you could say all 8 core nodes are connected to one switch, all 16 >> core nodes connected to another, etc. Slurm currently tries to do >> minimize the number of switches used by a job, but will still schedule >> jobs across switches. If there was an option in the topology plugin to >> not schedule across switches then I think it would work. I'm not sure >> how feasible it would be to add an option like that. > > User's can specify how many switches they want their job to use and > how long to wait. This is from man sbatch: > > --switches=<count>[@<max-time>] > When a tree topology is used, this defines the maximum count of > switches desired for the job allocation and optionally the maxi‐ > mum time to wait for that number of switches. If SLURM finds an > allocation containing more switches than the count specified, > the job remains pending until it either finds an allocation with > desired switch count or the time limit expires. It there is no > switch count limit, there is no delay in starting the job. > Acceptable time formats include "minutes", "minutes:seconds", > "hours:minutes:seconds", "days-hours", "days-hours:minutes" and > "days-hours:minutes:seconds". The job's maximum time delay may > be limited by the system administrator using the SchedulerParam‐ > eters configuration parameter with the max_switch_wait parameter > option. The default max-time is the max_switch_wait Scheduler‐ > Parameter. Note that our first experience trying to use the --switches option with the topology/tree plugin showed that having multiple jobs being submitted with this option can put a serious whammy on the interactivity of slurm intermittently (I posted a separate thread on this recently). Not sure if others have better experience using the tree plugin for large-scale production or not. -k
