On Fri, Jan 04, 2013 at 11:43:06AM -0700, Karl Schulz wrote: > Just a followup comment on the --switches inline below: > > On Jan 4, 2013, at 10:48 AM, Moe Jette <[email protected]> wrote: > > > > > Andy, See inline below: > > > > Quoting Andy Wettstein <[email protected]>: > > > >> > >> On Thu, Jan 03, 2013 at 03:50:05PM -0700, Ole Holm Nielsen wrote: > >>> > >>> On 03-01-2013 18:34, Moe Jette wrote: > >>>> > >>>> Slurm has an option --ntasks-per-node, so there are equivalents to > >>>>> qsub -l nodes=8,ppn=4 > >>>>> qsub -l nodes=4,ppn=8 > >>>>> qsub -l nodes=2,ppn=16 > >>>> like this > >>>> sbatch -N8 --ntasks-per-node=4 > >>>> sbatch -N4 --ntasks-per-node=8 > >>>> sbatch -N2 --ntasks-per-node=16 > >>>> > >>>> You could use a job_submit plugin to map sbatch -n32 to _one_ of the > >>>> above task distributions, but you would need to modify the Slurm code > >>>> for it to use only one of these distributions. > >>> > >>> Moe, thanks for the hint! I heard your impressive presentation at the > >>> SC'12 > >>> booth and wanted to investigate the option of switching our cluster > >>> to Slurm. > >>> > >>> Do I understand you correctly that the node layout would have to be > >>> decided > >>> manually with the Slurm scheduler? I.e., Slurm has no way of > >>> automatically and > >>> dynamically mapping jobs to whatever kinds of nodes are free (with > >>> different > >>> numbers of cores per node), just like the situation with Torque/Maui? > >> > >> I don't know of a way to do this currently in slurm either, but I have > >> kind of thought that it may be possible to use the topology/tree plugin > >> for something like this. The topology/tree plugin allows you to group > >> nodes by user defined switches. This is a completely arbitrary mapping, > >> so you could say all 8 core nodes are connected to one switch, all 16 > >> core nodes connected to another, etc. Slurm currently tries to do > >> minimize the number of switches used by a job, but will still schedule > >> jobs across switches. If there was an option in the topology plugin to > >> not schedule across switches then I think it would work. I'm not sure > >> how feasible it would be to add an option like that. > > > > User's can specify how many switches they want their job to use and > > how long to wait. This is from man sbatch: > > > > --switches=<count>[@<max-time>] > > When a tree topology is used, this defines the maximum count > > of > > switches desired for the job allocation and optionally the > > maxi‐ > > mum time to wait for that number of switches. If SLURM finds > > an > > allocation containing more switches than the count > > specified, > > the job remains pending until it either finds an allocation > > with > > desired switch count or the time limit expires. It there is > > no > > switch count limit, there is no delay in starting the > > job. > > Acceptable time formats include "minutes", > > "minutes:seconds", > > "hours:minutes:seconds", "days-hours", "days-hours:minutes" > > and > > "days-hours:minutes:seconds". The job's maximum time delay > > may > > be limited by the system administrator using the > > SchedulerParam‐ > > eters configuration parameter with the max_switch_wait > > parameter > > option. The default max-time is the max_switch_wait > > Scheduler‐ > > Parameter. >
I must have overlooked what the --switches option does. It seems like everything is there to do some experimentation to determine if using topology/tree could handle this type of usage. > Note that our first experience trying to use the --switches option with the > topology/tree plugin showed that having multiple jobs being submitted with > this option can put a serious whammy on the interactivity of slurm > intermittently (I posted a separate thread on this recently). Not sure if > others have better experience using the tree plugin for large-scale > production or not. I haven't seen performance issues with topology/tree, but whenever I enable it I seem to get some strange scheduling behavior (a large job won't be scheduled because the scheduler can't find an appropriate node configuration even though I've configured all the nodes to be on the same switch). I just have the topology/tree plugin disabled since it doesn't really matter with our current configuration. > > -k > > -- andy wettstein hpc system administrator research computing center university of chicago 773.702.1104
