The llnl documentation is very out of date. You should be consulting http://slurm.schedmd.com.
It might not be that different in this case but you should probably switch in any case. Teddy <[email protected]> wrote: >Hi Daniel and Moe, > >Thanks for your reply. > > >Some of users of ours said the srun with --distribution somehow not >working >as expected. > >When i read the tutorial carefully ( >https://computing.llnl.gov/linux/slurm/mc_support.html), i found this: > >"It is important to note that many of these flags are only meaningful >if >the processes' affinity is set. In order for the affinity to be set, >the >task/affinity plugin must be first enabled in slurm.conf: >TaskPlugin=task/affinity ># enable task affinity" > > >My slurm uses cgroup with the slurm.conf as follow: >".... >TaskPlugin=task/cgroup >TaskPluginParam=Sched >......" > > >Where cgroup.conf has the following : >" >CgroupMountpoint=/usr/local/etc/cgroup >CgroupAutomount=yes >CgroupReleaseAgentDir="/usr/local/etc/release" > >ConstrainCores=yes >TaskAffinity=yes" > > >Even though in cgroup.conf there is a line stating "TaskAffinity=yes", >must >slurm.conf use task/affinity instead of task/cgroup? > >What is the cgroup purpose exactly? > > >Thanks, > >Teddy > > > > >On Wed, Jul 10, 2013 at 12:56 AM, Moe Jette <[email protected]> wrote: > >> >> You might find this useful: >> http://www.schedmd.com/slurmdocs/mc_support.html >> >> Quoting "Daniel M. Weeks" <[email protected]>: >> >> > >> > On 07/08/2013 11:34 PM, Teddy - wrote: >> >> Dear all, >> >> >> >> >> >> I am not quite understand on how to assign slurm task by actual >physical >> >> core. >> >> >> >> Lets say i have 3 slurm nodes with different specs. >> >> Node A : 16 physical AMD processor, no hyperthreading ==> 16 cpu >threads >> >> Node B : 8 pyhsical Intel processor, with hyperthreading ==> 16 >cpu >> threads >> >> Node C : 4 physical Intel processor, no hyprethreading ==> 4 cpu >> threads. >> >> >> >> If i am not mistaken, by assigning standard slurm task (srun). >slurm >> >> will see all the nodes with the CPU threads, right? >> > >> > Yes, but don't think of the hardware threads as anything unique - >they >> > are part of each core. The Linux kernel maps all hardware threads >to >> > logical CPUs and that's where the number of "processors" is coming >from. >> > >> >> >> >> If yes, is there a way to run a slurm task and it will >automatically use >> >> actual physical cores ( >> >> So node B will be recognized as 8 cpu node) without have to set >the node >> >> manually in the slurm.conf? >> >> This is because only certain slurm tasks need to use actual cores, >other >> >> slurm tasks dont matter. >> > >> > This is tricky to answer in a simple form. >> > >> > If you specify only 8 tasks on the a system with 16 logical CPUs >but >> > only 8 physical cores (i.e. 2 threads per core), the kernel >scheduler is >> > still free to schedule those 8 tasks wherever it wants. This means >you >> > can still have 8 tasks running on only 4 physical cores and it >would be >> > nothing like hyperthreading being turned off. >> > >> > Affinity is what enables you to specify the hardware thread (a >logical >> > CPU) on which a task is allowed to be scheduled. However, each MPI >> > implementation handles affinity in a slightly different way. >(OpenMPI >> > being the most obnoxious because it wants to do everything itself.) >> > >> > To get back to your question, you can specify -n and -N and Slurm >with >> > balance tasks across the nodes. However, affinity usually isn't >assigned >> > automatically (IIRC, mvapich2 will do a bit). >> > >> > If you want to assign affinity using srun, take a look at the >cpu_bind >> > option. In the case of OpenMPI, you'll need to use its affinity >options >> > (useful options added ~1.4). You can check out the OpenMPI FAQ for >more >> > info: http://www.open-mpi.org/faq/?category=tuning#maffinity-defs >> > >> >> >> >> >> >> >> >> Regards, >> >> >> >> Teddy >> >> >> >> >> >> >> > >> > -- >> > Daniel M. Weeks >> > Systems Programmer >> > Computational Center for Nanotechnology Innovations >> > Rensselaer Polytechnic Institute >> > Troy, NY 12180 >> > 518-276-4458 >> > >> >>
