Hi Daniel and Moe, Thanks for your reply.
Some of users of ours said the srun with --distribution somehow not working as expected. When i read the tutorial carefully ( https://computing.llnl.gov/linux/slurm/mc_support.html), i found this: "It is important to note that many of these flags are only meaningful if the processes' affinity is set. In order for the affinity to be set, the task/affinity plugin must be first enabled in slurm.conf: TaskPlugin=task/affinity # enable task affinity" My slurm uses cgroup with the slurm.conf as follow: ".... TaskPlugin=task/cgroup TaskPluginParam=Sched ......" Where cgroup.conf has the following : " CgroupMountpoint=/usr/local/etc/cgroup CgroupAutomount=yes CgroupReleaseAgentDir="/usr/local/etc/release" ConstrainCores=yes TaskAffinity=yes" Even though in cgroup.conf there is a line stating "TaskAffinity=yes", must slurm.conf use task/affinity instead of task/cgroup? What is the cgroup purpose exactly? Thanks, Teddy On Wed, Jul 10, 2013 at 12:56 AM, Moe Jette <[email protected]> wrote: > > You might find this useful: > http://www.schedmd.com/slurmdocs/mc_support.html > > Quoting "Daniel M. Weeks" <[email protected]>: > > > > > On 07/08/2013 11:34 PM, Teddy - wrote: > >> Dear all, > >> > >> > >> I am not quite understand on how to assign slurm task by actual physical > >> core. > >> > >> Lets say i have 3 slurm nodes with different specs. > >> Node A : 16 physical AMD processor, no hyperthreading ==> 16 cpu threads > >> Node B : 8 pyhsical Intel processor, with hyperthreading ==> 16 cpu > threads > >> Node C : 4 physical Intel processor, no hyprethreading ==> 4 cpu > threads. > >> > >> If i am not mistaken, by assigning standard slurm task (srun). slurm > >> will see all the nodes with the CPU threads, right? > > > > Yes, but don't think of the hardware threads as anything unique - they > > are part of each core. The Linux kernel maps all hardware threads to > > logical CPUs and that's where the number of "processors" is coming from. > > > >> > >> If yes, is there a way to run a slurm task and it will automatically use > >> actual physical cores ( > >> So node B will be recognized as 8 cpu node) without have to set the node > >> manually in the slurm.conf? > >> This is because only certain slurm tasks need to use actual cores, other > >> slurm tasks dont matter. > > > > This is tricky to answer in a simple form. > > > > If you specify only 8 tasks on the a system with 16 logical CPUs but > > only 8 physical cores (i.e. 2 threads per core), the kernel scheduler is > > still free to schedule those 8 tasks wherever it wants. This means you > > can still have 8 tasks running on only 4 physical cores and it would be > > nothing like hyperthreading being turned off. > > > > Affinity is what enables you to specify the hardware thread (a logical > > CPU) on which a task is allowed to be scheduled. However, each MPI > > implementation handles affinity in a slightly different way. (OpenMPI > > being the most obnoxious because it wants to do everything itself.) > > > > To get back to your question, you can specify -n and -N and Slurm with > > balance tasks across the nodes. However, affinity usually isn't assigned > > automatically (IIRC, mvapich2 will do a bit). > > > > If you want to assign affinity using srun, take a look at the cpu_bind > > option. In the case of OpenMPI, you'll need to use its affinity options > > (useful options added ~1.4). You can check out the OpenMPI FAQ for more > > info: http://www.open-mpi.org/faq/?category=tuning#maffinity-defs > > > >> > >> > >> > >> Regards, > >> > >> Teddy > >> > >> > >> > > > > -- > > Daniel M. Weeks > > Systems Programmer > > Computational Center for Nanotechnology Innovations > > Rensselaer Polytechnic Institute > > Troy, NY 12180 > > 518-276-4458 > > > >
