srun --cpu_bind=cores On Wed, Feb 8, 2017 at 1:08 PM, Brendan Moloney <moloney.bren...@gmail.com> wrote:
> Hi, > > I want to allocate at the level of logical cores (each serial job gets one > thread on a hyperthreading system), which seems to be achievable only by > not setting threads_per_core on each node, and instead just setting > CPUs=<total_logical_cores>. > > However with core binding, this will pack two tasks onto the same physical > core while all other cores are left idle. On a system with 20 cores and 40 > threads I see this behavior: > > $ srun bash -c "lstopo | head -n 6 ; sleep 10" & > NUMANode L#0 (P#0 47GB) + Package L#0 + L3 L#0 (25MB) + L2 L#0 (256KB) + > L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0) > NUMANode L#1 (P#1 47GB) > HostBridge L#0 > PCIBridge > PCI 15b3:1003 > $ srun bash -c "lstopo | head -n 6 ; sleep 10" & > NUMANode L#0 (P#0 47GB) + Package L#0 + L3 L#0 (25MB) + L2 L#0 (256KB) + > L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#20) > NUMANode L#1 (P#1 47GB) > HostBridge L#0 > PCIBridge > PCI 15b3:1003 > > I expected to see the second job get logical core #1 (on the second > physical core) but instead it gets logical core #20 (the second thread on > the first physical core). I can't imagine that this is ever the desired > behavior, but I guess I could be missing some use case. > > I have spent quite a bit of time reading the documentation/mailing list > and experimenting with different options, all to no avail. Is it possible > to achieve my desired setup with Slurm? > > I also experimented with setting threads_per_core=2 and then setting > OverSubscribe=FORCE:2, but I am pretty unhappy with the results. I think > it is confusing that you request one cpu and end up with two (with your > --mem-per-cpu doubled), and best I can tell there is no way to only > oversubscribe if the user requested 1 core instead of two. > > Thanks for your time, > Brendan > > > >
