Matt: Thanks for posting this thorough study!
I'm not going to comment on your specific examples below. However, I will say that we have also found that a lot of the time SLURM's built-in task distribution options do not meet our needs. Most of the time we just need simple, automatic binding of tasks to cores, and occaisionally when a user needs something more complex than that, none of the task/affinity options seems to exactly meet their needs. For these rare cases it is possible to develop a plugin for slurm using the spank(8) framework to basically allow you to do any kind of task binding you would like. See for example our auto-affinity plugin: http://code.google.com/p/slurm-spank-plugins/source/browse/auto-affinity.c For something even more generic, you can use the spank/lua plugin along with lua-schedutils (lua interface to setaffinity), to write task binding methods in lua. http://code.google.com/p/slurm-spank-plugins/source/browse/#git%2Flua%2Flua-schedutils I just wanted to point out these extra tools in case the built in affinity tools in SLURM cannot meet your needs for some reason. mark On Thu, 20 Oct 2011 02:48:42 -0700, Matteo Guglielmi <[email protected]> wrote: > Lets start with the output of "likwid-topology -g" command: > > ------------------------------------------------------------- > CPU type: AMD Magny Cours processor > ************************************************************* > Hardware Thread Topology > ************************************************************* > Sockets: 4 > Cores per socket: 12 > Threads per core: 1 > ------------------------------------------------------------- > HWThread Thread Core Socket > 0 0 0 0 > 1 0 1 0 > 2 0 2 0 > 3 0 3 0 > 4 0 4 0 > 5 0 5 0 > 6 0 6 0 > 7 0 7 0 > 8 0 8 0 > 9 0 9 0 > 10 0 10 0 > 11 0 11 0 > 12 0 0 1 > 13 0 1 1 > 14 0 2 1 > 15 0 3 1 > 16 0 4 1 > 17 0 5 1 > 18 0 6 1 > 19 0 7 1 > 20 0 8 1 > 21 0 9 1 > 22 0 10 1 > 23 0 11 1 > 24 0 0 2 > 25 0 1 2 > 26 0 2 2 > 27 0 3 2 > 28 0 4 2 > 29 0 5 2 > 30 0 6 2 > 31 0 7 2 > 32 0 8 2 > 33 0 9 2 > 34 0 10 2 > 35 0 11 2 > 36 0 0 3 > 37 0 1 3 > 38 0 2 3 > 39 0 3 3 > 40 0 4 3 > 41 0 5 3 > 42 0 6 3 > 43 0 7 3 > 44 0 8 3 > 45 0 9 3 > 46 0 10 3 > 47 0 11 3 > ------------------------------------------------------------- > Socket 0: ( 0 1 2 3 4 5 6 7 8 9 10 11 ) > Socket 1: ( 12 13 14 15 16 17 18 19 20 21 22 23 ) > Socket 2: ( 24 25 26 27 28 29 30 31 32 33 34 35 ) > Socket 3: ( 36 37 38 39 40 41 42 43 44 45 46 47 ) > ------------------------------------------------------------- > > ************************************************************* > Cache Topology > ************************************************************* > Level: 1 > Size: 64 kB > Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( > 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( > 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) ( > 32 ) ( 33 ) ( 34 ) ( 35 ) ( 36 ) ( 37 ) ( 38 ) ( 39 ) ( 40 ) ( 41 ) ( 42 ) ( > 43 ) ( 44 ) ( 45 ) ( 46 ) ( 47 ) > ------------------------------------------------------------- > Level: 2 > Size: 512 kB > Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( > 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( > 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) ( > 32 ) ( 33 ) ( 34 ) ( 35 ) ( 36 ) ( 37 ) ( 38 ) ( 39 ) ( 40 ) ( 41 ) ( 42 ) ( > 43 ) ( 44 ) ( 45 ) ( 46 ) ( 47 ) > ------------------------------------------------------------- > Level: 3 > Size: 5 MB > Cache groups: ( 0 1 2 3 4 5 ) ( 6 7 8 9 10 11 ) ( 12 13 14 15 16 17 ) ( 18 > 19 20 21 22 23 ) ( 24 25 26 27 28 29 ) ( 30 31 32 33 34 35 ) ( 36 37 38 39 40 > 41 ) ( 42 43 44 45 46 47 ) > ------------------------------------------------------------- > > ************************************************************* > NUMA Topology > ************************************************************* > NUMA domains: 8 > ------------------------------------------------------------- > Domain 0: > Processors: 0 1 2 3 4 5 > Memory: 14449.4 MB free of total 16349.9 MB > ------------------------------------------------------------- > Domain 1: > Processors: 6 7 8 9 10 11 > Memory: 15753.2 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 2: > Processors: 12 13 14 15 16 17 > Memory: 15793.6 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 3: > Processors: 18 19 20 21 22 23 > Memory: 16112.3 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 4: > Processors: 24 25 26 27 28 29 > Memory: 16081.1 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 5: > Processors: 30 31 32 33 34 35 > Memory: 16104.1 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 6: > Processors: 36 37 38 39 40 41 > Memory: 16064.2 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 7: > Processors: 42 43 44 45 46 47 > Memory: 14706 MB free of total 16384 MB > ------------------------------------------------------------- > > [software@foff09:~]$ likwid-topology -g > ------------------------------------------------------------- > CPU type: AMD Magny Cours processor > ************************************************************* > Hardware Thread Topology > ************************************************************* > Sockets: 4 > Cores per socket: 12 > Threads per core: 1 > ------------------------------------------------------------- > HWThread Thread Core Socket > 0 0 0 0 > 1 0 1 0 > 2 0 2 0 > 3 0 3 0 > 4 0 4 0 > 5 0 5 0 > 6 0 6 0 > 7 0 7 0 > 8 0 8 0 > 9 0 9 0 > 10 0 10 0 > 11 0 11 0 > 12 0 0 1 > 13 0 1 1 > 14 0 2 1 > 15 0 3 1 > 16 0 4 1 > 17 0 5 1 > 18 0 6 1 > 19 0 7 1 > 20 0 8 1 > 21 0 9 1 > 22 0 10 1 > 23 0 11 1 > 24 0 0 2 > 25 0 1 2 > 26 0 2 2 > 27 0 3 2 > 28 0 4 2 > 29 0 5 2 > 30 0 6 2 > 31 0 7 2 > 32 0 8 2 > 33 0 9 2 > 34 0 10 2 > 35 0 11 2 > 36 0 0 3 > 37 0 1 3 > 38 0 2 3 > 39 0 3 3 > 40 0 4 3 > 41 0 5 3 > 42 0 6 3 > 43 0 7 3 > 44 0 8 3 > 45 0 9 3 > 46 0 10 3 > 47 0 11 3 > ------------------------------------------------------------- > Socket 0: ( 0 1 2 3 4 5 6 7 8 9 10 11 ) > Socket 1: ( 12 13 14 15 16 17 18 19 20 21 22 23 ) > Socket 2: ( 24 25 26 27 28 29 30 31 32 33 34 35 ) > Socket 3: ( 36 37 38 39 40 41 42 43 44 45 46 47 ) > ------------------------------------------------------------- > > ************************************************************* > Cache Topology > ************************************************************* > Level: 1 > Size: 64 kB > Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( > 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( > 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) ( > 32 ) ( 33 ) ( 34 ) ( 35 ) ( 36 ) ( 37 ) ( 38 ) ( 39 ) ( 40 ) ( 41 ) ( 42 ) ( > 43 ) ( 44 ) ( 45 ) ( 46 ) ( 47 ) > ------------------------------------------------------------- > Level: 2 > Size: 512 kB > Cache groups: ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( > 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) ( 16 ) ( 17 ) ( 18 ) ( 19 ) ( 20 ) ( > 21 ) ( 22 ) ( 23 ) ( 24 ) ( 25 ) ( 26 ) ( 27 ) ( 28 ) ( 29 ) ( 30 ) ( 31 ) ( > 32 ) ( 33 ) ( 34 ) ( 35 ) ( 36 ) ( 37 ) ( 38 ) ( 39 ) ( 40 ) ( 41 ) ( 42 ) ( > 43 ) ( 44 ) ( 45 ) ( 46 ) ( 47 ) > ------------------------------------------------------------- > Level: 3 > Size: 5 MB > Cache groups: ( 0 1 2 3 4 5 ) ( 6 7 8 9 10 11 ) ( 12 13 14 15 16 17 ) ( 18 > 19 20 21 22 23 ) ( 24 25 26 27 28 29 ) ( 30 31 32 33 34 35 ) ( 36 37 38 39 40 > 41 ) ( 42 43 44 45 46 47 ) > ------------------------------------------------------------- > > ************************************************************* > NUMA Topology > ************************************************************* > NUMA domains: 8 > ------------------------------------------------------------- > Domain 0: > Processors: 0 1 2 3 4 5 > Memory: 14449.3 MB free of total 16349.9 MB > ------------------------------------------------------------- > Domain 1: > Processors: 6 7 8 9 10 11 > Memory: 15753 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 2: > Processors: 12 13 14 15 16 17 > Memory: 15793.6 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 3: > Processors: 18 19 20 21 22 23 > Memory: 16112 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 4: > Processors: 24 25 26 27 28 29 > Memory: 16081.1 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 5: > Processors: 30 31 32 33 34 35 > Memory: 16104.1 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 6: > Processors: 36 37 38 39 40 41 > Memory: 16064.2 MB free of total 16384 MB > ------------------------------------------------------------- > Domain 7: > Processors: 42 43 44 45 46 47 > Memory: 14706.1 MB free of total 16384 MB > ------------------------------------------------------------- > > ************************************************************* > Graphical: > ************************************************************* > Socket 0: > +-------------------------------------------------------------------------------------------------------------------------+ > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 0 | | 1 | | 2 | | 3 | | 4 | | 5 | | 6 | | 7 > | | 8 | | 9 | | 10 | | 11 | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > | | 5MB | | > 5MB | | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > +-------------------------------------------------------------------------------------------------------------------------+ > Socket 1: > +-------------------------------------------------------------------------------------------------------------------------+ > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 12 | | 13 | | 14 | | 15 | | 16 | | 17 | | 18 | | > 19 | | 20 | | 21 | | 22 | | 23 | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > | | 5MB | | > 5MB | | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > +-------------------------------------------------------------------------------------------------------------------------+ > Socket 2: > +-------------------------------------------------------------------------------------------------------------------------+ > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 24 | | 25 | | 26 | | 27 | | 28 | | 29 | | 30 | | > 31 | | 32 | | 33 | | 34 | | 35 | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > | | 5MB | | > 5MB | | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > +-------------------------------------------------------------------------------------------------------------------------+ > Socket 3: > +-------------------------------------------------------------------------------------------------------------------------+ > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 36 | | 37 | | 38 | | 39 | | 40 | | 41 | | 42 | | > 43 | | 44 | | 45 | | 46 | | 47 | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > 64kB | | 64kB | | 64kB | | 64kB | | 64kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > 512kB | | 512kB | | 512kB | | 512kB | | 512kB | | > | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ > +-------+ +-------+ +-------+ +-------+ +-------+ | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > | | 5MB | | > 5MB | | > | +---------------------------------------------------------+ > +---------------------------------------------------------+ | > +-------------------------------------------------------------------------------------------------------------------------+ > > > 1st) > > Does SLURM use the same mapping for numbering the cores? > > Hopefully yes... so if slurm says "I've placed this task on core #6" I know > that he placed it on the first socket at position > 6 (beginning of second half of the 6+6=12 AMD 6176 SE cpu). > > 2nd) > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=block:block > --ntasks-per-node=48 -l hostname | sort > > 00: foff09 > 01: foff09 > 02: foff09 > 03: foff09 > 04: foff09 > 05: foff09 > 06: foff09 > 07: foff09 > 08: foff09 > 09: foff09 > 10: foff09 > 11: foff09 > 12: foff09 > 13: foff09 > 14: foff09 > 15: foff09 > 16: foff09 > 17: foff09 > 18: foff09 > 19: foff09 > 20: foff09 > 21: foff09 > 22: foff09 > 23: foff09 > 24: foff09 > 25: foff09 > 26: foff09 > 27: foff09 > 28: foff09 > 29: foff09 > 30: foff09 > 31: foff09 > 32: foff09 > 33: foff09 > 34: foff09 > 35: foff09 > 36: foff09 > 37: foff09 > 38: foff09 > 39: foff09 > 40: foff09 > 41: foff09 > 42: foff09 > 43: foff09 > 44: foff09 > 45: foff09 > 46: foff09 > 47: foff09 > 48: foff10 > 49: foff10 > 50: foff10 > 51: foff10 > 52: foff10 > 53: foff10 > 54: foff10 > 55: foff10 > 56: foff10 > 57: foff10 > 58: foff10 > 59: foff10 > 60: foff10 > 61: foff10 > 62: foff10 > 63: foff10 > 64: foff10 > 65: foff10 > 66: foff10 > 67: foff10 > 68: foff10 > 69: foff10 > 70: foff10 > 71: foff10 > 72: foff10 > 73: foff10 > 74: foff10 > 75: foff10 > 76: foff10 > 77: foff10 > 78: foff10 > 79: foff10 > 80: foff10 > 81: foff10 > 82: foff10 > 83: foff10 > 84: foff10 > 85: foff10 > 86: foff10 > 87: foff10 > 88: foff10 > 89: foff10 > 90: foff10 > 91: foff10 > 92: foff10 > 93: foff10 > 94: foff10 > 95: foff10 > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=block:block > --ntasks-per-node=48 -l cat /proc/self/status | grep Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 6 > 02: Cpus_allowed_list: 1 > 03: Cpus_allowed_list: 7 > 04: Cpus_allowed_list: 2 > 05: Cpus_allowed_list: 8 > 06: Cpus_allowed_list: 3 > 07: Cpus_allowed_list: 9 > 08: Cpus_allowed_list: 4 > 09: Cpus_allowed_list: 10 > 10: Cpus_allowed_list: 5 > 11: Cpus_allowed_list: 11 > 12: Cpus_allowed_list: 12 > 13: Cpus_allowed_list: 18 > 14: Cpus_allowed_list: 13 > 15: Cpus_allowed_list: 19 > 16: Cpus_allowed_list: 14 > 17: Cpus_allowed_list: 20 > 18: Cpus_allowed_list: 15 > 19: Cpus_allowed_list: 21 > 20: Cpus_allowed_list: 16 > 21: Cpus_allowed_list: 22 > 22: Cpus_allowed_list: 17 > 23: Cpus_allowed_list: 23 > 24: Cpus_allowed_list: 24 > 25: Cpus_allowed_list: 30 > 26: Cpus_allowed_list: 25 > 27: Cpus_allowed_list: 31 > 28: Cpus_allowed_list: 26 > 29: Cpus_allowed_list: 32 > 30: Cpus_allowed_list: 27 > 31: Cpus_allowed_list: 33 > 32: Cpus_allowed_list: 28 > 33: Cpus_allowed_list: 34 > 34: Cpus_allowed_list: 29 > 35: Cpus_allowed_list: 35 > 36: Cpus_allowed_list: 36 > 37: Cpus_allowed_list: 42 > 38: Cpus_allowed_list: 37 > 39: Cpus_allowed_list: 43 > 40: Cpus_allowed_list: 38 > 41: Cpus_allowed_list: 44 > 42: Cpus_allowed_list: 39 > 43: Cpus_allowed_list: 45 > 44: Cpus_allowed_list: 40 > 45: Cpus_allowed_list: 46 > 46: Cpus_allowed_list: 41 > 47: Cpus_allowed_list: 47 > 48: Cpus_allowed_list: 0 > 49: Cpus_allowed_list: 6 > 50: Cpus_allowed_list: 1 > 51: Cpus_allowed_list: 7 > 52: Cpus_allowed_list: 2 > 53: Cpus_allowed_list: 8 > 54: Cpus_allowed_list: 3 > 55: Cpus_allowed_list: 9 > 56: Cpus_allowed_list: 4 > 57: Cpus_allowed_list: 10 > 58: Cpus_allowed_list: 5 > 59: Cpus_allowed_list: 11 > 60: Cpus_allowed_list: 12 > 61: Cpus_allowed_list: 18 > 62: Cpus_allowed_list: 13 > 63: Cpus_allowed_list: 19 > 64: Cpus_allowed_list: 14 > 65: Cpus_allowed_list: 20 > 66: Cpus_allowed_list: 15 > 67: Cpus_allowed_list: 21 > 68: Cpus_allowed_list: 16 > 69: Cpus_allowed_list: 22 > 70: Cpus_allowed_list: 17 > 71: Cpus_allowed_list: 23 > 72: Cpus_allowed_list: 24 > 73: Cpus_allowed_list: 30 > 74: Cpus_allowed_list: 25 > 75: Cpus_allowed_list: 31 > 76: Cpus_allowed_list: 26 > 77: Cpus_allowed_list: 32 > 78: Cpus_allowed_list: 27 > 79: Cpus_allowed_list: 33 > 80: Cpus_allowed_list: 28 > 81: Cpus_allowed_list: 34 > 82: Cpus_allowed_list: 29 > 83: Cpus_allowed_list: 35 > 84: Cpus_allowed_list: 36 > 85: Cpus_allowed_list: 42 > 86: Cpus_allowed_list: 37 > 87: Cpus_allowed_list: 43 > 88: Cpus_allowed_list: 38 > 89: Cpus_allowed_list: 44 > 90: Cpus_allowed_list: 39 > 91: Cpus_allowed_list: 45 > 92: Cpus_allowed_list: 40 > 93: Cpus_allowed_list: 46 > 94: Cpus_allowed_list: 41 > 95: Cpus_allowed_list: 47 > > Perfect. > > 3rd) > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=block:cyclic > --ntasks-per-node=48 -l hostname | sort > > 00: foff09 > 01: foff09 > 02: foff09 > 03: foff09 > 04: foff09 > 05: foff09 > 06: foff09 > 07: foff09 > 08: foff09 > 09: foff09 > 10: foff09 > 11: foff09 > 12: foff09 > 13: foff09 > 14: foff09 > 15: foff09 > 16: foff09 > 17: foff09 > 18: foff09 > 19: foff09 > 20: foff09 > 21: foff09 > 22: foff09 > 23: foff09 > 24: foff09 > 25: foff09 > 26: foff09 > 27: foff09 > 28: foff09 > 29: foff09 > 30: foff09 > 31: foff09 > 32: foff09 > 33: foff09 > 34: foff09 > 35: foff09 > 36: foff09 > 37: foff09 > 38: foff09 > 39: foff09 > 40: foff09 > 41: foff09 > 42: foff09 > 43: foff09 > 44: foff09 > 45: foff09 > 46: foff09 > 47: foff09 > 48: foff10 > 49: foff10 > 50: foff10 > 51: foff10 > 52: foff10 > 53: foff10 > 54: foff10 > 55: foff10 > 56: foff10 > 57: foff10 > 58: foff10 > 59: foff10 > 60: foff10 > 61: foff10 > 62: foff10 > 63: foff10 > 64: foff10 > 65: foff10 > 66: foff10 > 67: foff10 > 68: foff10 > 69: foff10 > 70: foff10 > 71: foff10 > 72: foff10 > 73: foff10 > 74: foff10 > 75: foff10 > 76: foff10 > 77: foff10 > 78: foff10 > 79: foff10 > 80: foff10 > 81: foff10 > 82: foff10 > 83: foff10 > 84: foff10 > 85: foff10 > 86: foff10 > 87: foff10 > 88: foff10 > 89: foff10 > 90: foff10 > 91: foff10 > 92: foff10 > 93: foff10 > 94: foff10 > 95: foff10 > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=block:cyclic > --ntasks-per-node=48 -l cat /proc/self/status | grep Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 12 > 02: Cpus_allowed_list: 24 > 03: Cpus_allowed_list: 36 > 04: Cpus_allowed_list: 6 > 05: Cpus_allowed_list: 18 > 06: Cpus_allowed_list: 30 > 07: Cpus_allowed_list: 42 > 08: Cpus_allowed_list: 1 > 09: Cpus_allowed_list: 13 > 10: Cpus_allowed_list: 25 > 11: Cpus_allowed_list: 37 > 12: Cpus_allowed_list: 7 > 13: Cpus_allowed_list: 19 > 14: Cpus_allowed_list: 31 > 15: Cpus_allowed_list: 43 > 16: Cpus_allowed_list: 2 > 17: Cpus_allowed_list: 14 > 18: Cpus_allowed_list: 26 > 19: Cpus_allowed_list: 38 > 20: Cpus_allowed_list: 8 > 21: Cpus_allowed_list: 20 > 22: Cpus_allowed_list: 32 > 23: Cpus_allowed_list: 44 > 24: Cpus_allowed_list: 3 > 25: Cpus_allowed_list: 15 > 26: Cpus_allowed_list: 27 > 27: Cpus_allowed_list: 39 > 28: Cpus_allowed_list: 9 > 29: Cpus_allowed_list: 21 > 30: Cpus_allowed_list: 33 > 31: Cpus_allowed_list: 45 > 32: Cpus_allowed_list: 4 > 33: Cpus_allowed_list: 16 > 34: Cpus_allowed_list: 28 > 35: Cpus_allowed_list: 40 > 36: Cpus_allowed_list: 10 > 37: Cpus_allowed_list: 22 > 38: Cpus_allowed_list: 34 > 39: Cpus_allowed_list: 46 > 40: Cpus_allowed_list: 5 > 41: Cpus_allowed_list: 17 > 42: Cpus_allowed_list: 29 > 43: Cpus_allowed_list: 41 > 44: Cpus_allowed_list: 11 > 45: Cpus_allowed_list: 23 > 46: Cpus_allowed_list: 35 > 47: Cpus_allowed_list: 47 > 48: Cpus_allowed_list: 0 > 49: Cpus_allowed_list: 12 > 50: Cpus_allowed_list: 24 > 51: Cpus_allowed_list: 36 > 52: Cpus_allowed_list: 6 > 53: Cpus_allowed_list: 18 > 54: Cpus_allowed_list: 30 > 55: Cpus_allowed_list: 42 > 56: Cpus_allowed_list: 1 > 57: Cpus_allowed_list: 13 > 58: Cpus_allowed_list: 25 > 59: Cpus_allowed_list: 37 > 60: Cpus_allowed_list: 7 > 61: Cpus_allowed_list: 19 > 62: Cpus_allowed_list: 31 > 63: Cpus_allowed_list: 43 > 64: Cpus_allowed_list: 2 > 65: Cpus_allowed_list: 14 > 66: Cpus_allowed_list: 26 > 67: Cpus_allowed_list: 38 > 68: Cpus_allowed_list: 8 > 69: Cpus_allowed_list: 20 > 70: Cpus_allowed_list: 32 > 71: Cpus_allowed_list: 44 > 72: Cpus_allowed_list: 3 > 73: Cpus_allowed_list: 15 > 74: Cpus_allowed_list: 27 > 75: Cpus_allowed_list: 39 > 76: Cpus_allowed_list: 9 > 77: Cpus_allowed_list: 21 > 78: Cpus_allowed_list: 33 > 79: Cpus_allowed_list: 45 > 80: Cpus_allowed_list: 4 > 81: Cpus_allowed_list: 16 > 82: Cpus_allowed_list: 28 > 83: Cpus_allowed_list: 40 > 84: Cpus_allowed_list: 10 > 85: Cpus_allowed_list: 22 > 86: Cpus_allowed_list: 34 > 87: Cpus_allowed_list: 46 > 88: Cpus_allowed_list: 5 > 89: Cpus_allowed_list: 17 > 90: Cpus_allowed_list: 29 > 91: Cpus_allowed_list: 41 > 92: Cpus_allowed_list: 11 > 93: Cpus_allowed_list: 23 > 94: Cpus_allowed_list: 35 > 95: Cpus_allowed_list: 47 > > Perfect. > > 4th) > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=cyclic:block > --ntasks-per-node=48 -l hostname | sort > > 00: foff09 > 01: foff10 > 02: foff09 > 03: foff10 > 04: foff09 > 05: foff10 > 06: foff09 > 07: foff10 > 08: foff09 > 09: foff10 > 10: foff09 > 11: foff10 > 12: foff09 > 13: foff10 > 14: foff09 > 15: foff10 > 16: foff09 > 17: foff10 > 18: foff09 > 19: foff10 > 20: foff09 > 21: foff10 > 22: foff09 > 23: foff10 > 24: foff09 > 25: foff10 > 26: foff09 > 27: foff10 > 28: foff09 > 29: foff10 > 30: foff09 > 31: foff10 > 32: foff09 > 33: foff10 > 34: foff09 > 35: foff10 > 36: foff09 > 37: foff10 > 38: foff09 > 39: foff10 > 40: foff09 > 41: foff10 > 42: foff09 > 43: foff10 > 44: foff09 > 45: foff10 > 46: foff09 > 47: foff10 > 48: foff09 > 49: foff10 > 50: foff09 > 51: foff10 > 52: foff09 > 53: foff10 > 54: foff09 > 55: foff10 > 56: foff09 > 57: foff10 > 58: foff09 > 59: foff10 > 60: foff09 > 61: foff10 > 62: foff09 > 63: foff10 > 64: foff09 > 65: foff10 > 66: foff09 > 67: foff10 > 68: foff09 > 69: foff10 > 70: foff09 > 71: foff10 > 72: foff09 > 73: foff10 > 74: foff09 > 75: foff10 > 76: foff09 > 77: foff10 > 78: foff09 > 79: foff10 > 80: foff09 > 81: foff10 > 82: foff09 > 83: foff10 > 84: foff09 > 85: foff10 > 86: foff09 > 87: foff10 > 88: foff09 > 89: foff10 > 90: foff09 > 91: foff10 > 92: foff09 > 93: foff10 > 94: foff09 > 95: foff10 > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=cyclic:block > --ntasks-per-node=48 -l cat /proc/self/status | grep Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 0 > 02: Cpus_allowed_list: 6 > 03: Cpus_allowed_list: 6 > 04: Cpus_allowed_list: 1 > 05: Cpus_allowed_list: 1 > 06: Cpus_allowed_list: 7 > 07: Cpus_allowed_list: 7 > 08: Cpus_allowed_list: 2 > 09: Cpus_allowed_list: 2 > 10: Cpus_allowed_list: 8 > 11: Cpus_allowed_list: 8 > 12: Cpus_allowed_list: 3 > 13: Cpus_allowed_list: 3 > 14: Cpus_allowed_list: 9 > 15: Cpus_allowed_list: 9 > 16: Cpus_allowed_list: 4 > 17: Cpus_allowed_list: 4 > 18: Cpus_allowed_list: 10 > 19: Cpus_allowed_list: 10 > 20: Cpus_allowed_list: 5 > 21: Cpus_allowed_list: 5 > 22: Cpus_allowed_list: 11 > 23: Cpus_allowed_list: 11 > 24: Cpus_allowed_list: 12 > 25: Cpus_allowed_list: 12 > 26: Cpus_allowed_list: 18 > 27: Cpus_allowed_list: 18 > 28: Cpus_allowed_list: 13 > 29: Cpus_allowed_list: 13 > 30: Cpus_allowed_list: 19 > 31: Cpus_allowed_list: 19 > 32: Cpus_allowed_list: 14 > 33: Cpus_allowed_list: 14 > 34: Cpus_allowed_list: 20 > 35: Cpus_allowed_list: 20 > 36: Cpus_allowed_list: 15 > 37: Cpus_allowed_list: 15 > 38: Cpus_allowed_list: 21 > 39: Cpus_allowed_list: 21 > 40: Cpus_allowed_list: 16 > 41: Cpus_allowed_list: 16 > 42: Cpus_allowed_list: 22 > 43: Cpus_allowed_list: 22 > 44: Cpus_allowed_list: 17 > 45: Cpus_allowed_list: 17 > 46: Cpus_allowed_list: 23 > 47: Cpus_allowed_list: 23 > 48: Cpus_allowed_list: 24 > 49: Cpus_allowed_list: 24 > 50: Cpus_allowed_list: 30 > 51: Cpus_allowed_list: 30 > 52: Cpus_allowed_list: 25 > 53: Cpus_allowed_list: 25 > 54: Cpus_allowed_list: 31 > 55: Cpus_allowed_list: 31 > 56: Cpus_allowed_list: 26 > 57: Cpus_allowed_list: 26 > 58: Cpus_allowed_list: 32 > 59: Cpus_allowed_list: 32 > 60: Cpus_allowed_list: 27 > 61: Cpus_allowed_list: 27 > 62: Cpus_allowed_list: 33 > 63: Cpus_allowed_list: 33 > 64: Cpus_allowed_list: 28 > 65: Cpus_allowed_list: 28 > 66: Cpus_allowed_list: 34 > 67: Cpus_allowed_list: 34 > 68: Cpus_allowed_list: 29 > 69: Cpus_allowed_list: 29 > 70: Cpus_allowed_list: 35 > 71: Cpus_allowed_list: 35 > 72: Cpus_allowed_list: 36 > 73: Cpus_allowed_list: 36 > 74: Cpus_allowed_list: 42 > 75: Cpus_allowed_list: 42 > 76: Cpus_allowed_list: 37 > 77: Cpus_allowed_list: 37 > 78: Cpus_allowed_list: 43 > 79: Cpus_allowed_list: 43 > 80: Cpus_allowed_list: 38 > 81: Cpus_allowed_list: 38 > 82: Cpus_allowed_list: 44 > 83: Cpus_allowed_list: 44 > 84: Cpus_allowed_list: 39 > 85: Cpus_allowed_list: 39 > 86: Cpus_allowed_list: 45 > 87: Cpus_allowed_list: 45 > 88: Cpus_allowed_list: 40 > 89: Cpus_allowed_list: 40 > 90: Cpus_allowed_list: 46 > 91: Cpus_allowed_list: 46 > 92: Cpus_allowed_list: 41 > 93: Cpus_allowed_list: 41 > 94: Cpus_allowed_list: 47 > 95: Cpus_allowed_list: 47 > > Perfect. > > 5th) > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=cyclic:cyclic > --ntasks-per-node=48 -l hostname | sort > > 00: foff09 > 01: foff10 > 02: foff09 > 03: foff10 > 04: foff09 > 05: foff10 > 06: foff09 > 07: foff10 > 08: foff09 > 09: foff10 > 10: foff09 > 11: foff10 > 12: foff09 > 13: foff10 > 14: foff09 > 15: foff10 > 16: foff09 > 17: foff10 > 18: foff09 > 19: foff10 > 20: foff09 > 21: foff10 > 22: foff09 > 23: foff10 > 24: foff09 > 25: foff10 > 26: foff09 > 27: foff10 > 28: foff09 > 29: foff10 > 30: foff09 > 31: foff10 > 32: foff09 > 33: foff10 > 34: foff09 > 35: foff10 > 36: foff09 > 37: foff10 > 38: foff09 > 39: foff10 > 40: foff09 > 41: foff10 > 42: foff09 > 43: foff10 > 44: foff09 > 45: foff10 > 46: foff09 > 47: foff10 > 48: foff09 > 49: foff10 > 50: foff09 > 51: foff10 > 52: foff09 > 53: foff10 > 54: foff09 > 55: foff10 > 56: foff09 > 57: foff10 > 58: foff09 > 59: foff10 > 60: foff09 > 61: foff10 > 62: foff09 > 63: foff10 > 64: foff09 > 65: foff10 > 66: foff09 > 67: foff10 > 68: foff09 > 69: foff10 > 70: foff09 > 71: foff10 > 72: foff09 > 73: foff10 > 74: foff09 > 75: foff10 > 76: foff09 > 77: foff10 > 78: foff09 > 79: foff10 > 80: foff09 > 81: foff10 > 82: foff09 > 83: foff10 > 84: foff09 > 85: foff10 > 86: foff09 > 87: foff10 > 88: foff09 > 89: foff10 > 90: foff09 > 91: foff10 > 92: foff09 > 93: foff10 > 94: foff09 > 95: foff10 > > srun -A gr-fo -p foff2 -n 96 -N 2 --distribution=cyclic:cyclic > --ntasks-per-node=48 -l cat /proc/self/status | grep Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 0 > 02: Cpus_allowed_list: 12 > 03: Cpus_allowed_list: 12 > 04: Cpus_allowed_list: 24 > 05: Cpus_allowed_list: 24 > 06: Cpus_allowed_list: 36 > 07: Cpus_allowed_list: 36 > 08: Cpus_allowed_list: 6 > 09: Cpus_allowed_list: 6 > 10: Cpus_allowed_list: 18 > 11: Cpus_allowed_list: 18 > 12: Cpus_allowed_list: 30 > 13: Cpus_allowed_list: 30 > 14: Cpus_allowed_list: 42 > 15: Cpus_allowed_list: 42 > 16: Cpus_allowed_list: 1 > 17: Cpus_allowed_list: 1 > 18: Cpus_allowed_list: 13 > 19: Cpus_allowed_list: 13 > 20: Cpus_allowed_list: 25 > 21: Cpus_allowed_list: 25 > 22: Cpus_allowed_list: 37 > 23: Cpus_allowed_list: 37 > 24: Cpus_allowed_list: 7 > 25: Cpus_allowed_list: 7 > 26: Cpus_allowed_list: 19 > 27: Cpus_allowed_list: 19 > 28: Cpus_allowed_list: 31 > 29: Cpus_allowed_list: 31 > 30: Cpus_allowed_list: 43 > 31: Cpus_allowed_list: 43 > 32: Cpus_allowed_list: 2 > 33: Cpus_allowed_list: 2 > 34: Cpus_allowed_list: 14 > 35: Cpus_allowed_list: 14 > 36: Cpus_allowed_list: 26 > 37: Cpus_allowed_list: 26 > 38: Cpus_allowed_list: 38 > 39: Cpus_allowed_list: 38 > 40: Cpus_allowed_list: 8 > 41: Cpus_allowed_list: 8 > 42: Cpus_allowed_list: 20 > 43: Cpus_allowed_list: 20 > 44: Cpus_allowed_list: 32 > 45: Cpus_allowed_list: 32 > 46: Cpus_allowed_list: 44 > 47: Cpus_allowed_list: 44 > 48: Cpus_allowed_list: 3 > 49: Cpus_allowed_list: 3 > 50: Cpus_allowed_list: 15 > 51: Cpus_allowed_list: 15 > 52: Cpus_allowed_list: 27 > 53: Cpus_allowed_list: 27 > 54: Cpus_allowed_list: 39 > 55: Cpus_allowed_list: 39 > 56: Cpus_allowed_list: 9 > 57: Cpus_allowed_list: 9 > 58: Cpus_allowed_list: 21 > 59: Cpus_allowed_list: 21 > 60: Cpus_allowed_list: 33 > 61: Cpus_allowed_list: 33 > 62: Cpus_allowed_list: 45 > 63: Cpus_allowed_list: 45 > 64: Cpus_allowed_list: 4 > 65: Cpus_allowed_list: 4 > 66: Cpus_allowed_list: 16 > 67: Cpus_allowed_list: 16 > 68: Cpus_allowed_list: 28 > 69: Cpus_allowed_list: 28 > 70: Cpus_allowed_list: 40 > 71: Cpus_allowed_list: 40 > 72: Cpus_allowed_list: 10 > 73: Cpus_allowed_list: 10 > 74: Cpus_allowed_list: 22 > 75: Cpus_allowed_list: 22 > 76: Cpus_allowed_list: 34 > 77: Cpus_allowed_list: 34 > 78: Cpus_allowed_list: 46 > 79: Cpus_allowed_list: 46 > 80: Cpus_allowed_list: 5 > 81: Cpus_allowed_list: 5 > 82: Cpus_allowed_list: 17 > 83: Cpus_allowed_list: 17 > 84: Cpus_allowed_list: 29 > 85: Cpus_allowed_list: 29 > 86: Cpus_allowed_list: 41 > 87: Cpus_allowed_list: 41 > 88: Cpus_allowed_list: 11 > 89: Cpus_allowed_list: 11 > 90: Cpus_allowed_list: 23 > 91: Cpus_allowed_list: 23 > 92: Cpus_allowed_list: 35 > 93: Cpus_allowed_list: 35 > 94: Cpus_allowed_list: 47 > 95: Cpus_allowed_list: 47 > > Perfect. > > 6th) > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=block:block > --ntasks-per-node=24 -l hostname | sort > > 00: foff09 > 01: foff09 > 02: foff09 > 03: foff09 > 04: foff09 > 05: foff09 > 06: foff09 > 07: foff09 > 08: foff09 > 09: foff09 > 10: foff09 > 11: foff09 > 12: foff09 > 13: foff09 > 14: foff09 > 15: foff09 > 16: foff09 > 17: foff09 > 18: foff09 > 19: foff09 > 20: foff09 > 21: foff09 > 22: foff09 > 23: foff09 > 24: foff10 > 25: foff10 > 26: foff10 > 27: foff10 > 28: foff10 > 29: foff10 > 30: foff10 > 31: foff10 > 32: foff10 > 33: foff10 > 34: foff10 > 35: foff10 > 36: foff10 > 37: foff10 > 38: foff10 > 39: foff10 > 40: foff10 > 41: foff10 > 42: foff10 > 43: foff10 > 44: foff10 > 45: foff10 > 46: foff10 > 47: foff10 > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=block:block > --ntasks-per-node=24 -l cat /proc/self/status | grep Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 6 > 02: Cpus_allowed_list: 1 > 03: Cpus_allowed_list: 7 > 04: Cpus_allowed_list: 2 > 05: Cpus_allowed_list: 8 > 06: Cpus_allowed_list: 3 > 07: Cpus_allowed_list: 9 > 08: Cpus_allowed_list: 4 > 09: Cpus_allowed_list: 10 > 10: Cpus_allowed_list: 5 > 11: Cpus_allowed_list: 11 > 12: Cpus_allowed_list: 12 > 13: Cpus_allowed_list: 18 > 14: Cpus_allowed_list: 13 > 15: Cpus_allowed_list: 19 > 16: Cpus_allowed_list: 14 > 17: Cpus_allowed_list: 20 > 18: Cpus_allowed_list: 15 > 19: Cpus_allowed_list: 21 > 20: Cpus_allowed_list: 16 > 21: Cpus_allowed_list: 22 > 22: Cpus_allowed_list: 17 > 23: Cpus_allowed_list: 23 > 24: Cpus_allowed_list: 0 > 25: Cpus_allowed_list: 6 > 26: Cpus_allowed_list: 1 > 27: Cpus_allowed_list: 7 > 28: Cpus_allowed_list: 2 > 29: Cpus_allowed_list: 8 > 30: Cpus_allowed_list: 3 > 31: Cpus_allowed_list: 9 > 32: Cpus_allowed_list: 4 > 33: Cpus_allowed_list: 10 > 34: Cpus_allowed_list: 5 > 35: Cpus_allowed_list: 11 > 36: Cpus_allowed_list: 12 > 37: Cpus_allowed_list: 18 > 38: Cpus_allowed_list: 13 > 39: Cpus_allowed_list: 19 > 40: Cpus_allowed_list: 14 > 41: Cpus_allowed_list: 20 > 42: Cpus_allowed_list: 15 > 43: Cpus_allowed_list: 21 > 44: Cpus_allowed_list: 16 > 45: Cpus_allowed_list: 22 > 46: Cpus_allowed_list: 17 > 47: Cpus_allowed_list: 23 > > Perfect. > > 7th) > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=block:cyclic > --ntasks-per-node=24 -l hostname | sort > > 00: foff09 > 01: foff09 > 02: foff09 > 03: foff09 > 04: foff09 > 05: foff09 > 06: foff09 > 07: foff09 > 08: foff09 > 09: foff09 > 10: foff09 > 11: foff09 > 12: foff09 > 13: foff09 > 14: foff09 > 15: foff09 > 16: foff09 > 17: foff09 > 18: foff09 > 19: foff09 > 20: foff09 > 21: foff09 > 22: foff09 > 23: foff09 > 24: foff10 > 25: foff10 > 26: foff10 > 27: foff10 > 28: foff10 > 29: foff10 > 30: foff10 > 31: foff10 > 32: foff10 > 33: foff10 > 34: foff10 > 35: foff10 > 36: foff10 > 37: foff10 > 38: foff10 > 39: foff10 > 40: foff10 > 41: foff10 > 42: foff10 > 43: foff10 > 44: foff10 > 45: foff10 > 46: foff10 > 47: foff10 > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=block:cyclic > --ntasks-per-node=24 -l cat /proc/self/status | grep Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 12 > 02: Cpus_allowed_list: 6 > 03: Cpus_allowed_list: 18 > 04: Cpus_allowed_list: 1 > 05: Cpus_allowed_list: 13 > 06: Cpus_allowed_list: 7 > 07: Cpus_allowed_list: 19 > 08: Cpus_allowed_list: 2 > 09: Cpus_allowed_list: 14 > 10: Cpus_allowed_list: 8 > 11: Cpus_allowed_list: 20 > 12: Cpus_allowed_list: 3 > 13: Cpus_allowed_list: 15 > 14: Cpus_allowed_list: 9 > 15: Cpus_allowed_list: 21 > 16: Cpus_allowed_list: 4 > 17: Cpus_allowed_list: 16 > 18: Cpus_allowed_list: 10 > 19: Cpus_allowed_list: 22 > 20: Cpus_allowed_list: 5 > 21: Cpus_allowed_list: 17 > 22: Cpus_allowed_list: 11 > 23: Cpus_allowed_list: 23 > 24: Cpus_allowed_list: 0 > 25: Cpus_allowed_list: 12 > 26: Cpus_allowed_list: 6 > 27: Cpus_allowed_list: 18 > 28: Cpus_allowed_list: 1 > 29: Cpus_allowed_list: 13 > 30: Cpus_allowed_list: 7 > 31: Cpus_allowed_list: 19 > 32: Cpus_allowed_list: 2 > 33: Cpus_allowed_list: 14 > 34: Cpus_allowed_list: 8 > 35: Cpus_allowed_list: 20 > 36: Cpus_allowed_list: 3 > 37: Cpus_allowed_list: 15 > 38: Cpus_allowed_list: 9 > 39: Cpus_allowed_list: 21 > 40: Cpus_allowed_list: 4 > 41: Cpus_allowed_list: 16 > 42: Cpus_allowed_list: 10 > 43: Cpus_allowed_list: 22 > 44: Cpus_allowed_list: 5 > 45: Cpus_allowed_list: 17 > 46: Cpus_allowed_list: 11 > 47: Cpus_allowed_list: 23 > > Why is it NOT distributing the resources across ALL sockets (4) within a node? > > How can I force this behaviour? > > 8th) > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=cyclic:block > --ntasks-per-node=24 -l hostname | sort > > 00: foff09 > 01: foff10 > 02: foff09 > 03: foff10 > 04: foff09 > 05: foff10 > 06: foff09 > 07: foff10 > 08: foff09 > 09: foff10 > 10: foff09 > 11: foff10 > 12: foff09 > 13: foff10 > 14: foff09 > 15: foff10 > 16: foff09 > 17: foff10 > 18: foff09 > 19: foff10 > 20: foff09 > 21: foff10 > 22: foff09 > 23: foff10 > 24: foff09 > 25: foff10 > 26: foff09 > 27: foff10 > 28: foff09 > 29: foff10 > 30: foff09 > 31: foff10 > 32: foff09 > 33: foff10 > 34: foff09 > 35: foff10 > 36: foff09 > 37: foff10 > 38: foff09 > 39: foff10 > 40: foff09 > 41: foff10 > 42: foff09 > 43: foff10 > 44: foff09 > 45: foff10 > 46: foff09 > 47: foff10 > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=cyclic:block > --ntasks-per-node=24 --sockets-per-node=4 -l cat /proc/self/status | grep > Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 0 > 02: Cpus_allowed_list: 6 > 03: Cpus_allowed_list: 6 > 04: Cpus_allowed_list: 1 > 05: Cpus_allowed_list: 1 > 06: Cpus_allowed_list: 7 > 07: Cpus_allowed_list: 7 > 08: Cpus_allowed_list: 2 > 09: Cpus_allowed_list: 2 > 10: Cpus_allowed_list: 8 > 11: Cpus_allowed_list: 8 > 12: Cpus_allowed_list: 3 > 13: Cpus_allowed_list: 3 > 14: Cpus_allowed_list: 9 > 15: Cpus_allowed_list: 9 > 16: Cpus_allowed_list: 4 > 17: Cpus_allowed_list: 4 > 18: Cpus_allowed_list: 10 > 19: Cpus_allowed_list: 10 > 20: Cpus_allowed_list: 5 > 21: Cpus_allowed_list: 5 > 22: Cpus_allowed_list: 11 > 23: Cpus_allowed_list: 11 > 24: Cpus_allowed_list: 12 > 25: Cpus_allowed_list: 12 > 26: Cpus_allowed_list: 18 > 27: Cpus_allowed_list: 18 > 28: Cpus_allowed_list: 13 > 29: Cpus_allowed_list: 13 > 30: Cpus_allowed_list: 19 > 31: Cpus_allowed_list: 19 > 32: Cpus_allowed_list: 14 > 33: Cpus_allowed_list: 14 > 34: Cpus_allowed_list: 20 > 35: Cpus_allowed_list: 20 > 36: Cpus_allowed_list: 15 > 37: Cpus_allowed_list: 15 > 38: Cpus_allowed_list: 21 > 39: Cpus_allowed_list: 21 > 40: Cpus_allowed_list: 16 > 41: Cpus_allowed_list: 16 > 42: Cpus_allowed_list: 22 > 43: Cpus_allowed_list: 22 > 44: Cpus_allowed_list: 17 > 45: Cpus_allowed_list: 17 > 46: Cpus_allowed_list: 23 > 47: Cpus_allowed_list: 23 > > Perfect. > > 9th) > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=cyclic:cyclic > --ntasks-per-node=24 -l hostname | sort > > 00: foff09 > 01: foff10 > 02: foff09 > 03: foff10 > 04: foff09 > 05: foff10 > 06: foff09 > 07: foff10 > 08: foff09 > 09: foff10 > 10: foff09 > 11: foff10 > 12: foff09 > 13: foff10 > 14: foff09 > 15: foff10 > 16: foff09 > 17: foff10 > 18: foff09 > 19: foff10 > 20: foff09 > 21: foff10 > 22: foff09 > 23: foff10 > 24: foff09 > 25: foff10 > 26: foff09 > 27: foff10 > 28: foff09 > 29: foff10 > 30: foff09 > 31: foff10 > 32: foff09 > 33: foff10 > 34: foff09 > 35: foff10 > 36: foff09 > 37: foff10 > 38: foff09 > 39: foff10 > 40: foff09 > 41: foff10 > 42: foff09 > 43: foff10 > 44: foff09 > 45: foff10 > 46: foff09 > 47: foff10 > > srun -A gr-fo -p foff2 -n 48 -N 2 --distribution=cyclic:cyclic > --ntasks-per-node=24 --sockets-per-node=4 -l cat /proc/self/status | grep > Cpus_allowed_list | sort > > 00: Cpus_allowed_list: 0 > 01: Cpus_allowed_list: 0 > 02: Cpus_allowed_list: 12 > 03: Cpus_allowed_list: 12 > 04: Cpus_allowed_list: 6 > 05: Cpus_allowed_list: 6 > 06: Cpus_allowed_list: 18 > 07: Cpus_allowed_list: 18 > 08: Cpus_allowed_list: 1 > 09: Cpus_allowed_list: 1 > 10: Cpus_allowed_list: 13 > 11: Cpus_allowed_list: 13 > 12: Cpus_allowed_list: 7 > 13: Cpus_allowed_list: 7 > 14: Cpus_allowed_list: 19 > 15: Cpus_allowed_list: 19 > 16: Cpus_allowed_list: 2 > 17: Cpus_allowed_list: 2 > 18: Cpus_allowed_list: 14 > 19: Cpus_allowed_list: 14 > 20: Cpus_allowed_list: 8 > 21: Cpus_allowed_list: 8 > 22: Cpus_allowed_list: 20 > 23: Cpus_allowed_list: 20 > 24: Cpus_allowed_list: 3 > 25: Cpus_allowed_list: 3 > 26: Cpus_allowed_list: 15 > 27: Cpus_allowed_list: 15 > 28: Cpus_allowed_list: 9 > 29: Cpus_allowed_list: 9 > 30: Cpus_allowed_list: 21 > 31: Cpus_allowed_list: 21 > 32: Cpus_allowed_list: 4 > 33: Cpus_allowed_list: 4 > 34: Cpus_allowed_list: 16 > 35: Cpus_allowed_list: 16 > 36: Cpus_allowed_list: 10 > 37: Cpus_allowed_list: 10 > 38: Cpus_allowed_list: 22 > 39: Cpus_allowed_list: 22 > 40: Cpus_allowed_list: 5 > 41: Cpus_allowed_list: 5 > 42: Cpus_allowed_list: 17 > 43: Cpus_allowed_list: 17 > 44: Cpus_allowed_list: 11 > 45: Cpus_allowed_list: 11 > 46: Cpus_allowed_list: 23 > 47: Cpus_allowed_list: 23 > > Same question here: why is it NOT cycling over 4 sockets? > > > Thanks, > > --matt >
