> -----Original Message-----
> From: Jens Axboe [mailto:[email protected]]
> Sent: Wednesday, 26 February, 2014 6:08 PM
> To: Elliott, Robert (Server Storage); [email protected]
> Subject: Re: cpus_allowed per thread behavior
>
> On 2014-02-26 15:54, Elliott, Robert (Server Storage) wrote:
> > fio seems to assign the same cpus_allowed/cpumask value to all threads.
> > I think this allows the OS to move the threads around those CPUs.
>
> Correct. As long as the number of cpus in the mask is equal to (or
> larger than) the number of jobs within that group, the OS is free to
> place them wherever it wants. In practice, unless the CPU scheduling is
> horribly broken, they tend to "stick" for most intents and purposes.
>
> > In comparison, iometer assigns its worker threads to specific CPUs
> > within the cpumask in round-robin manner. Would that be worth adding
> > to fio, perhaps with an option like cpus_allowed_policy=roundrobin?
>
> Sure, we could add that feature. You can get the same setup now, if you
> "unroll" the job section, but that might not always be practical. How
> about cpus_allowed_policy, with 'shared' being the existing (and
> default) behavior and 'split' being each thread grabbing one of the CPUs?
Perhaps NUMA and hyperthreading aware allocation policies would
also be useful?
I don't know how consistent hyperthread CPU numbering is across
systems. On some servers I've tried, linux assigns 0-5 to the main
cores and 6-11 to the hyperthreaded siblings, while Windows assigns
0,2,4,6,8,10 to the main cores and 1,3,5,7,9,11 to their
hyperthreaded siblings.
Intel's OpenMP library offers two thread affinity types that might
be worth simulating:
COMPACT: pack them tightly
foreach (node)
foreach (core in the node)
foreach (hyperthreaded sibling)
SCATTER: spread across all the cores
foreach (hyperthreaded sibling)
foreach (core sharing a node)
foreach (node)
We could try:
cpus_allowed_policy=shared
cpus_allowed_policy=split (round-robin, don't care how the
CPU IDs were assigned)
cpus_allowed_policy=compact (NUMA/HT aware)
cpus_allowed_policy=scatter (NUMA/HT aware)
> --
> Jens Axboe
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
