Hi Brice, hi Chris, using the weights looks like a good solution to me. However, we need to think if and how we should propagate the weight via the upper levels of the hierarchy. So for example if you have a socket with 4 cores and cores 0&1 and 2&3 share L2 cache than the question is if and how the weight 1 for core 0 should propagate via L2 cache to core 1 (with some reduced factor).
L2 L2 L1 L1 L1 L1 C#0 C#1 C#2 C#3 hwloc-distrib --weight 1 socket:all.core:0 --single N should return N=1 => CORE=2 or 3 (definitely not CORE=1 because of common L2 cache between CORE#0 and CORE#1 ) N=2 => CORE=2,1 (not CORE#3 because of L2 cache topology) N=3 => CORE=2,1,3 N=4 => CORE=2,1,3,0 Implementing the weights is definitely a very nice feature but IMHO due to the need to propagate it through upper levels it can be quite complicated for the implementation, especially when multiple weights are specified at the command line. For example, consider this case hwloc-distrib --weight 10 socket:all.core:0 --single 2 for the topology described above. Should it return COREs 2 and 3 or COREs 2 and 1 ? (in other words, which weight should CORE#1 get when CORE#0 has weight 10 and they are connected via L2 cache ?) I think that --reverse option is much easier for the implementation and for the clear requirement and understanding how the output should look like. Any thoughts, comments on that? Thanks! Jirka On Tue, Aug 27, 2013 at 9:14 AM, Brice Goglin <brice.gog...@inria.fr> wrote: > Le 27/08/2013 05:07, Christopher Samuel a écrit : > > On 27/08/13 00:07, Brice Goglin wrote: > > > > > But there's a more general problem here, some people may want > > > something similar for other cases. I need to think about it. > > > > Something like a sort order perhaps, combined with some method to > > exclude or weight PUs based on some metrics (including a user defined > > weight)? > > Excluding is already supported with --restrict. > > > If you want to (if possible) avoid core 0 on each socket, and (at least) > avoid core 0 on the entire machine, you'd need a command-line like this: > > hwloc-distrib --weight 1 socket:all.core:0 --weight 2 core:0 ... > > Instead of doing > > if $(hwloc-calc -N pu all ~socket:all.core:0) -le $jobs; then > hwloc-distrib --restrict $(all ~socket:all.core:0) ... > else if $(hwloc-calc -N pu all ~core:0) -le $jobs; then > hwloc-distrib --restrict $(all ~core.0) ... > else > hwloc-distrib ... > fi > > > Another solution would be to have hwloc-distrib error-out when there are > not enough objects to distribute jobs. You'll do: > > hwloc-distrib --new-option --restrict $(all ~socket:all.core:0) ... > || hwloc-distrib --new-option --restrict $(all ~core.0) ... > || hwloc-distrib ... > > > And if you want to use entire cores instead of individual PUs, you can > still use "--to core" to stop distributing once you reach the core level. > > > > I had a quick poke around looking at /proc/irq/*/ and it would appear > > you can gather info about which CPUs are eligible to handle IRQs from > > the smp_affinity bitmask (or smp_affinity_list). > > smp_affinity_list is only accessible to root unfortunately, that's why > we never used it in hwloc. > > > The node file there just "shows the node to which the device using the > > IRQ reports itself as being attached. This hardware locality > > information does not include information about any possible driver > > locality preference." > > Ah, I missed the addition is "node" file. This one is world-accessible. > > Brice > > _______________________________________________ > hwloc-devel mailing list > hwloc-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel >
