Re: [hwloc-users] Build warnings with hwloc-2.0.3

2019-03-18 Thread Brice Goglin
Hello Pavan

I am planning to fix this in 2.1 (to be released before summer). I'll
backport the trivial pieces to 2.0.x too.

Do you care about a specific value passed to -Wstack-usage=X? or do you
just want to avoid dynamic/unbounded allocs on the stack?

Brice



Le 18/03/2019 à 15:04, Balaji, Pavan via hwloc-users a écrit :
> Brice, all,
>
> Any update on this?  Are you guys planning on fixing these?
>
>   -- Pavan
>
>> On Feb 25, 2019, at 7:33 AM, Balaji, Pavan via hwloc-users 
>>  wrote:
>>
>> Hi Brice,
>>
>>> On Feb 25, 2019, at 2:27 AM, Brice Goglin  wrote:
>>> Are you sure you're not passing -Wstack-usage? My Ubuntu 18.04 with
>>> latest gcc-7 (7.3.0-27ubuntu1~18.04) doesn't show any of those warnings.
>> Yes, you are right, -Wstack-usage was explicitly added too.  Sorry, I missed 
>> the fact that it wasn't default in -Wall.
>>
>>> It looks like all these warnings are caused by C99 variable-length
>>> arrays (except 2 that I don't understand). I know the kernel devs
>>> stopped using VLA recently, and it looks like C11 made them optional.
>>> But are we really supposed to stop using VLA already?
>> They are optional, which means we cannot assume them for portability 
>> reasons.  FWIW, we have made the rest of mpich stack-usage clean.
>>
>>  -- Pavan
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Build warnings with hwloc-2.0.3

2019-02-25 Thread Brice Goglin
Hello Pavan,

Are you sure you're not passing -Wstack-usage? My Ubuntu 18.04 with
latest gcc-7 (7.3.0-27ubuntu1~18.04) doesn't show any of those warnings.

It looks like all these warnings are caused by C99 variable-length
arrays (except 2 that I don't understand). I know the kernel devs
stopped using VLA recently, and it looks like C11 made them optional.
But are we really supposed to stop using VLA already?

Brice



Le 25/02/2019 à 02:07, Balaji, Pavan via hwloc-users a écrit :
> Folks,
>
> I'm getting the below build warnings with hwloc-2.0.3, gcc-7.3 on Ubuntu 
> (with -Wall -O2):
>
> 8<
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/distances.c:
>  In function 'hwloc__groups_by_distances':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/distances.c:817:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  hwloc__groups_by_distances(struct hwloc_topology *topology,
>  ^~
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology.c:
>  In function 'hwloc_propagate_symmetric_subtree':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology.c:2388:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  hwloc_propagate_symmetric_subtree(hwloc_topology_t topology, hwloc_obj_t 
> root)
>  ^
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-synthetic.c:
>  In function 'hwloc_synthetic_process_indexes':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-synthetic.c:71:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  hwloc_synthetic_process_indexes(struct hwloc_synthetic_backend_data_s *data,
>  ^~~
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-xml.c:
>  In function 'hwloc__xml_export_object_contents':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-xml.c:1920:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  hwloc__xml_export_object_contents (hwloc__xml_export_state_t state, 
> hwloc_topology_t topology, hwloc_obj_t obj, unsigned long flags)
>  ^
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:
>  In function 'hwloc_linux_get_area_membind':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:1883:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  hwloc_linux_get_area_membind(hwloc_topology_t topology, const void *addr, 
> size_t len, hwloc_nodeset_t nodeset, hwloc_membind_policy_t *policy, int 
> flags __hwloc_attribute_unused)
>  ^~~~
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:
>  In function 'hwloc_linux_set_thisthread_membind':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:1737:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  hwloc_linux_set_thisthread_membind(hwloc_topology_t topology, 
> hwloc_const_nodeset_t nodeset, hwloc_membind_policy_t policy, int flags)
>  ^~
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:
>  In function 'hwloc_linux_get_thisthread_membind':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:1848:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  hwloc_linux_get_thisthread_membind(hwloc_topology_t topology, 
> hwloc_nodeset_t nodeset, hwloc_membind_policy_t *policy, int flags 
> __hwloc_attribute_unused)
>  ^~
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:
>  In function 'hwloc_linux__get_allowed_resources':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-linux.c:4426:13:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  static void hwloc_linux__get_allowed_resources(hwloc_topology_t topology, 
> const char *root_path, int root_fd, char **cpuset_namep)
>  ^~
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-x86.c:
>  In function 'cpuiddump_read':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-x86.c:69:1:
>  warning: stack usage might be unbounded [-Wstack-usage=]
>  cpuiddump_read(const char *dirpath, unsigned idx)
>  ^~
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-x86.c:
>  In function 'hwloc_x86_component_instantiate':
> ../../../../../../../../../mpich/src/pm/hydra/tools/topo/hwloc/hwloc/hwloc/topology-x86.c:1456:1:
>  warning: stack usage might be unbounded 

Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Brice Goglin
Only the one in brackets is set, others are unset alternatives.

If you write "madvise" in that file, it'll become "always [madvise] never".

Brice


Le 29/01/2019 à 15:36, Biddiscombe, John A. a écrit :
> On the 8 numa node machine
>
> $cat /sys/kernel/mm/transparent_hugepage/enabled 
> [always] madvise never
>
> is set already, so I'm not really sure what should go in there to disable it.
>
> JB
>
> -Original Message-
> From: Brice Goglin  
> Sent: 29 January 2019 15:29
> To: Biddiscombe, John A. ; Hardware locality user list 
> 
> Subject: Re: [hwloc-users] unusual memory binding results
>
> Oh, that's very good to know. I guess lots of people using first touch will 
> be affected by this issue. We may want to add a hwloc memory flag doing 
> something similar.
>
> Do you have root access to verify that writing "never" or "madvise" in 
> /sys/kernel/mm/transparent_hugepage/enabled fixes the issue too?
>
> Brice
>
>
>
> Le 29/01/2019 à 14:02, Biddiscombe, John A. a écrit :
>> Brice
>>
>> madvise(addr, n * sizeof(T), MADV_NOHUGEPAGE)
>>
>> seems to make things behave much more sensibly. I had no idea it was a 
>> thing, but one of my colleagues pointed me to it.
>>
>> Problem seems to be solved for now. Thank you very much for your insights 
>> and suggestions/help.
>>
>> JB
>>
>> -Original Message-
>> From: Brice Goglin 
>> Sent: 29 January 2019 10:35
>> To: Biddiscombe, John A. ; Hardware locality user 
>> list 
>> Subject: Re: [hwloc-users] unusual memory binding results
>>
>> Crazy idea: 512 pages could be replaced with a single 2MB huge page.
>> You're not requesting huge pages in your allocation but some systems 
>> have transparent huge pages enabled by default (e.g. RHEL
>> https://access.redhat.com/solutions/46111)
>>
>> This could explain why 512 pages get allocated on the same node, but it 
>> wouldn't explain crazy patterns you've seen in the past.
>>
>> Brice
>>
>>
>>
>>
>> Le 29/01/2019 à 10:23, Biddiscombe, John A. a écrit :
>>> I simplified things and instead of writing to a 2D array, I allocate a 1D 
>>> array of bytes and touch pages in a linear fashion.
>>> Then I call syscall(NR)move_pages, ) and retrieve a status array for 
>>> each page in the data.
>>>
>>> When I allocate 511 pages and touch alternate pages on alternate numa 
>>> nodes
>>>
>>> Numa page binding 511
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
>>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>>
>>> but as soon as I increase to 512 pages, it breaks.
>>>
>>> Numa page binding 512
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 

Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Brice Goglin
Oh, that's very good to know. I guess lots of people using first touch
will be affected by this issue. We may want to add a hwloc memory flag
doing something similar.

Do you have root access to verify that writing "never" or "madvise" in
/sys/kernel/mm/transparent_hugepage/enabled fixes the issue too?

Brice



Le 29/01/2019 à 14:02, Biddiscombe, John A. a écrit :
> Brice
>
> madvise(addr, n * sizeof(T), MADV_NOHUGEPAGE)
>
> seems to make things behave much more sensibly. I had no idea it was a thing, 
> but one of my colleagues pointed me to it.
>
> Problem seems to be solved for now. Thank you very much for your insights and 
> suggestions/help.
>
> JB
>
> -Original Message-
> From: Brice Goglin  
> Sent: 29 January 2019 10:35
> To: Biddiscombe, John A. ; Hardware locality user list 
> 
> Subject: Re: [hwloc-users] unusual memory binding results
>
> Crazy idea: 512 pages could be replaced with a single 2MB huge page.
> You're not requesting huge pages in your allocation but some systems have 
> transparent huge pages enabled by default (e.g. RHEL
> https://access.redhat.com/solutions/46111)
>
> This could explain why 512 pages get allocated on the same node, but it 
> wouldn't explain crazy patterns you've seen in the past.
>
> Brice
>
>
>
>
> Le 29/01/2019 à 10:23, Biddiscombe, John A. a écrit :
>> I simplified things and instead of writing to a 2D array, I allocate a 1D 
>> array of bytes and touch pages in a linear fashion.
>> Then I call syscall(NR)move_pages, ) and retrieve a status array for 
>> each page in the data.
>>
>> When I allocate 511 pages and touch alternate pages on alternate numa 
>> nodes
>>
>> Numa page binding 511
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
>> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
>> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
>>
>> but as soon as I increase to 512 pages, it breaks.
>>
>> Numa page binding 512
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>
>> On the 8 numa node machine it sometimes gives the right answer even with 512 
>> pages.
>>
>> Still baffled
>>
>> JB
>>
>> -Original Message-
>> From: hwloc-users  On Behalf Of 
>> Biddiscombe, John A.
>> Sent: 28 January 2019 16:14
>> To: Brice Goglin 
>> Cc: Hardware locality user list 
>> Subject: Re: [hwloc-users] unusual memory binding results
>>
>> Brice
>>
>>> Can you print the pattern before and after thread 1 touched its pages, or 
>>> even in the middle ?
>>> It looks like somebody is touching too many pages here.
>> Expe

Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Brice Goglin
Crazy idea: 512 pages could be replaced with a single 2MB huge page.
You're not requesting huge pages in your allocation but some systems
have transparent huge pages enabled by default (e.g. RHEL
https://access.redhat.com/solutions/46111)

This could explain why 512 pages get allocated on the same node, but it
wouldn't explain crazy patterns you've seen in the past.

Brice




Le 29/01/2019 à 10:23, Biddiscombe, John A. a écrit :
> I simplified things and instead of writing to a 2D array, I allocate a 1D 
> array of bytes and touch pages in a linear fashion.
> Then I call syscall(NR)move_pages, ) and retrieve a status array for each 
> page in the data.
>
> When I allocate 511 pages and touch alternate pages on alternate numa nodes
>
> Numa page binding 511
> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
> 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 
> 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 
> 1 0 1 0
>
> but as soon as I increase to 512 pages, it breaks.
>
> Numa page binding 512
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
> 0 0 0 0 0
>
> On the 8 numa node machine it sometimes gives the right answer even with 512 
> pages.
>
> Still baffled
>
> JB
>
> -Original Message-
> From: hwloc-users  On Behalf Of 
> Biddiscombe, John A.
> Sent: 28 January 2019 16:14
> To: Brice Goglin 
> Cc: Hardware locality user list 
> Subject: Re: [hwloc-users] unusual memory binding results
>
> Brice
>
>> Can you print the pattern before and after thread 1 touched its pages, or 
>> even in the middle ?
>> It looks like somebody is touching too many pages here.
> Experimenting with different threads touching one or more pages, I get 
> unpredicatable results
>
> here on the 8 numa node device, the result is perfect. I am only allowing 
> thread 3 and 7 to write a single memory location
>
> get_numa_domain() 8 Domain Numa pattern
> 
> 
> 
> 3---
> 
> 
> 
> 7---
> 
>
> 
> Contents of memory locations
> 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 26 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 63 0 0 0 0 0 0 0 
> 
>
> you can see that core 26 (numa domain 3) wrote to memory, and so did core 63 
> (domain 8)
>
> Now I run it a second time and look, its rubbish
>
> get_numa_domain() 8 Domain Numa pattern
> 3---
> 3---
> 3---
> 3---
> 3---
> 3---
> 3---
> 3---
> 
>
> 
> Contents of memory locations
> 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 26 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 0 0 0 0 0 0 0 0 
> 0 0 0

Re: [hwloc-users] unusual memory binding results

2019-01-28 Thread Brice Goglin
Le 28/01/2019 à 11:28, Biddiscombe, John A. a écrit :
> If I disable thread 0 and allow thread 1 then I get this pattern on 1 machine 
> (clearly wrong)
> 
> 
> 
> 
> 


Can you print the pattern before and after thread 1 touched its pages,
or even in the middle ?

It looks like somebody is touching too many pages here.

Brice


> and on another I get
> -1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
> 1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
> -1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
> 1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
> -1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1
> 1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-
> which is correct because the '-' is a negative status. I will run again and 
> see if it's -14 or -2
>
> JB
>
>
> -Original Message-
> From: Brice Goglin  
> Sent: 28 January 2019 10:56
> To: Biddiscombe, John A. 
> Cc: Hardware locality user list 
> Subject: Re: [hwloc-users] unusual memory binding results
>
> Can you try again disabling the touching in one thread to check whether the 
> other thread only touched its own pages? (others' status should be
> -2 (ENOENT))
>
> Recent kernels have ways to migrate memory at runtime
> (CONFIG_NUMA_BALANCING) but this should only occur when it detects that some 
> thread does a lot of remote access, which shouldn't be the case here, at 
> least at the beginning of the program.
>
> Brice
>
>
>
> Le 28/01/2019 à 10:35, Biddiscombe, John A. a écrit :
>> Brice
>>
>> I might have been using the wrong params to hwloc_get_area_memlocation 
>> in my original version, but I bypassed it and have been calling
>>
>> int get_numa_domain(void *page)
>> {
>> HPX_ASSERT( (std::size_t(page) & 4095) ==0 );
>>
>> void *pages[1] = { page };
>> int  status[1] = { -1 };
>> if (syscall(__NR_move_pages, 0, 1, pages, nullptr, status, 0) == 
>> 0) {
>> if (status[0]>=0 && 
>> status[0]<=HPX_HAVE_MAX_NUMA_DOMAIN_COUNT) {
>> return status[0];
>> }
>> return -1;
>> }
>> throw std::runtime_error("Failed to get numa node for page");
>> }
>>
>> this function instead. Just testing one page address at a time. I 
>> still see this kind of pattern
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> 00101101010010101001010101011010011011010101110101110111010101
>> 010101
>> when I should see
>> 0101010101010101010101010101010101010101010101010101010101010101010101
>> 0101010101
>> 1010101010101010101010101010101010101010101010101010101010101010101010
>> 1010101010
>> 0101010101010101010101010101010101010101010101010101010101010101010101
>> 0101010101
>> 1010101010101010101010101010101010101010101010101010101010101010101010
>> 1010101010
>> 0101010101010101010101010101010101010101010101010101010101010101010101
>> 0101010101
>> 1010101010101010101010101010101010101010101010101010101010101010101010
>> 1010101010
>> 0101010101010101010101010101010101010101010101010101010101010

Re: [hwloc-users] unusual memory binding results

2019-01-28 Thread Brice Goglin
Can you try again disabling the touching in one thread to check whether
the other thread only touched its own pages? (others' status should be
-2 (ENOENT))

Recent kernels have ways to migrate memory at runtime
(CONFIG_NUMA_BALANCING) but this should only occur when it detects that
some thread does a lot of remote access, which shouldn't be the case
here, at least at the beginning of the program.

Brice



Le 28/01/2019 à 10:35, Biddiscombe, John A. a écrit :
> Brice
>
> I might have been using the wrong params to hwloc_get_area_memlocation in my 
> original version, but I bypassed it and have been calling
>
> int get_numa_domain(void *page)
> {
> HPX_ASSERT( (std::size_t(page) & 4095) ==0 );
>
> void *pages[1] = { page };
> int  status[1] = { -1 };
> if (syscall(__NR_move_pages, 0, 1, pages, nullptr, status, 0) == 
> 0) {
> if (status[0]>=0 && 
> status[0]<=HPX_HAVE_MAX_NUMA_DOMAIN_COUNT) {
> return status[0];
> }
> return -1;
> }
> throw std::runtime_error("Failed to get numa node for page");
> }
>
> this function instead. Just testing one page address at a time. I still see 
> this kind of pattern
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> 00101101010010101001010101011010011011010101110101110111010101010101
> when I should see
> 01010101010101010101010101010101010101010101010101010101010101010101010101010101
> 10101010101010101010101010101010101010101010101010101010101010101010101010101010
> 01010101010101010101010101010101010101010101010101010101010101010101010101010101
> 10101010101010101010101010101010101010101010101010101010101010101010101010101010
> 01010101010101010101010101010101010101010101010101010101010101010101010101010101
> 10101010101010101010101010101010101010101010101010101010101010101010101010101010
> 01010101010101010101010101010101010101010101010101010101010101010101010101010101
> 10101010101010101010101010101010101010101010101010101010101010101010101010101010
> 01010101010101010101010101010101010101010101010101010101010101010101010101010101
> 10101010101010101010101010101010101010101010101010101010101010101010101010101010
>
> I am deeply troubled by this and can't think of what to try next since I can 
> see the memory contents hold the correct CPU ID of the thread that touched 
> the memory, so either the syscall is wrong, or the kernel is doing something 
> else. I welcome any suggestions on what might be wrong.
>
> Thanks for trying to help.
>
> JB
>
> -Original Message-
> From: Brice Goglin  
> Sent: 26 January 2019 10:19
> To: Biddiscombe, John A. 
> Cc: Hardware locality user list 
> Subject: Re: [hwloc-users] unusual memory binding results
>
> Le 25/01/2019 à 23:16, Biddiscombe, John A. a écrit :
>>> move_pages() returning 0 with -14 in the status array? As opposed to 
>>> move_pages() returning -1 with errno set to 14, which would definitely be a 
>>> bug in hwloc.
>> I think it was move_pages returning zero with -14 in the status array, and 
>> then hwloc returning 0 with an empty nodeset (which I then messed up by 
>> calling get bitmap first and assuming 0 meant numa node zero and not 
>> checking for an empty nodeset).
>>
>> I'm not sure why I get -EFAULT status rather than -NOENT, but that's what 
>> I'm seeing in the status field when I pass the pointer returned from the 
>> alloc_membind call.
> The only reason I see for getting -EFAULT there would be that you pass the 
> buffer to move_pages (what hwloc_get_area_memlocation() wants, a start 
> pointer and length) instead of a pointer to an array of page addresses 
> (move_pages wants a void** pointing to individual pages).
>
> Brice
>
>
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] unusual memory binding results

2019-01-26 Thread Brice Goglin
Le 25/01/2019 à 23:16, Biddiscombe, John A. a écrit :
>> move_pages() returning 0 with -14 in the status array? As opposed to 
>> move_pages() returning -1 with errno set to 14, which would definitely be a 
>> bug in hwloc.
> I think it was move_pages returning zero with -14 in the status array, and 
> then hwloc returning 0 with an empty nodeset (which I then messed up by 
> calling get bitmap first and assuming 0 meant numa node zero and not checking 
> for an empty nodeset).
>
> I'm not sure why I get -EFAULT status rather than -NOENT, but that's what I'm 
> seeing in the status field when I pass the pointer returned from the 
> alloc_membind call.

The only reason I see for getting -EFAULT there would be that you pass
the buffer to move_pages (what hwloc_get_area_memlocation() wants, a
start pointer and length) instead of a pointer to an array of page
addresses (move_pages wants a void** pointing to individual pages).

Brice


___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] unusual memory binding results

2019-01-25 Thread Brice Goglin

Le 25/01/2019 à 14:17, Biddiscombe, John A. a écrit :
> Dear List/Brice
>
> I experimented with disabling the memory touch on threads except for 
> N=1,2,3,4 etc and found a problem in hwloc, which is that the function 
> hwloc_get_area_memlocation was returning '0' when the status of the memory 
> null move operation was -14 (#define EFAULT 14 /* Bad address */). This was 
> when I call get area memlocation immediately after allocating and then 'not' 
> touching. I think if the status is an error, then the function should 
> probably return -1, but anyway. I'll file a bug and send a patch if this is 
> considered to be a bug.


Just to be sure, you talking about move_pages() returning 0 with -14 in
the status array? As opposed to move_pages() returning -1 with errno set
to 14, which would definitely be a bug in hwloc.


When the page is valid but not allocated yet, move_pages() is supposed
to return status = -ENOENT. This case is not an error, so returning 0
with an empty nodeset looks fine to me (pages are not allocated, hence
they are allocated on an empty set of nodes).

-EFAULT means that the page is invalid (you'd get a segfault if you
touch it). I am not sure what we should return in that case. It's also
true that pages are allocated nowhere :)

Anyway, if you get -EFAULT in status, it should mean that an invalid
address was passed to hwloc_get_area_memlocation() or an invalid length.

Brice


___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] unusual memory binding results

2019-01-21 Thread Brice Goglin

Le 21/01/2019 à 17:08, Biddiscombe, John A. a écrit :
> Dear list,
>
> I'm allocating a matrix of size (say) 2048*2048 on a node with 2 numa domains 
> and initializing the matrix by using 2 threads, one pinned on each numa 
> domain - with the idea that I can create tiles of memory bound to each numa 
> domain rather than having pages assigned all to one, interleaved, or possibly 
> random. The tiling pattern can be user defined, but I am using a simple 
> strategy that touches pages based on a simple indexing scheme using (say) a 
> tile size of 256 elements and should give a pattern like this


Hello John,

First idea:

A title of 256 element means you're switching between tiles every 2kB
(if elements are double precision), hence half the page belongs to one
thread and the other half to the another thread, hence only the first
one touching his tile will actually allocate locally.

One way to debug would be to disable touching in N-1 thread to check
that everything allocated in on the right node.

Can you share the code, or at least part of it?

Brice


___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] mem bind

2018-12-21 Thread Brice Goglin
Hello

That's not how current operating systems work, hence hwloc cannot do it.
Usually you can bind a process virtual memory to a specific part of the
physical memory (a NUMA node is basically a big static range), but the
reverse isn't allowed by any OS I know.

If you can tweak the hardware, you could try tweaking the ACPI tables so
that a specific range of physical memory moves a new dedicated NUMA node :)

Another crazy idea is to tell the Linux kernel at boot that your ranges
aren't RAM but non-volatile memory. They won't be used by anybody by
default, but you can make them "dax" devices that programs could mmap.

Brice




Le 21/12/2018 à 21:11, Dahai Guo a écrit :
> Hi, 
>
> I was wondering if there is a good way in hwloc to bind a particular
> range of memory to a process? For example, suppose there are totally
> 1000MB on the node, how to bind memory range [50, 100]  to a process,
> and [101,200] to another one?
>
> If hwloc can, an example will be greatly appreciated.
>
> D.
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Travis CI unit tests failing with HW "operating system" error

2018-09-13 Thread Brice Goglin
If lstopo fails there, run "hwloc-gather-topology foo" and send foo.tar.bz2

As a workaround for ARMCI, you may try setting
HWLOC_COMPONENTS=no_os,stop in the environment so that hwloc behaves as
if the operating system had no topology support.

Brice



Le 14/09/2018 à 06:11, Jeff Hammond a écrit :
> All of the job failures have this warning so I am inclined to think
> they are related.  I don't know what I should expect from lstopo on
> inside of AWS, but I guess I'll try it.
>
> I was using the hwloc shipped with the mpich-3.3b1.  Talk to the MPICH
> team if you want them to upgrade :-)
>
> Jeff
>
> On Thu, Sep 13, 2018 at 8:42 AM, Brice Goglin  <mailto:brice.gog...@inria.fr>> wrote:
>
> This is actually just a warning. Usually it causes the topology to
> be wrong (like a missing object), but it shouldn't prevent the
> program from working. Are you sure your programs are failing
> because of hwloc? Do you have a way to run lstopo on that node?
>
> By the way, you shouldn't use hwloc 2.0.0rc2, at least because
> it's old, it has a broken ABI, and it's a RC :)
>
> Brice
>
>
>
> Le 13/09/2018 à 16:12, Jeff Hammond a écrit :
>> I am running ARMCI-MPI over MPICH in a Travis CI Linux instance
>> and topology is causing it to fail.  I do not care about topology
>> in a virtualized environment.  How do I fix this?
>>
>> 
>> 
>> * hwloc 2.0.0rc2-git has encountered what looks like an error
>> from the operating system.
>> *
>> * Group0 (cpuset 0x,0x) intersects with L3
>> (cpuset 0x1000,0x0212) without inclusion!
>> * Error occurred in topology.c line 1384
>> *
>> * The following FAQ entry in the hwloc documentation may help:
>> *   What should I do when hwloc reports "operating system" warnings?
>> * Otherwise please report this error message to the hwloc user's
>> mailing list
>> * along with the files generated by the hwloc-gather-topology script.
>> 
>> 
>>
>> https://travis-ci.org/jeffhammond/armci-mpi/jobs/425342479
>> <https://travis-ci.org/jeffhammond/armci-mpi/jobs/425342479> has
>> all of the details.
>>
>> Jeff
>>
>>
>> --
>> Jeff Hammond
>> jeff.scie...@gmail.com <mailto:jeff.scie...@gmail.com>
>> http://jeffhammond.github.io/
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> <mailto:hwloc-users@lists.open-mpi.org>
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> <https://lists.open-mpi.org/mailman/listinfo/hwloc-users>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org <mailto:hwloc-users@lists.open-mpi.org>
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> <https://lists.open-mpi.org/mailman/listinfo/hwloc-users>
>
>
>
>
> -- 
> Jeff Hammond
> jeff.scie...@gmail.com <mailto:jeff.scie...@gmail.com>
> http://jeffhammond.github.io/
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Travis CI unit tests failing with HW "operating system" error

2018-09-13 Thread Brice Goglin
This is actually just a warning. Usually it causes the topology to be
wrong (like a missing object), but it shouldn't prevent the program from
working. Are you sure your programs are failing because of hwloc? Do you
have a way to run lstopo on that node?

By the way, you shouldn't use hwloc 2.0.0rc2, at least because it's old,
it has a broken ABI, and it's a RC :)

Brice



Le 13/09/2018 à 16:12, Jeff Hammond a écrit :
> I am running ARMCI-MPI over MPICH in a Travis CI Linux instance and
> topology is causing it to fail.  I do not care about topology in a
> virtualized environment.  How do I fix this?
>
> 
> * hwloc 2.0.0rc2-git has encountered what looks like an error from the
> operating system.
> *
> * Group0 (cpuset 0x,0x) intersects with L3 (cpuset
> 0x1000,0x0212) without inclusion!
> * Error occurred in topology.c line 1384
> *
> * The following FAQ entry in the hwloc documentation may help:
> *   What should I do when hwloc reports "operating system" warnings?
> * Otherwise please report this error message to the hwloc user's
> mailing list
> * along with the files generated by the hwloc-gather-topology script.
> 
>
> https://travis-ci.org/jeffhammond/armci-mpi/jobs/425342479 has all of
> the details.
>
> Jeff
>
>
> --
> Jeff Hammond
> jeff.scie...@gmail.com 
> http://jeffhammond.github.io/
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] How to get pid in hwloc?

2018-09-04 Thread Brice Goglin
Hello

The only public portability layer we have for PIDs is hwloc_pid_t when
passed to things like set_proc_cpubind(). But we don't have a portable
getpid() or printf(). You'll have to use getpid() and printf("%ld",
(long)pid) on Unix.

On Windows, hwloc_pid_t is a HANDLE, you don't want to print that. You
can print a process number, and get a HANDLE from a process number using
something like
https://github.com/open-mpi/hwloc/blob/master/utils/hwloc/misc.h#L323

Brice



Le 05/09/2018 à 00:01, Junchao Zhang a écrit :
> Hi,
>   hwloc_set_proc_cpubind() has a pid argument. But how to get the pid
> portably? In addition, I want to convert a pid to an integer and then
> print it out. Does hwloc has APIs to support the needs?
>   Thank you.
> --Junchao Zhang
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] conflicts of multiple hwloc libraries

2018-09-01 Thread Brice Goglin
This was also addressed offline while the mailing was (again) broken.

Some symbols weren't renamed in old releases. This was fixed a couple
months ago. It will be in 2.0.2 and 1.11.11 (to be released on Monday
Sept 3rd).

Brice



Le 30/08/2018 à 06:31, Junchao Zhang a écrit :
> Hi,
>    My program calls a third party library, which in turn contains an
> embedded hwloc library.  My program itself also calls hwloc, and I
> installed a higher version of hwloc than the library's.  It seems this
> setting works with dynamic build. But on Cray machines with static
> library, I have a "multiple definition of `hwloc_linux_component'"
> error when linking my code. How to fix that?
>   Thank you.
> --Junchao Zhang
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Question about hwloc_bitmap_singlify

2018-08-28 Thread Brice Goglin
Hello

If you bind a thread to a newset that contains 4 PUs (4 bits), the
operating system scheduler is free to run that thread on any of these
PUs. It means it may run on it on one PU, then migrate it to the other
PU, then migrate it back, etc. If these PUs do not share all caches, you
will see a performance drop because the data you put in the cache when
running on PU1 has to be stored/migrated in the cache on another PU when
the thread is migrated by the OS scheduler. If the PU share all caches,
the performance drop is much lower, but still exists because migrating
tasks between PU takes a bit of time.

If you call hwloc_bitmap_signlify(newset) before binding, you basically
say "I want to run on any of these 4 PUs, I am actually going to run on
a specific one". Singlify takes your set of PUs in the bitmap and keeps
a single one. Your original binding is respected (you run inside the
original binding), but you don't use all of them.

HOWEVER if you bind multiple threads to the same identical newset, you
don't want to singlify because all of them would run on the SAME PU. You
can either bind without singlify() so that the OS scheduler spreads your
threads on different PUs among newset. Or you want to manually split
newset into multiple subset (hwloc_distrib can do that).

I'll try to improve the doc.

Brice



Le 29/08/2018 à 06:26, Junchao Zhang a écrit :
> Hi,   
>   On cpu binding, hwloc manual says "It is often useful to call
> hwloc_bitmap_singlify() first so that a single CPU remains in the set.
> This way, the process will not even migrate between different CPUs
> inside the given set" . I don't understand it. If I do not do
> hwloc_bitmap_singlify, what will happen? Suppose a process's old cpu
> binding is oldset, and I want to bind it to newset. What should I do
> to use hwloc_bitmap_singlify?
>   Thank you.
> --Junchao Zhang
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] How to combine bitmaps on MPI ranks?

2018-08-28 Thread Brice Goglin
This question was addressed offline while the mailing lists were offline.

We had things like hwloc_bitmap_set_ith_ulong() and
hwloc_bitmap_from_ith_ulong() for packing/unpacking but they weren't
very convenient unless you know multiple ulongs are actually needed to
store the bitmap.

We added new functions to ease things
(hwloc_bitmap_nr/from/to_ulongs()). They will be in the upcoming hwloc 2.1.

Brice



Le 23/08/2018 à 04:57, Junchao Zhang a écrit :
> Hello,
>   Suppose I call hwloc on two MPI ranks and get a bitmap on each.  On
> rank 0, I want to bitwise OR the two. How to do that?  I did not find
> bitmap APIs to pack/unpack bitmaps to/from ulongs for MPI send/recv
> purpose. 
>   Thank you.
> --Junchao Zhang
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Please help interpreting reported topology - possible bug?

2018-05-17 Thread Brice Goglin
Hello Hartmut

The mailing list address changed a while ago, there's an additional
"lists." in the domaine name.

Regarding your question, I would assume you are running in a cgroup with
the second NUMA node disallowed (while all the corresponding cores are
allowed). lstopo with --whole-system would confirm that by showing
disallowed stuff.

Brice



Le 17/05/2018 à 15:58, Hartmut Kaiser a écrit :
> Let me rephrase my question below:
>
> Why does the second socket does not show up as a NUMA domain (as the first
> socket does)?
> Is this a problem in HWLOC or is this expected?
>
> Thanks!
> Regards Hartmut
> ---
> http://stellar.cct.lsu.edu
> https://github.com/STEllAR-GROUP/hpx
>
>> -Original Message-
>> From: Hartmut Kaiser [mailto:hartmut.kai...@gmail.com]
>> Sent: Wednesday, May 16, 2018 3:48 PM
>> To: hwloc-us...@open-mpi.org
>> Subject: Please help interpreting reported topology
>>
>> All,
>>
>> We're seeing some topology reported by hwloc V2.0 we're not able to
>> interpret. Here is what lstopo gives us:
>>
>> Machine (63GB total)
>>   Package L#0
>> NUMANode L#0 (P#0 63GB)
>> L3 L#0 (30MB)
>>   L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0
>> (P#0)
>>   L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1
>> (P#1)
>>   L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2
>> (P#2)
>>   L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3
>> (P#3)
>>   L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4
>> (P#4)
>>   L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5
>> (P#5)
>>   L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6
>> (P#6)
>>   L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7
>> (P#7)
>>   L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8
>> (P#8)
>>   L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9
>> (P#9)
>>   L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU
>> L#10 (P#10)
>>   L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU
>> L#11 (P#11)
>> HostBridge
>>   PCIBridge
>> PCI 04:00.0 (VGA)
>>   PCI 00:11.4 (SATA)
>>   PCIBridge
>> PCI 07:00.0 (Ethernet)
>>   Net "eno1"
>>   PCIBridge
>> PCI 08:00.0 (Ethernet)
>>   Net "eno2"
>>   PCI 00:1f.2 (SATA)
>> Block(Removable Media Device) "sr0"
>> Block(Disk) "sda"
>>   PCI 00:1f.5 (IDE)
>>   Package L#1 + L3 L#1 (30MB)
>> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU
>> L#12 (P#12)
>> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU
>> L#13 (P#13)
>> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU
>> L#14 (P#14)
>> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU
>> L#15 (P#15)
>> L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU
>> L#16 (P#16)
>> L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU
>> L#17 (P#17)
>> L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU
>> L#18 (P#18)
>> L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU
>> L#19 (P#19)
>> L2 L#20 (256KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20 + PU
>> L#20 (P#20)
>> L2 L#21 (256KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21 + PU
>> L#21 (P#21)
>> L2 L#22 (256KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22 + PU
>> L#22 (P#22)
>> L2 L#23 (256KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23 + PU
>> L#23 (P#23)
>>
>> The machine has 2 sockets of Intel E5-2670 v3 and has HT disabled.
>> Everything looks ok for the first socket, but the second does not make any
>> sense to us.
>>
>> Any help would be appreciated.
>>
>> Thanks!
>> Regards Hartmut
>> ---
>> http://boost-spirit.com
>> http://stellar.cct.lsu.edu
>>
>
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Netloc integration with hwloc

2018-04-04 Thread Brice Goglin
Le 04/04/2018 à 16:49, Madhu, Kavitha Tiptur a écrit :
>
> — I tried building older netloc with hwloc 2.0 and it throws compiler errors. 
> Note that netloc was cloned from it’s git repo.

My guess is that the "map" part that joins netloc's info about the
fabric with hwloc's info about the nodes doesn't like hwloc 2.0. But
that should be easy to disable in the Makefiles and/or to update for
hwloc 2.0.

>>> The plan should rather be to tell us what you need from netloc so that
>>> we can reenable it with a good API. We hear lots of people saying they
>>> are interested in netloc, but *nobody* ever told us anything about what
>>> they want to do for real. And I am not even sure anybody ever played
>>> with the old API. This software cannot go forward unless we know where
>>> it's going. There are many ways to design the netloc API.
> — At this point, our requirement is to expose graph construction from raw 
> topology xml and mapping and traversal at best.
> I see some of these already defined in private/hwloc.h in the newer version. 
> Our problem here Is that we couldn’t build it in embedded mode, which is how 
> we are using hwloc.

Can't you hack your build system to build hwloc in standalone instead of
embedded mode for testing? Or use an external hwloc instead of your
embedded one?
I'd like to get feedback about private/netloc.h before making some of it
public.

I'll look at making libnetloc embeddable in 2.1.

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Netloc integration with hwloc

2018-04-03 Thread Brice Goglin
If you really want the old netloc API now, you could try hwloc 2.x with
the old netloc. But that's certainly not maintained anymore, and that
only works for IB while the new netloc should have OPA and Cray support
soon.

The plan should rather be to tell us what you need from netloc so that
we can reenable it with a good API. We hear lots of people saying they
are interested in netloc, but *nobody* ever told us anything about what
they want to do for real. And I am not even sure anybody ever played
with the old API. This software cannot go forward unless we know where
it's going. There are many ways to design the netloc API.

* We had an explicit graph API in the old netloc but that API implied
expensive graph algorithmics in the runtimes using it. It seemed
unusable for taking decision at runtime anyway, but again ever nobody
tried. Also it was rather strange to expose the full graph when you know
the fabric is a 3D dragonfly on Cray, etc.

* In the new netloc, we're thinking of having higher-level implicit
topologies for each class of fabric (dragon-fly, fat-tree, clos-network,
etc) that require more work on the netloc side and easier work in the
runtime using it. However that's less portable than exposing the full
graph. Not sure which one is best, or if both are needed.

* There are also issues regarding nodes/links failure etc. How do we
expose topology changes at runtime? Do we have a daemon running as root
in the background, etc?

Lots of question that need to be discussed before we expose a new API In
the wild. Unfortunately, we lost several years because of the lack of
users' feedback. I don't want to invest time and rush for a new API if
MPICH never actually uses it like other people did in the past.

Brice




Le 04/04/2018 à 01:36, Balaji, Pavan a écrit :
> Brice,
>
> We want to use both hwloc and netloc in mpich.  What are our options here?  
> Move back to hwloc-1.x?  That’d be a bummer because we already invested a lot 
> of effort to migrate to hwloc-2.x.
>
>   — Pavan
>
> Sent from my iPhone
>
>> On Apr 3, 2018, at 6:19 PM, Brice Goglin <brice.gog...@inria.fr> wrote:
>>
>> It's not possible now but that would certainly be considered whenever
>> people start using the API and linking against libnetloc.
>>
>> Brice
>>
>>
>>
>>
>>> Le 03/04/2018 à 21:34, Madhu, Kavitha Tiptur a écrit :
>>> Hi
>>> A follow up question, is it possible to build netloc along with hwloc in 
>>> embedded mode?
>>>
>>>
>>>> On Mar 30, 2018, at 1:34 PM, Brice Goglin <brice.gog...@inria.fr> wrote:
>>>>
>>>> Hello
>>>>
>>>> In 2.0, netloc is still highly experimental. Hopefully, a large rework
>>>> will be merged in git master next month for being released in hwloc 2.1.
>>>>
>>>> Most of the API from the old standalone netloc was made private when
>>>> integrated in hwloc because there wasn't any actual user. The API was
>>>> quite large (things for traversing the graph of both the fabric and the
>>>> servers' internals). We didn't want to expose such a large API before
>>>> getting actual user feedback.
>>>>
>>>> In short, in your need features, please let us know, so that we can
>>>> discuss what to expose in the public headers and how.
>>>>
>>>> Brice
>>>>
>>>>
>>>>
>>>>
>>>>> Le 30/03/2018 à 20:14, Madhu, Kavitha Tiptur a écrit :
>>>>> Hi
>>>>>
>>>>> I need some info on the status of netloc integration with hwloc. I see 
>>>>> the include/netloc.h header is almost empty in hwloc 2.0 and lots of 
>>>>> functionality missing compared to the previous standalone netloc release, 
>>>>> even in private/netloc.h. Am I missing something here?
>>>>>
>>>>> Thanks
>>>>> Kavitha
>>>>>
>>>> ___
>>>> hwloc-users mailing list
>>>> hwloc-users@lists.open-mpi.org
>>>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>>> ___
>>> hwloc-users mailing list
>>> hwloc-users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Netloc integration with hwloc

2018-04-03 Thread Brice Goglin
It's not possible now but that would certainly be considered whenever
people start using the API and linking against libnetloc.

Brice




Le 03/04/2018 à 21:34, Madhu, Kavitha Tiptur a écrit :
> Hi
> A follow up question, is it possible to build netloc along with hwloc in 
> embedded mode?
>
>
>> On Mar 30, 2018, at 1:34 PM, Brice Goglin <brice.gog...@inria.fr> wrote:
>>
>> Hello
>>
>> In 2.0, netloc is still highly experimental. Hopefully, a large rework
>> will be merged in git master next month for being released in hwloc 2.1.
>>
>> Most of the API from the old standalone netloc was made private when
>> integrated in hwloc because there wasn't any actual user. The API was
>> quite large (things for traversing the graph of both the fabric and the
>> servers' internals). We didn't want to expose such a large API before
>> getting actual user feedback.
>>
>> In short, in your need features, please let us know, so that we can
>> discuss what to expose in the public headers and how.
>>
>> Brice
>>
>>
>>
>>
>> Le 30/03/2018 à 20:14, Madhu, Kavitha Tiptur a écrit :
>>> Hi
>>>
>>> I need some info on the status of netloc integration with hwloc. I see the 
>>> include/netloc.h header is almost empty in hwloc 2.0 and lots of 
>>> functionality missing compared to the previous standalone netloc release, 
>>> even in private/netloc.h. Am I missing something here?
>>>
>>> Thanks
>>> Kavitha
>>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Netloc integration with hwloc

2018-03-30 Thread Brice Goglin
Hello

In 2.0, netloc is still highly experimental. Hopefully, a large rework
will be merged in git master next month for being released in hwloc 2.1.

Most of the API from the old standalone netloc was made private when
integrated in hwloc because there wasn't any actual user. The API was
quite large (things for traversing the graph of both the fabric and the
servers' internals). We didn't want to expose such a large API before
getting actual user feedback.

In short, in your need features, please let us know, so that we can
discuss what to expose in the public headers and how.

Brice




Le 30/03/2018 à 20:14, Madhu, Kavitha Tiptur a écrit :
> Hi
>
> I need some info on the status of netloc integration with hwloc. I see the 
> include/netloc.h header is almost empty in hwloc 2.0 and lots of 
> functionality missing compared to the previous standalone netloc release, 
> even in private/netloc.h. Am I missing something here?
>
> Thanks
> Kavitha
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

[hwloc-users] libhwloc soname change in 2.0.1rc1

2018-03-21 Thread Brice Goglin
Hello

In case you missed the announce yesterday, hwloc 2.0.1rc1 changes the
library soname from 12:0:0 to 15:0:0. On Linux, it means that we'll now
build libhwloc.so.15 instead of libhwloc.so.12. That means any
application built for hwloc 2.0.0 will need to be recompiled against 2.0.1.

I should have set the soname to 15:0:0 in 2.0.0 but I forgot. It may
cause issues because hwloc 1.11.x uses 12:x:y (we have "12" in both).
Given that 2.0.0 isn't widely used yet, I hope this way-too-late change
won't cause too many issues. Sorry.

As said on the download page, we want people to stop using 2.0.0 so that
we can forget this issue. If you already switched to hwloc 2.0.0 (and if
some applications are linked with libhwloc), please try to upgrade to
2.0.1 as soon as possible (final release expected next monday).

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] NUMA, io and miscellaneous object depths

2018-03-14 Thread Brice Goglin
Good point. In theory, that's possible because we only look at cpusets
(NUMA nodes have cpusets, I/O don't). So the name of the function still
matches its behavior.

However it won't happen in practice with the current code because I/O
are always attached to CPU objects. But it may change in the future with
things like processing-in-memory etc.

Instead of calling this function, you could do a while
(!hwloc_obj_type_is_normal(obj->type)) obj = obj->parent;

I'll update the doc too. Thanks.

Brice



Le 14/03/2018 à 22:16, Madhu, Kavitha Tiptur a écrit :
> A follow up question, can the call to hwloc_get_non_io_ancestor_obj() return 
> a numa object? 
>
>> On Mar 14, 2018, at 3:09 PM, Madhu, Kavitha Tiptur <kma...@anl.gov> wrote:
>>
>> Hi
>> This function was used to query depth of hardware objects of a certain type 
>> to bind processes to objects at the depth or above in Hydra previously. As 
>> you pointed out, the functionality makes no sense with NUMA/IO objects 
>> possibly being at different depths or for objects.
>>
>>> On Mar 14, 2018, at 3:00 PM, Brice Goglin <brice.gog...@inria.fr> wrote:
>>>
>>> Hello
>>>
>>> I can fix the documentation to say that the function always suceeds and
>>> returns the virtual depth for NUMA/IO/Misc.
>>>
>>> I don't understand your third sentence. If by "actual depth", you mean
>>> the depth of a (normal) parent where NUMA are attached (for instance the
>>> depth of Package if NUMAs are attached to Packages), see
>>> hwloc_get_memory_parents_depth(). However, you may have NUMA/IO/Misc
>>> attached to parents at different depths, so it doesn't make much sense
>>> in the general case.
>>>
>>> What do you use this function for? I thought of removing it from 2.0
>>> because it's hard to define a "usual" order for object types (for
>>> instance L3 can be above or below NUMA for different modern platforms).
>>>
>>> Brice
>>>
>>>
>>>
>>> Le 14/03/2018 à 20:24, Madhu, Kavitha Tiptur a écrit :
>>>> Hello folks,
>>>>
>>>> The function hwloc_get_type_or_above_depth() is supposed to return the 
>>>> depth of objects of type “type" or above. It internally calls 
>>>> hwloc_get_type_depth which returns virtual depths to NUMA, IO and misc 
>>>> objects. In order to retrieve the actual depth of these objects, one needs 
>>>> to call hwloc_get_obj_depth() with virtual depth. Can the documentation be 
>>>> updated to cover this? Or are there plans of changing this behavior?
>>>>
>>>> Thanks
>>>> Kavitha
>>>> ___
>>>> hwloc-users mailing list
>>>> hwloc-users@lists.open-mpi.org
>>>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>>> ___
>>> hwloc-users mailing list
>>> hwloc-users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] NUMA, io and miscellaneous object depths

2018-03-14 Thread Brice Goglin
Hello

I can fix the documentation to say that the function always suceeds and
returns the virtual depth for NUMA/IO/Misc.

I don't understand your third sentence. If by "actual depth", you mean
the depth of a (normal) parent where NUMA are attached (for instance the
depth of Package if NUMAs are attached to Packages), see
hwloc_get_memory_parents_depth(). However, you may have NUMA/IO/Misc
attached to parents at different depths, so it doesn't make much sense
in the general case.

What do you use this function for? I thought of removing it from 2.0
because it's hard to define a "usual" order for object types (for
instance L3 can be above or below NUMA for different modern platforms).

Brice



Le 14/03/2018 à 20:24, Madhu, Kavitha Tiptur a écrit :
> Hello folks,
>
> The function hwloc_get_type_or_above_depth() is supposed to return the depth 
> of objects of type “type" or above. It internally calls hwloc_get_type_depth 
> which returns virtual depths to NUMA, IO and misc objects. In order to 
> retrieve the actual depth of these objects, one needs to call 
> hwloc_get_obj_depth() with virtual depth. Can the documentation be updated to 
> cover this? Or are there plans of changing this behavior?
>
> Thanks
> Kavitha
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

[hwloc-users] call for testing on KNL

2018-02-09 Thread Brice Goglin
Hello

As you may know, hwloc only discovers KNL MCDRAM Cache details if
hwloc-dump-hwdata ran as root earlier. There's an issue with that tool
in 2.0, which was supposed to be a feature: we fixed the matching of
SMBIOS strings, and now it appears some vendors don't match anymore
because they didn't use the standardized strings :(

If you have root access on a KNL, could you build this tarball and run
utils/hwloc/hwloc-dump-hwdata -o foo (or $prefix/sbin/hwloc-dump-hwdata
-o foo).

https://ci.inria.fr/hwloc/job/zbgoglin-0-tarball/lastSuccessfulBuild/artifact/hwloc-hdh-20180209.1705.git6cd1efb.tar.gz


If things work, you should see something like:

Dumping KNL SMBIOS Memory-Side Cache information:
  File = /sys/firmware/dmi/entries/14-0/raw, size = 4096
Looking for "Group: Knights Landing Information" in group string "Group: 
Knights Landing Information"
   Found phi group
  Found KNL type = 160
  [... lots of other lines ...]


If it doesn't work:

|Dumping KNL SMBIOS Memory-Side Cache information: File =
/sys/firmware/dmi/entries/14-0/raw, size = 4096 Looking for "Group:
Knights Landing Information" in group string "foobar" Looking for
"Group: Knights Mill Information" in group string "foobar" Looking for
"Knights Landing Association" in group string "foobar" Failed to find
phi group SMBIOS table does not contain KNL entries|


In both cases, please send the model of the machine [1] as well as the
full output of the tool.

This issue might lead to a 2.0.1 release soon. In the meantime, you can
use hwloc-dump-hwdata from 1.11.9, its output is compatible with hwloc 2.0.

Thanks

Brice


[1] if you don't know, look at strings in "cat
/sys/devices/virtual/dmi/id/*"

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Machine nodes in hwloc topology

2018-02-05 Thread Brice Goglin
Yes, one and only one, the top-level root object of the topology.

This is somehow implied by the doc of HWLOC_OBJ_MACHINE itself ("The
root object type"), I guess I should make it even more clear there? If
there's another place in the doc where this should be added/clarified,
please let me know.

Brice



Le 05/02/2018 à 23:19, Madhu, Kavitha Tiptur a écrit :
> Hi
>
> Thanks for the response. Could you also confirm if hwloc topology
> object would have only machine node?
>
> Thanks,
> Kavitha
>
>
>
>> On Feb 5, 2018, at 4:14 PM, Brice Goglin <brice.gog...@inria.fr
>> <mailto:brice.gog...@inria.fr>> wrote:
>>
>> Hello,
>>
>> Oops, sorry, this sentence is obsolete, I am removing it from the doc
>> right now.
>>
>> We don't support the assembly of multiple machines in a single hwloc
>> topology anymore. For the record, this feature was a very small
>> corner case and it had important limitations (you couldn't bind
>> things or use cpusets unless you were very careful about which host
>> you were talking about), and it made the core hwloc code much more
>> complex.
>>
>> Thanks for the report
>> Brice
>>
>>
>> Le 05/02/2018 à 23:02, Madhu, Kavitha Tiptur a écrit :
>>> Hi
>>>
>>> I have a question on topology query. The hwloc 2.0.0 documentation
>>> states that "Additionally it may assemble the topologies of multiple
>>> machines into a single one so as to let applications consult the
>>> topology of an entire fabric or cluster at once.”. Since “system”
>>> object type has been removed from hwloc, does this statement mean
>>> that multiple “machine” nodes in the topology object would be
>>> combined to one? I can see in function“hwloc_topology_check” that
>>> machine node is at depth 0 and there are no machine nodes at depth
>>> other than 0. Can anyone confirm this?   
>>>
>>> Thanks 
>>> Kavitha
>>>
>>>
>>> ___
>>> hwloc-users mailing list
>>> hwloc-users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org <mailto:hwloc-users@lists.open-mpi.org>
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Machine nodes in hwloc topology

2018-02-05 Thread Brice Goglin
Hello,

Oops, sorry, this sentence is obsolete, I am removing it from the doc
right now.

We don't support the assembly of multiple machines in a single hwloc
topology anymore. For the record, this feature was a very small corner
case and it had important limitations (you couldn't bind things or use
cpusets unless you were very careful about which host you were talking
about), and it made the core hwloc code much more complex.

Thanks for the report
Brice


Le 05/02/2018 à 23:02, Madhu, Kavitha Tiptur a écrit :
> Hi
>
> I have a question on topology query. The hwloc 2.0.0 documentation
> states that "Additionally it may assemble the topologies of multiple
> machines into a single one so as to let applications consult the
> topology of an entire fabric or cluster at once.”. Since “system”
> object type has been removed from hwloc, does this statement mean that
> multiple “machine” nodes in the topology object would be combined to
> one? I can see in function“hwloc_topology_check” that machine node is
> at depth 0 and there are no machine nodes at depth other than 0. Can
> anyone confirm this?   
>
> Thanks 
> Kavitha
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

[hwloc-users] need help for testing new Mac OS support

2018-01-26 Thread Brice Goglin
Hello

I need people running Mac OS to test some patches before releasing them
in 2.0rc2 (which is likely delayed to Monday).

Just build this tarball, run lstopo, and report any difference with
older lstopo outputs:

https://ci.inria.fr/hwloc/job/zbgoglin-0-tarball/lastSuccessfulBuild/artifact/hwloc-master-20180126.1056.git55b1b3a.tar.gz

Hopefully, things won't change on most machines. On laptops without
hyperthreading, you should now see the right number of cores (instead of
N/2 dual-thread cores).

If anybody has a dual-socket machine, I am interesting in seeing whether
we properly detect the NUMA nodes. My only test case so far is a
dual-Nehalem machine which reports a single NUMA node. Not sure if the
BIOS is set to NUMA interleaving or if Mac OS reports wrong information
in sysctl.

Thanks

Brice


___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] Puzzled by the number of cores on i5-7500

2018-01-25 Thread Brice Goglin
It looks like our Mac OS X backend doesn't properly handle processors
that support hyperthreading without actually having hyperthreads enabled
in hardware. Your processor has 4-core without HT but it's based on a
processor with up to 8 cores and 16 threads. Our current code uses the
latter and therefore wrongly assume you have HT cores, hence reporting 2
HT cores instead of 4 noHT cores.

I guess we would also be wrong if some cores or HT are disabled in
software in Mac OS X.


Anybody reading this from a Mac, could you send the output of these
commands on your machine?
sysctl -a | grep ^hw
sysctl -a | grep ^machdep.cpu
lstopo -

Brice



>
>
> Le 25/01/2018 à 07:14, Olivier Cessenat a écrit :
>> Hello,
>>
>> I’m puzzled by the report from lstopo about the number of physical
>> cores on an iMac with
>> I5-7500. It is specified by Intel as a quad core processor and lstopo
>> reports only 2 cores:
>> lstopo
>> <<
>> Machine (8192MB total) + NUMANode L#0 (P#0 8192MB) + L3 L#0 (6144KB)
>>   Core L#0
>>     L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + PU L#0 (P#0)
>>     L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + PU L#1 (P#1)
>>   Core L#1
>>     L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + PU L#2 (P#2)
>>     L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + PU L#3 (P#3)
>> >>
>> When running system_profiler SPHardwareDataType
>> I obtain:
>> <<
>> Hardware:
>>
>>     Hardware Overview:
>>
>>       Model Name: iMac
>>       Model Identifier: iMac18,3
>>       Processor Name: Intel Core i5
>>       Processor Speed: 3,4 GHz
>>       Number of Processors: 1
>>       Total Number of Cores: 4
>>       L2 Cache (per Core): 256 KB
>>       L3 Cache: 6 MB
>>       Memory: 8 GB
>>       Boot ROM Version: IM183.0151.B00
>>       SMC Version (system): 2.41f1
>>       Serial Number (system): DGKV7HJCJ1GN
>>       Hardware UUID: 3FDAD77B-F4E8-50AB-B0FF-AA5C41CA35FA
>> >>
>>
>> Is there a trick ?
>>
>> Thanks for you help,
>>
>> Olivier Cessenat
>>
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc-2.0rc1 failure on Solaris

2018-01-25 Thread Brice Goglin
It is actually easy to fix, we just need to move hwloc's #include before
what base64.c actually #include's. That'll be fixed in rc2 too.

Brice



Le 25/01/2018 à 10:56, Brice Goglin a écrit :
> Like the error below?
>
> This code hasn't changed recently. Did you ever build with these flags
> before?
>
> I am not sure I'll have time to fix yet another header crazyness before rc2.
>
> Brice
>
>
>
>   CC   base64.lo
> In file included from
> /builds/hwloc-master-20180124.2347.gitf53fe3a/include/private/private.h:29:0,
>  from base64.c:128:
> /builds/hwloc-master-20180124.2347.gitf53fe3a/include/private/misc.h: In
> function 'hwloc_strncasecmp':
> /builds/hwloc-master-20180124.2347.gitf53fe3a/include/private/misc.h:370:10:
> error: implicit declaration of function 'strncasecmp'; did you mean
> 'strncmp'? [-Werror=implicit-function-declaration]
>    return strncasecmp(s1, s2, n);
>   ^~~
>   strncmp
> cc1: some warnings being treated as errors
>
>
> Le 25/01/2018 à 10:45, Balaji, Pavan a écrit :
>> Hello,
>>
>> hwloc-2.0rc1 build seems to fail on Solaris, with the following CFLAGS:
>>
>> CFLAGS="-Werror-implicit-function-declaration -std=c99"
>>
>> I'm using gcc-4.8.2
>>
>> Thanks,
>>
>>   -- Pavan
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc-2.0rc1 failure on Solaris

2018-01-25 Thread Brice Goglin
Like the error below?

This code hasn't changed recently. Did you ever build with these flags
before?

I am not sure I'll have time to fix yet another header crazyness before rc2.

Brice



  CC   base64.lo
In file included from
/builds/hwloc-master-20180124.2347.gitf53fe3a/include/private/private.h:29:0,
 from base64.c:128:
/builds/hwloc-master-20180124.2347.gitf53fe3a/include/private/misc.h: In
function 'hwloc_strncasecmp':
/builds/hwloc-master-20180124.2347.gitf53fe3a/include/private/misc.h:370:10:
error: implicit declaration of function 'strncasecmp'; did you mean
'strncmp'? [-Werror=implicit-function-declaration]
   return strncasecmp(s1, s2, n);
  ^~~
  strncmp
cc1: some warnings being treated as errors


Le 25/01/2018 à 10:45, Balaji, Pavan a écrit :
> Hello,
>
> hwloc-2.0rc1 build seems to fail on Solaris, with the following CFLAGS:
>
> CFLAGS="-Werror-implicit-function-declaration -std=c99"
>
> I'm using gcc-4.8.2
>
> Thanks,
>
>   -- Pavan
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc-2.0rc1 build warnings

2018-01-24 Thread Brice Goglin
#2 and #3 are OK

About #1, we could rename pkgconfig and pkgdata prefixes into something
like hwlocpkgconfig and hwlocpkgdata. I don't think the actual prefix
value matters. I'll try that tomorrow.

Brice


Le 24/01/2018 à 23:46, Balaji, Pavan a écrit :
> Hi Brice,
>
> Here are the other patches that we are currently maintaining for hwloc.  Can 
> you see if these can be integrated upstream too:
>
> https://github.com/pmodels/hwloc/commit/44fe0a500e7828bcb2390fbd24656a7a26b450ed
> https://github.com/pmodels/hwloc/commit/5b6d776a1226148030dcf4e26bd13fe16cc885f9
> https://github.com/pmodels/hwloc/commit/9bf3ff256511ea4092928438f5718904875e65e1
>
> The first one is definitely not usable as-is, since that breaks standalone 
> builds.  But I'm interested in hearing about any better solution that you 
> might have.
>
> Thanks,
>
>   -- Pavan
>
>> On Jan 24, 2018, at 4:43 PM, Brice Goglin <brice.gog...@inria.fr> wrote:
>>
>> Thanks, I am fixing this for rc2 tomorrow.
>>
>> Brice
>>
>>
>>
>> Le 24/01/2018 à 22:59, Balaji, Pavan a écrit :
>>> Folks,
>>>
>>> I'm seeing these warnings on the mac os when building hwloc-2.0rc1 with 
>>> clang:
>>>
>>> 8<
>>> CC   lstopo-lstopo.o
>>> lstopo.c: In function 'usage':
>>> lstopo.c:425:7: warning: "CAIRO_HAS_XLIB_SURFACE" is not defined, evaluates 
>>> to 0 [-Wundef]
>>> #elif CAIRO_HAS_XLIB_SURFACE && (defined HWLOC_HAVE_X11_KEYSYM)
>>>  ^~
>>> lstopo.c: In function 'main':
>>> lstopo.c:1041:5: warning: "CAIRO_HAS_XLIB_SURFACE" is not defined, 
>>> evaluates to 0 [-Wundef]
>>> #if CAIRO_HAS_XLIB_SURFACE && defined HWLOC_HAVE_X11_KEYSYM
>>> 8<
>>>
>>> 8<
>>> % clang --version
>>> Apple LLVM version 9.0.0 (clang-900.0.39.2)
>>> Target: x86_64-apple-darwin17.4.0
>>> Thread model: posix
>>> InstalledDir: 
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>>> 8<
>>>
>>> Thanks,
>>>
>>> -- Pavan
>>>
>>> ___
>>> hwloc-users mailing list
>>> hwloc-users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc-2.0rc1 build warnings

2018-01-24 Thread Brice Goglin
Thanks, I am fixing this for rc2 tomorrow.

Brice



Le 24/01/2018 à 22:59, Balaji, Pavan a écrit :
> Folks,
>
> I'm seeing these warnings on the mac os when building hwloc-2.0rc1 with clang:
>
> 8<
>  CC   lstopo-lstopo.o
> lstopo.c: In function 'usage':
> lstopo.c:425:7: warning: "CAIRO_HAS_XLIB_SURFACE" is not defined, evaluates 
> to 0 [-Wundef]
> #elif CAIRO_HAS_XLIB_SURFACE && (defined HWLOC_HAVE_X11_KEYSYM)
>   ^~
> lstopo.c: In function 'main':
> lstopo.c:1041:5: warning: "CAIRO_HAS_XLIB_SURFACE" is not defined, evaluates 
> to 0 [-Wundef]
> #if CAIRO_HAS_XLIB_SURFACE && defined HWLOC_HAVE_X11_KEYSYM
> 8<
>
> 8<
> % clang --version
> Apple LLVM version 9.0.0 (clang-900.0.39.2)
> Target: x86_64-apple-darwin17.4.0
> Thread model: posix
> InstalledDir: 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
> 8<
>
> Thanks,
>
>  -- Pavan
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] OFED requirements for netloc

2018-01-24 Thread Brice Goglin
OK

In the meantime, maybe you can diff the ibnetdiscover outputs to see if
anything obvious appears? You might need to sort the lines first if the
outputs aren't ordered the same.

Brice




Le 24/01/2018 à 23:33, Craig West a écrit :
> Brice,
>
> The output isn't big, just a pair of IB switches and a dozen hosts,
> some with single, some dual connections.
> However, we would need to sanitise the data, or at least look at it in
> detail first to see what it contains.
>
> I can say that the ibnetdiscover and ibroute commands report version
> 1.6.5 on the system that seq faults, and 1.6.6 on the one that succeeds. 
> And that the first looks to be the standard OFED release and the 1.6.6
> version a mellanox release of OFED.
>
> Craig.
>
> On Tue, 23 Jan 2018 at 17:10 Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>> wrote:
>
> Hello,
>
> If the output isn't too big, could you put the files gathered by
> netloc_ib_gather_raw online so that we look at them and try to
> reproduce the crash?
>
> Thanks
>
> Brice
>
>
>
> Le 23/01/2018 à 03:54, Craig West a écrit :
>> Hi,
>>
>> I can't find the version requirements for netloc. I've tried it
>> on an older version of OFED and a newer version of Mellanox OFED.
>> The newer version worked, the older segfaults when running the
>> "netloc_ib_extract_dats" process.
>>
>> Thanks,
>> Craig.
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> <mailto:hwloc-users@lists.open-mpi.org>
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org <mailto:hwloc-users@lists.open-mpi.org>
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Tags for pre-releases

2018-01-23 Thread Brice Goglin
Hello

I didn't know you use submodule. I just pushed tag "hwloc-2.0.0rc1" and
I'll try to remember pushing one for each future rc. If I don't, please
remind me.

I am not going to push all the previous ones because there are just too
many of them. If you need some specific ones, please let me know.

Brice




Le 23/01/2018 à 22:39, Balaji, Pavan a écrit :
> Folks,
>
> [resending to this mailing list; my email to the devel list failed]
>
> There don't seem to be any tags associated with hwloc prereleases (such as 
> hwloc-2.0rc1).  As you know, we embed hwloc into mpich, and we tend to use 
> the git version (through git submodules) rather than release tarballs.  Not 
> having tags for prereleases is making it hard for us to pick a prerelease 
> version.
>
> Can you add tags for the previous prereleases of hwloc?  Or, can you at least 
> add them for 2.0rc1 and other prereleases going forward?
>
> Thanks,
>
>  -- Pavan
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] OFED requirements for netloc

2018-01-22 Thread Brice Goglin
Hello,

If the output isn't too big, could you put the files gathered by
netloc_ib_gather_raw online so that we look at them and try to reproduce
the crash?

Thanks

Brice



Le 23/01/2018 à 03:54, Craig West a écrit :
> Hi,
>
> I can't find the version requirements for netloc. I've tried it on an
> older version of OFED and a newer version of Mellanox OFED. The newer
> version worked, the older segfaults when running the
> "netloc_ib_extract_dats" process.
>
> Thanks,
> Craig.
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] AMD EPYC topology

2017-12-29 Thread Brice Goglin


Le 29/12/2017 à 23:15, Bill Broadley a écrit :
>
>
> Very interesting, I was running parallel finite element code and was seeing
> great performance compared to Intel in most cases, but on larger runs it was 
> 20x
> slower.  This would explain it.
>
> Do you know which commit, or anything else that might help find any related
> discussion?  I tried a few google searches without luck.
>
> Is it specific to the 24-core?  The slowdown I described happened on a 32 core
> Epyc single socket as well as a dual socket 24 core AMD Epyc system.

Hello

Yes it's 24-core specific (that's the only core-count that doesn't have
8-core per zeppelin module).

The commit in Linux git master is 2b83809a5e6d619a780876fcaf68cdc42b50d28c

Brice


commit 2b83809a5e6d619a780876fcaf68cdc42b50d28c
Author: Suravee Suthikulpanit 
Date:   Mon Jul 31 10:51:59 2017 +0200

x86/cpu/amd: Derive L3 shared_cpu_map from cpu_llc_shared_mask

For systems with X86_FEATURE_TOPOEXT, current logic uses the APIC ID
to calculate shared_cpu_map. However, APIC IDs are not guaranteed to
be contiguous for cores across different L3s (e.g. family17h system
w/ downcore configuration). This breaks the logic, and results in an
incorrect L3 shared_cpu_map.

Instead, always use the previously calculated cpu_llc_shared_mask of
each CPU to derive the L3 shared_cpu_map.

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] AMD EPYC topology

2017-12-24 Thread Brice Goglin
Hello
Make sure you use a very recent Linux kernel. There was a bug regarding L3 
caches on 24-core Epyc processors which has been fixed in 4.14 and backported 
in 4.13.x (and maybe in distro kernels too).
However, that would likely not cause huge performance difference unless your 
application heavily depends on the L3 cache.
Brice


Le 24 décembre 2017 12:46:01 GMT+01:00, Matthew Scutter 
 a écrit :
>I'm getting poor performance on OpenMPI tasks on a new AMD 7401P EPYC
>server. I suspect hwloc providing a poor topology may have something to
>do
>with it as I receive this warning below when creating a job.
>Requested data files available at http://static.skysight.io/out.tgz
>Cheers,
>Matthew
>
>
>
>
>* hwloc 1.11.8 has encountered what looks like an error from the
>operating
>system.
>
>*
>
>
>* L3 (cpuset 0x6060) intersects with NUMANode (P#0 cpuset
>0x3f3f
>nodeset 0x0001) without inclusion!
>
>
>* Error occurred in topology.c line 1088
>
>
>
>*
>
>
>
>
>* The following FAQ entry in the hwloc documentation may help:
>
>
>*   What should I do when hwloc reports "operating system" warnings?
>
>
>* Otherwise please report this error message to the hwloc user's
>mailing
>list,
>
>* along with the files generated by the hwloc-gather-topology script.
>
>
>
>
>
>
>depth 0:1 Machine (type #1)
>
>
> depth 1:   1 Package (type #3)
>
>
>  depth 2:  4 NUMANode (type #2)
>
>
>   depth 3: 10 L3Cache (type #4)
>
>
>depth 4:24 L2Cache (type #4)
>
>
> depth 5:   24 L1dCache (type #4)
>
>
>  depth 6:  24 L1iCache (type #4)
>
>
>   depth 7: 24 Core (type #5)
>
>
>
>depth 8:48 PU (type #6)
>
>
>
>Special depth -3:   12 Bridge (type #9)
>
>
>Special depth -4:   9 PCI Device (type #10)
>
>
>Special depth -5:   4 OS Device (type #11)
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] How are processor groups under Windows reported?

2017-11-29 Thread Brice Goglin


Le 29/11/2017 15:21, David Creasy a écrit :
> Hello Brice,
>
> That makes perfect sense, and yes I do see group objects on larger
> systems. I was hoping to use the number of groups to see if it was
> safe to call hwloc_set_proc_cpubind(), but I obviously can't. I can of
> course simply call the Windows GetActiveProcessorGroupCount() function
> instead.
>
> As I'm sure you know, the windows implementation of
> hwloc_set_proc_cpubind has:
>
> hwloc_win_set_proc_cpubind(hwloc_topology_t topology...)
> {
>   assert(nr_processor_groups == 1);

This internal function should not be called at all when
nr_processor_groups>1.
Calling the API hwloc_set_proc_cpubind() should return an error and
never cause that assertion failure.

> Guess it might help to clarify when groups are assigned in the
> documentation.

I'll look at it, thanks.

Brice


>
> Thank you,
>
> David
>
> On 29/11/2017 13:35, Brice Goglin wrote:
>> Hello
>>
>> We only add hwloc Group objects when necessary. On your system, each
>> processor group contains a single NUMA node, so these Groups would not
>> really bring additional information about the hierarchy of resources.
>> If you had a bigger system with, let's say, 4 NUMA nodes, with 2 of them
>> in each processor groups, hwloc would report those as hwloc Group
>> objects.
>>
>> Does this help? I can clarify the FAQ if needed.
>>
>> Brice
>>
>>
>>
>> Le 29/11/2017 14:25, David Creasy a écrit :
>>> Hello,
>>>
>>> Thank you to all contributors to hwloc - very useful.
>>>
>>> In the FAQ,  under the section "What are these Group objects in my
>>> topology?" it says that they are used for "Windows processor groups".
>>> However, I'm either not seeing this, or I'm looking in the wrong
>>> place. On a system with two processor groups, I get:
>>>
>>> C:\temp\hwloc-win64-build-1.11.8\bin>hwloc-info.exe
>>> depth 0:1 Machine (type #1)
>>>   depth 1:   2 NUMANode (type #2)
>>>depth 2:  2 Package (type #3)
>>> depth 3: 2 L3Cache (type #4)
>>>  depth 4:12 L2Cache (type #4)
>>>   depth 5:   12 L1dCache (type #4)
>>>depth 6:  12 L1iCache (type #4)
>>> depth 7: 12 Core (type #5)
>>>  depth 8:24 PU (type #6)
>>>
>>> C:\temp\hwloc-win64-build-1.11.8\bin>hwloc-ls.exe
>>> Machine (1506MB total)
>>>NUMANode L#0 (P#0 346MB) + Package L#0 + L3 L#0 (12MB)
>>>  L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
>>>PU L#0 (P#0)
>>>PU L#1 (P#1)
>>>  L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
>>>PU L#2 (P#2)
>>>PU L#3 (P#3)
>>>  L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
>>>PU L#4 (P#4)
>>>PU L#5 (P#5)
>>>  L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
>>>PU L#6 (P#6)
>>>PU L#7 (P#7)
>>>  L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
>>>PU L#8 (P#8)
>>>PU L#9 (P#9)
>>>  L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
>>>PU L#10 (P#10)
>>>PU L#11 (P#11)
>>>NUMANode L#1 (P#1 1160MB) + Package L#1 + L3 L#1 (12MB)
>>>  L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
>>>PU L#12 (P#64)
>>>PU L#13 (P#65)
>>>  L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
>>>PU L#14 (P#66)
>>>PU L#15 (P#67)
>>>  L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8
>>>PU L#16 (P#68)
>>>PU L#17 (P#69)
>>>  L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9
>>>PU L#18 (P#70)
>>>PU L#19 (P#71)
>>>  L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
>>>PU L#20 (P#72)
>>>PU L#21 (P#73)
>>>  L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
>>>PU L#22 (P#74)
>>>PU L#23 (P#75)
>>>
>>> I definitely have 2 processor groups:
>>> C:\Windows\system32>bcdedit /enum | find "group"
>>> groupsize   6
>>> maxgroupYes
>>>
>>> And you can see this because the processor numbers above in the second
>>> numa node start at 64. Also, calling GetActiveProcessorGroupCount()
>>> returns 2.
>>>
>>> I was expecting to get "2" back from:
>>> hwloc_get_nbobjs_by_type(hwlocTopology_, HWLOC_OBJ_GROUP)
>>>
>>> but that returns 0. Am I doing something wrong?
>>>
>>> Thank you!
>>>
>>> David
>>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>>
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] How are processor groups under Windows reported?

2017-11-29 Thread Brice Goglin
Hello

We only add hwloc Group objects when necessary. On your system, each
processor group contains a single NUMA node, so these Groups would not
really bring additional information about the hierarchy of resources.
If you had a bigger system with, let's say, 4 NUMA nodes, with 2 of them
in each processor groups, hwloc would report those as hwloc Group objects.

Does this help? I can clarify the FAQ if needed.

Brice



Le 29/11/2017 14:25, David Creasy a écrit :
> Hello,
>
> Thank you to all contributors to hwloc - very useful.
>
> In the FAQ,  under the section "What are these Group objects in my
> topology?" it says that they are used for "Windows processor groups".
> However, I'm either not seeing this, or I'm looking in the wrong
> place. On a system with two processor groups, I get:
>
> C:\temp\hwloc-win64-build-1.11.8\bin>hwloc-info.exe
> depth 0:1 Machine (type #1)
>  depth 1:   2 NUMANode (type #2)
>   depth 2:  2 Package (type #3)
>depth 3: 2 L3Cache (type #4)
> depth 4:12 L2Cache (type #4)
>  depth 5:   12 L1dCache (type #4)
>   depth 6:  12 L1iCache (type #4)
>depth 7: 12 Core (type #5)
> depth 8:24 PU (type #6)
>
> C:\temp\hwloc-win64-build-1.11.8\bin>hwloc-ls.exe
> Machine (1506MB total)
>   NUMANode L#0 (P#0 346MB) + Package L#0 + L3 L#0 (12MB)
> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0
>   PU L#0 (P#0)
>   PU L#1 (P#1)
> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1
>   PU L#2 (P#2)
>   PU L#3 (P#3)
> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2
>   PU L#4 (P#4)
>   PU L#5 (P#5)
> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3
>   PU L#6 (P#6)
>   PU L#7 (P#7)
> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4
>   PU L#8 (P#8)
>   PU L#9 (P#9)
> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5
>   PU L#10 (P#10)
>   PU L#11 (P#11)
>   NUMANode L#1 (P#1 1160MB) + Package L#1 + L3 L#1 (12MB)
> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6
>   PU L#12 (P#64)
>   PU L#13 (P#65)
> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7
>   PU L#14 (P#66)
>   PU L#15 (P#67)
> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8
>   PU L#16 (P#68)
>   PU L#17 (P#69)
> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9
>   PU L#18 (P#70)
>   PU L#19 (P#71)
> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
>   PU L#20 (P#72)
>   PU L#21 (P#73)
> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
>   PU L#22 (P#74)
>   PU L#23 (P#75)
>
> I definitely have 2 processor groups:
> C:\Windows\system32>bcdedit /enum | find "group"
> groupsize   6
> maxgroupYes
>
> And you can see this because the processor numbers above in the second
> numa node start at 64. Also, calling GetActiveProcessorGroupCount()
> returns 2.
>
> I was expecting to get "2" back from:
> hwloc_get_nbobjs_by_type(hwlocTopology_, HWLOC_OBJ_GROUP)
>
> but that returns 0. Am I doing something wrong?
>
> Thank you!
>
> David
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] [WARNING: A/V UNSCANNABLE] Dual socket AMD Epyc error

2017-11-22 Thread Brice Goglin
FWIW, the kernel fix from 4.14 is going to linux stable. It should be in
the upcoming 4.13.16.
I also requested a backport to older kernels but I don't know if it can
happen. Hopefully, distributions will pick up the patch from the 4.13
stable branch and backport it to their own kernels.

Brice



Le 28/10/2017 09:31, Brice Goglin a écrit :
> Hello,
> The Linux kernel reports incorrect L3 information.
> Unfortunately, your old kernel seems to already contain patches for
> supporting the L3 on this hardware. I found two candidate patches for
> further fixing this, one is in 4.10 (cleanup of the above patch) and
> the other will only be in 4.14.
> I am going to ask AMD about this.
> Brice
>
>
>
> Le 28/10/2017 06:29, Bill Broadley a écrit :
>> Dual socket Epyc 7451 server running a linux 4.4.0 kernel.
>>
>> When I run OpenMPI jobs I get the message to email here:
>>
>>  L3 (cpuset 0x0060,0x0060) intersects with NUMANode (P#0 cpuset
>> 0x003f,0x003f) without inclusion!
>> * Error occurred in topology.c line 1048
>>
>> Here's the requested files:
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

[hwloc-users] RFCs about latest API changes

2017-11-19 Thread Brice Goglin
Hello

Here are 4 pull requests about the likely-last significant API changes
for hwloc 2.0. You'll get more details by clicking on the links. I'll
merge these next week unless somebody complains.

Only maintain allowed_cpuset and allowed_nodeset for the entire topology
https://github.com/open-mpi/hwloc/pull/277

Make all depths *signed* ints
https://github.com/open-mpi/hwloc/pull/276

Remove the "System" object type
https://github.com/open-mpi/hwloc/pull/275

Move local_memory to NUMA node specific attrs
https://github.com/open-mpi/hwloc/pull/274

Brice





Le 26/10/2017 17:36, Brice Goglin a écrit :
> Hello
>
> I finally merged the new memory model in master (mainly for properly
> supporting KNL-like heterogeneous memory). This was the main and last
> big change for hwloc 2.0. I still need to fix some caveats (and lstopo
> needs to better display NUMA nodes) but that part of the API should be
> ready.
>
> Now we encourage people to start porting their code to the new hwloc 2.0
> API. Here's a guide that should answer most questions about the upgrade:
> https://github.com/open-mpi/hwloc/wiki/Upgrading-to-v2.0-API
>
> The final 2.0 release isn't planned before at least the end of november,
> but we need to fix API issues before releasing it. So please start
> testing it and report issues, missing docs, etc. If there's any existing
> function/feature (either new or old) that needs to be changed, please
> report it too. We're only breaking the ABI once for 2.0, we cannot break
> it again 2 months later.
>
>
> Tarballs of git master are already available from
> https://ci.inria.fr/hwloc/job/master-0-tarball/lastBuild/
> and in nightly snapshots on the website starting tomorrow.
>
>
> There are still a couple things that may or may not change before the
> final 2.0 API. If you have an opinion, please let us know.
> * (likely) Make all depths *signed* ints: some objects have a negative
> depth (meaning "special depth", not a normal depth in the main tree).
> You'll have to cast to (int) whenever you printf a depth while
> supporting both hwloc 1.x and 2.x.
> * (likely) Drop obj->allowed_cpuset (and allowed_nodeset) and just keep
> one for the entire topology: It is very rarely used (only when you set
> HWLOC_TOPOLOGY_FLAG_WHOLESYSTEM) and can be emulated by doing a binary
> "and" or "intersects" between obj->cpuset and topology->allowed_cpuset.
> * (likely) obj->memory becomes obj->attr->numanode since it's only used
> for numa nodes. But obj->total_memory should remain in obj because it's
> available in all objects (accumulated memory in all children).
> * (likely) Remove HWLOC_OBJ_SYSTEM: not used anymore (we don't support
> multinode topologies anymore). The root is always MACHINE now. I guess
> we'd #define SYSTEM MACHINE so that you don't have to change your code.
> * (unlikely) rename some info objects for consistency (examples below).
>   + GPUVendor and PCIVendor and CPUVendor -> Vendor.
>   + GPUModel and PCIDevice and CPUModel -> Model
>   + NVIDIASerial and MICSerialNumber -> SerialNumber
> But that will make your life harder for looking up attributes while
> supporting hwloc 1.x and 2.x. And XML import from 1.x would be more
> expensive since we'd have to rename these.
> * (unlikely) Share information between osdev (e.g. eth0 or cuda0) and
> pcidev: Lots of attributes are identical (Vendor, Model, kind of device
> etc). We could merge those objects into a single generic "I/O object".
> However a single PCI device can contain multiple OS devices (for
> instance "mlx5_0"+"ib0", or "cuda0"+"opencl0d0", etc).
>
>
> --
> Brice
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-15 Thread Brice Goglin
A random guess would be that numactl-devel isn't installed on the
system. It's required for numa-related syscalls (there's a summary at
the end of configure saying whether libnuma was found, or you can ldd
libhwloc.so and see if libnuma is listed). This will go away in 2.0
because we don't use libnuma at all anymore.

Brice



Le 15/11/2017 09:51, Biddiscombe, John A. a écrit :
> Running my test on another machine (fedora 7.2) I get a 
> hwloc_get_area_memlocation fail
> with strerror = "Function not implemented"
>
> Does this mean that the OS has not implemented it (I'm using 1.11.8 hwloc 
> version - on the primary test machine I used 1.11.17) - am I doomed? - or 
> will things magically work if I upgrade to hwloc 2.0 etc etc
>
> Thanks
>
> JB
>
>
> -Original Message-
> From: hwloc-users [mailto:hwloc-users-boun...@lists.open-mpi.org] On Behalf 
> Of Biddiscombe, John A.
> Sent: 13 November 2017 15:37
> To: Hardware locality user list <hwloc-users@lists.open-mpi.org>
> Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset
>
> It's working and I'm seeing the binding pattern I hoped for.
>
> Thanks again
>
> JB
>
> ________
> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf of Brice 
> Goglin [brice.gog...@inria.fr]
> Sent: 13 November 2017 15:32
> To: Hardware locality user list
> Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset
>
> The doc is wrong, flags are used, only for BY_NODESET. I actually fixed that 
> in git very recently.
>
> Brice
>
>
>
> Le 13/11/2017 07:24, Biddiscombe, John A. a écrit :
>> In the documentation for get_area_memlocation it says "If 
>> HWLOC_MEMBIND_BYNODESET is specified, set is considered a nodeset. Otherwise 
>> it's a cpuset."
>>
>> but it also says "Flags are currently unused."
>>
>> so where should the BY_NODESET policy be used? Does it have to be used with 
>> the original alloc call?
>>
>> thanks
>>
>> JB
>>
>> 
>> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf 
>> of Biddiscombe, John A. [biddi...@cscs.ch]
>> Sent: 13 November 2017 14:59
>> To: Hardware locality user list
>> Subject: Re: [hwloc-users] question about 
>> hwloc_set_area_membind_nodeset
>>
>> Brice
>>
>> aha. thanks. I knew I'd seen a function for that, but couldn't remember what 
>> it was.
>>
>> Cheers
>>
>> JB
>> 
>> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf 
>> of Brice Goglin [brice.gog...@inria.fr]
>> Sent: 13 November 2017 14:57
>> To: Hardware locality user list
>> Subject: Re: [hwloc-users] question about 
>> hwloc_set_area_membind_nodeset
>>
>> Use get_area_memlocation()
>>
>> membind() returns where the pages are *allowed* to go (anywhere)
>> memlocation() returns where the pages are actually allocated.
>>
>> Brice
>>
>>
>>
>>
>> Le 13/11/2017 06:52, Biddiscombe, John A. a écrit :
>>> Thank you to you both.
>>>
>>> I modified the allocator to allocate one large block using 
>>> hwloc_alloc and then use one thread per numa domain to  touch each 
>>> page according to the tiling pattern - unfortunately, I hadn't 
>>> appreciated that now hwloc_get_area_membind_nodeset always returns 
>>> the full machine numa mask - and not the numa domain that the page 
>>> was touched by (I guess it only gives the expected answer when 
>>> set_area_membind is used first)
>>>
>>> I had hoped to use a dynamic query of the pages (using the first one of a 
>>> given tile) to schedule each task that operates on a given tile to run on 
>>> the numa node that touched it.
>>>
>>> I can work around this by using a matrix offset calculation to get the numa 
>>> node, but if there's a way of querying the page directly - then please let 
>>> me know.
>>>
>>> Thanks
>>>
>>> JB
>>> 
>>> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf 
>>> of Samuel Thibault [samuel.thiba...@inria.fr]
>>> Sent: 12 November 2017 10:48
>>> To: Hardware locality user list
>>> Subject: Re: [hwloc-users] question about 
>>> hwloc_set_area_membind_nodeset
>>>
>>> Brice Goglin, on dim. 12 nov. 2017 05:19:37 +0100, wrote:
>>>> That's likely w

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-13 Thread Brice Goglin
The doc is wrong, flags are used, only for BY_NODESET. I actually fixed
that in git very recently.

Brice



Le 13/11/2017 07:24, Biddiscombe, John A. a écrit :
> In the documentation for get_area_memlocation it says
> "If HWLOC_MEMBIND_BYNODESET is specified, set is considered a nodeset. 
> Otherwise it's a cpuset."
>
> but it also says "Flags are currently unused."
>
> so where should the BY_NODESET policy be used? Does it have to be used with 
> the original alloc call?
>
> thanks
>
> JB
>
> 
> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf of 
> Biddiscombe, John A. [biddi...@cscs.ch]
> Sent: 13 November 2017 14:59
> To: Hardware locality user list
> Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset
>
> Brice
>
> aha. thanks. I knew I'd seen a function for that, but couldn't remember what 
> it was.
>
> Cheers
>
> JB
> ____
> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf of Brice 
> Goglin [brice.gog...@inria.fr]
> Sent: 13 November 2017 14:57
> To: Hardware locality user list
> Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset
>
> Use get_area_memlocation()
>
> membind() returns where the pages are *allowed* to go (anywhere)
> memlocation() returns where the pages are actually allocated.
>
> Brice
>
>
>
>
> Le 13/11/2017 06:52, Biddiscombe, John A. a écrit :
>> Thank you to you both.
>>
>> I modified the allocator to allocate one large block using hwloc_alloc and 
>> then use one thread per numa domain to  touch each page according to the 
>> tiling pattern - unfortunately, I hadn't appreciated that now
>> hwloc_get_area_membind_nodeset
>> always returns the full machine numa mask - and not the numa domain that the 
>> page was touched by (I guess it only gives the expected answer when 
>> set_area_membind is used first)
>>
>> I had hoped to use a dynamic query of the pages (using the first one of a 
>> given tile) to schedule each task that operates on a given tile to run on 
>> the numa node that touched it.
>>
>> I can work around this by using a matrix offset calculation to get the numa 
>> node, but if there's a way of querying the page directly - then please let 
>> me know.
>>
>> Thanks
>>
>> JB
>> ________
>> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf of 
>> Samuel Thibault [samuel.thiba...@inria.fr]
>> Sent: 12 November 2017 10:48
>> To: Hardware locality user list
>> Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset
>>
>> Brice Goglin, on dim. 12 nov. 2017 05:19:37 +0100, wrote:
>>> That's likely what's happening. Each set_area() may be creating a new 
>>> "virtual
>>> memory area". The kernel tries to merge them with neighbors if they go to 
>>> the
>>> same NUMA node. Otherwise it creates a new VMA.
>> Mmmm, that sucks. Ideally we'd have a way to ask the kernel not to
>> strictly bind the memory, but just to allocate on a given memory
>> node, and just hope that the allocation will not go away (e.g. due to
>> swapping), which thus doesn't need a VMA to record the information. As
>> you describe below, first-touch achieves that but it's not necessarily
>> so convenient.
>>
>>> I can't find the exact limit but it's something like 64k so I guess
>>> you're exhausting that.
>> It's sysctl vm.max_map_count
>>
>>> Question 2 : Is there a better way of achieving the result I'm looking 
>>> for
>>> (such as a call to membind with a stride of some kind to say put N 
>>> pages in
>>> a row on each domain in alternation).
>>>
>>>
>>> Unfortunately, the interleave policy doesn't have a stride argument. It's 
>>> one
>>> page on node 0, one page on node 1, etc.
>>>
>>> The only idea I have is to use the first-touch policy: Make sure your buffer
>>> isn't is physical memory yet, and have a thread on node 0 read the "0" 
>>> pages,
>>> and another thread on node 1 read the "1" page.
>> Or "next-touch" if that was to ever get merged into mainline Linux :)
>>
>> Samuel
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
>> 

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-13 Thread Brice Goglin
Use get_area_memlocation()

membind() returns where the pages are *allowed* to go (anywhere)
memlocation() returns where the pages are actually allocated.

Brice




Le 13/11/2017 06:52, Biddiscombe, John A. a écrit :
> Thank you to you both.
>
> I modified the allocator to allocate one large block using hwloc_alloc and 
> then use one thread per numa domain to  touch each page according to the 
> tiling pattern - unfortunately, I hadn't appreciated that now
> hwloc_get_area_membind_nodeset
> always returns the full machine numa mask - and not the numa domain that the 
> page was touched by (I guess it only gives the expected answer when 
> set_area_membind is used first)
>
> I had hoped to use a dynamic query of the pages (using the first one of a 
> given tile) to schedule each task that operates on a given tile to run on the 
> numa node that touched it.
>
> I can work around this by using a matrix offset calculation to get the numa 
> node, but if there's a way of querying the page directly - then please let me 
> know.
>
> Thanks
>
> JB 
> 
> From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf of 
> Samuel Thibault [samuel.thiba...@inria.fr]
> Sent: 12 November 2017 10:48
> To: Hardware locality user list
> Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset
>
> Brice Goglin, on dim. 12 nov. 2017 05:19:37 +0100, wrote:
>> That's likely what's happening. Each set_area() may be creating a new 
>> "virtual
>> memory area". The kernel tries to merge them with neighbors if they go to the
>> same NUMA node. Otherwise it creates a new VMA.
> Mmmm, that sucks. Ideally we'd have a way to ask the kernel not to
> strictly bind the memory, but just to allocate on a given memory
> node, and just hope that the allocation will not go away (e.g. due to
> swapping), which thus doesn't need a VMA to record the information. As
> you describe below, first-touch achieves that but it's not necessarily
> so convenient.
>
>> I can't find the exact limit but it's something like 64k so I guess
>> you're exhausting that.
> It's sysctl vm.max_map_count
>
>> Question 2 : Is there a better way of achieving the result I'm looking 
>> for
>> (such as a call to membind with a stride of some kind to say put N pages 
>> in
>> a row on each domain in alternation).
>>
>>
>> Unfortunately, the interleave policy doesn't have a stride argument. It's one
>> page on node 0, one page on node 1, etc.
>>
>> The only idea I have is to use the first-touch policy: Make sure your buffer
>> isn't is physical memory yet, and have a thread on node 0 read the "0" pages,
>> and another thread on node 1 read the "1" page.
> Or "next-touch" if that was to ever get merged into mainline Linux :)
>
> Samuel
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-11 Thread Brice Goglin


Le 12/11/2017 00:14, Biddiscombe, John A. a écrit :
> I'm allocating some large matrices, from 10k squared elements up to
> 40k squared per node.
> I'm also using membind to place pages of the matrix memory across numa
> nodes so that the matrix might be bound according to the kind of
> pattern at the end of this email - where each 1 or 0 corresponds to a
> 256x256 block of memory.
>
> The way I'm doing this is by calling hwloc_set_area_membind_nodeset
> many thousands of times after allocation, and I've found that as the
> matrices get bigger, then after some N calls to area_membind then I
> get a failure and it returns -1 (errno does not seem to be set to
> either ENOSYS or EXDEV) - but strerror report "Cannot allocate memory".
>
> Question 1 : by calling area_setmembind too many times, am I causing
> some resource usage in the memory tables that is being exhausted.
>

Hello

That's likely what's happening. Each set_area() may be creating a new
"virtual memory area". The kernel tries to merge them with neighbors if
they go to the same NUMA node. Otherwise it creates a new VMA. I can't
find the exact limit but it's something like 64k so I guess you're
exhausting that.

> Question 2 : Is there a better way of achieving the result I'm looking
> for (such as a call to membind with a stride of some kind to say put N
> pages in a row on each domain in alternation).

Unfortunately, the interleave policy doesn't have a stride argument.
It's one page on node 0, one page on node 1, etc.

The only idea I have is to use the first-touch policy: Make sure your
buffer isn't is physical memory yet, and have a thread on node 0 read
the "0" pages, and another thread on node 1 read the "1" page.

Brice


>
> Many thanks
>
> JB
>
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ... etc
>
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] HWLOC_VERSION

2017-10-30 Thread Brice Goglin
Hello

It should have been 0x00010b03 but I forgot to increase it unfortunately
(and again in 1.11.6).
I need to add this to my release-TODO-list.

The upcoming 1.11.9 will have the proper HWLOC_API_VERSION (0x00010b06
unless we had something) so that people can at least check for these
features in newer releases...

If you have some configure checks for hwloc, you could add something
there to workaround the issue.

Sorry
Brice







Le 30/10/2017 09:09, Biddiscombe, John A. a écrit :
> Dear list
>
> According to the release notes Add HWLOC_MEMBIND_BYNODESET flag was added in 
> 1.11.3 - if I protect some code with 
>
> #if HWLOC_API_VERSION >= 0x00010b00
> then versions 1.11.0, 1.11.1, 1.11.2 still cause build failures.
>
> is there some VERSION flag that distinguishes between the patch version 
> releases?
>
> thanks
>
> JB
>
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] [WARNING: A/V UNSCANNABLE] Dual socket AMD Epyc error

2017-10-28 Thread Brice Goglin
Hello,
The Linux kernel reports incorrect L3 information.
Unfortunately, your old kernel seems to already contain patches for
supporting the L3 on this hardware. I found two candidate patches for
further fixing this, one is in 4.10 (cleanup of the above patch) and the
other will only be in 4.14.
I am going to ask AMD about this.
Brice



Le 28/10/2017 06:29, Bill Broadley a écrit :
> Dual socket Epyc 7451 server running a linux 4.4.0 kernel.
>
> When I run OpenMPI jobs I get the message to email here:
>
>  L3 (cpuset 0x0060,0x0060) intersects with NUMANode (P#0 cpuset
> 0x003f,0x003f) without inclusion!
> * Error occurred in topology.c line 1048
>
> Here's the requested files:
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

[hwloc-users] new memory model and API

2017-10-26 Thread Brice Goglin
Hello

I finally merged the new memory model in master (mainly for properly
supporting KNL-like heterogeneous memory). This was the main and last
big change for hwloc 2.0. I still need to fix some caveats (and lstopo
needs to better display NUMA nodes) but that part of the API should be
ready.

Now we encourage people to start porting their code to the new hwloc 2.0
API. Here's a guide that should answer most questions about the upgrade:
https://github.com/open-mpi/hwloc/wiki/Upgrading-to-v2.0-API

The final 2.0 release isn't planned before at least the end of november,
but we need to fix API issues before releasing it. So please start
testing it and report issues, missing docs, etc. If there's any existing
function/feature (either new or old) that needs to be changed, please
report it too. We're only breaking the ABI once for 2.0, we cannot break
it again 2 months later.


Tarballs of git master are already available from
https://ci.inria.fr/hwloc/job/master-0-tarball/lastBuild/
and in nightly snapshots on the website starting tomorrow.


There are still a couple things that may or may not change before the
final 2.0 API. If you have an opinion, please let us know.
* (likely) Make all depths *signed* ints: some objects have a negative
depth (meaning "special depth", not a normal depth in the main tree).
You'll have to cast to (int) whenever you printf a depth while
supporting both hwloc 1.x and 2.x.
* (likely) Drop obj->allowed_cpuset (and allowed_nodeset) and just keep
one for the entire topology: It is very rarely used (only when you set
HWLOC_TOPOLOGY_FLAG_WHOLESYSTEM) and can be emulated by doing a binary
"and" or "intersects" between obj->cpuset and topology->allowed_cpuset.
* (likely) obj->memory becomes obj->attr->numanode since it's only used
for numa nodes. But obj->total_memory should remain in obj because it's
available in all objects (accumulated memory in all children).
* (likely) Remove HWLOC_OBJ_SYSTEM: not used anymore (we don't support
multinode topologies anymore). The root is always MACHINE now. I guess
we'd #define SYSTEM MACHINE so that you don't have to change your code.
* (unlikely) rename some info objects for consistency (examples below).
  + GPUVendor and PCIVendor and CPUVendor -> Vendor.
  + GPUModel and PCIDevice and CPUModel -> Model
  + NVIDIASerial and MICSerialNumber -> SerialNumber
But that will make your life harder for looking up attributes while
supporting hwloc 1.x and 2.x. And XML import from 1.x would be more
expensive since we'd have to rename these.
* (unlikely) Share information between osdev (e.g. eth0 or cuda0) and
pcidev: Lots of attributes are identical (Vendor, Model, kind of device
etc). We could merge those objects into a single generic "I/O object".
However a single PCI device can contain multiple OS devices (for
instance "mlx5_0"+"ib0", or "cuda0"+"opencl0d0", etc).


--
Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] linkspeed in hwloc_obj_attr_u::hwloc_pcidev_attr_s struct while traversing topology

2017-10-13 Thread Brice Goglin
Hello

On Linux, the PCI linkspeed requires root privileges unfortunately
(except for the uplink above NVIDIA GPUs where we have another way to
find it).
The only way to workaround this is to dump the topology as XML as root
and then reload it at runtime (e.g. with HWLOC_XMLFILE) :/

Brice



Le 13/10/2017 10:53, TEJASWI k a écrit :
> I am trying to traverse the topology using hwloc APIs starting from a
> PCI device till Host Bridge
>
> My code snippet:
>
> unsigned long flags = HWLOC_TOPOLOGY_FLAG_IO_DEVICES |
> HWLOC_TOPOLOGY_FLAG_IO_BRIDGES;
>
> retval = hwloc_topology_init();
> retval = hwloc_topology_set_flags(topology, flags);
> retval = hwloc_topology_load(topology);
>
> pciObj = hwloc_get_pcidev_by_busidstring(topology, "");
> while(pciObj) {
> pciObj = pciObj->parent;
>
> if (pciObj->attr->bridge.upstream_type !=
> HWLOC_OBJ_BRIDGE_HOST)
> {
>   //Get all the required information about
> intermediate bridges like
>  
> // pciObj->attr->bridge.downstream.pci.secondary_bus, 
> pciObj->attr->bridge.upstream.pci.domain
>   // pciObj->attr->bridge.upstream.pci.bus,
> *_pciObj->attr->bridge.upstream.pci.linkspeed_*
> }
> }
>
> All the other details I am able to query but linkspeed
> (*_pciObj->attr->bridge.upstream.pci.linkspeed_*) is always 0.
> Do I need to enable any other flag to get linkspeed or am I going
> wrong somewhere?
>
> I want to get the PCI Bridges' generation or linkspeed for my usecase.
> Is there any other way to get this information?
>
>
> Thanks & Regards,
> Tejaswi K
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Why do I get such little information back about GPU's on my system

2017-07-07 Thread Brice Goglin
Does "ldd /path/to/libhwloc.so" say that it depends on libcuda?

Check the end of the configure output, you should see "CUDA" on the
"probe/display I/O devices" line:

-
Hwloc optional build support status (more details can be found above):

Probe / display I/O devices: PCI(pciaccess+linux) LinuxIO OpenCL CUDA NVML GL
Graphical output (Cairo):yes
XML input / output:  full
Netloc functionality:yes (with scotch: no)
Plugin support:  no
-

If not, go above and look for CUDA checks:

checking for cuda.h... yes
checking if CUDA_VERSION >= 3020... yes
checking for cuInit in -lcuda... yes
checking cuda_runtime_api.h usability... yes
checking cuda_runtime_api.h presence... yes
checking for cuda_runtime_api.h... yes
checking if CUDART_VERSION >= 3020... yes
checking for cudaGetDeviceProperties in -lcudart... yes

Brice




Le 07/07/2017 23:36, David Solt a écrit :
> Ok here we have a machine with cuda installed:
>
> # rpm -qa | grep cuda
> cuda-nvgraph-8-0-8.0.54-1.ppc64le
> cuda-curand-dev-8-0-8.0.54-1.ppc64le
> cuda-nvrtc-dev-8-0-8.0.54-1.ppc64le
> cuda-cudart-8-0-8.0.54-1.ppc64le
> cuda-cusolver-dev-8-0-8.0.54-1.ppc64le
> cuda-8.0.54-1.ppc64le
> cuda-driver-dev-8-0-8.0.54-1.ppc64le
> cuda-core-8-0-8.0.54-1.ppc64le
> etc, etc, etc
>
> #lspci | grep NVIDIA
>
> [root@c712f6n06 ~]# lspci | grep NVI
> 0002:01:00.0 3D controller: NVIDIA Corporation GP100GL (rev a1)
> 0003:01:00.0 3D controller: NVIDIA Corporation GP100GL (rev a1)
> 0006:01:00.0 3D controller: NVIDIA Corporation GP100GL (rev a1)
> 0007:01:00.0 3D controller: NVIDIA Corporation GP100GL (rev a1)
>
> But the only devices returned by hwloc are named "cardX" (same as what
> lstopo shows) and have osdev.type of HWLOC_OBJ_OSDEV_GPU and we see no
> devices of type HWLOC_OBJ_OSDEV_COPROC
>
> Sorry, I'm sure I'm doing something stupid here... I'm certainly new
> to using hwloc.
>
> Dave
>
>
>
> From:Brice Goglin <brice.gog...@inria.fr>
> To:Hardware locality user list <hwloc-users@lists.open-mpi.org>
> Date:07/07/2017 02:54 PM
> Subject:Re: [hwloc-users] Why do I get such little information
> back about GPU's on my system
> Sent by:"hwloc-users" <hwloc-users-boun...@lists.open-mpi.org>
> 
>
>
>
> Le 07/07/2017 21:51, David Solt a écrit :
> Oh, Geoff Paulsen will be there at Open MPI meeting and he can help
> with the discussion.   We tried searching for
>
>   // Iterate over each osdevice and identify the GPU's on each socket.
>   while ((obj = hwloc_get_next_osdev(machine_topology, obj)) != NULL) {
> if (HWLOC_OBJ_OSDEV_COPROC == obj->attr->osdev.type) {   // was
> HWLOC_OBJ_OSDEV_GPU
>
> Currently we are not finding any such devices.   Does this require
> that the cuda libraries be installed on the system for hwloc to find
> the hardware?
>
> Yes. We use the cuda API for listing these objects.
> Brice
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Why do I get such little information back about GPU's on my system

2017-07-07 Thread Brice Goglin
Le 07/07/2017 20:38, David Solt a écrit :
> We are using the hwloc api to identify GPUs on our cluster. While we
> are able to "discover" the GPUs, other information about them does not
> appear to be getting filled in. See below for example:

> (gdb) p *obj->attr
> $20 = {
>   cache = {
> size = 1,
> depth = 0,
> linesize = 0,
> associativity = 0,
> type = HWLOC_OBJ_CACHE_UNIFIED
>   },
>   group = {
> depth = 1
>   },
>   pcidev = {
> domain = 1,
> bus = 0 '\000',
> dev = 0 '\000',
> func = 0 '\000',
> class_id = 0,
> *vendor_id = 0,*
>*device_id = 0, *
> subvendor_id = 0,
> subdevice_id = 0,
> revision = 0 '\000',
> linkspeed = 0
>   },
>   bridge = {
> upstream = {
>   pci = {
> domain = 1,
> bus = 0 '\000',
> dev = 0 '\000',
> func = 0 '\000',
> class_id = 0,
> vendor_id = 0,
> device_id = 0,
> subvendor_id = 0,
> subdevice_id = 0,
> revision = 0 '\000',
> linkspeed = 0
>   }
> },
> upstream_type = HWLOC_OBJ_BRIDGE_HOST,
> downstream = {
>   pci = {
> domain = 0,
> secondary_bus = 0 '\000',
> subordinate_bus = 0 '\000'
>   }
> },
> downstream_type = HWLOC_OBJ_BRIDGE_HOST,
> depth = 0
>   },
>   osdev = {
> type = *HWLOC_OBJ_OSDEV_GPU*
>   }
> }

> The name is generally just "cardX". 


Hello

attr is an union so only the "osdev" portion above matters. "osdev" can
be a lot of different things. So instead of having all possible
attributes in a struct, we use info key/value pairs (hwloc_obj->infos).
But those "cardX" devices are the GPU reported by the Linux kernel DRM
subsystem, we don't have much information about them anyway.

If you're looking at Power machine, I am going to assume you care about
CUDA devices. Those are "osdev" objects of type "COPROC" instead of
"GPU". They have many more attributes. Here's what I see on one of our
machines:

  PCI 10de:1094 (P#540672 busid=:84:00.0 class=0302(3D) PCIVendor="NVIDIA 
Corporation" PCIDevice="Tesla M2075 Dual-Slot Computing Processor Module") 
"NVIDIA Corporation Tesla M2075 Dual-Slot Computing Processor Module"
Co-Processor L#5 (CoProcType=CUDA Backend=CUDA GPUVendor="NVIDIA 
Corporation" GPUModel="Tesla M2075" CUDAGlobalMemorySize=5428224 
CUDAL2CacheSize=768 CUDAMultiProcessors=14 CUDACoresPerMP=32 
CUDASharedMemorySizePerMP=48) "cuda2"


On recent kernels, you would see both a "cardX" GPU osdev and a "cudaX"
COPROC osdev in the PCI device. There can even be "nvmlX" and ":0.0" if
you have the nvml and nvctrl libraries. Those are basically different
ways to talk the GPU (Linux kernel DRM, CUDA, etc).

Given that I have never seen anybody use "cardX" for placing task/data
near a GPU, I am wondering if we should disable those by default. Or
maybe rename "GPU" into something that wouldn't attract people as much,
maybe "DRM".

> Does this mean that the cards are not configured correctly? Or is
> there an additional flag that needs to be set to get this information?


Make sure "cuda" appears in the summary at the end of the configure.

> Currently the code does:

>   hwloc_topology_init(_topology);
>   hwloc_topology_set_flags(machine_topology,
> HWLOC_TOPOLOGY_FLAG_IO_DEVICES);
>   hwloc_topology_load(machine_topology);

> And this is enough to identify the CPUs and GPUs, but any additional
> information - particularly the device and vendor id's - seem to not be
> there. 

> I tried this with the most recent release (1.11.7) and saw the same
> results.   

> We tried this on a variety of PowerPC machines and I think even some
> x86_64 machines with similar results.   

> Thoughts?
> Dave

BTW, it looks like you're not going to the OMPI dev meeting next week.
I'll be there if one of your colleague wants to discuss this face to face.

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc error in SuperMicro AMD Opteron 6238

2017-06-30 Thread Brice Goglin
Hello

We have seen _many_ reports like these. But there are different kinds of
errors. As far as I understand:

* Julio's error is caused by the Linux kernel improperly reporting L3
cache affinities. It's specific to multi-socket 12-core processors
because the kernel makes invalid assumptions about core APIC IDs in
these processors (because only 12 out of 16 cores are enabled).
HWLOC_COMPONENTS=x86 was designed to solve this issue until AMD fixed
the kernel, but it looks like they didn't.

* Your error looks like another issue where the BIOS reports invalid
NUMA affinity (likely in the SRAT table). A BIOS upgrade may help.
Fortunately, the x86 backend can also read NUMA affinity from CPUID
instructions on AMD. I didn't know/remember HWLOC_COMPONENTS=x86 could
help for this bug too.


I am going to add this workaround to the FAQ about these errors (this
FAQ is listed in the error below since 1.11).


By the way, you should upgrade. 1.10 is vey old :)

Brice




Le 30/06/2017 21:59, Belgin, Mehmet a écrit :
> We (Georgia Tech) too have been observing this on 16-core AMD AbuDhabi
> machines (6378). We weren’t aware of HWLOC_COMPONENTS workaround,
> which seems to mitigate the issue. 
>
> *Before:*
>
> # ./lstopo
> 
> * hwloc has encountered what looks like an error from the operating
> system.
> *
> * Socket (P#2 cpuset 0x,0x0) intersects with NUMANode (P#3
> cpuset 0xff00,0xff00) without inclusion!
> * Error occurred in topology.c line 940
> *
> * Please report this error message to the hwloc user's mailing list,
> * along with the output+tarball generated by the hwloc-gather-topology
> script.
> 
> Machine (128GB total)
>   Group0 L#0
> NUMANode L#0 (P#1 32GB)
> ...
>
> *After:*
>
> # export HWLOC_COMPONENTS=x86
> # ./lstopo
> Machine
>   Socket L#0
> NUMANode L#0 (P#0) + L3 L#0 (6144KB)
>   L2 L#0 (2048KB) + L1i L#0 (64KB)
> ...
>
> These nodes are the only one in our entire cluster to cause zombie
> processes using torque/moab. I have a feeling that they are related.
> We use hwloc/1.10.0.
>
> Not sure if this helps at all, but you are definitely not alone :)
>
> Thanks,
> -Mehmet
>
>
>
>> On Jun 29, 2017, at 1:24 AM, Brice Goglin <brice.gog...@inria.fr
>> <mailto:brice.gog...@inria.fr>> wrote:
>>
>> Hello
>>
>> We've seen this issue many times (it's specific to 12-core opterons),
>> but I am surprised it still occurs with such a recent kernel. AMD was
>> supposed to fix the kernel in early 2016 but I forgot checking
>> whether something was actually pushed.
>>
>> Anyway, you can likely ignore the issue as documented in the FAQ
>> https://www.open-mpi.org/projects/hwloc/doc/v1.11.7/a00305.php unless
>> you care about L3 affinity for binding. Otherwise, you can workaround
>> the issue by passing HWLOC_COMPONENTS=x86 in the environment so that
>> hwloc uses cpuid before of Linux sysfs files for discovery the topology.
>>
>> Brice
>>
>>
>>
>>
>> Le 29/06/2017 02:17, Julio Figueroa a écrit :
>>> Hi
>>>
>>> I am experincing the following issues when using pnetcdf version 1.8.1
>>> The machine is a Supermicro (H8DGi) dual socket AMD Opteron 6238
>>> (patch_level=0x0600063d)
>>> The BIOS is the lates from Supermicro (v3.5c 03/18/2016)
>>> OS: Debian 9.0 Kernel: 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u1
>>> (2017-06-18) x86_64 GNU/Linux
>>> 
>>> * hwloc 1.11.5 has encountered what looks like an error from the
>>> operating system.
>>> *
>>> * L3 (cpuset 0x03f0) intersects with NUMANode (P#0 cpuset
>>> 0x003f) without inclusion!
>>> * Error occurred in topology.c line 1074
>>> *
>>> * The following FAQ entry in the hwloc documentation may help:
>>> *   What should I do when hwloc reports "operating system" warnings?
>>> * Otherwise please report this error message to the hwloc user's
>>> mailing list,
>>> * along with the output+tarball generated by the
>>> hwloc-gather-topology script.
>>> 
>>>
>>> As suggested by the error message, here is the hwloc-gather-topology
>>> attached.
>>>
>>> Please let me know if you need more information.
>>>
>>> Julio Figueroa
>>> Oceanographer
>>>
>>>
>>&

Re: [hwloc-users] hwloc error in SuperMicro AMD Opteron 6238

2017-06-30 Thread Brice Goglin
Le 30/06/2017 22:08, fabricio a écrit :
> Em 30-06-2017 16:21, Brice Goglin escreveu:
>> Yes, it's possible but very easy. Before we go that way:
>> Can you also pass HWLOC_COMPONENTS_VERBOSE=1 in the environment and send
>> the verbose output?
>
> ///
> Registered cpu discovery component `no_os' with priority 40
> (statically build)
> Registered global discovery component `xml' with priority 30
> (statically build)
> Registered global discovery component `synthetic' with priority 30
> (statically build)
> Registered global discovery component `custom' with priority 30
> (statically build)
> Registered cpu discovery component `linux' with priority 50
> (statically build)
> Registered misc discovery component `linuxpci' with priority 19
> (statically build)
> Registered misc discovery component `pci' with priority 20 (statically
> build)
> Registered cpu discovery component `x86' with priority 45 (statically
> build)
> Enabling cpu discovery component `linux'
> Enabling cpu discovery component `x86'
> Enabling cpu discovery component `no_os'
> Excluding global discovery component `xml', conflicts with excludes 0x2
> Excluding global discovery component `synthetic', conflicts with
> excludes 0x2
> Excluding global discovery component `custom', conflicts with excludes
> 0x2
> Enabling misc discovery component `pci'
> Enabling misc discovery component `linuxpci'
> Final list of enabled discovery components: linux,x86,no_os,pci,linuxpci
> 
>
> * hwloc has encountered what looks like an error from the operating
> system.
> *
> * L3 (cpuset 0x03f0) intersects with NUMANode (P#0 cpuset
> 0x003f) without inclusion!
> * Error occurred in topology.c line 942
> *
> * The following FAQ entry in a recent hwloc documentation may help:
> *   What should I do when hwloc reports "operating system" warnings?
> * Otherwise please report this error message to the hwloc user's
> mailing list,
> * along with the output+tarball generated by the hwloc-gather-topology
> script.
> 
>
> Enabling global discovery component `xml'
> Excluding cpu discovery component `linux', conflicts with excludes
> 0x
> Excluding cpu discovery component `x86', conflicts with excludes
> 0x
> Excluding cpu discovery component `no_os', conflicts with excludes
> 0x
> Excluding global discovery component `xml', conflicts with excludes
> 0x
> Excluding global discovery component `synthetic', conflicts with
> excludes 0x
> Excluding global discovery component `custom', conflicts with excludes
> 0x
> Excluding misc discovery component `pci', conflicts with excludes
> 0x
> Excluding misc discovery component `linuxpci', conflicts with excludes
> 0x
> Final list of enabled discovery components: xml
> ///
>
>> I am wondering if the x86 backend was disabled somehow.
>> Please also send your config.log
>
> I'm using the embebbed hwloc in openmpi 1.10.7, whose version seems to
> be 1.9.1. I could not find a config.log file.

I thought you were using hwloc 1.11.5? HWLOC_COMPONENTS=x86 can help
there, but not in 1.9.1 from OMPI. Which one did you try?

>
>> Setting HWLOC_COMPONENTS=-linux could also work: It totally disables the
>> Linux backend. If the x86 is disabled as well, you would get an almost
>> empty topology.
>
> Will this leave the process allocation to the kernel, potentially
> diminishing performance?

This would basically ignore all topology information.
But it's not needed anymore here since the x86 backend is enabled above.

What you can do is one of these:
* tell OMPI to use an external hwloc >= 1.11.2
* use a more recent OMPI :)
* use a XML generated with hwloc >= 1.11.2 with HWLOC_COMPONENTS=x86,
and pass it to OMPI and/or hwloc with HWLOC_XMLFILE=/path/to/foo.xml and
HWLOC_THISSYSTEM=1 in the environment. If it doesn't work, I'll generate
the XML

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] hwloc error in SuperMicro AMD Opteron 6238

2017-06-28 Thread Brice Goglin
Hello

We've seen this issue many times (it's specific to 12-core opterons),
but I am surprised it still occurs with such a recent kernel. AMD was
supposed to fix the kernel in early 2016 but I forgot checking whether
something was actually pushed.

Anyway, you can likely ignore the issue as documented in the FAQ
https://www.open-mpi.org/projects/hwloc/doc/v1.11.7/a00305.php unless
you care about L3 affinity for binding. Otherwise, you can workaround
the issue by passing HWLOC_COMPONENTS=x86 in the environment so that
hwloc uses cpuid before of Linux sysfs files for discovery the topology.

Brice




Le 29/06/2017 02:17, Julio Figueroa a écrit :
> Hi
>
> I am experincing the following issues when using pnetcdf version 1.8.1
> The machine is a Supermicro (H8DGi) dual socket AMD Opteron 6238
> (patch_level=0x0600063d)
> The BIOS is the lates from Supermicro (v3.5c 03/18/2016)
> OS: Debian 9.0 Kernel: 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u1
> (2017-06-18) x86_64 GNU/Linux
> 
> * hwloc 1.11.5 has encountered what looks like an error from the
> operating system.
> *
> * L3 (cpuset 0x03f0) intersects with NUMANode (P#0 cpuset
> 0x003f) without inclusion!
> * Error occurred in topology.c line 1074
> *
> * The following FAQ entry in the hwloc documentation may help:
> *   What should I do when hwloc reports "operating system" warnings?
> * Otherwise please report this error message to the hwloc user's
> mailing list,
> * along with the output+tarball generated by the hwloc-gather-topology
> script.
> 
>
> As suggested by the error message, here is the hwloc-gather-topology
> attached.
>
> Please let me know if you need more information.
>
> Julio Figueroa
> Oceanographer
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] ? Finding cache & pci info on SPARC/Solaris 11.3

2017-06-09 Thread Brice Goglin
Thanks a lot for the input.
I opened https://github.com/open-mpi/hwloc/issues/243
I have access to a T5 but this will need investigation to actually find
where to get the info from.
Feel free to comment the issue if you find more. I am going to modify
Pg.pm to better understand where Caches come from.

Brice




Le 09/06/2017 09:11, Maureen Chew a écrit :
> Re: cache relationship… ah… so you’d need to parse both
> prtpicl(8) (to get sizes)  and something like pginfo(8) (perl script)
> to get
> relationship…..
>
> bash-4.3$ pginfo -v | more
> 0 (System [system]) CPUs: 0-511
> |-- 5 (Data_Pipe_to_memory [socket 0]) CPUs: 0-255
> |   |-- 4 (L3_Cache) CPUs: 0-31
> |   |   `-- 6 (CPU_PM_Active_Power_Domain) CPUs: 0-31
> |   |   |-- 3 (L2_Cache) CPUs: 0-15
> |   |   |   |-- 2 (Floating_Point_Unit [core 0]) CPUs: 0-7
> |   |   |   |   `-- 1 (Integer_Pipeline [core 0]) CPUs: 0-7
> |   |   |   `-- 8 (Floating_Point_Unit [core 1]) CPUs: 8-15
> |   |   |   `-- 7 (Integer_Pipeline [core 1]) CPUs: 8-15
> |   |   `-- 11 (L2_Cache) CPUs: 16-31
> |   |   |-- 10 (Floating_Point_Unit [core 2]) CPUs: 16-23
> |   |   |   `-- 9 (Integer_Pipeline [core 2]) CPUs: 16-23
> |   |   `-- 13 (Floating_Point_Unit [core 3]) CPUs: 24-31
> |   |   `-- 12 (Integer_Pipeline [core 3]) CPUs: 24-31
> |   |-- 17 (L3_Cache) CPUs: 32-63
> |   |   `-- 18 (CPU_PM_Active_Power_Domain) CPUs: 32-63
> |   |   |-- 16 (L2_Cache) CPUs: 32-47
> |   |   |   |-- 15 (Floating_Point_Unit [core 4]) CPUs: 32-39
> |   |   |   |   `-- 14 (Integer_Pipeline [core 4]) CPUs: 32-39
> |   |   |   `-- 20 (Floating_Point_Unit [core 5]) CPUs: 40-47
> |   |   |   `-- 19 (Integer_Pipeline [core 5]) CPUs: 40-47
> |   |   `-- 23 (L2_Cache) CPUs: 48-63
> |   |   |-- 22 (Floating_Point_Unit [core 6]) CPUs: 48-55
> |   |   |   `-- 21 (Integer_Pipeline [core 6]) CPUs: 48-55
> |   |   `-- 25 (Floating_Point_Unit [core 7]) CPUs: 56-63
> |   |   `-- 24 (Integer_Pipeline [core 7]) CPUs: 56-63
> |   |-- 29 (L3_Cache) CPUs: 64-95
> |   |   `-- 30 (CPU_PM_Active_Power_Domain) CPUs: 64-95
> |   |   |-- 28 (L2_Cache) CPUs: 64-79
> |   |   |   |-- 27 (Floating_Point_Unit [core 8]) CPUs: 64-71
> |   |   |   |   `-- 26 (Integer_Pipeline [core 8]) CPUs: 64-71
> |   |   |   `-- 32 (Floating_Point_Unit [core 9]) CPUs: 72-79
> |   |   |   `-- 31 (Integer_Pipeline [core 9]) CPUs: 72-79
> |   |   `-- 35 (L2_Cache) CPUs: 80-95
> |   |   |-- 34 (Floating_Point_Unit [core 10]) CPUs: 80-87
> |   |   |   `-- 33 (Integer_Pipeline [core 10]) CPUs: 80-87
> |   |   `-- 37 (Floating_Point_Unit [core 11]) CPUs: 88-95
> |   |   `-- 36 (Integer_Pipeline [core 11]) CPUs: 88-95
> |   |-- 41 (L3_Cache) CPUs: 96-127
> |   |   `-- 42 (CPU_PM_Active_Power_Domain) CPUs: 96-127
> |   |   |-- 40 (L2_Cache) CPUs: 96-111
> |   |   |   |-- 39 (Floating_Point_Unit [core 12]) CPUs: 96-103
> |   |   |   |   `-- 38 (Integer_Pipeline [core 12]) CPUs: 96-103
> |   |   |   `-- 44 (Floating_Point_Unit [core 13]) CPUs: 104-111
> |   |   |   `-- 43 (Integer_Pipeline [core 13]) CPUs: 104-111
> |   |   `-- 47 (L2_Cache) CPUs: 112-127
> |   |   |-- 46 (Floating_Point_Unit [core 14]) CPUs: 112-119
> |   |   |   `-- 45 (Integer_Pipeline [core 14]) CPUs: 112-119
> |   |   `-- 49 (Floating_Point_Unit [core 15]) CPUs: 120-127
> |   |   `-- 48 (Integer_Pipeline [core 15]) CPUs: 120-127
> |   |-- 53 (L3_Cache) CPUs: 128-159
> |   |   `-- 54 (CPU_PM_Active_Power_Domain) CPUs: 128-159
> |   |   |-- 52 (L2_Cache) CPUs: 128-143
> |   |   |   |-- 51 (Floating_Point_Unit [core 16]) CPUs: 128-135
> |   |   |   |   `-- 50 (Integer_Pipeline [core 16]) CPUs: 128-135
> |   |   |   `-- 56 (Floating_Point_Unit [core 17]) CPUs: 136-143
> |   |   |   `-- 55 (Integer_Pipeline [core 17]) CPUs: 136-143
> |   |   `-- 59 (L2_Cache) CPUs: 144-159
> |   |   |-- 58 (Floating_Point_Unit [core 18]) CPUs: 144-151
> |   |   |   `-- 57 (Integer_Pipeline [core 18]) CPUs: 144-151
> |   |   `-- 61 (Floating_Point_Unit [core 19]) CPUs: 152-159
> |   |   `-- 60 (Integer_Pipeline [core 19]) CPUs: 152-159
> |   |-- 65 (L3_Cache) CPUs: 160-191
> |   |   `-- 66 (CPU_PM_Active_Power_Domain) CPUs: 160-191
> |   |   |-- 64 (L2_Cache) CPUs: 160-175
> |   |   |   |-- 63 (Floating_Point_Unit [core 20]) CPUs: 160-167
> |   |   |   |   `-- 62 (Integer_Pipeline [core 20]) CPUs: 160-167
> |   |   |   `-- 68 (Floating_Point_Unit [core 21]) CPUs: 168-175
> |   |   |   `-- 67 (Integer_Pipeline [core 21]) CPUs: 168-175
> |   |   `-- 71 (L2_Cache) CPUs: 176-191
> |   |   |-- 70 (Floating_Point_Unit [core 22]) CPUs: 176-183
> |   |   |   `-- 69 (Integer_Pipeline [core 22]) CPUs: 176-183
> |   |   `-- 73 (Floating_Point_Unit 

Re: [hwloc-users] ? Finding cache & pci info on SPARC/Solaris 11.3

2017-06-08 Thread Brice Goglin


Le 08/06/2017 16:58, Samuel Thibault a écrit :
> Hello,
>
> Maureen Chew, on jeu. 08 juin 2017 10:51:56 -0400, wrote:
>> Should finding cache & pci info work?
> AFAWK, there is no user-available way to get cache information on
> Solaris, so it's not implemented in hwloc.

And even if prtpicl reports some information using the PICL API, I don't
think it says how caches are shared between cores.

> Concerning pci, you need libpciaccess to get PCI information.
>

And usually you need root access (I think it looks inside /devices/pci*
where files are root-only).

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] NetLoc subnets Problem

2017-02-22 Thread Brice Goglin
Cyril would know better but he's currently on vacation. I don't think
the required scotch release is already officially publicly available online.

I am not sure what problem you're trying to solve here. This feature
isn't really useful if you only have two nodes. It was rather designed
for process placement on large clusters (by the way, you'll need
something to build a communication pattern of your MPI application as a
communication matrix).

Regarding configure, once the right scotch release is installed, there's
no configure options if I remember correctly, you just need to point
your compiler to scotch include and lib directories. Something like this
may help:
export C_INCLUDE_PATH=/opt/scotch-6/include
export LD_LIBRARY_PATH=/opt/scotch-6/lib
export LIBRARY_PATH=/opt/scotch-6/lib

Brice



Le 22/02/2017 13:38, Михаил Халилов a écrit :
> I tried to configure hwloc with scotch, but I still haven't success
> with that case. I read Chapter 18 in doxygen and chapters about Netloc
> and installation, but not found about anything about Scotch configure.
> So, Scotch installed in /opt/scotch-6/ folder, and I want to install
> hwloc with netloc in /opt/hwloc/ . Which options should I use to give
> ./configure script information about Scotch?
>
> Best regards,
> Mikhail
>
> 2017-02-20 11:50 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>>:
>
> Inside the tarball that you downloaded, there's a
> doc/doxygen-doc/hwloc-a4.pdf with chapter 18 about Netloc with Scotch.
> Beware that this code is still under development.
>
> Brice
>
>
>
>
> Le 19/02/2017 20:20, Михаил Халилов a écrit :
>> Okay, but what configure options for Scotch should I use? I
>> didn't found any information about it in docs and readme
>>
>>
>>
>> 2017-02-19 20:52 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
>> <mailto:brice.gog...@inria.fr>>:
>>
>> The only publicly-installed netloc API is currently specific
>> to the scotch partitioner for process placement. It takes a
>> network topology and a communication pattern between a set of
>> process and it generates a topology-aware placement for these
>> processes.
>> This API only gets installed if you have scotch installed
>> (and tell configure where it is). That's why you don't get
>> any netloc API installed for now.
>>
>> We initially exposed the entire graph that netloc uses
>> internally (it's still true in v0.5 but not anymore in hwloc
>> 2.0) but there wasn't a clear list of what users want to do
>> with it. We didn't want to expose a random API without much
>> user feedback first. There are many ways to expose a graph
>> API, it was too risky. So it's not publicly installed anymore.
>>
>> You can use internal headers such as private/netloc.h for now
>> (you'll see edges, nodes, etc) and we'll make things public
>> once we know what you and others would like to do.
>>
>> Brice
>>
>>
>>
>>
>> Le 19/02/2017 17:29, Михаил Халилов a écrit :
>>> Hi again!
>>>
>>> Can I ask you, how can I use netloc API for my C programs?
>>> I configured hwloc only with --prefix=/opt/hwloc option. So,
>>> there are no netloc header files in /opt/hwloc/include
>>> directory. Also, I didn't understand how to use
>>> netloc_draw.html, because I found it only in extracted
>>> tarball. May be i should configure netloc with some other
>>> options?
>>>
>>> Best regards,
>>> Mikhail Khalilov
>>>
>>>
>>>
>>> ___
>>> hwloc-users mailing list
>>> hwloc-users@lists.open-mpi.org
>>> <mailto:hwloc-users@lists.open-mpi.org>
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
>> ___ hwloc-users
>> mailing list hwloc-users@lists.open-mpi.org
>> <mailto:hwloc-users@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
>>
>>
>> ___
>> 

Re: [hwloc-users] NetLoc subnets Problem

2017-02-20 Thread Brice Goglin
Inside the tarball that you downloaded, there's a
doc/doxygen-doc/hwloc-a4.pdf with chapter 18 about Netloc with Scotch.
Beware that this code is still under development.

Brice



Le 19/02/2017 20:20, Михаил Халилов a écrit :
> Okay, but what configure options for Scotch should I use? I didn't
> found any information about it in docs and readme
>
>
>
> 2017-02-19 20:52 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>>:
>
> The only publicly-installed netloc API is currently specific to
> the scotch partitioner for process placement. It takes a network
> topology and a communication pattern between a set of process and
> it generates a topology-aware placement for these processes.
> This API only gets installed if you have scotch installed (and
> tell configure where it is). That's why you don't get any netloc
> API installed for now.
>
> We initially exposed the entire graph that netloc uses internally
> (it's still true in v0.5 but not anymore in hwloc 2.0) but there
> wasn't a clear list of what users want to do with it. We didn't
> want to expose a random API without much user feedback first.
> There are many ways to expose a graph API, it was too risky. So
> it's not publicly installed anymore.
>
> You can use internal headers such as private/netloc.h for now
> (you'll see edges, nodes, etc) and we'll make things public once
> we know what you and others would like to do.
>
> Brice
>
>
>
>
> Le 19/02/2017 17:29, Михаил Халилов a écrit :
>> Hi again!
>>
>> Can I ask you, how can I use netloc API for my C programs?
>> I configured hwloc only with --prefix=/opt/hwloc option. So,
>> there are no netloc header files in /opt/hwloc/include directory.
>> Also, I didn't understand how to use netloc_draw.html, because I
>> found it only in extracted tarball. May be i should configure
>> netloc with some other options?
>>
>> Best regards,
>> Mikhail Khalilov
>>
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> <mailto:hwloc-users@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
> ___ hwloc-users
> mailing list hwloc-users@lists.open-mpi.org
> <mailto:hwloc-users@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users> 
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] NetLoc subnets Problem

2017-02-19 Thread Brice Goglin
The only publicly-installed netloc API is currently specific to the
scotch partitioner for process placement. It takes a network topology
and a communication pattern between a set of process and it generates a
topology-aware placement for these processes.
This API only gets installed if you have scotch installed (and tell
configure where it is). That's why you don't get any netloc API
installed for now.

We initially exposed the entire graph that netloc uses internally (it's
still true in v0.5 but not anymore in hwloc 2.0) but there wasn't a
clear list of what users want to do with it. We didn't want to expose a
random API without much user feedback first. There are many ways to
expose a graph API, it was too risky. So it's not publicly installed
anymore.

You can use internal headers such as private/netloc.h for now (you'll
see edges, nodes, etc) and we'll make things public once we know what
you and others would like to do.

Brice



Le 19/02/2017 17:29, Михаил Халилов a écrit :
> Hi again!
>
> Can I ask you, how can I use netloc API for my C programs?
> I configured hwloc only with --prefix=/opt/hwloc option. So, there are
> no netloc header files in /opt/hwloc/include directory. Also, I didn't
> understand how to use netloc_draw.html, because I found it only in
> extracted tarball. May be i should configure netloc with some other
> options?
>
> Best regards,
> Mikhail Khalilov
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] NetLoc subnets Problem

2017-02-17 Thread Brice Goglin
Please run "hwloc-gather-topology --io foo" on the head node 'Use the
gather-topology script that comes with the hwloc nightly snapshot).

The output tarball will likely be large, so feel free to send it to me
directly.

My guess is that your very old kernel may miss some files that we need
in the hwloc development snapshot (the I/O discovery changed
significantly in hwloc 2.0).

Brice




Le 17/02/2017 10:26, Михаил Халилов a écrit :
> I ran ibstat on head node it gives information in attach.
>
> 2017-02-17 12:16 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>>:
>
> For some reason, lstopo didn't find any InfiniBand information on
> the head node. I guess running lstopo won't show any "mlx4_0" or
> "ib0" object. Is the InfiniBand service really running on that
> machine?
>
> Brice
>
>
>
>
>
> Le 17/02/2017 10:04, Михаил Халилов a écrit :
>> All files in attach. I run netloc_ib_gather_raw with this
>> parameters netloc_ib_gather_raw /home/halilov/mycluster-data/
>>     --hwloc-dir=/home/halilov/mycluster-data/hwloc/ --verbose --sudo
>>
>> 2017-02-17 11:55 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
>> <mailto:brice.gog...@inria.fr>>:
>>
>> Please copy-paste the exact command line of your
>> "netloc_ib_gather_raw" and all the messages it printed. And
>> also send the output of the hwloc directory it created (it
>> will contain the lstopo XML output of the node where you ran
>> the command).
>>
>> Brice
>>
>>
>>
>> Le 17/02/2017 09:51, Михаил Халилов a écrit :
>>> I installed nightly tarball, but it still isn't working. In
>>> attach info of ibnetdiscover and ibroute. May be it wlii help...
>>> What could be the problem?
>>>
>>> Best regards,
>>> Mikhail Khalilov
>>>
>>> 2017-02-17 9:53 GMT+03:00 Brice Goglin
>>> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>>:
>>>
>>> Hello
>>>
>>> As identicated on the netloc webpages, the netloc
>>> development now occurs
>>> inside the hwloc git tree. netloc v0.5 is obsolete even
>>> if hwloc 2.0
>>> isn't released yet.
>>>
>>> If you want to use a development snapshot, take hwloc
>>> nightly tarballs
>>> from https://ci.inria.fr/hwloc/job/master-0-tarball/
>>> <https://ci.inria.fr/hwloc/job/master-0-tarball/> or
>>> https://www.open-mpi.org/software/hwloc/nightly/master/
>>> <https://www.open-mpi.org/software/hwloc/nightly/master/>
>>>
>>> Regards
>>> Brice
>>>
>>>
>>>
>>>
>>>
>>> Le 16/02/2017 19:15, miharuli...@gmail.com
>>> <mailto:miharuli...@gmail.com> a écrit :
>>> > I downloaded gunzip from openmpi site here:
>>> https://www.open-mpi.org/software/netloc/v0.5/
>>> >
>>> > There are three identical machines in my cluster, but
>>> now third node is broken, and i tried on two machines.
>>> They all connected by InfiniBand switch, and when I try
>>> to use ibnetdiscovery or ibroute, it works perfectly...
>>> >
>>> >
>>> >
>>> > Отправлено с iPad
>>> >> 16 февр. 2017 г., в 18:40, Cyril Bordage
>>> <cyril.bord...@inria.fr <mailto:cyril.bord...@inria.fr>>
>>> написал(а):
>>> >>
>>> >> Hi,
>>> >>
>>> >> What version did you use?
>>> >>
>>> >> I pushed some commits on master on ompi repository.
>>> With this version it
>>> >> seems to work.
>>> >> You have two machines because you tried netloc on
>>> these two?
>>> >>
>>> >>
>>> >> Cyril.
>>> >>
>>> >>> Le 15/02/2017 à 22:44, miharulidze a écrit :
>>&g

Re: [hwloc-users] NetLoc subnets Problem

2017-02-17 Thread Brice Goglin
For some reason, lstopo didn't find any InfiniBand information on the
head node. I guess running lstopo won't show any "mlx4_0" or "ib0"
object. Is the InfiniBand service really running on that machine?

Brice




Le 17/02/2017 10:04, Михаил Халилов a écrit :
> All files in attach. I run netloc_ib_gather_raw with this parameters
> netloc_ib_gather_raw /home/halilov/mycluster-data/
> --hwloc-dir=/home/halilov/mycluster-data/hwloc/ --verbose --sudo
>
> 2017-02-17 11:55 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>>:
>
> Please copy-paste the exact command line of your
> "netloc_ib_gather_raw" and all the messages it printed. And also
> send the output of the hwloc directory it created (it will contain
> the lstopo XML output of the node where you ran the command).
>
> Brice
>
>
>
> Le 17/02/2017 09:51, Михаил Халилов a écrit :
>> I installed nightly tarball, but it still isn't working. In
>> attach info of ibnetdiscover and ibroute. May be it wlii help...
>> What could be the problem?
>>
>> Best regards,
>> Mikhail Khalilov
>>
>> 2017-02-17 9:53 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
>> <mailto:brice.gog...@inria.fr>>:
>>
>> Hello
>>
>> As identicated on the netloc webpages, the netloc development
>> now occurs
>> inside the hwloc git tree. netloc v0.5 is obsolete even if
>> hwloc 2.0
>> isn't released yet.
>>
>> If you want to use a development snapshot, take hwloc nightly
>> tarballs
>> from https://ci.inria.fr/hwloc/job/master-0-tarball/
>> <https://ci.inria.fr/hwloc/job/master-0-tarball/> or
>> https://www.open-mpi.org/software/hwloc/nightly/master/
>> <https://www.open-mpi.org/software/hwloc/nightly/master/>
>>
>> Regards
>> Brice
>>
>>
>>
>>
>>
>> Le 16/02/2017 19:15, miharuli...@gmail.com
>> <mailto:miharuli...@gmail.com> a écrit :
>> > I downloaded gunzip from openmpi site here:
>> https://www.open-mpi.org/software/netloc/v0.5/
>> >
>> > There are three identical machines in my cluster, but now
>> third node is broken, and i tried on two machines. They all
>> connected by InfiniBand switch, and when I try to use
>> ibnetdiscovery or ibroute, it works perfectly...
>> >
>> >
>> >
>> > Отправлено с iPad
>> >> 16 февр. 2017 г., в 18:40, Cyril Bordage
>> <cyril.bord...@inria.fr <mailto:cyril.bord...@inria.fr>>
>> написал(а):
>> >>
>> >> Hi,
>> >>
>> >> What version did you use?
>> >>
>> >> I pushed some commits on master on ompi repository. With
>> this version it
>> >> seems to work.
>> >> You have two machines because you tried netloc on these two?
>> >>
>> >>
>> >> Cyril.
>> >>
>> >>> Le 15/02/2017 à 22:44, miharulidze a écrit :
>> >>> Hi!
>> >>>
>> >>> I'm trying to use NetLoc tool for detecting my cluster
>> topology.
>> >>>
>> >>> I have 2 node cluster with AMD Processors, connected by
>> InfiniBand. Also
>> >>> I installed latest versions of hwloc and netloc tools.
>> >>>
>> >>> I followed the instruction of netloc and when I tried to use
>> >>> netloc_ib_gather_raw as root, i recieved this message
>> >>> root:$ netloc_ib_gather_raw
>> >>> --out-dir=/home/halilov/mycluster-data/result/
>> >>> --hwloc-dir=/home/halilov/mycluster-data/hwloc/ --sudo
>> >>>
>> >>> Found 0 subnets in hwloc directory:
>> >>>
>> >>>
>> >>> There are two files in
>> /home/halilov/mycluster-data/hwloc/ generated by
>> >>> hwloc: head.xml and node01.xml
>> >>>
>> >>> P.S. in attach archieve with .xml files
>> >>>
>> >>>
>> &g

Re: [hwloc-users] NetLoc subnets Problem

2017-02-17 Thread Brice Goglin
Please copy-paste the exact command line of your "netloc_ib_gather_raw"
and all the messages it printed. And also send the output of the hwloc
directory it created (it will contain the lstopo XML output of the node
where you ran the command).

Brice


Le 17/02/2017 09:51, Михаил Халилов a écrit :
> I installed nightly tarball, but it still isn't working. In attach
> info of ibnetdiscover and ibroute. May be it wlii help...
> What could be the problem?
>
> Best regards,
> Mikhail Khalilov
>
> 2017-02-17 9:53 GMT+03:00 Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>>:
>
> Hello
>
> As identicated on the netloc webpages, the netloc development now
> occurs
> inside the hwloc git tree. netloc v0.5 is obsolete even if hwloc 2.0
> isn't released yet.
>
> If you want to use a development snapshot, take hwloc nightly tarballs
> from https://ci.inria.fr/hwloc/job/master-0-tarball/
> <https://ci.inria.fr/hwloc/job/master-0-tarball/> or
> https://www.open-mpi.org/software/hwloc/nightly/master/
> <https://www.open-mpi.org/software/hwloc/nightly/master/>
>
> Regards
> Brice
>
>
>
>
>
> Le 16/02/2017 19:15, miharuli...@gmail.com
> <mailto:miharuli...@gmail.com> a écrit :
> > I downloaded gunzip from openmpi site here:
> https://www.open-mpi.org/software/netloc/v0.5/
> <https://www.open-mpi.org/software/netloc/v0.5/>
> >
> > There are three identical machines in my cluster, but now third
> node is broken, and i tried on two machines. They all connected by
> InfiniBand switch, and when I try to use ibnetdiscovery or
> ibroute, it works perfectly...
> >
> >
> >
> > Отправлено с iPad
> >> 16 февр. 2017 г., в 18:40, Cyril Bordage
> <cyril.bord...@inria.fr <mailto:cyril.bord...@inria.fr>> написал(а):
> >>
> >> Hi,
> >>
> >> What version did you use?
> >>
> >> I pushed some commits on master on ompi repository. With this
> version it
> >> seems to work.
> >> You have two machines because you tried netloc on these two?
> >>
> >>
> >> Cyril.
> >>
> >>> Le 15/02/2017 à 22:44, miharulidze a écrit :
> >>> Hi!
> >>>
> >>> I'm trying to use NetLoc tool for detecting my cluster topology.
> >>>
> >>> I have 2 node cluster with AMD Processors, connected by
> InfiniBand. Also
> >>> I installed latest versions of hwloc and netloc tools.
> >>>
> >>> I followed the instruction of netloc and when I tried to use
> >>> netloc_ib_gather_raw as root, i recieved this message
> >>> root:$ netloc_ib_gather_raw
> >>> --out-dir=/home/halilov/mycluster-data/result/
> >>> --hwloc-dir=/home/halilov/mycluster-data/hwloc/ --sudo
> >>>
> >>> Found 0 subnets in hwloc directory:
> >>>
> >>>
> >>> There are two files in /home/halilov/mycluster-data/hwloc/
> generated by
> >>> hwloc: head.xml and node01.xml
> >>>
> >>> P.S. in attach archieve with .xml files
> >>>
> >>>
> >>> Best regards,
> >>> Mikhail Khalilov
> >>>
> >>>
> >>>
> >>> ___
> >>> hwloc-users mailing list
> >>> hwloc-users@lists.open-mpi.org
> <mailto:hwloc-users@lists.open-mpi.org>
> >>>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
> >> ___
> >> hwloc-users mailing list
> >> hwloc-users@lists.open-mpi.org
> <mailto:hwloc-users@lists.open-mpi.org>
> >>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
> > ___
> > hwloc-users mailing list
> > hwloc-users@lists.open-mpi.org
> <mailto:hwloc-users@lists.open-mpi.org>
> > https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org <mailto:hwloc-users@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
>
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] NetLoc subnets Problem

2017-02-16 Thread Brice Goglin
Hello

As identicated on the netloc webpages, the netloc development now occurs
inside the hwloc git tree. netloc v0.5 is obsolete even if hwloc 2.0
isn't released yet.

If you want to use a development snapshot, take hwloc nightly tarballs
from https://ci.inria.fr/hwloc/job/master-0-tarball/ or
https://www.open-mpi.org/software/hwloc/nightly/master/

Regards
Brice





Le 16/02/2017 19:15, miharuli...@gmail.com a écrit :
> I downloaded gunzip from openmpi site here: 
> https://www.open-mpi.org/software/netloc/v0.5/
>
> There are three identical machines in my cluster, but now third node is 
> broken, and i tried on two machines. They all connected by InfiniBand switch, 
> and when I try to use ibnetdiscovery or ibroute, it works perfectly...
>
>
>
> Отправлено с iPad
>> 16 февр. 2017 г., в 18:40, Cyril Bordage  написал(а):
>>
>> Hi,
>>
>> What version did you use?
>>
>> I pushed some commits on master on ompi repository. With this version it
>> seems to work.
>> You have two machines because you tried netloc on these two?
>>
>>
>> Cyril.
>>
>>> Le 15/02/2017 à 22:44, miharulidze a écrit :
>>> Hi!
>>>
>>> I'm trying to use NetLoc tool for detecting my cluster topology.
>>>
>>> I have 2 node cluster with AMD Processors, connected by InfiniBand. Also
>>> I installed latest versions of hwloc and netloc tools.
>>>
>>> I followed the instruction of netloc and when I tried to use
>>> netloc_ib_gather_raw as root, i recieved this message
>>> root:$ netloc_ib_gather_raw
>>> --out-dir=/home/halilov/mycluster-data/result/
>>> --hwloc-dir=/home/halilov/mycluster-data/hwloc/ --sudo
>>>
>>> Found 0 subnets in hwloc directory:
>>>
>>>
>>> There are two files in /home/halilov/mycluster-data/hwloc/ generated by
>>> hwloc: head.xml and node01.xml
>>>
>>> P.S. in attach archieve with .xml files
>>>
>>>
>>> Best regards,
>>> Mikhail Khalilov
>>>
>>>
>>>
>>> ___
>>> hwloc-users mailing list
>>> hwloc-users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] CPUSET shading using xml output of lstopo

2017-02-03 Thread Brice Goglin
Le 03/02/2017 23:01, James Elliott a écrit :
> On 2/3/17, Brice Goglin <brice.gog...@inria.fr> wrote:
>> What do you mean with shaded? Red or green? Red means unavailable.
>> Requires --whole-system everywhere. Green means that's where the
>> process is bound. But XML doesn't store the information about where
>> the process is bound, so you may only get Green in 2). 
> This is exactly what I am attempting to do (and finding it does not work).
> I would like to have a figure with green shadings so that I have a
> visual representation of where my MPI process lived on the machine.

Try this:

lstopo --whole-system --no-io -f hwloc-${rank}.xml
for pu in $(hwloc-calc --whole-system -H PU --sep " " $(hwloc-bind
--get)); do hwloc-annotate hwloc-${rank}.xml hwloc-${rank}.xml $pu info
lstopoStyle Background=#00ff00 ; done
...


How it works:
* hwloc-bind --get retrieves the current binding as a bitmask
* hwloc-calc converts this bitmask into a space-separated list of PU
indexes (there are other possible outputs if needed, such as cores, or
the largest object included in the binding, etc)
* the for loop iterates on these objects and hwloc-annotate adds an
attribute lstopoStyle Background=#00ff00 to each of them
* lstopo will use this attribute to change the background color of these
PU boxes in the graphical output

Make sure you have hwloc >= 1.11.1 for this to work.

Brice



>
> I currently have a function (in C) that I use in my codes that
> inspects affinities, but when I discuss app performance with others, I
> would like to be able to show (graphically) exactly how their app uses
> the resources.  I work mostly with hybrid MPI/OpenMP codes, developed
> by smart scientists who are not familiar with things like affinity.
>
>>> To test without MPI, you would just need to set a processes affinity
>>> and then use its PID instead.
>>>
>>> What I see, is that the XML generated in (1) is identical for all MPI
>>> processes, even though they have different PIDs and different CPUSETS.
>> Are you talking about different MPI runs, or different MPI ranks within
>> the same run?
>>
>> My feeling is that you think you should be seeing different cpusets for
>> each process, but they actually have the same cpuset but different
>> bindings. Cores outside the cpuset are red when --whole-system, or
>> totally ignored otherwise.
>>
>> In (2), you don't have --whole-system, no red cores. But you have --pid,
>> so you get one green core per process, it's its binding. That's why you
>> get different images for each process.
>> in (3), you inherit the missing --whole-system from (1) through XML, no
>> red cores either. But XML doesn't save the process binding, no green
>> cores either. Same image for each process.
>>
>>
>> Do you care about process binding (what mpirun applies to each rank?) or
>> about cpusets (what the batch scheduler applies to the entire job before
>> mpirun?)
>>
>> If cpuset, just add --whole-system everywhere, it should be enough.
>> If binding, there's no direct way with lstopo (but we have a way to save
>> custom colors for individual objects in the XML).
>>
>> Brice
>>
>>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] CPUSET shading using xml output of lstopo

2017-02-03 Thread Brice Goglin
Le 03/02/2017 21:57, James Elliott a écrit :
> Brice,
>  
> Thanks for you comments.  I have worked with this some, but this is
> not working.
>
> My goal is to generate images of the cpusets inuse when I run a
> parallel code using mpirun, aprun, srun, etc...  The compute nodes
> lack the mojo necessary to generate graphical formats, so I can only
> extract XML on the nodes.
>
> I am testing this locally on a 2 NUMA, dual socket workstation with 14
> cores per socket (so, 28 total cores).  I can use OpenMPI to easily
> spawn/bind processes.
>
> E.g.,
> mpirun --map-by ppr:2:NUMA:pe=7 ./hwloc_plot_mpi.sh
>
>
> hwloc_plot_mpi.sh is very simple:
>
> #!/bin/bash
>
> pid="$$"
>
> rank=${OMPI_COMM_WORLD_RANK}
>
> lstopo --pid ${pid} --no-io -f hwloc-${rank}.xml
>
> lstopo --pid ${pid} --no-io --append-legend "Rank: ${rank}" -f
> hwloc-${rank}-orig.png
>
> lstopo --append-legend "Rank: ${rank}" --whole-system --input
> hwloc-${rank}.xml -f hwloc-${rank}.png
>
>
>
> To test things,
> 1) write the XML
> 2) use the same command to write a PNG
> 3) use the generated XML to generate the PNG

Hello

You're missing --whole-system in 1) and 2)

Also --pid isn't very useful because you're basically looking at the
current process, and that's the default. The only difference is that the
process binding is reported in green when using --pid. Does it matter?
See below.

>
> (2) and (3) should produce the same image if I am doing things correctly.
>
> The image for (2) is unique for each process, showing 7 *different*
> cores shaded in each figure (4 images are generated since I spawn 4
> processes)
> The images from (3) are all identical (no shading)

What do you mean with shaded? Red or green?

Red means unavailable. Requires --whole-system everywhere.

Green means that's where the process is bound. But XML doesn't store the
information about where the process is bound, so you may only get Green
in 2).

>
> To test without MPI, you would just need to set a processes affinity
> and then use its PID instead.
>
> What I see, is that the XML generated in (1) is identical for all MPI
> processes, even though they have different PIDs and different CPUSETS.

Are you talking about different MPI runs, or different MPI ranks within
the same run?

My feeling is that you think you should be seeing different cpusets for
each process, but they actually have the same cpuset but different
bindings. Cores outside the cpuset are red when --whole-system, or
totally ignored otherwise.

In (2), you don't have --whole-system, no red cores. But you have --pid,
so you get one green core per process, it's its binding. That's why you
get different images for each process.
in (3), you inherit the missing --whole-system from (1) through XML, no
red cores either. But XML doesn't save the process binding, no green
cores either. Same image for each process.


Do you care about process binding (what mpirun applies to each rank?) or
about cpusets (what the batch scheduler applies to the entire job before
mpirun?)

If cpuset, just add --whole-system everywhere, it should be enough.
If binding, there's no direct way with lstopo (but we have a way to save
custom colors for individual objects in the XML).

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] CPUSET shading using xml output of lstopo

2017-01-31 Thread Brice Goglin
shade/highlight is included in the "cpuset" and "allowed_cpuset" fields
inside the XML (even when not using --pid).

By default, only what's "available" is displayed. If you want
"disallowed" things to appear (in different colors), add --whole-system
when drawing (in the second command-line).

Brice



Le 01/02/2017 06:56, James a écrit :
> Thanks Brice,
>
> I believe I am rebuilding it as you say, but I can retry tomorrow at
> my desk.
> I looked in the XML and can see the taskset data, but since I cannot
> do --pid ###, it seems to not shade/highlight the tasksets.
>
> I'll drop the args that are redundant and try the exact form you list.
>
> James
>
> On 1/31/2017 10:52 PM, Brice Goglin wrote:
>> Le 01/02/2017 00:19, James Elliott a écrit :
>>> Hi,
>>>
>>> I seem to be stuck. What I would like to do, is us lstopo to generate
>>> files that I can plot on another system (the nodes lack the necessary
>>> libraries for graphical output).
>>>
>>> That is, I would like to see something like
>>> lstopo --only core --pid ${pid} --taskset --no-io --no-bridges
>>> --append-legend "PID: ${pid}" -f hwloc-${pid}.png
>>>
>>> But I need to output to XML instead, and plot on another machine, e.g.
>>>
>>> lstopo --only core --pid ${pid} --taskset --no-io --no-bridges
>>> --append-legend "PID: ${pid}" -f hwloc-${pid}.png
>>> ...
>>> Then on another machine,
>>> lstopo --input hwloc-.xml output.png
>>>
>>> Where, the --pid shading of cpusets is produced in the output.png.
>>> This does not seem to work. I am fairly new to lstopo, is it possible
>>> to achieve this functionality? (I would also like to preserve the
>>> append-legend  stuff, but I could work out a way to do that on the
>>> other host.)
>> Hello
>>
>> My guess is that you would need to export to XML like this:
>> lstopo --pid ${pid} --no-io -f foo.xml
>>
>> and reload/draw on the other host like this:
>> lstopo --input foo.xml --only-core --taskset --append-legend "PID:
>> ${pid}" -f output.png
>>
>> Random comments:
>> * --no-bridges in implied by --no-io
>> * --only and --taskset only apply to the textual output, while you seem
>> to want graphical output as png
>> * --append-legend only applies to the graphical output
>>
>> Brice
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] CPUSET shading using xml output of lstopo

2017-01-31 Thread Brice Goglin
Le 01/02/2017 00:19, James Elliott a écrit :
> Hi,
>
> I seem to be stuck. What I would like to do, is us lstopo to generate
> files that I can plot on another system (the nodes lack the necessary
> libraries for graphical output).
>
> That is, I would like to see something like
> lstopo --only core --pid ${pid} --taskset --no-io --no-bridges
> --append-legend "PID: ${pid}" -f hwloc-${pid}.png
>
> But I need to output to XML instead, and plot on another machine, e.g.
>
> lstopo --only core --pid ${pid} --taskset --no-io --no-bridges
> --append-legend "PID: ${pid}" -f hwloc-${pid}.png
> ...
> Then on another machine,
> lstopo --input hwloc-.xml output.png
>
> Where, the --pid shading of cpusets is produced in the output.png.
> This does not seem to work. I am fairly new to lstopo, is it possible
> to achieve this functionality? (I would also like to preserve the
> append-legend  stuff, but I could work out a way to do that on the
> other host.)

Hello

My guess is that you would need to export to XML like this:
lstopo --pid ${pid} --no-io -f foo.xml

and reload/draw on the other host like this:
lstopo --input foo.xml --only-core --taskset --append-legend "PID:
${pid}" -f output.png

Random comments:
* --no-bridges in implied by --no-io
* --only and --taskset only apply to the textual output, while you seem
to want graphical output as png
* --append-legend only applies to the graphical output

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] Building hwloc on Cray with /opt/cray/craype/2.5.4/bin/cc

2017-01-05 Thread Brice Goglin
I think you are mixing hwloc libraries here. CPUType was removed a while
ago (except on Solaris). And CPUVendor was widely added later.
My feeling is that your lstopo uses a recent libhwloc while your test
program uses an old one.
Brice



Le 05/01/2017 15:16, Xavier LACOSTE a écrit :
> Hello Samuel,
>
> With my test I get : 
> CPU_Vendor : (null) 
> Socket : CPUModel="Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz" | CPUType=x86_64
>
> Meanwhile, the CPUVendor Field is in the generated xml.
>
> XL.
>
> -Message d'origine-
> De : Samuel Thibault [mailto:samuel.thiba...@inria.fr] 
> Envoyé : jeudi 5 janvier 2017 13:12
> À : Hardware locality user list
> Objet : Re: [hwloc-users] Building hwloc on Cray with 
> /opt/cray/craype/2.5.4/bin/cc
>
> Xavier LACOSTE, on Thu 05 Jan 2017 11:31:23 +, wrote:
>> Hwloc builds correctly with gcc compiler but the CPUVendor field is 
>> not available using this code :
> The code looks right and works here. What output do you get? Could you post 
> the output of
>
> lstopo test.xml
>
> Samuel
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Building hwloc on Cray with /opt/cray/craype/2.5.4/bin/cc

2017-01-05 Thread Brice Goglin
Does this even build with the cray compiler?

int main(int argc, char *argv[])
{
  return __builtin_ffsl((unsigned long) argc);
}




Le 05/01/2017 14:56, Xavier LACOSTE a écrit :
>
> Indeed,
>
>  
>
> If I had a #undef __GNUC__ in misc.h the compilation finished (I still
> have the link complaining about recompiling with –fPIE and linking
> with –pie, but I should be able to handle that)
>
> I tried all available cray cc (2.2.1 and 2.5.6) and they behave the same.
>
> I’ll see how to report bug to Cray and may ask for a new compiler
> installation.
>
>  
>
> XL.
>
>  
>
> *De :*Brice Goglin [mailto:brice.gog...@inria.fr]
> *Envoyé :* jeudi 5 janvier 2017 14:39
> *À :* Xavier LACOSTE
> *Cc :* Hardware locality user list
> *Objet :* Re: [hwloc-users] Building hwloc on Cray with
> /opt/cray/craype/2.5.4/bin/cc
>
>  
>
> Ah ok now I remember we've seen the same issue 4 years ago.
> https://mail-archive.com/hwloc-users@lists.open-mpi.org/msg00816.html
> Basically the Cray compiler claimed to be GCC compatible it was not.
> But that users explicitly requested the cray compiler to behave as GNU
> with "-h gnu". Did you pass any option to the compiler?
>
> We could workaround the issue by not using __builtin_ffsl() when we
> detect the cray compiler, but that would be overkill since earlier
> versions worked fine. Do you have ways to report bugs to Cray? Or
> install a different cray compiler version?
>
> Brice
>
>
> Le 05/01/2017 14:25, Xavier LACOSTE a écrit :
>
> It seems that the __GNU__ is defined so I don’t get into the
> HWLOC_HAVE_FFSL section.
>
>  
>
> Maybe because I already configured once with gcc before ? Do I
> have to do anything more than make clean an reconfigure to change
> compiler ?
>
>  
>
> *De :*Brice Goglin [mailto:brice.gog...@inria.fr]
> *Envoyé :* jeudi 5 janvier 2017 14:18
> *À :* Xavier LACOSTE
> *Cc :* Hardware locality user list
> *Objet :* Re: [hwloc-users] Building hwloc on Cray with
> /opt/cray/craype/2.5.4/bin/cc
>
>  
>
> configure seems to have detected ffsl() properly, but it looks
> like our ffsl() redefinition gets enabled anyway, and conflicts
> with the system-wide one.
>
> Does it help if you comment out line #66 of include/private/misc.h ?
> extern int ffsl(long) __hwloc_attribute_const;
>
> Brice
>
>
>
> Le 05/01/2017 13:52, Xavier LACOSTE a écrit :
>
> Hello Brice,
>
>  
>
> I attached the files.
>
>  
>
> I could build hwloc and link with it using the default (gcc)
> compiler.
>
> But If I use gcc to build the library I can link with
> gcc/icc/pgcc but not with cc from cray.
>
>  
>
> Thanks,
>
>  
>
> XL.
>
>  
>
>  
>
> *De :*Brice Goglin [mailto:brice.gog...@inria.fr]
> *Envoyé :* jeudi 5 janvier 2017 12:50
> *À :* Xavier LACOSTE
> *Cc :* Hardware locality user list
> *Objet :* Re: [hwloc-users] Building hwloc on Cray with
> /opt/cray/craype/2.5.4/bin/cc
>
>  
>
> Hello Xavier
> Can you send the /usr/include/string.h from that cray machine,
> your config.log and include/private/autogen/config.h from the
> hwloc build directory?
> Do you get the same error if building from the default
> compiler instead of /opt/cray/craype/2.5.4/bin/cc?
> thanks
> Brice
>
>
>
> Le 05/01/2017 12:31, Xavier LACOSTE a écrit :
>
>  
>
> Hello,
>
>  
>
> I’m trying to build hwloc on a cray machine with cray
> compiler :/opt/cray/craype/2.5.4/bin/cc
>
>  
>
> I get the following error :
>
> $> CC=cc ./configure --prefix=$PWD-cc-install
>
> $> make
>
> Making all in src
>
> make[1]: Entering directory
> `/home/j0306818/xavier/hwloc-1.11.5/src'
>
>   CC   bitmap.lo
>
> CC-147 craycc: ERROR
>
>   Declaration is incompatible with "int ffsl(long)"
> (declared at line 526 of
>
>   "/usr/include/string.h").
>
>  
>
>  
>
> Total errors detected in bitmap.c: 1
>
> make[1]: *** [bitmap.lo] Error 1
>
> make[1]: Leaving directory
>   

Re: [hwloc-users] Reporting an operating system warning

2017-01-03 Thread Brice Goglin
Thanks

Surprisingly, I don't see any L1i in the XML output either. Did you get
warnings during this run "HWLOC_COMPONENTS=x86 lstopo foo.xml" ?

Indeed, you (very likely) don't care about that warning in the AMD SDK.
Pass HWLOC_HIDE_ERRORS=1 in the environment to silence it.

Brice


Le 03/01/2017 07:59, Johannes Goller a écrit :
> Hi Brice,
>
> thank you very much for looking into this!
>
> I am attaching the generated foo.xml.
>
> I actually came across this error message when trying to play with
> OpenCL, using the AMD SDK API. My main interest is in getting that to
> work on the GPU, and that might still work on my current kernel (4.8),
> even if I get warnings like the one reported.
>
>
> johannes.
>
> 2017-01-03 15:15 GMT+09:00 Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>>:
>
> Hello Johannes
>
> I think there are two bugs here.
>
> First one is that each "dual-core compute unit" is reported as a
> single core with two hardware threads. That's a kernel bug that
> appeared in 4.6. There's a fix at
> https://lkml.org/lkml/2016/11/29/852
> <https://lkml.org/lkml/2016/11/29/852> but I don't think it has
> been applied yet.
>
> The second bug is a conflict between dual-core compute unit
> sharing and L1i. I am not sure which one is actually buggy. Can
> you run "HWLOC_COMPONENTS=x86 lstopo foo.xml" and send the
> generated foo.xml? (this is our raw detection that works around
> the kernel detection).
>
> Trying a Linux kernel <= 4.5 may help in the meantime.
>
> thanks
> Brice
>
>
>
>
> Le 03/01/2017 05:29, Johannes Goller a écrit :
>> As requested on
>> https://www.open-mpi.org/projects/hwloc/doc/v1.10.1/a00028.php
>> <https://www.open-mpi.org/projects/hwloc/doc/v1.10.1/a00028.php>
>> ("What should I do when hwloc reports 'operating system'
>> warnings?"), I am reporting the warning/error I received as follows
>>
>> 
>> 
>> * hwloc 1.11.0 has encountered what looks like an error from the
>> operating system.
>> *
>> * L1i (cpuset 0x0003) intersects with Core (P#0 cpuset
>> 0x0081) without inclusion!
>> * Error occurred in topology.c line 983
>> *
>> * The following FAQ entry in the hwloc documentation may help:
>> *   What should I do when hwloc reports "operating system" warnings?
>> * Otherwise please report this error message to the hwloc user's
>> mailing list,
>> * along with the output+tarball generated by the
>> hwloc-gather-topology script.
>> 
>> 
>>
>> Please find the tarball attached.
>>
>>
>>
>> regards,
>>
>> Johannes Goller.
>>
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> <mailto:hwloc-users@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users>
> ___ hwloc-users
> mailing list hwloc-users@lists.open-mpi.org
> <mailto:hwloc-users@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users> 
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] Reporting an operating system warning

2017-01-02 Thread Brice Goglin
Hello Johannes

I think there are two bugs here.

First one is that each "dual-core compute unit" is reported as a single
core with two hardware threads. That's a kernel bug that appeared in
4.6. There's a fix at https://lkml.org/lkml/2016/11/29/852 but I don't
think it has been applied yet.

The second bug is a conflict between dual-core compute unit sharing and
L1i. I am not sure which one is actually buggy. Can you run
"HWLOC_COMPONENTS=x86 lstopo foo.xml" and send the generated foo.xml?
(this is our raw detection that works around the kernel detection).

Trying a Linux kernel <= 4.5 may help in the meantime.

thanks
Brice



Le 03/01/2017 05:29, Johannes Goller a écrit :
> As requested on
> https://www.open-mpi.org/projects/hwloc/doc/v1.10.1/a00028.php ("What
> should I do when hwloc reports 'operating system' warnings?"), I am
> reporting the warning/error I received as follows
>
> 
> * hwloc 1.11.0 has encountered what looks like an error from the
> operating system.
> *
> * L1i (cpuset 0x0003) intersects with Core (P#0 cpuset 0x0081)
> without inclusion!
> * Error occurred in topology.c line 983
> *
> * The following FAQ entry in the hwloc documentation may help:
> *   What should I do when hwloc reports "operating system" warnings?
> * Otherwise please report this error message to the hwloc user's
> mailing list,
> * along with the output+tarball generated by the hwloc-gather-topology
> script.
> 
>
> Please find the tarball attached.
>
>
>
> regards,
>
> Johannes Goller.
>
>
>
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc 1.11 in Android

2016-10-25 Thread Brice Goglin
Thanks. I am applying the patch to git master and v1.11 branches.

I tested my static build of lstopo-no-graphics on Android 6 on aarch64
[1], it still works without disabling openat. It's not running as an
APK, but it's running inside a shell that was launched by an APK
(sshdroid, a ssh server shell). Not sure why this would be different?

Brice

[1]
http://people.bordeaux.inria.fr/goglin/tmp/lstopo-aarch64/lstopo-no-graphics



Le 25/10/2016 09:11, Marc Marí a écrit :
> Hello
>
> The patch is attached (if attachments are a problem in this list, I'll
> send it in plain).
>
> The only API functions I'm using are from this code:
>
> https://github.com/pocl/pocl/blob/master/lib/CL/devices/topology/pocl_topology.c
>
> I'm using it in Android 6 for ARM64, and I'm linking dynamically and
> packing in an APK.
>
> I suspect that, if you generate an executable and you run directly on
> the Android shell, you won't have any premission issues. The problem
> comes when you run it inside an APK, which is located in a restricted
> sandbox. I have not tested that assumption.
>
> Marc
>
>> On Mon, 24 Oct 2016 17:51:04 +0200
>> Brice Goglin <brice.gog...@inria.fr> wrote:
>>
>> Hello
>> I am interested in seeing the patch, at least. If it isn't too
>> intrusive, we could easily apply it.
>> I am surprised to hear that openat fails on Android. I don't remember
>> having to disable it last time I tried (on Android 4.0 or 4.1 iirc).
>> But I was building natively (basically just cross-compiling for armv7
>> and static linking), maybe you're doing something different that
>> comes with different constraints?
>> I'd like to have lstopo available in the play store too (I only had a
>> textual version last time I tried :))
>> Brice
>>
>>
>>
>> Le 24/10/2016 17:42, Marc Marí a écrit :
>>> I was porting a library to Android, and this library uses hwloc
>>> v1.11 in the background. I managed to make it work.
>>>
>>> Are you interested in adding Android support for hwloc version 1.11?
>>> Should I send a patch?
>>>
>>> The only change necessary is to disable "openat" (HAVE_OPENAT) when
>>> compiling for Android. Android does have support for it, but, due to
>>> the permission system, it cannot be used, so hwloc fails.
>>>
>>> Marc
>>>
>>> WARNING / LEGAL TEXT: This message is intended only for the use of
>>> the individual or entity to which it is addressed and may contain
>>> information which is privileged, confidential, proprietary, or
>>> exempt from disclosure under applicable law. If you are not the
>>> intended recipient or the person responsible for delivering the
>>> message to the intended recipient, you are strictly prohibited from
>>> disclosing, distributing, copying, or in any way using this
>>> message. If you have received this communication in error, please
>>> notify the sender and destroy and delete any copies you may have
>>> received.
>>>
>>> http://www.bsc.es/disclaimer
>>> ___
>>> hwloc-users mailing list
>>> hwloc-users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users  
>> ___
>> hwloc-users mailing list
>> hwloc-users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users
>
>
> WARNING / LEGAL TEXT: This message is intended only for the use of the
> individual or entity to which it is addressed and may contain
> information which is privileged, confidential, proprietary, or exempt
> from disclosure under applicable law. If you are not the intended
> recipient or the person responsible for delivering the message to the
> intended recipient, you are strictly prohibited from disclosing,
> distributing, copying, or in any way using this message. If you have
> received this communication in error, please notify the sender and
> destroy and delete any copies you may have received.
>
> http://www.bsc.es/disclaimer
>

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

Re: [hwloc-users] hwloc 1.11 in Android

2016-10-24 Thread Brice Goglin
Hello
I am interested in seeing the patch, at least. If it isn't too
intrusive, we could easily apply it.
I am surprised to hear that openat fails on Android. I don't remember
having to disable it last time I tried (on Android 4.0 or 4.1 iirc). But
I was building natively (basically just cross-compiling for armv7 and
static linking), maybe you're doing something different that comes with
different constraints?
I'd like to have lstopo available in the play store too (I only had a
textual version last time I tried :))
Brice



Le 24/10/2016 17:42, Marc Marí a écrit :
> I was porting a library to Android, and this library uses hwloc v1.11
> in the background. I managed to make it work.
>
> Are you interested in adding Android support for hwloc version 1.11?
> Should I send a patch?
>
> The only change necessary is to disable "openat" (HAVE_OPENAT) when
> compiling for Android. Android does have support for it, but, due to
> the permission system, it cannot be used, so hwloc fails.
>
> Marc
>
> WARNING / LEGAL TEXT: This message is intended only for the use of the
> individual or entity to which it is addressed and may contain
> information which is privileged, confidential, proprietary, or exempt
> from disclosure under applicable law. If you are not the intended
> recipient or the person responsible for delivering the message to the
> intended recipient, you are strictly prohibited from disclosing,
> distributing, copying, or in any way using this message. If you have
> received this communication in error, please notify the sender and
> destroy and delete any copies you may have received.
>
> http://www.bsc.es/disclaimer
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] memory binding on Knights Landing

2016-10-04 Thread Brice Goglin


Le 12/09/2016 04:20, Brice Goglin a écrit :
> So what's really slow is reading sysfs and/or inserting all hwloc
> objects in the tree. I need to do some profiling. And I am moving the
> item "parallelize the discovery" higher in the TODO list :) Brice 

Hello

I ran more benchmarks. What's really slow is the reading of all sysfs
files. About 90% of the topology building time is spent there on KNL.
We're reading more than 7000 files (most of them are 6 files for each
hardware thread and 6 files for each cache).
Reading from sysfs is significantly slower than reading normal files
that are cached (not surprising since the kernel doesn't cache sysfs
file contents).
And reading on KNL is about 30 times slower than on my laptop (70us for
each sysfs file). Don't know why.

And if you have one process doing this on each core simultaneously,
things become up to 30x slower.

Looks like XML is really the way to go on these platforms. One thing
that XML currently misses is cgroup support. You need to export the XML
inside the same cgroup or the topology will be wrong. I am adding an
option to read the current cgroup restrictions from the OS and apply it
to a XML imported topology (must be created outside of all cgroups).

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] memory binding on Knights Landing

2016-09-12 Thread Brice Goglin
Le 08/09/2016 19:17, Brice Goglin a écrit :
>
>> By the way, is it expected that binding will be slow on it?  hwloc-bind
>> is ~10 times slower (~1s) than on two-socket sandybridge, and ~3 times
>> slower than on a 128-core, 16-socket system.
> Binding itself shouldn't be slower. But hwloc's topology discovery
> (which is performed by hwloc-bind before actual binding) is slower on
> KNL than on "normal" nodes. The overhead is basically linear with the
> number of hyperthreads, and KNL sequential perf is lower than your other
> nodes.
>
> The easy fix is to export the topology to XML with lstopo foo.xml and
> then tell all hwloc users to load from XML:
> export HWLOC_XMLFILE=foo.xml
> export HWLOC_THISSYSTEM=1
> https://www.open-mpi.org/projects/hwloc/doc/v1.11.4/a00030.php#faq_xml
>
> For hwloc 2.0, I am trying to make sure we don't perform useless
> discovery steps. hwloc-bind (and many applications) don't require all
> topology details. v1.x gathers everything and filters things out later.
> For 2.0, the plan is rather to directly just gather what we need. What
> you can try for fun is:
> export HWLOC_COMPONENTS=-x86 (without the above XML env vars)
> It disables the x86-specific discovery which is useless for most cases
> on Linux.
>

Interesting, this last idea doesn't help. XML is much faster (0.14s),
but normal discovery is still 1s without the x86-specific code.

So what's really slow is reading sysfs and/or inserting all hwloc
objects in the tree. I need to do some profiling. And I am moving the
item "parallelize the discovery" higher in the TODO list :)

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] memory binding on Knights Landing

2016-09-09 Thread Brice Goglin
Le 09/09/2016 12:49, Dave Love a écrit :
>
>> Intel people are carrefully
>> working with RedHat so that hwloc is properly packaged for RHEL. I can
>> report bugs if needed.
> I can't see a recent hwloc for RHEL (e.g. in RHEL7 beta), but don't get
> me started on RHEL and HPC...
>

I am not sure where that hwloc for RHEL on KNL is available from. It
might be in Intel's "XPPSL" software suite.

Brice

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] memory binding on Knights Landing

2016-09-08 Thread Brice Goglin
Hello
It's not a feature. This should work fine.
Random guess: do you have NUMA headers on your build machine ? (package
libnuma-dev or numactl-devel)
(hwloc-info --support also report whether membinding is supported or not)
Brice



Le 08/09/2016 16:34, Dave Love a écrit :
> I'm somewhat confused by binding on Knights Landing -- which is probably
> a feature.
>
> I'm looking at a KNL box configured as "Cluster Mode: SNC4 Memory Mode:
> Cache" with hwloc 1.11.4; I've read the KNL hwloc FAQ entries.  I ran
> openmpi and it reported failure to bind memory (but binding to cores was
> OK).  So I tried hwloc-bind --membind and that seems to fail with no
> matter what I do, reporting
>
>   hwloc_set_membind 0x0002 (policy 2 flags 0) failed (errno 38 Function 
> not implemented)
>
> Is that expected, and is there a recommendation on how to do binding in
> that configuration with things that use hwloc?  I'm particularly
> interested in OMPI, but I guess this is a better place to ask.  Thanks.
> ___
> hwloc-users mailing list
> hwloc-users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users

___
hwloc-users mailing list
hwloc-users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/hwloc-users


Re: [hwloc-users] Topology Error

2016-05-09 Thread Brice Goglin
Le 09/05/2016 23:58, Mehmet Belgin a écrit :
> Greetings!
>
> We've been receiving this error for a while on our 64-core Interlagos
> AMD machines:
>
> 
>
> * hwloc has encountered what looks like an error from the operating
> system.
> *
> * Socket (P#2 cpuset 0x,0x0) intersects with NUMANode (P#3
> cpuset 0xff00,0xff00) without inclusion!
> * Error occurred in topology.c line 940
> *
> * Please report this error message to the hwloc user's mailing list,
> * along with the output+tarball generated by the hwloc-gather-topology
> script.
> 
>
>
> I've found some information in the hwloc list archives mentioning this
> is due to buggy AMD platform and the impact should be limited to hwloc
> missing L3 cache info (thanks Brice). If that's the case and processor
> representation is correct then I am sure we can live with this, but I
> still wanted to check with the list to confirm that (1) this is really
> harmless and (2) are there any known solutions other than upgrading
> BIOS/kernel?

Hello

The L3 bug only applies to 12-core Opteron 62xx/63xx, while you have
16-core Opterons. Your L3 locality is correct, but your NUMA locality is
wrong:
$ cat sys/devices/system/node/node*/cpumap 
,00ff
ff00,ff00
00ff,
,
You should have something like this instead:
,
,
,
,

This bug is not harmless since memory buffers have a good chance of
being physically allocated far away from your cores.

This is more likely a BIOS bug. Try upgrading.

Regards
Brice



Re: [hwloc-users] hwloc_alloc_membind with HWLOC_MEMBIND_BYNODESET

2016-05-09 Thread Brice Goglin
Hello Hugo,

Can you send your code and a description of the machine so that I try to
reproduce ?

By the way, BYNODESET is also available in 1.11.3.

Brice



Le 09/05/2016 16:18, Hugo Brunie a écrit :
> Hello,
>
> When I try to use hwloc_alloc_membind with HWLOC_MEMBIND_BYNODESET
> I obtain NULL as a pointer and the error message is : Invalid Argument.
>
> I try without, it works. It works also with HWLOC_MEMBIND_STRICMT
> and/or HWLOC_MEMBIND_THREAD.
> My hwloc version is :
> ~/usr/bin/hwloc-bind --version
> hwloc-bind 2.0.0a1-git
>
> Best regards,
>
> Hugo BRUNIE
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2016/05/1269.php



Re: [hwloc-users] HWLOC_get_membind: problem in getting right(specific) NODESET where data is allocated

2016-04-25 Thread Brice Goglin
**Please replace err with errno in that line:
printf("Error Occured, and error no:= %d \n", err);

You may need to #include  in the header.

Brice

*

*
Le 25/04/2016 00:27, Rezaul Karim Raju a écrit :
> Please find the attached, system Layout. 
> *uname -a*
> Linux crill-010 3.11.10-21-desktop #1 SMP PREEMPT Mon Jul 21 15:28:46
> UTC 2014 (9a9565d) x86_64 x86_64 x86_64 GNU/Linux
>
> and below is the code snippet where I am getting error: 
>
> /* Find Location of a: 3rd QUARTER */
> *err = hwloc_get_area_membind_nodeset(topology, array+ size/2, size/4,
> nodeset_c, , HWLOC_MEMBIND_THREAD ); *
> *if (err < 0) {*
> *printf("Error Occured, and error no:= %d \n", err);*
> fprintf(stderr, "failed to retrieve the buffer binding and policy\n");
> hwloc_topology_destroy(topology);
> hwloc_bitmap_free(nodeset_c);
> //return EXIT_FAILURE;
> }
>
> *Please ignore the segfault, here it gives the error no: = -1*
> *
> *
> *My question is allocate an array to a NUMA node and bind it over
> nodes partially is OK with hwloc API..?*
> *
> *
> Thank you again.
> - Raju
>
>
>
> On Sun, Apr 24, 2016 at 4:58 PM, Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>> wrote:
>
> Please find out which line is actually causing the segfault.
> Run your program under gdb. Once it crashes, type "bt full" and
> report the output here.
>
> By the way, what kind of machine are you using? (lstopo + uname -a)
>
> Brice
>
>
>
>
> Le 24/04/2016 23:46, Rezaul Karim Raju a écrit :
>> Hi Brice,
>>
>> Thank you very much for your prompt care. 
>>
>> I am retrieving as below:
>>
>> nodeset_c = hwloc_bitmap_alloc();
>>
>> */* Find Location of a: 3rd QUARTER */*
>> err = *hwloc_get_area_membind_nodeset(*topology, *array+ size/2,
>> size/4,* nodeset_c, , HWLOC_MEMBIND_THREAD ); 
>>
>> /* print the corresponding NUMA nodes */
>> hwloc_bitmap_asprintf(, nodeset_c);
>> printf("Address:= %p  Variable:=  bound
>> to*nodeset %s with contains:*\n", (array+size/2), s);
>> free(s);
>> hwloc_bitmap_foreach_begin(hw_i, nodeset_c) {
>> *obj_c = hwloc_get_numanode_obj_by_os_index(topology, hw_i);*
>> *printf("[3rd Q]  node #%u (OS index %u) with %lld bytes of
>> memory\n", obj_c->logical_index, hw_i, (unsigned long long)
>> obj_c->memory.local_memory)*;
>> } hwloc_bitmap_foreach_end();
>> hwloc_bitmap_free(nodeset_c);
>>
>> *It prints as below:*
>> *
>> *
>> *error no:= -1 and segmentation fault 
>> *
>> *my array size is =  262144 {data type long} and each Quarter =
>> size/4 =65536*
>> Address of array:= 0x7f350e515000, tmp:= 0x7f34fe515000,
>> tst_array:= 0x7f34ee515000
>> Address of array:= 0x7f350e515000, array+size/4:= 0x7f352e515000,
>> array+size/2:= 0x7f354e515000, array+3*size/4:= 0x7f356e515000
>>
>> Address:= 0x7f350e515000  Variable:= 
>> bound to nodeset 0x0001 with contains:
>>  [1st Q]  node #0 (OS index 0) with 8387047424 bytes of memory
>> Address:= 0x7f352e515000  Variable:= 
>> bound to nodeset 0x0004 with contains:
>> [2nd Q]  node #2 (OS index 2) with 8471621632 bytes of memory
>>
>> in case of [3rd Q]
>> Error Occured, and error no:= -1 and segmentation fault happened.
>>
>> Thanks.!
>>
>>
>> On Sun, Apr 24, 2016 at 4:08 PM, Brice Goglin
>> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>>
>> Hello,
>> What do you mean with " it can not bind the specified memory
>> section (addr, len) to the desired NUMA node"?
>> Did it fail? If so, what does errno contain?
>> If it didn't fail, what did it do instead?
>> thanks
>> Brice
>>
>>
>>
>>
>> Le 24/04/2016 23:02, Rezaul Karim Raju a écrit :
>>> Hi ...
>>>
>>> I was trying to bind each quarter of an array to 4 different
>>> NUMA nodes, and doing as below: 
>>>
>>> *//ALLOCATION *
>>> *obj_a = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 0);*
>>>
>>> *array =* hwloc_alloc_membind_nodeset( topology, size,
>>> obj_a->nodeset, HWLOC_MEMBIND_BIND, 1);
>>> *tmp *= hwloc_alloc_membind_nodeset( topology, size,
>>>

Re: [hwloc-users] HWLOC_get_membind: problem in getting right(specific) NODESET where data is allocated

2016-04-24 Thread Brice Goglin
Please find out which line is actually causing the segfault.
Run your program under gdb. Once it crashes, type "bt full" and report
the output here.

By the way, what kind of machine are you using? (lstopo + uname -a)

Brice



Le 24/04/2016 23:46, Rezaul Karim Raju a écrit :
> Hi Brice,
>
> Thank you very much for your prompt care. 
>
> I am retrieving as below:
>
> nodeset_c = hwloc_bitmap_alloc();
>
> */* Find Location of a: 3rd QUARTER */*
> err = *hwloc_get_area_membind_nodeset(*topology, *array+ size/2,
> size/4,* nodeset_c, , HWLOC_MEMBIND_THREAD ); 
>
> /* print the corresponding NUMA nodes */
> hwloc_bitmap_asprintf(, nodeset_c);
> printf("Address:= %p  Variable:=  bound
> to*nodeset %s with contains:*\n", (array+size/2), s);
> free(s);
> hwloc_bitmap_foreach_begin(hw_i, nodeset_c) {
> *obj_c = hwloc_get_numanode_obj_by_os_index(topology, hw_i);*
> *printf("[3rd Q]  node #%u (OS index %u) with %lld bytes of memory\n",
> obj_c->logical_index, hw_i, (unsigned long long)
> obj_c->memory.local_memory)*;
> } hwloc_bitmap_foreach_end();
> hwloc_bitmap_free(nodeset_c);
>
> *It prints as below:*
> *
> *
> *error no:= -1 and segmentation fault 
> *
> *my array size is =  262144 {data type long} and each Quarter = size/4
> =65536*
> Address of array:= 0x7f350e515000, tmp:= 0x7f34fe515000, tst_array:=
> 0x7f34ee515000
> Address of array:= 0x7f350e515000, array+size/4:= 0x7f352e515000,
> array+size/2:= 0x7f354e515000, array+3*size/4:= 0x7f356e515000
>
> Address:= 0x7f350e515000  Variable:=  bound
> to nodeset 0x0001 with contains:
>  [1st Q]  node #0 (OS index 0) with 8387047424 bytes of memory
> Address:= 0x7f352e515000  Variable:=  bound to
> nodeset 0x0004 with contains:
> [2nd Q]  node #2 (OS index 2) with 8471621632 bytes of memory
>
> in case of [3rd Q]
> Error Occured, and error no:= -1 and segmentation fault happened.
>
> Thanks.!
>
>
> On Sun, Apr 24, 2016 at 4:08 PM, Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>> wrote:
>
> Hello,
> What do you mean with " it can not bind the specified memory
> section (addr, len) to the desired NUMA node"?
> Did it fail? If so, what does errno contain?
> If it didn't fail, what did it do instead?
> thanks
> Brice
>
>
>
>
> Le 24/04/2016 23:02, Rezaul Karim Raju a écrit :
>> Hi ...
>>
>> I was trying to bind each quarter of an array to 4 different NUMA
>> nodes, and doing as below: 
>>
>> *//ALLOCATION *
>> *obj_a = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 0);*
>>
>> *array =* hwloc_alloc_membind_nodeset( topology, size,
>> obj_a->nodeset, HWLOC_MEMBIND_BIND, 1);
>> *tmp *= hwloc_alloc_membind_nodeset( topology, size,
>> obj_a->nodeset, HWLOC_MEMBIND_BIND, 1); 
>> *
>> *
>> *// DISTRIBUTED BINDING  [my system has 8 NUMA nodes (0-7)]*
>> printf("Address of array:= %p, array+size/4:= %p, array+size/2:=
>> %p, array+3*size/4:= %p \n", array, array+size/4, array+size/2,
>> array+3*size/4);
>> // bind 1st quarter to node (n-1)
>> hwloc_set_area_membind_nodeset(topology, (array), size/4,
>> obj_a->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>> hwloc_set_area_membind_nodeset(topology, (tmp), size/4,
>> obj_a->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>> // bind 2nd quarter to node (2)
>> *obj_b = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE,  2);*
>> hwloc_set_area_membind_nodeset(topology, (array+size/4), size/4,
>> obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>> hwloc_set_area_membind_nodeset(topology, (tmp +size/4), size/4,
>> obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>>
>> // bind 3rd quarter to node (4)
>> *obj_c = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 4);*
>> hwloc_set_area_membind_nodeset(topology, array+size/2, size/4,
>> obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>> hwloc_set_area_membind_nodeset(topology, tmp+size/2, size/4,
>> obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>> // bind 4th quarter to node (6)
>> *obj_d = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 6);*
>> hwloc_set_area_membind_nodeset(topology, array+3*size/4, size/4,
>> obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>> hwloc_set_area_membind_nodeset(topology, tmp+3*size/4, size/4,
>> obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_ME

Re: [hwloc-users] HWLOC_get_membind: problem in getting right(specific) NODESET where data is allocated

2016-04-24 Thread Brice Goglin
Hello,
What do you mean with " it can not bind the specified memory section
(addr, len) to the desired NUMA node"?
Did it fail? If so, what does errno contain?
If it didn't fail, what did it do instead?
thanks
Brice



Le 24/04/2016 23:02, Rezaul Karim Raju a écrit :
> Hi ...
>
> I was trying to bind each quarter of an array to 4 different NUMA
> nodes, and doing as below: 
>
> *//ALLOCATION *
> *obj_a = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 0);*
>
> *array =* hwloc_alloc_membind_nodeset( topology, size, obj_a->nodeset,
> HWLOC_MEMBIND_BIND, 1);
> *tmp *= hwloc_alloc_membind_nodeset( topology, size, obj_a->nodeset,
> HWLOC_MEMBIND_BIND, 1); 
> *
> *
> *// DISTRIBUTED BINDING  [my system has 8 NUMA nodes (0-7)]*
> printf("Address of array:= %p, array+size/4:= %p, array+size/2:= %p,
> array+3*size/4:= %p \n", array, array+size/4, array+size/2,
> array+3*size/4);
> // bind 1st quarter to node (n-1)
> hwloc_set_area_membind_nodeset(topology, (array), size/4,
> obj_a->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, (tmp), size/4,
> obj_a->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> // bind 2nd quarter to node (2)
> *obj_b = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE,  2);*
> hwloc_set_area_membind_nodeset(topology, (array+size/4), size/4,
> obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, (tmp +size/4), size/4,
> obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>
> // bind 3rd quarter to node (4)
> *obj_c = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 4);*
> hwloc_set_area_membind_nodeset(topology, array+size/2, size/4,
> obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, tmp+size/2, size/4,
> obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> // bind 4th quarter to node (6)
> *obj_d = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 6);*
> hwloc_set_area_membind_nodeset(topology, array+3*size/4, size/4,
> obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
> hwloc_set_area_membind_nodeset(topology, tmp+3*size/4, size/4,
> obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
>
>
> My intention here is to distribute 'array' (which is - long type
> element:  
> array = (ELM *) malloc(bots_arg_size * sizeof(ELM));
> tmp = (ELM *) malloc(bots_arg_size * sizeof(ELM));) over nodes through
> hwloc memory binding. 
>
> 1). But except only *obj_a, it can not bind the specified memory
> section (addr, len) to the desired NUMA node. *
> 2). I did tried with  MEMBIND_INTERLEAVE policy
> array = hwloc_alloc_membind_nodeset(topology, size, cset_available,
> HWLOC_MEMBIND_INTERLEAVE, HWLOC_MEMBIND_MIGRATE);
> tmp = hwloc_alloc_membind_nodeset(topology, size, cset_available,
> HWLOC_MEMBIND_INTERLEAVE, HWLOC_MEMBIND_MIGRATE);
> but I did get it working here as well. 
>
>
> *Can you please comment on this..?  *
>
> Thank you very much in advance..!! 
> - Raju
>
> On Mon, Mar 21, 2016 at 11:25 AM, Rezaul Karim Raju
> <raju.cse.b...@gmail.com <mailto:raju.cse.b...@gmail.com>> wrote:
>
> Thanks, Brice.!
>
> On Mon, Mar 21, 2016 at 11:22 AM, Brice Goglin
> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>
> For testing, you can use this tarball:
> 
> https://ci.inria.fr/hwloc/job/zcustombranch-0-tarball/lastSuccessfulBuild/artifact/hwloc-getmemlocation-20160320.2208.gitd2f6537.tar.gz
>
>
>
>
>
> Le 21/03/2016 17:21, Rezaul Karim Raju a écrit :
>> Hi Brice,
>>
>> Thanks for your email. 
>> I believe it is definitely helpful. Getting memory range
>> within the current process will be very good information to
>> drill down. 
>> Let me use this and I will get back if any
>> clarification/comment I have.
>>
>> Regards-
>> Raju 
>>
>> On Sun, Mar 20, 2016 at 4:26 PM, Brice Goglin
>> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>>
>> I just pushed a proposal, see
>> https://github.com/open-mpi/hwloc/issues/97
>>
>> Brice
>>
>>
>>
>>
>> Le 18/12/2015 20:45, Brice Goglin a écrit :
>>> Yes, we're "thinking" about it. But there are open
>>> questions as mentioned in the github issue.
>>> By the way, we wouldn't return NULL in case of
>>> non-physically-allocated buff

Re: [hwloc-users] [WARNING: A/V UNSCANNABLE] object intersection without inclusion

2016-02-10 Thread Brice Goglin
Hello

compute-0-12 reports totally buggy NUMA information:

$ cat compute-0-12/sys/devices/system/node/node*/cpumap
,00ff
,ff00ff00
,00ff
,
$ cat compute-0-0/sys/devices/system/node/node*/cpumap
,00ff
,ff00
,00ff
,ff00
00ff,
ff00,
00ff,
ff00,

This is likely a BIOS bug, and indeed the BIOS is older on compute-0-0
(3.0a instead of 3.5). I would suggest trying the latest 3.5a from
http://www.supermicro.com/support/resources/results.aspx
If it doesn't help, you should ask SuperMicro to provide the old 3.0a
(and report the issue)

Brice



Le 10/02/2016 21:30, Fabricio Cannini a écrit :
> Hello there
>
> I'm facing an issue with hwloc 1.5.3 (old, i know, but it's the stock
> centos 6 package) in that a single node emits this message whenever i
> run any MPI-enabled program.
>
> 
>
> * Hwloc has encountered what looks like an error from the operating
> system.
> *
> * object (Socket P#0 cpuset 0x) intersection without inclusion!
> * Error occurred in topology.c line 718
> *
> * Please report this error message to the hwloc user's mailing list,
> * along with the output from the hwloc-gather-topology.sh script.
> 
>
>
>
> This happen only in one node. Other similar nodes (same hardware, same
> OS, same software) run fine. The OS is centos 6.5 x86_64.
>
>
> In the attachments, 'compute-0-0' is the healthy node, 'compute-0-12'
> is the quirky one.
>
>
> Is it possible to point the faulty hardware from the attached outputs ?
>
>
> TIA,
> Fabricio



Re: [hwloc-users] lstopo hangs for centos 7

2016-02-03 Thread Brice Goglin
Thanks. I pushed the fix to git branches. It will be included in future
releases (but 1.11.3 isn't planned anytime soon).

It might be good to report a bug to VMware. I don't think they are
supposed to advertise the x2APIC CPU feature unless they support CPUID
0xb leaf.

Brice





Le 03/02/2016 05:45, Jianjun Wen a écrit :
> Confirmed!
> This patch fixes the problem.
>
> Thanks a lot!
> Jianjun
>
> On Tue, Feb 2, 2016 at 9:05 AM, Brice Goglin <brice.gog...@inria.fr
> <mailto:brice.gog...@inria.fr>> wrote:
>
> Does this patch help?
>
> diff --git a/src/topology-x86.c b/src/topology-x86.c
> index efd4300..a602121 100644
> --- a/src/topology-x86.c
> +++ b/src/topology-x86.c
> @@ -403,7 +403,7 @@ static void look_proc(struct hwloc_backend *backend, 
> struct procinfo *infos, uns
>/* Get package/core/thread information from cpuid 0x0b
> * (Intel x2APIC)
> */
> -  if (cpuid_type == intel && has_x2apic(features)) {
> +  if (cpuid_type == intel && highest_cpuid >= 0x0b && 
> has_x2apic(features)) {
>  unsigned level, apic_nextshift, apic_number, apic_type, apic_id = 0, 
> apic_shift = 0, id;
>  for (level = 0; ; level++) {
>ecx = level;
>
> It looks like VMware reports that the virtual reports x2APIC
> feature without 0xb CPUID. This looks buggy, but can be worked around.
>
> Brice
>
>
>
>
>
> Le 02/02/2016 05:50, Jianjun Wen a écrit :
>> Hi Brice,
>> Oh, didn't realize that. Only master has the gatther-cpuid.
>>
>> Attached.
>>
>> BTW, /proc/cpuinfo contain a field called flags. If it is an vm,
>> hypervisor will be there. 
>> sudo  dmidecode -s system-product-name
>> will output 
>> VMware Virtual Platform
>>
>> Jianjun
>>
>> On Mon, Feb 1, 2016 at 12:26 AM, Brice Goglin
>> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>>
>> Looks like you ran hwloc-gather-topology instead of
>> hwloc-gather-cpuid?
>> By the way, in (4)
>> tar cfj cpuid.tbz2 foo
>> should be
>> tar cfj cpuid.tbz2 cpuid
>>
>>
>>
>>
>> Le 01/02/2016 07:20, Jianjun Wen a écrit :
>>> Hi Brice,
>>> Thanks for the workaround -- it works very good.
>>>
>>> Attached please find the two output file after run
>>> hwloc-gather-cpuid.
>>> Let me after this is fixed!
>>>
>>> thanks,
>>> Jianjun
>>>
>>> On Sun, Jan 31, 2016 at 9:48 PM, Brice Goglin
>>> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>>>
>>> Thanks for the debugging. I guess VMware doesn't
>>> properly emulate the CPUID instruction.
>>>
>>> Please do:
>>> 1) take a tarball from git master at
>>> https://ci.inria.fr/hwloc/job/master-0-tarball/ and build it
>>> 2) export HWLOC_COMPONENTS=-x86 in your terminal
>>> 3) do utils/hwloc/hwloc-gather-cpuid
>>> 4) tar cfj cpuid.tbz2 foo and send that cpuid.tbz2
>>>
>>> Step (3) might do an infinite loop for the same reason,
>>> please replace
>>> for(i=0; ; i++) {
>>> with
>>> for(i=0; i<10; i++) {
>>> everywhere in utils/hwloc/hwloc-gather-cpuid.c
>>>
>>> This tarball will help me find what's buggy in VMware
>>> CPUID instruction.
>>>
>>>
>>> In the meantime, you can fix your hwloc by exporting
>>> HWLOC_COMPONENTS=-x86 in your environment.
>>>
>>> If somebody knows how do detect vmware by looking under
>>> /proc or /sys, we could use that to automatically set
>>> that environment variable.
>>>
>>> thanks
>>> Brice
>>>
>>>
>>>
>>>
>>>
>>> Le 01/02/2016 05:59, Jianjun Wen a écrit :
>>>> I did a debug build. Found it loops forever in this
>>>> loop in topology-x86.c:404.
>>>>   
>>>>
>>>> /* Get package/core/thread information from cpuid 0x0b
>>>>  

Re: [hwloc-users] lstopo hangs for centos 7

2016-02-01 Thread Brice Goglin
Thanks for the debugging. I guess VMware doesn't properly emulate the
CPUID instruction.

Please do:
1) take a tarball from git master at
https://ci.inria.fr/hwloc/job/master-0-tarball/ and build it
2) export HWLOC_COMPONENTS=-x86 in your terminal
3) do utils/hwloc/hwloc-gather-cpuid
4) tar cfj cpuid.tbz2 foo and send that cpuid.tbz2

Step (3) might do an infinite loop for the same reason, please replace
for(i=0; ; i++) {
with
for(i=0; i<10; i++) {
everywhere in utils/hwloc/hwloc-gather-cpuid.c

This tarball will help me find what's buggy in VMware CPUID instruction.


In the meantime, you can fix your hwloc by exporting
HWLOC_COMPONENTS=-x86 in your environment.

If somebody knows how do detect vmware by looking under /proc or /sys,
we could use that to automatically set that environment variable.

thanks
Brice




Le 01/02/2016 05:59, Jianjun Wen a écrit :
> I did a debug build. Found it loops forever in this loop in
> topology-x86.c:404.
>   
>
> /* Get package/core/thread information from cpuid 0x0b
>* (Intel x2APIC)
>*/
>   if (cpuid_type == intel && has_x2apic(features)) {
> unsigned level, apic_nextshift, apic_number, apic_type, apic_id =
> 0, apic_shift = 0, id;
> for (level = 0; ; level++) {
>   ecx = level;
>   eax = 0x0b;
>   hwloc_x86_cpuid(, , , );
>   if (!eax && !ebx)
> break;
> }
>
> On Sun, Jan 31, 2016 at 8:30 PM, Christopher Samuel
> > wrote:
>
> On 01/02/16 15:09, Jianjun Wen wrote:
>
> > 0x77bce13c in look_proc () from /lib64/libhwloc.so.5
> >
> > Always the same place.
>
> pstack on the process when stuck might give more of an insight as it
> should give more of a stack trace.
>
> Also running lstopo under strace should show what it is trying to
> do at
> that point.
>
> All the best,
> Chris
> --
>  Christopher SamuelSenior Systems Administrator
>  VLSCI - Victorian Life Sciences Computation Initiative
>  Email: sam...@unimelb.edu.au 
> Phone: +61 (0)3 903 55545 
>  http://www.vlsci.org.au/  http://twitter.com/vlsci
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post:
> http://www.open-mpi.org/community/lists/hwloc-users/2016/01/1251.php
>
>
>
>
> -- 
> -Jianjun Wen
> Wancube.com - 3D photography
> Phone: 408 888 7023
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2016/01/1252.php



Re: [hwloc-users] lstopo hangs for centos 7

2016-01-31 Thread Brice Goglin
Hello

Thanks for the report. I have never seen this issue. I have CentOS 7 VMs
(kvm), lstopo works fine. Did you try this in similar VMs in the past?

When you say "latest hwloc", do you mean "build latest tarball" (1.11.2)
or "installed latest centos package" (1.7)?

First thing to check: run lstopo, let it hang, and check under top
whether it uses 100% CPU or 0% CPU (to see if that's an infinite loop or
not).

Then, run it under gdb:
$ gdb lstopo
Type 'r' and Enter
When things hang, do ctrl-c
Type "where" and send the output to us.

If you got 100% in top above, you should do this multiple time. After
"where", type 'c' to go back to the execution, ctrl+c again, "where"
again and check whether the backtrace is similar.

Brice



Le 31/01/2016 04:48, Jianjun Wen a écrit :
> I installed the latest centos 7 (1151) on VM (vmware), then installed
> latest hwloc.
> lstopo command hangs.
>
> hwloc_topology_load()
> function call also hangs.
>
> Is this an know issue? How to find out what's wrong?
>
> thanks
> -- 
> -Jianjun Wen
> Wancube.com - 3D photography
> Phone: 408 888 7023
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2016/01/1247.php



Re: [hwloc-users] HWLOC_get_membind: problem in getting right(specific) NODESET where data is allocated

2016-01-27 Thread Brice Goglin
get_area_membind() would return a full nodeset if you don't membind first.

By the way, mydata_ptr is not a physical address, it's a virtual address.

Brice



Le 27/01/2016 02:23, Rezaul Karim Raju a écrit :
> Hi,
>
> I am interested to know about what 'hwloc_bind'  does beyond malloc()-
> the OS call. 
> If I do like this: 
>
> int mydata=11;
> int * mydata_ptr;
> *mydata_ptr = (int *)malloc(sizeof(int));  *
> * *
> *or,***
> *   mydata_ptr = (int *) hwloc_alloc (topology, (sizeof(int)));*
> **
> *mydata_ptr =  *
> **
> and then call: 
> *hwloc_get_area_membind_nodeset (*topology, *mydata_ptr,* sizeof(int),
> *nodeset,* , flags); flags:= 0/1 process / thread   
>
> it gives me the all available nodesets.
> According to Brice previous reply: hwloc (get_area_membind ) returns
> once Linux could have allocated (bound) memory buffer (the physical
> address I am sending as *mydata_ptr*). 
>
> *Is this something hwloc can only returns locations (& corresponding
> nodeset) once it is bound (*set_area_membind*) prior ..? *
> *
> *
> Thank you in advance.
> - Raju
>
> On Fri, Dec 18, 2015 at 11:57 AM, Rezaul Karim Raju
> <raju.cse.b...@gmail.com <mailto:raju.cse.b...@gmail.com>> wrote:
>
> Hi Brice, 
>
> Thanks for your time and nice explanation. 
> I have looked at the issue with location return (the page
> proportion across multiple node & physical allocation). Are you
> thinking to add this function..? Like if we think list of node or
> nodes where the array is allocated (only if physically allocated
> otherwise NULL) is it possible..? 
>
> I am looking for getting the physical location of data allocated
> by OS default policy. Appreciate any better idea and please share
> with me. 
>
> Best Regards,
> - Raju    
>
> On Tue, Dec 15, 2015 at 3:28 AM, Brice Goglin
> <brice.gog...@inria.fr <mailto:brice.gog...@inria.fr>> wrote:
>
>
>
> Le 15/12/2015 07:21, Brice Goglin a écrit :
>>
>>
>> Le 15/12/2015 05:57, Rezaul Karim Raju a écrit :
>>> *OUTPUT: *
>>> *Policy-->* buffer(Array: A) *membind [default OS binding]
>>> Policy is:= 1 [1 refers to *HWLOC_MEMBIND_FIRSTTOUCH
>>> 
>>> <https://www.open-mpi.org/projects/hwloc/doc/v1.11.1/a00083.php#ggac9764f79505775d06407b40f5e4661e8a979c7aa78dd32780858f30f47a72cca0>*]*
>>> *Nodeset --> *buffer(Array: A) bound to nodeset*0x00ff
>>> *with contains*:*
>>> * *node #0 (OS index 0) with 8387047424 bytes of memory
>>>  node #1 (OS index 1) with 8471617536 bytes of memory
>>>  node #2 (OS index 2) with 8471621632 bytes of memory
>>>  node #3 (OS index 3) with 8471617536 bytes of memory
>>>  node #4 (OS index 4) with 8471621632 bytes of memory
>>>  node #5 (OS index 5) with 8471617536 bytes of memory
>>>  node #6 (OS index 6) with 8471621632 bytes of memory
>>>  node #7 (OS index 7) with 8471564288 bytes of memory
>>> *
>>> *
>>> *why it shows allocated memory is bound to all available
>>> nodeset..? should it not be allocated to a specific nodeset
>>> one ..?*
>>> *
>>> *
>>
>> Hello
>>
>> You are confusing the "binding" and the "actual location".
>> Your memory buffer isn't bound to a specific location by
>> default. But Linux has to allocate it somewhere. So your
>> buffer is "located" in some node after the allocation, but it
>> is not "bound" there (what get_area_membind returns) which
>> means Linux could have allocated it somewhere else.
>>
>> hwloc cannot currently return the "location" of a memory
>> buffer. I have been thinking about adding this feature in the
>> past, but it looks like you are the first actual user
>> requesting this. We could add something like
>> hwloc_get_last_memory_location(topology, input buffer,
>> outputnodeset)
>> At least on Linux that's possible.
>>
>> For clarity, this is similar to the difference between
>> hwloc_get_cpubind() and hwloc_get_last_cpu_location(): A task
>> always runs on a specific PU, even if it is not bound to
>> anything specific PU.
>
> By the way, there is already an issue for this:
> https://git

Re: [hwloc-users] [WARNING: A/V UNSCANNABLE] hwloc error after upgrading from Centos 6.5 to Centos 7 on Supermicro with AMD Opteron 6344

2016-01-07 Thread Brice Goglin
Hello

Good to know, thanks.

There are two ways to workaround the issue:
* run "lstopo foo.xml" on a node that doesn't have the bug and do export
HWLOC_XMLFILE=foo.xml and HWLOC_THISSYSTEM=1 on buggy nodes. (that's
what you call a "map" below). Works with very old hwloc releases.
* export HWLOC_COMPONENTS=x86 (only works since hwloc >= 1.11.2)

Brice




Le 07/01/2016 16:20, David Winslow a écrit :
> Brice,
>
> Thanks for the information! It’s good to know it wasn’t a flaw in the 
> upgrade. This bug must have been introduced in kernel 3.x. I ran lstopo on on 
> of our servers that still have Centos 6.5 and it correctly reports L3 cache 
> for every 6 cores as shown below.
>
> We have 75 servers with the exact same specifications. I have only upgraded 
> two when I came across this problem during testing. Since I have a correct 
> map on the non-upgraded servers, can I use that map on the upgraded servers 
> somehow? Essentially hard code it?
>
> --- FROM Centos 6.5 ---
>   Socket L#0 (P#0 total=134215604KB CPUModel="AMD Opteron(tm) Processor 6344  
>" CPUType=x86_64)
> NUMANode L#0 (P#0 local=67106740KB total=67106740KB)
>   L3Cache L#0 (size=6144KB linesize=64 ways=64)
> L2Cache L#0 (size=2048KB linesize=64 ways=16)
>   L1iCache L#0 (size=64KB linesize=64 ways=2)
> L1dCache L#0 (size=16KB linesize=64 ways=4)
>   Core L#0 (P#0)
> PU L#0 (P#0)
> L1dCache L#1 (size=16KB linesize=64 ways=4)
>   Core L#1 (P#1)
> PU L#1 (P#1)
> L2Cache L#1 (size=2048KB linesize=64 ways=16)
>   L1iCache L#1 (size=64KB linesize=64 ways=2)
> L1dCache L#2 (size=16KB linesize=64 ways=4)
>   Core L#2 (P#2)
> PU L#2 (P#2)
> L1dCache L#3 (size=16KB linesize=64 ways=4)
>   Core L#3 (P#3)
> PU L#3 (P#3)
> L2Cache L#2 (size=2048KB linesize=64 ways=16)
>   L1iCache L#2 (size=64KB linesize=64 ways=2)
> L1dCache L#4 (size=16KB linesize=64 ways=4)
>   Core L#4 (P#4)
> PU L#4 (P#4)
>     L1dCache L#5 (size=16KB linesize=64 ways=4)
>   Core L#5 (P#5)
> PU L#5 (P#5)
>
>> On Jan 7, 2016, at 1:22 AM, Brice Goglin <brice.gog...@inria.fr> wrote:
>>
>> Hello
>>
>> This is a kernel bug for 12-core AMD Bulldozer/Piledriver (62xx/63xx) 
>> processors. hwloc is just complaining about buggy L3 information. lstopo 
>> should report one L3 above each set of 6 cores below each NUMA node. Instead 
>> you get strange L3s with 2, 4 or 6 cores.
>>
>> If you're not binding tasks based on L3 locality and if your applications do 
>> not care about L3, you can pass HWLOC_HIDE_ERRORS=1 in the environment to 
>> hide the message.
>>
>> AMD was working on a kernel patch but it doesn't seem to be in the upstream 
>> Linux yet. In hwloc v1.11.2, you can workaround the problem by passing 
>> HWLOC_COMPONENTS=x86 in the environment.
>>
>> I am not sure why CentOS 6.5 didn't complain. That 2.6.32 kernel should be 
>> buggy too, and old hwloc releases already complained about such bugs.
>>
>> thanks
>> Brice
>>
>>
>>
>>
>>
>>
>> Le 07/01/2016 04:10, David Winslow a écrit :
>>> I upgraded our servers from Centos 6.5 to Centos7.2. Since then, when I run 
>>> mpirun I get the following error but the software continues to run and it 
>>> appears to work fine.
>>>
>>> * hwloc 1.11.0rc3-git has encountered what looks like an error from the 
>>> operating system.
>>> *
>>> * L3 (cpuset 0x03f0) intersects with NUMANode (P#0 cpuset 0x003f) 
>>> without inclusion!
>>> * Error occurred in topology.c line 983
>>> *
>>> * The following FAQ entry in the hwloc documentation may help:
>>> *   What should I do when hwloc reports "operating system" warnings?
>>> * Otherwise please report this error message to the hwloc user's mailing 
>>> list,
>>> * along with the output+tarball generated by the hwloc-gather-topology 
>>> script.
>>>
>>> I can replicate the error by simply running hwloc-info.
>>>
>>> The version of hwloc used with mpirun is 1.9. The version installed on the 
>>> server that I ran is 1.7 that comes with Centos 7. They both give the error 
>>> with minor differences shown below.
>>>
>>> With hwloc 1.7
>>> * object (L3 cpuset 0x03f0) intersection wit

Re: [hwloc-users] error from the operating system - Solaris 11.3 - SOLVED

2016-01-07 Thread Brice Goglin
Thanks, I copied useful information from this thread and some links to
https://github.com/open-mpi/hwloc/issues/143

However, not sure I'll have time to look at this in the near future :/

Brice




Le 07/01/2016 09:03, Matthias Reich a écrit :
> Hello,
>
> To check whether kstat is able to report the psrset definitions, I
> defined a set consisting of 2 CPUs (psrset -c 1-2) CPU1 and CPU2. The
> remaining CPUs (CPU0, CPU2..CPU23) were left undefined.
>
> On the machine, we can execute the "kstat" command and receive (among
> 1000s of lines) the following info:
>
> module: unixinstance: 0
> name:   psetclass:misc
> avenrun_15min   70
> avenrun_1min53
> avenrun_5min47
> crtime  0
> ncpus   22
> runnable1146912
> snaptime80083.491239257
> updates 790784
> waiting 0
>
>
> module: unixinstance: 1
> name:   psetclass:misc
> avenrun_15min   0
> avenrun_1min0
> avenrun_5min0
> crtime  79983.070416351
> ncpus   2
> runnable0
> snaptime80083.595839172
> updates 1005
> waiting 0
>
> which is not very comprehensive and doesn't even tell, which CPUs are
> part of the particular set, but could probably be used to at least warn
> about the existence of a CPU set and prevent the (not very intuitive)
> error message and consequent abort.
>
> However, doing the same on the machine without the pset defined, we get:
>
> module: unixinstance: 0
> name:   psetclass:misc
> avenrun_15min   50
> avenrun_1min38
> avenrun_5min41
> crtime  0
> ncpus   24
> runnable1163866
> snaptime81105.346688035
> updates 801003
> waiting 0
>
> so the (only) processor set encompasses all 24 (virtual) cores. This
> may be the key to check for.
>
> The C-API to check for processor set(s) is available through the
> libpool library, which allows more resource pool configuration than
> just processor sets, but can probably act as an abstraction layer for
> different Solaris flavors...
>
> Matthias
>
>>  Hello
>> So processor sets are not taken into account when Solaris reports
>> topology information in kstat etc.
>> Do you know if hwloc can query processor sets from the C interface?
>> If so, we could apply the processor set mask to hwloc object cpusets
>> during discovery to avoid your error.
>> Brice
>>
>> Le 05/01/2016 10:18, Karl Behler a écrit :
>>> There was a processor set defined (command psrset) on this machine.
>>> Having removed the psrset hwloc-info produces a result without error
>>> messages:
>>>
>>> hwloc-info -v
>>> depth 0:1 Machine (type #1)
>>>  depth 1:   2 NUMANode (type #2)
>>>   depth 2:  2 Package (type #3)
>>>depth 3: 12 Core (type #5)
>>> depth 4:24 PU (type #6)
>>>
>>> It seems the concept of defining a psrset is in contradiction to what
>>> hwloc and/or openmpi expects/allows.
>>>
>>>
>>> On 04.01.16 18:16, Karl Behler wrote:
 We used to run our MPI application with the SUNWhpc implementation
 from Sun/Oracle. (This was derived from openmpi 1.5.)
 However, the Oracle HPC implementation fails for the new Solaris 11.3
 platform.
 So we downloaded and made openmpi 1.10.1 on this platform from
 scratch.

 All seems fine and a simple test application runs fine.
 However, with the real application we are running into a hwloc
 problem.

 So we also downloaded and made the hwloc package 1.11.2.

 Now examining hardware locality we get the following error:

 hwloc-info -v --whole-io
 


 * hwloc 1.11.2 has encountered what looks like an error from the
 operating system.
 *
 * Core (P#0 cpuset 0x1001) intersects with NUMANode (P#1 cpuset
 0x0003c001) without inclusion!
 * Error occurred in topology.c line 1046
 *
 * The following FAQ entry in the hwloc documentation may help:
 *   What should I do when hwloc reports "operating system" warnings?
 * Otherwise please report this error message to the hwloc user's
 mailing list,

Re: [hwloc-users] [WARNING: A/V UNSCANNABLE] hwloc error after upgrading from Centos 6.5 to Centos 7 on Supermicro with AMD Opteron 6344

2016-01-07 Thread Brice Goglin
Hello

This is a kernel bug for 12-core AMD Bulldozer/Piledriver (62xx/63xx)
processors. hwloc is just complaining about buggy L3 information. lstopo
should report one L3 above each set of 6 cores below each NUMA node.
Instead you get strange L3s with 2, 4 or 6 cores.

If you're not binding tasks based on L3 locality and if your
applications do not care about L3, you can pass HWLOC_HIDE_ERRORS=1 in
the environment to hide the message.

AMD was working on a kernel patch but it doesn't seem to be in the
upstream Linux yet. In hwloc v1.11.2, you can workaround the problem by
passing HWLOC_COMPONENTS=x86 in the environment.

I am not sure why CentOS 6.5 didn't complain. That 2.6.32 kernel should
be buggy too, and old hwloc releases already complained about such bugs.

thanks
Brice






Le 07/01/2016 04:10, David Winslow a écrit :
> I upgraded our servers from Centos 6.5 to Centos7.2. Since then, when I run 
> mpirun I get the following error but the software continues to run and it 
> appears to work fine.
>
> * hwloc 1.11.0rc3-git has encountered what looks like an error from the 
> operating system.
> *
> * L3 (cpuset 0x03f0) intersects with NUMANode (P#0 cpuset 0x003f) 
> without inclusion!
> * Error occurred in topology.c line 983
> *
> * The following FAQ entry in the hwloc documentation may help:
> *   What should I do when hwloc reports "operating system" warnings?
> * Otherwise please report this error message to the hwloc user's mailing list,
> * along with the output+tarball generated by the hwloc-gather-topology script.
>
> I can replicate the error by simply running hwloc-info.
>
> The version of hwloc used with mpirun is 1.9. The version installed on the 
> server that I ran is 1.7 that comes with Centos 7. They both give the error 
> with minor differences shown below.
>
> With hwloc 1.7
> * object (L3 cpuset 0x03f0) intersection without inclusion!
> * Error occurred in topology.c line 753
>
> With hwloc 1.9
> * L3 (cpuset 0x03f0) intersects with NUMANode (P#0 cpuset 0x003f) 
> without inclusion!
> * Error occurred in topology.c line 983
>
> The current kernel is 3.10.0-327.el7.x86_64. I’ve tried updating the kernel 
> to a minor release update and even tried to install kernel v4.4.3. None of 
> the kernels worked. Again, hwloc works fine in Centos 6.5 with kernel 
> 2.6.32-431.29.2.el6.x86_64.
>
> I’ve attached the files generated by hwloc-gather-topology.sh.  I compared 
> what this script says is the expected output to the actual output and, from 
> what I can tell, they look the same. Maybe I’m missing something after 
> staring all day at the information.
>
> I did a clean install of the OS to perform the upgrade from 6.5.
>
> I’ve attached the results of the hwloc-gather-topology.sh script. Any help 
> will be greatly appreciated.
>
>
>
>
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2016/01/1238.php



Re: [hwloc-users] error from the operating system - Solaris 11.3 - SOLVED

2016-01-05 Thread Brice Goglin
Hello
So processor sets are not taken into account when Solaris reports
topology information in kstat etc.
Do you know if hwloc can query processor sets from the C interface?
If so, we could apply the processor set mask to hwloc object cpusets
during discovery to avoid your error.
Brice




Le 05/01/2016 10:18, Karl Behler a écrit :
> There was a processor set defined (command psrset) on this machine.
> Having removed the psrset hwloc-info produces a result without error
> messages:
>
> hwloc-info -v
> depth 0:1 Machine (type #1)
>  depth 1:   2 NUMANode (type #2)
>   depth 2:  2 Package (type #3)
>depth 3: 12 Core (type #5)
> depth 4:24 PU (type #6)
>
> It seems the concept of defining a psrset is in contradiction to what
> hwloc and/or openmpi expects/allows.
>
>
> On 04.01.16 18:16, Karl Behler wrote:
>> We used to run our MPI application with the SUNWhpc implementation
>> from Sun/Oracle. (This was derived from openmpi 1.5.)
>> However, the Oracle HPC implementation fails for the new Solaris 11.3
>> platform.
>> So we downloaded and made openmpi 1.10.1 on this platform from scratch.
>>
>> All seems fine and a simple test application runs fine.
>> However, with the real application we are running into a hwloc problem.
>>
>> So we also downloaded and made the hwloc package 1.11.2.
>>
>> Now examining hardware locality we get the following error:
>>
>> hwloc-info -v --whole-io
>> 
>>
>> * hwloc 1.11.2 has encountered what looks like an error from the
>> operating system.
>> *
>> * Core (P#0 cpuset 0x1001) intersects with NUMANode (P#1 cpuset
>> 0x0003c001) without inclusion!
>> * Error occurred in topology.c line 1046
>> *
>> * The following FAQ entry in the hwloc documentation may help:
>> *   What should I do when hwloc reports "operating system" warnings?
>> * Otherwise please report this error message to the hwloc user's
>> mailing list,
>> * along with any relevant topology information from your platform.
>> 
>>
>> depth 0:1 Machine (type #1)
>>  depth 1:   2 Package (type #3)
>>   depth 2:  2 NUMANode (type #2)
>>depth 3: 1 Core (type #5)
>> depth 4:24 PU (type #6)
>>
>> Since I could not find the mentioned FAQ topic I'm asking the list
>> for advice.
>>
>> Our system is an Oracle/ Solaris 11.3 (latest patch level) on an
>> Intel hardware platform from Sun.
>>
>> output of uname -a -> SunOS sxaug28 5.11 11.3 i86pc i386 i86pc
>> output of psrinfo -v ->
>>
>> Status of virtual processor 0 as of: 01/04/2016 17:10:17
>>   on-line since 01/04/2016 14:44:28.
>>   The i386 processor operates at 1600 MHz,
>> and has an i387 compatible floating point processor.
>> Status of virtual processor 1 as of: 01/04/2016 17:10:17
>>   on-line since 01/04/2016 14:45:10.
>>   The i386 processor operates at 1600 MHz,
>> and has an i387 compatible floating point processor.
>> .
>> . (similar lines removed)
>> .
>> Status of virtual processor 23 as of: 01/04/2016 17:10:17
>>   on-line since 01/04/2016 14:45:11.
>>   The i386 processor operates at 1600 MHz,
>> and has an i387 compatible floating point processor.
>>
>> Following comes the script which was used to make hwloc: (used
>> compiler: Sunstudio 12.4, see config.log as bz2 attachment)
>>
>> setenv CFLAGS "-m64 -xtarget=generic -xarch=sse2 -xprefetch
>> -xprefetch_level=2 -xvector=simd -xdepend=yes -xbuiltin=%all -xO5"
>> setenv CXXFLAGS "$CFLAGS"
>> setenv FCFLAGS "-m64 -xtarget=generic -xarch=sse2 -xprefetch
>> -xprefetch_level=2 -xvector=simd -stackvar -xO5"
>> setenv FFLAGS "$FCFLAGS"
>> setenv PREFIX /usr/openmpi/hwloc-1.11.2
>> ./configure --prefix=$PREFIX --disable-debug
>> dmake -j 12
>> # as root: make install
>> #: cp -p config.status $PREFIX/config.status
>>
>> Any advice much appreciated.
>>
>> Karl
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>> Searchable archives: 
>> http://www.open-mpi.org/community/lists/hwloc-users/2016/01/1236.php
>
>
> -- 
> Dr. Karl Behler   
> CODAC & IT services ASDEX Upgrade
> phon +49 89 3299-1351 fax 3299-961351
>
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2016/01/1236.php



Re: [hwloc-users] [hwloc-announce] Hardware Locality (hwloc) v1.11.2 released

2015-12-19 Thread Brice Goglin
Applied, thanks !


Le 19/12/2015 06:52, Marco Atzeri a écrit :
> On 19/12/2015 00:38, Brice Goglin wrote:
>>
>>
>> Le 18/12/2015 12:14, Marco Atzeri a écrit :
>>> attached minor patch to solve a false "make check" failure
>>> on platform where EXEEXT in not empty.
>>>
>>> Tested on CYGWIN platforms.
>>>
>>> Regards
>>> Marco
>>> --- origsrc/hwloc-1.11.2/utils/hwloc/test-hwloc-assembler.sh.in   
>>> 2015-06-14 21:53:04.0 +0200
>>> +++ src/hwloc-1.11.2/utils/hwloc/test-hwloc-assembler.sh.in   
>>> 2015-12-18 07:47:38.743536900 +0100
>>> @@ -12,6 +12,7 @@ HWLOC_top_builddir="@HWLOC_top_builddir@
>>>   assembler="$HWLOC_top_builddir/utils/hwloc/hwloc-assembler"
>>>   HWLOC_top_srcdir="@HWLOC_top_srcdir@"
>>>   SED="@SED@"
>>> +EXEEXT="@EXEEXT@"
>>>
>>>   HWLOC_PLUGINS_PATH=${HWLOC_top_builddir}/src
>>>   export HWLOC_PLUGINS_PATH
>>> @@ -46,7 +47,7 @@ $assembler $file \
>>>   # filter ProcessName since it may be hwloc-info or lt-hwloc-info
>>>   cat $file \
>>>| $SED -e '/>> value=\"'$HWLOC_VERSION'\"\/>/d' \
>>> - | $SED -e '/>> value=\"hwloc-assembler\"\/>/d' -e '/>> value=\"lt-hwloc-assembler\"\/>/d' \
>>> + | $SED -e '/>> value=\"hwloc-assembler'$EXEEXT'\"\/>/d' -e '/>> name=\"ProcessName\" value=\"lt-hwloc-assembleri'$EXEEXT'\"\/>/d' \
>>>> ${file}.tmp
>>>   mv -f ${file}.tmp $file
>>> 
>>> 
>>> ^ here
>>
>> Hello
>> The 'i' at the end of 'lt-hwloc-assembleri' at the end of the last added
>> line above causes a failure here. Typo in your patch?
>> Thanks
>> Brice
>>
>
> sure. VI typo
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post:
> http://www.open-mpi.org/community/lists/hwloc-users/2015/12/1234.php



Re: [hwloc-users] [hwloc-announce] Hardware Locality (hwloc) v1.11.2 released

2015-12-18 Thread Brice Goglin


Le 18/12/2015 12:14, Marco Atzeri a écrit :
> attached minor patch to solve a false "make check" failure
> on platform where EXEEXT in not empty.
>
> Tested on CYGWIN platforms.
>
> Regards
> Marco
> --- origsrc/hwloc-1.11.2/utils/hwloc/test-hwloc-assembler.sh.in   
> 2015-06-14 21:53:04.0 +0200
> +++ src/hwloc-1.11.2/utils/hwloc/test-hwloc-assembler.sh.in   2015-12-18 
> 07:47:38.743536900 +0100
> @@ -12,6 +12,7 @@ HWLOC_top_builddir="@HWLOC_top_builddir@
>  assembler="$HWLOC_top_builddir/utils/hwloc/hwloc-assembler"
>  HWLOC_top_srcdir="@HWLOC_top_srcdir@"
>  SED="@SED@"
> +EXEEXT="@EXEEXT@"
>  
>  HWLOC_PLUGINS_PATH=${HWLOC_top_builddir}/src
>  export HWLOC_PLUGINS_PATH
> @@ -46,7 +47,7 @@ $assembler $file \
>  # filter ProcessName since it may be hwloc-info or lt-hwloc-info
>  cat $file \
>   | $SED -e '//d' \
> - | $SED -e '//d' -e 
> '//d' \
> + | $SED -e '/ value=\"hwloc-assembler'$EXEEXT'\"\/>/d' -e '/ value=\"lt-hwloc-assembleri'$EXEEXT'\"\/>/d' \
>   > ${file}.tmp
>  mv -f ${file}.tmp $file
>   
>   ^ here

Hello
The 'i' at the end of 'lt-hwloc-assembleri' at the end of the last added
line above causes a failure here. Typo in your patch?
Thanks
Brice



Re: [hwloc-users] [hwloc-announce] Hardware Locality (hwloc) v1.11.2 released

2015-12-18 Thread Brice Goglin
Hello
Release announces are sent to the hwloc-annonce mailing list only.
Yes your AMD bug is covered. You should pass HWLOC_COMPONENTS=x86 in the
environment to work around your Linux kernel bug.
Regards
Brice


Le 18/12/2015 12:26, Fabian Wein a écrit :
> Somehow I missed the announcement?!
>>> The Hardware Locality (hwloc) team is pleased to announce the release
>>> of v1.11.2:
>
> Does this cover the issue of certain Opteron systems?
>
> Thanks!
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post:
> http://www.open-mpi.org/community/lists/hwloc-users/2015/12/1228.php



Re: [hwloc-users] HWLOC_get_membind: problem in getting right(specific) NODESET where data is allocated

2015-12-15 Thread Brice Goglin


Le 15/12/2015 05:57, Rezaul Karim Raju a écrit :
> *OUTPUT: *
> *Policy-->* buffer(Array: A) *membind [default OS binding] Policy is:=
> 1 [1 refers to *HWLOC_MEMBIND_FIRSTTOUCH
> *]*
> *Nodeset --> *buffer(Array: A) bound to nodeset*0x00ff *with
> contains*:*
> * *node #0 (OS index 0) with 8387047424 bytes of memory
>  node #1 (OS index 1) with 8471617536 bytes of memory
>  node #2 (OS index 2) with 8471621632 bytes of memory
>  node #3 (OS index 3) with 8471617536 bytes of memory
>  node #4 (OS index 4) with 8471621632 bytes of memory
>  node #5 (OS index 5) with 8471617536 bytes of memory
>  node #6 (OS index 6) with 8471621632 bytes of memory
>  node #7 (OS index 7) with 8471564288 bytes of memory
> *
> *
> *why it shows allocated memory is bound to all available nodeset..?
> should it not be allocated to a specific nodeset one ..?*
> *
> *

Hello

You are confusing the "binding" and the "actual location". Your memory
buffer isn't bound to a specific location by default. But Linux has to
allocate it somewhere. So your buffer is "located" in some node after
the allocation, but it is not "bound" there (what get_area_membind
returns) which means Linux could have allocated it somewhere else.

hwloc cannot currently return the "location" of a memory buffer. I have
been thinking about adding this feature in the past, but it looks like
you are the first actual user requesting this. We could add something like
hwloc_get_last_memory_location(topology, input buffer, outputnodeset)
At least on Linux that's possible.

For clarity, this is similar to the difference between
hwloc_get_cpubind() and hwloc_get_last_cpu_location(): A task always
runs on a specific PU, even if it is not bound to anything specific PU.

Brice




> *If I write as below: *
>
> /* Get last node. */
> n = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_NODE);
> if (n) {
>
> 
> void *m;
> int prev_val, next_val;
> size = sizeof(int); //1024*1024;
>
> obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, n - 1);
> m = *hwloc_alloc_membind_nodeset(*topology, size,
> *obj->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_PROCESS*);
> hwloc_free(topology, m, size);
>
> m = malloc(size);
> // touch the m memory 
> m = _val;
> *(int*)m = 1010;
> *hwloc_set_area_membind_nodeset(*topology, m, size,
> *obj->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_STRICT*);
> //HWLOC_MEMBIND_DEFAULT:= Reset the memory allocation policy to the
> system default(HWLOC_MEMBIND_FIRSTTOUCH (Linux)).
> */* check where buffer(m) is allocated */*
> nodeset = hwloc_bitmap_alloc();
> *hwloc_get_area_membind_nodeset(topology, m, size, nodeset,
> , 0); *
>
> /* check the binding policy */
> printf("buffer(m) membind-ed policy is %d \n", nodepolicy);
> /* print the corresponding NUMA nodes */
> hwloc_bitmap_asprintf(, nodeset);
> printf("buffer bound to nodeset %s with contains:\n", s);
> free(s);
> hwloc_bitmap_foreach_begin(i, nodeset) {
> obj = hwloc_get_numanode_obj_by_os_index(topology, i);
> printf("  node #%u (OS index %u) with %lld bytes of memory\n",
> obj->logical_index, i, (unsigned long long) obj->memory.local_memory);
> } hwloc_bitmap_foreach_end();
> hwloc_bitmap_free(nodeset);
>
> *OUTPUT: *
> *Policy:* buffer(Array: A) membind *[default OS binding] Policy is:= 2*
> *Nodeset: *buffer(Array: A) *bound to nodeset 0x0080* with contains:
>   *node #7 (OS index 7) *with 8471564288 bytes of memory
> In this case it shows the specific nodeset one where the memory is
> allocated. 
>
> *Can you please comment and explain what is happening underneath ..?
> Thanking you in advance.* 
>
> 
> RaJu, Rezaul Karim
> Graduate Student (PhD) | Computer Science | University of Houston
> Research in High Performance Computing Tools  
> Houston, Texas-77004
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2015/12/1225.php



Re: [hwloc-users] Assembling multiple node XMLs

2015-10-30 Thread Brice Goglin
Hello
Can you have a startup script set
HWLOC_XMLFILE=/common/path/${hostname}.xml in the system-wide environment?
Brice



Le 30/10/2015 13:57, Andrej Prsa a écrit :
> Hi Brice,
>
>> When you assemble multiple nodes' topologies into a single one, the
>> resulting topology cannot be used for binding. Binding is only
>> possible when using objects/cpusets that correspond to the current
>> node.
> Ah, that explains it.
>
>> Open-MPI does not support these cases, hence the crash. I see that
>> individual XMLs worked fine. So why did you try this?
> I was trying to figure out a way to fix buggy BIOS topologies for all
> nodes from the server side. With our current setup, all the nodes are
> diskless, they boot up an identical image from NFS and I don't know how
> to tell each one what XML file to use. The reason I know that the XML
> file fixes the problem is by logging onto that node, exporting
> HWLOC_XMLFILE and running mpirun from there. Any suggestions on how to
> do that system-wide?
>
> Thanks,
> Andrej
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2015/10/1217.php



Re: [hwloc-users] Assembling multiple node XMLs

2015-10-30 Thread Brice Goglin
Hello

When you assemble multiple nodes' topologies into a single one, the
resulting topology cannot be used for binding. Binding is only possible
when using objects/cpusets that correspond to the current node. The
assembled topology contains many objects that can't be used for binding:
objects that contain multiple nodes, and objects that don't come from
the node where the current process is running.

Open-MPI does not support these cases, hence the crash. I see that
individual XMLs worked fine. So why did you try this?

By the way, the ability to assemble multiple topologies will be removed
in 2.0.

Brice



Le 30/10/2015 02:13, Andrej Prsa a écrit :
> Hi all,
>
> I have a 6-node cluster with the buggy L3 H8QG6 AMD boards. Brice
> Goglin recently provided a fix to Fabian Wein and I applied the same
> fix (by diffing Fabian's original and Brice's fixed XML and then
> incorporating the equivalent changes to our XML). It did the trick
> perfectly, using openmpi-1.10.0 and hwloc 1.11.1. I then proceeded to
> produce a patched XML for each node in our cluster.
>
> The problem arises when I try to combine the XMLs. To test the assembly
> of just two XMLs, I ran:
>
> hwloc-assembler combo.xml \
>   --name clusty clusty_fixed.xml \
>   --name node1 node1_fixed.xml
>
> I then set HWLOC_XMLFILE to combo.xml and, when trying to mpirun a test
> program on either of the two nodes, I get a segfault:
>
> andrej@clusty:~/MPI$ mpirun -np 44 python testmpi.py 
> [clusty:19136] *** Process received signal ***
> [clusty:19136] Signal: Segmentation fault (11)
> [clusty:19136] Signal code: Address not mapped (1)
> [clusty:19136] Failing at address: (nil)
> [clusty:19136]
> [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fdf37f38340]
> [clusty:19136]
> [ 1] /usr/local/hwloc/lib/libhwloc.so.5(hwloc_bitmap_and+0x17)[0x7fdf37934e77]
> [clusty:19136]
> [ 2] 
> /opt/openmpi-1.10.0/lib/libopen-pal.so.13(opal_hwloc_base_filter_cpus+0x37c)[0x7fdf381b239c]
> [clusty:19136]
> [ 3] 
> /opt/openmpi-1.10.0/lib/libopen-pal.so.13(opal_hwloc_base_get_topology+0xcb)[0x7fdf381b412b]
> [clusty:19136]
> [ 4] /opt/openmpi-1.10.0/lib/openmpi/mca_ess_hnp.so(+0x47ea)[0x7fdf35c1c7ea]
> [clusty:19136]
> [ 5] 
> /opt/openmpi-1.10.0/lib/libopen-rte.so.12(orte_init+0x168)[0x7fdf384062b8]
> [clusty:19136] [ 6] mpirun[0x404497] [clusty:19136] [ 7]
> mpirun[0x40363d] [clusty:19136]
> [ 8] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fdf37b81ec5]
> [clusty:19136] [ 9] mpirun[0x403559] [clusty:19136] *** End of error
> message *** Segmentation fault (core dumped)
>
> Each individual XML file works (i.e. no hwloc complaints and mpirun
> works perfectly), but the assembled one does not. I'm attaching all
> three XMLs: clusty.xml, node1.xml and combo.xml. Any ideas?
>
> Thanks,
> Andrej
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-users/2015/10/1214.php



Re: [hwloc-users] hwloc error for AMD Opteron 6300 processor family

2015-10-29 Thread Brice Goglin
Le 28/10/2015 18:04, Fabian Wein a écrit :
> I hope I'm still on the right list for my current problem.

Hello
It looks like this should go to us...@open-mpi.org now.

> -
> A request was made to bind a process, but at least one node does NOT
> support binding processes to cpus.
>
>   Node:  leo
> This usually is due to not having libnumactl and libnumactl-devel
> installed on the node.
> -
>
> I cannot find these packages for ubuntu 14.04.
>
> I find a lot of ubuntu deb packages on
> https://launchpad.net/ubuntu/+source/numactl
> But there I only find libnuma but no libnumactl.
>
> Where do I get the libnumactl and libnumactl-devel from?

On Deb-based (Debian, Ubuntu etc), the right package name is
"libnuma-dev". The OMPI message only cares about RPM distros.

> Is this the wrong thread and the wrong list?

Yeah, OpenMPI specific issues should go to OpenMPI list (hwloc is a
subproject of the OpenMPI consortium, but the software projects are
pretty much independent).

Brice


> I have a feeling that I'm quite close but just cannot reach it :(
>
> Thanks,
>
> Fabian
>
>
> On 10/27/2015 04:05 PM, Brice Goglin wrote:
>> I guess the next step would be to look at how these tasks are placed on
>> the machine. There are 8 NUMA nodes on the machine. Maybe 9 is where it
>> starts placing a second task per NUMA node?
>> For OMPI, --report-bindings may help. I am not sure about MPICH.
>>
>> Brice
>>
>>
>>
>> Le 27/10/2015 15:52, Fabian Wein a écrit :
>>> On 10/27/2015 03:42 PM, Brice Goglin wrote:
>>>> I guess the problem is that your OMPI uses an old hwloc internally.
>>>> That
>>>> one may be too old to understand recent XML exports.
>>>> Try replacing "Package" with "Socket" everywhere in the XML file.
>>>
>>> Thanks! That was it.
>>>
>>> I now get almost perfectly reproducible results.
>>>
>>> np  speedup
>>> 1 1.0
>>> 2 1.99
>>> 3 2.98
>>> 4 3.98
>>> 5 4.89
>>> 6 5.9
>>> 7 6.89
>>> 8 7.87
>>> 9 5.44
>>> 10 6.04
>>> 11 6.55
>>> 12 7.0
>>> 13 7.75
>>> 14 8.24
>>> 15 8.41
>>> 16 9.4
>>> 17 7.33
>>> 18 7.16
>>> 19 8.05
>>> 20 8.39
>>>
>>> What still puzzles me is the almost perfect speedup up to eight and
>>> than the
>>> drop down. But for the beginning 8 is already good!
>>>
>>> Thanks again,
>>>
>>> Fabian
>>>
>>> ___
>>> hwloc-users mailing list
>>> hwloc-us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/hwloc-users/2015/10/1210.php
>>
>> ___
>> hwloc-users mailing list
>> hwloc-us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/hwloc-users/2015/10/1210.php
>>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
> Link to this post:
> http://www.open-mpi.org/community/lists/hwloc-users/2015/10/1212.php



  1   2   3   4   >