Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Biddiscombe, John A.
- From: Brice Goglin Sent: 29 January 2019 15:39 To: Biddiscombe, John A. ; Hardware locality user list Subject: Re: [hwloc-users] unusual memory binding results Only the one in brackets is set, others are unset alternatives. If you write "madvise" in that file, it'll become &qu

Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Biddiscombe, John A.
On the 8 numa node machine $cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never is set already, so I'm not really sure what should go in there to disable it. JB -Original Message- From: Brice Goglin Sent: 29 January 2019 15:29 To: Biddiscombe, John A. ; Hardware

Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Biddiscombe, John A.
Message- From: Brice Goglin Sent: 29 January 2019 10:35 To: Biddiscombe, John A. ; Hardware locality user list Subject: Re: [hwloc-users] unusual memory binding results Crazy idea: 512 pages could be replaced with a single 2MB huge page. You're not requesting huge pages in your allocation

Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Biddiscombe, John A.
2019 10:35 To: Biddiscombe, John A. ; Hardware locality user list Subject: Re: [hwloc-users] unusual memory binding results Crazy idea: 512 pages could be replaced with a single 2MB huge page. You're not requesting huge pages in your allocation but some systems have transparent huge pages enabled

Re: [hwloc-users] unusual memory binding results

2019-01-29 Thread Biddiscombe, John A.
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 On the 8 numa node machine it sometimes gives the right answer even with 512 pages. Still baffled JB -Original Message- From: hwloc-users On Behalf Of Biddiscombe, John A. Sent: 28 January 2019 16:14 To: Brice

Re: [hwloc-users] unusual memory binding results

2019-01-28 Thread Biddiscombe, John A.
Brice >Can you print the pattern before and after thread 1 touched its pages, or even >in the middle ? >It looks like somebody is touching too many pages here. Experimenting with different threads touching one or more pages, I get unpredicatable results here on the 8 numa node device, the

Re: [hwloc-users] unusual memory binding results

2019-01-28 Thread Biddiscombe, John A.
To: Biddiscombe, John A. Cc: Hardware locality user list Subject: Re: [hwloc-users] unusual memory binding results Can you try again disabling the touching in one thread to check whether the other thread only touched its own pages? (others' status should be -2 (ENOENT)) Recent kernels have ways

Re: [hwloc-users] unusual memory binding results

2019-01-28 Thread Biddiscombe, John A.
either the syscall is wrong, or the kernel is doing something else. I welcome any suggestions on what might be wrong. Thanks for trying to help. JB -Original Message- From: Brice Goglin Sent: 26 January 2019 10:19 To: Biddiscombe, John A. Cc: Hardware locality user list Subject: Re: [hwloc-u

Re: [hwloc-users] unusual memory binding results

2019-01-25 Thread Biddiscombe, John A.
call. JB From: Brice Goglin [brice.gog...@inria.fr] Sent: 25 January 2019 21:08 To: Biddiscombe, John A. Cc: Hardware locality user list Subject: Re: [hwloc-users] unusual memory binding results Le 25/01/2019 à 14:17, Biddiscombe, John A. a écrit : > Dea

Re: [hwloc-users] unusual memory binding results

2019-01-25 Thread Biddiscombe, John A.
Dear List/Brice I experimented with disabling the memory touch on threads except for N=1,2,3,4 etc and found a problem in hwloc, which is that the function hwloc_get_area_memlocation was returning '0' when the status of the memory null move operation was -14 (#define EFAULT 14 /* Bad address

Re: [hwloc-users] unusual memory binding results

2019-01-21 Thread Biddiscombe, John A.
>One way to debug would be to disable touching in N-1 thread to check >that everything allocated in on the right node. I shall try that. Thanks ___ hwloc-users mailing list hwloc-users@lists.open-mpi.org

Re: [hwloc-users] unusual memory binding results

2019-01-21 Thread Biddiscombe, John A.
Brice Apologies, I didn't explain it very well, I do make sure that if the tile size 256*8 < 4096 (pagesize), then I double the number of tiles per page, I just wanted to keep the explanation simple. here are some code snippets to give you the flavour of it initializing the helper sruct

[hwloc-users] unusual memory binding results

2019-01-21 Thread Biddiscombe, John A.
Dear list, I'm allocating a matrix of size (say) 2048*2048 on a node with 2 numa domains and initializing the matrix by using 2 threads, one pinned on each numa domain - with the idea that I can create tiles of memory bound to each numa domain rather than having pages assigned all to one,

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-15 Thread Biddiscombe, John A.
ll things magically work if I upgrade to hwloc 2.0 etc etc Thanks JB -Original Message- From: hwloc-users [mailto:hwloc-users-boun...@lists.open-mpi.org] On Behalf Of Biddiscombe, John A. Sent: 13 November 2017 15:37 To: Hardware locality user list <hwloc-users@lists.open-mpi.org

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-13 Thread Biddiscombe, John A.
Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset The doc is wrong, flags are used, only for BY_NODESET. I actually fixed that in git very recently. Brice Le 13/11/2017 07:24, Biddiscombe, John A. a écrit : > In the documentation for get_area_memlocation it s

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-13 Thread Biddiscombe, John A.
the original alloc call? thanks JB From: hwloc-users [hwloc-users-boun...@lists.open-mpi.org] on behalf of Biddiscombe, John A. [biddi...@cscs.ch] Sent: 13 November 2017 14:59 To: Hardware locality user list Subject: Re: [hwloc-u

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-13 Thread Biddiscombe, John A.
locality user list Subject: Re: [hwloc-users] question about hwloc_set_area_membind_nodeset Use get_area_memlocation() membind() returns where the pages are *allowed* to go (anywhere) memlocation() returns where the pages are actually allocated. Brice Le 13/11/2017 06:52, Biddiscombe, John

Re: [hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-13 Thread Biddiscombe, John A.
Thank you to you both. I modified the allocator to allocate one large block using hwloc_alloc and then use one thread per numa domain to touch each page according to the tiling pattern - unfortunately, I hadn't appreciated that now hwloc_get_area_membind_nodeset always returns the full machine

[hwloc-users] question about hwloc_set_area_membind_nodeset

2017-11-11 Thread Biddiscombe, John A.
I'm allocating some large matrices, from 10k squared elements up to 40k squared per node. I'm also using membind to place pages of the matrix memory across numa nodes so that the matrix might be bound according to the kind of pattern at the end of this email - where each 1 or 0 corresponds to a

Re: [hwloc-users] HWLOC_VERSION

2017-10-30 Thread Biddiscombe, John A.
, Biddiscombe, John A. a écrit : > Dear list > > According to the release notes Add HWLOC_MEMBIND_BYNODESET flag was > added in 1.11.3 - if I protect some code with > > #if HWLOC_API_VERSION >= 0x00010b00 > then versions 1.11.0, 1.11.1, 1.11.2 still cause build failures. > &

[hwloc-users] HWLOC_VERSION

2017-10-30 Thread Biddiscombe, John A.
Dear list According to the release notes Add HWLOC_MEMBIND_BYNODESET flag was added in 1.11.3 - if I protect some code with #if HWLOC_API_VERSION >= 0x00010b00 then versions 1.11.0, 1.11.1, 1.11.2 still cause build failures. is there some VERSION flag that distinguishes between the patch

Re: [hwloc-users] BGQ question.

2014-03-28 Thread Biddiscombe, John A.
> -Original Message- > From: hwloc-users [mailto:hwloc-users-boun...@open-mpi.org] On Behalf > Of Chris Samuel > Sent: 26 March 2014 13:42 > To: Hardware locality user list > Subject: Re: [hwloc-users] BGQ question. > > On Wed, 26 Mar 2014 11:56:08 AM Biddiscombe, John A. wr

Re: [hwloc-users] BGQ question.

2014-03-25 Thread Biddiscombe, John A.
Chris > Out of interest, why on an I/O node? I'm targeting the BGQ BGAS nodes with flash cards installed. We've done tests with GPFS mounted on the flash and are trying to get comparable results with an in-house driver. JB

Re: [hwloc-users] BGQ question.

2014-03-25 Thread Biddiscombe, John A.
Goglin [mailto:brice.gog...@inria.fr] Sent: 25 March 2014 09:28 To: Biddiscombe, John A.; Hardware locality user list Subject: Re: [hwloc-users] BGQ question. Can you run hwloc-gather-topology foo and send the resulting foo.tar.bz2 ? If the tarball is too bug, feel free to send it to me in a private

Re: [hwloc-users] BGQ question.

2014-03-25 Thread Biddiscombe, John A.
for some special projects where we are trying to customise the IO. JB From: Brice Goglin [mailto:brice.gog...@inria.fr] Sent: 25 March 2014 08:43 To: Hardware locality user list; Biddiscombe, John A. Subject: Re: [hwloc-users] BGQ question. Wait, I missed the "io node" part of your first mai

Re: [hwloc-users] BGQ question.

2014-03-25 Thread Biddiscombe, John A.
with something set? Thanks JB From: hwloc-users [mailto:hwloc-users-boun...@open-mpi.org] On Behalf Of Brice Goglin Sent: 25 March 2014 08:04 To: Hardware locality user list Subject: Re: [hwloc-users] BGQ question. Le 25/03/2014 07:51, Biddiscombe, John A. a écrit : I'm compiling hwloc using

[hwloc-users] BGQ question.

2014-03-25 Thread Biddiscombe, John A.
I'm compiling hwloc using clang (bgclang++11 from ANL) to run on IO nodes af a BGQ. It seems to have compiled ok, and when I run lstopo, I get an output like this (below), which looks reasonable, but there are 15 sockets instead of 16. I'm a little worried because the first time I compiled, I