Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Brice Goglin
Le 25/09/2011 21:05, Gabriele Fatigati a écrit :
> Ok,
>
> so, set_membind() merged with HWLOC_MEMBIND_BIND is useless?

It's likely the most useful memory binding case. It's similar to what
numactl --membind does for instance, very common.

> The behaviour I want to set is it possible?

I just said "you have to touch right after malloc."

Brice


>
> 2011/9/25 Brice Goglin  >
>
> Le 25/09/2011 20:57, Gabriele Fatigati a écrit :
> > after done this, memory is allocated not in a local node of thread
> > that does set_membind and malloc, but in node of  thread that
> touches
> > it. And I don't understand this behaviour :(
>
> Memory is allocated when first-touched. If there's no area-specific
> policy, it's allocated according to the policy of the thread that
> touches, not according to the policy of the one that did malloc.
> If you
> want to follow the malloc'er thread, you have to touch right after
> malloc.
>
> Brice
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org 
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>
>
>
>
> -- 
> Ing. Gabriele Fatigati
>
> HPC specialist
>
> SuperComputing Applications and Innovation Department
>
> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>
> www.cineca.it Tel:   +39 051
> 6171722
>
> g.fatigati [AT] cineca.it   
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users



Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Gabriele Fatigati
Ok,

so, set_membind() merged with HWLOC_MEMBIND_BIND is useless?

The behaviour I want to set is it possible?

2011/9/25 Brice Goglin 

> Le 25/09/2011 20:57, Gabriele Fatigati a écrit :
> > after done this, memory is allocated not in a local node of thread
> > that does set_membind and malloc, but in node of  thread that touches
> > it. And I don't understand this behaviour :(
>
> Memory is allocated when first-touched. If there's no area-specific
> policy, it's allocated according to the policy of the thread that
> touches, not according to the policy of the one that did malloc. If you
> want to follow the malloc'er thread, you have to touch right after malloc.
>
> Brice
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>



-- 
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.itTel:   +39 051 6171722

g.fatigati [AT] cineca.it


Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Brice Goglin
Le 25/09/2011 20:57, Gabriele Fatigati a écrit :
> after done this, memory is allocated not in a local node of thread
> that does set_membind and malloc, but in node of  thread that touches
> it. And I don't understand this behaviour :(

Memory is allocated when first-touched. If there's no area-specific
policy, it's allocated according to the policy of the thread that
touches, not according to the policy of the one that did malloc. If you
want to follow the malloc'er thread, you have to touch right after malloc.

Brice




Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Gabriele Fatigati
My goal is to set memory binding policy for future allocations for each
thread.

So, according to definition of set_membind function:

"Set the default memory binding policy of the current process or thread to
prefer the
NUMA node(s) near the specified cpuset."

and according to definition of HWLOC_MEMBIND_BIND flag:

"Allocate memory on the specified nodes."

Merging togheter, I expect to set memory policy  of thread/process equal to
allocate memory on local node. But in my example,

after done this, memory is allocated not in a local node of thread that does
set_membind and malloc, but in node of  thread that touches it. And I don't
understand this behaviour :(


2011/9/25 Brice Goglin 

> **
> Le 25/09/2011 20:27, Gabriele Fatigati a écrit :
>
> if(tid==0){
>
>   set_membind(HWLOCMEMBIND_BIND, node 0)
>  malloc(array)...
>
>  }
>
>  if (tid==1){
>  set_membind(HWLOCMEMBIND_BIND, node 1)
>
>   for(i...)
>   array(i)
>  }
>
>  end parallel region
>
>
>  array is allocated on node 1, not node 0 as I expected So it seems
> set_membind() of second thread influence in some manner array allocation
> using first touch.
>
>
> Why do you call set_membind() here? It's whole point is to change the
> allocation policy of the current thread. If this thread then first-touches
> some data, this data will obviously get allocated acocording to
> set_membind().
>
> If you don't want set_membind() to modify the allocation policy of the
> current thread, why do you call it?
>
> Brice
>
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>



-- 
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.itTel:   +39 051 6171722

g.fatigati [AT] cineca.it


Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Brice Goglin
Le 25/09/2011 20:27, Gabriele Fatigati a écrit :
> if(tid==0){
>
>  set_membind(HWLOCMEMBIND_BIND, node 0)
>  malloc(array)...
>
> }
>
> if (tid==1){
>  set_membind(HWLOCMEMBIND_BIND, node 1)
>
> for(i...)
>   array(i)
> }
>
> end parallel region
>
>
> array is allocated on node 1, not node 0 as I expected So it seems
> set_membind() of second thread influence in some manner array
> allocation using first touch.

Why do you call set_membind() here? It's whole point is to change the
allocation policy of the current thread. If this thread then
first-touches some data, this data will obviously get allocated
acocording to set_membind().

If you don't want set_membind() to modify the allocation policy of the
current thread, why do you call it?

Brice




Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Gabriele Fatigati
>set_membind done by thread 2 has no reference with malloc(array) done by
first thread, so why it influence the real >allocation of this array?


It doesn't.


In my example seems yes...

if(tid==0){

 set_membind(HWLOCMEMBIND_BIND, node 0)
 malloc(array)...

}

if (tid==1){
 set_membind(HWLOCMEMBIND_BIND, node 1)

for(i...)
  array(i)
}

end parallel region


array is allocated on node 1, not node 0 as I expected So it seems
set_membind() of second thread influence in some manner array allocation
using first touch.



2011/9/25 Brice Goglin 

> **
> Le 25/09/2011 12:41, Gabriele Fatigati a écrit :
>
>
>>* doing two set_area_membind on the same entire array is useless, the
> second one will overwrite the first one.
>
>  But set_area_membind is for memory in general, not for a particular
> malloc. ( Is it rigth?)
>
>
> It's for an memory area.
>
>
>  set_membind done by thread 2 has no reference with malloc(array) done by
> first thread, so why it influence the real allocation of this array?
>
>
>
>
> Brice
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>



-- 
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.itTel:   +39 051 6171722

g.fatigati [AT] cineca.it


Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Gabriele Fatigati
2011/9/25 Gabriele Fatigati 

>
>> * doing two set_area_membind on the same entire array is useless, the
> second one will overwrite the first one.
>
> But set_area_membind is for memory in general, not for a particular malloc.
> ( Is it rigth?)
>


> set_membind done by thread 2 has no reference with malloc(array) done by
> first thread.
>
>
> 2011/9/25 Brice Goglin 
>
>> **
>> Le 25/09/2011 12:19, Gabriele Fatigati a écrit :
>>
>> Hi Brice,
>>
>>  >The flag says "when the first touch occurs and the physical memory is
>> allocated for real, don't allocate on the local node (default), but >rather
>> allocate where specified by set_membind".
>>
>>  If is it already allocated for real, how set_membind() can allocate on
>> other node?
>>
>>
>> Add the MIGRATE flag.
>>
>>
>>  So, what's the difference between HWLOC_MEMBIND_BIND and
>> HWLOC_MEMBIND_FIRSTTOUCH?
>>
>>
>> First touch makes the allocation on the node local to the thread that
>> touches first (default on Linux).
>> BIND makes the allocation on the node specified in set_membind.
>>
>>
>> Doing the follow test:
>>
>>  omp parallel region
>>
>>  if(tid==0){
>>  malloc(array)...
>>  set_area_membind(HWLOCMEMBIND_BIND, node 0)
>> }
>>
>>  if (tid==1){
>>  set_area_membind(HWLOCMEMBIND_BIND, node 1)
>>
>>
>> If both set_area_membind work on the same array (not on different halves),
>> this is doubly-wrong:
>> * you have no guarantee that thread 0 has already finished doing the
>> malloc before thread 1 does set_area_membind on the buffer.
>> * doing two set_area_membind on the same entire array is useless, the
>> second one will overwrite the first one.
>>
>> Brice
>>
>>
>> ___
>> hwloc-users mailing list
>> hwloc-us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>>
>
>
>
> --
> Ing. Gabriele Fatigati
>
> HPC specialist
>
> SuperComputing Applications and Innovation Department
>
> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>
> www.cineca.itTel:   +39 051 6171722
>
> g.fatigati [AT] cineca.it
>



-- 
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.itTel:   +39 051 6171722

g.fatigati [AT] cineca.it


Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Gabriele Fatigati
>
>
> * doing two set_area_membind on the same entire array is useless, the
second one will overwrite the first one.

But set_area_membind is for memory in general, not for a particular malloc.
( Is it rigth?)  In my example, I'm doing set_area_membind for thread 0
before to do some allocations, and set_area_membind for thread 2 for futures
allocations.

set_membind done by thread 2 has no reference with malloc(array) done by
first thread, so why it influence the real allocation of this array?


2011/9/25 Brice Goglin 

> **
> Le 25/09/2011 12:19, Gabriele Fatigati a écrit :
>
> Hi Brice,
>
>  >The flag says "when the first touch occurs and the physical memory is
> allocated for real, don't allocate on the local node (default), but >rather
> allocate where specified by set_membind".
>
>  If is it already allocated for real, how set_membind() can allocate on
> other node?
>
>
> Add the MIGRATE flag.
>
>
>  So, what's the difference between HWLOC_MEMBIND_BIND and
> HWLOC_MEMBIND_FIRSTTOUCH?
>
>
> First touch makes the allocation on the node local to the thread that
> touches first (default on Linux).
> BIND makes the allocation on the node specified in set_membind.
>
>
> Doing the follow test:
>
>  omp parallel region
>
>  if(tid==0){
>  malloc(array)...
>  set_area_membind(HWLOCMEMBIND_BIND, node 0)
> }
>
>  if (tid==1){
>  set_area_membind(HWLOCMEMBIND_BIND, node 1)
>
>
> If both set_area_membind work on the same array (not on different halves),
> this is doubly-wrong:
> * you have no guarantee that thread 0 has already finished doing the malloc
> before thread 1 does set_area_membind on the buffer.
> * doing two set_area_membind on the same entire array is useless, the
> second one will overwrite the first one.
>
> Brice
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>



-- 
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.itTel:   +39 051 6171722

g.fatigati [AT] cineca.it


Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Brice Goglin
Le 25/09/2011 12:19, Gabriele Fatigati a écrit :
> Hi Brice,
>
> >The flag says "when the first touch occurs and the physical memory is
> allocated for real, don't allocate on the local node (default), but
> >rather allocate where specified by set_membind".
>
> If is it already allocated for real, how set_membind() can allocate on
> other node?

Add the MIGRATE flag.

> So, what's the difference between HWLOC_MEMBIND_BIND and
> HWLOC_MEMBIND_FIRSTTOUCH?

First touch makes the allocation on the node local to the thread that
touches first (default on Linux).
BIND makes the allocation on the node specified in set_membind.

> Doing the follow test:
>
> omp parallel region
>
> if(tid==0){
>  malloc(array)...
>  set_area_membind(HWLOCMEMBIND_BIND, node 0)
> }
>
> if (tid==1){
>  set_area_membind(HWLOCMEMBIND_BIND, node 1)

If both set_area_membind work on the same array (not on different
halves), this is doubly-wrong:
* you have no guarantee that thread 0 has already finished doing the
malloc before thread 1 does set_area_membind on the buffer.
* doing two set_area_membind on the same entire array is useless, the
second one will overwrite the first one.

Brice



Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Gabriele Fatigati
Hi Brice,

>The flag says "when the first touch occurs and the physical memory is
allocated for real, don't allocate on the local node (default), but >rather
allocate where specified by set_membind".

If is it already allocated for real, how set_membind() can allocate on other
node?

So, what's the difference between HWLOC_MEMBIND_BIND and
HWLOC_MEMBIND_FIRSTTOUCH?


>set_membind_nodeset() with BIND and the nodeset containing the nodes where
physical memory should be allocated.

Mm In my tests, I have a process that allocates a memory, thread 0 is bound
on node 0 and thread 1 on node1:

malloc(array)...

omp parallel region


if(tid==0)
 set_area_membind_nodeset( first half of array)

if (tid==1)
  set_area_membind_nodeset( second half of array)


end parallel region


for(i...)
  array(i)


Checking the free memory on two nodes, supposing array is 1 Gb, I see 500 mb
less on first node and 500 mb less on second node and it is ok, because
first touch is done by the process, but on memory already bound on specific
nodes.

Doing the follow test:

omp parallel region


if(tid==0){
 malloc(array)...
 set_area_membind(HWLOCMEMBIND_BIND, node 0)

}

if (tid==1){
 set_area_membind(HWLOCMEMBIND_BIND, node 1)

for(i...)
  array(i)
}

end parallel region

Now, checking the free memory on two nodes, I see 1 GB less on second node,
because thread 1 is doing first touch. set_membind() for first thread has no
effect, I suppose, or I dont' understand how it works, because in the second
example only first touch appears to have some effects, indipendently which
hwloc function I'm using.

Sorry, but it is quite difficult to understand .. :(




2011/9/25 Brice Goglin 

> **
> Le 25/09/2011 11:14, Gabriele Fatigati a écrit :
>
>
> I report my  questions in a different way (in the first question i did a
> mistake):
>
>
>  1) I don't understand the means of set_membind() function. Why I should
> to allocate in a node "near" my cpuset and not in my local node (where
> thread or process runs?)
>
>
> It's exactly the same. Your local node is near the cpuset that contains the
> CPUs that are close to this node.
>
>
>  2) Which is the behaviour of HWLOC_MEMBIND_BIND  flag?
>
>  From the manual:
>
>  "Allocate memory on the specified nodes."
>
>  It means that I can allocate without binding the memory?
>
>
> It's about physical memory allocation (first touch causing a fault causing
> a page to be allocated), not about virtual memory (malloc).
>
>
>  What happens if one thread allocate and another thread in another node
>  read/write for the first time this memory? In a little example I see the
> memory is allocated on the second thread, not where first thread does
> malloc().  So, when I have to use HWLOC_MEMBIND_BIND flag? Or it has nothing
> to do with binding?
>
>  If the effective allocation is done when first thread touch the memory,
> which is the means of this flag?
>
>
> The flag says "when the first touch occurs and the physical memory is
> allocated for real, don't allocate on the local node (default), but rather
> allocate where specified by set_membind".
>
>
>  2) My goal is to replicate the behaviour of set_area_membind_nodeset() in
> some manner for all futures allocation without call this function each time
> I allocate some memory. Is it possible to do this?
>
>
> set_membind_nodeset() with BIND and the nodeset containing the nodes where
> physical memory should be allocated.
>
> Brice
>
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>



-- 
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.itTel:   +39 051 6171722

g.fatigati [AT] cineca.it


Re: [hwloc-users] hwloc set membind function

2011-09-25 Thread Brice Goglin
Le 25/09/2011 11:14, Gabriele Fatigati a écrit :
>
> I report my  questions in a different way (in the first question i did
> a mistake):
>
>
> 1) I don't understand the means of set_membind() function. Why I
> should to allocate in a node "near" my cpuset and not in my local node
> (where thread or process runs?)

It's exactly the same. Your local node is near the cpuset that contains
the CPUs that are close to this node.

> 2) Which is the behaviour of HWLOC_MEMBIND_BIND  flag? 
>
> From the manual:
>
> "Allocate memory on the specified nodes."
>
> It means that I can allocate without binding the memory?

It's about physical memory allocation (first touch causing a fault
causing a page to be allocated), not about virtual memory (malloc).

> What happens if one thread allocate and another thread in another node
>  read/write for the first time this memory? In a little example I see
> the memory is allocated on the second thread, not where first thread
> does malloc().  So, when I have to use HWLOC_MEMBIND_BIND flag? Or it
> has nothing to do with binding?
>
> If the effective allocation is done when first thread touch the
> memory, which is the means of this flag?

The flag says "when the first touch occurs and the physical memory is
allocated for real, don't allocate on the local node (default), but
rather allocate where specified by set_membind".

> 2) My goal is to replicate the behaviour of set_area_membind_nodeset()
> in some manner for all futures allocation without call this function
> each time I allocate some memory. Is it possible to do this?

set_membind_nodeset() with BIND and the nodeset containing the nodes
where physical memory should be allocated.

Brice



Re: [hwloc-users] hwloc set membind function

2011-09-22 Thread Gabriele Fatigati
Hi,

some questions:

1) I don't understand the means of HWLOC_MEMBIND_BIND policy. Why I should
to allocate in a node "near" my cpuset and not in my local node ( where
thread or process runs?)


2) My goal is to replicate the behaviour of set_area_membind_nodeset() in
some manner for all futures allocation without call this function each time
I allocate some memory. Is it possible to do this?



2011/9/22 Brice Goglin 

> Le 22/09/2011 12:20, Gabriele Fatigati a écrit :
> > NUMA node(s) near the specified cpuset.
> >
> > What does   "nodes near the specified cpuset" means? The node wherethe
> > specified cpuset lives?
> > Set the default memory binding policy of the current process or thread
> > to prefer the
>
> The node near the CPU specified in the cpuset.
>
> > The first thread allocates with a malloc an array. The second thread
> > (bound on other node) initialize the array.
> >
> > The free memory on the nodes decrease only on the node where the
> > second thread is. Is it rigth?
>
> Yes.
>
> >
> >  hwloc_set_membind involves all future allocations?
> >
>
> Yes. And already allocated pages if you add the migrate flag.
>
> Brice
>
> ___
> hwloc-users mailing list
> hwloc-us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>



-- 
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.itTel:   +39 051 6171722

g.fatigati [AT] cineca.it


Re: [hwloc-users] hwloc set membind function

2011-09-22 Thread Brice Goglin
Le 22/09/2011 12:20, Gabriele Fatigati a écrit :
> NUMA node(s) near the specified cpuset.
>
> What does   "nodes near the specified cpuset" means? The node wherethe
> specified cpuset lives?
> Set the default memory binding policy of the current process or thread
> to prefer the

The node near the CPU specified in the cpuset.

> The first thread allocates with a malloc an array. The second thread
> (bound on other node) initialize the array.
>
> The free memory on the nodes decrease only on the node where the
> second thread is. Is it rigth?

Yes.

>
>  hwloc_set_membind involves all future allocations?
>

Yes. And already allocated pages if you add the migrate flag.

Brice