Hi ...
I was trying to bind each quarter of an array to 4 different NUMA nodes,
and doing as below:
*//ALLOCATION *
*obj_a = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 0);*
*array =* hwloc_alloc_membind_nodeset( topology, size, obj_a->nodeset,
HWLOC_MEMBIND_BIND, 1);
*tmp *= hwloc_alloc_membind_nodeset( topology, size, obj_a->nodeset,
HWLOC_MEMBIND_BIND, 1);
*// DISTRIBUTED BINDING [my system has 8 NUMA nodes (0-7)]*
printf("Address of array:= %p, array+size/4:= %p, array+size/2:= %p,
array+3*size/4:= %p \n", array, array+size/4, array+size/2, array+3*size/4);
// bind 1st quarter to node (n-1)
hwloc_set_area_membind_nodeset(topology, (array), size/4, obj_a->nodeset,
HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
hwloc_set_area_membind_nodeset(topology, (tmp), size/4, obj_a->nodeset,
HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
// bind 2nd quarter to node (2)
*obj_b = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 2);*
hwloc_set_area_membind_nodeset(topology, (array+size/4), size/4,
obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
hwloc_set_area_membind_nodeset(topology, (tmp +size/4), size/4,
obj_b->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
// bind 3rd quarter to node (4)
* obj_c = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 4);*
hwloc_set_area_membind_nodeset(topology, array+size/2, size/4,
obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
hwloc_set_area_membind_nodeset(topology, tmp+size/2, size/4,
obj_c->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
// bind 4th quarter to node (6)
* obj_d = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, 6);*
hwloc_set_area_membind_nodeset(topology, array+3*size/4, size/4,
obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
hwloc_set_area_membind_nodeset(topology, tmp+3*size/4, size/4,
obj_d->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_MIGRATE);
My intention here is to distribute 'array' (which is - long type element:
array = (ELM *) malloc(bots_arg_size * sizeof(ELM));
tmp = (ELM *) malloc(bots_arg_size * sizeof(ELM));) over nodes through
hwloc memory binding.
1). But except only *obj_a, it can not bind the specified memory section
(addr, len) to the desired NUMA node. *
2). I did tried with MEMBIND_INTERLEAVE policy
array = hwloc_alloc_membind_nodeset(topology, size, cset_available,
HWLOC_MEMBIND_INTERLEAVE, HWLOC_MEMBIND_MIGRATE);
tmp = hwloc_alloc_membind_nodeset(topology, size, cset_available,
HWLOC_MEMBIND_INTERLEAVE, HWLOC_MEMBIND_MIGRATE);
but I did get it working here as well.
*Can you please comment on this..? *
Thank you very much in advance..!!
- Raju
On Mon, Mar 21, 2016 at 11:25 AM, Rezaul Karim Raju <raju.cse.b...@gmail.com
> wrote:
> Thanks, Brice.!
>
> On Mon, Mar 21, 2016 at 11:22 AM, Brice Goglin <brice.gog...@inria.fr>
> wrote:
>
>> For testing, you can use this tarball:
>>
>> https://ci.inria.fr/hwloc/job/zcustombranch-0-tarball/lastSuccessfulBuild/artifact/hwloc-getmemlocation-20160320.2208.gitd2f6537.tar.gz
>>
>>
>>
>>
>> Le 21/03/2016 17:21, Rezaul Karim Raju a écrit :
>>
>> Hi Brice,
>>
>> Thanks for your email.
>> I believe it is definitely helpful. Getting memory range within the
>> current process will be very good information to drill down.
>> Let me use this and I will get back if any clarification/comment I have.
>>
>> Regards-
>> Raju
>>
>> On Sun, Mar 20, 2016 at 4:26 PM, Brice Goglin <brice.gog...@inria.fr>
>> wrote:
>>
>>> I just pushed a proposal, see
>>> https://github.com/open-mpi/hwloc/issues/97
>>>
>>> Brice
>>>
>>>
>>>
>>>
>>> Le 18/12/2015 20:45, Brice Goglin a écrit :
>>>
>>> Yes, we're "thinking" about it. But there are open questions as
>>> mentioned in the github issue.
>>> By the way, we wouldn't return NULL in case of non-physically-allocated
>>> buffer, but rather set the output nodeset to 0.
>>> You should comment on the issue directly, instead of replying here,
>>> otherwise your comments may get lost.
>>>
>>> Brice
>>>
>>>
>>>
>>>
>>> Le 18/12/2015 18:57, Rezaul Karim Raju a écrit :
>>>
>>> Hi Brice,
>>>
>>> Thanks for your time and nice explanation.
>>> I have looked at the issue with location return (the page proportion
>>> across multiple node & physical allocation). Are you thinking to add this
>>> function..? Like if we think list of node or nodes where the array is
>>> allocated (only if physically allocated otherwise NULL) is it possible..?
>>>
>>> I am looking for getting the physical location of data allocated by OS
>>> default policy. Appreciate any better idea and please share with me.
>>>
>>> Best Regards,
>>> - Raju
>>>
>>> On Tue, Dec 15, 2015 at 3:28 AM, Brice Goglin < <brice.gog...@inria.fr>
>>> brice.gog...@inria.fr> wrote:
>>>
>>>>
>>>>
>>>> Le 15/12/2015 07:21, Brice Goglin a écrit :
>>>>
>>>>
>>>>
>>>> Le 15/12/2015 05:57, Rezaul Karim Raju a écrit :
>>>>
>>>> *OUTPUT: *
>>>> *Policy-->* buffer(Array: A) *membind [default OS binding] Policy is:=
>>>> 1 [1 refers to *HWLOC_MEMBIND_FIRSTTOUCH
>>>> <https://www.open-mpi.org/projects/hwloc/doc/v1.11.1/a00083.php#ggac9764f79505775d06407b40f5e4661e8a979c7aa78dd32780858f30f47a72cca0>
>>>> *]*
>>>> *Nodeset --> *buffer(Array: A) bound to nodeset* 0x000000ff *with
>>>> contains*:*
>>>> node #0 (OS index 0) with 8387047424 bytes of memory
>>>> node #1 (OS index 1) with 8471617536 bytes of memory
>>>> node #2 (OS index 2) with 8471621632 bytes of memory
>>>> node #3 (OS index 3) with 8471617536 bytes of memory
>>>> node #4 (OS index 4) with 8471621632 bytes of memory
>>>> node #5 (OS index 5) with 8471617536 bytes of memory
>>>> node #6 (OS index 6) with 8471621632 bytes of memory
>>>> node #7 (OS index 7) with 8471564288 bytes of memory
>>>>
>>>> *why it shows allocated memory is bound to all available nodeset..?
>>>> should it not be allocated to a specific nodeset one ..?*
>>>>
>>>>
>>>> Hello
>>>>
>>>> You are confusing the "binding" and the "actual location". Your memory
>>>> buffer isn't bound to a specific location by default. But Linux has to
>>>> allocate it somewhere. So your buffer is "located" in some node after the
>>>> allocation, but it is not "bound" there (what get_area_membind returns)
>>>> which means Linux could have allocated it somewhere else.
>>>>
>>>> hwloc cannot currently return the "location" of a memory buffer. I have
>>>> been thinking about adding this feature in the past, but it looks like you
>>>> are the first actual user requesting this. We could add something like
>>>> hwloc_get_last_memory_location(topology, input buffer, outputnodeset)
>>>> At least on Linux that's possible.
>>>>
>>>> For clarity, this is similar to the difference between
>>>> hwloc_get_cpubind() and hwloc_get_last_cpu_location(): A task always runs
>>>> on a specific PU, even if it is not bound to anything specific PU.
>>>>
>>>>
>>>> By the way, there is already an issue for this:
>>>> https://github.com/open-mpi/hwloc/issues/97
>>>>
>>>> Feel to comment there.
>>>>
>>>> Brice
>>>>
>>>>
>>>> _______________________________________________
>>>> hwloc-users mailing list
>>>> hwloc-us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
>>>> Link to this post:
>>>> <http://www.open-mpi.org/community/lists/hwloc-users/2015/12/1226.php>
>>>> http://www.open-mpi.org/community/lists/hwloc-users/2015/12/1226.php
>>>>
>>>
>>>
>>>
>>> --
>>> ------------------------
>>> RaJu, Rezaul Karim
>>> Graduate Student (PhD) | Computer Science | University of Houston
>>> Research in High Performance Computing Tools
>>> Houston, Texas-77004
>>>
>>>
>>>
>>>
>>
>>
>> --
>> ------------------------
>> RaJu, Rezaul Karim
>> Graduate Student (PhD) | Computer Science | University of Houston
>> Research in High Performance Computing Tools
>> Houston, Texas-77004
>>
>>
>>
>
>
> --
> ------------------------
> RaJu, Rezaul Karim
> Graduate Student (PhD) | Computer Science | University of Houston
> Research in High Performance Computing Tools
> Houston, Texas-77004
>
--
------------------------
RaJu, Rezaul Karim
Graduate Student (PhD) | Computer Science | University of Houston
Research in High Performance Computing Tools
Houston, Texas-77004
