Dear concern,
I am using hwloc library to find the node where my data is allocated. My
machine(*OS: LINUX*) topology is like below:
*** Objects at level 2 (node level): total node=8
Index 0: NUMANode#0(7999MB)
Index 1: NUMANode#1(8079MB)
Index 2: NUMANode#2(8079MB)
Index 3: NUMANode#3(8079MB)
Index 4: NUMANode#4(8079MB)
Index 5: NUMANode#5(8079MB)
Index 6: NUMANode#6(8079MB)
Index 7: NUMANode#7(8079MB)
My code is as below:
------------------ code snippet-----------
*/* declaration */*
int *p1, *p2;
int a1, b1, c1, N;
N=10;
double* A; A = (double*) malloc(N * sizeof(double));
*// touch the variable*
/* default allocation of *p1 which refers to int a1 variable address */
p1 = & a1;
// touch the memory reference of *p1
*p1 = a1 +2;
*//touch the array [A] first * for (int i = 0; i < N; i++) { A[i] = 0.0; }
*//------ call hwloc functions----- *
*/* check where buffer(array: A) is allocated */*
set = hwloc_bitmap_alloc();
err = hwloc_get_area_membind_nodeset(topology, *A, N * sizeof(double)*,
set, &policy, *HWLOC_MEMBIND_STRICT*);
if (err < 0) {
fprintf(stderr, "failed to retrieve the buffer binding and policy\n");
hwloc_topology_destroy(topology);
hwloc_bitmap_free(set);
return EXIT_FAILURE;
}
*/* check the binding policy: is identical for the current operating
system. */*
printf("--->buffer(Array: A) membind [default OS binding] Policy is:= %d
\n", policy);
*/* print the corresponding NUMA nodes */*
hwloc_bitmap_asprintf(&s, set);
printf("buffer(Array: A) bound to nodeset %s with contains:\n", s);
free(s);
hwloc_bitmap_foreach_begin(i, set) {
obj = hwloc_get_numanode_obj_by_os_index(topology, i);
printf(" -->node #%u (OS index %u) with %lld bytes of memory\n",
obj->logical_index, i, (unsigned long long) obj->memory.local_memory);
} hwloc_bitmap_foreach_end();
hwloc_bitmap_free(set);
*OUTPUT: *
*Policy-->* buffer(Array: A) *membind [default OS binding] Policy is:= 1 [1
refers to *HWLOC_MEMBIND_FIRSTTOUCH
<https://www.open-mpi.org/projects/hwloc/doc/v1.11.1/a00083.php#ggac9764f79505775d06407b40f5e4661e8a979c7aa78dd32780858f30f47a72cca0>
*]*
*Nodeset --> *buffer(Array: A) bound to nodeset* 0x000000ff *with contains
*:*
node #0 (OS index 0) with 8387047424 bytes of memory
node #1 (OS index 1) with 8471617536 bytes of memory
node #2 (OS index 2) with 8471621632 bytes of memory
node #3 (OS index 3) with 8471617536 bytes of memory
node #4 (OS index 4) with 8471621632 bytes of memory
node #5 (OS index 5) with 8471617536 bytes of memory
node #6 (OS index 6) with 8471621632 bytes of memory
node #7 (OS index 7) with 8471564288 bytes of memory
*why it shows allocated memory is bound to all available nodeset..? should
it not be allocated to a specific nodeset one ..?*
*If I write as below: *
/* Get last node. */
n = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_NODE);
if (n) {
void *m;
int prev_val, next_val;
size = sizeof(int); //1024*1024;
obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NODE, n - 1);
m = *hwloc_alloc_membind_nodeset(*topology, size, *obj->nodeset,
HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_PROCESS*);
hwloc_free(topology, m, size);
m = malloc(size);
// touch the m memory
m = &prev_val;
*(int*)m = 1010;
*hwloc_set_area_membind_nodeset(*topology, m, size, *obj->nodeset,
HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_STRICT*); //HWLOC_MEMBIND_DEFAULT:= Reset
the memory allocation policy to the system default(HWLOC_MEMBIND_FIRSTTOUCH
(Linux)).
*/* check where buffer(m) is allocated */*
nodeset = hwloc_bitmap_alloc();
*hwloc_get_area_membind_nodeset(topology, m, size, nodeset, &nodepolicy,
0); *
/* check the binding policy */
printf("buffer(m) membind-ed policy is %d \n", nodepolicy);
/* print the corresponding NUMA nodes */
hwloc_bitmap_asprintf(&s, nodeset);
printf("buffer bound to nodeset %s with contains:\n", s);
free(s);
hwloc_bitmap_foreach_begin(i, nodeset) {
obj = hwloc_get_numanode_obj_by_os_index(topology, i);
printf(" node #%u (OS index %u) with %lld bytes of memory\n",
obj->logical_index, i, (unsigned long long) obj->memory.local_memory);
} hwloc_bitmap_foreach_end();
hwloc_bitmap_free(nodeset);
*OUTPUT: *
*Policy:* buffer(Array: A) membind *[default OS binding] Policy is:= 2*
*Nodeset: * buffer(Array: A) *bound to nodeset 0x00000080* with contains:
*node #7 (OS index 7) *with 8471564288 bytes of memory
In this case it shows the specific nodeset one where the memory is
allocated.
*Can you please comment and explain what is happening underneath ..?
Thanking you in advance.*
------------------------
RaJu, Rezaul Karim
Graduate Student (PhD) | Computer Science | University of Houston
Research in High Performance Computing Tools
Houston, Texas-77004
