Hi Roland,
ibv_reg_mr() fails if I try to register a memory region backed by a
huge page, but is not aligned to huge page boundary. Digging deeper I
see that libibverbs aligns memory region to a regular page size and
calls madvise() and the call fails. See program below to reproduce.
The program assumes that hugetlbfs is mounted on /huge and there is at
least one huge page available. I am not use it is possible to know if a
memory buffer is backed by huge page to solve the problem.
Another issue with libibverbs is that after first ibv_reg_mr() fails the
second registration attempt of the same buffer succeed since
ibv_madvise_range() doesn't cleanup after madvice failure and thinks
that memory is already "madvised".
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <infiniband/verbs.h>
int main()
{
int num_devs, fd;
struct ibv_device **ib_devs;
struct ibv_context *ctx;
struct ibv_pd *pd;
struct ibv_mr *mr;
char *ptr;
size_t len = 1024*1024;
ibv_fork_init();
ib_devs = ibv_get_device_list(&num_devs);
ctx = ibv_open_device(ib_devs[0]);
pd = ibv_alloc_pd(ctx);
fd = open("/huge/test", O_CREAT | O_RDWR);
remove("/huge/test");
ptr = mmap(0, 2*len, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
mr = ibv_reg_mr(pd, ptr, len, IBV_ACCESS_LOCAL_WRITE |
IBV_ACCESS_REMOTE_WRITE |
IBV_ACCESS_REMOTE_READ);
fprintf(stderr, "mr = %p\n", mr);
return 0;
}
--
Gleb.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
