On Mon, Apr 21, 2008 at 02:53:51PM -0700, Roland Dreier wrote:
> > ibv_reg_mr() fails if I try to register a memory region backed by a
> > huge page, but is not aligned to huge page boundary. Digging deeper I
> > see that libibverbs aligns memory region to a regular page size and
> > calls madvise() and the call fails. See program below to reproduce.
> > The program assumes that hugetlbfs is mounted on /huge and there is at
> > least one huge page available. I am not use it is possible to know if a
> > memory buffer is backed by huge page to solve the problem.
>
> Hmm, not sure off the top of my head how we should deal with this.
Me too :(
>
> > Another issue with libibverbs is that after first ibv_reg_mr() fails the
> > second registration attempt of the same buffer succeed since
> > ibv_madvise_range() doesn't cleanup after madvice failure and thinks
> > that memory is already "madvised".
>
> I guess we shouldn't change the refcnt until after we know if madvise
> has succeeded or not. Does the patch below help? I'm not sure if this
> is a good enough fix -- we might have split up a node and want to
> remerge it if the madvise fails... rolling back is a little tricky... I
> think this will take a little more thought.
>
> - R.
>
> --- a/src/memory.c
> +++ b/src/memory.c
> @@ -506,8 +506,6 @@ static int ibv_madvise_range(void *base, size_t size, int
> advice)
> __mm_add(tmp);
> }
>
> - node->refcnt += inc;
> -
I suppose "if" below depends on updated refcnt, so update can't be moved
down without changing the "if" statement.
> if ((inc == -1 && node->refcnt == 0) ||
> (inc == 1 && node->refcnt == 1)) {
> /*
> @@ -532,6 +530,8 @@ static int ibv_madvise_range(void *base, size_t size, int
> advice)
> goto out;
> }
>
> + node->refcnt += inc;
> +
> node = __mm_next(node);
> }
>
--
Gleb.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
