Arjan van de Ven wrote:
On Mon, 2005-04-18 at 11:09 -0500, Timur Tabi wrote:
Roland Dreier wrote:
Troy> How is memory pinning handled? (I haven't had time to read
Troy> all the code, so please excuse my ignorance of something
Troy> obvious).
The userspace library calls mlock() and then
Arjan van de Ven wrote:
On Mon, 2005-04-18 at 11:09 -0500, Timur Tabi wrote:
Roland Dreier wrote:
Troy How is memory pinning handled? (I haven't had time to read
Troy all the code, so please excuse my ignorance of something
Troy obvious).
The userspace library calls mlock() and then the
Andy Isaacson <[EMAIL PROTECTED]> wrote:
> On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote:
>> I don't know if VM_REGISTERED is a good idea or not, but it should be
>> absolutely impossible for the kernel to reclaim "registered" (aka pinned)
>> memory, no matter what. For RDMA services
Hi!
> >Timur> Why do you call mlock() and get_user_pages()? In our
> >code,
> >Timur> we only call mlock(), and the memory is pinned. We have a
> >Timur> test case that fails if only get_user_pages() is called,
> >Timur> but it passes if only mlock() is called.
> >
> >What
On Thu, Apr 21, 2005 at 03:07:42PM -0500, Timur Tabi wrote:
> >*You* need to come up with a solution that looks good to *the community*
> >if you want it merged.
>
> True, but I'm not going to waste my time adding this support if the
> consensus I get from the kernel developers that they don't
On Thu, Apr 21, 2005 at 03:07:42PM -0500, Timur Tabi wrote:
*You* need to come up with a solution that looks good to *the community*
if you want it merged.
True, but I'm not going to waste my time adding this support if the
consensus I get from the kernel developers that they don't want
Hi!
Timur Why do you call mlock() and get_user_pages()? In our
code,
Timur we only call mlock(), and the memory is pinned. We have a
Timur test case that fails if only get_user_pages() is called,
Timur but it passes if only mlock() is called.
What if a
Andy Isaacson [EMAIL PROTECTED] wrote:
On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote:
I don't know if VM_REGISTERED is a good idea or not, but it should be
absolutely impossible for the kernel to reclaim registered (aka pinned)
memory, no matter what. For RDMA services (such as
On Thu, 2005-04-21 at 13:25 -0700, Chris Wright wrote:
> * Timur Tabi ([EMAIL PROTECTED]) wrote:
> > It works with every kernel I've tried. I'm sure there are plenty of kernel
> > configuration options that will break our driver. But as long as all the
> > distros our customers use work, as
* Timur Tabi ([EMAIL PROTECTED]) wrote:
> It works with every kernel I've tried. I'm sure there are plenty of kernel
> configuration options that will break our driver. But as long as all the
> distros our customers use work, as well as reasonably-configured custom
> kernels, we're happy.
>
Chris Wright wrote:
FYI, that will not work on all 2.6 kernels. Specifically anything that's
not using capabilities.
It works with every kernel I've tried. I'm sure there are plenty of kernel configuration
options that will break our driver. But as long as all the distros our customers use
* Timur Tabi ([EMAIL PROTECTED]) wrote:
> Andy Isaacson wrote:
> >Do you guys simply raise RLIMIT_MEMLOCK to allow apps to lock their
> >pages? Or are you doing something more nasty?
>
> A little more nasty. I raise RLIMIT_MEMLOCK in the driver to "unlimited"
> and also set
Andy Isaacson wrote:
I'm familiar with MPI 1.0 and 2.0, but I haven't been following the
development of modern messaging APIs, so I might not make sense here...
Assuming that the app calls into the library on a fairly regular basis,
Not really. The whole point is to have the adapter DMA the data
On Thu, Apr 21, 2005 at 01:39:35PM -0500, Timur Tabi wrote:
> Andy Isaacson wrote:
> >If you take the hardline position that "the app is the only thing that
> >matters", your code is unlikely to get merged. Linux is a
> >general-purpose OS.
>
> The problem is that our driver and library
Andy Isaacson wrote:
If you take the hardline position that "the app is the only thing that
matters", your code is unlikely to get merged. Linux is a
general-purpose OS.
The problem is that our driver and library implement an API that we don't fully control.
The API states that the application
On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote:
> Troy Benjegerdes wrote:
> >Someone (aka Tospin, infinicon, and Amasso) should probably post a patch
> >adding '#define VM_REGISTERD 0x0100', and some extensions to
> >something like 'madvise' to set pages to be registered.
> >
> >My
On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote:
Troy Benjegerdes wrote:
Someone (aka Tospin, infinicon, and Amasso) should probably post a patch
adding '#define VM_REGISTERD 0x0100', and some extensions to
something like 'madvise' to set pages to be registered.
My preference
Andy Isaacson wrote:
If you take the hardline position that the app is the only thing that
matters, your code is unlikely to get merged. Linux is a
general-purpose OS.
The problem is that our driver and library implement an API that we don't fully control.
The API states that the application
On Thu, Apr 21, 2005 at 01:39:35PM -0500, Timur Tabi wrote:
Andy Isaacson wrote:
If you take the hardline position that the app is the only thing that
matters, your code is unlikely to get merged. Linux is a
general-purpose OS.
The problem is that our driver and library implement an API
Andy Isaacson wrote:
I'm familiar with MPI 1.0 and 2.0, but I haven't been following the
development of modern messaging APIs, so I might not make sense here...
Assuming that the app calls into the library on a fairly regular basis,
Not really. The whole point is to have the adapter DMA the data
* Timur Tabi ([EMAIL PROTECTED]) wrote:
Andy Isaacson wrote:
Do you guys simply raise RLIMIT_MEMLOCK to allow apps to lock their
pages? Or are you doing something more nasty?
A little more nasty. I raise RLIMIT_MEMLOCK in the driver to unlimited
and also set cap_raise(IPC_LOCK). I do
Chris Wright wrote:
FYI, that will not work on all 2.6 kernels. Specifically anything that's
not using capabilities.
It works with every kernel I've tried. I'm sure there are plenty of kernel configuration
options that will break our driver. But as long as all the distros our customers use
* Timur Tabi ([EMAIL PROTECTED]) wrote:
It works with every kernel I've tried. I'm sure there are plenty of kernel
configuration options that will break our driver. But as long as all the
distros our customers use work, as well as reasonably-configured custom
kernels, we're happy.
Hey,
On Thu, 2005-04-21 at 13:25 -0700, Chris Wright wrote:
* Timur Tabi ([EMAIL PROTECTED]) wrote:
It works with every kernel I've tried. I'm sure there are plenty of kernel
configuration options that will break our driver. But as long as all the
distros our customers use work, as well as
Troy Benjegerdes wrote:
Someone (aka Tospin, infinicon, and Amasso) should probably post a patch
adding '#define VM_REGISTERD 0x0100', and some extensions to
something like 'madvise' to set pages to be registered.
My preference is said patch will also allow a way for the kernel to
reclaim
On Mon, Apr 18, 2005 at 10:07:12PM +0200, Bernhard Fischer wrote:
> On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote:
> >On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote:
> >> Arjan van de Ven wrote:
> >>
> >> > this is a myth; linux is free to move the page about in physical
On Mon, Apr 18, 2005 at 10:07:12PM +0200, Bernhard Fischer wrote:
On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote:
On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote:
Arjan van de Ven wrote:
this is a myth; linux is free to move the page about in physical memory
even
Troy Benjegerdes wrote:
Someone (aka Tospin, infinicon, and Amasso) should probably post a patch
adding '#define VM_REGISTERD 0x0100', and some extensions to
something like 'madvise' to set pages to be registered.
My preference is said patch will also allow a way for the kernel to
reclaim
Arjan van de Ven wrote:
you just said that you didn't care that it got munlock'd. So you don't
care that it gets freed either. And then reused.
Well, I can live with the app being able to call munlock(), because the apps that our
customers use don't call munlock(). What I can't live with is a
On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote:
>On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote:
>> Arjan van de Ven wrote:
>>
>> > this is a myth; linux is free to move the page about in physical memory
>> > even if it's mlock()ed!!
darn, yes, this is true.
I know people
On Mon, 2005-04-18 at 15:00 -0500, Timur Tabi wrote:
> Arjan van de Ven wrote:
>
> > you should since that physical page can be reused, say by a root
> > process, and you'd be majorly screwed
>
> I don't understand what you mean by "reused". The whole point behind pinning
> the memory
> is
Arjan van de Ven wrote:
you should since that physical page can be reused, say by a root
process, and you'd be majorly screwed
I don't understand what you mean by "reused". The whole point behind pinning the memory
is that it stays where it is. It doesn't get moved around and it doesn't get
On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote:
> Arjan van de Ven wrote:
>
> > this is a myth; linux is free to move the page about in physical memory
> > even if it's mlock()ed!!
>
> Then Linux has a very odd definition of the word "locked".
>
> > And even then, the user can munlock the
Libor Michalek wrote:
The problem we were seeing is that the minor fault by the app resulted
in a new physical page getting mapped for the application. The page that
had the elevated refcount was still waiting for the data to be written
to by the driver at the time that the app accessed the page
Roland Dreier wrote:
Timur> Why do you call mlock() and get_user_pages()? In our code,
Timur> we only call mlock(), and the memory is pinned. We have a
Timur> test case that fails if only get_user_pages() is called,
Timur> but it passes if only mlock() is called.
What if a
Timur> Why do you call mlock() and get_user_pages()? In our code,
Timur> we only call mlock(), and the memory is pinned. We have a
Timur> test case that fails if only get_user_pages() is called,
Timur> but it passes if only mlock() is called.
What if a buggy/malicious userspace
Christoph Hellwig wrote:
On Mon, Apr 18, 2005 at 11:22:29AM -0500, Timur Tabi wrote:
That's not what we're seeing. We have hardware that does DMA over the
network (much like the Infiniband stuff), and we have a testcase that fails
if get_user_pages() is used, but not if mlock() is used.
If you
On Mon, Apr 18, 2005 at 11:22:29AM -0500, Timur Tabi wrote:
> That's not what we're seeing. We have hardware that does DMA over the
> network (much like the Infiniband stuff), and we have a testcase that fails
> if get_user_pages() is used, but not if mlock() is used.
If you don't share your
Arjan van de Ven wrote:
this is a myth; linux is free to move the page about in physical memory
even if it's mlock()ed!!
Then Linux has a very odd definition of the word "locked".
And even then, the user can munlock the memory from another thread etc
etc. Not a good idea.
Well, that's okay,
Andrew Morton wrote:
Roland Dreier <[EMAIL PROTECTED]> wrote:
Troy> Do we even need the mlock in userspace then?
Yes, because the kernel may go through and unmap pages from userspace
while trying to swap. Since we have the page locked in the kernel,
the physical page won't go anywhere, but
On Mon, 2005-04-18 at 11:09 -0500, Timur Tabi wrote:
> Roland Dreier wrote:
> > Troy> How is memory pinning handled? (I haven't had time to read
> > Troy> all the code, so please excuse my ignorance of something
> > Troy> obvious).
> >
> > The userspace library calls mlock() and then
Roland Dreier wrote:
Troy> How is memory pinning handled? (I haven't had time to read
Troy> all the code, so please excuse my ignorance of something
Troy> obvious).
The userspace library calls mlock() and then the kernel does
get_user_pages().
Why do you call mlock() and
Roland Dreier wrote:
Troy How is memory pinning handled? (I haven't had time to read
Troy all the code, so please excuse my ignorance of something
Troy obvious).
The userspace library calls mlock() and then the kernel does
get_user_pages().
Why do you call mlock() and get_user_pages()?
On Mon, 2005-04-18 at 11:09 -0500, Timur Tabi wrote:
Roland Dreier wrote:
Troy How is memory pinning handled? (I haven't had time to read
Troy all the code, so please excuse my ignorance of something
Troy obvious).
The userspace library calls mlock() and then the kernel
Arjan van de Ven wrote:
this is a myth; linux is free to move the page about in physical memory
even if it's mlock()ed!!
Then Linux has a very odd definition of the word locked.
And even then, the user can munlock the memory from another thread etc
etc. Not a good idea.
Well, that's okay, because
Andrew Morton wrote:
Roland Dreier [EMAIL PROTECTED] wrote:
Troy Do we even need the mlock in userspace then?
Yes, because the kernel may go through and unmap pages from userspace
while trying to swap. Since we have the page locked in the kernel,
the physical page won't go anywhere, but
On Mon, Apr 18, 2005 at 11:22:29AM -0500, Timur Tabi wrote:
That's not what we're seeing. We have hardware that does DMA over the
network (much like the Infiniband stuff), and we have a testcase that fails
if get_user_pages() is used, but not if mlock() is used.
If you don't share your
Christoph Hellwig wrote:
On Mon, Apr 18, 2005 at 11:22:29AM -0500, Timur Tabi wrote:
That's not what we're seeing. We have hardware that does DMA over the
network (much like the Infiniband stuff), and we have a testcase that fails
if get_user_pages() is used, but not if mlock() is used.
If you
Timur Why do you call mlock() and get_user_pages()? In our code,
Timur we only call mlock(), and the memory is pinned. We have a
Timur test case that fails if only get_user_pages() is called,
Timur but it passes if only mlock() is called.
What if a buggy/malicious userspace
Roland Dreier wrote:
Timur Why do you call mlock() and get_user_pages()? In our code,
Timur we only call mlock(), and the memory is pinned. We have a
Timur test case that fails if only get_user_pages() is called,
Timur but it passes if only mlock() is called.
What if a
Libor Michalek wrote:
The problem we were seeing is that the minor fault by the app resulted
in a new physical page getting mapped for the application. The page that
had the elevated refcount was still waiting for the data to be written
to by the driver at the time that the app accessed the page
On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote:
Arjan van de Ven wrote:
this is a myth; linux is free to move the page about in physical memory
even if it's mlock()ed!!
Then Linux has a very odd definition of the word locked.
And even then, the user can munlock the memory from
Arjan van de Ven wrote:
you should since that physical page can be reused, say by a root
process, and you'd be majorly screwed
I don't understand what you mean by reused. The whole point behind pinning the memory
is that it stays where it is. It doesn't get moved around and it doesn't get
On Mon, 2005-04-18 at 15:00 -0500, Timur Tabi wrote:
Arjan van de Ven wrote:
you should since that physical page can be reused, say by a root
process, and you'd be majorly screwed
I don't understand what you mean by reused. The whole point behind pinning
the memory
is that it stays
On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote:
On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote:
Arjan van de Ven wrote:
this is a myth; linux is free to move the page about in physical memory
even if it's mlock()ed!!
darn, yes, this is true.
I know people who
Arjan van de Ven wrote:
you just said that you didn't care that it got munlock'd. So you don't
care that it gets freed either. And then reused.
Well, I can live with the app being able to call munlock(), because the apps that our
customers use don't call munlock(). What I can't live with is a
Roland Dreier <[EMAIL PROTECTED]> wrote:
>
> OK, I'm by no means an expert on this, but Libor and I looked at
> rmap.c a little more, and there is code:
>
> if ((vma->vm_flags & (VM_LOCKED|VM_RESERVED)) ||
> ptep_clear_flush_young(vma, address, pte)) {
>
OK, I'm by no means an expert on this, but Libor and I looked at
rmap.c a little more, and there is code:
if ((vma->vm_flags & (VM_LOCKED|VM_RESERVED)) ||
ptep_clear_flush_young(vma, address, pte)) {
ret = SWAP_FAIL;
goto out_unmap;
OK, I'm by no means an expert on this, but Libor and I looked at
rmap.c a little more, and there is code:
if ((vma-vm_flags (VM_LOCKED|VM_RESERVED)) ||
ptep_clear_flush_young(vma, address, pte)) {
ret = SWAP_FAIL;
goto out_unmap;
Roland Dreier [EMAIL PROTECTED] wrote:
OK, I'm by no means an expert on this, but Libor and I looked at
rmap.c a little more, and there is code:
if ((vma-vm_flags (VM_LOCKED|VM_RESERVED)) ||
ptep_clear_flush_young(vma, address, pte)) {
ret =
On Mon, Apr 11, 2005 at 05:13:47PM -0700, Andrew Morton wrote:
> Roland Dreier <[EMAIL PROTECTED]> wrote:
> >
> > Troy> Do we even need the mlock in userspace then?
> >
> > Yes, because the kernel may go through and unmap pages from userspace
> > while trying to swap. Since we have the page
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation
>
> Roland> Yes, because the kernel may go through and unmap pages
> Roland> from userspace while trying to swap. Since we have the
&
Quoting r. Roland Dreier [EMAIL PROTECTED]:
Subject: Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation
Roland Yes, because the kernel may go through and unmap pages
Roland from userspace while trying to swap. Since we have the
Roland page locked in the kernel
On Mon, Apr 11, 2005 at 05:13:47PM -0700, Andrew Morton wrote:
Roland Dreier [EMAIL PROTECTED] wrote:
Troy Do we even need the mlock in userspace then?
Yes, because the kernel may go through and unmap pages from userspace
while trying to swap. Since we have the page locked in the
Roland> Yes, because the kernel may go through and unmap pages
Roland> from userspace while trying to swap. Since we have the
Roland> page locked in the kernel, the physical page won't go
Roland> anywhere, but userspace might end up with a different page
Roland> mapped at the
Roland Dreier <[EMAIL PROTECTED]> wrote:
>
> Troy> Do we even need the mlock in userspace then?
>
> Yes, because the kernel may go through and unmap pages from userspace
> while trying to swap. Since we have the page locked in the kernel,
> the physical page won't go anywhere, but userspace
Troy> Do we even need the mlock in userspace then?
Yes, because the kernel may go through and unmap pages from userspace
while trying to swap. Since we have the page locked in the kernel,
the physical page won't go anywhere, but userspace might end up with a
different page mapped at the same
On Mon, Apr 11, 2005 at 09:56:53AM -0700, Roland Dreier wrote:
> Troy> Is there a check in the kernel that the memory is actually
> Troy> mlock()ed?
>
> No.
>
> Troy> What if a malicious (or broken) application does
> Troy> ibv_reg_mr() but doesn't lock the memory? Does the IB
Troy> Is there a check in the kernel that the memory is actually
Troy> mlock()ed?
No.
Troy> What if a malicious (or broken) application does
Troy> ibv_reg_mr() but doesn't lock the memory? Does the IB card
Troy> get a physical address for a page that might get swapped
On Mon, Apr 11, 2005 at 08:34:19AM -0700, Roland Dreier wrote:
> Troy> How is memory pinning handled? (I haven't had time to read
> Troy> all the code, so please excuse my ignorance of something
> Troy> obvious).
>
> The userspace library calls mlock() and then the kernel does
>
Troy> How is memory pinning handled? (I haven't had time to read
Troy> all the code, so please excuse my ignorance of something
Troy> obvious).
The userspace library calls mlock() and then the kernel does
get_user_pages().
Troy> The old mellanox drivers used to have a hack to
> In particular, the memory pinning code in in uverbs_mem.c could stand
> a looking over. In addition, a sanity check of the write()-based
> scheme for passing commands into the kernel in uverbs_main.c and
> uverbs_cmd.c is probably worthwhile.
How is memory pinning handled? (I haven't had time
In particular, the memory pinning code in in uverbs_mem.c could stand
a looking over. In addition, a sanity check of the write()-based
scheme for passing commands into the kernel in uverbs_main.c and
uverbs_cmd.c is probably worthwhile.
How is memory pinning handled? (I haven't had time to
Troy How is memory pinning handled? (I haven't had time to read
Troy all the code, so please excuse my ignorance of something
Troy obvious).
The userspace library calls mlock() and then the kernel does
get_user_pages().
Troy The old mellanox drivers used to have a hack to call
On Mon, Apr 11, 2005 at 08:34:19AM -0700, Roland Dreier wrote:
Troy How is memory pinning handled? (I haven't had time to read
Troy all the code, so please excuse my ignorance of something
Troy obvious).
The userspace library calls mlock() and then the kernel does
Troy Is there a check in the kernel that the memory is actually
Troy mlock()ed?
No.
Troy What if a malicious (or broken) application does
Troy ibv_reg_mr() but doesn't lock the memory? Does the IB card
Troy get a physical address for a page that might get swapped
Troy
On Mon, Apr 11, 2005 at 09:56:53AM -0700, Roland Dreier wrote:
Troy Is there a check in the kernel that the memory is actually
Troy mlock()ed?
No.
Troy What if a malicious (or broken) application does
Troy ibv_reg_mr() but doesn't lock the memory? Does the IB card
Troy Do we even need the mlock in userspace then?
Yes, because the kernel may go through and unmap pages from userspace
while trying to swap. Since we have the page locked in the kernel,
the physical page won't go anywhere, but userspace might end up with a
different page mapped at the same
Roland Dreier [EMAIL PROTECTED] wrote:
Troy Do we even need the mlock in userspace then?
Yes, because the kernel may go through and unmap pages from userspace
while trying to swap. Since we have the page locked in the kernel,
the physical page won't go anywhere, but userspace might end
Roland Yes, because the kernel may go through and unmap pages
Roland from userspace while trying to swap. Since we have the
Roland page locked in the kernel, the physical page won't go
Roland anywhere, but userspace might end up with a different page
Roland mapped at the same
Here is an initial implementation of InfiniBand userspace verbs. I
plan to commit this code to the OpenIB repository shortly, and submit
it for inclusion during the 2.6.13 cycle, so I am posting it early for
comments.
This code, in conjunction with the libibverbs and libmthca userspace
libraries
Here is an initial implementation of InfiniBand userspace verbs. I
plan to commit this code to the OpenIB repository shortly, and submit
it for inclusion during the 2.6.13 cycle, so I am posting it early for
comments.
This code, in conjunction with the libibverbs and libmthca userspace
libraries
82 matches
Mail list logo
