Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-22 Thread Bodo Eggert <[EMAIL PROTECTED]>
Andy Isaacson <[EMAIL PROTECTED]> wrote: > On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote: >> I don't know if VM_REGISTERED is a good idea or not, but it should be >> absolutely impossible for the kernel to reclaim "registered" (aka pinned) >> memory, no matter what. For RDMA services

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-22 Thread Greg KH
On Thu, Apr 21, 2005 at 03:07:42PM -0500, Timur Tabi wrote: > >*You* need to come up with a solution that looks good to *the community* > >if you want it merged. > > True, but I'm not going to waste my time adding this support if the > consensus I get from the kernel developers that they don't

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-22 Thread Greg KH
On Thu, Apr 21, 2005 at 03:07:42PM -0500, Timur Tabi wrote: *You* need to come up with a solution that looks good to *the community* if you want it merged. True, but I'm not going to waste my time adding this support if the consensus I get from the kernel developers that they don't want

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-22 Thread Bodo Eggert [EMAIL PROTECTED]
Andy Isaacson [EMAIL PROTECTED] wrote: On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote: I don't know if VM_REGISTERED is a good idea or not, but it should be absolutely impossible for the kernel to reclaim registered (aka pinned) memory, no matter what. For RDMA services (such as

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Arjan van de Ven
On Thu, 2005-04-21 at 13:25 -0700, Chris Wright wrote: > * Timur Tabi ([EMAIL PROTECTED]) wrote: > > It works with every kernel I've tried. I'm sure there are plenty of kernel > > configuration options that will break our driver. But as long as all the > > distros our customers use work, as

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Chris Wright
* Timur Tabi ([EMAIL PROTECTED]) wrote: > It works with every kernel I've tried. I'm sure there are plenty of kernel > configuration options that will break our driver. But as long as all the > distros our customers use work, as well as reasonably-configured custom > kernels, we're happy. >

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Timur Tabi
Chris Wright wrote: FYI, that will not work on all 2.6 kernels. Specifically anything that's not using capabilities. It works with every kernel I've tried. I'm sure there are plenty of kernel configuration options that will break our driver. But as long as all the distros our customers use

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Chris Wright
* Timur Tabi ([EMAIL PROTECTED]) wrote: > Andy Isaacson wrote: > >Do you guys simply raise RLIMIT_MEMLOCK to allow apps to lock their > >pages? Or are you doing something more nasty? > > A little more nasty. I raise RLIMIT_MEMLOCK in the driver to "unlimited" > and also set

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Timur Tabi
Andy Isaacson wrote: I'm familiar with MPI 1.0 and 2.0, but I haven't been following the development of modern messaging APIs, so I might not make sense here... Assuming that the app calls into the library on a fairly regular basis, Not really. The whole point is to have the adapter DMA the data

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Andy Isaacson
On Thu, Apr 21, 2005 at 01:39:35PM -0500, Timur Tabi wrote: > Andy Isaacson wrote: > >If you take the hardline position that "the app is the only thing that > >matters", your code is unlikely to get merged. Linux is a > >general-purpose OS. > > The problem is that our driver and library

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Timur Tabi
Andy Isaacson wrote: If you take the hardline position that "the app is the only thing that matters", your code is unlikely to get merged. Linux is a general-purpose OS. The problem is that our driver and library implement an API that we don't fully control. The API states that the application

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Andy Isaacson
On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote: > Troy Benjegerdes wrote: > >Someone (aka Tospin, infinicon, and Amasso) should probably post a patch > >adding '#define VM_REGISTERD 0x0100', and some extensions to > >something like 'madvise' to set pages to be registered. > > > >My

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Andy Isaacson
On Wed, Apr 20, 2005 at 10:07:45PM -0500, Timur Tabi wrote: Troy Benjegerdes wrote: Someone (aka Tospin, infinicon, and Amasso) should probably post a patch adding '#define VM_REGISTERD 0x0100', and some extensions to something like 'madvise' to set pages to be registered. My preference

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Timur Tabi
Andy Isaacson wrote: If you take the hardline position that the app is the only thing that matters, your code is unlikely to get merged. Linux is a general-purpose OS. The problem is that our driver and library implement an API that we don't fully control. The API states that the application

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Andy Isaacson
On Thu, Apr 21, 2005 at 01:39:35PM -0500, Timur Tabi wrote: Andy Isaacson wrote: If you take the hardline position that the app is the only thing that matters, your code is unlikely to get merged. Linux is a general-purpose OS. The problem is that our driver and library implement an API

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Timur Tabi
Andy Isaacson wrote: I'm familiar with MPI 1.0 and 2.0, but I haven't been following the development of modern messaging APIs, so I might not make sense here... Assuming that the app calls into the library on a fairly regular basis, Not really. The whole point is to have the adapter DMA the data

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Chris Wright
* Timur Tabi ([EMAIL PROTECTED]) wrote: Andy Isaacson wrote: Do you guys simply raise RLIMIT_MEMLOCK to allow apps to lock their pages? Or are you doing something more nasty? A little more nasty. I raise RLIMIT_MEMLOCK in the driver to unlimited and also set cap_raise(IPC_LOCK). I do

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Timur Tabi
Chris Wright wrote: FYI, that will not work on all 2.6 kernels. Specifically anything that's not using capabilities. It works with every kernel I've tried. I'm sure there are plenty of kernel configuration options that will break our driver. But as long as all the distros our customers use

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Chris Wright
* Timur Tabi ([EMAIL PROTECTED]) wrote: It works with every kernel I've tried. I'm sure there are plenty of kernel configuration options that will break our driver. But as long as all the distros our customers use work, as well as reasonably-configured custom kernels, we're happy. Hey,

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-21 Thread Arjan van de Ven
On Thu, 2005-04-21 at 13:25 -0700, Chris Wright wrote: * Timur Tabi ([EMAIL PROTECTED]) wrote: It works with every kernel I've tried. I'm sure there are plenty of kernel configuration options that will break our driver. But as long as all the distros our customers use work, as well as

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-20 Thread Timur Tabi
Troy Benjegerdes wrote: Someone (aka Tospin, infinicon, and Amasso) should probably post a patch adding '#define VM_REGISTERD 0x0100', and some extensions to something like 'madvise' to set pages to be registered. My preference is said patch will also allow a way for the kernel to reclaim

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-20 Thread Troy Benjegerdes
On Mon, Apr 18, 2005 at 10:07:12PM +0200, Bernhard Fischer wrote: > On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote: > >On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote: > >> Arjan van de Ven wrote: > >> > >> > this is a myth; linux is free to move the page about in physical

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-20 Thread Troy Benjegerdes
On Mon, Apr 18, 2005 at 10:07:12PM +0200, Bernhard Fischer wrote: On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote: On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote: Arjan van de Ven wrote: this is a myth; linux is free to move the page about in physical memory even

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-20 Thread Timur Tabi
Troy Benjegerdes wrote: Someone (aka Tospin, infinicon, and Amasso) should probably post a patch adding '#define VM_REGISTERD 0x0100', and some extensions to something like 'madvise' to set pages to be registered. My preference is said patch will also allow a way for the kernel to reclaim

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-18 Thread Bernhard Fischer
On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote: >On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote: >> Arjan van de Ven wrote: >> >> > this is a myth; linux is free to move the page about in physical memory >> > even if it's mlock()ed!! darn, yes, this is true. I know people

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-18 Thread Timur Tabi
Libor Michalek wrote: The problem we were seeing is that the minor fault by the app resulted in a new physical page getting mapped for the application. The page that had the elevated refcount was still waiting for the data to be written to by the driver at the time that the app accessed the page

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-18 Thread Timur Tabi
Libor Michalek wrote: The problem we were seeing is that the minor fault by the app resulted in a new physical page getting mapped for the application. The page that had the elevated refcount was still waiting for the data to be written to by the driver at the time that the app accessed the page

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-18 Thread Bernhard Fischer
On Mon, Apr 18, 2005 at 09:40:40PM +0200, Arjan van de Ven wrote: On Mon, 2005-04-18 at 11:25 -0500, Timur Tabi wrote: Arjan van de Ven wrote: this is a myth; linux is free to move the page about in physical memory even if it's mlock()ed!! darn, yes, this is true. I know people who

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-12 Thread Libor Michalek
On Mon, Apr 11, 2005 at 05:13:47PM -0700, Andrew Morton wrote: > Roland Dreier <[EMAIL PROTECTED]> wrote: > > > > Troy> Do we even need the mlock in userspace then? > > > > Yes, because the kernel may go through and unmap pages from userspace > > while trying to swap. Since we have the page

Re: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation

2005-04-12 Thread Libor Michalek
On Mon, Apr 11, 2005 at 05:13:47PM -0700, Andrew Morton wrote: Roland Dreier [EMAIL PROTECTED] wrote: Troy Do we even need the mlock in userspace then? Yes, because the kernel may go through and unmap pages from userspace while trying to swap. Since we have the page locked in the