Re: Infrastructure for zerocopy I/O

On Tue, 8 Dec 2015, Steinar H. Gunderson wrote: > Do you know what the status is for the LPM blacklisting patch we discussed > earlier? Last I heard, Baolu Lu was working on a more general solution. In its absence, maybe I should go ahead and submit the patch. Alan Stern -- To unsubscribe

Re: Infrastructure for zerocopy I/O

On Mon, 7 Dec 2015, Steinar H. Gunderson wrote: > On Mon, Dec 07, 2015 at 10:45:55AM -0500, Alan Stern wrote: > >> Here we are. Lightly tested, I believe all comments should be addressed. > > This looks quite good. I have only a couple of comments. > > Excellent. I'm sorry we've needed so many

Re: Infrastructure for zerocopy I/O

On Tue, Dec 08, 2015 at 11:07:55PM +0100, Steinar H. Gunderson wrote: > On Tue, Dec 08, 2015 at 05:01:46PM -0500, Alan Stern wrote: > > I don't see anything else to change. You can submit this to Greg KH > > and add: > > > > Acked-by: Alan Stern > > Great! > > How

Re: Infrastructure for zerocopy I/O

On Tue, 8 Dec 2015, Steinar H. Gunderson wrote: > On Tue, Dec 08, 2015 at 03:26:28PM -0500, Alan Stern wrote: > > I just noticed this -- usbm->memlist needs to be initialized before > > dec_usb_memory_use_count() can be called. INIT_LIST_HEAD before the > > "if" statement will fix the problem. >

Re: Infrastructure for zerocopy I/O

On Tue, Dec 08, 2015 at 05:01:46PM -0500, Alan Stern wrote: > I don't see anything else to change. You can submit this to Greg KH > and add: > > Acked-by: Alan Stern Great! How do I submit? Just do git send-email to some address? Do you know what the status is for

Re: Infrastructure for zerocopy I/O

On Tue, Dec 08, 2015 at 03:26:28PM -0500, Alan Stern wrote: > I just noticed this -- usbm->memlist needs to be initialized before > dec_usb_memory_use_count() can be called. INIT_LIST_HEAD before the > "if" statement will fix the problem. Done. > The patch could use one more ingredient. In >

Re: Infrastructure for zerocopy I/O

On Sun, 6 Dec 2015, Steinar H. Gunderson wrote: > On Sun, Dec 06, 2015 at 01:06:08AM +0100, Steinar H. Gunderson wrote: > > I'll try to update your patch with the other suggestions tomorrow. Thanks! > > Here we are. Lightly tested, I believe all comments should be addressed. This looks quite

Re: Infrastructure for zerocopy I/O

On Mon, Dec 07, 2015 at 10:45:55AM -0500, Alan Stern wrote: >> Here we are. Lightly tested, I believe all comments should be addressed. > This looks quite good. I have only a couple of comments. Excellent. I'm sorry we've needed so many rounds; this is what happens when you pick up someone

Re: Infrastructure for zerocopy I/O

On Sun, 6 Dec 2015, Steinar H. Gunderson wrote: > >>> You don't really need to do it earlier. An unnecessary calculation of > >>> num_sgs won't hurt. > >> Is it unnecessary? I thought the test was really to force num_sgs==0 for > >> the > >> DMA case, not to save a little arithmetic. > > Well,

Re: Infrastructure for zerocopy I/O

On Sun, Dec 06, 2015 at 01:06:08AM +0100, Steinar H. Gunderson wrote: > I'll try to update your patch with the other suggestions tomorrow. Thanks! Here we are. Lightly tested, I believe all comments should be addressed. /* Steinar */ >From 73833276a4f359c35edffc2a741dba57f2e82a12 Mon Sep 17

Re: Infrastructure for zerocopy I/O

On Sat, Dec 05, 2015 at 12:36:51PM -0500, Alan Stern wrote: > I meant that this "if" statement should test only uurb_start. If the > test succeeds, a nested "if" statement should test buffer_length and > return an error if it is too big. > > That's as opposed to the way you have it now, where

Re: Infrastructure for zerocopy I/O

On Wed, Dec 02, 2015 at 10:54:52AM -0500, Alan Stern wrote: > Well, it's partly a matter of taste. But there's also a race: This > code adds usbm to ps->memory_list (making it available to URB > submissions running on other CPUs) before usbm has been completely > initialized (vm_start isn't

Re: Infrastructure for zerocopy I/O

On Sat, 5 Dec 2015, Steinar H. Gunderson wrote: > > (Yes, I did.) On further thought, testing uurb_start is sufficient. > > If uurb->buffer_length then turns out to be too big, the function > > should return an error (or rather, an ERR_PTR) and the URB submission > > should fail. > > I don't

Re: Infrastructure for zerocopy I/O

On Sat, Dec 05, 2015 at 05:12:03PM -0500, Alan Stern wrote: > To tell the truth, I'm not sure whether it would be a problem or not. > That's why I said it "may" not be a good idea. Fair enough. >>> You don't really need to do it earlier. An unnecessary calculation of >>> num_sgs won't hurt.

Re: Infrastructure for zerocopy I/O

On Sat, 5 Dec 2015, Steinar H. Gunderson wrote: > On Sat, Dec 05, 2015 at 12:36:51PM -0500, Alan Stern wrote: > > I meant that this "if" statement should test only uurb_start. If the > > test succeeds, a nested "if" statement should test buffer_length and > > return an error if it is too big.

Re: Infrastructure for zerocopy I/O

On Thu, Dec 03, 2015 at 04:31:35PM -0500, Alan Stern wrote: >> Seemingly controller is already a pointer, so the & is redundant. No idea why >> it didn't crash plain out. Fixing that, I can allocate, although I have some >> other bug causing an oops somewhere else. > Ah, an easy mistake to miss.

Re: Infrastructure for zerocopy I/O

On Fri, Dec 04, 2015 at 01:29:05AM +0100, Steinar H. Gunderson wrote: > I must be doing something wrong, because I don't seem to get any frames > from my video capture, and when I close the application, I get a slowpath > warning: OK, the slowpath warning is a red herring -- it's just because I

Re: Infrastructure for zerocopy I/O

On Fri, 4 Dec 2015, Steinar H. Gunderson wrote: > On Fri, Dec 04, 2015 at 01:29:05AM +0100, Steinar H. Gunderson wrote: > > I must be doing something wrong, because I don't seem to get any frames > > from my video capture, and when I close the application, I get a slowpath > > warning: > > OK,

Re: Infrastructure for zerocopy I/O

On Thu, 3 Dec 2015, Steinar H. Gunderson wrote: > On Wed, Dec 02, 2015 at 10:54:52AM -0500, Alan Stern wrote: > >> [ 28.796244] DMAR: Allocating domain for 2-2 failed > > I don't know what the reason is for that. It may be that your kernel > > isn't configured to allocate as much coherent

Re: Infrastructure for zerocopy I/O

On Thu, 3 Dec 2015, Steinar H. Gunderson wrote: > On Thu, Dec 03, 2015 at 10:37:27AM -0500, Alan Stern wrote: > >> FWIW, I tried against >dev->bus->controller, and it didn't give me any > >> error messages, but still returns NULL. > > I'm at a loss. > > Seemingly controller is already a

Re: Infrastructure for zerocopy I/O

On Thu, Dec 03, 2015 at 10:37:27AM -0500, Alan Stern wrote: >> FWIW, I tried against >dev->bus->controller, and it didn't give me any >> error messages, but still returns NULL. > I'm at a loss. Seemingly controller is already a pointer, so the & is redundant. No idea why it didn't crash plain

Re: Infrastructure for zerocopy I/O

On Thu, Dec 03, 2015 at 10:18:20PM +0100, Steinar H. Gunderson wrote: > But you still think usb_alloc_coherent() is the better way to go? Should be > easy enough to change, just let me know. I must be doing something wrong, because I don't seem to get any frames from my video capture, and when I

Re: Infrastructure for zerocopy I/O

On Tue, 1 Dec 2015, Steinar H. Gunderson wrote: > >> + usbm->mem = mem; > >> + usbm->offset = virt_to_phys(mem); > >> + usbm->size = size; > >> + usbm->ps = ps; > >> + spin_lock_irqsave(>lock, flags); > >> + list_add_tail(>memlist, >memory_list); > >> + spin_unlock_irqrestore(>lock,

Re: Infrastructure for zerocopy I/O

On Wed, Dec 02, 2015 at 10:54:52AM -0500, Alan Stern wrote: >> [ 28.796244] DMAR: Allocating domain for 2-2 failed > I don't know what the reason is for that. It may be that your kernel > isn't configured to allocate as much coherent memory as you are asking > for. We'll have to

Re: Infrastructure for zerocopy I/O

On Thu, Dec 03, 2015 at 12:51:14AM +0100, Steinar H. Gunderson wrote: > I'm thinking; maybe should the memory not be allocated against the USB > device, but against the controller it hangs on? (Note that it complains about > allocating the _domain_, not the memory itself.) Do you know if that's >

Re: Infrastructure for zerocopy I/O

On Mon, Nov 30, 2015 at 12:14:59PM -0500, Alan Stern wrote: > As I recall, Markus's original patch took care of this by checking to > see whether the transfer buffer was in one of the mmap'ed areas. If it > was then the transfer would be zerocopy; otherwise it would be normal. > That seems like

Re: Infrastructure for zerocopy I/O

On Mon, 2015-11-30 at 12:14 -0500, Alan Stern wrote: > > +static struct usb_memory * > > +find_memory_area(struct usb_dev_state *ps, const struct > usbdevfs_urb *uurb) > > +{ > > + struct usb_memory *usbm = NULL, *iter; > > + unsigned long flags; > > + unsigned long uurb_start =

Re: Infrastructure for zerocopy I/O

On Mon, 30 Nov 2015, Oliver Neukum wrote: > On Mon, 2015-11-30 at 12:14 -0500, Alan Stern wrote: > > > +static struct usb_memory * > > > +find_memory_area(struct usb_dev_state *ps, const struct > > usbdevfs_urb *uurb) > > > +{ > > > + struct usb_memory *usbm = NULL, *iter; > > > +

Re: Infrastructure for zerocopy I/O

On Fri, 2015-11-27 at 00:04 +0100, Steinar H. Gunderson wrote: > On Thu, Nov 26, 2015 at 01:26:32AM +0100, Steinar H. Gunderson wrote: > >> There are numerous smaller issues: The allocated memory needs to be > >> charged against usbfs_memory_usage > > I'll fix this. > > Fixed in updated patch

Re: Infrastructure for zerocopy I/O

On Thu, 26 Nov 2015, Steinar H. Gunderson wrote: > On Wed, Nov 25, 2015 at 10:29:53AM -0500, Alan Stern wrote: > > I want to see a modified version of your patch. Several things need to > > be changed or fixed, but the major change needs to be the way memory is > > allocated. It should be

Re: Infrastructure for zerocopy I/O

On Thu, Nov 26, 2015 at 01:26:32AM +0100, Steinar H. Gunderson wrote: >> There are numerous smaller issues: The allocated memory needs to be >> charged against usbfs_memory_usage > I'll fix this. Fixed in updated patch (attached). > I'll fix this, too. Also fixed. Now also checkpatch clean.

Re: Infrastructure for zerocopy I/O

On Wed, 25 Nov 2015, Markus Rechberger wrote: > Seems like it's getting stuck again, since we're entering devices with > a higher bandwidth need now it seems to be necessary to really fix it > in the kernel. > > I propose add it as it is, it's proven that it works and there are 2 > testcases out

Re: Infrastructure for zerocopy I/O

On Wed, Nov 25, 2015 at 10:29:53AM -0500, Alan Stern wrote: > I want to see a modified version of your patch. Several things need to > be changed or fixed, but the major change needs to be the way memory is > allocated. It should be done as part of the mmap system call, not as a > separate

Re: Infrastructure for zerocopy I/O

On Wed, Nov 25, 2015 at 10:29:53AM -0500, Alan Stern wrote: > I want to see a modified version of your patch. Several things need to > be changed or fixed, but the major change needs to be the way memory is > allocated. It should be done as part of the mmap system call, not as a > separate

Re: Infrastructure for zerocopy I/O

Seems like it's getting stuck again, since we're entering devices with a higher bandwidth need now it seems to be necessary to really fix it in the kernel. I propose add it as it is, it's proven that it works and there are 2 testcases out there right now and it decreases the CPU load on intel

Re: Infrastructure for zerocopy I/O

On Wed, 18 Nov 2015, Peter Stuge wrote: > Alan Stern wrote: > > if we can directly map the user-provided buffer for DMA. > > Given a memory buffer either kernel or userspace has to apply the > hardware constraints to find out what part if any is usable for DMA. > > I think it's fine to push

Re: Infrastructure for zerocopy I/O

"Steinar H. Gunderson" writes: > On Tue, Nov 17, 2015 at 03:16:55PM -0500, Alan Stern wrote: >> xHCI always uses 64-bit addresses. But many EHCI controllers don't, >> and only a few of the EHCI platform drivers support 64-bit DMA. > > OK, sure. But are systems with USB2

Re: Infrastructure for zerocopy I/O

On Wed, Nov 18, 2015 at 3:02 PM, Christoph Hellwig wrote: > On Tue, Nov 17, 2015 at 02:07:58PM -0500, Alan Stern wrote: >> If we really want to do zerocopy I/O then we should not use a bounce >> buffer. But the only way to avoid bounce buffers may be to give user >> programs a way

Re: Infrastructure for zerocopy I/O

On Tue, Nov 17, 2015 at 02:07:58PM -0500, Alan Stern wrote: > If we really want to do zerocopy I/O then we should not use a bounce > buffer. But the only way to avoid bounce buffers may be to give user > programs a way of allocating memory pages below 4 GB, because lots of > USB hardware can

RE: Infrastructure for zerocopy I/O

From: Christoph Hellwig > Sent: 18 November 2015 14:02 > On Tue, Nov 17, 2015 at 02:07:58PM -0500, Alan Stern wrote: > > If we really want to do zerocopy I/O then we should not use a bounce > > buffer. But the only way to avoid bounce buffers may be to give user > > programs a way of allocating

Re: Infrastructure for zerocopy I/O

By the way QNAP NAS systems are shipped with a 64bit kernel but a 32bit system environment. Those systems support USB 2.0 and USB 3.0. You can expect any kind of combination out there. On Wed, Nov 18, 2015 at 7:23 PM, Bjørn Mork wrote: > "Steinar H. Gunderson"

Re: Infrastructure for zerocopy I/O

On Mon, Nov 16, 2015 at 03:22:06PM -0500, Alan Stern wrote: > In other words, you're suggesting we do this: > > Check that userspace requested zerocopy (otherwise the user > program might try to access other data stored in the same cache > lines as the buffer while the I/O is

Re: Infrastructure for zerocopy I/O

On Tue, 2015-11-17 at 13:07 +0100, Steinar H. Gunderson wrote: > On Mon, Nov 16, 2015 at 03:22:06PM -0500, Alan Stern wrote: > > Check that userspace requested zerocopy (otherwise the user > > program might try to access other data stored in the same cache > > lines as the buffer

Re: Infrastructure for zerocopy I/O

On Tue, 17 Nov 2015, Christoph Hellwig wrote: > On Mon, Nov 16, 2015 at 03:22:06PM -0500, Alan Stern wrote: > > In other words, you're suggesting we do this: > > > > Check that userspace requested zerocopy (otherwise the user > > program might try to access other data stored in the same

Re: Infrastructure for zerocopy I/O

On Mon, Nov 16, 2015 at 03:22:06PM -0500, Alan Stern wrote: > Check that userspace requested zerocopy (otherwise the user > program might try to access other data stored in the same cache > lines as the buffer while the I/O is in progres); Needed for send only, right? For

Re: Infrastructure for zerocopy I/O

There are 2 problems with the current implementation: 1. the memset on isochronous transfers to empty the buffers in order to avoid leaking raw memory to userspace (this costs a lot on intel Atoms and is also noticeable on other systems). 2. the memory fragmentation. Seems like recent systems

Re: Infrastructure for zerocopy I/O

On Tue, Nov 17, 2015 at 07:17:31PM +0100, Markus Rechberger wrote: > 1. the memset on isochronous transfers to empty the buffers in order > to avoid leaking raw memory to userspace (this costs a lot on intel > Atoms and is also noticeable on other systems). > > 2. the memory fragmentation. Seems

Re: Infrastructure for zerocopy I/O

On Tue, 17 Nov 2015, Markus Rechberger wrote: > There are 2 problems with the current implementation: > > 1. the memset on isochronous transfers to empty the buffers in order > to avoid leaking raw memory to userspace (this costs a lot on intel > Atoms and is also noticeable on other systems).

Re: Infrastructure for zerocopy I/O

Just a side note, for older videodevices it was also common to allocate the DMA memory during bootup because it might have failed during runtime because of memory fragmentation. The reason for the memset on the isochronous transfer is that USB might not fill up the entire buffer leaving half of

Re: Infrastructure for zerocopy I/O

Hi, Markus Rechberger writes: > Such kind of security is a little bit odd for a stack that doesn't > work nearly as well as Mac or Windows. And you say this based on what, exactly ? Linux supports many more devices, it's usually faster with most devices (actually Mac's

Re: Infrastructure for zerocopy I/O

On Tue, Nov 17, 2015 at 02:07:58PM -0500, Alan Stern wrote: > Is there an API for allocating user memory below 4 GB? Would a new > MMAP flag be acceptable? MAP_32BIT (to mmap) can seemingly do this, but from the man page, it sounds more like a kludge than anything else. /* Steinar */ --

Re: Infrastructure for zerocopy I/O

On Tue, 17 Nov 2015, Steinar H. Gunderson wrote: > On Tue, Nov 17, 2015 at 02:13:49PM -0500, Alan Stern wrote: > > But what other way of allocating memory is there? > > For my part, GPU memory versus malloc(). (You can ask OpenGL to permanently > map up a chunk of GPU memory for you into

Re: Infrastructure for zerocopy I/O

By the way on MacOSX we also pre-allocate some buffer in userspace first for everything. Well it's up to you anyway I haven't seen a move into a good direction in that area for years anyway on Linux so I'm used to deal with what is currently available. On Tue, Nov 17, 2015 at 8:13 PM, Alan Stern

Re: Infrastructure for zerocopy I/O

On Tue, Nov 17, 2015 at 02:13:49PM -0500, Alan Stern wrote: > But what other way of allocating memory is there? For my part, GPU memory versus malloc(). (You can ask OpenGL to permanently map up a chunk of GPU memory for you into userspace, but you have no guarantees as of if it's DMA-able. But

Re: Infrastructure for zerocopy I/O

Alan Stern wrote: > if we can directly map the user-provided buffer for DMA. Given a memory buffer either kernel or userspace has to apply the hardware constraints to find out what part if any is usable for DMA. I think it's fine to push this task to userspace - as long as userspace has a way to

Re: Infrastructure for zerocopy I/O

On Tue, 2015-11-17 at 14:13 -0500, Alan Stern wrote: > On Tue, 17 Nov 2015, Steinar H. Gunderson wrote: > > > Alan, could we perhaps let the zerocopy flag make the request fail (instead > > of going through a bounce buffer) if direct DMA is not possible? That way, > > it would be quite obvious

Re: Infrastructure for zerocopy I/O

On Tue, 17 Nov 2015, Steinar H. Gunderson wrote: > On Tue, Nov 17, 2015 at 07:17:31PM +0100, Markus Rechberger wrote: > > 1. the memset on isochronous transfers to empty the buffers in order > > to avoid leaking raw memory to userspace (this costs a lot on intel > > Atoms and is also noticeable

Re: Infrastructure for zerocopy I/O

On Tue, 17 Nov 2015, Steinar H. Gunderson wrote: > On Tue, Nov 17, 2015 at 02:07:58PM -0500, Alan Stern wrote: > > Is there an API for allocating user memory below 4 GB? Would a new > > MMAP flag be acceptable? > > MAP_32BIT (to mmap) can seemingly do this, but from the man page, it sounds >

Re: Infrastructure for zerocopy I/O

On Tue, 17 Nov 2015, Steinar H. Gunderson wrote: > > xHCI always uses 64-bit addresses. But many EHCI controllers don't, > > and only a few of the EHCI platform drivers support 64-bit DMA. > > OK, sure. But are systems with USB2 only and more than 4GB of RAM common? > (And will they not

Re: Infrastructure for zerocopy I/O

On Mon, 16 Nov 2015, Christoph Hellwig wrote: > On Mon, Nov 16, 2015 at 08:03:12PM +0100, Steinar H. Gunderson wrote: > > The original use case: DVB capture on embedded devices. > > > > My use case: USB3 uncompressed HD video capture (from multiple cards). > > > > Both of these hit (beyond the

Re: Infrastructure for zerocopy I/O

On Tue, Nov 17, 2015 at 03:01:39PM -0500, Alan Stern wrote: >> On Tue, Nov 17, 2015 at 02:07:58PM -0500, Alan Stern wrote: >>> Is there an API for allocating user memory below 4 GB? Would a new >>> MMAP flag be acceptable? >> MAP_32BIT (to mmap) can seemingly do this, but from the man page, it

Re: Infrastructure for zerocopy I/O

On Tue, Nov 17, 2015 at 03:16:55PM -0500, Alan Stern wrote: >>> But what other way of allocating memory is there? >> For my part, GPU memory versus malloc(). (You can ask OpenGL to permanently >> map up a chunk of GPU memory for you into userspace, but you have no >> guarantees as of if it's

Infrastructure for zerocopy I/O

Christoph: Can you help answer some questions about implementing zerocopy I/O? We would like to add zerocopy support for USB devices. Basically this means: Allocating memory pages that match the device's DMA mask, for use as I/O buffers, and locking them so their physical

Re: Infrastructure for zerocopy I/O

On Mon, Nov 16, 2015 at 01:29:58PM -0500, Alan Stern wrote: > A proposed patch has been posted > (http://marc.info/?l=linux-usb=144763502829452=2), but I'm not > convinced that it is the best approach. For instance, it always tries > to get contiguous pages and so is vulnerable to memory

Re: Infrastructure for zerocopy I/O

On Mon, Nov 16, 2015 at 07:55:45PM +0100, Christoph Hellwig wrote: >> A proposed patch has been posted >> (http://marc.info/?l=linux-usb=144763502829452=2), but I'm not >> convinced that it is the best approach. For instance, it always tries >> to get contiguous pages and so is vulnerable to

Re: Infrastructure for zerocopy I/O

On Mon, Nov 16, 2015 at 01:29:58PM -0500, Alan Stern wrote: > Allocating memory pages that match the device's DMA mask, for > use as I/O buffers, and locking them so their physical > addresses don't change (and they don't get paged out); > > Mapping those pages into a

Re: Infrastructure for zerocopy I/O

On Mon, Nov 16, 2015 at 08:03:12PM +0100, Steinar H. Gunderson wrote: > The original use case: DVB capture on embedded devices. > > My use case: USB3 uncompressed HD video capture (from multiple cards). > > Both of these hit (beyond the speed boost from zerocopy) the problem that > as time goes

Re: Infrastructure for zerocopy I/O

On Mon, 16 Nov 2015, Christoph Hellwig wrote: > On Mon, Nov 16, 2015 at 08:03:12PM +0100, Steinar H. Gunderson wrote: > > The original use case: DVB capture on embedded devices. > > > > My use case: USB3 uncompressed HD video capture (from multiple cards). > > > > Both of these hit (beyond the