Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-05 Thread Ming Lei
On Sat, Oct 5, 2013 at 4:34 AM, Alan Stern st...@rowland.harvard.edu wrote: On Fri, 4 Oct 2013, Markus Rechberger wrote: I was only testing reading the data so I didn't see any caching effects since I don't have a device or driver which I can send a lot data out. As far as I understand

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-05 Thread Alan Stern
On Sat, 5 Oct 2013, Ming Lei wrote: The buffer should be cached. The userspace program will have to make sure that it doesn't try to access the buffer while DMA is in progress. As long as that restriction is obeyed, the USB core will take care of mapping the buffer for DMA (which flushes

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-05 Thread Ming Lei
On Sat, Oct 5, 2013 at 11:10 PM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 5 Oct 2013, Ming Lei wrote: The buffer should be cached. The userspace program will have to make sure that it doesn't try to access the buffer while DMA is in progress. As long as that restriction is

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-04 Thread Markus Rechberger
On Mon, Sep 30, 2013 at 5:12 PM, Markus Rechberger mrechber...@gmail.com wrote: On Mon, Sep 30, 2013 at 4:59 PM, Alan Stern st...@rowland.harvard.edu wrote: On Mon, 30 Sep 2013, Markus Rechberger wrote: to explain why Isochronous makes such a difference, the kernel driver doesn't do the

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-04 Thread Alan Stern
On Sun, 29 Sep 2013, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The CPU usage decreases 1-2% on my 1.3ghz U7300 notebook The CPU usage decreases 6-8% on an Intel Atom n270

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-04 Thread Markus Rechberger
On Fri, Oct 4, 2013 at 8:41 PM, Alan Stern st...@rowland.harvard.edu wrote: On the whole this seems reasonable. There are a few stylistic things that could be cleaned up (missing blank lines after variable declarations, for example, and other checkpatch issues), but they are minor. Why do

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-04 Thread Alan Stern
On Fri, 4 Oct 2013, Markus Rechberger wrote: The biggest problem is that your proc_alloc_memory() routine doesn't call usbfs_increase_memory_usage(). Without that, there's nothing to prevent a user from allocating all the available kernel memory. only root is supposed to have raw USB

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-10-04 Thread Markus Rechberger
On Fri, Oct 4, 2013 at 10:34 PM, Alan Stern st...@rowland.harvard.edu wrote: On Fri, 4 Oct 2013, Markus Rechberger wrote: The biggest problem is that your proc_alloc_memory() routine doesn't call usbfs_increase_memory_usage(). Without that, there's nothing to prevent a user from

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-30 Thread Markus Rechberger
On Mon, Sep 30, 2013 at 5:51 AM, Marcel Holtmann mar...@holtmann.org wrote: Hi Markus, Do you have a userspace test program that we can use to verify that this does work, and that others can use to run on some different platforms to verify that this is actually faster? You will need one of

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-30 Thread Markus Rechberger
On Mon, Sep 30, 2013 at 1:26 PM, Markus Rechberger mrechber...@gmail.com wrote: On Mon, Sep 30, 2013 at 5:51 AM, Marcel Holtmann mar...@holtmann.org wrote: Hi Markus, Do you have a userspace test program that we can use to verify that this does work, and that others can use to run on some

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-30 Thread Alan Stern
On Mon, 30 Sep 2013, Markus Rechberger wrote: to explain why Isochronous makes such a difference, the kernel driver doesn't do the memset anymore for each urb packet. However that patch addresses multiple issues * Isochronous improvement by removing memset for each packet * Pre-Allocation to

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-30 Thread Markus Rechberger
On Mon, Sep 30, 2013 at 4:59 PM, Alan Stern st...@rowland.harvard.edu wrote: On Mon, 30 Sep 2013, Markus Rechberger wrote: to explain why Isochronous makes such a difference, the kernel driver doesn't do the memset anymore for each urb packet. However that patch addresses multiple issues *

Re: [PATCH] Memory mapping for USBFS

2013-09-29 Thread Markus Rechberger
On Sun, Sep 29, 2013 at 5:10 AM, Alan Stern st...@rowland.harvard.edu wrote: On Sun, 29 Sep 2013, Markus Rechberger wrote: to follow up I know why it doesn't happen anymore because customers are using a kernel module which does optimized transfers and pre-allocated the URBs and transfer

Re: [PATCH] Memory mapping for USBFS

2013-09-29 Thread Ming Lei
On Sun, Sep 29, 2013 at 7:51 PM, Markus Rechberger mrechber...@gmail.com wrote: I agree, but here not only small buffers are the problem, also latency. I will send another patch version which overrides the SG transfer once a preallocated buffer is submitted. The current patch will only use

Re: [PATCH] Memory mapping for USBFS

2013-09-29 Thread Markus Rechberger
On Sun, Sep 29, 2013 at 2:07 PM, Ming Lei tom.leim...@gmail.com wrote: On Sun, Sep 29, 2013 at 7:51 PM, Markus Rechberger mrechber...@gmail.com wrote: I agree, but here not only small buffers are the problem, also latency. I will send another patch version which overrides the SG transfer

Re: [PATCH] Memory mapping for USBFS

2013-09-29 Thread Ming Lei
On Sun, Sep 29, 2013 at 8:18 PM, Markus Rechberger mrechber...@gmail.com wrote: On Sun, Sep 29, 2013 at 2:07 PM, Ming Lei tom.leim...@gmail.com wrote: On Sun, Sep 29, 2013 at 7:51 PM, Markus Rechberger mrechber...@gmail.com wrote: I agree, but here not only small buffers are the problem,

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Ming Lei
On Sun, Sep 29, 2013 at 10:02 PM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The CPU usage decreases 1-2% on my 1.3ghz U7300 notebook The CPU usage

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Greg KH
On Sun, Sep 29, 2013 at 04:02:56PM +0200, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The CPU usage decreases 1-2% on my 1.3ghz U7300 notebook The CPU usage decreases 6-8% on

Re: [PATCH] Memory mapping for USBFS

2013-09-29 Thread Greg KH
On Sun, Sep 29, 2013 at 01:51:58PM +0200, Markus Rechberger wrote: I searched with google, obviously no one is using USBFS/userspace USB drivers that intensive as we do because (streaming 6mb for DVB or 20mb for Analog over days) the only bug reports that come up are related to our devices and

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Markus Rechberger
On Sun, Sep 29, 2013 at 5:14 PM, Greg KH gre...@linuxfoundation.org wrote: On Sun, Sep 29, 2013 at 04:02:56PM +0200, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The CPU usage

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Markus Rechberger
On Sun, Sep 29, 2013 at 4:24 PM, Ming Lei tom.leim...@gmail.com wrote: On Sun, Sep 29, 2013 at 10:02 PM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The

Re: [PATCH] Memory mapping for USBFS

2013-09-29 Thread Markus Rechberger
On Sun, Sep 29, 2013 at 5:22 PM, Greg KH gre...@linuxfoundation.org wrote: On Sun, Sep 29, 2013 at 01:51:58PM +0200, Markus Rechberger wrote: I searched with google, obviously no one is using USBFS/userspace USB drivers that intensive as we do because (streaming 6mb for DVB or 20mb for Analog

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Greg KH
On Sun, Sep 29, 2013 at 07:32:56PM +0200, Markus Rechberger wrote: On Sun, Sep 29, 2013 at 5:14 PM, Greg KH gre...@linuxfoundation.org wrote: On Sun, Sep 29, 2013 at 04:02:56PM +0200, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Markus Rechberger
On Sun, Sep 29, 2013 at 9:01 PM, Greg KH gre...@linuxfoundation.org wrote: On Sun, Sep 29, 2013 at 07:32:56PM +0200, Markus Rechberger wrote: On Sun, Sep 29, 2013 at 5:14 PM, Greg KH gre...@linuxfoundation.org wrote: On Sun, Sep 29, 2013 at 04:02:56PM +0200, Markus Rechberger wrote: This

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Alan Stern
On Sun, 29 Sep 2013, Markus Rechberger wrote: Do you have a userspace test program that we can use to verify that this does work, and that others can use to run on some different platforms to verify that this is actually faster? You will need one of our devices for testing I

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Markus Rechberger
On Sun, Sep 29, 2013 at 9:58 PM, Alan Stern st...@rowland.harvard.edu wrote: On Sun, 29 Sep 2013, Markus Rechberger wrote: Do you have a userspace test program that we can use to verify that this does work, and that others can use to run on some different platforms to verify that this

Re: [PATCH] memory mapping for usbfs (v0.4)

2013-09-29 Thread Marcel Holtmann
Hi Markus, Do you have a userspace test program that we can use to verify that this does work, and that others can use to run on some different platforms to verify that this is actually faster? You will need one of our devices for testing I guess. Some scanners (which use USBFS) or other

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Markus Rechberger
On Sat, Sep 28, 2013 at 4:39 AM, Ming Lei tom.leim...@gmail.com wrote: On Sat, Sep 28, 2013 at 10:29 AM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 28 Sep 2013, Ming Lei wrote: On Wed, Sep 25, 2013 at 3:12 AM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Alan Stern
On Sat, 28 Sep 2013, Ming Lei wrote: On Sat, Sep 28, 2013 at 10:29 AM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 28 Sep 2013, Ming Lei wrote: On Wed, Sep 25, 2013 at 3:12 AM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory mapping support to USBFS for

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Alan Stern
On Sat, 28 Sep 2013, Markus Rechberger wrote: Actually I observed both throughput and cpu utilization can be improved with the 4GB of DMA limit on either 32bit arch or 64bit arch, wrt. direct I/O over usb mass storage block device. I didn't look into the sg API in detail yet, but

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Markus Rechberger
On Sat, Sep 28, 2013 at 3:45 PM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 28 Sep 2013, Markus Rechberger wrote: Actually I observed both throughput and cpu utilization can be improved with the 4GB of DMA limit on either 32bit arch or 64bit arch, wrt. direct I/O over usb mass

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Ming Lei
On Sat, Sep 28, 2013 at 9:38 PM, Alan Stern st...@rowland.harvard.edu wrote: Very few non-xHCI controllers can do DMA above the 4 GB limit. Yes, but I am wondering non-xHCI need this kind of zero copy optimization, since very few user space drivers complain or care performance or cpu

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Alan Stern
On Sat, 28 Sep 2013, Markus Rechberger wrote: How about the initial patch, is it okay so far? I haven't had time to look through it yet. There's a settopbox manufacturer who pushed it to his next firmware release so we can do some additional performance measuring as well. My next step would

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Markus Rechberger
On Sat, Sep 28, 2013 at 5:17 PM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 28 Sep 2013, Markus Rechberger wrote: How about the initial patch, is it okay so far? I haven't had time to look through it yet. There's a settopbox manufacturer who pushed it to his next firmware release

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Alan Stern
On Sat, 28 Sep 2013, Markus Rechberger wrote: My next step would be to eliminate the kmallocs/kfrees for the single urbs (just by allocating the urbs with the same mechanism and passing the returned pointer to SUBMIT_URB). The time spent allocating and freeing URBs is probably pretty

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Markus Rechberger
On Sat, Sep 28, 2013 at 7:43 PM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 28 Sep 2013, Markus Rechberger wrote: My next step would be to eliminate the kmallocs/kfrees for the single urbs (just by allocating the urbs with the same mechanism and passing the returned pointer to

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Markus Rechberger
On Sat, Sep 28, 2013 at 8:52 PM, Markus Rechberger mrechber...@gmail.com wrote: On Sat, Sep 28, 2013 at 8:28 PM, Markus Rechberger mrechber...@gmail.com wrote: On Sat, Sep 28, 2013 at 7:43 PM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 28 Sep 2013, Markus Rechberger wrote: My

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Alan Stern
On Sat, 28 Sep 2013, Markus Rechberger wrote: I hope the SG transfer does not introduce any latency issues we have some usb chipsets which are very picky about latency issues on embedded systems. The latency for SG transfers should be smaller than for non-SG. We got it work by tuning the

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Alan Stern
On Sat, 28 Sep 2013, Markus Rechberger wrote: I put together a few reports: http://support.sundtek.com/index.php/topic,350.0.html http://support.sundtek.com/index.php/topic,1097.msg7780.html#msg7780 http://support.sundtek.com/index.php/topic,411.msg2153.html#msg2153

Re: [PATCH] Memory mapping for USBFS

2013-09-28 Thread Alan Stern
On Sun, 29 Sep 2013, Markus Rechberger wrote: to follow up I know why it doesn't happen anymore because customers are using a kernel module which does optimized transfers and pre-allocated the URBs and transfer buffer in kernelspace, the interface just copies the datastream to our actual

Re: [PATCH] Memory mapping for USBFS

2013-09-27 Thread Markus Rechberger
On Thu, Sep 26, 2013 at 2:13 AM, Greg KH gre...@linuxfoundation.org wrote: On Tue, Sep 24, 2013 at 09:12:39PM +0200, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. Does libusb

Re: [PATCH] Memory mapping for USBFS

2013-09-27 Thread Ming Lei
On Wed, Sep 25, 2013 at 3:12 AM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The CPU usage decreases 1-2% on my 1.3ghz U7300 notebook when transferring

Re: [PATCH] Memory mapping for USBFS

2013-09-27 Thread Alan Stern
On Sat, 28 Sep 2013, Ming Lei wrote: On Wed, Sep 25, 2013 at 3:12 AM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The CPU usage decreases 1-2% on

Re: [PATCH] Memory mapping for USBFS

2013-09-27 Thread Ming Lei
On Sat, Sep 28, 2013 at 10:29 AM, Alan Stern st...@rowland.harvard.edu wrote: On Sat, 28 Sep 2013, Ming Lei wrote: On Wed, Sep 25, 2013 at 3:12 AM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it

Re: [PATCH] Memory mapping for USBFS

2013-09-25 Thread Greg KH
On Tue, Sep 24, 2013 at 09:12:39PM +0200, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. Does libusb support this? The CPU usage decreases 1-2% on my 1.3ghz U7300 notebook when

Re: [PATCH] Memory mapping for USBFS

2013-09-25 Thread Markus Rechberger
On Thu, Sep 26, 2013 at 2:13 AM, Greg KH gre...@linuxfoundation.org wrote: On Tue, Sep 24, 2013 at 09:12:39PM +0200, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. Does libusb

Re: [PATCH] Memory mapping for USBFS

2013-09-25 Thread Markus Rechberger
On Thu, Sep 26, 2013 at 5:17 AM, Markus Rechberger mrechber...@gmail.com wrote: On Thu, Sep 26, 2013 at 2:13 AM, Greg KH gre...@linuxfoundation.org wrote: On Tue, Sep 24, 2013 at 09:12:39PM +0200, Markus Rechberger wrote: This patch adds memory mapping support to USBFS for isochronous and bulk

Re: [PATCH] Memory mapping for USBFS

2013-09-24 Thread Markus Rechberger
On Tue, Sep 24, 2013 at 9:12 PM, Markus Rechberger mrechber...@gmail.com wrote: This patch adds memory mapping support to USBFS for isochronous and bulk data transfers, it allows to pre-allocate usb transfer buffers. The CPU usage decreases 1-2% on my 1.3ghz U7300 notebook when transferring