Hi Alan, thanks for your clear answer. A few questions follow.
On Wednesday 09 May 2007, Alan Stern wrote: > On Wed, 9 May 2007, Laurent Pinchart wrote: > > Hi everybody, > > > > I'm running into an issue where large bulk transfers require lots of host > > memory. To fix the problem (without upgrading the system memory), I tried > > to implement "split bulk transfers". The first test results are > > promising, but there are a few borderline cases that I'm not able to > > test. I'd like to have your opinion on those. First of all, a little > > background (I'll be talking about IN bulk transfers here) to make sure I > > got everything right. > > > > Bulk transfers are payload oriented. The device usually reports the > > maximum payload size, and the host allocated URBs big enough to transfer > > a complete payload. The URBs are then divided in packets (by the host USB > > stack or the host controller, I'm not sure about this), and data is > > transfered packet by packet until the device signals the end of the > > payload by a packet shorter than the maximum packet size, which can lead > > to a 0-byte packet if the payload size is an exact multiple of the packet > > size. The host detects that short packet, finishes processing the URB and > > calls the completion handler. The short packet is required to detect > > payload end, as the actual payload size might be smaller than the maximum > > payload size reported by the device (payloads can have variable sizes). > > > > This works fine, but requires a lot of memory when the maximum payload > > size is big. Someone reported running across a device with a maximum > > payload size of 2.7MB, and as my driver allocates 5 bulk URBs, that makes > > a lot of memory especially for embedded or other memory constrained > > hosts. To work around this, the idea was to use smaller URBs that would > > not carry a complete payload, but part of a payload. A payload would thus > > be divided across several URBs, which would in turn be divided in USB > > packets. > > The scatter-gather routines in drivers/usb/core/message.c do something > similar to what you're describing. But they assume that all the memory > buffers have already been allocated, so they won't help solve your main > problem (memory usage). I wasn't aware of those functions. Just to make sure I get this right, am I right to say that URB buffers should be allocated as contiguous cache coherent physical memory (using usb_buffer_alloc, which in turns call dma_alloc_coherent for big DMAable buffers) or as normal memory using scatter-gather routines ? I currently allocate big bulk buffers using usb_buffer_alloc, which puts a lot of pressure on the memory allocator. Using scatter-gather mappings would help. However, if I'm not mistaken, the usb_sg_* routines are designed for synchronous use (with usb_sg_wait), and thus can't be used for streaming data. > > The problem I now have is to make sure I can detect the payload end > > reliably. When the payload size is not an exact multiple of the URB size, > > all the URBs except the last one will be filled with data. The last URB > > will have an actual size smaller than its maximum size, which is how I > > detect the payload end. When the payload size is an exact multiple of the > > URB size, all URBs will have the same actual length. I would like to know > > (and I can't test this as my test device has fixed-size payloads with > > sizes that are not an exact multiple of the bulk packet size) if the USB > > stack will return an empty URB to the completion handler, much like the > > USB device sends an empty packet. > > Yes, it will. Great. > > If so, are there any other caveats I should be aware of when splitting the > > payload across multiple bulk URBs ? And if no empty URB is currently > > returned by the Linux USB stack, is that something that could/should/will > > be fixed ? > > I assume that you submit your URBs one at a time, waiting for each one to > complete before submitting the next, since the idea is to minimize memory > usage. If you were to submit several URBs at once, you would need to set > the URB_SHORT_NOT_OK flag transfer flag on all but the last -- just > theoretical, since you're not in this situation. Actually I'm not. I submit a few URBs to get high throughput. Even with several URBs, memory usage is smaller than with a single payload-sized URB. Why do I need to set URB_SHORT_NOT_OK on all but the last URB ? As the actual payload size isn't known (only the maximum payload size is known), a short packet can be received at any time. The next URBs will be filled with the next payload, right ? > You should make sure that each URB's transfer_buffer_length is a multiple > of the maxpacket size. This is probably obvious; if you didn't do it then > you would get an EOVERFLOW error. There aren't any other constraints on > the transfer sizes; you can make them as big or as small as you see fit. That was indeed obvious, but still worth mentioning. > There's a possible ambiguity if the maximum payload size is a multiple of > the maxpacket size. If it is, and if the device wants to send a > maximum-sized payload, does it also send a 0-length packet at the end? That's a very good question. I haven't come across devices with a maximum payload size multiple of the maximum packet size. > In theory it should not, but devices don't always obey the spec in this > regard. Anyway, you will need to cope with the device's behavior. I didn't know about that. Thanks. > If it doesn't send the extra 0-length packet, then you have to make sure > that your last URB's length is exactly the right amount to make up a > maximum-sized payload. Otherwise the last URB will be filled with packet from the next payload, right ? > If it does send an extra 0-length packet then your last URB needs a larger > length, so that the 0-length packet will be received properly. If the last URB doesn't have a larger length, won't the 0-length packet be received by the next URB, making the USB stack call the completion handler with a 0-length URB ? > Other than that, you should be okay. Thanks a lot for your help once again. Laurent Pinchart ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ [email protected] To unsubscribe, use the last form field at: https://lists.sourceforge.net/lists/listinfo/linux-usb-devel
