Re: libusb performance on 8.1
While doing some speed testing I find that FreeBSD does ~4Mb/sec, but
Linux does 17Mb/sec OS X does 8-10Mb/sec.
Check out the bulk read-ahead support in NetBSD's ugen. This was
written at BBN in 2006 to enable higher data rates with the USRP (for
GNU Radio).
From your description it's not clear that this would help, but given
what you're doing you should probably be aware of it.
UGEN(4) NetBSD Kernel Interfaces ManualUGEN(4)
NAME
ugen -- USB generic device support
SYNOPSIS
ugen* at uhub? flags N
options UGEN_BULK_RA_WB
DESCRIPTION
The ugen driver provides support for all USB devices that do not have a
special driver. It supports access to all parts of the device, but not
in a way that is as convenient as a special purpose driver.
Normally the ugen driver is used when no other driver attaches to a
device. If ``flags 1'' is specified, the ugen will instead attach with a
very high priority and always be used. Together with the vendor and
product locators this can be used to force the ugen driver to be used for
a certain device.
There can be up to 127 USB devices connected to a USB bus. Each USB
device can have up to 16 endpoints. Each of these endpoints will commu-
nicate in one of four different modes: control, isochronous, bulk, or
interrupt. Each of the endpoints will have a different device node. The
four least significant bits in the minor device number determines which
endpoint the device accesses and the rest of the bits determines which
USB device.
If an endpoint address is used both for input and output the device can
be opened for both read or write.
To find out what endpoints exist there are a series of ioctl(2) opera-
tions on the control endpoint that return the USB descriptors of the
device, configurations, interfaces, and endpoints.
The control transfer mode can only happen on the control endpoint which
is always endpoint 0. The control endpoint accepts requests and may
respond with an answer to such requests. Control requests are issued by
ioctl(2) calls.
The bulk transfer mode can be in or out depending on the endpoint. To
perform IO on a bulk endpoint read(2) and write(2) should be used. All
IO operations on a bulk endpoint are normally unbuffered. On kernels
built with the UGEN_BULK_RA_WB option, the USB_SET_BULK_RA and
USB_SET_BULK_WB ioctl(2) calls are available, and enable read-ahead and
write-behind buffering respectively. When read-ahead or write-behind are
enabled, the file descriptor may be set to use non-blocking IO.
When in a UGEN_BULK_RA_WB mode, select(2) for read and write operates
normally, returning true if there is data in the read buffer and space in
the write buffer, respectively. When not in a UGEN_BULK_RA_WB mode,
select(2) always returns true, because there is no way to predict how the
device will respond to a read or write request.
The interrupt transfer mode can be in or out depending on the endpoint.
To perform IO on an interrupt endpoint read(2) and write(2) should be
used. A moderate amount of buffering is done by the driver.
All endpoints handle the following ioctl(2) calls:
USB_SET_SHORT_XFER (int)
Allow short read transfer. Normally a transfer from the device
which is shorter than the request specified is reported as an
error.
USB_SET_TIMEOUT (int)
Set the timeout on the device operations, the time is specified
in milliseconds. The value 0 is used to indicate that there is
no timeout.
The control endpoint (endpoint 0) handles the following ioctl(2) calls:
USB_GET_CONFIG (int)
Get the device configuration number.
USB_SET_CONFIG (int)
Set the device into the given configuration number.
This operation can only be performed when the control endpoint is
the sole open endpoint.
USB_GET_ALTINTERFACE (struct usb_alt_interface)
Get the alternative setting number for the interface with the
given index. The config_index is ignored in this call.
struct usb_alt_interface {
int uai_config_index;
int uai_interface_index;
int uai_alt_no;
};
USB_SET_ALTINTERFACE (struct usb_alt_interface)
Set the alternative setting to the given number in the interface
with the given index. The uai_config_index is ignored in this
call.
This operation can only be performed when no endpoints for the
interface are open.
USB_GET_NO_ALT (struct usb_alt_interface)
Return the number of different alternate settings in the
aui_alt_no
Re: libusb performance on 8.1
On 19/02/2011, at 1:38, Hans Petter Selasky wrote: Which harddisk driver are you using? ATA? Yes. My guess would be that taskqueues() in the HDD drivers are using swi-queues, instead of ordinary lower-priority queues. For example in sys/dev/ata I found: TASK_INIT(request-task, 0, ata_completed, request); ATA_DEBUG_RQ(request, finish taskqueue_swi); taskqueue_enqueue(taskqueue_swi, request-task); Which should perhaps just be taskqueue_thread instead of taskqueue_swi. I'll try changing it and seeing if it improves things. I've noticed during USB debugging that if certain non-DATA-xfer SCSI commands take time to complete, the whole system is waiting apparently, at least X11. This might indicate that synchronous code is being run from interrupt context. Interesting.. Although in the dual core case I wouldn't have thought it would be a huge deal would it? I had to give the dual core system I was using for testing back so I am currently using a single core with a non-ACHI capable chipset. I'll try and get access to the previous system again for some more testing. Do you have any suggestions for how I can find out exactly where it's sleeping in libusb? Or I suppose once it's in the kernel.. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On Friday 18 February 2011 15:40:07 Daniel O'Connor wrote: On 03/02/2011, at 17:26, Daniel O'Connor wrote: BTW do you have a feel for the latency in bulk vs iso? I currently have 5-10 msec of buffering in the hardware which I plan on increasing but I'm not sure what a reasonable amount would be :) I increased this by a factor of 8 but I still get drop outs during disk activity. I timed libusb_handle_events_timeout() with a timeout of 1 millisecond and it quite often exceeds this (by a significant amount) eg.. Triggered at 1298039742 2011/02/18 14:35:27:ugsio_child: libusb took 10msec - 32.44 ms 2011/02/18 14:35:27:ugsio_child: libusb took 10msec - 27.19 ms 2011/02/18 14:35:27:ugsio_child: libusb took 10msec - 25.27 ms 2011/02/18 14:35:27:ugsio_child: libusb took 10msec - 21.83 ms 2011/02/18 14:35:36:ugsio_child: libusb took 10msec - 48.23 ms Error during acquisition - ERROR:305 UGSIO aborted transfer: stalled (still using bulk transfers for now) Hi, Which harddisk driver are you using? ATA? My guess would be that taskqueues() in the HDD drivers are using swi-queues, instead of ordinary lower-priority queues. For example in sys/dev/ata I found: TASK_INIT(request-task, 0, ata_completed, request); ATA_DEBUG_RQ(request, finish taskqueue_swi); taskqueue_enqueue(taskqueue_swi, request-task); Which should perhaps just be taskqueue_thread instead of taskqueue_swi. I've noticed during USB debugging that if certain non-DATA-xfer SCSI commands take time to complete, the whole system is waiting apparently, at least X11. This might indicate that synchronous code is being run from interrupt context. --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On 03/02/2011, at 17:52, Hans Petter Selasky wrote: I am trying to get it working at the moment, however I'm only finding it capable of 4 or 8 Mb/sec (512 or 1024 byte EP), although perhaps I don't understand how to do ISO transfer properly. Hi, You need to set the multiplier to 2 or 3. Then you get 3*1024 bytes at maximum. OK, so I need.. usb_xf[i].xf = libusb_alloc_transfer(3); p = malloc(3 * 1024); libusb_fill_iso_transfer(usb_xf[i].xf, h, 0x82 p, 3 * 1024, 3, usbcb, usb_xf[i], 2000); ? BTW do you have a feel for the latency in bulk vs iso? I currently have 5-10 msec of buffering in the hardware which I plan on increasing but I'm not sure what a reasonable amount would be :) I put a logic analyser on my board and it shows fairly regular requests from the hardware (16kbyte bursts every 2msec or so) however I see glitches occasionally - 5.5ms, 7.5ms. I am not sure if they are attributable to userland scheduling (in which case writing a kernel driver should help) or some subtlety in USB itself. Are you using two isochronous transfers or just one? It is a single in endpoint. The device also has 2 in bulk and 2 out bulk endpoints for other data but they are idle while the iso one is running. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On Thursday 03 February 2011 11:24:23 Daniel O'Connor wrote: On 03/02/2011, at 17:52, Hans Petter Selasky wrote: I am trying to get it working at the moment, however I'm only finding it capable of 4 or 8 Mb/sec (512 or 1024 byte EP), although perhaps I don't understand how to do ISO transfer properly. Hi, You need to set the multiplier to 2 or 3. Then you get 3*1024 bytes at maximum. OK, so I need.. usb_xf[i].xf = libusb_alloc_transfer(3); p = malloc(3 * 1024); libusb_fill_iso_transfer(usb_xf[i].xf, h, 0x82 p, 3 * 1024, 3, usbcb, usb_xf[i], 2000); No. Please read the description of wMaxPacketSize in the USB2.0 spec, and the multiplier bits. High-speed USB executes 8 isoc packets per second! Number of packets should not be less than 56 for High-speed USB due to underflow risc. usb_xf[i].xf = libusb_alloc_transfer(56); p = malloc(3 * 1024 * 56); libusb_fill_iso_transfer(usb_xf[i].xf, 0x82, p, 3 * 56 * 1024, 56, usbcb, usb_xf[i], 2000); libusb_set_iso_packet_lengths(usb_xf[i].xf, 3 * 1024); You need to allocate 2x libusb_alloc_transfer(56) and submit to get double buffering! --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On 04/02/2011, at 14:32, Daniel O'Connor wrote: OK, I have that, it seems I need to work on the firmware as it's not sending data properly when I set it to iso. I'll work on that and try and figure it out. Unfortunately there isn't much example code on isochronous data transfer with the FX2 :( However I see the requests return saying they are complete but the amount of data transferred is 0. Setting the short not OK flag doesn't change the behaviour. I'm not sure if this is a bug in libusb, expected behaviour, or a bug in my program. I think this might be the FX2 sending zero length packets because its FIFO is empty. Now to figure out why it isn't triggering :-/ -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On 04/02/2011, at 15:18, Daniel O'Connor wrote: I'm not sure if this is a bug in libusb, expected behaviour, or a bug in my program. I think this might be the FX2 sending zero length packets because its FIFO is empty. Now to figure out why it isn't triggering :-/ Looks like my firmware managed to confuse the device enough that a power cycle was required, it now transfers data. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On 28/01/2011, at 18:28, Hans Petter Selasky wrote: For this kind of applications ISOCHRONOUS transfers should be used. Then you can have a double buffer guard in the range 1-56ms, regardless of the buffer size the hardware uses. That sounds nice :) I am trying to get it working at the moment, however I'm only finding it capable of 4 or 8 Mb/sec (512 or 1024 byte EP), although perhaps I don't understand how to do ISO transfer properly. BTW do you have a feel for the latency in bulk vs iso? I currently have 5-10 msec of buffering in the hardware which I plan on increasing but I'm not sure what a reasonable amount would be :) I put a logic analyser on my board and it shows fairly regular requests from the hardware (16kbyte bursts every 2msec or so) however I see glitches occasionally - 5.5ms, 7.5ms. I am not sure if they are attributable to userland scheduling (in which case writing a kernel driver should help) or some subtlety in USB itself. Thanks :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On Thursday 03 February 2011 07:56:25 Daniel O'Connor wrote: On 28/01/2011, at 18:28, Hans Petter Selasky wrote: For this kind of applications ISOCHRONOUS transfers should be used. Then you can have a double buffer guard in the range 1-56ms, regardless of the buffer size the hardware uses. That sounds nice :) I am trying to get it working at the moment, however I'm only finding it capable of 4 or 8 Mb/sec (512 or 1024 byte EP), although perhaps I don't understand how to do ISO transfer properly. Hi, You need to set the multiplier to 2 or 3. Then you get 3*1024 bytes at maximum. BTW do you have a feel for the latency in bulk vs iso? I currently have 5-10 msec of buffering in the hardware which I plan on increasing but I'm not sure what a reasonable amount would be :) I put a logic analyser on my board and it shows fairly regular requests from the hardware (16kbyte bursts every 2msec or so) however I see glitches occasionally - 5.5ms, 7.5ms. I am not sure if they are attributable to userland scheduling (in which case writing a kernel driver should help) or some subtlety in USB itself. Are you using two isochronous transfers or just one? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On 28/01/2011, at 18:28, Hans Petter Selasky wrote: For this kind of applications ISOCHRONOUS transfers should be used. Then you can have a double buffer guard in the range 1-56ms, regardless of the buffer size the hardware uses. Hmm, OK, I will have a look at changing it, although it is a bit of a pain because you can't stall an ISO EP and I'm using stall to indicate an error. Is it possible to change the amount that is buffered? This is a specialised application so a custom kernel is no problem. Even a hint in the right direction would be greatly appreciated :) You could also try an XHCI controller, because the BULK buffering is done differently there. OK thanks, I'll try that too. I obviously don't need any more throughput, however my application is very sensitive to latency, as I am reading out of a fairly small FIFO and if it fills up my entire run has to be aborted. --HPS -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On Friday 28 January 2011 09:42:57 Daniel O'Connor wrote: On 28/01/2011, at 18:28, Hans Petter Selasky wrote: For this kind of applications ISOCHRONOUS transfers should be used. Then you can have a double buffer guard in the range 1-56ms, regardless of the buffer size the hardware uses. Hmm, OK, I will have a look at changing it, although it is a bit of a pain because you can't stall an ISO EP and I'm using stall to indicate an error. Hi, There are some hard limits in the libusb source code. Just search for 16K in decimal. In the kernel the limit is bigger and I guess you don't need to change anything there. Is it possible to change the amount that is buffered? This is a specialised application so a custom kernel is no problem. Even a hint in the right direction would be greatly appreciated :) You could also try an XHCI controller, because the BULK buffering is done differently there. OK thanks, I'll try that too. --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On 23/01/2011, at 24:03, Hans Petter Selasky wrote: You need to change the way you buffer the data. FreeBSD does not queue more than 2 URB's at any time, and the turnaround time varies from 1ms to 125us due to hardware IRQ restrictions. Linux queues up all it can get, which leads to other kind of problems. The current internal buffer limit is 16Kbyte 8000 times per second which gives a MAX of 128 MByte/second. How difficult would it be to increase this? I obviously don't need any more throughput, however my application is very sensitive to latency, as I am reading out of a fairly small FIFO and if it fills up my entire run has to be aborted. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On Friday 28 January 2011 02:44:43 Daniel O'Connor wrote: On 23/01/2011, at 24:03, Hans Petter Selasky wrote: You need to change the way you buffer the data. FreeBSD does not queue more than 2 URB's at any time, and the turnaround time varies from 1ms to 125us due to hardware IRQ restrictions. Linux queues up all it can get, which leads to other kind of problems. The current internal buffer limit is 16Kbyte 8000 times per second which gives a MAX of 128 MByte/second. How difficult would it be to increase this? Hi, For this kind of applications ISOCHRONOUS transfers should be used. Then you can have a double buffer guard in the range 1-56ms, regardless of the buffer size the hardware uses. You could also try an XHCI controller, because the BULK buffering is done differently there. I obviously don't need any more throughput, however my application is very sensitive to latency, as I am reading out of a fairly small FIFO and if it fills up my entire run has to be aborted. --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On Thursday 20 January 2011 07:37:32 Daniel O'Connor wrote: [re-sent from the correct address] Hi, I am building a USB interface to a radar data acquisition chassis based on the Cypress CY7C68013A. While doing some speed testing I find that FreeBSD does ~4Mb/sec, but Linux does 17Mb/sec OS X does 8-10Mb/sec. Hi, You need to change the way you buffer the data. FreeBSD does not queue more than 2 URB's at any time, and the turnaround time varies from 1ms to 125us due to hardware IRQ restrictions. Linux queues up all it can get, which leads to other kind of problems. The current internal buffer limit is 16Kbyte 8000 times per second which gives a MAX of 128 MByte/second. In general during hardware design: 1) Avoid short packets. 2) Make sure transfer buffers are matched between host and device. From my experience it is very possible to reach +30Mbyte/second with LibUSB under FreeBSD 8.2+, given the buffering is optimal for the kernel. Hint: The libusb20 API has features to transfer multi-short packets in a more optimised way, which are not exposed in the libusb10 API. It is using libusb-1.0 with the async API in a threaded app. I spawn a thread which sites in the following loop: --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
Re: libusb performance on 8.1
On 23/01/2011, at 24:03, Hans Petter Selasky wrote: While doing some speed testing I find that FreeBSD does ~4Mb/sec, but Linux does 17Mb/sec OS X does 8-10Mb/sec. You need to change the way you buffer the data. FreeBSD does not queue more than 2 URB's at any time, and the turnaround time varies from 1ms to 125us due to hardware IRQ restrictions. Linux queues up all it can get, which leads to other kind of problems. The current internal buffer limit is 16Kbyte 8000 times per second which gives a MAX of 128 MByte/second. In general during hardware design: 1) Avoid short packets. 2) Make sure transfer buffers are matched between host and device. From my experience it is very possible to reach +30Mbyte/second with LibUSB under FreeBSD 8.2+, given the buffering is optimal for the kernel. Hmm... I am transferring fixed size packets of 512 bytes, so there shouldn't be any short packets (unless I misunderstand what you mean). Ahh, I I see what you mean now.. I was transferring a single packet at a time because I didn't realise you could transfer more at once. If I switch to transferring 2k at once it goes up to 16Mb/sec. Thanks :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
libusb performance on 8.1
[re-sent from the correct address]
Hi,
I am building a USB interface to a radar data acquisition chassis based on the
Cypress CY7C68013A.
While doing some speed testing I find that FreeBSD does ~4Mb/sec, but Linux
does 17Mb/sec OS X does 8-10Mb/sec.
It is using libusb-1.0 with the async API in a threaded app. I spawn a thread
which sites in the following loop:
while (!child_exit) {
tout.tv_sec = 0;
tout.tv_usec = 10;
libusb_handle_events_timeout(ctx, tout);
}
ugsio_cancelusb();
The parent thread is the rest of the program which and handles reading data out
of the completed transfers and forwarding them to the data processing children
and re-submitting them to the USB stack.
I keep 8 transactions of 512 bytes each in flight. The device itself holds up
to 4 packets in a FIFO until the PC is ready. Increasing the number of
transactions in flight has no effect on performance.
Does anyone have a suggestion as to how I can look at improving performance?
Deploying Linux is not really an option :)
The Linux PC is..
Linux fpgawork 2.6.32-28-generic #55-Ubuntu SMP Mon Jan 10 23:42:43 UTC 2011
x86_64 GNU/Linux
The CPU is a Core 2 E6400 @ 2.13GHz on a Gigabyte 965P-DS3 rev 1
(http://www.gigabyte.com/products/product-page.aspx?pid=2314#ov) which has an
ICH8 chipset.
The FreeBSD PC is..
FreeBSD aumond8.gsoft.com.au 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19
02:55:53 UTC 2010 r...@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
i386
The CPU is a Core 2 E7500 @ 2.93GHz on a Supermicro C2SBA+
(http://www.supermicro.com/products/motherboard/Core/G33/C2SBA_.cfm) which has
an ICH9 chipset.
Any patches welcome!
Thanks.
--
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
The nice thing about standards is that there
are so many of them to choose from.
-- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to freebsd-usb-unsubscr...@freebsd.org
