RE: PROBLEM: USB isochronous urb leak on EHCI driver
Okay; I did my homeworks. We've loaded kernel V3.16 (Oct 14th, 2015) on an i.MX51 plattform and the problem is still there. Unless an important change occured in V3.19, it appears that the latest kernel is not the solution for us. So we're still not able to use 4 codecs on our i.MX plattform. So just to make things clearer: - We have customers waiting for a solution with that hardware (this hardware is already delivred AND used); - We have important comittments and severe penalties ($$$) if we're not able to deliver on time (due for March 15th); - We've already looked at a hardware solution, which corresponds to replace current units ($), so that is not really an option for us; So as a last resort, I'm wondering, where is the USB expert who could help us solving our problem? Any suggestions? That would be me. Great. What do you need to make it a priority? If we are to get into debugging the USB driver, we would do it with the current kernel used (V2.6.31). What are the better tools to get into that? I guess a USB analyzer (hardware) would be the smart thing? Any brand name to suggest? It really would be better to start by debugging the most recent kernel possible. Once the problem has been solved there, it should be fairly straightforward to port it back. How much time do you think you'd spend solving that kind of issue? Few days? Few weeks, or few months? Could you commit on a certain number of hours? (We are trying to evaluate a possible date where we could start delivering products) When you say straightforward, again do you have an idea of the amount of time to do such work? Any other ideas for a solution will be really appreciated. You should begin by using usbmon during a short test (one or two seconds ought to be enough). See the instructions in the kernel source file Documentation/usb/usbmon.txt. Thanks for your support, Michael Tessier -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: PROBLEM: USB isochronous urb leak on EHCI driver
That is interresting, however, I have an older kernel running an OHCI driver which is able to handle 4 codecs. Same usb hardware (codecs and hub), but older kernel on a different CPU, with much less power. This makes me believe that there's a solution to make it work... Of course there is: Install an OHCI host controller and use it to drive your codecs. It should work fine. What do you mean by that? The host controller is embedded in the i.MX CPU... Changing the CPU is not really an option to me. Unless I am missing something? I didn't realize you were talking about an i.MX-based system. On a computer with a free PCI slot, it's easy to add an OHCI controller. iMX isn't as accomodating. If there's no way to add an extra USB controller to your system then the only choice is to upgrade the driver software. Alan Stern Okay; I did my homeworks. We've loaded kernel V3.16 (Oct 14th, 2015) on an i.MX51 plattform and the problem is still there. Unless an important change occured in V3.19, it appears that the latest kernel is not the solution for us. So we're still not able to use 4 codecs on our i.MX plattform. So just to make things clearer: - We have customers waiting for a solution with that hardware (this hardware is already delivred AND used); - We have important comittments and severe penalties ($$$) if we're not able to deliver on time (due for March 15th); - We've already looked at a hardware solution, which corresponds to replace current units ($), so that is not really an option for us; So as a last resort, I'm wondering, where is the USB expert who could help us solving our problem? Any suggestions? If we are to get into debugging the USB driver, we would do it with the current kernel used (V2.6.31). What are the better tools to get into that? I guess a USB analyzer (hardware) would be the smart thing? Any brand name to suggest? Any other ideas for a solution will be really appreciated. Regards, Michael Tessier. -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: PROBLEM: USB isochronous urb leak on EHCI driver
That is interresting, however, I have an older kernel running an OHCI driver which is able to handle 4 codecs. Same usb hardware (codecs and hub), but older kernel on a different CPU, with much less power. This makes me believe that there's a solution to make it work... Of course there is: Install an OHCI host controller and use it to drive your codecs. It should work fine. What do you mean by that? The host controller is embedded in the i.MX CPU... Changing the CPU is not really an option to me. Unless I am missing something? The periodic scheduling algorithm for OHCI is very different from the algorithm for EHCI. According to your knowledge, how much time would you think it takes to change the EHCI driver with an OHCI one? I don't understand the question. And can you tell if the OHCI driver will work on my hardware given that the Host controller of the i.MX512 is a USB2.0 host controller? (OHCI was implemented for USB 1.x from what I understand) The OHCI driver works with OHCI hardware and the EHCI driver works with EHCI hardware. OHCI is USB-1.1 and EHCI is USB-2. They are not interchangeable. That was what I thought first... I tried to do so several days ago with the built-in configurator (I am using ltib), but the configurator does not allow selecting the OHCI driver; I tried manually but it turned into compiler errors... It looks like the configurator is smart; it won't let you select the wrong driver for your hardware. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: PROBLEM: USB isochronous urb leak on EHCI driver
that the patch to fix it has not yet been merged. I understand. However, if one could test the following with a 3.x kernel: - CPU with USB 2.0 Host controller (using EHCI-hcd driver) - 4-port USB 1.1 Hub - 4x USB codecs (configured at 32khz-mono, 16-bits audio) Then try to stream audio on each of the 4 codecs at the same time (this includes one Read and one Write stream on each codec, so total of 4 Read and 4 Write streams. Then listen to the output... The result is likely to depend on what other USB hardware is attached. If sound is ok when using only 1 codec and becomes choppy when adding a second codec, then it means that this issue is still in the 3.x kernel. This answer will tell me if it is worth working on using a newer kernel or not. I have to say that I'm not a linux expert, so I see the migration to a newer kernel as a quite big amount of work... Why don't you try this yourself? It's easy to do; borrow a regular PC with a USB-2 host controller, boot it from a Live-CD version of Linux, plug in your hub with the codecs, and see what happens. Good point. I'll try that for sure. This will at least let me know if this issue has been corrected in the latest kernel. Michael Tessier -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: PROBLEM: USB isochronous urb leak on EHCI driver
If sound is ok when using only 1 codec and becomes choppy when adding
a second codec, then it means that this issue is still in the 3.x
kernel. This answer will tell me if it is worth working on using a newer
kernel or not.
I have to say that I'm not a linux expert, so I see the migration to a
newer kernel as a quite big amount of work...
We have support for mx51 on the latest kernel. All you need to do is to
describe your hardware on a device tree file. You can refer to
arch/arm/boot/dts/imx51-babbage.dts as an example.
Should be simple for you to make such test with the latest kernel.
I'll try to do so. My main concern is that even if it has been corrected
in the last kernel, I'm not free to just use the last kernel; our customers
already have units in their hands, and they bought us a Form Fit Function
plattform. We have a huge bunch of documentation coming with each deliverable
(part of an ISO process, following IEEE software standards). So for the
company, migrating to the latest kernel is the last resort because of the
amount of work it would require. Thanks for the idea, I'll let you know.
Michael Tessier
N�r��yb�X��ǧv�^�){.n�+{��^n�r���z���h��� �G���h�(�階�ݢj���m��z�ޖ���f���h���~�m�
RE: PROBLEM: USB isochronous urb leak on EHCI driver
On Mon, 15 Dec 2014, Michael Tessier wrote: Hi, I am dealing with a USB EHCI driver bug. Here is the info: My configuration: - Host: Freescale i.MX512 with ARM Cortex A8 (USB 2.0 host controller) Linux kernel: 2.6.31, using EHCI USB driver As mentioned by other people, the age of that kernel makes any bug report completely irrelevant. It's hard to count the number of non-trivial changes that have been made to the isochronous code in ehci-hcd since 2.6.31, but there have been quite a few. Hub: 4-PORT USB 1.1 HUB (Texas Instruments PN: tusb2046b) Devices: 4 USB 1.1 audio codecs (Texas Instruments PN: pcm2901) Note: each codec is being used in R/W access, so with 4 codecs, I have 4 playback and 4 capture streams. My problem: --- I have usb urb leaks when connecting more than 1 codec to the USB 1.1 Hub. What do you mean by urb leak? Normally, people use the word leak to refer to memory that is dynamically allocated and never deallocated, but you seem to mean something else. You are right. What I mean by leak is the following: At application level, all my calls to Read or Write operation to the codec driver will return with the correct amount of bytes read/written, with a choppy sound. Then when looking at lower levels: snd_pcm_oss_write (pcm_oss.c) - OK snd_pcm_lib_write (pcm_lib.c) - OK usb_submit_urb (urb.c) - FAIL with 3 codecs The FAIL here indicates that the total amount of bytes transferred does not correspond to what was expected. And indeed the sound is choppy when using more than a certain amount of bandwidth. However this amount of bandwidth is higher when connecting only 1 codec with different settings (48khz-stereo 16-bits instead of 32 khz-mono 16-bits).So at some point it looks like the bug is in the scheduler, only with several isochronous links. (the result is that some of the audio data is not transferred, part of the sound is simply missing) No problem when using only 1 of the 4 codecs connected to the hub; When I connect a second codec, the sound quality starts to degrade. With 3 codecs, we just cannot recognize a speach. Tests and observations: --- Since I have 3 usb ports available on the i.MX512, I tried to connect 3 codecs directly on USB ports: the sound is perfect on each of the three ports. I bought a consumer USB 2.0 Hub: no problem when using 3 codecs connected to that Hub, however, the audio will completly stop on all channels when connecting the 4th codec. Above you said the sound started to degrade when the second codec was connected; here you say there is no problem when using 3 of them. Which is it? Do you mean that the high-speed hub works better than the full-speed hub? Yes, that's it. Using the high-speed hub will allow for more data throughput before starting to miss some usb packets (and result in a choppy sound). I checked the communication between the Hub (USB 1.1) and the Host controller (USB 2.0) with a scope and concluded that the communication speed is 1.5 MBytes/s has expected (so the communication is downgraded to USB 1.1, since codecs and hub are USB 1.1 devices). Also, I know that there is physically enough bandwidth to transfer the data for two reasons: 1) I have an older CPU with a USB 1.1 host controller (using the OHCI driver), using the same hub and the same codecs: works like a champ, using less than 50% of the available bandwidth (observed with a scope) 2) 1 audio stream is 32khz-mono, 16 bits = 64 kB/s, 4 codecs = 8 streams(R/W) x 64 kB/s = 512 kB/s (out of 1.5MB/s) The amount of bandwidth available is usually not as much of an issue as the ability of the scheduling alogorithm to divide the bandwidth among the streams. The algorithm is not very smart and it often runs into a wall even when lots of physical bandwidth is still available. That is interresting, however, I have an older kernel running an OHCI driver which is able to handle 4 codecs. Same usb hardware (codecs and hub), but older kernel on a different CPU, with much less power. This makes me believe that there's a solution to make it work... I noticed that my sound problem starts happening with only 2 codecs (4 streams, 256 kB/s). I first thought that it was a bandwidth limitation, so I decided to connect only 1 codec using more bandwidth. I configured it to 48khz-stereo (16-bits), using 384 kB/s for both read and write streams: no problem. With that configuration, the scope shows about 30% of total bandwidth usage (300us used out of 1ms periods). Then, I added a second codec (48khz-stereo-16bits): very strange, now the total bandwidth usage felt down to about 200us, which seems to keep the same, whatever the number of codec I add (I also tried 3 and 4...). So it looks like the scheduler is not able to properly allocate Isochronous
RE: PROBLEM: USB isochronous urb leak on EHCI driver
My configuration: - Host: Freescale i.MX512 with ARM Cortex A8 (USB 2.0 host controller) Linux kernel: 2.6.31, using EHCI USB driver Hub: 4-PORT USB 1.1 HUB (Texas Instruments PN: tusb2046b) Devices: 4 USB 1.1 audio codecs (Texas Instruments PN: pcm2901) Note: each codec is being used in R/W access, so with 4 codecs, I have 4 playback and 4 capture streams. My problem: --- I have usb urb leaks when connecting more than 1 codec to the USB 1.1 Hub. (the result is that some of the audio data is not transferred, part of the sound is simply missing) No problem when using only 1 of the 4 codecs connected to the hub; When I connect a second codec, the sound quality starts to degrade. With 3 codecs, we just cannot recognize a speach. Tests and observations: --- Since I have 3 usb ports available on the i.MX512, I tried to connect 3 codecs directly on USB ports: the sound is perfect on each of the three ports. I bought a consumer USB 2.0 Hub: no problem when using 3 codecs connected to that Hub, however, the audio will completly stop on all channels when connecting the 4th codec. I checked the communication between the Hub (USB 1.1) and the Host controller (USB 2.0) with a scope and concluded that the communication speed is 1.5 MBytes/s has expected (so the communication is downgraded to USB 1.1, since codecs and hub are USB 1.1 devices). Also, I know that there is physically enough bandwidth to transfer the data for two reasons: 1) I have an older CPU with a USB 1.1 host controller (using the OHCI driver), using the same hub and the same codecs: works like a champ, using less than 50% of the available bandwidth (observed with a scope) 2) 1 audio stream is 32khz-mono, 16 bits = 64 kB/s, 4 codecs = 8 streams(R/W) x 64 kB/s = 512 kB/s (out of 1.5MB/s) I noticed that my sound problem starts happening with only 2 codecs (4 streams, 256 kB/s). I first thought that it was a bandwidth limitation, so I decided to connect only 1 codec using more bandwidth. I configured it to 48khz-stereo (16-bits), using 384 kB/s for both read and write streams: no problem. With that configuration, the scope shows about 30% of total bandwidth usage (300us used out of 1ms periods). Then, I added a second codec (48khz-stereo-16bits): very strange, now the total bandwidth usage felt down to about 200us, which seems to keep the same, whatever the number of codec I add (I also tried 3 and 4...). So it looks like the scheduler is not able to properly allocate Isochronous time slots when more than one device is connected to the hub. However, without the hub, it works perfectly. I am wonder if it is similar problem I met when using multiple interrupt transfers, when you find you lose the data, try to run 'top' to show cpu utilization, if it is close to 100%, it means the ehci can't queue request in time, so the host can't send IN token in time. Using a USB bus analyzer can also verify it. Peter Thanks Peter, I did it, the CPU usage varies between 0% and 4% so this is not the case here. -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
PROBLEM: USB isochronous urb leak on EHCI driver
Hi, I am dealing with a USB EHCI driver bug. Here is the info: My configuration: - Host: Freescale i.MX512 with ARM Cortex A8 (USB 2.0 host controller) Linux kernel: 2.6.31, using EHCI USB driver Hub: 4-PORT USB 1.1 HUB (Texas Instruments PN: tusb2046b) Devices: 4 USB 1.1 audio codecs (Texas Instruments PN: pcm2901) Note: each codec is being used in R/W access, so with 4 codecs, I have 4 playback and 4 capture streams. My problem: --- I have usb urb leaks when connecting more than 1 codec to the USB 1.1 Hub. (the result is that some of the audio data is not transferred, part of the sound is simply missing) No problem when using only 1 of the 4 codecs connected to the hub; When I connect a second codec, the sound quality starts to degrade. With 3 codecs, we just cannot recognize a speach. Tests and observations: --- Since I have 3 usb ports available on the i.MX512, I tried to connect 3 codecs directly on USB ports: the sound is perfect on each of the three ports. I bought a consumer USB 2.0 Hub: no problem when using 3 codecs connected to that Hub, however, the audio will completly stop on all channels when connecting the 4th codec. I checked the communication between the Hub (USB 1.1) and the Host controller (USB 2.0) with a scope and concluded that the communication speed is 1.5 MBytes/s has expected (so the communication is downgraded to USB 1.1, since codecs and hub are USB 1.1 devices). Also, I know that there is physically enough bandwidth to transfer the data for two reasons: 1) I have an older CPU with a USB 1.1 host controller (using the OHCI driver), using the same hub and the same codecs: works like a champ, using less than 50% of the available bandwidth (observed with a scope) 2) 1 audio stream is 32khz-mono, 16 bits = 64 kB/s, 4 codecs = 8 streams(R/W) x 64 kB/s = 512 kB/s (out of 1.5MB/s) I noticed that my sound problem starts happening with only 2 codecs (4 streams, 256 kB/s). I first thought that it was a bandwidth limitation, so I decided to connect only 1 codec using more bandwidth. I configured it to 48khz-stereo (16-bits), using 384 kB/s for both read and write streams: no problem. With that configuration, the scope shows about 30% of total bandwidth usage (300us used out of 1ms periods). Then, I added a second codec (48khz-stereo-16bits): very strange, now the total bandwidth usage felt down to about 200us, which seems to keep the same, whatever the number of codec I add (I also tried 3 and 4...). So it looks like the scheduler is not able to properly allocate Isochronous time slots when more than one device is connected to the hub. However, without the hub, it works perfectly. Another interresting fact is that at application level, the Read and Write operations are returning the good amount of bytes read/written. This is not the case at kernel level: I noticed that function usb_submit_urb (from /drivers/usb/core/urb.c) will only tranfer part of the urbs when the sound is degraded. I tried to figure out where the leak comes from without success. Also, there are no error messages from kernel so everything appears to work well, excepted that part of the sound is missing! I can't change my hardware (this is in the hand of customers), so the only possible solution for me is to correct the software. I tried to change my ehci driver with the one from kernel 2.6.39.4 but did not work, same problem. Question: - Before attempting to upgrade to an earlier kernel driver (this is a fairly big amount of work), I would really like to know if this problem would still be in the 3.x kernels. Has anyone seen that issue in 3.x kernels? I am pretty new to USB driver debugging, so any ideas of where/how to find solutions will be appreciated. Thank you very much in advance for the support. Also don't hesitate to redirect me if I'm not at the right place to ask these questions. I can also provide some code if someone need it to help. Attached is a dump of my dmesg after startup. Michael Tessier 5Linux version 2.6.31-770-g0e46b52-0897 (michael.tess...@vsvr-compile-01.pocatec.com) (gcc version 4.1.2) #12 PREEMPT Mon Nov 24 18:34:19 EST 2014 4CPU: ARMv7 Processor [412fc085] revision 5 (ARMv7), cr=10c53c7f 4CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache 4Machine: MX51 Xanthus Board 4Memory policy: ECC disabled, Data cache writeback 7On node 0 totalpages: 65536 7free_area_init_node: node 0, pgdat c02f137c, node_mem_map c0307000 7 DMA zone: 128 pages used for memmap 7 DMA zone: 0 pages reserved 7 DMA zone: 16256 pages, LIFO batch:3 7 Normal zone: 384 pages used for memmap 7 Normal zone: 48768 pages, LIFO batch:15 4Built 1 zonelists in Zone order, mobility grouping on. Total pages: 65024 5Kernel command line: console=ttymxc2,115200 CNFGmagic=XATS CNFGversion=1.0.0.0 vendorID=EXPT platformID=LINX productID=XATS mac1=00:1D:9E:00:17:2e mac2=00:1D:9E:00:17:2f XanthusPart=8100050 XanthusSerial=51
