Re: [RFC] How to handle delays in isochronous transfers?
On Tue, 25 Sep 2012, Laurent Pinchart wrote: All right. It sounds like as far as uvcvideo is concerned, recovering video-in frame synchronization after a drop-out doesn't matter very much. You don't much care if some of the frames are gone from the data stream with no indication as to how many were skipped over. Or if you do care, you don't think there's much to be done about it. I care, but I don't think the USB core can provide me with any useful information. In that case I'll go ahead with the proposed API change. Leaving URB_ISO_ASAP enabled for all your URBs should continue to work as well as could be expected. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Hi Alan, On Thursday 13 September 2012 12:08:35 Alan Stern wrote: On Thu, 13 Sep 2012, Laurent Pinchart wrote: Hi Alan, Sorry for the late answer, I've been travelling with no internet connection. No problem; thanks for joining the discussion. For video ISO IN problems can be split in two categories. Small delays ( one video frame) should already be properly supported as far as I can see. Data will be lost, drivers will detect that, either immediately if the protocol allows such detection, or at some video frame boundary. This will result in a corrupt video frame with missing data, and drivers will mark it as such when passing it to userspace through V4L2. Is this true even when the lost packets span a video frame boundary? Yes. Each ISO packet starts with a header that includes a frame ID bit toggled with every video frame. When receiving the first ISO packet of the next video frame the driver will mark the current video buffer as complete and flag an error as its size will be lower than expected (this is valid for uncompressed formats only, missing data in compressed video can't be detected). The next video frame will be missing data as well (at the start of the frame now), and will be marked as erroneous as well based on its size. Applications will thus receive two erroneous video frames. Larger delays that result in completely dropped video frames can cause more serious problems. Drivers won't know how many video frames have been dropped, and the V4L2 video frame sequence number will get out of sync. Applications can work around the problem using the video frame timestamps, but that's more a hack than a proper solution. I'm not sure if there is a way to solve the problem properly. ISO IN with video typically uses the ASAP flag, There's no particular reason for this, is there? Currently the flag doesn't really do anything. To be honest I thought the ASAP flag did what it now does with your patch :-) The uvcvideo driver doesn't care much about when URBs will be transferred, I just set the frame number to 0 and set the ASAP flag. and making the USB core report an artificial number of dropped ISO packets won't help much as many (most/all ?) devices transmit video frames in a variable number of ISO packets. If the number of lost packets is small enough, the video driver can get it directly by looking at the status fields in the isochronous packet descriptor structures. If the number is too big, though, it won't be reported at all -- the driver will know only that a big gap occurred. From what you say, this would be okay. Using timestamps the lost frames can be detected, assuming the frame rate is constant. With a variable frame rate we're probably screwed. For video output the situation is currently unclear. The only V4L2 output driver for USB devices that I know of is the uvcvideo driver, and the only UVC output device I know of uses bulk transfers. Assuming we will get video ISO OUT devices in the future, the right way to fix the problem will be to drop complete video frames on the host side if a large delays occurs. Feedback endpoints (currently not supported in the uvcvideo driver) might help (I've never worked with them, so I'd have to take a closer look). As Clemens mentioned, feedback endpoints are meant to help with small differences in clock rates between the host and the device. They are not meant to help with large gaps in the data stream. Another option would be to just check timestamps on the host side when starting a new video frame. I'm not sure whether the USB core could help with this problem. The best solution may be, if an URB submission fails with a -EXDEV error, to move forward to the next video frame boundary and restart transmitting from there. Depending on the protocol, you might want to delay the next URB submission so that the device doesn't see two frame boundaries too close together. Either way, you would set the URB_ISO_ASAP flag for the first URB and leave it turned off for all the following URBs. I'm not sure to get that. Why should it be turned off for the following URBs ? Basically what I'm asking is whether the API described in the documentation RFC (http://marc.info/?l=linux-usbm=134382746009339w=2) That's [PATCH] usb: storage: stop all current urbs when device is disconnected. I'm pretty sure you meant something else :-) would be suitable for your needs. The main point is that there should be a way to let class drivers know when a large gap in the isochronous stream has occurred. In the absence of such a mechanism the driver would proceed merrily on its way, totally unaware that the stream had lost synchronization. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at
Re: [RFC] How to handle delays in isochronous transfers?
On Wed, 19 Sep 2012, Laurent Pinchart wrote: To be honest I thought the ASAP flag did what it now does with your patch :-) The uvcvideo driver doesn't care much about when URBs will be transferred, I just set the frame number to 0 and set the ASAP flag. If the number of lost packets is small enough, the video driver can get it directly by looking at the status fields in the isochronous packet descriptor structures. If the number is too big, though, it won't be reported at all -- the driver will know only that a big gap occurred. From what you say, this would be okay. Using timestamps the lost frames can be detected, assuming the frame rate is constant. With a variable frame rate we're probably screwed. All right. It sounds like as far as uvcvideo is concerned, recovering video-in frame synchronization after a drop-out doesn't matter very much. You don't much care if some of the frames are gone from the data stream with no indication as to how many were skipped over. Or if you do care, you don't think there's much to be done about it. (The expected length of these drop-outs is not likely to be much more than 100 ms. At reasonable video frame rates, that's not going to amount to more than a few frames.) The best solution may be, if an URB submission fails with a -EXDEV error, to move forward to the next video frame boundary and restart transmitting from there. Depending on the protocol, you might want to delay the next URB submission so that the device doesn't see two frame boundaries too close together. Either way, you would set the URB_ISO_ASAP flag for the first URB and leave it turned off for all the following URBs. I'm not sure to get that. Why should it be turned off for the following URBs ? Because if the ASAP flag is turned on, you won't be notified about large gaps in the data stream. Maybe you don't care about this; I suppose the proper action would depend in part on how the device reacts to the missing data. Here's a simplified example to illustrate the point. Let's label the data packets you want to send as A, B, C, and so on. You transmit A, B, C, and D. Then there's a large gap, during which the device expects to receive E and F, but nothing actually gets sent. After the gap, if USB_ISO_ASAP is set then the driver would be unaware that anything went wrong. The computer would end up sending: A B C D . . E F G In other words, all the packets from E onward would be sent later than they should have been. On the other hand, if you turn off URB_ISO_ASAP for all the packets besides the first, then after the gap one of two things would happen (depending on the big the gap is compared to the depth of your isochronous queue). The first possibility is that packets E and F would be ignored, so you would end up transmitting: A B C D . . G H I There's still a gap but now the later packets are sent at the right times. The other possibility is that the usb_submit_urb() call for packet E would get an error. The driver would then recognize that something had gone wrong and it would try to recover, perhaps by jumping forward to the next video frame boundary and setting URB_ISO_ASAP for the next packet. Basically what I'm asking is whether the API described in the documentation RFC (http://marc.info/?l=linux-usbm=134382746009339w=2) That's [PATCH] usb: storage: stop all current urbs when device is disconnected. I'm pretty sure you meant something else :-) Weird copy/paste error. Pretend that I wrote what I meant instead. :-) Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Hi Alan, Sorry for the late answer, I've been travelling with no internet connection. On Tuesday 04 September 2012 10:23:41 Alan Stern wrote: On Tue, 4 Sep 2012, Peter Chen wrote: In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. It is ISO transfer, if the delay is too much, and the buffer at device side is empty, it is normal the screen is stopped like we watch movie on Internet (buffering). So if the delay is too much, I don't think there is a way can deal with it as host does not send any data to device. Yes, some data is lost. We want to make sure that when the data starts flowing again, it is properly synchronized. Furthermore, if packets are lost, frequency feedback becomes impossible because the device doesn't know how many samples were lost. Where the packets are lost? If the packets are lost at class driver/usb driver, class driver will know it and should take the responsible to re-send it. If the packets are lost on the USB bus (during the transfer), as it is ISO transfer, then the data should be lost, and host doesn't know the data is lost, how can it re-sends the packet? Data should not be resent in any case. Isochronous data is never resent. Here's the problem we face: The class driver submits packets 1, 2, 3, and 4. They get sent properly, but the completion interrupt is delayed. As a result, the class driver's completion handler doesn't get called until too late; the frames for packets 5 - 44 have already expired. The data that should have been sent during those frames is lost. But the class driver doesn't know this, so when the completion handler does get called, it submits packets 5, 6, 7, and 8. They end up getting sent in frames 45, 46, 47, and 48. This means the data is now out of synchronization by 40 frames. That's the problem I want to solve. There's an equivalent problem for data flowing into the host. In this case, the URB that should have been submitted for frame 5 would receive data that the device sent during frame 45. (Whether this data contains the signals for frame 5 or for frame 45 would depend on the device; the host can't do anything about it.) For video ISO IN problems can be split in two categories. Small delays ( one video frame) should already be properly supported as far as I can see. Data will be lost, drivers will detect that, either immediately if the protocol allows such detection, or at some video frame boundary. This will result in a corrupt video frame with missing data, and drivers will mark it as such when passing it to userspace through V4L2. Larger delays that result in completely dropped video frames can cause more serious problems. Drivers won't know how many video frames have been dropped, and the V4L2 video frame sequence number will get out of sync. Applications can work around the problem using the video frame timestamps, but that's more a hack than a proper solution. I'm not sure if there is a way to solve the problem properly. ISO IN with video typically uses the ASAP flag, and making the USB core report an artificial number of dropped ISO packets won't help much as many (most/all ?) devices transmit video frames in a variable number of ISO packets. For video output the situation is currently unclear. The only V4L2 output driver for USB devices that I know of is the uvcvideo driver, and the only UVC output device I know of uses bulk transfers. Assuming we will get video ISO OUT devices in the future, the right way to fix the problem will be to drop complete video frames on the host side if a large delays occurs. Feedback endpoints (currently not supported in the uvcvideo driver) might help (I've never worked with them, so I'd have to take a closer look). Another option would be to just check timestamps on the host side when starting a new video frame. I'm not sure whether the USB core could help with this problem. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Tue, 11 Sep 2012, Clemens Ladisch wrote:
The test patch should be ready for posting soon. Would you like to try
it out?
Yes (if I find time).
Here it is. This is actually five separate patches combined into one.
It implements the new documented behavior in ehci-hcd and adds
debugging code for performing tests. It also changes the snd-usb-audio
driver by removing the URB_ISO_ASAP flag from the data URBs.
Here's how the testing code should be used. The patch creates a file
named test under /sys/kernel/debug/usb/ehci/X/ where X is the PCI
device name of the particular controller (for example, :00:1d.7).
You write two integer values to the test file. The first tells
ehci-hcd how many isochronous completion events to ignore and the
second tells ehci-hcd how many microseconds to delay before then
scanning for isochronous completions.
For example, with my USB audio device, the snd-usb-audio driver uses a
queue of 2 iso data URBs each containing 3 packets (usually).
Therefore I test by playing an audio file and doing:
echo 2 1000 test
while the audio is running. The driver ignores the completion events
for the two URBs in the queue when they occur, and then handles them
1000 us after the second completion. As a result, the next URB
submitted by the sound driver is one frame too late. The submission
succeeds, but I can see the delay in a usbmon trace as well as the fact
that when the URB completes, its first packet gets a status of -EXDEV
instead of 0.
If instead I do echo 2 3000 test then the next URB is submitted 3
frames too late. Since the URB contains only 3 packets, all of its
slots have already expired. The submission fails and the audio stops
playing, although the ogg123 program continues running and the synch
URBs continue to be submitted.
If the URB_ISO_ASAP flag is retained in the sound driver then the
delayed URB submission does not fail. Instead there is a brief gap and
the audio continues, now out of synchronization. In theory this gap
could be made as large as you want, although in practice the udelay()
routine has an upper limit on the size of the argument it accepts.
Alan Stern
Index: usb-3.6/drivers/usb/host/ehci.h
===
--- usb-3.6.orig/drivers/usb/host/ehci.h
+++ usb-3.6/drivers/usb/host/ehci.h
@@ -112,6 +112,7 @@ struct ehci_hcd { /* one per controlle
__u32 hcs_params; /* cached register copy */
spinlock_t lock;
enum ehci_rh_state rh_state;
+ int test1, test2;
/* general schedule support */
boolscanning:1;
@@ -143,7 +144,7 @@ struct ehci_hcd { /* one per controlle
struct ehci_qh *intr_unlink_last;
unsignedintr_unlink_cycle;
unsignednow_frame; /* frame from HC hardware */
- unsignednext_frame; /* scan periodic, start here */
+ unsignedlast_iso_frame; /* last frame scanned for iso */
unsignedintr_count; /* intr activity count */
unsignedisoc_count; /* isoc activity count */
unsignedperiodic_count; /* periodic activity count */
@@ -193,7 +194,6 @@ struct ehci_hcd { /* one per controlle
unsignedhas_amcc_usb23:1;
unsignedneed_io_watchdog:1;
unsignedamd_pll_fix:1;
- unsignedfs_i_thresh:1; /* Intel iso scheduling */
unsigneduse_dummy_qh:1; /* AMD Frame List table quirk*/
unsignedhas_synopsys_hc_bug:1; /* Synopsys HC */
unsignedframe_index_bug:1; /* MosChip (AKA NetMos) */
Index: usb-3.6/drivers/usb/host/ehci-sched.c
===
--- usb-3.6.orig/drivers/usb/host/ehci-sched.c
+++ usb-3.6/drivers/usb/host/ehci-sched.c
@@ -1361,7 +1361,7 @@ sitd_slot_ok (
* given EHCI_TUNE_FLS and the slop). Or, write a smarter scheduler!
*/
-#define SCHEDULE_SLOP 80 /* microframes */
+#define SCHEDULING_DELAY 40 /* microframes */
static int
iso_stream_schedule (
@@ -1370,7 +1370,7 @@ iso_stream_schedule (
struct ehci_iso_stream *stream
)
{
- u32 now, next, start, period, span;
+ u32 now, base, next, start, period, span;
int status;
unsignedmod = ehci-periodic_size 3;
struct ehci_iso_sched *sched = urb-hcpriv;
@@ -1382,62 +1382,75 @@ iso_stream_schedule (
span = 3;
}
- if (span mod - SCHEDULE_SLOP) {
- ehci_dbg (ehci, iso request %p too long\n, urb);
- status = -EFBIG;
- goto fail;
-
Re: [RFC] How to handle delays in isochronous transfers?
On Thu, 13 Sep 2012, Laurent Pinchart wrote: Hi Alan, Sorry for the late answer, I've been travelling with no internet connection. No problem; thanks for joining the discussion. For video ISO IN problems can be split in two categories. Small delays ( one video frame) should already be properly supported as far as I can see. Data will be lost, drivers will detect that, either immediately if the protocol allows such detection, or at some video frame boundary. This will result in a corrupt video frame with missing data, and drivers will mark it as such when passing it to userspace through V4L2. Is this true even when the lost packets span a video frame boundary? Larger delays that result in completely dropped video frames can cause more serious problems. Drivers won't know how many video frames have been dropped, and the V4L2 video frame sequence number will get out of sync. Applications can work around the problem using the video frame timestamps, but that's more a hack than a proper solution. I'm not sure if there is a way to solve the problem properly. ISO IN with video typically uses the ASAP flag, There's no particular reason for this, is there? Currently the flag doesn't really do anything. and making the USB core report an artificial number of dropped ISO packets won't help much as many (most/all ?) devices transmit video frames in a variable number of ISO packets. If the number of lost packets is small enough, the video driver can get it directly by looking at the status fields in the isochronous packet descriptor structures. If the number is too big, though, it won't be reported at all -- the driver will know only that a big gap occurred. From what you say, this would be okay. For video output the situation is currently unclear. The only V4L2 output driver for USB devices that I know of is the uvcvideo driver, and the only UVC output device I know of uses bulk transfers. Assuming we will get video ISO OUT devices in the future, the right way to fix the problem will be to drop complete video frames on the host side if a large delays occurs. Feedback endpoints (currently not supported in the uvcvideo driver) might help (I've never worked with them, so I'd have to take a closer look). As Clemens mentioned, feedback endpoints are meant to help with small differences in clock rates between the host and the device. They are not meant to help with large gaps in the data stream. Another option would be to just check timestamps on the host side when starting a new video frame. I'm not sure whether the USB core could help with this problem. The best solution may be, if an URB submission fails with a -EXDEV error, to move forward to the next video frame boundary and restart transmitting from there. Depending on the protocol, you might want to delay the next URB submission so that the device doesn't see two frame boundaries too close together. Either way, you would set the URB_ISO_ASAP flag for the first URB and leave it turned off for all the following URBs. Basically what I'm asking is whether the API described in the documentation RFC (http://marc.info/?l=linux-usbm=134382746009339w=2) would be suitable for your needs. The main point is that there should be a way to let class drivers know when a large gap in the isochronous stream has occurred. In the absence of such a mechanism the driver would proceed merrily on its way, totally unaware that the stream had lost synchronization. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Alan Stern wrote: If instead I do echo 2 3000 test then the next URB is submitted 3 frames too late. Since the URB contains only 3 packets, all of its slots have already expired. The submission fails and the audio stops playing, although the ogg123 program continues running and the synch URBs continue to be submitted. You might try this untested patch (ogg123 should then restart): --- a/sound/usb/endpoint.c +++ b/sound/usb/endpoint.c @@ -386,7 +386,7 @@ static void snd_complete_urb(struct urb *urb) return; snd_printk(KERN_ERR cannot submit urb (err = %d)\n, err); - //snd_pcm_stop(substream, SNDRV_PCM_STATE_XRUN); + snd_pcm_stop(substream, SNDRV_PCM_STATE_XRUN); exit_clear: clear_bit(ctx-index, ep-active_mask); -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Thu, 13 Sep 2012, Clemens Ladisch wrote: Alan Stern wrote: If instead I do echo 2 3000 test then the next URB is submitted 3 frames too late. Since the URB contains only 3 packets, all of its slots have already expired. The submission fails and the audio stops playing, although the ogg123 program continues running and the synch URBs continue to be submitted. You might try this untested patch (ogg123 should then restart): --- a/sound/usb/endpoint.c +++ b/sound/usb/endpoint.c @@ -386,7 +386,7 @@ static void snd_complete_urb(struct urb *urb) return; snd_printk(KERN_ERR cannot submit urb (err = %d)\n, err); - //snd_pcm_stop(substream, SNDRV_PCM_STATE_XRUN); + snd_pcm_stop(substream, SNDRV_PCM_STATE_XRUN); This doesn't compile because substream is undefined. I tried the following instead: --- usb-3.6.orig/sound/usb/endpoint.c +++ usb-3.6/sound/usb/endpoint.c @@ -376,7 +376,7 @@ static void snd_complete_urb(struct urb return; snd_printk(KERN_ERR cannot submit urb (err = %d)\n, err); - //snd_pcm_stop(substream, SNDRV_PCM_STATE_XRUN); + snd_pcm_stop(ctx-subs-pcm_substream, SNDRV_PCM_STATE_XRUN); exit_clear: clear_bit(ctx-index, ep-active_mask); But it caused an immediate oops during testing. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Tue, Sep 11, 2012 at 11:46 PM, Clemens Ladisch clem...@ladisch.de wrote: Alan Stern wrote: I have tried some initial testing of my updates, using data-OUT transfers with URB_ISO_ASAP turned off for the data URBs in sound/usb/endpoint.c. When interrupts are delayed so long that synchronization is lost and a data URB submission fails, the sound stops playing and doesn't restart. But the synch URBs continue to be submitted and ogg123 doesn't end for quite some time. When URB submission fails, the stream should be stopped. In theory. Since ISOCH transfers are not expected to be lossless (i.e, not retried), perhaps we should simply lump all kinds of failures together and ignore them and simply keep moving on? Stream stopping should only be done in response to standard USB state changes that warrant it (ex switching to AltSetting-0) ? -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Jassi Brar wrote: On Tue, Sep 11, 2012 at 11:46 PM, Clemens Ladisch clem...@ladisch.de wrote: When URB submission fails, the stream should be stopped. Since ISOCH transfers are not expected to be lossless (i.e, not retried), perhaps we should simply lump all kinds of failures together and ignore them and simply keep moving on? URB submission errors cannot be ignored -- when this happens, the USB stream _is_ already stopped; stopping the PCM device just informs the application about this. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Mon, 10 Sep 2012, Clemens Ladisch wrote: Peter Chen wrote: Essentially, the stream is restarted with known timing. So, if big delay occurs (like 10ms), you would like the controller layer tells you that, then, you drop 10ms packet? I want to be able to know when the next submitted packet will be transmitted. You can find out after usb_submit_urb(urb) returns by looking at urb-start_frame. I have tried some initial testing of my updates, using data-OUT transfers with URB_ISO_ASAP turned off for the data URBs in sound/usb/endpoint.c. When interrupts are delayed so long that synchronization is lost and a data URB submission fails, the sound stops playing and doesn't restart. But the synch URBs continue to be submitted and ogg123 doesn't end for quite some time. This doesn't seem like the desired behavior. I'm not sure where to pin any blame, though. The test patch should be ready for posting soon. Would you like to try it out? Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Alan Stern wrote: I have tried some initial testing of my updates, using data-OUT transfers with URB_ISO_ASAP turned off for the data URBs in sound/usb/endpoint.c. When interrupts are delayed so long that synchronization is lost and a data URB submission fails, the sound stops playing and doesn't restart. But the synch URBs continue to be submitted and ogg123 doesn't end for quite some time. When URB submission fails, the stream should be stopped. In theory. Daniel, is there a reason that you commented out the snd_pcm_stop() call in snd_complete_urb()? The test patch should be ready for posting soon. Would you like to try it out? Yes (if I find time). Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
If the feedback is supported, the device will know host sends data slowly, it will give speed up feedback information after it receives packet 5 or other packets depends on its interval at descriptor. At this information, it can tell the host to increase the packet size, then the transaction length and transaction numbers at iTD can be increased(Assume it was not maximum). Clemens pointed out that this won't work if the delay is too long. Clements said In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. before. I don't understand what will not be helped? The frequency feedback mechanism is designed to compensate for small differences in the host's and the device's clocks; it is not suitable for situations where the host sends less data than it should (or none at all). Devices have a buffer that is no larger than two or three milliseconds. Furthermore, the maximum packet size usually is only about 10 % larger than needed for the nominal sample rate, so it would in no case be possible to compensate for even a single lost packet. OK, if host takes responsible for re-sync the data, - When and where the class driver knows out-of-sync? At completion function? It may also several milliseconds later than last packet. - How to compensate the data which is out-of sync quicker than feedback way? Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Peter Chen wrote: Essentially, the stream is restarted with known timing. So, if big delay occurs (like 10ms), you would like the controller layer tells you that, then, you drop 10ms packet? I want to be able to know when the next submitted packet will be transmitted. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Sat, 8 Sep 2012, Peter Chen wrote: If the feedback is supported, the device will know host sends data slowly, it will give speed up feedback information after it receives packet 5 or other packets depends on its interval at descriptor. At this information, it can tell the host to increase the packet size, then the transaction length and transaction numbers at iTD can be increased(Assume it was not maximum). Clemens pointed out that this won't work if the delay is too long. Clements said In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. before. I don't understand what will not be helped? Maybe Clemens can explain. I have never worked with feedback endpoints. And what will happen when the thing goes to wrong? The isochronous stream will be delayed and the output will not be synchronized with the input. For example, an speaker might produce sounds 50 or 100 ms after receiving the audio data from the microphone. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Peter Chen wrote: On Fri, Sep 7, 2012 at 11:44 PM, Alan Stern st...@rowland.harvard.edu wrote: On Fri, 7 Sep 2012, Peter Chen wrote: If the feedback is supported, the device will know host sends data slowly, it will give speed up feedback information after it receives packet 5 or other packets depends on its interval at descriptor. At this information, it can tell the host to increase the packet size, then the transaction length and transaction numbers at iTD can be increased(Assume it was not maximum). Clemens pointed out that this won't work if the delay is too long. Clements said In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. before. I don't understand what will not be helped? The frequency feedback mechanism is designed to compensate for small differences in the host's and the device's clocks; it is not suitable for situations where the host sends less data than it should (or none at all). Devices have a buffer that is no larger than two or three milliseconds. Furthermore, the maximum packet size usually is only about 10 % larger than needed for the nominal sample rate, so it would in no case be possible to compensate for even a single lost packet. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Fri, 7 Sep 2012, Peter Chen wrote: Here's the problem we face: The class driver submits packets 1, 2, 3, and 4. They get sent properly, but the completion interrupt is delayed. As a result, the class driver's completion handler doesn't get called until too late; the frames for packets 5 - 44 have already expired. The data that should have been sent during those frames is lost. But the class driver doesn't know this, so when the completion handler does get called, it submits packets 5, 6, 7, and 8. They end up getting sent in frames 45, 46, 47, and 48. This means the data is now out of synchronization by 40 frames. That's the problem I want to solve. Can I understand this problem like below: It is OUT ISO transfer, at each completion handler, it prepares 4 iTDs for 4 packets. When the above problem occurs, the class driver will submit packet from 5 - 48, and the controller driver will prepare 44 iTDs. (or the class driver still submits packet 5-8?) The class driver submits packets 5-8, because each time the completion handler runs it prepares 4 iTDs. If the feedback is supported, the device will know host sends data slowly, it will give speed up feedback information after it receives packet 5 or other packets depends on its interval at descriptor. At this information, it can tell the host to increase the packet size, then the transaction length and transaction numbers at iTD can be increased(Assume it was not maximum). Clemens pointed out that this won't work if the delay is too long. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Fri, Sep 7, 2012 at 11:44 PM, Alan Stern st...@rowland.harvard.edu wrote: On Fri, 7 Sep 2012, Peter Chen wrote: Here's the problem we face: The class driver submits packets 1, 2, 3, and 4. They get sent properly, but the completion interrupt is delayed. As a result, the class driver's completion handler doesn't get called until too late; the frames for packets 5 - 44 have already expired. The data that should have been sent during those frames is lost. But the class driver doesn't know this, so when the completion handler does get called, it submits packets 5, 6, 7, and 8. They end up getting sent in frames 45, 46, 47, and 48. This means the data is now out of synchronization by 40 frames. That's the problem I want to solve. Can I understand this problem like below: It is OUT ISO transfer, at each completion handler, it prepares 4 iTDs for 4 packets. When the above problem occurs, the class driver will submit packet from 5 - 48, and the controller driver will prepare 44 iTDs. (or the class driver still submits packet 5-8?) The class driver submits packets 5-8, because each time the completion handler runs it prepares 4 iTDs. If the feedback is supported, the device will know host sends data slowly, it will give speed up feedback information after it receives packet 5 or other packets depends on its interval at descriptor. At this information, it can tell the host to increase the packet size, then the transaction length and transaction numbers at iTD can be increased(Assume it was not maximum). Clemens pointed out that this won't work if the delay is too long. Clements said In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. before. I don't understand what will not be helped? And what will happen when the thing goes to wrong? Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Here's the problem we face: The class driver submits packets 1, 2, 3, and 4. They get sent properly, but the completion interrupt is delayed. As a result, the class driver's completion handler doesn't get called until too late; the frames for packets 5 - 44 have already expired. The data that should have been sent during those frames is lost. But the class driver doesn't know this, so when the completion handler does get called, it submits packets 5, 6, 7, and 8. They end up getting sent in frames 45, 46, 47, and 48. This means the data is now out of synchronization by 40 frames. That's the problem I want to solve. Can I understand this problem like below: It is OUT ISO transfer, at each completion handler, it prepares 4 iTDs for 4 packets. When the above problem occurs, the class driver will submit packet from 5 - 48, and the controller driver will prepare 44 iTDs. (or the class driver still submits packet 5-8?) If the feedback is supported, the device will know host sends data slowly, it will give speed up feedback information after it receives packet 5 or other packets depends on its interval at descriptor. At this information, it can tell the host to increase the packet size, then the transaction length and transaction numbers at iTD can be increased(Assume it was not maximum). There's an equivalent problem for data flowing into the host. In this case, the URB that should have been submitted for frame 5 would receive data that the device sent during frame 45. (Whether this data contains the signals for frame 5 or for frame 45 would depend on the device; the host can't do anything about it.) Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Alan Stern wrote: There are two possibilities to consider when an URB containing several packets of data is submitted: 1. The slots for the first few packets have already expired, but the remaining packets will be transferred okay. 2. The slots for all the packets in the URB have expired. In case 1 there is data loss but the queuing remains intact. In case 2 the queuing is broken. It sounds like you're saying that case 1 submissions should succeed (and return -EXDEV status for the statuses of the missed packets), whereas case 2 submissions should fail outright (say with an -EXDEV error, which is currently not used for URB submission). This wasn't exactly what I meant, but your proposal is turns out to be the only logical way of reporting errors when only some of the URB's packets fail. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Alan Stern wrote: On Tue, 4 Sep 2012, Jassi Brar wrote: Second, people involved in real-time programming (such as audio or video) generally want to keep latency to a minimum. If we progress the h/w pointer of ALSA ring buffer at URB completion (and not at URB submission), I don't understand this comment. As far as I can tell, the relevant pointer already is being advanced during URB completion. It's advanced directly after submission. The queued data is double- buffered because the USB API lacks the ability to scatter/gather from a coherent buffer. this shouldn't affect the latency. How would this make any difference? The time between the application writing samples to the buffer and the samples actually being played by the device would not change. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Wed, Sep 5, 2012 at 5:37 PM, Clemens Ladisch clem...@ladisch.de wrote: On Tue, 4 Sep 2012, Jassi Brar wrote: If we progress the h/w pointer of ALSA ring buffer at URB completion (and not at URB submission) this shouldn't affect the latency. How would this make any difference? The time between the application writing samples to the buffer and the samples actually being played by the device would not change. If we advance the pointer immediately after submission, the total time a sample spends in flight is ring-buffer + queue length. If we advance the pointer upon completion, the total time a sample spends in flight is ring-buffer. Or I am doped. Cheers -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Wed, 5 Sep 2012, Clemens Ladisch wrote: Alan Stern wrote: On Tue, 4 Sep 2012, Jassi Brar wrote: Second, people involved in real-time programming (such as audio or video) generally want to keep latency to a minimum. If we progress the h/w pointer of ALSA ring buffer at URB completion (and not at URB submission), I don't understand this comment. As far as I can tell, the relevant pointer already is being advanced during URB completion. It's advanced directly after submission. The queued data is double- buffered because the USB API lacks the ability to scatter/gather from a coherent buffer. Actually it does have this ability, but only to a limited extent. I don't remember what the restrictions are (and they probably aren't documented anywhere). Certainly the data for each packet has to be contiguous. There may be an assumption that the packets are stored in sequential increasing order in memory, and there may be a limitation on the distance between the starts of neighboring packets. I'm also not sure to what extent these restrictions could be relaxed. this shouldn't affect the latency. How would this make any difference? The time between the application writing samples to the buffer and the samples actually being played by the device would not change. This depends on how rapidly the program writes the samples. If they are written at about the same rate as they are played out then using a longer queue _will_ increase the latency, because the samples won't start playing until the queue is full. (I don't see how this is related to whether the buffer pointer is advanced at URB submission vs. URB completion, but never mind...) Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Tue, 4 Sep 2012, Peter Chen wrote: In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. It is ISO transfer, if the delay is too much, and the buffer at device side is empty, it is normal the screen is stopped like we watch movie on Internet (buffering). So if the delay is too much, I don't think there is a way can deal with it as host does not send any data to device. Yes, some data is lost. We want to make sure that when the data starts flowing again, it is properly synchronized. Furthermore, if packets are lost, frequency feedback becomes impossible because the device doesn't know how many samples were lost. Where the packets are lost? If the packets are lost at class driver/usb driver, class driver will know it and should take the responsible to re-send it. If the packets are lost on the USB bus (during the transfer), as it is ISO transfer, then the data should be lost, and host doesn't know the data is lost, how can it re-sends the packet? Data should not be resent in any case. Isochronous data is never resent. Here's the problem we face: The class driver submits packets 1, 2, 3, and 4. They get sent properly, but the completion interrupt is delayed. As a result, the class driver's completion handler doesn't get called until too late; the frames for packets 5 - 44 have already expired. The data that should have been sent during those frames is lost. But the class driver doesn't know this, so when the completion handler does get called, it submits packets 5, 6, 7, and 8. They end up getting sent in frames 45, 46, 47, and 48. This means the data is now out of synchronization by 40 frames. That's the problem I want to solve. There's an equivalent problem for data flowing into the host. In this case, the URB that should have been submitted for frame 5 would receive data that the device sent during frame 45. (Whether this data contains the signals for frame 5 or for frame 45 would depend on the device; the host can't do anything about it.) Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Tue, 4 Sep 2012, Jassi Brar wrote: How about effectively increasing the queue length from 10ms to 50ms (max anticipated latency) ? There are two problems with that approach. First, 50 ms isn't really the max anticipated latency; it's merely the largest that I've seen so far. (In fact, the max anticipated latency is probably 10 ms; these 50-ms delays were definitely exceptional.) It doesn't have to be hardcoded - maybe USB audio/video code could get the hint via some module parameter? That's even worse. Fortunately it isn't necessary; ehci-hcd uses an I/O watchdog timer so the latency can never be much longer than 100 ms. Second, people involved in real-time programming (such as audio or video) generally want to keep latency to a minimum. If we progress the h/w pointer of ALSA ring buffer at URB completion (and not at URB submission), this shouldn't affect the latency. And I don't understand this comment. As far as I can tell, the relevant pointer already is being advanced during URB completion. IIRC, USB isn't anyway recommended for real-time usage. People still want to minimize latency, though. I doubt that anybody would be happy with 100 ms latency. Imagine trying to talk over a loudspeaker system if the sounds coming out of the speaker were 100 ms behind your voice. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Alan Stern wrote: How should the lower USB layers handle delays in transferring isochronous data? [...] when this happens the endpoint's queue drains completely. Clearly this will cause a glitch in the data stream. The question is: What should we do to recover and re-synchronize? The ALSA API promises that the samples in a stream run at a fixed clock, i.e., samples are never delayed or accelerated. We could pretend nothing happened and continue to handle URBs normally, scheduling each submission for the next available slot. But for an isochronous-OUT transfer, this would mean that all the future data values are delayed by some 40 ms relative to the earlier data. If another glitch occurs then the following data will be delayed by even more. For some applications this might not matter, but for others (real-time things like voice) it could be quite bad. Similar problems arise with IN transfers. This would not be acceptable; future samples must be scheduled for the same frame for which they would have been scheduled without the delay. Alternatively, the host controller driver could fail the next 40 ms worth of isochronous URBs, so that the higher-level client catches up to where it should be. The failure could occur during submission, or the URBs could be accepted and then forced to complete immediately, with no data transferred. Both would be possible. At the moment, a submission failure would cause the driver to stop the stream and report an error to the application. In this kind of situation (where it's known that multiple packets have not been transferred), it would be somewhat preferrable to report the error instead of ignoring it. What's the right thing to do? My feeling is that the behavior should be decided not by the host controller driver but rather by the higher-level client. Yeah, schedule directly after the previous packet, or fail and schedule for the next possible frame might be useful. (But the audio driver would always use the first option.) Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Peter Chen wrote: How should the lower USB layers handle delays in transferring isochronous data? I have seen reports from users where URB completion interrupts were delayed by as much as 50 ms. What should we do to recover and re-synchronize? I am not sure if feedback endpoint is implemented at our ISO-transfer class driver. If it is implemented, the class driver will take responsible to speed up/slow down transferring according to the device's feedback information. In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. Furthermore, if packets are lost, frequency feedback becomes impossible because the device doesn't know how many samples were lost. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Mon, 3 Sep 2012, Clemens Ladisch wrote: Alan Stern wrote: How should the lower USB layers handle delays in transferring isochronous data? [...] when this happens the endpoint's queue drains completely. Clearly this will cause a glitch in the data stream. The question is: What should we do to recover and re-synchronize? The ALSA API promises that the samples in a stream run at a fixed clock, i.e., samples are never delayed or accelerated. We could pretend nothing happened and continue to handle URBs normally, scheduling each submission for the next available slot. But for an isochronous-OUT transfer, this would mean that all the future data values are delayed by some 40 ms relative to the earlier data. If another glitch occurs then the following data will be delayed by even more. For some applications this might not matter, but for others (real-time things like voice) it could be quite bad. Similar problems arise with IN transfers. This would not be acceptable; future samples must be scheduled for the same frame for which they would have been scheduled without the delay. I suspected as much. It's good to have the suspicions confirmed. Alternatively, the host controller driver could fail the next 40 ms worth of isochronous URBs, so that the higher-level client catches up to where it should be. The failure could occur during submission, or the URBs could be accepted and then forced to complete immediately, with no data transferred. Both would be possible. At the moment, a submission failure would cause the driver to stop the stream and report an error to the application. A submission failure is cleaner than an immediate completion, because we could then avoid making a whole series of doomed submissions (and using up a lot of stack space). On the other hand, it would be necessary to tell the client how many slots have to be skipped. Any idea how to do that? And of course, it would be necessary to change the clients to take this new information and new failure mode into account. In this kind of situation (where it's known that multiple packets have not been transferred), it would be somewhat preferrable to report the error instead of ignoring it. That's a good idea. I can add a warning message to ehci-hcd. Should it trigger when any packets have been skipped, or only when the number of skipped packets is so large that an entire URB has to be rejected? What's the right thing to do? My feeling is that the behavior should be decided not by the host controller driver but rather by the higher-level client. Yeah, schedule directly after the previous packet, or fail and schedule for the next possible frame might be useful. (But the audio driver would always use the first option.) I'll use URB_ISO_ASAP to mean the next possible slot. Leaving that flag turned off will mean the slot after the previous packet. I can do this for ehci-hcd, ohci-hcd, and uhci-hcd. I don't know if xhci-hcd will be able to support this, however. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Fri, Aug 31, 2012 at 11:56 PM, Alan Stern st...@rowland.harvard.edu wrote: Clemens and Laurent (and anyone else who's interested): How should the lower USB layers handle delays in transferring isochronous data? I'm asking you because the most common usages of isochronous transfers are for audio and video. Here's an example to illustrate what I mean. Typically an audio or video driver will keep a queue of around 10 ms of data submitted to an isochronous endpoint. I have seen reports from users where URB completion interrupts were delayed by as much as 50 ms. In one case the delay was caused by a bug in a wireless drivers that left interrupts disabled; in another case the cause was unknown -- it might have been a hardware problem. At any rate, when this happens the endpoint's queue drains completely. Clearly this will cause a glitch in the data stream. The question is: What should we do to recover and re-synchronize? How about effectively increasing the queue length from 10ms to 50ms (max anticipated latency) ? -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Alan Stern wrote: On Mon, 3 Sep 2012, Clemens Ladisch wrote: Alan Stern wrote: Alternatively, the host controller driver could fail the next 40 ms worth of isochronous URBs, so that the higher-level client catches up to where it should be. The failure could occur during submission, or the URBs could be accepted and then forced to complete immediately, with no data transferred. Both would be possible. At the moment, a submission failure would cause the driver to stop the stream and report an error to the application. A submission failure is cleaner than an immediate completion, because we could then avoid making a whole series of doomed submissions (and using up a lot of stack space). On the other hand, it would be necessary to tell the client how many slots have to be skipped. The audio driver wouldn't care. Logically, it starts a new stream. In this kind of situation (where it's known that multiple packets have not been transferred), it would be somewhat preferrable to report the error instead of ignoring it. That's a good idea. I can add a warning message to ehci-hcd. I meant report to the client as an error, instead of silently dropping packets. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Mon, 3 Sep 2012, Jassi Brar wrote: On Fri, Aug 31, 2012 at 11:56 PM, Alan Stern st...@rowland.harvard.edu wrote: Clemens and Laurent (and anyone else who's interested): How should the lower USB layers handle delays in transferring isochronous data? I'm asking you because the most common usages of isochronous transfers are for audio and video. Here's an example to illustrate what I mean. Typically an audio or video driver will keep a queue of around 10 ms of data submitted to an isochronous endpoint. I have seen reports from users where URB completion interrupts were delayed by as much as 50 ms. In one case the delay was caused by a bug in a wireless drivers that left interrupts disabled; in another case the cause was unknown -- it might have been a hardware problem. At any rate, when this happens the endpoint's queue drains completely. Clearly this will cause a glitch in the data stream. The question is: What should we do to recover and re-synchronize? How about effectively increasing the queue length from 10ms to 50ms (max anticipated latency) ? There are two problems with that approach. First, 50 ms isn't really the max anticipated latency; it's merely the largest that I've seen so far. (In fact, the max anticipated latency is probably 10 ms; these 50-ms delays were definitely exceptional.) Second, people involved in real-time programming (such as audio or video) generally want to keep latency to a minimum. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Mon, 3 Sep 2012, Clemens Ladisch wrote: Alan Stern wrote: On Mon, 3 Sep 2012, Clemens Ladisch wrote: Alan Stern wrote: Alternatively, the host controller driver could fail the next 40 ms worth of isochronous URBs, so that the higher-level client catches up to where it should be. The failure could occur during submission, or the URBs could be accepted and then forced to complete immediately, with no data transferred. Both would be possible. At the moment, a submission failure would cause the driver to stop the stream and report an error to the application. A submission failure is cleaner than an immediate completion, because we could then avoid making a whole series of doomed submissions (and using up a lot of stack space). On the other hand, it would be necessary to tell the client how many slots have to be skipped. The audio driver wouldn't care. Logically, it starts a new stream. Really? Why not just skip a few packets and carry on with the existing stream? Does the behavior vary (or need to vary) according to the stream's direction? Logically, the situation isn't very different from what happens when packets are lost in transit (except for the fact that outgoing packets can be lost without the host's knowledge). Isochronous makes no guarantees about packet delivery, and never retries. In this kind of situation (where it's known that multiple packets have not been transferred), it would be somewhat preferrable to report the error instead of ignoring it. That's a good idea. I can add a warning message to ehci-hcd. I meant report to the client as an error, instead of silently dropping packets. In fact this missing data is already reported. The status member of the usb_iso_packet_descriptor structure returns -EXDEV if a packet was submitted too late for its scheduled time slot. (The error-codes.txt documentation file says partially completed, which isn't a very good description.) My reason for bringing this up is because I want to improve the way ehci-hcd handles such things. Right now it doesn't do a very good job; when faced with delays longer than the built-in slop setting (currently 10 ms) it completely loses track of what's happening. It thinks the late submissions are actually far too early (it's dealing with a hardware schedule arranged as a ring buffer with length 512 or 1024 ms) and never gets back on track. I've got a patch ready for testing that improves this behavior, but it fixes only part of the problem. Resynchronization afterward remains an issue. Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
Alan Stern wrote: On Mon, 3 Sep 2012, Clemens Ladisch wrote: The audio driver wouldn't care. Logically, it starts a new stream. Really? Why not just skip a few packets and carry on with the existing stream? Logically, the situation isn't very different from what happens when packets are lost in transit (except for the fact that outgoing packets can be lost without the host's knowledge). A few lost packets do not affect the timing of completion interrupts/ callbacks. However, when the queuing is broken, the completions arrive in a way that makes the stream to appear to slow down, and then to speed up when the HCD is trying to resynchronize. Some jitter in the reported stream progress is acceptable (and USB audio already is infamous for that, due to the 1 ms frame granularity and the fact that URBs often span many frames). But when the timing errors become too big, synchronization with other real-time events gets lost. Where exactly the boundary between small errors to be ignored and big fatal errors lies is of course rather fuzzy. However, by selecting a particular queue length, drivers have control over this boundary. Does the behavior vary (or need to vary) according to the stream's direction? Oh well. There are no sample counters, so if any capture packet is dropped, we do not know how many samples are missing. (I've never heard of this happening in practice though.) My reason for bringing this up is because I want to improve the way ehci-hcd handles such things. Right now it doesn't do a very good job; when faced with delays longer than the built-in slop setting (currently 10 ms) it completely loses track of what's happening. It thinks the late submissions are actually far too early (it's dealing with a hardware schedule arranged as a ring buffer with length 512 or 1024 ms) and never gets back on track. I've got a patch ready for testing that improves this behavior, but it fixes only part of the problem. Resynchronization afterward remains an issue. From the audio driver's point of view, resynchronization by the HCD is not even necessary. Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
On Mon, 3 Sep 2012, Clemens Ladisch wrote: Alan Stern wrote: On Mon, 3 Sep 2012, Clemens Ladisch wrote: The audio driver wouldn't care. Logically, it starts a new stream. Really? Why not just skip a few packets and carry on with the existing stream? Logically, the situation isn't very different from what happens when packets are lost in transit (except for the fact that outgoing packets can be lost without the host's knowledge). A few lost packets do not affect the timing of completion interrupts/ callbacks. However, when the queuing is broken, the completions arrive in a way that makes the stream to appear to slow down, and then to speed up when the HCD is trying to resynchronize. ... From the audio driver's point of view, resynchronization by the HCD is not even necessary. There are two possibilities to consider when an URB containing several packets of data is submitted: 1. The slots for the first few packets have already expired, but the remaining packets will be transferred okay. 2. The slots for all the packets in the URB have expired. In case 1 there is data loss but the queuing remains intact. In case 2 the queuing is broken. It sounds like you're saying that case 1 submissions should succeed (and return -EXDEV status for the statuses of the missed packets), whereas case 2 submissions should fail outright (say with an -EXDEV error, which is currently not used for URB submission). Then resynchronization becomes the audio driver's problem entirely. Would that be okay? Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
In such a situation, the delay is much bigger than the device's buffer, so just sending more samples afterwards will not help. It is ISO transfer, if the delay is too much, and the buffer at device side is empty, it is normal the screen is stopped like we watch movie on Internet (buffering). So if the delay is too much, I don't think there is a way can deal with it as host does not send any data to device. Furthermore, if packets are lost, frequency feedback becomes impossible because the device doesn't know how many samples were lost. Where the packets are lost? If the packets are lost at class driver/usb driver, class driver will know it and should take the responsible to re-send it. If the packets are lost on the USB bus (during the transfer), as it is ISO transfer, then the data should be lost, and host doesn't know the data is lost, how can it re-sends the packet? Regards, Clemens -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
[CC'ing an USB audio heavyweight - Daniel Mack] On Tue, Sep 4, 2012 at 12:39 AM, Alan Stern st...@rowland.harvard.edu wrote: On Mon, 3 Sep 2012, Jassi Brar wrote: On Fri, Aug 31, 2012 at 11:56 PM, Alan Stern st...@rowland.harvard.edu wrote: Clemens and Laurent (and anyone else who's interested): How should the lower USB layers handle delays in transferring isochronous data? I'm asking you because the most common usages of isochronous transfers are for audio and video. Here's an example to illustrate what I mean. Typically an audio or video driver will keep a queue of around 10 ms of data submitted to an isochronous endpoint. I have seen reports from users where URB completion interrupts were delayed by as much as 50 ms. In one case the delay was caused by a bug in a wireless drivers that left interrupts disabled; in another case the cause was unknown -- it might have been a hardware problem. At any rate, when this happens the endpoint's queue drains completely. Clearly this will cause a glitch in the data stream. The question is: What should we do to recover and re-synchronize? How about effectively increasing the queue length from 10ms to 50ms (max anticipated latency) ? There are two problems with that approach. First, 50 ms isn't really the max anticipated latency; it's merely the largest that I've seen so far. (In fact, the max anticipated latency is probably 10 ms; these 50-ms delays were definitely exceptional.) It doesn't have to be hardcoded - maybe USB audio/video code could get the hint via some module parameter? Second, people involved in real-time programming (such as audio or video) generally want to keep latency to a minimum. If we progress the h/w pointer of ALSA ring buffer at URB completion (and not at URB submission), this shouldn't affect the latency. And IIRC, USB isn't anyway recommended for real-time usage. cheers. -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] How to handle delays in isochronous transfers?
How should the lower USB layers handle delays in transferring isochronous data? I'm asking you because the most common usages of isochronous transfers are for audio and video. Here's an example to illustrate what I mean. Typically an audio or video driver will keep a queue of around 10 ms of data submitted to an isochronous endpoint. I have seen reports from users where URB completion interrupts were delayed by as much as 50 ms. In one case the delay was caused by a bug in a wireless drivers that left interrupts disabled; in another case the cause was unknown -- it might have been a hardware problem. At any rate, when this happens the endpoint's queue drains completely. Clearly this will cause a glitch in the data stream. The question is: What should we do to recover and re-synchronize? I am not sure if feedback endpoint is implemented at our ISO-transfer class driver. If it is implemented, the class driver will take responsible to speed up/slow down transferring according to the device's feedback information. For non-supported feedback device, the device should take responsible for data loss or data too fast by changing codec clock freq or give up some data. Yes, the host controller may know if the data is really on the bus in time, and tell the class driver adjust the packet size, but device may have already changed its codec clock already or its behavior. Both host/device system high loading and clock not always the same between usb and codec at device side will cause the total payload is not the same between producer (host) and consumer (device) after sometime, so the feedback between device and host is the best way to keep data integrity. We could pretend nothing happened and continue to handle URBs normally, scheduling each submission for the next available slot. But for an isochronous-OUT transfer, this would mean that all the future data values are delayed by some 40 ms relative to the earlier data. If another glitch occurs then the following data will be delayed by even more. For some applications this might not matter, but for others (real-time things like voice) it could be quite bad. Similar problems arise with IN transfers. Alternatively, the host controller driver could fail the next 40 ms worth of isochronous URBs, so that the higher-level client catches up to where it should be. The failure could occur during submission, or the URBs could be accepted and then forced to complete immediately, with no data transferred. What's the right thing to do? My feeling is that the behavior should be decided not by the host controller driver but rather by the higher-level client. We could use the URB_ISO_ASAP flag for this purpose -- right now it is essentially useless. Or maybe we should do something else I haven't thought of. What would be the best approach for your purposes? Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC] How to handle delays in isochronous transfers?
Clemens and Laurent (and anyone else who's interested): How should the lower USB layers handle delays in transferring isochronous data? I'm asking you because the most common usages of isochronous transfers are for audio and video. Here's an example to illustrate what I mean. Typically an audio or video driver will keep a queue of around 10 ms of data submitted to an isochronous endpoint. I have seen reports from users where URB completion interrupts were delayed by as much as 50 ms. In one case the delay was caused by a bug in a wireless drivers that left interrupts disabled; in another case the cause was unknown -- it might have been a hardware problem. At any rate, when this happens the endpoint's queue drains completely. Clearly this will cause a glitch in the data stream. The question is: What should we do to recover and re-synchronize? We could pretend nothing happened and continue to handle URBs normally, scheduling each submission for the next available slot. But for an isochronous-OUT transfer, this would mean that all the future data values are delayed by some 40 ms relative to the earlier data. If another glitch occurs then the following data will be delayed by even more. For some applications this might not matter, but for others (real-time things like voice) it could be quite bad. Similar problems arise with IN transfers. Alternatively, the host controller driver could fail the next 40 ms worth of isochronous URBs, so that the higher-level client catches up to where it should be. The failure could occur during submission, or the URBs could be accepted and then forced to complete immediately, with no data transferred. What's the right thing to do? My feeling is that the behavior should be decided not by the host controller driver but rather by the higher-level client. We could use the URB_ISO_ASAP flag for this purpose -- right now it is essentially useless. Or maybe we should do something else I haven't thought of. What would be the best approach for your purposes? Alan Stern -- To unsubscribe from this list: send the line unsubscribe linux-usb in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
