Hi Alan, On Thursday 13 September 2012 12:08:35 Alan Stern wrote: > On Thu, 13 Sep 2012, Laurent Pinchart wrote: > > Hi Alan, > > > > Sorry for the late answer, I've been travelling with no internet > > connection. > > No problem; thanks for joining the discussion. > > > For video ISO IN problems can be split in two categories. > > > > Small delays (< one video frame) should already be properly supported as > > far as I can see. Data will be lost, drivers will detect that, either > > immediately if the protocol allows such detection, or at some video frame > > boundary. This will result in a corrupt video frame with missing data, > > and drivers will mark it as such when passing it to userspace through > > V4L2. > > Is this true even when the lost packets span a video frame boundary?
Yes. Each ISO packet starts with a header that includes a frame ID bit toggled with every video frame. When receiving the first ISO packet of the next video frame the driver will mark the current video buffer as complete and flag an error as its size will be lower than expected (this is valid for uncompressed formats only, missing data in compressed video can't be detected). The next video frame will be missing data as well (at the start of the frame now), and will be marked as erroneous as well based on its size. Applications will thus receive two erroneous video frames. > > Larger delays that result in completely dropped video frames can cause > > more serious problems. Drivers won't know how many video frames have been > > dropped, and the V4L2 video frame sequence number will get out of sync. > > Applications can work around the problem using the video frame > > timestamps, but that's more a hack than a proper solution. > > > > I'm not sure if there is a way to solve the problem properly. ISO IN with > > video typically uses the ASAP flag, > > There's no particular reason for this, is there? Currently the flag doesn't > really do anything. To be honest I thought the ASAP flag did what it now does with your patch :-) The uvcvideo driver doesn't care much about when URBs will be transferred, I just set the frame number to 0 and set the ASAP flag. > > and making the USB core report an artificial number of dropped ISO > > packets won't help much as many (most/all ?) devices transmit video > > frames in a variable number of ISO packets. > > If the number of lost packets is small enough, the video driver can get > it directly by looking at the status fields in the isochronous packet > descriptor structures. If the number is too big, though, it won't be > reported at all -- the driver will know only that a big gap occurred. > From what you say, this would be okay. Using timestamps the lost frames can be detected, assuming the frame rate is constant. With a variable frame rate we're probably screwed. > > For video output the situation is currently unclear. The only V4L2 output > > driver for USB devices that I know of is the uvcvideo driver, and the only > > UVC output device I know of uses bulk transfers. > > > > Assuming we will get video ISO OUT devices in the future, the right way to > > fix the problem will be to drop complete video frames on the host side if > > a large delays occurs. Feedback endpoints (currently not supported in the > > uvcvideo driver) might help (I've never worked with them, so I'd have to > > take a closer look). > > As Clemens mentioned, feedback endpoints are meant to help with small > differences in clock rates between the host and the device. They are not > meant to help with large gaps in the data stream. > > > Another option would be to just check timestamps on the host side when > > starting a new video frame. I'm not sure whether the USB core could help > > with this problem. > > The best solution may be, if an URB submission fails with a -EXDEV error, to > move forward to the next video frame boundary and restart transmitting from > there. Depending on the protocol, you might want to delay the next URB > submission so that the device doesn't see two frame boundaries too close > together. Either way, you would set the URB_ISO_ASAP flag for the first URB > and leave it turned off for all the following URBs. I'm not sure to get that. Why should it be turned off for the following URBs ? > Basically what I'm asking is whether the API described in the documentation > RFC (http://marc.info/?l=linux-usb&m=134382746009339&w=2) That's "[PATCH] usb: storage: stop all current urbs when device is disconnected". I'm pretty sure you meant something else :-) > would be suitable for your needs. The main point is that there should be a > way to let class drivers know when a large gap in the isochronous stream has > occurred. In the absence of such a mechanism the driver would proceed > merrily on its way, totally unaware that the stream had lost > synchronization. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
