Re: [RFCv11 PATCH 00/29] Request API
Hi Alexandre, On 04/19/2018 09:48 AM, Paul Kocialkowski wrote: > Hi, > > On Wed, 2018-04-18 at 02:06 +, Alexandre Courbot wrote: >> On Tue, Apr 17, 2018 at 8:41 PM Paul Kocialkowski < >> paul.kocialkow...@bootlin.com> wrote: >> On Tue, 2018-04-17 at 06:17 +, Alexandre Courbot wrote: On Tue, Apr 17, 2018 at 3:12 PM Hans Verkuilwrote: > On 04/17/2018 06:33 AM, Alexandre Courbot wrote: >> On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil > nl> >> wrote: >> >>> From: Hans Verkuil >>> Hi all, >>> This is a cleaned up version of the v10 series (never posted >>> to >>> the list since it was messy). >> >> Hi Hans, >> >> It took me a while to test and review this, but finally have >> been >> able to >> do it. >> >> First the result of the test: I have tried porting my dummy >> vim2m >> test >> program >> (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a >> 42 >> for >> reference), >> and am getting a hang when trying to queue the second OUTPUT >> buffer (right >> after >> queuing the first request). If I move the calls the >> VIDIOC_STREAMON after >> the >> requests are queued, the hang seems to happen at that moment. >> Probably a >> deadlock, haven't looked in detail yet. >> >> I have a few other comments, will follow up per-patch. >> > > I had a similar/same (?) report about this from Paul: > https://www.mail-archive.com/linux-media@vger.kernel.org/msg1291 > 77.h > tml I saw this and tried to move the call to STREAMON to after the requests are queued in my example program, but it then hanged there. So there is probably something more intricate taking place. >>> I figured out the issue (but forgot to report back to the list): >>> Hans' >>> version of the request API doesn't set the POLLIN bit but POLLPRI >>> instead, so you need to select for expect_fds instead of read_fds in >>> the >>> select call. That's pretty much all there is to it. >> >> I am not using select() but poll() in my test program (see the gist >> link >> above) and have set POLLPRI as the event to poll for. I may be missing >> something but this looks correct to me? > > You're right, I overlooked your email and assumed you were hitting the > same issue that I had at first. > > Anyway, I also had a similar issue when calling the STREAMON ioctl > *before* enqueuing the request. What happens here is that > v4l2_m2m_streamon (called from the ioctl) will also try to schedule a > device run (v4l2_m2m_try_schedule). When the ioctl is called and the > request was not queued yet, the lack of buffers will delay the streamon > call. Then, when the request is later submitted, vb2_streamon is called > with the queue directly, and there is no m2m-specific provision there, > so no device run is scheduled and nothing happens. If the STREAMON ioctl > happens after queuing the request, then things should be fine (but > that's definitely not what we want userspace to be doing) since > the vb2_streamon call from the ioctl handler will take effect > and v4l2_m2m_try_schedule will be called. > > The way that I have solved this with the Sunxi-Cedrus driver is to add a > req_complete callback function pointer (holding a call > to v4l2_m2m_try_schedule) in media_device_ops and call it (if available) > from media_request_ioctl_queue. I initially put this in req_queue > directly, but since it is wrapped by the request queue mutex lock and > provided that device_run needs to access the request queue, we need an > extra op called out of this lock, before completing the ioctl handler. > > I will be sending v2 of my driver with preliminary patches to fix this > (perhaps I should also fix vim2m that way along the way). Can you verify that this indeed solves your problem? Or is there still something else going on? Regards, Hans
Re: [RFCv11 PATCH 00/29] Request API
Hi, On Wed, 2018-04-18 at 02:06 +, Alexandre Courbot wrote: > On Tue, Apr 17, 2018 at 8:41 PM Paul Kocialkowski < > paul.kocialkow...@bootlin.com> wrote: > On Tue, 2018-04-17 at 06:17 +, Alexandre Courbot wrote: > > > On Tue, Apr 17, 2018 at 3:12 PM Hans Verkuil> > > wrote: > > > > > > > On 04/17/2018 06:33 AM, Alexandre Courbot wrote: > > > > > On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil > > > > nl> > > > > > wrote: > > > > > > > > > > > From: Hans Verkuil > > > > > > Hi all, > > > > > > This is a cleaned up version of the v10 series (never posted > > > > > > to > > > > > > the list since it was messy). > > > > > > > > > > Hi Hans, > > > > > > > > > > It took me a while to test and review this, but finally have > > > > > been > > > > > able > > > > > > to > > > > > do it. > > > > > > > > > > First the result of the test: I have tried porting my dummy > > > > > vim2m > > > > > test > > > > > program > > > > > (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a > > > > > 42 > > > > > for > > > > > reference), > > > > > and am getting a hang when trying to queue the second OUTPUT > > > > > buffer > > > > > > (right > > > > > after > > > > > queuing the first request). If I move the calls the > > > > > VIDIOC_STREAMON > > > > > > after > > > > > the > > > > > requests are queued, the hang seems to happen at that moment. > > > > > Probably a > > > > > deadlock, haven't looked in detail yet. > > > > > > > > > > I have a few other comments, will follow up per-patch. > > > > > > > > > > > > > I had a similar/same (?) report about this from Paul: > > > > https://www.mail-archive.com/linux-media@vger.kernel.org/msg1291 > > > > 77.h > > > > tml > > > > > > I saw this and tried to move the call to STREAMON to after the > > > requests are queued in my example program, but it then hanged > > > there. > > > So there is probably something more intricate taking place. > > I figured out the issue (but forgot to report back to the list): > > Hans' > > version of the request API doesn't set the POLLIN bit but POLLPRI > > instead, so you need to select for expect_fds instead of read_fds in > > the > > select call. That's pretty much all there is to it. > > I am not using select() but poll() in my test program (see the gist > link > above) and have set POLLPRI as the event to poll for. I may be missing > something but this looks correct to me? You're right, I overlooked your email and assumed you were hitting the same issue that I had at first. Anyway, I also had a similar issue when calling the STREAMON ioctl *before* enqueuing the request. What happens here is that v4l2_m2m_streamon (called from the ioctl) will also try to schedule a device run (v4l2_m2m_try_schedule). When the ioctl is called and the request was not queued yet, the lack of buffers will delay the streamon call. Then, when the request is later submitted, vb2_streamon is called with the queue directly, and there is no m2m-specific provision there, so no device run is scheduled and nothing happens. If the STREAMON ioctl happens after queuing the request, then things should be fine (but that's definitely not what we want userspace to be doing) since the vb2_streamon call from the ioctl handler will take effect and v4l2_m2m_try_schedule will be called. The way that I have solved this with the Sunxi-Cedrus driver is to add a req_complete callback function pointer (holding a call to v4l2_m2m_try_schedule) in media_device_ops and call it (if available) from media_request_ioctl_queue. I initially put this in req_queue directly, but since it is wrapped by the request queue mutex lock and provided that device_run needs to access the request queue, we need an extra op called out of this lock, before completing the ioctl handler. I will be sending v2 of my driver with preliminary patches to fix this (perhaps I should also fix vim2m that way along the way). Cheers, -- Paul Kocialkowski, Bootlin (formerly Free Electrons) Embedded Linux and kernel engineering https://bootlin.com signature.asc Description: This is a digitally signed message part
Re: [RFCv11 PATCH 00/29] Request API
On Tue, Apr 17, 2018 at 8:41 PM Paul Kocialkowski < paul.kocialkow...@bootlin.com> wrote: > Hi, > On Tue, 2018-04-17 at 06:17 +, Alexandre Courbot wrote: > > On Tue, Apr 17, 2018 at 3:12 PM Hans Verkuil> > wrote: > > > > > On 04/17/2018 06:33 AM, Alexandre Courbot wrote: > > > > On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil > > > > wrote: > > > > > > > > > From: Hans Verkuil > > > > > Hi all, > > > > > This is a cleaned up version of the v10 series (never posted to > > > > > the list since it was messy). > > > > > > > > Hi Hans, > > > > > > > > It took me a while to test and review this, but finally have been > > > > able > > > > to > > > > do it. > > > > > > > > First the result of the test: I have tried porting my dummy vim2m > > > > test > > > > program > > > > (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42 > > > > for > > > > reference), > > > > and am getting a hang when trying to queue the second OUTPUT > > > > buffer > > > > (right > > > > after > > > > queuing the first request). If I move the calls the > > > > VIDIOC_STREAMON > > > > after > > > > the > > > > requests are queued, the hang seems to happen at that moment. > > > > Probably a > > > > deadlock, haven't looked in detail yet. > > > > > > > > I have a few other comments, will follow up per-patch. > > > > > > > I had a similar/same (?) report about this from Paul: > > > https://www.mail-archive.com/linux-media@vger.kernel.org/msg129177.h > > > tml > > > > I saw this and tried to move the call to STREAMON to after the > > requests are queued in my example program, but it then hanged there. > > So there is probably something more intricate taking place. > I figured out the issue (but forgot to report back to the list): Hans' > version of the request API doesn't set the POLLIN bit but POLLPRI > instead, so you need to select for expect_fds instead of read_fds in the > select call. That's pretty much all there is to it. I am not using select() but poll() in my test program (see the gist link above) and have set POLLPRI as the event to poll for. I may be missing something but this looks correct to me?
Re: [RFCv11 PATCH 00/29] Request API
Hi, On Tue, 2018-04-17 at 06:17 +, Alexandre Courbot wrote: > On Tue, Apr 17, 2018 at 3:12 PM Hans Verkuil> wrote: > > > On 04/17/2018 06:33 AM, Alexandre Courbot wrote: > > > On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil > > > wrote: > > > > > > > From: Hans Verkuil > > > > Hi all, > > > > This is a cleaned up version of the v10 series (never posted to > > > > the list since it was messy). > > > > > > Hi Hans, > > > > > > It took me a while to test and review this, but finally have been > > > able > > to > > > do it. > > > > > > First the result of the test: I have tried porting my dummy vim2m > > > test > > > program > > > (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42 > > > for > > > reference), > > > and am getting a hang when trying to queue the second OUTPUT > > > buffer > > (right > > > after > > > queuing the first request). If I move the calls the > > > VIDIOC_STREAMON > > after > > > the > > > requests are queued, the hang seems to happen at that moment. > > > Probably a > > > deadlock, haven't looked in detail yet. > > > > > > I have a few other comments, will follow up per-patch. > > > > > I had a similar/same (?) report about this from Paul: > > https://www.mail-archive.com/linux-media@vger.kernel.org/msg129177.h > > tml > > I saw this and tried to move the call to STREAMON to after the > requests are queued in my example program, but it then hanged there. > So there is probably something more intricate taking place. I figured out the issue (but forgot to report back to the list): Hans' version of the request API doesn't set the POLLIN bit but POLLPRI instead, so you need to select for expect_fds instead of read_fds in the select call. That's pretty much all there is to it. Hope this helps, -- Paul Kocialkowski, Bootlin (formerly Free Electrons) Embedded Linux and kernel engineering https://bootlin.com signature.asc Description: This is a digitally signed message part
Re: [RFCv11 PATCH 00/29] Request API
On Tue, Apr 17, 2018 at 3:12 PM Hans Verkuilwrote: > On 04/17/2018 06:33 AM, Alexandre Courbot wrote: > > On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil wrote: > > > >> From: Hans Verkuil > > > >> Hi all, > > > >> This is a cleaned up version of the v10 series (never posted to > >> the list since it was messy). > > > > Hi Hans, > > > > It took me a while to test and review this, but finally have been able to > > do it. > > > > First the result of the test: I have tried porting my dummy vim2m test > > program > > (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42 for > > reference), > > and am getting a hang when trying to queue the second OUTPUT buffer (right > > after > > queuing the first request). If I move the calls the VIDIOC_STREAMON after > > the > > requests are queued, the hang seems to happen at that moment. Probably a > > deadlock, haven't looked in detail yet. > > > > I have a few other comments, will follow up per-patch. > > > I had a similar/same (?) report about this from Paul: > https://www.mail-archive.com/linux-media@vger.kernel.org/msg129177.html I saw this and tried to move the call to STREAMON to after the requests are queued in my example program, but it then hanged there. So there is probably something more intricate taking place.
Re: [RFCv11 PATCH 00/29] Request API
On 04/17/2018 06:33 AM, Alexandre Courbot wrote: > On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuilwrote: > >> From: Hans Verkuil > >> Hi all, > >> This is a cleaned up version of the v10 series (never posted to >> the list since it was messy). > > Hi Hans, > > It took me a while to test and review this, but finally have been able to > do it. > > First the result of the test: I have tried porting my dummy vim2m test > program > (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42 for > reference), > and am getting a hang when trying to queue the second OUTPUT buffer (right > after > queuing the first request). If I move the calls the VIDIOC_STREAMON after > the > requests are queued, the hang seems to happen at that moment. Probably a > deadlock, haven't looked in detail yet. > > I have a few other comments, will follow up per-patch. > I had a similar/same (?) report about this from Paul: https://www.mail-archive.com/linux-media@vger.kernel.org/msg129177.html I hope to resume my Request API work on Thursday. Regards, Hans
Re: [RFCv11 PATCH 00/29] Request API
On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuilwrote: > From: Hans Verkuil > Hi all, > This is a cleaned up version of the v10 series (never posted to > the list since it was messy). Hi Hans, It took me a while to test and review this, but finally have been able to do it. First the result of the test: I have tried porting my dummy vim2m test program (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42 for reference), and am getting a hang when trying to queue the second OUTPUT buffer (right after queuing the first request). If I move the calls the VIDIOC_STREAMON after the requests are queued, the hang seems to happen at that moment. Probably a deadlock, haven't looked in detail yet. I have a few other comments, will follow up per-patch.
Re: [RFCv11 PATCH 00/29] Request API
Em Mon, 9 Apr 2018 16:19:57 +0200 Hans Verkuilescreveu: > From: Hans Verkuil > > Hi all, > > This is a cleaned up version of the v10 series (never posted to > the list since it was messy). Just reviewed the full series. I found v11 still a little messy, specially at the part that touches VB2 core. I wasn't capable of understanding the changes there on a reasonable time. As mentioned there, please split it into more palatable patches :-) > > The main changes compared to v9 are in the control framework which > is (hopefully!) now in sync with the RFC. Specifically, when queueing > a request it will 'chain' itself correctly with the previously queued > request. Control values that were not explicitly set in the request > will get the value from the first request in the queue that sets it, > or the hardware value if no request in the queue ever touches it. > > However, I have not yet had the opportunity to test this in > v4l2-compliance! You should. I'd expect some troubles, expecially when read() interface is used. > > Sakari: I did not have the time to incorporate your comments. I'll > probably wait until I have more feedback and then post a new series > next week. > > Other changes: > > - various request state checks were missing (i.e. you could set > a control in a queued request). > > - a new cancel op was added to handle the corner case where a > request was queued but never reached the driver since STREAMOFF > was called before the buffers were ever queued in the driver. > > - various random fixes. > > - added the patch "videobuf2-v4l2: export request_fd" as requested > by Sakari. > > - changed some inconsistent error codes. - Review locks. IMHO, they're currently broken. > > This has been tested with vim2m and vivid using v4l2-compliance. > The v4l-utils repo supporting requests is here: > https://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=request > > TODO/Remarks: > > 1) missing prototype documentation in media-requests.c/h. Some >is documented, but not everything. > > 2) No VIDIOC_REQUEST_ALLOC 'shortcut' ioctl. Sorry, I just ran out >of time. Alexandre, Tomasz, feel free to add it back (it should >be quite easy to do) and post a patch. I'll add it to my patch >series. As mentioned before: whether or not we actually want this >has not been decided yet. > > 3) vim2m: the media topology is a bit bogus, this needs to be fixed >(i.e. a proper HW entity should be added). But for now it is >good enough for testing. > > 4) I think this should slide in fairly easy after the fence support >is merged. I made sure the request API changes in public headers >did not clash with the changes made by the fence API. > > 5) I did not verify the Request API documentation patch. I did update >it with the new buffer flags and 'which' value, but it probably is >out of date in places. That's why I think documentation patch should come first, with the "basic" stuff, and, as new features/API headers are touched, more stuff gets added there. > > 6) More testing. I think I have fairly good coverage in v4l2-compliance, >but no doubt I've missed a few test cases. > > 7) debugfs: it would be really useful to expose the number of request >and request objects in debugfs to simplify debugging. Esp. to make >sure everything is freed correctly. > > 8) Review copyright/authorship lines. I'm not sure if everything is >correct. Alexandre, Sakari, if you see something that is not >right in this respect, just let me know! > > 9) The req_queue op should likely be split into a req_validate and >a req_queue op (based on a discussion on the ML with Sakari). >That avoids a race condition. > > Everything seemed to slip nicely into place while working on this, > so I hope this is finally an implementation that we can proceed to > upstream and build upon for complex camera pipelines in the future. > > This patch series is also available here: > > https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=reqv11 > > Regards, > > Hans > > > Alexandre Courbot (3): > videodev2.h: add request_fd field to v4l2_ext_controls > Documentation: v4l: document request API > media: vim2m: add media device > > Hans Verkuil (26): > v4l2-device.h: always expose mdev > uapi/linux/media.h: add request API > media-request: allocate media requests > media-request: core request support > media-request: add request ioctls > media-request: add media_request_find > media-request: add media_request_object_find > videodev2.h: add V4L2_CTRL_FLAG_IN_REQUEST > v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev > v4l2-ctrls: prepare internal structs for request API > v4l2-ctrls: alloc memory for p_req > v4l2-ctrls: use ref in helper instead of ctrl > v4l2-ctrls: add core request support > v4l2-ctrls: support g/s_ext_ctrls for requests > videodev2.h: Add request_fd
[RFCv11 PATCH 00/29] Request API
From: Hans VerkuilHi all, This is a cleaned up version of the v10 series (never posted to the list since it was messy). The main changes compared to v9 are in the control framework which is (hopefully!) now in sync with the RFC. Specifically, when queueing a request it will 'chain' itself correctly with the previously queued request. Control values that were not explicitly set in the request will get the value from the first request in the queue that sets it, or the hardware value if no request in the queue ever touches it. However, I have not yet had the opportunity to test this in v4l2-compliance! Sakari: I did not have the time to incorporate your comments. I'll probably wait until I have more feedback and then post a new series next week. Other changes: - various request state checks were missing (i.e. you could set a control in a queued request). - a new cancel op was added to handle the corner case where a request was queued but never reached the driver since STREAMOFF was called before the buffers were ever queued in the driver. - various random fixes. - added the patch "videobuf2-v4l2: export request_fd" as requested by Sakari. - changed some inconsistent error codes. This has been tested with vim2m and vivid using v4l2-compliance. The v4l-utils repo supporting requests is here: https://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=request TODO/Remarks: 1) missing prototype documentation in media-requests.c/h. Some is documented, but not everything. 2) No VIDIOC_REQUEST_ALLOC 'shortcut' ioctl. Sorry, I just ran out of time. Alexandre, Tomasz, feel free to add it back (it should be quite easy to do) and post a patch. I'll add it to my patch series. As mentioned before: whether or not we actually want this has not been decided yet. 3) vim2m: the media topology is a bit bogus, this needs to be fixed (i.e. a proper HW entity should be added). But for now it is good enough for testing. 4) I think this should slide in fairly easy after the fence support is merged. I made sure the request API changes in public headers did not clash with the changes made by the fence API. 5) I did not verify the Request API documentation patch. I did update it with the new buffer flags and 'which' value, but it probably is out of date in places. 6) More testing. I think I have fairly good coverage in v4l2-compliance, but no doubt I've missed a few test cases. 7) debugfs: it would be really useful to expose the number of request and request objects in debugfs to simplify debugging. Esp. to make sure everything is freed correctly. 8) Review copyright/authorship lines. I'm not sure if everything is correct. Alexandre, Sakari, if you see something that is not right in this respect, just let me know! 9) The req_queue op should likely be split into a req_validate and a req_queue op (based on a discussion on the ML with Sakari). That avoids a race condition. Everything seemed to slip nicely into place while working on this, so I hope this is finally an implementation that we can proceed to upstream and build upon for complex camera pipelines in the future. This patch series is also available here: https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=reqv11 Regards, Hans Alexandre Courbot (3): videodev2.h: add request_fd field to v4l2_ext_controls Documentation: v4l: document request API media: vim2m: add media device Hans Verkuil (26): v4l2-device.h: always expose mdev uapi/linux/media.h: add request API media-request: allocate media requests media-request: core request support media-request: add request ioctls media-request: add media_request_find media-request: add media_request_object_find videodev2.h: add V4L2_CTRL_FLAG_IN_REQUEST v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev v4l2-ctrls: prepare internal structs for request API v4l2-ctrls: alloc memory for p_req v4l2-ctrls: use ref in helper instead of ctrl v4l2-ctrls: add core request support v4l2-ctrls: support g/s_ext_ctrls for requests videodev2.h: Add request_fd field to v4l2_buffer vb2: store userspace data in vb2_v4l2_buffer videobuf2-core: embed media_request_object videobuf2-core: integrate with media requests videobuf2-v4l2: integrate with media requests videobuf2-core: add vb2_core_request_has_buffers videobuf2-v4l2: add vb2_request_queue helper videobuf2-v4l2: export request_fd vim2m: use workqueue vim2m: support requests vivid: add mc vivid: add request support Documentation/media/uapi/v4l/buffer.rst| 19 +- Documentation/media/uapi/v4l/common.rst| 1 + Documentation/media/uapi/v4l/request-api.rst | 199 + Documentation/media/uapi/v4l/user-func.rst | 1 + .../media/uapi/v4l/vidioc-g-ext-ctrls.rst | 22 +- .../media/uapi/v4l/vidioc-new-request.rst | 64 +++ Documentation/media/uapi/v4l/vidioc-qbuf.rst