Re: Invite for IRC meeting: Re: [PATCHv4 01/10] videodev2.h: add tag support

[Alexandre Courbot]

Hi Hans,

On Fri, Dec 7, 2018 at 12:08 AM Hans Verkuil  wrote:
>
> Mauro raised a number of objections on irc regarding tags:
>
> https://linuxtv.org/irc/irclogger_log/media-maint?date=2018-12-06,Thu
>
> I would like to setup an irc meeting to discuss this and come to a
> conclusion, since we need to decide this soon since this is critical
> for stateless codec support.
>
> Unfortunately timezone-wise this is a bit of a nightmare. I think
> that at least Mauro, myself and Tomasz Figa should be there, so UTC-2,
> UTC+1 and UTC+9 (if I got that right).
>
> I propose 9 AM UTC which I think will work for everyone except Nicolas.
> Any day next week works for me, and (for now) as well for Mauro. Let's pick
> Monday to start with, and if you want to join in, then let me know. If that
> day doesn't work for you, let me know what other days next week do work for
> you.

Monday (or any other day next week) 9AM UTC should work for me!

Thanks,
Alex.

Re: [PATCHv2 0/9] vb2/cedrus: add tag support

[Alexandre Courbot]

On Wed, Nov 14, 2018 at 10:47 PM Hans Verkuil  wrote:
>
> From: Hans Verkuil 
>
> As was discussed here (among other places):
>
> https://lkml.org/lkml/2018/10/19/440
>
> using capture queue buffer indices to refer to reference frames is
> not a good idea. A better idea is to use a 'tag' where the
> application can assign a u64 tag to an output buffer, which is then
> copied to the capture buffer(s) derived from the output buffer.
>
> A u64 is chosen since this allows userspace to also use pointers to
> internal structures as 'tag'.
>
> The first three patches add core tag support, the next patch document
> the tag support, then a new helper function is added to v4l2-mem2mem.c
> to easily copy data from a source to a destination buffer that drivers
> can use.
>
> Next a new supports_tags vb2_queue flag is added to indicate that
> the driver supports tags. Ideally this should not be necessary, but
> that would require that all m2m drivers are converted to using the
> new helper function introduced in the previous patch. That takes more
> time then I have now since we want to get this in for 4.20.
>
> Finally the vim2m, vicodec and cedrus drivers are converted to support
> tags.
>
> I also removed the 'pad' fields from the mpeg2 control structs (it
> should never been added in the first place) and aligned the structs
> to a u32 boundary (u64 for the tag values).
>
> Note that this might change further (Paul suggested using bitfields).
>
> Also note that the cedrus code doesn't set the sequence counter, that's
> something that should still be added before this driver can be moved
> out of staging.
>
> Note: if no buffer is found for a certain tag, then the dma address
> is just set to 0. That happened before as well with invalid buffer
> indices. This should be checked in the driver!
>
> The previous RFC series was tested successfully with the cedrus driver.
>
> Regards,
>
> Hans
>
> Changes since v1:
>
> - changed to a u32 tag. Using a 64 bit tag was overly complicated due
>   to the bad layout of the v4l2_buffer struct, and there is no real
>   need for it by applications.
>
> Main changes since the RFC:
>
> - Added new buffer capability flag
> - Added m2m helper to copy data between buffers
> - Added documentation
> - Added tag logging in v4l2-ioctl.c
>
> Hans Verkuil (9):
>   videodev2.h: add tag support
>   vb2: add tag support
>   v4l2-ioctl.c: log v4l2_buffer tag
>   buffer.rst: document the new buffer tag feature.
>   v4l2-mem2mem: add v4l2_m2m_buf_copy_data helper function
>   vb2: add new supports_tags queue flag
>   vim2m: add tag support
>   vicodec: add tag support
>   cedrus: add tag support

Good call on the v4l2_m2m_buf_copy_data() function!

Reviewed-by: Alexandre Courbot 

>
>  Documentation/media/uapi/v4l/buffer.rst   | 32 +
>  .../media/uapi/v4l/vidioc-reqbufs.rst |  4 ++
>  .../media/common/videobuf2/videobuf2-v4l2.c   | 45 ---
>  drivers/media/platform/vicodec/vicodec-core.c | 14 ++
>  drivers/media/platform/vim2m.c| 14 ++
>  drivers/media/v4l2-core/v4l2-ctrls.c  |  9 
>  drivers/media/v4l2-core/v4l2-ioctl.c  |  9 ++--
>  drivers/media/v4l2-core/v4l2-mem2mem.c| 23 ++
>  drivers/staging/media/sunxi/cedrus/cedrus.h   |  9 ++--
>  .../staging/media/sunxi/cedrus/cedrus_dec.c   |  2 +
>  .../staging/media/sunxi/cedrus/cedrus_mpeg2.c | 21 -
>  .../staging/media/sunxi/cedrus/cedrus_video.c |  2 +
>  include/media/v4l2-mem2mem.h  | 21 +
>  include/media/videobuf2-core.h|  2 +
>  include/media/videobuf2-v4l2.h| 21 -
>  include/uapi/linux/v4l2-controls.h| 14 +++---
>  include/uapi/linux/videodev2.h|  9 +++-
>  17 files changed, 178 insertions(+), 73 deletions(-)
>
> --
> 2.19.1
>

Re: [RFC PATCH 0/5] vb2/cedrus: add cookie support

[Alexandre Courbot]

Hi Hans,

On Fri, Nov 9, 2018 at 6:56 PM Hans Verkuil  wrote:
>
> As was discussed here (among other places):
>
> https://lkml.org/lkml/2018/10/19/440
>
> using capture queue buffer indices to refer to reference frames is
> not a good idea. A better idea is to use 'cookies' (a better name is
> welcome!)

Maybe "tag" is more common for that kind of usage, but "cookie" is
fine as well IMHO.

I can only comment on patches 1-4, but so far it seems to me that this
would work. I will use this to base the next stateless codec API RFC.

Thanks!
Alex.

Re: [RFC PATCH 1/5] videodev2.h: add cookie support

[Alexandre Courbot]

On Fri, Nov 9, 2018 at 6:56 PM Hans Verkuil  wrote:
>
> From: Hans Verkuil 
>
> Add support for 'cookies' to struct v4l2_buffer. These can be used to

This "to" seems unneeded.

Re: [PATCH] media: venus: support VB2_USERPTR IO mode

[Alexandre Courbot]

On Thu, Oct 11, 2018 at 5:26 PM Stanimir Varbanov
 wrote:
>
> Hi Alex,
>
> On 10/11/2018 09:50 AM, Alexandre Courbot wrote:
> > Please ignore this patch - I did not notice that a similar one has
> > been sent before.
>
> The difference is that you made it for decoder as well. Do you need
> userptr for decoder?

Not at the moment, so this part has not been tested. Better to take
Malathi's patch.

Re: [PATCH] media: venus: add support for USERPTR to queue

[Alexandre Courbot]

On Tue, Oct 9, 2018 at 4:52 PM Malathi Gottam  wrote:
>
> Add USERPTR to queue access methods by adding this
> support to io_modes on both the planes.
>
> Signed-off-by: Malathi Gottam 

Tested-by: Alexandre Courbot 

Re: [PATCH] media: venus: support VB2_USERPTR IO mode

[Alexandre Courbot]

Please ignore this patch - I did not notice that a similar one has
been sent before. >_<
On Thu, Oct 11, 2018 at 3:46 PM Alexandre Courbot  wrote:
>
> The venus codec can work just fine with USERPTR buffers. Enable this
> possibility.
>
> Signed-off-by: Alexandre Courbot 
> ---
>  drivers/media/platform/qcom/venus/vdec.c | 4 ++--
>  drivers/media/platform/qcom/venus/venc.c | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/media/platform/qcom/venus/vdec.c 
> b/drivers/media/platform/qcom/venus/vdec.c
> index 33320c5025313f..dfc2260e8d213a 100644
> --- a/drivers/media/platform/qcom/venus/vdec.c
> +++ b/drivers/media/platform/qcom/venus/vdec.c
> @@ -984,7 +984,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
> *src_vq,
> int ret;
>
> src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> -   src_vq->io_modes = VB2_MMAP | VB2_DMABUF;
> +   src_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
> src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
> src_vq->ops = _vb2_ops;
> src_vq->mem_ops = _dma_sg_memops;
> @@ -999,7 +999,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
> *src_vq,
> return ret;
>
> dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> -   dst_vq->io_modes = VB2_MMAP | VB2_DMABUF;
> +   dst_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
> dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
> dst_vq->ops = _vb2_ops;
> dst_vq->mem_ops = _dma_sg_memops;
> diff --git a/drivers/media/platform/qcom/venus/venc.c 
> b/drivers/media/platform/qcom/venus/venc.c
> index d2805b5e28a1b2..71ca59a24991be 100644
> --- a/drivers/media/platform/qcom/venus/venc.c
> +++ b/drivers/media/platform/qcom/venus/venc.c
> @@ -1073,7 +1073,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
> *src_vq,
> int ret;
>
> src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> -   src_vq->io_modes = VB2_MMAP | VB2_DMABUF;
> +   src_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
> src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
> src_vq->ops = _vb2_ops;
> src_vq->mem_ops = _dma_sg_memops;
> @@ -1090,7 +1090,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
> *src_vq,
> return ret;
>
> dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> -   dst_vq->io_modes = VB2_MMAP | VB2_DMABUF;
> +   dst_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
> dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
> dst_vq->ops = _vb2_ops;
> dst_vq->mem_ops = _dma_sg_memops;
> --
> 2.19.0.605.g01d371f741-goog
>

[PATCH] media: venus: support VB2_USERPTR IO mode

[Alexandre Courbot]

The venus codec can work just fine with USERPTR buffers. Enable this
possibility.

Signed-off-by: Alexandre Courbot 
---
 drivers/media/platform/qcom/venus/vdec.c | 4 ++--
 drivers/media/platform/qcom/venus/venc.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/qcom/venus/vdec.c 
b/drivers/media/platform/qcom/venus/vdec.c
index 33320c5025313f..dfc2260e8d213a 100644
--- a/drivers/media/platform/qcom/venus/vdec.c
+++ b/drivers/media/platform/qcom/venus/vdec.c
@@ -984,7 +984,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
*src_vq,
int ret;
 
src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
-   src_vq->io_modes = VB2_MMAP | VB2_DMABUF;
+   src_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
src_vq->ops = _vb2_ops;
src_vq->mem_ops = _dma_sg_memops;
@@ -999,7 +999,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
*src_vq,
return ret;
 
dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
-   dst_vq->io_modes = VB2_MMAP | VB2_DMABUF;
+   dst_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
dst_vq->ops = _vb2_ops;
dst_vq->mem_ops = _dma_sg_memops;
diff --git a/drivers/media/platform/qcom/venus/venc.c 
b/drivers/media/platform/qcom/venus/venc.c
index d2805b5e28a1b2..71ca59a24991be 100644
--- a/drivers/media/platform/qcom/venus/venc.c
+++ b/drivers/media/platform/qcom/venus/venc.c
@@ -1073,7 +1073,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
*src_vq,
int ret;
 
src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
-   src_vq->io_modes = VB2_MMAP | VB2_DMABUF;
+   src_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
src_vq->ops = _vb2_ops;
src_vq->mem_ops = _dma_sg_memops;
@@ -1090,7 +1090,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue 
*src_vq,
return ret;
 
dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
-   dst_vq->io_modes = VB2_MMAP | VB2_DMABUF;
+   dst_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
dst_vq->ops = _vb2_ops;
dst_vq->mem_ops = _dma_sg_memops;
-- 
2.19.0.605.g01d371f741-goog

Re: [RFC] Request API and V4L2 capabilities

[Alexandre Courbot]

On Tue, Aug 21, 2018 at 6:15 PM Tomasz Figa  wrote:
>
> On Wed, Aug 15, 2018 at 11:18 PM Hans Verkuil  wrote:
> >
> > On 15/08/18 14:11, Mauro Carvalho Chehab wrote:
> > > Em Sat, 4 Aug 2018 15:50:04 +0200
> > > Hans Verkuil  escreveu:
> > >
> > >> Hi all,
> > >>
> > >> While the Request API patch series addresses all the core API issues, 
> > >> there
> > >> are some high-level considerations as well:
> > >>
> > >> 1) How can the application tell that the Request API is supported and for
> > >>which buffer types (capture/output) and pixel formats?
> > >>
> > >> 2) How can the application tell if the Request API is required as 
> > >> opposed to being
> > >>optional?
> > >
> > > Huh? Why would it be mandatory?
> >
> > It is mandatory for stateless codecs: you can't use them without the 
> > Request API since
> > each frame needs the state as well. If you could make a driver for a 
> > stateless codec
> > without the Request API we wouldn't have had to spend ages on developing it 
> > in the first
> > place, would we? :-)
>
> Technically, one could still do the synchronous S_EXT_CTRL, QBUF,
> DQBUF, S_EXT_CTRL... loop, but I'm not sure if this is something worth
> considering.

Having this option is useful for driver bringup at the very least. And
why add artificial constraints if the two APIs are able to work
independently, albeit in a sub-optimal way?

>
> [snip]
> > >> Regarding point 3: I think this should be documented next to the pixel 
> > >> format. I.e.
> > >> the MPEG-2 Slice format used by the stateless cedrus codec requires the 
> > >> request API
> > >> and that two MPEG-2 controls (slice params and quantization matrices) 
> > >> must be present
> > >> in each request.
> > >
> > > Makes sense to document with the pixel format...
> > >
> > >> I am not sure a control flag (e.g. V4L2_CTRL_FLAG_REQUIRED_IN_REQ) is 
> > >> needed here.
> > >
> > > but it sounds worth to also have a flag.
> >
> > I'll wait to get some more feedback. I don't have a very strong opinion on
> > this.
> >
>
> I don't see any value in a flag for codecs. Querying the controls for
> the flag, if it's already required as a part of the Stateless Codec
> Interface for given pixel format, would only mean some redundant code
> both in kernel and user space.
>
> For other use cases, I'm not sure if we can say that a control is
> really required in a request. I think it should be something for the
> user space to decide, depending on the synchronization (and other)
> needs of given use case.
>
> Best regards,
> Tomasz

Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19

[Alexandre Courbot]

 should allow allow external closed-source algorithms to run as well.
>
> The 3A library API must be standardized, to allow the closed-source
> vendor implementation to be replaced by an open-source implementation
> should someone have the time and energy (and qualifications) to write
> one.
>
> Sandboxed execution of the 3A library must be possible as closed-source
> can't always be blindly trusted. This includes the ability to wrap the
> library in a daemon should the platform's multimedia stack wishes
> and to avoid any direct access to the kernel devices by the 3A library
> itself (all accesses should be marshaled by the camera stack).
>
> Please note that this daemon is *not* a camera daemon that would
> communicates with the V4L2 driver through a custom back channel.
>
> The decision to run the 3A library in a sandboxed process or to call
> it directly from the camera stack should be left to the camera stack
> and to the platform integrator, and should not be visible by the 3A
> library.
>
> The 3A library must be usable on major Linux-based camera stacks (the
> Android and Chrome OS camera HALs are certainly important targets,
> more can be added) unmodified, which will allow usage of the vendor
> binary provided for Chrome OS or Android on regular Linux systems.
>
> It would make sense to design a modular camera stack, and try to make
> most components as platform-independent as possible. This should include:
>
> - the kernel drivers (V4L2-compliant and usable without any closed-source
>   userspace component);
> - the 3A library
> - any other component that could be shared (for instance a possible
>   request API library).
>
> The rest of the code will mostly be glue around those components to
> integrate them in a particular camera stack, and should be as
> platform-agnostic as possible.
>
> In the case of the Android camera HAL, ideally it would be a glue that
> could be used with different camera vendors (probably with some kind of
> vendor-specific configuration, or possibly with a separate vendor-specific
> component to handle pipeline configuration).
>
> 4 Complex camera workshop
> =
>
> The workshop will be happening in Tokyo, Japan, at Jun, 19, at the
> google offices. The location is:
>
> 〒106-6126 Tokyo, Minato, Roppongi, 6 Chome−10−1 Roppongi Hills Mori Tower 44F
>
> 4.1 Physical Attendees
> ==
>
> Tomasz Figa 
> Mauro Carvalho Chehab 
> Kieran Bingham 
> Laurent Pinchart 
> Niklas Söderlund 
> Zheng, Jian Xu Zheng 
>
> Anywone else?

I will probably be there as well.

Alexandre Courbot 

Cheers,
Alex.

>
> 4.2. Attendees Via Google Hangouts
> ==
>
> Hans Verkuil  - Via Google Hangouts - maybe only on 
> afternoon
> Javier Martinez Canillas  - Via Google Hangouts - only 
> on reasonable TZ-compatible-hours
> Ricky - Google camera team in Taipei - Via Google Hangouts
>
> Anywone else?

Re: [PATCH v2 02/10] media-request: Add a request complete operation to allow m2m scheduling

[Alexandre Courbot]

On Fri, Apr 20, 2018 at 12:43 AM Paul Kocialkowski <
paul.kocialkow...@bootlin.com> wrote:

> When using the request API in the context of a m2m driver, the
> operations that come with a m2m run scheduling call in their
> (m2m-specific) ioctl handler are delayed until the request is queued
> (for instance, this includes queuing buffers and streamon).

> Thus, the m2m run scheduling calls are not called in due time since the
> request AP's internal plumbing will (rightfully) use the relevant core
> functions directly instead of the ioctl handler.

> This ends up in a situation where nothing happens if there is no
> run-scheduling ioctl called after queuing the request.

> In order to circumvent the issue, a new media operation is introduced,
> called at the time of handling the media request queue ioctl. It gives
> m2m drivers a chance to schedule a m2m device run at that time.

> The existing req_queue operation cannot be used for this purpose, since
> it is called with the request queue mutex held, that is eventually needed
> in the device_run call to apply relevant controls.

> Signed-off-by: Paul Kocialkowski 
> ---
>   drivers/media/media-request.c | 3 +++
>   include/media/media-device.h  | 2 ++
>   2 files changed, 5 insertions(+)

> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> index 415f7e31019d..28ac5ccfe6a2 100644
> --- a/drivers/media/media-request.c
> +++ b/drivers/media/media-request.c
> @@ -157,6 +157,9 @@ static long media_request_ioctl_queue(struct
media_request *req)
>  media_request_get(req);
>  }

> +   if (mdev->ops->req_complete)
> +   mdev->ops->req_complete(req);
> +
>  return ret;
>   }

> diff --git a/include/media/media-device.h b/include/media/media-device.h
> index 07e323c57202..c7dcf2079cc9 100644
> --- a/include/media/media-device.h
> +++ b/include/media/media-device.h
> @@ -55,6 +55,7 @@ struct media_entity_notify {
>* @req_alloc: Allocate a request
>* @req_free: Free a request
>* @req_queue: Queue a request
> + * @req_complete: Complete a request
>*/
>   struct media_device_ops {
>  int (*link_notify)(struct media_link *link, u32 flags,
> @@ -62,6 +63,7 @@ struct media_device_ops {
>  struct media_request *(*req_alloc)(struct media_device *mdev);
>  void (*req_free)(struct media_request *req);
>  int (*req_queue)(struct media_request *req);
> +   void (*req_complete)(struct media_request *req);

This is called *before* the request is actually run, isn't it? In that
case, wouldn't something like "req_schedule" be less confusing?
req_complete implies that the request is already completed.

Re: [RFCv11 PATCH 00/29] Request API

[Alexandre Courbot]

On Tue, Apr 17, 2018 at 8:41 PM Paul Kocialkowski <
paul.kocialkow...@bootlin.com> wrote:

> Hi,

> On Tue, 2018-04-17 at 06:17 +0000, Alexandre Courbot wrote:
> > On Tue, Apr 17, 2018 at 3:12 PM Hans Verkuil <hverk...@xs4all.nl>
> > wrote:
> >
> > > On 04/17/2018 06:33 AM, Alexandre Courbot wrote:
> > > > On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil <hverk...@xs4all.nl>
> > > > wrote:
> > > >
> > > > > From: Hans Verkuil <hans.verk...@cisco.com>
> > > > > Hi all,
> > > > > This is a cleaned up version of the v10 series (never posted to
> > > > > the list since it was messy).
> > > >
> > > > Hi Hans,
> > > >
> > > > It took me a while to test and review this, but finally have been
> > > > able
> >
> > to
> > > > do it.
> > > >
> > > > First the result of the test: I have tried porting my dummy vim2m
> > > > test
> > > > program
> > > > (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42
> > > > for
> > > > reference),
> > > > and am getting a hang when trying to queue the second OUTPUT
> > > > buffer
> >
> > (right
> > > > after
> > > > queuing the first request). If I move the calls the
> > > > VIDIOC_STREAMON
> >
> > after
> > > > the
> > > > requests are queued, the hang seems to happen at that moment.
> > > > Probably a
> > > > deadlock, haven't looked in detail yet.
> > > >
> > > > I have a few other comments, will follow up per-patch.
> > > >
> > > I had a similar/same (?) report about this from Paul:
> > > https://www.mail-archive.com/linux-media@vger.kernel.org/msg129177.h
> > > tml
> >
> > I saw this and tried to move the call to STREAMON to after the
> > requests are queued in my example program, but it then hanged there.
> > So there is probably something more intricate taking place.

> I figured out the issue (but forgot to report back to the list): Hans'
> version of the request API doesn't set the POLLIN bit but POLLPRI
> instead, so you need to select for expect_fds instead of read_fds in the
> select call. That's pretty much all there is to it.

I am not using select() but poll() in my test program (see the gist link
above) and have set POLLPRI as the event to poll for. I may be missing
something but this looks correct to me?

Re: [RFCv11 PATCH 00/29] Request API

[Alexandre Courbot]

On Tue, Apr 17, 2018 at 3:12 PM Hans Verkuil <hverk...@xs4all.nl> wrote:

> On 04/17/2018 06:33 AM, Alexandre Courbot wrote:
> > On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil <hverk...@xs4all.nl> wrote:
> >
> >> From: Hans Verkuil <hans.verk...@cisco.com>
> >
> >> Hi all,
> >
> >> This is a cleaned up version of the v10 series (never posted to
> >> the list since it was messy).
> >
> > Hi Hans,
> >
> > It took me a while to test and review this, but finally have been able
to
> > do it.
> >
> > First the result of the test: I have tried porting my dummy vim2m test
> > program
> > (https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42 for
> > reference),
> > and am getting a hang when trying to queue the second OUTPUT buffer
(right
> > after
> > queuing the first request). If I move the calls the VIDIOC_STREAMON
after
> > the
> > requests are queued, the hang seems to happen at that moment. Probably a
> > deadlock, haven't looked in detail yet.
> >
> > I have a few other comments, will follow up per-patch.
> >

> I had a similar/same (?) report about this from Paul:

> https://www.mail-archive.com/linux-media@vger.kernel.org/msg129177.html

I saw this and tried to move the call to STREAMON to after the requests are
queued in my example program, but it then hanged there. So there is
probably something more intricate taking place.

Re: [RFCv11 PATCH 25/29] media: vim2m: add media device

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil <hverk...@xs4all.nl> wrote:

> From: Alexandre Courbot <acour...@chromium.org>

> Request API requires a media node. Add one to the vim2m driver so we can
> use requests with it.

> This probably needs a bit more work to correctly represent m2m
> hardware in the media topology.

> Signed-off-by: Alexandre Courbot <acour...@chromium.org>

I don't remember writing this - actually IIRC you came with this patch
initially.

> ---
>   drivers/media/platform/vim2m.c | 43
+-
>   1 file changed, 38 insertions(+), 5 deletions(-)

> diff --git a/drivers/media/platform/vim2m.c
b/drivers/media/platform/vim2m.c
> index 065483e62db4..ef970434af13 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -140,6 +140,10 @@ static struct vim2m_fmt *find_format(struct
v4l2_format *f)
>   struct vim2m_dev {
>  struct v4l2_device  v4l2_dev;
>  struct video_device vfd;
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +   struct media_device mdev;
> +   struct media_padpad[2];
> +#endif

>  atomic_tnum_inst;
>  struct mutexdev_mutex;
> @@ -1000,11 +1004,6 @@ static int vim2m_probe(struct platform_device
*pdev)
>  return -ENOMEM;

>  spin_lock_init(>irqlock);
> -
> -   ret = v4l2_device_register(>dev, >v4l2_dev);
> -   if (ret)
> -   return ret;
> -
>  atomic_set(>num_inst, 0);
>  mutex_init(>dev_mutex);

> @@ -1013,6 +1012,22 @@ static int vim2m_probe(struct platform_device
*pdev)
>  vfd->lock = >dev_mutex;
>  vfd->v4l2_dev = >v4l2_dev;

> +#ifdef CONFIG_MEDIA_CONTROLLER
> +   dev->mdev.dev = >dev;
> +   strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
> +   media_device_init(>mdev);
> +   dev->v4l2_dev.mdev = >mdev;
> +   dev->pad[0].flags = MEDIA_PAD_FL_SINK;
> +   dev->pad[1].flags = MEDIA_PAD_FL_SOURCE;
> +   ret = media_entity_pads_init(>entity, 2, dev->pad);
> +   if (ret)
> +   return ret;
> +#endif
> +
> +   ret = v4l2_device_register(>dev, >v4l2_dev);
> +   if (ret)
> +   goto unreg_media;
> +
>  ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
>  if (ret) {
>  v4l2_err(>v4l2_dev, "Failed to register video
device\n");
> @@ -1034,6 +1049,13 @@ static int vim2m_probe(struct platform_device
*pdev)
>  goto err_m2m;
>  }

> +#ifdef CONFIG_MEDIA_CONTROLLER
> +   /* Register the media device node */
> +   ret = media_device_register(>mdev);
> +   if (ret)
> +   goto err_m2m;
> +#endif
> +
>  return 0;

>   err_m2m:
> @@ -1041,6 +1063,10 @@ static int vim2m_probe(struct platform_device
*pdev)
>  video_unregister_device(>vfd);
>   unreg_dev:
>  v4l2_device_unregister(>v4l2_dev);
> +unreg_media:
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +   media_device_unregister(>mdev);
> +#endif

>  return ret;
>   }
> @@ -1050,6 +1076,13 @@ static int vim2m_remove(struct platform_device
*pdev)
>  struct vim2m_dev *dev = platform_get_drvdata(pdev);

>  v4l2_info(>v4l2_dev, "Removing " MEM2MEM_NAME);
> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +   if (media_devnode_is_registered(dev->mdev.devnode))
> +   media_device_unregister(>mdev);
> +   media_device_cleanup(>mdev);
> +#endif
> +
>  v4l2_m2m_release(dev->m2m_dev);
>  del_timer_sync(>timer);
>  video_unregister_device(>vfd);
> --
> 2.16.3

Re: [RFCv11 PATCH 27/29] vim2m: support requests

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil  wrote:

> From: Hans Verkuil 

> Add support for requests to vim2m.

> Signed-off-by: Hans Verkuil 
> ---
>   drivers/media/platform/vim2m.c | 25 +
>   1 file changed, 25 insertions(+)

> diff --git a/drivers/media/platform/vim2m.c
b/drivers/media/platform/vim2m.c
> index 9b18b32c255d..2dcf0ea85705 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -387,8 +387,26 @@ static void device_run(void *priv)
>  src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
>  dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);

> +   /* Apply request if needed */
> +   if (src_buf->vb2_buf.req_obj.req)
> +   v4l2_ctrl_request_setup(src_buf->vb2_buf.req_obj.req,
> +   >hdl);
> +   if (dst_buf->vb2_buf.req_obj.req &&
> +   dst_buf->vb2_buf.req_obj.req != src_buf->vb2_buf.req_obj.req)
> +   v4l2_ctrl_request_setup(dst_buf->vb2_buf.req_obj.req,
> +   >hdl);

This implies that we can have two different requests active at the same time
for the same device. Do we want to open that can of worms?

Valid scenarios for requests should be clearly defined. In the case of a
simple
buffer processor like vim2m, something like "request required for source
buffer,
optional and must be same as source if specified for destination", looks
simple
and sane.

> +
>  device_process(ctx, src_buf, dst_buf);

> +   /* Complete request if needed */
> +   if (src_buf->vb2_buf.req_obj.req)
> +   v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req,
> +   >hdl);
> +   if (dst_buf->vb2_buf.req_obj.req &&
> +   dst_buf->vb2_buf.req_obj.req != src_buf->vb2_buf.req_obj.req)
> +   v4l2_ctrl_request_complete(dst_buf->vb2_buf.req_obj.req,
> +   >hdl);
> +
>  /* Run a timer, which simulates a hardware irq  */
>  schedule_irq(dev, ctx->transtime);
>   }
> @@ -823,6 +841,8 @@ static void vim2m_stop_streaming(struct vb2_queue *q)
>  vbuf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx);
>  if (vbuf == NULL)
>  return;
> +   v4l2_ctrl_request_complete(vbuf->vb2_buf.req_obj.req,
> +  >hdl);
>  spin_lock_irqsave(>dev->irqlock, flags);
>  v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR);
>  spin_unlock_irqrestore(>dev->irqlock, flags);
> @@ -1003,6 +1023,10 @@ static const struct v4l2_m2m_ops m2m_ops = {
>  .job_abort  = job_abort,
>   };

> +static const struct media_device_ops m2m_media_ops = {
> +   .req_queue = vb2_request_queue,
> +};
> +
>   static int vim2m_probe(struct platform_device *pdev)
>   {
>  struct vim2m_dev *dev;
> @@ -1027,6 +1051,7 @@ static int vim2m_probe(struct platform_device *pdev)
>  dev->mdev.dev = >dev;
>  strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
>  media_device_init(>mdev);
> +   dev->mdev.ops = _media_ops;
>  dev->v4l2_dev.mdev = >mdev;
>  dev->pad[0].flags = MEDIA_PAD_FL_SINK;
>  dev->pad[1].flags = MEDIA_PAD_FL_SOURCE;
> --
> 2.16.3

Re: [RFCv11 PATCH 20/29] videobuf2-v4l2: integrate with media requests

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:21 PM Hans Verkuil  wrote:

> From: Hans Verkuil 

> This implements the V4L2 part of the request support. The main
> change is that vb2_qbuf and vb2_prepare_buf now have a new
> media_device pointer. This required changes to several drivers
> that did not use the vb2_ioctl_qbuf/prepare_buf helper functions.

> Signed-off-by: Hans Verkuil 
> ---
>   drivers/media/common/videobuf2/videobuf2-v4l2.c  | 84

>   drivers/media/platform/omap3isp/ispvideo.c   |  2 +-
>   drivers/media/platform/s3c-camif/camif-capture.c |  4 +-
>   drivers/media/platform/s5p-mfc/s5p_mfc_dec.c |  4 +-
>   drivers/media/platform/s5p-mfc/s5p_mfc_enc.c |  4 +-
>   drivers/media/platform/soc_camera/soc_camera.c   |  4 +-
>   drivers/media/usb/uvc/uvc_queue.c|  5 +-
>   drivers/media/usb/uvc/uvc_v4l2.c |  3 +-
>   drivers/media/usb/uvc/uvcvideo.h |  1 +
>   drivers/media/v4l2-core/v4l2-mem2mem.c   |  7 +-
>   drivers/staging/media/davinci_vpfe/vpfe_video.c  |  3 +-
>   drivers/staging/media/omap4iss/iss_video.c   |  3 +-
>   drivers/usb/gadget/function/uvc_queue.c  |  2 +-
>   include/media/videobuf2-v4l2.h   | 12 +++-
>   14 files changed, 106 insertions(+), 32 deletions(-)

> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index b8d370b97cca..73c1fd4da58a 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -25,6 +25,7 @@
>   #include 

>   #include 
> +#include 
>   #include 
>   #include 
>   #include 
> @@ -40,10 +41,12 @@ module_param(debug, int, 0644);
>  pr_info("vb2-v4l2: %s: " fmt, __func__, ## arg); \
>  } while (0)

> -/* Flags that are set by the vb2 core */
> +/* Flags that are set by us */
>   #define V4L2_BUFFER_MASK_FLAGS (V4L2_BUF_FLAG_MAPPED |
V4L2_BUF_FLAG_QUEUED | \
>   V4L2_BUF_FLAG_DONE | V4L2_BUF_FLAG_ERROR
| \
>   V4L2_BUF_FLAG_PREPARED | \
> +V4L2_BUF_FLAG_IN_REQUEST | \
> +V4L2_BUF_FLAG_REQUEST_FD | \
>   V4L2_BUF_FLAG_TIMESTAMP_MASK)
>   /* Output buffer flags that should be passed on to the driver */
>   #define V4L2_BUFFER_OUT_FLAGS  (V4L2_BUF_FLAG_PFRAME |
V4L2_BUF_FLAG_BFRAME | \
> @@ -318,13 +321,17 @@ static int vb2_fill_vb2_v4l2_buffer(struct
vb2_buffer *vb, struct v4l2_buffer *b
>  return 0;
>   }

> -static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct
v4l2_buffer *b,
> -   const char *opname)
> +static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct
media_device *mdev,
> +   struct v4l2_buffer *b,
> +   const char *opname,
> +   struct media_request **p_req)
>   {
> +   struct media_request *req;
>  struct vb2_v4l2_buffer *vbuf;
>  struct vb2_buffer *vb;
>  int ret;

> +   *p_req = NULL;
>  if (b->type != q->type) {
>  dprintk(1, "%s: invalid buffer type\n", opname);
>  return -EINVAL;
> @@ -354,7 +361,38 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue
*q, struct v4l2_buffer *b,

>  /* Copy relevant information provided by the userspace */
>  memset(vbuf->planes, 0, sizeof(vbuf->planes[0]) * vb->num_planes);
> -   return vb2_fill_vb2_v4l2_buffer(vb, b);
> +   ret = vb2_fill_vb2_v4l2_buffer(vb, b);
> +   if (ret)
> +   return ret;
> +
> +   if (!(b->flags & V4L2_BUF_FLAG_REQUEST_FD))
> +   return 0;
> +
> +   if (vb->state != VB2_BUF_STATE_DEQUEUED) {
> +   dprintk(1, "%s: buffer is not in dequeued state\n",
opname);
> +   return -EINVAL;
> +   }
> +
> +   if (b->request_fd < 0) {
> +   dprintk(1, "%s: request_fd < 0\n", opname);
> +   return -EINVAL;
> +   }
> +
> +   req = media_request_find(mdev, b->request_fd);
> +   if (IS_ERR(req)) {
> +   dprintk(1, "%s: invalid request_fd\n", opname);
> +   return PTR_ERR(req);
> +   }
> +
> +   if (req->state != MEDIA_REQUEST_STATE_IDLE) {
> +   dprintk(1, "%s: request is not idle\n", opname);
> +   media_request_put(req);
> +   return -EBUSY;
> +   }
> +
> +   *p_req = req;
> +
> +   return 0;
>   }

>   /*
> @@ -437,6 +475,9 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb,
void *pb)
>  case VB2_BUF_STATE_ACTIVE:
>  b->flags |= V4L2_BUF_FLAG_QUEUED;
>  break;
> +   case VB2_BUF_STATE_IN_REQUEST:
> +   b->flags |= 

Re: [RFCv11 PATCH 07/29] media-request: add media_request_object_find

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:21 PM Hans Verkuil  wrote:

> From: Hans Verkuil 

> Add media_request_object_find to find a request object inside a
> request based on ops and/or priv values.

> Objects of the same type (vb2 buffer, control handler) will have
> the same ops value. And objects that refer to the same 'parent'
> object (e.g. the v4l2_ctrl_handler that has the current driver
> state) will have the same priv value.

> The caller has to call media_request_object_put() for the returned
> object since this function increments the refcount.

> Signed-off-by: Hans Verkuil 
> ---
>   drivers/media/media-request.c | 26 ++
>   include/media/media-request.h | 25 +
>   2 files changed, 51 insertions(+)

> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> index 02b620c81de5..415f7e31019d 100644
> --- a/drivers/media/media-request.c
> +++ b/drivers/media/media-request.c
> @@ -322,6 +322,32 @@ static void media_request_object_release(struct kref
*kref)
>  obj->ops->release(obj);
>   }

> +struct media_request_object *
> +media_request_object_find(struct media_request *req,
> + const struct media_request_object_ops *ops,
> + void *priv)
> +{
> +   struct media_request_object *obj;
> +   struct media_request_object *found = NULL;
> +   unsigned long flags;
> +
> +   if (!ops && !priv)
> +   return NULL;
> +
> +   spin_lock_irqsave(>lock, flags);
> +   list_for_each_entry(obj, >objects, list) {
> +   if ((!ops || obj->ops == ops) &&
> +   (!priv || obj->priv == priv)) {
> +   media_request_object_get(obj);
> +   found = obj;
> +   break;
> +   }
> +   }
> +   spin_unlock_irqrestore(>lock, flags);
> +   return found;
> +}
> +EXPORT_SYMBOL_GPL(media_request_object_find);
> +
>   void media_request_object_put(struct media_request_object *obj)
>   {
>  kref_put(>kref, media_request_object_release);
> diff --git a/include/media/media-request.h b/include/media/media-request.h
> index 033697d493cd..ea990c8f76bc 100644
> --- a/include/media/media-request.h
> +++ b/include/media/media-request.h
> @@ -130,6 +130,23 @@ static inline void media_request_object_get(struct
media_request_object *obj)
>*/
>   void media_request_object_put(struct media_request_object *obj);

> +/**
> + * media_request_object_find - Find an object in a request
> + *
> + * @ops: Find an object with this ops value, may be NULL.
> + * @priv: Find an object with this priv value, may be NULL.
> + *
> + * At least one of @ops and @priv must be non-NULL. If one of
> + * these is NULL, then skip checking for that field.
> + *
> + * Returns NULL if not found or the object (the refcount is increased
> + * in that case).
> + */
> +struct media_request_object *
> +media_request_object_find(struct media_request *req,
> + const struct media_request_object_ops *ops,
> + void *priv);

Mm, that signature is weird. I don't yet know how this function is going to
be
called, but wouldn't priv be enough? If we look for ops, this means we are
looking for the first object of a given class (IIUC the class/objects
mechanism
here), and I cannot see where we would want to do that.

> +
>   /**
>* media_request_object_init - Initialise a media request object
>*
> @@ -162,6 +179,14 @@ static inline void media_request_object_put(struct
media_request_object *obj)
>   {
>   }

> +static inline struct media_request_object *
> +media_request_object_find(struct media_request *req,
> + const struct media_request_object_ops *ops,
> + void *priv)
> +{
> +   return NULL;
> +}
> +
>   static inline void media_request_object_init(struct media_request_object
*obj)
>   {
>  obj->ops = NULL;
> --
> 2.16.3

Re: [RFCv11 PATCH 04/29] media-request: core request support

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:21 PM Hans Verkuil  wrote:

> From: Hans Verkuil 

> Implement the core of the media request processing.

> Drivers can bind request objects to a request. These objects
> can then be marked completed if the driver finished using them,
> or just be unbound if the results do not need to be kept (e.g.
> in the case of buffers).

> Once all objects that were added are either unbound or completed,
> the request is marked 'complete' and a POLLPRI signal is sent
> via poll.

> Both requests and request objects are refcounted.

> While a request is queued its refcount is incremented (since it
> is in use by a driver). Once it is completed the refcount is
> decremented. When the user closes the request file descriptor
> the refcount is also decremented. Once it reaches 0 all request
> objects in the request are unbound and put() and the request
> itself is freed.

> Signed-off-by: Hans Verkuil 
> ---
>   drivers/media/media-request.c | 284
+-
>   include/media/media-request.h | 156 +++
>   2 files changed, 439 insertions(+), 1 deletion(-)

> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> index ead78613fdbe..dffc290e4ada 100644
> --- a/drivers/media/media-request.c
> +++ b/drivers/media/media-request.c
> @@ -16,8 +16,290 @@
>   #include 
>   #include 

> +static const char * const request_state[] = {
> +   "idle",
> +   "queueing",
> +   "queued",
> +   "complete",
> +   "cleaning",
> +};
> +
> +static const char *
> +media_request_state_str(enum media_request_state state)
> +{
> +   if (WARN_ON(state >= ARRAY_SIZE(request_state)))
> +   return "unknown";
> +   return request_state[state];
> +}
> +
> +static void media_request_clean(struct media_request *req)
> +{
> +   struct media_request_object *obj, *obj_safe;
> +
> +   WARN_ON(req->state != MEDIA_REQUEST_STATE_CLEANING);
> +
> +   list_for_each_entry_safe(obj, obj_safe, >objects, list) {
> +   media_request_object_unbind(obj);
> +   media_request_object_put(obj);
> +   }
> +
> +   req->num_incomplete_objects = 0;
> +   wake_up_interruptible(>poll_wait);
> +}
> +
> +static void media_request_release(struct kref *kref)
> +{
> +   struct media_request *req =
> +   container_of(kref, struct media_request, kref);
> +   struct media_device *mdev = req->mdev;
> +   unsigned long flags;
> +
> +   dev_dbg(mdev->dev, "request: release %s\n", req->debug_str);
> +
> +   spin_lock_irqsave(>lock, flags);
> +   req->state = MEDIA_REQUEST_STATE_CLEANING;
> +   spin_unlock_irqrestore(>lock, flags);
> +
> +   media_request_clean(req);
> +
> +   if (mdev->ops->req_free)
> +   mdev->ops->req_free(req);
> +   else
> +   kfree(req);

Adding a third (different) opinion on this: if requests are to be embedded
into
other struct, then shouldn't we mandate an implementation for req_free
anyway?
Making it optional sounds error-prone to me.

> +}
> +
> +void media_request_put(struct media_request *req)
> +{
> +   kref_put(>kref, media_request_release);
> +}
> +EXPORT_SYMBOL_GPL(media_request_put);
> +
> +void media_request_cancel(struct media_request *req)
> +{
> +   struct media_request_object *obj, *obj_safe;
> +
> +   if (req->state != MEDIA_REQUEST_STATE_QUEUED)
> +   return;
> +
> +   list_for_each_entry_safe(obj, obj_safe, >objects, list)
> +   if (obj->ops->cancel)
> +   obj->ops->cancel(obj);
> +}
> +EXPORT_SYMBOL_GPL(media_request_cancel);
> +
> +static int media_request_close(struct inode *inode, struct file *filp)
> +{
> +   struct media_request *req = filp->private_data;
> +
> +   media_request_put(req);
> +   return 0;
> +}
> +
> +static unsigned int media_request_poll(struct file *filp,
> +  struct poll_table_struct *wait)
> +{
> +   struct media_request *req = filp->private_data;
> +   unsigned long flags;
> +   enum media_request_state state;
> +
> +   if (!(poll_requested_events(wait) & POLLPRI))
> +   return 0;
> +
> +   spin_lock_irqsave(>lock, flags);
> +   state = req->state;
> +   spin_unlock_irqrestore(>lock, flags);
> +
> +   if (state == MEDIA_REQUEST_STATE_COMPLETE)
> +   return POLLPRI;
> +   if (state == MEDIA_REQUEST_STATE_IDLE)
> +   return POLLERR;
> +
> +   poll_wait(filp, >poll_wait, wait);
> +   return 0;
> +}
> +
> +static long media_request_ioctl(struct file *filp, unsigned int cmd,
> +   unsigned long __arg)
> +{
> +   return -ENOIOCTLCMD;
> +}
> +
> +static const struct file_operations request_fops = {
> +   .owner = THIS_MODULE,
> +   .poll = media_request_poll,
> +   

Re: [RFCv11 PATCH 01/29] v4l2-device.h: always expose mdev

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:21 PM Hans Verkuil  wrote:

> From: Hans Verkuil 

> The mdev field is only present if CONFIG_MEDIA_CONTROLLER is set.
> But since we will need to pass the media_device to vb2 snd the

Typo: s/snd/and

> control framework it is very convenient to just make this field
> available all the time. If CONFIG_MEDIA_CONTROLLER is not set,
> then it will just be NULL.

> Signed-off-by: Hans Verkuil 
> ---
>   include/media/v4l2-device.h | 4 +---
>   1 file changed, 1 insertion(+), 3 deletions(-)

> diff --git a/include/media/v4l2-device.h b/include/media/v4l2-device.h
> index 0c9e4da55499..b330e4a08a6b 100644
> --- a/include/media/v4l2-device.h
> +++ b/include/media/v4l2-device.h
> @@ -33,7 +33,7 @@ struct v4l2_ctrl_handler;
>* struct v4l2_device - main struct to for V4L2 device drivers
>*
>* @dev: pointer to struct device.
> - * @mdev: pointer to struct media_device
> + * @mdev: pointer to struct media_device, may be NULL.
>* @subdevs: used to keep track of the registered subdevs
>* @lock: lock this struct; can be used by the driver as well
>* if this struct is embedded into a larger struct.
> @@ -58,9 +58,7 @@ struct v4l2_ctrl_handler;
>*/
>   struct v4l2_device {
>  struct device *dev;
> -#if defined(CONFIG_MEDIA_CONTROLLER)
>  struct media_device *mdev;
> -#endif
>  struct list_head subdevs;
>  spinlock_t lock;
>  char name[V4L2_DEVICE_NAME_SIZE];
> --
> 2.16.3

Re: [RFCv11 PATCH 02/29] uapi/linux/media.h: add request API

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil <hverk...@xs4all.nl> wrote:

> From: Hans Verkuil <hans.verk...@cisco.com>

> Define the public request API.

> This adds the new MEDIA_IOC_REQUEST_ALLOC ioctl to allocate a request
> and two ioctls that operate on a request in order to queue the
> contents of the request to the driver and to re-initialize the
> request.

> Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>

Like it that the API stays simple, with request ioctls tied to request FDs.

Acked-by: Alexandre Courbot <acour...@chromium.org>

> ---
>   include/uapi/linux/media.h | 8 
>   1 file changed, 8 insertions(+)

> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index c7e9a5cba24e..f8769e74f847 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -342,11 +342,19 @@ struct media_v2_topology {

>   /* ioctls */

> +struct __attribute__ ((packed)) media_request_alloc {
> +   __s32 fd;
> +};
> +
>   #define MEDIA_IOC_DEVICE_INFO  _IOWR('|', 0x00, struct media_device_info)
>   #define MEDIA_IOC_ENUM_ENTITIES_IOWR('|', 0x01, struct
media_entity_desc)
>   #define MEDIA_IOC_ENUM_LINKS   _IOWR('|', 0x02, struct media_links_enum)
>   #define MEDIA_IOC_SETUP_LINK   _IOWR('|', 0x03, struct media_link_desc)
>   #define MEDIA_IOC_G_TOPOLOGY   _IOWR('|', 0x04, struct media_v2_topology)
> +#define MEDIA_IOC_REQUEST_ALLOC_IOWR('|', 0x05, struct
media_request_alloc)
> +
> +#define MEDIA_REQUEST_IOC_QUEUE_IO('|',  0x80)
> +#define MEDIA_REQUEST_IOC_REINIT   _IO('|',  0x81)

>   #if !defined(__KERNEL__) || defined(__NEED_MEDIA_LEGACY_API)

> --
> 2.16.3

Re: [RFCv11 PATCH 00/29] Request API

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil  wrote:

> From: Hans Verkuil 

> Hi all,

> This is a cleaned up version of the v10 series (never posted to
> the list since it was messy).

Hi Hans,

It took me a while to test and review this, but finally have been able to
do it.

First the result of the test: I have tried porting my dummy vim2m test
program
(https://gist.github.com/Gnurou/34c35f1f8e278dad454b51578d239a42 for
reference),
and am getting a hang when trying to queue the second OUTPUT buffer (right
after
queuing the first request). If I move the calls the VIDIOC_STREAMON after
the
requests are queued, the hang seems to happen at that moment. Probably a
deadlock, haven't looked in detail yet.

I have a few other comments, will follow up per-patch.

Re: [RFCv11 PATCH 03/29] media-request: allocate media requests

[Alexandre Courbot]

On Mon, Apr 9, 2018 at 11:20 PM Hans Verkuil  wrote:

> From: Hans Verkuil 

> Add support for allocating a new request. This is only supported
> if mdev->ops->req_queue is set, i.e. the driver indicates that it
> supports queueing requests.

> Signed-off-by: Hans Verkuil 
> ---
>   drivers/media/Makefile|  3 ++-
>   drivers/media/media-device.c  | 14 ++
>   drivers/media/media-request.c | 23 +++
>   include/media/media-device.h  | 13 +
>   include/media/media-request.h | 22 ++
>   5 files changed, 74 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/media/media-request.c
>   create mode 100644 include/media/media-request.h

> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
> index 594b462ddf0e..985d35ec6b29 100644
> --- a/drivers/media/Makefile
> +++ b/drivers/media/Makefile
> @@ -3,7 +3,8 @@
>   # Makefile for the kernel multimedia device drivers.
>   #

> -media-objs := media-device.o media-devnode.o media-entity.o
> +media-objs := media-device.o media-devnode.o media-entity.o \
> +  media-request.o

>   #
>   # I2C drivers should come before other drivers, otherwise they'll fail
> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
> index 35e81f7c0d2f..acb583c0eacd 100644
> --- a/drivers/media/media-device.c
> +++ b/drivers/media/media-device.c
> @@ -32,6 +32,7 @@
>   #include 
>   #include 
>   #include 
> +#include 

>   #ifdef CONFIG_MEDIA_CONTROLLER

> @@ -366,6 +367,15 @@ static long media_device_get_topology(struct
media_device *mdev,
>  return ret;
>   }

> +static long media_device_request_alloc(struct media_device *mdev,
> +  struct media_request_alloc *alloc)
> +{
> +   if (!mdev->ops || !mdev->ops->req_queue)
> +   return -ENOTTY;
> +
> +   return media_request_alloc(mdev, alloc);
> +}
> +
>   static long copy_arg_from_user(void *karg, void __user *uarg, unsigned
int cmd)
>   {
>  /* All media IOCTLs are _IOWR() */
> @@ -414,6 +424,7 @@ static const struct media_ioctl_info ioctl_info[] = {
>  MEDIA_IOC(ENUM_LINKS, media_device_enum_links,
MEDIA_IOC_FL_GRAPH_MUTEX),
>  MEDIA_IOC(SETUP_LINK, media_device_setup_link,
MEDIA_IOC_FL_GRAPH_MUTEX),
>  MEDIA_IOC(G_TOPOLOGY, media_device_get_topology,
MEDIA_IOC_FL_GRAPH_MUTEX),
> +   MEDIA_IOC(REQUEST_ALLOC, media_device_request_alloc, 0),
>   };

>   static long media_device_ioctl(struct file *filp, unsigned int cmd,
> @@ -686,6 +697,9 @@ void media_device_init(struct media_device *mdev)
>  INIT_LIST_HEAD(>pads);
>  INIT_LIST_HEAD(>links);
>  INIT_LIST_HEAD(>entity_notify);
> +
> +   spin_lock_init(>req_lock);
> +   mutex_init(>req_queue_mutex);
>  mutex_init(>graph_mutex);
>  ida_init(>entity_internal_idx);

> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> new file mode 100644
> index ..ead78613fdbe
> --- /dev/null
> +++ b/drivers/media/media-request.c
> @@ -0,0 +1,23 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Media device request objects
> + *
> + * Copyright (C) 2018 Intel Corporation
> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.

Seconding Tomasz' comment - Chromium authorship should be something like

Copyright (C) 2018 Google, Inc.

(see
https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/kernel-faq#TOC-Which-copyright-header-should-I-use-
)

Sorry since I think I am the one who introduced this line in my early
patches.

> + *
> + * Author: Sakari Ailus 
> + */
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +
> +#include 
> +#include 
> +
> +int media_request_alloc(struct media_device *mdev,
> +   struct media_request_alloc *alloc)
> +{
> +   return -ENOMEM;

Probably not the best error code for now? And this function can probably be
declared later (i.e. along with its full definition) in the series anyway.

> +}
> diff --git a/include/media/media-device.h b/include/media/media-device.h
> index bcc6ec434f1f..07e323c57202 100644
> --- a/include/media/media-device.h
> +++ b/include/media/media-device.h
> @@ -19,6 +19,7 @@
>   #ifndef _MEDIA_DEVICE_H
>   #define _MEDIA_DEVICE_H

> +#include 
>   #include 
>   #include 

> @@ -27,6 +28,7 @@

>   struct ida;
>   struct device;
> +struct media_device;

>   /**
>* struct media_entity_notify - Media Entity Notify
> @@ -50,10 +52,16 @@ struct media_entity_notify {
>* struct media_device_ops - Media device operations
>* @link_notify: Link state change notification callback. This callback
is
>*  called with the graph_mutex held.
> + * @req_alloc: Allocate a request
> + * @req_free: Free a request
> + * @req_queue: Queue a request
>*/
>   struct media_device_ops {
>

Re: [RFC v2 00/10] Preparing the request API

[Alexandre Courbot]

On Sat, Mar 24, 2018 at 6:17 AM, Sakari Ailus
<sakari.ai...@linux.intel.com> wrote:
> Hi folks,
>
> This preliminary RFC patchset prepares for the request API. What's new
> here is support for binding arbitrary configuration or resources to
> requests.
>
> There are a few new concepts here:
>
> Class --- a type of configuration or resource a driver (or e.g. the V4L2
> framework) can attach to a resource. E.g. a video buffer queue would be a
> class.
>
> Object --- an instance of the class. This may be either configuration (in
> which case the setting will stay until changed, e.g. V4L2 format on a
> video node) or a resource (such as a video buffer).
>
> Reference --- a reference to an object. If a configuration is not changed
> in a request, instead of allocating a new object, a reference to an
> existing object is used. This saves time and memory.
>
> I expect Laurent to comment on aligning the concept names between the
> request API and DRM. As far as I understand, the respective DRM names for
> "class" and "object" used in this set would be "object" and "state".
>
> The drivers will need to interact with the requests in three ways:
>
> - Allocate new configurations or resources. Drivers are free to store
>   their own data into request objects as well. These callbacks are
>   specific to classes.
>
> - Validate and queue callbacks. These callbacks are used to try requests
>   (validate only) as well as queue them (validate and queue). These
>   callbacks are media device wide, at least for now.
>
> The lifetime of the objects related to requests is based on refcounting
> both requests and request objects. This fits well for existing use cases
> whether or not based on refcounting; what still needs most of the
> attention is likely that the number of gets and puts matches once the
> object is no longer needed.
>
> Configuration can be bound to the request the usual way (V4L2 IOCTLs with
> the request_fd field set to the request). Once queued, request completion
> is signalled through polling the request file handle (POLLPRI).
>
> I'm posting this as an RFC because it's not complete yet. The code
> compiles but no testing has been done yet.
>
> Todo list:
>
> - Testing! (And fixing the bugs.)
>
> - Request support in a few drivers as well as the control framework.
>
> - Request support for V4L2 formats?

Also, conformance tests in v4l2-compliance.

>
> In the future, support for changing e.g. Media controller link state or
> V4L2 sub-device formats will need to be added. Those should receive more
> attention when the core is in a good shape and the more simple use cases
> are already functional.
>
> Comments and questions are welcome.
>
> since v1:
>
> - Provide an iterator helper for request objects in a request.
>
> - Remove the request lists in the media device (they were not used)
>
> - Move request queing to request fd and add reinit (Alexandre's patchset);
>   this roughly corresponds to Request API RFC v2 from Hans.
>   (MEDIA_IOC_REQUEST_ALLOC argument is a struct pointer instead of an
>   __s32 pointer.)
>
> - Provide a way to unbind request objects from an unqueued request
>   (reinit, closing request fd).
>
> - v4l2-mem2mem + vivid implementation without control support.

Looks like vivid is not here yet, although it doesn't really matter for now.

>
> - More states for requests. In order to take a spinlock (or a mutex) for
>   an extended period of time, add a "QUEUEING" and "REINIT" states.
>
> - Move non-IOCTL code to media-request.c, remove extra filp argument that
>   was added in v1.
>
> - SPDX license header, other small changes.
>
> Open questions:
>
> - How to tell at complete time whether a request failed? Return error code
>   on release? What's the behaviour with reinit then --- fail on error? Add
>   another IOCTL to ask for completion status?

We could look for POLLERR when polling the request in order to know
that something went wrong. When this happens, an extra ioctl on the
request fd (e.g. REQUEST_GET_ERROR) could be used to get more
information about what has failed. Due to the extensible nature of
requests, we should be careful to make this ioctl widely extensible as
well. Anything that is performed in a deferred way can fail (for now,
S_EXT_CTRLS and QBUF), so maybe this ioctl should return errors in the
same format as these calls. The request would be interrupted upon
first error, so we could just return the code of the ioctl that would
have failed + its relevant user-readable state).

>
>
> Alexandre Courbot (1):
>   videodev2.h: add request_fd field to v4l2_ext_controls
>
> Hans Verkuil (1):
>   videodev2

[PATCH v2] venus: vdec: fix format enumeration

[Alexandre Courbot]

find_format_by_index() stops enumerating formats as soon as the index
matches, and returns NULL if venus_helper_check_codec() finds out that
the format is not supported. This prevents formats to be properly
enumerated if a non-supported format is present, as the enumeration will
end with it.

Fix this by moving the call to venus_helper_check_codec() into the loop,
and keep enumerating when it fails.

Fixes: 29f0133ec6 media: venus: use helper function to check supported codecs

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/qcom/venus/vdec.c | 13 +++--
 drivers/media/platform/qcom/venus/venc.c | 13 +++--
 2 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/drivers/media/platform/qcom/venus/vdec.c 
b/drivers/media/platform/qcom/venus/vdec.c
index c9e9576bb08a..49bbd1861d3a 100644
--- a/drivers/media/platform/qcom/venus/vdec.c
+++ b/drivers/media/platform/qcom/venus/vdec.c
@@ -135,20 +135,21 @@ find_format_by_index(struct venus_inst *inst, unsigned 
int index, u32 type)
return NULL;
 
for (i = 0; i < size; i++) {
+   bool valid;
+
if (fmt[i].type != type)
continue;
-   if (k == index)
+   valid = type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
+   venus_helper_check_codec(inst, fmt[i].pixfmt);
+   if (k == index && valid)
break;
-   k++;
+   if (valid)
+   k++;
}
 
if (i == size)
return NULL;
 
-   if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE &&
-   !venus_helper_check_codec(inst, fmt[i].pixfmt))
-   return NULL;
-
return [i];
 }
 
diff --git a/drivers/media/platform/qcom/venus/venc.c 
b/drivers/media/platform/qcom/venus/venc.c
index e3a10a852cad..6b2ce479584e 100644
--- a/drivers/media/platform/qcom/venus/venc.c
+++ b/drivers/media/platform/qcom/venus/venc.c
@@ -120,20 +120,21 @@ find_format_by_index(struct venus_inst *inst, unsigned 
int index, u32 type)
return NULL;
 
for (i = 0; i < size; i++) {
+   bool valid;
+
if (fmt[i].type != type)
continue;
-   if (k == index)
+   valid = type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE ||
+   venus_helper_check_codec(inst, fmt[i].pixfmt);
+   if (k == index && valid)
break;
-   k++;
+   if (valid)
+   k++;
}
 
if (i == size)
return NULL;
 
-   if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE &&
-   !venus_helper_check_codec(inst, fmt[i].pixfmt))
-   return NULL;
-
return [i];
 }
 
-- 
2.16.2.804.g6dcf76e118-goog

Re: [PATCH] venus: vdec: fix format enumeration

[Alexandre Courbot]

On Thu, Mar 15, 2018 at 6:52 PM, Stanimir Varbanov
<stanimir.varba...@linaro.org> wrote:
> Hi Alex,
>
> Thanks for the patch!
>
> On 13.03.2018 11:11, Alexandre Courbot wrote:
>> find_format_by_index() stops enumerating formats as soon as the index
>> matches, and returns NULL if venus_helper_check_codec() finds out that
>> the format is not supported. This prevents formats to be properly
>> enumerated if a non-supported format is present, as the enumeration will
>> end with it.
>
> Please add fixes tag,
>
> Fixes: 29f0133ec6 media: venus: use helper function to check supported
> codecs

Sure.

>
>>
>> Fix this by moving the call to venus_helper_check_codec() into the loop,
>> and keep enumerating when it fails.
>
> Good catch!
>
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>>
>> Change-Id: I4ff66e0b85172598efa59a6f01da8cb60597a6a5
>
> You forgot to delete gerrit id.

D'oh, I should know better than send patches from my Chrome OS tree...

>
>> ---
>>   drivers/media/platform/qcom/venus/vdec.c | 13 +++--
>>   drivers/media/platform/qcom/venus/venc.c |  9 +++--
>>   2 files changed, 14 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/media/platform/qcom/venus/vdec.c 
>> b/drivers/media/platform/qcom/venus/vdec.c
>> index c9e9576bb08a..3677302cfe43 100644
>> --- a/drivers/media/platform/qcom/venus/vdec.c
>> +++ b/drivers/media/platform/qcom/venus/vdec.c
>> @@ -135,20 +135,21 @@ find_format_by_index(struct venus_inst *inst, unsigned 
>> int index, u32 type)
>>   return NULL;
>>
>>   for (i = 0; i < size; i++) {
>> + bool valid;
>> +
>>   if (fmt[i].type != type)
>>   continue;
>> - if (k == index)
>> + valid = (type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
>> +  venus_helper_check_codec(inst, fmt[i].pixfmt));
>
> open and close braces are not needed.

Removed them.

>
>> + if (k == index && valid)
>>   break;
>> - k++;
>> + if (valid)
>> + k++;
>>   }
>>
>>   if (i == size)
>>   return NULL;
>>
>> - if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE &&
>> - !venus_helper_check_codec(inst, fmt[i].pixfmt))
>> - return NULL;
>> -
>>   return [i];
>>   }
>>
>> diff --git a/drivers/media/platform/qcom/venus/venc.c 
>> b/drivers/media/platform/qcom/venus/venc.c
>> index e3a10a852cad..5eba4c7cd52e 100644
>> --- a/drivers/media/platform/qcom/venus/venc.c
>> +++ b/drivers/media/platform/qcom/venus/venc.c
>> @@ -120,11 +120,16 @@ find_format_by_index(struct venus_inst *inst, unsigned 
>> int index, u32 type)
>>   return NULL;
>>
>>   for (i = 0; i < size; i++) {
>> + bool valid;
>> +
>>   if (fmt[i].type != type)
>>   continue;
>> - if (k == index)
>> + valid = (type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE ||
>> +  venus_helper_check_codec(inst, fmt[i].pixfmt));
>> + if (k == index && valid)
>>   break;
>> - k++;
>> + if (valid)
>> + k++;
>>   }
>>
>>   if (i == size)
>
> maybe we should delete the condition:
>
> if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE &&
> !venus_helper_check_codec(inst, fmt[i].pixfmt))
>
> as we do for the decoder?

We should indeed, missed that.

Re: [RFCv4,19/21] media: vim2m: add request support

[Alexandre Courbot]

On Wed, Mar 14, 2018 at 10:25 PM, Paul Kocialkowski
<paul.kocialkow...@bootlin.com> wrote:
> Hi,
>
> On Tue, 2018-03-13 at 19:24 +0900, Alexandre Courbot wrote:
>> On Fri, Mar 9, 2018 at 11:35 PM, Paul Kocialkowski
>> <paul.kocialkow...@bootlin.com> wrote:
>> > Hi,
>> >
>> > On Thu, 2018-03-08 at 22:48 +0900, Alexandre Courbot wrote:
>> > > Hi Paul!
>> > >
>> > > Thanks a lot for taking the time to try this! I am also working on
>> > > getting it to work with an actual driver, but you apparently found
>> > > rough edges that I missed.
>> > >
>> > > On Thu, Mar 8, 2018 at 1:37 AM, Paul Kocialkowski
>> > > <paul.kocialkow...@bootlin.com> wrote:
>> > > > On Tue, 2018-02-20 at 13:44 +0900, Alexandre Courbot wrote:
>
> [...]
>
>> > > > > +static int vim2m_request_submit(struct media_request *req,
>> > > > > + struct media_request_entity_data
>> > > > > *_data)
>> > > > > +{
>> > > > > + struct v4l2_request_entity_data *data;
>> > > > > +
>> > > > > + data = to_v4l2_entity_data(_data);
>> > > >
>> > > > We need to call v4l2_m2m_try_schedule here so that m2m
>> > > > scheduling
>> > > > can
>> > > > happen when only 2 buffers were queued and no other action was
>> > > > taken
>> > > > from usespace. In that scenario, m2m scheduling currently
>> > > > doesn't
>> > > > happen.
>> > >
>> > > I don't think I understand the sequence of events that results in
>> > > v4l2_m2m_try_schedule() not being called. Do you mean something
>> > > like:
>> > >
>> > > *
>> > > * QBUF on output queue with request set
>> > > * QBUF on capture queue
>> > > * SUBMIT_REQUEST
>> > >
>> > > ?
>> > >
>> > > The call to vb2_request_submit() right after should trigger
>> > > v4l2_m2m_try_schedule(), since the buffers associated to the
>> > > request
>> > > will enter the vb2 queue and be passed to the m2m framework, which
>> > > will then call v4l2_m2m_try_schedule(). Or maybe you are thinking
>> > > about a different sequence of events?
>> >
>> > This is indeed the sequence of events that I'm seeing, but the
>> > scheduling call simply did not happen on vb2_request_submit. I
>> > suppose I will need to investigate some more to find out exactly
>> > why.
>> >
>> > IIRC, the m2m qbuf function is called (and fails to schedule) when
>> > the
>> > ioctl happens, not when the task is submitted.
>> >
>> > This issue is seen with vim2m as well as the rencently-submitted
>> > sunxi-
>> > cedrus driver (with the in-driver calls to v4l2_m2m_try_schedule
>> > removed, obviously). If needs be, I could provide a standalone test
>> > program to reproduce it.
>>
>> If you have a standalone program that can reproduce this on vim2m,
>> then I would like to see it indeed, if only to understand what I have
>> missed.
>
> You can find the test file for this use case at:
> https://gist.github.com/paulkocialkowski/4cfa350e1bbe8e3bf714480bba83ea72

Thanks, I have been able to see what the issue was. One indeed needs
to call v4l2_m2m_try_schedule() when the request is queued, since the
driver qbuf() hook does not do it automatically.

Re: [RFCv4,19/21] media: vim2m: add request support

[Alexandre Courbot]

On Fri, Mar 9, 2018 at 11:35 PM, Paul Kocialkowski
<paul.kocialkow...@bootlin.com> wrote:
> Hi,
>
> On Thu, 2018-03-08 at 22:48 +0900, Alexandre Courbot wrote:
>> Hi Paul!
>>
>> Thanks a lot for taking the time to try this! I am also working on
>> getting it to work with an actual driver, but you apparently found
>> rough edges that I missed.
>>
>> On Thu, Mar 8, 2018 at 1:37 AM, Paul Kocialkowski
>> <paul.kocialkow...@bootlin.com> wrote:
>> > Hi,
>> >
>> > First off, I'd like to take the occasion to say thank-you for your
>> > work.
>> > This is a major piece of plumbing that is required for me to add
>> > support
>> > for the Allwinner CedarX VPU hardware in upstream Linux. Other
>> > drivers,
>> > such as tegra-vde (that was recently merged in staging) are also
>> > badly
>> > in need of this API.
>> >
>> > I have a few comments based on my experience integrating this
>> > request
>> > API with the Cedrus VPU driver (and the associated libva backend),
>> > that
>> > also concern the vim2m driver.
>> >
>> > On Tue, 2018-02-20 at 13:44 +0900, Alexandre Courbot wrote:
>> > > Set the necessary ops for supporting requests in vim2m.
>> > >
>> > > Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> > > ---
>> > >  drivers/media/platform/Kconfig |  1 +
>> > >  drivers/media/platform/vim2m.c | 75
>> > > ++
>> > >  2 files changed, 76 insertions(+)
>> > >
>> > > diff --git a/drivers/media/platform/Kconfig
>> > > b/drivers/media/platform/Kconfig
>> > > index 614fbef08ddc..09be0b5f9afe 100644
>> > > --- a/drivers/media/platform/Kconfig
>> > > +++ b/drivers/media/platform/Kconfig
>> >
>> > [...]
>> >
>> > > +static int vim2m_request_submit(struct media_request *req,
>> > > + struct media_request_entity_data
>> > > *_data)
>> > > +{
>> > > + struct v4l2_request_entity_data *data;
>> > > +
>> > > + data = to_v4l2_entity_data(_data);
>> >
>> > We need to call v4l2_m2m_try_schedule here so that m2m scheduling
>> > can
>> > happen when only 2 buffers were queued and no other action was taken
>> > from usespace. In that scenario, m2m scheduling currently doesn't
>> > happen.
>>
>> I don't think I understand the sequence of events that results in
>> v4l2_m2m_try_schedule() not being called. Do you mean something like:
>>
>> *
>> * QBUF on output queue with request set
>> * QBUF on capture queue
>> * SUBMIT_REQUEST
>>
>> ?
>>
>> The call to vb2_request_submit() right after should trigger
>> v4l2_m2m_try_schedule(), since the buffers associated to the request
>> will enter the vb2 queue and be passed to the m2m framework, which
>> will then call v4l2_m2m_try_schedule(). Or maybe you are thinking
>> about a different sequence of events?
>
> This is indeed the sequence of events that I'm seeing, but the
> scheduling call simply did not happen on vb2_request_submit. I suppose I will 
> need to investigate some more to find out exactly why.
>
> IIRC, the m2m qbuf function is called (and fails to schedule) when the
> ioctl happens, not when the task is submitted.
>
> This issue is seen with vim2m as well as the rencently-submitted sunxi-
> cedrus driver (with the in-driver calls to v4l2_m2m_try_schedule
> removed, obviously). If needs be, I could provide a standalone test
> program to reproduce it.

If you have a standalone program that can reproduce this on vim2m,
then I would like to see it indeed, if only to understand what I have
missed.

Thanks,
Alex.

[PATCH] venus: vdec: fix format enumeration

[Alexandre Courbot]

find_format_by_index() stops enumerating formats as soon as the index
matches, and returns NULL if venus_helper_check_codec() finds out that
the format is not supported. This prevents formats to be properly
enumerated if a non-supported format is present, as the enumeration will
end with it.

Fix this by moving the call to venus_helper_check_codec() into the loop,
and keep enumerating when it fails.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>

Change-Id: I4ff66e0b85172598efa59a6f01da8cb60597a6a5
---
 drivers/media/platform/qcom/venus/vdec.c | 13 +++--
 drivers/media/platform/qcom/venus/venc.c |  9 +++--
 2 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/drivers/media/platform/qcom/venus/vdec.c 
b/drivers/media/platform/qcom/venus/vdec.c
index c9e9576bb08a..3677302cfe43 100644
--- a/drivers/media/platform/qcom/venus/vdec.c
+++ b/drivers/media/platform/qcom/venus/vdec.c
@@ -135,20 +135,21 @@ find_format_by_index(struct venus_inst *inst, unsigned 
int index, u32 type)
return NULL;
 
for (i = 0; i < size; i++) {
+   bool valid;
+
if (fmt[i].type != type)
continue;
-   if (k == index)
+   valid = (type != V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
+venus_helper_check_codec(inst, fmt[i].pixfmt));
+   if (k == index && valid)
break;
-   k++;
+   if (valid)
+   k++;
}
 
if (i == size)
return NULL;
 
-   if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE &&
-   !venus_helper_check_codec(inst, fmt[i].pixfmt))
-   return NULL;
-
return [i];
 }
 
diff --git a/drivers/media/platform/qcom/venus/venc.c 
b/drivers/media/platform/qcom/venus/venc.c
index e3a10a852cad..5eba4c7cd52e 100644
--- a/drivers/media/platform/qcom/venus/venc.c
+++ b/drivers/media/platform/qcom/venus/venc.c
@@ -120,11 +120,16 @@ find_format_by_index(struct venus_inst *inst, unsigned 
int index, u32 type)
return NULL;
 
for (i = 0; i < size; i++) {
+   bool valid;
+
if (fmt[i].type != type)
continue;
-   if (k == index)
+   valid = (type != V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE ||
+venus_helper_check_codec(inst, fmt[i].pixfmt));
+   if (k == index && valid)
break;
-   k++;
+   if (valid)
+   k++;
}
 
if (i == size)
-- 
2.16.2.660.g709887971b-goog

Re: [RFCv4,19/21] media: vim2m: add request support

[Alexandre Courbot]

On Mon, Mar 12, 2018 at 5:15 PM, Tomasz Figa  wrote:
> Hi Paul, Dmitry,
>
> On Mon, Mar 12, 2018 at 5:10 PM, Paul Kocialkowski
>  wrote:
>> Hi,
>>
>> On Sun, 2018-03-11 at 22:42 +0300, Dmitry Osipenko wrote:
>>> Hello,
>>>
>>> On 07.03.2018 19:37, Paul Kocialkowski wrote:
>>> > Hi,
>>> >
>>> > First off, I'd like to take the occasion to say thank-you for your
>>> > work.
>>> > This is a major piece of plumbing that is required for me to add
>>> > support
>>> > for the Allwinner CedarX VPU hardware in upstream Linux. Other
>>> > drivers,
>>> > such as tegra-vde (that was recently merged in staging) are also
>>> > badly
>>> > in need of this API.
>>>
>>> Certainly it would be good to have a common UAPI. Yet I haven't got my
>>> hands on
>>> trying to implement the V4L interface for the tegra-vde driver, but
>>> I've taken a
>>> look at Cedrus driver and for now I've one question:
>>>
>>> Would it be possible (or maybe already is) to have a single IOCTL that
>>> takes input/output buffers with codec parameters, processes the
>>> request(s) and returns to userspace when everything is done? Having 5
>>> context switches for a single frame decode (like Cedrus VAAPI driver
>>> does) looks like a bit of overhead.
>>
>> The V4L2 interface exposes ioctls for differents actions and I don't
>> think there's a combined ioctl for this. The request API was introduced
>> precisely because we need to have consistency between the various ioctls
>> needed for each frame. Maybe one single (atomic) ioctl would have worked
>> too, but that's apparently not how the V4L2 API was designed.
>>
>> I don't think there is any particular overhead caused by having n ioctls
>> instead of a single one. At least that would be very surprising IMHO.
>
> Well, there is small syscall overhead, which normally shouldn't be
> very painful, although with all the speculative execution hardening,
> can't be sure of anything anymore. :)
>
> Hans and Alex can correct me if I'm wrong, but I believe there is a
> more atomic-like API being planned, which would only need one IOCTL to
> do everything. However, that would be a more serious change to the
> V4L2 interfaces, so should be decoupled from Request API itself.

Indeed, we discussed the possibility to setup and submit requests in
one syscall, similarly (at least in spirit) to the DRM atomic API.

This has only been discussed though, and as a feature to consider
*after* the request API is merged for codecs (as the more complex
camera use-cases would benefit more from it). As Tomasz mentioned, the
overhead of ioctls is somehow negligible compared to the workload of
the encoding/decoding itself, although I suppose it can still add up.
The main advantage I can see for this is a simpler and less
error-prone setup of requests for user-space.

Re: [RFCv4,13/21] media: videobuf2-v4l2: support for requests

[Alexandre Courbot]

On Thu, Mar 8, 2018 at 1:50 AM, Paul Kocialkowski
<paul.kocialkow...@bootlin.com> wrote:
> Hi,
>
> On Tue, 2018-02-20 at 13:44 +0900, Alexandre Courbot wrote:
>> Add a new vb2_qbuf_request() (a request-aware version of vb2_qbuf())
>> that request-aware drivers can call to queue a buffer into a request
>> instead of directly into the vb2 queue if relevent.
>>
>> This function expects that drivers invoking it are using instances of
>> v4l2_request_entity and v4l2_request_entity_data to describe their
>> entity and entity data respectively.
>>
>> Also add the vb2_request_submit() helper function which drivers can
>> invoke in order to queue all the buffers previously queued into a
>> request into the regular vb2 queue.
>
> See a comment/proposal below about an issue I encountered when using
> multi-planar formats.
>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  .../media/common/videobuf2/videobuf2-v4l2.c   | 129
>> +-
>>  include/media/videobuf2-v4l2.h|  59 
>>  2 files changed, 187 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> index 6d4d184aa68e..0627c3339572 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>
> [...]
>
>> +#if IS_ENABLED(CONFIG_MEDIA_REQUEST_API)
>> +int vb2_qbuf_request(struct vb2_queue *q, struct v4l2_buffer *b,
>> +  struct media_request_entity *entity)
>> +{
>> + struct v4l2_request_entity_data *data;
>> + struct v4l2_vb2_request_buffer *qb;
>> + struct media_request *req;
>> + struct vb2_buffer *vb;
>> + int ret = 0;
>> +
>> + if (b->request_fd <= 0)
>> + return vb2_qbuf(q, b);
>> +
>> + if (!q->allow_requests)
>> + return -EINVAL;
>> +
>> + req = media_request_get_from_fd(b->request_fd);
>> + if (!req)
>> + return -EINVAL;
>> +
>> + data = to_v4l2_entity_data(media_request_get_entity_data(req,
>> entity));
>> + if (IS_ERR(data)) {
>> + ret = PTR_ERR(data);
>> + goto out;
>> + }
>> +
>> + mutex_lock(>lock);
>> +
>> + if (req->state != MEDIA_REQUEST_STATE_IDLE) {
>> + ret = -EINVAL;
>> + goto out;
>> + }
>> +
>> + ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
>> + if (ret)
>> + goto out;
>> +
>> + vb = q->bufs[b->index];
>> + switch (vb->state) {
>> + case VB2_BUF_STATE_DEQUEUED:
>> + break;
>> + case VB2_BUF_STATE_PREPARED:
>> + break;
>> + case VB2_BUF_STATE_PREPARING:
>> + dprintk(1, "buffer still being prepared\n");
>> + ret = -EINVAL;
>> + goto out;
>> + default:
>> + dprintk(1, "invalid buffer state %d\n", vb->state);
>> + ret = -EINVAL;
>> + goto out;
>> + }
>> +
>> + /* do we already have a buffer for this request in the queue?
>> */
>> + list_for_each_entry(qb, >queued_buffers, node) {
>> + if (qb->queue == q) {
>> + ret = -EBUSY;
>> + goto out;
>> + }
>> + }
>> +
>> + qb = kzalloc(sizeof(*qb), GFP_KERNEL);
>> + if (!qb) {
>> + ret = -ENOMEM;
>> + goto out;
>> + }
>> +
>> + /*
>> +  * TODO should be prepare the buffer here if needed, to
>> report errors
>> +  * early?
>> +  */
>> + qb->pre_req_state = vb->state;
>> + qb->queue = q;
>> + memcpy(>v4l2_buf, b, sizeof(*b));
>
> I am getting data corruption on qb->v4l2_buf.m.planes from this when
> using planar buffers, only after exiting the ioctl handler (i.e. when
> accessing this buffer later from the queue).
>
> I initially thought this was because the planes pointer was copied as-is
> from userspace, but Maxime Ripard suggested that this would have
> automatically triggered a visible fault in the kernel.
>
> Thus, my best guess is that the data is properly copied from userspace
> but freed when leaving the ioctl handler, which doesn't work our with
> the request API.
>
> A dirty fix that I came up with consists in re-allocating the planes
> buffer here and copying its contents from "b", so that it can live
> beyond the ioctl call.
>
> I am not too sure whether this should be fixed here or in the part of
> the v4l2 common code that frees this data. What do you think?

Oh, nice catch. Copying plane information indeed requires more work
than a dumb memcpy(). I suppose this should be handled here, but let
me come back to this after a good night of sleep. :)

Thanks! I will try to fix this tomorrow and push a temporary version
somewhere so you can move forward.

Alex.

Re: [RFCv4,19/21] media: vim2m: add request support

[Alexandre Courbot]

Hi Paul!

Thanks a lot for taking the time to try this! I am also working on
getting it to work with an actual driver, but you apparently found
rough edges that I missed.

On Thu, Mar 8, 2018 at 1:37 AM, Paul Kocialkowski
<paul.kocialkow...@bootlin.com> wrote:
> Hi,
>
> First off, I'd like to take the occasion to say thank-you for your work.
> This is a major piece of plumbing that is required for me to add support
> for the Allwinner CedarX VPU hardware in upstream Linux. Other drivers,
> such as tegra-vde (that was recently merged in staging) are also badly
> in need of this API.
>
> I have a few comments based on my experience integrating this request
> API with the Cedrus VPU driver (and the associated libva backend), that
> also concern the vim2m driver.
>
> On Tue, 2018-02-20 at 13:44 +0900, Alexandre Courbot wrote:
>> Set the necessary ops for supporting requests in vim2m.
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  drivers/media/platform/Kconfig |  1 +
>>  drivers/media/platform/vim2m.c | 75
>> ++
>>  2 files changed, 76 insertions(+)
>>
>> diff --git a/drivers/media/platform/Kconfig
>> b/drivers/media/platform/Kconfig
>> index 614fbef08ddc..09be0b5f9afe 100644
>> --- a/drivers/media/platform/Kconfig
>> +++ b/drivers/media/platform/Kconfig
>
> [...]
>
>> +static int vim2m_request_submit(struct media_request *req,
>> + struct media_request_entity_data
>> *_data)
>> +{
>> + struct v4l2_request_entity_data *data;
>> +
>> + data = to_v4l2_entity_data(_data);
>
> We need to call v4l2_m2m_try_schedule here so that m2m scheduling can
> happen when only 2 buffers were queued and no other action was taken
> from usespace. In that scenario, m2m scheduling currently doesn't
> happen.

I don't think I understand the sequence of events that results in
v4l2_m2m_try_schedule() not being called. Do you mean something like:

*
* QBUF on output queue with request set
* QBUF on capture queue
* SUBMIT_REQUEST

?

The call to vb2_request_submit() right after should trigger
v4l2_m2m_try_schedule(), since the buffers associated to the request
will enter the vb2 queue and be passed to the m2m framework, which
will then call v4l2_m2m_try_schedule(). Or maybe you are thinking
about a different sequence of events?

>
> However, this requires access to the m2m context, which is not easy to
> get from req or _data. I'm not sure that some container_of magic would
> even do the trick here.

data_->entity will give you a pointer to the media_request_entity,
which is part of vim2m_ctx. You can thus get the m2m context by doing
container_of(data_->entity, struct vim2m_ctx, req_entity). See
vim2m_entity_data_alloc() for an example.

>
>> + return vb2_request_submit(data);
>
> vb2_request_submit does not lock the associated request mutex although
> it accesses the associated queued buffers list, which I believe this
> mutex is supposed to protect.

After a request is submitted, the data protected by the mutex can only
be accessed by the driver when it processes the request. It cannot be
modified concurrently, so I think we are safe here.

I am also wondering whether the ioctl locking doesn't make the request
locking redundant. Request information can only be modified and
accessed through ioctls until it is submitted, and after that there
are no concurrent accesses. I need to think a bit more about it
though.

Cheers,
Alex.

>
> We could either wrap this call with media_request_lock(req) and
> media_request_unlock(req) or have the lock in the function itself, which
> would require passing it the req pointer.
>
> The latter would probably be safer for future use of the function.
>
>> +}
>> +
>> +static const struct media_request_entity_ops vim2m_request_entity_ops
>> = {
>> + .data_alloc = vim2m_entity_data_alloc,
>> + .data_free  = v4l2_request_entity_data_free,
>> + .submit = vim2m_request_submit,
>> +};
>> +
>>  /*
>>   * File operations
>>   */
>> @@ -900,6 +967,9 @@ static int vim2m_open(struct file *file)
>>   ctx->dev = dev;
>>   hdl = >hdl;
>>   v4l2_ctrl_handler_init(hdl, 4);
>> + v4l2_request_entity_init(>req_entity,
>> _request_entity_ops,
>> +  >dev->vfd);
>> + ctx->fh.entity = >req_entity.base;
>>   v4l2_ctrl_new_std(hdl, _ctrl_ops, V4L2_CID_HFLIP, 0, 1,
>> 1, 0);
>>   v4l2_ctrl_new_std(hdl, _ctrl_ops, V4L2_CID_VFLIP, 0, 1,
>> 1, 0);
>>   v4l2_ctrl_new_custom(hdl, _ctrl_trans_time_msec, NULL);
>> @@ -

[PATCH v4l-utils] v4l2-compliance/v4l2-test-formats: fix typo

[Alexandre Courbot]

When using planar formats, we want to check pix_mp, not pix.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 utils/v4l2-compliance/v4l2-test-formats.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utils/v4l2-compliance/v4l2-test-formats.cpp 
b/utils/v4l2-compliance/v4l2-test-formats.cpp
index a336afdd..e9170688 100644
--- a/utils/v4l2-compliance/v4l2-test-formats.cpp
+++ b/utils/v4l2-compliance/v4l2-test-formats.cpp
@@ -452,7 +452,7 @@ static int testFormatsType(struct node *node, int ret,  
unsigned type, struct v4
if (!node->is_m2m)
fail_on_test(testColorspace(pix_mp.pixelformat, 
pix_mp.colorspace, 
 pix_mp.ycbcr_enc, 
pix_mp.quantization));
-   fail_on_test(pix.field == V4L2_FIELD_ANY);
+   fail_on_test(pix_mp.field == V4L2_FIELD_ANY);
ret = check_0(pix_mp.reserved, sizeof(pix_mp.reserved));
if (ret)
return fail("pix_mp.reserved not zeroed\n");
-- 
2.16.2.395.g2e18187dfd-goog

Re: [RFCv4 01/21] media: add request API core and UAPI

[Alexandre Courbot]

On Wed, Feb 21, 2018 at 4:29 PM, Hans Verkuil <hverk...@xs4all.nl> wrote:
> On 02/21/2018 07:01 AM, Alexandre Courbot wrote:
>> Hi Hans,
>>
>> On Tue, Feb 20, 2018 at 7:36 PM, Hans Verkuil <hverk...@xs4all.nl> wrote:
>>> On 02/20/18 05:44, Alexandre Courbot wrote:
>
> 
>
>>>> +#define MEDIA_REQUEST_IOC(__cmd, func)
>>>>\
>>>> + [_IOC_NR(MEDIA_REQUEST_IOC_##__cmd) - 0x80] = { \
>>>> + .cmd = MEDIA_REQUEST_IOC_##__cmd,   \
>>>> + .fn = func, \
>>>> + }
>>>> +
>>>> +struct media_request_ioctl_info {
>>>> + unsigned int cmd;
>>>> + long (*fn)(struct media_request *req);
>>>> +};
>>>> +
>>>> +static const struct media_request_ioctl_info ioctl_info[] = {
>>>> + MEDIA_REQUEST_IOC(SUBMIT, media_request_ioctl_submit),
>>>> + MEDIA_REQUEST_IOC(REINIT, media_request_ioctl_reinit),
>>>
>>> There are only two ioctls, so there is really no need for the
>>> MEDIA_REQUEST_IOC define. Just keep it simple.
>>
>> The number of times it is used doesn't change the fact that it helps
>> with readability IMHO.
>
> But this macro just boils down to:
>
> static const struct media_request_ioctl_info ioctl_info[] = {
> { MEDIA_REQUEST_IOC_SUBMIT, media_request_ioctl_submit },
> { MEDIA_REQUEST_IOC_REINIT, media_request_ioctl_reinit },
> };
>
> It's absolutely identical! So it seems senseless to me.

This expands to more than that - the index needs to be offset by 0x80,
something we probably don't want to repeat every line.

>
>>
>>>
>>>> +};
>>>> +
>>>> +static long media_request_ioctl(struct file *filp, unsigned int cmd,
>>>> + unsigned long __arg)
>>>> +{
>>>> + struct media_request *req = filp->private_data;
>>>> + const struct media_request_ioctl_info *info;
>>>> +
>>>> + if ((_IOC_NR(cmd) < 0x80) ||
>>>
>>> Why start the ioctl number at 0x80? Why not just 0?
>>> It avoids all this hassle with the 0x80 offset.
>
> There is no clash with the MC ioctls, so I really don't believe the 0x80
> offset is needed.

I suppose your comment in patch 16 supersedes this one. :)

>
>>>
>>>> +  _IOC_NR(cmd) >= 0x80 + ARRAY_SIZE(ioctl_info) ||
>>>> +  ioctl_info[_IOC_NR(cmd) - 0x80].cmd != cmd)
>>>> + return -ENOIOCTLCMD;
>>>> +
>>>> + info = _info[_IOC_NR(cmd) - 0x80];
>>>> +
>>>> + return info->fn(req);
>>>> +}
>
> 
>
>>>> diff --git a/include/uapi/linux/media-request.h 
>>>> b/include/uapi/linux/media-request.h
>>>> new file mode 100644
>>>> index ..5d30f731a442
>>>> --- /dev/null
>>>> +++ b/include/uapi/linux/media-request.h
>>>> @@ -0,0 +1,37 @@
>>>> +/*
>>>> + * Media requests UAPI
>>>> + *
>>>> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify
>>>> + * it under the terms of the GNU General Public License version 2 as
>>>> + * published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>> + * GNU General Public License for more details.
>>>> + */
>>>> +
>>>> +#ifndef __LINUX_MEDIA_REQUEST_H
>>>> +#define __LINUX_MEDIA_REQUEST_H
>>>> +
>>>> +#ifndef __KERNEL__
>>>> +#include 
>>>> +#endif
>>>> +#include 
>>>> +#include 
>>>> +#include 
>>>> +
>>>> +/* Only check that requests can be used, do not allocate */
>>>> +#define MEDIA_REQUEST_FLAG_TEST  0x0001
>>>> +
>>>> +struct media_request_new {
>>>> + __u32 flags;
>>>> + __s32 fd;
>>>> +} __attribute__ ((packed));
>>>> +
>>>> +#define MEDIA_REQUEST_IOC_SUBMIT   _IO('|',  128)
>>>> +#define MEDI

Re: [RFCv4 01/21] media: add request API core and UAPI

[Alexandre Courbot]

Hi Hans,

On Tue, Feb 20, 2018 at 7:36 PM, Hans Verkuil <hverk...@xs4all.nl> wrote:
> On 02/20/18 05:44, Alexandre Courbot wrote:
>> The request API provides a way to group buffers and device parameters
>> into units of work to be queued and executed. This patch introduces the
>> UAPI and core framework.
>>
>> This patch is based on the previous work by Laurent Pinchart. The core
>> has changed considerably, but the UAPI is mostly untouched.
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  drivers/media/Kconfig  |   3 +
>>  drivers/media/Makefile |   6 +
>>  drivers/media/media-request.c  | 341 
>>  include/media/media-request.h  | 349 +
>>  include/uapi/linux/media-request.h |  37 +++
>>  5 files changed, 736 insertions(+)
>>  create mode 100644 drivers/media/media-request.c
>>  create mode 100644 include/media/media-request.h
>>  create mode 100644 include/uapi/linux/media-request.h
>>
>> diff --git a/drivers/media/Kconfig b/drivers/media/Kconfig
>> index 145e12bfb819..db30fc9547d2 100644
>> --- a/drivers/media/Kconfig
>> +++ b/drivers/media/Kconfig
>> @@ -130,6 +130,9 @@ config VIDEO_V4L2_SUBDEV_API
>>
>> This API is mostly used by camera interfaces in embedded platforms.
>>
>> +config MEDIA_REQUEST_API
>> + tristate
>> +
>>  source "drivers/media/v4l2-core/Kconfig"
>>
>>  #
>> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
>> index 594b462ddf0e..03c0a39ad344 100644
>> --- a/drivers/media/Makefile
>> +++ b/drivers/media/Makefile
>> @@ -5,6 +5,12 @@
>>
>>  media-objs   := media-device.o media-devnode.o media-entity.o
>>
>> +#
>> +# Request API support comes as its own module since it can be used by
>> +# both media and video devices
>> +#
>> +obj-$(CONFIG_MEDIA_REQUEST_API) += media-request.o
>> +
>>  #
>>  # I2C drivers should come before other drivers, otherwise they'll fail
>>  # when compiled as builtin drivers
>> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
>> new file mode 100644
>> index ..b88362028561
>> --- /dev/null
>> +++ b/drivers/media/media-request.c
>> @@ -0,0 +1,341 @@
>> +/*
>> + * Request base implementation
>> + *
>> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +
>> +#include 
>> +
>> +const struct file_operations request_fops;
>> +
>> +static const char * const media_request_states[] __maybe_unused = {
>
> Why 'maybe_unused'?
>
>> + "IDLE",
>> + "SUBMITTED",
>> + "COMPLETED",
>> + "INVALID",
>
> I don't like to yell. I prefer "Idle", "Submitted", etc.

Sure.

>
>> +};
>> +
>> +struct media_request *media_request_get(struct media_request *req)
>> +{
>> + get_file(req->file);
>> + return req;
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_get);
>> +
>> +struct media_request *media_request_get_from_fd(int fd)
>> +{
>> + struct file *f;
>> +
>> + f = fget(fd);
>> + if (!f)
>> + return NULL;
>> +
>> + /* Not a request FD? */
>> + if (f->f_op != _fops) {
>> + fput(f);
>> + return NULL;
>> + }
>> +
>> + return f->private_data;
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_get_from_fd);
>> +
>> +void media_request_put(struct media_request *req)
>> +{
>> + if (WARN_ON(req == NULL))
>> + return;
>> +
>> + fput(req->file);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_put);
>> +
>> +struct media_request_entity_data *
>&g

Re: [RFCv4 10/21] videodev2.h: Add request_fd field to v4l2_buffer

[Alexandre Courbot]

On Wed, Feb 21, 2018 at 12:20 AM, Hans Verkuil <hverk...@xs4all.nl> wrote:
> On 02/20/18 05:44, Alexandre Courbot wrote:
>> From: Hans Verkuil <hans.verk...@cisco.com>
>>
>> When queuing buffers allow for passing the request that should
>> be associated with this buffer.
>>
>> Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
>> [acour...@chromium.org: make request ID 32-bit]
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  drivers/media/common/videobuf2/videobuf2-v4l2.c | 2 +-
>>  drivers/media/usb/cpia2/cpia2_v4l.c | 2 +-
>>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c   | 9 ++---
>>  drivers/media/v4l2-core/v4l2-ioctl.c| 4 ++--
>>  include/uapi/linux/videodev2.h  | 3 ++-
>>  5 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> index 886a2d8d5c6c..6d4d184aa68e 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> @@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, 
>> void *pb)
>>   b->timestamp = ns_to_timeval(vb->timestamp);
>>   b->timecode = vbuf->timecode;
>>   b->sequence = vbuf->sequence;
>> - b->reserved2 = 0;
>> + b->request_fd = 0;
>>   b->reserved = 0;
>>
>>   if (q->is_multiplanar) {
>> diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
>> b/drivers/media/usb/cpia2/cpia2_v4l.c
>> index 99f106b13280..af42ce3ceb48 100644
>> --- a/drivers/media/usb/cpia2/cpia2_v4l.c
>> +++ b/drivers/media/usb/cpia2/cpia2_v4l.c
>> @@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, 
>> struct v4l2_buffer *buf)
>>   buf->sequence = cam->buffers[buf->index].seq;
>>   buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
>>   buf->length = cam->frame_size;
>> - buf->reserved2 = 0;
>> + buf->request_fd = 0;
>>   buf->reserved = 0;
>>   memset(>timecode, 0, sizeof(buf->timecode));
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
>> b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>> index 5198c9eeb348..32bf47489a2e 100644
>> --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>> +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>> @@ -386,7 +386,7 @@ struct v4l2_buffer32 {
>>   __s32   fd;
>>   } m;
>>   __u32   length;
>> - __u32   reserved2;
>> + __s32   request_fd;
>>   __u32   reserved;
>>  };
>>
>> @@ -486,6 +486,7 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user 
>> *kp,
>>  {
>>   u32 type;
>>   u32 length;
>> + s32 request_fd;
>>   enum v4l2_memory memory;
>>   struct v4l2_plane32 __user *uplane32;
>>   struct v4l2_plane __user *uplane;
>> @@ -500,7 +501,9 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user 
>> *kp,
>>   get_user(memory, >memory) ||
>>   put_user(memory, >memory) ||
>>   get_user(length, >length) ||
>> - put_user(length, >length))
>> + put_user(length, >length) ||
>> + get_user(request_fd, >request_fd) ||
>> + put_user(request_fd, >request_fd))
>>   return -EFAULT;
>>
>>   if (V4L2_TYPE_IS_OUTPUT(type))
>> @@ -604,7 +607,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user 
>> *kp,
>>   assign_in_user(>timestamp.tv_usec, >timestamp.tv_usec) ||
>>   copy_in_user(>timecode, >timecode, sizeof(kp->timecode)) ||
>>   assign_in_user(>sequence, >sequence) ||
>> - assign_in_user(>reserved2, >reserved2) ||
>> + assign_in_user(>request_fd, >request_fd) ||
>>   assign_in_user(>reserved, >reserved) ||
>>   get_user(length, >length) ||
>>   put_user(length, >length))
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
>> b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index 260288ca4f55..7bfeaf233d5a 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool 
>> write_only)
>>   con

Re: [RFCv4 13/21] media: videobuf2-v4l2: support for requests

[Alexandre Courbot]

On Wed, Feb 21, 2018 at 1:18 AM, Hans Verkuil <hverk...@xs4all.nl> wrote:
> On 02/20/2018 05:44 AM, Alexandre Courbot wrote:
>> Add a new vb2_qbuf_request() (a request-aware version of vb2_qbuf())
>> that request-aware drivers can call to queue a buffer into a request
>> instead of directly into the vb2 queue if relevent.
>>
>> This function expects that drivers invoking it are using instances of
>> v4l2_request_entity and v4l2_request_entity_data to describe their
>> entity and entity data respectively.
>>
>> Also add the vb2_request_submit() helper function which drivers can
>> invoke in order to queue all the buffers previously queued into a
>> request into the regular vb2 queue.
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  .../media/common/videobuf2/videobuf2-v4l2.c   | 129 +-
>>  include/media/videobuf2-v4l2.h|  59 
>>  2 files changed, 187 insertions(+), 1 deletion(-)
>>
>
> 
>
>> @@ -776,10 +899,14 @@ EXPORT_SYMBOL_GPL(vb2_ioctl_querybuf);
>>  int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
>>  {
>>   struct video_device *vdev = video_devdata(file);
>> + struct v4l2_fh *fh = NULL;
>> +
>> + if (test_bit(V4L2_FL_USES_V4L2_FH, >flags))
>> + fh = file->private_data;
>
> No need for this. All drivers using vb2 will also use v4l2_fh.

Fixed.

>
>>
>>   if (vb2_queue_is_busy(vdev, file))
>>   return -EBUSY;
>> - return vb2_qbuf(vdev->queue, p);
>> + return vb2_qbuf_request(vdev->queue, p, fh ? fh->entity : NULL);
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_ioctl_qbuf);
>>
>> diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
>> index 3d5e2d739f05..d4dfa266a0da 100644
>> --- a/include/media/videobuf2-v4l2.h
>> +++ b/include/media/videobuf2-v4l2.h
>> @@ -23,6 +23,12 @@
>>  #error VB2_MAX_PLANES != VIDEO_MAX_PLANES
>>  #endif
>>
>> +struct media_entity;
>> +struct v4l2_fh;
>> +struct media_request;
>> +struct media_request_entity;
>> +struct v4l2_request_entity_data;
>> +
>>  /**
>>   * struct vb2_v4l2_buffer - video buffer information for v4l2.
>>   *
>> @@ -116,6 +122,59 @@ int vb2_prepare_buf(struct vb2_queue *q, struct 
>> v4l2_buffer *b);
>>   */
>>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b);
>>
>> +#if IS_ENABLED(CONFIG_MEDIA_REQUEST_API)
>> +
>> +/**
>> + * vb2_qbuf_request() - Queue a buffer, with request support
>> + * @q:   pointer to  vb2_queue with videobuf2 queue.
>> + * @b:   buffer structure passed from userspace to
>> + *   _ioctl_ops->vidioc_qbuf handler in driver
>> + * @entity:  request entity to queue for if requests are used.
>> + *
>> + * Should be called from _ioctl_ops->vidioc_qbuf handler of a driver.
>> + *
>> + * If requests are not in use, calling this is equivalent to calling 
>> vb2_qbuf().
>> + *
>> + * If the request_fd member of b is set, then the buffer represented by b is
>> + * queued in the request instead of the vb2 queue. The buffer will be passed
>> + * to the vb2 queue when the request is submitted.
>
> I would definitely also prepare the buffer at this time. That way you'll see 
> any
> errors relating to the prepare early on.

I was wondering about that, so glad to have your opinion on this. Will
make sure buffers are prepared before queuing them to a request.

>
>> + *
>> + * The return values from this function are intended to be directly returned
>> + * from _ioctl_ops->vidioc_qbuf handler in driver.
>> + */
>> +int vb2_qbuf_request(struct vb2_queue *q, struct v4l2_buffer *b,
>> +  struct media_request_entity *entity);
>> +
>> +/**
>> + * vb2_request_submit() - Queue all the buffers in a v4l2 request.
>> + * @data:request entity data to queue buffers of
>> + *
>> + * This function should be called from the media_request_entity_ops::submit
>> + * hook for instances of media_request_v4l2_entity_data. It will immediately
>> + * queue all the request-bound buffers to their respective vb2 queues.
>> + *
>> + * Errors from vb2_core_qbuf() are returned if something happened. Also, 
>> since
>> + * v4l2 request entities require at least one buffer for the request to 
>> trigger,
>> + * this function will return -EINVAL if no buffer have been bound at all for
>> + * this entity.
>> + */
>> +int vb2_request_submit(struct v4l2_request_entity_data *data);
>> +
>> +#else /* CONFIG_MEDIA_REQUEST_API */
>> +
>> +static inline int vb2_qbuf_request(struct vb2_queue *q, struct v4l2_buffer 
>> *b,
>> +struct media_request_entity *entity)
>> +{
>> + return vb2_qbuf(q, b);
>> +}
>> +
>> +static inline int vb2_request_submit(struct v4l2_request_entity_data *data)
>> +{
>> + return -ENOTSUPP;
>> +}
>> +
>> +#endif /* CONFIG_MEDIA_REQUEST_API */
>> +
>>  /**
>>   * vb2_expbuf() - Export a buffer as a file descriptor
>>   * @q:   pointer to  vb2_queue with videobuf2 queue.
>>
>
> Regards,
>
> Hans

Re: [RFCv4 11/21] media: v4l2_fh: add request entity field

[Alexandre Courbot]

On Wed, Feb 21, 2018 at 12:24 AM, Hans Verkuil <hverk...@xs4all.nl> wrote:
> On 02/20/18 05:44, Alexandre Courbot wrote:
>> Allow drivers to assign a request entity to v4l2_fh. This will be useful
>> for request-aware ioctls to find out which request entity to use.
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  include/media/v4l2-fh.h | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
>> index ea73fef8bdc0..f54cb319dd64 100644
>> --- a/include/media/v4l2-fh.h
>> +++ b/include/media/v4l2-fh.h
>> @@ -28,6 +28,7 @@
>>
>>  struct video_device;
>>  struct v4l2_ctrl_handler;
>> +struct media_request_entity;
>>
>>  /**
>>   * struct v4l2_fh - Describes a V4L2 file handler
>> @@ -43,6 +44,7 @@ struct v4l2_ctrl_handler;
>>   * @navailable: number of available events at @available list
>>   * @sequence: event sequence number
>>   * @m2m_ctx: pointer to  v4l2_m2m_ctx
>> + * @entity: the request entity this fh operates on behalf of
>>   */
>>  struct v4l2_fh {
>>   struct list_headlist;
>> @@ -60,6 +62,7 @@ struct v4l2_fh {
>>  #if IS_ENABLED(CONFIG_V4L2_MEM2MEM_DEV)
>>   struct v4l2_m2m_ctx *m2m_ctx;
>>  #endif
>> + struct media_request_entity *entity;
>
> The name 'media_request_entity' is very confusing.
>
> In the media controller API terminology an entity represents a piece
> of hardware with inputs and outputs (very rough description), but a
> request is not an entity. It may be associated with an entity, though.
>
> So calling this field 'entity' is also very misleading.

Note that this field is not a request though, it is a pointer to a
piece of hardware referenced by a request, which is closer to the MC
terminology. Or do you mean this should just be renamed
"request_entity"?

If we go all the way, the media_ prefix is also misleading - it
implies a dependency to the media controller framework, while there is
none (in this patchset at least).

However I thought that 'request' alone (instead of media_request) may
name-conflict with something else, and since 'media' is also the
umbrella term for anything under drivers/media it sounds fitting on
the other hand. Suggestions are welcome though.

>
> As with previous patches, I'll have to think about this and try and
> come up with better, less confusing names.

I will gladly take suggestions, have been trying to come with a better
name to reply to your comment above but could not find any. :)

Re: [RFCv4 16/21] v4l2: video_device: support for creating requests

[Alexandre Courbot]

On Wed, Feb 21, 2018 at 1:35 AM, Hans Verkuil <hverk...@xs4all.nl> wrote:
> On 02/20/2018 05:44 AM, Alexandre Courbot wrote:
>> Add a new VIDIOC_NEW_REQUEST ioctl, which allows to instanciate requests
>> on devices that support the request API. Requests created that way can
>> only control the device they originate from, making them suitable for
>> simple devices, but not complex pipelines.
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  Documentation/ioctl/ioctl-number.txt |  1 +
>>  drivers/media/v4l2-core/v4l2-dev.c   |  2 ++
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 25 +
>>  include/media/v4l2-dev.h |  2 ++
>>  include/uapi/linux/videodev2.h   |  3 +++
>>  5 files changed, 33 insertions(+)
>>
>> diff --git a/Documentation/ioctl/ioctl-number.txt 
>> b/Documentation/ioctl/ioctl-number.txt
>> index 6501389d55b9..afdc9ed255b0 100644
>> --- a/Documentation/ioctl/ioctl-number.txt
>> +++ b/Documentation/ioctl/ioctl-number.txt
>> @@ -286,6 +286,7 @@ Code  Seq#(hex)   Include FileComments
>>   <mailto:o...@port.de>
>>  'z'  10-4F   drivers/s390/crypto/zcrypt_api.hconflict!
>>  '|'  00-7F   linux/media.h
>> +'|'  80-9F   linux/media-request.h
>>  0x80 00-1F   linux/fb.h
>>  0x89 00-06   arch/x86/include/asm/sockios.h
>>  0x89 0B-DF   linux/sockios.h
>
> This  doesn't belong in this patch.

Do you mean we need a separate patch for this?

>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c 
>> b/drivers/media/v4l2-core/v4l2-dev.c
>> index 0301fe426a43..062ebee5bffc 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -559,6 +559,8 @@ static void determine_valid_ioctls(struct video_device 
>> *vdev)
>>   set_bit(_IOC_NR(VIDIOC_TRY_EXT_CTRLS), valid_ioctls);
>>   if (vdev->ctrl_handler || ops->vidioc_querymenu)
>>   set_bit(_IOC_NR(VIDIOC_QUERYMENU), valid_ioctls);
>> + if (vdev->req_mgr)
>> + set_bit(_IOC_NR(VIDIOC_NEW_REQUEST), valid_ioctls);
>>   SET_VALID_IOCTL(ops, VIDIOC_G_FREQUENCY, vidioc_g_frequency);
>>   SET_VALID_IOCTL(ops, VIDIOC_S_FREQUENCY, vidioc_s_frequency);
>>   SET_VALID_IOCTL(ops, VIDIOC_LOG_STATUS, vidioc_log_status);
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
>> b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index ab4968ea443f..a45fe078f8ae 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -21,6 +21,7 @@
>>
>>  #include 
>>
>> +#include 
>>  #include 
>>  #include 
>>  #include 
>> @@ -842,6 +843,13 @@ static void v4l_print_freq_band(const void *arg, bool 
>> write_only)
>>   p->rangehigh, p->modulation);
>>  }
>>
>> +static void vidioc_print_new_request(const void *arg, bool write_only)
>> +{
>> + const struct media_request_new *new = arg;
>> +
>> + pr_cont("fd=0x%x\n", new->fd);
>
> I'd use %d since fds are typically shown as signed integers.

Right.

>
>> +}
>> +
>>  static void v4l_print_edid(const void *arg, bool write_only)
>>  {
>>   const struct v4l2_edid *p = arg;
>> @@ -2486,6 +2494,22 @@ static int v4l_enum_freq_bands(const struct 
>> v4l2_ioctl_ops *ops,
>>   return -ENOTTY;
>>  }
>>
>> +static int vidioc_new_request(const struct v4l2_ioctl_ops *ops,
>> +   struct file *file, void *fh, void *arg)
>> +{
>> +#if IS_ENABLED(CONFIG_MEDIA_REQUEST_API)
>> + struct media_request_new *new = arg;
>> + struct video_device *vfd = video_devdata(file);
>> +
>> + if (!vfd->req_mgr)
>> + return -ENOTTY;
>> +
>> + return media_request_ioctl_new(vfd->req_mgr, new);
>> +#else
>> + return -ENOTTY;
>> +#endif
>> +}
>
> You don't need the #ifdef's here. media_request_ioctl_new() will be stubbed if
> CONFIG_MEDIA_REQUEST_API isn't set.

Correct.

>
>> +
>>  struct v4l2_ioctl_info {
>>   unsigned int ioctl;
>>   u32 flags;
>> @@ -2617,6 +2641,7 @@ static struct v4l2_ioctl_info v4l2_ioctls[] = {
>>   IOCTL_INFO_FNC(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, 
>> v4l_print_freq_band, 0),
>>   IOCTL_INFO_FNC(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, 
>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)),
>>   IOC

Re: [RFCv4 09/21] v4l2: add request API support

[Alexandre Courbot]

On Tue, Feb 20, 2018 at 10:25 PM, Hans Verkuil <hverk...@xs4all.nl> wrote:
> On 02/20/18 05:44, Alexandre Courbot wrote:
>> Add a v4l2 request entity data structure that takes care of storing the
>> request-related state of a V4L2 device ; in this case, its controls.
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  drivers/media/v4l2-core/Makefile   |   1 +
>>  drivers/media/v4l2-core/v4l2-request.c | 178 +
>>  include/media/v4l2-request.h   | 159 ++
>>  3 files changed, 338 insertions(+)
>>  create mode 100644 drivers/media/v4l2-core/v4l2-request.c
>>  create mode 100644 include/media/v4l2-request.h
>>
>> diff --git a/drivers/media/v4l2-core/Makefile 
>> b/drivers/media/v4l2-core/Makefile
>> index 80de2cb9c476..13d0477535bd 100644
>> --- a/drivers/media/v4l2-core/Makefile
>> +++ b/drivers/media/v4l2-core/Makefile
>> @@ -16,6 +16,7 @@ ifeq ($(CONFIG_TRACEPOINTS),y)
>>videodev-objs += vb2-trace.o v4l2-trace.o
>>  endif
>>  videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o
>> +videodev-$(CONFIG_MEDIA_REQUEST_API) += v4l2-request.o
>>
>>  obj-$(CONFIG_VIDEO_V4L2) += videodev.o
>>  obj-$(CONFIG_VIDEO_V4L2) += v4l2-common.o
>> diff --git a/drivers/media/v4l2-core/v4l2-request.c 
>> b/drivers/media/v4l2-core/v4l2-request.c
>> new file mode 100644
>> index ..e8ad10e2f525
>> --- /dev/null
>> +++ b/drivers/media/v4l2-core/v4l2-request.c
>> @@ -0,0 +1,178 @@
>> +/*
>> + * Media requests support for V4L2
>> + *
>> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include 
>> +
>> +#include 
>> +#include 
>> +#include 
>> +
>> +void v4l2_request_entity_init(struct v4l2_request_entity *entity,
>> +   const struct media_request_entity_ops *ops,
>> +   struct video_device *vdev)
>> +{
>> + media_request_entity_init(>base, 
>> MEDIA_REQUEST_ENTITY_TYPE_V4L2, ops);
>> + entity->vdev = vdev;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_request_entity_init);
>> +
>> +struct media_request_entity_data *
>> +v4l2_request_entity_data_alloc(struct media_request *req,
>> +struct v4l2_ctrl_handler *hdl)
>> +{
>> + struct v4l2_request_entity_data *data;
>> + int ret;
>> +
>> + data = kzalloc(sizeof(*data), GFP_KERNEL);
>> + if (!data)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + ret = v4l2_ctrl_request_init(>ctrls);
>> + if (ret) {
>> + kfree(data);
>> + return ERR_PTR(ret);
>> + }
>> + ret = v4l2_ctrl_request_clone(>ctrls, hdl, NULL);
>> + if (ret) {
>> + kfree(data);
>> + return ERR_PTR(ret);
>> + }
>> +
>> + INIT_LIST_HEAD(>queued_buffers);
>> +
>> + return >base;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_request_entity_data_alloc);
>> +
>> +void v4l2_request_entity_data_free(struct media_request_entity_data *_data)
>> +{
>> + struct v4l2_request_entity_data *data;
>> + struct v4l2_vb2_request_buffer *qb, *n;
>> +
>> + data = to_v4l2_entity_data(_data);
>> +
>> + list_for_each_entry_safe(qb, n, >queued_buffers, node) {
>> + struct vb2_buffer *buf;
>> + dev_warn(_data->request->mgr->dev,
>> +  "entity data freed while buffer still queued!\n");
>> +
>> + /* give buffer back to user-space */
>> + buf = qb->queue->bufs[qb->v4l2_buf.index];
>> + buf->state = qb->pre_req_state;
>> + buf->request = NULL;
>> +
>> + kfree(qb);
>> + }
>> +
>> + v4l2_ctrl_handler_free(>ctrls);
>> + kfree(data);
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_request_entity_data_free);
>> +
>> +
>> +
>

Re: [RFCv4 09/21] v4l2: add request API support

[Alexandre Courbot]

Hi Philippe,

On Tue, Feb 20, 2018 at 4:36 PM, Philippe Ombredanne
<pombreda...@nexb.com> wrote:
> On Tue, Feb 20, 2018 at 5:44 AM, Alexandre Courbot
> <acour...@chromium.org> wrote:
>> Add a v4l2 request entity data structure that takes care of storing the
>> request-related state of a V4L2 device ; in this case, its controls.
>>
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>
> 
>
>> --- /dev/null
>> +++ b/drivers/media/v4l2-core/v4l2-request.c
>> @@ -0,0 +1,178 @@
>> +/*
>> + * Media requests support for V4L2
>> + *
>> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>
> Do you mind using SPDX tags per [1] rather that this fine but long
> legalese. (Here and in the whole patch series)
>
> [1] 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/license-rules.rst

I need to check what is the stance of Chromium OS regarding these, but
will gladly comply if possible.

[RFCv4 00/21] Request API

[Alexandre Courbot]

, while vivid
gives an example of a global request entity.
* Improved documentation.

Issues remaining with this version:
* Vivid support exposed what seems to be a limitation in the current controls
cloning implementation. See patch 8 (which works the issue around) for details.
* Media controller implementation of requests is still incomplete. I plan to
complete and exert it on an actual camera driver sometime soon.


Alexandre Courbot (14):
  media: add request API core and UAPI
  [WAR] v4l2-ctrls: do not clone non-standard controls
  v4l2: add request API support
  media: v4l2_fh: add request entity field
  media: videobuf2: add support for requests
  media: videobuf2-v4l2: support for requests
  videodev2.h: add request_fd field to v4l2_ext_controls
  v4l2-ctrls: support requests in EXT_CTRLS ioctls
  v4l2: video_device: support for creating requests
  media: mem2mem: support for requests
  Documentation: v4l: document request API
  media: vim2m: add request support
  media: vivid: add request support for the video capture device
  [WIP] media: media-device: support for creating requests

Hans Verkuil (7):
  v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev
  v4l2-ctrls: prepare internal structs for request API
  v4l2-ctrls: add core request API
  v4l2-ctrls: use ref in helper instead of ctrl
  v4l2-ctrls: support g/s_ext_ctrls for requests
  v4l2-ctrls: add v4l2_ctrl_request_setup
  videodev2.h: Add request_fd field to v4l2_buffer

 Documentation/ioctl/ioctl-number.txt  |   1 +
 Documentation/media/uapi/v4l/buffer.rst   |   9 +-
 Documentation/media/uapi/v4l/common.rst   |   1 +
 Documentation/media/uapi/v4l/request-api.rst  | 199 ++
 Documentation/media/uapi/v4l/user-func.rst|   1 +
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst |  16 +-
 .../media/uapi/v4l/vidioc-new-request.rst |  64 
 Documentation/media/uapi/v4l/vidioc-qbuf.rst  |   7 +
 drivers/media/Kconfig |   3 +
 drivers/media/Makefile|   6 +
 .../media/common/videobuf2/videobuf2-core.c   |   3 +
 .../media/common/videobuf2/videobuf2-v4l2.c   | 131 ++-
 drivers/media/dvb-frontends/rtl2832_sdr.c |   5 +-
 drivers/media/media-device.c  |  11 +
 drivers/media/media-request.c | 341 +
 drivers/media/pci/bt8xx/bttv-driver.c |   2 +-
 drivers/media/pci/cx23885/cx23885-417.c   |   2 +-
 drivers/media/pci/cx88/cx88-blackbird.c   |   2 +-
 drivers/media/pci/cx88/cx88-video.c   |   2 +-
 drivers/media/pci/saa7134/saa7134-empress.c   |   4 +-
 drivers/media/pci/saa7134/saa7134-video.c |   2 +-
 drivers/media/platform/Kconfig|   1 +
 .../media/platform/exynos4-is/fimc-capture.c  |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c|   2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c   |   3 +-
 drivers/media/platform/rcar_drif.c|   2 +-
 .../media/platform/soc_camera/soc_camera.c|   3 +-
 drivers/media/platform/vim2m.c|  75 
 drivers/media/platform/vivid/Kconfig  |   1 +
 drivers/media/platform/vivid/vivid-core.c |  63 +++-
 drivers/media/platform/vivid/vivid-core.h |   3 +
 drivers/media/platform/vivid/vivid-ctrls.c|  46 +--
 .../media/platform/vivid/vivid-kthread-cap.c  |  17 +
 drivers/media/usb/cpia2/cpia2_v4l.c   |   2 +-
 drivers/media/usb/cx231xx/cx231xx-417.c   |   2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c |   4 +-
 drivers/media/usb/msi2500/msi2500.c   |   2 +-
 drivers/media/usb/tm6000/tm6000-video.c   |   2 +-
 drivers/media/v4l2-core/Makefile  |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c |   9 +-
 drivers/media/v4l2-core/v4l2-ctrls.c  | 341 +++--
 drivers/media/v4l2-core/v4l2-dev.c|   2 +
 drivers/media/v4l2-core/v4l2-device.c |   3 +-
 drivers/media/v4l2-core/v4l2-ioctl.c  |  37 +-
 drivers/media/v4l2-core/v4l2-mem2mem.c|   3 +-
 drivers/media/v4l2-core/v4l2-request.c| 178 +
 drivers/media/v4l2-core/v4l2-subdev.c |   2 +-
 drivers/staging/media/imx/imx-media-dev.c |   2 +-
 drivers/staging/media/imx/imx-media-fim.c |   2 +-
 include/media/mc-request.h|  33 ++
 include/media/media-device.h  |   1 +
 include/media/media-entity.h  |   5 +
 include/media/media-request.h | 349 ++
 include/media/v4l2-ctrls.h|  20 +-
 include/media/v4l2-dev.h  |   2 +
 include/media/v4l2-fh.h   |   3 +
 include/media/v4l2-request.h  | 159 
 include/media/videobuf2-core.h|   4 +
 include/media/videobuf2-v4l2.h|  59 +++
 include/uapi/linux/media-request.h|  37 ++
 include/uapi/linux/media.h|   2 +
 include/uapi/linux/videodev2.h|   9 +-
 62

[RFCv4 02/21] v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a 'bool from_other_dev' argument: set to true if the two
handlers refer to different devices (e.g. it is true when
inheriting controls from a subdev into a main v4l2 bridge
driver).

This will be used later when implementing support for the
request API since we need to skip such controls.

TODO: check drivers/staging/media/imx/imx-media-fim.c change.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/dvb-frontends/rtl2832_sdr.c |  5 +-
 drivers/media/pci/bt8xx/bttv-driver.c |  2 +-
 drivers/media/pci/cx23885/cx23885-417.c   |  2 +-
 drivers/media/pci/cx88/cx88-blackbird.c   |  2 +-
 drivers/media/pci/cx88/cx88-video.c   |  2 +-
 drivers/media/pci/saa7134/saa7134-empress.c   |  4 +-
 drivers/media/pci/saa7134/saa7134-video.c |  2 +-
 .../media/platform/exynos4-is/fimc-capture.c  |  2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c   |  3 +-
 drivers/media/platform/rcar_drif.c|  2 +-
 .../media/platform/soc_camera/soc_camera.c|  3 +-
 drivers/media/platform/vivid/vivid-ctrls.c| 46 +--
 drivers/media/usb/cx231xx/cx231xx-417.c   |  2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c |  4 +-
 drivers/media/usb/msi2500/msi2500.c   |  2 +-
 drivers/media/usb/tm6000/tm6000-video.c   |  2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c  | 11 +++--
 drivers/media/v4l2-core/v4l2-device.c |  3 +-
 drivers/staging/media/imx/imx-media-dev.c |  2 +-
 drivers/staging/media/imx/imx-media-fim.c |  2 +-
 include/media/v4l2-ctrls.h|  4 +-
 21 files changed, 58 insertions(+), 49 deletions(-)

diff --git a/drivers/media/dvb-frontends/rtl2832_sdr.c 
b/drivers/media/dvb-frontends/rtl2832_sdr.c
index c6e78d870ccd..6064d28224e8 100644
--- a/drivers/media/dvb-frontends/rtl2832_sdr.c
+++ b/drivers/media/dvb-frontends/rtl2832_sdr.c
@@ -1394,7 +1394,8 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
case RTL2832_SDR_TUNER_E4000:
v4l2_ctrl_handler_init(>hdl, 9);
if (subdev)
-   v4l2_ctrl_add_handler(>hdl, subdev->ctrl_handler, 
NULL);
+   v4l2_ctrl_add_handler(>hdl, subdev->ctrl_handler,
+ NULL, true);
break;
case RTL2832_SDR_TUNER_R820T:
case RTL2832_SDR_TUNER_R828D:
@@ -1423,7 +1424,7 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
v4l2_ctrl_handler_init(>hdl, 2);
if (subdev)
v4l2_ctrl_add_handler(>hdl, subdev->ctrl_handler,
- NULL);
+ NULL, true);
break;
default:
v4l2_ctrl_handler_init(>hdl, 0);
diff --git a/drivers/media/pci/bt8xx/bttv-driver.c 
b/drivers/media/pci/bt8xx/bttv-driver.c
index f697698fe38d..cdcb36d8c5c3 100644
--- a/drivers/media/pci/bt8xx/bttv-driver.c
+++ b/drivers/media/pci/bt8xx/bttv-driver.c
@@ -4211,7 +4211,7 @@ static int bttv_probe(struct pci_dev *dev, const struct 
pci_device_id *pci_id)
/* register video4linux + input */
if (!bttv_tvcards[btv->c.type].no_video) {
v4l2_ctrl_add_handler(>radio_ctrl_handler, hdl,
-   v4l2_ctrl_radio_filter);
+   v4l2_ctrl_radio_filter, false);
if (btv->radio_ctrl_handler.error) {
result = btv->radio_ctrl_handler.error;
goto fail2;
diff --git a/drivers/media/pci/cx23885/cx23885-417.c 
b/drivers/media/pci/cx23885/cx23885-417.c
index a71f3c7569ce..762823871c78 100644
--- a/drivers/media/pci/cx23885/cx23885-417.c
+++ b/drivers/media/pci/cx23885/cx23885-417.c
@@ -1527,7 +1527,7 @@ int cx23885_417_register(struct cx23885_dev *dev)
dev->cxhdl.priv = dev;
dev->cxhdl.func = cx23885_api_func;
cx2341x_handler_set_50hz(>cxhdl, tsport->height == 576);
-   v4l2_ctrl_add_handler(>ctrl_handler, >cxhdl.hdl, NULL);
+   v4l2_ctrl_add_handler(>ctrl_handler, >cxhdl.hdl, NULL, false);
 
/* Allocate and initialize V4L video device */
dev->v4l_device = cx23885_video_dev_alloc(tsport,
diff --git a/drivers/media/pci/cx88/cx88-blackbird.c 
b/drivers/media/pci/cx88/cx88-blackbird.c
index 0e0952e60795..39f69d89a663 100644
--- a/drivers/media/pci/cx88/cx88-blackbird.c
+++ b/drivers/media/pci/cx88/cx88-blackbird.c
@@ -1183,7 +1183,7 @@ static int cx8802_blackbird_probe(struct cx8802_driver 
*drv)
err = cx2341x_handler_init(>cxhdl, 36);
if (err)
goto fail_core;
-   v4l2_ctrl_add_handler(>cxhdl.hdl, >video_hdl, NULL);
+   v4l2_ctrl_add_handler(>cxhdl.hdl, >video_hdl, N

[RFCv4 00/21] Request API

[Alexandre Courbot]

 an example of a global request entity.
* Improved documentation.

Issues remaining with this version:
* Vivid support exposed what seems to be a limitation in the current controls 
cloning implementation. See patch 8 (which works the issue around) for details.
* Media controller implementation of requests is still incomplete. I plan to 
complete and exert it on an actual camera driver sometime soon.


Alexandre Courbot (14):
  media: add request API core and UAPI
  [WAR] v4l2-ctrls: do not clone non-standard controls
  v4l2: add request API support
  media: v4l2_fh: add request entity field
  media: videobuf2: add support for requests
  media: videobuf2-v4l2: support for requests
  videodev2.h: add request_fd field to v4l2_ext_controls
  v4l2-ctrls: support requests in EXT_CTRLS ioctls
  v4l2: video_device: support for creating requests
  media: mem2mem: support for requests
  Documentation: v4l: document request API
  media: vim2m: add request support
  media: vivid: add request support for the video capture device
  [WIP] media: media-device: support for creating requests

Hans Verkuil (7):
  v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev
  v4l2-ctrls: prepare internal structs for request API
  v4l2-ctrls: add core request API
  v4l2-ctrls: use ref in helper instead of ctrl
  v4l2-ctrls: support g/s_ext_ctrls for requests
  v4l2-ctrls: add v4l2_ctrl_request_setup
  videodev2.h: Add request_fd field to v4l2_buffer

 Documentation/ioctl/ioctl-number.txt  |   1 +
 Documentation/media/uapi/v4l/buffer.rst   |   9 +-
 Documentation/media/uapi/v4l/common.rst   |   1 +
 Documentation/media/uapi/v4l/request-api.rst  | 199 ++
 Documentation/media/uapi/v4l/user-func.rst|   1 +
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst |  16 +-
 .../media/uapi/v4l/vidioc-new-request.rst |  64 
 Documentation/media/uapi/v4l/vidioc-qbuf.rst  |   7 +
 drivers/media/Kconfig |   3 +
 drivers/media/Makefile|   6 +
 .../media/common/videobuf2/videobuf2-core.c   |   3 +
 .../media/common/videobuf2/videobuf2-v4l2.c   | 131 ++-
 drivers/media/dvb-frontends/rtl2832_sdr.c |   5 +-
 drivers/media/media-device.c  |  11 +
 drivers/media/media-request.c | 341 +
 drivers/media/pci/bt8xx/bttv-driver.c |   2 +-
 drivers/media/pci/cx23885/cx23885-417.c   |   2 +-
 drivers/media/pci/cx88/cx88-blackbird.c   |   2 +-
 drivers/media/pci/cx88/cx88-video.c   |   2 +-
 drivers/media/pci/saa7134/saa7134-empress.c   |   4 +-
 drivers/media/pci/saa7134/saa7134-video.c |   2 +-
 drivers/media/platform/Kconfig|   1 +
 .../media/platform/exynos4-is/fimc-capture.c  |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c|   2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c   |   3 +-
 drivers/media/platform/rcar_drif.c|   2 +-
 .../media/platform/soc_camera/soc_camera.c|   3 +-
 drivers/media/platform/vim2m.c|  75 
 drivers/media/platform/vivid/Kconfig  |   1 +
 drivers/media/platform/vivid/vivid-core.c |  63 +++-
 drivers/media/platform/vivid/vivid-core.h |   3 +
 drivers/media/platform/vivid/vivid-ctrls.c|  46 +--
 .../media/platform/vivid/vivid-kthread-cap.c  |  17 +
 drivers/media/usb/cpia2/cpia2_v4l.c   |   2 +-
 drivers/media/usb/cx231xx/cx231xx-417.c   |   2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c |   4 +-
 drivers/media/usb/msi2500/msi2500.c   |   2 +-
 drivers/media/usb/tm6000/tm6000-video.c   |   2 +-
 drivers/media/v4l2-core/Makefile  |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c |   9 +-
 drivers/media/v4l2-core/v4l2-ctrls.c  | 341 +++--
 drivers/media/v4l2-core/v4l2-dev.c|   2 +
 drivers/media/v4l2-core/v4l2-device.c |   3 +-
 drivers/media/v4l2-core/v4l2-ioctl.c  |  37 +-
 drivers/media/v4l2-core/v4l2-mem2mem.c|   3 +-
 drivers/media/v4l2-core/v4l2-request.c| 178 +
 drivers/media/v4l2-core/v4l2-subdev.c |   2 +-
 drivers/staging/media/imx/imx-media-dev.c |   2 +-
 drivers/staging/media/imx/imx-media-fim.c |   2 +-
 include/media/mc-request.h|  33 ++
 include/media/media-device.h  |   1 +
 include/media/media-entity.h  |   5 +
 include/media/media-request.h | 349 ++
 include/media/v4l2-ctrls.h|  20 +-
 include/media/v4l2-dev.h  |   2 +
 include/media/v4l2-fh.h   |   3 +
 include/media/v4l2-request.h  | 159 
 include/media/videobuf2-core.h|   4 +
 include/media/videobuf2-v4l2.h|  59 +++
 include/uapi/linux/media-request.h|  37 ++
 include/uapi/linux/media.h|   2 +
 include/uapi/linux/videodev2.h|   9 +-
 62 files changed

[RFCv4 03/21] v4l2-ctrls: prepare internal structs for request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a refcount and is_request bool to struct v4l2_ctrl_handler:
this is used to refcount a handler that represents a request.

Add a p_req field to struct v4l2_ctrl_ref that will store the
request value.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 1 +
 include/media/v4l2-ctrls.h   | 4 
 2 files changed, 5 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 5c705f5dde9b..eac70598635d 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1761,6 +1761,7 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler 
*hdl,
  sizeof(hdl->buckets[0]),
  GFP_KERNEL | __GFP_ZERO);
hdl->error = hdl->buckets ? 0 : -ENOMEM;
+   hdl->is_request = false;
return hdl->error;
 }
 EXPORT_SYMBOL(v4l2_ctrl_handler_init_class);
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 96f5c63c12d0..bcabbf8a44b5 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -18,6 +18,7 @@
 #define _V4L2_CTRLS_H
 
 #include 
+#include 
 #include 
 #include 
 
@@ -257,6 +258,7 @@ struct v4l2_ctrl_ref {
struct v4l2_ctrl_ref *next;
struct v4l2_ctrl *ctrl;
struct v4l2_ctrl_helper *helper;
+   union v4l2_ctrl_ptr p_req;
bool from_other_dev;
 };
 
@@ -292,7 +294,9 @@ struct v4l2_ctrl_handler {
v4l2_ctrl_notify_fnc notify;
void *notify_priv;
u16 nr_of_buckets;
+   bool is_request;
int error;
+   struct kref ref;
 };
 
 /**
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 04/21] v4l2-ctrls: add core request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add the four core request functions:

v4l2_ctrl_request_init() initializes a new (empty) request.
v4l2_ctrl_request_clone() resets a request based on another request
(or clears it if that request is NULL).
v4l2_ctrl_request_get(): increase refcount
v4l2_ctrl_request_put(): decrease refcount and delete if it reaches 0.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
[acour...@chromium.org: turn v4l2_ctrl_request_alloc into init function]
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 106 ++-
 include/media/v4l2-ctrls.h   |   7 ++
 2 files changed, 110 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index eac70598635d..784879816c24 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1878,6 +1878,7 @@ EXPORT_SYMBOL(v4l2_ctrl_find);
 /* Allocate a new v4l2_ctrl_ref and hook it into the handler. */
 static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
   struct v4l2_ctrl *ctrl,
+  struct v4l2_ctrl_ref **ctrl_ref,
   bool from_other_dev)
 {
struct v4l2_ctrl_ref *ref;
@@ -1885,6 +1886,10 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
u32 id = ctrl->id;
u32 class_ctrl = V4L2_CTRL_ID2WHICH(id) | 1;
int bucket = id % hdl->nr_of_buckets;   /* which bucket to use */
+   unsigned int sz_extra = 0;
+
+   if (ctrl_ref)
+   *ctrl_ref = NULL;
 
/*
 * Automatically add the control class if it is not yet present and
@@ -1898,11 +1903,16 @@ static int handler_new_ref(struct v4l2_ctrl_handler 
*hdl,
if (hdl->error)
return hdl->error;
 
-   new_ref = kzalloc(sizeof(*new_ref), GFP_KERNEL);
+   if (hdl->is_request)
+   sz_extra = ctrl->elems * ctrl->elem_size;
+   new_ref = kzalloc(sizeof(*new_ref) + sz_extra, GFP_KERNEL);
if (!new_ref)
return handler_set_err(hdl, -ENOMEM);
new_ref->ctrl = ctrl;
new_ref->from_other_dev = from_other_dev;
+   if (sz_extra)
+   new_ref->p_req.p = _ref[1];
+
if (ctrl->handler == hdl) {
/* By default each control starts in a cluster of its own.
   new_ref->ctrl is basically a cluster array with one
@@ -1942,6 +1952,8 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
/* Insert the control node in the hash */
new_ref->next = hdl->buckets[bucket];
hdl->buckets[bucket] = new_ref;
+   if (ctrl_ref)
+   *ctrl_ref = new_ref;
 
 unlock:
mutex_unlock(hdl->lock);
@@ -2083,7 +2095,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct 
v4l2_ctrl_handler *hdl,
ctrl->type_ops->init(ctrl, idx, ctrl->p_new);
}
 
-   if (handler_new_ref(hdl, ctrl, false)) {
+   if (handler_new_ref(hdl, ctrl, NULL, false)) {
kvfree(ctrl);
return NULL;
}
@@ -2276,7 +2288,7 @@ int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
/* Filter any unwanted controls */
if (filter && !filter(ctrl))
continue;
-   ret = handler_new_ref(hdl, ctrl, from_other_dev);
+   ret = handler_new_ref(hdl, ctrl, NULL, from_other_dev);
if (ret)
break;
}
@@ -2685,6 +2697,94 @@ int v4l2_querymenu(struct v4l2_ctrl_handler *hdl, struct 
v4l2_querymenu *qm)
 }
 EXPORT_SYMBOL(v4l2_querymenu);
 
+int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl)
+{
+   int err;
+
+   err = v4l2_ctrl_handler_init(hdl, 0);
+   if (err)
+   return err;
+   hdl->is_request = true;
+   kref_init(>ref);
+
+   return 0;
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_init);
+
+int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
+   const struct v4l2_ctrl_handler *from,
+   bool (*filter)(const struct v4l2_ctrl *ctrl))
+{
+   struct v4l2_ctrl_ref *ref;
+   int err;
+
+   if (WARN_ON(!hdl || hdl == from))
+   return -EINVAL;
+
+   if (hdl->error)
+   return hdl->error;
+
+   WARN_ON(hdl->lock != >_lock);
+   v4l2_ctrl_handler_free(hdl);
+   err = v4l2_ctrl_handler_init(hdl, (from->nr_of_buckets - 1) * 8);
+   hdl->is_request = true;
+   if (err)
+   return err;
+   if (!from)
+   return 0;
+
+   mutex_lock(from->lock);
+   list_for_each_entry(ref, >ctrl_refs, node) {
+   struct v4l2_ctrl *ctrl = ref->ctrl;
+   struct v4l2_ctrl_ref *new_ref;
+
+   /*

[RFCv4 06/21] v4l2-ctrls: support g/s_ext_ctrls for requests

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

The v4l2_g/s_ext_ctrls functions now support control handlers that
represent requests.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 37 +---
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index b3be022b219f..00c4488ca1da 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1528,6 +1528,13 @@ static int new_to_user(struct v4l2_ext_control *c,
return ptr_to_user(c, ctrl, ctrl->p_new);
 }
 
+/* Helper function: copy the request value back to the caller */
+static int req_to_user(struct v4l2_ext_control *c,
+  struct v4l2_ctrl_ref *ref)
+{
+   return ptr_to_user(c, ref->ctrl, ref->p_req);
+}
+
 /* Helper function: copy the initial control value back to the caller */
 static int def_to_user(struct v4l2_ext_control *c, struct v4l2_ctrl *ctrl)
 {
@@ -1647,6 +1654,14 @@ static void cur_to_new(struct v4l2_ctrl *ctrl)
ptr_to_ptr(ctrl, ctrl->p_cur, ctrl->p_new);
 }
 
+/* Copy the new value to the request value */
+static void new_to_req(struct v4l2_ctrl_ref *ref)
+{
+   if (!ref)
+   return;
+   ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -2971,7 +2986,8 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
struct v4l2_ctrl *ctrl);
struct v4l2_ctrl *master;
 
-   ctrl_to_user = def_value ? def_to_user : cur_to_user;
+   ctrl_to_user = def_value ? def_to_user :
+  (hdl->is_request ? NULL : cur_to_user);
 
if (helpers[i].mref == NULL)
continue;
@@ -2997,8 +3013,12 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
u32 idx = i;
 
do {
-   ret = ctrl_to_user(cs->controls + idx,
-  helpers[idx].ref->ctrl);
+   if (ctrl_to_user)
+   ret = ctrl_to_user(cs->controls + idx,
+   helpers[idx].ref->ctrl);
+   else
+   ret = req_to_user(cs->controls + idx,
+   helpers[idx].ref);
idx = helpers[idx].next;
} while (!ret && idx);
}
@@ -3271,7 +3291,16 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
} while (!ret && idx);
 
if (!ret)
-   ret = try_or_set_cluster(fh, master, set, 0);
+   ret = try_or_set_cluster(fh, master,
+!hdl->is_request && set, 0);
+   if (!ret && hdl->is_request && set) {
+   for (j = 0; j < master->ncontrols; j++) {
+   struct v4l2_ctrl_ref *ref =
+   find_ref(hdl, master->cluster[j]->id);
+
+   new_to_req(ref);
+   }
+   }
 
/* Copy the new values back to userspace. */
if (!ret) {
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 07/21] v4l2-ctrls: add v4l2_ctrl_request_setup

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a helper function that can set controls from a request.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 71 
 include/media/v4l2-ctrls.h   |  2 +
 2 files changed, 73 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 00c4488ca1da..166647817efb 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1662,6 +1662,14 @@ static void new_to_req(struct v4l2_ctrl_ref *ref)
ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
 }
 
+/* Copy the request value to the new value */
+static void req_to_new(struct v4l2_ctrl_ref *ref)
+{
+   if (!ref)
+   return;
+   ptr_to_ptr(ref->ctrl, ref->p_req, ref->ctrl->p_new);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -3427,6 +3435,69 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, 
const char *s)
 }
 EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
 
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl)
+{
+   struct v4l2_ctrl_ref *ref;
+
+   if (!hdl)
+   return;
+
+   mutex_lock(hdl->lock);
+
+   list_for_each_entry(ref, >ctrl_refs, node)
+   ref->done = false;
+
+   list_for_each_entry(ref, >ctrl_refs, node) {
+   struct v4l2_ctrl *ctrl = ref->ctrl;
+   struct v4l2_ctrl *master = ctrl->cluster[0];
+   int i;
+
+   /* Skip if this control was already handled by a cluster. */
+   /* Skip button controls and read-only controls. */
+   if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
+   (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
+   continue;
+
+   v4l2_ctrl_lock(master);
+   for (i = 0; i < master->ncontrols; i++) {
+   if (master->cluster[i]) {
+   struct v4l2_ctrl_ref *r =
+   find_ref(hdl, master->cluster[i]->id);
+
+   req_to_new(r);
+   master->cluster[i]->is_new = 1;
+   r->done = true;
+   }
+   }
+   /*
+* For volatile autoclusters that are currently in auto mode
+* we need to discover if it will be set to manual mode.
+* If so, then we have to copy the current volatile values
+* first since those will become the new manual values (which
+* may be overwritten by explicit new values from this set
+* of controls).
+*/
+   if (master->is_auto && master->has_volatiles &&
+   !is_cur_manual(master)) {
+   s32 new_auto_val = *master->p_new.p_s32;
+
+   /*
+* If the new value == the manual value, then copy
+* the current volatile values.
+*/
+   if (new_auto_val == master->manual_mode_value)
+   update_from_auto_cluster(master);
+   }
+
+   try_or_set_cluster(NULL, master, true, 0);
+
+   v4l2_ctrl_unlock(master);
+   }
+
+   mutex_unlock(hdl->lock);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_setup);
+
 void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, 
void *priv)
 {
if (ctrl == NULL)
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index c51e1cacc09d..3a10fb3419e3 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -259,6 +259,7 @@ struct v4l2_ctrl_ref {
struct v4l2_ctrl *ctrl;
struct v4l2_ctrl_helper *helper;
union v4l2_ctrl_ptr p_req;
+   bool done;
bool from_other_dev;
 };
 
@@ -1056,6 +1057,7 @@ int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl);
 int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
const struct v4l2_ctrl_handler *from,
bool (*filter)(const struct v4l2_ctrl *ctrl));
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_get(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_put(struct v4l2_ctrl_handler *hdl);
 
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 05/21] v4l2-ctrls: use ref in helper instead of ctrl

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

The next patch needs the reference to a control instead of the
control itself, so change struct v4l2_ctrl_helper accordingly.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 784879816c24..b3be022b219f 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -37,8 +37,8 @@
 struct v4l2_ctrl_helper {
/* Pointer to the control reference of the master control */
struct v4l2_ctrl_ref *mref;
-   /* The control corresponding to the v4l2_ext_control ID field. */
-   struct v4l2_ctrl *ctrl;
+   /* The control ref corresponding to the v4l2_ext_control ID field. */
+   struct v4l2_ctrl_ref *ref;
/* v4l2_ext_control index of the next control belonging to the
   same cluster, or 0 if there isn't any. */
u32 next;
@@ -2856,6 +2856,7 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler 
*hdl,
ref = find_ref_lock(hdl, id);
if (ref == NULL)
return -EINVAL;
+   h->ref = ref;
ctrl = ref->ctrl;
if (ctrl->flags & V4L2_CTRL_FLAG_DISABLED)
return -EINVAL;
@@ -2878,7 +2879,6 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler 
*hdl,
}
/* Store the ref to the master control of the cluster */
h->mref = ref;
-   h->ctrl = ctrl;
/* Initially set next to 0, meaning that there is no other
   control in this helper array belonging to the same
   cluster */
@@ -2963,7 +2963,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
cs->error_idx = cs->count;
 
for (i = 0; !ret && i < cs->count; i++)
-   if (helpers[i].ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
+   if (helpers[i].ref->ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
ret = -EACCES;
 
for (i = 0; !ret && i < cs->count; i++) {
@@ -2998,7 +2998,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
 
do {
ret = ctrl_to_user(cs->controls + idx,
-  helpers[idx].ctrl);
+  helpers[idx].ref->ctrl);
idx = helpers[idx].next;
} while (!ret && idx);
}
@@ -3137,7 +3137,7 @@ static int validate_ctrls(struct v4l2_ext_controls *cs,
 
cs->error_idx = cs->count;
for (i = 0; i < cs->count; i++) {
-   struct v4l2_ctrl *ctrl = helpers[i].ctrl;
+   struct v4l2_ctrl *ctrl = helpers[i].ref->ctrl;
union v4l2_ctrl_ptr p_new;
 
cs->error_idx = i;
@@ -3249,7 +3249,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
do {
/* Check if the auto control is part of the
   list, and remember the new value. */
-   if (helpers[tmp_idx].ctrl == master)
+   if (helpers[tmp_idx].ref->ctrl == master)
new_auto_val = 
cs->controls[tmp_idx].value;
tmp_idx = helpers[tmp_idx].next;
} while (tmp_idx);
@@ -3262,7 +3262,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
/* Copy the new caller-supplied control values.
   user_to_new() sets 'is_new' to 1. */
do {
-   struct v4l2_ctrl *ctrl = helpers[idx].ctrl;
+   struct v4l2_ctrl *ctrl = helpers[idx].ref->ctrl;
 
ret = user_to_new(cs->controls + idx, ctrl);
if (!ret && ctrl->is_ptr)
@@ -3278,7 +3278,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
idx = i;
do {
ret = new_to_user(cs->controls + idx,
-   helpers[idx].ctrl);
+   helpers[idx].ref->ctrl);
idx = helpers[idx].next;
} while (!ret && idx);
}
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 08/21] [WAR] v4l2-ctrls: do not clone non-standard controls

[Alexandre Courbot]

Only standard controls can be successfully cloned: handler_new_ref, used
by v4l2_ctrl_request_clone(), forcibly calls v4l2_ctrl_new_std() which
fails to find custom controls names, and we eventually hit the condition
that name == NULL in v4l2_ctrl_new().

This prevents us from using non-standard controls with requests, but
that is enough for testing purposes.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 166647817efb..7a81aa5959c3 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -2772,6 +2772,11 @@ int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler 
*hdl,
if (filter && !filter(ctrl))
continue;
err = handler_new_ref(hdl, ctrl, _ref, false);
+   if (err) {
+   printk("%s: handler_new_ref on control %x (%s) returned 
%d\n", __func__, ctrl->id, ctrl->name, err);
+   err = 0;
+   continue;
+   }
if (err)
break;
if (from->is_request)
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 11/21] media: v4l2_fh: add request entity field

[Alexandre Courbot]

Allow drivers to assign a request entity to v4l2_fh. This will be useful
for request-aware ioctls to find out which request entity to use.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 include/media/v4l2-fh.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
index ea73fef8bdc0..f54cb319dd64 100644
--- a/include/media/v4l2-fh.h
+++ b/include/media/v4l2-fh.h
@@ -28,6 +28,7 @@
 
 struct video_device;
 struct v4l2_ctrl_handler;
+struct media_request_entity;
 
 /**
  * struct v4l2_fh - Describes a V4L2 file handler
@@ -43,6 +44,7 @@ struct v4l2_ctrl_handler;
  * @navailable: number of available events at @available list
  * @sequence: event sequence number
  * @m2m_ctx: pointer to  v4l2_m2m_ctx
+ * @entity: the request entity this fh operates on behalf of
  */
 struct v4l2_fh {
struct list_headlist;
@@ -60,6 +62,7 @@ struct v4l2_fh {
 #if IS_ENABLED(CONFIG_V4L2_MEM2MEM_DEV)
struct v4l2_m2m_ctx *m2m_ctx;
 #endif
+   struct media_request_entity *entity;
 };
 
 /**
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 09/21] v4l2: add request API support

[Alexandre Courbot]

Add a v4l2 request entity data structure that takes care of storing the
request-related state of a V4L2 device ; in this case, its controls.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/Makefile   |   1 +
 drivers/media/v4l2-core/v4l2-request.c | 178 +
 include/media/v4l2-request.h   | 159 ++
 3 files changed, 338 insertions(+)
 create mode 100644 drivers/media/v4l2-core/v4l2-request.c
 create mode 100644 include/media/v4l2-request.h

diff --git a/drivers/media/v4l2-core/Makefile b/drivers/media/v4l2-core/Makefile
index 80de2cb9c476..13d0477535bd 100644
--- a/drivers/media/v4l2-core/Makefile
+++ b/drivers/media/v4l2-core/Makefile
@@ -16,6 +16,7 @@ ifeq ($(CONFIG_TRACEPOINTS),y)
   videodev-objs += vb2-trace.o v4l2-trace.o
 endif
 videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o
+videodev-$(CONFIG_MEDIA_REQUEST_API) += v4l2-request.o
 
 obj-$(CONFIG_VIDEO_V4L2) += videodev.o
 obj-$(CONFIG_VIDEO_V4L2) += v4l2-common.o
diff --git a/drivers/media/v4l2-core/v4l2-request.c 
b/drivers/media/v4l2-core/v4l2-request.c
new file mode 100644
index ..e8ad10e2f525
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-request.c
@@ -0,0 +1,178 @@
+/*
+ * Media requests support for V4L2
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include 
+
+#include 
+#include 
+#include 
+
+void v4l2_request_entity_init(struct v4l2_request_entity *entity,
+ const struct media_request_entity_ops *ops,
+ struct video_device *vdev)
+{
+   media_request_entity_init(>base, 
MEDIA_REQUEST_ENTITY_TYPE_V4L2, ops);
+   entity->vdev = vdev;
+}
+EXPORT_SYMBOL_GPL(v4l2_request_entity_init);
+
+struct media_request_entity_data *
+v4l2_request_entity_data_alloc(struct media_request *req,
+  struct v4l2_ctrl_handler *hdl)
+{
+   struct v4l2_request_entity_data *data;
+   int ret;
+
+   data = kzalloc(sizeof(*data), GFP_KERNEL);
+   if (!data)
+   return ERR_PTR(-ENOMEM);
+
+   ret = v4l2_ctrl_request_init(>ctrls);
+   if (ret) {
+   kfree(data);
+   return ERR_PTR(ret);
+   }
+   ret = v4l2_ctrl_request_clone(>ctrls, hdl, NULL);
+   if (ret) {
+   kfree(data);
+   return ERR_PTR(ret);
+   }
+
+   INIT_LIST_HEAD(>queued_buffers);
+
+   return >base;
+}
+EXPORT_SYMBOL_GPL(v4l2_request_entity_data_alloc);
+
+void v4l2_request_entity_data_free(struct media_request_entity_data *_data)
+{
+   struct v4l2_request_entity_data *data;
+   struct v4l2_vb2_request_buffer *qb, *n;
+
+   data = to_v4l2_entity_data(_data);
+
+   list_for_each_entry_safe(qb, n, >queued_buffers, node) {
+   struct vb2_buffer *buf;
+   dev_warn(_data->request->mgr->dev,
+"entity data freed while buffer still queued!\n");
+
+   /* give buffer back to user-space */
+   buf = qb->queue->bufs[qb->v4l2_buf.index];
+   buf->state = qb->pre_req_state;
+   buf->request = NULL;
+
+   kfree(qb);
+   }
+
+   v4l2_ctrl_handler_free(>ctrls);
+   kfree(data);
+}
+EXPORT_SYMBOL_GPL(v4l2_request_entity_data_free);
+
+
+
+
+
+static struct media_request *v4l2_request_alloc(struct media_request_mgr *mgr)
+{
+   struct media_request *req;
+
+   req = kzalloc(sizeof(*req), GFP_KERNEL);
+   if (!req)
+   return ERR_PTR(-ENOMEM);
+
+   req->mgr = mgr;
+   req->state = MEDIA_REQUEST_STATE_IDLE;
+   INIT_LIST_HEAD(>data);
+   init_waitqueue_head(>complete_wait);
+   mutex_init(>lock);
+
+   mutex_lock(>mutex);
+   list_add_tail(>list, >requests);
+   mutex_unlock(>mutex);
+
+   return req;
+}
+
+static void v4l2_request_free(struct media_request *req)
+{
+   struct media_request_mgr *mgr = req->mgr;
+   struct media_request_entity_data *data, *next;
+
+   mutex_lock(>mutex);
+   list_del(>list);
+   mutex_unlock(>mutex);
+
+   list_for_each_entry_safe(data, next, >data, list) {
+   list_del(>list);
+   data->entity->ops->data_free(data);
+   }
+
+   kfree(req);
+}
+
+static bool v4l2_entity_valid(const struct media_request *req,
+   

[RFCv4 10/21] videodev2.h: Add request_fd field to v4l2_buffer

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

When queuing buffers allow for passing the request that should
be associated with this buffer.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
[acour...@chromium.org: make request ID 32-bit]
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 2 +-
 drivers/media/usb/cpia2/cpia2_v4l.c | 2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   | 9 ++---
 drivers/media/v4l2-core/v4l2-ioctl.c| 4 ++--
 include/uapi/linux/videodev2.h  | 3 ++-
 5 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 886a2d8d5c6c..6d4d184aa68e 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->request_fd = 0;
b->reserved = 0;
 
if (q->is_multiplanar) {
diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
b/drivers/media/usb/cpia2/cpia2_v4l.c
index 99f106b13280..af42ce3ceb48 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct 
v4l2_buffer *buf)
buf->sequence = cam->buffers[buf->index].seq;
buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
buf->length = cam->frame_size;
-   buf->reserved2 = 0;
+   buf->request_fd = 0;
buf->reserved = 0;
memset(>timecode, 0, sizeof(buf->timecode));
 
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 5198c9eeb348..32bf47489a2e 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -386,7 +386,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __s32   request_fd;
__u32   reserved;
 };
 
@@ -486,6 +486,7 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *kp,
 {
u32 type;
u32 length;
+   s32 request_fd;
enum v4l2_memory memory;
struct v4l2_plane32 __user *uplane32;
struct v4l2_plane __user *uplane;
@@ -500,7 +501,9 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *kp,
get_user(memory, >memory) ||
put_user(memory, >memory) ||
get_user(length, >length) ||
-   put_user(length, >length))
+   put_user(length, >length) ||
+   get_user(request_fd, >request_fd) ||
+   put_user(request_fd, >request_fd))
return -EFAULT;
 
if (V4L2_TYPE_IS_OUTPUT(type))
@@ -604,7 +607,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
assign_in_user(>timestamp.tv_usec, >timestamp.tv_usec) ||
copy_in_user(>timecode, >timecode, sizeof(kp->timecode)) ||
assign_in_user(>sequence, >sequence) ||
-   assign_in_user(>reserved2, >reserved2) ||
+   assign_in_user(>request_fd, >request_fd) ||
assign_in_user(>reserved, >reserved) ||
get_user(length, >length) ||
put_user(length, >length))
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index 260288ca4f55..7bfeaf233d5a 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool 
write_only)
const struct v4l2_plane *plane;
int i;
 
-   pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, 
field=%s, sequence=%d, memory=%s",
+   pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request_fd=%u, 
flags=0x%08x, field=%s, sequence=%d, memory=%s",
p->timestamp.tv_sec / 3600,
(int)(p->timestamp.tv_sec / 60) % 60,
(int)(p->timestamp.tv_sec % 60),
(long)p->timestamp.tv_usec,
p->index,
-   prt_names(p->type, v4l2_type_names),
+   prt_names(p->type, v4l2_type_names), p->request_fd,
p->flags, prt_names(p->field, v4l2_field_names),
p->sequence, prt_names(p->memory, v4l2_memory_names));
 
diff --git a/include/uapi/linux/videode

[RFCv4 12/21] media: videobuf2: add support for requests

[Alexandre Courbot]

Make vb2 core aware of requests. Drivers can specify whether a given
queue accepts requests or not.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 3 +++
 include/media/videobuf2-core.h  | 4 
 2 files changed, 7 insertions(+)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c 
b/drivers/media/common/videobuf2/videobuf2-core.c
index debe35fc66b4..355fe7dc99d7 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -930,6 +930,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
vb->state = state;
}
atomic_dec(>owned_by_drv_count);
+
+   vb->request = NULL;
+
spin_unlock_irqrestore(>done_lock, flags);
 
trace_vb2_buf_done(q, vb);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 5b6c541e4e1b..6e9e814886e7 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -238,6 +238,7 @@ struct vb2_queue;
  * @num_planes:number of planes in the buffer
  * on an internal driver queue.
  * @timestamp: frame timestamp in ns.
+ * @request:   request the buffer belongs to, if any.
  */
 struct vb2_buffer {
struct vb2_queue*vb2_queue;
@@ -246,6 +247,7 @@ struct vb2_buffer {
unsigned intmemory;
unsigned intnum_planes;
u64 timestamp;
+   struct media_request*request;
 
/* private: internal use only
 *
@@ -446,6 +448,7 @@ struct vb2_buf_ops {
  * @quirk_poll_must_check_waiting_for_buffers: Return %EPOLLERR at poll when 
QBUF
  *  has not been called. This is a vb1 idiom that has been adopted
  *  also by vb2.
+ * @allow_requests:whether requests are supported on this queue.
  * @lock:  pointer to a mutex that protects the  vb2_queue. The
  * driver can set this to a mutex to let the v4l2 core serialize
  * the queuing ioctls. If the driver wants to handle locking
@@ -513,6 +516,7 @@ struct vb2_queue {
unsignedfileio_write_immediately:1;
unsignedallow_zero_bytesused:1;
unsigned   quirk_poll_must_check_waiting_for_buffers:1;
+   unsignedallow_requests:1;
 
struct mutex*lock;
void*owner;
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 17/21] media: mem2mem: support for requests

[Alexandre Courbot]

Add generic support for requests to the mem2mem framework. This just
requires calling vb2_qbuf_request() instead of vb2_qbuf() in
v4l2_m2m_qbuf().

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c 
b/drivers/media/v4l2-core/v4l2-mem2mem.c
index c4f963d96a79..d9ae428651c4 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -388,11 +388,12 @@ EXPORT_SYMBOL_GPL(v4l2_m2m_querybuf);
 int v4l2_m2m_qbuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
  struct v4l2_buffer *buf)
 {
+   struct v4l2_fh *fh = file->private_data;
struct vb2_queue *vq;
int ret;
 
vq = v4l2_m2m_get_vq(m2m_ctx, buf->type);
-   ret = vb2_qbuf(vq, buf);
+   ret = vb2_qbuf_request(vq, buf, fh->entity);
if (!ret)
v4l2_m2m_try_schedule(m2m_ctx);
 
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 13/21] media: videobuf2-v4l2: support for requests

[Alexandre Courbot]

Add a new vb2_qbuf_request() (a request-aware version of vb2_qbuf())
that request-aware drivers can call to queue a buffer into a request
instead of directly into the vb2 queue if relevent.

This function expects that drivers invoking it are using instances of
v4l2_request_entity and v4l2_request_entity_data to describe their
entity and entity data respectively.

Also add the vb2_request_submit() helper function which drivers can
invoke in order to queue all the buffers previously queued into a
request into the regular vb2 queue.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 .../media/common/videobuf2/videobuf2-v4l2.c   | 129 +-
 include/media/videobuf2-v4l2.h|  59 
 2 files changed, 187 insertions(+), 1 deletion(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 6d4d184aa68e..0627c3339572 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -28,8 +28,11 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 
 #include 
+#include 
 
 static int debug;
 module_param(debug, int, 0644);
@@ -569,11 +572,131 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
return -EBUSY;
}
 
+   /* drivers supporting requests must call vb2_qbuf_request instead */
+   if (b->request_fd > 0) {
+   dprintk(1, "invalid call to vb2_qbuf with request_fd set\n");
+   return -EINVAL;
+   }
+
ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
return ret ? ret : vb2_core_qbuf(q, b->index, b);
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
+#if IS_ENABLED(CONFIG_MEDIA_REQUEST_API)
+int vb2_qbuf_request(struct vb2_queue *q, struct v4l2_buffer *b,
+struct media_request_entity *entity)
+{
+   struct v4l2_request_entity_data *data;
+   struct v4l2_vb2_request_buffer *qb;
+   struct media_request *req;
+   struct vb2_buffer *vb;
+   int ret = 0;
+
+   if (b->request_fd <= 0)
+   return vb2_qbuf(q, b);
+
+   if (!q->allow_requests)
+   return -EINVAL;
+
+   req = media_request_get_from_fd(b->request_fd);
+   if (!req)
+   return -EINVAL;
+
+   data = to_v4l2_entity_data(media_request_get_entity_data(req, entity));
+   if (IS_ERR(data)) {
+   ret = PTR_ERR(data);
+   goto out;
+   }
+
+   mutex_lock(>lock);
+
+   if (req->state != MEDIA_REQUEST_STATE_IDLE) {
+   ret = -EINVAL;
+   goto out;
+   }
+
+   ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
+   if (ret)
+   goto out;
+
+   vb = q->bufs[b->index];
+   switch (vb->state) {
+   case VB2_BUF_STATE_DEQUEUED:
+   break;
+   case VB2_BUF_STATE_PREPARED:
+   break;
+   case VB2_BUF_STATE_PREPARING:
+   dprintk(1, "buffer still being prepared\n");
+   ret = -EINVAL;
+   goto out;
+   default:
+   dprintk(1, "invalid buffer state %d\n", vb->state);
+   ret = -EINVAL;
+   goto out;
+   }
+
+   /* do we already have a buffer for this request in the queue? */
+   list_for_each_entry(qb, >queued_buffers, node) {
+   if (qb->queue == q) {
+   ret = -EBUSY;
+   goto out;
+   }
+   }
+
+   qb = kzalloc(sizeof(*qb), GFP_KERNEL);
+   if (!qb) {
+   ret = -ENOMEM;
+   goto out;
+   }
+
+   /*
+* TODO should be prepare the buffer here if needed, to report errors
+* early?
+*/
+   qb->pre_req_state = vb->state;
+   qb->queue = q;
+   memcpy(>v4l2_buf, b, sizeof(*b));
+   vb->request = req;
+   vb->state = VB2_BUF_STATE_QUEUED;
+   list_add_tail(>node, >queued_buffers);
+
+out:
+   mutex_unlock(>lock);
+   media_request_put(req);
+
+   return ret;
+}
+EXPORT_SYMBOL_GPL(vb2_qbuf_request);
+
+int vb2_request_submit(struct v4l2_request_entity_data *data)
+{
+   struct v4l2_vb2_request_buffer *qb, *n;
+
+   /* v4l2 requests require at least one buffer to reach the device */
+   if (list_empty(>queued_buffers)) {
+   return -EINVAL;
+   }
+
+   list_for_each_entry_safe(qb, n, >queued_buffers, node) {
+   struct vb2_queue *q = qb->queue;
+   struct vb2_buffer *vb = q->bufs[qb->v4l2_buf.index];
+   int ret;
+
+   list_del(>node);
+   vb->state = qb->pre_req_state;
+   ret = vb2_core_qbuf(q, vb->index, >v4l2_buf);
+   kfree(qb);
+   if (ret)
+   return ret;
+   }
+
+   r

[RFCv4 14/21] videodev2.h: add request_fd field to v4l2_ext_controls

[Alexandre Courbot]

Allow to specify a request to be used with the S_EXT_CTRLS and
G_EXT_CTRLS operations.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 2 +-
 include/uapi/linux/videodev2.h   | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index 7bfeaf233d5a..2f40ac0cdf6e 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -870,7 +870,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int 
allow_priv)
__u32 i;
 
/* zero the reserved fields */
-   c->reserved[0] = c->reserved[1] = 0;
+   c->reserved[0] = 0;
for (i = 0; i < c->count; i++)
c->controls[i].reserved2[0] = 0;
 
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 4fd46ae8fad5..91cfe0cbd5c5 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1592,7 +1592,8 @@ struct v4l2_ext_controls {
};
__u32 count;
__u32 error_idx;
-   __u32 reserved[2];
+   __s32 request_fd;
+   __u32 reserved[1];
struct v4l2_ext_control *controls;
 };
 
-- 
2.16.1.291.g4437f3f132-goog

[RFCv4 15/21] v4l2-ctrls: support requests in EXT_CTRLS ioctls

[Alexandre Courbot]

Read and use the request_fd field of struct v4l2_ext_controls to apply
VIDIOC_G_EXT_CTRLS or VIDIOC_S_EXT_CTRLS to a request when asked by
userspace.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/omap3isp/ispvideo.c |  2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c   | 98 +-
 drivers/media/v4l2-core/v4l2-ioctl.c   |  6 +-
 drivers/media/v4l2-core/v4l2-subdev.c  |  2 +-
 include/media/v4l2-ctrls.h |  3 +-
 5 files changed, 102 insertions(+), 9 deletions(-)

diff --git a/drivers/media/platform/omap3isp/ispvideo.c 
b/drivers/media/platform/omap3isp/ispvideo.c
index a751c89a3ea8..3976cd9ac2f2 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1028,7 +1028,7 @@ static int isp_video_check_external_subdevs(struct 
isp_video *video,
ctrls.count = 1;
ctrls.controls = 
 
-   ret = v4l2_g_ext_ctrls(pipe->external->ctrl_handler, );
+   ret = v4l2_g_ext_ctrls(NULL, pipe->external->ctrl_handler, );
if (ret < 0) {
dev_warn(isp->dev, "no pixel rate control in subdev %s\n",
 pipe->external->name);
diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 7a81aa5959c3..d7b1aeb32470 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -27,6 +27,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #define has_op(master, op) \
(master->ops && master->ops->op)
@@ -2959,9 +2960,44 @@ static int class_check(struct v4l2_ctrl_handler *hdl, 
u32 which)
 }
 
 
+#if IS_ENABLED(CONFIG_MEDIA_REQUEST_API)
+static struct media_request *
+get_handler_for_request(struct v4l2_fh *fh, struct v4l2_ext_controls *cs,
+   struct v4l2_ctrl_handler **hdl)
+{
+   struct media_request *req;
+   struct v4l2_request_entity_data *data;
+
+   if (!fh || !fh->entity)
+   return ERR_PTR(-EINVAL);
+
+   req = media_request_get_from_fd(cs->request_fd);
+   if (!req)
+   return ERR_PTR(-EINVAL);
+
+   data = to_v4l2_entity_data(media_request_get_entity_data(req,
+fh->entity));
+   if (IS_ERR(data)) {
+   media_request_put(req);
+   return (void *)data;
+   }
+
+   *hdl = >ctrls;
+
+   return req;
+}
+#else
+static struct media_request *
+get_handler_for_request(struct v4l2_fh *fh, struct v4l2_ext_controls *cs,
+   struct v4l2_ctrl_handler **hdl)
+{
+   return ERR_PTR(-ENOTSUPP);
+}
+#endif
 
 /* Get extended controls. Allocates the helpers array if needed. */
-int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls 
*cs)
+int __v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
+  struct v4l2_ext_controls *cs)
 {
struct v4l2_ctrl_helper helper[4];
struct v4l2_ctrl_helper *helpers = helper;
@@ -3042,6 +3078,30 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
kvfree(helpers);
return ret;
 }
+
+int v4l2_g_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
+struct v4l2_ext_controls *cs)
+{
+   struct media_request *req = NULL;
+   int ret;
+
+   if (cs->request_fd > 0) {
+   req = get_handler_for_request(fh, cs, );
+   if (IS_ERR(req))
+   return PTR_ERR(req);
+
+media_request_lock(req);
+   }
+
+   ret = __v4l2_g_ext_ctrls(hdl, cs);
+
+   if (req) {
+   media_request_unlock(req);
+   media_request_put(req);
+   }
+
+   return ret;
+}
 EXPORT_SYMBOL(v4l2_g_ext_ctrls);
 
 /* Helper function to get a single control */
@@ -3217,9 +3277,9 @@ static void update_from_auto_cluster(struct v4l2_ctrl 
*master)
 }
 
 /* Try or try-and-set controls */
-static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
-struct v4l2_ext_controls *cs,
-bool set)
+static int __try_set_ext_ctrls(struct v4l2_fh *fh,
+  struct v4l2_ctrl_handler *hdl,
+  struct v4l2_ext_controls *cs, bool set)
 {
struct v4l2_ctrl_helper helper[4];
struct v4l2_ctrl_helper *helpers = helper;
@@ -3332,6 +3392,36 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
return ret;
 }
 
+static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
+struct v4l2_ext_controls *cs, bool set)
+{
+   struct media_request *req = NULL;
+   int ret;
+
+   if (cs->request_fd > 0) {
+   req = get_handler_for_request(fh, c

[RFCv4 16/21] v4l2: video_device: support for creating requests

[Alexandre Courbot]

Add a new VIDIOC_NEW_REQUEST ioctl, which allows to instanciate requests
on devices that support the request API. Requests created that way can
only control the device they originate from, making them suitable for
simple devices, but not complex pipelines.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 Documentation/ioctl/ioctl-number.txt |  1 +
 drivers/media/v4l2-core/v4l2-dev.c   |  2 ++
 drivers/media/v4l2-core/v4l2-ioctl.c | 25 +
 include/media/v4l2-dev.h |  2 ++
 include/uapi/linux/videodev2.h   |  3 +++
 5 files changed, 33 insertions(+)

diff --git a/Documentation/ioctl/ioctl-number.txt 
b/Documentation/ioctl/ioctl-number.txt
index 6501389d55b9..afdc9ed255b0 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt
@@ -286,6 +286,7 @@ Code  Seq#(hex) Include FileComments
<mailto:o...@port.de>
 'z'10-4F   drivers/s390/crypto/zcrypt_api.hconflict!
 '|'00-7F   linux/media.h
+'|'80-9F   linux/media-request.h
 0x80   00-1F   linux/fb.h
 0x89   00-06   arch/x86/include/asm/sockios.h
 0x89   0B-DF   linux/sockios.h
diff --git a/drivers/media/v4l2-core/v4l2-dev.c 
b/drivers/media/v4l2-core/v4l2-dev.c
index 0301fe426a43..062ebee5bffc 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -559,6 +559,8 @@ static void determine_valid_ioctls(struct video_device 
*vdev)
set_bit(_IOC_NR(VIDIOC_TRY_EXT_CTRLS), valid_ioctls);
if (vdev->ctrl_handler || ops->vidioc_querymenu)
set_bit(_IOC_NR(VIDIOC_QUERYMENU), valid_ioctls);
+   if (vdev->req_mgr)
+   set_bit(_IOC_NR(VIDIOC_NEW_REQUEST), valid_ioctls);
SET_VALID_IOCTL(ops, VIDIOC_G_FREQUENCY, vidioc_g_frequency);
SET_VALID_IOCTL(ops, VIDIOC_S_FREQUENCY, vidioc_s_frequency);
SET_VALID_IOCTL(ops, VIDIOC_LOG_STATUS, vidioc_log_status);
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index ab4968ea443f..a45fe078f8ae 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -21,6 +21,7 @@
 
 #include 
 
+#include 
 #include 
 #include 
 #include 
@@ -842,6 +843,13 @@ static void v4l_print_freq_band(const void *arg, bool 
write_only)
p->rangehigh, p->modulation);
 }
 
+static void vidioc_print_new_request(const void *arg, bool write_only)
+{
+   const struct media_request_new *new = arg;
+
+   pr_cont("fd=0x%x\n", new->fd);
+}
+
 static void v4l_print_edid(const void *arg, bool write_only)
 {
const struct v4l2_edid *p = arg;
@@ -2486,6 +2494,22 @@ static int v4l_enum_freq_bands(const struct 
v4l2_ioctl_ops *ops,
return -ENOTTY;
 }
 
+static int vidioc_new_request(const struct v4l2_ioctl_ops *ops,
+ struct file *file, void *fh, void *arg)
+{
+#if IS_ENABLED(CONFIG_MEDIA_REQUEST_API)
+   struct media_request_new *new = arg;
+   struct video_device *vfd = video_devdata(file);
+
+   if (!vfd->req_mgr)
+   return -ENOTTY;
+
+   return media_request_ioctl_new(vfd->req_mgr, new);
+#else
+   return -ENOTTY;
+#endif
+}
+
 struct v4l2_ioctl_info {
unsigned int ioctl;
u32 flags;
@@ -2617,6 +2641,7 @@ static struct v4l2_ioctl_info v4l2_ioctls[] = {
IOCTL_INFO_FNC(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, 
v4l_print_freq_band, 0),
IOCTL_INFO_FNC(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, 
v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)),
IOCTL_INFO_FNC(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, 
v4l_print_query_ext_ctrl, INFO_FL_CTRL | INFO_FL_CLEAR(v4l2_query_ext_ctrl, 
id)),
+   IOCTL_INFO_FNC(VIDIOC_NEW_REQUEST, vidioc_new_request, 
vidioc_print_new_request, 0),
 };
 #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls)
 
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 53f32022fabe..e6c4e10889bc 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -209,6 +209,7 @@ struct v4l2_file_operations {
  * @entity:  media_entity
  * @intf_devnode: pointer to  media_intf_devnode
  * @pipe:  media_pipeline
+ * @req_mgr: request manager to use if this device supports creating requests
  * @fops: pointer to  v4l2_file_operations for the video device
  * @device_caps: device capabilities as used in v4l2_capabilities
  * @dev:  device for the video device
@@ -251,6 +252,7 @@ struct video_device
struct media_intf_devnode *intf_devnode;
struct media_pipeline pipe;
 #endif
+   struct media_request_mgr *req_mgr;
const struct v4l2_file_operations *fops;
 
u32 device_caps;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 91cfe0cbd5c5..35706204e81d 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/vide

[RFCv4 20/21] media: vivid: add request support for the video capture device

[Alexandre Courbot]

Allow to use requests with the video capture device.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/vivid/Kconfig  |  1 +
 drivers/media/platform/vivid/vivid-core.c | 63 ++-
 drivers/media/platform/vivid/vivid-core.h |  3 +
 .../media/platform/vivid/vivid-kthread-cap.c  | 17 +
 4 files changed, 83 insertions(+), 1 deletion(-)

diff --git a/drivers/media/platform/vivid/Kconfig 
b/drivers/media/platform/vivid/Kconfig
index 154de92dd809..a6494dabae95 100644
--- a/drivers/media/platform/vivid/Kconfig
+++ b/drivers/media/platform/vivid/Kconfig
@@ -7,6 +7,7 @@ config VIDEO_VIVID
select FB_CFB_FILLRECT
select FB_CFB_COPYAREA
select FB_CFB_IMAGEBLIT
+   select MEDIA_REQUEST_API
select VIDEOBUF2_VMALLOC
select VIDEOBUF2_DMA_CONTIG
select VIDEO_V4L2_TPG
diff --git a/drivers/media/platform/vivid/vivid-core.c 
b/drivers/media/platform/vivid/vivid-core.c
index 82ec216f2ad8..c1cf8e7ca2c9 100644
--- a/drivers/media/platform/vivid/vivid-core.c
+++ b/drivers/media/platform/vivid/vivid-core.c
@@ -23,6 +23,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "vivid-core.h"
 #include "vivid-vid-common.h"
@@ -486,6 +487,33 @@ static const struct v4l2_file_operations vivid_fops = {
.mmap   = vb2_fop_mmap,
 };
 
+static int vivid_cap_open(struct file *filp)
+{
+   struct vivid_dev *dev;
+   struct v4l2_fh *fh;
+   int ret;
+
+   ret = v4l2_fh_open(filp);
+   if (ret)
+   return ret;
+
+   dev = container_of(video_devdata(filp), struct vivid_dev, vid_cap_dev);
+   fh = filp->private_data;
+   fh->entity = >vid_cap_req_entity.base;
+
+   return ret;
+}
+
+static const struct v4l2_file_operations vivid_cap_fops = {
+   .owner  = THIS_MODULE,
+   .open   = vivid_cap_open,
+   .release= vivid_fop_release,
+   .read   = vb2_fop_read,
+   .write  = vb2_fop_write,
+   .poll   = vb2_fop_poll,
+   .unlocked_ioctl = video_ioctl2,
+   .mmap   = vb2_fop_mmap,
+};
 static const struct v4l2_file_operations vivid_radio_fops = {
.owner  = THIS_MODULE,
.open   = v4l2_fh_open,
@@ -607,6 +635,31 @@ static const struct v4l2_ioctl_ops vivid_ioctl_ops = {
.vidioc_unsubscribe_event   = v4l2_event_unsubscribe,
 };
 
+struct media_request_entity_data *
+vid_cap_entity_data_alloc(struct media_request *req,
+   struct media_request_entity *entity)
+{
+   struct vivid_dev *dev;
+
+   dev = container_of(entity, struct vivid_dev, vid_cap_req_entity.base);
+   return v4l2_request_entity_data_alloc(req, >ctrl_hdl_vid_cap);
+}
+
+static int vid_cap_request_submit(struct media_request *req,
+   struct media_request_entity_data *_data)
+{
+   struct v4l2_request_entity_data *data;
+
+   data = to_v4l2_entity_data(_data);
+   return vb2_request_submit(data);
+}
+
+static const struct media_request_entity_ops vivid_request_entity_ops = {
+   .data_alloc = vid_cap_entity_data_alloc,
+   .data_free  = v4l2_request_entity_data_free,
+   .submit = vid_cap_request_submit,
+};
+
 /* -
Initialization and module stuff
--*/
@@ -1057,6 +1110,7 @@ static int vivid_create_instance(struct platform_device 
*pdev, int inst)
q->mem_ops = vivid_mem_ops[allocator];
q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
q->min_buffers_needed = 2;
+   q->allow_requests = true;
q->lock = >mutex;
q->dev = dev->v4l2_dev.dev;
 
@@ -1158,13 +1212,19 @@ static int vivid_create_instance(struct platform_device 
*pdev, int inst)
vfd = >vid_cap_dev;
snprintf(vfd->name, sizeof(vfd->name),
 "vivid-%03d-vid-cap", inst);
-   vfd->fops = _fops;
+   vfd->fops = _cap_fops;
vfd->ioctl_ops = _ioctl_ops;
vfd->device_caps = dev->vid_cap_caps;
vfd->release = video_device_release_empty;
vfd->v4l2_dev = >v4l2_dev;
vfd->queue = >vb_vid_cap_q;
vfd->tvnorms = tvnorms_cap;
+   vfd->req_mgr = >vid_cap_req_mgr.base;
+   v4l2_request_mgr_init(>vid_cap_req_mgr, vfd,
+ _request_ops);
+   v4l2_request_entity_init(>vid_cap_req_entity,
+_request_entity_ops,
+vfd);
 
/*
 * Provi

[RFCv4 18/21] Documentation: v4l: document request API

[Alexandre Courbot]

Document the request API for V4L2 devices, and amend the documentation
of system calls influenced by it.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 Documentation/media/uapi/v4l/buffer.rst   |   9 +-
 Documentation/media/uapi/v4l/common.rst   |   1 +
 Documentation/media/uapi/v4l/request-api.rst  | 199 ++
 Documentation/media/uapi/v4l/user-func.rst|   1 +
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst |  16 +-
 .../media/uapi/v4l/vidioc-new-request.rst |  64 ++
 Documentation/media/uapi/v4l/vidioc-qbuf.rst  |   7 +
 7 files changed, 293 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst
 create mode 100644 Documentation/media/uapi/v4l/vidioc-new-request.rst

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..62caa8c9120d 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,13 @@ struct v4l2_buffer
elements in the ``planes`` array. The driver will fill in the
actual number of valid elements in that array.
 * - __u32
-  - ``reserved2``
+  - ``request_fd``
   -
-  - A place holder for future extensions. Drivers and applications
-   must set this to 0.
+  - The file descriptor of the request to queue the buffer to. If 
specified,
+   the buffer will be queued to that request. If zero, the buffer will
+   be directly queued to the queue designated by the ``type`` field. This
+   is set by the user when calling :ref:`VIDIOC_QBUF` and ignored by other
+   ioctls.
 * - __u32
   - ``reserved``
   -
diff --git a/Documentation/media/uapi/v4l/common.rst 
b/Documentation/media/uapi/v4l/common.rst
index 13f2ed3fc5a6..a4aa0059d45a 100644
--- a/Documentation/media/uapi/v4l/common.rst
+++ b/Documentation/media/uapi/v4l/common.rst
@@ -44,3 +44,4 @@ applicable to all devices.
 crop
 selection-api
 streaming-par
+request-api
diff --git a/Documentation/media/uapi/v4l/request-api.rst 
b/Documentation/media/uapi/v4l/request-api.rst
new file mode 100644
index ..0c1f2896e197
--- /dev/null
+++ b/Documentation/media/uapi/v4l/request-api.rst
@@ -0,0 +1,199 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _media-request-api:
+
+Request API
+===
+
+The Request API has been designed to allow V4L2 to deal with requirements of
+modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android Codec
+v2). One such requirement is the ability for devices belonging to the same
+pipeline to reconfigure and collaborate closely on a per-frame basis. Another 
is
+efficient support of stateless codecs, which need per-frame controls to be set
+asynchronously in order to be efficiently used.
+
+Supporting these features without the Request API is possible but terribly
+inefficient: user-space would have to flush all activity on the media pipeline,
+reconfigure it for the next frame, queue the buffers to be processed with that
+configuration, and wait until they are all available for dequeing before
+considering the next frame. This defeats the purpose of having buffer queues
+since in practice only one buffer would be queued at a time.
+
+The Request API allows a specific configuration of the pipeline (media
+controller topology + controls for each device) to be associated with specific
+buffers. The parameters are applied by each participating device as buffers
+associated to a request flow in. This allows user-space to schedule several
+tasks ("requests") with different parameters in advance, knowing that the
+parameters will be applied when needed to get the expected result. Controls
+values at the time of request completion are also available for reading.
+
+Usage
+=
+
+The Request API is used on top of standard media controller and V4L2 calls,
+which are augmented with an extra ``request_fd`` parameter. Request themselves
+are allocated from either a supporting V4L2 device node, or a supporting media
+controller node. The origin of requests determine their scope: requests
+allocated from a V4L2 device node can only act on that device, whereas requests
+allocated from a media controller node can control the whole pipeline of the
+controller.
+
+Request Allocation
+--
+
+User-space allocates requests using the ``VIDIOC_NEW_REQUEST`` (for V4L2 device
+requests) or ``MEDIA_IOC_NEW_REQUEST`` (for media controller requests) on an
+opened device or media node. This returns a file descriptor representing the
+request. Typically, several such requests will be allocated.
+
+Request Preparation
+---
+
+Standard V4L2 ioctls can then receive a request file descriptor to express the
+fact that the ioctl is part of said request, and is not to be applied
+immediately. V4L2 ioctls supporting this are :c:func:`VIDIOC_S_EXT_CTRLS` and
+:c:func:`VIDIOC_QBUF`. Controls set with a r

[RFCv4 21/21] [WIP] media: media-device: support for creating requests

[Alexandre Courbot]

Add a new MEDIA_IOC_NEW_REQUEST ioctl, which can be used to instantiate
a request suitable to control the media device topology as well as the
parameters and buffer flow of its entities.

This is still very early work, and mainly here to demonstrate that it is
still possible to bind requests to media entities.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/media-device.c | 11 +++
 include/media/mc-request.h   | 33 +
 include/media/media-device.h |  1 +
 include/media/media-entity.h |  5 +
 include/uapi/linux/media.h   |  2 ++
 5 files changed, 52 insertions(+)
 create mode 100644 include/media/mc-request.h

diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index e79f72b8b858..2fb8b9c5ec85 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -32,6 +32,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
@@ -359,6 +360,15 @@ static long media_device_get_topology(struct media_device 
*mdev,
return ret;
 }
 
+static long media_device_new_request(struct media_device *mdev,
+struct media_request_new *new)
+{
+   if (!mdev->req_mgr)
+   return -ENOTTY;
+
+   return media_request_ioctl_new(mdev->req_mgr, new);
+}
+
 static long copy_arg_from_user(void *karg, void __user *uarg, unsigned int cmd)
 {
/* All media IOCTLs are _IOWR() */
@@ -407,6 +417,7 @@ static const struct media_ioctl_info ioctl_info[] = {
MEDIA_IOC(ENUM_LINKS, media_device_enum_links, 
MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(SETUP_LINK, media_device_setup_link, 
MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, 
MEDIA_IOC_FL_GRAPH_MUTEX),
+   MEDIA_IOC(NEW_REQUEST, media_device_new_request, 0),
 };
 
 static long media_device_ioctl(struct file *filp, unsigned int cmd,
diff --git a/include/media/mc-request.h b/include/media/mc-request.h
new file mode 100644
index ..c14d38a93019
--- /dev/null
+++ b/include/media/mc-request.h
@@ -0,0 +1,33 @@
+/*
+ * Media requests support for media controller
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MEDIA_MC_REQUEST_H
+#define _MEDIA_MC_REQUEST_H
+
+#include 
+#include 
+
+#if IS_ENABLED(CONFIG_MEDIA_REQUEST_API)
+
+struct mc_request_entity {
+   struct media_request_entity base;
+   struct media_entity *entity;
+};
+
+#else  /* CONFIG_MEDIA_REQUEST_API */
+
+#endif  /* CONFIG_MEDIA_REQUEST_API */
+
+#endif
diff --git a/include/media/media-device.h b/include/media/media-device.h
index bcc6ec434f1f..e931e8b9f60e 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -127,6 +127,7 @@ struct media_device {
/* dev->driver_data points to this struct. */
struct device *dev;
struct media_devnode *devnode;
+   struct media_request_mgr *req_mgr;
 
char model[32];
char driver_name[32];
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index a732af1dbba0..e3525d1ec386 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -26,6 +26,8 @@
 #include 
 #include 
 
+struct mc_request_entity;
+
 /* Enums used internally at the media controller to represent graphs */
 
 /**
@@ -243,6 +245,7 @@ enum media_entity_type {
  * re-used if entities are unregistered or registered again.
  * @pads:  Pads array with the size defined by @num_pads.
  * @links: List of data links.
+ * @req_entity:Pointer to the request entity representing this entity, 
if any
  * @ops:   Entity operations.
  * @stream_count: Stream count for the entity.
  * @use_count: Use count for the entity.
@@ -279,6 +282,8 @@ struct media_entity {
struct media_pad *pads;
struct list_head links;
 
+   struct mc_request_entity *req_entity;
+
const struct media_entity_operations *ops;
 
int stream_count;
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index b9b9446095e9..eb0014c4eb40 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -30,6 +30,7 @@
 #include 
 #include 
 #include 
+#include 
 
 struct media_device_info {
char driver[16];
@@ -413,5 +414,6 @@ struct media_v2_topology {
 #define MEDIA_IOC_ENUM_LINKS   _IOWR('|', 0x02, struct 
media_links_enum)
 #define MEDIA_IOC_SETUP_LINK   _IOWR('|', 0x03, struct media_link

[RFCv4 19/21] media: vim2m: add request support

[Alexandre Courbot]

Set the necessary ops for supporting requests in vim2m.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/Kconfig |  1 +
 drivers/media/platform/vim2m.c | 75 ++
 2 files changed, 76 insertions(+)

diff --git a/drivers/media/platform/Kconfig b/drivers/media/platform/Kconfig
index 614fbef08ddc..09be0b5f9afe 100644
--- a/drivers/media/platform/Kconfig
+++ b/drivers/media/platform/Kconfig
@@ -534,6 +534,7 @@ source "drivers/media/platform/vivid/Kconfig"
 config VIDEO_VIM2M
tristate "Virtual Memory-to-Memory Driver"
depends on VIDEO_DEV && VIDEO_V4L2
+   select MEDIA_REQUEST_API
select VIDEOBUF2_VMALLOC
select V4L2_MEM2MEM_DEV
default n
diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 065483e62db4..02793dd9a330 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -30,6 +30,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 
 MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
 MODULE_AUTHOR("Pawel Osciak, <pa...@osciak.com>");
@@ -148,6 +150,8 @@ struct vim2m_dev {
struct timer_list   timer;
 
struct v4l2_m2m_dev *m2m_dev;
+
+   struct v4l2_request_mgr req_mgr;
 };
 
 struct vim2m_ctx {
@@ -155,6 +159,7 @@ struct vim2m_ctx {
struct vim2m_dev*dev;
 
struct v4l2_ctrl_handler hdl;
+   struct v4l2_request_entity req_entity;
 
/* Processed buffers in this transaction */
u8  num_processed;
@@ -367,6 +372,24 @@ static void job_abort(void *priv)
ctx->aborting = 1;
 }
 
+static int apply_request_params(struct media_request *req,
+   struct vim2m_ctx *ctx)
+{
+   struct v4l2_request_entity_data *data;
+
+   if (!req)
+   return -EINVAL;
+
+   data = to_v4l2_entity_data(media_request_get_entity_data(req,
+   >req_entity.base));
+   if (WARN_ON(IS_ERR(data)))
+   return PTR_ERR(data);
+
+   v4l2_ctrl_request_setup(>ctrls);
+
+   return 0;
+}
+
 /* device_run() - prepares and starts the device
  *
  * This simulates all the immediate preparations required before starting
@@ -382,6 +405,19 @@ static void device_run(void *priv)
src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
 
+   WARN_ON(dst_buf->vb2_buf.request != NULL &&
+   dst_buf->vb2_buf.request != src_buf->vb2_buf.request);
+
+   /* Apply request if needed */
+   if (src_buf->vb2_buf.request) {
+   int ret = apply_request_params(src_buf->vb2_buf.request, ctx);
+   if (ret) {
+   dprintk(dev, "error applying request parameters: %d\n",
+   ret);
+   return;
+   }
+   }
+
device_process(ctx, src_buf, dst_buf);
 
/* Run a timer, which simulates a hardware irq  */
@@ -393,6 +429,7 @@ static void device_isr(struct timer_list *t)
struct vim2m_dev *vim2m_dev = from_timer(vim2m_dev, t, timer);
struct vim2m_ctx *curr_ctx;
struct vb2_v4l2_buffer *src_vb, *dst_vb;
+   struct media_request *req;
unsigned long flags;
 
curr_ctx = v4l2_m2m_get_curr_priv(vim2m_dev->m2m_dev);
@@ -404,6 +441,7 @@ static void device_isr(struct timer_list *t)
 
src_vb = v4l2_m2m_src_buf_remove(curr_ctx->fh.m2m_ctx);
dst_vb = v4l2_m2m_dst_buf_remove(curr_ctx->fh.m2m_ctx);
+   req = src_vb->vb2_buf.request;
 
curr_ctx->num_processed++;
 
@@ -411,6 +449,8 @@ static void device_isr(struct timer_list *t)
v4l2_m2m_buf_done(src_vb, VB2_BUF_STATE_DONE);
v4l2_m2m_buf_done(dst_vb, VB2_BUF_STATE_DONE);
spin_unlock_irqrestore(_dev->irqlock, flags);
+   if (req)
+   media_request_entity_complete(req, _ctx->req_entity.base);
 
if (curr_ctx->num_processed == curr_ctx->translen
|| curr_ctx->aborting) {
@@ -838,6 +878,7 @@ static int queue_init(void *priv, struct vb2_queue *src_vq, 
struct vb2_queue *ds
src_vq->mem_ops = _vmalloc_memops;
src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
src_vq->lock = >dev->dev_mutex;
+   src_vq->allow_requests = true;
 
ret = vb2_queue_init(src_vq);
if (ret)
@@ -851,6 +892,7 @@ static int queue_init(void *priv, struct vb2_queue *src_vq, 
struct vb2_queue *ds
dst_vq->mem_ops = _vmalloc_memops;
dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
dst_vq->lock = >dev->dev_mutex;
+   dst_vq->allow_requests = true;
 
return vb2_queue_init(dst_vq);
 }
@@ -877

[RFCv4 01/21] media: add request API core and UAPI

[Alexandre Courbot]

The request API provides a way to group buffers and device parameters
into units of work to be queued and executed. This patch introduces the
UAPI and core framework.

This patch is based on the previous work by Laurent Pinchart. The core
has changed considerably, but the UAPI is mostly untouched.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/Kconfig  |   3 +
 drivers/media/Makefile |   6 +
 drivers/media/media-request.c  | 341 
 include/media/media-request.h  | 349 +
 include/uapi/linux/media-request.h |  37 +++
 5 files changed, 736 insertions(+)
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request.h
 create mode 100644 include/uapi/linux/media-request.h

diff --git a/drivers/media/Kconfig b/drivers/media/Kconfig
index 145e12bfb819..db30fc9547d2 100644
--- a/drivers/media/Kconfig
+++ b/drivers/media/Kconfig
@@ -130,6 +130,9 @@ config VIDEO_V4L2_SUBDEV_API
 
  This API is mostly used by camera interfaces in embedded platforms.
 
+config MEDIA_REQUEST_API
+   tristate
+
 source "drivers/media/v4l2-core/Kconfig"
 
 #
diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 594b462ddf0e..03c0a39ad344 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -5,6 +5,12 @@
 
 media-objs := media-device.o media-devnode.o media-entity.o
 
+#
+# Request API support comes as its own module since it can be used by
+# both media and video devices
+#
+obj-$(CONFIG_MEDIA_REQUEST_API) += media-request.o
+
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
 # when compiled as builtin drivers
diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
new file mode 100644
index ..b88362028561
--- /dev/null
+++ b/drivers/media/media-request.c
@@ -0,0 +1,341 @@
+/*
+ * Request base implementation
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+
+const struct file_operations request_fops;
+
+static const char * const media_request_states[] __maybe_unused = {
+   "IDLE",
+   "SUBMITTED",
+   "COMPLETED",
+   "INVALID",
+};
+
+struct media_request *media_request_get(struct media_request *req)
+{
+   get_file(req->file);
+   return req;
+}
+EXPORT_SYMBOL_GPL(media_request_get);
+
+struct media_request *media_request_get_from_fd(int fd)
+{
+   struct file *f;
+
+   f = fget(fd);
+   if (!f)
+   return NULL;
+
+   /* Not a request FD? */
+   if (f->f_op != _fops) {
+   fput(f);
+   return NULL;
+   }
+
+   return f->private_data;
+}
+EXPORT_SYMBOL_GPL(media_request_get_from_fd);
+
+void media_request_put(struct media_request *req)
+{
+   if (WARN_ON(req == NULL))
+   return;
+
+   fput(req->file);
+}
+EXPORT_SYMBOL_GPL(media_request_put);
+
+struct media_request_entity_data *
+media_request_get_entity_data(struct media_request *req,
+ struct media_request_entity *entity)
+{
+   struct media_request_entity_data *data;
+
+   /* First check that this entity is valid for this request at all */
+   if (!req->mgr->ops->entity_valid(req, entity))
+   return ERR_PTR(-EINVAL);
+
+   mutex_lock(>lock);
+
+   /* Lookup whether we already have entity data */
+   list_for_each_entry(data, >data, list) {
+   if (data->entity == entity)
+   goto out;
+   }
+
+   /* No entity data found, let's create it */
+   data = entity->ops->data_alloc(req, entity);
+   if (IS_ERR(data))
+   goto out;
+
+   data->entity = entity;
+   list_add_tail(>list, >data);
+
+out:
+   mutex_unlock(>lock);
+
+   return data;
+}
+EXPORT_SYMBOL_GPL(media_request_get_entity_data);
+
+static unsigned int media_request_poll(struct file *file, poll_table *wait)
+{
+   struct media_request *req = file->private_data;
+
+   poll_wait(file, >complete_wait, wait);
+
+   if (req->state == MEDIA_REQUEST_STATE_COMPLETED)
+   return POLLIN | POLLRDNORM;
+
+   return 0;
+}
+
+static int media_request_release(struct inode *inode, struct file *filp)
+{
+   struct medi

[RFCv3 00/17] Request API, take three

[Alexandre Courbot]

As discussed yesterday, here is a rebase on the media master branch. A
few minor fixes for typos have also slept in, but otherwise this is
equivalent to v2. I expect to have the buffer queueing behavior fixed in
the next version.

Alexandre Courbot (9):
  media: add request API core and UAPI
  media: videobuf2: add support for requests
  media: vb2: add support for requests in QBUF ioctl
  v4l2: add request API support
  videodev2.h: add request_fd field to v4l2_ext_controls
  v4l2-ctrls: support requests in EXT_CTRLS ioctls
  v4l2: document the request API interface
  media: vim2m: add media device
  media: vim2m: add request support

Hans Verkuil (7):
  videodev2.h: Add request_fd field to v4l2_buffer
  v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev
  v4l2-ctrls: prepare internal structs for request API
  v4l2-ctrls: add core request API
  v4l2-ctrls: use ref in helper instead of ctrl
  v4l2-ctrls: support g/s_ext_ctrls for requests
  v4l2-ctrls: add v4l2_ctrl_request_setup

Laurent Pinchart (1):
  media: Document the media request API

 Documentation/media/uapi/mediactl/media-funcs.rst  |   1 +
 .../media/uapi/mediactl/media-ioc-request-cmd.rst  | 142 ++
 Documentation/media/uapi/v4l/buffer.rst|  10 +-
 Documentation/media/uapi/v4l/common.rst|   1 +
 Documentation/media/uapi/v4l/request-api.rst   | 236 
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst  |  16 +-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst   |  21 ++
 drivers/media/Makefile |   3 +-
 drivers/media/common/videobuf2/videobuf2-core.c| 133 -
 drivers/media/common/videobuf2/videobuf2-v4l2.c|  31 +-
 drivers/media/dvb-frontends/rtl2832_sdr.c  |   5 +-
 drivers/media/media-device.c   |   7 +
 drivers/media/media-request-mgr.c  | 105 +++
 drivers/media/media-request.c  | 311 +
 drivers/media/pci/bt8xx/bttv-driver.c  |   2 +-
 drivers/media/pci/cx23885/cx23885-417.c|   2 +-
 drivers/media/pci/cx88/cx88-blackbird.c|   2 +-
 drivers/media/pci/cx88/cx88-video.c|   2 +-
 drivers/media/pci/saa7134/saa7134-empress.c|   4 +-
 drivers/media/pci/saa7134/saa7134-video.c  |   2 +-
 drivers/media/platform/exynos4-is/fimc-capture.c   |   2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c|   3 +-
 drivers/media/platform/rcar_drif.c |   2 +-
 drivers/media/platform/soc_camera/soc_camera.c |   3 +-
 drivers/media/platform/vim2m.c |  79 ++
 drivers/media/platform/vivid/vivid-ctrls.c |  46 +--
 drivers/media/usb/cpia2/cpia2_v4l.c|   2 +-
 drivers/media/usb/cx231xx/cx231xx-417.c|   2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c  |   4 +-
 drivers/media/usb/msi2500/msi2500.c|   2 +-
 drivers/media/usb/tm6000/tm6000-video.c|   2 +-
 drivers/media/v4l2-core/Makefile   |   2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c  |   9 +-
 drivers/media/v4l2-core/v4l2-ctrls.c   | 238 ++--
 drivers/media/v4l2-core/v4l2-device.c  |   3 +-
 drivers/media/v4l2-core/v4l2-ioctl.c   | 125 -
 drivers/media/v4l2-core/v4l2-request.c |  54 
 drivers/staging/media/imx/imx-media-dev.c  |   2 +-
 drivers/staging/media/imx/imx-media-fim.c  |   2 +-
 include/media/media-device.h   |   3 +
 include/media/media-entity.h   |   9 +
 include/media/media-request-mgr.h  |  73 +
 include/media/media-request.h  | 186 
 include/media/v4l2-ctrls.h |  17 +-
 include/media/v4l2-request.h   |  34 +++
 include/media/videobuf2-core.h |  15 +-
 include/media/videobuf2-v4l2.h |   2 +
 include/uapi/linux/media.h |  10 +
 include/uapi/linux/videodev2.h |   6 +-
 49 files changed, 1886 insertions(+), 87 deletions(-)
 create mode 100644 Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst
 create mode 100644 drivers/media/media-request-mgr.c
 create mode 100644 drivers/media/media-request.c
 create mode 100644 drivers/media/v4l2-core/v4l2-request.c
 create mode 100644 include/media/media-request-mgr.h
 create mode 100644 include/media/media-request.h
 create mode 100644 include/media/v4l2-request.h

-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 02/17] videodev2.h: Add request_fd field to v4l2_buffer

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

When queuing buffers allow for passing the request that should
be associated with this buffer.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
[acour...@chromium.org: make request ID 32-bit]
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 3 ++-
 drivers/media/usb/cpia2/cpia2_v4l.c | 2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   | 9 ++---
 drivers/media/v4l2-core/v4l2-ioctl.c| 4 ++--
 include/media/videobuf2-v4l2.h  | 2 ++
 include/uapi/linux/videodev2.h  | 3 ++-
 6 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index fac3cd6f901d..0034f4d190f2 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->request_fd = vbuf->request_fd;
b->reserved = 0;
 
if (q->is_multiplanar) {
@@ -320,6 +320,7 @@ static int __fill_vb2_buffer(struct vb2_buffer *vb,
}
vb->timestamp = 0;
vbuf->sequence = 0;
+   vbuf->request_fd = b->request_fd;
 
if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
if (b->memory == VB2_MEMORY_USERPTR) {
diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
b/drivers/media/usb/cpia2/cpia2_v4l.c
index a1c59f19cf2d..54c5aa0ecd26 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct 
v4l2_buffer *buf)
buf->sequence = cam->buffers[buf->index].seq;
buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
buf->length = cam->frame_size;
-   buf->reserved2 = 0;
+   buf->request_fd = 0;
buf->reserved = 0;
memset(>timecode, 0, sizeof(buf->timecode));
 
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 5198c9eeb348..32bf47489a2e 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -386,7 +386,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __s32   request_fd;
__u32   reserved;
 };
 
@@ -486,6 +486,7 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *kp,
 {
u32 type;
u32 length;
+   s32 request_fd;
enum v4l2_memory memory;
struct v4l2_plane32 __user *uplane32;
struct v4l2_plane __user *uplane;
@@ -500,7 +501,9 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *kp,
get_user(memory, >memory) ||
put_user(memory, >memory) ||
get_user(length, >length) ||
-   put_user(length, >length))
+   put_user(length, >length) ||
+   get_user(request_fd, >request_fd) ||
+   put_user(request_fd, >request_fd))
return -EFAULT;
 
if (V4L2_TYPE_IS_OUTPUT(type))
@@ -604,7 +607,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
assign_in_user(>timestamp.tv_usec, >timestamp.tv_usec) ||
copy_in_user(>timecode, >timecode, sizeof(kp->timecode)) ||
assign_in_user(>sequence, >sequence) ||
-   assign_in_user(>reserved2, >reserved2) ||
+   assign_in_user(>request_fd, >request_fd) ||
assign_in_user(>reserved, >reserved) ||
get_user(length, >length) ||
put_user(length, >length))
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index e5109e5b8bf5..2f40ac0cdf6e 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool 
write_only)
const struct v4l2_plane *plane;
int i;
 
-   pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, 
field=%s, sequence=%d, memory=%s",
+   pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request_fd=%u, 
flags=0x%08x, field=%s, sequence=%d, memory=%s",
p->timestamp.tv_sec / 3600,
(int)(p->timestamp.tv_sec / 60) % 60,
(int)(p->timestamp.tv_sec % 60),
(long)p->timestamp.tv_usec,
  

[RFCv3 01/17] media: add request API core and UAPI

[Alexandre Courbot]

The request API provides a way to group buffers and device parameters
into units of work to be queued and executed. This patch introduces the
UAPI and core framework.

This patch is based on the previous work by Laurent Pinchart. The core
has changed considerably, but the UAPI is mostly untouched.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/Makefile   |   3 +-
 drivers/media/media-device.c |   7 +
 drivers/media/media-request-mgr.c| 105 
 drivers/media/media-request.c| 311 +++
 drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
 include/media/media-device.h |   3 +
 include/media/media-entity.h |   9 +
 include/media/media-request-mgr.h|  73 
 include/media/media-request.h| 186 +
 include/uapi/linux/media.h   |  10 ++
 10 files changed, 707 insertions(+), 2 deletions(-)
 create mode 100644 drivers/media/media-request-mgr.c
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request-mgr.h
 create mode 100644 include/media/media-request.h

diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 594b462ddf0e..06c43ddb52ea 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -3,7 +3,8 @@
 # Makefile for the kernel multimedia device drivers.
 #
 
-media-objs := media-device.o media-devnode.o media-entity.o
+media-objs := media-device.o media-devnode.o media-entity.o \
+  media-request.o media-request-mgr.o
 
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index e79f72b8b858..024ee81a8334 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -32,6 +32,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
@@ -407,6 +409,7 @@ static const struct media_ioctl_info ioctl_info[] = {
MEDIA_IOC(ENUM_LINKS, media_device_enum_links, 
MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(SETUP_LINK, media_device_setup_link, 
MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, 
MEDIA_IOC_FL_GRAPH_MUTEX),
+   MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
 };
 
 static long media_device_ioctl(struct file *filp, unsigned int cmd,
@@ -688,6 +691,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
 
 void media_device_cleanup(struct media_device *mdev)
 {
+   if (mdev->req_mgr) {
+   media_request_mgr_free(mdev->req_mgr);
+   mdev->req_mgr = NULL;
+   }
ida_destroy(>entity_internal_idx);
mdev->entity_internal_idx_max = 0;
media_graph_walk_cleanup(>pm_count_walk);
diff --git a/drivers/media/media-request-mgr.c 
b/drivers/media/media-request-mgr.c
new file mode 100644
index ..686e877a884b
--- /dev/null
+++ b/drivers/media/media-request-mgr.c
@@ -0,0 +1,105 @@
+/*
+ * Generic request manager implementation.
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+static struct media_request *
+media_request_alloc(struct media_request_mgr *mgr)
+{
+   struct media_request *req;
+
+   req = kzalloc(sizeof(*req), GFP_KERNEL);
+   if (!req)
+   return ERR_PTR(-ENOMEM);
+
+   req->mgr = mgr;
+   req->state = MEDIA_REQUEST_STATE_IDLE;
+   kref_init(>kref);
+   INIT_LIST_HEAD(>data);
+   init_waitqueue_head(>complete_wait);
+   ATOMIC_INIT_NOTIFIER_HEAD(>submit_notif);
+   mutex_init(>lock);
+
+   mutex_lock(>mutex);
+   req->id = ++mgr->req_id;
+   list_add_tail(>list, >requests);
+   mutex_unlock(>mutex);
+
+   return req;
+}
+
+static void media_request_free(struct media_request *req)
+{
+   struct media_request_mgr *mgr = req->mgr;
+   struct media_request_entity_data *data, *next;
+
+   mutex_lock(>mutex);
+   list_del(>list);
+   mutex_unlock(>mutex);
+
+   list_for_each_entry_safe(data, next, >data, list) {
+   list_del(>list);
+   data->entity->req_ops->data_free(data);
+   }
+
+   kfree(req);
+}
+
+void media_request_mgr_free(struct media_request_mgr *mgr)
+{
+   struct media_device *mdev = mgr->mdev;
+
+   /* Just a sanity check - we should have no remaining r

[RFCv3 06/17] v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a 'bool from_other_dev' argument: set to true if the two
handlers refer to different devices (e.g. it is true when
inheriting controls from a subdev into a main v4l2 bridge
driver).

This will be used later when implementing support for the
request API since we need to skip such controls.

TODO: check drivers/staging/media/imx/imx-media-fim.c change.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/dvb-frontends/rtl2832_sdr.c|  5 +--
 drivers/media/pci/bt8xx/bttv-driver.c|  2 +-
 drivers/media/pci/cx23885/cx23885-417.c  |  2 +-
 drivers/media/pci/cx88/cx88-blackbird.c  |  2 +-
 drivers/media/pci/cx88/cx88-video.c  |  2 +-
 drivers/media/pci/saa7134/saa7134-empress.c  |  4 +--
 drivers/media/pci/saa7134/saa7134-video.c|  2 +-
 drivers/media/platform/exynos4-is/fimc-capture.c |  2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c  |  3 +-
 drivers/media/platform/rcar_drif.c   |  2 +-
 drivers/media/platform/soc_camera/soc_camera.c   |  3 +-
 drivers/media/platform/vivid/vivid-ctrls.c   | 46 
 drivers/media/usb/cx231xx/cx231xx-417.c  |  2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c|  4 +--
 drivers/media/usb/msi2500/msi2500.c  |  2 +-
 drivers/media/usb/tm6000/tm6000-video.c  |  2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c | 11 +++---
 drivers/media/v4l2-core/v4l2-device.c|  3 +-
 drivers/staging/media/imx/imx-media-dev.c|  2 +-
 drivers/staging/media/imx/imx-media-fim.c|  2 +-
 include/media/v4l2-ctrls.h   |  4 ++-
 21 files changed, 58 insertions(+), 49 deletions(-)

diff --git a/drivers/media/dvb-frontends/rtl2832_sdr.c 
b/drivers/media/dvb-frontends/rtl2832_sdr.c
index c6e78d870ccd..6064d28224e8 100644
--- a/drivers/media/dvb-frontends/rtl2832_sdr.c
+++ b/drivers/media/dvb-frontends/rtl2832_sdr.c
@@ -1394,7 +1394,8 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
case RTL2832_SDR_TUNER_E4000:
v4l2_ctrl_handler_init(>hdl, 9);
if (subdev)
-   v4l2_ctrl_add_handler(>hdl, subdev->ctrl_handler, 
NULL);
+   v4l2_ctrl_add_handler(>hdl, subdev->ctrl_handler,
+ NULL, true);
break;
case RTL2832_SDR_TUNER_R820T:
case RTL2832_SDR_TUNER_R828D:
@@ -1423,7 +1424,7 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
v4l2_ctrl_handler_init(>hdl, 2);
if (subdev)
v4l2_ctrl_add_handler(>hdl, subdev->ctrl_handler,
- NULL);
+ NULL, true);
break;
default:
v4l2_ctrl_handler_init(>hdl, 0);
diff --git a/drivers/media/pci/bt8xx/bttv-driver.c 
b/drivers/media/pci/bt8xx/bttv-driver.c
index b366a7e1d976..91874f775d37 100644
--- a/drivers/media/pci/bt8xx/bttv-driver.c
+++ b/drivers/media/pci/bt8xx/bttv-driver.c
@@ -4211,7 +4211,7 @@ static int bttv_probe(struct pci_dev *dev, const struct 
pci_device_id *pci_id)
/* register video4linux + input */
if (!bttv_tvcards[btv->c.type].no_video) {
v4l2_ctrl_add_handler(>radio_ctrl_handler, hdl,
-   v4l2_ctrl_radio_filter);
+   v4l2_ctrl_radio_filter, false);
if (btv->radio_ctrl_handler.error) {
result = btv->radio_ctrl_handler.error;
goto fail2;
diff --git a/drivers/media/pci/cx23885/cx23885-417.c 
b/drivers/media/pci/cx23885/cx23885-417.c
index a71f3c7569ce..762823871c78 100644
--- a/drivers/media/pci/cx23885/cx23885-417.c
+++ b/drivers/media/pci/cx23885/cx23885-417.c
@@ -1527,7 +1527,7 @@ int cx23885_417_register(struct cx23885_dev *dev)
dev->cxhdl.priv = dev;
dev->cxhdl.func = cx23885_api_func;
cx2341x_handler_set_50hz(>cxhdl, tsport->height == 576);
-   v4l2_ctrl_add_handler(>ctrl_handler, >cxhdl.hdl, NULL);
+   v4l2_ctrl_add_handler(>ctrl_handler, >cxhdl.hdl, NULL, false);
 
/* Allocate and initialize V4L video device */
dev->v4l_device = cx23885_video_dev_alloc(tsport,
diff --git a/drivers/media/pci/cx88/cx88-blackbird.c 
b/drivers/media/pci/cx88/cx88-blackbird.c
index 0e0952e60795..39f69d89a663 100644
--- a/drivers/media/pci/cx88/cx88-blackbird.c
+++ b/drivers/media/pci/cx88/cx88-blackbird.c
@@ -1183,7 +1183,7 @@ static int cx8802_blackbird_probe(struct cx8802_driver 
*drv)
err = cx2341x_handler_init(>cxhdl, 36);
if (err)
goto fail_core;
-   v4l2_ctrl_add_handler(>cxhdl

[RFCv3 08/17] v4l2-ctrls: add core request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add the four core request functions:

v4l2_ctrl_request_init() initializes a new (empty) request.
v4l2_ctrl_request_clone() resets a request based on another request
(or clears it if that request is NULL).
v4l2_ctrl_request_get(): increase refcount
v4l2_ctrl_request_put(): decrease refcount and delete if it reaches 0.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
[acour...@chromium.org: turn v4l2_ctrl_request_alloc into init function]
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 106 ++-
 include/media/v4l2-ctrls.h   |   7 +++
 2 files changed, 110 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 1ff8fc59fff5..c692a6d925c6 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1878,6 +1878,7 @@ EXPORT_SYMBOL(v4l2_ctrl_find);
 /* Allocate a new v4l2_ctrl_ref and hook it into the handler. */
 static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
   struct v4l2_ctrl *ctrl,
+  struct v4l2_ctrl_ref **ctrl_ref,
   bool from_other_dev)
 {
struct v4l2_ctrl_ref *ref;
@@ -1885,6 +1886,10 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
u32 id = ctrl->id;
u32 class_ctrl = V4L2_CTRL_ID2WHICH(id) | 1;
int bucket = id % hdl->nr_of_buckets;   /* which bucket to use */
+   unsigned int sz_extra = 0;
+
+   if (ctrl_ref)
+   *ctrl_ref = NULL;
 
/*
 * Automatically add the control class if it is not yet present and
@@ -1898,11 +1903,16 @@ static int handler_new_ref(struct v4l2_ctrl_handler 
*hdl,
if (hdl->error)
return hdl->error;
 
-   new_ref = kzalloc(sizeof(*new_ref), GFP_KERNEL);
+   if (hdl->is_request)
+   sz_extra = ctrl->elems * ctrl->elem_size;
+   new_ref = kzalloc(sizeof(*new_ref) + sz_extra, GFP_KERNEL);
if (!new_ref)
return handler_set_err(hdl, -ENOMEM);
new_ref->ctrl = ctrl;
new_ref->from_other_dev = from_other_dev;
+   if (sz_extra)
+   new_ref->p_req.p = _ref[1];
+
if (ctrl->handler == hdl) {
/* By default each control starts in a cluster of its own.
   new_ref->ctrl is basically a cluster array with one
@@ -1942,6 +1952,8 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
/* Insert the control node in the hash */
new_ref->next = hdl->buckets[bucket];
hdl->buckets[bucket] = new_ref;
+   if (ctrl_ref)
+   *ctrl_ref = new_ref;
 
 unlock:
mutex_unlock(hdl->lock);
@@ -2083,7 +2095,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct 
v4l2_ctrl_handler *hdl,
ctrl->type_ops->init(ctrl, idx, ctrl->p_new);
}
 
-   if (handler_new_ref(hdl, ctrl, false)) {
+   if (handler_new_ref(hdl, ctrl, NULL, false)) {
kvfree(ctrl);
return NULL;
}
@@ -2276,7 +2288,7 @@ int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
/* Filter any unwanted controls */
if (filter && !filter(ctrl))
continue;
-   ret = handler_new_ref(hdl, ctrl, from_other_dev);
+   ret = handler_new_ref(hdl, ctrl, NULL, from_other_dev);
if (ret)
break;
}
@@ -2685,6 +2697,94 @@ int v4l2_querymenu(struct v4l2_ctrl_handler *hdl, struct 
v4l2_querymenu *qm)
 }
 EXPORT_SYMBOL(v4l2_querymenu);
 
+int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl)
+{
+   int err;
+
+   err = v4l2_ctrl_handler_init(hdl, 0);
+   if (err)
+   return err;
+   hdl->is_request = true;
+   kref_init(>ref);
+
+   return 0;
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_init);
+
+int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
+   const struct v4l2_ctrl_handler *from,
+   bool (*filter)(const struct v4l2_ctrl *ctrl))
+{
+   struct v4l2_ctrl_ref *ref;
+   int err;
+
+   if (WARN_ON(!hdl || hdl == from))
+   return -EINVAL;
+
+   if (hdl->error)
+   return hdl->error;
+
+   WARN_ON(hdl->lock != >_lock);
+   v4l2_ctrl_handler_free(hdl);
+   err = v4l2_ctrl_handler_init(hdl, (from->nr_of_buckets - 1) * 8);
+   hdl->is_request = true;
+   if (err)
+   return err;
+   if (!from)
+   return 0;
+
+   mutex_lock(from->lock);
+   list_for_each_entry(ref, >ctrl_refs, node) {
+   struct v4l2_ctrl *ctrl = ref->ctrl;
+   struct v4l2_ctrl_ref *new_ref;
+
+   /*

[RFCv3 04/17] media: vb2: add support for requests in QBUF ioctl

[Alexandre Courbot]

Support the request argument of the QBUF ioctl.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 83 +++-
 1 file changed, 82 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index 2f40ac0cdf6e..235acdde3111 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -29,6 +29,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -968,6 +969,71 @@ static int check_fmt(struct file *file, enum v4l2_buf_type 
type)
return -EINVAL;
 }
 
+/*
+ * Validate that a given request can be used during an ioctl.
+ *
+ * When using the request API, request file descriptors must be matched against
+ * the actual request object. User-space can pass any file descriptor, so we
+ * need to make sure the call is valid before going further.
+ *
+ * This function looks up the request and associated data and performs the
+ * following sanity checks:
+ *
+ * * Make sure that the entity belongs to the media_device managing the passed
+ *   request,
+ * * Make sure that the entity data (if any) is associated to the current file
+ *   handler.
+ *
+ * This function returns a pointer to the valid request, or and error code in
+ * case of failure. When successful, a reference to the request is acquired and
+ * must be properly released by the caller. If _data is not NULL, then it is
+ * filled with a pointer to the corresponding entity data.
+ */
+#ifdef CONFIG_MEDIA_CONTROLLER
+static struct media_request *
+check_request(int request, struct file *file, void *fh, struct 
media_request_entity_data **_data)
+{
+   struct media_request *req = NULL;
+   struct video_device *vfd = video_devdata(file);
+   struct v4l2_fh *vfh =
+   test_bit(V4L2_FL_USES_V4L2_FH, >flags) ? fh : NULL;
+   struct media_entity *entity = >entity;
+   struct media_request_entity_data *data;
+
+   if (!entity)
+   return ERR_PTR(-EINVAL);
+
+   req = media_request_get_from_fd(request);
+   if (!req)
+   return ERR_PTR(-EINVAL);
+
+   /* Validate that the entity belongs to the correct media_device */
+   if (vfd->v4l2_dev->mdev->req_mgr != req->mgr) {
+   media_request_put(req);
+   return ERR_PTR(-EINVAL);
+   }
+
+   /* Validate that the entity's data belongs to the correct fh */
+   data = media_request_get_entity_data(req, entity, vfh);
+   if (IS_ERR(data)) {
+   media_request_put(req);
+   return ERR_PTR(PTR_ERR(data));
+   }
+
+   if (_data)
+   *_data = data;
+
+   return req;
+}
+#else /* CONFIG_MEDIA_CONTROLLER */
+static struct media_request *
+check_request(int request, struct file *file, void *fh)
+{
+   return ERR_PTR(-ENOSYS);
+}
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 static void v4l_sanitize_format(struct v4l2_format *fmt)
 {
unsigned int offset;
@@ -1878,10 +1944,25 @@ static int v4l_querybuf(const struct v4l2_ioctl_ops 
*ops,
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
struct file *file, void *fh, void *arg)
 {
+   struct media_request *req = NULL;
struct v4l2_buffer *p = arg;
int ret = check_fmt(file, p->type);
 
-   return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+   if (ret)
+   return ret;
+
+   if (p->request_fd > 0) {
+   req = check_request(p->request_fd, file, fh, NULL);
+   if (IS_ERR(req))
+   return PTR_ERR(req);
+   }
+
+   ret = ops->vidioc_qbuf(file, fh, p);
+
+   if (req)
+   media_request_put(req);
+
+   return ret;
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 07/17] v4l2-ctrls: prepare internal structs for request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a refcount and is_request bool to struct v4l2_ctrl_handler:
this is used to refcount a handler that represents a request.

Add a p_req field to struct v4l2_ctrl_ref that will store the
request value.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 1 +
 include/media/v4l2-ctrls.h   | 4 
 2 files changed, 5 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 2e58381444d1..1ff8fc59fff5 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1761,6 +1761,7 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler 
*hdl,
  sizeof(hdl->buckets[0]),
  GFP_KERNEL | __GFP_ZERO);
hdl->error = hdl->buckets ? 0 : -ENOMEM;
+   hdl->is_request = false;
return hdl->error;
 }
 EXPORT_SYMBOL(v4l2_ctrl_handler_init_class);
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index a4750a8090e3..22b6be78057f 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -18,6 +18,7 @@
 #define _V4L2_CTRLS_H
 
 #include 
+#include 
 #include 
 #include 
 
@@ -257,6 +258,7 @@ struct v4l2_ctrl_ref {
struct v4l2_ctrl_ref *next;
struct v4l2_ctrl *ctrl;
struct v4l2_ctrl_helper *helper;
+   union v4l2_ctrl_ptr p_req;
bool from_other_dev;
 };
 
@@ -292,7 +294,9 @@ struct v4l2_ctrl_handler {
v4l2_ctrl_notify_fnc notify;
void *notify_priv;
u16 nr_of_buckets;
+   bool is_request;
int error;
+   struct kref ref;
 };
 
 /**
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 05/17] media: Document the media request API

[Alexandre Courbot]

From: Laurent Pinchart <laurent.pinchart+rene...@ideasonboard.com>

The media request API is made of a new ioctl to implement request
management. Document it.

Signed-off-by: Laurent Pinchart <laurent.pinchart+rene...@ideasonboard.com>
[acour...@chromium.org: adapt for newest API]
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 Documentation/media/uapi/mediactl/media-funcs.rst  |   1 +
 .../media/uapi/mediactl/media-ioc-request-cmd.rst  | 142 +
 2 files changed, 143 insertions(+)
 create mode 100644 Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst

diff --git a/Documentation/media/uapi/mediactl/media-funcs.rst 
b/Documentation/media/uapi/mediactl/media-funcs.rst
index 076856501cdb..e3a45d82ffcb 100644
--- a/Documentation/media/uapi/mediactl/media-funcs.rst
+++ b/Documentation/media/uapi/mediactl/media-funcs.rst
@@ -15,4 +15,5 @@ Function Reference
 media-ioc-g-topology
 media-ioc-enum-entities
 media-ioc-enum-links
+media-ioc-request-cmd
 media-ioc-setup-link
diff --git a/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst 
b/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
new file mode 100644
index ..ced76ff3498d
--- /dev/null
+++ b/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
@@ -0,0 +1,142 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _media_ioc_request_cmd:
+
+***
+ioctl MEDIA_IOC_REQUEST_CMD
+***
+
+Name
+
+
+MEDIA_IOC_REQUEST_CMD - Manage media device requests
+
+
+Synopsis
+
+
+.. c:function:: int ioctl( int fd, MEDIA_IOC_REQUEST_CMD, struct 
media_request_cmd *argp )
+:name: MEDIA_IOC_REQUEST_CMD
+
+
+Arguments
+=
+
+``fd``
+File descriptor returned by :ref:`open() `.
+
+``argp``
+
+
+Description
+===
+
+The MEDIA_IOC_REQUEST_CMD ioctl allows applications to manage media device
+requests. A request is an object that can group media device configuration
+parameters, including subsystem-specific parameters, in order to apply all the
+parameters atomically. Applications are responsible for allocating and
+deleting requests, filling them with configuration parameters and submitting
+them.
+
+Request operations are performed by calling the MEDIA_IOC_REQUEST_CMD ioctl
+with a pointer to a struct :c:type:`media_request_cmd` with the cmd field set
+to the appropriate command. :ref:`media-request-command` lists the commands
+supported by the ioctl.
+
+The struct :c:type:`media_request_cmd` request field contains the file
+descriptor of the request on which the command operates. For the
+``MEDIA_REQ_CMD_ALLOC`` command the field is set to zero by applications and
+filled by the driver. For all other commands the field is set by applications
+and left untouched by the driver.
+
+To allocate a new request applications use the ``MEDIA_REQ_CMD_ALLOC``
+command. The driver will allocate a new request and return its FD in the
+request field. After allocation, the request is "empty", which means that it
+does not hold any state of its own, and that the hardware's state will not be
+affected by it unless it is passed as argument to V4L2 or media controller
+commands.
+
+Requests are reference-counted. A newly allocated request is referenced
+by the returned file descriptor, and can be later referenced by
+subsystem-specific operations. Requests will thus be automatically deleted
+when they're no longer used after the returned file descriptor is closed.
+
+If a request isn't needed applications can delete it by calling ``close()``
+on it. The driver will drop the file handle reference. The request will not
+be usable through the MEDIA_IOC_REQUEST_CMD ioctl anymore, but will only be
+deleted when the last reference is released. If no other reference exists when
+``close()`` is invoked the request will be deleted immediately.
+
+After creating a request applications should fill it with configuration
+parameters. This is performed through subsystem-specific request APIs outside
+the scope of the media controller API. See the appropriate subsystem APIs for
+more information, including how they interact with the MEDIA_IOC_REQUEST_CMD
+ioctl.
+
+Once a request contains all the desired configuration parameters it can be
+submitted using the ``MEDIA_REQ_CMD_SUBMIT`` command. This will let the
+buffers queued for the request be passed to their respective drivers, which
+will then apply the request's parameters before processing them.
+
+Once a request has been queued applications are not allowed to modify its
+configuration parameters until the request has been fully processed. Any
+attempt to do so will result in the related subsystem API returning an error.
+The application that submitted the request can wait for its completion by
+polling on the request's file descriptor.
+
+Once a request has completed, it can be reused. The ``MEDIA_REQ_CMD_REINIT``
+command will bring it back to its initial 

[RFCv3 10/17] v4l2-ctrls: support g/s_ext_ctrls for requests

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

The v4l2_g/s_ext_ctrls functions now support control handlers that
represent requests.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 37 
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 9090a49eef91..4fa7adef7531 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1528,6 +1528,13 @@ static int new_to_user(struct v4l2_ext_control *c,
return ptr_to_user(c, ctrl, ctrl->p_new);
 }
 
+/* Helper function: copy the request value back to the caller */
+static int req_to_user(struct v4l2_ext_control *c,
+  struct v4l2_ctrl_ref *ref)
+{
+   return ptr_to_user(c, ref->ctrl, ref->p_req);
+}
+
 /* Helper function: copy the initial control value back to the caller */
 static int def_to_user(struct v4l2_ext_control *c, struct v4l2_ctrl *ctrl)
 {
@@ -1647,6 +1654,14 @@ static void cur_to_new(struct v4l2_ctrl *ctrl)
ptr_to_ptr(ctrl, ctrl->p_cur, ctrl->p_new);
 }
 
+/* Copy the new value to the request value */
+static void new_to_req(struct v4l2_ctrl_ref *ref)
+{
+   if (!ref)
+   return;
+   ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -2971,7 +2986,8 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
struct v4l2_ctrl *ctrl);
struct v4l2_ctrl *master;
 
-   ctrl_to_user = def_value ? def_to_user : cur_to_user;
+   ctrl_to_user = def_value ? def_to_user :
+  (hdl->is_request ? NULL : cur_to_user);
 
if (helpers[i].mref == NULL)
continue;
@@ -2997,8 +3013,12 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
u32 idx = i;
 
do {
-   ret = ctrl_to_user(cs->controls + idx,
-  helpers[idx].ref->ctrl);
+   if (ctrl_to_user)
+   ret = ctrl_to_user(cs->controls + idx,
+   helpers[idx].ref->ctrl);
+   else
+   ret = req_to_user(cs->controls + idx,
+   helpers[idx].ref);
idx = helpers[idx].next;
} while (!ret && idx);
}
@@ -3271,7 +3291,16 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
} while (!ret && idx);
 
if (!ret)
-   ret = try_or_set_cluster(fh, master, set, 0);
+   ret = try_or_set_cluster(fh, master,
+!hdl->is_request && set, 0);
+   if (!ret && hdl->is_request && set) {
+   for (j = 0; j < master->ncontrols; j++) {
+   struct v4l2_ctrl_ref *ref =
+   find_ref(hdl, master->cluster[j]->id);
+
+   new_to_req(ref);
+   }
+   }
 
/* Copy the new values back to userspace. */
if (!ret) {
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 12/17] v4l2: add request API support

[Alexandre Courbot]

Add a v4l2 request entity data structure that takes care of storing the
request-related state of a V4L2 device ; in this case, its controls.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/Makefile   |  2 +-
 drivers/media/v4l2-core/v4l2-request.c | 54 ++
 include/media/v4l2-request.h   | 34 +
 3 files changed, 89 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-request.c
 create mode 100644 include/media/v4l2-request.h

diff --git a/drivers/media/v4l2-core/Makefile b/drivers/media/v4l2-core/Makefile
index 80de2cb9c476..1113dea1f4f9 100644
--- a/drivers/media/v4l2-core/Makefile
+++ b/drivers/media/v4l2-core/Makefile
@@ -15,7 +15,7 @@ obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o
 ifeq ($(CONFIG_TRACEPOINTS),y)
   videodev-objs += vb2-trace.o v4l2-trace.o
 endif
-videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o
+videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o v4l2-request.o
 
 obj-$(CONFIG_VIDEO_V4L2) += videodev.o
 obj-$(CONFIG_VIDEO_V4L2) += v4l2-common.o
diff --git a/drivers/media/v4l2-core/v4l2-request.c 
b/drivers/media/v4l2-core/v4l2-request.c
new file mode 100644
index ..7bc29d3cc332
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-request.c
@@ -0,0 +1,54 @@
+/*
+ * Media requests support for V4L2
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include 
+
+#include 
+
+struct media_request_entity_data *media_request_v4l2_entity_data_alloc(
+   struct v4l2_ctrl_handler *hdl)
+{
+   struct media_request_v4l2_entity_data *data;
+   int ret;
+
+   data = kzalloc(sizeof(*data), GFP_KERNEL);
+
+   ret = v4l2_ctrl_request_init(>ctrls);
+   if (ret) {
+   kfree(data);
+   return ERR_PTR(ret);
+   }
+
+   ret = v4l2_ctrl_request_clone(>ctrls, hdl, NULL);
+   if (ret) {
+   kfree(data);
+   return ERR_PTR(ret);
+   }
+
+   return >base;
+}
+EXPORT_SYMBOL_GPL(media_request_v4l2_entity_data_alloc);
+
+void
+media_request_v4l2_entity_data_free(struct media_request_entity_data *_data)
+{
+   struct media_request_v4l2_entity_data *data;
+
+   data = to_v4l2_entity_data(_data);
+
+   v4l2_ctrl_handler_free(>ctrls);
+   kfree(data);
+}
+EXPORT_SYMBOL_GPL(media_request_v4l2_entity_data_free);
diff --git a/include/media/v4l2-request.h b/include/media/v4l2-request.h
new file mode 100644
index ..db38dc5fc460
--- /dev/null
+++ b/include/media/v4l2-request.h
@@ -0,0 +1,34 @@
+/*
+ * Media requests support for V4L2
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MEDIA_REQUEST_V4L2_H
+#define _MEDIA_REQUEST_V4L2_H
+
+#include 
+#include 
+
+struct media_request_v4l2_entity_data {
+   struct media_request_entity_data base;
+
+   struct v4l2_ctrl_handler ctrls;
+};
+#define to_v4l2_entity_data(d) \
+   container_of(d, struct media_request_v4l2_entity_data, base)
+
+struct media_request_entity_data *media_request_v4l2_entity_data_alloc(
+   struct v4l2_ctrl_handler *hdl);
+void media_request_v4l2_entity_data_free(struct media_request_entity_data 
*data);
+
+#endif
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 11/17] v4l2-ctrls: add v4l2_ctrl_request_setup

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a helper function that can set controls from a request.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 71 
 include/media/v4l2-ctrls.h   |  2 +
 2 files changed, 73 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 4fa7adef7531..dccc27968df1 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1662,6 +1662,14 @@ static void new_to_req(struct v4l2_ctrl_ref *ref)
ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
 }
 
+/* Copy the request value to the new value */
+static void req_to_new(struct v4l2_ctrl_ref *ref)
+{
+   if (!ref)
+   return;
+   ptr_to_ptr(ref->ctrl, ref->p_req, ref->ctrl->p_new);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -3427,6 +3435,69 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, 
const char *s)
 }
 EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
 
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl)
+{
+   struct v4l2_ctrl_ref *ref;
+
+   if (!hdl)
+   return;
+
+   mutex_lock(hdl->lock);
+
+   list_for_each_entry(ref, >ctrl_refs, node)
+   ref->done = false;
+
+   list_for_each_entry(ref, >ctrl_refs, node) {
+   struct v4l2_ctrl *ctrl = ref->ctrl;
+   struct v4l2_ctrl *master = ctrl->cluster[0];
+   int i;
+
+   /* Skip if this control was already handled by a cluster. */
+   /* Skip button controls and read-only controls. */
+   if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
+   (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
+   continue;
+
+   v4l2_ctrl_lock(master);
+   for (i = 0; i < master->ncontrols; i++) {
+   if (master->cluster[i]) {
+   struct v4l2_ctrl_ref *r =
+   find_ref(hdl, master->cluster[i]->id);
+
+   req_to_new(r);
+   master->cluster[i]->is_new = 1;
+   r->done = true;
+   }
+   }
+   /*
+* For volatile autoclusters that are currently in auto mode
+* we need to discover if it will be set to manual mode.
+* If so, then we have to copy the current volatile values
+* first since those will become the new manual values (which
+* may be overwritten by explicit new values from this set
+* of controls).
+*/
+   if (master->is_auto && master->has_volatiles &&
+   !is_cur_manual(master)) {
+   s32 new_auto_val = *master->p_new.p_s32;
+
+   /*
+* If the new value == the manual value, then copy
+* the current volatile values.
+*/
+   if (new_auto_val == master->manual_mode_value)
+   update_from_auto_cluster(master);
+   }
+
+   try_or_set_cluster(NULL, master, true, 0);
+
+   v4l2_ctrl_unlock(master);
+   }
+
+   mutex_unlock(hdl->lock);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_setup);
+
 void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, 
void *priv)
 {
if (ctrl == NULL)
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index bddceb794961..922824dbd7db 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -259,6 +259,7 @@ struct v4l2_ctrl_ref {
struct v4l2_ctrl *ctrl;
struct v4l2_ctrl_helper *helper;
union v4l2_ctrl_ptr p_req;
+   bool done;
bool from_other_dev;
 };
 
@@ -1056,6 +1057,7 @@ int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl);
 int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
const struct v4l2_ctrl_handler *from,
bool (*filter)(const struct v4l2_ctrl *ctrl));
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_get(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_put(struct v4l2_ctrl_handler *hdl);
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 16/17] media: vim2m: add media device

[Alexandre Courbot]

Request API requires a media node. Add one to the vim2m driver so we can
use requests with it.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/vim2m.c | 24 
 1 file changed, 24 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 065483e62db4..e0eb60310717 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -140,6 +140,9 @@ static struct vim2m_fmt *find_format(struct v4l2_format *f)
 struct vim2m_dev {
struct v4l2_device  v4l2_dev;
struct video_device vfd;
+#ifdef CONFIG_MEDIA_CONTROLLER
+   struct media_device mdev;
+#endif
 
atomic_tnum_inst;
struct mutexdev_mutex;
@@ -1001,6 +1004,13 @@ static int vim2m_probe(struct platform_device *pdev)
 
spin_lock_init(>irqlock);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+   dev->mdev.dev = >dev;
+   strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
+   media_device_init(>mdev);
+   dev->v4l2_dev.mdev = >mdev;
+#endif
+
ret = v4l2_device_register(>dev, >v4l2_dev);
if (ret)
return ret;
@@ -1034,6 +1044,13 @@ static int vim2m_probe(struct platform_device *pdev)
goto err_m2m;
}
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+   /* Register the media device node */
+   ret = media_device_register(>mdev);
+   if (ret)
+   goto err_m2m;
+#endif
+
return 0;
 
 err_m2m:
@@ -1050,6 +1067,13 @@ static int vim2m_remove(struct platform_device *pdev)
struct vim2m_dev *dev = platform_get_drvdata(pdev);
 
v4l2_info(>v4l2_dev, "Removing " MEM2MEM_NAME);
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+   if (media_devnode_is_registered(dev->mdev.devnode))
+   media_device_unregister(>mdev);
+   media_device_cleanup(>mdev);
+#endif
+
v4l2_m2m_release(dev->m2m_dev);
del_timer_sync(>timer);
video_unregister_device(>vfd);
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 13/17] videodev2.h: add request_fd field to v4l2_ext_controls

[Alexandre Courbot]

Allow to specify a request to be used with the S_EXT_CTRLS and
G_EXT_CTRLS operations.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 include/uapi/linux/videodev2.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 4fd46ae8fad5..91cfe0cbd5c5 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1592,7 +1592,8 @@ struct v4l2_ext_controls {
};
__u32 count;
__u32 error_idx;
-   __u32 reserved[2];
+   __s32 request_fd;
+   __u32 reserved[1];
struct v4l2_ext_control *controls;
 };
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 15/17] v4l2: document the request API interface

[Alexandre Courbot]

Document how the request API can be used along with the existing V4L2
interface.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 Documentation/media/uapi/v4l/buffer.rst|  10 +-
 Documentation/media/uapi/v4l/common.rst|   1 +
 Documentation/media/uapi/v4l/request-api.rst   | 236 +
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst  |  16 +-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst   |  21 ++
 5 files changed, 280 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..9d082784081d 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,14 @@ struct v4l2_buffer
elements in the ``planes`` array. The driver will fill in the
actual number of valid elements in that array.
 * - __u32
-  - ``reserved2``
+  - ``request_fd``
   -
-  - A place holder for future extensions. Drivers and applications
-   must set this to 0.
+  - The file descriptor of the request associated with this buffer.
+   user-space can set this when calling :ref:`VIDIOC_QBUF`, and drivers
+   will return the request used when processing a buffer (if any) upon
+   :ref:`VIDIOC_DQBUF`.
+
+   A value of 0 means the buffer is not associated with any request.
 * - __u32
   - ``reserved``
   -
diff --git a/Documentation/media/uapi/v4l/common.rst 
b/Documentation/media/uapi/v4l/common.rst
index 13f2ed3fc5a6..a4aa0059d45a 100644
--- a/Documentation/media/uapi/v4l/common.rst
+++ b/Documentation/media/uapi/v4l/common.rst
@@ -44,3 +44,4 @@ applicable to all devices.
 crop
 selection-api
 streaming-par
+request-api
diff --git a/Documentation/media/uapi/v4l/request-api.rst 
b/Documentation/media/uapi/v4l/request-api.rst
new file mode 100644
index ..4c61a0dbe3a9
--- /dev/null
+++ b/Documentation/media/uapi/v4l/request-api.rst
@@ -0,0 +1,236 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _media-request-api:
+
+Request API
+===
+
+The Request API has been designed to allow V4L2 to deal with requirements of
+modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android Codec
+v2). One such requirement is the ability for devices belonging to the same
+pipeline to reconfigure and collaborate closely on a per-frame basis. Another 
is
+efficient support of stateless codecs, which need per-frame controls to be set
+asynchronously in order to be efficiently used.
+
+Supporting these features without the Request API is possible but terribly
+inefficient: user-space would have to flush all activity on the media pipeline,
+reconfigure it for the next frame, queue the buffers to be processed with that
+configuration, and wait until they are all available for dequeing before
+considering the next frame. This defeats the purpose of having buffer queues
+since in practice only one buffer would be queued at a time.
+
+The Request API allows a specific configuration of the pipeline (media
+controller topology + controls for each device) to be associated with specific
+buffers. The parameters are applied by each participating device as buffers
+associated to a request flow in. This allows user-space to schedule several
+tasks ("requests") with different parameters in advance, knowing that the
+parameters will be applied when needed to get the expected result. Controls
+values at the time of request completion are also available for reading.
+
+Usage
+=
+
+The Request API is used on top of standard media controller and V4L2 calls,
+which are augmented with an extra ``request_fd`` parameter. All operations on
+requests themselves are performed using the command parameter of the
+:c:func:`MEDIA_IOC_REQUEST_CMD` ioctl.
+
+Request Allocation
+--
+
+User-space allocates requests using the ``MEDIA_REQ_CMD_ALLOC`` command on
+an opened media device. This returns a file descriptor representing the
+request. Typically, several such requests will be allocated.
+
+Request Preparation
+---
+
+Standard V4L2 ioctls can then receive a request file descriptor to express the
+fact that the ioctl is part of said request, and is not to be applied
+immediately. V4L2 ioctls supporting this are :c:func:`VIDIOC_S_EXT_CTRLS` and
+:c:func:`VIDIOC_QBUF`. Controls set with a request parameter are stored instead
+of being immediately applied, and queued buffers will block the buffer queue
+until the request becomes active.
+
+RFC Note: currently several buffers can be queued to the same queue with the
+same request. The request parameters will be only be applied when processing
+the first buffer. Does it make more sense to allow at most one buffer per
+request per queue instead?
+
+Request Submission
+--
+
+Once the parameters and buffer

[RFCv3 17/17] media: vim2m: add request support

[Alexandre Courbot]

Set the necessary ops for supporting requests in vim2m.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/vim2m.c | 55 ++
 1 file changed, 55 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index e0eb60310717..96dba60d3c74 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -30,6 +30,9 @@
 #include 
 #include 
 #include 
+#include 
+#include 
+#include 
 
 MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
 MODULE_AUTHOR("Pawel Osciak, <pa...@osciak.com>");
@@ -370,6 +373,28 @@ static void job_abort(void *priv)
ctx->aborting = 1;
 }
 
+static int apply_request_params(struct media_request *req,
+struct vim2m_ctx *ctx)
+{
+   struct media_request_entity_data *_data;
+   struct media_request_v4l2_entity_data *data;
+
+   if (!req)
+   return -EINVAL;
+
+   _data = media_request_get_entity_data(req, >dev->vfd.entity,
+ >fh);
+   if (WARN_ON(!_data))
+   return -EINVAL;
+   data = container_of(_data, struct media_request_v4l2_entity_data, base);
+
+   v4l2_ctrl_request_setup(>ctrls);
+
+   data->base.applied = true;
+
+   return 0;
+}
+
 /* device_run() - prepares and starts the device
  *
  * This simulates all the immediate preparations required before starting
@@ -381,12 +406,22 @@ static void device_run(void *priv)
struct vim2m_ctx *ctx = priv;
struct vim2m_dev *dev = ctx->dev;
struct vb2_v4l2_buffer *src_buf, *dst_buf;
+   int ret;
 
src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
 
+   WARN_ON(dst_buf->vb2_buf.request != NULL);
+
+   /* Apply request if needed */
+   ret = apply_request_params(src_buf->vb2_buf.request, ctx);
+   WARN_ON(ret);
+
device_process(ctx, src_buf, dst_buf);
 
+   /* Inform user about which request produced the destination buffer */
+   dst_buf->request_fd = src_buf->request_fd;
+
/* Run a timer, which simulates a hardware irq  */
schedule_irq(dev, ctx->transtime);
 }
@@ -841,6 +876,7 @@ static int queue_init(void *priv, struct vb2_queue *src_vq, 
struct vb2_queue *ds
src_vq->mem_ops = _vmalloc_memops;
src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
src_vq->lock = >dev->dev_mutex;
+   src_vq->allow_requests = true;
 
ret = vb2_queue_init(src_vq);
if (ret)
@@ -992,6 +1028,19 @@ static const struct v4l2_m2m_ops m2m_ops = {
.job_abort  = job_abort,
 };
 
+struct media_request_entity_data *vim2m_entity_data_alloc(struct media_entity 
*entity,
+ void *fh)
+{
+   struct vim2m_ctx *ctx = container_of(fh, struct vim2m_ctx, fh);
+
+   return media_request_v4l2_entity_data_alloc(>hdl);
+}
+
+static const struct media_entity_request_ops vim2m_entity_req_ops = {
+   .data_alloc = vim2m_entity_data_alloc,
+   .data_free  = media_request_v4l2_entity_data_free,
+};
+
 static int vim2m_probe(struct platform_device *pdev)
 {
struct vim2m_dev *dev;
@@ -1006,6 +1055,9 @@ static int vim2m_probe(struct platform_device *pdev)
 
 #ifdef CONFIG_MEDIA_CONTROLLER
dev->mdev.dev = >dev;
+   dev->mdev.req_mgr = media_request_mgr_alloc(>mdev);
+   if (IS_ERR(dev->mdev.req_mgr))
+   return PTR_ERR(dev->mdev.req_mgr);
strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
media_device_init(>mdev);
dev->v4l2_dev.mdev = >mdev;
@@ -1030,6 +1082,9 @@ static int vim2m_probe(struct platform_device *pdev)
}
 
video_set_drvdata(vfd, dev);
+#ifdef CONFIG_MEDIA_CONTROLLER
+   dev->vfd.entity.req_ops = _entity_req_ops;
+#endif
snprintf(vfd->name, sizeof(vfd->name), "%s", vim2m_videodev.name);
v4l2_info(>v4l2_dev,
"Device registered as /dev/video%d\n", vfd->num);
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 14/17] v4l2-ctrls: support requests in EXT_CTRLS ioctls

[Alexandre Courbot]

Read and use the request_fd field of struct v4l2_ext_controls to apply
VIDIOC_G_EXT_CTRLS or VIDIOC_S_EXT_CTRLS to a request when asked by
userspace.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 36 
 1 file changed, 36 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index 235acdde3111..cbaefcad9694 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -30,6 +30,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -2158,6 +2159,24 @@ static int v4l_g_ext_ctrls(const struct v4l2_ioctl_ops 
*ops,
test_bit(V4L2_FL_USES_V4L2_FH, >flags) ? fh : NULL;
 
p->error_idx = p->count;
+
+   if (p->request_fd > 0) {
+   struct media_request *req = NULL;
+   struct media_request_entity_data *_data;
+   struct media_request_v4l2_entity_data *data;
+   int ret;
+
+   req = check_request(p->request_fd, file, fh, &_data);
+   if (IS_ERR(req))
+   return PTR_ERR(req);
+   data = to_v4l2_entity_data(_data);
+
+   ret = v4l2_g_ext_ctrls(>ctrls, p);
+
+   media_request_put(req);
+   return ret;
+   }
+
if (vfh && vfh->ctrl_handler)
return v4l2_g_ext_ctrls(vfh->ctrl_handler, p);
if (vfd->ctrl_handler)
@@ -2177,6 +2196,23 @@ static int v4l_s_ext_ctrls(const struct v4l2_ioctl_ops 
*ops,
test_bit(V4L2_FL_USES_V4L2_FH, >flags) ? fh : NULL;
 
p->error_idx = p->count;
+   if (p->request_fd > 0) {
+   struct media_request *req = NULL;
+   struct media_request_entity_data *_data;
+   struct media_request_v4l2_entity_data *data;
+   int ret;
+
+   req = check_request(p->request_fd, file, fh, &_data);
+   if (IS_ERR(req))
+   return PTR_ERR(req);
+   data = to_v4l2_entity_data(_data);
+
+   ret = v4l2_s_ext_ctrls(vfh, >ctrls, p);
+
+   media_request_put(req);
+   return ret;
+   }
+
if (vfh && vfh->ctrl_handler)
return v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler, p);
if (vfd->ctrl_handler)
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 09/17] v4l2-ctrls: use ref in helper instead of ctrl

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

The next patch needs the reference to a control instead of the
control itself, so change struct v4l2_ctrl_helper accordingly.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index c692a6d925c6..9090a49eef91 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -37,8 +37,8 @@
 struct v4l2_ctrl_helper {
/* Pointer to the control reference of the master control */
struct v4l2_ctrl_ref *mref;
-   /* The control corresponding to the v4l2_ext_control ID field. */
-   struct v4l2_ctrl *ctrl;
+   /* The control ref corresponding to the v4l2_ext_control ID field. */
+   struct v4l2_ctrl_ref *ref;
/* v4l2_ext_control index of the next control belonging to the
   same cluster, or 0 if there isn't any. */
u32 next;
@@ -2856,6 +2856,7 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler 
*hdl,
ref = find_ref_lock(hdl, id);
if (ref == NULL)
return -EINVAL;
+   h->ref = ref;
ctrl = ref->ctrl;
if (ctrl->flags & V4L2_CTRL_FLAG_DISABLED)
return -EINVAL;
@@ -2878,7 +2879,6 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler 
*hdl,
}
/* Store the ref to the master control of the cluster */
h->mref = ref;
-   h->ctrl = ctrl;
/* Initially set next to 0, meaning that there is no other
   control in this helper array belonging to the same
   cluster */
@@ -2963,7 +2963,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
cs->error_idx = cs->count;
 
for (i = 0; !ret && i < cs->count; i++)
-   if (helpers[i].ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
+   if (helpers[i].ref->ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
ret = -EACCES;
 
for (i = 0; !ret && i < cs->count; i++) {
@@ -2998,7 +2998,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
 
do {
ret = ctrl_to_user(cs->controls + idx,
-  helpers[idx].ctrl);
+  helpers[idx].ref->ctrl);
idx = helpers[idx].next;
} while (!ret && idx);
}
@@ -3137,7 +3137,7 @@ static int validate_ctrls(struct v4l2_ext_controls *cs,
 
cs->error_idx = cs->count;
for (i = 0; i < cs->count; i++) {
-   struct v4l2_ctrl *ctrl = helpers[i].ctrl;
+   struct v4l2_ctrl *ctrl = helpers[i].ref->ctrl;
union v4l2_ctrl_ptr p_new;
 
cs->error_idx = i;
@@ -3249,7 +3249,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
do {
/* Check if the auto control is part of the
   list, and remember the new value. */
-   if (helpers[tmp_idx].ctrl == master)
+   if (helpers[tmp_idx].ref->ctrl == master)
new_auto_val = 
cs->controls[tmp_idx].value;
tmp_idx = helpers[tmp_idx].next;
} while (tmp_idx);
@@ -3262,7 +3262,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
/* Copy the new caller-supplied control values.
   user_to_new() sets 'is_new' to 1. */
do {
-   struct v4l2_ctrl *ctrl = helpers[idx].ctrl;
+   struct v4l2_ctrl *ctrl = helpers[idx].ref->ctrl;
 
ret = user_to_new(cs->controls + idx, ctrl);
if (!ret && ctrl->is_ptr)
@@ -3278,7 +3278,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
idx = i;
do {
ret = new_to_user(cs->controls + idx,
-   helpers[idx].ctrl);
+   helpers[idx].ref->ctrl);
idx = helpers[idx].next;
} while (!ret && idx);
}
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 03/17] media: videobuf2: add support for requests

[Alexandre Courbot]

Make vb2 aware of requests. Drivers can specify whether a given queue
can accept requests or not. Queues that accept requests will block on a
buffer that is part of a request until that request is submitted.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 133 ++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  28 -
 include/media/videobuf2-core.h  |  15 ++-
 3 files changed, 168 insertions(+), 8 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c 
b/drivers/media/common/videobuf2/videobuf2-core.c
index f7109f827f6e..c1b9ccbdecb3 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -28,6 +28,7 @@
 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -930,6 +931,17 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
vb->state = state;
}
atomic_dec(>owned_by_drv_count);
+   if (vb->request) {
+   struct media_request *req = vb->request;
+
+   if (atomic_dec_and_test(>buf_cpt))
+   media_request_complete(vb->request);
+
+   /* release reference acquired during qbuf */
+   vb->request = NULL;
+   media_request_put(req);
+   }
+
spin_unlock_irqrestore(>done_lock, flags);
 
trace_vb2_buf_done(q, vb);
@@ -1306,6 +1318,53 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+/*
+ * vb2_check_buf_req_status() - Validate request state of a buffer
+ * @vb:buffer to check
+ *
+ * Returns true if a buffer is ready to be passed to the driver request-wise.
+ * This means that neither this buffer nor any previously-queued buffer is
+ * associated to a request that is not yet submitted.
+ *
+ * If this function returns false, then the buffer shall not be passed to its
+ * driver since the request state is not completely built yet. In that case,
+ * this function will register a notifier to be called when the request is
+ * submitted and the queue can be unblocked.
+ *
+ * This function must be called with req_lock held.
+ */
+static bool vb2_check_buf_req_status(struct vb2_buffer *vb)
+{
+   struct media_request *req = vb->request;
+   struct vb2_queue *q = vb->vb2_queue;
+   int ret = false;
+
+   mutex_lock(>req_lock);
+
+   if (!req) {
+   ret = !q->waiting_req;
+   goto done;
+   }
+
+   mutex_lock(>lock);
+   if (req->state == MEDIA_REQUEST_STATE_SUBMITTED) {
+   mutex_unlock(>lock);
+   ret = !q->waiting_req;
+   goto done;
+   }
+
+   if (!q->waiting_req) {
+   q->waiting_req = true;
+   atomic_notifier_chain_register(>submit_notif,
+  >req_blk);
+   }
+   mutex_unlock(>lock);
+
+done:
+   mutex_unlock(>req_lock);
+   return ret;
+}
+
 /*
  * vb2_start_streaming() - Attempt to start streaming.
  * @q: videobuf2 queue
@@ -1326,8 +1385,11 @@ static int vb2_start_streaming(struct vb2_queue *q)
 * If any buffers were queued before streamon,
 * we can now pass them to driver for processing.
 */
-   list_for_each_entry(vb, >queued_list, queued_entry)
+   list_for_each_entry(vb, >queued_list, queued_entry) {
+   if (!vb2_check_buf_req_status(vb))
+   break;
__enqueue_in_driver(vb);
+   }
 
/* Tell the driver to start streaming */
q->start_streaming_called = 1;
@@ -1369,7 +1431,46 @@ static int vb2_start_streaming(struct vb2_queue *q)
return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+/*
+ * vb2_unblock_requests() - unblock a queue waiting for a request submission
+ * @nb:notifier block that has been registered
+ * @action:unused
+ * @data:  request that has been submitted
+ *
+ * This is a callback function that is registered when
+ * vb2_check_buf_req_status() returns false. It is invoked when the request
+ * blocking the queue has been submitted. This means its buffers (and all
+ * following valid buffers) can be passed to drivers.
+ */
+static int vb2_unblock_requests(struct notifier_block *nb, unsigned long 
action,
+   void *data)
+{
+   struct vb2_queue *q = container_of(nb, struct vb2_queue, req_blk);
+   struct media_request *req = data;
+   struct vb2_buffer *vb;
+   bool found_request = false;
+
+   mutex_lock(>req_lock);
+   atomic_notifier_chain_unregister(>submit_notif, >req_blk);
+   q->waiting_req = false;
+   mutex_unlock(>req_lock);
+
+   list_fo

Re: Compliance tests for new public API features

[Alexandre Courbot]

Hi Hans,

On Sun, Feb 4, 2018 at 10:30 PM, Hans Verkuil  wrote:
> Hi Gustavo, Alexandre,
>
> As you may have seen I have been extending the v4l2-compliance utility with 
> tests
> for v4l-subdevX and mediaX devices. In the process of doing that I promptly
> found a bunch of bugs. Mostly little things that are easy to fix, but much
> harder to find without proper tests.
>
> You are both working on new API additions and I want to give a heads-up that
> I want to have patches for v4l2-compliance that test the new additions to a
> reasonable extent.
>
> I say 'reasonable' because I suspect that e.g. in-fence support might be hard
> to test. But out-fences can definitely be tested and for in-fences you can
> at minimum test what happens when you give it an invalid file descriptor.
>
> For the request API is it obviously too early to start writing tests, but
> as the API crystallizes you may consider starting to work on it.
>
> I've decided to be more strict about this based on my experiences. I'm more
> than happy to assist and give advice, especially since this is a bit of a late
> notice, particularly for Gustavo.
>
> But every time we skip this step it bites us in the ass later on.
>
> It is also highly recommended to add fence/request support to the vivid and
> vim2m drivers. It makes it much easier to find regressions in the API due to
> future changes since you don't need 'real' hardware for these drivers.
>
> Again my apologies for the late notice, but it is better to make these tests
> now while you are working on the new feature, rather than doing it much later
> and finding all sorts of gremlins in the API.

I completely agree with your reasoning and will consider updating
v4l2-compliance as soon as we have something stable on the user-facing
side (which hopefully should be close by now).

Alex.

[PATCH] media: v4l2_fh.h: add missing kconfig.h include

[Alexandre Courbot]

v4l2_fh.h uses the IS_ENABLED() macro and thus should include kconfig.h.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 include/media/v4l2-fh.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
index 62633e7d2630..ea73fef8bdc0 100644
--- a/include/media/v4l2-fh.h
+++ b/include/media/v4l2-fh.h
@@ -22,6 +22,7 @@
 #define V4L2_FH_H
 
 #include 
+#include 
 #include 
 #include 
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[PATCH] v4l: vidioc-prepare-buf.rst: fix link to VIDIOC_QBUF

[Alexandre Courbot]

The description for VIDIOC_PREPARE_BUF results in the following
sentence: "...before actually enqueuing it, using the ioctl VIDIOC_QBUF,
VIDIOC_DQBUF ioctl...".

The intent is to only refer to VIDIOC_QBUF though, so fix this.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 Documentation/media/uapi/v4l/vidioc-prepare-buf.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-prepare-buf.rst 
b/Documentation/media/uapi/v4l/vidioc-prepare-buf.rst
index 70687a86ae38..49f9f4c181de 100644
--- a/Documentation/media/uapi/v4l/vidioc-prepare-buf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-prepare-buf.rst
@@ -34,7 +34,7 @@ Description
 
 Applications can optionally call the :ref:`VIDIOC_PREPARE_BUF` ioctl to
 pass ownership of the buffer to the driver before actually enqueuing it,
-using the :ref:`VIDIOC_QBUF` ioctl, and to prepare it for future I/O. Such
+using the :ref:`VIDIOC_QBUF ` ioctl, and to prepare it for future 
I/O. Such
 preparations may include cache invalidation or cleaning. Performing them
 in advance saves time during the actual I/O. In case such cache
 operations are not required, the application can use one of
-- 
2.16.0.rc1.238.g530d649a79-goog

[PATCH] media: media-types.rst: fix typo

[Alexandre Courbot]

with -> which

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 Documentation/media/uapi/mediactl/media-types.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/media/uapi/mediactl/media-types.rst 
b/Documentation/media/uapi/mediactl/media-types.rst
index 8d64b0c06ebc..932823e3cca8 100644
--- a/Documentation/media/uapi/mediactl/media-types.rst
+++ b/Documentation/media/uapi/mediactl/media-types.rst
@@ -26,7 +26,7 @@ Types and flags used to represent the media graph elements
  ``MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN``
 
-  Unknown entity. That generally indicates that a driver didn't
- initialize properly the entity, with is a Kernel bug
+ initialize properly the entity, which is a Kernel bug
 
 -  .. row 2
 
-- 
2.16.0.rc1.238.g530d649a79-goog

Re: [RFCv2 05/17] media: Document the media request API

[Alexandre Courbot]

Hi Randy,

On Fri, Feb 2, 2018 at 3:14 AM, Randy Dunlap <rdun...@infradead.org> wrote:
> On 01/31/2018 02:24 AM, Alexandre Courbot wrote:
>> From: Laurent Pinchart <laurent.pinchart+rene...@ideasonboard.com>
>>
>> The media request API is made of a new ioctl to implement request
>> management. Document it.
>>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart+rene...@ideasonboard.com>
>> [acour...@chromium.org: adapt for newest API]
>> Signed-off-by: Alexandre Courbot <acour...@chromium.org>
>> ---
>>  Documentation/media/uapi/mediactl/media-funcs.rst  |   1 +
>>  .../media/uapi/mediactl/media-ioc-request-cmd.rst  | 141 
>> +
>>  2 files changed, 142 insertions(+)
>>  create mode 100644 
>> Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
>>
>> diff --git a/Documentation/media/uapi/mediactl/media-funcs.rst 
>> b/Documentation/media/uapi/mediactl/media-funcs.rst
>> index 076856501cdb..e3a45d82ffcb 100644
>> --- a/Documentation/media/uapi/mediactl/media-funcs.rst
>> +++ b/Documentation/media/uapi/mediactl/media-funcs.rst
>> @@ -15,4 +15,5 @@ Function Reference
>>  media-ioc-g-topology
>>  media-ioc-enum-entities
>>  media-ioc-enum-links
>> +media-ioc-request-cmd
>>  media-ioc-setup-link
>> diff --git a/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst 
>> b/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
>> new file mode 100644
>> index ..723b422afcce
>> --- /dev/null
>> +++ b/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
>> @@ -0,0 +1,141 @@
>> +.. -*- coding: utf-8; mode: rst -*-
>> +
>> +.. _media_ioc_request_cmd:
>> +
>> +***
>> +ioctl MEDIA_IOC_REQUEST_CMD
>> +***
>> +
>> +Name
>> +
>> +
>> +MEDIA_IOC_REQUEST_CMD - Manage media device requests
>> +
>> +
>> +Synopsis
>> +
>> +
>> +.. c:function:: int ioctl( int fd, MEDIA_IOC_REQUEST_CMD, struct 
>> media_request_cmd *argp )
>> +:name: MEDIA_IOC_REQUEST_CMD
>> +
>> +
>> +Arguments
>> +=
>> +
>> +``fd``
>> +File descriptor returned by :ref:`open() `.
>> +
>> +``argp``
>> +
>> +
>> +Description
>> +===
>> +
>> +The MEDIA_IOC_REQUEST_CMD ioctl allow applications to manage media device
>
>allows
>
>> +requests. A request is an object that can group media device configuration
>> +parameters, including subsystem-specific parameters, in order to apply all 
>> the
>> +parameters atomically. Applications are responsible for allocating and
>> +deleting requests, filling them with configuration parameters submitting 
>> them.
>
> and 
> submitting them.
>
>> +
>> +Request operations are performed by calling the MEDIA_IOC_REQUEST_CMD ioctl
>> +with a pointer to a struct :c:type:`media_request_cmd` with the cmd field 
>> set
>> +to the appropriate command. :ref:`media-request-command` lists the commands
>> +supported by the ioctl.
>> +
>> +The struct :c:type:`media_request_cmd` request field contains the file
>> +descriptorof the request on which the command operates. For the
>
>descriptor of
>
>> +``MEDIA_REQ_CMD_ALLOC`` command the field is set to zero by applications and
>> +filled by the driver. For all other commands the field is set by 
>> applications
>> +and left untouched by the driver.
>> +
>> +To allocate a new request applications use the ``MEDIA_REQ_CMD_ALLOC``
>> +command. The driver will allocate a new request and return its FD in the
>> +request field. After allocation, the request is "empty", which means that it
>> +does not hold any state of its own, and that the hardware's state will not 
>> be
>> +affected by it unless it is passed as argument to V4L2 or media controller
>> +commands.
>> +
>> +Requests are reference-counted. A newly allocated request is referenced
>> +by the returned file descriptor, and can be later referenced by
>> +subsystem-specific operations. Requests will thus be automatically deleted
>> +when they're not longer used after the returned file descriptor is closed.
>
>  no longer
>
>> +
>> +If a request isn't needed applications can delete it by calling ``close()``
>> +on it. The driver will drop the file handle reference. The 

[RFCv2 02/17] videodev2.h: Add request_fd field to v4l2_buffer

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

When queuing buffers allow for passing the request that should
be associated with this buffer.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
[acour...@chromium.org: make request ID 32-bit]
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/usb/cpia2/cpia2_v4l.c   | 2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 7 ---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 4 ++--
 drivers/media/v4l2-core/videobuf2-v4l2.c  | 3 ++-
 include/media/videobuf2-v4l2.h| 2 ++
 include/uapi/linux/videodev2.h| 3 ++-
 6 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
b/drivers/media/usb/cpia2/cpia2_v4l.c
index 3dedd83f0b19..7acb6807b306 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct 
v4l2_buffer *buf)
buf->sequence = cam->buffers[buf->index].seq;
buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
buf->length = cam->frame_size;
-   buf->reserved2 = 0;
+   buf->request_fd = 0;
buf->reserved = 0;
memset(>timecode, 0, sizeof(buf->timecode));
 
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 821f2aa299ae..7e4440950c76 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -370,7 +370,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __u32   request_fd;
__u32   reserved;
 };
 
@@ -438,7 +438,8 @@ static int get_v4l2_buffer32(struct v4l2_buffer *kp, struct 
v4l2_buffer32 __user
get_user(kp->type, >type) ||
get_user(kp->flags, >flags) ||
get_user(kp->memory, >memory) ||
-   get_user(kp->length, >length))
+   get_user(kp->length, >length) ||
+   get_user(kp->request_fd, >request_fd))
return -EFAULT;
 
if (V4L2_TYPE_IS_OUTPUT(kp->type))
@@ -533,7 +534,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct 
v4l2_buffer32 __user
put_user(kp->timestamp.tv_usec, >timestamp.tv_usec) ||
copy_to_user(>timecode, >timecode, sizeof(struct 
v4l2_timecode)) ||
put_user(kp->sequence, >sequence) ||
-   put_user(kp->reserved2, >reserved2) ||
+   put_user(kp->request_fd, >request_fd) ||
put_user(kp->reserved, >reserved) ||
put_user(kp->length, >length))
return -EFAULT;
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index ec4ecd5aa8bf..fdd2f784c264 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool 
write_only)
const struct v4l2_plane *plane;
int i;
 
-   pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, 
field=%s, sequence=%d, memory=%s",
+   pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request_fd=%u, 
flags=0x%08x, field=%s, sequence=%d, memory=%s",
p->timestamp.tv_sec / 3600,
(int)(p->timestamp.tv_sec / 60) % 60,
(int)(p->timestamp.tv_sec % 60),
(long)p->timestamp.tv_usec,
p->index,
-   prt_names(p->type, v4l2_type_names),
+   prt_names(p->type, v4l2_type_names), p->request_fd,
p->flags, prt_names(p->field, v4l2_field_names),
p->sequence, prt_names(p->memory, v4l2_memory_names));
 
diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c 
b/drivers/media/v4l2-core/videobuf2-v4l2.c
index 0c0669976bdc..0f8edbdebe30 100644
--- a/drivers/media/v4l2-core/videobuf2-v4l2.c
+++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->request_fd = vbuf->request_fd;
b->reserved = 0;
 
if (q->is_multiplanar) {
@@ -320,6 +320,7 @@ static int __fill_vb2_buffer(struct vb2_buffer *vb,
}
vb->timestamp = 0;
vbuf->sequence = 0;
+   vbuf->request_fd = b->reques

[RFCv2 04/17] media: vb2: add support for requests in QBUF ioctl

[Alexandre Courbot]

Support the request argument of the QBUF ioctl.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 83 +++-
 1 file changed, 82 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index fdd2f784c264..3b44f1fe4f23 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -29,6 +29,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -965,6 +966,71 @@ static int check_fmt(struct file *file, enum v4l2_buf_type 
type)
return -EINVAL;
 }
 
+/*
+ * Validate that a given request can be used during an ioctl.
+ *
+ * When using the request API, request file descriptors must be matched against
+ * the actual request object. User-space can pass any file descriptor, so we
+ * need to make sure the call is valid before going further.
+ *
+ * This function looks up the request and associated data and performs the
+ * following sanity checks:
+ *
+ * * Make sure that the entity belongs to the media_device managing the passed
+ *   request,
+ * * Make sure that the entity data (if any) is associated to the current file
+ *   handler.
+ *
+ * This function returns a pointer to the valid request, or and error code in
+ * case of failure. When successful, a reference to the request is acquired and
+ * must be properly released by the caller. If _data is not NULL, then it is
+ * filled with a pointer to the corresponding entity data.
+ */
+#ifdef CONFIG_MEDIA_CONTROLLER
+static struct media_request *
+check_request(int request, struct file *file, void *fh, struct 
media_request_entity_data **_data)
+{
+   struct media_request *req = NULL;
+   struct video_device *vfd = video_devdata(file);
+   struct v4l2_fh *vfh =
+   test_bit(V4L2_FL_USES_V4L2_FH, >flags) ? fh : NULL;
+   struct media_entity *entity = >entity;
+   struct media_request_entity_data *data;
+
+   if (!entity)
+   return ERR_PTR(-EINVAL);
+
+   req = media_request_get_from_fd(request);
+   if (!req)
+   return ERR_PTR(-EINVAL);
+
+   /* Validate that the entity belongs to the correct media_device */
+   if (vfd->v4l2_dev->mdev->req_mgr != req->mgr) {
+   media_request_put(req);
+   return ERR_PTR(-EINVAL);
+   }
+
+   /* Validate that the entity's data belongs to the correct fh */
+   data = media_request_get_entity_data(req, entity, vfh);
+   if (IS_ERR(data)) {
+   media_request_put(req);
+   return ERR_PTR(PTR_ERR(data));
+   }
+
+   if (_data)
+   *_data = data;
+
+   return req;
+}
+#else /* CONFIG_MEDIA_CONTROLLER */
+static struct media_request *
+check_request(int request, struct file *file, void *fh)
+{
+   return ERR_PTR(-ENOSYS);
+}
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 static void v4l_sanitize_format(struct v4l2_format *fmt)
 {
unsigned int offset;
@@ -1902,10 +1968,25 @@ static int v4l_querybuf(const struct v4l2_ioctl_ops 
*ops,
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
struct file *file, void *fh, void *arg)
 {
+   struct media_request *req = NULL;
struct v4l2_buffer *p = arg;
int ret = check_fmt(file, p->type);
 
-   return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+   if (ret)
+   return ret;
+
+   if (p->request_fd > 0) {
+   req = check_request(p->request_fd, file, fh, NULL);
+   if (IS_ERR(req))
+   return PTR_ERR(req);
+   }
+
+   ret = ops->vidioc_qbuf(file, fh, p);
+
+   if (req)
+   media_request_put(req);
+
+   return ret;
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 01/17] media: add request API core and UAPI

[Alexandre Courbot]

The request API provides a way to group buffers and device parameters
into units of work to be queued and executed. This patch introduces the
UAPI and core framework.

This patch is based on the previous work by Laurent Pinchart. The core
has changed considerably, but the UAPI is mostly untouched.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/Makefile   |   3 +-
 drivers/media/media-device.c |   7 +
 drivers/media/media-request-mgr.c| 105 
 drivers/media/media-request.c| 311 +++
 drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
 include/media/media-device.h |   3 +
 include/media/media-entity.h |   9 +
 include/media/media-request-mgr.h|  73 
 include/media/media-request.h| 186 +
 include/uapi/linux/media.h   |  10 ++
 10 files changed, 707 insertions(+), 2 deletions(-)
 create mode 100644 drivers/media/media-request-mgr.c
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request-mgr.h
 create mode 100644 include/media/media-request.h

diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 594b462ddf0e..06c43ddb52ea 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -3,7 +3,8 @@
 # Makefile for the kernel multimedia device drivers.
 #
 
-media-objs := media-device.o media-devnode.o media-entity.o
+media-objs := media-device.o media-devnode.o media-entity.o \
+  media-request.o media-request-mgr.o
 
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index e79f72b8b858..024ee81a8334 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -32,6 +32,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
@@ -407,6 +409,7 @@ static const struct media_ioctl_info ioctl_info[] = {
MEDIA_IOC(ENUM_LINKS, media_device_enum_links, 
MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(SETUP_LINK, media_device_setup_link, 
MEDIA_IOC_FL_GRAPH_MUTEX),
MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, 
MEDIA_IOC_FL_GRAPH_MUTEX),
+   MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
 };
 
 static long media_device_ioctl(struct file *filp, unsigned int cmd,
@@ -688,6 +691,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
 
 void media_device_cleanup(struct media_device *mdev)
 {
+   if (mdev->req_mgr) {
+   media_request_mgr_free(mdev->req_mgr);
+   mdev->req_mgr = NULL;
+   }
ida_destroy(>entity_internal_idx);
mdev->entity_internal_idx_max = 0;
media_graph_walk_cleanup(>pm_count_walk);
diff --git a/drivers/media/media-request-mgr.c 
b/drivers/media/media-request-mgr.c
new file mode 100644
index ..686e877a884b
--- /dev/null
+++ b/drivers/media/media-request-mgr.c
@@ -0,0 +1,105 @@
+/*
+ * Generic request manager implementation.
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+static struct media_request *
+media_request_alloc(struct media_request_mgr *mgr)
+{
+   struct media_request *req;
+
+   req = kzalloc(sizeof(*req), GFP_KERNEL);
+   if (!req)
+   return ERR_PTR(-ENOMEM);
+
+   req->mgr = mgr;
+   req->state = MEDIA_REQUEST_STATE_IDLE;
+   kref_init(>kref);
+   INIT_LIST_HEAD(>data);
+   init_waitqueue_head(>complete_wait);
+   ATOMIC_INIT_NOTIFIER_HEAD(>submit_notif);
+   mutex_init(>lock);
+
+   mutex_lock(>mutex);
+   req->id = ++mgr->req_id;
+   list_add_tail(>list, >requests);
+   mutex_unlock(>mutex);
+
+   return req;
+}
+
+static void media_request_free(struct media_request *req)
+{
+   struct media_request_mgr *mgr = req->mgr;
+   struct media_request_entity_data *data, *next;
+
+   mutex_lock(>mutex);
+   list_del(>list);
+   mutex_unlock(>mutex);
+
+   list_for_each_entry_safe(data, next, >data, list) {
+   list_del(>list);
+   data->entity->req_ops->data_free(data);
+   }
+
+   kfree(req);
+}
+
+void media_request_mgr_free(struct media_request_mgr *mgr)
+{
+   struct media_device *mdev = mgr->mdev;
+
+   /* Just a sanity check - we should have no remaining r

[RFCv2 08/17] v4l2-ctrls: add core request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add the four core request functions:

v4l2_ctrl_request_init() initializes a new (empty) request.
v4l2_ctrl_request_clone() resets a request based on another request
(or clears it if that request is NULL).
v4l2_ctrl_request_get(): increase refcount
v4l2_ctrl_request_put(): decrease refcount and delete if it reaches 0.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
[acour...@chromium.org: turn v4l2_ctrl_request_alloc into init function]
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 106 ++-
 include/media/v4l2-ctrls.h   |   7 +++
 2 files changed, 110 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 9d99679b543c..ca4c320f64c9 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1868,6 +1868,7 @@ EXPORT_SYMBOL(v4l2_ctrl_find);
 /* Allocate a new v4l2_ctrl_ref and hook it into the handler. */
 static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
   struct v4l2_ctrl *ctrl,
+  struct v4l2_ctrl_ref **ctrl_ref,
   bool from_other_dev)
 {
struct v4l2_ctrl_ref *ref;
@@ -1875,6 +1876,10 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
u32 id = ctrl->id;
u32 class_ctrl = V4L2_CTRL_ID2WHICH(id) | 1;
int bucket = id % hdl->nr_of_buckets;   /* which bucket to use */
+   unsigned int sz_extra = 0;
+
+   if (ctrl_ref)
+   *ctrl_ref = NULL;
 
/*
 * Automatically add the control class if it is not yet present and
@@ -1888,11 +1893,16 @@ static int handler_new_ref(struct v4l2_ctrl_handler 
*hdl,
if (hdl->error)
return hdl->error;
 
-   new_ref = kzalloc(sizeof(*new_ref), GFP_KERNEL);
+   if (hdl->is_request)
+   sz_extra = ctrl->elems * ctrl->elem_size;
+   new_ref = kzalloc(sizeof(*new_ref) + sz_extra, GFP_KERNEL);
if (!new_ref)
return handler_set_err(hdl, -ENOMEM);
new_ref->ctrl = ctrl;
new_ref->from_other_dev = from_other_dev;
+   if (sz_extra)
+   new_ref->p_req.p = _ref[1];
+
if (ctrl->handler == hdl) {
/* By default each control starts in a cluster of its own.
   new_ref->ctrl is basically a cluster array with one
@@ -1932,6 +1942,8 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
/* Insert the control node in the hash */
new_ref->next = hdl->buckets[bucket];
hdl->buckets[bucket] = new_ref;
+   if (ctrl_ref)
+   *ctrl_ref = new_ref;
 
 unlock:
mutex_unlock(hdl->lock);
@@ -2077,7 +2089,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct 
v4l2_ctrl_handler *hdl,
ctrl->type_ops->init(ctrl, idx, ctrl->p_new);
}
 
-   if (handler_new_ref(hdl, ctrl, false)) {
+   if (handler_new_ref(hdl, ctrl, NULL, false)) {
kvfree(ctrl);
return NULL;
}
@@ -2270,7 +2282,7 @@ int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
/* Filter any unwanted controls */
if (filter && !filter(ctrl))
continue;
-   ret = handler_new_ref(hdl, ctrl, from_other_dev);
+   ret = handler_new_ref(hdl, ctrl, NULL, from_other_dev);
if (ret)
break;
}
@@ -2681,6 +2693,94 @@ int v4l2_querymenu(struct v4l2_ctrl_handler *hdl, struct 
v4l2_querymenu *qm)
 }
 EXPORT_SYMBOL(v4l2_querymenu);
 
+int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl)
+{
+   int err;
+
+   err = v4l2_ctrl_handler_init(hdl, 0);
+   if (err)
+   return err;
+   hdl->is_request = true;
+   kref_init(>ref);
+
+   return 0;
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_init);
+
+int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
+   const struct v4l2_ctrl_handler *from,
+   bool (*filter)(const struct v4l2_ctrl *ctrl))
+{
+   struct v4l2_ctrl_ref *ref;
+   int err;
+
+   if (WARN_ON(!hdl || hdl == from))
+   return -EINVAL;
+
+   if (hdl->error)
+   return hdl->error;
+
+   WARN_ON(hdl->lock != >_lock);
+   v4l2_ctrl_handler_free(hdl);
+   err = v4l2_ctrl_handler_init(hdl, (from->nr_of_buckets - 1) * 8);
+   hdl->is_request = true;
+   if (err)
+   return err;
+   if (!from)
+   return 0;
+
+   mutex_lock(from->lock);
+   list_for_each_entry(ref, >ctrl_refs, node) {
+   struct v4l2_ctrl *ctrl = ref->ctrl;
+   struct v4l2_ctrl_ref *new_ref;
+
+   /*

[RFCv2 12/17] v4l2: add request API support

[Alexandre Courbot]

Add a v4l2 request entity data structure that takes care of storing the
request-related state of a V4L2 device ; in this case, its controls.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/Makefile   |  2 +-
 drivers/media/v4l2-core/v4l2-request.c | 54 ++
 include/media/v4l2-request.h   | 34 +
 3 files changed, 89 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-request.c
 create mode 100644 include/media/v4l2-request.h

diff --git a/drivers/media/v4l2-core/Makefile b/drivers/media/v4l2-core/Makefile
index 77303286aef7..5d885932f68f 100644
--- a/drivers/media/v4l2-core/Makefile
+++ b/drivers/media/v4l2-core/Makefile
@@ -15,7 +15,7 @@ obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o
 ifeq ($(CONFIG_TRACEPOINTS),y)
   videodev-objs += vb2-trace.o v4l2-trace.o
 endif
-videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o
+videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o v4l2-request.o
 
 obj-$(CONFIG_VIDEO_V4L2) += videodev.o
 obj-$(CONFIG_VIDEO_V4L2) += v4l2-common.o
diff --git a/drivers/media/v4l2-core/v4l2-request.c 
b/drivers/media/v4l2-core/v4l2-request.c
new file mode 100644
index ..7bc29d3cc332
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-request.c
@@ -0,0 +1,54 @@
+/*
+ * Media requests support for V4L2
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include 
+
+#include 
+
+struct media_request_entity_data *media_request_v4l2_entity_data_alloc(
+   struct v4l2_ctrl_handler *hdl)
+{
+   struct media_request_v4l2_entity_data *data;
+   int ret;
+
+   data = kzalloc(sizeof(*data), GFP_KERNEL);
+
+   ret = v4l2_ctrl_request_init(>ctrls);
+   if (ret) {
+   kfree(data);
+   return ERR_PTR(ret);
+   }
+
+   ret = v4l2_ctrl_request_clone(>ctrls, hdl, NULL);
+   if (ret) {
+   kfree(data);
+   return ERR_PTR(ret);
+   }
+
+   return >base;
+}
+EXPORT_SYMBOL_GPL(media_request_v4l2_entity_data_alloc);
+
+void
+media_request_v4l2_entity_data_free(struct media_request_entity_data *_data)
+{
+   struct media_request_v4l2_entity_data *data;
+
+   data = to_v4l2_entity_data(_data);
+
+   v4l2_ctrl_handler_free(>ctrls);
+   kfree(data);
+}
+EXPORT_SYMBOL_GPL(media_request_v4l2_entity_data_free);
diff --git a/include/media/v4l2-request.h b/include/media/v4l2-request.h
new file mode 100644
index ..db38dc5fc460
--- /dev/null
+++ b/include/media/v4l2-request.h
@@ -0,0 +1,34 @@
+/*
+ * Media requests support for V4L2
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MEDIA_REQUEST_V4L2_H
+#define _MEDIA_REQUEST_V4L2_H
+
+#include 
+#include 
+
+struct media_request_v4l2_entity_data {
+   struct media_request_entity_data base;
+
+   struct v4l2_ctrl_handler ctrls;
+};
+#define to_v4l2_entity_data(d) \
+   container_of(d, struct media_request_v4l2_entity_data, base)
+
+struct media_request_entity_data *media_request_v4l2_entity_data_alloc(
+   struct v4l2_ctrl_handler *hdl);
+void media_request_v4l2_entity_data_free(struct media_request_entity_data 
*data);
+
+#endif
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 13/17] videodev2.h: add request_fd field to v4l2_ext_controls

[Alexandre Courbot]

Allow to specify a request to be used with the S_EXT_CTRLS and
G_EXT_CTRLS operations.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 include/uapi/linux/videodev2.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 89bd716c66a6..0aeb8861908a 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1585,7 +1585,8 @@ struct v4l2_ext_controls {
};
__u32 count;
__u32 error_idx;
-   __u32 reserved[2];
+   __s32 request_fd;
+   __u32 reserved[1];
struct v4l2_ext_control *controls;
 };
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 10/17] v4l2-ctrls: support g/s_ext_ctrls for requests

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

The v4l2_g/s_ext_ctrls functions now support control handlers that
represent requests.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 37 
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 0fa20c89ece1..e4d29cc76ca8 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1518,6 +1518,13 @@ static int new_to_user(struct v4l2_ext_control *c,
return ptr_to_user(c, ctrl, ctrl->p_new);
 }
 
+/* Helper function: copy the request value back to the caller */
+static int req_to_user(struct v4l2_ext_control *c,
+  struct v4l2_ctrl_ref *ref)
+{
+   return ptr_to_user(c, ref->ctrl, ref->p_req);
+}
+
 /* Helper function: copy the initial control value back to the caller */
 static int def_to_user(struct v4l2_ext_control *c, struct v4l2_ctrl *ctrl)
 {
@@ -1637,6 +1644,14 @@ static void cur_to_new(struct v4l2_ctrl *ctrl)
ptr_to_ptr(ctrl, ctrl->p_cur, ctrl->p_new);
 }
 
+/* Copy the new value to the request value */
+static void new_to_req(struct v4l2_ctrl_ref *ref)
+{
+   if (!ref)
+   return;
+   ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -2967,7 +2982,8 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
struct v4l2_ctrl *ctrl);
struct v4l2_ctrl *master;
 
-   ctrl_to_user = def_value ? def_to_user : cur_to_user;
+   ctrl_to_user = def_value ? def_to_user :
+  (hdl->is_request ? NULL : cur_to_user);
 
if (helpers[i].mref == NULL)
continue;
@@ -2993,8 +3009,12 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, 
struct v4l2_ext_controls *cs
u32 idx = i;
 
do {
-   ret = ctrl_to_user(cs->controls + idx,
-  helpers[idx].ref->ctrl);
+   if (ctrl_to_user)
+   ret = ctrl_to_user(cs->controls + idx,
+   helpers[idx].ref->ctrl);
+   else
+   ret = req_to_user(cs->controls + idx,
+   helpers[idx].ref);
idx = helpers[idx].next;
} while (!ret && idx);
}
@@ -3267,7 +3287,16 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct 
v4l2_ctrl_handler *hdl,
} while (!ret && idx);
 
if (!ret)
-   ret = try_or_set_cluster(fh, master, set, 0);
+   ret = try_or_set_cluster(fh, master,
+!hdl->is_request && set, 0);
+   if (!ret && hdl->is_request && set) {
+   for (j = 0; j < master->ncontrols; j++) {
+   struct v4l2_ctrl_ref *ref =
+   find_ref(hdl, master->cluster[j]->id);
+
+   new_to_req(ref);
+   }
+   }
 
/* Copy the new values back to userspace. */
if (!ret) {
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 14/17] v4l2-ctrls: support requests in EXT_CTRLS ioctls

[Alexandre Courbot]

Read and use the request_fd field of struct v4l2_ext_controls to apply
VIDIOC_G_EXT_CTRLS or VIDIOC_S_EXT_CTRLS to a request when asked by
userspace.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 36 
 1 file changed, 36 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c 
b/drivers/media/v4l2-core/v4l2-ioctl.c
index 3b44f1fe4f23..0b4c9024c96b 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -30,6 +30,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -2182,6 +2183,24 @@ static int v4l_g_ext_ctrls(const struct v4l2_ioctl_ops 
*ops,
test_bit(V4L2_FL_USES_V4L2_FH, >flags) ? fh : NULL;
 
p->error_idx = p->count;
+
+   if (p->request_fd > 0) {
+   struct media_request *req = NULL;
+   struct media_request_entity_data *_data;
+   struct media_request_v4l2_entity_data *data;
+   int ret;
+
+   req = check_request(p->request_fd, file, fh, &_data);
+   if (IS_ERR(req))
+   return PTR_ERR(req);
+   data = to_v4l2_entity_data(_data);
+
+   ret = v4l2_g_ext_ctrls(>ctrls, p);
+
+   media_request_put(req);
+   return ret;
+   }
+
if (vfh && vfh->ctrl_handler)
return v4l2_g_ext_ctrls(vfh->ctrl_handler, p);
if (vfd->ctrl_handler)
@@ -2201,6 +2220,23 @@ static int v4l_s_ext_ctrls(const struct v4l2_ioctl_ops 
*ops,
test_bit(V4L2_FL_USES_V4L2_FH, >flags) ? fh : NULL;
 
p->error_idx = p->count;
+   if (p->request_fd > 0) {
+   struct media_request *req = NULL;
+   struct media_request_entity_data *_data;
+   struct media_request_v4l2_entity_data *data;
+   int ret;
+
+   req = check_request(p->request_fd, file, fh, &_data);
+   if (IS_ERR(req))
+   return PTR_ERR(req);
+   data = to_v4l2_entity_data(_data);
+
+   ret = v4l2_s_ext_ctrls(vfh, >ctrls, p);
+
+   media_request_put(req);
+   return ret;
+   }
+
if (vfh && vfh->ctrl_handler)
return v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler, p);
if (vfd->ctrl_handler)
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 17/17] media: vim2m: add request support

[Alexandre Courbot]

Set the necessary ops for supporting requests in vim2m.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/vim2m.c | 55 ++
 1 file changed, 55 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index a32e8a7950eb..a5ac95f8832a 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -30,6 +30,9 @@
 #include 
 #include 
 #include 
+#include 
+#include 
+#include 
 
 MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
 MODULE_AUTHOR("Pawel Osciak, <pa...@osciak.com>");
@@ -370,6 +373,28 @@ static void job_abort(void *priv)
ctx->aborting = 1;
 }
 
+static int apply_request_params(struct media_request *req,
+struct vim2m_ctx *ctx)
+{
+   struct media_request_entity_data *_data;
+   struct media_request_v4l2_entity_data *data;
+
+   if (!req)
+   return -EINVAL;
+
+   _data = media_request_get_entity_data(req, >dev->vfd.entity,
+ >fh);
+   if (WARN_ON(!_data))
+   return -EINVAL;
+   data = container_of(_data, struct media_request_v4l2_entity_data, base);
+
+   v4l2_ctrl_request_setup(>ctrls);
+
+   data->base.applied = true;
+
+   return 0;
+}
+
 /* device_run() - prepares and starts the device
  *
  * This simulates all the immediate preparations required before starting
@@ -381,12 +406,22 @@ static void device_run(void *priv)
struct vim2m_ctx *ctx = priv;
struct vim2m_dev *dev = ctx->dev;
struct vb2_v4l2_buffer *src_buf, *dst_buf;
+   int ret;
 
src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
 
+   WARN_ON(dst_buf->vb2_buf.request != NULL);
+
+   /* Apply request if needed */
+   ret = apply_request_params(src_buf->vb2_buf.request, ctx);
+   WARN_ON(ret);
+
device_process(ctx, src_buf, dst_buf);
 
+   /* Inform user about which request produced the destination buffer */
+   dst_buf->request_fd = src_buf->request_fd;
+
/* Run a timer, which simulates a hardware irq  */
schedule_irq(dev, ctx->transtime);
 }
@@ -841,6 +876,7 @@ static int queue_init(void *priv, struct vb2_queue *src_vq, 
struct vb2_queue *ds
src_vq->mem_ops = _vmalloc_memops;
src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
src_vq->lock = >dev->dev_mutex;
+   src_vq->allow_requests = true;
 
ret = vb2_queue_init(src_vq);
if (ret)
@@ -992,6 +1028,19 @@ static const struct v4l2_m2m_ops m2m_ops = {
.job_abort  = job_abort,
 };
 
+struct media_request_entity_data *vim2m_entity_data_alloc(struct media_entity 
*entity,
+ void *fh)
+{
+   struct vim2m_ctx *ctx = container_of(fh, struct vim2m_ctx, fh);
+
+   return media_request_v4l2_entity_data_alloc(>hdl);
+}
+
+static const struct media_entity_request_ops vim2m_entity_req_ops = {
+   .data_alloc = vim2m_entity_data_alloc,
+   .data_free  = media_request_v4l2_entity_data_free,
+};
+
 static int vim2m_probe(struct platform_device *pdev)
 {
struct vim2m_dev *dev;
@@ -1006,6 +1055,9 @@ static int vim2m_probe(struct platform_device *pdev)
 
 #ifdef CONFIG_MEDIA_CONTROLLER
dev->mdev.dev = >dev;
+   dev->mdev.req_mgr = media_request_mgr_alloc(>mdev);
+   if (IS_ERR(dev->mdev.req_mgr))
+   return PTR_ERR(dev->mdev.req_mgr);
strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
media_device_init(>mdev);
dev->v4l2_dev.mdev = >mdev;
@@ -1030,6 +1082,9 @@ static int vim2m_probe(struct platform_device *pdev)
}
 
video_set_drvdata(vfd, dev);
+#ifdef CONFIG_MEDIA_CONTROLLER
+   dev->vfd.entity.req_ops = _entity_req_ops;
+#endif
snprintf(vfd->name, sizeof(vfd->name), "%s", vim2m_videodev.name);
v4l2_info(>v4l2_dev,
"Device registered as /dev/video%d\n", vfd->num);
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 16/17] media: vim2m: add media device

[Alexandre Courbot]

Request API requires a media node. Add one to the vim2m driver so we can
use requests with it.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/platform/vim2m.c | 24 
 1 file changed, 24 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index b01fba020d5f..a32e8a7950eb 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -140,6 +140,9 @@ static struct vim2m_fmt *find_format(struct v4l2_format *f)
 struct vim2m_dev {
struct v4l2_device  v4l2_dev;
struct video_device vfd;
+#ifdef CONFIG_MEDIA_CONTROLLER
+   struct media_device mdev;
+#endif
 
atomic_tnum_inst;
struct mutexdev_mutex;
@@ -1001,6 +1004,13 @@ static int vim2m_probe(struct platform_device *pdev)
 
spin_lock_init(>irqlock);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+   dev->mdev.dev = >dev;
+   strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
+   media_device_init(>mdev);
+   dev->v4l2_dev.mdev = >mdev;
+#endif
+
ret = v4l2_device_register(>dev, >v4l2_dev);
if (ret)
return ret;
@@ -1034,6 +1044,13 @@ static int vim2m_probe(struct platform_device *pdev)
goto err_m2m;
}
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+   /* Register the media device node */
+   ret = media_device_register(>mdev);
+   if (ret)
+   goto err_m2m;
+#endif
+
return 0;
 
 err_m2m:
@@ -1050,6 +1067,13 @@ static int vim2m_remove(struct platform_device *pdev)
struct vim2m_dev *dev = platform_get_drvdata(pdev);
 
v4l2_info(>v4l2_dev, "Removing " MEM2MEM_NAME);
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+   if (media_devnode_is_registered(dev->mdev.devnode))
+   media_device_unregister(>mdev);
+   media_device_cleanup(>mdev);
+#endif
+
v4l2_m2m_release(dev->m2m_dev);
del_timer_sync(>timer);
video_unregister_device(>vfd);
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 15/17] v4l2: document the request API interface

[Alexandre Courbot]

Document how the request API can be used along with the existing V4L2
interface.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 Documentation/media/uapi/v4l/buffer.rst|  10 +-
 Documentation/media/uapi/v4l/common.rst|   1 +
 Documentation/media/uapi/v4l/request-api.rst   | 236 +
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst  |  16 +-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst   |  21 ++
 5 files changed, 280 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..9d082784081d 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,14 @@ struct v4l2_buffer
elements in the ``planes`` array. The driver will fill in the
actual number of valid elements in that array.
 * - __u32
-  - ``reserved2``
+  - ``request_fd``
   -
-  - A place holder for future extensions. Drivers and applications
-   must set this to 0.
+  - The file descriptor of the request associated with this buffer.
+   user-space can set this when calling :ref:`VIDIOC_QBUF`, and drivers
+   will return the request used when processing a buffer (if any) upon
+   :ref:`VIDIOC_DQBUF`.
+
+   A value of 0 means the buffer is not associated with any request.
 * - __u32
   - ``reserved``
   -
diff --git a/Documentation/media/uapi/v4l/common.rst 
b/Documentation/media/uapi/v4l/common.rst
index 13f2ed3fc5a6..a4aa0059d45a 100644
--- a/Documentation/media/uapi/v4l/common.rst
+++ b/Documentation/media/uapi/v4l/common.rst
@@ -44,3 +44,4 @@ applicable to all devices.
 crop
 selection-api
 streaming-par
+request-api
diff --git a/Documentation/media/uapi/v4l/request-api.rst 
b/Documentation/media/uapi/v4l/request-api.rst
new file mode 100644
index ..4c61a0dbe3a9
--- /dev/null
+++ b/Documentation/media/uapi/v4l/request-api.rst
@@ -0,0 +1,236 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _media-request-api:
+
+Request API
+===
+
+The Request API has been designed to allow V4L2 to deal with requirements of
+modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android Codec
+v2). One such requirement is the ability for devices belonging to the same
+pipeline to reconfigure and collaborate closely on a per-frame basis. Another 
is
+efficient support of stateless codecs, which need per-frame controls to be set
+asynchronously in order to be efficiently used.
+
+Supporting these features without the Request API is possible but terribly
+inefficient: user-space would have to flush all activity on the media pipeline,
+reconfigure it for the next frame, queue the buffers to be processed with that
+configuration, and wait until they are all available for dequeing before
+considering the next frame. This defeats the purpose of having buffer queues
+since in practice only one buffer would be queued at a time.
+
+The Request API allows a specific configuration of the pipeline (media
+controller topology + controls for each device) to be associated with specific
+buffers. The parameters are applied by each participating device as buffers
+associated to a request flow in. This allows user-space to schedule several
+tasks ("requests") with different parameters in advance, knowing that the
+parameters will be applied when needed to get the expected result. Controls
+values at the time of request completion are also available for reading.
+
+Usage
+=
+
+The Request API is used on top of standard media controller and V4L2 calls,
+which are augmented with an extra ``request_fd`` parameter. All operations on
+requests themselves are performed using the command parameter of the
+:c:func:`MEDIA_IOC_REQUEST_CMD` ioctl.
+
+Request Allocation
+--
+
+User-space allocates requests using the ``MEDIA_REQ_CMD_ALLOC`` command on
+an opened media device. This returns a file descriptor representing the
+request. Typically, several such requests will be allocated.
+
+Request Preparation
+---
+
+Standard V4L2 ioctls can then receive a request file descriptor to express the
+fact that the ioctl is part of said request, and is not to be applied
+immediately. V4L2 ioctls supporting this are :c:func:`VIDIOC_S_EXT_CTRLS` and
+:c:func:`VIDIOC_QBUF`. Controls set with a request parameter are stored instead
+of being immediately applied, and queued buffers will block the buffer queue
+until the request becomes active.
+
+RFC Note: currently several buffers can be queued to the same queue with the
+same request. The request parameters will be only be applied when processing
+the first buffer. Does it make more sense to allow at most one buffer per
+request per queue instead?
+
+Request Submission
+--
+
+Once the parameters and buffer

[RFCv2 07/17] v4l2-ctrls: prepare internal structs for request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a refcount and is_request bool to struct v4l2_ctrl_handler:
this is used to refcount a handler that represents a request.

Add a p_req field to struct v4l2_ctrl_ref that will store the
request value.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 1 +
 include/media/v4l2-ctrls.h   | 4 
 2 files changed, 5 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index 5e3b8883..9d99679b543c 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1751,6 +1751,7 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler 
*hdl,
  sizeof(hdl->buckets[0]),
  GFP_KERNEL | __GFP_ZERO);
hdl->error = hdl->buckets ? 0 : -ENOMEM;
+   hdl->is_request = false;
return hdl->error;
 }
 EXPORT_SYMBOL(v4l2_ctrl_handler_init_class);
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index a762f3392d90..a215f25a82cf 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -18,6 +18,7 @@
 #define _V4L2_CTRLS_H
 
 #include 
+#include 
 #include 
 #include 
 
@@ -250,6 +251,7 @@ struct v4l2_ctrl_ref {
struct v4l2_ctrl_ref *next;
struct v4l2_ctrl *ctrl;
struct v4l2_ctrl_helper *helper;
+   union v4l2_ctrl_ptr p_req;
bool from_other_dev;
 };
 
@@ -285,7 +287,9 @@ struct v4l2_ctrl_handler {
v4l2_ctrl_notify_fnc notify;
void *notify_priv;
u16 nr_of_buckets;
+   bool is_request;
int error;
+   struct kref ref;
 };
 
 /**
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 11/17] v4l2-ctrls: add v4l2_ctrl_request_setup

[Alexandre Courbot]

From: Hans Verkuil <hans.verk...@cisco.com>

Add a helper function that can set controls from a request.

Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>
Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 71 
 include/media/v4l2-ctrls.h   |  2 +
 2 files changed, 73 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c 
b/drivers/media/v4l2-core/v4l2-ctrls.c
index e4d29cc76ca8..da3db6268e9c 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1652,6 +1652,14 @@ static void new_to_req(struct v4l2_ctrl_ref *ref)
ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
 }
 
+/* Copy the request value to the new value */
+static void req_to_new(struct v4l2_ctrl_ref *ref)
+{
+   if (!ref)
+   return;
+   ptr_to_ptr(ref->ctrl, ref->p_req, ref->ctrl->p_new);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -3423,6 +3431,69 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, 
const char *s)
 }
 EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
 
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl)
+{
+   struct v4l2_ctrl_ref *ref;
+
+   if (!hdl)
+   return;
+
+   mutex_lock(hdl->lock);
+
+   list_for_each_entry(ref, >ctrl_refs, node)
+   ref->done = false;
+
+   list_for_each_entry(ref, >ctrl_refs, node) {
+   struct v4l2_ctrl *ctrl = ref->ctrl;
+   struct v4l2_ctrl *master = ctrl->cluster[0];
+   int i;
+
+   /* Skip if this control was already handled by a cluster. */
+   /* Skip button controls and read-only controls. */
+   if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
+   (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
+   continue;
+
+   v4l2_ctrl_lock(master);
+   for (i = 0; i < master->ncontrols; i++) {
+   if (master->cluster[i]) {
+   struct v4l2_ctrl_ref *r =
+   find_ref(hdl, master->cluster[i]->id);
+
+   req_to_new(r);
+   master->cluster[i]->is_new = 1;
+   r->done = true;
+   }
+   }
+   /*
+* For volatile autoclusters that are currently in auto mode
+* we need to discover if it will be set to manual mode.
+* If so, then we have to copy the current volatile values
+* first since those will become the new manual values (which
+* may be overwritten by explicit new values from this set
+* of controls).
+*/
+   if (master->is_auto && master->has_volatiles &&
+   !is_cur_manual(master)) {
+   s32 new_auto_val = *master->p_new.p_s32;
+
+   /*
+* If the new value == the manual value, then copy
+* the current volatile values.
+*/
+   if (new_auto_val == master->manual_mode_value)
+   update_from_auto_cluster(master);
+   }
+
+   try_or_set_cluster(NULL, master, true, 0);
+
+   v4l2_ctrl_unlock(master);
+   }
+
+   mutex_unlock(hdl->lock);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_setup);
+
 void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, 
void *priv)
 {
if (ctrl == NULL)
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 24a60ca1a0dd..9cc9c242ed9d 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -252,6 +252,7 @@ struct v4l2_ctrl_ref {
struct v4l2_ctrl *ctrl;
struct v4l2_ctrl_helper *helper;
union v4l2_ctrl_ptr p_req;
+   bool done;
bool from_other_dev;
 };
 
@@ -1049,6 +1050,7 @@ int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl);
 int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
const struct v4l2_ctrl_handler *from,
bool (*filter)(const struct v4l2_ctrl *ctrl));
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_get(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_put(struct v4l2_ctrl_handler *hdl);
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv2 03/17] media: videobuf2: add support for requests

[Alexandre Courbot]

Make vb2 aware of requests. Drivers can specify whether a given queue
can accept requests or not. Queues that accept requests will block on a
buffer that is part of a request until that request is submitted.

Signed-off-by: Alexandre Courbot <acour...@chromium.org>
---
 drivers/media/v4l2-core/videobuf2-core.c | 133 +--
 drivers/media/v4l2-core/videobuf2-v4l2.c |  28 ++-
 include/media/videobuf2-core.h   |  15 +++-
 3 files changed, 168 insertions(+), 8 deletions(-)

diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
b/drivers/media/v4l2-core/videobuf2-core.c
index cb115ba6a1d2..4a69ac12ee88 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -26,6 +26,7 @@
 
 #include 
 #include 
+#include 
 
 #include 
 
@@ -922,6 +923,17 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
vb->state = state;
}
atomic_dec(>owned_by_drv_count);
+   if (vb->request) {
+   struct media_request *req = vb->request;
+
+   if (atomic_dec_and_test(>buf_cpt))
+   media_request_complete(vb->request);
+
+   /* release reference acquired during qbuf */
+   vb->request = NULL;
+   media_request_put(req);
+   }
+
spin_unlock_irqrestore(>done_lock, flags);
 
trace_vb2_buf_done(q, vb);
@@ -1298,6 +1310,53 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+/**
+ * vb2_check_buf_req_status() - Validate request state of a buffer
+ * @vb:buffer to check
+ *
+ * Returns true if a buffer is ready to be passed to the driver request-wise.
+ * This means that neither this buffer nor any previously-queued buffer is
+ * associated to a request that is not yet submitted.
+ *
+ * If this function returns false, then the buffer shall not be passed to its
+ * driver since the request state is not completely built yet. In that case,
+ * this function will register a notifier to be called when the request is
+ * submitted and the queue can be unblocked.
+ *
+ * This function must be called with req_lock held.
+ */
+static bool vb2_check_buf_req_status(struct vb2_buffer *vb)
+{
+   struct media_request *req = vb->request;
+   struct vb2_queue *q = vb->vb2_queue;
+   int ret = false;
+
+   mutex_lock(>req_lock);
+
+   if (!req) {
+   ret = !q->waiting_req;
+   goto done;
+   }
+
+   mutex_lock(>lock);
+   if (req->state == MEDIA_REQUEST_STATE_SUBMITTED) {
+   mutex_unlock(>lock);
+   ret = !q->waiting_req;
+   goto done;
+   }
+
+   if (!q->waiting_req) {
+   q->waiting_req = true;
+   atomic_notifier_chain_register(>submit_notif,
+  >req_blk);
+   }
+   mutex_unlock(>lock);
+
+done:
+   mutex_unlock(>req_lock);
+   return ret;
+}
+
 /**
  * vb2_start_streaming() - Attempt to start streaming.
  * @q: videobuf2 queue
@@ -1318,8 +1377,11 @@ static int vb2_start_streaming(struct vb2_queue *q)
 * If any buffers were queued before streamon,
 * we can now pass them to driver for processing.
 */
-   list_for_each_entry(vb, >queued_list, queued_entry)
+   list_for_each_entry(vb, >queued_list, queued_entry) {
+   if (!vb2_check_buf_req_status(vb))
+   break;
__enqueue_in_driver(vb);
+   }
 
/* Tell the driver to start streaming */
q->start_streaming_called = 1;
@@ -1361,7 +1423,46 @@ static int vb2_start_streaming(struct vb2_queue *q)
return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+/**
+ * vb2_unblock_requests() - unblock a queue waiting for a request submission
+ * @nb:notifier block that has been registered
+ * @action:unused
+ * @data:  request that has been submitted
+ *
+ * This is a callback function that is registered when
+ * vb2_check_buf_req_status() returns false. It is invoked when the request
+ * blocking the queue has been submitted. This means its buffers (and all
+ * following valid buffers) can be passed to drivers.
+ */
+static int vb2_unblock_requests(struct notifier_block *nb, unsigned long 
action,
+   void *data)
+{
+   struct vb2_queue *q = container_of(nb, struct vb2_queue, req_blk);
+   struct media_request *req = data;
+   struct vb2_buffer *vb;
+   bool found_request = false;
+
+   mutex_lock(>req_lock);
+   atomic_notifier_chain_unregister(>submit_notif, >req_blk);
+   q->waiting_req = false;
+   mutex_unlock(>req_lock);
+
+   list_for_each_entry(vb, >queued_list,