Re: Linux Testing Microconference at LPC

[Gustavo Padovan]

Hi, 
 
On Thursday, April 11, 2019 14:37 -03, Dhaval Giani  
wrote: 
 
> Hi Folks,
> 
> This is a call for participation for the Linux Testing microconference
> at LPC this year.
> 
> For those who were at LPC last year, as the closing panel mentioned,
> testing is probably the next big push needed to improve quality. From
> getting more selftests in, to regression testing to ensure we don't
> break realtime as more of PREEMPT_RT comes in, to more stable distros,
> we need more testing around the kernel.
> 
> We have talked about different efforts around testing, such as fuzzing
> (using syzkaller and trinity), automating fuzzing with syzbot, 0day
> testing, test frameworks such as ktests, smatch to find bugs in the
> past. We want to push this discussion further this year and are
> interested in hearing from you what you want to talk about, and where
> kernel testing needs to go next.
> 
> Please let us know what topics you believe should be a part of the
> micro conference this year.

Guillaume would like to talk about the his work on kernelCI on automated 
bisection, functional testing and modular pipelines.

Regards,

Gustavo

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Gustavo Padovan]

Hi Dhaval,

On 9/19/18 7:13 PM, Dhaval Giani wrote:

Hi folks,

Sasha and I are pleased to announce the Testing and Fuzzing track at
LPC [ 1 ]. We are planning to continue the discussions from last
year's microconference [2]. Many discussions from the Automated
Testing Summit [3] will also continue, and a final agenda will come up
only soon after that.

Suggested Topics

- Syzbot/syzkaller
- ATS
- Distro/stable testing
- kernelci
- kernelci auto bisection


Having 2 kernelci talks don't make too much sense, I discussed with 
Kevin and we thing it would be a good idea to merge them together. Could 
you do that?


Thanks,

Gustavo


Gustavo Padovan
Collabora Ltd

Re: [Announce] LPC 2018: Testing and Fuzzing Microconference

[Gustavo Padovan]

Hi Dhaval,

On 9/19/18 7:13 PM, Dhaval Giani wrote:

Hi folks,

Sasha and I are pleased to announce the Testing and Fuzzing track at
LPC [ 1 ]. We are planning to continue the discussions from last
year's microconference [2]. Many discussions from the Automated
Testing Summit [3] will also continue, and a final agenda will come up
only soon after that.

Suggested Topics

- Syzbot/syzkaller
- ATS
- Distro/stable testing
- kernelci
- kernelci auto bisection


Having 2 kernelci talks don't make too much sense, I discussed with 
Kevin and we thing it would be a good idea to merge them together. Could 
you do that?


Thanks,

Gustavo


Gustavo Padovan
Collabora Ltd

Re: [RFC RESEND PATCH] drm/rockchip: update cursors asynchronously through atomic.

[Gustavo Padovan]

Hi Enric,

On 07/23/2018 11:36 AM, Sean Paul wrote:

On Wed, Jun 27, 2018 at 11:14:47PM +0200, Enric Balletbo i Serra wrote:

Add support to async updates of cursors by using the new atomic
interface for that.

Signed-off-by: Enric Balletbo i Serra 


LGTM. Given rockchip hasn't weighed in on the patch, and that you've tested it
on real hardware, let's land it.

Reviewed-by: Sean Paul 


Your patch don't apply cleanly anymore. Can you rebase it and add the 
r-b to the patch while at it? Thanks!


Gustavo





---
I am sending this as RFC because I still don't have a deep knowledge of
the hw and I am not sure if the vop_plane_update function can be reused
in both cases, atomic_updates and atomic_async_updates. I think that
someone with more knowledge should take a look. The patch was tested on
a Samsung Chromebook Plus in two ways.

1. Running all igt kms_cursor_legacy and kms_atomic@plane_cursor_legacy
tests and see that there is no regression after the patch.

2. Running weston using the atomic API.

Best regards,
   Enric

  drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 80 -
  1 file changed, 64 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop.c 
b/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
index 53d4afe15278..1eb6bda924af 100644
--- a/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
@@ -688,8 +688,7 @@ static void vop_plane_atomic_disable(struct drm_plane 
*plane,
spin_unlock(>reg_lock);
  }
  
-static void vop_plane_atomic_update(struct drm_plane *plane,

-   struct drm_plane_state *old_state)
+static void vop_plane_update(struct drm_plane *plane)
  {
struct drm_plane_state *state = plane->state;
struct drm_crtc *crtc = state->crtc;
@@ -710,20 +709,6 @@ static void vop_plane_atomic_update(struct drm_plane 
*plane,
bool rb_swap;
int format;
  
-	/*

-* can't update plane when vop is disabled.
-*/
-   if (WARN_ON(!crtc))
-   return;
-
-   if (WARN_ON(!vop->is_enabled))
-   return;
-
-   if (!state->visible) {
-   vop_plane_atomic_disable(plane, old_state);
-   return;
-   }
-
obj = rockchip_fb_get_gem_obj(fb, 0);
rk_obj = to_rockchip_obj(obj);
  
@@ -794,10 +779,73 @@ static void vop_plane_atomic_update(struct drm_plane *plane,

spin_unlock(>reg_lock);
  }
  
+static void vop_plane_atomic_update(struct drm_plane *plane,

+   struct drm_plane_state *old_state)
+{
+   struct drm_plane_state *state = plane->state;
+   struct vop *vop = to_vop(state->crtc);
+
+   /*
+* can't update plane when vop is disabled.
+*/
+   if (WARN_ON(!state->crtc))
+   return;
+
+   if (WARN_ON(!vop->is_enabled))
+   return;
+
+   if (!state->visible) {
+   vop_plane_atomic_disable(plane, old_state);
+   return;
+   }
+
+   vop_plane_update(plane);
+}
+
+static int vop_plane_atomic_async_check(struct drm_plane *plane,
+   struct drm_plane_state *state)
+{
+   struct drm_crtc_state *crtc_state;
+
+   crtc_state = drm_atomic_get_existing_crtc_state(state->state,
+   state->crtc);
+   if (WARN_ON(!crtc_state))
+   return -EINVAL;
+
+   if (!crtc_state->active)
+   return -EINVAL;
+
+   if (plane->state->crtc != state->crtc ||
+   plane->state->src_w != state->src_w ||
+   plane->state->src_h != state->src_h ||
+   plane->state->crtc_w != state->crtc_w ||
+   plane->state->crtc_h != state->crtc_h ||
+   !plane->state->fb ||
+   plane->state->fb != state->fb)
+   return -EINVAL;
+
+   return 0;
+}
+
+static void vop_plane_atomic_async_update(struct drm_plane *plane,
+ struct drm_plane_state *new_state)
+{
+   plane->state->src_x = new_state->src_x;
+   plane->state->src_y = new_state->src_y;
+   plane->state->crtc_x = new_state->crtc_x;
+   plane->state->crtc_y = new_state->crtc_y;
+   plane->state->fb = new_state->fb;
+   *plane->state = *new_state;
+
+   vop_plane_update(plane);
+}
+
  static const struct drm_plane_helper_funcs plane_helper_funcs = {
.atomic_check = vop_plane_atomic_check,
.atomic_update = vop_plane_atomic_update,
.atomic_disable = vop_plane_atomic_disable,
+   .atomic_async_check = vop_plane_atomic_async_check,
+       .atomic_async_update = vop_plane_atomic_async_update,
  };
  
  static const struct drm_plane_funcs vop_plane_funcs = {

--
2.18.0





--
Gustavo Padovan
Collabora Ltd

Re: [RFC RESEND PATCH] drm/rockchip: update cursors asynchronously through atomic.

[Gustavo Padovan]

Hi Enric,

On 07/23/2018 11:36 AM, Sean Paul wrote:

On Wed, Jun 27, 2018 at 11:14:47PM +0200, Enric Balletbo i Serra wrote:

Add support to async updates of cursors by using the new atomic
interface for that.

Signed-off-by: Enric Balletbo i Serra 


LGTM. Given rockchip hasn't weighed in on the patch, and that you've tested it
on real hardware, let's land it.

Reviewed-by: Sean Paul 


Your patch don't apply cleanly anymore. Can you rebase it and add the 
r-b to the patch while at it? Thanks!


Gustavo





---
I am sending this as RFC because I still don't have a deep knowledge of
the hw and I am not sure if the vop_plane_update function can be reused
in both cases, atomic_updates and atomic_async_updates. I think that
someone with more knowledge should take a look. The patch was tested on
a Samsung Chromebook Plus in two ways.

1. Running all igt kms_cursor_legacy and kms_atomic@plane_cursor_legacy
tests and see that there is no regression after the patch.

2. Running weston using the atomic API.

Best regards,
   Enric

  drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 80 -
  1 file changed, 64 insertions(+), 16 deletions(-)

diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop.c 
b/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
index 53d4afe15278..1eb6bda924af 100644
--- a/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop.c
@@ -688,8 +688,7 @@ static void vop_plane_atomic_disable(struct drm_plane 
*plane,
spin_unlock(>reg_lock);
  }
  
-static void vop_plane_atomic_update(struct drm_plane *plane,

-   struct drm_plane_state *old_state)
+static void vop_plane_update(struct drm_plane *plane)
  {
struct drm_plane_state *state = plane->state;
struct drm_crtc *crtc = state->crtc;
@@ -710,20 +709,6 @@ static void vop_plane_atomic_update(struct drm_plane 
*plane,
bool rb_swap;
int format;
  
-	/*

-* can't update plane when vop is disabled.
-*/
-   if (WARN_ON(!crtc))
-   return;
-
-   if (WARN_ON(!vop->is_enabled))
-   return;
-
-   if (!state->visible) {
-   vop_plane_atomic_disable(plane, old_state);
-   return;
-   }
-
obj = rockchip_fb_get_gem_obj(fb, 0);
rk_obj = to_rockchip_obj(obj);
  
@@ -794,10 +779,73 @@ static void vop_plane_atomic_update(struct drm_plane *plane,

spin_unlock(>reg_lock);
  }
  
+static void vop_plane_atomic_update(struct drm_plane *plane,

+   struct drm_plane_state *old_state)
+{
+   struct drm_plane_state *state = plane->state;
+   struct vop *vop = to_vop(state->crtc);
+
+   /*
+* can't update plane when vop is disabled.
+*/
+   if (WARN_ON(!state->crtc))
+   return;
+
+   if (WARN_ON(!vop->is_enabled))
+   return;
+
+   if (!state->visible) {
+   vop_plane_atomic_disable(plane, old_state);
+   return;
+   }
+
+   vop_plane_update(plane);
+}
+
+static int vop_plane_atomic_async_check(struct drm_plane *plane,
+   struct drm_plane_state *state)
+{
+   struct drm_crtc_state *crtc_state;
+
+   crtc_state = drm_atomic_get_existing_crtc_state(state->state,
+   state->crtc);
+   if (WARN_ON(!crtc_state))
+   return -EINVAL;
+
+   if (!crtc_state->active)
+   return -EINVAL;
+
+   if (plane->state->crtc != state->crtc ||
+   plane->state->src_w != state->src_w ||
+   plane->state->src_h != state->src_h ||
+   plane->state->crtc_w != state->crtc_w ||
+   plane->state->crtc_h != state->crtc_h ||
+   !plane->state->fb ||
+   plane->state->fb != state->fb)
+   return -EINVAL;
+
+   return 0;
+}
+
+static void vop_plane_atomic_async_update(struct drm_plane *plane,
+ struct drm_plane_state *new_state)
+{
+   plane->state->src_x = new_state->src_x;
+   plane->state->src_y = new_state->src_y;
+   plane->state->crtc_x = new_state->crtc_x;
+   plane->state->crtc_y = new_state->crtc_y;
+   plane->state->fb = new_state->fb;
+   *plane->state = *new_state;
+
+   vop_plane_update(plane);
+}
+
  static const struct drm_plane_helper_funcs plane_helper_funcs = {
.atomic_check = vop_plane_atomic_check,
.atomic_update = vop_plane_atomic_update,
.atomic_disable = vop_plane_atomic_disable,
+   .atomic_async_check = vop_plane_atomic_async_check,
+       .atomic_async_update = vop_plane_atomic_async_update,
  };
  
  static const struct drm_plane_funcs vop_plane_funcs = {

--
2.18.0





--
Gustavo Padovan
Collabora Ltd

Re: [RFC PATCH] drm/atomic: add ASYNC_UPDATE flag to the Atomic IOCTL.

[Gustavo Padovan]

Hi Ville,

On Thu, 2018-06-28 at 16:35 +0300, Ville Syrjälä wrote:
> On Wed, Jun 27, 2018 at 11:25:06PM +0200, Enric Balletbo i Serra
> wrote:
> > From: Gustavo Padovan 
> > 
> > This flag tells core to jump ahead the queued update if the
> > conditions
> > in drm_atomic_async_check() are met. That means we are only able to
> > do an
> > async update if no modeset is pending and update for the same plane
> > is
> > not queued.
> > 
> > It uses the already in place infrastructure for async updates.
> 
> I still dislike the name. On Intel hw "async flip" means "flip
> asap and tear". Whereas the legcay cursor thing i915 has is a normal
> sync flip. "unthrottled" or something like that would be less
> confusing.

Maybe "amend"? However if we submit an amend commit when no commit is
pending it will become a regular atomic commit. 

> 
> As far as introducing this flag, at least i915 totally lacks a
> mechanism for deferring the buffer unpinning after the vblank. Hence
> this can't be used by i915 currently. I think the only reason we get
> away with the cursor hack is that we unbind the vma lazily and it's
> unlikely that the small cursor vma is going to get knocked out before
> the next vblank. For larger buffers that risk grows. We would
> probably
> want a stress test that smashes the gtt hard while doing "async
> updates" to catch this.

Right. We'll look into more detail to this in i915.

> 
> There's also the question of how out fences work with the async
> updates. I see that you disallow such an update if there's a previous
> sync update still pending. That simplifies things a little bit I
> suppose. But still you would need to track the fences per-plane and
> signal the old fence ones as soon as the new update overrides the
> pending update. And if you want to allow multiple planes in one async
> update then I think the uapi would need to be changed to have
> per-plane fences as well because subsequent updates could override 
> only a part of a previous update.

The only usecase (and userspace) we have on our side for now is for
cursors. I'm not sure an uAPI that only work for cursor in the first
stage would be acceptable. That would make things easier.

For updates on other planes or multiples planes I agree with you on
having per-plane fences - but I guess we need a userspace usecase for
it first.

> 
> > It is useful for cursor updates and async PageFlips over the atomic
> > ioctl, otherwise in some cases updates may be delayed to the point
> > the
> > user will notice it.
> > 
> > DRM_MODE_ATOMIC_ASYNC_UPDATE should be passed to the Atomic IOCTL
> > to use
> > this feature.
> > 
> > Signed-off-by: Gustavo Padovan 
> > Signed-off-by: Enric Balletbo i Serra  > >
> > ---
> > Hi,
> > 
> > This is an attempt to introduce the new ASYNC_UPDATE flag for
> > atomic
> > operations, see the commit message for a more detailed description.
> > 
> > To test this patch we have created an IGT test that we plan to send
> > to
> > the ML but also was tested using a small program that exercises the
> > uAPI
> > for easy sanity testing. The program created by Alexandros can be
> > found here
> > [2]. To test, just build the program and use the --atomic flag to
> > use the
> > cursor plane in normal (blocking mode), and --atomic-async to use
> > the cursor
> > plane with the ASYNC_UPDATE flag.E.g.
> > 
> >   drm_cursor --atomic
> > 
> > or
> > 
> >   drm_cursor --atomic-async
> > 
> > The test worked on a Samsung Chromebook Plus on top of mainline
> > plus
> > the patch to update cursors asynchronously through atomic for the
> > drm/rockchip driver [3].
> > 
> > Alexandros also did a proof-of-concept to use this flag and draw
> > cursors
> > using atomic if possible on ozone [1].
> > 
> > Best regards,
> >  Enric
> > 
> > [1] https://chromium-review.googlesource.com/c/chromium/src/+/10927
> > 11
> > [2] https://gitlab.collabora.com/alf/drm-cursor
> > [3] https://patchwork.kernel.org/patch/10492693/
> > 
> > 
> >  drivers/gpu/drm/drm_atomic.c| 6 ++
> >  drivers/gpu/drm/drm_atomic_helper.c | 9 ++---
> >  include/uapi/drm/drm_mode.h | 4 +++-
> >  3 files changed, 15 insertions(+), 4 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/drm_atomic.c
> > b/drivers/gpu/drm/drm_atomic.c
> > index c825c76edc1d..15b799f46982 100644
> > --- a/drivers/gpu/drm/drm_atomic.c
> > +++ b/drivers/gpu/drm/drm_atomic.c
> >

Re: [RFC PATCH] drm/atomic: add ASYNC_UPDATE flag to the Atomic IOCTL.

[Gustavo Padovan]

Hi Ville,

On Thu, 2018-06-28 at 16:35 +0300, Ville Syrjälä wrote:
> On Wed, Jun 27, 2018 at 11:25:06PM +0200, Enric Balletbo i Serra
> wrote:
> > From: Gustavo Padovan 
> > 
> > This flag tells core to jump ahead the queued update if the
> > conditions
> > in drm_atomic_async_check() are met. That means we are only able to
> > do an
> > async update if no modeset is pending and update for the same plane
> > is
> > not queued.
> > 
> > It uses the already in place infrastructure for async updates.
> 
> I still dislike the name. On Intel hw "async flip" means "flip
> asap and tear". Whereas the legcay cursor thing i915 has is a normal
> sync flip. "unthrottled" or something like that would be less
> confusing.

Maybe "amend"? However if we submit an amend commit when no commit is
pending it will become a regular atomic commit. 

> 
> As far as introducing this flag, at least i915 totally lacks a
> mechanism for deferring the buffer unpinning after the vblank. Hence
> this can't be used by i915 currently. I think the only reason we get
> away with the cursor hack is that we unbind the vma lazily and it's
> unlikely that the small cursor vma is going to get knocked out before
> the next vblank. For larger buffers that risk grows. We would
> probably
> want a stress test that smashes the gtt hard while doing "async
> updates" to catch this.

Right. We'll look into more detail to this in i915.

> 
> There's also the question of how out fences work with the async
> updates. I see that you disallow such an update if there's a previous
> sync update still pending. That simplifies things a little bit I
> suppose. But still you would need to track the fences per-plane and
> signal the old fence ones as soon as the new update overrides the
> pending update. And if you want to allow multiple planes in one async
> update then I think the uapi would need to be changed to have
> per-plane fences as well because subsequent updates could override 
> only a part of a previous update.

The only usecase (and userspace) we have on our side for now is for
cursors. I'm not sure an uAPI that only work for cursor in the first
stage would be acceptable. That would make things easier.

For updates on other planes or multiples planes I agree with you on
having per-plane fences - but I guess we need a userspace usecase for
it first.

> 
> > It is useful for cursor updates and async PageFlips over the atomic
> > ioctl, otherwise in some cases updates may be delayed to the point
> > the
> > user will notice it.
> > 
> > DRM_MODE_ATOMIC_ASYNC_UPDATE should be passed to the Atomic IOCTL
> > to use
> > this feature.
> > 
> > Signed-off-by: Gustavo Padovan 
> > Signed-off-by: Enric Balletbo i Serra  > >
> > ---
> > Hi,
> > 
> > This is an attempt to introduce the new ASYNC_UPDATE flag for
> > atomic
> > operations, see the commit message for a more detailed description.
> > 
> > To test this patch we have created an IGT test that we plan to send
> > to
> > the ML but also was tested using a small program that exercises the
> > uAPI
> > for easy sanity testing. The program created by Alexandros can be
> > found here
> > [2]. To test, just build the program and use the --atomic flag to
> > use the
> > cursor plane in normal (blocking mode), and --atomic-async to use
> > the cursor
> > plane with the ASYNC_UPDATE flag.E.g.
> > 
> >   drm_cursor --atomic
> > 
> > or
> > 
> >   drm_cursor --atomic-async
> > 
> > The test worked on a Samsung Chromebook Plus on top of mainline
> > plus
> > the patch to update cursors asynchronously through atomic for the
> > drm/rockchip driver [3].
> > 
> > Alexandros also did a proof-of-concept to use this flag and draw
> > cursors
> > using atomic if possible on ozone [1].
> > 
> > Best regards,
> >  Enric
> > 
> > [1] https://chromium-review.googlesource.com/c/chromium/src/+/10927
> > 11
> > [2] https://gitlab.collabora.com/alf/drm-cursor
> > [3] https://patchwork.kernel.org/patch/10492693/
> > 
> > 
> >  drivers/gpu/drm/drm_atomic.c| 6 ++
> >  drivers/gpu/drm/drm_atomic_helper.c | 9 ++---
> >  include/uapi/drm/drm_mode.h | 4 +++-
> >  3 files changed, 15 insertions(+), 4 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/drm_atomic.c
> > b/drivers/gpu/drm/drm_atomic.c
> > index c825c76edc1d..15b799f46982 100644
> > --- a/drivers/gpu/drm/drm_atomic.c
> > +++ b/drivers/gpu/drm/drm_atomic.c
> >

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Gustavo Padovan]

Hi Hans,

On Mon, 2018-05-07 at 14:07 +0200, Hans Verkuil wrote:
> On 04/05/18 22:06, Ezequiel Garcia wrote:
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued
> > to the
> > driver before its fences signal. And a buffer can't be queued to
> > the driver
> > out of the order they were queued from userspace. That means that
> > even if
> > its fence signals it must wait for all other buffers, ahead of it
> > in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the
> > driver,
> > instead we mark it as error and wait until the previous buffer is
> > done
> > to notify userspace of the error. We wait here to deliver the
> > buffers back
> > to userspace in order.
> > 
> > v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
> > 
> > v11: - minor doc/comments fixes (Hans Verkuil)
> >  - reviewed the in-fence path at __fill_v4l2_buffer()
> > 
> > v10: - rename fence to in_fence in many places
> >  - handle fences signalling with error better (Hans Verkuil)
> > 
> > v9: - improve comments and docs (Hans Verkuil)
> > - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans
> > Verkuil)
> > - move in-fences code that was in the out-fences patch here
> > (Alex)
> > 
> > v8: - improve comments about fences with errors
> > 
> > v7: - get rid of the fence array stuff for ordering and just use
> >   get_num_buffers_ready() (Hans)
> > - fix issue of queuing the buffer twice (Hans)
> > - avoid the dma_fence_wait() in core_qbuf() (Alex)
> > - merge preparation commit in
> > 
> > v6: - With fences always keep the order userspace queues the
> > buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence
> > array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian
> > Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> >   needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4: - Add a comment about dma_fence_add_callback() not returning a
> >   error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver()
> > (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> >   vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2: - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > Signed-off-by: Ezequiel Garcia <ezequ...@collabora.com>
> > ---
> >  drivers/media/common/videobuf2/videobuf2-core.c | 197
> > 
> >  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
> >  drivers/media/dvb-core/dvb_vb2.c|   2 +-
> >  drivers/media/v4l2-core/Kconfig |  33 
> >  include/media/videobuf2-core.h  |  14 +-
> >  5 files changed, 249 insertions(+), 31 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> > b/drivers/media/common/videobuf2/videobuf2-core.c
> > index 6b8e083893ad..996b99497a98 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > +++ b/drive

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Gustavo Padovan]

Hi Hans,

On Mon, 2018-05-07 at 14:07 +0200, Hans Verkuil wrote:
> On 04/05/18 22:06, Ezequiel Garcia wrote:
> > From: Gustavo Padovan 
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued
> > to the
> > driver before its fences signal. And a buffer can't be queued to
> > the driver
> > out of the order they were queued from userspace. That means that
> > even if
> > its fence signals it must wait for all other buffers, ahead of it
> > in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the
> > driver,
> > instead we mark it as error and wait until the previous buffer is
> > done
> > to notify userspace of the error. We wait here to deliver the
> > buffers back
> > to userspace in order.
> > 
> > v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
> > 
> > v11: - minor doc/comments fixes (Hans Verkuil)
> >  - reviewed the in-fence path at __fill_v4l2_buffer()
> > 
> > v10: - rename fence to in_fence in many places
> >  - handle fences signalling with error better (Hans Verkuil)
> > 
> > v9: - improve comments and docs (Hans Verkuil)
> > - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans
> > Verkuil)
> > - move in-fences code that was in the out-fences patch here
> > (Alex)
> > 
> > v8: - improve comments about fences with errors
> > 
> > v7: - get rid of the fence array stuff for ordering and just use
> >   get_num_buffers_ready() (Hans)
> > - fix issue of queuing the buffer twice (Hans)
> > - avoid the dma_fence_wait() in core_qbuf() (Alex)
> > - merge preparation commit in
> > 
> > v6: - With fences always keep the order userspace queues the
> > buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence
> > array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian
> > Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> >   needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4: - Add a comment about dma_fence_add_callback() not returning a
> >   error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver()
> > (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> >   vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2: - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan 
> > Signed-off-by: Ezequiel Garcia 
> > ---
> >  drivers/media/common/videobuf2/videobuf2-core.c | 197
> > 
> >  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
> >  drivers/media/dvb-core/dvb_vb2.c|   2 +-
> >  drivers/media/v4l2-core/Kconfig |  33 
> >  include/media/videobuf2-core.h  |  14 +-
> >  5 files changed, 249 insertions(+), 31 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> > b/drivers/media/common/videobuf2/videobuf2-core.c
> > index 6b8e083893ad..996b99497a98 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > @@ -352,6 +352,7 @@ static int __vb2_que

[PATCH v8 02/13] [media] hackrf: group device capabilities

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Instead of putting V4L2_CAP_STREAMING and V4L2_CAP_READWRITE
everywhere, set device_caps earlier with these values.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/usb/hackrf/hackrf.c | 11 ---
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/drivers/media/usb/hackrf/hackrf.c 
b/drivers/media/usb/hackrf/hackrf.c
index 7eb53517a82f..6d692fb3e8dd 100644
--- a/drivers/media/usb/hackrf/hackrf.c
+++ b/drivers/media/usb/hackrf/hackrf.c
@@ -909,18 +909,15 @@ static int hackrf_querycap(struct file *file, void *fh,
 
dev_dbg(>dev, "\n");
 
+   cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
if (vdev->vfl_dir == VFL_DIR_RX)
-   cap->device_caps = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
-  V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
-
+   cap->device_caps |= V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER;
else
-   cap->device_caps = V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-  V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+   cap->device_caps |= V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR;
 
cap->capabilities = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
-   V4L2_CAP_DEVICE_CAPS;
+   V4L2_CAP_DEVICE_CAPS | cap->device_caps;
strlcpy(cap->driver, KBUILD_MODNAME, sizeof(cap->driver));
strlcpy(cap->card, dev->rx_vdev.name, sizeof(cap->card));
usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
-- 
2.14.3

[PATCH v8 02/13] [media] hackrf: group device capabilities

[Gustavo Padovan]

From: Gustavo Padovan 

Instead of putting V4L2_CAP_STREAMING and V4L2_CAP_READWRITE
everywhere, set device_caps earlier with these values.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/usb/hackrf/hackrf.c | 11 ---
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/drivers/media/usb/hackrf/hackrf.c 
b/drivers/media/usb/hackrf/hackrf.c
index 7eb53517a82f..6d692fb3e8dd 100644
--- a/drivers/media/usb/hackrf/hackrf.c
+++ b/drivers/media/usb/hackrf/hackrf.c
@@ -909,18 +909,15 @@ static int hackrf_querycap(struct file *file, void *fh,
 
dev_dbg(>dev, "\n");
 
+   cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
if (vdev->vfl_dir == VFL_DIR_RX)
-   cap->device_caps = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
-  V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
-
+   cap->device_caps |= V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER;
else
-   cap->device_caps = V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-  V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+   cap->device_caps |= V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR;
 
cap->capabilities = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
-   V4L2_CAP_DEVICE_CAPS;
+   V4L2_CAP_DEVICE_CAPS | cap->device_caps;
strlcpy(cap->driver, KBUILD_MODNAME, sizeof(cap->driver));
strlcpy(cap->card, dev->rx_vdev.name, sizeof(cap->card));
usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
-- 
2.14.3

[PATCH v8 05/13] [media] v4l: add 'unordered' flag to format description ioctl

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

v2  - Improve documentation (Hans)

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 7 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 8 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst 
b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..df8e039b9ac2 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,13 @@ one until ``EINVAL`` is returned.
   - This format is not native to the device but emulated through
software (usually libv4l2), where possible try to use a native
format instead for better performance.
+* - ``V4L2_FMT_FLAG_UNORDERED``
+  - 0x0004
+  - This format doesn't guarantee ordered buffer handling. I.e. the order
+   in which buffers are dequeued with
+   :ref:`VIDIOC_DQBUF ` may be different
+   from the order in which they were queued with
+   :ref:`VIDIOC_QBUF `.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 982718965180..58894cfe9479 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -716,6 +716,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
/* Frame Size and frame rate enumeration */
 /*
-- 
2.14.3

[PATCH v8 05/13] [media] v4l: add 'unordered' flag to format description ioctl

[Gustavo Padovan]

From: Gustavo Padovan 

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

v2  - Improve documentation (Hans)

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 7 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 8 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst 
b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..df8e039b9ac2 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,13 @@ one until ``EINVAL`` is returned.
   - This format is not native to the device but emulated through
software (usually libv4l2), where possible try to use a native
format instead for better performance.
+* - ``V4L2_FMT_FLAG_UNORDERED``
+  - 0x0004
+  - This format doesn't guarantee ordered buffer handling. I.e. the order
+   in which buffers are dequeued with
+   :ref:`VIDIOC_DQBUF ` may be different
+   from the order in which they were queued with
+   :ref:`VIDIOC_QBUF `.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 982718965180..58894cfe9479 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -716,6 +716,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
/* Frame Size and frame rate enumeration */
 /*
-- 
2.14.3

[PATCH v8 07/13] [media] vb2: mark codec drivers as unordered

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

In preparation to have full support to explicit fence we are
marking codec as non-ordered preventively. It is easier and safer from an
uAPI point of view to move from unordered to ordered than the opposite.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/platform/coda/coda-common.c  | 1 +
 drivers/media/platform/exynos-gsc/gsc-m2m.c| 1 +
 drivers/media/platform/exynos4-is/fimc-m2m.c   | 1 +
 drivers/media/platform/m2m-deinterlace.c   | 1 +
 drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c| 1 +
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c   | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
 drivers/media/platform/mx2_emmaprp.c   | 1 +
 drivers/media/platform/qcom/venus/vdec.c   | 1 +
 drivers/media/platform/qcom/venus/venc.c   | 1 +
 drivers/media/platform/rcar_fdp1.c | 1 +
 drivers/media/platform/rcar_jpu.c  | 1 +
 drivers/media/platform/rockchip/rga/rga-buf.c  | 1 +
 drivers/media/platform/s5p-g2d/g2d.c   | 1 +
 drivers/media/platform/s5p-jpeg/jpeg-core.c| 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c   | 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c   | 1 +
 drivers/media/platform/sh_veu.c| 1 +
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c  | 1 +
 drivers/media/platform/ti-vpe/vpe.c| 1 +
 drivers/media/platform/vim2m.c | 1 +
 22 files changed, 22 insertions(+)

diff --git a/drivers/media/platform/coda/coda-common.c 
b/drivers/media/platform/coda/coda-common.c
index 04e35d70ce2e..6deb29fe6eb7 100644
--- a/drivers/media/platform/coda/coda-common.c
+++ b/drivers/media/platform/coda/coda-common.c
@@ -1649,6 +1649,7 @@ static const struct vb2_ops coda_qops = {
.stop_streaming = coda_stop_streaming,
.wait_prepare   = vb2_ops_wait_prepare,
.wait_finish= vb2_ops_wait_finish,
+   .is_unordered   = vb2_ops_set_unordered,
 };
 
 static int coda_s_ctrl(struct v4l2_ctrl *ctrl)
diff --git a/drivers/media/platform/exynos-gsc/gsc-m2m.c 
b/drivers/media/platform/exynos-gsc/gsc-m2m.c
index e9ff27949a91..10c3e4659d38 100644
--- a/drivers/media/platform/exynos-gsc/gsc-m2m.c
+++ b/drivers/media/platform/exynos-gsc/gsc-m2m.c
@@ -286,6 +286,7 @@ static const struct vb2_ops gsc_m2m_qops = {
.wait_finish = vb2_ops_wait_finish,
.stop_streaming  = gsc_m2m_stop_streaming,
.start_streaming = gsc_m2m_start_streaming,
+   .is_unordered= vb2_ops_set_unordered,
 };
 
 static int gsc_m2m_querycap(struct file *file, void *fh,
diff --git a/drivers/media/platform/exynos4-is/fimc-m2m.c 
b/drivers/media/platform/exynos4-is/fimc-m2m.c
index a19f8b164a47..dfc487a582c0 100644
--- a/drivers/media/platform/exynos4-is/fimc-m2m.c
+++ b/drivers/media/platform/exynos4-is/fimc-m2m.c
@@ -227,6 +227,7 @@ static const struct vb2_ops fimc_qops = {
.wait_finish = vb2_ops_wait_finish,
.stop_streaming  = stop_streaming,
.start_streaming = start_streaming,
+   .is_unordered= vb2_ops_set_unordered,
 };
 
 /*
diff --git a/drivers/media/platform/m2m-deinterlace.c 
b/drivers/media/platform/m2m-deinterlace.c
index 1e4195144f39..35a0f45d2a51 100644
--- a/drivers/media/platform/m2m-deinterlace.c
+++ b/drivers/media/platform/m2m-deinterlace.c
@@ -856,6 +856,7 @@ static const struct vb2_ops deinterlace_qops = {
.queue_setup = deinterlace_queue_setup,
.buf_prepare = deinterlace_buf_prepare,
.buf_queue   = deinterlace_buf_queue,
+   .is_unordered= vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c 
b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
index 226f90886484..34a4b5b2e1b5 100644
--- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
+++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
@@ -764,6 +764,7 @@ static const struct vb2_ops mtk_jpeg_qops = {
.wait_finish= vb2_ops_wait_finish,
.start_streaming= mtk_jpeg_start_streaming,
.stop_streaming = mtk_jpeg_stop_streaming,
+   .is_unordered   = vb2_ops_set_unordered,
 };
 
 static void mtk_jpeg_set_dec_src(struct mtk_jpeg_ctx *ctx,
diff --git a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c 
b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
index 583d47724ee8..f3bb9f277f55 100644
--- a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
+++ b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
@@ -629,6 +629,7 @@ static const struct vb2_ops mtk_mdp_m2m_qops = {
.wait_finish = mtk_mdp_ctx_lock,
.stop_streaming  = mtk_mdp_m2m_stop_streaming,
.start_streaming = mtk_mdp_m2m_start_streaming,
+   .

[PATCH v8 07/13] [media] vb2: mark codec drivers as unordered

[Gustavo Padovan]

From: Gustavo Padovan 

In preparation to have full support to explicit fence we are
marking codec as non-ordered preventively. It is easier and safer from an
uAPI point of view to move from unordered to ordered than the opposite.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/platform/coda/coda-common.c  | 1 +
 drivers/media/platform/exynos-gsc/gsc-m2m.c| 1 +
 drivers/media/platform/exynos4-is/fimc-m2m.c   | 1 +
 drivers/media/platform/m2m-deinterlace.c   | 1 +
 drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c| 1 +
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c   | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
 drivers/media/platform/mx2_emmaprp.c   | 1 +
 drivers/media/platform/qcom/venus/vdec.c   | 1 +
 drivers/media/platform/qcom/venus/venc.c   | 1 +
 drivers/media/platform/rcar_fdp1.c | 1 +
 drivers/media/platform/rcar_jpu.c  | 1 +
 drivers/media/platform/rockchip/rga/rga-buf.c  | 1 +
 drivers/media/platform/s5p-g2d/g2d.c   | 1 +
 drivers/media/platform/s5p-jpeg/jpeg-core.c| 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c   | 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c   | 1 +
 drivers/media/platform/sh_veu.c| 1 +
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c  | 1 +
 drivers/media/platform/ti-vpe/vpe.c| 1 +
 drivers/media/platform/vim2m.c | 1 +
 22 files changed, 22 insertions(+)

diff --git a/drivers/media/platform/coda/coda-common.c 
b/drivers/media/platform/coda/coda-common.c
index 04e35d70ce2e..6deb29fe6eb7 100644
--- a/drivers/media/platform/coda/coda-common.c
+++ b/drivers/media/platform/coda/coda-common.c
@@ -1649,6 +1649,7 @@ static const struct vb2_ops coda_qops = {
.stop_streaming = coda_stop_streaming,
.wait_prepare   = vb2_ops_wait_prepare,
.wait_finish= vb2_ops_wait_finish,
+   .is_unordered   = vb2_ops_set_unordered,
 };
 
 static int coda_s_ctrl(struct v4l2_ctrl *ctrl)
diff --git a/drivers/media/platform/exynos-gsc/gsc-m2m.c 
b/drivers/media/platform/exynos-gsc/gsc-m2m.c
index e9ff27949a91..10c3e4659d38 100644
--- a/drivers/media/platform/exynos-gsc/gsc-m2m.c
+++ b/drivers/media/platform/exynos-gsc/gsc-m2m.c
@@ -286,6 +286,7 @@ static const struct vb2_ops gsc_m2m_qops = {
.wait_finish = vb2_ops_wait_finish,
.stop_streaming  = gsc_m2m_stop_streaming,
.start_streaming = gsc_m2m_start_streaming,
+   .is_unordered= vb2_ops_set_unordered,
 };
 
 static int gsc_m2m_querycap(struct file *file, void *fh,
diff --git a/drivers/media/platform/exynos4-is/fimc-m2m.c 
b/drivers/media/platform/exynos4-is/fimc-m2m.c
index a19f8b164a47..dfc487a582c0 100644
--- a/drivers/media/platform/exynos4-is/fimc-m2m.c
+++ b/drivers/media/platform/exynos4-is/fimc-m2m.c
@@ -227,6 +227,7 @@ static const struct vb2_ops fimc_qops = {
.wait_finish = vb2_ops_wait_finish,
.stop_streaming  = stop_streaming,
.start_streaming = start_streaming,
+   .is_unordered= vb2_ops_set_unordered,
 };
 
 /*
diff --git a/drivers/media/platform/m2m-deinterlace.c 
b/drivers/media/platform/m2m-deinterlace.c
index 1e4195144f39..35a0f45d2a51 100644
--- a/drivers/media/platform/m2m-deinterlace.c
+++ b/drivers/media/platform/m2m-deinterlace.c
@@ -856,6 +856,7 @@ static const struct vb2_ops deinterlace_qops = {
.queue_setup = deinterlace_queue_setup,
.buf_prepare = deinterlace_buf_prepare,
.buf_queue   = deinterlace_buf_queue,
+   .is_unordered= vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c 
b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
index 226f90886484..34a4b5b2e1b5 100644
--- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
+++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
@@ -764,6 +764,7 @@ static const struct vb2_ops mtk_jpeg_qops = {
.wait_finish= vb2_ops_wait_finish,
.start_streaming= mtk_jpeg_start_streaming,
.stop_streaming = mtk_jpeg_stop_streaming,
+   .is_unordered   = vb2_ops_set_unordered,
 };
 
 static void mtk_jpeg_set_dec_src(struct mtk_jpeg_ctx *ctx,
diff --git a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c 
b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
index 583d47724ee8..f3bb9f277f55 100644
--- a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
+++ b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
@@ -629,6 +629,7 @@ static const struct vb2_ops mtk_mdp_m2m_qops = {
.wait_finish = mtk_mdp_ctx_lock,
.stop_streaming  = mtk_mdp_m2m_stop_streaming,
.start_streaming = mtk_mdp_m2m_start_streaming,
+   .is_unordered= vb2_ops_set_unordered,
 };
 
 static int mtk_mdp_m2m_querycap(struct file *file

[PATCH v8 06/13] [media] cobalt: add .is_unordered() for cobalt

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

The cobalt driver may reorder the capture buffers so we need to report
it as such.

v2: - use vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/pci/cobalt/cobalt-v4l2.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c 
b/drivers/media/pci/cobalt/cobalt-v4l2.c
index e2a4c705d353..6b6611a0e190 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -430,6 +430,7 @@ static const struct vb2_ops cobalt_qops = {
.stop_streaming = cobalt_stop_streaming,
.wait_prepare = vb2_ops_wait_prepare,
.wait_finish = vb2_ops_wait_finish,
+   .is_unordered = vb2_ops_set_unordered,
 };
 
 /* V4L2 ioctls */
-- 
2.14.3

[PATCH v8 06/13] [media] cobalt: add .is_unordered() for cobalt

[Gustavo Padovan]

From: Gustavo Padovan 

The cobalt driver may reorder the capture buffers so we need to report
it as such.

v2: - use vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan 
---
 drivers/media/pci/cobalt/cobalt-v4l2.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c 
b/drivers/media/pci/cobalt/cobalt-v4l2.c
index e2a4c705d353..6b6611a0e190 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -430,6 +430,7 @@ static const struct vb2_ops cobalt_qops = {
.stop_streaming = cobalt_stop_streaming,
.wait_prepare = vb2_ops_wait_prepare,
.wait_finish = vb2_ops_wait_finish,
+   .is_unordered = vb2_ops_set_unordered,
 };
 
 /* V4L2 ioctls */
-- 
2.14.3

[PATCH v8 11/13] [media] v4l: introduce the fences capability

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Drivers capable of using fences (vb2 drivers) should report the
V4L2_CAP_FENCES to userspace, so add this flag to the uapi.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-querycap.rst | 3 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-querycap.rst 
b/Documentation/media/uapi/v4l/vidioc-querycap.rst
index 66fb1b3d6e6e..414016065309 100644
--- a/Documentation/media/uapi/v4l/vidioc-querycap.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querycap.rst
@@ -254,6 +254,9 @@ specification the ioctl returns an ``EINVAL`` error code.
 * - ``V4L2_CAP_TOUCH``
   - 0x1000
   - This is a touch device.
+* - ``V4L2_CAP_FENCES``
+  - 0x2000
+  - The device support explicit synchronization.
 * - ``V4L2_CAP_DEVICE_CAPS``
   - 0x8000
   - The driver fills the ``device_caps`` field. This capability can
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 2d424aebdd1e..db58204e346e 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -460,6 +460,7 @@ struct v4l2_capability {
 #define V4L2_CAP_STREAMING  0x0400  /* streaming I/O ioctls */
 
 #define V4L2_CAP_TOUCH  0x1000  /* Is a touch device */
+#define V4L2_CAP_FENCES 0x2000  /* Supports explicit 
synchronization */
 
 #define V4L2_CAP_DEVICE_CAPS0x8000  /* sets device 
capabilities field */
 
-- 
2.14.3

[PATCH v8 08/13] [media] vb2: add explicit fence user API

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel or return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending a fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v6: - big improvement on doc (Hans Verkuil)

v5:
- keep using reserved2 field for cpia2
- set fence_fd to 0 for now, for compat with userspace(Mauro)

v4:
- make it a union with reserved2 and fence_fd (Hans Verkuil)

v3:
- make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/buffer.rst | 45 +++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
 include/uapi/linux/videodev2.h  |  7 +++-
 4 files changed, 51 insertions(+), 7 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index e2c85ddc990b..49273026740f 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,22 @@ struct v4l2_buffer
elements in the ``planes`` array. The driver will fill in the
actual number of valid elements in that array.
 * - __u32
-  - ``reserved2``
+  - ``fence_fd``
   -
-  - A place holder for future extensions. Drivers and applications
-   must set this to 0.
+  - Used to communicate fences file descriptors from userspace to kernel
+   and vice-versa. On :ref:`VIDIOC_QBUF ` when sending
+   an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag 
must
+   be used and this field set to the fence file descriptor of the in-fence
+   If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
+   must be set, the kernel will return the out-fence file descriptor on
+   this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
+an error.
+
+   In all other ioctls V4L2 sets this field to -1 if
+   ``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
+   otherwise this field is set to 0 for backward compatibility.
 * - __u32
   - ``reserved``
   -
@@ -648,6 +660,33 @@ Buffer Flags
   - Start Of Exposure. The buffer timestamp has been taken when the
exposure of the frame has begun. This is only valid for the
``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+* .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+  - ``V4L2_BUF_FLAG_IN_FENCE``
+  - 0x0020
+  - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
+   buffer won't be queued to the driver until the fence signals. The order
+   in which buffers are queued is guaranteed to be preserved, so any
+   buffers queued after this buffer will also be blocked until this fence
+   signals. This flag must be set before calling ``VIDIOC_QBUF``. For
+   other ioctls the driver just report the value of the flag.
+
+If the fence signals the flag is cleared and not reported anymore.
+   If the fence is not valid ``VIDIOC_QBUF`` returns an error.
+
+
+* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+  - ``V4L2_BUF_FLAG_OUT_FENCE``
+  - 0x0040
+  - Request for a fence to be attached to the buffer. The driver will fill
+   in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
+   ` returns. This flag must be set before calling
+   ``VIDIOC_QBUF``. For other ioctls the driver just report the value of
+   the flag.
+
+If the creation of the  out-fence  fails ``VIDIOC_QBUF`` returns an
+   error.
 
 
 
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 68291ba8632d..ad1e032c3bf5 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->fence_fd = 0;
b->reserved = 0;
 
if (q->is_multiplanar) {
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 5198c9eeb348..3de2252e3632 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -386,7 +386,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;

[PATCH v8 08/13] [media] vb2: add explicit fence user API

[Gustavo Padovan]

From: Gustavo Padovan 

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel or return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending a fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v6: - big improvement on doc (Hans Verkuil)

v5:
- keep using reserved2 field for cpia2
- set fence_fd to 0 for now, for compat with userspace(Mauro)

v4:
- make it a union with reserved2 and fence_fd (Hans Verkuil)

v3:
- make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/buffer.rst | 45 +++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
 include/uapi/linux/videodev2.h  |  7 +++-
 4 files changed, 51 insertions(+), 7 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index e2c85ddc990b..49273026740f 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,22 @@ struct v4l2_buffer
elements in the ``planes`` array. The driver will fill in the
actual number of valid elements in that array.
 * - __u32
-  - ``reserved2``
+  - ``fence_fd``
   -
-  - A place holder for future extensions. Drivers and applications
-   must set this to 0.
+  - Used to communicate fences file descriptors from userspace to kernel
+   and vice-versa. On :ref:`VIDIOC_QBUF ` when sending
+   an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag 
must
+   be used and this field set to the fence file descriptor of the in-fence
+   If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
+   must be set, the kernel will return the out-fence file descriptor on
+   this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
+an error.
+
+   In all other ioctls V4L2 sets this field to -1 if
+   ``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
+   otherwise this field is set to 0 for backward compatibility.
 * - __u32
   - ``reserved``
   -
@@ -648,6 +660,33 @@ Buffer Flags
   - Start Of Exposure. The buffer timestamp has been taken when the
exposure of the frame has begun. This is only valid for the
``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+* .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+  - ``V4L2_BUF_FLAG_IN_FENCE``
+  - 0x0020
+  - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
+   buffer won't be queued to the driver until the fence signals. The order
+   in which buffers are queued is guaranteed to be preserved, so any
+   buffers queued after this buffer will also be blocked until this fence
+   signals. This flag must be set before calling ``VIDIOC_QBUF``. For
+   other ioctls the driver just report the value of the flag.
+
+If the fence signals the flag is cleared and not reported anymore.
+   If the fence is not valid ``VIDIOC_QBUF`` returns an error.
+
+
+* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+  - ``V4L2_BUF_FLAG_OUT_FENCE``
+  - 0x0040
+  - Request for a fence to be attached to the buffer. The driver will fill
+   in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
+   ` returns. This flag must be set before calling
+   ``VIDIOC_QBUF``. For other ioctls the driver just report the value of
+   the flag.
+
+If the creation of the  out-fence  fails ``VIDIOC_QBUF`` returns an
+   error.
 
 
 
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 68291ba8632d..ad1e032c3bf5 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->fence_fd = 0;
b->reserved = 0;
 
if (q->is_multiplanar) {
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 5198c9eeb348..3de2252e3632 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -386,7 +386,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __s32

[PATCH v8 11/13] [media] v4l: introduce the fences capability

[Gustavo Padovan]

From: Gustavo Padovan 

Drivers capable of using fences (vb2 drivers) should report the
V4L2_CAP_FENCES to userspace, so add this flag to the uapi.

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/vidioc-querycap.rst | 3 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-querycap.rst 
b/Documentation/media/uapi/v4l/vidioc-querycap.rst
index 66fb1b3d6e6e..414016065309 100644
--- a/Documentation/media/uapi/v4l/vidioc-querycap.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querycap.rst
@@ -254,6 +254,9 @@ specification the ioctl returns an ``EINVAL`` error code.
 * - ``V4L2_CAP_TOUCH``
   - 0x1000
   - This is a touch device.
+* - ``V4L2_CAP_FENCES``
+  - 0x2000
+  - The device support explicit synchronization.
 * - ``V4L2_CAP_DEVICE_CAPS``
   - 0x8000
   - The driver fills the ``device_caps`` field. This capability can
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 2d424aebdd1e..db58204e346e 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -460,6 +460,7 @@ struct v4l2_capability {
 #define V4L2_CAP_STREAMING  0x0400  /* streaming I/O ioctls */
 
 #define V4L2_CAP_TOUCH  0x1000  /* Is a touch device */
+#define V4L2_CAP_FENCES 0x2000  /* Supports explicit 
synchronization */
 
 #define V4L2_CAP_DEVICE_CAPS0x8000  /* sets device 
capabilities field */
 
-- 
2.14.3

[PATCH v8 12/13] [media] v4l: Add V4L2_CAP_FENCES to drivers

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Drivers that use videobuf2 are capable of using fences and
should report that to userspace.

The coding style is following what each drivers was already
doing.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/pci/cobalt/cobalt-v4l2.c | 3 ++-
 drivers/media/pci/cx23885/cx23885-417.c| 2 +-
 drivers/media/pci/cx23885/cx23885-video.c  | 3 ++-
 drivers/media/pci/cx88/cx88-video.c| 3 ++-
 drivers/media/pci/dt3155/dt3155.c  | 2 +-
 drivers/media/pci/saa7134/saa7134-video.c  | 2 ++
 drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c | 3 ++-
 drivers/media/pci/solo6x10/solo6x10-v4l2.c | 3 ++-
 drivers/media/pci/sta2x11/sta2x11_vip.c| 2 +-
 drivers/media/pci/tw68/tw68-video.c| 3 ++-
 drivers/media/pci/tw686x/tw686x-video.c| 2 +-
 drivers/media/platform/am437x/am437x-vpfe.c| 2 +-
 drivers/media/platform/blackfin/bfin_capture.c | 3 ++-
 drivers/media/platform/coda/coda-common.c  | 3 ++-
 drivers/media/platform/davinci/vpbe_display.c  | 3 ++-
 drivers/media/platform/davinci/vpfe_capture.c  | 3 ++-
 drivers/media/platform/davinci/vpif_capture.c  | 3 ++-
 drivers/media/platform/davinci/vpif_display.c  | 3 ++-
 drivers/media/platform/exynos-gsc/gsc-m2m.c| 3 ++-
 drivers/media/platform/exynos4-is/fimc-capture.c   | 3 ++-
 drivers/media/platform/exynos4-is/fimc-isp-video.c | 3 ++-
 drivers/media/platform/exynos4-is/fimc-lite.c  | 2 +-
 drivers/media/platform/exynos4-is/fimc-m2m.c   | 3 ++-
 drivers/media/platform/m2m-deinterlace.c   | 3 ++-
 drivers/media/platform/marvell-ccic/mcam-core.c| 2 +-
 drivers/media/platform/mx2_emmaprp.c   | 3 ++-
 drivers/media/platform/omap3isp/ispvideo.c | 2 +-
 drivers/media/platform/pxa_camera.c| 3 ++-
 drivers/media/platform/rcar_jpu.c  | 3 ++-
 drivers/media/platform/s3c-camif/camif-capture.c   | 3 ++-
 drivers/media/platform/s5p-g2d/g2d.c   | 3 ++-
 drivers/media/platform/s5p-jpeg/jpeg-core.c| 3 ++-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c   | 3 ++-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c   | 3 ++-
 drivers/media/platform/sh_veu.c| 3 ++-
 drivers/media/platform/sh_vou.c| 2 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c  | 3 ++-
 drivers/media/platform/ti-vpe/cal.c| 2 +-
 drivers/media/platform/ti-vpe/vpe.c| 3 ++-
 drivers/media/platform/vim2m.c | 3 ++-
 drivers/media/platform/vivid/vivid-core.c  | 2 +-
 drivers/media/platform/vsp1/vsp1_histo.c   | 2 +-
 drivers/media/platform/vsp1/vsp1_video.c   | 2 +-
 drivers/media/platform/xilinx/xilinx-dma.c | 2 +-
 drivers/media/usb/airspy/airspy.c  | 2 +-
 drivers/media/usb/au0828/au0828-video.c| 3 ++-
 drivers/media/usb/em28xx/em28xx-video.c| 1 +
 drivers/media/usb/go7007/go7007-v4l2.c | 2 +-
 drivers/media/usb/hackrf/hackrf.c  | 3 ++-
 drivers/media/usb/msi2500/msi2500.c| 2 +-
 drivers/media/usb/pwc/pwc-v4l.c| 2 +-
 drivers/media/usb/s2255/s2255drv.c | 2 +-
 drivers/media/usb/stk1160/stk1160-v4l.c| 3 ++-
 drivers/media/usb/usbtv/usbtv-video.c  | 3 ++-
 drivers/media/usb/uvc/uvc_driver.c | 1 +
 55 files changed, 89 insertions(+), 52 deletions(-)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c 
b/drivers/media/pci/cobalt/cobalt-v4l2.c
index 6b6611a0e190..ef1014b5d4a7 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -484,7 +484,8 @@ static int cobalt_querycap(struct file *file, void *priv_fh,
strlcpy(vcap->card, "cobalt", sizeof(vcap->card));
snprintf(vcap->bus_info, sizeof(vcap->bus_info),
 "PCIe:%s", pci_name(cobalt->pci_dev));
-   vcap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+   vcap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
+   V4L2_CAP_FENCES;
if (s->is_output)
vcap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
else
diff --git a/drivers/media/pci/cx23885/cx23885-417.c 
b/drivers/media/pci/cx23885/cx23885-417.c
index a71f3c7569ce..56bf7ec4e25f 100644
--- a/drivers/media/pci/cx23885/cx23885-417.c
+++ b/drivers/media/pci/cx23885/cx23885-417.c
@@ -1334,7 +1334,7 @@ static int vidioc_querycap(struct file *file, void  *priv,
sizeof(cap->card));
sprintf(cap->bus_info, "PCIe:%s", pci_name(dev->pci));
cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE |
-  V4L2_CAP_STREAMING;
+  

[PATCH v8 12/13] [media] v4l: Add V4L2_CAP_FENCES to drivers

[Gustavo Padovan]

From: Gustavo Padovan 

Drivers that use videobuf2 are capable of using fences and
should report that to userspace.

The coding style is following what each drivers was already
doing.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/pci/cobalt/cobalt-v4l2.c | 3 ++-
 drivers/media/pci/cx23885/cx23885-417.c| 2 +-
 drivers/media/pci/cx23885/cx23885-video.c  | 3 ++-
 drivers/media/pci/cx88/cx88-video.c| 3 ++-
 drivers/media/pci/dt3155/dt3155.c  | 2 +-
 drivers/media/pci/saa7134/saa7134-video.c  | 2 ++
 drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c | 3 ++-
 drivers/media/pci/solo6x10/solo6x10-v4l2.c | 3 ++-
 drivers/media/pci/sta2x11/sta2x11_vip.c| 2 +-
 drivers/media/pci/tw68/tw68-video.c| 3 ++-
 drivers/media/pci/tw686x/tw686x-video.c| 2 +-
 drivers/media/platform/am437x/am437x-vpfe.c| 2 +-
 drivers/media/platform/blackfin/bfin_capture.c | 3 ++-
 drivers/media/platform/coda/coda-common.c  | 3 ++-
 drivers/media/platform/davinci/vpbe_display.c  | 3 ++-
 drivers/media/platform/davinci/vpfe_capture.c  | 3 ++-
 drivers/media/platform/davinci/vpif_capture.c  | 3 ++-
 drivers/media/platform/davinci/vpif_display.c  | 3 ++-
 drivers/media/platform/exynos-gsc/gsc-m2m.c| 3 ++-
 drivers/media/platform/exynos4-is/fimc-capture.c   | 3 ++-
 drivers/media/platform/exynos4-is/fimc-isp-video.c | 3 ++-
 drivers/media/platform/exynos4-is/fimc-lite.c  | 2 +-
 drivers/media/platform/exynos4-is/fimc-m2m.c   | 3 ++-
 drivers/media/platform/m2m-deinterlace.c   | 3 ++-
 drivers/media/platform/marvell-ccic/mcam-core.c| 2 +-
 drivers/media/platform/mx2_emmaprp.c   | 3 ++-
 drivers/media/platform/omap3isp/ispvideo.c | 2 +-
 drivers/media/platform/pxa_camera.c| 3 ++-
 drivers/media/platform/rcar_jpu.c  | 3 ++-
 drivers/media/platform/s3c-camif/camif-capture.c   | 3 ++-
 drivers/media/platform/s5p-g2d/g2d.c   | 3 ++-
 drivers/media/platform/s5p-jpeg/jpeg-core.c| 3 ++-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c   | 3 ++-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c   | 3 ++-
 drivers/media/platform/sh_veu.c| 3 ++-
 drivers/media/platform/sh_vou.c| 2 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c  | 3 ++-
 drivers/media/platform/ti-vpe/cal.c| 2 +-
 drivers/media/platform/ti-vpe/vpe.c| 3 ++-
 drivers/media/platform/vim2m.c | 3 ++-
 drivers/media/platform/vivid/vivid-core.c  | 2 +-
 drivers/media/platform/vsp1/vsp1_histo.c   | 2 +-
 drivers/media/platform/vsp1/vsp1_video.c   | 2 +-
 drivers/media/platform/xilinx/xilinx-dma.c | 2 +-
 drivers/media/usb/airspy/airspy.c  | 2 +-
 drivers/media/usb/au0828/au0828-video.c| 3 ++-
 drivers/media/usb/em28xx/em28xx-video.c| 1 +
 drivers/media/usb/go7007/go7007-v4l2.c | 2 +-
 drivers/media/usb/hackrf/hackrf.c  | 3 ++-
 drivers/media/usb/msi2500/msi2500.c| 2 +-
 drivers/media/usb/pwc/pwc-v4l.c| 2 +-
 drivers/media/usb/s2255/s2255drv.c | 2 +-
 drivers/media/usb/stk1160/stk1160-v4l.c| 3 ++-
 drivers/media/usb/usbtv/usbtv-video.c  | 3 ++-
 drivers/media/usb/uvc/uvc_driver.c | 1 +
 55 files changed, 89 insertions(+), 52 deletions(-)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c 
b/drivers/media/pci/cobalt/cobalt-v4l2.c
index 6b6611a0e190..ef1014b5d4a7 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -484,7 +484,8 @@ static int cobalt_querycap(struct file *file, void *priv_fh,
strlcpy(vcap->card, "cobalt", sizeof(vcap->card));
snprintf(vcap->bus_info, sizeof(vcap->bus_info),
 "PCIe:%s", pci_name(cobalt->pci_dev));
-   vcap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+   vcap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
+   V4L2_CAP_FENCES;
if (s->is_output)
vcap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
else
diff --git a/drivers/media/pci/cx23885/cx23885-417.c 
b/drivers/media/pci/cx23885/cx23885-417.c
index a71f3c7569ce..56bf7ec4e25f 100644
--- a/drivers/media/pci/cx23885/cx23885-417.c
+++ b/drivers/media/pci/cx23885/cx23885-417.c
@@ -1334,7 +1334,7 @@ static int vidioc_querycap(struct file *file, void  *priv,
sizeof(cap->card));
sprintf(cap->bus_info, "PCIe:%s", pci_name(dev->pci));
cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE |
-  V4L2_CAP_STREAMING;
+  V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
if (dev->tuner_

[PATCH v8 13/13] [media] v4l: Document explicit synchronization behavior

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it

v6: - Close some gaps in the docs (Hans)

v5:
- Remove V4L2_CAP_ORDERED
- Add doc about V4L2_FMT_FLAG_UNORDERED

v4:
- Document ordering behavior for in-fences
- Document V4L2_CAP_ORDERED capability
- Remove doc about OUT_FENCE event
- Document immediate return of out-fence in QBUF

v3:
- make the out_fence refer to the current buffer (Hans)
- Note what happens when the IN_FENCE is not set (Hans)

v2:
- mention that fences are files (Hans)
- rework for the new API

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst | 55 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 --
 2 files changed, 63 insertions(+), 4 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst 
b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..371d84966e34 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API `, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,59 @@ immediately with an ``EAGAIN`` error code when no buffer 
is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer. On the other
+side, the kernel can create out-fences for the buffers it queues to the
+drivers. Out-fences signal when the driver is finished with buffer, i.e., the
+buffer is ready. The fences are represented as a file and passed as a file
+descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with an error. The fence_fd field will be ignored if the
+``V4L2_BUF_FLAG_IN_FENCE`` is not set.
+
+The videobuf2-core will guarantee that all buffers queued with an in-fence will
+be queued to the drivers in the same order. Fences may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering. So when
+waiting on a fence to signal all buffers queued after will be also block until
+that fence signal.
+
+If the in-fence signals with an error the buffer will be marked with
+``V4L2_BUF_FLAG_ERROR`` when returned to userspace at ``VIDIOC_DQBUF``.
+Even with the error the order of dequeueing the buffers are preserved.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.
+
+Note the the same `fence_fd` field is used for both sending the in-fence as
+input argument to receive the out-fence as a return argument. A buffer can
+have both in-fence ond out-fence.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. They will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the  when using this format the order in which buffers are dequeued can
+be different from the order in which they were queued.
+
+Ordering is important to fences because it can optimize the pipeline with
+other drivers like a DRM/KMS display driver. For example, if a capture from the
+camera is happening in an orderly manner one can send the capture buffer
+out-fence to the DRM/KMS driver and rest sure that the buffers will be shown on
+the screen at the correct order. If an ordered queue can not be set then such
+arrangements with other drivers may not be possible.
 
 Return Value
 ===

[PATCH v8 10/13] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace on the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v9: - remove in-fences changes from this patch (Alex Courbot)
- improve fence context creation (Hans Verkuil)
- clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)

v8:
- return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
- fix crash when checking not using fences in vb2_buffer_done()

v7:
- merge patch that add the infrastructure to out-fences into
this one (Alex Courbot)
- Do not install the fd if there is no fence. (Alex Courbot)
- do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6
- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
ordering in vb2 for queueing in the driver, so the event is not
necessary anymore and the out_fence_fd is sent back to userspace
on QBUF call return arg
- do not allow requeueing with out-fences, instead mark the buffer
with an error and wake up to userspace.
- send the out_fence_fd back to userspace on the fence_fd field

v5:
- delay fd_install to DQ_EVENT (Hans)
- if queue is fully ordered send OUT_FENCE event right away
(Brian)
- rename 'q->ordered' to 'q->ordered_in_driver'
- merge change to implement OUT_FENCE event here

v4:
- return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
- set the OUT_FENCE flag if there is a fence pending (Hans)
- call fd_install() after vb2_core_qbuf() (Hans)
- clean up fence if vb2_core_qbuf() fails (Hans)
- add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 88 +
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 20 +-
 include/media/videobuf2-core.h  | 25 +++
 3 files changed, 132 insertions(+), 1 deletion(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c 
b/drivers/media/common/videobuf2/videobuf2-core.c
index 5de5e35cfc40..dd18a9f345c7 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -25,6 +25,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #include 
@@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
}
+   vb->out_fence_fd = -1;
q->bufs[vb->index] = vb;
 
/* Allocate video buffer memory for the MMAP type */
@@ -934,10 +936,22 @@ static void vb2_process_buffer_done(struct vb2_buffer 
*vb, enum vb2_buffer_state
case VB2_BUF_STATE_QUEUED:
return;
case VB2_BUF_STATE_REQUEUEING:
+   /* Requeuing with explicit synchronization, spit warning */
+   WARN_ON_ONCE(vb->out_fence);
+
if (q->start_streaming_called)
__enqueue_in_driver(vb);
return;
default:
+   if (vb->out_fence) {
+   if (state == VB2_BUF_STATE_ERROR)
+   dma_fence_set_error(vb->out_fence, -EFAULT);
+   dma_fence_signal(vb->out_fence);
+   dma_fence_put(vb->out_fence);
+   vb->out_fence = NULL;
+   vb->out_fence_fd = -1;
+   }
+
/* Inform any processes that may be waiting for buffers */
wake_up(>done_wq);
break;
@@ -1353,6 +1367,62 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+   return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+   return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+   return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+   .get_driver_name = vb2_fence_get_driver_name,
+   .get_timeline_name = vb2_fence_get_timeline_name,
+   .enable_signaling = vb2_fence_enable_signaling,
+   .wait = dma_fence_default_wait,
+};
+
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
+{
+

[PATCH v8 13/13] [media] v4l: Document explicit synchronization behavior

[Gustavo Padovan]

From: Gustavo Padovan 

Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it

v6: - Close some gaps in the docs (Hans)

v5:
- Remove V4L2_CAP_ORDERED
- Add doc about V4L2_FMT_FLAG_UNORDERED

v4:
- Document ordering behavior for in-fences
- Document V4L2_CAP_ORDERED capability
- Remove doc about OUT_FENCE event
- Document immediate return of out-fence in QBUF

v3:
- make the out_fence refer to the current buffer (Hans)
- Note what happens when the IN_FENCE is not set (Hans)

v2:
- mention that fences are files (Hans)
- rework for the new API

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst | 55 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 --
 2 files changed, 63 insertions(+), 4 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst 
b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..371d84966e34 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API `, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,59 @@ immediately with an ``EAGAIN`` error code when no buffer 
is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer. On the other
+side, the kernel can create out-fences for the buffers it queues to the
+drivers. Out-fences signal when the driver is finished with buffer, i.e., the
+buffer is ready. The fences are represented as a file and passed as a file
+descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with an error. The fence_fd field will be ignored if the
+``V4L2_BUF_FLAG_IN_FENCE`` is not set.
+
+The videobuf2-core will guarantee that all buffers queued with an in-fence will
+be queued to the drivers in the same order. Fences may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering. So when
+waiting on a fence to signal all buffers queued after will be also block until
+that fence signal.
+
+If the in-fence signals with an error the buffer will be marked with
+``V4L2_BUF_FLAG_ERROR`` when returned to userspace at ``VIDIOC_DQBUF``.
+Even with the error the order of dequeueing the buffers are preserved.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.
+
+Note the the same `fence_fd` field is used for both sending the in-fence as
+input argument to receive the out-fence as a return argument. A buffer can
+have both in-fence ond out-fence.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. They will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the  when using this format the order in which buffers are dequeued can
+be different from the order in which they were queued.
+
+Ordering is important to fences because it can optimize the pipeline with
+other drivers like a DRM/KMS display driver. For example, if a capture from the
+camera is happening in an orderly manner one can send the capture buffer
+out-fence to the DRM/KMS driver and rest sure that the buffers will be shown on
+the screen at the correct order. If an ordered queue can not be set then such
+arrangements with other drivers may not be possible.
 
 Return Value
 
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst 
b

[PATCH v8 10/13] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan 

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace on the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v9: - remove in-fences changes from this patch (Alex Courbot)
- improve fence context creation (Hans Verkuil)
- clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)

v8:
- return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
- fix crash when checking not using fences in vb2_buffer_done()

v7:
- merge patch that add the infrastructure to out-fences into
this one (Alex Courbot)
- Do not install the fd if there is no fence. (Alex Courbot)
- do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6
- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
ordering in vb2 for queueing in the driver, so the event is not
necessary anymore and the out_fence_fd is sent back to userspace
on QBUF call return arg
- do not allow requeueing with out-fences, instead mark the buffer
with an error and wake up to userspace.
- send the out_fence_fd back to userspace on the fence_fd field

v5:
- delay fd_install to DQ_EVENT (Hans)
- if queue is fully ordered send OUT_FENCE event right away
(Brian)
- rename 'q->ordered' to 'q->ordered_in_driver'
- merge change to implement OUT_FENCE event here

v4:
- return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
- set the OUT_FENCE flag if there is a fence pending (Hans)
- call fd_install() after vb2_core_qbuf() (Hans)
- clean up fence if vb2_core_qbuf() fails (Hans)
- add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/common/videobuf2/videobuf2-core.c | 88 +
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 20 +-
 include/media/videobuf2-core.h  | 25 +++
 3 files changed, 132 insertions(+), 1 deletion(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c 
b/drivers/media/common/videobuf2/videobuf2-core.c
index 5de5e35cfc40..dd18a9f345c7 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -25,6 +25,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #include 
@@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
}
+   vb->out_fence_fd = -1;
q->bufs[vb->index] = vb;
 
/* Allocate video buffer memory for the MMAP type */
@@ -934,10 +936,22 @@ static void vb2_process_buffer_done(struct vb2_buffer 
*vb, enum vb2_buffer_state
case VB2_BUF_STATE_QUEUED:
return;
case VB2_BUF_STATE_REQUEUEING:
+   /* Requeuing with explicit synchronization, spit warning */
+   WARN_ON_ONCE(vb->out_fence);
+
if (q->start_streaming_called)
__enqueue_in_driver(vb);
return;
default:
+   if (vb->out_fence) {
+   if (state == VB2_BUF_STATE_ERROR)
+   dma_fence_set_error(vb->out_fence, -EFAULT);
+   dma_fence_signal(vb->out_fence);
+   dma_fence_put(vb->out_fence);
+   vb->out_fence = NULL;
+   vb->out_fence_fd = -1;
+   }
+
/* Inform any processes that may be waiting for buffers */
wake_up(>done_wq);
break;
@@ -1353,6 +1367,62 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+   return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+   return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+   return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+   .get_driver_name = vb2_fence_get_driver_name,
+   .get_timeline_name = vb2_fence_get_timeline_name,
+   .enable_signaling = vb2_fence_enable_signaling,
+   .wait = dma_fence_default_wait,
+};
+
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
+{
+   struct vb2_buffer *vb;
+
+   vb = q->bufs[index];
+
+   v

[PATCH v8 09/13] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queue to the driver
out of the order they were queued from userspace. That means that even if
it fence signal it must wait all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v9: - rename fence to in_fence in many places
- handle fences signalling with error better (Hans Verkuil)

v8: - improve comments and docs (Hans Verkuil)
- fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
- move in-fences code that was in the out-fences patch here (Alex)

v8: - improve comments about fences with errors

v7:
- get rid of the fence array stuff for ordering and just use
get_num_buffers_ready() (Hans)
- fix issue of queuing the buffer twice (Hans)
- avoid the dma_fence_wait() in core_qbuf() (Alex)
- merge preparation commit in

v6:
- With fences always keep the order userspace queues the buffers.
- Protect in_fence manipulation with a lock (Brian Starkey)
- check if fences have the same context before adding a fence array
- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
- Clean up fence if __set_in_fence() fails (Brian Starkey)
- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
needed (Brian Starkey)
- keep backward compat on the reserved2 field (Brian Starkey)
- protect fence callback removal with lock (Brian Starkey)

v4:
- Add a comment about dma_fence_add_callback() not returning a
error (Hans)
- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
vb2_start_streaming() (Hans)
- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
- Queue buffers to the driver as soon as they are ready (Hans)
- call fill_user_buffer() after queuing the buffer (Hans)
- add err: label to clean up fence
- add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
- remove ternary if at vb2_qbuf() return (Mauro)
- do not change if conditions behaviour (Mauro)

v2:
- fix vb2_queue_or_prepare_buf() ret check
- remove check for VB2_MEMORY_DMABUF only (Javier)
- check num of ready buffers to start streaming
- when queueing, start from the first ready buffer
- handle queue cancel

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 197 
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
 drivers/media/v4l2-core/Kconfig |  33 
 include/media/videobuf2-core.h  |  14 +-
 4 files changed, 248 insertions(+), 30 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c 
b/drivers/media/common/videobuf2/videobuf2-core.c
index d3f7bb33a54d..5de5e35cfc40 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->index = q->num_buffers + buffer;
vb->type = q->type;
vb->memory = memory;
+   spin_lock_init(>fence_cb_lock);
for (plane = 0; plane < num_planes; ++plane) {
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
@@ -891,20 +892,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned 
int plane_no)
 }
 EXPORT_SYMBOL_GPL(vb2_plane_cookie);
 
-void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+static void vb2_process_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
 {
struct vb2_queue *q = vb->vb2_queue;
unsigned long flags;
unsigned int plane;
 
-   if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
-   return;
-
-   if (WARN_ON(state != VB2_BUF_STATE_DONE &&
-   state != VB2_BUF_STATE_ERROR &&
-   state != VB2_BUF

[PATCH v8 09/13] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan 

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queue to the driver
out of the order they were queued from userspace. That means that even if
it fence signal it must wait all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v9: - rename fence to in_fence in many places
- handle fences signalling with error better (Hans Verkuil)

v8: - improve comments and docs (Hans Verkuil)
- fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
- move in-fences code that was in the out-fences patch here (Alex)

v8: - improve comments about fences with errors

v7:
- get rid of the fence array stuff for ordering and just use
get_num_buffers_ready() (Hans)
- fix issue of queuing the buffer twice (Hans)
- avoid the dma_fence_wait() in core_qbuf() (Alex)
- merge preparation commit in

v6:
- With fences always keep the order userspace queues the buffers.
- Protect in_fence manipulation with a lock (Brian Starkey)
- check if fences have the same context before adding a fence array
- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
- Clean up fence if __set_in_fence() fails (Brian Starkey)
- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
needed (Brian Starkey)
- keep backward compat on the reserved2 field (Brian Starkey)
- protect fence callback removal with lock (Brian Starkey)

v4:
- Add a comment about dma_fence_add_callback() not returning a
error (Hans)
- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
vb2_start_streaming() (Hans)
- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
- Queue buffers to the driver as soon as they are ready (Hans)
- call fill_user_buffer() after queuing the buffer (Hans)
- add err: label to clean up fence
- add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
- remove ternary if at vb2_qbuf() return (Mauro)
- do not change if conditions behaviour (Mauro)

v2:
- fix vb2_queue_or_prepare_buf() ret check
- remove check for VB2_MEMORY_DMABUF only (Javier)
- check num of ready buffers to start streaming
- when queueing, start from the first ready buffer
- handle queue cancel

Signed-off-by: Gustavo Padovan 
---
 drivers/media/common/videobuf2/videobuf2-core.c | 197 
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
 drivers/media/v4l2-core/Kconfig |  33 
 include/media/videobuf2-core.h  |  14 +-
 4 files changed, 248 insertions(+), 30 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c 
b/drivers/media/common/videobuf2/videobuf2-core.c
index d3f7bb33a54d..5de5e35cfc40 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->index = q->num_buffers + buffer;
vb->type = q->type;
vb->memory = memory;
+   spin_lock_init(>fence_cb_lock);
for (plane = 0; plane < num_planes; ++plane) {
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
@@ -891,20 +892,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned 
int plane_no)
 }
 EXPORT_SYMBOL_GPL(vb2_plane_cookie);
 
-void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+static void vb2_process_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
 {
struct vb2_queue *q = vb->vb2_queue;
unsigned long flags;
unsigned int plane;
 
-   if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
-   return;
-
-   if (WARN_ON(state != VB2_BUF_STATE_DONE &&
-   state != VB2_BUF_STATE_ERROR &&
-   state != VB2_BUF_STATE_QUEUED &&
-   state != VB2_BUF_STATE_R

[PATCH v8 03/13] [media] omap3isp: group device capabilities

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Instead of putting V4L2_CAP_STREAMING everywhere, set device_caps
earlier with this value.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/platform/omap3isp/ispvideo.c | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/omap3isp/ispvideo.c 
b/drivers/media/platform/omap3isp/ispvideo.c
index a751c89a3ea8..b4d4ef926749 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -658,13 +658,14 @@ isp_video_querycap(struct file *file, void *fh, struct 
v4l2_capability *cap)
strlcpy(cap->card, video->video.name, sizeof(cap->card));
strlcpy(cap->bus_info, "media", sizeof(cap->bus_info));
 
-   cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT
-   | V4L2_CAP_STREAMING | V4L2_CAP_DEVICE_CAPS;
+   cap->device_caps = V4L2_CAP_STREAMING;
+   cap->capabilities = cap->device_caps | V4L2_CAP_VIDEO_CAPTURE |
+   V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_DEVICE_CAPS;
 
if (video->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-   cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
else
-   cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
 
return 0;
 }
-- 
2.14.3

[PATCH v8 04/13] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

v2: - improve comments for is_unordered flag (Hans)

v3: - make it bool (Hans)
- create vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  6 ++
 include/media/videobuf2-core.h  |  6 ++
 include/media/videobuf2-v4l2.h  | 10 ++
 3 files changed, 22 insertions(+)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 886a2d8d5c6c..68291ba8632d 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -961,6 +961,12 @@ void vb2_ops_wait_finish(struct vb2_queue *vq)
 }
 EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
 
+bool vb2_ops_set_unordered(struct vb2_queue *q)
+{
+   return true;
+}
+EXPORT_SYMBOL_GPL(vb2_ops_set_unordered);
+
 MODULE_DESCRIPTION("Driver helper framework for Video for Linux 2");
 MODULE_AUTHOR("Pawel Osciak <pa...@osciak.com>, Marek Szyprowski");
 MODULE_LICENSE("GPL");
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 5b6c541e4e1b..46a9e674f7e1 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -370,6 +370,10 @@ struct vb2_buffer {
  * callback by calling vb2_buffer_done() with either
  * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  * vb2_wait_for_all_buffers() function
+ * @is_unordered:  tell if the queue is unordered, i.e. buffers can be
+ * dequeued in a different order from how they were queued.
+ * The default is assumed to be ordered and this function
+ * only needs to be implemented for unordered queues.
  * @buf_queue: passes buffer vb to the driver; driver may start
  * hardware operation on this buffer; driver should give
  * the buffer back by calling vb2_buffer_done() function;
@@ -393,6 +397,7 @@ struct vb2_ops {
 
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
void (*stop_streaming)(struct vb2_queue *q);
+   bool (*is_unordered)(struct vb2_queue *q);
 
void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -566,6 +571,7 @@ struct vb2_queue {
u32 cnt_wait_finish;
u32 cnt_start_streaming;
u32 cnt_stop_streaming;
+   u32 cnt_is_unordered;
 #endif
 };
 
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 3d5e2d739f05..9de3c887c875 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -291,4 +291,14 @@ void vb2_ops_wait_prepare(struct vb2_queue *vq);
  */
 void vb2_ops_wait_finish(struct vb2_queue *vq);
 
+/**
+ * vb2_ops_set_unordered - helper function to mark queue as unordered
+ *
+ * @vq: pointer to  vb2_queue
+ *
+ * This helper just return true to notify that the driver can't deal with
+ * ordered queues.
+ */
+bool vb2_ops_set_unordered(struct vb2_queue *q);
+
 #endif /* _MEDIA_VIDEOBUF2_V4L2_H */
-- 
2.14.3

[PATCH v8 03/13] [media] omap3isp: group device capabilities

[Gustavo Padovan]

From: Gustavo Padovan 

Instead of putting V4L2_CAP_STREAMING everywhere, set device_caps
earlier with this value.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/platform/omap3isp/ispvideo.c | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/omap3isp/ispvideo.c 
b/drivers/media/platform/omap3isp/ispvideo.c
index a751c89a3ea8..b4d4ef926749 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -658,13 +658,14 @@ isp_video_querycap(struct file *file, void *fh, struct 
v4l2_capability *cap)
strlcpy(cap->card, video->video.name, sizeof(cap->card));
strlcpy(cap->bus_info, "media", sizeof(cap->bus_info));
 
-   cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT
-   | V4L2_CAP_STREAMING | V4L2_CAP_DEVICE_CAPS;
+   cap->device_caps = V4L2_CAP_STREAMING;
+   cap->capabilities = cap->device_caps | V4L2_CAP_VIDEO_CAPTURE |
+   V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_DEVICE_CAPS;
 
if (video->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-   cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
else
-   cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
 
return 0;
 }
-- 
2.14.3

[PATCH v8 04/13] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

From: Gustavo Padovan 

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

v2: - improve comments for is_unordered flag (Hans)

v3: - make it bool (Hans)
- create vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan 
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  6 ++
 include/media/videobuf2-core.h  |  6 ++
 include/media/videobuf2-v4l2.h  | 10 ++
 3 files changed, 22 insertions(+)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c 
b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 886a2d8d5c6c..68291ba8632d 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -961,6 +961,12 @@ void vb2_ops_wait_finish(struct vb2_queue *vq)
 }
 EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
 
+bool vb2_ops_set_unordered(struct vb2_queue *q)
+{
+   return true;
+}
+EXPORT_SYMBOL_GPL(vb2_ops_set_unordered);
+
 MODULE_DESCRIPTION("Driver helper framework for Video for Linux 2");
 MODULE_AUTHOR("Pawel Osciak , Marek Szyprowski");
 MODULE_LICENSE("GPL");
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 5b6c541e4e1b..46a9e674f7e1 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -370,6 +370,10 @@ struct vb2_buffer {
  * callback by calling vb2_buffer_done() with either
  * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  * vb2_wait_for_all_buffers() function
+ * @is_unordered:  tell if the queue is unordered, i.e. buffers can be
+ * dequeued in a different order from how they were queued.
+ * The default is assumed to be ordered and this function
+ * only needs to be implemented for unordered queues.
  * @buf_queue: passes buffer vb to the driver; driver may start
  * hardware operation on this buffer; driver should give
  * the buffer back by calling vb2_buffer_done() function;
@@ -393,6 +397,7 @@ struct vb2_ops {
 
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
void (*stop_streaming)(struct vb2_queue *q);
+   bool (*is_unordered)(struct vb2_queue *q);
 
void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -566,6 +571,7 @@ struct vb2_queue {
u32 cnt_wait_finish;
u32 cnt_start_streaming;
u32 cnt_stop_streaming;
+   u32 cnt_is_unordered;
 #endif
 };
 
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 3d5e2d739f05..9de3c887c875 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -291,4 +291,14 @@ void vb2_ops_wait_prepare(struct vb2_queue *vq);
  */
 void vb2_ops_wait_finish(struct vb2_queue *vq);
 
+/**
+ * vb2_ops_set_unordered - helper function to mark queue as unordered
+ *
+ * @vq: pointer to  vb2_queue
+ *
+ * This helper just return true to notify that the driver can't deal with
+ * ordered queues.
+ */
+bool vb2_ops_set_unordered(struct vb2_queue *q);
+
 #endif /* _MEDIA_VIDEOBUF2_V4L2_H */
-- 
2.14.3

[PATCH v8 01/13] [media] xilinx: regroup caps on querycap

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

To better organize the code we concentrate the setting of
V4L2_CAP_STREAMING in one place.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/platform/xilinx/xilinx-dma.c | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/media/platform/xilinx/xilinx-dma.c 
b/drivers/media/platform/xilinx/xilinx-dma.c
index 522cdfdd3345..565e466ba4fa 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -494,13 +494,14 @@ xvip_dma_querycap(struct file *file, void *fh, struct 
v4l2_capability *cap)
struct v4l2_fh *vfh = file->private_data;
struct xvip_dma *dma = to_xvip_dma(vfh->vdev);
 
-   cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
+   cap->device_caps = V4L2_CAP_STREAMING;
+   cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS
  | dma->xdev->v4l2_caps;
 
if (dma->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-   cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
else
-   cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
 
strlcpy(cap->driver, "xilinx-vipp", sizeof(cap->driver));
strlcpy(cap->card, dma->video.name, sizeof(cap->card));
-- 
2.14.3

[PATCH v8 00/13] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Hi,

So v8 is finally out addressing the comments from the previous version[1].
For more info see v5 cover letter[2]. The most important points I address
here is the handling of fences that signal with error to follow Hans
suggestion. I also added V4L2_CAP_FENCES to all vb2 drivers and marked all
codec and cobalt as unordered. More specific changelog are noted on the
patches itself.

The first 3 patches are just clean ups in preparation to add the new cap flag
and can go upstream earlier. Then there are patches to add unordered info,
the actual fences implementation and later the V4L2_CAP_FENCES flag.
The last patch contains the Documentation.

You can find the code at:

https://gitlab.collabora.com/padovan/linux/tree/v4l2-fences

The test tools I've been using are:
https://gitlab.collabora.com/padovan/drm-v4l2-test
https://gitlab.collabora.com/padovan/v4l2-fences-test

Please review,

Gustavo

[1] https://lkml.org/lkml/2018/1/10/644
[2] https://lkml.org/lkml/2017/11/15/550

Gustavo Padovan (13):
  [media] xilinx: regroup caps on querycap
  [media] hackrf: group device capabilities
  [media] omap3isp: group device capabilities
  [media] vb2: add is_unordered callback for drivers
  [media] v4l: add 'unordered' flag to format description ioctl
  [media] cobalt: add .is_unordered() for cobalt
  [media] vb2: mark codec drivers as unordered
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add out-fence support to QBUF
  [media] v4l: introduce the fences capability
  [media] v4l: Add V4L2_CAP_FENCES to drivers
  [media] v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst|  45 +++-
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst   |   7 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst   |  55 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst   |  12 +-
 Documentation/media/uapi/v4l/vidioc-querycap.rst   |   3 +
 drivers/media/common/videobuf2/videobuf2-core.c| 289 ++---
 drivers/media/common/videobuf2/videobuf2-v4l2.c|  58 -
 drivers/media/pci/cobalt/cobalt-v4l2.c |   4 +-
 drivers/media/pci/cx23885/cx23885-417.c|   2 +-
 drivers/media/pci/cx23885/cx23885-video.c  |   3 +-
 drivers/media/pci/cx88/cx88-video.c|   3 +-
 drivers/media/pci/dt3155/dt3155.c  |   2 +-
 drivers/media/pci/saa7134/saa7134-video.c  |   2 +
 drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c |   3 +-
 drivers/media/pci/solo6x10/solo6x10-v4l2.c |   3 +-
 drivers/media/pci/sta2x11/sta2x11_vip.c|   2 +-
 drivers/media/pci/tw68/tw68-video.c|   3 +-
 drivers/media/pci/tw686x/tw686x-video.c|   2 +-
 drivers/media/platform/am437x/am437x-vpfe.c|   2 +-
 drivers/media/platform/blackfin/bfin_capture.c |   3 +-
 drivers/media/platform/coda/coda-common.c  |   4 +-
 drivers/media/platform/davinci/vpbe_display.c  |   3 +-
 drivers/media/platform/davinci/vpfe_capture.c  |   3 +-
 drivers/media/platform/davinci/vpif_capture.c  |   3 +-
 drivers/media/platform/davinci/vpif_display.c  |   3 +-
 drivers/media/platform/exynos-gsc/gsc-m2m.c|   4 +-
 drivers/media/platform/exynos4-is/fimc-capture.c   |   3 +-
 drivers/media/platform/exynos4-is/fimc-isp-video.c |   3 +-
 drivers/media/platform/exynos4-is/fimc-lite.c  |   2 +-
 drivers/media/platform/exynos4-is/fimc-m2m.c   |   4 +-
 drivers/media/platform/m2m-deinterlace.c   |   4 +-
 drivers/media/platform/marvell-ccic/mcam-core.c|   2 +-
 drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c|   1 +
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c   |   1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c |   1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c |   1 +
 drivers/media/platform/mx2_emmaprp.c   |   4 +-
 drivers/media/platform/omap3isp/ispvideo.c |   9 +-
 drivers/media/platform/pxa_camera.c|   3 +-
 drivers/media/platform/qcom/venus/vdec.c   |   1 +
 drivers/media/platform/qcom/venus/venc.c   |   1 +
 drivers/media/platform/rcar_fdp1.c |   1 +
 drivers/media/platform/rcar_jpu.c  |   4 +-
 drivers/media/platform/rockchip/rga/rga-buf.c  |   1 +
 drivers/media/platform/s3c-camif/camif-capture.c   |   3 +-
 drivers/media/platform/s5p-g2d/g2d.c   |   4 +-
 drivers/media/platform/s5p-jpeg/jpeg-core.c|   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c   |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c   |   4 +-
 drivers/media/platform/sh_veu.c|   4 +-
 drivers/media/platform/sh_vou.c|   2 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c  |   4 +-
 drivers/media/platform/ti-vpe/cal.c|   2 +-
 drivers/media/platform/ti-vpe

[PATCH v8 01/13] [media] xilinx: regroup caps on querycap

[Gustavo Padovan]

From: Gustavo Padovan 

To better organize the code we concentrate the setting of
V4L2_CAP_STREAMING in one place.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/platform/xilinx/xilinx-dma.c | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/media/platform/xilinx/xilinx-dma.c 
b/drivers/media/platform/xilinx/xilinx-dma.c
index 522cdfdd3345..565e466ba4fa 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -494,13 +494,14 @@ xvip_dma_querycap(struct file *file, void *fh, struct 
v4l2_capability *cap)
struct v4l2_fh *vfh = file->private_data;
struct xvip_dma *dma = to_xvip_dma(vfh->vdev);
 
-   cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
+   cap->device_caps = V4L2_CAP_STREAMING;
+   cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS
  | dma->xdev->v4l2_caps;
 
if (dma->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-   cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
else
-   cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+   cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
 
strlcpy(cap->driver, "xilinx-vipp", sizeof(cap->driver));
strlcpy(cap->card, dma->video.name, sizeof(cap->card));
-- 
2.14.3

[PATCH v8 00/13] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan 

Hi,

So v8 is finally out addressing the comments from the previous version[1].
For more info see v5 cover letter[2]. The most important points I address
here is the handling of fences that signal with error to follow Hans
suggestion. I also added V4L2_CAP_FENCES to all vb2 drivers and marked all
codec and cobalt as unordered. More specific changelog are noted on the
patches itself.

The first 3 patches are just clean ups in preparation to add the new cap flag
and can go upstream earlier. Then there are patches to add unordered info,
the actual fences implementation and later the V4L2_CAP_FENCES flag.
The last patch contains the Documentation.

You can find the code at:

https://gitlab.collabora.com/padovan/linux/tree/v4l2-fences

The test tools I've been using are:
https://gitlab.collabora.com/padovan/drm-v4l2-test
https://gitlab.collabora.com/padovan/v4l2-fences-test

Please review,

Gustavo

[1] https://lkml.org/lkml/2018/1/10/644
[2] https://lkml.org/lkml/2017/11/15/550

Gustavo Padovan (13):
  [media] xilinx: regroup caps on querycap
  [media] hackrf: group device capabilities
  [media] omap3isp: group device capabilities
  [media] vb2: add is_unordered callback for drivers
  [media] v4l: add 'unordered' flag to format description ioctl
  [media] cobalt: add .is_unordered() for cobalt
  [media] vb2: mark codec drivers as unordered
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add out-fence support to QBUF
  [media] v4l: introduce the fences capability
  [media] v4l: Add V4L2_CAP_FENCES to drivers
  [media] v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst|  45 +++-
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst   |   7 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst   |  55 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst   |  12 +-
 Documentation/media/uapi/v4l/vidioc-querycap.rst   |   3 +
 drivers/media/common/videobuf2/videobuf2-core.c| 289 ++---
 drivers/media/common/videobuf2/videobuf2-v4l2.c|  58 -
 drivers/media/pci/cobalt/cobalt-v4l2.c |   4 +-
 drivers/media/pci/cx23885/cx23885-417.c|   2 +-
 drivers/media/pci/cx23885/cx23885-video.c  |   3 +-
 drivers/media/pci/cx88/cx88-video.c|   3 +-
 drivers/media/pci/dt3155/dt3155.c  |   2 +-
 drivers/media/pci/saa7134/saa7134-video.c  |   2 +
 drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c |   3 +-
 drivers/media/pci/solo6x10/solo6x10-v4l2.c |   3 +-
 drivers/media/pci/sta2x11/sta2x11_vip.c|   2 +-
 drivers/media/pci/tw68/tw68-video.c|   3 +-
 drivers/media/pci/tw686x/tw686x-video.c|   2 +-
 drivers/media/platform/am437x/am437x-vpfe.c|   2 +-
 drivers/media/platform/blackfin/bfin_capture.c |   3 +-
 drivers/media/platform/coda/coda-common.c  |   4 +-
 drivers/media/platform/davinci/vpbe_display.c  |   3 +-
 drivers/media/platform/davinci/vpfe_capture.c  |   3 +-
 drivers/media/platform/davinci/vpif_capture.c  |   3 +-
 drivers/media/platform/davinci/vpif_display.c  |   3 +-
 drivers/media/platform/exynos-gsc/gsc-m2m.c|   4 +-
 drivers/media/platform/exynos4-is/fimc-capture.c   |   3 +-
 drivers/media/platform/exynos4-is/fimc-isp-video.c |   3 +-
 drivers/media/platform/exynos4-is/fimc-lite.c  |   2 +-
 drivers/media/platform/exynos4-is/fimc-m2m.c   |   4 +-
 drivers/media/platform/m2m-deinterlace.c   |   4 +-
 drivers/media/platform/marvell-ccic/mcam-core.c|   2 +-
 drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c|   1 +
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c   |   1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c |   1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c |   1 +
 drivers/media/platform/mx2_emmaprp.c   |   4 +-
 drivers/media/platform/omap3isp/ispvideo.c |   9 +-
 drivers/media/platform/pxa_camera.c|   3 +-
 drivers/media/platform/qcom/venus/vdec.c   |   1 +
 drivers/media/platform/qcom/venus/venc.c   |   1 +
 drivers/media/platform/rcar_fdp1.c |   1 +
 drivers/media/platform/rcar_jpu.c  |   4 +-
 drivers/media/platform/rockchip/rga/rga-buf.c  |   1 +
 drivers/media/platform/s3c-camif/camif-capture.c   |   3 +-
 drivers/media/platform/s5p-g2d/g2d.c   |   4 +-
 drivers/media/platform/s5p-jpeg/jpeg-core.c|   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c   |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c   |   4 +-
 drivers/media/platform/sh_veu.c|   4 +-
 drivers/media/platform/sh_vou.c|   2 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c  |   4 +-
 drivers/media/platform/ti-vpe/cal.c|   2 +-
 drivers/media/platform/ti-vpe/vpe.c|   4 +-
 drivers/media/platform

Re: [PATCH v7 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

2018-01-15 Alexandre Courbot <acour...@chromium.org>:

> On Thu, Jan 11, 2018 at 1:07 AM, Gustavo Padovan <gust...@padovan.org> wrote:
> >  /*
> >   * vb2_start_streaming() - Attempt to start streaming.
> >   * @q: videobuf2 queue
> > @@ -1489,18 +1562,16 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int 
> > index, void *pb,
> > if (vb->in_fence) {
> > ret = dma_fence_add_callback(vb->in_fence, >fence_cb,
> >  vb2_qbuf_fence_cb);
> > -   if (ret == -EINVAL) {
> > +   /* is the fence signaled? */
> > +   if (ret == -ENOENT) {
> > +   dma_fence_put(vb->in_fence);
> > +   vb->in_fence = NULL;
> > +   } else if (ret) {
> > spin_unlock_irqrestore(>fence_cb_lock, flags);
> > goto err;
> > -   } else if (!ret) {
> > -   goto fill;
> > }
> > -
> > -   dma_fence_put(vb->in_fence);
> > -   vb->in_fence = NULL;
> 
> This chunk seems to deal with input fences, shouldn't it be part of
> the previous patch instead of this one?
> 
> >
> > -   if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> > -   !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > +   if (b->fence_fd > 0 && !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", 
> > opname);
> > return -EINVAL;
> > }
> >
> > +   if (b->fence_fd == -1 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > +   dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", 
> > opname);
> > +   return -EINVAL;
> > +   }
> > +
> 
> Same here?
> 
> > return __verify_planes_array(q->bufs[b->index], b);
> >  }
> >
> > @@ -212,7 +216,12 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, 
> > void *pb)
> > b->sequence = vbuf->sequence;
> > b->reserved = 0;
> >
> > -   b->fence_fd = 0;
> > +   if (b->flags & V4L2_BUF_FLAG_OUT_FENCE) {
> > +   b->fence_fd = vb->out_fence_fd;
> > +   } else {
> > +   b->fence_fd = 0;
> > +   }
> 
> Sorry if this has already been discussed, but I don't remember the
> outcome if it has.
> 
> I wonder if doing this here could not make out_fence_fd leak in
> situations where we don't need/want it to. Let's take for instance a
> multi-process user program. One process queues a buffer with an
> OUT_FENCE and gets a valid fd in fence_fd upon return. Then the other
> process performs a QUERYBUF and gets the same fence_fd - which is
> invalid in its context. Would it not be preferable fill the out fence
> information only when queuing buffers, since it is the only time where
> we are guaranteed it will be usable by the caller?
> 
> Similarly, when a buffer is processed and user-space performs a DQBUF,
> the V4L2_BUF_FLAG_OUT_FENCE will be set but fence_fd will be 0. Again,
> limiting the return of out fence information to QBUF would prevent
> this.

Right. So in summary as this is something Hans commented on another
e-mail in this thread.

Your proposal is to only return the out_fence fd number on QBUF, right?
And DQBUF and QUERYBUF would only return -1 in the fence_fd field.

What I understood from Hans comment is that he is okay with sharing the
fd in such cases and v4l2 already does that for dmabuf fds.

I believe sharing is okay, as it will be either the same process or a
process we gave the device fd in the first place.

I'm not invested in any particular approach here. Thoughts?

> 
> If we go that route, out_fence_fd could maybe become a local variable
> of vb2_qbuf() instead of being a member of vb2_buffer, and would be
> returned by vb2_setup_out_fence(). This would guarantee it does not
> leak anywhere else.


Gustavo

Re: [PATCH v7 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

2018-01-15 Alexandre Courbot :

> On Thu, Jan 11, 2018 at 1:07 AM, Gustavo Padovan  wrote:
> >  /*
> >   * vb2_start_streaming() - Attempt to start streaming.
> >   * @q: videobuf2 queue
> > @@ -1489,18 +1562,16 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int 
> > index, void *pb,
> > if (vb->in_fence) {
> > ret = dma_fence_add_callback(vb->in_fence, >fence_cb,
> >  vb2_qbuf_fence_cb);
> > -   if (ret == -EINVAL) {
> > +   /* is the fence signaled? */
> > +   if (ret == -ENOENT) {
> > +   dma_fence_put(vb->in_fence);
> > +   vb->in_fence = NULL;
> > +   } else if (ret) {
> > spin_unlock_irqrestore(>fence_cb_lock, flags);
> > goto err;
> > -   } else if (!ret) {
> > -   goto fill;
> > }
> > -
> > -   dma_fence_put(vb->in_fence);
> > -   vb->in_fence = NULL;
> 
> This chunk seems to deal with input fences, shouldn't it be part of
> the previous patch instead of this one?
> 
> >
> > -   if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> > -   !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > +   if (b->fence_fd > 0 && !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", 
> > opname);
> > return -EINVAL;
> > }
> >
> > +   if (b->fence_fd == -1 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > +   dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", 
> > opname);
> > +   return -EINVAL;
> > +   }
> > +
> 
> Same here?
> 
> > return __verify_planes_array(q->bufs[b->index], b);
> >  }
> >
> > @@ -212,7 +216,12 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, 
> > void *pb)
> > b->sequence = vbuf->sequence;
> > b->reserved = 0;
> >
> > -   b->fence_fd = 0;
> > +   if (b->flags & V4L2_BUF_FLAG_OUT_FENCE) {
> > +   b->fence_fd = vb->out_fence_fd;
> > +   } else {
> > +   b->fence_fd = 0;
> > +   }
> 
> Sorry if this has already been discussed, but I don't remember the
> outcome if it has.
> 
> I wonder if doing this here could not make out_fence_fd leak in
> situations where we don't need/want it to. Let's take for instance a
> multi-process user program. One process queues a buffer with an
> OUT_FENCE and gets a valid fd in fence_fd upon return. Then the other
> process performs a QUERYBUF and gets the same fence_fd - which is
> invalid in its context. Would it not be preferable fill the out fence
> information only when queuing buffers, since it is the only time where
> we are guaranteed it will be usable by the caller?
> 
> Similarly, when a buffer is processed and user-space performs a DQBUF,
> the V4L2_BUF_FLAG_OUT_FENCE will be set but fence_fd will be 0. Again,
> limiting the return of out fence information to QBUF would prevent
> this.

Right. So in summary as this is something Hans commented on another
e-mail in this thread.

Your proposal is to only return the out_fence fd number on QBUF, right?
And DQBUF and QUERYBUF would only return -1 in the fence_fd field.

What I understood from Hans comment is that he is okay with sharing the
fd in such cases and v4l2 already does that for dmabuf fds.

I believe sharing is okay, as it will be either the same process or a
process we gave the device fd in the first place.

I'm not invested in any particular approach here. Thoughts?

> 
> If we go that route, out_fence_fd could maybe become a local variable
> of vb2_qbuf() instead of being a member of vb2_buffer, and would be
> returned by vb2_setup_out_fence(). This would guarantee it does not
> leak anywhere else.


Gustavo

Re: [PATCH v7 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

2018-01-12 Hans Verkuil <hverk...@xs4all.nl>:

> On 01/10/18 17:07, Gustavo Padovan wrote:
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
> > an out_fence and send its fd to userspace on the fence_fd field as a
> > return arg for the QBUF call.
> > 
> > The fence is signaled on buffer_done(), when the job on the buffer is
> > finished.
> > 
> > v8:
> > - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
> > - fix crash when checking not using fences in vb2_buffer_done()
> > 
> > v7:
> > - merge patch that add the infrastructure to out-fences into
> > this one (Alex Courbot)
> > - Do not install the fd if there is no fence. (Alex Courbot)
> > - do not report error on requeueing, just WARN_ON_ONCE() (Hans)
> > 
> > v6
> > - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
> > ordering in vb2 for queueing in the driver, so the event is not
> > necessary anymore and the out_fence_fd is sent back to userspace
> > on QBUF call return arg
> > - do not allow requeueing with out-fences, instead mark the buffer
> > with an error and wake up to userspace.
> > - send the out_fence_fd back to userspace on the fence_fd field
> > 
> > v5:
> > - delay fd_install to DQ_EVENT (Hans)
> > - if queue is fully ordered send OUT_FENCE event right away
> > (Brian)
> > - rename 'q->ordered' to 'q->ordered_in_driver'
> > - merge change to implement OUT_FENCE event here
> > 
> > v4:
> > - return the out_fence_fd in the BUF_QUEUED event(Hans)
> > 
> > v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
> > - set the OUT_FENCE flag if there is a fence pending (Hans)
> > - call fd_install() after vb2_core_qbuf() (Hans)
> > - clean up fence if vb2_core_qbuf() fails (Hans)
> > - add list to store sync_file and fence for the next queued buffer
> > 
> > v2: check if the queue is ordered.
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  drivers/media/common/videobuf/videobuf2-core.c | 101 
> > +++--
> >  drivers/media/common/videobuf/videobuf2-v4l2.c |  28 ++-
> >  include/media/videobuf2-core.h |  22 ++
> >  3 files changed, 140 insertions(+), 11 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
> > b/drivers/media/common/videobuf/videobuf2-core.c
> > index 777e3a2bc746..1f30d9efb7c8 100644
> > --- a/drivers/media/common/videobuf/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf/videobuf2-core.c
> > @@ -25,6 +25,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  
> >  #include 
> >  #include 
> > @@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->planes[plane].length = plane_sizes[plane];
> > vb->planes[plane].min_length = plane_sizes[plane];
> > }
> > +   vb->out_fence_fd = -1;
> > q->bufs[vb->index] = vb;
> >  
> > /* Allocate video buffer memory for the MMAP type */
> > @@ -939,10 +941,22 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
> > vb2_buffer_state state)
> > case VB2_BUF_STATE_QUEUED:
> > break;
> > case VB2_BUF_STATE_REQUEUEING:
> > +   /* Requeuing with explicit synchronization, spit warning */
> > +   WARN_ON_ONCE(vb->out_fence);
> > +
> > if (q->start_streaming_called)
> > __enqueue_in_driver(vb);
> > -   return;
> > +   break;
> > default:
> > +   if (vb->out_fence) {
> > +   if (state == VB2_BUF_STATE_ERROR)
> > +   dma_fence_set_error(vb->out_fence, -EFAULT);
> > +   dma_fence_signal(vb->out_fence);
> > +   dma_fence_put(vb->out_fence);
> > +   vb->out_fence = NULL;
> > +   vb->out_fence_fd = -1;
> > +   }
> > +
> > /* Inform any processes that may be waiting for buffers */
> > wake_up(>done_wq);
> > break;
> > @@ -1341,6 +1355,65 @@ int vb2_core_prepare_buf(struct vb2_queue *q, 
> > unsigned int index, void *pb)
> 

Re: [PATCH v7 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

2018-01-12 Hans Verkuil :

> On 01/10/18 17:07, Gustavo Padovan wrote:
> > From: Gustavo Padovan 
> > 
> > If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
> > an out_fence and send its fd to userspace on the fence_fd field as a
> > return arg for the QBUF call.
> > 
> > The fence is signaled on buffer_done(), when the job on the buffer is
> > finished.
> > 
> > v8:
> > - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
> > - fix crash when checking not using fences in vb2_buffer_done()
> > 
> > v7:
> > - merge patch that add the infrastructure to out-fences into
> > this one (Alex Courbot)
> > - Do not install the fd if there is no fence. (Alex Courbot)
> > - do not report error on requeueing, just WARN_ON_ONCE() (Hans)
> > 
> > v6
> > - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
> > ordering in vb2 for queueing in the driver, so the event is not
> > necessary anymore and the out_fence_fd is sent back to userspace
> > on QBUF call return arg
> > - do not allow requeueing with out-fences, instead mark the buffer
> > with an error and wake up to userspace.
> > - send the out_fence_fd back to userspace on the fence_fd field
> > 
> > v5:
> > - delay fd_install to DQ_EVENT (Hans)
> > - if queue is fully ordered send OUT_FENCE event right away
> > (Brian)
> > - rename 'q->ordered' to 'q->ordered_in_driver'
> > - merge change to implement OUT_FENCE event here
> > 
> > v4:
> > - return the out_fence_fd in the BUF_QUEUED event(Hans)
> > 
> > v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
> > - set the OUT_FENCE flag if there is a fence pending (Hans)
> >     - call fd_install() after vb2_core_qbuf() (Hans)
> > - clean up fence if vb2_core_qbuf() fails (Hans)
> > - add list to store sync_file and fence for the next queued buffer
> > 
> > v2: check if the queue is ordered.
> > 
> > Signed-off-by: Gustavo Padovan 
> > ---
> >  drivers/media/common/videobuf/videobuf2-core.c | 101 
> > +++--
> >  drivers/media/common/videobuf/videobuf2-v4l2.c |  28 ++-
> >  include/media/videobuf2-core.h |  22 ++
> >  3 files changed, 140 insertions(+), 11 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
> > b/drivers/media/common/videobuf/videobuf2-core.c
> > index 777e3a2bc746..1f30d9efb7c8 100644
> > --- a/drivers/media/common/videobuf/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf/videobuf2-core.c
> > @@ -25,6 +25,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  
> >  #include 
> >  #include 
> > @@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->planes[plane].length = plane_sizes[plane];
> > vb->planes[plane].min_length = plane_sizes[plane];
> > }
> > +   vb->out_fence_fd = -1;
> > q->bufs[vb->index] = vb;
> >  
> > /* Allocate video buffer memory for the MMAP type */
> > @@ -939,10 +941,22 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
> > vb2_buffer_state state)
> > case VB2_BUF_STATE_QUEUED:
> > break;
> > case VB2_BUF_STATE_REQUEUEING:
> > +   /* Requeuing with explicit synchronization, spit warning */
> > +   WARN_ON_ONCE(vb->out_fence);
> > +
> > if (q->start_streaming_called)
> > __enqueue_in_driver(vb);
> > -   return;
> > +   break;
> > default:
> > +   if (vb->out_fence) {
> > +   if (state == VB2_BUF_STATE_ERROR)
> > +   dma_fence_set_error(vb->out_fence, -EFAULT);
> > +   dma_fence_signal(vb->out_fence);
> > +   dma_fence_put(vb->out_fence);
> > +   vb->out_fence = NULL;
> > +   vb->out_fence_fd = -1;
> > +   }
> > +
> > /* Inform any processes that may be waiting for buffers */
> > wake_up(>done_wq);
> > break;
> > @@ -1341,6 +1355,65 @@ int vb2_core_prepare_buf(struct vb2_queue *q, 
> > unsigned int index, void *pb)
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
> >  
> > +static inline const char

Re: [PATCH v7 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

Hi Hans,

2018-01-12 Hans Verkuil <hverk...@xs4all.nl>:

> On 01/10/18 17:07, Gustavo Padovan wrote:
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the driver,
> > instead we mark it as error and wait until the previous buffer is done
> > to notify userspace of the error. We wait here to deliver the buffers back
> > to userspace in order.
> > 
> > v8: - improve comments about fences with errors
> > 
> > v7:
> > - get rid of the fence array stuff for ordering and just use
> > get_num_buffers_ready() (Hans)
> > - fix issue of queuing the buffer twice (Hans)
> > - avoid the dma_fence_wait() in core_qbuf() (Alex)
> > - merge preparation commit in
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  drivers/media/common/videobuf/videobuf2-core.c | 166 
> > ++---
> >  drivers/media/common/videobuf/videobuf2-v4l2.c |  29 -
> >  drivers/media/v4l2-core/Kconfig|  33 +
> >  include/media/videobuf2-core.h |  14 ++-
> >  4 files changed, 221 insertions(+), 21 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
> > b/drivers/media/common/videobuf/videobuf2-core.c
> > index f7109f827f6e..777e3a2bc746 100644
> > --- a/drivers/media/common/videobuf/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf/videobuf2-core.c
> > @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   spin_lock_init(>fence_cb_lock);
> > for (plane = 0; plane < num_planes; ++plane) {
> > vb->planes[plane].length = plane_sizes[plane];
> > vb->planes[plane].min_length = plane_sizes[plane];
> > @@ -936,7 +937,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
> > vb2_buffer_state state)
> >  
> > switch (state) {
> > case VB2_BU

Re: [PATCH v7 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

Hi Hans,

2018-01-12 Hans Verkuil :

> On 01/10/18 17:07, Gustavo Padovan wrote:
> > From: Gustavo Padovan 
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the driver,
> > instead we mark it as error and wait until the previous buffer is done
> > to notify userspace of the error. We wait here to deliver the buffers back
> > to userspace in order.
> > 
> > v8: - improve comments about fences with errors
> > 
> > v7:
> > - get rid of the fence array stuff for ordering and just use
> > get_num_buffers_ready() (Hans)
> > - fix issue of queuing the buffer twice (Hans)
> > - avoid the dma_fence_wait() in core_qbuf() (Alex)
> > - merge preparation commit in
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan 
> > ---
> >  drivers/media/common/videobuf/videobuf2-core.c | 166 
> > ++---
> >  drivers/media/common/videobuf/videobuf2-v4l2.c |  29 -
> >  drivers/media/v4l2-core/Kconfig|  33 +
> >  include/media/videobuf2-core.h |  14 ++-
> >  4 files changed, 221 insertions(+), 21 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
> > b/drivers/media/common/videobuf/videobuf2-core.c
> > index f7109f827f6e..777e3a2bc746 100644
> > --- a/drivers/media/common/videobuf/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf/videobuf2-core.c
> > @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   spin_lock_init(>fence_cb_lock);
> > for (plane = 0; plane < num_planes; ++plane) {
> > vb->planes[plane].length = plane_sizes[plane];
> > vb->planes[plane].min_length = plane_sizes[plane];
> > @@ -936,7 +937,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
> > vb2_buffer_state state)
> >  
> > switch (state) {
> > case VB2_BUF_STATE_QUEUED:
> > -   return;
> > +   break;
> > case VB2_BUF_STATE_REQUEUEI

Re: [PATCH v7 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

2018-01-15 Hans Verkuil <hverk...@xs4all.nl>:

> On 01/15/2018 01:01 PM, Gustavo Padovan wrote:
> > 2018-01-15 Alexandre Courbot <acour...@chromium.org>:
> > 
> >> On Thu, Jan 11, 2018 at 1:07 AM, Gustavo Padovan <gust...@padovan.org> 
> >> wrote:
> >>> From: Gustavo Padovan <gustavo.pado...@collabora.com>
> >>>
> >>> Explicit synchronization benefits a lot from ordered queues, they fit
> >>> better in a pipeline with DRM for example so create a opt-in way for
> >>> drivers notify videobuf2 that the queue is unordered.
> >>>
> >>> Drivers don't need implement it if the queue is ordered.
> >>
> >> This is going to make user-space believe that *all* vb2 drivers use
> >> ordered queues by default, at least until non-ordered drivers catch up
> >> with this change. Wouldn't it be less dangerous to do the opposite
> >> (make queues non-ordered by default)?
> > 
> > The rational behind this decision was because most formats/drivers are
> > ordered so only a small amount of drivers need to changed. I think this
> > was proposed by Hans on the Media Summit.
> > 
> > I understand your concern. My question is how dangerous will it be. If
> > you are building a product you will make the changes in the driver if
> > they are not there yet, or if it is a distribution you'd never know
> > which driver/format you are using so you should be prepared for
> > everything.
> > 
> > AFAIK all Capture drivers are ordered and that is where I think fences
> > is most useful.
> 
> Right. What could be done is to mark all codec drivers as unordered initially
> ask the driver authors to verify this. All capture drivers using vb2 and not
> using REQUEUE are ordered.

That is a good way out.

> 
> One thing we haven't looked at is what to do with drivers that do not use vb2.
> Those won't support fences, but how will userspace know that fences are not
> supported? I'm not sure what the best method is for that.
> 
> I am leaning towards a new capability since this has to be advertised clearly.

The capability flag makes sense to me, I'll incorporate it as part of my
next patchset.

Gustavo

Re: [PATCH v7 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

2018-01-15 Hans Verkuil :

> On 01/15/2018 01:01 PM, Gustavo Padovan wrote:
> > 2018-01-15 Alexandre Courbot :
> > 
> >> On Thu, Jan 11, 2018 at 1:07 AM, Gustavo Padovan  
> >> wrote:
> >>> From: Gustavo Padovan 
> >>>
> >>> Explicit synchronization benefits a lot from ordered queues, they fit
> >>> better in a pipeline with DRM for example so create a opt-in way for
> >>> drivers notify videobuf2 that the queue is unordered.
> >>>
> >>> Drivers don't need implement it if the queue is ordered.
> >>
> >> This is going to make user-space believe that *all* vb2 drivers use
> >> ordered queues by default, at least until non-ordered drivers catch up
> >> with this change. Wouldn't it be less dangerous to do the opposite
> >> (make queues non-ordered by default)?
> > 
> > The rational behind this decision was because most formats/drivers are
> > ordered so only a small amount of drivers need to changed. I think this
> > was proposed by Hans on the Media Summit.
> > 
> > I understand your concern. My question is how dangerous will it be. If
> > you are building a product you will make the changes in the driver if
> > they are not there yet, or if it is a distribution you'd never know
> > which driver/format you are using so you should be prepared for
> > everything.
> > 
> > AFAIK all Capture drivers are ordered and that is where I think fences
> > is most useful.
> 
> Right. What could be done is to mark all codec drivers as unordered initially
> ask the driver authors to verify this. All capture drivers using vb2 and not
> using REQUEUE are ordered.

That is a good way out.

> 
> One thing we haven't looked at is what to do with drivers that do not use vb2.
> Those won't support fences, but how will userspace know that fences are not
> supported? I'm not sure what the best method is for that.
> 
> I am leaning towards a new capability since this has to be advertised clearly.

The capability flag makes sense to me, I'll incorporate it as part of my
next patchset.

Gustavo

Re: [PATCH v7 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

2018-01-15 Alexandre Courbot <acour...@chromium.org>:

> On Thu, Jan 11, 2018 at 1:07 AM, Gustavo Padovan <gust...@padovan.org> wrote:
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> >
> > Explicit synchronization benefits a lot from ordered queues, they fit
> > better in a pipeline with DRM for example so create a opt-in way for
> > drivers notify videobuf2 that the queue is unordered.
> >
> > Drivers don't need implement it if the queue is ordered.
> 
> This is going to make user-space believe that *all* vb2 drivers use
> ordered queues by default, at least until non-ordered drivers catch up
> with this change. Wouldn't it be less dangerous to do the opposite
> (make queues non-ordered by default)?

The rational behind this decision was because most formats/drivers are
ordered so only a small amount of drivers need to changed. I think this
was proposed by Hans on the Media Summit.

I understand your concern. My question is how dangerous will it be. If
you are building a product you will make the changes in the driver if
they are not there yet, or if it is a distribution you'd never know
which driver/format you are using so you should be prepared for
everything.

AFAIK all Capture drivers are ordered and that is where I think fences
is most useful.

Gustavo

Re: [PATCH v7 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

2018-01-15 Alexandre Courbot :

> On Thu, Jan 11, 2018 at 1:07 AM, Gustavo Padovan  wrote:
> > From: Gustavo Padovan 
> >
> > Explicit synchronization benefits a lot from ordered queues, they fit
> > better in a pipeline with DRM for example so create a opt-in way for
> > drivers notify videobuf2 that the queue is unordered.
> >
> > Drivers don't need implement it if the queue is ordered.
> 
> This is going to make user-space believe that *all* vb2 drivers use
> ordered queues by default, at least until non-ordered drivers catch up
> with this change. Wouldn't it be less dangerous to do the opposite
> (make queues non-ordered by default)?

The rational behind this decision was because most formats/drivers are
ordered so only a small amount of drivers need to changed. I think this
was proposed by Hans on the Media Summit.

I understand your concern. My question is how dangerous will it be. If
you are building a product you will make the changes in the driver if
they are not there yet, or if it is a distribution you'd never know
which driver/format you are using so you should be prepared for
everything.

AFAIK all Capture drivers are ordered and that is where I think fences
is most useful.

Gustavo

Re: [PATCH v7 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

2018-01-10 Nicolas Dufresne <nico...@ndufresne.ca>:

> Le mercredi 10 janvier 2018 à 14:07 -0200, Gustavo Padovan a écrit :
> > v7 bring a fix for a crash when not using fences and a uAPI fix.
> > I've done a bit more of testing on it and also measured some
> > performance. On a intel laptop a DRM<->V4L2 pipeline with fences is
> > runnning twice as faster than the same pipeline with no fences.
> 
> What does it mean twice faster here ?

That capture then display on the screen for a given number of frames was
completing in about half of the time when using fences and passing then
to the other driver right away when they are received.

Gustavo

Re: [PATCH v7 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

2018-01-10 Nicolas Dufresne :

> Le mercredi 10 janvier 2018 à 14:07 -0200, Gustavo Padovan a écrit :
> > v7 bring a fix for a crash when not using fences and a uAPI fix.
> > I've done a bit more of testing on it and also measured some
> > performance. On a intel laptop a DRM<->V4L2 pipeline with fences is
> > runnning twice as faster than the same pipeline with no fences.
> 
> What does it mean twice faster here ?

That capture then display on the screen for a given number of frames was
completing in about half of the time when using fences and passing then
to the other driver right away when they are received.

Gustavo

[PATCH v7 6/6] [media] v4l: Document explicit synchronization behavior

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Add section to VIDIOC_QBUF about it

v5:
- Remove V4L2_CAP_ORDERED
- Add doc about V4L2_FMT_FLAG_UNORDERED

v4:
- Document ordering behavior for in-fences
- Document V4L2_CAP_ORDERED capability
- Remove doc about OUT_FENCE event
- Document immediate return of out-fence in QBUF

v3:
- make the out_fence refer to the current buffer (Hans)
- Note what happens when the IN_FENCE is not set (Hans)

v2:
- mention that fences are files (Hans)
- rework for the new API

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst | 47 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |  9 -
 2 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst 
b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..8809397fb110 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API `, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,51 @@ immediately with an ``EAGAIN`` error code when no buffer 
is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer, i.e., 
queueing
+it to the driver. On the other side, the kernel can create out-fences for the
+buffers it queues to the drivers. Out-fences signal when the driver is
+finished with buffer, i.e., the buffer is ready. The fences are represented
+as a file and passed as a file descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with error. The fence_fd field will be ignored if the
+``V4L2_BUF_FLAG_IN_FENCE`` is not set.
+
+The videobuf2-core will guarantee that all buffers queued with in-fence will
+be queued to the drivers in the same order. Fence may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering.
+
+If the in-fence signals with an error the videobuf2 won't queue the buffer to
+the driver, instead it will flag it with an error. And then wait for the
+previous buffer to complete before asking userspace dequeue the buffer with
+error - to make sure we deliver the buffers back in the correct order.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.
+
+Note the the same `fence_fd` field is used for both sending the in-fence as
+input argument to receive the out-fence as a return argument.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. It will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the current buffer queues is able to keep the ordering of buffers, i.e.,
+the dequeing of buffers will happen at the same order we queue them, with no
+reordering by the driver.
 
 Return Value
 
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst 
b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
index dd54747fabc9..df964c4d916b 100644
--- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
@@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero 
to the
 number of buffers allocated with
 :ref:`VIDIOC_REQBUFS` (struct
 :c:type:`v4l2_requestbuffers` ``count``) minus
-one. The ``reserved`` and ``reserved2``

[PATCH v7 6/6] [media] v4l: Document explicit synchronization behavior

[Gustavo Padovan]

From: Gustavo Padovan 

Add section to VIDIOC_QBUF about it

v5:
- Remove V4L2_CAP_ORDERED
- Add doc about V4L2_FMT_FLAG_UNORDERED

v4:
- Document ordering behavior for in-fences
- Document V4L2_CAP_ORDERED capability
- Remove doc about OUT_FENCE event
- Document immediate return of out-fence in QBUF

v3:
- make the out_fence refer to the current buffer (Hans)
- Note what happens when the IN_FENCE is not set (Hans)

v2:
- mention that fences are files (Hans)
- rework for the new API

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst | 47 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |  9 -
 2 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst 
b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..8809397fb110 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API `, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,51 @@ immediately with an ``EAGAIN`` error code when no buffer 
is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer, i.e., 
queueing
+it to the driver. On the other side, the kernel can create out-fences for the
+buffers it queues to the drivers. Out-fences signal when the driver is
+finished with buffer, i.e., the buffer is ready. The fences are represented
+as a file and passed as a file descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with error. The fence_fd field will be ignored if the
+``V4L2_BUF_FLAG_IN_FENCE`` is not set.
+
+The videobuf2-core will guarantee that all buffers queued with in-fence will
+be queued to the drivers in the same order. Fence may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering.
+
+If the in-fence signals with an error the videobuf2 won't queue the buffer to
+the driver, instead it will flag it with an error. And then wait for the
+previous buffer to complete before asking userspace dequeue the buffer with
+error - to make sure we deliver the buffers back in the correct order.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.
+
+Note the the same `fence_fd` field is used for both sending the in-fence as
+input argument to receive the out-fence as a return argument.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. It will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the current buffer queues is able to keep the ordering of buffers, i.e.,
+the dequeing of buffers will happen at the same order we queue them, with no
+reordering by the driver.
 
 Return Value
 
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst 
b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
index dd54747fabc9..df964c4d916b 100644
--- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
@@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero 
to the
 number of buffers allocated with
 :ref:`VIDIOC_REQBUFS` (struct
 :c:type:`v4l2_requestbuffers` ``count``) minus
-one. The ``reserved`` and ``reserved2`` fields must be set to 0. When
+one. The ``reserved`` field must be set to 0

[PATCH v7 2/6] [media] v4l: add 'unordered' flag to format description ioctl

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 3 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst 
b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..368115f44fc0 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,9 @@ one until ``EINVAL`` is returned.
   - This format is not native to the device but emulated through
software (usually libv4l2), where possible try to use a native
format instead for better performance.
+* - ``V4L2_FMT_FLAG_UNORDERED``
+  - 0x0004
+  - This is a format that doesn't guarantee timely order of frames.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 982718965180..58894cfe9479 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -716,6 +716,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
/* Frame Size and frame rate enumeration */
 /*
-- 
2.14.3

[PATCH v7 2/6] [media] v4l: add 'unordered' flag to format description ioctl

[Gustavo Padovan]

From: Gustavo Padovan 

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 3 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst 
b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..368115f44fc0 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,9 @@ one until ``EINVAL`` is returned.
   - This format is not native to the device but emulated through
software (usually libv4l2), where possible try to use a native
format instead for better performance.
+* - ``V4L2_FMT_FLAG_UNORDERED``
+  - 0x0004
+  - This is a format that doesn't guarantee timely order of frames.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 982718965180..58894cfe9479 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -716,6 +716,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
/* Frame Size and frame rate enumeration */
 /*
-- 
2.14.3

[PATCH v7 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace on the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v8:
- return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
- fix crash when checking not using fences in vb2_buffer_done()

v7:
- merge patch that add the infrastructure to out-fences into
this one (Alex Courbot)
- Do not install the fd if there is no fence. (Alex Courbot)
- do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6
- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
ordering in vb2 for queueing in the driver, so the event is not
necessary anymore and the out_fence_fd is sent back to userspace
on QBUF call return arg
- do not allow requeueing with out-fences, instead mark the buffer
with an error and wake up to userspace.
- send the out_fence_fd back to userspace on the fence_fd field

v5:
- delay fd_install to DQ_EVENT (Hans)
- if queue is fully ordered send OUT_FENCE event right away
(Brian)
- rename 'q->ordered' to 'q->ordered_in_driver'
- merge change to implement OUT_FENCE event here

v4:
- return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
- set the OUT_FENCE flag if there is a fence pending (Hans)
- call fd_install() after vb2_core_qbuf() (Hans)
- clean up fence if vb2_core_qbuf() fails (Hans)
- add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/common/videobuf/videobuf2-core.c | 101 +++--
 drivers/media/common/videobuf/videobuf2-v4l2.c |  28 ++-
 include/media/videobuf2-core.h |  22 ++
 3 files changed, 140 insertions(+), 11 deletions(-)

diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
b/drivers/media/common/videobuf/videobuf2-core.c
index 777e3a2bc746..1f30d9efb7c8 100644
--- a/drivers/media/common/videobuf/videobuf2-core.c
+++ b/drivers/media/common/videobuf/videobuf2-core.c
@@ -25,6 +25,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #include 
@@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
}
+   vb->out_fence_fd = -1;
q->bufs[vb->index] = vb;
 
/* Allocate video buffer memory for the MMAP type */
@@ -939,10 +941,22 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
case VB2_BUF_STATE_QUEUED:
break;
case VB2_BUF_STATE_REQUEUEING:
+   /* Requeuing with explicit synchronization, spit warning */
+   WARN_ON_ONCE(vb->out_fence);
+
if (q->start_streaming_called)
__enqueue_in_driver(vb);
-   return;
+   break;
default:
+   if (vb->out_fence) {
+   if (state == VB2_BUF_STATE_ERROR)
+   dma_fence_set_error(vb->out_fence, -EFAULT);
+   dma_fence_signal(vb->out_fence);
+   dma_fence_put(vb->out_fence);
+   vb->out_fence = NULL;
+   vb->out_fence_fd = -1;
+   }
+
/* Inform any processes that may be waiting for buffers */
wake_up(>done_wq);
break;
@@ -1341,6 +1355,65 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+   return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+   return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+   return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+   .get_driver_name = vb2_fence_get_driver_name,
+   .get_timeline_name = vb2_fence_get_timeline_name,
+   .enable_signaling = vb2_fence_enable_signaling,
+   .wait = dma_fence_default_wait,
+};
+
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
+{
+   struct vb2_buffer *vb;
+   u64 context;
+
+   vb = q->bufs[index];
+
+   vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
+
+   if (call_qop(q, is_unordered, q))
+

[PATCH v7 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan 

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace on the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v8:
- return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
- fix crash when checking not using fences in vb2_buffer_done()

v7:
- merge patch that add the infrastructure to out-fences into
this one (Alex Courbot)
- Do not install the fd if there is no fence. (Alex Courbot)
- do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6
- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
ordering in vb2 for queueing in the driver, so the event is not
necessary anymore and the out_fence_fd is sent back to userspace
on QBUF call return arg
- do not allow requeueing with out-fences, instead mark the buffer
with an error and wake up to userspace.
- send the out_fence_fd back to userspace on the fence_fd field

v5:
- delay fd_install to DQ_EVENT (Hans)
- if queue is fully ordered send OUT_FENCE event right away
(Brian)
- rename 'q->ordered' to 'q->ordered_in_driver'
- merge change to implement OUT_FENCE event here

v4:
- return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
- set the OUT_FENCE flag if there is a fence pending (Hans)
- call fd_install() after vb2_core_qbuf() (Hans)
- clean up fence if vb2_core_qbuf() fails (Hans)
- add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/common/videobuf/videobuf2-core.c | 101 +++--
 drivers/media/common/videobuf/videobuf2-v4l2.c |  28 ++-
 include/media/videobuf2-core.h |  22 ++
 3 files changed, 140 insertions(+), 11 deletions(-)

diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
b/drivers/media/common/videobuf/videobuf2-core.c
index 777e3a2bc746..1f30d9efb7c8 100644
--- a/drivers/media/common/videobuf/videobuf2-core.c
+++ b/drivers/media/common/videobuf/videobuf2-core.c
@@ -25,6 +25,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #include 
@@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
}
+   vb->out_fence_fd = -1;
q->bufs[vb->index] = vb;
 
/* Allocate video buffer memory for the MMAP type */
@@ -939,10 +941,22 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
case VB2_BUF_STATE_QUEUED:
break;
case VB2_BUF_STATE_REQUEUEING:
+   /* Requeuing with explicit synchronization, spit warning */
+   WARN_ON_ONCE(vb->out_fence);
+
if (q->start_streaming_called)
__enqueue_in_driver(vb);
-   return;
+   break;
default:
+   if (vb->out_fence) {
+   if (state == VB2_BUF_STATE_ERROR)
+   dma_fence_set_error(vb->out_fence, -EFAULT);
+   dma_fence_signal(vb->out_fence);
+   dma_fence_put(vb->out_fence);
+   vb->out_fence = NULL;
+   vb->out_fence_fd = -1;
+   }
+
/* Inform any processes that may be waiting for buffers */
wake_up(>done_wq);
break;
@@ -1341,6 +1355,65 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+   return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+   return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+   return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+   .get_driver_name = vb2_fence_get_driver_name,
+   .get_timeline_name = vb2_fence_get_timeline_name,
+   .enable_signaling = vb2_fence_enable_signaling,
+   .wait = dma_fence_default_wait,
+};
+
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
+{
+   struct vb2_buffer *vb;
+   u64 context;
+
+   vb = q->bufs[index];
+
+   vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
+
+   if (call_qop(q, is_unordered, q))
+   context = dma_fence_context_alloc(1);
+   else
+   

[PATCH v7 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 include/media/videobuf2-core.h | 5 +
 1 file changed, 5 insertions(+)

diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index f3ee4c7c2fb3..583cdc06de79 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -370,6 +370,9 @@ struct vb2_buffer {
  * callback by calling vb2_buffer_done() with either
  * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  * vb2_wait_for_all_buffers() function
+ * @is_unordered:  tell if the queue format is unordered. The default is
+ * assumed to be ordered and this function only needs to
+ * be implemented for unordered queues.
  * @buf_queue: passes buffer vb to the driver; driver may start
  * hardware operation on this buffer; driver should give
  * the buffer back by calling vb2_buffer_done() function;
@@ -393,6 +396,7 @@ struct vb2_ops {
 
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
void (*stop_streaming)(struct vb2_queue *q);
+   int (*is_unordered)(struct vb2_queue *q);
 
void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -566,6 +570,7 @@ struct vb2_queue {
u32 cnt_wait_finish;
u32 cnt_start_streaming;
u32 cnt_stop_streaming;
+   u32 cnt_is_unordered;
 #endif
 };
 
-- 
2.14.3

[PATCH v7 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

From: Gustavo Padovan 

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

Signed-off-by: Gustavo Padovan 
---
 include/media/videobuf2-core.h | 5 +
 1 file changed, 5 insertions(+)

diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index f3ee4c7c2fb3..583cdc06de79 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -370,6 +370,9 @@ struct vb2_buffer {
  * callback by calling vb2_buffer_done() with either
  * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  * vb2_wait_for_all_buffers() function
+ * @is_unordered:  tell if the queue format is unordered. The default is
+ * assumed to be ordered and this function only needs to
+ * be implemented for unordered queues.
  * @buf_queue: passes buffer vb to the driver; driver may start
  * hardware operation on this buffer; driver should give
  * the buffer back by calling vb2_buffer_done() function;
@@ -393,6 +396,7 @@ struct vb2_ops {
 
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
void (*stop_streaming)(struct vb2_queue *q);
+   int (*is_unordered)(struct vb2_queue *q);
 
void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -566,6 +570,7 @@ struct vb2_queue {
u32 cnt_wait_finish;
u32 cnt_start_streaming;
u32 cnt_stop_streaming;
+   u32 cnt_is_unordered;
 #endif
 };
 
-- 
2.14.3

[PATCH v7 3/6] [media] vb2: add explicit fence user API

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel and return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending a fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v5:
- keep using reserved2 field for cpia2
- set fence_fd to 0 for now, for compat with userspace(Mauro)

v4:
- make it a union with reserved2 and fence_fd (Hans Verkuil)

v3:
- make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/buffer.rst| 15 +++
 drivers/media/common/videobuf/videobuf2-v4l2.c |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c  |  4 ++--
 include/uapi/linux/videodev2.h |  7 ++-
 4 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..eeefbd2547e7 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -648,6 +648,21 @@ Buffer Flags
   - Start Of Exposure. The buffer timestamp has been taken when the
exposure of the frame has begun. This is only valid for the
``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+* .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+  - ``V4L2_BUF_FLAG_IN_FENCE``
+  - 0x0020
+  - Ask V4L2 to wait on fence passed in ``fence_fd`` field. The buffer
+   won't be queued to the driver until the fence signals.
+
+* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+  - ``V4L2_BUF_FLAG_OUT_FENCE``
+  - 0x0040
+  - Request a fence to be attached to the buffer. The ``fence_fd``
+   field on
+   :ref:`VIDIOC_QBUF` is used as a return argument to send the out-fence
+   fd to userspace.
 
 
 
diff --git a/drivers/media/common/videobuf/videobuf2-v4l2.c 
b/drivers/media/common/videobuf/videobuf2-v4l2.c
index fac3cd6f901d..d838524a459e 100644
--- a/drivers/media/common/videobuf/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->fence_fd = 0;
b->reserved = 0;
 
if (q->is_multiplanar) {
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index e48d59046086..a11a0a2bed47 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -370,7 +370,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __s32   fence_fd;
__u32   reserved;
 };
 
@@ -533,7 +533,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct 
v4l2_buffer32 __user
put_user(kp->timestamp.tv_usec, >timestamp.tv_usec) ||
copy_to_user(>timecode, >timecode, sizeof(struct 
v4l2_timecode)) ||
put_user(kp->sequence, >sequence) ||
-   put_user(kp->reserved2, >reserved2) ||
+   put_user(kp->fence_fd, >fence_fd) ||
put_user(kp->reserved, >reserved) ||
put_user(kp->length, >length))
return -EFAULT;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 58894cfe9479..2d424aebdd1e 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -933,7 +933,10 @@ struct v4l2_buffer {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   union {
+   __s32   fence_fd;
+   __u32   reserved2;
+   };
__u32   reserved;
 };
 
@@ -970,6 +973,8 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE   0x0001
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST 0x0010
+#define V4L2_BUF_FLAG_IN_FENCE 0x0020
+#define V4L2_BUF_FLAG_OUT_FENCE0x0040
 
 /**
  * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
-- 
2.14.3

[PATCH v7 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queue to the driver
out of the order they were queued from userspace. That means that even if
it fence signal it must wait all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v8: - improve comments about fences with errors

v7:
- get rid of the fence array stuff for ordering and just use
get_num_buffers_ready() (Hans)
- fix issue of queuing the buffer twice (Hans)
- avoid the dma_fence_wait() in core_qbuf() (Alex)
- merge preparation commit in

v6:
- With fences always keep the order userspace queues the buffers.
- Protect in_fence manipulation with a lock (Brian Starkey)
- check if fences have the same context before adding a fence array
- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
- Clean up fence if __set_in_fence() fails (Brian Starkey)
- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
needed (Brian Starkey)
- keep backward compat on the reserved2 field (Brian Starkey)
- protect fence callback removal with lock (Brian Starkey)

v4:
- Add a comment about dma_fence_add_callback() not returning a
error (Hans)
- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
vb2_start_streaming() (Hans)
- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
- Queue buffers to the driver as soon as they are ready (Hans)
- call fill_user_buffer() after queuing the buffer (Hans)
- add err: label to clean up fence
- add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
- remove ternary if at vb2_qbuf() return (Mauro)
- do not change if conditions behaviour (Mauro)

v2:
- fix vb2_queue_or_prepare_buf() ret check
- remove check for VB2_MEMORY_DMABUF only (Javier)
- check num of ready buffers to start streaming
- when queueing, start from the first ready buffer
- handle queue cancel

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/common/videobuf/videobuf2-core.c | 166 ++---
 drivers/media/common/videobuf/videobuf2-v4l2.c |  29 -
 drivers/media/v4l2-core/Kconfig|  33 +
 include/media/videobuf2-core.h |  14 ++-
 4 files changed, 221 insertions(+), 21 deletions(-)

diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
b/drivers/media/common/videobuf/videobuf2-core.c
index f7109f827f6e..777e3a2bc746 100644
--- a/drivers/media/common/videobuf/videobuf2-core.c
+++ b/drivers/media/common/videobuf/videobuf2-core.c
@@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->index = q->num_buffers + buffer;
vb->type = q->type;
vb->memory = memory;
+   spin_lock_init(>fence_cb_lock);
for (plane = 0; plane < num_planes; ++plane) {
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
@@ -936,7 +937,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
 
switch (state) {
case VB2_BUF_STATE_QUEUED:
-   return;
+   break;
case VB2_BUF_STATE_REQUEUEING:
if (q->start_streaming_called)
__enqueue_in_driver(vb);
@@ -946,6 +947,19 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
wake_up(>done_wq);
break;
}
+
+   /*
+* The check below verifies if there is a buffer in queue with an
+* error state. They are added to queue in the error state when
+* their in-fence fails to signal.
+* To not mess with buffer ordering we wait until the previous buffer
+* is done to mark the buffer in the error state as done and notify
+* userspace. So everytime a buffer is done we check the next one for
+* VB2_BUF_STATE_ERROR.
+ 

[PATCH v7 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Hi,

v7 bring a fix for a crash when not using fences and a uAPI fix.
I've done a bit more of testing on it and also measured some
performance. On a intel laptop a DRM<->V4L2 pipeline with fences is
runnning twice as faster than the same pipeline with no fences.

For more details on how fences work refer to patch 6 in this series.

The test tools I've been using are:
https://gitlab.collabora.com/padovan/drm-v4l2-test
https://gitlab.collabora.com/padovan/v4l2-fences-test

Please review,

Gustavo

Gustavo Padovan (6):
  [media] vb2: add is_unordered callback for drivers
  [media] v4l: add 'unordered' flag to format description ioctl
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add out-fence support to QBUF
  [media] v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst  |  15 ++
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst |   3 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst |  47 -
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |   9 +-
 drivers/media/common/videobuf/videobuf2-core.c   | 253 +--
 drivers/media/common/videobuf/videobuf2-v4l2.c   |  51 -
 drivers/media/v4l2-core/Kconfig  |  33 +++
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c|   4 +-
 include/media/videobuf2-core.h   |  41 +++-
 include/uapi/linux/videodev2.h   |   8 +-
 10 files changed, 437 insertions(+), 27 deletions(-)

-- 
2.14.3

[PATCH v7 3/6] [media] vb2: add explicit fence user API

[Gustavo Padovan]

From: Gustavo Padovan 

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel and return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending a fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v5:
- keep using reserved2 field for cpia2
- set fence_fd to 0 for now, for compat with userspace(Mauro)

v4:
- make it a union with reserved2 and fence_fd (Hans Verkuil)

v3:
- make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/buffer.rst| 15 +++
 drivers/media/common/videobuf/videobuf2-v4l2.c |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c  |  4 ++--
 include/uapi/linux/videodev2.h |  7 ++-
 4 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..eeefbd2547e7 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -648,6 +648,21 @@ Buffer Flags
   - Start Of Exposure. The buffer timestamp has been taken when the
exposure of the frame has begun. This is only valid for the
``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+* .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+  - ``V4L2_BUF_FLAG_IN_FENCE``
+  - 0x0020
+  - Ask V4L2 to wait on fence passed in ``fence_fd`` field. The buffer
+   won't be queued to the driver until the fence signals.
+
+* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+  - ``V4L2_BUF_FLAG_OUT_FENCE``
+  - 0x0040
+  - Request a fence to be attached to the buffer. The ``fence_fd``
+   field on
+   :ref:`VIDIOC_QBUF` is used as a return argument to send the out-fence
+   fd to userspace.
 
 
 
diff --git a/drivers/media/common/videobuf/videobuf2-v4l2.c 
b/drivers/media/common/videobuf/videobuf2-v4l2.c
index fac3cd6f901d..d838524a459e 100644
--- a/drivers/media/common/videobuf/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->fence_fd = 0;
b->reserved = 0;
 
if (q->is_multiplanar) {
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index e48d59046086..a11a0a2bed47 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -370,7 +370,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __s32   fence_fd;
__u32   reserved;
 };
 
@@ -533,7 +533,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct 
v4l2_buffer32 __user
put_user(kp->timestamp.tv_usec, >timestamp.tv_usec) ||
copy_to_user(>timecode, >timecode, sizeof(struct 
v4l2_timecode)) ||
put_user(kp->sequence, >sequence) ||
-   put_user(kp->reserved2, >reserved2) ||
+   put_user(kp->fence_fd, >fence_fd) ||
put_user(kp->reserved, >reserved) ||
put_user(kp->length, >length))
return -EFAULT;
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 58894cfe9479..2d424aebdd1e 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -933,7 +933,10 @@ struct v4l2_buffer {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   union {
+   __s32   fence_fd;
+   __u32   reserved2;
+   };
__u32   reserved;
 };
 
@@ -970,6 +973,8 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE   0x0001
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST 0x0010
+#define V4L2_BUF_FLAG_IN_FENCE 0x0020
+#define V4L2_BUF_FLAG_OUT_FENCE0x0040
 
 /**
  * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
-- 
2.14.3

[PATCH v7 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan 

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queue to the driver
out of the order they were queued from userspace. That means that even if
it fence signal it must wait all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v8: - improve comments about fences with errors

v7:
- get rid of the fence array stuff for ordering and just use
get_num_buffers_ready() (Hans)
- fix issue of queuing the buffer twice (Hans)
- avoid the dma_fence_wait() in core_qbuf() (Alex)
- merge preparation commit in

v6:
- With fences always keep the order userspace queues the buffers.
- Protect in_fence manipulation with a lock (Brian Starkey)
- check if fences have the same context before adding a fence array
- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
- Clean up fence if __set_in_fence() fails (Brian Starkey)
- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
needed (Brian Starkey)
- keep backward compat on the reserved2 field (Brian Starkey)
- protect fence callback removal with lock (Brian Starkey)

v4:
- Add a comment about dma_fence_add_callback() not returning a
error (Hans)
- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
vb2_start_streaming() (Hans)
- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
- Queue buffers to the driver as soon as they are ready (Hans)
- call fill_user_buffer() after queuing the buffer (Hans)
- add err: label to clean up fence
- add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
- remove ternary if at vb2_qbuf() return (Mauro)
- do not change if conditions behaviour (Mauro)

v2:
- fix vb2_queue_or_prepare_buf() ret check
- remove check for VB2_MEMORY_DMABUF only (Javier)
- check num of ready buffers to start streaming
- when queueing, start from the first ready buffer
- handle queue cancel

Signed-off-by: Gustavo Padovan 
---
 drivers/media/common/videobuf/videobuf2-core.c | 166 ++---
 drivers/media/common/videobuf/videobuf2-v4l2.c |  29 -
 drivers/media/v4l2-core/Kconfig|  33 +
 include/media/videobuf2-core.h |  14 ++-
 4 files changed, 221 insertions(+), 21 deletions(-)

diff --git a/drivers/media/common/videobuf/videobuf2-core.c 
b/drivers/media/common/videobuf/videobuf2-core.c
index f7109f827f6e..777e3a2bc746 100644
--- a/drivers/media/common/videobuf/videobuf2-core.c
+++ b/drivers/media/common/videobuf/videobuf2-core.c
@@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->index = q->num_buffers + buffer;
vb->type = q->type;
vb->memory = memory;
+   spin_lock_init(>fence_cb_lock);
for (plane = 0; plane < num_planes; ++plane) {
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
@@ -936,7 +937,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
 
switch (state) {
case VB2_BUF_STATE_QUEUED:
-   return;
+   break;
case VB2_BUF_STATE_REQUEUEING:
if (q->start_streaming_called)
__enqueue_in_driver(vb);
@@ -946,6 +947,19 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
wake_up(>done_wq);
break;
}
+
+   /*
+* The check below verifies if there is a buffer in queue with an
+* error state. They are added to queue in the error state when
+* their in-fence fails to signal.
+* To not mess with buffer ordering we wait until the previous buffer
+* is done to mark the buffer in the error state as done and notify
+* userspace. So everytime a buffer is done we check the next one for
+* VB2_BUF_STATE_ERROR.
+*/
+   vb = list_next_entry(vb, queued_entry);
+   if 

[PATCH v7 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan 

Hi,

v7 bring a fix for a crash when not using fences and a uAPI fix.
I've done a bit more of testing on it and also measured some
performance. On a intel laptop a DRM<->V4L2 pipeline with fences is
runnning twice as faster than the same pipeline with no fences.

For more details on how fences work refer to patch 6 in this series.

The test tools I've been using are:
https://gitlab.collabora.com/padovan/drm-v4l2-test
https://gitlab.collabora.com/padovan/v4l2-fences-test

Please review,

Gustavo

Gustavo Padovan (6):
  [media] vb2: add is_unordered callback for drivers
  [media] v4l: add 'unordered' flag to format description ioctl
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add out-fence support to QBUF
  [media] v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst  |  15 ++
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst |   3 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst |  47 -
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |   9 +-
 drivers/media/common/videobuf/videobuf2-core.c   | 253 +--
 drivers/media/common/videobuf/videobuf2-v4l2.c   |  51 -
 drivers/media/v4l2-core/Kconfig  |  33 +++
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c|   4 +-
 include/media/videobuf2-core.h   |  41 +++-
 include/uapi/linux/videodev2.h   |   8 +-
 10 files changed, 437 insertions(+), 27 deletions(-)

-- 
2.14.3

Re: [PATCH v6 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab <mche...@osg.samsung.com>:

> Em Mon, 11 Dec 2017 16:27:39 -0200
> Gustavo Padovan <gust...@padovan.org> escreveu:
> 
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the driver,
> > instead we mark it as error and wait until the previous buffer is done
> > to notify userspace of the error. We wait here to deliver the buffers back
> > to userspace in order.
> > 
> > v7:
> > - get rid of the fence array stuff for ordering and just use
> > get_num_buffers_ready() (Hans)
> > - fix issue of queuing the buffer twice (Hans)
> > - avoid the dma_fence_wait() in core_qbuf() (Alex)
> > - merge preparation commit in
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  drivers/media/v4l2-core/Kconfig  |   1 +
> >  drivers/media/v4l2-core/videobuf2-core.c | 154 
> > +++
> >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 +-
> >  include/media/videobuf2-core.h   |  14 ++-
> >  4 files changed, 177 insertions(+), 21 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/Kconfig 
> > b/drivers/media/v4l2-core/Kconfig
> > index a35c33686abf..3f988c407c80 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> > select DMA_SHARED_BUFFER
> > +   select SYNC_FILE
> > tristate
> >  
> >  config VIDEOBUF2_MEMOPS
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index a8589d96ef72..520aa3c7d9f0 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -346,6 +346,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   

Re: [PATCH v6 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab :

> Em Mon, 11 Dec 2017 16:27:39 -0200
> Gustavo Padovan  escreveu:
> 
> > From: Gustavo Padovan 
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the driver,
> > instead we mark it as error and wait until the previous buffer is done
> > to notify userspace of the error. We wait here to deliver the buffers back
> > to userspace in order.
> > 
> > v7:
> > - get rid of the fence array stuff for ordering and just use
> > get_num_buffers_ready() (Hans)
> > - fix issue of queuing the buffer twice (Hans)
> > - avoid the dma_fence_wait() in core_qbuf() (Alex)
> > - merge preparation commit in
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan 
> > ---
> >  drivers/media/v4l2-core/Kconfig  |   1 +
> >  drivers/media/v4l2-core/videobuf2-core.c | 154 
> > +++
> >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 +-
> >  include/media/videobuf2-core.h   |  14 ++-
> >  4 files changed, 177 insertions(+), 21 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/Kconfig 
> > b/drivers/media/v4l2-core/Kconfig
> > index a35c33686abf..3f988c407c80 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> > select DMA_SHARED_BUFFER
> > +   select SYNC_FILE
> > tristate
> >  
> >  config VIDEOBUF2_MEMOPS
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index a8589d96ef72..520aa3c7d9f0 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -346,6 +346,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   spin_lock_init(>fence_cb_lock);
> > for (plane = 0; plane < num_planes; ++plane) {
> > 

Re: [PATCH v6 3/6] [media] vb2: add explicit fence user API

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab <mche...@osg.samsung.com>:

> Em Mon, 11 Dec 2017 16:27:38 -0200
> Gustavo Padovan <gust...@padovan.org> escreveu:
> 
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Turn the reserved2 field into fence_fd that we will use to send
> > an in-fence to the kernel and return an out-fence from the kernel to
> > userspace.
> > 
> > Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> > when sending a fence to the kernel to be waited on, and
> > V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> > 
> > v4:
> > - make it a union with reserved2 and fence_fd (Hans Verkuil)
> > 
> > v3:
> > - make the out_fence refer to the current buffer (Hans Verkuil)
> > 
> > v2: add documentation
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  Documentation/media/uapi/v4l/buffer.rst   | 15 +++
> >  drivers/media/usb/cpia2/cpia2_v4l.c   |  2 +-
> >  drivers/media/v4l2-core/v4l2-compat-ioctl32.c |  4 ++--
> >  drivers/media/v4l2-core/videobuf2-v4l2.c  |  2 +-
> >  include/uapi/linux/videodev2.h|  7 ++-
> >  5 files changed, 25 insertions(+), 5 deletions(-)
> > 
> > diff --git a/Documentation/media/uapi/v4l/buffer.rst 
> > b/Documentation/media/uapi/v4l/buffer.rst
> > index ae6ee73f151c..eeefbd2547e7 100644
> > --- a/Documentation/media/uapi/v4l/buffer.rst
> > +++ b/Documentation/media/uapi/v4l/buffer.rst
> > @@ -648,6 +648,21 @@ Buffer Flags
> >- Start Of Exposure. The buffer timestamp has been taken when the
> > exposure of the frame has begun. This is only valid for the
> > ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> > +* .. _`V4L2-BUF-FLAG-IN-FENCE`:
> > +
> > +  - ``V4L2_BUF_FLAG_IN_FENCE``
> > +  - 0x0020
> > +  - Ask V4L2 to wait on fence passed in ``fence_fd`` field. The buffer
> > +   won't be queued to the driver until the fence signals.
> > +
> > +* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> > +
> > +  - ``V4L2_BUF_FLAG_OUT_FENCE``
> > +  - 0x0040
> > +  - Request a fence to be attached to the buffer. The ``fence_fd``
> > +   field on
> > +   :ref:`VIDIOC_QBUF` is used as a return argument to send the out-fence
> > +   fd to userspace.
> >  
> >  
> >  
> > diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
> > b/drivers/media/usb/cpia2/cpia2_v4l.c
> > index a1c59f19cf2d..7d7459fa2077 100644
> > --- a/drivers/media/usb/cpia2/cpia2_v4l.c
> > +++ b/drivers/media/usb/cpia2/cpia2_v4l.c
> > @@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, 
> > struct v4l2_buffer *buf)
> > buf->sequence = cam->buffers[buf->index].seq;
> > buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
> > buf->length = cam->frame_size;
> > -   buf->reserved2 = 0;
> > +   buf->fence_fd = -1;
> > buf->reserved = 0;
> > memset(>timecode, 0, sizeof(buf->timecode));
> >  
> > diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
> > b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> > index 821f2aa299ae..3a31d318df2a 100644
> > --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> > +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> > @@ -370,7 +370,7 @@ struct v4l2_buffer32 {
> > __s32   fd;
> > } m;
> > __u32   length;
> > -   __u32   reserved2;
> > +   __s32   fence_fd;
> > __u32   reserved;
> >  };
> >  
> > @@ -533,7 +533,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, 
> > struct v4l2_buffer32 __user
> > put_user(kp->timestamp.tv_usec, >timestamp.tv_usec) ||
> > copy_to_user(>timecode, >timecode, sizeof(struct 
> > v4l2_timecode)) ||
> > put_user(kp->sequence, >sequence) ||
> > -   put_user(kp->reserved2, >reserved2) ||
> > +   put_user(kp->fence_fd, >fence_fd) ||
> > put_user(kp->reserved, >reserved) ||
> > put_user(kp->length, >length))
> > return -EFAULT;
> > diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c 
> > b/drivers/media/v4l2-core/videobuf2-v4l2.c
> > index 4075314a6989..4a5244ee2c58 100644
> > --- a/drivers/media/v4l2-core/videobuf2-v4l2.c
> > +++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
> > @@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, 
> > void *pb)
> > b->timestamp = ns_to_timeval(vb->timestamp);
> > b->timecode = vbuf->timecode;
> > b->sequence = vbuf->sequence;
> > -   b->reserved2 = 0;
> > +   b->fence_fd = -1;
> 
> The patch itself looks ok. I'm just in doubt here, but it is probably
> ok to change its default to -1.

Right. What we can do is return 0 if the OUT_FENCE flag is not set.

Gustavo

Re: [PATCH v6 3/6] [media] vb2: add explicit fence user API

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab :

> Em Mon, 11 Dec 2017 16:27:38 -0200
> Gustavo Padovan  escreveu:
> 
> > From: Gustavo Padovan 
> > 
> > Turn the reserved2 field into fence_fd that we will use to send
> > an in-fence to the kernel and return an out-fence from the kernel to
> > userspace.
> > 
> > Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> > when sending a fence to the kernel to be waited on, and
> > V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> > 
> > v4:
> > - make it a union with reserved2 and fence_fd (Hans Verkuil)
> > 
> > v3:
> > - make the out_fence refer to the current buffer (Hans Verkuil)
> > 
> > v2: add documentation
> > 
> > Signed-off-by: Gustavo Padovan 
> > ---
> >  Documentation/media/uapi/v4l/buffer.rst   | 15 +++
> >  drivers/media/usb/cpia2/cpia2_v4l.c   |  2 +-
> >  drivers/media/v4l2-core/v4l2-compat-ioctl32.c |  4 ++--
> >  drivers/media/v4l2-core/videobuf2-v4l2.c  |  2 +-
> >  include/uapi/linux/videodev2.h|  7 ++-
> >  5 files changed, 25 insertions(+), 5 deletions(-)
> > 
> > diff --git a/Documentation/media/uapi/v4l/buffer.rst 
> > b/Documentation/media/uapi/v4l/buffer.rst
> > index ae6ee73f151c..eeefbd2547e7 100644
> > --- a/Documentation/media/uapi/v4l/buffer.rst
> > +++ b/Documentation/media/uapi/v4l/buffer.rst
> > @@ -648,6 +648,21 @@ Buffer Flags
> >- Start Of Exposure. The buffer timestamp has been taken when the
> > exposure of the frame has begun. This is only valid for the
> > ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> > +* .. _`V4L2-BUF-FLAG-IN-FENCE`:
> > +
> > +  - ``V4L2_BUF_FLAG_IN_FENCE``
> > +  - 0x0020
> > +  - Ask V4L2 to wait on fence passed in ``fence_fd`` field. The buffer
> > +   won't be queued to the driver until the fence signals.
> > +
> > +* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> > +
> > +  - ``V4L2_BUF_FLAG_OUT_FENCE``
> > +  - 0x0040
> > +  - Request a fence to be attached to the buffer. The ``fence_fd``
> > +   field on
> > +   :ref:`VIDIOC_QBUF` is used as a return argument to send the out-fence
> > +   fd to userspace.
> >  
> >  
> >  
> > diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
> > b/drivers/media/usb/cpia2/cpia2_v4l.c
> > index a1c59f19cf2d..7d7459fa2077 100644
> > --- a/drivers/media/usb/cpia2/cpia2_v4l.c
> > +++ b/drivers/media/usb/cpia2/cpia2_v4l.c
> > @@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, 
> > struct v4l2_buffer *buf)
> > buf->sequence = cam->buffers[buf->index].seq;
> > buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
> > buf->length = cam->frame_size;
> > -   buf->reserved2 = 0;
> > +   buf->fence_fd = -1;
> > buf->reserved = 0;
> > memset(>timecode, 0, sizeof(buf->timecode));
> >  
> > diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
> > b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> > index 821f2aa299ae..3a31d318df2a 100644
> > --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> > +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> > @@ -370,7 +370,7 @@ struct v4l2_buffer32 {
> > __s32   fd;
> > } m;
> > __u32   length;
> > -   __u32   reserved2;
> > +   __s32   fence_fd;
> > __u32   reserved;
> >  };
> >  
> > @@ -533,7 +533,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, 
> > struct v4l2_buffer32 __user
> > put_user(kp->timestamp.tv_usec, >timestamp.tv_usec) ||
> > copy_to_user(>timecode, >timecode, sizeof(struct 
> > v4l2_timecode)) ||
> > put_user(kp->sequence, >sequence) ||
> > -   put_user(kp->reserved2, >reserved2) ||
> > +   put_user(kp->fence_fd, >fence_fd) ||
> > put_user(kp->reserved, >reserved) ||
> > put_user(kp->length, >length))
> > return -EFAULT;
> > diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c 
> > b/drivers/media/v4l2-core/videobuf2-v4l2.c
> > index 4075314a6989..4a5244ee2c58 100644
> > --- a/drivers/media/v4l2-core/videobuf2-v4l2.c
> > +++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
> > @@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, 
> > void *pb)
> > b->timestamp = ns_to_timeval(vb->timestamp);
> > b->timecode = vbuf->timecode;
> > b->sequence = vbuf->sequence;
> > -   b->reserved2 = 0;
> > +   b->fence_fd = -1;
> 
> The patch itself looks ok. I'm just in doubt here, but it is probably
> ok to change its default to -1.

Right. What we can do is return 0 if the OUT_FENCE flag is not set.

Gustavo

Re: [PATCH v6 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab <mche...@osg.samsung.com>:

> Em Mon, 11 Dec 2017 16:27:36 -0200
> Gustavo Padovan <gust...@padovan.org> escreveu:
> 
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Explicit synchronization benefits a lot from ordered queues, they fit
> > better in a pipeline with DRM for example so create a opt-in way for
> > drivers notify videobuf2 that the queue is unordered.
> > 
> > Drivers don't need implement it if the queue is ordered.
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  include/media/videobuf2-core.h | 5 +
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> > index ef9b64398c8c..eddb38a2a2f3 100644
> > --- a/include/media/videobuf2-core.h
> > +++ b/include/media/videobuf2-core.h
> > @@ -368,6 +368,9 @@ struct vb2_buffer {
> >   * callback by calling vb2_buffer_done() with either
> >   * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
> >   * vb2_wait_for_all_buffers() function
> > + * @is_unordered:  tell if the queue format is unordered. The default is
> > + * assumed to be ordered and this function only needs to
> > + * be implemented for unordered queues.
> >   * @buf_queue: passes buffer vb to the driver; driver may start
> >   * hardware operation on this buffer; driver should give
> >   * the buffer back by calling vb2_buffer_done() function;
> > @@ -391,6 +394,7 @@ struct vb2_ops {
> >  
> > int (*start_streaming)(struct vb2_queue *q, unsigned int count);
> > void (*stop_streaming)(struct vb2_queue *q);
> > +   int (*is_unordered)(struct vb2_queue *q);
> >  
> > void (*buf_queue)(struct vb2_buffer *vb);
> >  };
> > @@ -564,6 +568,7 @@ struct vb2_queue {
> > u32 cnt_wait_finish;
> > u32 cnt_start_streaming;
> > u32 cnt_stop_streaming;
> > +   u32 cnt_is_unordered;
> 
> If I understand, this is just a bit, right?
> 
> if so, better to declare it as:
> 
>   u32 cnt_is_unordered : 1;

no, is_unordered() is a vb2 callback to ask drivers if their current
queue is not ordered and cnt_is_unordered is the counter of how many
times this callback was called. This happens for all vb2 callbacks, see
the other counters above, so I decided to add one for is_unordered() as
well.

Gustavo

Re: [PATCH v6 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab :

> Em Mon, 11 Dec 2017 16:27:36 -0200
> Gustavo Padovan  escreveu:
> 
> > From: Gustavo Padovan 
> > 
> > Explicit synchronization benefits a lot from ordered queues, they fit
> > better in a pipeline with DRM for example so create a opt-in way for
> > drivers notify videobuf2 that the queue is unordered.
> > 
> > Drivers don't need implement it if the queue is ordered.
> > 
> > Signed-off-by: Gustavo Padovan 
> > ---
> >  include/media/videobuf2-core.h | 5 +
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> > index ef9b64398c8c..eddb38a2a2f3 100644
> > --- a/include/media/videobuf2-core.h
> > +++ b/include/media/videobuf2-core.h
> > @@ -368,6 +368,9 @@ struct vb2_buffer {
> >   * callback by calling vb2_buffer_done() with either
> >   * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
> >   * vb2_wait_for_all_buffers() function
> > + * @is_unordered:  tell if the queue format is unordered. The default is
> > + * assumed to be ordered and this function only needs to
> > + * be implemented for unordered queues.
> >   * @buf_queue: passes buffer vb to the driver; driver may start
> >   * hardware operation on this buffer; driver should give
> >   * the buffer back by calling vb2_buffer_done() function;
> > @@ -391,6 +394,7 @@ struct vb2_ops {
> >  
> > int (*start_streaming)(struct vb2_queue *q, unsigned int count);
> > void (*stop_streaming)(struct vb2_queue *q);
> > +   int (*is_unordered)(struct vb2_queue *q);
> >  
> > void (*buf_queue)(struct vb2_buffer *vb);
> >  };
> > @@ -564,6 +568,7 @@ struct vb2_queue {
> > u32 cnt_wait_finish;
> > u32 cnt_start_streaming;
> > u32 cnt_stop_streaming;
> > +   u32 cnt_is_unordered;
> 
> If I understand, this is just a bit, right?
> 
> if so, better to declare it as:
> 
>   u32 cnt_is_unordered : 1;

no, is_unordered() is a vb2 callback to ask drivers if their current
queue is not ordered and cnt_is_unordered is the counter of how many
times this callback was called. This happens for all vb2 callbacks, see
the other counters above, so I decided to add one for is_unordered() as
well.

Gustavo

Re: [PATCH v6 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab <mche...@osg.samsung.com>:

> Em Mon, 11 Dec 2017 16:27:35 -0200
> Gustavo Padovan <gust...@padovan.org> escreveu:
> 
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Hi,
> > 
> > One more iteration of the explicit fences patches, please refer
> > to the previous version[1] for more details about the general
> > mechanism
> > 
> > This version makes the patchset and the implementation much more
> > simple, to start we are not using a ordered capability anymore,
> > but instead we have a VIDIOC_ENUM_FMT flag to tell when the
> > queue in not ordered. Drivers with ordered queues/formats don't
> > need implement anything. See patches 1 and 2 for more details.
> > 
> > The implementation of in-fences and out-fences were condensed in
> > just patches 4 and 5, making it more self-contained and easy to
> > understand. See the patches for detailed changelog.
> > 
> > Please review! Thanks.
> 
> Hi Gustavo,
> 
> As I was afraid, the changes at the VB2 core makes it non-generic,
> breaking support for the DVB VB2 patchset. That's a branch with
> both patchsets applied:
> 
>   
> https://git.linuxtv.org/mchehab/experimental.git/log/?h=dvb_mmap%2bexplicit_fences
> 
> 
> With the explicit fences patchset, the DVB streaming breaks:
> 
>   $ sudo perf stat dvbv5-zap -c ~/dvb_channel.conf "TV Brasilia RedeTV!"  
> -o /dev/null -t120 -R
>   Usando demux 'dvb0.demux0'
>   lendo canais do arquivo  '/home/mchehab/dvb_channel.conf'
>   sintonizando em 557142857 Hz
>   PID de vídeo 273
> dvb_set_pesfilter 273
>   PID de áudio 274
> dvb_set_pesfilter 274
>   Travado  (0x1f) Sinal= -85,22dBm C/N= 18,57dB UCB= 8589933955 pós-BER= 0
>   Travado  (0x1f) Sinal= -85,24dBm C/N= 18,57dB UCB= 8589933955 pós-BER= 0
>   Gravação iniciada para o arquivo '/dev/null'
>   ERRO:DMX_REQBUFS failed: error=-1 (Invalid argument)
>   ERRO:[stream_to_file] Failed to setup buffers!!! (Invalid argument)
>   start streaming!!!
>   copied 0 bytes (0 Kbytes/sec)
>   Travado  (0x1f) Sinal= -85,25dBm C/N= 18,57dB UCB= 8589933955 pós-BER= 0

I haven't tried it, not sure what could break it. I'll take a look on
it.

>  Performance counter stats for 'dvbv5-zap -c /home/mchehab/dvb_channel.conf 
> TV Brasilia RedeTV! -o /dev/null -t120 -R':
> 
>7.001647  task-clock-msecs #  0.003 CPUs 
> 251  context-switches #  0.036 M/sec
>  18  CPU-migrations   #  0.003 M/sec
> 181  page-faults  #  0.026 M/sec
>17001058  cycles   #   2428.151 M/sec
>11342660  instructions #  0.667 IPC  
>  349075  cache-references # 49.856 M/sec
>   70802  cache-misses # 10.112 M/sec
> 
> 2.133343557  seconds time elapsed
> 
> It also breaks support on V4L2, when fences is not used:
> 
> $ ./contrib/test/v4l2grab 
> (kernel crashes)
> 
> I don't have a serial console on this machine to print what's
> wrong, but clearly there's something not right there :-)

2 days ago I fixed the a crash when not using fences:

https://git.kernel.org/pub/scm/linux/kernel/git/padovan/linux.git/commit/?h=v4l2-fences=704419e59437e9f7bdc369b44a612685e8663880

Gustavo

Re: [PATCH v6 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

2017-12-21 Mauro Carvalho Chehab :

> Em Mon, 11 Dec 2017 16:27:35 -0200
> Gustavo Padovan  escreveu:
> 
> > From: Gustavo Padovan 
> > 
> > Hi,
> > 
> > One more iteration of the explicit fences patches, please refer
> > to the previous version[1] for more details about the general
> > mechanism
> > 
> > This version makes the patchset and the implementation much more
> > simple, to start we are not using a ordered capability anymore,
> > but instead we have a VIDIOC_ENUM_FMT flag to tell when the
> > queue in not ordered. Drivers with ordered queues/formats don't
> > need implement anything. See patches 1 and 2 for more details.
> > 
> > The implementation of in-fences and out-fences were condensed in
> > just patches 4 and 5, making it more self-contained and easy to
> > understand. See the patches for detailed changelog.
> > 
> > Please review! Thanks.
> 
> Hi Gustavo,
> 
> As I was afraid, the changes at the VB2 core makes it non-generic,
> breaking support for the DVB VB2 patchset. That's a branch with
> both patchsets applied:
> 
>   
> https://git.linuxtv.org/mchehab/experimental.git/log/?h=dvb_mmap%2bexplicit_fences
> 
> 
> With the explicit fences patchset, the DVB streaming breaks:
> 
>   $ sudo perf stat dvbv5-zap -c ~/dvb_channel.conf "TV Brasilia RedeTV!"  
> -o /dev/null -t120 -R
>   Usando demux 'dvb0.demux0'
>   lendo canais do arquivo  '/home/mchehab/dvb_channel.conf'
>   sintonizando em 557142857 Hz
>   PID de vídeo 273
> dvb_set_pesfilter 273
>   PID de áudio 274
> dvb_set_pesfilter 274
>   Travado  (0x1f) Sinal= -85,22dBm C/N= 18,57dB UCB= 8589933955 pós-BER= 0
>   Travado  (0x1f) Sinal= -85,24dBm C/N= 18,57dB UCB= 8589933955 pós-BER= 0
>   Gravação iniciada para o arquivo '/dev/null'
>   ERRO:DMX_REQBUFS failed: error=-1 (Invalid argument)
>   ERRO:[stream_to_file] Failed to setup buffers!!! (Invalid argument)
>   start streaming!!!
>   copied 0 bytes (0 Kbytes/sec)
>   Travado  (0x1f) Sinal= -85,25dBm C/N= 18,57dB UCB= 8589933955 pós-BER= 0

I haven't tried it, not sure what could break it. I'll take a look on
it.

>  Performance counter stats for 'dvbv5-zap -c /home/mchehab/dvb_channel.conf 
> TV Brasilia RedeTV! -o /dev/null -t120 -R':
> 
>7.001647  task-clock-msecs #  0.003 CPUs 
> 251  context-switches #  0.036 M/sec
>  18  CPU-migrations   #  0.003 M/sec
> 181  page-faults  #  0.026 M/sec
>17001058  cycles   #   2428.151 M/sec
>11342660  instructions #  0.667 IPC  
>  349075  cache-references # 49.856 M/sec
>   70802  cache-misses # 10.112 M/sec
> 
> 2.133343557  seconds time elapsed
> 
> It also breaks support on V4L2, when fences is not used:
> 
> $ ./contrib/test/v4l2grab 
> (kernel crashes)
> 
> I don't have a serial console on this machine to print what's
> wrong, but clearly there's something not right there :-)

2 days ago I fixed the a crash when not using fences:

https://git.kernel.org/pub/scm/linux/kernel/git/padovan/linux.git/commit/?h=v4l2-fences=704419e59437e9f7bdc369b44a612685e8663880

Gustavo

Re: [RFC PATCH 9/9] media: vim2m: add request support

[Gustavo Padovan]

Hi Alex,

2017-12-15 Alexandre Courbot :

> Set the necessary ops for supporting requests in vim2m.
> 
> Signed-off-by: Alexandre Courbot 
> ---
>  drivers/media/platform/vim2m.c | 22 ++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index a32e8a7950eb..ffe94ef9214d 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -30,6 +30,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>  
>  MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
>  MODULE_AUTHOR("Pawel Osciak, ");
> @@ -142,6 +143,7 @@ struct vim2m_dev {
>   struct video_device vfd;
>  #ifdef CONFIG_MEDIA_CONTROLLER
>   struct media_device mdev;
> + struct media_request_queue *req_queue;
>  #endif
>  
>   atomic_tnum_inst;
> @@ -937,6 +939,11 @@ static int vim2m_open(struct file *file)
>   goto open_unlock;
>   }
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> + v4l2_mem_ctx_request_init(ctx->fh.m2m_ctx, dev->req_queue,
> +   >vfd.entity);
> +#endif
> +
>   v4l2_fh_add(>fh);
>   atomic_inc(>num_inst);
>  
> @@ -992,6 +999,12 @@ static const struct v4l2_m2m_ops m2m_ops = {
>   .job_abort  = job_abort,
>  };
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +static const struct media_entity_operations vim2m_entity_ops = {
> + .process_request = v4l2_m2m_process_request,
> +};
> +#endif
> +
>  static int vim2m_probe(struct platform_device *pdev)
>  {
>   struct vim2m_dev *dev;
> @@ -1006,6 +1019,10 @@ static int vim2m_probe(struct platform_device *pdev)
>  
>  #ifdef CONFIG_MEDIA_CONTROLLER
>   dev->mdev.dev = >dev;
> + dev->req_queue = media_request_queue_generic_alloc(>mdev);
> + if (IS_ERR(dev->req_queue))
> + return PTR_ERR(dev->req_queue);
> + dev->mdev.req_queue = dev->req_queue;
>   strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
>   media_device_init(>mdev);
>   dev->v4l2_dev.mdev = >mdev;
> @@ -1023,6 +1040,11 @@ static int vim2m_probe(struct platform_device *pdev)
>   vfd->lock = >dev_mutex;
>   vfd->v4l2_dev = >v4l2_dev;
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> + vfd->entity.ops = _entity_ops;
> + vfd->entity.req_ops = _entity_request_generic_ops;
> +#endif
> +

It seems to me that we can avoid most of this patch and just setup the
request support automatically when the media device node is registered.
The less change we impose to drivers the better I think.

Gustavo

Re: [RFC PATCH 9/9] media: vim2m: add request support

[Gustavo Padovan]

Hi Alex,

2017-12-15 Alexandre Courbot :

> Set the necessary ops for supporting requests in vim2m.
> 
> Signed-off-by: Alexandre Courbot 
> ---
>  drivers/media/platform/vim2m.c | 22 ++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index a32e8a7950eb..ffe94ef9214d 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -30,6 +30,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>  
>  MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
>  MODULE_AUTHOR("Pawel Osciak, ");
> @@ -142,6 +143,7 @@ struct vim2m_dev {
>   struct video_device vfd;
>  #ifdef CONFIG_MEDIA_CONTROLLER
>   struct media_device mdev;
> + struct media_request_queue *req_queue;
>  #endif
>  
>   atomic_tnum_inst;
> @@ -937,6 +939,11 @@ static int vim2m_open(struct file *file)
>   goto open_unlock;
>   }
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> + v4l2_mem_ctx_request_init(ctx->fh.m2m_ctx, dev->req_queue,
> +   >vfd.entity);
> +#endif
> +
>   v4l2_fh_add(>fh);
>   atomic_inc(>num_inst);
>  
> @@ -992,6 +999,12 @@ static const struct v4l2_m2m_ops m2m_ops = {
>   .job_abort  = job_abort,
>  };
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +static const struct media_entity_operations vim2m_entity_ops = {
> + .process_request = v4l2_m2m_process_request,
> +};
> +#endif
> +
>  static int vim2m_probe(struct platform_device *pdev)
>  {
>   struct vim2m_dev *dev;
> @@ -1006,6 +1019,10 @@ static int vim2m_probe(struct platform_device *pdev)
>  
>  #ifdef CONFIG_MEDIA_CONTROLLER
>   dev->mdev.dev = >dev;
> + dev->req_queue = media_request_queue_generic_alloc(>mdev);
> + if (IS_ERR(dev->req_queue))
> + return PTR_ERR(dev->req_queue);
> + dev->mdev.req_queue = dev->req_queue;
>   strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
>   media_device_init(>mdev);
>   dev->v4l2_dev.mdev = >mdev;
> @@ -1023,6 +1040,11 @@ static int vim2m_probe(struct platform_device *pdev)
>   vfd->lock = >dev_mutex;
>   vfd->v4l2_dev = >v4l2_dev;
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> + vfd->entity.ops = _entity_ops;
> + vfd->entity.req_ops = _entity_request_generic_ops;
> +#endif
> +

It seems to me that we can avoid most of this patch and just setup the
request support automatically when the media device node is registered.
The less change we impose to drivers the better I think.

Gustavo

[PATCH v6 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 include/media/videobuf2-core.h | 5 +
 1 file changed, 5 insertions(+)

diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index ef9b64398c8c..eddb38a2a2f3 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -368,6 +368,9 @@ struct vb2_buffer {
  * callback by calling vb2_buffer_done() with either
  * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  * vb2_wait_for_all_buffers() function
+ * @is_unordered:  tell if the queue format is unordered. The default is
+ * assumed to be ordered and this function only needs to
+ * be implemented for unordered queues.
  * @buf_queue: passes buffer vb to the driver; driver may start
  * hardware operation on this buffer; driver should give
  * the buffer back by calling vb2_buffer_done() function;
@@ -391,6 +394,7 @@ struct vb2_ops {
 
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
void (*stop_streaming)(struct vb2_queue *q);
+   int (*is_unordered)(struct vb2_queue *q);
 
void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -564,6 +568,7 @@ struct vb2_queue {
u32 cnt_wait_finish;
u32 cnt_start_streaming;
u32 cnt_stop_streaming;
+   u32 cnt_is_unordered;
 #endif
 };
 
-- 
2.13.6

[PATCH v6 1/6] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

From: Gustavo Padovan 

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

Signed-off-by: Gustavo Padovan 
---
 include/media/videobuf2-core.h | 5 +
 1 file changed, 5 insertions(+)

diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index ef9b64398c8c..eddb38a2a2f3 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -368,6 +368,9 @@ struct vb2_buffer {
  * callback by calling vb2_buffer_done() with either
  * %VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  * vb2_wait_for_all_buffers() function
+ * @is_unordered:  tell if the queue format is unordered. The default is
+ * assumed to be ordered and this function only needs to
+ * be implemented for unordered queues.
  * @buf_queue: passes buffer vb to the driver; driver may start
  * hardware operation on this buffer; driver should give
  * the buffer back by calling vb2_buffer_done() function;
@@ -391,6 +394,7 @@ struct vb2_ops {
 
int (*start_streaming)(struct vb2_queue *q, unsigned int count);
void (*stop_streaming)(struct vb2_queue *q);
+   int (*is_unordered)(struct vb2_queue *q);
 
void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -564,6 +568,7 @@ struct vb2_queue {
u32 cnt_wait_finish;
u32 cnt_start_streaming;
u32 cnt_stop_streaming;
+   u32 cnt_is_unordered;
 #endif
 };
 
-- 
2.13.6

[PATCH v6 3/6] [media] vb2: add explicit fence user API

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel and return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending a fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v4:
- make it a union with reserved2 and fence_fd (Hans Verkuil)

v3:
- make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/buffer.rst   | 15 +++
 drivers/media/usb/cpia2/cpia2_v4l.c   |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c |  4 ++--
 drivers/media/v4l2-core/videobuf2-v4l2.c  |  2 +-
 include/uapi/linux/videodev2.h|  7 ++-
 5 files changed, 25 insertions(+), 5 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..eeefbd2547e7 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -648,6 +648,21 @@ Buffer Flags
   - Start Of Exposure. The buffer timestamp has been taken when the
exposure of the frame has begun. This is only valid for the
``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+* .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+  - ``V4L2_BUF_FLAG_IN_FENCE``
+  - 0x0020
+  - Ask V4L2 to wait on fence passed in ``fence_fd`` field. The buffer
+   won't be queued to the driver until the fence signals.
+
+* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+  - ``V4L2_BUF_FLAG_OUT_FENCE``
+  - 0x0040
+  - Request a fence to be attached to the buffer. The ``fence_fd``
+   field on
+   :ref:`VIDIOC_QBUF` is used as a return argument to send the out-fence
+   fd to userspace.
 
 
 
diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
b/drivers/media/usb/cpia2/cpia2_v4l.c
index a1c59f19cf2d..7d7459fa2077 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct 
v4l2_buffer *buf)
buf->sequence = cam->buffers[buf->index].seq;
buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
buf->length = cam->frame_size;
-   buf->reserved2 = 0;
+   buf->fence_fd = -1;
buf->reserved = 0;
memset(>timecode, 0, sizeof(buf->timecode));
 
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 821f2aa299ae..3a31d318df2a 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -370,7 +370,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __s32   fence_fd;
__u32   reserved;
 };
 
@@ -533,7 +533,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct 
v4l2_buffer32 __user
put_user(kp->timestamp.tv_usec, >timestamp.tv_usec) ||
copy_to_user(>timecode, >timecode, sizeof(struct 
v4l2_timecode)) ||
put_user(kp->sequence, >sequence) ||
-   put_user(kp->reserved2, >reserved2) ||
+   put_user(kp->fence_fd, >fence_fd) ||
put_user(kp->reserved, >reserved) ||
put_user(kp->length, >length))
return -EFAULT;
diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c 
b/drivers/media/v4l2-core/videobuf2-v4l2.c
index 4075314a6989..4a5244ee2c58 100644
--- a/drivers/media/v4l2-core/videobuf2-v4l2.c
+++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->fence_fd = -1;
b->reserved = 0;
 
if (q->is_multiplanar) {
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index a8ea632c14f0..17e163b5036d 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -926,7 +926,10 @@ struct v4l2_buffer {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   union {
+   __s32   fence_fd;
+   __u32   reserved2;
+   };
__u32   reserved;
 };
 
@@ -963,6 +966,8 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE   0x0001
 /* mem2mem encoder/de

[PATCH v6 3/6] [media] vb2: add explicit fence user API

[Gustavo Padovan]

From: Gustavo Padovan 

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel and return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending a fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v4:
- make it a union with reserved2 and fence_fd (Hans Verkuil)

v3:
- make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/buffer.rst   | 15 +++
 drivers/media/usb/cpia2/cpia2_v4l.c   |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c |  4 ++--
 drivers/media/v4l2-core/videobuf2-v4l2.c  |  2 +-
 include/uapi/linux/videodev2.h|  7 ++-
 5 files changed, 25 insertions(+), 5 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst 
b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..eeefbd2547e7 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -648,6 +648,21 @@ Buffer Flags
   - Start Of Exposure. The buffer timestamp has been taken when the
exposure of the frame has begun. This is only valid for the
``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+* .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+  - ``V4L2_BUF_FLAG_IN_FENCE``
+  - 0x0020
+  - Ask V4L2 to wait on fence passed in ``fence_fd`` field. The buffer
+   won't be queued to the driver until the fence signals.
+
+* .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+  - ``V4L2_BUF_FLAG_OUT_FENCE``
+  - 0x0040
+  - Request a fence to be attached to the buffer. The ``fence_fd``
+   field on
+   :ref:`VIDIOC_QBUF` is used as a return argument to send the out-fence
+   fd to userspace.
 
 
 
diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c 
b/drivers/media/usb/cpia2/cpia2_v4l.c
index a1c59f19cf2d..7d7459fa2077 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct 
v4l2_buffer *buf)
buf->sequence = cam->buffers[buf->index].seq;
buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
buf->length = cam->frame_size;
-   buf->reserved2 = 0;
+   buf->fence_fd = -1;
buf->reserved = 0;
memset(>timecode, 0, sizeof(buf->timecode));
 
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c 
b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 821f2aa299ae..3a31d318df2a 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -370,7 +370,7 @@ struct v4l2_buffer32 {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   __s32   fence_fd;
__u32   reserved;
 };
 
@@ -533,7 +533,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct 
v4l2_buffer32 __user
put_user(kp->timestamp.tv_usec, >timestamp.tv_usec) ||
copy_to_user(>timecode, >timecode, sizeof(struct 
v4l2_timecode)) ||
put_user(kp->sequence, >sequence) ||
-   put_user(kp->reserved2, >reserved2) ||
+   put_user(kp->fence_fd, >fence_fd) ||
put_user(kp->reserved, >reserved) ||
put_user(kp->length, >length))
return -EFAULT;
diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c 
b/drivers/media/v4l2-core/videobuf2-v4l2.c
index 4075314a6989..4a5244ee2c58 100644
--- a/drivers/media/v4l2-core/videobuf2-v4l2.c
+++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void 
*pb)
b->timestamp = ns_to_timeval(vb->timestamp);
b->timecode = vbuf->timecode;
b->sequence = vbuf->sequence;
-   b->reserved2 = 0;
+   b->fence_fd = -1;
b->reserved = 0;
 
if (q->is_multiplanar) {
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index a8ea632c14f0..17e163b5036d 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -926,7 +926,10 @@ struct v4l2_buffer {
__s32   fd;
} m;
__u32   length;
-   __u32   reserved2;
+   union {
+   __s32   fence_fd;
+   __u32   reserved2;
+   };
__u32   reserved;
 };
 
@@ -963,6 +966,8 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE   0x0001
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST 

[PATCH v6 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace on the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v7:
- merge patch that add the infrastructure to out-fences into
this one (Alex Courbot)
- Do not install the fd if there is no fence. (Alex Courbot)
- do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6
- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
ordering in vb2 for queueing in the driver, so the event is not
necessary anymore and the out_fence_fd is sent back to userspace
on QBUF call return arg
- do not allow requeueing with out-fences, instead mark the buffer
with an error and wake up to userspace.
- send the out_fence_fd back to userspace on the fence_fd field

v5:
- delay fd_install to DQ_EVENT (Hans)
- if queue is fully ordered send OUT_FENCE event right away
(Brian)
- rename 'q->ordered' to 'q->ordered_in_driver'
- merge change to implement OUT_FENCE event here

v4:
- return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
- set the OUT_FENCE flag if there is a fence pending (Hans)
- call fd_install() after vb2_core_qbuf() (Hans)
- clean up fence if vb2_core_qbuf() fails (Hans)
- add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/v4l2-core/videobuf2-core.c | 99 +---
 drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +-
 include/media/videobuf2-core.h   | 22 +++
 3 files changed, 139 insertions(+), 11 deletions(-)

diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
b/drivers/media/v4l2-core/videobuf2-core.c
index 520aa3c7d9f0..cbe115f00736 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -23,6 +23,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #include 
@@ -351,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
}
+   vb->out_fence_fd = -1;
q->bufs[vb->index] = vb;
 
/* Allocate video buffer memory for the MMAP type */
@@ -931,10 +933,20 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
case VB2_BUF_STATE_QUEUED:
break;
case VB2_BUF_STATE_REQUEUEING:
+   /* Requeuing with explicit synchronization, spit warning */
+   WARN_ON_ONCE(vb->out_fence);
+
if (q->start_streaming_called)
__enqueue_in_driver(vb);
-   return;
+   break;
default:
+   if (state == VB2_BUF_STATE_ERROR)
+   dma_fence_set_error(vb->out_fence, -EFAULT);
+   dma_fence_signal(vb->out_fence);
+   dma_fence_put(vb->out_fence);
+   vb->out_fence = NULL;
+   vb->out_fence_fd = -1;
+
/* Inform any processes that may be waiting for buffers */
wake_up(>done_wq);
break;
@@ -1330,6 +1342,65 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+   return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+   return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+   return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+   .get_driver_name = vb2_fence_get_driver_name,
+   .get_timeline_name = vb2_fence_get_timeline_name,
+   .enable_signaling = vb2_fence_enable_signaling,
+   .wait = dma_fence_default_wait,
+};
+
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
+{
+   struct vb2_buffer *vb;
+   u64 context;
+
+   vb = q->bufs[index];
+
+   vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
+
+   if (call_qop(q, is_unordered, q))
+   context = dma_fence_context_alloc(1);
+   else
+   context = q->out_fence_context;
+
+   vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
+   if (!vb->out_fence)
+   return -ENOMEM;
+
+   dma_fence_in

[PATCH v6 5/6] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan 

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace on the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v7:
- merge patch that add the infrastructure to out-fences into
this one (Alex Courbot)
- Do not install the fd if there is no fence. (Alex Courbot)
- do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6
- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
ordering in vb2 for queueing in the driver, so the event is not
necessary anymore and the out_fence_fd is sent back to userspace
on QBUF call return arg
- do not allow requeueing with out-fences, instead mark the buffer
with an error and wake up to userspace.
- send the out_fence_fd back to userspace on the fence_fd field

v5:
- delay fd_install to DQ_EVENT (Hans)
- if queue is fully ordered send OUT_FENCE event right away
(Brian)
- rename 'q->ordered' to 'q->ordered_in_driver'
- merge change to implement OUT_FENCE event here

v4:
- return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
- set the OUT_FENCE flag if there is a fence pending (Hans)
- call fd_install() after vb2_core_qbuf() (Hans)
- clean up fence if vb2_core_qbuf() fails (Hans)
- add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan 
---
 drivers/media/v4l2-core/videobuf2-core.c | 99 +---
 drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +-
 include/media/videobuf2-core.h   | 22 +++
 3 files changed, 139 insertions(+), 11 deletions(-)

diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
b/drivers/media/v4l2-core/videobuf2-core.c
index 520aa3c7d9f0..cbe115f00736 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -23,6 +23,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include 
 #include 
@@ -351,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
}
+   vb->out_fence_fd = -1;
q->bufs[vb->index] = vb;
 
/* Allocate video buffer memory for the MMAP type */
@@ -931,10 +933,20 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
case VB2_BUF_STATE_QUEUED:
break;
case VB2_BUF_STATE_REQUEUEING:
+   /* Requeuing with explicit synchronization, spit warning */
+   WARN_ON_ONCE(vb->out_fence);
+
if (q->start_streaming_called)
__enqueue_in_driver(vb);
-   return;
+   break;
default:
+   if (state == VB2_BUF_STATE_ERROR)
+   dma_fence_set_error(vb->out_fence, -EFAULT);
+   dma_fence_signal(vb->out_fence);
+   dma_fence_put(vb->out_fence);
+   vb->out_fence = NULL;
+   vb->out_fence_fd = -1;
+
/* Inform any processes that may be waiting for buffers */
wake_up(>done_wq);
break;
@@ -1330,6 +1342,65 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned 
int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+   return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+   return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+   return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+   .get_driver_name = vb2_fence_get_driver_name,
+   .get_timeline_name = vb2_fence_get_timeline_name,
+   .enable_signaling = vb2_fence_enable_signaling,
+   .wait = dma_fence_default_wait,
+};
+
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
+{
+   struct vb2_buffer *vb;
+   u64 context;
+
+   vb = q->bufs[index];
+
+   vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
+
+   if (call_qop(q, is_unordered, q))
+   context = dma_fence_context_alloc(1);
+   else
+   context = q->out_fence_context;
+
+   vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
+   if (!vb->out_fence)
+   return -ENOMEM;
+
+   dma_fence_init(vb->out_fence, _fence_ops, >out_fence_lock,
+   

[PATCH v6 6/6] [media] v4l: Document explicit synchronization behavior

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Add section to VIDIOC_QBUF about it

v5:
- Remove V4L2_CAP_ORDERED
- Add doc about V4L2_FMT_FLAG_UNORDERED

v4:
- Document ordering behavior for in-fences
- Document V4L2_CAP_ORDERED capability
- Remove doc about OUT_FENCE event
- Document immediate return of out-fence in QBUF

v3:
- make the out_fence refer to the current buffer (Hans)
- Note what happens when the IN_FENCE is not set (Hans)

v2:
- mention that fences are files (Hans)
- rework for the new API

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst | 47 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |  9 -
 2 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst 
b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..8809397fb110 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API `, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,51 @@ immediately with an ``EAGAIN`` error code when no buffer 
is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer, i.e., 
queueing
+it to the driver. On the other side, the kernel can create out-fences for the
+buffers it queues to the drivers. Out-fences signal when the driver is
+finished with buffer, i.e., the buffer is ready. The fences are represented
+as a file and passed as a file descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with error. The fence_fd field will be ignored if the
+``V4L2_BUF_FLAG_IN_FENCE`` is not set.
+
+The videobuf2-core will guarantee that all buffers queued with in-fence will
+be queued to the drivers in the same order. Fence may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering.
+
+If the in-fence signals with an error the videobuf2 won't queue the buffer to
+the driver, instead it will flag it with an error. And then wait for the
+previous buffer to complete before asking userspace dequeue the buffer with
+error - to make sure we deliver the buffers back in the correct order.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.
+
+Note the the same `fence_fd` field is used for both sending the in-fence as
+input argument to receive the out-fence as a return argument.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. It will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the current buffer queues is able to keep the ordering of buffers, i.e.,
+the dequeing of buffers will happen at the same order we queue them, with no
+reordering by the driver.
 
 Return Value
 
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst 
b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
index dd54747fabc9..df964c4d916b 100644
--- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
@@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero 
to the
 number of buffers allocated with
 :ref:`VIDIOC_REQBUFS` (struct
 :c:type:`v4l2_requestbuffers` ``count``) minus
-one. The ``reserved`` and ``reserved2``

[PATCH v6 6/6] [media] v4l: Document explicit synchronization behavior

[Gustavo Padovan]

From: Gustavo Padovan 

Add section to VIDIOC_QBUF about it

v5:
- Remove V4L2_CAP_ORDERED
- Add doc about V4L2_FMT_FLAG_UNORDERED

v4:
- Document ordering behavior for in-fences
- Document V4L2_CAP_ORDERED capability
- Remove doc about OUT_FENCE event
- Document immediate return of out-fence in QBUF

v3:
- make the out_fence refer to the current buffer (Hans)
- Note what happens when the IN_FENCE is not set (Hans)

v2:
- mention that fences are files (Hans)
- rework for the new API

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst | 47 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |  9 -
 2 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst 
b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..8809397fb110 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API `, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,51 @@ immediately with an ``EAGAIN`` error code when no buffer 
is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer, i.e., 
queueing
+it to the driver. On the other side, the kernel can create out-fences for the
+buffers it queues to the drivers. Out-fences signal when the driver is
+finished with buffer, i.e., the buffer is ready. The fences are represented
+as a file and passed as a file descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with error. The fence_fd field will be ignored if the
+``V4L2_BUF_FLAG_IN_FENCE`` is not set.
+
+The videobuf2-core will guarantee that all buffers queued with in-fence will
+be queued to the drivers in the same order. Fence may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering.
+
+If the in-fence signals with an error the videobuf2 won't queue the buffer to
+the driver, instead it will flag it with an error. And then wait for the
+previous buffer to complete before asking userspace dequeue the buffer with
+error - to make sure we deliver the buffers back in the correct order.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.
+
+Note the the same `fence_fd` field is used for both sending the in-fence as
+input argument to receive the out-fence as a return argument.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. It will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the current buffer queues is able to keep the ordering of buffers, i.e.,
+the dequeing of buffers will happen at the same order we queue them, with no
+reordering by the driver.
 
 Return Value
 
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst 
b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
index dd54747fabc9..df964c4d916b 100644
--- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
@@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero 
to the
 number of buffers allocated with
 :ref:`VIDIOC_REQBUFS` (struct
 :c:type:`v4l2_requestbuffers` ``count``) minus
-one. The ``reserved`` and ``reserved2`` fields must be set to 0. When
+one. The ``reserved`` field must be set to 0

[PATCH v6 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queue to the driver
out of the order they were queued from userspace. That means that even if
it fence signal it must wait all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v7:
- get rid of the fence array stuff for ordering and just use
get_num_buffers_ready() (Hans)
- fix issue of queuing the buffer twice (Hans)
- avoid the dma_fence_wait() in core_qbuf() (Alex)
- merge preparation commit in

v6:
- With fences always keep the order userspace queues the buffers.
- Protect in_fence manipulation with a lock (Brian Starkey)
- check if fences have the same context before adding a fence array
- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
- Clean up fence if __set_in_fence() fails (Brian Starkey)
- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
needed (Brian Starkey)
- keep backward compat on the reserved2 field (Brian Starkey)
- protect fence callback removal with lock (Brian Starkey)

v4:
- Add a comment about dma_fence_add_callback() not returning a
error (Hans)
- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
vb2_start_streaming() (Hans)
- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
- Queue buffers to the driver as soon as they are ready (Hans)
- call fill_user_buffer() after queuing the buffer (Hans)
- add err: label to clean up fence
- add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
- remove ternary if at vb2_qbuf() return (Mauro)
- do not change if conditions behaviour (Mauro)

v2:
- fix vb2_queue_or_prepare_buf() ret check
- remove check for VB2_MEMORY_DMABUF only (Javier)
- check num of ready buffers to start streaming
- when queueing, start from the first ready buffer
- handle queue cancel

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 drivers/media/v4l2-core/Kconfig  |   1 +
 drivers/media/v4l2-core/videobuf2-core.c | 154 +++
 drivers/media/v4l2-core/videobuf2-v4l2.c |  29 +-
 include/media/videobuf2-core.h   |  14 ++-
 4 files changed, 177 insertions(+), 21 deletions(-)

diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
index a35c33686abf..3f988c407c80 100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -83,6 +83,7 @@ config VIDEOBUF_DVB
 # Used by drivers that need Videobuf2 modules
 config VIDEOBUF2_CORE
select DMA_SHARED_BUFFER
+   select SYNC_FILE
tristate
 
 config VIDEOBUF2_MEMOPS
diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
b/drivers/media/v4l2-core/videobuf2-core.c
index a8589d96ef72..520aa3c7d9f0 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -346,6 +346,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->index = q->num_buffers + buffer;
vb->type = q->type;
vb->memory = memory;
+   spin_lock_init(>fence_cb_lock);
for (plane = 0; plane < num_planes; ++plane) {
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
@@ -928,7 +929,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
 
switch (state) {
case VB2_BUF_STATE_QUEUED:
-   return;
+   break;
case VB2_BUF_STATE_REQUEUEING:
if (q->start_streaming_called)
__enqueue_in_driver(vb);
@@ -938,6 +939,16 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
wake_up(>done_wq);
break;
}
+
+   /*
+* If the in-fence fails the buffer is not queued to the driver
+* and we have to wait until the previous buffer is done before
+*

[PATCH v6 4/6] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan 

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queue to the driver
out of the order they were queued from userspace. That means that even if
it fence signal it must wait all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v7:
- get rid of the fence array stuff for ordering and just use
get_num_buffers_ready() (Hans)
- fix issue of queuing the buffer twice (Hans)
- avoid the dma_fence_wait() in core_qbuf() (Alex)
- merge preparation commit in

v6:
- With fences always keep the order userspace queues the buffers.
- Protect in_fence manipulation with a lock (Brian Starkey)
- check if fences have the same context before adding a fence array
- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
- Clean up fence if __set_in_fence() fails (Brian Starkey)
- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
needed (Brian Starkey)
- keep backward compat on the reserved2 field (Brian Starkey)
- protect fence callback removal with lock (Brian Starkey)

v4:
- Add a comment about dma_fence_add_callback() not returning a
error (Hans)
- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
vb2_start_streaming() (Hans)
- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
- Queue buffers to the driver as soon as they are ready (Hans)
- call fill_user_buffer() after queuing the buffer (Hans)
- add err: label to clean up fence
- add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
- remove ternary if at vb2_qbuf() return (Mauro)
- do not change if conditions behaviour (Mauro)

v2:
- fix vb2_queue_or_prepare_buf() ret check
- remove check for VB2_MEMORY_DMABUF only (Javier)
- check num of ready buffers to start streaming
- when queueing, start from the first ready buffer
- handle queue cancel

Signed-off-by: Gustavo Padovan 
---
 drivers/media/v4l2-core/Kconfig  |   1 +
 drivers/media/v4l2-core/videobuf2-core.c | 154 +++
 drivers/media/v4l2-core/videobuf2-v4l2.c |  29 +-
 include/media/videobuf2-core.h   |  14 ++-
 4 files changed, 177 insertions(+), 21 deletions(-)

diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
index a35c33686abf..3f988c407c80 100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -83,6 +83,7 @@ config VIDEOBUF_DVB
 # Used by drivers that need Videobuf2 modules
 config VIDEOBUF2_CORE
select DMA_SHARED_BUFFER
+   select SYNC_FILE
tristate
 
 config VIDEOBUF2_MEMOPS
diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
b/drivers/media/v4l2-core/videobuf2-core.c
index a8589d96ef72..520aa3c7d9f0 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -346,6 +346,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
vb2_memory memory,
vb->index = q->num_buffers + buffer;
vb->type = q->type;
vb->memory = memory;
+   spin_lock_init(>fence_cb_lock);
for (plane = 0; plane < num_planes; ++plane) {
vb->planes[plane].length = plane_sizes[plane];
vb->planes[plane].min_length = plane_sizes[plane];
@@ -928,7 +929,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
 
switch (state) {
case VB2_BUF_STATE_QUEUED:
-   return;
+   break;
case VB2_BUF_STATE_REQUEUEING:
if (q->start_streaming_called)
__enqueue_in_driver(vb);
@@ -938,6 +939,16 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum 
vb2_buffer_state state)
wake_up(>done_wq);
break;
}
+
+   /*
+* If the in-fence fails the buffer is not queued to the driver
+* and we have to wait until the previous buffer is done before
+* we notify userspace that the buffer with error can be dequeued.
+*

[PATCH v6 2/6] [media] v4l: add 'unordered' flag to format description ioctl

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 3 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst 
b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..368115f44fc0 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,9 @@ one until ``EINVAL`` is returned.
   - This format is not native to the device but emulated through
software (usually libv4l2), where possible try to use a native
format instead for better performance.
+* - ``V4L2_FMT_FLAG_UNORDERED``
+  - 0x0004
+  - This is a format that doesn't guarantee timely order of frames.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 1c095b5a99c5..a8ea632c14f0 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -709,6 +709,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
/* Frame Size and frame rate enumeration */
 /*
-- 
2.13.6

[PATCH v6 2/6] [media] v4l: add 'unordered' flag to format description ioctl

[Gustavo Padovan]

From: Gustavo Padovan 

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

Signed-off-by: Gustavo Padovan 
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 3 +++
 include/uapi/linux/videodev2.h   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst 
b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..368115f44fc0 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,9 @@ one until ``EINVAL`` is returned.
   - This format is not native to the device but emulated through
software (usually libv4l2), where possible try to use a native
format instead for better performance.
+* - ``V4L2_FMT_FLAG_UNORDERED``
+  - 0x0004
+  - This is a format that doesn't guarantee timely order of frames.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 1c095b5a99c5..a8ea632c14f0 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -709,6 +709,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
/* Frame Size and frame rate enumeration */
 /*
-- 
2.13.6

[PATCH v6 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.pado...@collabora.com>

Hi,

One more iteration of the explicit fences patches, please refer
to the previous version[1] for more details about the general
mechanism

This version makes the patchset and the implementation much more
simple, to start we are not using a ordered capability anymore,
but instead we have a VIDIOC_ENUM_FMT flag to tell when the
queue in not ordered. Drivers with ordered queues/formats don't
need implement anything. See patches 1 and 2 for more details.

The implementation of in-fences and out-fences were condensed in
just patches 4 and 5, making it more self-contained and easy to
understand. See the patches for detailed changelog.

Please review! Thanks.

Gustavo.

[1] https://lkml.org/lkml/2017/11/15/550

Gustavo Padovan (6):
  [media] vb2: add is_unordered callback for drivers
  [media] v4l: add 'unordered' flag to format description ioctl
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add out-fence support to QBUF
  [media] v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst  |  15 ++
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst |   3 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst |  47 -
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |   9 +-
 drivers/media/usb/cpia2/cpia2_v4l.c  |   2 +-
 drivers/media/v4l2-core/Kconfig  |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c|   4 +-
 drivers/media/v4l2-core/videobuf2-core.c | 239 +--
 drivers/media/v4l2-core/videobuf2-v4l2.c |  52 -
 include/media/videobuf2-core.h   |  41 +++-
 include/uapi/linux/videodev2.h   |   8 +-
 11 files changed, 393 insertions(+), 28 deletions(-)

-- 
2.13.6

[PATCH v6 0/6] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan 

Hi,

One more iteration of the explicit fences patches, please refer
to the previous version[1] for more details about the general
mechanism

This version makes the patchset and the implementation much more
simple, to start we are not using a ordered capability anymore,
but instead we have a VIDIOC_ENUM_FMT flag to tell when the
queue in not ordered. Drivers with ordered queues/formats don't
need implement anything. See patches 1 and 2 for more details.

The implementation of in-fences and out-fences were condensed in
just patches 4 and 5, making it more self-contained and easy to
understand. See the patches for detailed changelog.

Please review! Thanks.

Gustavo.

[1] https://lkml.org/lkml/2017/11/15/550

Gustavo Padovan (6):
  [media] vb2: add is_unordered callback for drivers
  [media] v4l: add 'unordered' flag to format description ioctl
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add out-fence support to QBUF
  [media] v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst  |  15 ++
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst |   3 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst |  47 -
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |   9 +-
 drivers/media/usb/cpia2/cpia2_v4l.c  |   2 +-
 drivers/media/v4l2-core/Kconfig  |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c|   4 +-
 drivers/media/v4l2-core/videobuf2-core.c | 239 +--
 drivers/media/v4l2-core/videobuf2-v4l2.c |  52 -
 include/media/videobuf2-core.h   |  41 +++-
 include/uapi/linux/videodev2.h   |   8 +-
 11 files changed, 393 insertions(+), 28 deletions(-)

-- 
2.13.6

Re: [RFC v5 00/11] V4L2 Explicit Synchronization

[Gustavo Padovan]

Hi Smitha,

2017-11-20 Smitha T Murthy :

> Hi Gustavo,
> 
> I am currently referring to your implementation for explicit
> synchronisation. For the same I needed your testapp, but I am unable
> to download the same at the link provided
> “https://gitlab.collabora.com/padovan/v4l2-fences-test”
> 
> Could you please help me out with the same.

Sorry about that, our gitlab instance creates the repos as internal by
default. I just changed it to Public.

Gustavo

Re: [RFC v5 00/11] V4L2 Explicit Synchronization

[Gustavo Padovan]

Hi Smitha,

2017-11-20 Smitha T Murthy :

> Hi Gustavo,
> 
> I am currently referring to your implementation for explicit
> synchronisation. For the same I needed your testapp, but I am unable
> to download the same at the link provided
> “https://gitlab.collabora.com/padovan/v4l2-fences-test”
> 
> Could you please help me out with the same.

Sorry about that, our gitlab instance creates the repos as internal by
default. I just changed it to Public.

Gustavo

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Gustavo Padovan <gust...@padovan.org>:

> 2017-11-17 Hans Verkuil <hverk...@xs4all.nl>:
> 
> > On 15/11/17 18:10, Gustavo Padovan wrote:
> > > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > > 
> > > Receive in-fence from userspace and add support for waiting on them
> > > before queueing the buffer to the driver. Buffers can't be queued to the
> > > driver before its fences signal. And a buffer can't be queue to the driver
> > > out of the order they were queued from userspace. That means that even if
> > > it fence signal it must wait all other buffers, ahead of it in the queue,
> > > to signal first.
> > > 
> > > To make that possible we use fence_array to keep that ordering. Basically
> > > we create a fence_array that contains both the current fence and the fence
> > > from the previous buffer (which might be a fence array as well). The base
> > > fence class for the fence_array becomes the new buffer fence, waiting on
> > > that one guarantees that it won't be queued out of order.
> > > 
> > > v6:
> > >   - With fences always keep the order userspace queues the buffers.
> > >   - Protect in_fence manipulation with a lock (Brian Starkey)
> > >   - check if fences have the same context before adding a fence array
> > >   - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > >   - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > >   - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > > 
> > > v5:   - use fence_array to keep buffers ordered in vb2 core when
> > >   needed (Brian Starkey)
> > >   - keep backward compat on the reserved2 field (Brian Starkey)
> > >   - protect fence callback removal with lock (Brian Starkey)
> > > 
> > > v4:
> > >   - Add a comment about dma_fence_add_callback() not returning a
> > >   error (Hans)
> > >   - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > >   - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > >   - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > >   - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > >   -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > >   vb2_start_streaming() (Hans)
> > >   - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > >   - Queue buffers to the driver as soon as they are ready (Hans)
> > >   - call fill_user_buffer() after queuing the buffer (Hans)
> > >   - add err: label to clean up fence
> > >   - add dma_fence_wait() before calling vb2_start_streaming()
> > > 
> > > v3:   - document fence parameter
> > >   - remove ternary if at vb2_qbuf() return (Mauro)
> > >   - do not change if conditions behaviour (Mauro)
> > > 
> > > v2:
> > >   - fix vb2_queue_or_prepare_buf() ret check
> > >   - remove check for VB2_MEMORY_DMABUF only (Javier)
> > >   - check num of ready buffers to start streaming
> > >   - when queueing, start from the first ready buffer
> > >   - handle queue cancel
> > > 
> > > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > > ---
> > >  drivers/media/v4l2-core/Kconfig  |   1 +
> > >  drivers/media/v4l2-core/videobuf2-core.c | 202 
> > > ---
> > >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 -
> > >  include/media/videobuf2-core.h   |  17 ++-
> > >  4 files changed, 231 insertions(+), 18 deletions(-)
> > > 
> > > diff --git a/drivers/media/v4l2-core/Kconfig 
> > > b/drivers/media/v4l2-core/Kconfig
> > > index a35c33686abf..3f988c407c80 100644
> > > --- a/drivers/media/v4l2-core/Kconfig
> > > +++ b/drivers/media/v4l2-core/Kconfig
> > > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> > >  # Used by drivers that need Videobuf2 modules
> > >  config VIDEOBUF2_CORE
> > >   select DMA_SHARED_BUFFER
> > > + select SYNC_FILE
> > >   tristate
> > >  
> > >  config VIDEOBUF2_MEMOPS
> > > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > > b/drivers/media/v4l2-core/videobuf2-core.c
> > > index 60f8b582396a..26de4c80717d 100644
> > > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > > @@ -23,6 +23,7 @@
> > >  #include 
> > >  #include 
> > >  #include 
> >

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Gustavo Padovan :

> 2017-11-17 Hans Verkuil :
> 
> > On 15/11/17 18:10, Gustavo Padovan wrote:
> > > From: Gustavo Padovan 
> > > 
> > > Receive in-fence from userspace and add support for waiting on them
> > > before queueing the buffer to the driver. Buffers can't be queued to the
> > > driver before its fences signal. And a buffer can't be queue to the driver
> > > out of the order they were queued from userspace. That means that even if
> > > it fence signal it must wait all other buffers, ahead of it in the queue,
> > > to signal first.
> > > 
> > > To make that possible we use fence_array to keep that ordering. Basically
> > > we create a fence_array that contains both the current fence and the fence
> > > from the previous buffer (which might be a fence array as well). The base
> > > fence class for the fence_array becomes the new buffer fence, waiting on
> > > that one guarantees that it won't be queued out of order.
> > > 
> > > v6:
> > >   - With fences always keep the order userspace queues the buffers.
> > >   - Protect in_fence manipulation with a lock (Brian Starkey)
> > >   - check if fences have the same context before adding a fence array
> > >   - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > >   - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > >   - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > > 
> > > v5:   - use fence_array to keep buffers ordered in vb2 core when
> > >   needed (Brian Starkey)
> > >   - keep backward compat on the reserved2 field (Brian Starkey)
> > >   - protect fence callback removal with lock (Brian Starkey)
> > > 
> > > v4:
> > >   - Add a comment about dma_fence_add_callback() not returning a
> > >   error (Hans)
> > >   - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > >   - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > >   - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > >   - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > >   -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > >   vb2_start_streaming() (Hans)
> > >   - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > >   - Queue buffers to the driver as soon as they are ready (Hans)
> > >   - call fill_user_buffer() after queuing the buffer (Hans)
> > >   - add err: label to clean up fence
> > >   - add dma_fence_wait() before calling vb2_start_streaming()
> > > 
> > > v3:   - document fence parameter
> > >   - remove ternary if at vb2_qbuf() return (Mauro)
> > >   - do not change if conditions behaviour (Mauro)
> > > 
> > > v2:
> > >   - fix vb2_queue_or_prepare_buf() ret check
> > >   - remove check for VB2_MEMORY_DMABUF only (Javier)
> > >   - check num of ready buffers to start streaming
> > >   - when queueing, start from the first ready buffer
> > >   - handle queue cancel
> > > 
> > > Signed-off-by: Gustavo Padovan 
> > > ---
> > >  drivers/media/v4l2-core/Kconfig  |   1 +
> > >  drivers/media/v4l2-core/videobuf2-core.c | 202 
> > > ---
> > >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 -
> > >  include/media/videobuf2-core.h   |  17 ++-
> > >  4 files changed, 231 insertions(+), 18 deletions(-)
> > > 
> > > diff --git a/drivers/media/v4l2-core/Kconfig 
> > > b/drivers/media/v4l2-core/Kconfig
> > > index a35c33686abf..3f988c407c80 100644
> > > --- a/drivers/media/v4l2-core/Kconfig
> > > +++ b/drivers/media/v4l2-core/Kconfig
> > > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> > >  # Used by drivers that need Videobuf2 modules
> > >  config VIDEOBUF2_CORE
> > >   select DMA_SHARED_BUFFER
> > > + select SYNC_FILE
> > >   tristate
> > >  
> > >  config VIDEOBUF2_MEMOPS
> > > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > > b/drivers/media/v4l2-core/videobuf2-core.c
> > > index 60f8b582396a..26de4c80717d 100644
> > > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > > @@ -23,6 +23,7 @@
> > >  #include 
> > >  #include 
> > >  #include 
> > > +#include 
> > >  
> > >  #include 
> > >  #include 
> > > @@ -346,6 +347,7 @@ static int __vb2

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Hans Verkuil <hverk...@xs4all.nl>:

> On 15/11/17 18:10, Gustavo Padovan wrote:
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > To make that possible we use fence_array to keep that ordering. Basically
> > we create a fence_array that contains both the current fence and the fence
> > from the previous buffer (which might be a fence array as well). The base
> > fence class for the fence_array becomes the new buffer fence, waiting on
> > that one guarantees that it won't be queued out of order.
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  drivers/media/v4l2-core/Kconfig  |   1 +
> >  drivers/media/v4l2-core/videobuf2-core.c | 202 
> > ---
> >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 -
> >  include/media/videobuf2-core.h   |  17 ++-
> >  4 files changed, 231 insertions(+), 18 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/Kconfig 
> > b/drivers/media/v4l2-core/Kconfig
> > index a35c33686abf..3f988c407c80 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> > select DMA_SHARED_BUFFER
> > +   select SYNC_FILE
> > tristate
> >  
> >  config VIDEOBUF2_MEMOPS
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index 60f8b582396a..26de4c80717d 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -23,6 +23,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  
> >  #include 
> >  #include 
> > @@ -346,6 +347,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   spin_lock_init(>fence_cb_lock);
> > for (plane = 0; plane < num_planes;

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Hans Verkuil :

> On 15/11/17 18:10, Gustavo Padovan wrote:
> > From: Gustavo Padovan 
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > To make that possible we use fence_array to keep that ordering. Basically
> > we create a fence_array that contains both the current fence and the fence
> > from the previous buffer (which might be a fence array as well). The base
> > fence class for the fence_array becomes the new buffer fence, waiting on
> > that one guarantees that it won't be queued out of order.
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan 
> > ---
> >  drivers/media/v4l2-core/Kconfig  |   1 +
> >  drivers/media/v4l2-core/videobuf2-core.c | 202 
> > ---
> >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 -
> >  include/media/videobuf2-core.h   |  17 ++-
> >  4 files changed, 231 insertions(+), 18 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/Kconfig 
> > b/drivers/media/v4l2-core/Kconfig
> > index a35c33686abf..3f988c407c80 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> > select DMA_SHARED_BUFFER
> > +   select SYNC_FILE
> > tristate
> >  
> >  config VIDEOBUF2_MEMOPS
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index 60f8b582396a..26de4c80717d 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -23,6 +23,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  
> >  #include 
> >  #include 
> > @@ -346,6 +347,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   spin_lock_init(>fence_cb_lock);
> > for (plane = 0; plane < num_planes; ++plane) {
> > vb->planes[plane].length = plane_sizes[plane];
> >

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Mauro Carvalho Chehab <mche...@osg.samsung.com>:

> Em Fri, 17 Nov 2017 11:12:48 -0200
> Gustavo Padovan <gust...@padovan.org> escreveu:
> 
> > > > /*
> > > >  * If streamon has been called, and we haven't yet called
> > > >  * start_streaming() since not enough buffers were queued, and
> > > >  * we now have reached the minimum number of queued buffers,
> > > >  * then we can finally call start_streaming().
> > > > -*
> > > > -* If already streaming, give the buffer to driver for 
> > > > processing.
> > > > -* If not, the buffer will be given to driver on next streamon.
> > > >  */
> > > > if (q->streaming && !q->start_streaming_called &&
> > > > -   q->queued_count >= q->min_buffers_needed) {
> > > > +   __get_num_ready_buffers(q) >= q->min_buffers_needed) {  
> > > 
> > > I guess the case where fences is not used is not covered here.
> > > 
> > > You probably should add a check at __get_num_ready_buffers(q)
> > > as well, making it just return q->queued_count if fences isn't
> > > used.  
> > 
> > We can't know that beforehand, some buffer ahead may have a fence,
> > but there is already a check for !fence for each buffer. If none of
> > them have fences the return will be equal to q->queued_count.
> 
> Hmm... are we willing to support the case where just some
> buffers have fences? Why?

It may be that some fences are already signaled before the QBUF call
happens, so the app may just pass -1 instead.

> In any case, we should add a notice at the documentation telling
> about what happens if not all buffers have fences, and if fences
> are required to be setup before starting streaming, or if it is
> possible to dynamically change the fances behavior while streaming.

We don't have such thing. The fence behavior is tied to each QBUF call,
the stream can be setup without knowing anything about if fences are
going to be used or not. I think we need a good reason to do otherwise.
Yet, I can add something to the docs saying that fences are exclusively
per QBUF call.

Gustavo

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Mauro Carvalho Chehab :

> Em Fri, 17 Nov 2017 11:12:48 -0200
> Gustavo Padovan  escreveu:
> 
> > > > /*
> > > >  * If streamon has been called, and we haven't yet called
> > > >  * start_streaming() since not enough buffers were queued, and
> > > >  * we now have reached the minimum number of queued buffers,
> > > >  * then we can finally call start_streaming().
> > > > -*
> > > > -* If already streaming, give the buffer to driver for 
> > > > processing.
> > > > -* If not, the buffer will be given to driver on next streamon.
> > > >  */
> > > > if (q->streaming && !q->start_streaming_called &&
> > > > -   q->queued_count >= q->min_buffers_needed) {
> > > > +   __get_num_ready_buffers(q) >= q->min_buffers_needed) {  
> > > 
> > > I guess the case where fences is not used is not covered here.
> > > 
> > > You probably should add a check at __get_num_ready_buffers(q)
> > > as well, making it just return q->queued_count if fences isn't
> > > used.  
> > 
> > We can't know that beforehand, some buffer ahead may have a fence,
> > but there is already a check for !fence for each buffer. If none of
> > them have fences the return will be equal to q->queued_count.
> 
> Hmm... are we willing to support the case where just some
> buffers have fences? Why?

It may be that some fences are already signaled before the QBUF call
happens, so the app may just pass -1 instead.

> In any case, we should add a notice at the documentation telling
> about what happens if not all buffers have fences, and if fences
> are required to be setup before starting streaming, or if it is
> possible to dynamically change the fances behavior while streaming.

We don't have such thing. The fence behavior is tied to each QBUF call,
the stream can be setup without knowing anything about if fences are
going to be used or not. I think we need a good reason to do otherwise.
Yet, I can add something to the docs saying that fences are exclusively
per QBUF call.

Gustavo

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Mauro Carvalho Chehab <mche...@osg.samsung.com>:

> Em Wed, 15 Nov 2017 15:10:53 -0200
> Gustavo Padovan <gust...@padovan.org> escreveu:
> 
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > To make that possible we use fence_array to keep that ordering. Basically
> > we create a fence_array that contains both the current fence and the fence
> > from the previous buffer (which might be a fence array as well). The base
> > fence class for the fence_array becomes the new buffer fence, waiting on
> > that one guarantees that it won't be queued out of order.
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  drivers/media/v4l2-core/Kconfig  |   1 +
> >  drivers/media/v4l2-core/videobuf2-core.c | 202 
> > ---
> >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 -
> >  include/media/videobuf2-core.h   |  17 ++-
> >  4 files changed, 231 insertions(+), 18 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/Kconfig 
> > b/drivers/media/v4l2-core/Kconfig
> > index a35c33686abf..3f988c407c80 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> > select DMA_SHARED_BUFFER
> > +   select SYNC_FILE
> > tristate
> >  
> >  config VIDEOBUF2_MEMOPS
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index 60f8b582396a..26de4c80717d 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -23,6 +23,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  
> >  #include 
> >  #include 
> > @@ -346,6 +347,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   spin_lock_init(>fence_c

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Mauro Carvalho Chehab :

> Em Wed, 15 Nov 2017 15:10:53 -0200
> Gustavo Padovan  escreveu:
> 
> > From: Gustavo Padovan 
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > To make that possible we use fence_array to keep that ordering. Basically
> > we create a fence_array that contains both the current fence and the fence
> > from the previous buffer (which might be a fence array as well). The base
> > fence class for the fence_array becomes the new buffer fence, waiting on
> > that one guarantees that it won't be queued out of order.
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan 
> > ---
> >  drivers/media/v4l2-core/Kconfig  |   1 +
> >  drivers/media/v4l2-core/videobuf2-core.c | 202 
> > ---
> >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 -
> >  include/media/videobuf2-core.h   |  17 ++-
> >  4 files changed, 231 insertions(+), 18 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/Kconfig 
> > b/drivers/media/v4l2-core/Kconfig
> > index a35c33686abf..3f988c407c80 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> > select DMA_SHARED_BUFFER
> > +   select SYNC_FILE
> > tristate
> >  
> >  config VIDEOBUF2_MEMOPS
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index 60f8b582396a..26de4c80717d 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -23,6 +23,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  
> >  #include 
> >  #include 
> > @@ -346,6 +347,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum 
> > vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> > vb->type = q->type;
> > vb->memory = memory;
> > +   spin_lock_init(>fence_cb_lock);
> > for (plane = 0; plane < num_planes; ++plane) {
> > vb->planes[

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Mauro Carvalho Chehab :

> Em Fri, 17 Nov 2017 15:49:23 +0900
> Alexandre Courbot  escreveu:
> 
> > > @@ -178,6 +179,12 @@ static int vb2_queue_or_prepare_buf(struct 
> > > vb2_queue *q, struct v4l2_buffer *b,
> > >   return -EINVAL;
> > >   }
> > >  
> > > + if ((b->fence_fd != 0 && b->fence_fd != -1) &&  
> > 
> > Why do we need to consider both values invalid? Can 0 ever be a valid fence 
> > fd?
> 
> Programs that don't use fences will initialize reserved2/fence_fd field
> at the uAPI call to zero.
> 
> So, I guess using fd=0 here could be a problem. Anyway, I would, instead,
> do:
> 
>   if ((b->fence_fd < 1) &&
>   ...
> 
> as other negative values are likely invalid as well.

We are checking when the fence_fd is set but the flag wasn't. Checking
for < 1 is exactly the opposite. so we keep as is or do it fence_fd > 0.

Gustavo

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Mauro Carvalho Chehab :

> Em Fri, 17 Nov 2017 15:49:23 +0900
> Alexandre Courbot  escreveu:
> 
> > > @@ -178,6 +179,12 @@ static int vb2_queue_or_prepare_buf(struct 
> > > vb2_queue *q, struct v4l2_buffer *b,
> > >   return -EINVAL;
> > >   }
> > >  
> > > + if ((b->fence_fd != 0 && b->fence_fd != -1) &&  
> > 
> > Why do we need to consider both values invalid? Can 0 ever be a valid fence 
> > fd?
> 
> Programs that don't use fences will initialize reserved2/fence_fd field
> at the uAPI call to zero.
> 
> So, I guess using fd=0 here could be a problem. Anyway, I would, instead,
> do:
> 
>   if ((b->fence_fd < 1) &&
>   ...
> 
> as other negative values are likely invalid as well.

We are checking when the fence_fd is set but the flag wasn't. Checking
for < 1 is exactly the opposite. so we keep as is or do it fence_fd > 0.

Gustavo

Re: [RFC v5 07/11] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

2017-11-17 Alexandre Courbot <acour...@chromium.org>:

> Hi Gustavo,
> 
> I am coming a bit late in this series' review, so apologies if some of my
> comments have already have been discussed in an earlier revision.
> 
> On Thursday, November 16, 2017 2:10:53 AM JST, Gustavo Padovan wrote:
> > From: Gustavo Padovan <gustavo.pado...@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queue to the driver
> > out of the order they were queued from userspace. That means that even if
> > it fence signal it must wait all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > To make that possible we use fence_array to keep that ordering. Basically
> > we create a fence_array that contains both the current fence and the fence
> > from the previous buffer (which might be a fence array as well). The base
> > fence class for the fence_array becomes the new buffer fence, waiting on
> > that one guarantees that it won't be queued out of order.
> > 
> > v6:
> > - With fences always keep the order userspace queues the buffers.
> > - Protect in_fence manipulation with a lock (Brian Starkey)
> > - check if fences have the same context before adding a fence array
> > - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> > - Clean up fence if __set_in_fence() fails (Brian Starkey)
> > - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> > needed (Brian Starkey)
> > - keep backward compat on the reserved2 field (Brian Starkey)
> > - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4:
> > - Add a comment about dma_fence_add_callback() not returning a
> > error (Hans)
> > - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> > - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> > - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> > - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> > -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> > vb2_start_streaming() (Hans)
> > - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> > - Queue buffers to the driver as soon as they are ready (Hans)
> > - call fill_user_buffer() after queuing the buffer (Hans)
> > - add err: label to clean up fence
> > - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> > - remove ternary if at vb2_qbuf() return (Mauro)
> > - do not change if conditions behaviour (Mauro)
> > 
> > v2:
> > - fix vb2_queue_or_prepare_buf() ret check
> > - remove check for VB2_MEMORY_DMABUF only (Javier)
> > - check num of ready buffers to start streaming
> > - when queueing, start from the first ready buffer
> > - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.pado...@collabora.com>
> > ---
> >  drivers/media/v4l2-core/Kconfig  |   1 +
> >  drivers/media/v4l2-core/videobuf2-core.c | 202
> > ---
> >  drivers/media/v4l2-core/videobuf2-v4l2.c |  29 -
> >  include/media/videobuf2-core.h   |  17 ++-
> >  4 files changed, 231 insertions(+), 18 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/Kconfig
> > b/drivers/media/v4l2-core/Kconfig
> > index a35c33686abf..3f988c407c80 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -83,6 +83,7 @@ config VIDEOBUF_DVB
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> > select DMA_SHARED_BUFFER
> > +   select SYNC_FILE
> > tristate
> >  config VIDEOBUF2_MEMOPS
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index 60f8b582396a..26de4c80717d 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -23,6 +23,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> > @@ -346,6 +347,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q,
> > enum vb2_memory memory,
> > vb->index = q->num_buffers + buffer;
> >