Re: [PATCH] uvcvideo: add a D4M camera description

[Guennadi Liakhovetski]

Hi Laurent,

On Sat, 25 Aug 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Friday, 3 August 2018 14:07:12 EEST Guennadi Liakhovetski wrote:
> > Hi Laurent,
> > 
> > Thanks for the review. A general note: I think you're requesting a rather
> > detailed information about many parameters. That isn't a problem by
> > itself, however, it is difficult to obtain some of that information. I'll
> > address whatever comments I can in an updated version, just answering some
> > questions here. I directed youor questions, that I couldn't answer myself
> > to respective people, but I have no idea if and when I get replies. So,
> > it's up to you whether to wait for that additional information or to take
> > at least what we have now.
> 
> I've replied to v2, and apart from a few minor points, I think we can apply 
> the current version. There are a few small questions I would still like to 
> have answers to, but if it takes to long to obtain that, let's not miss v4.20.
> 
> > On Sun, 29 Jul 2018, Laurent Pinchart wrote:
> > > On Saturday, 23 December 2017 13:11:00 EEST Guennadi Liakhovetski wrote:
> > >> From: Guennadi Liakhovetski 
> > >> 
> > >> D4M is a mobile model from the D4XX family of Intel RealSense cameras.
> > >> This patch adds a descriptor for it, which enables reading per-frame
> > >> metadata from it.
> > >> 
> > >> Signed-off-by: Guennadi Liakhovetski 
> > >> ---
> > >> 
> > >>  Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst | 202 
> > >>  drivers/media/usb/uvc/uvc_driver.c|  11 ++
> > >>  include/uapi/linux/videodev2.h|   1 +
> > >>  3 files changed, 214 insertions(+)
> > >>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > >> 
> > >> diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > >> b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst new file mode 100644
> > >> index 000..950780d
> > >> --- /dev/null
> > >> +++ b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> 
> [snip]
> 
> > >> +* - :cspan:`1` *Configuration*
> > >> +* - __u32 ID
> > >> +  - 0x8002
> > >> +* - __u32 Size
> > >> +  - Size in bytes (currently 40)
> > >> +* - __u32 Version
> > >> +  - Version of the struct
> > >> +* - __u32 Flags
> > >> +  - A bitmask of flags: see [4_] below
> > >> +* - __u8 Hardware type
> > >> +  - Camera hardware version [5_]
> > >> +* - __u8 SKU ID
> > >> +  - Camera hardware configuration [6_]
> > >> +* - __u32 Cookie
> > >> +  - Internal synchronisation
> > > 
> > > Internal synchronisation with what ? :-)
> 
> This is still something I'd like to understand (and I understand it may still 
> take time to receive an answer from the right person).

Sorry, no idea, that flag wasn't even set when I was testing it.

> > >> +* - __u16 Format
> > >> +  - Image format code [7_]
> > >> +* - __u16 Width
> > >> +  - Width in pixels
> > >> +* - __u16 Height
> > >> +  - Height in pixels
> > >> +* - __u16 Framerate
> > >> +  - Requested framerate
> > > 
> > > What's the unit of this value ?
> > 
> > Is anything other than frames per second used in V4L?
> 
> V4L2 expresses the frame rate as a fraction, hence my question, to know 
> whether this field contained the number of frames per second as an integer, 
> or 
> used a different representation (such as a fixed point decimal value for 
> instance).

Nono, sorry, just an integer FPS.

Thanks
Guennadi

> > >> +* - __u16 Trigger
> > >> +  - Byte 0: bit 0:  depth and RGB are synchronised, bit 1: external
> > >> trigger
> > >> +
> > >> +.. _1:
> > >> +
> > >> +[1]
> > >> https://docs.microsoft.com/en-us/windows-hardware/drivers/stream/uvc-ext
> > >> ensions-1-5
> > > 
> > > Should we at some point replicate that documentation in the V4L2 spec ?
> > > Without copying it of course, as that would be a copyright violation.
> > 
> > Well, we don't replicate the UVC itself or any other standards, do we? Of
> > course, that document doesn't have the same status as an official
> > vendor-neutral standard, but still, we don't replicate data sheets either.
> > Besides, I think there are cameras that use this, and windows supports
> > this, so, don't think it will disappear overnight...
> 
> Probably not overnight, you're right. I'm a bit worried about the link 
> becoming invalid though. In any case that's not a blocker, but I might at 
> some 
> point decide to replicate the documentation.
> 
> [snip]
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 
> 
> 

Re: [PATCH v2 2/2] uvcvideo: add a D4M camera description

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the review.

On Sat, 25 Aug 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> Overall this looks good to me, I only have small comments. Please see below, 
> with a summary at the end.
> 
> On Friday, 3 August 2018 14:37:08 EEST Guennadi Liakhovetski wrote:
> > D4M is a mobile model from the D4XX family of Intel RealSense cameras.
> > This patch adds a descriptor for it, which enables reading per-frame
> > metadata from it.
> > 
> > Signed-off-by: Guennadi Liakhovetski 
> > ---
> >  Documentation/media/uapi/v4l/meta-formats.rst |   1 +
> >  Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst | 204 +++
> >  drivers/media/usb/uvc/uvc_driver.c|  11 ++
> >  include/uapi/linux/videodev2.h|   1 +
> >  4 files changed, 217 insertions(+)
> >  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> 
> [snip]
> 
> > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst new file mode 100644
> > index 000..57ecfd9
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > @@ -0,0 +1,204 @@
> > +.. -*- coding: utf-8; mode: rst -*-
> > +
> > +.. _v4l2-meta-fmt-d4xx:
> > +
> > +***
> > +V4L2_META_FMT_D4XX ('D4XX')
> > +***
> > +
> > +Intel D4xx UVC Cameras Metadata
> > +
> > +
> > +Description
> > +===
> > +
> > +Intel D4xx (D435 and other) cameras include per-frame metadata in their UVC
> > +payload headers, following the Microsoft(R) UVC extension proposal [1_].
> > That +means, that the private D4XX metadata, following the standard UVC
> > header, is +organised in blocks. D4XX cameras implement several standard
> > block types, +proposed by Microsoft, and several proprietary ones.
> > Supported standard metadata +types are MetadataId_CaptureStats (ID 3),
> > MetadataId_CameraExtrinsics (ID 4), +and MetadataId_CameraIntrinsics (ID
> > 5). For their description see [1_]. This +document describes proprietary
> > metadata types, used by D4xx cameras.
> > +
> > +V4L2_META_FMT_D4XX buffers follow the metadata buffer layout of
> > +V4L2_META_FMT_UVC with the only difference, that it also includes
> > proprietary +payload header data. D4xx cameras use bulk transfers and only
> > send one payload +per frame, therefore their headers cannot be larger than
> > 255 bytes.
> > +
> > +Below are proprietary Microsoft style metadata types, used by D4xx cameras,
> > +where all fields are in little endian order:
> > +
> > +.. flat-table:: D4xx metadata
> > +:widths: 1 4
> > +:header-rows:  1
> > +:stub-columns: 0
> > +
> > +* - Field
> > +  - Description
> > +* - :cspan:`1` *Depth Control*
> 
> Does this mean that all fields in this structure apply to the depth image 
> only 
> ? If so, do you mind if I mention that explicitly ?

I don't think that's the case. E.g. only this struct has laser parameters 
and the laser can be used without the depth node being active. In fact I 
just tried streaming from the other node and reading metadata from its 
metadata node - it also included ID 0x8000 with flags set to 0xff, 
i.e. all fields valid.

> > +* - __u32 ID
> > +  - 0x8000
> > +* - __u32 Size
> > +  - Size in bytes (currently 56)
> > +* - __u32 Version
> > +  - Version of the struct
> 
> I would elaborate a bit here, how about the following ?
> 
> "Version of this structure. The documentation herein corresponds to version 
> xxx. The version number will be incremented when new fields are added."
> 
> If you can provide me with the current version number I can update this when 
> applying the patch (I would get it myself, but I'm about to board a plane and 
> haven't taken the camera with me :-/).

Sure, sounds good, the "depth control" is version 2, the rest is version 
1.

> > +* - __u32 Flags
> > +  - A bitmask of flags: see [2_] below
> > +* - __u32 Gain
> > +  - Gain value in internal units, same as the V4L2_CID_GAIN control,
> > used to
> > +capture the frame
> > +* - __u32 Exposure
> > +  - Exposure time (in microseconds) used to capture the frame
> > +* - __u32 Laser power
> > +  - Power of the laser LED 0-360, used for depth measurement
> > +* - __u32 AE mode
> > +  - 0: manual; 1: automatic expo

Re: [PATCH v2 1/2] uvcvideo: rename UVC_QUIRK_INFO to UVC_INFO_QUIRK

[Guennadi Liakhovetski]

Hi Laurent,

As far as I understand we've missed the 4.19 cycle. Can we move on with 
this please?

Thanks
Guennadi

On Fri, 3 Aug 2018, Guennadi Liakhovetski wrote:

> This macro defines "information about quirks," not "quirks for
> information."
> 
> Signed-off-by: Guennadi Liakhovetski 
> ---
>  drivers/media/usb/uvc/uvc_driver.c | 18 +-
>  1 file changed, 9 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/media/usb/uvc/uvc_driver.c 
> b/drivers/media/usb/uvc/uvc_driver.c
> index d46dc43..699984b 100644
> --- a/drivers/media/usb/uvc/uvc_driver.c
> +++ b/drivers/media/usb/uvc/uvc_driver.c
> @@ -2344,7 +2344,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
>   .quirks = UVC_QUIRK_FORCE_Y8,
>  };
>  
> -#define UVC_QUIRK_INFO(q) (kernel_ulong_t)&(struct uvc_device_info){.quirks 
> = q}
> +#define UVC_INFO_QUIRK(q) (kernel_ulong_t)&(struct uvc_device_info){.quirks 
> = q}
>  
>  /*
>   * The Logitech cameras listed below have their interface class set to
> @@ -2453,7 +2453,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
> .bInterfaceClass  = USB_CLASS_VIDEO,
> .bInterfaceSubClass   = 1,
> .bInterfaceProtocol   = 0,
> -   .driver_info  = 
> UVC_QUIRK_INFO(UVC_QUIRK_RESTORE_CTRLS_ON_INIT) },
> +   .driver_info  = 
> UVC_INFO_QUIRK(UVC_QUIRK_RESTORE_CTRLS_ON_INIT) },
>   /* Chicony CNF7129 (Asus EEE 100HE) */
>   { .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
>   | USB_DEVICE_ID_MATCH_INT_INFO,
> @@ -2462,7 +2462,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
> .bInterfaceClass  = USB_CLASS_VIDEO,
> .bInterfaceSubClass   = 1,
> .bInterfaceProtocol   = 0,
> -   .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_RESTRICT_FRAME_RATE) 
> },
> +   .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_RESTRICT_FRAME_RATE) 
> },
>   /* Alcor Micro AU3820 (Future Boy PC USB Webcam) */
>   { .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
>   | USB_DEVICE_ID_MATCH_INT_INFO,
> @@ -2525,7 +2525,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
> .bInterfaceClass  = USB_CLASS_VIDEO,
> .bInterfaceSubClass   = 1,
> .bInterfaceProtocol   = 0,
> -   .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_MINMAX
> +   .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_MINMAX
>   | UVC_QUIRK_BUILTIN_ISIGHT) },
>   /* Apple Built-In iSight via iBridge */
>   { .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
> @@ -2607,7 +2607,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
> .bInterfaceClass  = USB_CLASS_VIDEO,
> .bInterfaceSubClass   = 1,
> .bInterfaceProtocol   = 0,
> -   .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_MINMAX
> +   .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_MINMAX
>   | UVC_QUIRK_PROBE_DEF) },
>   /* IMC Networks (Medion Akoya) */
>   { .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
> @@ -2707,7 +2707,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
> .bInterfaceClass  = USB_CLASS_VIDEO,
> .bInterfaceSubClass   = 1,
> .bInterfaceProtocol   = 0,
> -   .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_MINMAX
> +   .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_MINMAX
>   | UVC_QUIRK_PROBE_EXTRAFIELDS) },
>   /* Aveo Technology USB 2.0 Camera (Tasco USB Microscope) */
>   { .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
> @@ -2725,7 +2725,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
> .bInterfaceClass  = USB_CLASS_VIDEO,
> .bInterfaceSubClass   = 1,
> .bInterfaceProtocol   = 0,
> -   .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_EXTRAFIELDS) },
> +   .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_EXTRAFIELDS) },
>   /* Manta MM-353 Plako */
>   { .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
>   | USB_DEVICE_ID_MATCH_INT_INFO,
> @@ -2771,7 +2771,7 @@ static int uvc_clock_param_set(const char *val, const 
> struct kernel_param *kp)
> .bInterfaceClass  = USB_CLASS_VIDEO,
> .bInterfaceSubClass   = 1,
> .bInterfaceProtocol   = 0,
> -   .driver_inf

[PATCH v2 1/2] uvcvideo: rename UVC_QUIRK_INFO to UVC_INFO_QUIRK

[Guennadi Liakhovetski]

This macro defines "information about quirks," not "quirks for
information."

Signed-off-by: Guennadi Liakhovetski 
---
 drivers/media/usb/uvc/uvc_driver.c | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index d46dc43..699984b 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -2344,7 +2344,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
.quirks = UVC_QUIRK_FORCE_Y8,
 };
 
-#define UVC_QUIRK_INFO(q) (kernel_ulong_t)&(struct uvc_device_info){.quirks = 
q}
+#define UVC_INFO_QUIRK(q) (kernel_ulong_t)&(struct uvc_device_info){.quirks = 
q}
 
 /*
  * The Logitech cameras listed below have their interface class set to
@@ -2453,7 +2453,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = 
UVC_QUIRK_INFO(UVC_QUIRK_RESTORE_CTRLS_ON_INIT) },
+ .driver_info  = 
UVC_INFO_QUIRK(UVC_QUIRK_RESTORE_CTRLS_ON_INIT) },
/* Chicony CNF7129 (Asus EEE 100HE) */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
| USB_DEVICE_ID_MATCH_INT_INFO,
@@ -2462,7 +2462,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_RESTRICT_FRAME_RATE) 
},
+ .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_RESTRICT_FRAME_RATE) 
},
/* Alcor Micro AU3820 (Future Boy PC USB Webcam) */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
| USB_DEVICE_ID_MATCH_INT_INFO,
@@ -2525,7 +2525,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_MINMAX
+ .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_MINMAX
| UVC_QUIRK_BUILTIN_ISIGHT) },
/* Apple Built-In iSight via iBridge */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
@@ -2607,7 +2607,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_MINMAX
+ .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_MINMAX
| UVC_QUIRK_PROBE_DEF) },
/* IMC Networks (Medion Akoya) */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
@@ -2707,7 +2707,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_MINMAX
+ .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_MINMAX
| UVC_QUIRK_PROBE_EXTRAFIELDS) },
/* Aveo Technology USB 2.0 Camera (Tasco USB Microscope) */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
@@ -2725,7 +2725,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_PROBE_EXTRAFIELDS) },
+ .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_PROBE_EXTRAFIELDS) },
/* Manta MM-353 Plako */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
| USB_DEVICE_ID_MATCH_INT_INFO,
@@ -2771,7 +2771,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_INFO(UVC_QUIRK_STATUS_INTERVAL) },
+ .driver_info  = UVC_INFO_QUIRK(UVC_QUIRK_STATUS_INTERVAL) },
/* MSI StarCam 370i */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
| USB_DEVICE_ID_MATCH_INT_INFO,
@@ -2798,7 +2798,7 @@ static int uvc_clock_param_set(const char *val, const 
struct kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_in

[PATCH v2 2/2] uvcvideo: add a D4M camera description

[Guennadi Liakhovetski]

D4M is a mobile model from the D4XX family of Intel RealSense cameras.
This patch adds a descriptor for it, which enables reading per-frame
metadata from it.

Signed-off-by: Guennadi Liakhovetski 
---
 Documentation/media/uapi/v4l/meta-formats.rst |   1 +
 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst | 204 ++
 drivers/media/usb/uvc/uvc_driver.c|  11 ++
 include/uapi/linux/videodev2.h|   1 +
 4 files changed, 217 insertions(+)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst

diff --git a/Documentation/media/uapi/v4l/meta-formats.rst 
b/Documentation/media/uapi/v4l/meta-formats.rst
index 0c4e1ec..cf971d5 100644
--- a/Documentation/media/uapi/v4l/meta-formats.rst
+++ b/Documentation/media/uapi/v4l/meta-formats.rst
@@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
 .. toctree::
 :maxdepth: 1
 
+pixfmt-meta-d4xx
 pixfmt-meta-uvc
 pixfmt-meta-vsp1-hgo
 pixfmt-meta-vsp1-hgt
diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst 
b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
new file mode 100644
index 000..57ecfd9
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
@@ -0,0 +1,204 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _v4l2-meta-fmt-d4xx:
+
+***
+V4L2_META_FMT_D4XX ('D4XX')
+***
+
+Intel D4xx UVC Cameras Metadata
+
+
+Description
+===
+
+Intel D4xx (D435 and other) cameras include per-frame metadata in their UVC
+payload headers, following the Microsoft(R) UVC extension proposal [1_]. That
+means, that the private D4XX metadata, following the standard UVC header, is
+organised in blocks. D4XX cameras implement several standard block types,
+proposed by Microsoft, and several proprietary ones. Supported standard 
metadata
+types are MetadataId_CaptureStats (ID 3), MetadataId_CameraExtrinsics (ID 4),
+and MetadataId_CameraIntrinsics (ID 5). For their description see [1_]. This
+document describes proprietary metadata types, used by D4xx cameras.
+
+V4L2_META_FMT_D4XX buffers follow the metadata buffer layout of
+V4L2_META_FMT_UVC with the only difference, that it also includes proprietary
+payload header data. D4xx cameras use bulk transfers and only send one payload
+per frame, therefore their headers cannot be larger than 255 bytes.
+
+Below are proprietary Microsoft style metadata types, used by D4xx cameras,
+where all fields are in little endian order:
+
+.. flat-table:: D4xx metadata
+:widths: 1 4
+:header-rows:  1
+:stub-columns: 0
+
+* - Field
+  - Description
+* - :cspan:`1` *Depth Control*
+* - __u32 ID
+  - 0x8000
+* - __u32 Size
+  - Size in bytes (currently 56)
+* - __u32 Version
+  - Version of the struct
+* - __u32 Flags
+  - A bitmask of flags: see [2_] below
+* - __u32 Gain
+  - Gain value in internal units, same as the V4L2_CID_GAIN control, used 
to
+   capture the frame
+* - __u32 Exposure
+  - Exposure time (in microseconds) used to capture the frame
+* - __u32 Laser power
+  - Power of the laser LED 0-360, used for depth measurement
+* - __u32 AE mode
+  - 0: manual; 1: automatic exposure
+* - __u32 Exposure priority
+  - Exposure priority value: 0 - constant frame rate
+* - __u32 AE ROI left
+  - Left border of the AE Region of Interest (all ROI values are in pixels
+   and lie between 0 and maximum width or height respectively)
+* - __u32 AE ROI right
+  - Right border of the AE Region of Interest
+* - __u32 AE ROI top
+  - Top border of the AE Region of Interest
+* - __u32 AE ROI bottom
+  - Bottom border of the AE Region of Interest
+* - __u32 Preset
+  - Preset selector value, default: 0, unless changed by the user
+* - __u32 Laser mode
+  - 0: off, 1: on
+* - :cspan:`1` *Capture Timing*
+* - __u32 ID
+  - 0x8001
+* - __u32 Size
+  - Size in bytes (currently 40)
+* - __u32 Version
+  - Version of the struct
+* - __u32 Flags
+  - A bitmask of flags: see [3_] below
+* - __u32 Frame counter
+  - Monotonically increasing counter
+* - __u32 Optical time
+  - Time in microseconds from the beginning of a frame till its middle
+* - __u32 Readout time
+  - Time, used to read out a frame in microseconds
+* - __u32 Exposure time
+  - Frame exposure time in microseconds
+* - __u32 Frame interval
+  - In microseconds = 100 / framerate
+* - __u32 Pipe latency
+  - Time in microseconds from start of frame to data in USB buffer
+* - :cspan:`1` *Configuration*
+* - __u32 ID
+  - 0x8002
+* - __u32 Size
+  - Size in bytes (currently 40)
+* - __u32 Version
+  - Version of the struct
+* - __u32 Flags
+  - A bitmask of flags: see [4_] below
+* - __u8 Hardware type
+  - Camera

Re: [PATCH] uvcvideo: add a D4M camera description

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the review. A general note: I think you're requesting a rather 
detailed information about many parameters. That isn't a problem by 
itself, however, it is difficult to obtain some of that information. I'll 
address whatever comments I can in an updated version, just answering some 
questions here. I directed youor questions, that I couldn't answer myself 
to respective people, but I have no idea if and when I get replies. So, 
it's up to you whether to wait for that additional information or to take 
at least what we have now.

On Sun, 29 Jul 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> On Saturday, 23 December 2017 13:11:00 EEST Guennadi Liakhovetski wrote:
> > From: Guennadi Liakhovetski 
> > 
> > D4M is a mobile model from the D4XX family of Intel RealSense cameras.
> > This patch adds a descriptor for it, which enables reading per-frame
> > metadata from it.
> > 
> > Signed-off-by: Guennadi Liakhovetski 
> > ---
> >  Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst | 202 +++
> >  drivers/media/usb/uvc/uvc_driver.c|  11 ++
> >  include/uapi/linux/videodev2.h|   1 +
> >  3 files changed, 214 insertions(+)
> >  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > 
> > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst new file mode 100644
> > index 000..950780d
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > @@ -0,0 +1,202 @@
> > +.. -*- coding: utf-8; mode: rst -*-
> > +
> > +.. _v4l2-meta-fmt-d4xx:
> > +
> > +***
> > +V4L2_META_FMT_D4XX ('D4XX')
> > +***
> > +
> > +D4XX Metadata
> 
> How about "Intel D4xx UVC Cameras Metadata" ?
> 
> > +
> > +
> > +Description
> > +===
> > +
> > +D4XX (D435 and other) cameras include per-frame metadata in their UVC
> > payload
> 
> Should this be "Intel D4XX" ?
> 
> > +headers, following the Microsoft(R) UVC extension proposal [1_]. That
> > means,
> > +that the private D4XX metadata, following the standard UVC header, is
> > organised
> > +in blocks. D4XX cameras implement several standard block types, proposed by
> > +Microsoft, and several proprietary ones. Supported standard metadata types
> > +include MetadataId_CaptureStats (ID 3), MetadataId_CameraExtrinsics (ID 4),
> > and
> > +MetadataId_CameraIntrinsics (ID 5). For their description see [1_].
> 
> Does "including" mean that the list isn't exhaustive and that other standard 
> types could be returned too ? If so, would it be possible to get an 
> exhaustive 
> list ? And if the list is exhaustive, could you word this paragraph to make 
> that clear ?
> 
> > This
> > +document describes proprietary metadata types, used by DS4XX cameras.
> 
> Is it D4XX or DS4XX ?
> 
> > +V4L2_META_FMT_D4XX buffers follow the metadata buffer layout of
> > +V4L2_META_FMT_UVC with the only difference, that it also includes
> > proprietary
> > +payload header data. D4XX cameras use bulk transfers and only send one
> > payload
> > +per frame, therefore their headers cannot be larger than 255 bytes.
> > +
> > +Below are proprietary Microsoft style metadata types, used by D4XX cameras,
> > +where all fields are in little endian order:
> > +
> > +.. flat-table:: D4XX metadata
> > +:widths: 1 4
> > +:header-rows:  1
> > +:stub-columns: 0
> > +
> > +* - Field
> > +  - Description
> > +* - :cspan:`1` *Depth Control*
> > +* - __u32 ID
> > +  - 0x8000
> > +* - __u32 Size
> > +  - Size in bytes (currently 56)
> > +* - __u32 Version
> > +  - Version of the struct
> 
> What is this field used for ?

For future changes to this (and all other) struct(s). If in the future a 
new field is added to this struct, the version will be incremented to 
inform the user.

> > +* - __u32 Flags
> > +  - A bitmask of flags: see [2_] below
> > +* - __u32 Gain
> > +  - Manual gain value
> 
> What is the gain unit ?

It's in internal units. I guess librealsense has formulas to convert them 
to ISO or something else standard. It's the same units as the 
V4L2_CID_GAIN control.

> > +* - __u32 Exposure
> > +  - Manual exposure time in microseconds
> 
> When auto-exposure is enabled, does this reflect the actual exposure t

Re: [PATCH v9] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

Thanks. Now we can get to the next one: 
https://patchwork.linuxtv.org/patch/46184/ - without that one nobody can 
get complete D4XX metadata. After I get your comments to it I'll address 
them together with Sakari's ones.

Thanks
Guennadi

On Thu, 26 Jul 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> On Thursday, 26 July 2018 11:17:53 EEST Guennadi Liakhovetski wrote:
> > From: Guennadi Liakhovetski 
> > 
> > UVC defines a method of handling asynchronous controls, which sends a
> > USB packet over the interrupt pipe. This patch implements support for
> > such packets by sending a control event to the user. Since this can
> > involve USB traffic and, therefore, scheduling, this has to be done
> > in a work queue.
> > 
> > Signed-off-by: Guennadi Liakhovetski 
> 
> Reviewed-by: Laurent Pinchart 
> 
> applied to my tree and pushed to the uvc/next branch.
> 
> > ---
> > 
> > v9:
> > - multiple optimisations and style improvements from Laurent - thanks
> > - avoid the handle rewriting race at least in cases, when the user only
> > sends a new control after receiving an event for the previous one
> > 
> > Laurent, you added a couple of comments, using DocBook markup like
> > "@param" but you didn't mark those comments for DocBook processing with
> > "/**" - I don't care that much, just wanted to check, that that was
> > intentional.
> 
> Yes, it's not worth compiling that as kerneldoc, I just wanted to mark 
> references to variable names for clarity.
> 
> >  drivers/media/usb/uvc/uvc_ctrl.c   | 211
> > - drivers/media/usb/uvc/uvc_status.c |
> > 121 ++---
> >  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> >  drivers/media/usb/uvc/uvcvideo.h   |  15 ++-
> >  include/uapi/linux/uvcvideo.h  |   2 +
> >  5 files changed, 286 insertions(+), 67 deletions(-)
> > 
> > diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > b/drivers/media/usb/uvc/uvc_ctrl.c index 2a213c8..ddd069b 100644
> > --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> > @@ -20,6 +20,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> > 
> > @@ -971,12 +972,30 @@ static int uvc_ctrl_populate_cache(struct
> > uvc_video_chain *chain, return 0;
> >  }
> > 
> > +static s32 __uvc_ctrl_get_value(struct uvc_control_mapping *mapping,
> > +   const u8 *data)
> > +{
> > +   s32 value = mapping->get(mapping, UVC_GET_CUR, data);
> > +
> > +   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
> > +   struct uvc_menu_info *menu = mapping->menu_info;
> > +   unsigned int i;
> > +
> > +   for (i = 0; i < mapping->menu_count; ++i, ++menu) {
> > +   if (menu->value == value) {
> > +   value = i;
> > +   break;
> > +   }
> > +   }
> > +   }
> > +
> > +   return value;
> > +}
> > +
> >  static int __uvc_ctrl_get(struct uvc_video_chain *chain,
> > struct uvc_control *ctrl, struct uvc_control_mapping *mapping,
> > s32 *value)
> >  {
> > -   struct uvc_menu_info *menu;
> > -   unsigned int i;
> > int ret;
> > 
> > if ((ctrl->info.flags & UVC_CTRL_FLAG_GET_CUR) == 0)
> > @@ -993,18 +1012,8 @@ static int __uvc_ctrl_get(struct uvc_video_chain
> > *chain, ctrl->loaded = 1;
> > }
> > 
> > -   *value = mapping->get(mapping, UVC_GET_CUR,
> > -   uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
> > -
> > -   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
> > -   menu = mapping->menu_info;
> > -   for (i = 0; i < mapping->menu_count; ++i, ++menu) {
> > -   if (menu->value == *value) {
> > -   *value = i;
> > -   break;
> > -   }
> > -   }
> > -   }
> > +   *value = __uvc_ctrl_get_value(mapping,
> > +   uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
> > 
> > return 0;
> >  }
> > @@ -1216,53 +1225,135 @@ static void uvc_ctrl_fill_event(struct
> > uvc_video_chain *chain, ev->u.ctrl.default_value = v4l2_ctrl.default_value;
> >  }
> > 
> > -static void uvc_ctrl_send_event(struct uvc_fh *handle,
> > -   struct uvc_control *ctrl,

[PATCH v9] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

From: Guennadi Liakhovetski 

UVC defines a method of handling asynchronous controls, which sends a
USB packet over the interrupt pipe. This patch implements support for
such packets by sending a control event to the user. Since this can
involve USB traffic and, therefore, scheduling, this has to be done
in a work queue.

Signed-off-by: Guennadi Liakhovetski 
---

v9:
- multiple optimisations and style improvements from Laurent - thanks
- avoid the handle rewriting race at least in cases, when the user only 
sends a new control after receiving an event for the previous one

Laurent, you added a couple of comments, using DocBook markup like 
"@param" but you didn't mark those comments for DocBook processing with 
"/**" - I don't care that much, just wanted to check, that that was 
intentional.

 drivers/media/usb/uvc/uvc_ctrl.c   | 211 -
 drivers/media/usb/uvc/uvc_status.c | 121 ++---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvcvideo.h   |  15 ++-
 include/uapi/linux/uvcvideo.h  |   2 +
 5 files changed, 286 insertions(+), 67 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c
index 2a213c8..ddd069b 100644
--- a/drivers/media/usb/uvc/uvc_ctrl.c
+++ b/drivers/media/usb/uvc/uvc_ctrl.c
@@ -20,6 +20,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 
@@ -971,12 +972,30 @@ static int uvc_ctrl_populate_cache(struct uvc_video_chain 
*chain,
return 0;
 }
 
+static s32 __uvc_ctrl_get_value(struct uvc_control_mapping *mapping,
+   const u8 *data)
+{
+   s32 value = mapping->get(mapping, UVC_GET_CUR, data);
+
+   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
+   struct uvc_menu_info *menu = mapping->menu_info;
+   unsigned int i;
+
+   for (i = 0; i < mapping->menu_count; ++i, ++menu) {
+   if (menu->value == value) {
+   value = i;
+   break;
+   }
+   }
+   }
+
+   return value;
+}
+
 static int __uvc_ctrl_get(struct uvc_video_chain *chain,
struct uvc_control *ctrl, struct uvc_control_mapping *mapping,
s32 *value)
 {
-   struct uvc_menu_info *menu;
-   unsigned int i;
int ret;
 
if ((ctrl->info.flags & UVC_CTRL_FLAG_GET_CUR) == 0)
@@ -993,18 +1012,8 @@ static int __uvc_ctrl_get(struct uvc_video_chain *chain,
ctrl->loaded = 1;
}
 
-   *value = mapping->get(mapping, UVC_GET_CUR,
-   uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
-
-   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
-   menu = mapping->menu_info;
-   for (i = 0; i < mapping->menu_count; ++i, ++menu) {
-   if (menu->value == *value) {
-   *value = i;
-   break;
-   }
-   }
-   }
+   *value = __uvc_ctrl_get_value(mapping,
+   uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
 
return 0;
 }
@@ -1216,53 +1225,135 @@ static void uvc_ctrl_fill_event(struct uvc_video_chain 
*chain,
ev->u.ctrl.default_value = v4l2_ctrl.default_value;
 }
 
-static void uvc_ctrl_send_event(struct uvc_fh *handle,
-   struct uvc_control *ctrl, struct uvc_control_mapping *mapping,
-   s32 value, u32 changes)
+/*
+ * Send control change events to all subscribers for the @ctrl control. By
+ * default the subscriber that generated the event, as identified by @handle,
+ * is not notified unless it has set the V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK flag.
+ * @handle can be NULL for asynchronous events related to auto-update controls,
+ * in which case all subscribers are notified.
+ */
+static void uvc_ctrl_send_event(struct uvc_video_chain *chain,
+   struct uvc_fh *handle, struct uvc_control *ctrl,
+   struct uvc_control_mapping *mapping, s32 value, u32 changes)
 {
+   struct v4l2_fh *originator = handle ? >vfh : NULL;
struct v4l2_subscribed_event *sev;
struct v4l2_event ev;
 
if (list_empty(>ev_subs))
return;
 
-   uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
+   uvc_ctrl_fill_event(chain, , ctrl, mapping, value, changes);
 
list_for_each_entry(sev, >ev_subs, node) {
-   if (sev->fh != >vfh ||
+   if (sev->fh != originator ||
(sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
(changes & V4L2_EVENT_CTRL_CH_FLAGS))
v4l2_event_queue_fh(sev->fh, );
}
 }
 
-static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
-   struct uvc_control *master, u32 slave_id,
-   const struct v4l2_ext_con

Re: [PATCH v8 2/3] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the diff. I'm about to go over it and check your comments, but 
I cannot seem to find a git tree with your current stack. Particularly, 
you wanted to apply patches 1 and 3 from this series before applying this 
one, so, I'd prefer to work on top of the same tree as you to make sure to 
avoid rejects.

Thanks
Guennaddi

On Wed, 25 Jul 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Wednesday, 25 July 2018 20:21:54 EEST Guennadi Liakhovetski wrote:
> > On Wed, 25 Jul 2018, Laurent Pinchart wrote:
> > > On Wednesday, 18 July 2018 09:55:27 EEST Guennadi Liakhovetski wrote:
> > >> On Wed, 18 Jul 2018, Laurent Pinchart wrote:
> > >>> On Wednesday, 18 July 2018 00:30:45 EEST Guennadi Liakhovetski wrote:
> > >>>> On Tue, 17 Jul 2018, Laurent Pinchart wrote:
> > >>>>> On Thursday, 12 July 2018 10:30:46 EEST Guennadi Liakhovetski wrote:
> > >>>>>> On Thu, 12 Jul 2018, Laurent Pinchart wrote:
> > >>>>>>> On Tuesday, 8 May 2018 18:07:43 EEST Guennadi Liakhovetski wrote:
> > >>>>>>>> UVC defines a method of handling asynchronous controls, which
> > >>>>>>>> sends a USB packet over the interrupt pipe. This patch implements
> > >>>>>>>> support for such packets by sending a control event to the user.
> > >>>>>>>> Since this can involve USB traffic and, therefore, scheduling, this
> > >>>>>>>> has to be done in a work queue.
> > >>>>>>>> 
> > >>>>>>>> Signed-off-by: Guennadi Liakhovetski
> > >>>>>>>> 
> > >>>>>>>> ---
> > >>>>>>>> 
> > >>>>>>>> v8:
> > >>>>>>>> 
> > >>>>>>>> * avoid losing events by delaying the status URB resubmission
> > >>>>>>>>   until after completion of the current event
>  >>>>>>>> * extract control value calculation into __uvc_ctrl_get_value()
> > >>>>>>>> * do not proactively return EBUSY if the previous control hasn't
> > >>>>>>>>   completed yet, let the camera handle such cases
> > >>>>>>>> * multiple cosmetic changes
> > >>>>>>>> 
> > >>>>>>>>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 +++--
> > >>>>>>>>  drivers/media/usb/uvc/uvc_status.c | 112 ++---
> > >>>>>>>>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> > >>>>>>>>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> > >>>>>>>>  include/uapi/linux/uvcvideo.h  |   2 +
> > >>>>>>>>  5 files changed, 255 insertions(+), 44 deletions(-)
> > >>>>>>>> 
> > >>>>>>>> diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>>>>>> b/drivers/media/usb/uvc/uvc_ctrl.c index 2a213c8..796f86a 100644
> > >>>>>>>> --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>>>>>> +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>>>> 
> > >>>>>> [snip]
> > >>>>>> 
> > >>>>>>>> +static void uvc_ctrl_status_event_work(struct work_struct *work)
> > >>>>>>>> +{
> > >>>>>>>> +  struct uvc_device *dev = container_of(work, struct uvc_device,
> > >>>>>>>> +async_ctrl.work);
> > >>>>>>>> +  struct uvc_ctrl_work *w = >async_ctrl;
> > >>>>>>>> +  struct uvc_control_mapping *mapping;
> > >>>>>>>> +  struct uvc_control *ctrl = w->ctrl;
> > >>>>>>>> +  unsigned int i;
> > >>>>>>>> +  int ret;
> > >>>>>>>> +
> > >>>>>>>> +  mutex_lock(>chain->ctrl_mutex);
> > >>>>>>>> +
> > >>>>>>>> +  list_for_each_entry(mapping, >info.mappings, list) {
> > >>>>>>>> +  s32 value = __uvc_ctrl_get_value(mapping, w->data);
> > >>>>>>>> +
> > >>>>>>>> +  /*
> > >&

Re: [PATCH v8 2/3] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

On Wed, 25 Jul 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Wednesday, 18 July 2018 09:55:27 EEST Guennadi Liakhovetski wrote:
> > On Wed, 18 Jul 2018, Laurent Pinchart wrote:
> > > On Wednesday, 18 July 2018 00:30:45 EEST Guennadi Liakhovetski wrote:
> > >> On Tue, 17 Jul 2018, Laurent Pinchart wrote:
> > >>> On Thursday, 12 July 2018 10:30:46 EEST Guennadi Liakhovetski wrote:
> > >>>> On Thu, 12 Jul 2018, Laurent Pinchart wrote:
> > >>>>> On Tuesday, 8 May 2018 18:07:43 EEST Guennadi Liakhovetski wrote:
> > >>>>>> UVC defines a method of handling asynchronous controls, which sends
> > >>>>>> a USB packet over the interrupt pipe. This patch implements support
> > >>>>>> for such packets by sending a control event to the user. Since this
> > >>>>>> can involve USB traffic and, therefore, scheduling, this has to be
> > >>>>>> done in a work queue.
> > >>>>>> 
> > >>>>>> Signed-off-by: Guennadi Liakhovetski
> > >>>>>> 
> > >>>>>> ---
> > >>>>>> 
> > >>>>>> v8:
> > >>>>>> 
> > >>>>>> * avoid losing events by delaying the status URB resubmission until
> > >>>>>>   after completion of the current event
> > >>>>>> * extract control value calculation into __uvc_ctrl_get_value()
> > >>>>>> * do not proactively return EBUSY if the previous control hasn't
> > >>>>>>   completed yet, let the camera handle such cases
> > >>>>>> * multiple cosmetic changes
> > >>>>>> 
> > >>>>>>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 --
> > >>>>>>  drivers/media/usb/uvc/uvc_status.c | 112 ++---
> > >>>>>>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> > >>>>>>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> > >>>>>>  include/uapi/linux/uvcvideo.h  |   2 +
> > >>>>>>  5 files changed, 255 insertions(+), 44 deletions(-)
> > >>>>>> 
> > >>>>>> diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>>>> b/drivers/media/usb/uvc/uvc_ctrl.c index 2a213c8..796f86a 100644
> > >>>>>> --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>>>> +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>> 
> > >>>> [snip]
> > >>>> 
> > >>>>>> +static void uvc_ctrl_status_event_work(struct work_struct *work)
> > >>>>>> +{
> > >>>>>> +struct uvc_device *dev = container_of(work, struct uvc_device,
> > >>>>>> +  async_ctrl.work);
> > >>>>>> +struct uvc_ctrl_work *w = >async_ctrl;
> > >>>>>> +struct uvc_control_mapping *mapping;
> > >>>>>> +struct uvc_control *ctrl = w->ctrl;
> > >>>>>> +unsigned int i;
> > >>>>>> +int ret;
> > >>>>>> +
> > >>>>>> +mutex_lock(>chain->ctrl_mutex);
> > >>>>>> +
> > >>>>>> +list_for_each_entry(mapping, >info.mappings, list) {
> > >>>>>> +s32 value = __uvc_ctrl_get_value(mapping, w->data);
> > >>>>>> +
> > >>>>>> +/*
> > >>>>>> + * So far none of the auto-update controls in the 
> > >>>>>> uvc_ctrls[]
> > >>>>>> + * table is mapped to a V4L control with slaves in the
> > >>>>>> + * uvc_ctrl_mappings[] list, so slave controls so far 
> > >>>>>> never have
> > >>>>>> + * handle == NULL, but this can change in the future
> > >>>>>> + */
> > >>>>>> +for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
> > >>>>>> +if (!mapping->slave_ids[i])
> > >>>>>> +break;
> > >>>>>> +
> > >>>>>> +__uvc_ctrl_send_slave_event(ctrl->h

Re: [PATCH v8 2/3] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

On Wed, 18 Jul 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Wednesday, 18 July 2018 00:30:45 EEST Guennadi Liakhovetski wrote:
> > On Tue, 17 Jul 2018, Laurent Pinchart wrote:
> > > On Thursday, 12 July 2018 10:30:46 EEST Guennadi Liakhovetski wrote:
> > >> On Thu, 12 Jul 2018, Laurent Pinchart wrote:
> > >>> On Tuesday, 8 May 2018 18:07:43 EEST Guennadi Liakhovetski wrote:
> > >>>> UVC defines a method of handling asynchronous controls, which sends a
> > >>>> USB packet over the interrupt pipe. This patch implements support for
> > >>>> such packets by sending a control event to the user. Since this can
> > >>>> involve USB traffic and, therefore, scheduling, this has to be done
> > >>>> in a work queue.
> > >>>> 
> > >>>> Signed-off-by: Guennadi Liakhovetski
> > >>>> 
> > >>>> ---
> > >>>> 
> > >>>> v8:
> > >>>> 
> > >>>> * avoid losing events by delaying the status URB resubmission until
> > >>>>   after completion of the current event
> > >>>> * extract control value calculation into __uvc_ctrl_get_value()
> > >>>> * do not proactively return EBUSY if the previous control hasn't
> > >>>>   completed yet, let the camera handle such cases
> > >>>> * multiple cosmetic changes
> > >>>> 
> > >>>>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 +--
> > >>>>  drivers/media/usb/uvc/uvc_status.c | 112 ++---
> > >>>>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> > >>>>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> > >>>>  include/uapi/linux/uvcvideo.h  |   2 +
> > >>>>  5 files changed, 255 insertions(+), 44 deletions(-)
> > >>>> 
> > >>>> diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>> b/drivers/media/usb/uvc/uvc_ctrl.c index 2a213c8..796f86a 100644
> > >>>> --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > >>>> +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> > >> 
> > >> [snip]
> > >> 
> > >>>> +static void uvc_ctrl_status_event_work(struct work_struct *work)
> > >>>> +{
> > >>>> +  struct uvc_device *dev = container_of(work, struct uvc_device,
> > >>>> +async_ctrl.work);
> > >>>> +  struct uvc_ctrl_work *w = >async_ctrl;
> > >>>> +  struct uvc_control_mapping *mapping;
> > >>>> +  struct uvc_control *ctrl = w->ctrl;
> > >>>> +  unsigned int i;
> > >>>> +  int ret;
> > >>>> +
> > >>>> +  mutex_lock(>chain->ctrl_mutex);
> > >>>> +
> > >>>> +  list_for_each_entry(mapping, >info.mappings, list) {
> > >>>> +  s32 value = __uvc_ctrl_get_value(mapping, w->data);
> > >>>> +
> > >>>> +  /*
> > >>>> +   * So far none of the auto-update controls in the 
> > >>>> uvc_ctrls[]
> > >>>> +   * table is mapped to a V4L control with slaves in the
> > >>>> +   * uvc_ctrl_mappings[] list, so slave controls so far 
> > >>>> never have
> > >>>> +   * handle == NULL, but this can change in the future
> > >>>> +   */
> > >>>> +  for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
> > >>>> +  if (!mapping->slave_ids[i])
> > >>>> +  break;
> > >>>> +
> > >>>> +  __uvc_ctrl_send_slave_event(ctrl->handle, 
> > >>>> w->chain,
> > >>>> +  ctrl, 
> > >>>> mapping->slave_ids[i]);
> > >>>> +  }
> > >>>> +
> > >>>> +  uvc_ctrl_send_event(ctrl->handle, ctrl, mapping, value,
> > >>>> +  V4L2_EVENT_CTRL_CH_VALUE);
> > >>>> +  }
> > >>>> +
> > >>>> +  mutex_unlock(>chain->ctrl_mutex);
> > >>>>

Re: [PATCH v8 2/3] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

On Tue, 17 Jul 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Thursday, 12 July 2018 10:30:46 EEST Guennadi Liakhovetski wrote:
> > On Thu, 12 Jul 2018, Laurent Pinchart wrote:
> > > On Tuesday, 8 May 2018 18:07:43 EEST Guennadi Liakhovetski wrote:
> > >> UVC defines a method of handling asynchronous controls, which sends a
> > >> USB packet over the interrupt pipe. This patch implements support for
> > >> such packets by sending a control event to the user. Since this can
> > >> involve USB traffic and, therefore, scheduling, this has to be done
> > >> in a work queue.
> > >> 
> > >> Signed-off-by: Guennadi Liakhovetski 
> > >> ---
> > >> 
> > >> v8:
> > >> 
> > >> * avoid losing events by delaying the status URB resubmission until
> > >>   after completion of the current event
> > >> * extract control value calculation into __uvc_ctrl_get_value()
> > >> * do not proactively return EBUSY if the previous control hasn't
> > >>   completed yet, let the camera handle such cases
> > >> * multiple cosmetic changes
> > >> 
> > >>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 ---
> > >>  drivers/media/usb/uvc/uvc_status.c | 112 ++---
> > >>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> > >>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> > >>  include/uapi/linux/uvcvideo.h  |   2 +
> > >>  5 files changed, 255 insertions(+), 44 deletions(-)
> > >> 
> > >> diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > >> b/drivers/media/usb/uvc/uvc_ctrl.c index 2a213c8..796f86a 100644
> > >> --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > >> +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> > 
> > [snip]
> > 
> > >> +static void uvc_ctrl_status_event_work(struct work_struct *work)
> > >> +{
> > >> +struct uvc_device *dev = container_of(work, struct uvc_device,
> > >> +  async_ctrl.work);
> > >> +struct uvc_ctrl_work *w = >async_ctrl;
> > >> +struct uvc_control_mapping *mapping;
> > >> +struct uvc_control *ctrl = w->ctrl;
> > >> +unsigned int i;
> > >> +int ret;
> > >> +
> > >> +mutex_lock(>chain->ctrl_mutex);
> > >> +
> > >> +list_for_each_entry(mapping, >info.mappings, list) {
> > >> +s32 value = __uvc_ctrl_get_value(mapping, w->data);
> > >> +
> > >> +/*
> > >> + * So far none of the auto-update controls in the 
> > >> uvc_ctrls[]
> > >> + * table is mapped to a V4L control with slaves in the
> > >> + * uvc_ctrl_mappings[] list, so slave controls so far 
> > >> never have
> > >> + * handle == NULL, but this can change in the future
> > >> + */
> > >> +for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
> > >> +if (!mapping->slave_ids[i])
> > >> +break;
> > >> +
> > >> +__uvc_ctrl_send_slave_event(ctrl->handle, 
> > >> w->chain,
> > >> +ctrl, 
> > >> mapping->slave_ids[i]);
> > >> +}
> > >> +
> > >> +uvc_ctrl_send_event(ctrl->handle, ctrl, mapping, value,
> > >> +V4L2_EVENT_CTRL_CH_VALUE);
> > >> +}
> > >> +
> > >> +mutex_unlock(>chain->ctrl_mutex);
> > >> +
> > >> +ctrl->handle = NULL;
> > > 
> > > Can't this race with a uvc_ctrl_set() call, resulting in ctrl->handle
> > > being NULL after the control gets set ?
> > 
> > Right, it's better to set .handle to NULL before sending events. Something
> > like
> > 
> > mutex_lock();
> > 
> > handle = ctrl->handle;
> > ctrl->handle = NULL;
> > 
> > list_for_each_entry() {
> > ...
> > uvc_ctrl_send_event(handle,...);
> > }
> > 
> > mutex_unlock();
> > 
> > ?
> 
> I think you also have to take the same lock in the uvc_ctrl_set() function 

Re: [PATCH v8 2/3] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

On Thu, 12 Jul 2018, Guennadi Liakhovetski wrote:

> Hi Laurent,
> 
> Thanks for the review.

[snip]

> Once I get your review of patch #3, I'll fix these two issues and 
> resubmit, so you can also tell me your "minor concerns," since I'll be 
> resubmitting anyway.

Any progress on this?

Thanks
Guennadi

Re: [PATCH v8 2/3] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the review.

On Thu, 12 Jul 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> On Tuesday, 8 May 2018 18:07:43 EEST Guennadi Liakhovetski wrote:
> > UVC defines a method of handling asynchronous controls, which sends a
> > USB packet over the interrupt pipe. This patch implements support for
> > such packets by sending a control event to the user. Since this can
> > involve USB traffic and, therefore, scheduling, this has to be done
> > in a work queue.
> > 
> > Signed-off-by: Guennadi Liakhovetski 
> > ---
> > 
> > v8:
> > 
> > * avoid losing events by delaying the status URB resubmission until
> >   after completion of the current event
> > * extract control value calculation into __uvc_ctrl_get_value()
> > * do not proactively return EBUSY if the previous control hasn't
> >   completed yet, let the camera handle such cases
> > * multiple cosmetic changes
> > 
> >  drivers/media/usb/uvc/uvc_ctrl.c   | 166 +++---
> >  drivers/media/usb/uvc/uvc_status.c | 112 ++---
> >  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> >  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> >  include/uapi/linux/uvcvideo.h  |   2 +
> >  5 files changed, 255 insertions(+), 44 deletions(-)
> > 
> > diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > b/drivers/media/usb/uvc/uvc_ctrl.c index 2a213c8..796f86a 100644
> > --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > +++ b/drivers/media/usb/uvc/uvc_ctrl.c

[snip]

> > +static void uvc_ctrl_status_event_work(struct work_struct *work)
> > +{
> > +   struct uvc_device *dev = container_of(work, struct uvc_device,
> > + async_ctrl.work);
> > +   struct uvc_ctrl_work *w = >async_ctrl;
> > +   struct uvc_control_mapping *mapping;
> > +   struct uvc_control *ctrl = w->ctrl;
> > +   unsigned int i;
> > +   int ret;
> > +
> > +   mutex_lock(>chain->ctrl_mutex);
> > +
> > +   list_for_each_entry(mapping, >info.mappings, list) {
> > +   s32 value = __uvc_ctrl_get_value(mapping, w->data);
> > +
> > +   /*
> > +* So far none of the auto-update controls in the uvc_ctrls[]
> > +* table is mapped to a V4L control with slaves in the
> > +* uvc_ctrl_mappings[] list, so slave controls so far never have
> > +* handle == NULL, but this can change in the future
> > +*/
> > +   for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
> > +   if (!mapping->slave_ids[i])
> > +   break;
> > +
> > +   __uvc_ctrl_send_slave_event(ctrl->handle, w->chain,
> > +   ctrl, mapping->slave_ids[i]);
> > +   }
> > +
> > +   uvc_ctrl_send_event(ctrl->handle, ctrl, mapping, value,
> > +   V4L2_EVENT_CTRL_CH_VALUE);
> > +   }
> > +
> > +   mutex_unlock(>chain->ctrl_mutex);
> > +
> > +   ctrl->handle = NULL;
> 
> Can't this race with a uvc_ctrl_set() call, resulting in ctrl->handle being 
> NULL after the control gets set ?

Right, it's better to set .handle to NULL before sending events. Something 
like

mutex_lock();

handle = ctrl->handle;
ctrl->handle = NULL;

list_for_each_entry() {
...
uvc_ctrl_send_event(handle,...);
}

mutex_unlock();

?

> 
> > +   /* Resubmit the URB. */
> > +   w->urb->interval = dev->int_ep->desc.bInterval;
> > +   ret = usb_submit_urb(w->urb, GFP_KERNEL);
> > +   if (ret < 0)
> > +   uvc_printk(KERN_ERR, "Failed to resubmit status URB (%d).\n",
> > +  ret);
> > +}
> > +
> > +bool uvc_ctrl_status_event(struct urb *urb, struct uvc_video_chain *chain,
> > +  struct uvc_control *ctrl, const u8 *data)
> > +{
> > +   struct uvc_device *dev = chain->dev;
> > +   struct uvc_ctrl_work *w = >async_ctrl;
> > +
> > +   if (list_empty(>info.mappings)) {
> > +   ctrl->handle = NULL;
> > +   return false;
> > +   }
> > +
> > +   w->data = data;
> > +   w->urb = urb;
> > +   w->chain = chain;
> > +   w->ctrl = ctrl;
> > +
> > +   schedule_work(>work);
> > +
> > +   return true;
> > +}
> > +
> > +static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
> > +   struct uvc_contro

Re: [PATCH v8 0/3] uvcvideo: asynchronous controls

[Guennadi Liakhovetski]

Hi Laurent,

6.5 weeks and counting. Can we please schedule a review of these patches 
for the next week? Not much time is left to make it for 4.19.

Thanks
Guennadi

On Thu, 31 May 2018, Guennadi Liakhovetski wrote:

> Hi Laurent,
> 
> More than 3 weeks since v8 has been posted. Seems like we've missed 4.18. 
> Could you please review them ASAP to make sure we merge them into 4.19?
> 
> Thanks
> Guennadi
> 
> On Tue, 8 May 2018, Guennadi Liakhovetski wrote:
> 
> > Added a patch to remove a redundant check, addressed Laurent's
> > comments.
> > 
> > Guennadi Liakhovetski (3):
> >   uvcvideo: remove a redundant check
> >   uvcvideo: send a control event when a Control Change interrupt arrives
> >   uvcvideo: handle control pipe protocol STALLs
> > 
> >  drivers/media/usb/uvc/uvc_ctrl.c   | 168 
> > ++---
> >  drivers/media/usb/uvc/uvc_status.c | 112 ++---
> >  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> >  drivers/media/usb/uvc/uvc_video.c  |  52 ++--
> >  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> >  include/uapi/linux/uvcvideo.h  |   2 +
> >  6 files changed, 302 insertions(+), 51 deletions(-)
> > 
> > -- 
> > 1.9.3
> > 
> > Thanks
> > Guennadi
> > 
> 

Re: [PATCH v8 0/3] uvcvideo: asynchronous controls

[Guennadi Liakhovetski]

Hi Laurent,

More than 3 weeks since v8 has been posted. Seems like we've missed 4.18. 
Could you please review them ASAP to make sure we merge them into 4.19?

Thanks
Guennadi

On Tue, 8 May 2018, Guennadi Liakhovetski wrote:

> Added a patch to remove a redundant check, addressed Laurent's
> comments.
> 
> Guennadi Liakhovetski (3):
>   uvcvideo: remove a redundant check
>   uvcvideo: send a control event when a Control Change interrupt arrives
>   uvcvideo: handle control pipe protocol STALLs
> 
>  drivers/media/usb/uvc/uvc_ctrl.c   | 168 
> ++---
>  drivers/media/usb/uvc/uvc_status.c | 112 ++---
>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
>  drivers/media/usb/uvc/uvc_video.c  |  52 ++--
>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
>  include/uapi/linux/uvcvideo.h  |   2 +
>  6 files changed, 302 insertions(+), 51 deletions(-)
> 
> -- 
> 1.9.3
> 
> Thanks
> Guennadi
> 

[PATCH v8 2/3] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

UVC defines a method of handling asynchronous controls, which sends a
USB packet over the interrupt pipe. This patch implements support for
such packets by sending a control event to the user. Since this can
involve USB traffic and, therefore, scheduling, this has to be done
in a work queue.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v8:

* avoid losing events by delaying the status URB resubmission until
  after completion of the current event
* extract control value calculation into __uvc_ctrl_get_value()
* do not proactively return EBUSY if the previous control hasn't
  completed yet, let the camera handle such cases
* multiple cosmetic changes

 drivers/media/usb/uvc/uvc_ctrl.c   | 166 ++---
 drivers/media/usb/uvc/uvc_status.c | 112 ++---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
 include/uapi/linux/uvcvideo.h  |   2 +
 5 files changed, 255 insertions(+), 44 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c
index 2a213c8..796f86a 100644
--- a/drivers/media/usb/uvc/uvc_ctrl.c
+++ b/drivers/media/usb/uvc/uvc_ctrl.c
@@ -20,6 +20,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 
@@ -971,12 +972,30 @@ static int uvc_ctrl_populate_cache(struct uvc_video_chain 
*chain,
return 0;
 }
 
+static s32 __uvc_ctrl_get_value(struct uvc_control_mapping *mapping,
+   const u8 *data)
+{
+   s32 value = mapping->get(mapping, UVC_GET_CUR, data);
+
+   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
+   struct uvc_menu_info *menu = mapping->menu_info;
+   unsigned int i;
+
+   for (i = 0; i < mapping->menu_count; ++i, ++menu) {
+   if (menu->value == value) {
+   value = i;
+   break;
+   }
+   }
+   }
+
+   return value;
+}
+
 static int __uvc_ctrl_get(struct uvc_video_chain *chain,
struct uvc_control *ctrl, struct uvc_control_mapping *mapping,
s32 *value)
 {
-   struct uvc_menu_info *menu;
-   unsigned int i;
int ret;
 
if ((ctrl->info.flags & UVC_CTRL_FLAG_GET_CUR) == 0)
@@ -993,18 +1012,8 @@ static int __uvc_ctrl_get(struct uvc_video_chain *chain,
ctrl->loaded = 1;
}
 
-   *value = mapping->get(mapping, UVC_GET_CUR,
-   uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
-
-   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
-   menu = mapping->menu_info;
-   for (i = 0; i < mapping->menu_count; ++i, ++menu) {
-   if (menu->value == *value) {
-   *value = i;
-   break;
-   }
-   }
-   }
+   *value = __uvc_ctrl_get_value(mapping,
+   uvc_ctrl_data(ctrl, UVC_CTRL_DATA_CURRENT));
 
return 0;
 }
@@ -1222,30 +1231,121 @@ static void uvc_ctrl_send_event(struct uvc_fh *handle,
 {
struct v4l2_subscribed_event *sev;
struct v4l2_event ev;
+   bool autoupdate;
 
if (list_empty(>ev_subs))
return;
 
+   if (!handle) {
+   autoupdate = true;
+   sev = list_first_entry(>ev_subs,
+  struct v4l2_subscribed_event, node);
+   handle = container_of(sev->fh, struct uvc_fh, vfh);
+   } else {
+   autoupdate = false;
+   }
+
uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
 
list_for_each_entry(sev, >ev_subs, node) {
if (sev->fh != >vfh ||
(sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
-   (changes & V4L2_EVENT_CTRL_CH_FLAGS))
+   (changes & V4L2_EVENT_CTRL_CH_FLAGS) || autoupdate)
v4l2_event_queue_fh(sev->fh, );
}
 }
 
-static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
-   struct uvc_control *master, u32 slave_id,
-   const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
+static void __uvc_ctrl_send_slave_event(struct uvc_fh *handle,
+   struct uvc_video_chain *chain, struct uvc_control *master, u32 slave_id)
 {
struct uvc_control_mapping *mapping = NULL;
struct uvc_control *ctrl = NULL;
u32 changes = V4L2_EVENT_CTRL_CH_FLAGS;
-   unsigned int i;
s32 val = 0;
 
+   __uvc_find_control(master->entity, slave_id, , , 0);
+   if (ctrl == NULL)
+   return;
+
+   if (__uvc_ctrl_get(handle ? handle->chain : chain, ctrl, mapping,
+  ) == 0)
+   changes |= V4L2_EVENT_CTRL_CH_VALUE;
+
+   uvc_ctrl_sen

[PATCH v8 3/3] uvcvideo: handle control pipe protocol STALLs

[Guennadi Liakhovetski]

When a command ends up in a STALL on the control pipe, use the Request
Error Code control to provide a more precise error information to the
user. For example, if a camera is still busy processing a control,
when the same or an interrelated control set request arrives, the
camera can react with a STALL and then return the "Not ready" status
in response to a UVC_VC_REQUEST_ERROR_CODE_CONTROL command. With this
patch the user would then get an EBUSY error code instead of a
generic EPIPE.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v8:

* restrict UVC_VC_REQUEST_ERROR_CODE_CONTROL to the control interface
* fix the wIndex value
* improve error codes
* cosmetic changes

 drivers/media/usb/uvc/uvc_video.c | 52 ++-
 1 file changed, 46 insertions(+), 6 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_video.c 
b/drivers/media/usb/uvc/uvc_video.c
index aa0082f..ce65cd6 100644
--- a/drivers/media/usb/uvc/uvc_video.c
+++ b/drivers/media/usb/uvc/uvc_video.c
@@ -73,17 +73,57 @@ int uvc_query_ctrl(struct uvc_device *dev, u8 query, u8 
unit,
u8 intfnum, u8 cs, void *data, u16 size)
 {
int ret;
+   u8 error;
+   u8 tmp;
 
ret = __uvc_query_ctrl(dev, query, unit, intfnum, cs, data, size,
UVC_CTRL_CONTROL_TIMEOUT);
-   if (ret != size) {
-   uvc_printk(KERN_ERR, "Failed to query (%s) UVC control %u on "
-   "unit %u: %d (exp. %u).\n", uvc_query_name(query), cs,
-   unit, ret, size);
-   return -EIO;
+   if (likely(ret == size))
+   return 0;
+
+   uvc_printk(KERN_ERR,
+  "Failed to query (%s) UVC control %u on unit %u: %d (exp. 
%u).\n",
+  uvc_query_name(query), cs, unit, ret, size);
+
+   if (ret != -EPIPE)
+   return ret;
+
+   tmp = *(u8 *)data;
+
+   ret = __uvc_query_ctrl(dev, UVC_GET_CUR, 0, intfnum,
+  UVC_VC_REQUEST_ERROR_CODE_CONTROL, data, 1,
+  UVC_CTRL_CONTROL_TIMEOUT);
+
+   error = *(u8 *)data;
+   *(u8 *)data = tmp;
+
+   if (ret != 1)
+   return ret < 0 ? ret : -EPIPE;
+
+   uvc_trace(UVC_TRACE_CONTROL, "Control error %u\n", error);
+
+   switch (error) {
+   case 0:
+   /* Cannot happen - we received a STALL */
+   return -EPIPE;
+   case 1: /* Not ready */
+   return -EBUSY;
+   case 2: /* Wrong state */
+   return -EILSEQ;
+   case 3: /* Power */
+   return -EREMOTE;
+   case 4: /* Out of range */
+   return -ERANGE;
+   case 5: /* Invalid unit */
+   case 6: /* Invalid control */
+   case 7: /* Invalid Request */
+   case 8: /* Invalid value within range */
+   return -EINVAL;
+   default: /* reserved or unknown */
+   break;
}
 
-   return 0;
+   return -EPIPE;
 }
 
 static void uvc_fixup_video_ctrl(struct uvc_streaming *stream,
-- 
1.9.3

[PATCH v8 1/3] uvcvideo: remove a redundant check

[Guennadi Liakhovetski]

Event subscribers cannot have a NULL file handle. They are only added
at a single location in the code, and the .fh pointer is used without
checking there.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_ctrl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c
index a36b4fb..2a213c8 100644
--- a/drivers/media/usb/uvc/uvc_ctrl.c
+++ b/drivers/media/usb/uvc/uvc_ctrl.c
@@ -1229,9 +1229,9 @@ static void uvc_ctrl_send_event(struct uvc_fh *handle,
uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
 
list_for_each_entry(sev, >ev_subs, node) {
-   if (sev->fh && (sev->fh != >vfh ||
+   if (sev->fh != >vfh ||
(sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
-   (changes & V4L2_EVENT_CTRL_CH_FLAGS)))
+   (changes & V4L2_EVENT_CTRL_CH_FLAGS))
v4l2_event_queue_fh(sev->fh, );
}
 }
-- 
1.9.3

[PATCH v8 0/3] uvcvideo: asynchronous controls

[Guennadi Liakhovetski]

Added a patch to remove a redundant check, addressed Laurent's
comments.

Guennadi Liakhovetski (3):
  uvcvideo: remove a redundant check
  uvcvideo: send a control event when a Control Change interrupt arrives
  uvcvideo: handle control pipe protocol STALLs

 drivers/media/usb/uvc/uvc_ctrl.c   | 168 ++---
 drivers/media/usb/uvc/uvc_status.c | 112 ++---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvc_video.c  |  52 ++--
 drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
 include/uapi/linux/uvcvideo.h  |   2 +
 6 files changed, 302 insertions(+), 51 deletions(-)

-- 
1.9.3

Thanks
Guennadi

Re: [PATCH v7 1/2] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

One more comment-to-comment:

On Mon, 7 May 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday, 10 April 2018 14:31:35 EEST Guennadi Liakhovetski wrote:
> > On Fri, 23 Mar 2018, Laurent Pinchart wrote:
> > > On Friday, 23 March 2018 11:24:00 EET Laurent Pinchart wrote:
> > >> From: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> > >> 
> > >> UVC defines a method of handling asynchronous controls, which sends a
> > >> USB packet over the interrupt pipe. This patch implements support for
> > >> such packets by sending a control event to the user. Since this can
> > >> involve USB traffic and, therefore, scheduling, this has to be done
> > >> in a work queue.
> > >> 
> > >> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >> ---
> > >> 
> > >>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 +++---
> > >>  drivers/media/usb/uvc/uvc_status.c | 111 ++---
> > >>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> > >>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> > >>  include/uapi/linux/uvcvideo.h  |   2 +
> > >>  5 files changed, 269 insertions(+), 29 deletions(-)

[snip]

> > >> diff --git a/drivers/media/usb/uvc/uvc_status.c
> > >> b/drivers/media/usb/uvc/uvc_status.c index 7b710410584a..d1d83aed6a1d
> > >> 100644
> > >> --- a/drivers/media/usb/uvc/uvc_status.c
> > >> +++ b/drivers/media/usb/uvc/uvc_status.c

[snip]

> > >> +ctrl = uvc_event_entity_ctrl(entity,
> > >> + 
> > >> status->bSelector);
> > >> +/*
> > >> + * Some buggy cameras send asynchronous 
> > >> Control
> > >> + * Change events for control, other 
> > >> than the
> > >> + * ones, that had been changed, even 
> > >> though the
> > >> + * AutoUpdate flag isn't set for the 
> > >> control.
> > >> + */
> > > 
> > > That's lots of commas, I'm not sure what you mean here. Are there cameras
> > > that send event for controls that haven't changed ? Or cameras that send
> > > events for controls that don't have the auto-update flag set ? Do you
> > > know what cameras are affected ?
> > 
> > I meant a case like
> > 
> > set_control(x=X)
> > interrupt(x=X)
> > interrupt(y=Y)
> > 
> > where y is a different control and it doesn't have an auto-update flag
> > set. I think those were some early versions of our cameras, but as far as
> > I can see, we need the check anyway for autoupdate controls, so, I can
> > just remove the comment.
> 
> OK. But now that I read the comment again, how is it related to the check ? 
> Why do you need ctrl->handle->chain == *chain ? Isn't that only a partial 
> guard for the case above, as both x and y could be part of the same chain ?

The uvc_event_entity_ctrl() function checks the selector, so, a selector + 
chain should be unique?

Thanks
Guennadi

Re: [PATCH v7 1/2] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the replies. One follow-up question:

On Mon, 7 May 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday, 10 April 2018 14:31:35 EEST Guennadi Liakhovetski wrote:
> > On Fri, 23 Mar 2018, Laurent Pinchart wrote:
> > > On Friday, 23 March 2018 11:24:00 EET Laurent Pinchart wrote:
> > >> From: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> > >> 
> > >> UVC defines a method of handling asynchronous controls, which sends a
> > >> USB packet over the interrupt pipe. This patch implements support for
> > >> such packets by sending a control event to the user. Since this can
> > >> involve USB traffic and, therefore, scheduling, this has to be done
> > >> in a work queue.
> > >> 
> > >> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >> ---
> > >> 
> > >>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 +++---
> > >>  drivers/media/usb/uvc/uvc_status.c | 111 ++---
> > >>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> > >>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> > >>  include/uapi/linux/uvcvideo.h  |   2 +
> > >>  5 files changed, 269 insertions(+), 29 deletions(-)
> > >> 
> > >> diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > >> b/drivers/media/usb/uvc/uvc_ctrl.c index 4042cbdb721b..f4773c56438c
> > >> 100644
> > >> --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > >> +++ b/drivers/media/usb/uvc/uvc_ctrl.c

[snip]

> > >> +void uvc_ctrl_status_event(struct uvc_video_chain *chain,
> > >> +   struct uvc_control *ctrl, u8 *data, size_t 
> > >> len)
> > >> +{
> > >> +struct uvc_device *dev = chain->dev;
> > >> +struct uvc_ctrl_work *w = >async_ctrl;
> > >> +
> > >> +if (list_empty(>info.mappings))
> > >> +return;
> > >> +
> > >> +spin_lock(>lock);
> > >> +if (w->data)
> > >> +/* A previous event work hasn't run yet, we lose 1 
> > >> event */
> > >> +kfree(w->data);
> > > 
> > > I really don't like losing events :/
> > 
> > Well, I'm not sure whether having no available status URBs isn't
> > equivalent to losing events, but if you prefer that - no problem.
> > 
> > >> +w->data = kmalloc(len, GFP_ATOMIC);
> > > 
> > > GFP_ATOMIC allocation isn't very nice either.
> > > 
> > > How about if we instead delayed resubmitting the status URB until the
> > > event is fully processed by the work queue ? That way we wouldn't lose
> > > events, we wouldn't need memory allocation in atomic context, and if the
> > > work queue becomes a bottleneck we could even queue multiple status URBs
> > > and easily add them to a list for processing by the work queue.
> > 
> > You mean only for control status events? Can do, sure.
> 
> I mean the status endpoint URB in general, so this would affect both control 
> events and button events.

I don't think any of my UVC cameras have such a button, do you have any of 
those? I'd rather not change something, that I cannot test myself and 
cannot have tested. I could leave the button processing as is and only 
change the URB submission path for control change events?

Thanks
Guennadi

Re: [PATCH v7 1/2] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

On Tue, 10 Apr 2018, Guennadi Liakhovetski wrote:

[snip]

> > > @@ -1488,6 +1591,25 @@ int uvc_ctrl_set(struct uvc_video_chain *chain,
> > >   return -EINVAL;
> > >   if (!(ctrl->info.flags & UVC_CTRL_FLAG_SET_CUR))
> > >   return -EACCES;
> > > + if (ctrl->info.flags & UVC_CTRL_FLAG_ASYNCHRONOUS) {
> > > + if (ctrl->handle)
> > > + /*
> > > +  * We have already sent this control to the camera
> > > +  * recently and are currently waiting for a completion
> > > +  * notification. The camera might already have completed
> > > +  * its processing and is ready to accept a new control
> > > +  * or it's still busy processing. If we send a new
> > > +  * instance of this control now, in the former case the
> > > +  * camera will process this one too and we'll get
> > > +  * completions for both, but we will only deliver an
> > > +  * event for one of them back to the user. In the latter
> > > +  * case the camera will reply with a STALL. It's easier
> > > +  * and more reliable to return an error now and let the
> > > +  * user retry.
> > > +  */
> > > + return -EBUSY;
> > > + ctrl->handle = handle;
> > 
> > This part worries me. If the control change event isn't received for any 
> > reason (such as a buggy device for instance, or uvc_ctrl_status_event() 
> > being 
> > called with the previous event not processed yet), the control will stay 
> > busy 
> > forever.
> > 
> > I see two approaches to fix this. One would be to forward all received 
> > control 
> > change events to all file handles unconditionally and remove the handle 
> > field 
> > from the uvc_control structure.
> 
> How is this a solution? A case of senging a repeated control to the camera 
> and causing a STALL would still be possible. If you prefer STALLs, you 
> could just remove this check here.
> 
> > Another one would be to add a timeout, storing 
> > the time at which the control has been set in the uvc_control structure, 
> > and 
> > checking whether the time difference exceeds a fixed timeout here. We could 
> > also combine the two, replacing the handle field with a timestamp field.
> 
> Don't think you can remove the handle field, there are a couple of things, 
> that need it, also you'll have to send events to all listeners, including 
> the thread, that has sent the control, which contradicts the API? Assuming 
> it hasn't set the V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK flag.
> 
> I can add a timeout, even though that doesn't seem to be very clean to me. 
> According to the UVC standard some controls can take a while to complete, 
> possibly seconds. How long would you propose that timeout to be?

How about just not checking when setting control and letting the camera 
decide? That's what the STALL mechanism is there for. The only advantage 
of using this is to provide a short-cut when we're "sure," that the 
hardware isn't ready to take a new request yet, but if implementing such a 
shortcut becomes too cumbersome, we can give control back to the hardware.

Thanks
Guennadi

Re: [PATCH v7 2/2] uvcvideo: handle control pipe protocol STALLs

[Guennadi Liakhovetski]

Hi Laurent,

I've addressed those your comments, that I had no questions for, but I'm 
waiting for your replies to my clarification requests to finalise and post 
the next revision of the patches.

Thanks
Guennadi

On Wed, 11 Apr 2018, Guennadi Liakhovetski wrote:

> Hi Laurent,
> 
> First just answers to your questions:
> 
> On Sat, 7 Apr 2018, Laurent Pinchart wrote:
> 
> > Hi Guennadi,
> > 
> > Thank you for the patch.
> > 
> > On Friday, 23 March 2018 11:24:01 EEST Laurent Pinchart wrote:
> > > From: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> > > 
> > > When a command ends up in a STALL on the control pipe, use the Request
> > > Error Code control to provide a more precise error information to the
> > > user.
> > 
> > This is the kind of change that I believe is completely right, but that 
> > nonetheless worries me. All the years I've spent working with UVC webcams 
> > taught me that lots of cameras have buggy firmware, and that any change in 
> > how 
> > the host driver issues requests can have dire consequences for users. This 
> > is 
> > especially true when the host driver issues a request that was never issued 
> > before.
> > 
> > The UVC specification also doesn't clearly tell whether the request error 
> > code 
> > control is mandatory for a device to implement. My interpretation of the 
> > specification is that it is, but it would have been nice for the 
> > specification 
> > to be explicit on this topic. Have you encountered devices that don't 
> > implement this control ?
> 
> No, I haven't. But I haven't explicitly tested too many either. This patch 
> would only issue that control if a STALL condition is detected, and 
> normally that doesn't happen.
> 
> > This being said, I don't claim that would be a reason not to use the 
> > request 
> > error code control as proposed by this patch, but I would feel less worried 
> > if 
> > I knew whether the Windows driver used that control as well. If it does 
> > there's a high chance that cameras will implement it correctly, while if it 
> > doesn't we will most certainly hit bugs with several cameras. I was thus 
> > wondering if in the course of your UVC developments you would have happened 
> > to 
> > find out whether this control is used by Windows.
> 
> No, sorry, I never tried to analyse the behaviour of the Windows UVC 
> driver.
> 
> > I would also like to know a bit more about the purpose of this patch. 
> > Logging 
> > the error code is certainly useful for diagnosis purpose. Have you also 
> > found 
> > it useful to report different errors back to userspace ? If so, could you 
> > explain your use cases ?
> 
> Yes, with this patch the user-space can with certainty identify the reason 
> of a stall, specifically you would know, when the camera is in a "not 
> ready" state. With the previous patch in this series the driver shouldn't 
> be sending a second SET_CUR command to the same control, before the first 
> one has completed, but on some cameras different controls can also be 
> interrelated. In such a case trying to set a different control, while a 
> previous one is still being processed, can also cause a STALL with a "Not 
> ready" error state.
> 
> > > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > > ---
> > >  drivers/media/usb/uvc/uvc_video.c | 59 
> > > 
> > >  1 file changed, 53 insertions(+), 6 deletions(-)
> > > 
> > > diff --git a/drivers/media/usb/uvc/uvc_video.c
> > > b/drivers/media/usb/uvc/uvc_video.c index aa0082fe5833..eb9e04a59427 
> > > 100644
> > > --- a/drivers/media/usb/uvc/uvc_video.c
> > > +++ b/drivers/media/usb/uvc/uvc_video.c
> > > @@ -34,15 +34,59 @@ static int __uvc_query_ctrl(struct uvc_device *dev, u8
> > > query, u8 unit, u8 intfnum, u8 cs, void *data, u16 size,
> > >   int timeout)
> > >  {
> > > - u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE;
> > > + u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE, tmp, error;
> > 
> > Would you mind declaring one variable per line to match the style of the 
> > rest 
> > of the driver ?
> 
> In fact I would, but well... ;-)
> 
> > >   unsigned int pipe;
> > > + int ret;
> > > 
> > >   pipe = (query & 0x80) ? usb_rcvctrlpipe(dev->udev, 0)
> > > : usb_sndctrlpipe(dev->udev, 0);
&g

Re: a 4.16 kernel with Debian 9.4 "stretch" causes a log explosion

[Guennadi Liakhovetski]

On Wed, 11 Apr 2018, Kieran Bingham wrote:

> Hi Guennadi,
> 
> On 11/04/18 18:06, Guennadi Liakhovetski wrote:
> 
>  
> 
> >>>>
> >>>> Just figured out this commit
> >>>>
> >>>> From: Edgar Thier <i...@edgarthier.net>
> >>>> Date: Thu, 12 Oct 2017 03:54:17 -0400
> >>>> Subject: [PATCH] media: uvcvideo: Apply flags from device to actual 
> >>>> properties
> >>>>
> >>>> as the culprit. Without it everything is back to normal.
> >>>
> >>> I've already investigated and fixed this:
> >>>
> >>> Please apply:
> >>>   https://patchwork.kernel.org/patch/10299735/
> > 
> > Great, thanks! That seems to fix my problem.
> 
> Fantastic. I'm glad it helped.
> 
>  - Can I call that a Tested-by: ? :-D

Now you officially can - I replied to the patch.

Thanks
Guennadi

> 
> Regards
> 
> Kieran
> 

Re: [PATCH] media: uvcvideo: Prevent setting unavailable flags

[Guennadi Liakhovetski]

Hi Kieran,

Thanks for the patch, it fixed a problem I was having with media master, 
working with a Logitech UVC 1.5 camera.

On Wed, 21 Mar 2018, Kieran Bingham wrote:

> The addition of an extra operation to use the GET_INFO command
> overwrites all existing flags from the uvc_ctrls table. This includes
> setting all controls as supporting  GET_MIN, GET_MAX, GET_RES, and
> GET_DEF regardless of whether they do or not.
> 
> Move the initialisation of these control capabilities directly to the
> uvc_ctrl_fill_xu_info() call where they were originally located in that
> use case, and ensure that the new functionality in uvc_ctrl_get_flags()
> will only set flags based on their reported capability from the GET_INFO
> call.
> 
> Fixes: 859086ae3636 ("media: uvcvideo: Apply flags from device to actual
> properties")
> 
> Signed-off-by: Kieran Bingham <kieran.bing...@ideasonboard.com>

Tested-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>

Thanks
Guennadi

> ---
>  drivers/media/usb/uvc/uvc_ctrl.c | 17 +
>  1 file changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/usb/uvc/uvc_ctrl.c 
> b/drivers/media/usb/uvc/uvc_ctrl.c
> index 1daf444371be..4042cbdb721b 100644
> --- a/drivers/media/usb/uvc/uvc_ctrl.c
> +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> @@ -1607,14 +1607,12 @@ static int uvc_ctrl_get_flags(struct uvc_device *dev,
>   ret = uvc_query_ctrl(dev, UVC_GET_INFO, ctrl->entity->id, dev->intfnum,
>info->selector, data, 1);
>   if (!ret)
> - info->flags = UVC_CTRL_FLAG_GET_MIN | UVC_CTRL_FLAG_GET_MAX
> - | UVC_CTRL_FLAG_GET_RES | UVC_CTRL_FLAG_GET_DEF
> - | (data[0] & UVC_CONTROL_CAP_GET ?
> -UVC_CTRL_FLAG_GET_CUR : 0)
> - | (data[0] & UVC_CONTROL_CAP_SET ?
> -UVC_CTRL_FLAG_SET_CUR : 0)
> - | (data[0] & UVC_CONTROL_CAP_AUTOUPDATE ?
> -UVC_CTRL_FLAG_AUTO_UPDATE : 0);
> + info->flags |= (data[0] & UVC_CONTROL_CAP_GET ?
> + UVC_CTRL_FLAG_GET_CUR : 0)
> + |  (data[0] & UVC_CONTROL_CAP_SET ?
> + UVC_CTRL_FLAG_SET_CUR : 0)
> + |  (data[0] & UVC_CONTROL_CAP_AUTOUPDATE ?
> + UVC_CTRL_FLAG_AUTO_UPDATE : 0);
>  
>   kfree(data);
>   return ret;
> @@ -1689,6 +1687,9 @@ static int uvc_ctrl_fill_xu_info(struct uvc_device *dev,
>  
>   info->size = le16_to_cpup((__le16 *)data);
>  
> + info->flags = UVC_CTRL_FLAG_GET_MIN | UVC_CTRL_FLAG_GET_MAX
> + | UVC_CTRL_FLAG_GET_RES | UVC_CTRL_FLAG_GET_DEF;
> +
>   ret = uvc_ctrl_get_flags(dev, ctrl, info);
>   if (ret < 0) {
>   uvc_trace(UVC_TRACE_CONTROL,
> -- 
> 2.7.4
> 

Re: a 4.16 kernel with Debian 9.4 "stretch" causes a log explosion

[Guennadi Liakhovetski]

Hi Kieran,

On Wed, 11 Apr 2018, Kieran Bingham wrote:

> Hi Guennadi,
> 
> On 11/04/18 17:17, Kieran Bingham wrote:
> > Hi Guennadi,
> > 
> > On 11/04/18 10:56, Guennadi Liakhovetski wrote:
> >> Hi Laurent,
> >>
> >> Not sure whether that's a kernel or a user-space problem, but UVC related 
> >> anyway. I've got a UVC 1.5 (!) Logitech camera, that used to work fine 
> >> with earlier kernels. I now installed "media 4.16" and saw, that the 
> >> kernel log was filling with messages like
> >>
> >> uvcvideo: Failed to query (GET_MIN) UVC control 2 on unit 1: -32 (exp. 1).
> >>
> >> The expected /dev/video[01] nodes were not created correctly, and the 
> >> hard-drive was getting full very quickly. The latter was happening because 
> >> the the /var/log/uvcdynctrl-udev.log file was growing. A truncated sample 
> >> is attached. At its bottom you see messages
> >>
> >> [libwebcam] Warning: The driver behind device video0 has a slightly buggy 
> >> implementation
> >>   of the V4L2_CTRL_FLAG_NEXT_CTRL flag. It does not return the next higher
> >>   control ID if a control query fails. A workaround has been enabled.
> >>
> >> repeating, which continues even if the camera is unplugged. The kernel is 
> >> the head of the master branch of git://linuxtv.org/media_tree.git
> >>
> >> Just figured out this commit
> >>
> >> From: Edgar Thier <i...@edgarthier.net>
> >> Date: Thu, 12 Oct 2017 03:54:17 -0400
> >> Subject: [PATCH] media: uvcvideo: Apply flags from device to actual 
> >> properties
> >>
> >> as the culprit. Without it everything is back to normal.
> > 
> > I've already investigated and fixed this:
> > 
> > Please apply:
> > https://patchwork.kernel.org/patch/10299735/

Great, thanks! That seems to fix my problem.

Regards
Guennadi

> > 
> > You stated that this is showing up on a v4.16 kernel ... but as far as I'm 
> > aware
> > - this feature shouldn't make it in until v4.17. Are you using linux-next 
> > or a
> > media/master or such ?
> 
> Aha - never mind - I just re-read your message.
> 
> I expect the fix to make it into the media mainline when Mauro pulls from
> Laurent? So I'm sure it will find its way soon.
> 
> --
> Cheers
> 
> Kieran
> 
> 
> > 
> > Regards
> > 
> > Kieran
> > 
> >> Thanks
> >> Guennadi
> >>
> 

Re: [PATCH v7 2/2] uvcvideo: handle control pipe protocol STALLs

[Guennadi Liakhovetski]

Hi Laurent,

First just answers to your questions:

On Sat, 7 Apr 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> On Friday, 23 March 2018 11:24:01 EEST Laurent Pinchart wrote:
> > From: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> > 
> > When a command ends up in a STALL on the control pipe, use the Request
> > Error Code control to provide a more precise error information to the
> > user.
> 
> This is the kind of change that I believe is completely right, but that 
> nonetheless worries me. All the years I've spent working with UVC webcams 
> taught me that lots of cameras have buggy firmware, and that any change in 
> how 
> the host driver issues requests can have dire consequences for users. This is 
> especially true when the host driver issues a request that was never issued 
> before.
> 
> The UVC specification also doesn't clearly tell whether the request error 
> code 
> control is mandatory for a device to implement. My interpretation of the 
> specification is that it is, but it would have been nice for the 
> specification 
> to be explicit on this topic. Have you encountered devices that don't 
> implement this control ?

No, I haven't. But I haven't explicitly tested too many either. This patch 
would only issue that control if a STALL condition is detected, and 
normally that doesn't happen.

> This being said, I don't claim that would be a reason not to use the request 
> error code control as proposed by this patch, but I would feel less worried 
> if 
> I knew whether the Windows driver used that control as well. If it does 
> there's a high chance that cameras will implement it correctly, while if it 
> doesn't we will most certainly hit bugs with several cameras. I was thus 
> wondering if in the course of your UVC developments you would have happened 
> to 
> find out whether this control is used by Windows.

No, sorry, I never tried to analyse the behaviour of the Windows UVC 
driver.

> I would also like to know a bit more about the purpose of this patch. Logging 
> the error code is certainly useful for diagnosis purpose. Have you also found 
> it useful to report different errors back to userspace ? If so, could you 
> explain your use cases ?

Yes, with this patch the user-space can with certainty identify the reason 
of a stall, specifically you would know, when the camera is in a "not 
ready" state. With the previous patch in this series the driver shouldn't 
be sending a second SET_CUR command to the same control, before the first 
one has completed, but on some cameras different controls can also be 
interrelated. In such a case trying to set a different control, while a 
previous one is still being processed, can also cause a STALL with a "Not 
ready" error state.

> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> >  drivers/media/usb/uvc/uvc_video.c | 59 
> >  1 file changed, 53 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/media/usb/uvc/uvc_video.c
> > b/drivers/media/usb/uvc/uvc_video.c index aa0082fe5833..eb9e04a59427 100644
> > --- a/drivers/media/usb/uvc/uvc_video.c
> > +++ b/drivers/media/usb/uvc/uvc_video.c
> > @@ -34,15 +34,59 @@ static int __uvc_query_ctrl(struct uvc_device *dev, u8
> > query, u8 unit, u8 intfnum, u8 cs, void *data, u16 size,
> > int timeout)
> >  {
> > -   u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE;
> > +   u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE, tmp, error;
> 
> Would you mind declaring one variable per line to match the style of the rest 
> of the driver ?

In fact I would, but well... ;-)

> > unsigned int pipe;
> > +   int ret;
> > 
> > pipe = (query & 0x80) ? usb_rcvctrlpipe(dev->udev, 0)
> >   : usb_sndctrlpipe(dev->udev, 0);
> > 
> > type |= (query & 0x80) ? USB_DIR_IN : USB_DIR_OUT;
> > 
> > -   return usb_control_msg(dev->udev, pipe, query, type, cs << 8,
> > +   ret = usb_control_msg(dev->udev, pipe, query, type, cs << 8,
> > unit << 8 | intfnum, data, size, timeout);
> > +
> 
> Nitpicking, you can remove the blank line here.
> 
> > +   if (ret != -EPIPE)
> > +   return ret;
> > +
> > +   tmp = *(u8 *)data;
> > +
> > +   pipe = usb_rcvctrlpipe(dev->udev, 0);
> > +   type = USB_TYPE_CLASS | USB_RECIP_INTERFACE | USB_DIR_IN;
> > +   ret = usb_control_msg(dev->udev, pipe, UVC_GET_CUR, type,
> > + UVC_VC_REQUEST_ERROR_CODE_CONTROL << 8,
> > +  

a 4.16 kernel with Debian 9.4 "stretch" causes a log explosion

[Guennadi Liakhovetski]

Hi Laurent,

Not sure whether that's a kernel or a user-space problem, but UVC related 
anyway. I've got a UVC 1.5 (!) Logitech camera, that used to work fine 
with earlier kernels. I now installed "media 4.16" and saw, that the 
kernel log was filling with messages like

uvcvideo: Failed to query (GET_MIN) UVC control 2 on unit 1: -32 (exp. 1).

The expected /dev/video[01] nodes were not created correctly, and the 
hard-drive was getting full very quickly. The latter was happening because 
the the /var/log/uvcdynctrl-udev.log file was growing. A truncated sample 
is attached. At its bottom you see messages

[libwebcam] Warning: The driver behind device video0 has a slightly buggy 
implementation
  of the V4L2_CTRL_FLAG_NEXT_CTRL flag. It does not return the next higher
  control ID if a control query fails. A workaround has been enabled.

repeating, which continues even if the camera is unplugged. The kernel is 
the head of the master branch of git://linuxtv.org/media_tree.git

Just figured out this commit

From: Edgar Thier 
Date: Thu, 12 Oct 2017 03:54:17 -0400
Subject: [PATCH] media: uvcvideo: Apply flags from device to actual properties

as the culprit. Without it everything is back to normal.

Thanks
Guennadi


==
uvcdynctrl script version 0.3 running from '/lib/udev/uvcdynctrl'
==
uvcdynctrl script version 0.3 running from '/lib/udev/uvcdynctrl'
Triggered at Wed Apr 11 09:08:26 CEST 2018

ACTION='add'
DEVLINKS='/dev/v4l/by-path/pci-:00:14.0-usb-0:1:1.0-video-index0 
/dev/v4l/by-id/usb-046d_Logitech_Webcam_C930e_816415EE-video-index0'
DEVNAME='/dev/video0'
DEVPATH='/devices/pci:00/:00:14.0/usb2/2-1/2-1:1.0/video4linux/video0'
ID_BUS='usb'
ID_FOR_SEAT='video4linux-pci-_00_14_0-usb-0_1_1_0'
ID_MODEL='Logitech_Webcam_C930e'
ID_MODEL_ENC='Logitech\x20Webcam\x20C930e'
ID_MODEL_ID='0843'
ID_PATH='pci-:00:14.0-usb-0:1:1.0'
ID_PATH_TAG='pci-_00_14_0-usb-0_1_1_0'
ID_REVISION='0013'
ID_SERIAL='046d_Logitech_Webcam_C930e_816415EE'
ID_SERIAL_SHORT='816415EE'
ID_TYPE='video'
ID_USB_DRIVER='uvcvideo'
ID_USB_INTERFACES=':0e0100:0e0200:010100:010200:'
ID_USB_INTERFACE_NUM='00'
ID_V4L_CAPABILITIES=':capture:'
ID_V4L_PRODUCT='Logitech Webcam C930e'
ID_V4L_VERSION='2'
ID_VENDOR='046d'
ID_VENDOR_ENC='046d'
ID_VENDOR_ID='046d'
IFS='   
'
MAJOR='81'
MINOR='0'
OPTIND='1'
PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PPID='853'
PS1='# '
PS2='> '
PS4='+ '
PWD='/'
SEQNUM='1895'
SUBSYSTEM='video4linux'
TAGS=':uaccess:seat:'
USEC_INITIALIZED='175470181'
debug='1'
logfile='/var/log/uvcdynctrl-udev.log'
uvcdynctrlpath='uvcdynctrl'
version='0.3'

Triggered at Wed Apr 11 09:08:26 CEST 2018
VID of new device: '046d'
PID of new device: '0843'

Executing command: 'uvcdynctrl -d /dev/video0 --addctrl=046d:0843'
ACTION='add'
DEVLINKS='/dev/v4l/by-id/usb-046d_Logitech_Webcam_C930e_816415EE-video-index1 
/dev/v4l/by-path/pci-:00:14.0-usb-0:1:1.0-video-index1'
DEVNAME='/dev/video1'
DEVPATH='/devices/pci:00/:00:14.0/usb2/2-1/2-1:1.0/video4linux/video1'
ID_BUS='usb'
ID_FOR_SEAT='video4linux-pci-_00_14_0-usb-0_1_1_0'
ID_MODEL='Logitech_Webcam_C930e'
ID_MODEL_ENC='Logitech\x20Webcam\x20C930e'
ID_MODEL_ID='0843'
ID_PATH='pci-:00:14.0-usb-0:1:1.0'
ID_PATH_TAG='pci-_00_14_0-usb-0_1_1_0'
ID_REVISION='0013'
ID_SERIAL='046d_Logitech_Webcam_C930e_816415EE'
ID_SERIAL_SHORT='816415EE'
ID_TYPE='video'
ID_USB_DRIVER='uvcvideo'
ID_USB_INTERFACES=':0e0100:0e0200:010100:010200:'
ID_USB_INTERFACE_NUM='00'
ID_V4L_CAPABILITIES=':capture:'
ID_V4L_PRODUCT='Logitech Webcam C930e'
ID_V4L_VERSION='2'
ID_VENDOR='046d'
ID_VENDOR_ENC='046d'
ID_VENDOR_ID='046d'
IFS='   
'
MAJOR='81'
MINOR='1'
OPTIND='1'
PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PPID='858'
PS1='# '
PS2='> '
PS4='+ '
PWD='/'
SEQNUM='1896'
SUBSYSTEM='video4linux'
TAGS=':uaccess:seat:'
USEC_INITIALIZED='175470183'
debug='1'
logfile='/var/log/uvcdynctrl-udev.log'
uvcdynctrlpath='uvcdynctrl'
version='0.3'

VID of new device: '046d'
PID of new device: '0843'
Executing command: 'uvcdynctrl -d /dev/video1 --addctrl=046d:0843'
[libwebcam] Warning: The driver behind device video0 has a slightly buggy 
implementation
  of the V4L2_CTRL_FLAG_NEXT_CTRL flag. It does not return the next higher
  control ID if a control query fails. A workaround has been enabled.
[libwebcam] Warning: The driver behind device video0 has a slightly buggy 
implementation
  of the V4L2_CTRL_FLAG_NEXT_CTRL flag. It does not return the next higher
  control ID if a control query fails. A workaround has been enabled.
[libwebcam] Warning: The driver behind device video0 has a slightly buggy 
implementation
  of the V4L2_CTRL_FLAG_NEXT_CTRL flag. It does not return the next higher
  control ID if a control query fails. A workaround has been enabled.
[libwebcam] Warning: The 

Re: [PATCH v7 1/2] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

First a couple of replies to your questions.

On Fri, 23 Mar 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> On Friday, 23 March 2018 11:24:00 EET Laurent Pinchart wrote:
> > From: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> > 
> > UVC defines a method of handling asynchronous controls, which sends a
> > USB packet over the interrupt pipe. This patch implements support for
> > such packets by sending a control event to the user. Since this can
> > involve USB traffic and, therefore, scheduling, this has to be done
> > in a work queue.
> > 
> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> >  drivers/media/usb/uvc/uvc_ctrl.c   | 166 ++
> >  drivers/media/usb/uvc/uvc_status.c | 111 ++---
> >  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> >  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> >  include/uapi/linux/uvcvideo.h  |   2 +
> >  5 files changed, 269 insertions(+), 29 deletions(-)
> > 
> > diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > b/drivers/media/usb/uvc/uvc_ctrl.c index 4042cbdb721b..f4773c56438c 100644
> > --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> > @@ -20,6 +20,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> > 
> > @@ -1222,30 +1223,134 @@ static void uvc_ctrl_send_event(struct uvc_fh
> > *handle, {
> > struct v4l2_subscribed_event *sev;
> > struct v4l2_event ev;
> > +   bool autoupdate;
> > 
> > if (list_empty(>ev_subs))
> > return;
> > 
> > +   if (!handle) {
> 
> In which circumstances does this happen ? Is it when the device reports a 
> control change event without an prior control set ? Have you seen that 
> happening in practice ?

This happens with autoupdate controls. Yes, I've seen this happen.

> 
> > +   autoupdate = true;
> > +   sev = list_first_entry(>ev_subs,
> > +  struct v4l2_subscribed_event, node);
> > +   handle = container_of(sev->fh, struct uvc_fh, vfh);
> 
> There's a check below that guards against sev->fh being NULL. Could sev->fh 
> be 
> NULL here ?

I cannot see how that can happen, but if you see such a possibility, I can 
add a check.

> > +   } else {
> > +   autoupdate = false;
> > +   }
> > +
> > uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
> > 
> > list_for_each_entry(sev, >ev_subs, node) {
> > if (sev->fh && (sev->fh != >vfh ||
> > (sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
> > -   (changes & V4L2_EVENT_CTRL_CH_FLAGS)))
> > +   (changes & V4L2_EVENT_CTRL_CH_FLAGS) || autoupdate))
> > v4l2_event_queue_fh(sev->fh, );
> > }
> >  }
> > 
> > -static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
> > -   struct uvc_control *master, u32 slave_id,
> > -   const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
> > +static void __uvc_ctrl_send_slave_event(struct uvc_fh *handle,
> > +   struct uvc_control *master, u32 slave_id)
> >  {
> > struct uvc_control_mapping *mapping = NULL;
> > struct uvc_control *ctrl = NULL;
> > u32 changes = V4L2_EVENT_CTRL_CH_FLAGS;
> > -   unsigned int i;
> > s32 val = 0;
> > 
> > +   __uvc_find_control(master->entity, slave_id, , , 0);
> > +   if (ctrl == NULL)
> > +   return;
> > +
> > +   if (__uvc_ctrl_get(handle->chain, ctrl, mapping, ) == 0)
> > +   changes |= V4L2_EVENT_CTRL_CH_VALUE;
> > +
> > +   uvc_ctrl_send_event(handle, ctrl, mapping, val, changes);
> > +}
> > +
> > +static void uvc_ctrl_status_event_work(struct work_struct *work)
> > +{
> > +   struct uvc_device *dev = container_of(work, struct uvc_device,
> > + async_ctrl.work);
> > +   struct uvc_video_chain *chain;
> > +   struct uvc_ctrl_work *w = >async_ctrl;
> > +   struct uvc_control_mapping *mapping;
> > +   struct uvc_control *ctrl;
> > +   struct uvc_fh *handle;
> > +   unsigned int i;
> > +   u8 *data;
> > +
> > +   spin_lock_irq(>lock);
> > +   data = w->data;
> > +   w->data = NULL;
> > +   chain = w->chain;
> > 

Re: [PATCH v7 1/2] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the review. As soon as the second patch has also been reviewed, 
I'll work on both of them.

Thanks
Guennadi

On Fri, 23 Mar 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> On Friday, 23 March 2018 11:24:00 EET Laurent Pinchart wrote:
> > From: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> > 
> > UVC defines a method of handling asynchronous controls, which sends a
> > USB packet over the interrupt pipe. This patch implements support for
> > such packets by sending a control event to the user. Since this can
> > involve USB traffic and, therefore, scheduling, this has to be done
> > in a work queue.
> > 
> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> >  drivers/media/usb/uvc/uvc_ctrl.c   | 166 ++
> >  drivers/media/usb/uvc/uvc_status.c | 111 ++---
> >  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
> >  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
> >  include/uapi/linux/uvcvideo.h  |   2 +
> >  5 files changed, 269 insertions(+), 29 deletions(-)
> > 
> > diff --git a/drivers/media/usb/uvc/uvc_ctrl.c
> > b/drivers/media/usb/uvc/uvc_ctrl.c index 4042cbdb721b..f4773c56438c 100644
> > --- a/drivers/media/usb/uvc/uvc_ctrl.c
> > +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> > @@ -20,6 +20,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> > 
> > @@ -1222,30 +1223,134 @@ static void uvc_ctrl_send_event(struct uvc_fh
> > *handle, {
> > struct v4l2_subscribed_event *sev;
> > struct v4l2_event ev;
> > +   bool autoupdate;
> > 
> > if (list_empty(>ev_subs))
> > return;
> > 
> > +   if (!handle) {
> 
> In which circumstances does this happen ? Is it when the device reports a 
> control change event without an prior control set ? Have you seen that 
> happening in practice ?
> 
> > +   autoupdate = true;
> > +   sev = list_first_entry(>ev_subs,
> > +  struct v4l2_subscribed_event, node);
> > +   handle = container_of(sev->fh, struct uvc_fh, vfh);
> 
> There's a check below that guards against sev->fh being NULL. Could sev->fh 
> be 
> NULL here ?
> 
> > +   } else {
> > +   autoupdate = false;
> > +   }
> > +
> > uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
> > 
> > list_for_each_entry(sev, >ev_subs, node) {
> > if (sev->fh && (sev->fh != >vfh ||
> > (sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
> > -   (changes & V4L2_EVENT_CTRL_CH_FLAGS)))
> > +   (changes & V4L2_EVENT_CTRL_CH_FLAGS) || autoupdate))
> > v4l2_event_queue_fh(sev->fh, );
> > }
> >  }
> > 
> > -static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
> > -   struct uvc_control *master, u32 slave_id,
> > -   const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
> > +static void __uvc_ctrl_send_slave_event(struct uvc_fh *handle,
> > +   struct uvc_control *master, u32 slave_id)
> >  {
> > struct uvc_control_mapping *mapping = NULL;
> > struct uvc_control *ctrl = NULL;
> > u32 changes = V4L2_EVENT_CTRL_CH_FLAGS;
> > -   unsigned int i;
> > s32 val = 0;
> > 
> > +   __uvc_find_control(master->entity, slave_id, , , 0);
> > +   if (ctrl == NULL)
> > +   return;
> > +
> > +   if (__uvc_ctrl_get(handle->chain, ctrl, mapping, ) == 0)
> > +   changes |= V4L2_EVENT_CTRL_CH_VALUE;
> > +
> > +   uvc_ctrl_send_event(handle, ctrl, mapping, val, changes);
> > +}
> > +
> > +static void uvc_ctrl_status_event_work(struct work_struct *work)
> > +{
> > +   struct uvc_device *dev = container_of(work, struct uvc_device,
> > + async_ctrl.work);
> > +   struct uvc_video_chain *chain;
> > +   struct uvc_ctrl_work *w = >async_ctrl;
> > +   struct uvc_control_mapping *mapping;
> > +   struct uvc_control *ctrl;
> > +   struct uvc_fh *handle;
> > +   unsigned int i;
> > +   u8 *data;
> > +
> > +   spin_lock_irq(>lock);
> > +   data = w->data;
> > +   w->data = NULL;
> > +   chain = w->chain;
> > +   ctrl = w->ctrl;
> > +   handle = ctrl->handle;
> > +   ctrl->

[PATCH] V4L: remove myself as soc-camera maintainer

[Guennadi Liakhovetski]

The soc-camera framework is deprecated, patches for it are very rare
and only contain trivial clean up. Further I haven't got any more
soc-camera systems running modern kernels.

Signed-off-by: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
---
 MAINTAINERS | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index 64cd083..80655f0 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12942,10 +12942,9 @@ S: Maintained
 F: drivers/net/ethernet/smsc/smsc9420.*
 
 SOC-CAMERA V4L2 SUBSYSTEM
-M: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
 L: linux-media@vger.kernel.org
 T: git git://linuxtv.org/media_tree.git
-S: Maintained
+S: Orphan
 F: include/media/soc*
 F: drivers/media/i2c/soc_camera/
 F: drivers/media/platform/soc_camera/
-- 
1.9.3

[PATCH RESEND 2/2 v6] uvcvideo: handle control pipe protocol STALLs

[Guennadi Liakhovetski]

When a command ends up in a STALL on the control pipe, use the Request
Error Code control to provide a more precise error information to the
user.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_video.c | 59 +++
 1 file changed, 53 insertions(+), 6 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_video.c 
b/drivers/media/usb/uvc/uvc_video.c
index 2fc0bf2..cfcc4861 100644
--- a/drivers/media/usb/uvc/uvc_video.c
+++ b/drivers/media/usb/uvc/uvc_video.c
@@ -34,15 +34,59 @@ static int __uvc_query_ctrl(struct uvc_device *dev, __u8 
query, __u8 unit,
__u8 intfnum, __u8 cs, void *data, __u16 size,
int timeout)
 {
-   __u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE;
+   __u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE, tmp, error;
unsigned int pipe;
+   int ret;
 
pipe = (query & 0x80) ? usb_rcvctrlpipe(dev->udev, 0)
  : usb_sndctrlpipe(dev->udev, 0);
type |= (query & 0x80) ? USB_DIR_IN : USB_DIR_OUT;
 
-   return usb_control_msg(dev->udev, pipe, query, type, cs << 8,
+   ret = usb_control_msg(dev->udev, pipe, query, type, cs << 8,
unit << 8 | intfnum, data, size, timeout);
+
+   if (ret != -EPIPE)
+   return ret;
+
+   tmp = *(u8 *)data;
+
+   pipe = usb_rcvctrlpipe(dev->udev, 0);
+   type = USB_TYPE_CLASS | USB_RECIP_INTERFACE | USB_DIR_IN;
+   ret = usb_control_msg(dev->udev, pipe, UVC_GET_CUR, type,
+ UVC_VC_REQUEST_ERROR_CODE_CONTROL << 8,
+ unit << 8 | intfnum, data, 1, timeout);
+   error = *(u8 *)data;
+   *(u8 *)data = tmp;
+
+   if (ret < 0)
+   return ret;
+
+   if (!ret)
+   return -EINVAL;
+
+   uvc_trace(UVC_TRACE_CONTROL, "Control error %u\n", error);
+
+   switch (error) {
+   case 0:
+   /* Cannot happen - we received a STALL */
+   return -EPIPE;
+   case 1: /* Not ready */
+   return -EAGAIN;
+   case 2: /* Wrong state */
+   return -EILSEQ;
+   case 3: /* Power */
+   return -EREMOTE;
+   case 4: /* Out of range */
+   return -ERANGE;
+   case 5: /* Invalid unit */
+   case 6: /* Invalid control */
+   case 7: /* Invalid Request */
+   case 8: /* Invalid value within range */
+   default: /* reserved or unknown */
+   break;
+   }
+
+   return -EINVAL;
 }
 
 static const char *uvc_query_name(__u8 query)
@@ -80,7 +124,7 @@ int uvc_query_ctrl(struct uvc_device *dev, __u8 query, __u8 
unit,
uvc_printk(KERN_ERR, "Failed to query (%s) UVC control %u on "
"unit %u: %d (exp. %u).\n", uvc_query_name(query), cs,
unit, ret, size);
-   return -EIO;
+   return ret < 0 ? ret : -EIO;
}
 
return 0;
@@ -203,13 +247,15 @@ static int uvc_get_video_ctrl(struct uvc_streaming 
*stream,
uvc_warn_once(stream->dev, UVC_WARN_PROBE_DEF, "UVC non "
"compliance - GET_DEF(PROBE) not supported. "
"Enabling workaround.\n");
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
goto out;
} else if (ret != size) {
uvc_printk(KERN_ERR, "Failed to query (%u) UVC %s control : "
"%d (exp. %u).\n", query, probe ? "probe" : "commit",
ret, size);
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
goto out;
}
 
@@ -290,7 +336,8 @@ static int uvc_set_video_ctrl(struct uvc_streaming *stream,
uvc_printk(KERN_ERR, "Failed to set UVC %s control : "
"%d (exp. %u).\n", probe ? "probe" : "commit",
ret, size);
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
}
 
kfree(data);
-- 
1.9.3

[PATCH RESEND 1/2 v6] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

UVC defines a method of handling asynchronous controls, which sends a
USB packet over the interrupt pipe. This patch implements support for
such packets by sending a control event to the user. Since this can
involve USB traffic and, therefore, scheduling, this has to be done
in a work queue.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_ctrl.c   | 166 +
 drivers/media/usb/uvc/uvc_status.c | 111 ++---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
 include/uapi/linux/uvcvideo.h  |   2 +
 5 files changed, 269 insertions(+), 29 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c
index 20397ab..2a592c2 100644
--- a/drivers/media/usb/uvc/uvc_ctrl.c
+++ b/drivers/media/usb/uvc/uvc_ctrl.c
@@ -20,6 +20,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 
@@ -1222,30 +1223,134 @@ static void uvc_ctrl_send_event(struct uvc_fh *handle,
 {
struct v4l2_subscribed_event *sev;
struct v4l2_event ev;
+   bool autoupdate;
 
if (list_empty(>ev_subs))
return;
 
+   if (!handle) {
+   autoupdate = true;
+   sev = list_first_entry(>ev_subs,
+  struct v4l2_subscribed_event, node);
+   handle = container_of(sev->fh, struct uvc_fh, vfh);
+   } else {
+   autoupdate = false;
+   }
+
uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
 
list_for_each_entry(sev, >ev_subs, node) {
if (sev->fh && (sev->fh != >vfh ||
(sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
-   (changes & V4L2_EVENT_CTRL_CH_FLAGS)))
+   (changes & V4L2_EVENT_CTRL_CH_FLAGS) || autoupdate))
v4l2_event_queue_fh(sev->fh, );
}
 }
 
-static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
-   struct uvc_control *master, u32 slave_id,
-   const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
+static void __uvc_ctrl_send_slave_event(struct uvc_fh *handle,
+   struct uvc_control *master, u32 slave_id)
 {
struct uvc_control_mapping *mapping = NULL;
struct uvc_control *ctrl = NULL;
u32 changes = V4L2_EVENT_CTRL_CH_FLAGS;
-   unsigned int i;
s32 val = 0;
 
+   __uvc_find_control(master->entity, slave_id, , , 0);
+   if (ctrl == NULL)
+   return;
+
+   if (__uvc_ctrl_get(handle->chain, ctrl, mapping, ) == 0)
+   changes |= V4L2_EVENT_CTRL_CH_VALUE;
+
+   uvc_ctrl_send_event(handle, ctrl, mapping, val, changes);
+}
+
+static void uvc_ctrl_status_event_work(struct work_struct *work)
+{
+   struct uvc_device *dev = container_of(work, struct uvc_device,
+ async_ctrl.work);
+   struct uvc_video_chain *chain;
+   struct uvc_ctrl_work *w = >async_ctrl;
+   struct uvc_control_mapping *mapping;
+   struct uvc_control *ctrl;
+   struct uvc_fh *handle;
+   __u8 *data;
+   unsigned int i;
+
+   spin_lock_irq(>lock);
+   data = w->data;
+   w->data = NULL;
+   chain = w->chain;
+   ctrl = w->ctrl;
+   handle = ctrl->handle;
+   ctrl->handle = NULL;
+   spin_unlock_irq(>lock);
+
+   if (mutex_lock_interruptible(>ctrl_mutex))
+   goto free;
+
+   list_for_each_entry(mapping, >info.mappings, list) {
+   s32 value = mapping->get(mapping, UVC_GET_CUR, data);
+
+   for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
+   if (!mapping->slave_ids[i])
+   break;
+
+   __uvc_ctrl_send_slave_event(handle, ctrl,
+   mapping->slave_ids[i]);
+   }
+
+   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
+   struct uvc_menu_info *menu = mapping->menu_info;
+   unsigned int i;
+
+   for (i = 0; i < mapping->menu_count; ++i, ++menu)
+   if (menu->value == value) {
+   value = i;
+   break;
+   }
+   }
+
+   uvc_ctrl_send_event(handle, ctrl, mapping, value,
+   V4L2_EVENT_CTRL_CH_VALUE);
+   }
+
+   mutex_unlock(>ctrl_mutex);
+
+free:
+   kfree(data);
+}
+
+void uvc_ctrl_status_event(struct uvc_video_chain *chain,
+  struct uvc_control *ctrl, __u8 *data, size_t len)
+{
+   struct uvc_device *d

[PATCH RESEND 0/2 v6] uvcvideo: asynchronous controls

[Guennadi Liakhovetski]

This is an update of the two patches, adding asynchronous control
support to the uvcvideo driver. If a control is sent, while the camera
is still processing an earlier control, it will generate a protocol
STALL condition on the control pipe.

Thanks
Guennadi

Guennadi Liakhovetski (2):
  uvcvideo: send a control event when a Control Change interrupt arrives
  uvcvideo: handle control pipe protocol STALLs

 drivers/media/usb/uvc/uvc_ctrl.c   | 166 +
 drivers/media/usb/uvc/uvc_status.c | 111 ++---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvc_video.c  |  59 +++--
 drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
 include/uapi/linux/uvcvideo.h  |   2 +
 6 files changed, 322 insertions(+), 35 deletions(-)

-- 
1.9.3

Re: Patch review

[Guennadi Liakhovetski]

Hi Laurent,

Yes, that's correct.

Thanks
Guennadi

On Wed, 28 Feb 2018, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Wednesday, 28 February 2018 17:07:00 EET Guennadi Liakhovetski wrote:
> > Hi,
> > 
> > I know the "development process and responsibilities" was the main topic
> > during the last media summit. Unfortunately I haven't attended it, from
> > the etherpad notes I also cannot quite conclude what decisions have been
> > made. Have any measures been discussed and agreed upon for cases, when
> > patches don't get reviewed for many months, adding up to more than a year
> > (in this specific case the first version submitted in June 2016)?
> 
> I assume you're talking about the "[PATCH 0/2 v6] uvcvideo: asynchronous 
> controls" series, is that correct ?
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

Patch review

[Guennadi Liakhovetski]

Hi,

I know the "development process and responsibilities" was the main topic 
during the last media summit. Unfortunately I haven't attended it, from 
the etherpad notes I also cannot quite conclude what decisions have been 
made. Have any measures been discussed and agreed upon for cases, when 
patches don't get reviewed for many months, adding up to more than a year 
(in this specific case the first version submitted in June 2016)?

Thanks
Guennadi

Re: Bug: Two device nodes created in /dev for a single UVC webcam

[Guennadi Liakhovetski]

Hi Kieran,

On Mon, 19 Feb 2018, Kieran Bingham wrote:

> Hi Alexandre,
> 
> Thankyou for your bug report,
> 
> On 17/02/18 20:47, Alexandre-Xavier Labonté-Lamoureux wrote:
> > Hi,
> > 
> > I'm running Linux 4.9.0-5-amd64 on Debian. I built the drivers from
> > the latest git and installed the modules.
> 
> Could you please be specific here?
> 
> Are you referring to linux-media/master branch or such? The latest from 
> Linus' tree?
> 
> Please also detail the steps you have taken to reproduce this issue - and of
> course - if you have made any code changes to make the latest UVC module 
> compile
> against a v4.9 kernel...
> 
> Building the latest git tree and installing as a module on a v4.9 kernel is
> quite a leap... I wouldn't have expected that to work.
> 
> The code would have to be compiled against a v4.9 kernel directly, and I 
> didn't
> think compiling the UVC driver against older kernels worked.
> 
> (at least it didn't work cleanly when I tried to compile v4.15 against a v4.14
> kernel last month)
> 
> > Now, two device nodes are
> > created for my webcam. This is not normal as it has never happened to
> > me before. If I connect another webcam to my laptop, two more device
> > nodes will be created for this webcam. So two new device nodes are
> > created for a single webcam.
> 
> I believe Guennadi's latest work for handling meta-data (in the latest 
> v4.16-rc1
> I think) will create two device nodes.

That's correct. The lower index node (/dev/video0) is a video node, the 
higher videoo node (/dev/video1) is a metadata node.

> > The name of my webcam appears twice in the device comobox in Guvcview
> > because of this. One of them will not work if I select it.
> 
> It would be expected that only the device with video functions as a streaming
> camera device, while the other would not.

Exactly.

> > My webcam has completely stopped working with Cheese and VLC.
> 
> This part is of particular concern however.
> 
> Guennadi - Have you tested Cheese/VLC with your series?

Sure, with cheese you can specify which camera you need by using its 
--device= parameter. Eventually it's expected, that those programs will be 
updated to recognise metadata nodes and not attempt to use them.

Thanks
Guennadi

> Are there any known issues that need looking at ?
> 
> >> v4l2-ctl --list-devices
> > Laptop_Integrated_Webcam_E4HD:  (usb-:00:1a.0-1.5):
> > /dev/video0
> > /dev/video1
> > 
> >> ls /dev/video*
> > /dev/video0  /dev/video1
> >
> > Have a nice day,
> > Alexandre-Xavier
> 
> Regards
> 
> Kieran Bingham
> 
> 

Re: [PATCH 0/2 v6] uvcvideo: asynchronous controls

[Guennadi Liakhovetski]

Hi Laurent,

Any update on this?

Thanks
Guennadi

On Wed, 13 Dec 2017, Guennadi Liakhovetski wrote:

> This is an update of the two patches, adding asynchronous control
> support to the uvcvideo driver. If a control is sent, while the camera
> is still processing an earlier control, it will generate a protocol
> STALL condition on the control pipe.
> 
> Thanks
> Guennadi
> 
> Guennadi Liakhovetski (2):
>   uvcvideo: send a control event when a Control Change interrupt arrives
>   uvcvideo: handle control pipe protocol STALLs
> 
>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 
> +
>  drivers/media/usb/uvc/uvc_status.c | 111 ++---
>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
>  drivers/media/usb/uvc/uvc_video.c  |  59 +++--
>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
>  include/uapi/linux/uvcvideo.h  |   2 +
>  6 files changed, 322 insertions(+), 35 deletions(-)
> 
> -- 
> 1.9.3
> 

Re: [RFT PATCH v3 6/6] uvcvideo: Move decode processing to process context

[Guennadi Liakhovetski]

Hi Kieran,

On Fri, 12 Jan 2018, Kieran Bingham wrote:

> Newer high definition cameras, and cameras with multiple lenses such as
> the range of stereo-vision cameras now available have ever increasing
> data rates.
> 
> The inclusion of a variable length packet header in URB packets mean
> that we must memcpy the frame data out to our destination 'manually'.
> This can result in data rates of up to 2 gigabits per second being
> processed.
> 
> To improve efficiency, and maximise throughput, handle the URB decode
> processing through a work queue to move it from interrupt context, and
> allow multiple processors to work on URBs in parallel.
> 
> Signed-off-by: Kieran Bingham 
> 
> ---
> v2:
>  - Lock full critical section of usb_submit_urb()
> 
> v3:
>  - Fix race on submitting uvc_video_decode_data_work() to work queue.
>  - Rename uvc_decode_op -> uvc_copy_op (Generic to encode/decode)
>  - Rename decodes -> copy_operations
>  - Don't queue work if there is no async task
>  - obtain copy op structure directly in uvc_video_decode_data()
>  - uvc_video_decode_data_work() -> uvc_video_copy_data_work()
> ---
>  drivers/media/usb/uvc/uvc_queue.c |  12 +++-
>  drivers/media/usb/uvc/uvc_video.c | 116 +++
>  drivers/media/usb/uvc/uvcvideo.h  |  24 ++-
>  3 files changed, 138 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/media/usb/uvc/uvc_queue.c 
> b/drivers/media/usb/uvc/uvc_queue.c
> index 5a9987e547d3..598087eeb5c2 100644
> --- a/drivers/media/usb/uvc/uvc_queue.c
> +++ b/drivers/media/usb/uvc/uvc_queue.c
> @@ -179,10 +179,22 @@ static void uvc_stop_streaming(struct vb2_queue *vq)
>   struct uvc_video_queue *queue = vb2_get_drv_priv(vq);
>   struct uvc_streaming *stream = uvc_queue_to_stream(queue);
>  
> + /* Prevent new buffers coming in. */
> + spin_lock_irq(>irqlock);
> + queue->flags |= UVC_QUEUE_STOPPING;
> + spin_unlock_irq(>irqlock);
> +
> + /*
> +  * All pending work should be completed before disabling the stream, as
> +  * all URBs will be free'd during uvc_video_enable(s, 0).
> +  */
> + flush_workqueue(stream->async_wq);

What if we manage to get one last URB here, then...

> +
>   uvc_video_enable(stream, 0);
>  
>   spin_lock_irq(>irqlock);
>   uvc_queue_return_buffers(queue, UVC_BUF_STATE_ERROR);
> + queue->flags &= ~UVC_QUEUE_STOPPING;
>   spin_unlock_irq(>irqlock);
>  }
>  
> diff --git a/drivers/media/usb/uvc/uvc_video.c 
> b/drivers/media/usb/uvc/uvc_video.c
> index 3878bec3276e..fb6b5af17380 100644
> --- a/drivers/media/usb/uvc/uvc_video.c
> +++ b/drivers/media/usb/uvc/uvc_video.c

[snip]

> + /*
> +  * When the stream is stopped, all URBs are freed as part of the call to
> +  * uvc_stop_streaming() and must not be handled asynchronously. In that
> +  * event we can safely complete the packet work directly in this
> +  * context, without resubmitting the URB.
> +  */
> + spin_lock_irqsave(>irqlock, flags);
> + if (!(queue->flags & UVC_QUEUE_STOPPING)) {
> + INIT_WORK(_urb->work, uvc_video_copy_data_work);
> + queue_work(stream->async_wq, _urb->work);
> + } else {
> + uvc_video_copy_packets(uvc_urb);

Can it not happen, that if the stream is currently being stopped, the 
queue has been flushed, possibly the previous URB or a couple of them 
don't get decoded, but you do decode this one, creating a corrupted frame? 
Wouldn't it be better to just drop this URB too?

>   }
> + spin_unlock_irqrestore(>irqlock, flags);
>  }
>  
>  /*

Thanks
Guennadi

Re: [RFC/RFT PATCH 6/6] uvcvideo: Move decode processing to process context

[Guennadi Liakhovetski]

On Wed, 3 Jan 2018, Kieran Bingham wrote:

> From: Kieran Bingham 
> 
> Newer high definition cameras, and cameras with multiple lenses such as
> the range of stereovision cameras now available have ever increasing
> data rates.
> 
> The inclusion of a variable length packet header in URB packets mean
> that we must memcpy the frame data out to our destination 'manually'.
> This can result in data rates of up to 2 gigabits per second being
> processed.
> 
> To improve efficiency, and maximise throughput, handle the URB decode
> processing through a work queue to move it from interrupt context, and
> allow multiple processors to work on URBs in parallel.
> 
> Signed-off-by: Kieran Bingham 
> ---
>  drivers/media/usb/uvc/uvc_queue.c |  12 +++-
>  drivers/media/usb/uvc/uvc_video.c | 114 ++-
>  drivers/media/usb/uvc/uvcvideo.h  |  24 +++-
>  3 files changed, 132 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/media/usb/uvc/uvc_queue.c 
> b/drivers/media/usb/uvc/uvc_queue.c
> index 204dd91a8526..07fcbfc132c9 100644
> --- a/drivers/media/usb/uvc/uvc_queue.c
> +++ b/drivers/media/usb/uvc/uvc_queue.c
> @@ -179,10 +179,22 @@ static void uvc_stop_streaming(struct vb2_queue *vq)
>   struct uvc_video_queue *queue = vb2_get_drv_priv(vq);
>   struct uvc_streaming *stream = uvc_queue_to_stream(queue);
>  
> + /* Prevent new buffers coming in. */
> + spin_lock_irq(>irqlock);
> + queue->flags |= UVC_QUEUE_STOPPING;
> + spin_unlock_irq(>irqlock);
> +
> + /*
> +  * All pending work should be completed before disabling the stream, as
> +  * all URBs will be free'd during uvc_video_enable(s, 0).
> +  */
> + flush_workqueue(stream->async_wq);
> +
>   uvc_video_enable(stream, 0);
>  
>   spin_lock_irq(>irqlock);
>   uvc_queue_return_buffers(queue, UVC_BUF_STATE_ERROR);
> + queue->flags &= ~UVC_QUEUE_STOPPING;
>   spin_unlock_irq(>irqlock);
>  }
>  
> diff --git a/drivers/media/usb/uvc/uvc_video.c 
> b/drivers/media/usb/uvc/uvc_video.c
> index 045ac655313c..b7b32a6bc2dc 100644
> --- a/drivers/media/usb/uvc/uvc_video.c
> +++ b/drivers/media/usb/uvc/uvc_video.c
> @@ -1058,21 +1058,70 @@ static int uvc_video_decode_start(struct 
> uvc_streaming *stream,
>   return data[0];
>  }
>  
> -static void uvc_video_decode_data(struct uvc_streaming *stream,
> - struct uvc_buffer *buf, const __u8 *data, int len)
> +/*
> + * uvc_video_decode_data_work: Asynchronous memcpy processing
> + *
> + * Perform memcpy tasks in process context, with completion handlers
> + * to return the URB, and buffer handles.
> + *
> + * The work submitter must pre-determine that the work is safe
> + */
> +static void uvc_video_decode_data_work(struct work_struct *work)
>  {
> - unsigned int maxlen, nbytes;
> - void *mem;
> + struct uvc_urb *uvc_urb = container_of(work, struct uvc_urb, work);
> + struct uvc_streaming *stream = uvc_urb->stream;
> + struct uvc_video_queue *queue = >queue;
> + unsigned int i;
> + bool stopping;
> + int ret;
> +
> + for (i = 0; i < uvc_urb->packets; i++) {
> + struct uvc_decode_op *op = _urb->decodes[i];
> +
> + memcpy(op->dst, op->src, op->len);
> +
> + /* Release reference taken on this buffer */
> + uvc_queue_buffer_release(op->buf);
> + }
> +
> + /*
> +  * Prevent resubmitting URBs when shutting down to ensure that no new
> +  * work item will be scheduled after uvc_stop_streaming() flushes the
> +  * work queue.
> +  */
> + spin_lock_irq(>irqlock);
> + stopping = queue->flags & UVC_QUEUE_STOPPING;
> + spin_unlock_irq(>irqlock);

Are you sure this locking really helps? What if uvc_stop_streaming() runs 
here?

Thanks
Guennadi

> +
> + if (stopping)
> + return;
> +
> + ret = usb_submit_urb(uvc_urb->urb, GFP_ATOMIC);
> + if (ret < 0)
> + uvc_printk(KERN_ERR, "Failed to resubmit video URB (%d).\n",
> +ret);
> +}

Re: [RFC/RFT PATCH 1/6] uvcvideo: Refactor URB descriptors

[Guennadi Liakhovetski]

Hi Kieran,

Just minor suggestions below:

On Wed, 3 Jan 2018, Kieran Bingham wrote:

> From: Kieran Bingham 
> 
> We currently store three separate arrays for each URB reference we hold.
> 
> Objectify the data needed to track URBs into a single uvc_urb structure,
> allowing better object management and tracking of the URB.
> 
> All accesses to the data pointers through stream, are converted to use a
> uvc_urb pointer for consistency.
> 
> Signed-off-by: Kieran Bingham 
> Reviewed-by: Laurent Pinchart 
> ---
>  drivers/media/usb/uvc/uvc_video.c | 46 
>  drivers/media/usb/uvc/uvcvideo.h  | 18 ++---
>  2 files changed, 44 insertions(+), 20 deletions(-)

[snip]

> diff --git a/drivers/media/usb/uvc/uvcvideo.h 
> b/drivers/media/usb/uvc/uvcvideo.h
> index 19e725e2bda5..4afa8ce13ea7 100644
> --- a/drivers/media/usb/uvc/uvcvideo.h
> +++ b/drivers/media/usb/uvc/uvcvideo.h
> @@ -479,6 +479,20 @@ struct uvc_stats_stream {
>   unsigned int max_sof;   /* Maximum STC.SOF value */
>  };
>  
> +/**
> + * struct uvc_urb - URB context management structure
> + *
> + * @urb: described URB. Must be allocated with usb_alloc_urb()

Didn't you mean "describes?"

> + * @urb_buffer: memory storage for the URB
> + * @urb_dma: DMA coherent addressing for the urb_buffer

The whole struct describes URBs, so, I wouldn't repeat that in these two 
field names, I'd just call them "buffer" and "dma." OTOH, later you add 
more fields like "stream," which aren't per-URB, so, maybe you want to 
keep these prefixes.

Thanks
Guennadi

> + */
> +struct uvc_urb {
> + struct urb *urb;
> +
> + char *urb_buffer;
> + dma_addr_t urb_dma;
> +};
> +
>  struct uvc_streaming {
>   struct list_head list;
>   struct uvc_device *dev;
> @@ -521,9 +535,7 @@ struct uvc_streaming {
>   __u32 max_payload_size;
>   } bulk;
>  
> - struct urb *urb[UVC_URBS];
> - char *urb_buffer[UVC_URBS];
> - dma_addr_t urb_dma[UVC_URBS];
> + struct uvc_urb uvc_urb[UVC_URBS];
>   unsigned int urb_size;
>  
>   __u32 sequence;
> -- 
> git-series 0.9.1
> 

Re: [RFC/RFT PATCH 3/6] uvcvideo: Protect queue internals with helper

[Guennadi Liakhovetski]

Hi Kieran,

On Wed, 3 Jan 2018, Kieran Bingham wrote:

> From: Kieran Bingham 
> 
> The URB completion operation obtains the current buffer by reading
> directly into the queue internal interface.
> 
> Protect this queue abstraction by providing a helper
> uvc_queue_get_current_buffer() which can be used by both the decode
> task, and the uvc_queue_next_buffer() functions.
> 
> Signed-off-by: Kieran Bingham 
> Reviewed-by: Laurent Pinchart 
> ---
>  drivers/media/usb/uvc/uvc_queue.c | 34 +++-
>  drivers/media/usb/uvc/uvc_video.c |  7 +--
>  drivers/media/usb/uvc/uvcvideo.h  |  2 ++-
>  3 files changed, 32 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/media/usb/uvc/uvc_queue.c 
> b/drivers/media/usb/uvc/uvc_queue.c
> index c8d78b2f3de4..0711e3d9ff76 100644
> --- a/drivers/media/usb/uvc/uvc_queue.c
> +++ b/drivers/media/usb/uvc/uvc_queue.c
> @@ -399,6 +399,34 @@ void uvc_queue_cancel(struct uvc_video_queue *queue, int 
> disconnect)
>   spin_unlock_irqrestore(>irqlock, flags);
>  }
>  
> +/*
> + * uvc_queue_get_current_buffer: Obtain the current working output buffer
> + *
> + * Buffers may span multiple packets, and even URBs, therefore the active 
> buffer
> + * remains on the queue until the EOF marker.
> + */
> +static struct uvc_buffer *
> +__uvc_queue_get_current_buffer(struct uvc_video_queue *queue)
> +{
> + if (!list_empty(>irqqueue))
> + return list_first_entry(>irqqueue, struct uvc_buffer,
> + queue);
> + else
> + return NULL;

I think the preferred style is not to use "else" in such cases. It might 
even be prettier to write

if (list_empty(...))
return NULL;

return list_first_entry(...);

Thanks
Guennadi

Re: [PATCH] uvcvideo: add a D4M camera description

[Guennadi Liakhovetski]

Hi Sakari,

On Wed, 27 Dec 2017, Sakari Ailus wrote:

> Hi Guennadi,
> 
> Thanks for the patch!
> 
> On Sat, Dec 23, 2017 at 12:11:00PM +0100, Guennadi Liakhovetski wrote:
> > From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > 
> > D4M is a mobile model from the D4XX family of Intel RealSense cameras.
> > This patch adds a descriptor for it, which enables reading per-frame
> > metadata from it.
> > 
> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> >  Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst | 202 
> > ++
> >  drivers/media/usb/uvc/uvc_driver.c|  11 ++
> >  include/uapi/linux/videodev2.h|   1 +
> >  3 files changed, 214 insertions(+)
> >  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > 
> > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst 
> > b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > new file mode 100644
> > index 000..950780d
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
> > @@ -0,0 +1,202 @@
> > +.. -*- coding: utf-8; mode: rst -*-
> > +
> > +.. _v4l2-meta-fmt-d4xx:
> > +
> > +***
> > +V4L2_META_FMT_D4XX ('D4XX')
> > +***
> > +
> > +D4XX Metadata
> > +
> > +
> > +Description
> > +===
> > +
> > +D4XX (D435 and other) cameras include per-frame metadata in their UVC 
> > payload
> 
> If this is D435 and some others, I'd simply call this D435. Say, if you get
> another device in D4xx series that implements a different format, how do
> you call that? Up to you.

As far as I understand, all D4XX cameras are using this format, but they 
might use different sets of metadata blocks, in which case any new such 
blocks should be added to this documentation. The format remains the same 
though, hence the name.

> Is there a specific list of devices that use this format? The driver patch
> only appears to introduce one USB ID.

I don't have such a list, sorry. I think any D4XX camera should work with 
this, so it would be a matter of adding those product ID values.

Thanks
Guennadi

[PATCH] uvcvideo: add a D4M camera description

[Guennadi Liakhovetski]

From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>

D4M is a mobile model from the D4XX family of Intel RealSense cameras.
This patch adds a descriptor for it, which enables reading per-frame
metadata from it.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst | 202 ++
 drivers/media/usb/uvc/uvc_driver.c|  11 ++
 include/uapi/linux/videodev2.h|   1 +
 3 files changed, 214 insertions(+)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst

diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst 
b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
new file mode 100644
index 000..950780d
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-d4xx.rst
@@ -0,0 +1,202 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _v4l2-meta-fmt-d4xx:
+
+***
+V4L2_META_FMT_D4XX ('D4XX')
+***
+
+D4XX Metadata
+
+
+Description
+===
+
+D4XX (D435 and other) cameras include per-frame metadata in their UVC payload
+headers, following the Microsoft(R) UVC extension proposal [1_]. That means,
+that the private D4XX metadata, following the standard UVC header, is organised
+in blocks. D4XX cameras implement several standard block types, proposed by
+Microsoft, and several proprietary ones. Supported standard metadata types
+include MetadataId_CaptureStats (ID 3), MetadataId_CameraExtrinsics (ID 4), and
+MetadataId_CameraIntrinsics (ID 5). For their description see [1_]. This
+document describes proprietary metadata types, used by DS4XX cameras.
+
+V4L2_META_FMT_D4XX buffers follow the metadata buffer layout of
+V4L2_META_FMT_UVC with the only difference, that it also includes proprietary
+payload header data. D4XX cameras use bulk transfers and only send one payload
+per frame, therefore their headers cannot be larger than 255 bytes.
+
+Below are proprietary Microsoft style metadata types, used by D4XX cameras,
+where all fields are in little endian order:
+
+.. flat-table:: D4XX metadata
+:widths: 1 4
+:header-rows:  1
+:stub-columns: 0
+
+* - Field
+  - Description
+* - :cspan:`1` *Depth Control*
+* - __u32 ID
+  - 0x8000
+* - __u32 Size
+  - Size in bytes (currently 56)
+* - __u32 Version
+  - Version of the struct
+* - __u32 Flags
+  - A bitmask of flags: see [2_] below
+* - __u32 Gain
+  - Manual gain value
+* - __u32 Exposure
+  - Manual exposure time in microseconds
+* - __u32 Laser power
+  - Power of the laser LED 0-360, used for depth measurement
+* - __u32 AE mode
+  - 0: manual; 1: automatic exposure
+* - __u32 Exposure priority
+  - Exposure priority value: 0 - constant frameerate
+* - __u32 AE ROI left
+  - Left border of the AE Region of Interest
+* - __u32 AE ROI right
+  - Right border of the AE Region of Interest
+* - __u32 AE ROI top
+  - Top border of the AE Region of Interest
+* - __u32 AE ROI bottom
+  - Bottom border of the AE Region of Interest
+* - __u32 Preset
+  - Preset selector value
+* - __u32 Laser mode
+  - 0: off, 1: on
+* - :cspan:`1` *Capture Timing*
+* - __u32 ID
+  - 0x8001
+* - __u32 Size
+  - Size in bytes (currently 40)
+* - __u32 Version
+  - Version of the struct
+* - __u32 Flags
+  - A bitmask of flags: see [3_] below
+* - __u32 Frame counter
+  - Monotonically increasing counter
+* - __u32 Optical time
+  - Time in microseconds from the beginning of a frame till its middle
+* - __u32 Readout time
+  - Time, used to read out a frame in microseconds
+* - __u32 Exposure time
+  - Frame exposure time in microseconds
+* - __u32 Frame interval
+  - In microseconds = 100 / framerate
+* - __u32 Pipe latency
+  - Time in microseconds from start of frame to data in USB buffer
+* - :cspan:`1` *Configuration*
+* - __u32 ID
+  - 0x8002
+* - __u32 Size
+  - Size in bytes (currently 40)
+* - __u32 Version
+  - Version of the struct
+* - __u32 Flags
+  - A bitmask of flags: see [4_] below
+* - __u8 Hardware type
+  - Camera hardware version [5_]
+* - __u8 SKU ID
+  - Camera hardware configuration [6_]
+* - __u32 Cookie
+  - Internal synchronisation
+* - __u16 Format
+  - Image format code [7_]
+* - __u16 Width
+  - Width in pixels
+* - __u16 Height
+  - Height in pixels
+* - __u16 Framerate
+  - Requested framerate
+* - __u16 Trigger
+  - Byte 0: bit 0:  depth and RGB are synchronised, bit 1: external trigger
+
+.. _1:
+
+[1] 
https://docs.microsoft.com/en-us/windows-hardware/drivers/stream/uvc-extensions-1-5
+
+.. _2:
+
+[2] Depth Control flags specify, which fields are valid: ::
+
+  0x0001 Gain
+  0x0002 Man

Re: [PATCH 1/2 v6] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

Sorry, forgot to mention a change from the previous version: now 
autoupdate Control Change events are also delivered.

Thanks
Guennadi

On Wed, 13 Dec 2017, Guennadi Liakhovetski wrote:

> UVC defines a method of handling asynchronous controls, which sends a
> USB packet over the interrupt pipe. This patch implements support for
> such packets by sending a control event to the user. Since this can
> involve USB traffic and, therefore, scheduling, this has to be done
> in a work queue.
> 
> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> ---
>  drivers/media/usb/uvc/uvc_ctrl.c   | 166 
> +
>  drivers/media/usb/uvc/uvc_status.c | 111 ++---
>  drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
>  drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
>  include/uapi/linux/uvcvideo.h  |   2 +
>  5 files changed, 269 insertions(+), 29 deletions(-)
> 
> diff --git a/drivers/media/usb/uvc/uvc_ctrl.c 
> b/drivers/media/usb/uvc/uvc_ctrl.c
> index 20397ab..2a592c2 100644
> --- a/drivers/media/usb/uvc/uvc_ctrl.c
> +++ b/drivers/media/usb/uvc/uvc_ctrl.c
> @@ -20,6 +20,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>  #include 
>  #include 
>  
> @@ -1222,30 +1223,134 @@ static void uvc_ctrl_send_event(struct uvc_fh 
> *handle,
>  {
>   struct v4l2_subscribed_event *sev;
>   struct v4l2_event ev;
> + bool autoupdate;
>  
>   if (list_empty(>ev_subs))
>   return;
>  
> + if (!handle) {
> + autoupdate = true;
> + sev = list_first_entry(>ev_subs,
> +struct v4l2_subscribed_event, node);
> + handle = container_of(sev->fh, struct uvc_fh, vfh);
> + } else {
> + autoupdate = false;
> + }
> +
>   uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
>  
>   list_for_each_entry(sev, >ev_subs, node) {
>   if (sev->fh && (sev->fh != >vfh ||
>   (sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
> - (changes & V4L2_EVENT_CTRL_CH_FLAGS)))
> + (changes & V4L2_EVENT_CTRL_CH_FLAGS) || autoupdate))
>   v4l2_event_queue_fh(sev->fh, );
>   }
>  }
>  
> -static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
> - struct uvc_control *master, u32 slave_id,
> - const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
> +static void __uvc_ctrl_send_slave_event(struct uvc_fh *handle,
> + struct uvc_control *master, u32 slave_id)
>  {
>   struct uvc_control_mapping *mapping = NULL;
>   struct uvc_control *ctrl = NULL;
>   u32 changes = V4L2_EVENT_CTRL_CH_FLAGS;
> - unsigned int i;
>   s32 val = 0;
>  
> + __uvc_find_control(master->entity, slave_id, , , 0);
> + if (ctrl == NULL)
> + return;
> +
> + if (__uvc_ctrl_get(handle->chain, ctrl, mapping, ) == 0)
> + changes |= V4L2_EVENT_CTRL_CH_VALUE;
> +
> + uvc_ctrl_send_event(handle, ctrl, mapping, val, changes);
> +}
> +
> +static void uvc_ctrl_status_event_work(struct work_struct *work)
> +{
> + struct uvc_device *dev = container_of(work, struct uvc_device,
> +   async_ctrl.work);
> + struct uvc_video_chain *chain;
> + struct uvc_ctrl_work *w = >async_ctrl;
> + struct uvc_control_mapping *mapping;
> + struct uvc_control *ctrl;
> + struct uvc_fh *handle;
> + __u8 *data;
> + unsigned int i;
> +
> + spin_lock_irq(>lock);
> + data = w->data;
> + w->data = NULL;
> + chain = w->chain;
> + ctrl = w->ctrl;
> + handle = ctrl->handle;
> + ctrl->handle = NULL;
> + spin_unlock_irq(>lock);
> +
> + if (mutex_lock_interruptible(>ctrl_mutex))
> + goto free;
> +
> + list_for_each_entry(mapping, >info.mappings, list) {
> + s32 value = mapping->get(mapping, UVC_GET_CUR, data);
> +
> + for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
> + if (!mapping->slave_ids[i])
> + break;
> +
> + __uvc_ctrl_send_slave_event(handle, ctrl,
> + mapping->slave_ids[i]);
> + }
> +
> + if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
> + struct uvc_menu_info *menu = mapping->menu_info;
> + unsigned int i;
> +
> +  

[PATCH 2/2 v6] uvcvideo: handle control pipe protocol STALLs

[Guennadi Liakhovetski]

When a command ends up in a STALL on the control pipe, use the Request
Error Code control to provide a more precise error information to the
user.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_video.c | 59 +++
 1 file changed, 53 insertions(+), 6 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_video.c 
b/drivers/media/usb/uvc/uvc_video.c
index 2fc0bf2..cfcc4861 100644
--- a/drivers/media/usb/uvc/uvc_video.c
+++ b/drivers/media/usb/uvc/uvc_video.c
@@ -34,15 +34,59 @@ static int __uvc_query_ctrl(struct uvc_device *dev, __u8 
query, __u8 unit,
__u8 intfnum, __u8 cs, void *data, __u16 size,
int timeout)
 {
-   __u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE;
+   __u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE, tmp, error;
unsigned int pipe;
+   int ret;
 
pipe = (query & 0x80) ? usb_rcvctrlpipe(dev->udev, 0)
  : usb_sndctrlpipe(dev->udev, 0);
type |= (query & 0x80) ? USB_DIR_IN : USB_DIR_OUT;
 
-   return usb_control_msg(dev->udev, pipe, query, type, cs << 8,
+   ret = usb_control_msg(dev->udev, pipe, query, type, cs << 8,
unit << 8 | intfnum, data, size, timeout);
+
+   if (ret != -EPIPE)
+   return ret;
+
+   tmp = *(u8 *)data;
+
+   pipe = usb_rcvctrlpipe(dev->udev, 0);
+   type = USB_TYPE_CLASS | USB_RECIP_INTERFACE | USB_DIR_IN;
+   ret = usb_control_msg(dev->udev, pipe, UVC_GET_CUR, type,
+ UVC_VC_REQUEST_ERROR_CODE_CONTROL << 8,
+ unit << 8 | intfnum, data, 1, timeout);
+   error = *(u8 *)data;
+   *(u8 *)data = tmp;
+
+   if (ret < 0)
+   return ret;
+
+   if (!ret)
+   return -EINVAL;
+
+   uvc_trace(UVC_TRACE_CONTROL, "Control error %u\n", error);
+
+   switch (error) {
+   case 0:
+   /* Cannot happen - we received a STALL */
+   return -EPIPE;
+   case 1: /* Not ready */
+   return -EAGAIN;
+   case 2: /* Wrong state */
+   return -EILSEQ;
+   case 3: /* Power */
+   return -EREMOTE;
+   case 4: /* Out of range */
+   return -ERANGE;
+   case 5: /* Invalid unit */
+   case 6: /* Invalid control */
+   case 7: /* Invalid Request */
+   case 8: /* Invalid value within range */
+   default: /* reserved or unknown */
+   break;
+   }
+
+   return -EINVAL;
 }
 
 static const char *uvc_query_name(__u8 query)
@@ -80,7 +124,7 @@ int uvc_query_ctrl(struct uvc_device *dev, __u8 query, __u8 
unit,
uvc_printk(KERN_ERR, "Failed to query (%s) UVC control %u on "
"unit %u: %d (exp. %u).\n", uvc_query_name(query), cs,
unit, ret, size);
-   return -EIO;
+   return ret < 0 ? ret : -EIO;
}
 
return 0;
@@ -203,13 +247,15 @@ static int uvc_get_video_ctrl(struct uvc_streaming 
*stream,
uvc_warn_once(stream->dev, UVC_WARN_PROBE_DEF, "UVC non "
"compliance - GET_DEF(PROBE) not supported. "
"Enabling workaround.\n");
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
goto out;
} else if (ret != size) {
uvc_printk(KERN_ERR, "Failed to query (%u) UVC %s control : "
"%d (exp. %u).\n", query, probe ? "probe" : "commit",
ret, size);
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
goto out;
}
 
@@ -290,7 +336,8 @@ static int uvc_set_video_ctrl(struct uvc_streaming *stream,
uvc_printk(KERN_ERR, "Failed to set UVC %s control : "
"%d (exp. %u).\n", probe ? "probe" : "commit",
ret, size);
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
}
 
kfree(data);
-- 
1.9.3

[PATCH 1/2 v6] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

UVC defines a method of handling asynchronous controls, which sends a
USB packet over the interrupt pipe. This patch implements support for
such packets by sending a control event to the user. Since this can
involve USB traffic and, therefore, scheduling, this has to be done
in a work queue.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_ctrl.c   | 166 +
 drivers/media/usb/uvc/uvc_status.c | 111 ++---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
 include/uapi/linux/uvcvideo.h  |   2 +
 5 files changed, 269 insertions(+), 29 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c
index 20397ab..2a592c2 100644
--- a/drivers/media/usb/uvc/uvc_ctrl.c
+++ b/drivers/media/usb/uvc/uvc_ctrl.c
@@ -20,6 +20,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 
@@ -1222,30 +1223,134 @@ static void uvc_ctrl_send_event(struct uvc_fh *handle,
 {
struct v4l2_subscribed_event *sev;
struct v4l2_event ev;
+   bool autoupdate;
 
if (list_empty(>ev_subs))
return;
 
+   if (!handle) {
+   autoupdate = true;
+   sev = list_first_entry(>ev_subs,
+  struct v4l2_subscribed_event, node);
+   handle = container_of(sev->fh, struct uvc_fh, vfh);
+   } else {
+   autoupdate = false;
+   }
+
uvc_ctrl_fill_event(handle->chain, , ctrl, mapping, value, changes);
 
list_for_each_entry(sev, >ev_subs, node) {
if (sev->fh && (sev->fh != >vfh ||
(sev->flags & V4L2_EVENT_SUB_FL_ALLOW_FEEDBACK) ||
-   (changes & V4L2_EVENT_CTRL_CH_FLAGS)))
+   (changes & V4L2_EVENT_CTRL_CH_FLAGS) || autoupdate))
v4l2_event_queue_fh(sev->fh, );
}
 }
 
-static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
-   struct uvc_control *master, u32 slave_id,
-   const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
+static void __uvc_ctrl_send_slave_event(struct uvc_fh *handle,
+   struct uvc_control *master, u32 slave_id)
 {
struct uvc_control_mapping *mapping = NULL;
struct uvc_control *ctrl = NULL;
u32 changes = V4L2_EVENT_CTRL_CH_FLAGS;
-   unsigned int i;
s32 val = 0;
 
+   __uvc_find_control(master->entity, slave_id, , , 0);
+   if (ctrl == NULL)
+   return;
+
+   if (__uvc_ctrl_get(handle->chain, ctrl, mapping, ) == 0)
+   changes |= V4L2_EVENT_CTRL_CH_VALUE;
+
+   uvc_ctrl_send_event(handle, ctrl, mapping, val, changes);
+}
+
+static void uvc_ctrl_status_event_work(struct work_struct *work)
+{
+   struct uvc_device *dev = container_of(work, struct uvc_device,
+ async_ctrl.work);
+   struct uvc_video_chain *chain;
+   struct uvc_ctrl_work *w = >async_ctrl;
+   struct uvc_control_mapping *mapping;
+   struct uvc_control *ctrl;
+   struct uvc_fh *handle;
+   __u8 *data;
+   unsigned int i;
+
+   spin_lock_irq(>lock);
+   data = w->data;
+   w->data = NULL;
+   chain = w->chain;
+   ctrl = w->ctrl;
+   handle = ctrl->handle;
+   ctrl->handle = NULL;
+   spin_unlock_irq(>lock);
+
+   if (mutex_lock_interruptible(>ctrl_mutex))
+   goto free;
+
+   list_for_each_entry(mapping, >info.mappings, list) {
+   s32 value = mapping->get(mapping, UVC_GET_CUR, data);
+
+   for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
+   if (!mapping->slave_ids[i])
+   break;
+
+   __uvc_ctrl_send_slave_event(handle, ctrl,
+   mapping->slave_ids[i]);
+   }
+
+   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
+   struct uvc_menu_info *menu = mapping->menu_info;
+   unsigned int i;
+
+   for (i = 0; i < mapping->menu_count; ++i, ++menu)
+   if (menu->value == value) {
+   value = i;
+   break;
+   }
+   }
+
+   uvc_ctrl_send_event(handle, ctrl, mapping, value,
+   V4L2_EVENT_CTRL_CH_VALUE);
+   }
+
+   mutex_unlock(>ctrl_mutex);
+
+free:
+   kfree(data);
+}
+
+void uvc_ctrl_status_event(struct uvc_video_chain *chain,
+  struct uvc_control *ctrl, __u8 *data, size_t len)
+{
+   struct uvc_device *d

[PATCH 0/2 v6] uvcvideo: asynchronous controls

[Guennadi Liakhovetski]

This is an update of the two patches, adding asynchronous control
support to the uvcvideo driver. If a control is sent, while the camera
is still processing an earlier control, it will generate a protocol
STALL condition on the control pipe.

Thanks
Guennadi

Guennadi Liakhovetski (2):
  uvcvideo: send a control event when a Control Change interrupt arrives
  uvcvideo: handle control pipe protocol STALLs

 drivers/media/usb/uvc/uvc_ctrl.c   | 166 +
 drivers/media/usb/uvc/uvc_status.c | 111 ++---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvc_video.c  |  59 +++--
 drivers/media/usb/uvc/uvcvideo.h   |  15 +++-
 include/uapi/linux/uvcvideo.h  |   2 +
 6 files changed, 322 insertions(+), 35 deletions(-)

-- 
1.9.3

Re: [PATCH v8] uvcvideo: Add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

On Wed, 13 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday, 12 December 2017 10:30:39 EET Guennadi Liakhovetski wrote:
> > On Mon, 11 Dec 2017, Laurent Pinchart wrote:
> > > On Monday, 11 December 2017 23:44:09 EET Guennadi Liakhovetski wrote:
> > >> On Mon, 11 Dec 2017, Laurent Pinchart wrote:
> > >>> On Monday, 11 December 2017 22:16:23 EET Laurent Pinchart wrote:
> > >>>> On Wednesday, 6 December 2017 17:15:40 EET Guennadi Liakhovetski wrote:
> > >>>>> From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >>>>> 
> > >>>>> Some UVC video cameras contain metadata in their payload headers.
> > >>>>> This patch extracts that data, adding more clock synchronisation
> > >>>>> information, on both bulk and isochronous endpoints and makes it
> > >>>>> available to the user space on a separate video node, using the
> > >>>>> V4L2_CAP_META_CAPTURE capability and the V4L2_BUF_TYPE_META_CAPTURE
> > >>>>> buffer queue type. By default, only the V4L2_META_FMT_UVC pixel
> > >>>>> format is available from those nodes. However, cameras can be added to
> > >>>>> the device ID table to additionally specify their own metadata format,
> > >>>>> in which case that format will also become available from the metadata
> > >>>>> node.
> > >>>>> 
> > >>>>> Signed-off-by: Guennadi Liakhovetski
> > >>>>> <guennadi.liakhovet...@intel.com>
> > >>>>> ---
> > >>>>> 
> > >>>>> v8: addressed comments and integrated changes from Laurent, thanks
> > >>>>> again, e.g.:
> > >>>>> 
> > >>>>> - multiple stylistic changes
> > >>>>> - remove the UVC_DEV_FLAG_METADATA_NODE flag / quirk: nodes are now
> > >>>>>   created unconditionally
> > >>>>> - reuse uvc_ioctl_querycap()
> > >>>>> - reuse code in uvc_register_video()
> > >>>>> - set an error flag when the metadata buffer overflows
> > >>>>> 
> > >>>>>  drivers/media/usb/uvc/Makefile   |   2 +-
> > >>>>>  drivers/media/usb/uvc/uvc_driver.c   |  15 ++-
> > >>>>>  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> > >>>>>  drivers/media/usb/uvc/uvc_metadata.c | 179+++
> > >>>>>  drivers/media/usb/uvc/uvc_queue.c|  44 +++--
> > >>>>>  drivers/media/usb/uvc/uvc_video.c| 132 +++--
> > >>>>>  drivers/media/usb/uvc/uvcvideo.h |  16 +++-
> > >>>>>  include/uapi/linux/uvcvideo.h|  26 +
> > >>>>>  8 files changed, 394 insertions(+), 22 deletions(-)
> > >>>>>  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> > >>>> 
> > >>>> [snip]
> > >>>> 
> > >>>>> diff --git a/drivers/media/usb/uvc/uvc_video.c
> > >>>>> b/drivers/media/usb/uvc/uvc_video.c index 13f459e..2fc0bf2 100644
> > >>>>> --- a/drivers/media/usb/uvc/uvc_video.c
> > >>>>> +++ b/drivers/media/usb/uvc/uvc_video.c
> > >>>> 
> > >>>> [snip]
> > >>>> 
> > >>>>> +static void uvc_video_decode_meta(struct uvc_streaming *stream,
> > >>>>> +   struct uvc_buffer *meta_buf,
> > >>>>> +   const u8 *mem, unsigned int length)
> > >>>>> +{
> > >>>>> + struct uvc_meta_buf *meta;
> > >>>>> + size_t len_std = 2;
> > >>>>> + bool has_pts, has_scr;
> > >>>>> + unsigned long flags;
> > >>>>> + ktime_t time;
> > >>>>> + const u8 *scr;
> 
> [snip]
> 
> > >>>>> + meta = (struct uvc_meta_buf *)((u8 *)meta_buf->mem +
> > >>>>> meta_buf->bytesused);
> > >>>>> + local_irq_save(flags);
> > >>>>> + time = uvc_video_get_time();
> > >>>>> + meta->sof = usb_get_current_frame_number(stream->dev->udev);
> > >>>> 
> > >>>> You need a put_unaligned here too. If you'r

Re: [PATCH v8] uvcvideo: Add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

On Mon, 11 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Monday, 11 December 2017 23:44:09 EET Guennadi Liakhovetski wrote:
> > On Mon, 11 Dec 2017, Laurent Pinchart wrote:
> > > On Monday, 11 December 2017 22:16:23 EET Laurent Pinchart wrote:
> > >> On Wednesday, 6 December 2017 17:15:40 EET Guennadi Liakhovetski wrote:
> > >>> From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >>> 
> > >>> Some UVC video cameras contain metadata in their payload headers. This
> > >>> patch extracts that data, adding more clock synchronisation
> > >>> information, on both bulk and isochronous endpoints and makes it
> > >>> available to the user space on a separate video node, using the
> > >>> V4L2_CAP_META_CAPTURE capability and the V4L2_BUF_TYPE_META_CAPTURE
> > >>> buffer queue type. By default, only the V4L2_META_FMT_UVC pixel format
> > >>> is available from those nodes. However, cameras can be added to the
> > >>> device ID table to additionally specify their own metadata format, in
> > >>> which case that format will also become available from the metadata
> > >>> node.
> > >>> 
> > >>> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >>> ---
> > >>> 
> > >>> v8: addressed comments and integrated changes from Laurent, thanks
> > >>> again, e.g.:
> > >>> 
> > >>> - multiple stylistic changes
> > >>> - remove the UVC_DEV_FLAG_METADATA_NODE flag / quirk: nodes are now
> > >>>   created unconditionally
> > >>> - reuse uvc_ioctl_querycap()
> > >>> - reuse code in uvc_register_video()
> > >>> - set an error flag when the metadata buffer overflows
> > >>> 
> > >>>  drivers/media/usb/uvc/Makefile   |   2 +-
> > >>>  drivers/media/usb/uvc/uvc_driver.c   |  15 ++-
> > >>>  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> > >>>  drivers/media/usb/uvc/uvc_metadata.c | 179 
> > >>>  drivers/media/usb/uvc/uvc_queue.c|  44 +++--
> > >>>  drivers/media/usb/uvc/uvc_video.c| 132 --
> > >>>  drivers/media/usb/uvc/uvcvideo.h |  16 +++-
> > >>>  include/uapi/linux/uvcvideo.h|  26 +
> > >>>  8 files changed, 394 insertions(+), 22 deletions(-)
> > >>>  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> > >> 
> > >> [snip]
> > >> 
> > >> > diff --git a/drivers/media/usb/uvc/uvc_video.c
> > >> > b/drivers/media/usb/uvc/uvc_video.c index 13f459e..2fc0bf2 100644
> > >> > --- a/drivers/media/usb/uvc/uvc_video.c
> > >> > +++ b/drivers/media/usb/uvc/uvc_video.c
> > >> 
> > >> [snip]
> > >> 
> > >>> +static void uvc_video_decode_meta(struct uvc_streaming *stream,
> > >>> + struct uvc_buffer *meta_buf,
> > >>> + const u8 *mem, unsigned int length)
> > >>> +{
> > >>> +   struct uvc_meta_buf *meta;
> > >>> +   size_t len_std = 2;
> > >>> +   bool has_pts, has_scr;
> > >>> +   unsigned long flags;
> > >>> +   ktime_t time;
> > >>> +   const u8 *scr;
> > >>> +
> > >>> +   if (!meta_buf || length == 2)
> > >>> +   return;
> > >>> +
> > >>> +   if (meta_buf->length - meta_buf->bytesused <
> > >>> +   length + sizeof(meta->ns) + sizeof(meta->sof)) {
> > >>> +   meta_buf->error = 1;
> > >>> +   return;
> > >>> +   }
> > >>> +
> > >>> +   has_pts = mem[1] & UVC_STREAM_PTS;
> > >>> +   has_scr = mem[1] & UVC_STREAM_SCR;
> > >>> +
> > >>> +   if (has_pts) {
> > >>> +   len_std += 4;
> > >>> +   scr = mem + 6;
> > >>> +   } else {
> > >>> +   scr = mem + 2;
> > >>> +   }
> > >>> +
> > >>> +   if (has_scr)
> > >>> +   len_std += 6;
> > >>> +
> > >>> +   if

Re: [PATCH 0/2] uvcvideo: Refactor code to ease metadata implementation

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the patches. Please feel free to add either or both of

Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
Tested-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>

to both of the patches. Whereas in fact strictly speaking your current 
tree has updated improved versions of the patches, at least of the first 
of them - it now correctly handles the struct video_device::vfl_dir field, 
even thoough I'd still find merging that "if" with the following "switch" 
prettier ;-) So, strictly speaking you'd have to post those updated 
versions, in any case my approval tags refer to versions in your tree with 
commit IDs

53464c9f76da054ac3c291d27f348170d2a346c6
and
b6c5f10563c4ee8437cd9131bc3d389514456519

Thanks
Guennadi

On Tue, 5 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> This small patch series refactors the uvc_video_register() function to extract
> the code that you need into a new uvc_video_register_device() function. Please
> let me know if it can help.
> 
> Laurent Pinchart (2):
>   uvcvideo: Factor out video device registration to a function
>   uvcvideo: Report V4L2 device caps through the video_device structure
> 
>  drivers/media/usb/uvc/uvc_driver.c | 77 
> +-
>  drivers/media/usb/uvc/uvc_v4l2.c   |  4 --
>  drivers/media/usb/uvc/uvcvideo.h   |  8 
>  3 files changed, 60 insertions(+), 29 deletions(-)
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

Re: [PATCH v8] uvcvideo: Add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

On Mon, 11 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Monday, 11 December 2017 22:16:23 EET Laurent Pinchart wrote:
> > On Wednesday, 6 December 2017 17:15:40 EET Guennadi Liakhovetski wrote:
> > > From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > > 
> > > Some UVC video cameras contain metadata in their payload headers. This
> > > patch extracts that data, adding more clock synchronisation information,
> > > on both bulk and isochronous endpoints and makes it available to the user
> > > space on a separate video node, using the V4L2_CAP_META_CAPTURE capability
> > > and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. By default, only the
> > > V4L2_META_FMT_UVC pixel format is available from those nodes. However,
> > > cameras can be added to the device ID table to additionally specify their
> > > own metadata format, in which case that format will also become available
> > > from the metadata node.
> > > 
> > > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > > ---
> > > 
> > > v8: addressed comments and integrated changes from Laurent, thanks again,
> > > e.g.:
> > > 
> > > - multiple stylistic changes
> > > - remove the UVC_DEV_FLAG_METADATA_NODE flag / quirk: nodes are now
> > >   created unconditionally
> > > - reuse uvc_ioctl_querycap()
> > > - reuse code in uvc_register_video()
> > > - set an error flag when the metadata buffer overflows
> > > 
> > >  drivers/media/usb/uvc/Makefile   |   2 +-
> > >  drivers/media/usb/uvc/uvc_driver.c   |  15 ++-
> > >  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> > >  drivers/media/usb/uvc/uvc_metadata.c | 179 ++
> > >  drivers/media/usb/uvc/uvc_queue.c|  44 +++--
> > >  drivers/media/usb/uvc/uvc_video.c| 132 --
> > >  drivers/media/usb/uvc/uvcvideo.h |  16 +++-
> > >  include/uapi/linux/uvcvideo.h|  26 +
> > >  8 files changed, 394 insertions(+), 22 deletions(-)
> > >  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> > 
> > [snip]
> > 
> > > diff --git a/drivers/media/usb/uvc/uvc_video.c
> > > b/drivers/media/usb/uvc/uvc_video.c index 13f459e..2fc0bf2 100644
> > > --- a/drivers/media/usb/uvc/uvc_video.c
> > > +++ b/drivers/media/usb/uvc/uvc_video.c
> > 
> > [snip]
> > 
> > > +static void uvc_video_decode_meta(struct uvc_streaming *stream,
> > > +   struct uvc_buffer *meta_buf,
> > > +   const u8 *mem, unsigned int length)
> > > +{
> > > + struct uvc_meta_buf *meta;
> > > + size_t len_std = 2;
> > > + bool has_pts, has_scr;
> > > + unsigned long flags;
> > > + ktime_t time;
> > > + const u8 *scr;
> > > +
> > > + if (!meta_buf || length == 2)
> > > + return;
> > > +
> > > + if (meta_buf->length - meta_buf->bytesused <
> > > + length + sizeof(meta->ns) + sizeof(meta->sof)) {
> > > + meta_buf->error = 1;
> > > + return;
> > > + }
> > > +
> > > + has_pts = mem[1] & UVC_STREAM_PTS;
> > > + has_scr = mem[1] & UVC_STREAM_SCR;
> > > +
> > > + if (has_pts) {
> > > + len_std += 4;
> > > + scr = mem + 6;
> > > + } else {
> > > + scr = mem + 2;
> > > + }
> > > +
> > > + if (has_scr)
> > > + len_std += 6;
> > > +
> > > + if (stream->meta.format == V4L2_META_FMT_UVC)
> > > + length = len_std;
> > > +
> > > + if (length == len_std && (!has_scr ||
> > > +   !memcmp(scr, stream->clock.last_scr, 6)))
> > > + return;
> > > +
> > > + meta = (struct uvc_meta_buf *)((u8 *)meta_buf->mem +
> > > meta_buf->bytesused); +   local_irq_save(flags);
> > > + time = uvc_video_get_time();
> > > + meta->sof = usb_get_current_frame_number(stream->dev->udev);
> > 
> > You need a put_unaligned here too. If you're fine with the patch below
> > there's no need to resubmit, and
> 
> One more thing, __put_unaligned_cpu16() and __put_unaligned_cpu64() don't 
> compile on x86_64 with v4.12 (using media_build.git). I propose replacing 
> them 
> with put_unaligned() which compiles and should do the right 

[PATCH v8] uvcvideo: Add a metadata device node

[Guennadi Liakhovetski]

From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>

Some UVC video cameras contain metadata in their payload headers. This
patch extracts that data, adding more clock synchronisation information,
on both bulk and isochronous endpoints and makes it available to the user
space on a separate video node, using the V4L2_CAP_META_CAPTURE capability
and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. By default, only the
V4L2_META_FMT_UVC pixel format is available from those nodes. However,
cameras can be added to the device ID table to additionally specify their
own metadata format, in which case that format will also become available
from the metadata node.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v8: addressed comments and integrated changes from Laurent, thanks again, 
e.g.:

- multiple stylistic changes
- remove the UVC_DEV_FLAG_METADATA_NODE flag / quirk: nodes are now 
  created unconditionally
- reuse uvc_ioctl_querycap()
- reuse code in uvc_register_video()
- set an error flag when the metadata buffer overflows

 drivers/media/usb/uvc/Makefile   |   2 +-
 drivers/media/usb/uvc/uvc_driver.c   |  15 ++-
 drivers/media/usb/uvc/uvc_isight.c   |   2 +-
 drivers/media/usb/uvc/uvc_metadata.c | 179 +++
 drivers/media/usb/uvc/uvc_queue.c|  44 +++--
 drivers/media/usb/uvc/uvc_video.c| 132 --
 drivers/media/usb/uvc/uvcvideo.h |  16 +++-
 include/uapi/linux/uvcvideo.h|  26 +
 8 files changed, 394 insertions(+), 22 deletions(-)
 create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

diff --git a/drivers/media/usb/uvc/Makefile b/drivers/media/usb/uvc/Makefile
index c26d12f..06c7cd3 100644
--- a/drivers/media/usb/uvc/Makefile
+++ b/drivers/media/usb/uvc/Makefile
@@ -1,5 +1,5 @@
 uvcvideo-objs  := uvc_driver.o uvc_queue.o uvc_v4l2.o uvc_video.o uvc_ctrl.o \
- uvc_status.o uvc_isight.o uvc_debugfs.o
+ uvc_status.o uvc_isight.o uvc_debugfs.o uvc_metadata.o
 ifeq ($(CONFIG_MEDIA_CONTROLLER),y)
 uvcvideo-objs  += uvc_entity.o
 endif
diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index 88032c2..36061f3 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -1883,6 +1883,7 @@ static void uvc_unregister_video(struct uvc_device *dev)
continue;
 
video_unregister_device(>vdev);
+   video_unregister_device(>meta.vdev);
 
uvc_debugfs_cleanup_stream(stream);
}
@@ -1930,6 +1931,9 @@ int uvc_register_video_device(struct uvc_device *dev,
case V4L2_BUF_TYPE_VIDEO_OUTPUT:
vdev->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
break;
+   case V4L2_BUF_TYPE_META_CAPTURE:
+   vdev->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
+   break;
}
 
strlcpy(vdev->name, dev->name, sizeof vdev->name);
@@ -1965,7 +1969,8 @@ static int uvc_register_video(struct uvc_device *dev,
}
 
if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-   stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE;
+   stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE
+   | V4L2_CAP_META_CAPTURE;
else
stream->chain->caps |= V4L2_CAP_VIDEO_OUTPUT;
 
@@ -2003,6 +2008,11 @@ static int uvc_register_terms(struct uvc_device *dev,
if (ret < 0)
return ret;
 
+   /* Register a metadata node, but ignore a possible failure,
+* complete registration of video nodes anyway.
+*/
+   uvc_meta_register(stream);
+
term->vdev = >vdev;
}
 
@@ -2037,6 +2047,7 @@ static int uvc_register_chains(struct uvc_device *dev)
 
 struct uvc_device_info {
u32 quirks;
+   u32 meta_format;
 };
 
 static int uvc_probe(struct usb_interface *intf,
@@ -2074,6 +2085,8 @@ static int uvc_probe(struct usb_interface *intf,
dev->intfnum = intf->cur_altsetting->desc.bInterfaceNumber;
dev->quirks = (uvc_quirks_param == -1)
? quirks : uvc_quirks_param;
+   if (info)
+   dev->meta_format = info->meta_format;
 
if (udev->product != NULL)
strlcpy(dev->name, udev->product, sizeof dev->name);
diff --git a/drivers/media/usb/uvc/uvc_isight.c 
b/drivers/media/usb/uvc/uvc_isight.c
index 8510e725..fb940cf 100644
--- a/drivers/media/usb/uvc/uvc_isight.c
+++ b/drivers/media/usb/uvc/uvc_isight.c
@@ -100,7 +100,7 @@ static int isight_decode(struct uvc_video_queue *queue, 
struct uvc_buffer *buf,
 }
 
 void uvc_video_decode_isight(struct urb *urb, struct uvc_streaming *stream,
-   struct uvc_buffer *buf)
+   struc

Re: [PATCH 3/3 v7] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

While testing the new patch version, we did introduce a couple of 
differences:

1. We cannot (easily) reuse .vidioc_querycap() - the metadata node uses 
v4l2_fh_open() directly, so, it has a different struct file::private_data 
pointer.

2. After your video device unification, the order has swapped: now 
/dev/video0 is a metadata node and /dev/video1 is a video node. Is that 
how you wanted to have this or you don't mind or shall I swap them back? 
For now I've swapped them back, I think that would be more appropriate.

Thanks
Guennadi

Re: [PATCH 3/3 v7] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

On Tue, 5 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday, 5 December 2017 15:44:34 EET Guennadi Liakhovetski wrote:
> > On Tue, 5 Dec 2017, Laurent Pinchart wrote:
> > > On Tuesday, 5 December 2017 12:56:53 EET Guennadi Liakhovetski wrote:
> > >> On Tue, 5 Dec 2017, Laurent Pinchart wrote:
> > >>> On Wednesday, 8 November 2017 18:00:14 EET Guennadi Liakhovetski wrote:
> > >>>> Some UVC video cameras contain metadata in their payload headers.
> > >>>> This patch extracts that data, adding more clock synchronisation
> > >>>> information, on both bulk and isochronous endpoints and makes it
> > >>>> available to the user space on a separate video node, using the
> > >>>> V4L2_CAP_META_CAPTURE capability and the V4L2_BUF_TYPE_META_CAPTURE
> > >>>> buffer queue type. Even though different cameras will have different
> > >>>> metadata formats, we use the same V4L2_META_FMT_UVC pixel format for
> > >>>> all of them. Users have to parse data, based on the specific camera
> > >>>> model information. This version of the patch only creates such metadata
> > >>>> nodes for cameras, specifying a UVC_QUIRK_METADATA_NODE quirk flag.
> > >>> 
> > >>> I don't think this is correct anymore, as we'll use different 4CCs for
> > >>> different vendor metadata. How would you like to rephrase the commit
> > >>> message ?
> > >> 
> > >> Something like
> > >> 
> > >> "
> > >> Some UVC video cameras contain metadata in their payload headers. This
> > >> patch extracts that data, adding more clock synchronisation information,
> > >> on both bulk and isochronous endpoints and makes it available to the
> > >> user space on a separate video node, using the V4L2_CAP_META_CAPTURE
> > >> capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. By
> > >> default, only the V4L2_META_FMT_UVC pixel format is available from those
> > >> nodes. However, cameras can be added to the device ID table to
> > >> additionally specify their own metadata format, in which case that
> > >> format will also become available from the metadata node.
> > >> "
> > > 
> > > Sounds good to me.
> > > 
> > >>>> Signed-off-by: Guennadi Liakhovetski
> > >>>> <guennadi.liakhovet...@intel.com>
> > >>>> ---
> > >>>> 
> > >>>> v7: support up to two metadata formats per camera - the standard one
> > >>>> and an optional private one, if specified in device information
> > >>>> 
> > >>>>  drivers/media/usb/uvc/Makefile   |   2 +-
> > >>>>  drivers/media/usb/uvc/uvc_driver.c   |  15 +++
> > >>>>  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> > >>>>  drivers/media/usb/uvc/uvc_metadata.c | 204 +++
> > >>>>  drivers/media/usb/uvc/uvc_queue.c|  41 +--
> > >>>>  drivers/media/usb/uvc/uvc_video.c| 127 --
> > >>>>  drivers/media/usb/uvc/uvcvideo.h |  19 +++-
> > >>>>  drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
> > >>>>  include/uapi/linux/uvcvideo.h|  26 +
> > >>>>  9 files changed, 416 insertions(+), 21 deletions(-)
> > >>>>  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> > > 
> > > [snip]
> > > 
> > >>>> diff --git a/drivers/media/usb/uvc/uvc_metadata.c
> > >>>> b/drivers/media/usb/uvc/uvc_metadata.c new file mode 100644
> > >>>> index 000..219
> > >>>> --- /dev/null
> > >>>> +++ b/drivers/media/usb/uvc/uvc_metadata.c
> > > 
> > > [snip]
> > > 
> > >>>> +static int uvc_meta_v4l2_querycap(struct file *file, void *fh,
> > >>>> +struct v4l2_capability *cap)
> > >>>> +{
> > >>>> +  struct v4l2_fh *vfh = file->private_data;
> > >>>> +  struct uvc_streaming *stream = video_get_drvdata(vfh->vdev);
> > >>>> +
> > >>>> +  strlcpy(cap->driver, "uvcvideo", sizeof(cap->driver));
> > >>>> +  strlcpy(cap->card, vfh->vdev->name, sizeof(cap->card));
> > >>>> +  u

Re: [PATCH 3/3 v7] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

On Tue, 5 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday, 5 December 2017 12:56:53 EET Guennadi Liakhovetski wrote:
> > On Tue, 5 Dec 2017, Laurent Pinchart wrote:
> > > On Wednesday, 8 November 2017 18:00:14 EET Guennadi Liakhovetski wrote:
> > >> Some UVC video cameras contain metadata in their payload headers. This
> > >> patch extracts that data, adding more clock synchronisation information,
> > >> on both bulk and isochronous endpoints and makes it available to the
> > >> user space on a separate video node, using the V4L2_CAP_META_CAPTURE
> > >> capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
> > >> though different cameras will have different metadata formats, we use
> > >> the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
> > >> parse data, based on the specific camera model information. This
> > >> version of the patch only creates such metadata nodes for cameras,
> > >> specifying a UVC_QUIRK_METADATA_NODE quirk flag.
> > > 
> > > I don't think this is correct anymore, as we'll use different 4CCs for
> > > different vendor metadata. How would you like to rephrase the commit
> > > message ?
> > 
> > Something like
> > 
> > "
> > Some UVC video cameras contain metadata in their payload headers. This
> > patch extracts that data, adding more clock synchronisation information,
> > on both bulk and isochronous endpoints and makes it available to the user
> > space on a separate video node, using the V4L2_CAP_META_CAPTURE capability
> > and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. By default, only the
> > V4L2_META_FMT_UVC pixel format is available from those nodes. However,
> > cameras can be added to the device ID table to additionally specify their
> > own metadata format, in which case that format will also become available
> > from the metadata node.
> > "
> 
> Sounds good to me.
> 
> > >> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >> ---
> > >> 
> > >> v7: support up to two metadata formats per camera - the standard one and
> > >> an optional private one, if specified in device information
> > >> 
> > >>  drivers/media/usb/uvc/Makefile   |   2 +-
> > >>  drivers/media/usb/uvc/uvc_driver.c   |  15 +++
> > >>  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> > >>  drivers/media/usb/uvc/uvc_metadata.c | 204 +
> > >>  drivers/media/usb/uvc/uvc_queue.c|  41 +--
> > >>  drivers/media/usb/uvc/uvc_video.c| 127 --
> > >>  drivers/media/usb/uvc/uvcvideo.h |  19 +++-
> > >>  drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
> > >>  include/uapi/linux/uvcvideo.h|  26 +
> > >>  9 files changed, 416 insertions(+), 21 deletions(-)
> > >>  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> [snip]
> 
> > >> diff --git a/drivers/media/usb/uvc/uvc_metadata.c
> > >> b/drivers/media/usb/uvc/uvc_metadata.c new file mode 100644
> > >> index 000..219
> > >> --- /dev/null
> > >> +++ b/drivers/media/usb/uvc/uvc_metadata.c
> 
> [snip]
> 
> > >> +static int uvc_meta_v4l2_querycap(struct file *file, void *fh,
> > >> +  struct v4l2_capability *cap)
> > >> +{
> > >> +struct v4l2_fh *vfh = file->private_data;
> > >> +struct uvc_streaming *stream = video_get_drvdata(vfh->vdev);
> > >> +
> > >> +strlcpy(cap->driver, "uvcvideo", sizeof(cap->driver));
> > >> +strlcpy(cap->card, vfh->vdev->name, sizeof(cap->card));
> > >> +usb_make_path(stream->dev->udev, cap->bus_info,
> > >> sizeof(cap->bus_info));
> > >> +
> > >> +return 0;
> > >> +}
> > > 
> > > Do you think we could reuse uvc_ioctl_querycap() as-is ?
> > 
> > AFAICS it still has
> > 
> > cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
> >   | chain->caps;
> > 
> > in it, which doesn't suit the metadata node?
> 
> I'd say this is debatable, isn't the capabilities field supposed to include 
> all capabilities from all video nodes for the device ? chain->caps would need 
> to include metadata capability in that case.
> 
> Code re

Re: [PATCH 3/3 v7] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for a review.

On Tue, 5 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch. We're getting very close, I only have small 
> comments, 
> please see below.
> 
> On Wednesday, 8 November 2017 18:00:14 EET Guennadi Liakhovetski wrote:
> > Some UVC video cameras contain metadata in their payload headers. This
> > patch extracts that data, adding more clock synchronisation information,
> > on both bulk and isochronous endpoints and makes it available to the
> > user space on a separate video node, using the V4L2_CAP_META_CAPTURE
> > capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
> > though different cameras will have different metadata formats, we use
> > the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
> > parse data, based on the specific camera model information. This
> > version of the patch only creates such metadata nodes for cameras,
> > specifying a UVC_QUIRK_METADATA_NODE quirk flag.
> 
> I don't think this is correct anymore, as we'll use different 4CCs for 
> different vendor metadata. How would you like to rephrase the commit message ?

Something like

"
Some UVC video cameras contain metadata in their payload headers. This 
patch extracts that data, adding more clock synchronisation information, 
on both bulk and isochronous endpoints and makes it available to the user 
space on a separate video node, using the V4L2_CAP_META_CAPTURE capability 
and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. By default, only the 
V4L2_META_FMT_UVC pixel format is available from those nodes. However, 
cameras can be added to the device ID table to additionally specify their 
own metadata format, in which case that format will also become available 
from the metadata node.
"

> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> > 
> > v7: support up to two metadata formats per camera - the standard one and
> > an optional private one, if specified in device information
> > 
> >  drivers/media/usb/uvc/Makefile   |   2 +-
> >  drivers/media/usb/uvc/uvc_driver.c   |  15 +++
> >  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> >  drivers/media/usb/uvc/uvc_metadata.c | 204 
> >  drivers/media/usb/uvc/uvc_queue.c|  41 +--
> >  drivers/media/usb/uvc/uvc_video.c| 127 --
> >  drivers/media/usb/uvc/uvcvideo.h |  19 +++-
> >  drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
> >  include/uapi/linux/uvcvideo.h|  26 +
> >  9 files changed, 416 insertions(+), 21 deletions(-)
> >  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> 
> [snip]
> 
> > diff --git a/drivers/media/usb/uvc/uvc_driver.c
> > b/drivers/media/usb/uvc/uvc_driver.c index cbf79b9..5f7ce97 100644
> > --- a/drivers/media/usb/uvc/uvc_driver.c
> > +++ b/drivers/media/usb/uvc/uvc_driver.c
> > @@ -1877,6 +1877,7 @@ static void uvc_unregister_video(struct uvc_device
> > *dev) continue;
> > 
> > video_unregister_device(>vdev);
> > +   video_unregister_device(>meta.vdev);
> > 
> > uvc_debugfs_cleanup_stream(stream);
> > }
> > @@ -1934,6 +1935,11 @@ static int uvc_register_video(struct uvc_device *dev,
> > return ret;
> > }
> > 
> > +   /* Register a metadata node, but ignore a possible failure, complete
> > +* registration of video nodes anyway.
> > +*/
> > +   uvc_meta_register(stream);
> > +
> > if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> > stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE;
> > else
> > @@ -2003,6 +2009,7 @@ static int uvc_register_chains(struct uvc_device *dev)
> > 
> >  struct uvc_device_info {
> > u32 quirks;
> > +   u32 meta_format;
> >  };
> > 
> >  static int uvc_probe(struct usb_interface *intf,
> > @@ -2038,8 +2045,16 @@ static int uvc_probe(struct usb_interface *intf,
> > dev->udev = usb_get_dev(udev);
> > dev->intf = usb_get_intf(intf);
> > dev->intfnum = intf->cur_altsetting->desc.bInterfaceNumber;
> > +   if (uvc_quirks_param != -1 &&
> > +   uvc_quirks_param & UVC_DEV_FLAG_METADATA_NODE) {
> > +   uvc_quirks_param &= ~UVC_DEV_FLAG_METADATA_NODE;
> > +   if (uvc_quirks_param == 0)
> > +   uvc_quirks_param = -1;
> > +   }
> 
> I think we can remove the UVC_DEV_FLAG_METADATA_NODE flag. We can use the 
> metadata format specified in the device information structure if present, and 
>

Re: [PATCH 1/2] uvcvideo: Factor out video device registration to a function

[Guennadi Liakhovetski]

On Tue, 5 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday, 5 December 2017 11:14:18 EET Guennadi Liakhovetski wrote:
> > On Tue, 5 Dec 2017, Laurent Pinchart wrote:
> > > The function will then be used to register the video device for metadata
> > > capture.
> > > 
> > > Signed-off-by: Laurent Pinchart <laurent.pinch...@ideasonboard.com>
> > > ---
> > > 
> > >  drivers/media/usb/uvc/uvc_driver.c | 66 -
> > >  drivers/media/usb/uvc/uvcvideo.h   |  8 +
> > >  2 files changed, 49 insertions(+), 25 deletions(-)
> > > 
> > > diff --git a/drivers/media/usb/uvc/uvc_driver.c
> > > b/drivers/media/usb/uvc/uvc_driver.c index f77e31fcfc57..b832929d3382
> > > 100644
> > > --- a/drivers/media/usb/uvc/uvc_driver.c
> > > +++ b/drivers/media/usb/uvc/uvc_driver.c
> > > @@ -24,6 +24,7 @@
> > >  #include 
> > >  
> > >  #include 
> > > +#include 
> > > 
> > >  #include "uvcvideo.h"
> > > @@ -1895,52 +1896,63 @@ static void uvc_unregister_video(struct uvc_device
> > > *dev)
> > 
> > [snip]
> > 
> > >   vdev->release = uvc_release;
> > >   vdev->prio = >chain->prio;
> > > 
> > > - if (stream->type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> > > + if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> > >   vdev->vfl_dir = VFL_DIR_TX;
> > 
> > Why isn't .vfl_dir set for other stream types? Are you jusut relying on
> > VFL_DIR_RX == 0? I'd use a switch (type) here which then would be extended
> > in your next patch with .device_caps fields.
> 
> Yes, and I agree it's not right. How about
> 
>   if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
>   vdev->vfl_dir = VFL_DIR_TX;
>   else
>   vdev->vfl_dir = VFL_DIR_RX;
> 
> Then it won't need to be touched when adding metadata support.

Well, I personally find it a bit less than elegant to have

if (x = a)
...
else
...

switch (x) {
case a:
...
...
}

Also, if I did end up having these two separate, I'd also rather use the 
?: operator for the first one, but in the end it's up to you, I won't 
fight for that :-)

Thanks
Guennadi

Re: [PATCH 1/2] uvcvideo: Factor out video device registration to a function

[Guennadi Liakhovetski]

Hi Laurent,

On Tue, 5 Dec 2017, Laurent Pinchart wrote:

> The function will then be used to register the video device for metadata
> capture.
> 
> Signed-off-by: Laurent Pinchart 
> ---
>  drivers/media/usb/uvc/uvc_driver.c | 66 
> +++---
>  drivers/media/usb/uvc/uvcvideo.h   |  8 +
>  2 files changed, 49 insertions(+), 25 deletions(-)
> 
> diff --git a/drivers/media/usb/uvc/uvc_driver.c 
> b/drivers/media/usb/uvc/uvc_driver.c
> index f77e31fcfc57..b832929d3382 100644
> --- a/drivers/media/usb/uvc/uvc_driver.c
> +++ b/drivers/media/usb/uvc/uvc_driver.c
> @@ -24,6 +24,7 @@
>  #include 
>  
>  #include 
> +#include 
>  
>  #include "uvcvideo.h"
>  
> @@ -1895,52 +1896,63 @@ static void uvc_unregister_video(struct uvc_device 
> *dev)

[snip]

>   vdev->release = uvc_release;
>   vdev->prio = >chain->prio;
> - if (stream->type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
> + if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT)
>   vdev->vfl_dir = VFL_DIR_TX;

Why isn't .vfl_dir set for other stream types? Are you jusut relying on 
VFL_DIR_RX == 0? I'd use a switch (type) here which then would be extended 
in your next patch with .device_caps fields.

Thanks
Guennadi

Re: [PATCH 3/3 v7] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

On Tue, 5 Dec 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday, 5 December 2017 02:24:30 EET Laurent Pinchart wrote:
> > On Wednesday, 8 November 2017 18:00:14 EET Guennadi Liakhovetski wrote:
> 
> [snip]
> 
> > > +static void uvc_video_decode_meta(struct uvc_streaming *stream,
> > > + struct uvc_buffer *buf, struct uvc_buffer *meta_buf,
> > > + u8 *mem, unsigned int length)
> > 
> > The buf parameter is unused, you can remove it. mem isn't modified, I would
> > make it const.
> > 
> > > +{
> > > + struct uvc_meta_buf *meta;
> > > + size_t len_std = 2;
> > > + bool has_pts, has_scr;
> > > + unsigned long flags;
> > > + struct timespec ts;
> > > + u8 *scr;
> > 
> > And scr should be const too.
> > 
> > > +
> > > + if (!meta_buf || length == 2 ||
> > > + meta_buf->length - meta_buf->bytesused <
> > > + length + sizeof(meta->ns) + sizeof(meta->sof))
> > > + return;
> > 
> > If the metadata buffer overflows should we also set the error bit like we do
> > for video buffers ? I have mixed feelings about this, I'd appreciate your
> > input.
> > 
> > > + has_pts = mem[1] & UVC_STREAM_PTS;
> > > + has_scr = mem[1] & UVC_STREAM_SCR;
> > > +
> > > + if (has_pts) {
> > > + len_std += 4;
> > > + scr = mem + 6;
> > > + } else {
> > > + scr = mem + 2;
> > > + }
> > > +
> > > + if (has_scr)
> > > + len_std += 6;
> > > +
> > > + if (stream->cur_meta_format == V4L2_META_FMT_UVC)
> > > + length = len_std;
> > > +
> > > + if (length == len_std && (!has_scr ||
> > > +   !memcmp(scr, stream->clock.last_scr, 6)))
> > > + return;
> > > +
> > > + meta = (struct uvc_meta_buf *)((u8 *)meta_buf->mem +
> > > meta_buf->bytesused); +   local_irq_save(flags);
> > > + uvc_video_get_ts();
> > 
> > FYI, Arnd has posted https://patchwork.kernel.org/patch/10076887/. If the
> > patch gets merged first I can help with the rebasing.
> 
> I've reviewed and merged Arnd patches in my tree, and...
> 
> > > + meta->sof = usb_get_current_frame_number(stream->dev->udev);
> > > + local_irq_restore(flags);
> > > + meta->ns = timespec_to_ns();
> > 
> > The meta pointer can be unaligned as the structure is packed and its size
> > isn't a multiple of the size of the largest field (and it can contain an
> > unspecified amount of vendor data anyway). You thus can't access it directly
> > on all architectures, you will need to use the put_unaligned macro. As I
> > haven't checked whether all architectures can handle unaligned accesses
> > without generating a trap, I would store the USB frame number in a local
> > variable and use the put_unaligned macro output of the IRQ disabled section
> > (feel free to show me that I'm unnecessarily cautious :-)).
> > 
> > > + if (has_scr)
> > > + memcpy(stream->clock.last_scr, scr, 6);
> > > +
> > > + memcpy(>length, mem, length);
> > > + meta_buf->bytesused += length + sizeof(meta->ns) + sizeof(meta->sof);
> > > +
> > > + uvc_trace(UVC_TRACE_FRAME,
> > > +   "%s(): t-sys %lu.%09lus, SOF %u, len %u, flags 0x%x, PTS %u, 
> > > STC 
> %u
> > > frame SOF %u\n",
> > > +   __func__, ts.tv_sec, ts.tv_nsec, meta->sof,
> > > +   meta->length, meta->flags, has_pts ? *(u32 *)meta->buf : 0,
> > > +   has_scr ? *(u32 *)scr : 0,
> > > +   has_scr ? *(u32 *)(scr + 4) & 0x7ff : 0);
> > > +}
> 
> [snip]
> 
> > For your convenience I've rebased your patch series on top of the two
> > patches I mentioned and added another patch on top that contains fixes for
> > all the small issues mentioned above. The result is available at

In your rebased version you've also dropped the hunk for 
drivers/media/v4l2-core/v4l2-ioctl.c adding a description for the new 
V4L2_META_FMT_UVC format - is that on purpose?

Thanks
Guennadi

> > 
> > git://linuxtv.org/pinchartl/media.git uvc/metadata
> > 
> > There are just a handful of issues or questions I haven't addressed, if we
> > handle them I think we'll be good to go.
> 
> ... updated the above branch with a rebased version of the series.
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

Re: [PATCH 15/22] media: soc_camera: fix a kernel-doc markup

[Guennadi Liakhovetski]

On Wed, 29 Nov 2017, Mauro Carvalho Chehab wrote:

> Remove this warning:
>   drivers/media/platform/soc_camera/soc_scale_crop.c:309: warning: Cannot 
> understand  * @icd  - soc-camera device
>on line 309 - I thought it was a doc line
> 
> Signed-off-by: Mauro Carvalho Chehab <mche...@s-opensource.com>

Acked-by: Guennadi Liakhovetski <g.liakhovet...@gmx.de>

Thanks
Guennadi

> ---
>  drivers/media/platform/soc_camera/soc_scale_crop.c | 21 +++--
>  1 file changed, 11 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/media/platform/soc_camera/soc_scale_crop.c 
> b/drivers/media/platform/soc_camera/soc_scale_crop.c
> index 0116097c0c0f..270ec613c27c 100644
> --- a/drivers/media/platform/soc_camera/soc_scale_crop.c
> +++ b/drivers/media/platform/soc_camera/soc_scale_crop.c
> @@ -306,16 +306,17 @@ static int client_set_fmt(struct soc_camera_device *icd,
>  }
>  
>  /**
> - * @icd  - soc-camera device
> - * @rect - camera cropping window
> - * @subrect  - part of rect, sent to the user
> - * @mf   - in- / output camera output window
> - * @width- on input: max host input width
> - * on output: user width, mapped back to input
> - * @height   - on input: max host input height
> - * on output: user height, mapped back to input
> - * @host_can_scale - host can scale this pixel format
> - * @shift- shift, used for scaling
> + * soc_camera_client_scale
> + * @icd: soc-camera device
> + * @rect:camera cropping window
> + * @subrect: part of rect, sent to the user
> + * @mf:  in- / output camera output window
> + * @width:   on input: max host input width;
> + *   on output: user width, mapped back to input
> + * @height:  on input: max host input height;
> + *   on output: user height, mapped back to input
> + * @host_can_scale:  host can scale this pixel format
> + * @shift:   shift, used for scaling
>   */
>  int soc_camera_client_scale(struct soc_camera_device *icd,
>   struct v4l2_rect *rect, struct v4l2_rect *subrect,
> -- 
> 2.14.3
> 

Re: [PATCH 2/3 v7] uvcvideo: add extensible device information

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for reviewing.

On Tue, 28 Nov 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> On Wednesday, 8 November 2017 18:00:13 EET Guennadi Liakhovetski wrote:
> > From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > 
> > Currently the UVC driver assigns a quirk bitmask to the .driver_info
> > field of struct usb_device_id. This patch instroduces a struct to store
> > quirks and possibly other per-device parameters in the future.
> 
> I'd drop the "possibly" as I'm fairly certain we will have other parameters 
> in 
> the future. Otherwise the patch would be a bit pointless :-)
> 
> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> > 
> > v7: this is a new patch in the series. Needed to enable specifying
> > private camera metadata formats
> > 
> >  drivers/media/usb/uvc/uvc_driver.c | 127 --
> >  1 file changed, 78 insertions(+), 49 deletions(-)
> > 
> > diff --git a/drivers/media/usb/uvc/uvc_driver.c
> > b/drivers/media/usb/uvc/uvc_driver.c index 6d22b22..cbf79b9 100644
> > --- a/drivers/media/usb/uvc/uvc_driver.c
> > +++ b/drivers/media/usb/uvc/uvc_driver.c
> > @@ -2001,11 +2001,18 @@ static int uvc_register_chains(struct uvc_device
> > *dev) * USB probe, disconnect, suspend and resume
> >   */
> > 
> > +struct uvc_device_info {
> > +   u32 quirks;
> > +};
> > +
> >  static int uvc_probe(struct usb_interface *intf,
> >  const struct usb_device_id *id)
> >  {
> > struct usb_device *udev = interface_to_usbdev(intf);
> > struct uvc_device *dev;
> > +   const struct uvc_device_info *info =
> > +   (const struct uvc_device_info *)id->driver_info;
> > +   u32 quirks = info ? info->quirks : 0;
> > int function;
> > int ret;
> > 
> > @@ -2032,7 +2039,7 @@ static int uvc_probe(struct usb_interface *intf,
> > dev->intf = usb_get_intf(intf);
> > dev->intfnum = intf->cur_altsetting->desc.bInterfaceNumber;
> > dev->quirks = (uvc_quirks_param == -1)
> > -   ? id->driver_info : uvc_quirks_param;
> > +   ? quirks : uvc_quirks_param;
> > 
> > if (udev->product != NULL)
> > strlcpy(dev->name, udev->product, sizeof dev->name);
> > @@ -2073,7 +2080,7 @@ static int uvc_probe(struct usb_interface *intf,
> > le16_to_cpu(udev->descriptor.idVendor),
> > le16_to_cpu(udev->descriptor.idProduct));
> > 
> > -   if (dev->quirks != id->driver_info) {
> > +   if (dev->quirks != quirks) {
> > uvc_printk(KERN_INFO, "Forcing device quirks to 0x%x by module "
> > "parameter for testing purpose.\n", dev->quirks);
> > uvc_printk(KERN_INFO, "Please report required quirks to the "
> > @@ -2271,6 +2278,28 @@ static int uvc_clock_param_set(const char *val,
> > struct kernel_param *kp) * Driver initialization and cleanup
> >   */
> > 
> > +static struct uvc_device_info uvc_quirk_probe_minmax = {
> > +   .quirks = UVC_QUIRK_PROBE_MINMAX,
> > +};
> > +
> > +static struct uvc_device_info uvc_quirk_fix_bandwidth = {
> > +   .quirks = UVC_QUIRK_FIX_BANDWIDTH,
> > +};
> > +
> > +static struct uvc_device_info uvc_quirk_probe_def = {
> > +   .quirks = UVC_QUIRK_PROBE_DEF,
> > +};
> > +
> > +static struct uvc_device_info uvc_quirk_stream_no_fid = {
> > +   .quirks = UVC_QUIRK_STREAM_NO_FID,
> > +};
> > +
> > +static struct uvc_device_info uvc_quirk_force_y8 = {
> > +   .quirks = UVC_QUIRK_FORCE_Y8,
> > +};
> 
> You can make all of these static const.
> 
> > +#define UVC_QUIRK_INFO(q) (kernel_ulong_t)&(struct uvc_device_info){.quirks
> > = q}
> > +
> >  /*
> >   * The Logitech cameras listed below have their interface class set to
> >   * VENDOR_SPEC because they don't announce themselves as UVC devices, even
> > @@ -2285,7 +2314,7 @@ static int uvc_clock_param_set(const char *val, struct
> > kernel_param *kp) .bInterfaceClass  = USB_CLASS_VIDEO,
> >   .bInterfaceSubClass   = 1,
> >   .bInterfaceProtocol   = 0,
> > - .driver_info  = UVC_QUIRK_PROBE_MINMAX },
> > + .driver_info  = (kernel_ulong_t)_quirk_probe_minmax },
> > /* Genius eFace 2025 */
> > { .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
> > 
> > 

Re: [PATCH 1/3 v7] V4L: Add a UVC Metadata format

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the review.

On Tue, 28 Nov 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> Overall this looks good to me. Please see below for one small comment.

Yes, looks good to me, feel free to use your wording.

Thanks
Guennadi

> On Wednesday, 8 November 2017 18:00:12 EET Guennadi Liakhovetski wrote:
> > From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > 
> > Add a pixel format, used by the UVC driver to stream metadata.
> > 
> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> > 
> > v7: alphabetic order, update documentation.
> > 
> >  Documentation/media/uapi/v4l/meta-formats.rst|  1 +
> >  Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst | 50 +
> >  include/uapi/linux/videodev2.h   |  1 +
> >  3 files changed, 52 insertions(+)
> >  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
> > 
> > diff --git a/Documentation/media/uapi/v4l/meta-formats.rst
> > b/Documentation/media/uapi/v4l/meta-formats.rst index 01e24e3..0c4e1ec
> > 100644
> > --- a/Documentation/media/uapi/v4l/meta-formats.rst
> > +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> > @@ -12,5 +12,6 @@ These formats are used for the :ref:`metadata` interface
> > only.
> >  .. toctree::
> >  :maxdepth: 1
> > 
> > +pixfmt-meta-uvc
> >  pixfmt-meta-vsp1-hgo
> >  pixfmt-meta-vsp1-hgt
> > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
> > b/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst new file mode 100644
> > index 000..06f603c
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
> > @@ -0,0 +1,50 @@
> > +.. -*- coding: utf-8; mode: rst -*-
> > +
> > +.. _v4l2-meta-fmt-uvc:
> > +
> > +***
> > +V4L2_META_FMT_UVC ('UVCH')
> > +***
> > +
> > +UVC Payload Header Data
> > +
> > +
> > +Description
> > +===
> > +
> > +This format describes standard UVC metadata, extracted from UVC packet
> > headers +and provided by the UVC driver through metadata video nodes. That
> > data includes +exact copies of the standard part of UVC Payload Header
> > contents and auxiliary +timing information, required for precise
> > interpretation of timestamps, contained +in those headers. See section
> > "2.4.3.3 Video and Still Image Payload Headers" of +the "UVC 1.5 Class
> > specification" for details.
> > +
> > +Each UVC payload header can be between 2 and 12 bytes large. Buffers can
> > contain +multiple headers, if multiple such headers have been transmitted
> > by the camera +for the respective frame. However, headers, containing no
> > useful information, +e.g. those without the SCR field or with that field
> > identical to the previous +header, will be dropped by the driver.
> 
> If the driver receives too many headers with different SCR (more than the 
> buffer can hold for instance) it will have to drop some of them. The simplest 
> implementation would be to start dropping them when the buffer is full, but 
> I'd like to leave room for the driver to be a bit more clever and drop 
> headers 
> that have a SCR too close to the previous one for instance. I propose wording 
> the above paragraph as follows.
> 
> "Each UVC payload header can be between 2 and 12 bytes large. Buffers can
> contain multiple headers, if multiple such headers have been transmitted
> by the camera for the respective frame. However, the driver may drop headers 
> when the buffer is full, when they contain no useful information (e.g. those 
> without the SCR field or with that field identical to the previous header), 
> or 
> generally to perform rate limiting when the device sends a large number of 
> headers".
> 
> If you're fine with this there's no need to resent, I can update the 
> documentation when applying, and
> 
> Reviewed-by: Laurent Pinchart <laurent.pinch...@ideasonboard.com>
> 
> > +Each individual block contains the following fields:
> > +
> > +.. flat-table:: UVC Metadata Block
> > +:widths: 1 4
> > +:header-rows:  1
> > +:stub-columns: 0
> > +
> > +* - Field
> > +  - Description
> > +* - __u64 ts;
> > +  - system timestamp in host byte order, measured by the driver upon
> > +reception of the payload
> > +* - __u16 sof;
> > +  - USB Frame Number in host byte order, also obtained by

Re: [PATCH 2/6 v5] V4L: Add a UVC Metadata format

[Guennadi Liakhovetski]

Hi Laurent,

On Thu, 9 Nov 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Wednesday, 8 November 2017 12:43:46 EET Guennadi Liakhovetski wrote:

[snip]

> > To recall the metadata buffer layout should be
> > 
> > struct uvc_meta_buf {
> > uint64_t ns;
> > uint16_t sof;
> > uint8_t length;
> > uint8_t flags;
> > uint8_t buf[];
> > } __attribute__((packed));
> > 
> > where all the fields, beginning with "length" are a direct copy from the
> > UVC payload header. If multiple such payload headers have arrived for a
> > single frame, they will be appended and .bytesused will as usually have
> > the total byte count, used up in this frame. An application would then
> > calculate lengths of those individual metadata blocks as
> > 
> > sizeof(.ns) + sizeof(.sof) + .length
> > 
> > But this won't work with the "standard" UVC metadata format where any
> > private data, following standard fields, are dropped. In that case
> > applications would have to look at .flags and calculate the block length
> > based on them. Another possibility would be to rewrite the .length field
> > in the driver to only include standard fields, but I really don't think
> > that would be a good idea.
> 
> For the standard header the length can indeed be easily computed from the 
> flags. I wonder, however, why you think rewriting length would be a bad idea ?

Because I like the guarantee of the least intrusion. We (so far) 
guarantee, that the buffer contents beyond the system and the USB 
timestamps are a direct unmodified copy from the camera. This seems to be 
a good principle to stick to.

Thanks
Guennadi

[PATCH 3/3 v7] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Some UVC video cameras contain metadata in their payload headers. This
patch extracts that data, adding more clock synchronisation information,
on both bulk and isochronous endpoints and makes it available to the
user space on a separate video node, using the V4L2_CAP_META_CAPTURE
capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
though different cameras will have different metadata formats, we use
the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
parse data, based on the specific camera model information. This
version of the patch only creates such metadata nodes for cameras,
specifying a UVC_QUIRK_METADATA_NODE quirk flag.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v7: support up to two metadata formats per camera - the standard one and
an optional private one, if specified in device information

 drivers/media/usb/uvc/Makefile   |   2 +-
 drivers/media/usb/uvc/uvc_driver.c   |  15 +++
 drivers/media/usb/uvc/uvc_isight.c   |   2 +-
 drivers/media/usb/uvc/uvc_metadata.c | 204 +++
 drivers/media/usb/uvc/uvc_queue.c|  41 +--
 drivers/media/usb/uvc/uvc_video.c| 127 --
 drivers/media/usb/uvc/uvcvideo.h |  19 +++-
 drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
 include/uapi/linux/uvcvideo.h|  26 +
 9 files changed, 416 insertions(+), 21 deletions(-)
 create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

diff --git a/drivers/media/usb/uvc/Makefile b/drivers/media/usb/uvc/Makefile
index c26d12f..06c7cd3 100644
--- a/drivers/media/usb/uvc/Makefile
+++ b/drivers/media/usb/uvc/Makefile
@@ -1,5 +1,5 @@
 uvcvideo-objs  := uvc_driver.o uvc_queue.o uvc_v4l2.o uvc_video.o uvc_ctrl.o \
- uvc_status.o uvc_isight.o uvc_debugfs.o
+ uvc_status.o uvc_isight.o uvc_debugfs.o uvc_metadata.o
 ifeq ($(CONFIG_MEDIA_CONTROLLER),y)
 uvcvideo-objs  += uvc_entity.o
 endif
diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index cbf79b9..5f7ce97 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -1877,6 +1877,7 @@ static void uvc_unregister_video(struct uvc_device *dev)
continue;
 
video_unregister_device(>vdev);
+   video_unregister_device(>meta.vdev);
 
uvc_debugfs_cleanup_stream(stream);
}
@@ -1934,6 +1935,11 @@ static int uvc_register_video(struct uvc_device *dev,
return ret;
}
 
+   /* Register a metadata node, but ignore a possible failure, complete
+* registration of video nodes anyway.
+*/
+   uvc_meta_register(stream);
+
if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE;
else
@@ -2003,6 +2009,7 @@ static int uvc_register_chains(struct uvc_device *dev)
 
 struct uvc_device_info {
u32 quirks;
+   u32 meta_format;
 };
 
 static int uvc_probe(struct usb_interface *intf,
@@ -2038,8 +2045,16 @@ static int uvc_probe(struct usb_interface *intf,
dev->udev = usb_get_dev(udev);
dev->intf = usb_get_intf(intf);
dev->intfnum = intf->cur_altsetting->desc.bInterfaceNumber;
+   if (uvc_quirks_param != -1 &&
+   uvc_quirks_param & UVC_DEV_FLAG_METADATA_NODE) {
+   uvc_quirks_param &= ~UVC_DEV_FLAG_METADATA_NODE;
+   if (uvc_quirks_param == 0)
+   uvc_quirks_param = -1;
+   }
dev->quirks = (uvc_quirks_param == -1)
? quirks : uvc_quirks_param;
+   if (info)
+   dev->meta_format = info->meta_format;
 
if (udev->product != NULL)
strlcpy(dev->name, udev->product, sizeof dev->name);
diff --git a/drivers/media/usb/uvc/uvc_isight.c 
b/drivers/media/usb/uvc/uvc_isight.c
index 8510e725..fb940cf 100644
--- a/drivers/media/usb/uvc/uvc_isight.c
+++ b/drivers/media/usb/uvc/uvc_isight.c
@@ -100,7 +100,7 @@ static int isight_decode(struct uvc_video_queue *queue, 
struct uvc_buffer *buf,
 }
 
 void uvc_video_decode_isight(struct urb *urb, struct uvc_streaming *stream,
-   struct uvc_buffer *buf)
+   struct uvc_buffer *buf, struct uvc_buffer *meta_buf)
 {
int ret, i;
 
diff --git a/drivers/media/usb/uvc/uvc_metadata.c 
b/drivers/media/usb/uvc/uvc_metadata.c
new file mode 100644
index 000..219
--- /dev/null
+++ b/drivers/media/usb/uvc/uvc_metadata.c
@@ -0,0 +1,204 @@
+/*
+ *  uvc_metadata.c  --  USB Video Class driver - Metadata handling
+ *
+ *  Copyright (C) 2016
+ *  Guennadi Liakhovetski (guennadi.liakhovet...@intel.com)
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published 

[PATCH 2/3 v7] uvcvideo: add extensible device information

[Guennadi Liakhovetski]

From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>

Currently the UVC driver assigns a quirk bitmask to the .driver_info
field of struct usb_device_id. This patch instroduces a struct to store
quirks and possibly other per-device parameters in the future.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v7: this is a new patch in the series. Needed to enable specifying
private camera metadata formats

 drivers/media/usb/uvc/uvc_driver.c | 127 +++--
 1 file changed, 78 insertions(+), 49 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index 6d22b22..cbf79b9 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -2001,11 +2001,18 @@ static int uvc_register_chains(struct uvc_device *dev)
  * USB probe, disconnect, suspend and resume
  */
 
+struct uvc_device_info {
+   u32 quirks;
+};
+
 static int uvc_probe(struct usb_interface *intf,
 const struct usb_device_id *id)
 {
struct usb_device *udev = interface_to_usbdev(intf);
struct uvc_device *dev;
+   const struct uvc_device_info *info =
+   (const struct uvc_device_info *)id->driver_info;
+   u32 quirks = info ? info->quirks : 0;
int function;
int ret;
 
@@ -2032,7 +2039,7 @@ static int uvc_probe(struct usb_interface *intf,
dev->intf = usb_get_intf(intf);
dev->intfnum = intf->cur_altsetting->desc.bInterfaceNumber;
dev->quirks = (uvc_quirks_param == -1)
-   ? id->driver_info : uvc_quirks_param;
+   ? quirks : uvc_quirks_param;
 
if (udev->product != NULL)
strlcpy(dev->name, udev->product, sizeof dev->name);
@@ -2073,7 +2080,7 @@ static int uvc_probe(struct usb_interface *intf,
le16_to_cpu(udev->descriptor.idVendor),
le16_to_cpu(udev->descriptor.idProduct));
 
-   if (dev->quirks != id->driver_info) {
+   if (dev->quirks != quirks) {
uvc_printk(KERN_INFO, "Forcing device quirks to 0x%x by module "
"parameter for testing purpose.\n", dev->quirks);
uvc_printk(KERN_INFO, "Please report required quirks to the "
@@ -2271,6 +2278,28 @@ static int uvc_clock_param_set(const char *val, struct 
kernel_param *kp)
  * Driver initialization and cleanup
  */
 
+static struct uvc_device_info uvc_quirk_probe_minmax = {
+   .quirks = UVC_QUIRK_PROBE_MINMAX,
+};
+
+static struct uvc_device_info uvc_quirk_fix_bandwidth = {
+   .quirks = UVC_QUIRK_FIX_BANDWIDTH,
+};
+
+static struct uvc_device_info uvc_quirk_probe_def = {
+   .quirks = UVC_QUIRK_PROBE_DEF,
+};
+
+static struct uvc_device_info uvc_quirk_stream_no_fid = {
+   .quirks = UVC_QUIRK_STREAM_NO_FID,
+};
+
+static struct uvc_device_info uvc_quirk_force_y8 = {
+   .quirks = UVC_QUIRK_FORCE_Y8,
+};
+
+#define UVC_QUIRK_INFO(q) (kernel_ulong_t)&(struct uvc_device_info){.quirks = 
q}
+
 /*
  * The Logitech cameras listed below have their interface class set to
  * VENDOR_SPEC because they don't announce themselves as UVC devices, even
@@ -2285,7 +2314,7 @@ static int uvc_clock_param_set(const char *val, struct 
kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_PROBE_MINMAX },
+ .driver_info  = (kernel_ulong_t)_quirk_probe_minmax },
/* Genius eFace 2025 */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
| USB_DEVICE_ID_MATCH_INT_INFO,
@@ -2294,7 +2323,7 @@ static int uvc_clock_param_set(const char *val, struct 
kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_PROBE_MINMAX },
+ .driver_info  = (kernel_ulong_t)_quirk_probe_minmax },
/* Microsoft Lifecam NX-6000 */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
| USB_DEVICE_ID_MATCH_INT_INFO,
@@ -2303,7 +2332,7 @@ static int uvc_clock_param_set(const char *val, struct 
kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VIDEO,
  .bInterfaceSubClass   = 1,
  .bInterfaceProtocol   = 0,
- .driver_info  = UVC_QUIRK_PROBE_MINMAX },
+ .driver_info  = (kernel_ulong_t)_quirk_probe_minmax },
/* Microsoft Lifecam NX-3000 */
{ .match_flags  = USB_DEVICE_ID_MATCH_DEVICE
| USB_DEVICE_ID_MATCH_INT_INFO,
@@ -2312,7 +2341,7 @@ static int uvc_clock_param_set(const char *val, struct 
kernel_param *kp)
  .bInterfaceClass  = USB_CLASS_VID

[PATCH 0/3 v7] uvcvideo: metadata nodes

[Guennadi Liakhovetski]

From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>

Comments by Laurent and Hans addressed, thanks.

Regards
Guennadi

Guennadi Liakhovetski (3):
  V4L: Add a UVC Metadata format
  uvcvideo: add extensible device information
  uvcvideo: add a metadata device node

 Documentation/media/uapi/v4l/meta-formats.rst|   1 +
 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst |  50 ++
 drivers/media/usb/uvc/Makefile   |   2 +-
 drivers/media/usb/uvc/uvc_driver.c   | 142 ++--
 drivers/media/usb/uvc/uvc_isight.c   |   2 +-
 drivers/media/usb/uvc/uvc_metadata.c | 204 +++
 drivers/media/usb/uvc/uvc_queue.c|  41 -
 drivers/media/usb/uvc/uvc_video.c| 127 --
 drivers/media/usb/uvc/uvcvideo.h |  19 ++-
 drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
 include/uapi/linux/uvcvideo.h|  26 +++
 include/uapi/linux/videodev2.h   |   1 +
 12 files changed, 546 insertions(+), 70 deletions(-)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
 create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

-- 
1.9.3

[PATCH 1/3 v7] V4L: Add a UVC Metadata format

[Guennadi Liakhovetski]

From: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>

Add a pixel format, used by the UVC driver to stream metadata.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v7: alphabetic order, update documentation.

 Documentation/media/uapi/v4l/meta-formats.rst|  1 +
 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst | 50 
 include/uapi/linux/videodev2.h   |  1 +
 3 files changed, 52 insertions(+)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst

diff --git a/Documentation/media/uapi/v4l/meta-formats.rst 
b/Documentation/media/uapi/v4l/meta-formats.rst
index 01e24e3..0c4e1ec 100644
--- a/Documentation/media/uapi/v4l/meta-formats.rst
+++ b/Documentation/media/uapi/v4l/meta-formats.rst
@@ -12,5 +12,6 @@ These formats are used for the :ref:`metadata` interface only.
 .. toctree::
 :maxdepth: 1
 
+pixfmt-meta-uvc
 pixfmt-meta-vsp1-hgo
 pixfmt-meta-vsp1-hgt
diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst 
b/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
new file mode 100644
index 000..06f603c
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
@@ -0,0 +1,50 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _v4l2-meta-fmt-uvc:
+
+***
+V4L2_META_FMT_UVC ('UVCH')
+***
+
+UVC Payload Header Data
+
+
+Description
+===
+
+This format describes standard UVC metadata, extracted from UVC packet headers
+and provided by the UVC driver through metadata video nodes. That data includes
+exact copies of the standard part of UVC Payload Header contents and auxiliary
+timing information, required for precise interpretation of timestamps, 
contained
+in those headers. See section "2.4.3.3 Video and Still Image Payload Headers" 
of
+the "UVC 1.5 Class specification" for details.
+
+Each UVC payload header can be between 2 and 12 bytes large. Buffers can 
contain
+multiple headers, if multiple such headers have been transmitted by the camera
+for the respective frame. However, headers, containing no useful information,
+e.g. those without the SCR field or with that field identical to the previous
+header, will be dropped by the driver.
+
+Each individual block contains the following fields:
+
+.. flat-table:: UVC Metadata Block
+:widths: 1 4
+:header-rows:  1
+:stub-columns: 0
+
+* - Field
+  - Description
+* - __u64 ts;
+  - system timestamp in host byte order, measured by the driver upon
+reception of the payload
+* - __u16 sof;
+  - USB Frame Number in host byte order, also obtained by the driver as
+close as possible to the above timestamp to enable correlation between
+them
+* - :cspan:`1` *The rest is an exact copy of the UVC payload header:*
+* - __u8 length;
+  - length of the rest of the block, including this field
+* - __u8 flags;
+  - Flags, indicating presence of other standard UVC fields
+* - __u8 buf[];
+  - The rest of the header, possibly including UVC PTS and SCR fields
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 185d6a0..0d07b2d 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -687,6 +687,7 @@ struct v4l2_pix_format {
 /* Meta-data formats */
 #define V4L2_META_FMT_VSP1_HGOv4l2_fourcc('V', 'S', 'P', 'H') /* R-Car 
VSP1 1-D Histogram */
 #define V4L2_META_FMT_VSP1_HGTv4l2_fourcc('V', 'S', 'P', 'T') /* R-Car 
VSP1 2-D Histogram */
+#define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC 
Payload Header metadata */
 
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC0xfeedcafe
-- 
1.9.3

Re: [PATCH 2/6 v5] V4L: Add a UVC Metadata format

[Guennadi Liakhovetski]

Hi Laurent, Hans,

On Tue, 7 Nov 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Monday, 6 November 2017 16:53:10 EET Guennadi Liakhovetski wrote:
> > On Mon, 30 Oct 2017, Hans Verkuil wrote:
> > > On 07/28/2017 02:46 PM, Hans Verkuil wrote:
> > >> On 07/28/2017 02:33 PM, Guennadi Liakhovetski wrote:
> > >>> Add a pixel format, used by the UVC driver to stream metadata.
> > >>> 
> > >>> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >>> ---
> > >>> 
> > >>>  Documentation/media/uapi/v4l/meta-formats.rst|  1 +
> > >>>  Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst | 39 +
> > >>>  include/uapi/linux/videodev2.h   |  1 +
> > >>>  3 files changed, 41 insertions(+)
> > >>>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
> > 
> > [snip]
> > 
> > >>> diff --git a/include/uapi/linux/videodev2.h
> > >>> b/include/uapi/linux/videodev2.h index 45cf735..0aad91c 100644
> > >>> --- a/include/uapi/linux/videodev2.h
> > >>> +++ b/include/uapi/linux/videodev2.h
> > >>> @@ -682,6 +682,7 @@ struct v4l2_pix_format {
> > >>> 
> > >>>  /* Meta-data formats */
> > >>>  #define V4L2_META_FMT_VSP1_HGOv4l2_fourcc('V', 'S', 'P', 'H') /*
> > >>>  R-Car VSP1 1-D Histogram */ #define V4L2_META_FMT_VSP1_HGT   
> > >>>  v4l2_fourcc('V', 'S', 'P', 'T') /* R-Car VSP1 2-D Histogram */> >> 
> > >>> +#define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /*
> > >>> UVC Payload Header metadata */
> > > 
> > > I discussed this with Laurent last week and since the metadata for UVC
> > > starts with a standard header followed by vendor-specific data it makes
> > > sense to use V4L2_META_FMT_UVC for just the standard header. Any vendor
> > > specific formats should have their own fourcc which starts with the
> > > standard header followed by the custom data. The UVC driver would
> > > enumerate both the standard and the vendor specific fourcc. This would
> > > allow generic UVC applications to use the standard header. Applications
> > > that know about the vendor specific data can select the vendor specific
> > > format.
> > > 
> > > This change would make this much more convenient to use.
> > 
> > Then the driver should be able to decide, which private fourcc code to use
> > for each of those devices. A natural way to do that seems to be to put
> > that in the .driver_info field of struct usb_device_id. For that I'd
> > replace the current use of that field for quirks with a pointer to a
> > struct in a separate patch. Laurent, would that be acceptable? Then add a
> > field to that struct for a private metadata fourcc code.
> 
> I've been thinking about doing so for some time now. If you can write a patch 
> it would be great ! What I've been wondering is how to keep the code both 
> readable and small. If we declared those structures separately from the 
> devices array we could use one instance for multiple devices, but naming 
> might 
> become awkward. On the other hand, if we defined them inline within the 
> devices array, we'd get rid of the naming issue, but at the expense of 
> increased memory usage.

We have in the meantime agreed to always use structs and to create named 
structs for reoccurring quirk flags and in-place structs for 
single-occurrance flags.

While implementing this I came across yet one more compromise, that we'll 
have to accept with this approach:

To recall the metadata buffer layout should be

struct uvc_meta_buf {
uint64_t ns;
uint16_t sof;
uint8_t length;
uint8_t flags;
uint8_t buf[];
} __attribute__((packed));

where all the fields, beginning with "length" are a direct copy from the 
UVC payload header. If multiple such payload headers have arrived for a 
single frame, they will be appended and .bytesused will as usually have 
the total byte count, used up in this frame. An application would then 
calculate lengths of those individual metadata blocks as

sizeof(.ns) + sizeof(.sof) + .length

But this won't work with the "standard" UVC metadata format where any 
private data, following standard fields, are dropped. In that case 
applications would have to look at .flags and calculate the block length 
based on them. Another possibility would be to rewrite the .length field 
in the driver to only include standard fields, but I really don't think 
that would be a good idea.

Thanks
Guennadi

> One middle-ground option would be to allow storing either a structure pointer 
> or quirks flags in the field, relying on the fact that the low order bit of a 
> pointer will be NULL. We could repurpose flag BIT(0) to indicate that the 
> field contains flags instead of a pointer.
> 
> Maybe I'm over-engineering this and that the extra memory consumption won't 
> be 
> too bad, or separately defined structures will be easy to name. I'd 
> appreciate 
> your opinion on this matter.
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

Re: [PATCH 2/6 v5] V4L: Add a UVC Metadata format

[Guennadi Liakhovetski]

Hi Hans,

Thanks for the comments.

On Mon, 30 Oct 2017, Hans Verkuil wrote:

> Hi Guennadi,
> 
> On 07/28/2017 02:46 PM, Hans Verkuil wrote:
> > On 07/28/2017 02:33 PM, Guennadi Liakhovetski wrote:
> >> Add a pixel format, used by the UVC driver to stream metadata.
> >>
> >> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> >> ---
> >>  Documentation/media/uapi/v4l/meta-formats.rst|  1 +
> >>  Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst | 39 
> >> 
> >>  include/uapi/linux/videodev2.h   |  1 +
> >>  3 files changed, 41 insertions(+)
> >>  create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst

[snip]

> >> diff --git a/include/uapi/linux/videodev2.h 
> >> b/include/uapi/linux/videodev2.h
> >> index 45cf735..0aad91c 100644
> >> --- a/include/uapi/linux/videodev2.h
> >> +++ b/include/uapi/linux/videodev2.h
> >> @@ -682,6 +682,7 @@ struct v4l2_pix_format {
> >>  /* Meta-data formats */
> >>  #define V4L2_META_FMT_VSP1_HGOv4l2_fourcc('V', 'S', 'P', 'H') /* 
> >> R-Car VSP1 1-D Histogram */
> >>  #define V4L2_META_FMT_VSP1_HGTv4l2_fourcc('V', 'S', 'P', 'T') /* 
> >> R-Car VSP1 2-D Histogram */
> >> +#define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC 
> >> Payload Header metadata */
> 
> I discussed this with Laurent last week and since the metadata for UVC starts
> with a standard header followed by vendor-specific data it makes sense to
> use V4L2_META_FMT_UVC for just the standard header. Any vendor specific 
> formats
> should have their own fourcc which starts with the standard header followed by
> the custom data. The UVC driver would enumerate both the standard and the 
> vendor
> specific fourcc. This would allow generic UVC applications to use the standard
> header. Applications that know about the vendor specific data can select the
> vendor specific format.
> 
> This change would make this much more convenient to use.

Then the driver should be able to decide, which private fourcc code to use 
for each of those devices. A natural way to do that seems to be to put 
that in the .driver_info field of struct usb_device_id. For that I'd 
replace the current use of that field for quirks with a pointer to a 
struct in a separate patch. Laurent, would that be acceptable? Then add a 
field to that struct for a private metadata fourcc code.

Thanks
Guennadi

> Regards,
> 
>   Hans
> 
> >>  /* priv field value to indicates that subsequent fields are valid. */
> >>  #define V4L2_PIX_FMT_PRIV_MAGIC   0xfeedcafe

Re: [PATCH resend] [media] uvcvideo: zero seq number when disabling stream

[Guennadi Liakhovetski]

Hi Laurent,

On Mon, 16 Oct 2017, Laurent Pinchart wrote:

> Hi Hans,
> 
> (CC'ing Guennadi Liakhovetski)
> 
> Thank you for the patch.
> 
> On Friday, 15 September 2017 09:27:51 EEST Hans Yang wrote:
> > For bulk-based devices, when disabling the video stream,
> > in addition to issue CLEAR_FEATURE(HALT), it is better to set
> > alternate setting 0 as well or the sequnce number in host
> > side will probably not reset to zero.
> 
> The USB 2.0 specificatin states in the description of the SET_INTERFACE 
> request that "If a device only supports a default setting for the specified 
> interface, then a STALL may be returned in the Status stage of the request".
> 
> The Linux implementation of usb_set_interface() deals with this by handling 
> STALL conditions and manually clearing HALT for all endpoints in the 
> interface, but I'm still concerned that this change could break existing bulk-
> based cameras. Do you know what other operating systems do when disabling the 
> stream on bulk cameras ? According to a comment in the driver Windows calls 
> CLEAR_FEATURE(HALT), but the situation might have changed since that was 
> tested.
> 
> Guennadi, how do your bulk-based cameras handle this ?

I don't know what design decisions are implemented there, but I tested a 
couple of cameras for sending a STREAMOFF after a few captured buffers, 
sleeping for a couple of seconds, requeuing buffers, sending STREAMON and 
capturing a few more - that seems to have worked. "Seems" because I only 
used a modified version of capture-example, I haven't checked buffer 
contents.

BTW, Hans, may I ask - what cameras did you use?

Thanks
Guennadi

> > Then in next time video stream start, the device will expect
> > host starts packet from 0 sequence number but host actually
> > continue the sequence number from last transaction and this
> > causes transaction errors.
> 
> Do you mean the DATA0/DATA1 toggle ? Why does the host continue toggling the 
> PID from the last transation ? The usb_clear_halt() function calls 
> usb_reset_endpoint() which should reset the DATA PID toggle.
> 
> > This commit fixes this by adding set alternate setting 0 back
> > as what isoch-based devices do.
> > 
> > Below error message will also be eliminated for some devices:
> > uvcvideo: Non-zero status (-71) in video completion handler.
> > 
> > Signed-off-by: Hans Yang <ha...@nvidia.com>
> > ---
> >  drivers/media/usb/uvc/uvc_video.c | 5 ++---
> >  1 file changed, 2 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/media/usb/uvc/uvc_video.c
> > b/drivers/media/usb/uvc/uvc_video.c index fb86d6af398d..ad80c2a6da6a 100644
> > --- a/drivers/media/usb/uvc/uvc_video.c
> > +++ b/drivers/media/usb/uvc/uvc_video.c
> > @@ -1862,10 +1862,9 @@ int uvc_video_enable(struct uvc_streaming *stream,
> > int enable)
> > 
> > if (!enable) {
> > uvc_uninit_video(stream, 1);
> > -   if (stream->intf->num_altsetting > 1) {
> > -   usb_set_interface(stream->dev->udev,
> > +   usb_set_interface(stream->dev->udev,
> >   stream->intfnum, 0);
> > -   } else {
> > +   if (stream->intf->num_altsetting == 1) {
> > /* UVC doesn't specify how to inform a bulk-based device
> >  * when the video stream is stopped. Windows sends a
> >  * CLEAR_FEATURE(HALT) request to the video streaming
> 
> -- 
> Regards,
> 
> Laurent Pinchart
> 

[PATCH 4/6 v6] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Some UVC video cameras contain metadata in their payload headers. This
patch extracts that data, adding more clock synchronisation information,
on both bulk and isochronous endpoints and makes it available to the
user space on a separate video node, using the V4L2_CAP_META_CAPTURE
capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
though different cameras will have different metadata formats, we use
the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
parse data, based on the specific camera model information. This
version of the patch only creates such metadata nodes for cameras,
specifying a UVC_QUIRK_METADATA_NODE quirk flag.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v6:
 - remove an unrelated uvc_ctrl.c chunk
 - update comments in uvc_meta_register()
 - only process metadata buffers when a metadata node is used

 drivers/media/usb/uvc/Makefile   |   2 +-
 drivers/media/usb/uvc/uvc_driver.c   |  12 
 drivers/media/usb/uvc/uvc_isight.c   |   2 +-
 drivers/media/usb/uvc/uvc_metadata.c | 132 +++
 drivers/media/usb/uvc/uvc_queue.c|  41 +--
 drivers/media/usb/uvc/uvc_video.c| 124 +---
 drivers/media/usb/uvc/uvcvideo.h |  17 -
 drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
 include/uapi/linux/uvcvideo.h|  26 +++
 9 files changed, 336 insertions(+), 21 deletions(-)
 create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

diff --git a/drivers/media/usb/uvc/Makefile b/drivers/media/usb/uvc/Makefile
index c26d12f..06c7cd3 100644
--- a/drivers/media/usb/uvc/Makefile
+++ b/drivers/media/usb/uvc/Makefile
@@ -1,5 +1,5 @@
 uvcvideo-objs  := uvc_driver.o uvc_queue.o uvc_v4l2.o uvc_video.o uvc_ctrl.o \
- uvc_status.o uvc_isight.o uvc_debugfs.o
+ uvc_status.o uvc_isight.o uvc_debugfs.o uvc_metadata.o
 ifeq ($(CONFIG_MEDIA_CONTROLLER),y)
 uvcvideo-objs  += uvc_entity.o
 endif
diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index cfe33bf..3d61cec 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -1884,6 +1884,7 @@ static void uvc_unregister_video(struct uvc_device *dev)
continue;
 
video_unregister_device(>vdev);
+   video_unregister_device(>meta.vdev);
 
uvc_debugfs_cleanup_stream(stream);
}
@@ -1944,6 +1945,11 @@ static int uvc_register_video(struct uvc_device *dev,
return ret;
}
 
+   /* Register a metadata node, but ignore a possible failure, complete
+* registration of video nodes anyway.
+*/
+   uvc_meta_register(stream);
+
if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE;
else
@@ -2041,6 +2047,12 @@ static int uvc_probe(struct usb_interface *intf,
dev->udev = usb_get_dev(udev);
dev->intf = usb_get_intf(intf);
dev->intfnum = intf->cur_altsetting->desc.bInterfaceNumber;
+   if (uvc_quirks_param != -1 &&
+   uvc_quirks_param & UVC_DEV_FLAG_METADATA_NODE) {
+   uvc_quirks_param &= ~UVC_DEV_FLAG_METADATA_NODE;
+   if (uvc_quirks_param == 0)
+   uvc_quirks_param = -1;
+   }
dev->quirks = (uvc_quirks_param == -1)
? id->driver_info : uvc_quirks_param;
 
diff --git a/drivers/media/usb/uvc/uvc_isight.c 
b/drivers/media/usb/uvc/uvc_isight.c
index 8510e725..fb940cf 100644
--- a/drivers/media/usb/uvc/uvc_isight.c
+++ b/drivers/media/usb/uvc/uvc_isight.c
@@ -100,7 +100,7 @@ static int isight_decode(struct uvc_video_queue *queue, 
struct uvc_buffer *buf,
 }
 
 void uvc_video_decode_isight(struct urb *urb, struct uvc_streaming *stream,
-   struct uvc_buffer *buf)
+   struct uvc_buffer *buf, struct uvc_buffer *meta_buf)
 {
int ret, i;
 
diff --git a/drivers/media/usb/uvc/uvc_metadata.c 
b/drivers/media/usb/uvc/uvc_metadata.c
new file mode 100644
index 000..cbcddcd
--- /dev/null
+++ b/drivers/media/usb/uvc/uvc_metadata.c
@@ -0,0 +1,132 @@
+/*
+ *  uvc_metadata.c  --  USB Video Class driver - Metadata handling
+ *
+ *  Copyright (C) 2016
+ *  Guennadi Liakhovetski (guennadi.liakhovet...@intel.com)
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "uvcvideo.h"
+
+/* 

Re: [PATCH 4/6 v5] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Hi Hans,

Thanks for your comment.

On Fri, 28 Jul 2017, Hans Verkuil wrote:

> On 07/28/2017 02:33 PM, Guennadi Liakhovetski wrote:
> > Some UVC video cameras contain metadata in their payload headers. This
> > patch extracts that data, adding more clock synchronisation information,
> > on both bulk and isochronous endpoints and makes it available to the
> > user space on a separate video node, using the V4L2_CAP_META_CAPTURE
> > capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
> > though different cameras will have different metadata formats, we use
> > the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
> > parse data, based on the specific camera model information. This
> > version of the patch only creates such metadata nodes for cameras,
> > specifying a UVC_QUIRK_METADATA_NODE quirk flag.
> > 
> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> >  drivers/media/usb/uvc/Makefile   |   2 +-
> >  drivers/media/usb/uvc/uvc_ctrl.c |   3 +
> >  drivers/media/usb/uvc/uvc_driver.c   |  12 +++
> >  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> >  drivers/media/usb/uvc/uvc_metadata.c | 139 
> > +++
> >  drivers/media/usb/uvc/uvc_queue.c|  41 +--
> >  drivers/media/usb/uvc/uvc_video.c| 119 +++---
> >  drivers/media/usb/uvc/uvcvideo.h |  17 -
> >  drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
> >  include/uapi/linux/uvcvideo.h|  26 +++
> >  10 files changed, 341 insertions(+), 21 deletions(-)
> >  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

[snip]

> > diff --git a/include/uapi/linux/uvcvideo.h b/include/uapi/linux/uvcvideo.h
> > index 3b08186..ffe17ec 100644
> > --- a/include/uapi/linux/uvcvideo.h
> > +++ b/include/uapi/linux/uvcvideo.h
> > @@ -67,4 +67,30 @@ struct uvc_xu_control_query {
> >  #define UVCIOC_CTRL_MAP_IOWR('u', 0x20, struct 
> > uvc_xu_control_mapping)
> >  #define UVCIOC_CTRL_QUERY  _IOWR('u', 0x21, struct uvc_xu_control_query)
> >  
> > +/*
> > + * Metadata node
> > + */
> > +
> > +/**
> > + * struct uvc_meta_buf - metadata buffer building block
> > + * @ns - system timestamp of the payload in nanoseconds
> > + * @sof- USB Frame Number
> > + * @length - length of the payload header
> > + * @flags  - payload header flags
> > + * @buf- optional device-specific header data
> > + *
> > + * UVC metadata nodes fill buffers with possibly multiple instances of this
> > + * struct. The first two fields are added by the driver, they can be used 
> > for
> > + * clock synchronisation. The rest is an exact copy of a UVC payload 
> > header.
> > + * Only complete objects with complete buffers are included. Therefore it's
> > + * always sizeof(meta->ts) + sizeof(meta->sof) + meta->length bytes large.
> > + */
> > +struct uvc_meta_buf {
> > +   __u64 ns;
> > +   __u16 sof;
> > +   __u8 length;
> > +   __u8 flags;
> 
> I think the struct layout is architecture dependent. I could be wrong, but I 
> think
> for x64 there is a 4-byte hole here, but not for arm32/arm64.
> 
> Please double-check the struct layout.

You mean this can be a problem for 64- / 32-bit compatibility? If the 
difference is just between ARM and X86 then we don't care, do we? Or do 
video buffers have to be safe for cross-system transfer?

> BTW: __u8 for length? The payload can never be longer in UVC?

No, not the payload, a single payload header cannot be larger than 255 
bytes, that's right.

Thanks
Guennadi

> Regards,
> 
>   Hans
> 
> > +   __u8 buf[];
> > +};
> > +
> >  #endif
> > 
> 

Re: UVC property auto update

[Guennadi Liakhovetski]

On Mon, 4 Sep 2017, Edgar Thier wrote:

> Hi Guennadi,
> 
> 
> > But that patch only re-reads the flags. What does that give you? Do those 
> > flags change? In which way? As far as I understand the UVC Autoupdate
> > feature, a change in GET_INFO data is only one possibility, (arguably) a 
> > more important one is changes in GET_CUR data.

Sorry, I misunderstood, please, ignore that.

> My understanding of the driver is rather narrow, so please correct me if I am 
> wrong.
> From what I can see the uvc driver is currently not asking the device if a 
> property has self
> modifying properties (and thus would require the AUTO_UPDATE flag).
> This is only done for properties in the extension unit but not for 'standard' 
> properties.
> Thus properties exhibit different behavior depending on where they are 
> defined.
> By changing this the driver now assumes that a property with AUTO_UPDATE has 
> to be re-read when
> getting/setting a property and does not rely on cached values, no matter if 
> extension unit or not.
> 
> I did not consider the possibility that a lower level change would be 
> necessary or that a more
> previce update mechanism for different property parts was possible.
> 
> Basically the entry point for my change would be here:
> https://git.linuxtv.org/media_tree.git/tree/drivers/media/usb/uvc/uvc_ctrl.c#n1405
> 
> How an update is handled by the driver did not seem important for this patch 
> as the retrieval of a
> correct property value seemed more important.

Ok, looking more at the spec, the driver and your patch, here's what I 
come up with:

1. UVC defines which standard controls should have which flags. Among 
those flags it specifies, which controls should specify the Autoupdate 
flag. E.g. see the first of them as it appears in my copy of the spec 
"4.2.2.4.8 Average Bit Rate Control"
2. The driver could read out flags from descriptors, but it hard-codes 
them instead. So, (arguably), there's no need to actually read them at 
probe time. XUs on the other hand aren't standard, therefore their flags 
have to be read out.
3. In your patch you provide gain as an example. Do you mean the 
PU_GAIN_CONTROL? The spec doesn't specify, that it should have Autoupdate 
set. Now, whether that means, that using that flag with PU_GAIN_CONTROL is 
a violation of the spec - I'm not sure about.

So, I think, the question really is - are hard-coded flags a proper and 
sufficient approach or should flags be read from descriptors?

> > In either case, this should 
> > be implemented using the UVC Interrupt endpoint. Here's my latest 
> > asynchronous control patch:
> > 
> > https://patchwork.linuxtv.org/patch/42800/
> > 
> > As you can see, it only handles the VALUE_CHANGE (GET_CUR) case. I would 
> > suggest implementing a patch on top of it to add support for INFO_CHANGE 
> > and you'd be the best person to test it then!
> 
> I will try it out. I should be able to give you feedback tomorrow.

Thanks.

> I will also ask the firmware developer if only value changes are available or 
> flag changes are also
> a possibility.

Well, flags aren't likely to change, perhaps. I think min and max values 
are more likely to be updated.

Regards
Guennadi

> Cheers,
> Edgar

Re: UVC property auto update

[Guennadi Liakhovetski]

Hi Edgar,

Thanks for the explanation.

On Mon, 4 Sep 2017, Edgar Thier wrote:

> Hi Guennadi,
> 
> The cameras in question are USB-3.0 industrial cameras from The Imaging 
> Source.
> The ones I tested were the DFK UX250 and DFK UX264 models.
> I do not know if there are other devices that have the AUTO_UPDATE flag for 
> various properties.
> 
> Since I received no immediate answer I tried fixing it myself.
> The result can be found here:
> https://patchwork.linuxtv.org/patch/43289/

But that patch only re-reads the flags. What does that give you? Do those 
flags change? In which way? As far as I understand the UVC Autoupdate 
feature, a change in GET_INFO data is only one possibility, (arguably) a 
more important one is changes in GET_CUR data. In either case, this should 
be implemented using the UVC Interrupt endpoint. Here's my latest 
asynchronous control patch:

https://patchwork.linuxtv.org/patch/42800/

As you can see, it only handles the VALUE_CHANGE (GET_CUR) case. I would 
suggest implementing a patch on top of it to add support for INFO_CHANGE 
and you'd be the best person to test it then!

Thanks
Guennadi

> Cheers,
> 
> Edgar

Re: UVC property auto update

[Guennadi Liakhovetski]

Hi Edgar,

I'm adding UVC maintainer to CC, he might have missed this email. Could 
you also tell us what cameras have those features? A recent patch from me 
extends the UVC driver use of the Interrupt endpoint for asynchronous 
control completion notifications. This would be another extension for the 
same interface. I guess the way to implement it would be using V4L events.

Thanks
Guennadi

On Mon, 7 Aug 2017, Edgar Thier wrote:

> Hi all,
> 
> I have some USB-3.0 cameras that use UVC.
> These cameras offer auto updates for various properties.
> An example of such a property would be gain, that will be adjusted when 
> activating the auto-gain
> property. These property changes are not queried by the UVC driver, unless it 
> already has the
> property marked as auto update via UVC_CTRL_FLAG_AUTO_UPDATE.
> >From what I have seen, it seems that this flag is not checked when indexing 
> >the camera controls.
> However it is checked when using extension units, so all properties loaded 
> through such a unit are
> being updates as one would hope.
> 
> My questions:
> 
> Is it intended that properties cannot mark themselves as autoupdate?
> If yes:
>   Is there a recommended way of working around this?
>   Do all autoupdate properties have to be in an extension unit?
> If no:
>   What should a fix look like?
> 
> Regards,
> 
> Edgar
> 

Re: [PATCH] v4l: use struct v4l2_buffer explicitly instead of void *

[Guennadi Liakhovetski]

Hi Mauro,

On Sun, 27 Aug 2017, Mauro Carvalho Chehab wrote:

> Em Fri, 28 Jul 2017 14:45:37 +0200 (CEST)
> Guennadi Liakhovetski <g.liakhovet...@gmx.de> escreveu:
> 
> > A number of functions use void * for a struct v4l2_buffer parameter.
> > Avoid that to improve compile-time checking.
> 
> Nack.
> 
> The videbuf2-core should be be independent of videobuf2-v4l2. The
> plan is to use it for DVB too. There's a patch floating around,
> lacking people to take a look.
> 
> I intend to review and merge it when I have some time along
> the year.

Ok, I thought there would be a reason similar to this, thanks for 
explaining.

Regards
Guennadi

> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> > 
> > This probably was done on purpose, maybe to reuse the video buffers by 
> > other than V4L2 users, but I haven't found any, and the code doesn't seem 
> > very new...
> > 
> >  drivers/media/v4l2-core/videobuf2-core.c | 17 +
> >  drivers/media/v4l2-core/videobuf2-v4l2.c | 15 +++
> >  include/media/videobuf2-core.h   | 19 ---
> >  3 files changed, 28 insertions(+), 23 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
> > b/drivers/media/v4l2-core/videobuf2-core.c
> > index 14f83cec..170a416 100644
> > --- a/drivers/media/v4l2-core/videobuf2-core.c
> > +++ b/drivers/media/v4l2-core/videobuf2-core.c
> > @@ -958,7 +958,7 @@ void vb2_discard_done(struct vb2_queue *q)
> >  /**
> >   * __prepare_mmap() - prepare an MMAP buffer
> >   */
> > -static int __prepare_mmap(struct vb2_buffer *vb, const void *pb)
> > +static int __prepare_mmap(struct vb2_buffer *vb, const struct v4l2_buffer 
> > *pb)
> >  {
> > int ret = 0;
> >  
> > @@ -971,7 +971,7 @@ static int __prepare_mmap(struct vb2_buffer *vb, const 
> > void *pb)
> >  /**
> >   * __prepare_userptr() - prepare a USERPTR buffer
> >   */
> > -static int __prepare_userptr(struct vb2_buffer *vb, const void *pb)
> > +static int __prepare_userptr(struct vb2_buffer *vb, const struct 
> > v4l2_buffer *pb)
> >  {
> > struct vb2_plane planes[VB2_MAX_PLANES];
> > struct vb2_queue *q = vb->vb2_queue;
> > @@ -1089,7 +1089,7 @@ static int __prepare_userptr(struct vb2_buffer *vb, 
> > const void *pb)
> >  /**
> >   * __prepare_dmabuf() - prepare a DMABUF buffer
> >   */
> > -static int __prepare_dmabuf(struct vb2_buffer *vb, const void *pb)
> > +static int __prepare_dmabuf(struct vb2_buffer *vb, const struct 
> > v4l2_buffer *pb)
> >  {
> > struct vb2_plane planes[VB2_MAX_PLANES];
> > struct vb2_queue *q = vb->vb2_queue;
> > @@ -1236,7 +1236,7 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
> > call_void_vb_qop(vb, buf_queue, vb);
> >  }
> >  
> > -static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
> > +static int __buf_prepare(struct vb2_buffer *vb, const struct v4l2_buffer 
> > *pb)
> >  {
> > struct vb2_queue *q = vb->vb2_queue;
> > unsigned int plane;
> > @@ -1279,7 +1279,8 @@ static int __buf_prepare(struct vb2_buffer *vb, const 
> > void *pb)
> > return 0;
> >  }
> >  
> > -int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
> > +int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index,
> > +struct v4l2_buffer *pb)
> >  {
> > struct vb2_buffer *vb;
> > int ret;
> > @@ -1514,7 +1515,7 @@ static int __vb2_wait_for_done_vb(struct vb2_queue 
> > *q, int nonblocking)
> >   * Will sleep if required for nonblocking == false.
> >   */
> >  static int __vb2_get_done_vb(struct vb2_queue *q, struct vb2_buffer **vb,
> > -void *pb, int nonblocking)
> > +struct v4l2_buffer *pb, int nonblocking)
> >  {
> > unsigned long flags;
> > int ret = 0;
> > @@ -1583,8 +1584,8 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
> > }
> >  }
> >  
> > -int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
> > -  bool nonblocking)
> > +int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex,
> > +  struct v4l2_buffer *pb, bool nonblocking)
> >  {
> > struct vb2_buffer *vb = NULL;
> > int ret;
> > diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c 
> > b/drivers/media/v4l2-core/videobuf2-v4l2.c
> > index 0c06699..3c425ea 100644
> >

Re: [PATCH 03/15] [media] i2c: make device_type const

[Guennadi Liakhovetski]

On Sat, 19 Aug 2017, Bhumika Goyal wrote:

> Make this const as it is only stored in the type field of a device
> structure, which is const.
> Done using Coccinelle.
> 
> Signed-off-by: Bhumika Goyal <bhumi...@gmail.com>

Acked-by: Guennadi Liakhovetski <g.liakhovet...@gmx.de>

Thanks
Guennadi

> ---
>  drivers/media/i2c/soc_camera/mt9t031.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/media/i2c/soc_camera/mt9t031.c 
> b/drivers/media/i2c/soc_camera/mt9t031.c
> index 714fb35..4802d30 100644
> --- a/drivers/media/i2c/soc_camera/mt9t031.c
> +++ b/drivers/media/i2c/soc_camera/mt9t031.c
> @@ -592,7 +592,7 @@ static int mt9t031_runtime_resume(struct device *dev)
>   .runtime_resume = mt9t031_runtime_resume,
>  };
>  
> -static struct device_type mt9t031_dev_type = {
> +static const struct device_type mt9t031_dev_type = {
>   .name   = "MT9T031",
>   .pm = _dev_pm_ops,
>  };
> -- 
> 1.9.1
> 

Re: [PATCH 4/6 v5] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Hi Hans,

On Fri, 28 Jul 2017, Hans Verkuil wrote:

> On 07/28/2017 02:33 PM, Guennadi Liakhovetski wrote:

[snip]

> > +/**
> > + * struct uvc_meta_buf - metadata buffer building block
> > + * @ns - system timestamp of the payload in nanoseconds
> > + * @sof- USB Frame Number
> > + * @length - length of the payload header
> > + * @flags  - payload header flags
> > + * @buf- optional device-specific header data
> > + *
> > + * UVC metadata nodes fill buffers with possibly multiple instances of this
> > + * struct. The first two fields are added by the driver, they can be used 
> > for
> > + * clock synchronisation. The rest is an exact copy of a UVC payload 
> > header.
> > + * Only complete objects with complete buffers are included. Therefore it's
> > + * always sizeof(meta->ts) + sizeof(meta->sof) + meta->length bytes large.
> > + */
> > +struct uvc_meta_buf {
> > +   __u64 ns;
> > +   __u16 sof;
> > +   __u8 length;
> > +   __u8 flags;
> 
> I think the struct layout is architecture dependent. I could be wrong, but I 
> think
> for x64 there is a 4-byte hole here, but not for arm32/arm64.
> 
> Please double-check the struct layout.

It worked for me so far on an x86-64 system, I was able to access buffer 
data correctly, but yes, it's probably safer to use __packed.

> BTW: __u8 for length? The payload can never be longer in UVC?

No, it cannot. That's exactly how UVC defines it.

Thanks
Guennadi

> 
> Regards,
> 
>   Hans
> 
> > +   __u8 buf[];
> > +};
> > +
> >  #endif
> > 
> 

[PATCH] v4l: use struct v4l2_buffer explicitly instead of void *

[Guennadi Liakhovetski]

A number of functions use void * for a struct v4l2_buffer parameter.
Avoid that to improve compile-time checking.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

This probably was done on purpose, maybe to reuse the video buffers by 
other than V4L2 users, but I haven't found any, and the code doesn't seem 
very new...

 drivers/media/v4l2-core/videobuf2-core.c | 17 +
 drivers/media/v4l2-core/videobuf2-v4l2.c | 15 +++
 include/media/videobuf2-core.h   | 19 ---
 3 files changed, 28 insertions(+), 23 deletions(-)

diff --git a/drivers/media/v4l2-core/videobuf2-core.c 
b/drivers/media/v4l2-core/videobuf2-core.c
index 14f83cec..170a416 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -958,7 +958,7 @@ void vb2_discard_done(struct vb2_queue *q)
 /**
  * __prepare_mmap() - prepare an MMAP buffer
  */
-static int __prepare_mmap(struct vb2_buffer *vb, const void *pb)
+static int __prepare_mmap(struct vb2_buffer *vb, const struct v4l2_buffer *pb)
 {
int ret = 0;
 
@@ -971,7 +971,7 @@ static int __prepare_mmap(struct vb2_buffer *vb, const void 
*pb)
 /**
  * __prepare_userptr() - prepare a USERPTR buffer
  */
-static int __prepare_userptr(struct vb2_buffer *vb, const void *pb)
+static int __prepare_userptr(struct vb2_buffer *vb, const struct v4l2_buffer 
*pb)
 {
struct vb2_plane planes[VB2_MAX_PLANES];
struct vb2_queue *q = vb->vb2_queue;
@@ -1089,7 +1089,7 @@ static int __prepare_userptr(struct vb2_buffer *vb, const 
void *pb)
 /**
  * __prepare_dmabuf() - prepare a DMABUF buffer
  */
-static int __prepare_dmabuf(struct vb2_buffer *vb, const void *pb)
+static int __prepare_dmabuf(struct vb2_buffer *vb, const struct v4l2_buffer 
*pb)
 {
struct vb2_plane planes[VB2_MAX_PLANES];
struct vb2_queue *q = vb->vb2_queue;
@@ -1236,7 +1236,7 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
call_void_vb_qop(vb, buf_queue, vb);
 }
 
-static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
+static int __buf_prepare(struct vb2_buffer *vb, const struct v4l2_buffer *pb)
 {
struct vb2_queue *q = vb->vb2_queue;
unsigned int plane;
@@ -1279,7 +1279,8 @@ static int __buf_prepare(struct vb2_buffer *vb, const 
void *pb)
return 0;
 }
 
-int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
+int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index,
+struct v4l2_buffer *pb)
 {
struct vb2_buffer *vb;
int ret;
@@ -1514,7 +1515,7 @@ static int __vb2_wait_for_done_vb(struct vb2_queue *q, 
int nonblocking)
  * Will sleep if required for nonblocking == false.
  */
 static int __vb2_get_done_vb(struct vb2_queue *q, struct vb2_buffer **vb,
-void *pb, int nonblocking)
+struct v4l2_buffer *pb, int nonblocking)
 {
unsigned long flags;
int ret = 0;
@@ -1583,8 +1584,8 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
}
 }
 
-int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
-  bool nonblocking)
+int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex,
+  struct v4l2_buffer *pb, bool nonblocking)
 {
struct vb2_buffer *vb = NULL;
int ret;
diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c 
b/drivers/media/v4l2-core/videobuf2-v4l2.c
index 0c06699..3c425ea 100644
--- a/drivers/media/v4l2-core/videobuf2-v4l2.c
+++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
@@ -53,7 +53,8 @@
  * __verify_planes_array() - verify that the planes array passed in struct
  * v4l2_buffer from userspace can be safely used
  */
-static int __verify_planes_array(struct vb2_buffer *vb, const struct 
v4l2_buffer *b)
+static int __verify_planes_array(struct vb2_buffer *vb,
+const struct v4l2_buffer *b)
 {
if (!V4L2_TYPE_IS_MULTIPLANAR(b->type))
return 0;
@@ -73,7 +74,8 @@ static int __verify_planes_array(struct vb2_buffer *vb, const 
struct v4l2_buffer
return 0;
 }
 
-static int __verify_planes_array_core(struct vb2_buffer *vb, const void *pb)
+static int __verify_planes_array_core(struct vb2_buffer *vb,
+ const struct v4l2_buffer *pb)
 {
return __verify_planes_array(vb, pb);
 }
@@ -118,9 +120,8 @@ static int __verify_length(struct vb2_buffer *vb, const 
struct v4l2_buffer *b)
return 0;
 }
 
-static void __copy_timestamp(struct vb2_buffer *vb, const void *pb)
+static void __copy_timestamp(struct vb2_buffer *vb, const struct v4l2_buffer 
*b)
 {
-   const struct v4l2_buffer *b = pb;
struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
struct vb2_queue *q = vb->vb2_queue;
 
@@ -185,9 +186,8 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, 
struct v4l2_buffer *b,
 

[PATCH 1/6 v5] UVC: fix .queue_setup() to check the number of planes

[Guennadi Liakhovetski]

According to documentation of struct vb2_ops the .queue_setup() callback
should return an error if the number of planes parameter contains an
invalid value on input. Fix this instead of ignoring the value.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_queue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/usb/uvc/uvc_queue.c 
b/drivers/media/usb/uvc/uvc_queue.c
index aa21997..371a4ad 100644
--- a/drivers/media/usb/uvc/uvc_queue.c
+++ b/drivers/media/usb/uvc/uvc_queue.c
@@ -84,7 +84,7 @@ static int uvc_queue_setup(struct vb2_queue *vq,
 
/* Make sure the image size is large enough. */
if (*nplanes)
-   return sizes[0] < size ? -EINVAL : 0;
+   return sizes[0] < size || *nplanes != 1 ? -EINVAL : 0;
*nplanes = 1;
sizes[0] = size;
return 0;
-- 
1.9.3

[PATCH 4/6 v5] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Some UVC video cameras contain metadata in their payload headers. This
patch extracts that data, adding more clock synchronisation information,
on both bulk and isochronous endpoints and makes it available to the
user space on a separate video node, using the V4L2_CAP_META_CAPTURE
capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
though different cameras will have different metadata formats, we use
the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
parse data, based on the specific camera model information. This
version of the patch only creates such metadata nodes for cameras,
specifying a UVC_QUIRK_METADATA_NODE quirk flag.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/Makefile   |   2 +-
 drivers/media/usb/uvc/uvc_ctrl.c |   3 +
 drivers/media/usb/uvc/uvc_driver.c   |  12 +++
 drivers/media/usb/uvc/uvc_isight.c   |   2 +-
 drivers/media/usb/uvc/uvc_metadata.c | 139 +++
 drivers/media/usb/uvc/uvc_queue.c|  41 +--
 drivers/media/usb/uvc/uvc_video.c| 119 +++---
 drivers/media/usb/uvc/uvcvideo.h |  17 -
 drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
 include/uapi/linux/uvcvideo.h|  26 +++
 10 files changed, 341 insertions(+), 21 deletions(-)
 create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

diff --git a/drivers/media/usb/uvc/Makefile b/drivers/media/usb/uvc/Makefile
index c26d12f..06c7cd3 100644
--- a/drivers/media/usb/uvc/Makefile
+++ b/drivers/media/usb/uvc/Makefile
@@ -1,5 +1,5 @@
 uvcvideo-objs  := uvc_driver.o uvc_queue.o uvc_v4l2.o uvc_video.o uvc_ctrl.o \
- uvc_status.o uvc_isight.o uvc_debugfs.o
+ uvc_status.o uvc_isight.o uvc_debugfs.o uvc_metadata.o
 ifeq ($(CONFIG_MEDIA_CONTROLLER),y)
 uvcvideo-objs  += uvc_entity.o
 endif
diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c
index c2ee6e3..91ff2c7 100644
--- a/drivers/media/usb/uvc/uvc_ctrl.c
+++ b/drivers/media/usb/uvc/uvc_ctrl.c
@@ -2179,6 +2179,9 @@ int uvc_ctrl_init_device(struct uvc_device *dev)
ctrl->entity = entity;
ctrl->index = i;
 
+   if (UVC_ENTITY_TYPE(entity) == UVC_VC_EXTENSION_UNIT)
+   uvc_trace(UVC_TRACE_CONTROL, "XU %u: ctrl 
%d\n", entity->id, i);
+
uvc_ctrl_init_ctrl(dev, ctrl);
ctrl++;
}
diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index cfe33bf..3d61cec 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -1884,6 +1884,7 @@ static void uvc_unregister_video(struct uvc_device *dev)
continue;
 
video_unregister_device(>vdev);
+   video_unregister_device(>meta.vdev);
 
uvc_debugfs_cleanup_stream(stream);
}
@@ -1944,6 +1945,11 @@ static int uvc_register_video(struct uvc_device *dev,
return ret;
}
 
+   /* Register a metadata node, but ignore a possible failure, complete
+* registration of video nodes anyway.
+*/
+   uvc_meta_register(stream);
+
if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE;
else
@@ -2041,6 +2047,12 @@ static int uvc_probe(struct usb_interface *intf,
dev->udev = usb_get_dev(udev);
dev->intf = usb_get_intf(intf);
dev->intfnum = intf->cur_altsetting->desc.bInterfaceNumber;
+   if (uvc_quirks_param != -1 &&
+   uvc_quirks_param & UVC_DEV_FLAG_METADATA_NODE) {
+   uvc_quirks_param &= ~UVC_DEV_FLAG_METADATA_NODE;
+   if (uvc_quirks_param == 0)
+   uvc_quirks_param = -1;
+   }
dev->quirks = (uvc_quirks_param == -1)
? id->driver_info : uvc_quirks_param;
 
diff --git a/drivers/media/usb/uvc/uvc_isight.c 
b/drivers/media/usb/uvc/uvc_isight.c
index 8510e725..fb940cf 100644
--- a/drivers/media/usb/uvc/uvc_isight.c
+++ b/drivers/media/usb/uvc/uvc_isight.c
@@ -100,7 +100,7 @@ static int isight_decode(struct uvc_video_queue *queue, 
struct uvc_buffer *buf,
 }
 
 void uvc_video_decode_isight(struct urb *urb, struct uvc_streaming *stream,
-   struct uvc_buffer *buf)
+   struct uvc_buffer *buf, struct uvc_buffer *meta_buf)
 {
int ret, i;
 
diff --git a/drivers/media/usb/uvc/uvc_metadata.c 
b/drivers/media/usb/uvc/uvc_metadata.c
new file mode 100644
index 000..062e6ec
--- /dev/null
+++ b/drivers/media/usb/uvc/uvc_metadata.c
@@ -0,0 +1,139 @@
+/*
+ *  uvc_metadata.c  --  USB Video Class driver - Metadata handling
+ *
+ *  Copyright (C) 2016
+ *  Guennadi Liakhovetski (guennadi.liakhovet..

[PATCH 0/6 v5] uvcvideo: metadata nodes and controls

[Guennadi Liakhovetski]

The first four patches are for UVC metadata nodes, the last two patches
are for asynchronous controls and control error reporting.

Thanks
Guennadi

Guennadi Liakhovetski (6):
  UVC: fix .queue_setup() to check the number of planes
  V4L: Add a UVC Metadata format
  uvcvideo: convert from using an atomic variable to a reference count
  uvcvideo: add a metadata device node
  uvcvideo: send a control event when a Control Change interrupt arrives
  uvcvideo: handle control pipe protocol STALLs

 Documentation/media/uapi/v4l/meta-formats.rst|   1 +
 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst |  39 +
 drivers/media/usb/uvc/Makefile   |   2 +-
 drivers/media/usb/uvc/uvc_ctrl.c | 150 +--
 drivers/media/usb/uvc/uvc_driver.c   |  43 --
 drivers/media/usb/uvc/uvc_isight.c   |   2 +-
 drivers/media/usb/uvc/uvc_metadata.c | 139 ++
 drivers/media/usb/uvc/uvc_queue.c|  43 +-
 drivers/media/usb/uvc/uvc_status.c   | 112 --
 drivers/media/usb/uvc/uvc_v4l2.c |   4 +-
 drivers/media/usb/uvc/uvc_video.c| 178 +--
 drivers/media/usb/uvc/uvcvideo.h |  33 -
 drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
 include/uapi/linux/uvcvideo.h|  28 
 include/uapi/linux/videodev2.h   |   1 +
 15 files changed, 705 insertions(+), 71 deletions(-)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
 create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

-- 
1.9.3

[PATCH 6/6 v5] uvcvideo: handle control pipe protocol STALLs

[Guennadi Liakhovetski]

When a command ends up in a STALL on the control pipe, use the Request
Error Code control to provide a more precise error information to the
user.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_video.c | 59 +++
 1 file changed, 53 insertions(+), 6 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_video.c 
b/drivers/media/usb/uvc/uvc_video.c
index 006691e..887561b 100644
--- a/drivers/media/usb/uvc/uvc_video.c
+++ b/drivers/media/usb/uvc/uvc_video.c
@@ -34,15 +34,59 @@ static int __uvc_query_ctrl(struct uvc_device *dev, __u8 
query, __u8 unit,
__u8 intfnum, __u8 cs, void *data, __u16 size,
int timeout)
 {
-   __u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE;
+   __u8 type = USB_TYPE_CLASS | USB_RECIP_INTERFACE, tmp, error;
unsigned int pipe;
+   int ret;
 
pipe = (query & 0x80) ? usb_rcvctrlpipe(dev->udev, 0)
  : usb_sndctrlpipe(dev->udev, 0);
type |= (query & 0x80) ? USB_DIR_IN : USB_DIR_OUT;
 
-   return usb_control_msg(dev->udev, pipe, query, type, cs << 8,
+   ret = usb_control_msg(dev->udev, pipe, query, type, cs << 8,
unit << 8 | intfnum, data, size, timeout);
+
+   if (ret != -EPIPE)
+   return ret;
+
+   tmp = *(u8 *)data;
+
+   pipe = usb_rcvctrlpipe(dev->udev, 0);
+   type = USB_TYPE_CLASS | USB_RECIP_INTERFACE | USB_DIR_IN;
+   ret = usb_control_msg(dev->udev, pipe, UVC_GET_CUR, type,
+ UVC_VC_REQUEST_ERROR_CODE_CONTROL << 8,
+ unit << 8 | intfnum, data, 1, timeout);
+   error = *(u8 *)data;
+   *(u8 *)data = tmp;
+
+   if (ret < 0)
+   return ret;
+
+   if (!ret)
+   return -EINVAL;
+
+   uvc_trace(UVC_TRACE_CONTROL, "Control error %u\n", error);
+
+   switch (error) {
+   case 0:
+   /* Cannot happen - we received a STALL */
+   return -EPIPE;
+   case 1: /* Not ready */
+   return -EAGAIN;
+   case 2: /* Wrong state */
+   return -EILSEQ;
+   case 3: /* Power */
+   return -EREMOTE;
+   case 4: /* Out of range */
+   return -ERANGE;
+   case 5: /* Invalid unit */
+   case 6: /* Invalid control */
+   case 7: /* Invalid Request */
+   case 8: /* Invalid value within range */
+   default: /* reserved or unknown */
+   break;
+   }
+
+   return -EINVAL;
 }
 
 static const char *uvc_query_name(__u8 query)
@@ -80,7 +124,7 @@ int uvc_query_ctrl(struct uvc_device *dev, __u8 query, __u8 
unit,
uvc_printk(KERN_ERR, "Failed to query (%s) UVC control %u on "
"unit %u: %d (exp. %u).\n", uvc_query_name(query), cs,
unit, ret, size);
-   return -EIO;
+   return ret < 0 ? ret : -EIO;
}
 
return 0;
@@ -203,13 +247,15 @@ static int uvc_get_video_ctrl(struct uvc_streaming 
*stream,
uvc_warn_once(stream->dev, UVC_WARN_PROBE_DEF, "UVC non "
"compliance - GET_DEF(PROBE) not supported. "
"Enabling workaround.\n");
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
goto out;
} else if (ret != size) {
uvc_printk(KERN_ERR, "Failed to query (%u) UVC %s control : "
"%d (exp. %u).\n", query, probe ? "probe" : "commit",
ret, size);
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
goto out;
}
 
@@ -290,7 +336,8 @@ static int uvc_set_video_ctrl(struct uvc_streaming *stream,
uvc_printk(KERN_ERR, "Failed to set UVC %s control : "
"%d (exp. %u).\n", probe ? "probe" : "commit",
ret, size);
-   ret = -EIO;
+   if (ret >= 0)
+   ret = -EIO;
}
 
kfree(data);
-- 
1.9.3

[PATCH 2/6 v5] V4L: Add a UVC Metadata format

[Guennadi Liakhovetski]

Add a pixel format, used by the UVC driver to stream metadata.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 Documentation/media/uapi/v4l/meta-formats.rst|  1 +
 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst | 39 
 include/uapi/linux/videodev2.h   |  1 +
 3 files changed, 41 insertions(+)
 create mode 100644 Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst

diff --git a/Documentation/media/uapi/v4l/meta-formats.rst 
b/Documentation/media/uapi/v4l/meta-formats.rst
index 01e24e3..1bb45a3f 100644
--- a/Documentation/media/uapi/v4l/meta-formats.rst
+++ b/Documentation/media/uapi/v4l/meta-formats.rst
@@ -14,3 +14,4 @@ These formats are used for the :ref:`metadata` interface only.
 
 pixfmt-meta-vsp1-hgo
 pixfmt-meta-vsp1-hgt
+pixfmt-meta-uvc
diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst 
b/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
new file mode 100644
index 000..58f78cb
--- /dev/null
+++ b/Documentation/media/uapi/v4l/pixfmt-meta-uvc.rst
@@ -0,0 +1,39 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _v4l2-meta-fmt-uvc:
+
+***
+V4L2_META_FMT_UVC ('UVCH')
+***
+
+UVC Payload Header Data
+
+
+Description
+===
+
+This format describes data, supplied by the UVC driver from metadata video
+nodes. That data includes UVC Payload Header contents and auxiliary timing
+information, required for precise interpretation of timestamps, contained in
+those headers. Buffers, streamed via UVC metadata nodes, are composed of blocks
+of variable length. Those blocks contain are described by struct uvc_meta_buf
+and contain the following fields:
+
+.. flat-table:: UVC Metadata Block
+:widths: 1 4
+:header-rows:  1
+:stub-columns: 0
+
+* - Field
+  - Description
+* - struct timespec ts;
+  - system timestamp, measured by the driver upon reception of the payload
+* - __u16 sof;
+  - USB Frame Number, also obtained by the driver
+* - :cspan:`1` *The rest is an exact copy of the payload header:*
+* - __u8 length;
+  - length of the rest of the block, including this field
+* - __u8 flags;
+  - Flags, indicating presence of other standard UVC fields
+* - __u8 buf[];
+  - The rest of the header, possibly including UVC PTS and SCR fields
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 45cf735..0aad91c 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -682,6 +682,7 @@ struct v4l2_pix_format {
 /* Meta-data formats */
 #define V4L2_META_FMT_VSP1_HGOv4l2_fourcc('V', 'S', 'P', 'H') /* R-Car 
VSP1 1-D Histogram */
 #define V4L2_META_FMT_VSP1_HGTv4l2_fourcc('V', 'S', 'P', 'T') /* R-Car 
VSP1 2-D Histogram */
+#define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC 
Payload Header metadata */
 
 /* priv field value to indicates that subsequent fields are valid. */
 #define V4L2_PIX_FMT_PRIV_MAGIC0xfeedcafe
-- 
1.9.3

[PATCH 3/6 v5] uvcvideo: convert from using an atomic variable to a reference count

[Guennadi Liakhovetski]

When adding support for metadata nodes, we'll have to keep video
devices registered until all metadata nodes are closed too. Since
this has nothing to do with stream counting, replace the nstreams
atomic variable with a reference counter.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_driver.c | 31 +--
 drivers/media/usb/uvc/uvcvideo.h   |  2 +-
 2 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index 70842c5..cfe33bf 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -1849,16 +1849,20 @@ static void uvc_delete(struct uvc_device *dev)
kfree(dev);
 }
 
+static void uvc_kref_release(struct kref *kref)
+{
+   struct uvc_device *dev = container_of(kref, struct uvc_device, ref);
+
+   uvc_delete(dev);
+}
+
 static void uvc_release(struct video_device *vdev)
 {
struct uvc_streaming *stream = video_get_drvdata(vdev);
struct uvc_device *dev = stream->dev;
 
-   /* Decrement the registered streams count and delete the device when it
-* reaches zero.
-*/
-   if (atomic_dec_and_test(>nstreams))
-   uvc_delete(dev);
+   /* Decrement the refcount and delete the device when it reaches zero */
+   kref_put(>ref, uvc_kref_release);
 }
 
 /*
@@ -1870,10 +1874,10 @@ static void uvc_unregister_video(struct uvc_device *dev)
 
/* Unregistering all video devices might result in uvc_delete() being
 * called from inside the loop if there's no open file handle. To avoid
-* that, increment the stream count before iterating over the streams
-* and decrement it when done.
+* that, increment the refcount before iterating over the streams and
+* decrement it when done.
 */
-   atomic_inc(>nstreams);
+   kref_get(>ref);
 
list_for_each_entry(stream, >streams, list) {
if (!video_is_registered(>vdev))
@@ -1884,11 +1888,10 @@ static void uvc_unregister_video(struct uvc_device *dev)
uvc_debugfs_cleanup_stream(stream);
}
 
-   /* Decrement the stream count and call uvc_delete explicitly if there
-* are no stream left.
+   /* Decrement the refcount and call uvc_delete explicitly if there are no
+* stream left.
 */
-   if (atomic_dec_and_test(>nstreams))
-   uvc_delete(dev);
+   kref_put(>ref, uvc_kref_release);
 }
 
 static int uvc_register_video(struct uvc_device *dev,
@@ -1946,7 +1949,7 @@ static int uvc_register_video(struct uvc_device *dev,
else
stream->chain->caps |= V4L2_CAP_VIDEO_OUTPUT;
 
-   atomic_inc(>nstreams);
+   kref_get(>ref);
return 0;
 }
 
@@ -2031,7 +2034,7 @@ static int uvc_probe(struct usb_interface *intf,
INIT_LIST_HEAD(>entities);
INIT_LIST_HEAD(>chains);
INIT_LIST_HEAD(>streams);
-   atomic_set(>nstreams, 0);
+   kref_init(>ref);
atomic_set(>nmappings, 0);
mutex_init(>lock);
 
diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h
index 15e415e..f157cf7 100644
--- a/drivers/media/usb/uvc/uvcvideo.h
+++ b/drivers/media/usb/uvc/uvcvideo.h
@@ -575,7 +575,7 @@ struct uvc_device {
 
/* Video Streaming interfaces */
struct list_head streams;
-   atomic_t nstreams;
+   struct kref ref;
 
/* Status Interrupt Endpoint */
struct usb_host_endpoint *int_ep;
-- 
1.9.3

[PATCH 5/6 v5] uvcvideo: send a control event when a Control Change interrupt arrives

[Guennadi Liakhovetski]

UVC defines a method of handling asynchronous controls, which sends a
USB packet over the interrupt pipe. This patch implements support for
such packets by sending a control event to the user. Since this can
involve USB traffic and, therefore, scheduling, this has to be done
in a work queue.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---
 drivers/media/usb/uvc/uvc_ctrl.c   | 147 +
 drivers/media/usb/uvc/uvc_status.c | 112 +---
 drivers/media/usb/uvc/uvc_v4l2.c   |   4 +-
 drivers/media/usb/uvc/uvcvideo.h   |  14 +++-
 include/uapi/linux/uvcvideo.h  |   2 +
 5 files changed, 251 insertions(+), 28 deletions(-)

diff --git a/drivers/media/usb/uvc/uvc_ctrl.c b/drivers/media/usb/uvc/uvc_ctrl.c
index 91ff2c7..be18707 100644
--- a/drivers/media/usb/uvc/uvc_ctrl.c
+++ b/drivers/media/usb/uvc/uvc_ctrl.c
@@ -20,6 +20,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 
@@ -1236,16 +1237,110 @@ static void uvc_ctrl_send_event(struct uvc_fh *handle,
}
 }
 
-static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
-   struct uvc_control *master, u32 slave_id,
-   const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
+static void __uvc_ctrl_send_slave_event(struct uvc_fh *handle,
+   struct uvc_control *master, u32 slave_id)
 {
struct uvc_control_mapping *mapping = NULL;
struct uvc_control *ctrl = NULL;
u32 changes = V4L2_EVENT_CTRL_CH_FLAGS;
-   unsigned int i;
s32 val = 0;
 
+   __uvc_find_control(master->entity, slave_id, , , 0);
+   if (ctrl == NULL)
+   return;
+
+   if (__uvc_ctrl_get(handle->chain, ctrl, mapping, ) == 0)
+   changes |= V4L2_EVENT_CTRL_CH_VALUE;
+
+   uvc_ctrl_send_event(handle, ctrl, mapping, val, changes);
+}
+
+static void uvc_ctrl_status_event_work(struct work_struct *work)
+{
+   struct uvc_device *dev = container_of(work, struct uvc_device,
+ async_ctrl.work);
+   struct uvc_ctrl_work *w = >async_ctrl;
+   struct uvc_control_mapping *mapping;
+   struct uvc_control *ctrl;
+   struct uvc_fh *handle;
+   __u8 *data;
+   unsigned int i;
+
+   spin_lock_irq(>lock);
+   data = w->data;
+   w->data = NULL;
+   ctrl = w->ctrl;
+   handle = ctrl->handle;
+   ctrl->handle = NULL;
+   spin_unlock_irq(>lock);
+
+   if (mutex_lock_interruptible(>chain->ctrl_mutex))
+   goto free;
+
+   list_for_each_entry(mapping, >info.mappings, list) {
+   s32 value = mapping->get(mapping, UVC_GET_CUR, data);
+
+   for (i = 0; i < ARRAY_SIZE(mapping->slave_ids); ++i) {
+   if (!mapping->slave_ids[i])
+   break;
+
+   __uvc_ctrl_send_slave_event(handle, ctrl,
+   mapping->slave_ids[i]);
+   }
+
+   if (mapping->v4l2_type == V4L2_CTRL_TYPE_MENU) {
+   struct uvc_menu_info *menu = mapping->menu_info;
+   unsigned int i;
+
+   for (i = 0; i < mapping->menu_count; ++i, ++menu)
+   if (menu->value == value) {
+   value = i;
+   break;
+   }
+   }
+
+   uvc_ctrl_send_event(handle, ctrl, mapping, value,
+   V4L2_EVENT_CTRL_CH_VALUE);
+   }
+
+   mutex_unlock(>chain->ctrl_mutex);
+
+free:
+   kfree(data);
+}
+
+void uvc_ctrl_status_event(struct uvc_device *dev, struct uvc_control *ctrl,
+  __u8 *data, size_t len)
+{
+   struct uvc_ctrl_work *w = >async_ctrl;
+
+   if (list_empty(>info.mappings))
+   return;
+
+   if (!ctrl->handle)
+   /* This is an auto-update, they are unsupported */
+   return;
+
+   spin_lock(>lock);
+   if (w->data)
+   /* A previous event work hasn't run yet, we lose 1 event */
+   kfree(w->data);
+
+   w->data = kmalloc(len, GFP_ATOMIC);
+   if (w->data) {
+   memcpy(w->data, data, len);
+   w->ctrl = ctrl;
+   schedule_work(>work);
+   }
+   spin_unlock(>lock);
+}
+
+static void uvc_ctrl_send_slave_event(struct uvc_fh *handle,
+   struct uvc_control *master, u32 slave_id,
+   const struct v4l2_ext_control *xctrls, unsigned int xctrls_count)
+{
+   unsigned int i;
+
/*
 * We can skip sending an event for the slave if the slave
 * is being modified in the same transaction.
@@ -1255,14 +1350,7 @@ static void uvc_ctrl_send_slav

Re: [PATCH v4] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

On Fri, 28 Jul 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Tuesday 25 Jul 2017 15:27:24 Guennadi Liakhovetski wrote:
> > On Fri, 21 Jul 2017, Laurent Pinchart wrote:
> > > Hi Guennadi,
> > > 
> > > Thank you for the patch.
> > > 
> > >> Some UVC video cameras contain metadata in their payload headers. This
> > >> patch extracts that data, adding more clock synchronisation information,
> > >> on both bulk and isochronous endpoints and makes it available to the
> > >> user space on a separate video node, using the V4L2_CAP_META_CAPTURE
> > >> capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
> > >> though different cameras will have different metadata formats, we use
> > >> the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
> > >> parse data, based on the specific camera model information.
> > > 
> > > The issue we still haven't addressed is how to ensure that vendors will
> > > document their metadata format :-S
> > 
> > Uhm, add a black list of offending vendors and drop 60% of their frames?
> > ;-)
> 
> This was actually a serious question :-)
> 
> How about white-listing cameras instead, and enabling extended metadata 
> (after 
> the standard header) support only for vendors who have documented their 
> format 
> ?
> 
> Speaking of which, where's the documentation for the camera you're working on 
> ? :-)

The metadata definition is at the known to you page 
https://docs.microsoft.com/en-us/windows-hardware/drivers/stream/uvc-extensions-1-5
 
:-) But yes, I'll check with managers and submit a dev-flag patch too.

> > >> This version of the patch only creates such metadata nodes for cameras,
> > >> specifying a UVC_QUIRK_METADATA_NODE quirk flag.
> > >> 
> > >> Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > >> ---
> > >> 
> > >> v4:
> > >> - add support for isochronous cameras. Metadata is now collected from as
> > >> many payloads as they fit in the buffer
> > >> - add a USB Start Of Frame and a system timestamp to each metadata block
> > >> for user-space clock synchronisation
> > >> - use a default buffer size of 1024 bytes
> > >> 
> > >> Thanks to Laurent for patient long discussions and to everybody, who
> > >> helped me conduct all the investigation into various past, present and
> > >> future UVC cameras :-)
> > >> 
> > >>  drivers/media/usb/uvc/Makefile   |   2 +-
> > >>  drivers/media/usb/uvc/uvc_driver.c   |   4 +
> > >>  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> > >>  drivers/media/usb/uvc/uvc_metadata.c | 158 
> > >>  drivers/media/usb/uvc/uvc_queue.c|  68 ++-
> > >>  drivers/media/usb/uvc/uvc_video.c| 101 +++---
> > >>  drivers/media/usb/uvc/uvcvideo.h |  23 -
> > >>  drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
> > >>  include/uapi/linux/uvcvideo.h|  19 +
> > >>  include/uapi/linux/videodev2.h   |   3 +
> > >>  10 files changed, 347 insertions(+), 34 deletions(-)
> > >>  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> > > 
> > > [snip]
> > > 
> > >> diff --git a/drivers/media/usb/uvc/uvc_driver.c
> > >> b/drivers/media/usb/uvc/uvc_driver.c
> > >> index 70842c5..9f23dcf 100644
> > >> --- a/drivers/media/usb/uvc/uvc_driver.c
> > >> +++ b/drivers/media/usb/uvc/uvc_driver.c
> 
> [snip]
> 
> > >> @@ -1941,6 +1942,9 @@ static int uvc_register_video(struct uvc_device
> > >> *dev,
> > > > return ret;
> > > > }
> > > > 
> > > > +   /* Register a metadata node. */
> > > > +   uvc_meta_register(stream);
> > > 
> > > This can fail, you should handle errors.
> > 
> > Yes, this is in a way deliberate. If metadata node registration fails, the
> > driver continues without one. Would you prefer to fail and unregister the
> > main node in this case?
> 
> No, that's fine, but you should then add a comment to explain why the error 
> isn't handled.
> 
> Please also make sure that no damage can occur at device removal time if the 
> metadata node hasn't been registered.

Don't see any potential problems in that case.

> > > > if (stream->type == V4L

Re: [PATCH v4] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

On Tue, 25 Jul 2017, Guennadi Liakhovetski wrote:

[snip]

> > > +struct uvc_meta_buf {
> > > + struct timespec ts;
> > 
> > timespec has a different size on 32-bit and 64-bit architectures, so there
> > could be issues on 32-bit userspace running on a 64-bit kernel.
> > 
> > Additionally, on 32-bit platforms, timespec is not year 2038-safe. I thought
> > that timespec64 was exposed to userspace nowadays, but it doesn't seem to be
> > the case. I'm not sure how to handle this.
> 
> Oh, that isn't good :-/ I'll have to think more about this. If you get any 
> more ideas, I'd be glad to hear them too.

Shall we just use nanoseconds here too then, as returned by 
timespec_to_ns(), just like in frame timestamps?

Thanks
Guennadi

Re: [PATCH v4] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for your comments!

On Fri, 21 Jul 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> Thank you for the patch.
> 
> > Some UVC video cameras contain metadata in their payload headers. This
> > patch extracts that data, adding more clock synchronisation information,
> > on both bulk and isochronous endpoints and makes it available to the
> > user space on a separate video node, using the V4L2_CAP_META_CAPTURE
> > capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
> > though different cameras will have different metadata formats, we use
> > the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
> > parse data, based on the specific camera model information.
> 
> The issue we still haven't addressed is how to ensure that vendors will
> document their metadata format :-S

Uhm, add a black list of offending vendors and drop 60% of their frames? 
;-)

> > This version of the patch only creates such metadata nodes for cameras,
> > specifying a UVC_QUIRK_METADATA_NODE quirk flag.
> > 
> > Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
> > ---
> > 
> > v4:
> > - add support for isochronous cameras. Metadata is now collected from as 
> > many payloads as they fit in the buffer
> > - add a USB Start Of Frame and a system timestamp to each metadata block 
> > for user-space clock synchronisation
> > - use a default buffer size of 1024 bytes
> > 
> > Thanks to Laurent for patient long discussions and to everybody, who 
> > helped me conduct all the investigation into various past, present and 
> > future UVC cameras :-)
> > 
> >  drivers/media/usb/uvc/Makefile   |   2 +-
> >  drivers/media/usb/uvc/uvc_driver.c   |   4 +
> >  drivers/media/usb/uvc/uvc_isight.c   |   2 +-
> >  drivers/media/usb/uvc/uvc_metadata.c | 158 +++
> >  drivers/media/usb/uvc/uvc_queue.c|  68 ++-
> >  drivers/media/usb/uvc/uvc_video.c| 101 +++---
> >  drivers/media/usb/uvc/uvcvideo.h |  23 -
> >  drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
> >  include/uapi/linux/uvcvideo.h|  19 +
> >  include/uapi/linux/videodev2.h   |   3 +
> >  10 files changed, 347 insertions(+), 34 deletions(-)
> >  create mode 100644 drivers/media/usb/uvc/uvc_metadata.c
> 
> [snip]
> 
> > diff --git a/drivers/media/usb/uvc/uvc_driver.c
> > b/drivers/media/usb/uvc/uvc_driver.c
> > index 70842c5..9f23dcf 100644
> > --- a/drivers/media/usb/uvc/uvc_driver.c
> > +++ b/drivers/media/usb/uvc/uvc_driver.c
> > @@ -1880,6 +1880,7 @@ static void uvc_unregister_video(struct uvc_device
> > *dev)
> > continue;
> >  
> > video_unregister_device(>vdev);
> > +   video_unregister_device(>meta.vdev);
> >  
> > uvc_debugfs_cleanup_stream(stream);
> > }
> > @@ -1941,6 +1942,9 @@ static int uvc_register_video(struct uvc_device *dev,
> > return ret;
> > }
> >  
> > +   /* Register a metadata node. */
> > +   uvc_meta_register(stream);
> 
> This can fail, you should handle errors.

Yes, this is in a way deliberate. If metadata node registration fails, the 
driver continues without one. Would you prefer to fail and unregister the 
main node in this case?

> > if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> > stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE;
> > else
> 
> [snip]
> 
> > diff --git a/drivers/media/usb/uvc/uvc_metadata.c 
> b/drivers/media/usb/uvc/uvc_metadata.c
> > new file mode 100644
> > index 000..876badd
> > --- /dev/null
> > +++ b/drivers/media/usb/uvc/uvc_metadata.c
> > @@ -0,0 +1,158 @@
> > +/*
> > + *  uvc_metadata.c  --  USB Video Class driver - Metadata handling
> > + *
> > + *  Copyright (C) 2016
> > + *  Guennadi Liakhovetski (guennadi.liakhovet...@intel.com)
> > + *
> > + *  This program is free software; you can redistribute it and/or
> > modify
> > + *  it under the terms of the GNU General Public License as published
> > by
> > + *  the Free Software Foundation; either version 2 of the License, or
> > + *  (at your option) any later version.
> > + */
> > +
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +
> > +#include 
> > +#include 
> > +#include 
> > +
> > +#include "uvcvideo.h"
> > +
> > +/* 

[PATCH v4] uvcvideo: add a metadata device node

[Guennadi Liakhovetski]

Some UVC video cameras contain metadata in their payload headers. This
patch extracts that data, adding more clock synchronisation information,
on both bulk and isochronous endpoints and makes it available to the
user space on a separate video node, using the V4L2_CAP_META_CAPTURE
capability and the V4L2_BUF_TYPE_META_CAPTURE buffer queue type. Even
though different cameras will have different metadata formats, we use
the same V4L2_META_FMT_UVC pixel format for all of them. Users have to
parse data, based on the specific camera model information. This
version of the patch only creates such metadata nodes for cameras,
specifying a UVC_QUIRK_METADATA_NODE quirk flag.

Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovet...@intel.com>
---

v4:
- add support for isochronous cameras. Metadata is now collected from as 
many payloads as they fit in the buffer
- add a USB Start Of Frame and a system timestamp to each metadata block 
for user-space clock synchronisation
- use a default buffer size of 1024 bytes

Thanks to Laurent for patient long discussions and to everybody, who 
helped me conduct all the investigation into various past, present and 
future UVC cameras :-)

 drivers/media/usb/uvc/Makefile   |   2 +-
 drivers/media/usb/uvc/uvc_driver.c   |   4 +
 drivers/media/usb/uvc/uvc_isight.c   |   2 +-
 drivers/media/usb/uvc/uvc_metadata.c | 158 +++
 drivers/media/usb/uvc/uvc_queue.c|  68 ++-
 drivers/media/usb/uvc/uvc_video.c| 101 +++---
 drivers/media/usb/uvc/uvcvideo.h |  23 -
 drivers/media/v4l2-core/v4l2-ioctl.c |   1 +
 include/uapi/linux/uvcvideo.h|  19 +
 include/uapi/linux/videodev2.h   |   3 +
 10 files changed, 347 insertions(+), 34 deletions(-)
 create mode 100644 drivers/media/usb/uvc/uvc_metadata.c

diff --git a/drivers/media/usb/uvc/Makefile b/drivers/media/usb/uvc/Makefile
index c26d12f..06c7cd3 100644
--- a/drivers/media/usb/uvc/Makefile
+++ b/drivers/media/usb/uvc/Makefile
@@ -1,5 +1,5 @@
 uvcvideo-objs  := uvc_driver.o uvc_queue.o uvc_v4l2.o uvc_video.o uvc_ctrl.o \
- uvc_status.o uvc_isight.o uvc_debugfs.o
+ uvc_status.o uvc_isight.o uvc_debugfs.o uvc_metadata.o
 ifeq ($(CONFIG_MEDIA_CONTROLLER),y)
 uvcvideo-objs  += uvc_entity.o
 endif
diff --git a/drivers/media/usb/uvc/uvc_driver.c 
b/drivers/media/usb/uvc/uvc_driver.c
index 70842c5..9f23dcf 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -1880,6 +1880,7 @@ static void uvc_unregister_video(struct uvc_device *dev)
continue;
 
video_unregister_device(>vdev);
+   video_unregister_device(>meta.vdev);
 
uvc_debugfs_cleanup_stream(stream);
}
@@ -1941,6 +1942,9 @@ static int uvc_register_video(struct uvc_device *dev,
return ret;
}
 
+   /* Register a metadata node. */
+   uvc_meta_register(stream);
+
if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE;
else
diff --git a/drivers/media/usb/uvc/uvc_isight.c 
b/drivers/media/usb/uvc/uvc_isight.c
index 8510e725..fb940cf 100644
--- a/drivers/media/usb/uvc/uvc_isight.c
+++ b/drivers/media/usb/uvc/uvc_isight.c
@@ -100,7 +100,7 @@ static int isight_decode(struct uvc_video_queue *queue, 
struct uvc_buffer *buf,
 }
 
 void uvc_video_decode_isight(struct urb *urb, struct uvc_streaming *stream,
-   struct uvc_buffer *buf)
+   struct uvc_buffer *buf, struct uvc_buffer *meta_buf)
 {
int ret, i;
 
diff --git a/drivers/media/usb/uvc/uvc_metadata.c 
b/drivers/media/usb/uvc/uvc_metadata.c
new file mode 100644
index 000..876badd
--- /dev/null
+++ b/drivers/media/usb/uvc/uvc_metadata.c
@@ -0,0 +1,158 @@
+/*
+ *  uvc_metadata.c  --  USB Video Class driver - Metadata handling
+ *
+ *  Copyright (C) 2016
+ *  Guennadi Liakhovetski (guennadi.liakhovet...@intel.com)
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+#include 
+#include 
+
+#include "uvcvideo.h"
+
+/* 
-
+ * videobuf2 Queue Operations
+ */
+
+static struct vb2_ops uvc_meta_queue_ops = {
+   .queue_setup = uvc_queue_setup,
+   .buf_prepare = uvc_buffer_prepare,
+   .buf_queue = uvc_buffer_queue,
+   .wait_prepare = vb2_ops_wait_prepare,
+   .wait_finish = vb2_ops_wait_finish,
+   .stop_streaming = uvc_stop_streaming,
+};
+
+/* 
-
+ *

Re: [PATCH] v4l: soc-camera: Remove videobuf1 support

[Guennadi Liakhovetski]

On Mon, 13 Mar 2017, Hans Verkuil wrote:

> On 03/12/2017 01:06 PM, Guennadi Liakhovetski wrote:
> > Hi Laurent,
> > 
> > Thanks for the patch. I just checked in the current media/master, there
> > are still 2 users of vb1: sh_mobile_ceu_camera.c and atmel-isi.c. I
> > understand, that they are about to be removed either completely or out of
> > soc-camera, maybe patches for that have already beed submitted, but they
> > haven't been committed yet. Shall we wait until then with this patch?
> > Would be easier to handle dependencies, there isn't any hurry with it,
> > right?
> 
> 
> 
> Both drivers use vb2.

Uhm, ok, I stand corrected. I just grepped for symbols, that get deleted 
by the patch and my grep was too generous.

Thanks
Guennadi

> I've already added this patch to a pull request of mine.
> 
> Regards,
> 
>   Hans
> 

[GIT PULL] soc-camera for 4.12

[Guennadi Liakhovetski]

Hi Mauro,

The following changes since commit 700ea5e0e0dd70420a04e703ff264cc133834cba:

  Merge tag 'v4.11-rc1' into patchwork (2017-03-06 06:49:34 -0300)

are available in the git repository at:

  git://linuxtv.org/gliakhovetski/v4l-dvb.git for-4.12-1

for you to fetch changes up to c259da29a447dbb5737c5c85e99c039263df94cc:

  soc-camera: fix rectangle adjustment in cropping (2017-03-12 12:53:25 +0100)


Bhumika Goyal (1):
  media: i2c: soc_camera: constify v4l2_subdev_* structures

Geliang Tang (1):
  sh_mobile_ceu_camera: use module_platform_driver

Janusz Krzysztofik (1):
  media: i2c/soc_camera: fix ov6650 sensor getting wrong clock

Koji Matsuoka (1):
  soc-camera: fix rectangle adjustment in cropping

 drivers/media/i2c/soc_camera/imx074.c|  6 +++---
 drivers/media/i2c/soc_camera/mt9m001.c   |  6 +++---
 drivers/media/i2c/soc_camera/mt9t031.c   |  6 +++---
 drivers/media/i2c/soc_camera/mt9t112.c   |  6 +++---
 drivers/media/i2c/soc_camera/mt9v022.c   |  6 +++---
 drivers/media/i2c/soc_camera/ov2640.c|  6 +++---
 drivers/media/i2c/soc_camera/ov5642.c|  6 +++---
 drivers/media/i2c/soc_camera/ov6650.c|  8 
 drivers/media/i2c/soc_camera/ov772x.c|  6 +++---
 drivers/media/i2c/soc_camera/ov9640.c|  6 +++---
 drivers/media/i2c/soc_camera/ov9740.c|  6 +++---
 drivers/media/i2c/soc_camera/rj54n1cb0c.c|  6 +++---
 drivers/media/i2c/soc_camera/tw9910.c|  6 +++---
 drivers/media/platform/soc_camera/sh_mobile_ceu_camera.c | 13 +
 drivers/media/platform/soc_camera/soc_scale_crop.c   | 11 ++-
 15 files changed, 47 insertions(+), 57 deletions(-)

Thanks
Guennadi

Re: [PATCHv5 07/16] atmel-isi: remove dependency of the soc-camera framework

[Guennadi Liakhovetski]

Hi Hans,

Thanks for the patch. Why hasn't this patch been CCed to Josh Wu? Is he 
still at Atmel? Adding to CC to check.

A general comment: this patch doesn't only remove soc-camera dependencies, 
it also changes the code and the behaviour. I would prefer to break that 
down in multiple patches, or remove this driver completely and add a new 
one. I'll provide some comments, but of course I cannot make an extensive 
review of a 1200-line of change patch without knowing the hardware and the 
set ups well enough.

On Sat, 11 Mar 2017, Hans Verkuil wrote:

> From: Hans Verkuil 
> 
> This patch converts the atmel-isi driver from a soc-camera driver to a driver
> that is stand-alone.
> 
> Signed-off-by: Hans Verkuil 
> ---
>  drivers/media/platform/soc_camera/Kconfig |3 +-
>  drivers/media/platform/soc_camera/atmel-isi.c | 1209 
> +++--
>  2 files changed, 714 insertions(+), 498 deletions(-)
> 
> diff --git a/drivers/media/platform/soc_camera/Kconfig 
> b/drivers/media/platform/soc_camera/Kconfig
> index 86d74788544f..a37ec91b026e 100644
> --- a/drivers/media/platform/soc_camera/Kconfig
> +++ b/drivers/media/platform/soc_camera/Kconfig
> @@ -29,9 +29,8 @@ config VIDEO_SH_MOBILE_CEU
>  
>  config VIDEO_ATMEL_ISI
>   tristate "ATMEL Image Sensor Interface (ISI) support"
> - depends on VIDEO_DEV && SOC_CAMERA
> + depends on VIDEO_V4L2 && OF && HAS_DMA
>   depends on ARCH_AT91 || COMPILE_TEST
> - depends on HAS_DMA
>   select VIDEOBUF2_DMA_CONTIG
>   ---help---
> This module makes the ATMEL Image Sensor Interface available
> diff --git a/drivers/media/platform/soc_camera/atmel-isi.c 
> b/drivers/media/platform/soc_camera/atmel-isi.c
> index 46de657c3e6d..a6d60c2e207d 100644
> --- a/drivers/media/platform/soc_camera/atmel-isi.c
> +++ b/drivers/media/platform/soc_camera/atmel-isi.c
> @@ -22,18 +22,22 @@
>  #include 
>  #include 
>  #include 
> -
> -#include 
> -#include 
> +#include 
> +
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
> +#include 
>  #include 
>  #include 
> +#include 
>  
>  #include "atmel-isi.h"
>  
> -#define MAX_BUFFER_NUM   32
>  #define MAX_SUPPORT_WIDTH2048
>  #define MAX_SUPPORT_HEIGHT   2048
> -#define VID_LIMIT_BYTES  (16 * 1024 * 1024)
>  #define MIN_FRAME_RATE   15
>  #define FRAME_INTERVAL_MILLI_SEC (1000 / MIN_FRAME_RATE)
>  
> @@ -65,9 +69,37 @@ struct frame_buffer {
>   struct list_head list;
>  };
>  
> +struct isi_graph_entity {
> + struct device_node *node;
> +
> + struct v4l2_async_subdev asd;
> + struct v4l2_subdev *subdev;
> +};
> +
> +/*
> + * struct isi_format - ISI media bus format information
> + * @fourcc:  Fourcc code for this format
> + * @mbus_code:   V4L2 media bus format code.
> + * @bpp: Bytes per pixel (when stored in memory)
> + * @swap:Byte swap configuration value
> + * @support: Indicates format supported by subdev
> + * @skip:Skip duplicate format supported by subdev
> + */
> +struct isi_format {
> + u32 fourcc;
> + u32 mbus_code;
> + u8  bpp;
> + u32 swap;
> +
> + boolsupport;
> + boolskip;
> +};
> +
> +
>  struct atmel_isi {
>   /* Protects the access of variables shared with the ISR */
> - spinlock_t  lock;
> + spinlock_t  irqlock;
> + struct device   *dev;
>   void __iomem*regs;
>  
>   int sequence;
> @@ -76,7 +108,7 @@ struct atmel_isi {
>   struct fbd  *p_fb_descriptors;
>   dma_addr_t  fb_descriptors_phys;
>   struct  list_head dma_desc_head;
> - struct isi_dma_desc dma_desc[MAX_BUFFER_NUM];
> + struct isi_dma_desc dma_desc[VIDEO_MAX_FRAME];
>   boolenable_preview_path;
>  
>   struct completion   complete;
> @@ -90,9 +122,22 @@ struct atmel_isi {
>   struct list_headvideo_buffer_list;
>   struct frame_buffer *active;
>  
> - struct soc_camera_host  soc_host;
> + struct v4l2_device  v4l2_dev;
> + struct video_device *vdev;
> + struct v4l2_async_notifier  notifier;
> + struct isi_graph_entity entity;
> + struct v4l2_format  fmt;
> +
> + struct isi_format   **user_formats;
> + unsigned intnum_user_formats;
> + const struct isi_format *current_fmt;
> +
> + struct mutexlock;
> + struct vb2_queuequeue;
>  };
>  
> +#define notifier_to_isi(n) container_of(n, struct atmel_isi, notifier)
> +
>  static void isi_writel(struct atmel_isi 

Re: [PATCHv5 09/16] ov2640: fix colorspace handling

[Guennadi Liakhovetski]

On Sat, 11 Mar 2017, Hans Verkuil wrote:

> From: Hans Verkuil <hans.verk...@cisco.com>
> 
> The colorspace is independent of whether YUV or RGB is sent to the SoC.
> Fix this.
> 
> Signed-off-by: Hans Verkuil <hans.verk...@cisco.com>

I'm not sure why the first hunk is needed and how it is related :-) But it 
doesn't break anything. I understand, this patch should better go in as a 
part of a series, so

Acked-by: Guennadi Liakhovetski <g.liakhovet...@gmx.de>

Thanks
Guennadi

> ---
>  drivers/media/i2c/soc_camera/ov2640.c | 23 +++
>  1 file changed, 7 insertions(+), 16 deletions(-)
> 
> diff --git a/drivers/media/i2c/soc_camera/ov2640.c 
> b/drivers/media/i2c/soc_camera/ov2640.c
> index 56de18263359..b9a0069f5b33 100644
> --- a/drivers/media/i2c/soc_camera/ov2640.c
> +++ b/drivers/media/i2c/soc_camera/ov2640.c
> @@ -794,10 +794,11 @@ static int ov2640_set_params(struct i2c_client *client, 
> u32 *width, u32 *height,
>   dev_dbg(>dev, "%s: Selected cfmt YUYV (YUV422)", 
> __func__);
>   selected_cfmt_regs = ov2640_yuyv_regs;
>   break;
> - default:
>   case MEDIA_BUS_FMT_UYVY8_2X8:
> + default:
>   dev_dbg(>dev, "%s: Selected cfmt UYVY", __func__);
>   selected_cfmt_regs = ov2640_uyvy_regs;
> + break;
>   }
>  
>   /* reset hardware */
> @@ -865,17 +866,7 @@ static int ov2640_get_fmt(struct v4l2_subdev *sd,
>   mf->width   = priv->win->width;
>   mf->height  = priv->win->height;
>   mf->code= priv->cfmt_code;
> -
> - switch (mf->code) {
> - case MEDIA_BUS_FMT_RGB565_2X8_BE:
> - case MEDIA_BUS_FMT_RGB565_2X8_LE:
> - mf->colorspace = V4L2_COLORSPACE_SRGB;
> - break;
> - default:
> - case MEDIA_BUS_FMT_YUYV8_2X8:
> - case MEDIA_BUS_FMT_UYVY8_2X8:
> - mf->colorspace = V4L2_COLORSPACE_JPEG;
> - }
> + mf->colorspace  = V4L2_COLORSPACE_SRGB;
>   mf->field   = V4L2_FIELD_NONE;
>  
>   return 0;
> @@ -897,17 +888,17 @@ static int ov2640_set_fmt(struct v4l2_subdev *sd,
>   ov2640_select_win(>width, >height);
>  
>   mf->field   = V4L2_FIELD_NONE;
> + mf->colorspace  = V4L2_COLORSPACE_SRGB;
>  
>   switch (mf->code) {
>   case MEDIA_BUS_FMT_RGB565_2X8_BE:
>   case MEDIA_BUS_FMT_RGB565_2X8_LE:
> - mf->colorspace = V4L2_COLORSPACE_SRGB;
> + case MEDIA_BUS_FMT_YUYV8_2X8:
> + case MEDIA_BUS_FMT_UYVY8_2X8:
>   break;
>   default:
>   mf->code = MEDIA_BUS_FMT_UYVY8_2X8;
> - case MEDIA_BUS_FMT_YUYV8_2X8:
> - case MEDIA_BUS_FMT_UYVY8_2X8:
> - mf->colorspace = V4L2_COLORSPACE_JPEG;
> + break;
>   }
>  
>   if (format->which == V4L2_SUBDEV_FORMAT_ACTIVE)
> -- 
> 2.11.0
> 

Re: [PATCH] v4l: soc-camera: Remove videobuf1 support

[Guennadi Liakhovetski]

Hi Laurent,

Thanks for the patch. I just checked in the current media/master, there 
are still 2 users of vb1: sh_mobile_ceu_camera.c and atmel-isi.c. I 
understand, that they are about to be removed either completely or out of 
soc-camera, maybe patches for that have already beed submitted, but they 
haven't been committed yet. Shall we wait until then with this patch? 
Would be easier to handle dependencies, there isn't any hurry with it, 
right?

Thanks
Guennaddi

On Wed, 8 Mar 2017, Laurent Pinchart wrote:

> All remaining soc-camera drivers use videobuf2, drop support for
> videobuf1.
> 
> Signed-off-by: Laurent Pinchart 
> ---
>  drivers/media/platform/soc_camera/soc_camera.c | 103 
> +
>  include/media/soc_camera.h |  14 +---
>  2 files changed, 20 insertions(+), 97 deletions(-)
> 
> diff --git a/drivers/media/platform/soc_camera/soc_camera.c 
> b/drivers/media/platform/soc_camera/soc_camera.c
> index edd1c1de4e33..3c9421f4d8e3 100644
> --- a/drivers/media/platform/soc_camera/soc_camera.c
> +++ b/drivers/media/platform/soc_camera/soc_camera.c
> @@ -37,18 +37,12 @@
>  #include 
>  #include 
>  #include 
> -#include 
>  #include 
>  
>  /* Default to VGA resolution */
>  #define DEFAULT_WIDTH640
>  #define DEFAULT_HEIGHT   480
>  
> -#define is_streaming(ici, icd)   \
> - (((ici)->ops->init_videobuf) ?  \
> -  (icd)->vb_vidq.streaming : \
> -  vb2_is_streaming(&(icd)->vb2_vidq))
> -
>  #define MAP_MAX_NUM 32
>  static DECLARE_BITMAP(device_map, MAP_MAX_NUM);
>  static LIST_HEAD(hosts);
> @@ -367,23 +361,13 @@ static int soc_camera_reqbufs(struct file *file, void 
> *priv,
>  {
>   int ret;
>   struct soc_camera_device *icd = file->private_data;
> - struct soc_camera_host *ici = to_soc_camera_host(icd->parent);
>  
>   WARN_ON(priv != file->private_data);
>  
>   if (icd->streamer && icd->streamer != file)
>   return -EBUSY;
>  
> - if (ici->ops->init_videobuf) {
> - ret = videobuf_reqbufs(>vb_vidq, p);
> - if (ret < 0)
> - return ret;
> -
> - ret = ici->ops->reqbufs(icd, p);
> - } else {
> - ret = vb2_reqbufs(>vb2_vidq, p);
> - }
> -
> + ret = vb2_reqbufs(>vb2_vidq, p);
>   if (!ret)
>   icd->streamer = p->count ? file : NULL;
>   return ret;
> @@ -393,61 +377,44 @@ static int soc_camera_querybuf(struct file *file, void 
> *priv,
>  struct v4l2_buffer *p)
>  {
>   struct soc_camera_device *icd = file->private_data;
> - struct soc_camera_host *ici = to_soc_camera_host(icd->parent);
>  
>   WARN_ON(priv != file->private_data);
>  
> - if (ici->ops->init_videobuf)
> - return videobuf_querybuf(>vb_vidq, p);
> - else
> - return vb2_querybuf(>vb2_vidq, p);
> + return vb2_querybuf(>vb2_vidq, p);
>  }
>  
>  static int soc_camera_qbuf(struct file *file, void *priv,
>  struct v4l2_buffer *p)
>  {
>   struct soc_camera_device *icd = file->private_data;
> - struct soc_camera_host *ici = to_soc_camera_host(icd->parent);
>  
>   WARN_ON(priv != file->private_data);
>  
>   if (icd->streamer != file)
>   return -EBUSY;
>  
> - if (ici->ops->init_videobuf)
> - return videobuf_qbuf(>vb_vidq, p);
> - else
> - return vb2_qbuf(>vb2_vidq, p);
> + return vb2_qbuf(>vb2_vidq, p);
>  }
>  
>  static int soc_camera_dqbuf(struct file *file, void *priv,
>   struct v4l2_buffer *p)
>  {
>   struct soc_camera_device *icd = file->private_data;
> - struct soc_camera_host *ici = to_soc_camera_host(icd->parent);
>  
>   WARN_ON(priv != file->private_data);
>  
>   if (icd->streamer != file)
>   return -EBUSY;
>  
> - if (ici->ops->init_videobuf)
> - return videobuf_dqbuf(>vb_vidq, p, file->f_flags & 
> O_NONBLOCK);
> - else
> - return vb2_dqbuf(>vb2_vidq, p, file->f_flags & O_NONBLOCK);
> + return vb2_dqbuf(>vb2_vidq, p, file->f_flags & O_NONBLOCK);
>  }
>  
>  static int soc_camera_create_bufs(struct file *file, void *priv,
>   struct v4l2_create_buffers *create)
>  {
>   struct soc_camera_device *icd = file->private_data;
> - struct soc_camera_host *ici = to_soc_camera_host(icd->parent);
>   int ret;
>  
> - /* videobuf2 only */
> - if (ici->ops->init_videobuf)
> - return -ENOTTY;
> -
>   if (icd->streamer && icd->streamer != file)
>   return -EBUSY;
>  
> @@ -461,24 +428,14 @@ static int soc_camera_prepare_buf(struct file *file, 
> void *priv,
> struct v4l2_buffer *b)
>  {
>   struct soc_camera_device *icd = file->private_data;
> - struct soc_camera_host *ici = 

[PATCH v2] soc-camera: fix rectangle adjustment in cropping

[Guennadi Liakhovetski]

From: Koji Matsuoka <koji.matsuoka...@renesas.com>

update_subrect() adjusts the sub-rectangle to be inside a base area.
It checks width and height to not exceed those of the area, then it
checks the low border (left or top) to lie within the area, then the
high border (right or bottom) to lie there too. This latter check has
a bug, which is fixed by this patch.

Signed-off-by: Koji Matsuoka <koji.matsuoka...@renesas.com>
Signed-off-by: Yoshihiro Kaneko <ykaneko0...@gmail.com>
[g.liakhovet...@gmx.de: dropped supposedly wrong hunks]
Signed-off-by: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
---

v2: fix a silly typo, reported by Laurent, thanks! Also renamed the 
function, according to his suggestion.

 drivers/media/platform/soc_camera/soc_scale_crop.c | 11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/media/platform/soc_camera/soc_scale_crop.c 
b/drivers/media/platform/soc_camera/soc_scale_crop.c
index f77252d..0116097 100644
--- a/drivers/media/platform/soc_camera/soc_scale_crop.c
+++ b/drivers/media/platform/soc_camera/soc_scale_crop.c
@@ -62,7 +62,8 @@ int soc_camera_client_g_rect(struct v4l2_subdev *sd, struct 
v4l2_rect *rect)
 EXPORT_SYMBOL(soc_camera_client_g_rect);
 
 /* Client crop has changed, update our sub-rectangle to remain within the area 
*/
-static void update_subrect(struct v4l2_rect *rect, struct v4l2_rect *subrect)
+static void move_and_crop_subrect(struct v4l2_rect *rect,
+ struct v4l2_rect *subrect)
 {
if (rect->width < subrect->width)
subrect->width = rect->width;
@@ -72,14 +73,14 @@ static void update_subrect(struct v4l2_rect *rect, struct 
v4l2_rect *subrect)
 
if (rect->left > subrect->left)
subrect->left = rect->left;
-   else if (rect->left + rect->width >
+   else if (rect->left + rect->width <
 subrect->left + subrect->width)
subrect->left = rect->left + rect->width -
subrect->width;
 
if (rect->top > subrect->top)
subrect->top = rect->top;
-   else if (rect->top + rect->height >
+   else if (rect->top + rect->height <
 subrect->top + subrect->height)
subrect->top = rect->top + rect->height -
subrect->height;
@@ -216,7 +217,7 @@ int soc_camera_client_s_selection(struct v4l2_subdev *sd,
 
if (!ret) {
*target_rect = *cam_rect;
-   update_subrect(target_rect, subrect);
+   move_and_crop_subrect(target_rect, subrect);
}
 
return ret;
@@ -299,7 +300,7 @@ static int client_set_fmt(struct soc_camera_device *icd,
if (host_1to1)
*subrect = *rect;
else
-   update_subrect(rect, subrect);
+   move_and_crop_subrect(rect, subrect);
 
return 0;
 }
-- 
1.9.3

Re: [PATCH] soc-camera: fix rectangle adjustment in cropping

[Guennadi Liakhovetski]

On Mon, 27 Feb 2017, Laurent Pinchart wrote:

> Hi Guennadi,
> 
> On Monday 27 Feb 2017 09:54:19 Guennadi Liakhovetski wrote:
> > On Mon, 27 Feb 2017, Laurent Pinchart wrote:
> > > On Sunday 26 Feb 2017 21:58:16 Guennadi Liakhovetski wrote:
> > >> From: Koji Matsuoka <koji.matsuoka...@renesas.com>
> > >> 
> > >> update_subrect() adjusts the sub-rectangle to be inside a base area.
> > >> It checks width and height to not exceed those of the area, then it
> > >> checks the low border (left or top) to lie within the area, then the
> > >> high border (right or bottom) to lie there too. This latter check has
> > >> a bug, which is fixed by this patch.
> > >> 
> > >> Signed-off-by: Koji Matsuoka <koji.matsuoka...@renesas.com>
> > >> Signed-off-by: Yoshihiro Kaneko <ykaneko0...@gmail.com>
> > >> [g.liakhovet...@gmx.de: dropped supposedly wrong hunks]
> > >> Signed-off-by: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> > >> ---
> > >> 
> > >> This is a part of the https://patchwork.linuxtv.org/patch/26441/
> > >> submitted almost 2.5 years ago. Back then I commented to the patch but
> > >> never got a reply or an update. I preserved original authorship and Sob
> > >> tags, although this version only uses a small portion of the original
> > >> patch. This version is of course completely untested, any testing (at
> > >> least regression) would be highly appreciated! This code is only used by
> > >> the SH CEU driver and only in cropping / zooming scenarios.
> > >> 
> > >>  drivers/media/platform/soc_camera/soc_scale_crop.c | 4 ++--
> > >>  1 file changed, 2 insertions(+), 2 deletions(-)
> > >> 
> > >> diff --git a/drivers/media/platform/soc_camera/soc_scale_crop.c
> > >> b/drivers/media/platform/soc_camera/soc_scale_crop.c index
> > >> f77252d..4bfc1bf
> > >> 100644
> > >> --- a/drivers/media/platform/soc_camera/soc_scale_crop.c
> > >> +++ b/drivers/media/platform/soc_camera/soc_scale_crop.c
> > >> @@ -70,14 +70,14 @@ static void update_subrect(struct v4l2_rect *rect,
> > >> struct v4l2_rect *subrect)
> > >>  if (rect->height < subrect->height)
> > >>  subrect->height = rect->height;
> > >> 
> > >> -if (rect->left > subrect->left)
> > >> +if (rect->left < subrect->left)
> > > 
> > > This looks wrong to me. If the purpose of the function is indeed to adjust
> > > subrect to stay within rect, the condition doesn't need to be changed.
> > > 
> > >>  subrect->left = rect->left;
> > >>  else if (rect->left + rect->width >
> > >>   subrect->left + subrect->width)
> > > 
> > > This condition, however, is wrong.
> > 
> > Agh, of course, I meant to change this one! Thanks for catching.
> > 
> > >>  subrect->left = rect->left + rect->width -
> > >>  subrect->width;
> > > 
> > > More than that, adjusting the width first and then the left coordinate can
> > > result in an incorrect width.
> > 
> > The width is adjusted in the beginning only to stay within the area, you
> > cannot go beyond it anyway. So, that has to be done anyway. And then the
> > origin is adjusted.
> > 
> > > It looks to me like you should drop the width
> > > check at the beginning of this function, and turn the "else if" here into
> > > an "if" with the right condition. Or, even better in my opinion, use the
> > > min/max/clamp macros.
> > 
> > Well, that depends on what result we want to achieve, what parameter we
> > prioritise. This approach prioritises width and height, and then adjusts
> > edges to accommodate as much of them as possible. A different approach
> > would be to prioritise the origin (top and left) and adjust width and
> > height to stay within the area. Do we have a preference for this?
> 
> Don't you need both ? "Inside the area" is a pretty well-defined concept :-)
> 
>   subrect->left = max(subrect->left, rect->left);

I prefer to avoid assignments like "a = max(a, b)" to avoid a redundant 
"a = a" assignment, when a >= b :-)

>   subrect->top = max(subrect->top, rect->top);
>

Re: [PATCH] soc-camera: fix rectangle adjustment in cropping

[Guennadi Liakhovetski]

On Mon, 27 Feb 2017, Hans Verkuil wrote:

> On 02/27/2017 10:02 AM, Laurent Pinchart wrote:
> > Hi Guennadi,
> > 
> > On Monday 27 Feb 2017 09:54:19 Guennadi Liakhovetski wrote:
> >> On Mon, 27 Feb 2017, Laurent Pinchart wrote:
> >>> On Sunday 26 Feb 2017 21:58:16 Guennadi Liakhovetski wrote:
> >>>> From: Koji Matsuoka <koji.matsuoka...@renesas.com>
> >>>>
> >>>> update_subrect() adjusts the sub-rectangle to be inside a base area.
> >>>> It checks width and height to not exceed those of the area, then it
> >>>> checks the low border (left or top) to lie within the area, then the
> >>>> high border (right or bottom) to lie there too. This latter check has
> >>>> a bug, which is fixed by this patch.
> >>>>
> >>>> Signed-off-by: Koji Matsuoka <koji.matsuoka...@renesas.com>
> >>>> Signed-off-by: Yoshihiro Kaneko <ykaneko0...@gmail.com>
> >>>> [g.liakhovet...@gmx.de: dropped supposedly wrong hunks]
> >>>> Signed-off-by: Guennadi Liakhovetski <g.liakhovet...@gmx.de>
> >>>> ---
> >>>>
> >>>> This is a part of the https://patchwork.linuxtv.org/patch/26441/
> >>>> submitted almost 2.5 years ago. Back then I commented to the patch but
> >>>> never got a reply or an update. I preserved original authorship and Sob
> >>>> tags, although this version only uses a small portion of the original
> >>>> patch. This version is of course completely untested, any testing (at
> >>>> least regression) would be highly appreciated! This code is only used by
> >>>> the SH CEU driver and only in cropping / zooming scenarios.
> >>>>
> >>>>  drivers/media/platform/soc_camera/soc_scale_crop.c | 4 ++--
> >>>>  1 file changed, 2 insertions(+), 2 deletions(-)
> >>>>
> >>>> diff --git a/drivers/media/platform/soc_camera/soc_scale_crop.c
> >>>> b/drivers/media/platform/soc_camera/soc_scale_crop.c index
> >>>> f77252d..4bfc1bf
> >>>> 100644
> >>>> --- a/drivers/media/platform/soc_camera/soc_scale_crop.c
> >>>> +++ b/drivers/media/platform/soc_camera/soc_scale_crop.c
> >>>> @@ -70,14 +70,14 @@ static void update_subrect(struct v4l2_rect *rect,
> >>>> struct v4l2_rect *subrect)
> >>>>  if (rect->height < subrect->height)
> >>>>  subrect->height = rect->height;
> >>>>
> >>>> -if (rect->left > subrect->left)
> >>>> +if (rect->left < subrect->left)
> >>>
> >>> This looks wrong to me. If the purpose of the function is indeed to adjust
> >>> subrect to stay within rect, the condition doesn't need to be changed.
> >>>
> >>>>  subrect->left = rect->left;
> >>>>  else if (rect->left + rect->width >
> >>>>   subrect->left + subrect->width)
> >>>
> >>> This condition, however, is wrong.
> >>
> >> Agh, of course, I meant to change this one! Thanks for catching.
> >>
> >>>>  subrect->left = rect->left + rect->width -
> >>>>  subrect->width;
> >>>
> >>> More than that, adjusting the width first and then the left coordinate can
> >>> result in an incorrect width.
> >>
> >> The width is adjusted in the beginning only to stay within the area, you
> >> cannot go beyond it anyway. So, that has to be done anyway. And then the
> >> origin is adjusted.
> >>
> >>> It looks to me like you should drop the width
> >>> check at the beginning of this function, and turn the "else if" here into
> >>> an "if" with the right condition. Or, even better in my opinion, use the
> >>> min/max/clamp macros.
> >>
> >> Well, that depends on what result we want to achieve, what parameter we
> >> prioritise. This approach prioritises width and height, and then adjusts
> >> edges to accommodate as much of them as possible. A different approach
> >> would be to prioritise the origin (top and left) and adjust width and
> >> height to stay within the area. Do we have a preference for this?
> > 
> > Don't you need both ? "Inside the area" is a pretty well-defined concept :-)
>