Re: [PATCH] media: unify some sony camera sensors pattern naming
Hi Bingbu,
On Mon, Nov 26, 2018 at 7:56 PM wrote:
>
> From: Bingbu Cao
>
> Some Sony camera sensors have same test pattern
> definitions, this patch unify the pattern naming
> to make it more clear to the userspace.
>
> Suggested-by: Sakari Ailus
> Signed-off-by: Bingbu Cao
> ---
> drivers/media/i2c/imx258.c | 8
> drivers/media/i2c/imx319.c | 8
> drivers/media/i2c/imx355.c | 8
> 3 files changed, 12 insertions(+), 12 deletions(-)
>
Thanks for the patch! One comment inline.
> diff --git a/drivers/media/i2c/imx258.c b/drivers/media/i2c/imx258.c
> index 31a1e2294843..a8a2880c6b4e 100644
> --- a/drivers/media/i2c/imx258.c
> +++ b/drivers/media/i2c/imx258.c
> @@ -504,10 +504,10 @@ struct imx258_mode {
>
> static const char * const imx258_test_pattern_menu[] = {
> "Disabled",
> - "Color Bars",
> - "Solid Color",
> - "Grey Color Bars",
> - "PN9"
> + "Solid Colour",
> + "Eight Vertical Colour Bars",
Is it just me or "solid color" and "color bars" are being swapped
here? Did the driver had the names mixed up before or the order of
modes is different between these sensors?
Best regards,
Tomasz
Re: [PATCH v7 00/16] Intel IPU3 ImgU patchset
On Thu, Nov 29, 2018 at 6:43 AM Laurent Pinchart wrote: > > Hello Yong, > > On Tuesday, 30 October 2018 00:22:54 EET Yong Zhi wrote: > > Hi, > > > > This series adds support for the Intel IPU3 (Image Processing Unit) > > ImgU which is essentially a modern memory-to-memory ISP. It implements > > raw Bayer to YUV image format conversion as well as a large number of > > other pixel processing algorithms for improving the image quality. > > > > Meta data formats are defined for image statistics (3A, i.e. automatic > > white balance, exposure and focus, histogram and local area contrast > > enhancement) as well as for the pixel processing algorithm parameters. > > The documentation for these formats is currently not included in the > > patchset but will be added in a future version of this set. > > > > The algorithm parameters need to be considered specific to a given frame > > and typically a large number of these parameters change on frame to frame > > basis. Additionally, the parameters are highly structured (and not a flat > > space of independent configuration primitives). They also reflect the > > data structures used by the firmware and the hardware. On top of that, > > the algorithms require highly specialized user space to make meaningful > > use of them. For these reasons it has been chosen video buffers to pass > > the parameters to the device. > > > > On individual patches: > > > > The heart of ImgU is the CSS, or Camera Subsystem, which contains the > > image processors and HW accelerators. > > > > The 3A statistics and other firmware parameter computation related > > functions are implemented in patch 11. > > > > All IPU3 pipeline default settings can be found in patch 10. > > > > To access DDR via ImgU's own memory space, IPU3 is also equipped with > > its own MMU unit, the driver is implemented in patch 6. > > > > Patch 7 uses above driver for DMA mapping operation. > > > > The communication between IPU3 firmware and driver is implemented with > > circular queues in patch 8. > > > > Patch 9 provide some utility functions and manage IPU3 fw download and > > install. > > > > The firmware which is called ipu3-fw.bin can be downloaded from: > > > > git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git > > (commit 2c27b0cb02f18c022d8378e0e1abaf8b7ae8188f) > > > > Firmware ABI is defined in patches 4 and 5. > > > > Patches 12 and 13 are of the same file, the former contains all h/w > > programming related code, the latter implements interface functions for > > access fw & hw capabilities. > > > > Patch 14 has a dependency on Sakari's V4L2_BUF_TYPE_META_OUTPUT work: > > > > https://patchwork.kernel.org/patch/9976295/> > > > > Patch 15 represents the top level that glues all of the other components > > together, passing arguments between the components. > > > > Patch 16 is a recent effort to extend v6 for advanced camera features like > > Continuous View Finder (CVF) and Snapshot During Video(SDV) support. > > > > Link to user space implementation: > > > > git clone https://chromium.googlesource.com/chromiumos/platform/arc-camera > > > > ImgU media topology print: > > [snip] > > > v4l2-compliance utility is built with Sakari's patches for meta data > > output support(rebased): > > > > https://patchwork.linuxtv.org/patch/43370/> > > https://patchwork.linuxtv.org/patch/43369/> > > > > The test (v4l2-compliance -m 0) passes without error, outputs are appended > > at the end of revision history. > > > > Note: > > > > 1. Link pad flag of video nodes (i.e. ipu3-imgu 0 output) need to be enabled > > prior to the test. > > 2. Stream tests are not performed since it requires pre-configuration for > > each case. > > And that's a bit of an issue. I've tested the driver with a small script based > on media-ctl to configure links and yavta to interface with the video nodes, > and got the following oops: > > [ 136.927788] divide error: [#1] PREEMPT SMP PTI > [ 136.927801] CPU: 2 PID: 2069 Comm: yavta Not tainted 4.20.0-rc1+ #9 > [ 136.927806] Hardware name: HP Soraka/Soraka, BIOS 08/30/2018 > [ 136.927820] RIP: 0010:ipu3_css_osys_calc+0xc54/0xe14 [ipu3_imgu] > [ 136.927825] Code: 89 44 24 28 42 8b 44 86 6c f7 54 24 04 81 64 24 28 00 fd > ff ff 81 64 24 04 00 03 00 00 8d 44 03 ff 81 44 24 28 80 03 00 00 99 fb > 0f af c3 bb 20 00 00 00 99 f7 fb 8b 5c 24 40 83 fd 01 19 d2 > [ 136.927830] RSP: 0018:9af2c0b837c8 EFLAGS: 00010202 > [ 136.927835] RAX: RBX: RCX: > 9af2c3e353c0 > [ 136.927839] RDX: RSI: 9af2c0b838e0 RDI: > 9af2c3e353c0 > [ 136.927843] RBP: 0001 R08: R09: > 9af2c0b83880 > [ 136.927846] R10: 9af2c3e353c0 R11: 9af2c3e357c0 R12: > 03a0 > [ 136.927849] R13: 00025a0a R14: R15: > > [ 136.927854] FS: 7f1eca167700() GS:8c19fab0() knlGS: > > [ 136.927858] CS: 0010 DS: ES:
Re: [PATCH v10 4/4] media: add Rockchip VPU JPEG encoder driver
On Wed, Nov 28, 2018 at 10:05 AM Ezequiel Garcia wrote:
>
> On Tue, 2018-11-27 at 19:09 +0900, Tomasz Figa wrote:
> > On Fri, Nov 23, 2018 at 5:24 AM Ezequiel Garcia
> > wrote:
> > [snip]
> > > > > +const struct rockchip_vpu_variant rk3288_vpu_variant = {
> > > > > + .enc_offset = 0x0,
> > > > > + .enc_fmts = rk3288_vpu_enc_fmts,
> > > > > + .num_enc_fmts = ARRAY_SIZE(rk3288_vpu_enc_fmts),
> > > > > + .codec_ops = rk3288_vpu_codec_ops,
> > > > > + .codec = RK_VPU_CODEC_JPEG,
> > > > > + .vepu_irq = rk3288_vepu_irq,
> > > > > + .init = rk3288_vpu_hw_init,
> > > > > + .clk_names = {"aclk", "hclk"},
> > > >
> > > > nit: Spaces inside the brackets.
> > > >
> > >
> > > You mean you this style is prefered?
> > >
> > > .clk_names = { "aclk", "hclk" },
> > >
> > > Grepping thru sources, it seems there is no convention on this,
> > > so it's your call.
> > >
> >
> > I thought this is a part of CodingStyle, but it doesn't seem to
> > mention it. I personally prefer to have the spaces there, but we can
> > go with your personal preferences here. :)
>
> OK.
>
> > [snip]
> > > > > +* by .vidioc_s_fmt_vid_cap_mplane() callback
> > > > > +*/
> > > > > + reg = VEPU_REG_IN_IMG_CTRL_ROW_LEN(pix_fmt->width);
> > > > > + vepu_write_relaxed(vpu, reg, VEPU_REG_INPUT_LUMA_INFO);
> > > > > +
> > > > > + reg = VEPU_REG_IN_IMG_CTRL_OVRFLR_D4(0) |
> > > > > + VEPU_REG_IN_IMG_CTRL_OVRFLB(0);
> > > >
> > > > For reference, this register controls the input crop, as the offset
> > > > from the right/bottom within the last macroblock. The offset from the
> > > > right must be divided by 4 and so the crop must be aligned to 4 pixels
> > > > horizontally.
> > > >
> > >
> > > OK, I'll add a comment.
> > >
> >
> > I meant to refer to the comment Hans had, about input images of
> > resolutions that are not of full macroblocks, so the comment would
> > probably go to the TODO file together with Hans's note.
>
> Got it.
>
> > [snip]
> > > > > +static inline u32 vepu_read(struct rockchip_vpu_dev *vpu, u32 reg)
> > > > > +{
> > > > > + u32 val = readl(vpu->enc_base + reg);
> > > > > +
> > > > > + vpu_debug(6, "MARK: get reg[%03d]: %08x\n", reg / 4, val);
> > > >
> > > > I remember seeing this "MARK" in the logs when debugging. I don't
> > > > think it's desired here.
> > > >
> > > > How about printing "%s(%03d) = %08x\n" for reads and "%s(%08x,
> > > > %03d)\n" for writes?
> >
> > Actually, I missed the 0x prefix for hex values and possibly also
> > changing the decimal register offset to hexadecimal:
> > "%s(0x%04x) = 0x%08x\n"
> >
> > >
> > > Makes sense, but why a %s string format?
> > >
> >
> > For __func__, so that we end up with messages like:
> >
> > vepu_write(0x12345678, 0x0123)
> > vepu_read(0x0123) = 0x12345678
> >
>
> vepu_debug already uses __func__, so it would look like this:
>
> [ 27.125090] vepu_write_relaxed:215: 0x001f = 0x01010101
> [ 27.127413] vepu_write:221: 0x0069 = 0xfc00
> [ 27.129673] vepu_write_relaxed:215: 0x0036 = 0x1000
> [ 27.132079] vepu_write:221: 0x0067 = 0x00f01461
> [ 27.135042] vepu_read:229: 0x006d = 0x1003
> [ 27.137450] vepu_read:229: 0x0035 = 0x00020318
> [ 27.139778] vepu_write:221: 0x006d = 0x
> [ 27.142144] vepu_write:221: 0x0036 = 0x
>
> Unless we use a different debug macro for i/o access.
>
Okay, I missed that part of vepu_debug(). We can drop the %s and
explicit __func__ from the message then.
> > [snip]
> > > > > + dst->field = src->field;
> > > > > + dst->timecode = src->timecode;
> > > >
> > > > Time code is only valid if the buffer has V4L2_BUF_FLAG_TIMECODE set.
> > > > I don't think there is any use case for mem2mem devices for it.
> > > >
> > >
> > > Right. Other mem2mem drivers seem to pass thru the timecode like this:
> > >
> > > if (in_vb->flags & V4L2_BUF_FLAG_TIMECODE)
> > > out_vb->timecode = in_vb->timecode;
> > >
> > > It fails a v4l2-compliance test without it.
> > >
> >
> > Hmm, I don't see the spec defining it to be copied by a mem2mem device
> > and I wonder if it's actually of any use for those. Hans, could you
> > comment on this?
> >
>
> I asked Hans about this and he said timecode should behave as timestamp,
> and be carried from the output queue to the capture queue.
>
> This helper will take care of it: https://patchwork.linuxtv.org/patch/52961/
>
Fair enough.
Best regards,
Tomasz
Re: [PATCH v10 4/4] media: add Rockchip VPU JPEG encoder driver
On Fri, Nov 23, 2018 at 5:24 AM Ezequiel Garcia wrote:
[snip]
> > > +const struct rockchip_vpu_variant rk3288_vpu_variant = {
> > > + .enc_offset = 0x0,
> > > + .enc_fmts = rk3288_vpu_enc_fmts,
> > > + .num_enc_fmts = ARRAY_SIZE(rk3288_vpu_enc_fmts),
> > > + .codec_ops = rk3288_vpu_codec_ops,
> > > + .codec = RK_VPU_CODEC_JPEG,
> > > + .vepu_irq = rk3288_vepu_irq,
> > > + .init = rk3288_vpu_hw_init,
> > > + .clk_names = {"aclk", "hclk"},
> >
> > nit: Spaces inside the brackets.
> >
>
> You mean you this style is prefered?
>
> .clk_names = { "aclk", "hclk" },
>
> Grepping thru sources, it seems there is no convention on this,
> so it's your call.
>
I thought this is a part of CodingStyle, but it doesn't seem to
mention it. I personally prefer to have the spaces there, but we can
go with your personal preferences here. :)
[snip]
> > > +* by .vidioc_s_fmt_vid_cap_mplane() callback
> > > +*/
> > > + reg = VEPU_REG_IN_IMG_CTRL_ROW_LEN(pix_fmt->width);
> > > + vepu_write_relaxed(vpu, reg, VEPU_REG_INPUT_LUMA_INFO);
> > > +
> > > + reg = VEPU_REG_IN_IMG_CTRL_OVRFLR_D4(0) |
> > > + VEPU_REG_IN_IMG_CTRL_OVRFLB(0);
> >
> > For reference, this register controls the input crop, as the offset
> > from the right/bottom within the last macroblock. The offset from the
> > right must be divided by 4 and so the crop must be aligned to 4 pixels
> > horizontally.
> >
>
> OK, I'll add a comment.
>
I meant to refer to the comment Hans had, about input images of
resolutions that are not of full macroblocks, so the comment would
probably go to the TODO file together with Hans's note.
[snip]
> > > +static inline u32 vepu_read(struct rockchip_vpu_dev *vpu, u32 reg)
> > > +{
> > > + u32 val = readl(vpu->enc_base + reg);
> > > +
> > > + vpu_debug(6, "MARK: get reg[%03d]: %08x\n", reg / 4, val);
> >
> > I remember seeing this "MARK" in the logs when debugging. I don't
> > think it's desired here.
> >
> > How about printing "%s(%03d) = %08x\n" for reads and "%s(%08x,
> > %03d)\n" for writes?
Actually, I missed the 0x prefix for hex values and possibly also
changing the decimal register offset to hexadecimal:
"%s(0x%04x) = 0x%08x\n"
> >
>
> Makes sense, but why a %s string format?
>
For __func__, so that we end up with messages like:
vepu_write(0x12345678, 0x0123)
vepu_read(0x0123) = 0x12345678
[snip]
> > > + dst->field = src->field;
> > > + dst->timecode = src->timecode;
> >
> > Time code is only valid if the buffer has V4L2_BUF_FLAG_TIMECODE set.
> > I don't think there is any use case for mem2mem devices for it.
> >
>
> Right. Other mem2mem drivers seem to pass thru the timecode like this:
>
> if (in_vb->flags & V4L2_BUF_FLAG_TIMECODE)
> out_vb->timecode = in_vb->timecode;
>
> It fails a v4l2-compliance test without it.
>
Hmm, I don't see the spec defining it to be copied by a mem2mem device
and I wonder if it's actually of any use for those. Hans, could you
comment on this?
> > > + dst->vb2_buf.timestamp = src->vb2_buf.timestamp;
> > > + dst->flags &= ~V4L2_BUF_FLAG_TSTAMP_SRC_MASK;
> > > + dst->flags |= src->flags & V4L2_BUF_FLAG_TSTAMP_SRC_MASK;
> >
> > Not V4L2_BUF_FLAG_TIMESTAMP_COPY?
> >
>
> I believe v4l core should take care of it in __fill_v4l2_buffer,
> as timestamp_flags is set when the vb2_queue structs are init'ed.
>
Ah, okay, I confused this with V4L2_BUF_FLAG_TIMESTAMP_MASK.
> > > +
> > > + if (bytesused) {
> >
> > Should we check whether bytesused (read from hardware) is not bigger
> > than size of the buffer?
> >
>
> Good catch, makes sense. OTOH, if bytesused is bigger than the dst
> buffer, it is also bigger than the bounce buffer. I guess the IOMMU
> helps prevents nasty issues?
>
Yes, in that case it would likely trigger an IOMMU fault.
Another thing is whether there isn't some special bit somewhere high
in the register we're reading from that we're not aware of, which
would make the bytesused bigger than the buffer.
[snip]
> > > +void rockchip_vpu_irq_done(struct rockchip_vpu_dev *vpu,
> > > + unsigned int bytesused,
> > > + enum vb2_buffer_state result)
> > > +{
> > > + struct rockchip_vpu_ctx *ctx =
> > > + (struct rockchip_vpu_ctx
> > > *)v4l2_m2m_get_curr_priv(vpu->m2m_dev);
> >
> > I don't think we need to cast from void *?
> >
>
> Right.
>
> > > +
> > > + /* Atomic watchdog cancel. The worker may still be
> > > +* running after calling this.
> > > +*/
> >
> > Wrong multi-line comment style.
> >
>
> Right.
>
> > > + cancel_delayed_work(>watchdog_work);
> > > + if (ctx)
> > > + rockchip_vpu_job_finish(vpu, ctx, bytesused, result);
> > > +}
> > > +
> > > +void rockchip_vpu_watchdog(struct work_struct *work)
> > > +{
> > > + struct rockchip_vpu_dev *vpu;
> > > + struct rockchip_vpu_ctx *ctx;
> > > +
> >
Re: [PATCH] v4l2-ioctl: Zero v4l2_pix_format_mplane reserved fields
On Tue, Nov 27, 2018 at 8:29 AM Ezequiel Garcia wrote:
>
> On Mon, 2018-11-26 at 13:14 +0900, Tomasz Figa wrote:
> > Hi Ezequiel,
> >
> > On Sat, Nov 24, 2018 at 2:20 AM Ezequiel Garcia
> > wrote:
> > > Make the core set the reserved fields to zero in
> > > v4l2_pix_format_mplane and v4l2_plane_pix_format structs,
> > > for _MPLANE queue types.
> > >
> > > Moving this to the core avoids having to do so in each
> > > and every driver.
> > >
> > > Suggested-by: Tomasz Figa
> > > Signed-off-by: Ezequiel Garcia
> > > ---
> > > drivers/media/v4l2-core/v4l2-ioctl.c | 51
> > > 1 file changed, 45 insertions(+), 6 deletions(-)
> > >
> >
> > Thanks for the patch. Please see my comments inline.
> >
> > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > index 10b862dcbd86..3858fffc3e68 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > > @@ -1420,6 +1420,7 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops
> > > *ops,
> > > {
> > > struct v4l2_format *p = arg;
> > > int ret = check_fmt(file, p->type);
> > > + int i;
> > >
> > > if (ret)
> > > return ret;
> > > @@ -1458,7 +1459,13 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops
> > > *ops,
> > > p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> > > return ret;
> > > case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> > > - return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> > > + ret = ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> > > + memset(p->fmt.pix_mp.reserved, 0,
> > > + sizeof(p->fmt.pix_mp.reserved));
> > > + for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> > > + memset(p->fmt.pix_mp.plane_fmt[i].reserved, 0,
> > > +
> > > sizeof(p->fmt.pix_mp.plane_fmt[i].reserved));
> > > + return ret;
> > > case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> > > return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
> > > case V4L2_BUF_TYPE_VBI_CAPTURE:
> > > @@ -1474,7 +1481,13 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops
> > > *ops,
> > > p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> > > return ret;
> > > case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> > > - return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> > > + ret = ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> > > + memset(p->fmt.pix_mp.reserved, 0,
> > > + sizeof(p->fmt.pix_mp.reserved));
> > > + for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> > > + memset(p->fmt.pix_mp.plane_fmt[i].reserved, 0,
> > > +
> > > sizeof(p->fmt.pix_mp.plane_fmt[i].reserved));
> > > + return ret;
> >
> > I wonder if we need this for G_FMT. The driver can just memset() the
> > whole struct itself and then just initialize the fields it cares
> > about, but actually in many cases the driver will just include an
> > instance of the pix_fmt(_mp) struct in its internal state (which has
> > the reserved fields already zeroed) and just copy it to the target
> > struct in the callback.
> >
>
> Perhaps in many cases, but from code inspection it seems not
> all of them (randomly opened vicodec & mtk-jpeg and both need
> a memset!).
>
> I'm thinkig it'd best to keep it this way for consistency
> and to avoid having the worry at all about this in the drivers.
I guess it makes sense indeed. The structure isn't terribly big, so
there shouldn't be any significant performance penalty I suppose.
Perhaps the next step would be to have someone clean up the existing drivers.
>
> Should we use CLEAR_AFTER_FIELD here as well?
>
I think so.
Best regards,
Tomasz
Re: [PATCH] v4l2-ioctl: Zero v4l2_pix_format_mplane reserved fields
Hi Ezequiel,
On Sat, Nov 24, 2018 at 2:20 AM Ezequiel Garcia wrote:
>
> Make the core set the reserved fields to zero in
> v4l2_pix_format_mplane and v4l2_plane_pix_format structs,
> for _MPLANE queue types.
>
> Moving this to the core avoids having to do so in each
> and every driver.
>
> Suggested-by: Tomasz Figa
> Signed-off-by: Ezequiel Garcia
> ---
> drivers/media/v4l2-core/v4l2-ioctl.c | 51
> 1 file changed, 45 insertions(+), 6 deletions(-)
>
Thanks for the patch. Please see my comments inline.
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c
> b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 10b862dcbd86..3858fffc3e68 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1420,6 +1420,7 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
> {
> struct v4l2_format *p = arg;
> int ret = check_fmt(file, p->type);
> + int i;
>
> if (ret)
> return ret;
> @@ -1458,7 +1459,13 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
> p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> return ret;
> case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> - return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> + ret = ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> + memset(p->fmt.pix_mp.reserved, 0,
> + sizeof(p->fmt.pix_mp.reserved));
> + for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> + memset(p->fmt.pix_mp.plane_fmt[i].reserved, 0,
> + sizeof(p->fmt.pix_mp.plane_fmt[i].reserved));
> + return ret;
> case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
> case V4L2_BUF_TYPE_VBI_CAPTURE:
> @@ -1474,7 +1481,13 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
> p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> return ret;
> case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> - return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> + ret = ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> + memset(p->fmt.pix_mp.reserved, 0,
> + sizeof(p->fmt.pix_mp.reserved));
> + for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> + memset(p->fmt.pix_mp.plane_fmt[i].reserved, 0,
> + sizeof(p->fmt.pix_mp.plane_fmt[i].reserved));
> + return ret;
I wonder if we need this for G_FMT. The driver can just memset() the
whole struct itself and then just initialize the fields it cares
about, but actually in many cases the driver will just include an
instance of the pix_fmt(_mp) struct in its internal state (which has
the reserved fields already zeroed) and just copy it to the target
struct in the callback.
> case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
> return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
> case V4L2_BUF_TYPE_VBI_OUTPUT:
> @@ -1512,6 +1525,7 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
> struct v4l2_format *p = arg;
> struct video_device *vfd = video_devdata(file);
> int ret = check_fmt(file, p->type);
> + int i;
>
> if (ret)
> return ret;
> @@ -1536,7 +1550,13 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
> if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
> break;
> CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> - return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> + ret = ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> + memset(p->fmt.pix_mp.reserved, 0,
> + sizeof(p->fmt.pix_mp.reserved));
Note that we're already zeroing this field before calling driver's callback.
> + for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> + memset(p->fmt.pix_mp.plane_fmt[i].reserved, 0,
> + sizeof(p->fmt.pix_mp.plane_fmt[i].reserved));
Should we use the CLEAR_AFTER_FIELD() macro? Also, should we do before
calling the driver, as with pix_mp.reserved?
> + return ret;
> case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
> break;
> @@ -1564,7 +1584,13 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
> if (unlikely(!ops->vidioc_s_fmt
Re: is it possible to use single IOCTL to setup media pipeline?
On Thu, Nov 22, 2018 at 10:46 PM Sakari Ailus wrote: > > Hi Tomasz, > > On Thu, Nov 22, 2018 at 04:06:36PM +0900, Tomasz Figa wrote: > > Hi Ning, > > > > On Thu, Nov 22, 2018 at 11:52 AM Zhang, Ning A > > wrote: > > > > > > Hello everyone > > > > > > when we need to setup media pipeline, eg, for camera capture, media-ctl > > > needs to be called multiple time to setup media link and subdev > > > formats, or similar code in a single application. this will use > > > multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY". > > > > > > to setup media pipeline in userspace requires to fully understanding > > > the topology of the media stack. but the fact is only media driver > > > developer could know how to setup media pipeline. each time driver > > > updates, this would break userspace application if application > > > engineers don't know this change. > > > > That's obviously a bug in the driver. Kernel interfaces must not > > change in a way that are not compatible with the userspace. > > Alternatively, this could result from fixing a bug in a driver. Or adding > features that were not previously supported. > > In this case there are often no perfect solutions to address all the issues > at hand. An upstream kernel driver must maintain compatibility even in such cases, which isn't the most convenient thing for driver developers, but that's something we have to live with if we want to have users. > > > > > > In this case, if a IOCTL is designed > > > to setup media pipeline, no need to update applications, after driver > > > is updated. > > > > > > this will not only benefit for design a single IOCTL, this also helps > > > to hide the detail of media pipeline, by load a binary blob which holds > > > information about how to setup pipeline, or hide it in bootloader/ACPI > > > tables/device tree, etc. > > > > Media pipeline configuration is specific to the use case. If you > > hardcode it in the driver or bootloader, the user will not be able to > > use any other use case than the hardcoded blob, which is unacceptable > > for Linux drivers. > > > > Instead, it sounds like your userspace should be designed in a way > > that the media topology configuration is loaded from a configuration > > file that you could either get from your kernel driver developer or > > just maintain yourself based on any changes the media developers do. > > Of course that's unrelated to the backwards compatibility issue, which > > should not happen normally. The configuration file would be helpful > > for handling future extensions and new hardware platforms. > > > > > > > > another benefit is save time for setup media pipeline, if there is a > > > PKI like "time for open camera". as my test, this will saves hundreds > > > of milliseconds. > > > > For this problem, the proper solution would be to create an ioctl that > > can aggregate setting multiple parts of the topology in one go. For > > example, V4L2 has VIDIOC_S_CTRL for setting a control, but there is > > also VIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one > > call. Something like VIDIOC_S_EXT_CTRLS for configuring the media > > topology would solve the performance problem. > > There have been ideas of moving all IOCTL handling to the media device, in > a way that would allow issuing them in the same fashion than controls. This > was discussed in last year's media summit actually. I think this could be > done once the request API is otherwise working for e.g. camera devices. I > don't expect this to materialise in near (or even medium) term though. I'm aware of those talks and we actually took this into consideration when developing the Request API. Hopefully we can speed up the work on this to make it happen medium term at latest, if not short term, as it's going to be quite important for the complex camera subsystems, especially in the days of all the counter measures against the speculative execution vulnerabilities. Best regards, Tomasz
Re: [GIT PULL FOR v4.21] Add Rockchip VPU JPEG encoder
On Thu, Nov 22, 2018 at 7:29 PM Hans Verkuil wrote: > > Hi Ezeguiel, > > Just saw Tomasz' in-depth review and decided to drop this pull request. > > He found a few too many issues and I prefer those are addressed first. > > Sorry, still more work for you, on to v11! I'm really sorry for the late review. Hopefully I can do better from now on. (Still have quite a big pile of backlog, though...) Best regards, Tomasz > > Regards, > > Hans > > On 11/22/2018 10:39 AM, Hans Verkuil wrote: > > The following changes since commit 5200ab6a32d6055428896a49ec9e3b1652c1a100: > > > > media: vidioc_cropcap -> vidioc_g_pixelaspect (2018-11-20 13:57:21 -0500) > > > > are available in the Git repository at: > > > > git://linuxtv.org/hverkuil/media_tree.git tags/br-jpeg > > > > for you to fetch changes up to cbf7592cb57ec9986c4d1becfd80b85486fd318a: > > > > media: add Rockchip VPU JPEG encoder driver (2018-11-22 10:12:29 +0100) > > > > > > Tag branch > > > > > > Ezequiel Garcia (1): > > media: add Rockchip VPU JPEG encoder driver > > > > MAINTAINERS | 7 + > > drivers/staging/media/Kconfig | 2 + > > drivers/staging/media/Makefile | 1 + > > drivers/staging/media/rockchip/vpu/Kconfig | 13 + > > drivers/staging/media/rockchip/vpu/Makefile | 10 + > > drivers/staging/media/rockchip/vpu/TODO | 6 + > > drivers/staging/media/rockchip/vpu/rk3288_vpu_hw.c | 118 > > drivers/staging/media/rockchip/vpu/rk3288_vpu_hw_jpeg_enc.c | 133 > > drivers/staging/media/rockchip/vpu/rk3288_vpu_regs.h| 442 > > +++ > > drivers/staging/media/rockchip/vpu/rk3399_vpu_hw.c | 118 > > drivers/staging/media/rockchip/vpu/rk3399_vpu_hw_jpeg_enc.c | 160 > > ++ > > drivers/staging/media/rockchip/vpu/rk3399_vpu_regs.h| 600 > > > > drivers/staging/media/rockchip/vpu/rockchip_vpu.h | 237 > > +++ > > drivers/staging/media/rockchip/vpu/rockchip_vpu_common.h| 29 ++ > > drivers/staging/media/rockchip/vpu/rockchip_vpu_drv.c | 535 > > + > > drivers/staging/media/rockchip/vpu/rockchip_vpu_enc.c | 702 > > +++ > > drivers/staging/media/rockchip/vpu/rockchip_vpu_hw.h| 58 > > drivers/staging/media/rockchip/vpu/rockchip_vpu_jpeg.c | 290 > > ++ > > drivers/staging/media/rockchip/vpu/rockchip_vpu_jpeg.h | 14 + > > 19 files changed, 3475 insertions(+) > > create mode 100644 drivers/staging/media/rockchip/vpu/Kconfig > > create mode 100644 drivers/staging/media/rockchip/vpu/Makefile > > create mode 100644 drivers/staging/media/rockchip/vpu/TODO > > create mode 100644 drivers/staging/media/rockchip/vpu/rk3288_vpu_hw.c > > create mode 100644 > > drivers/staging/media/rockchip/vpu/rk3288_vpu_hw_jpeg_enc.c > > create mode 100644 drivers/staging/media/rockchip/vpu/rk3288_vpu_regs.h > > create mode 100644 drivers/staging/media/rockchip/vpu/rk3399_vpu_hw.c > > create mode 100644 > > drivers/staging/media/rockchip/vpu/rk3399_vpu_hw_jpeg_enc.c > > create mode 100644 drivers/staging/media/rockchip/vpu/rk3399_vpu_regs.h > > create mode 100644 drivers/staging/media/rockchip/vpu/rockchip_vpu.h > > create mode 100644 drivers/staging/media/rockchip/vpu/rockchip_vpu_common.h > > create mode 100644 drivers/staging/media/rockchip/vpu/rockchip_vpu_drv.c > > create mode 100644 drivers/staging/media/rockchip/vpu/rockchip_vpu_enc.c > > create mode 100644 drivers/staging/media/rockchip/vpu/rockchip_vpu_hw.h > > create mode 100644 drivers/staging/media/rockchip/vpu/rockchip_vpu_jpeg.c > > create mode 100644 drivers/staging/media/rockchip/vpu/rockchip_vpu_jpeg.h > > >
Re: [PATCH v10 3/4] arm64: dts: rockchip: add VPU device node for RK3399
On Thu, Nov 22, 2018 at 4:58 AM Ezequiel Garcia wrote:
>
> Add the Video Processing Unit node for the RK3399 SoC.
>
> Also, fix the VPU IOMMU node, which was disabled and lacking
> its power domain property.
>
> Signed-off-by: Ezequiel Garcia
> ---
> arch/arm64/boot/dts/rockchip/rk3399.dtsi | 14 +-
> 1 file changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/boot/dts/rockchip/rk3399.dtsi
> b/arch/arm64/boot/dts/rockchip/rk3399.dtsi
> index 99e7f65c1779..040d3080565f 100644
> --- a/arch/arm64/boot/dts/rockchip/rk3399.dtsi
> +++ b/arch/arm64/boot/dts/rockchip/rk3399.dtsi
> @@ -1226,6 +1226,18 @@
> status = "disabled";
> };
>
> + vpu: video-codec@ff65 {
> + compatible = "rockchip,rk3399-vpu";
> + reg = <0x0 0xff65 0x0 0x800>;
> + interrupts = ,
> +;
> + interrupt-names = "vepu", "vdpu";
> + clocks = < ACLK_VCODEC>, < HCLK_VCODEC>;
> + clock-names = "aclk", "hclk";
> + power-domains = < RK3399_PD_VCODEC>;
> + iommus = <_mmu>;
> + };
> +
> vpu_mmu: iommu@ff650800 {
> compatible = "rockchip,iommu";
> reg = <0x0 0xff650800 0x0 0x40>;
> @@ -1233,8 +1245,8 @@
> interrupt-names = "vpu_mmu";
> clocks = < ACLK_VCODEC>, < HCLK_VCODEC>;
> clock-names = "aclk", "iface";
> + power-domains = < RK3399_PD_VCODEC>;
> #iommu-cells = <0>;
> - status = "disabled";
> };
>
> vdec_mmu: iommu@ff660480 {
Reviewed-by: Tomasz Figa
Best regards,
Tomasz
Re: [PATCH v10 2/4] ARM: dts: rockchip: add VPU device node for RK3288
Hi Ezequiel,
On Thu, Nov 22, 2018 at 4:17 AM Ezequiel Garcia wrote:
>
> Add the Video Processing Unit node for RK3288 SoC.
>
> Fix the VPU IOMMU node, which was disabled and lacking
> its power domain property.
>
> Signed-off-by: Ezequiel Garcia
> ---
> arch/arm/boot/dts/rk3288.dtsi | 14 +-
> 1 file changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm/boot/dts/rk3288.dtsi b/arch/arm/boot/dts/rk3288.dtsi
> index 0840ffb3205c..40d203cdca09 100644
> --- a/arch/arm/boot/dts/rk3288.dtsi
> +++ b/arch/arm/boot/dts/rk3288.dtsi
> @@ -1223,6 +1223,18 @@
> };
> };
>
> + vpu: video-codec@ff9a {
> + compatible = "rockchip,rk3288-vpu";
> + reg = <0x0 0xff9a 0x0 0x800>;
> + interrupts = ,
> +;
> + interrupt-names = "vepu", "vdpu";
> + clocks = < ACLK_VCODEC>, < HCLK_VCODEC>;
> + clock-names = "aclk", "hclk";
> + power-domains = < RK3288_PD_VIDEO>;
> + iommus = <_mmu>;
> + };
> +
> vpu_mmu: iommu@ff9a0800 {
> compatible = "rockchip,iommu";
> reg = <0x0 0xff9a0800 0x0 0x100>;
> @@ -1230,8 +1242,8 @@
> interrupt-names = "vpu_mmu";
> clocks = < ACLK_VCODEC>, < HCLK_VCODEC>;
> clock-names = "aclk", "iface";
> + power-domains = < RK3288_PD_VIDEO>;
> #iommu-cells = <0>;
> - status = "disabled";
> };
>
> hevc_mmu: iommu@ff9c0440 {
Reviewed-by: Tomasz Figa
Best regards,
Tomasz
Re: [PATCH v10 4/4] media: add Rockchip VPU JPEG encoder driver
gt; + rockchip_vpu_jpeg_get_qtable(_ctx, 1));
> +
> + /* Make sure that all registers are written at this point. */
> + wmb();
> +
Perhaps the next vepu_write_relaxed() should be turned into
vepu_write() instead? writel() basically starts with a wmb() on
arm/arm64.
> + reg = VEPU_REG_AXI_CTRL_OUTPUT_SWAP16
> + | VEPU_REG_AXI_CTRL_INPUT_SWAP16
> + | VEPU_REG_AXI_CTRL_BURST_LEN(16)
> + | VEPU_REG_AXI_CTRL_OUTPUT_SWAP32
> + | VEPU_REG_AXI_CTRL_INPUT_SWAP32
> + | VEPU_REG_AXI_CTRL_OUTPUT_SWAP8
> + | VEPU_REG_AXI_CTRL_INPUT_SWAP8;
> + vepu_write_relaxed(vpu, reg, VEPU_REG_AXI_CTRL);
> +
> + reg = VEPU_REG_ENC_CTRL_WIDTH(MB_WIDTH(ctx->src_fmt.width))
> + | VEPU_REG_ENC_CTRL_HEIGHT(MB_HEIGHT(ctx->src_fmt.height))
> + | VEPU_REG_ENC_CTRL_ENC_MODE_JPEG
> + | VEPU_REG_ENC_PIC_INTRA
> + | VEPU_REG_ENC_CTRL_EN_BIT;
> + /* Kick the watchdog and start encoding */
> + schedule_delayed_work(>watchdog_work, msecs_to_jiffies(2000));
> + vepu_write(vpu, reg, VEPU_REG_ENC_CTRL);
> +}
> diff --git a/drivers/staging/media/rockchip/vpu/rk3288_vpu_regs.h
> b/drivers/staging/media/rockchip/vpu/rk3288_vpu_regs.h
> new file mode 100644
> index ..b5a464844dce
> --- /dev/null
> +++ b/drivers/staging/media/rockchip/vpu/rk3288_vpu_regs.h
> @@ -0,0 +1,442 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Rockchip VPU codec driver
> + *
> + * Copyright (C) 2018 Google, Inc.
Should be:
Copyright 2018 Google LLC.
> + * Tomasz Figa
> + */
> +
[snip]
> diff --git a/drivers/staging/media/rockchip/vpu/rk3399_vpu_hw.c
> b/drivers/staging/media/rockchip/vpu/rk3399_vpu_hw.c
> new file mode 100644
> index ..f9338745afe9
> --- /dev/null
> +++ b/drivers/staging/media/rockchip/vpu/rk3399_vpu_hw.c
> @@ -0,0 +1,118 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Rockchip VPU codec driver
> + *
> + * Copyright (C) 2018 Rockchip Electronics Co., Ltd.
> + * Jeffy Chen
> + */
> +
> +#include
> +
> +#include "rockchip_vpu.h"
> +#include "rockchip_vpu_jpeg.h"
> +#include "rk3399_vpu_regs.h"
> +
> +#define RK3399_ACLK_MAX_FREQ (400 * 1000 * 1000)
> +
> +/*
> + * Supported formats.
> + */
> +
> +static const struct rockchip_vpu_fmt rk3399_vpu_enc_fmts[] = {
> + {
> + .fourcc = V4L2_PIX_FMT_YUV420M,
> + .codec_mode = RK_VPU_MODE_NONE,
> + .enc_fmt = RK3288_VPU_ENC_FMT_YUV420P,
> + },
> + {
> + .fourcc = V4L2_PIX_FMT_NV12M,
> + .codec_mode = RK_VPU_MODE_NONE,
> + .enc_fmt = RK3288_VPU_ENC_FMT_YUV420SP,
> + },
> + {
> + .fourcc = V4L2_PIX_FMT_YUYV,
> + .codec_mode = RK_VPU_MODE_NONE,
> + .enc_fmt = RK3288_VPU_ENC_FMT_YUYV422,
> + },
> + {
> + .fourcc = V4L2_PIX_FMT_UYVY,
> + .codec_mode = RK_VPU_MODE_NONE,
> + .enc_fmt = RK3288_VPU_ENC_FMT_UYVY422,
> + },
> + {
> + .fourcc = V4L2_PIX_FMT_JPEG,
> + .codec_mode = RK_VPU_MODE_JPEG_ENC,
> + .max_depth = 2,
> + .header_size = JPEG_HEADER_SIZE,
> + .frmsize = {
> + .min_width = 96,
> + .max_width = 8192,
> + .step_width = MB_DIM,
> + .min_height = 32,
> + .max_height = 8192,
> + .step_height = MB_DIM,
> + },
> + },
> +};
> +
> +static irqreturn_t rk3399_vepu_irq(int irq, void *dev_id)
> +{
> + struct rockchip_vpu_dev *vpu = dev_id;
> + enum vb2_buffer_state state;
> + u32 status, bytesused;
> +
> + status = vepu_read(vpu, VEPU_REG_INTERRUPT);
> + bytesused = vepu_read(vpu, VEPU_REG_STR_BUF_LIMIT) / 8;
> + state = (status & VEPU_REG_INTERRUPT_FRAME_READY) ?
> + VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
> +
> + vepu_write(vpu, 0, VEPU_REG_INTERRUPT);
> + vepu_write(vpu, 0, VEPU_REG_AXI_CTRL);
> +
> + rockchip_vpu_irq_done(vpu, bytesused, state);
> +
> + return IRQ_HANDLED;
> +}
> +
> +static int rk3399_vpu_hw_init(struct rockchip_vpu_dev *vpu)
> +{
> + /* Bump ACLK to max. possible freq. to improve performance. */
> + clk_set_rate(vpu->clocks[0].clk, RK3399_ACLK_MAX_FREQ);
> + return 0;
> +}
> +
> +static void rk3399_v
Re: is it possible to use single IOCTL to setup media pipeline?
Hi Ning, On Thu, Nov 22, 2018 at 11:52 AM Zhang, Ning A wrote: > > Hello everyone > > when we need to setup media pipeline, eg, for camera capture, media-ctl > needs to be called multiple time to setup media link and subdev > formats, or similar code in a single application. this will use > multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY". > > to setup media pipeline in userspace requires to fully understanding > the topology of the media stack. but the fact is only media driver > developer could know how to setup media pipeline. each time driver > updates, this would break userspace application if application > engineers don't know this change. That's obviously a bug in the driver. Kernel interfaces must not change in a way that are not compatible with the userspace. > In this case, if a IOCTL is designed > to setup media pipeline, no need to update applications, after driver > is updated. > > this will not only benefit for design a single IOCTL, this also helps > to hide the detail of media pipeline, by load a binary blob which holds > information about how to setup pipeline, or hide it in bootloader/ACPI > tables/device tree, etc. Media pipeline configuration is specific to the use case. If you hardcode it in the driver or bootloader, the user will not be able to use any other use case than the hardcoded blob, which is unacceptable for Linux drivers. Instead, it sounds like your userspace should be designed in a way that the media topology configuration is loaded from a configuration file that you could either get from your kernel driver developer or just maintain yourself based on any changes the media developers do. Of course that's unrelated to the backwards compatibility issue, which should not happen normally. The configuration file would be helpful for handling future extensions and new hardware platforms. > > another benefit is save time for setup media pipeline, if there is a > PKI like "time for open camera". as my test, this will saves hundreds > of milliseconds. For this problem, the proper solution would be to create an ioctl that can aggregate setting multiple parts of the topology in one go. For example, V4L2 has VIDIOC_S_CTRL for setting a control, but there is also VIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one call. Something like VIDIOC_S_EXT_CTRLS for configuring the media topology would solve the performance problem. Best regards, Tomasz
Re: [PATCH 1/2] vb2: add waiting_in_dqbuf flag
On Mon, Nov 19, 2018 at 6:54 PM Hans Verkuil wrote: > > On 11/19/2018 09:44 AM, Hans Verkuil wrote: > > On 11/19/2018 06:27 AM, Tomasz Figa wrote: > >> On Fri, Nov 16, 2018 at 6:45 PM Hans Verkuil wrote: > >>> > >>> On 11/16/2018 09:43 AM, Tomasz Figa wrote: > >>>> Hi Hans, > >>>> > >>>> On Wed, Nov 14, 2018 at 12:08 AM Hans Verkuil wrote: > >>>>> > >>>>> Calling VIDIOC_DQBUF can release the core serialization lock pointed to > >>>>> by vb2_queue->lock if it has to wait for a new buffer to arrive. > >>>>> > >>>>> However, if userspace dup()ped the video device filehandle, then it is > >>>>> possible to read or call DQBUF from two filehandles at the same time. > >>>>> > >>>> > >>>> What side effects would reading have? > >>>> > >>>> As for another DQBUF in parallel, perhaps that's actually a valid > >>>> operation that should be handled? I can imagine that one could want to > >>>> have multiple threads dequeuing buffers as they become available, so > >>>> that no dispatch thread is needed. > >>> > >>> I think parallel DQBUFs can be done, but it has never been tested, nor > >>> has vb2 been designed with that in mind. I also don't see the use-case > >>> since if you have, say, two DQBUFs in parallel, then it will be random > >>> which DQBUF gets which frame. > >>> > >> > >> Any post processing that operates only on single frame data would be > >> able to benefit from multiple threads, with results ordered after the > >> processing, based on timestamps. > >> > >> Still, if that's not something we've ever claimed as supported and > >> couldn't work correctly with current code, it sounds fair to > >> completely forbid it for now. > >> > >>> If we ever see a need for this, then that needs to be designed and tested > >>> properly. > >>> > >>>> > >>>>> It is also possible to call REQBUFS from one filehandle while the other > >>>>> is waiting for a buffer. This will remove all the buffers and reallocate > >>>>> new ones. Removing all the buffers isn't the problem here (that's > >>>>> already > >>>>> handled correctly by DQBUF), but the reallocating part is: DQBUF isn't > >>>>> aware that the buffers have changed. > >>>>> > >>>>> This is fixed by setting a flag whenever the lock is released while > >>>>> waiting > >>>>> for a buffer to arrive. And checking the flag where needed so we can > >>>>> return > >>>>> -EBUSY. > >>>> > >>>> Maybe it would make more sense to actually handle those side effects? > >>>> Such waiting DQBUF would then just fail in the same way as if it > >>>> couldn't get a buffer (or if it's blocking, just retry until a correct > >>>> buffer becomes available?). > >>> > >>> That sounds like a good idea, but it isn't. > >>> > >>> With this patch you can't call REQBUFS to reallocate buffers while a > >>> thread > >>> is waiting for a buffer. > >>> > >>> If I allow this, then the problem moves to when the thread that called > >>> REQBUFS > >>> calls DQBUF next. Since we don't allow multiple DQBUFs this second DQBUF > >>> will > >>> mysteriously fail. If we DO allow multiple DQBUFs, then how does REQBUFS > >>> ensure > >>> that only the DQBUF that relied on the old buffers is stopped? > >>> > >>> It sounds nice, but the more I think about it, the more problems I see > >>> with it. > >>> > >>> I think it is perfectly reasonable to expect REQBUFS to return EBUSY if > >>> some > >>> thread is still waiting for a buffer. > >>> > >>> That said, I think one test is missing in vb2_core_create_bufs: there too > >>> it > >>> should check waiting_in_dqbuf if q->num_buffers == 0: it is possible to do > >>> REQBUFS(0) followed by CREATE_BUFS() while another thread is waiting for a > >>> buffer. CREATE_BUFS acts like REQBUFS(count >= 1) in that case. > >>> > >>> Admittedly, that would require some extremely unfortunate scheduling, but > >>> it is easy enough to check this. > >> > >> I thought a bit more about this and I agree with you. We should keep > >> things as simple as possible. > >> > >> Another thing that came to my mind is that the problematic scenario > >> described in the commit message can happen only if queue->lock == > >> dev->lock. I wonder how likely it would be to mandate queue->lock != > >> dev->lock? > > > > My plan is to switch vivid to that model. Expect patches for that today. > > One thing I noticed is that there is an issue with calling queue_setup > > in that case. I have a separate patch for that, so just read it when I > > post it. > > Note that this specific scenario can happen regardless of whether > queue->lock == dev->lock or not. Ah, good point. Somehow I assumed that only QBUF/DQBUF would use queue->lock, while anything else would use dev->lock, but that's not the case. Then I can't find any simpler and/or more general fix for now, so I'm okay with this. Another note, don't we need similar error in case of REQBUFS(0), while DQBUF() is waiting? Current patch seems to add one only for count != 0. Best regards, Tomasz
Re: [PATCH 1/2] vb2: add waiting_in_dqbuf flag
On Fri, Nov 16, 2018 at 6:45 PM Hans Verkuil wrote: > > On 11/16/2018 09:43 AM, Tomasz Figa wrote: > > Hi Hans, > > > > On Wed, Nov 14, 2018 at 12:08 AM Hans Verkuil wrote: > >> > >> Calling VIDIOC_DQBUF can release the core serialization lock pointed to > >> by vb2_queue->lock if it has to wait for a new buffer to arrive. > >> > >> However, if userspace dup()ped the video device filehandle, then it is > >> possible to read or call DQBUF from two filehandles at the same time. > >> > > > > What side effects would reading have? > > > > As for another DQBUF in parallel, perhaps that's actually a valid > > operation that should be handled? I can imagine that one could want to > > have multiple threads dequeuing buffers as they become available, so > > that no dispatch thread is needed. > > I think parallel DQBUFs can be done, but it has never been tested, nor > has vb2 been designed with that in mind. I also don't see the use-case > since if you have, say, two DQBUFs in parallel, then it will be random > which DQBUF gets which frame. > Any post processing that operates only on single frame data would be able to benefit from multiple threads, with results ordered after the processing, based on timestamps. Still, if that's not something we've ever claimed as supported and couldn't work correctly with current code, it sounds fair to completely forbid it for now. > If we ever see a need for this, then that needs to be designed and tested > properly. > > > > >> It is also possible to call REQBUFS from one filehandle while the other > >> is waiting for a buffer. This will remove all the buffers and reallocate > >> new ones. Removing all the buffers isn't the problem here (that's already > >> handled correctly by DQBUF), but the reallocating part is: DQBUF isn't > >> aware that the buffers have changed. > >> > >> This is fixed by setting a flag whenever the lock is released while waiting > >> for a buffer to arrive. And checking the flag where needed so we can return > >> -EBUSY. > > > > Maybe it would make more sense to actually handle those side effects? > > Such waiting DQBUF would then just fail in the same way as if it > > couldn't get a buffer (or if it's blocking, just retry until a correct > > buffer becomes available?). > > That sounds like a good idea, but it isn't. > > With this patch you can't call REQBUFS to reallocate buffers while a thread > is waiting for a buffer. > > If I allow this, then the problem moves to when the thread that called REQBUFS > calls DQBUF next. Since we don't allow multiple DQBUFs this second DQBUF will > mysteriously fail. If we DO allow multiple DQBUFs, then how does REQBUFS > ensure > that only the DQBUF that relied on the old buffers is stopped? > > It sounds nice, but the more I think about it, the more problems I see with > it. > > I think it is perfectly reasonable to expect REQBUFS to return EBUSY if some > thread is still waiting for a buffer. > > That said, I think one test is missing in vb2_core_create_bufs: there too it > should check waiting_in_dqbuf if q->num_buffers == 0: it is possible to do > REQBUFS(0) followed by CREATE_BUFS() while another thread is waiting for a > buffer. CREATE_BUFS acts like REQBUFS(count >= 1) in that case. > > Admittedly, that would require some extremely unfortunate scheduling, but > it is easy enough to check this. I thought a bit more about this and I agree with you. We should keep things as simple as possible. Another thing that came to my mind is that the problematic scenario described in the commit message can happen only if queue->lock == dev->lock. I wonder how likely it would be to mandate queue->lock != dev->lock? Best regards, Tomasz
Re: [ANN] Edinburgh Media Summit 2018 meeting report
On Sun, Nov 18, 2018 at 7:45 AM Sakari Ailus wrote: > > Hello everyone, > > > Here's the report on the Media Summit held on 25th October in Edinburgh. > The report is followed by the stateless codec discussion two days earlier. > > Note: this is bcc'd to the meeting attendees plus a few others. I didn't > use cc as the list servers tend to reject messages with too many > recipients in cc / to headers. > > Most presenters used slides some of which are already available here > (expect more in the near future): > > https://www.linuxtv.org/downloads/presentations/media_summit_2018/> > > The original announcement for the meeting is here: > > https://www.spinics.net/lists/linux-media/msg141095.html> > > The raw notes can be found here: > > http://www.retiisi.org.uk/~sailus/v4l2/notes/osseu18-media.html> Thanks Sakari for editing the notes. Let me share my thoughts inline. [snip] > Automated testing - Ezequiel Garcia > --- > > Ideal Continuous Integration process consists of the following steps: > > 1. patch submission > 2. review and approval > 3. merge > > The core question is "what level of quality standards do we want to > enforce". The maintenance process should be modelled around this question, > and not the other way around. Automated testing can be a part of enforcing > the quality standards. > > There are three steps: > > 1. Define the quality standard > 2. Define how to quantify quality in respect to the standard > 3. Define how to enforce the standards > > On the tooling side, an uAPI test tool exists. It's called v4l2-compliance, > and new drivers are required to pass the v4l2-compliance test. > It has quite a few favourable properties: > > - Complete in terms of the uAPI coverage > - Quick and easy to run > - Nice output format for humans & scripts > > There are some issues as well: > > - No codec support (stateful or stateless) > - No SDR or touch support > - Frequently updated (distribution shipped v4l2-compliance useless) > - Only one contributor > > Ezequiel noted that some people think that v4l2-compliance is changing too > often but Hans responded that this is a necessity. The API gets amended > occasionally and the existing API gets new tests. Mauro proposed moving > v4l2-compliance to the kernel source tree but Hans preferred keeping it > separate. That way it's easier to develop it. > > To address the problem of only a single contributor, it was suggested that > people implementing new APIs would need to provide the tests for > v4l2-compliance as well. To achieve this, the v4l2-compliance codebase > needs some cleanup to make it easier to contribute. The codebase is larger > and there is no documentation. > > V4l2-compliance also covers MC, V4L2 and V4L2 sub-device uAPIs. > > DVB will require its own test tooling; it is not covered by > v4l2-compliance. In order to facilitate automated testing, a virtual DVB > driver would be useful as well. The task was added to the list of projects > needing volunteers: > > https://linuxtv.org/wiki/index.php/Media_Open_Source_Projects:_Looking_for_Volunteers> > > There are some other test tools that could cover V4L2 but at the moment it > seems somewhat far-fetched any of them would be used to test V4L2 in the > near future: > > - kselftest > - kunit > - gst-validate > - ktf (https://github.com/oracle/ktf, > http://heim.ifi.uio.no/~knuto/ktf/) > > KernelCI is a test automation system that supports automated compile and > boot testing. As a newly added feature, additional tests may be > implemented. This is what Collabora has implemented, effectively the > current demo system runs v4l2-compliance on virtual drivers in a virtual > machines (LAVA slaves). > > A sample of the current test report is here: > > https://www.mail-archive.com/linux-media@vger.kernel.org/msg135787.html> > > The established way to run KernelCI tests is off the head of the branches of > the stable and development kernel trees, including linux-next. This is not > useful as such to support automated testing of patches for the media tree: > the patches need to be tested before they are merged, not after merging. > > In the discusion that followed among a slightly smaller group of people, it > was suggested that tests could be run from select developer kernel trees, > from any branch. If a developer needs long-term storage, (s)he could have > another tree which would not be subject automated test builds. > Alternatively, the branch name could be used as a basis for triggering > an automated build, but this could end up being too restrictive. > > Merging the next rc1 by the maintainer would be no special case: the branch > would be tested in similar way than the developer branches containing > patches, and tests should need to pass before pushing the content to the > media tree master branch. > > Ezequiel wished that people would reply to his e-mail to express their > wishes
Re: [PATCHv2 0/9] vb2/cedrus: add tag support
Hi Hans, On Wed, Nov 14, 2018 at 10:47 PM Hans Verkuil wrote: > > From: Hans Verkuil > > As was discussed here (among other places): > > https://lkml.org/lkml/2018/10/19/440 > > using capture queue buffer indices to refer to reference frames is > not a good idea. A better idea is to use a 'tag' where the > application can assign a u64 tag to an output buffer, which is then > copied to the capture buffer(s) derived from the output buffer. > Thanks for the patches. Please see my comments below. > A u64 is chosen since this allows userspace to also use pointers to > internal structures as 'tag'. > I think this is not true anymore in this version. > The first three patches add core tag support, the next patch document > the tag support, then a new helper function is added to v4l2-mem2mem.c > to easily copy data from a source to a destination buffer that drivers > can use. > > Next a new supports_tags vb2_queue flag is added to indicate that > the driver supports tags. Ideally this should not be necessary, but > that would require that all m2m drivers are converted to using the > new helper function introduced in the previous patch. That takes more > time then I have now since we want to get this in for 4.20. > > Finally the vim2m, vicodec and cedrus drivers are converted to support > tags. > > I also removed the 'pad' fields from the mpeg2 control structs (it > should never been added in the first place) and aligned the structs > to a u32 boundary (u64 for the tag values). u32 in this version > > Note that this might change further (Paul suggested using bitfields). > > Also note that the cedrus code doesn't set the sequence counter, that's > something that should still be added before this driver can be moved > out of staging. > > Note: if no buffer is found for a certain tag, then the dma address > is just set to 0. That happened before as well with invalid buffer > indices. This should be checked in the driver! Note that DMA address 0 may as well be a valid address. Should we have another way of signaling that? > > The previous RFC series was tested successfully with the cedrus driver. > > Regards, > > Hans > > Changes since v1: > > - changed to a u32 tag. Using a 64 bit tag was overly complicated due > to the bad layout of the v4l2_buffer struct, and there is no real > need for it by applications. I hope these won't become famous last words. :) For Chromium it should be okay indeed. Since we've been thinking about a redesign of the buffer struct, perhaps we can live with u32 for now and we can design the new struct to handle u64 nicely? Best regards, Tomasz
Re: [PATCH 2/2] vb2: don't allow queueing buffers when canceling queue
On Fri, Nov 16, 2018 at 5:42 PM Hans Verkuil wrote:
>
> On 11/16/2018 09:34 AM, Tomasz Figa wrote:
> > Hi Hans,
> >
> > On Wed, Nov 14, 2018 at 12:08 AM Hans Verkuil wrote:
> >>
> >> Calling the stop_streaming op can release the core serialization lock
> >> pointed to by vb2_queue->lock if it has to wait for buffers to finish.
> >> An example of that behavior is the vivid driver.
> >
> > Why would vb2_queue->lock have to be released to wait for buffer to
> > finish? The drivers I worked with never had to do anything like that.
>
> Actually, they all do. It's done through the wait_prepare/finish callbacks
> and by setting those to vb2_ops_wait_prepare/finish.
>
> If you don't, then while one thread is waiting for a buffer to arrive,
> another thread cannot queue a new buffer since it will be serialized by
> queue->lock.
>
> v4l2-compliance even tests for this.
Why would you need the userspace to queue more buffers when you're
stopping the queue?
>
> >
> >>
> >> However, if userspace dup()ped the video device filehandle, then it is
> >> possible to stop streaming on one filehandle and call read/write or
> >> VIDIOC_QBUF from the other.
> >
> > How about other ioctls? I can imagine at least STREAMON could be
> > called at the same time too, but not sure if it would have any side
> > effects.
>
> STREAMON would return an error since q->streaming is still set while
> in the stop_streaming callback.
>
> So that combination is safe.
>
Okay, thanks. I'm still slightly worried that this approach with a
flag makes it possible to miss some non-trivial cases, though...
> Regards,
>
> Hans
>
> >
> > Best regards,
> > Tomasz
> >
> >>
> >> This is fixed by setting a flag whenever stop_streaming is called and
> >> checking the flag where needed so we can return -EBUSY.
> >>
> >> Signed-off-by: Hans Verkuil
> >> Reported-by: syzbot+736c3aae4af7b50d9...@syzkaller.appspotmail.com
> >> ---
> >> drivers/media/common/videobuf2/videobuf2-core.c | 14 +-
> >> include/media/videobuf2-core.h | 1 +
> >> 2 files changed, 14 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> >> b/drivers/media/common/videobuf2/videobuf2-core.c
> >> index 138223af701f..560577321fe7 100644
> >> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> >> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> >> @@ -1503,6 +1503,10 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int
> >> index, void *pb,
> >> dprintk(1, "fatal error occurred on queue\n");
> >> return -EIO;
> >> }
> >> + if (q->in_stop_streaming) {
> >> + dprintk(1, "stop_streaming is called\n");
> >> + return -EBUSY;
> >> + }
> >>
> >> vb = q->bufs[index];
> >>
> >> @@ -1834,8 +1838,11 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> >> * Tell driver to stop all transactions and release all queued
> >> * buffers.
> >> */
> >> - if (q->start_streaming_called)
> >> + if (q->start_streaming_called) {
> >> + q->in_stop_streaming = 1;
> >> call_void_qop(q, stop_streaming, q);
> >> + q->in_stop_streaming = 0;
> >> + }
> >>
> >> /*
> >> * If you see this warning, then the driver isn't cleaning up
> >> properly
> >> @@ -2565,6 +2572,11 @@ static size_t __vb2_perform_fileio(struct vb2_queue
> >> *q, char __user *data, size_
> >> return -EBUSY;
> >> }
> >>
> >> + if (q->in_stop_streaming) {
> >> + dprintk(3, "stop_streaming is called\n");
> >> + return -EBUSY;
> >> + }
> >> +
> >> /*
> >> * Initialize emulator on first call.
> >> */
> >> diff --git a/include/media/videobuf2-core.h
> >> b/include/media/videobuf2-core.h
> >> index 613f22910174..5a3d3ada5940 100644
> >> --- a/include/media/videobuf2-core.h
> >> +++ b/include/media/videobuf2-core.h
> >> @@ -585,6 +585,7 @@ struct vb2_queue {
> >> unsigned interror:1;
> >> unsigned intwaiting_for_buffers:1;
> >> unsigned intwaiting_in_dqbuf:1;
> >> + unsigned intin_stop_streaming:1;
> >> unsigned intis_multiplanar:1;
> >> unsigned intis_output:1;
> >> unsigned intcopy_timestamp:1;
> >> --
> >> 2.19.1
> >>
>
Re: [PATCH 1/2] vb2: add waiting_in_dqbuf flag
Hi Hans,
On Wed, Nov 14, 2018 at 12:08 AM Hans Verkuil wrote:
>
> Calling VIDIOC_DQBUF can release the core serialization lock pointed to
> by vb2_queue->lock if it has to wait for a new buffer to arrive.
>
> However, if userspace dup()ped the video device filehandle, then it is
> possible to read or call DQBUF from two filehandles at the same time.
>
What side effects would reading have?
As for another DQBUF in parallel, perhaps that's actually a valid
operation that should be handled? I can imagine that one could want to
have multiple threads dequeuing buffers as they become available, so
that no dispatch thread is needed.
> It is also possible to call REQBUFS from one filehandle while the other
> is waiting for a buffer. This will remove all the buffers and reallocate
> new ones. Removing all the buffers isn't the problem here (that's already
> handled correctly by DQBUF), but the reallocating part is: DQBUF isn't
> aware that the buffers have changed.
>
> This is fixed by setting a flag whenever the lock is released while waiting
> for a buffer to arrive. And checking the flag where needed so we can return
> -EBUSY.
Maybe it would make more sense to actually handle those side effects?
Such waiting DQBUF would then just fail in the same way as if it
couldn't get a buffer (or if it's blocking, just retry until a correct
buffer becomes available?).
Best regards,
Tomasz
>
> Signed-off-by: Hans Verkuil
> ---
> .../media/common/videobuf2/videobuf2-core.c| 18 ++
> include/media/videobuf2-core.h | 1 +
> 2 files changed, 19 insertions(+)
>
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> b/drivers/media/common/videobuf2/videobuf2-core.c
> index 03954c13024c..138223af701f 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -672,6 +672,11 @@ int vb2_core_reqbufs(struct vb2_queue *q, enum
> vb2_memory memory,
> return -EBUSY;
> }
>
> + if (q->waiting_in_dqbuf && *count) {
> + dprintk(1, "another dup()ped fd is waiting for a buffer\n");
> + return -EBUSY;
> + }
> +
> if (*count == 0 || q->num_buffers != 0 ||
> (q->memory != VB2_MEMORY_UNKNOWN && q->memory != memory)) {
> /*
> @@ -1624,6 +1629,11 @@ static int __vb2_wait_for_done_vb(struct vb2_queue *q,
> int nonblocking)
> for (;;) {
> int ret;
>
> + if (q->waiting_in_dqbuf) {
> + dprintk(1, "another dup()ped fd is waiting for a
> buffer\n");
> + return -EBUSY;
> + }
> +
> if (!q->streaming) {
> dprintk(1, "streaming off, will not wait for
> buffers\n");
> return -EINVAL;
> @@ -1651,6 +1661,7 @@ static int __vb2_wait_for_done_vb(struct vb2_queue *q,
> int nonblocking)
> return -EAGAIN;
> }
>
> + q->waiting_in_dqbuf = 1;
> /*
> * We are streaming and blocking, wait for another buffer to
> * become ready or for streamoff. Driver's lock is released to
> @@ -1671,6 +1682,7 @@ static int __vb2_wait_for_done_vb(struct vb2_queue *q,
> int nonblocking)
> * the locks or return an error if one occurred.
> */
> call_void_qop(q, wait_finish, q);
> + q->waiting_in_dqbuf = 0;
> if (ret) {
> dprintk(1, "sleep was interrupted\n");
> return ret;
> @@ -2547,6 +2559,12 @@ static size_t __vb2_perform_fileio(struct vb2_queue
> *q, char __user *data, size_
> if (!data)
> return -EINVAL;
>
> + if (q->waiting_in_dqbuf) {
> + dprintk(3, "another dup()ped fd is %s\n",
> + read ? "reading" : "writing");
> + return -EBUSY;
> + }
> +
> /*
> * Initialize emulator on first call.
> */
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index e86981d615ae..613f22910174 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -584,6 +584,7 @@ struct vb2_queue {
> unsigned intstart_streaming_called:1;
> unsigned interror:1;
> unsigned intwaiting_for_buffers:1;
> + unsigned intwaiting_in_dqbuf:1;
> unsigned intis_multiplanar:1;
> unsigned intis_output:1;
> unsigned intcopy_timestamp:1;
> --
> 2.19.1
>
Re: [PATCH 2/2] vb2: don't allow queueing buffers when canceling queue
Hi Hans,
On Wed, Nov 14, 2018 at 12:08 AM Hans Verkuil wrote:
>
> Calling the stop_streaming op can release the core serialization lock
> pointed to by vb2_queue->lock if it has to wait for buffers to finish.
> An example of that behavior is the vivid driver.
Why would vb2_queue->lock have to be released to wait for buffer to
finish? The drivers I worked with never had to do anything like that.
>
> However, if userspace dup()ped the video device filehandle, then it is
> possible to stop streaming on one filehandle and call read/write or
> VIDIOC_QBUF from the other.
How about other ioctls? I can imagine at least STREAMON could be
called at the same time too, but not sure if it would have any side
effects.
Best regards,
Tomasz
>
> This is fixed by setting a flag whenever stop_streaming is called and
> checking the flag where needed so we can return -EBUSY.
>
> Signed-off-by: Hans Verkuil
> Reported-by: syzbot+736c3aae4af7b50d9...@syzkaller.appspotmail.com
> ---
> drivers/media/common/videobuf2/videobuf2-core.c | 14 +-
> include/media/videobuf2-core.h | 1 +
> 2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> b/drivers/media/common/videobuf2/videobuf2-core.c
> index 138223af701f..560577321fe7 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -1503,6 +1503,10 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int
> index, void *pb,
> dprintk(1, "fatal error occurred on queue\n");
> return -EIO;
> }
> + if (q->in_stop_streaming) {
> + dprintk(1, "stop_streaming is called\n");
> + return -EBUSY;
> + }
>
> vb = q->bufs[index];
>
> @@ -1834,8 +1838,11 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> * Tell driver to stop all transactions and release all queued
> * buffers.
> */
> - if (q->start_streaming_called)
> + if (q->start_streaming_called) {
> + q->in_stop_streaming = 1;
> call_void_qop(q, stop_streaming, q);
> + q->in_stop_streaming = 0;
> + }
>
> /*
> * If you see this warning, then the driver isn't cleaning up properly
> @@ -2565,6 +2572,11 @@ static size_t __vb2_perform_fileio(struct vb2_queue
> *q, char __user *data, size_
> return -EBUSY;
> }
>
> + if (q->in_stop_streaming) {
> + dprintk(3, "stop_streaming is called\n");
> + return -EBUSY;
> + }
> +
> /*
> * Initialize emulator on first call.
> */
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 613f22910174..5a3d3ada5940 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -585,6 +585,7 @@ struct vb2_queue {
> unsigned interror:1;
> unsigned intwaiting_for_buffers:1;
> unsigned intwaiting_in_dqbuf:1;
> + unsigned intin_stop_streaming:1;
> unsigned intis_multiplanar:1;
> unsigned intis_output:1;
> unsigned intcopy_timestamp:1;
> --
> 2.19.1
>
Re: [PATCH v3] media: vb2: Allow reqbufs(0) with "in use" MMAP buffers
On Thu, Nov 15, 2018 at 4:59 AM Laurent Pinchart wrote: > > Hi Philipp, > > Thank you for the patch. > > On Wednesday, 14 November 2018 17:04:49 EET Philipp Zabel wrote: > > From: John Sheu > > > > Videobuf2 presently does not allow VIDIOC_REQBUFS to destroy outstanding > > buffers if the queue is of type V4L2_MEMORY_MMAP, and if the buffers are > > considered "in use". This is different behavior than for other memory > > types and prevents us from deallocating buffers in following two cases: > > > > 1) There are outstanding mmap()ed views on the buffer. However even if > >we put the buffer in reqbufs(0), there will be remaining references, > >due to vma .open/close() adjusting vb2 buffer refcount appropriately. > >This means that the buffer will be in fact freed only when the last > >mmap()ed view is unmapped. > > While I agree that we should remove this restriction, it has helped me in the > past to find missing munmap() in userspace. This patch thus has the potential > of causing memory leaks in userspace. Is there a way we could assist > application developers with this ? > I'm not very convinced that it's something we need to have, but possibly one could have it as a settable capability, that's given to REQBUF/CREATE_BUFS at allocation (count > 0) time? > > 2) Buffer has been exported as a DMABUF. Refcount of the vb2 buffer > >is managed properly by VB2 DMABUF ops, i.e. incremented on DMABUF > >get and decremented on DMABUF release. This means that the buffer > >will be alive until all importers release it. > > > > Considering both cases above, there does not seem to be any need to > > prevent reqbufs(0) operation, because buffer lifetime is already > > properly managed by both mmap() and DMABUF code paths. Let's remove it > > and allow userspace freeing the queue (and potentially allocating a new > > one) even though old buffers might be still in processing. > > > > To let userspace know that the kernel now supports orphaning buffers > > that are still in use, add a new V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS > > to be set by reqbufs and create_bufs. > > > > Signed-off-by: John Sheu > > Reviewed-by: Pawel Osciak > > Reviewed-by: Tomasz Figa > > Signed-off-by: Tomasz Figa > > [p.za...@pengutronix.de: moved __vb2_queue_cancel out of the mmap_lock > > and added V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS] > > Signed-off-by: Philipp Zabel > > Acked-by: Sakari Ailus > > --- > > Changes since v2: > > - Added documentation for V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS > > --- > > .../media/uapi/v4l/vidioc-reqbufs.rst | 15 --- > > .../media/common/videobuf2/videobuf2-core.c | 26 +-- > > .../media/common/videobuf2/videobuf2-v4l2.c | 2 +- > > include/uapi/linux/videodev2.h| 1 + > > 4 files changed, 15 insertions(+), 29 deletions(-) > > > > diff --git a/Documentation/media/uapi/v4l/vidioc-reqbufs.rst > > b/Documentation/media/uapi/v4l/vidioc-reqbufs.rst index > > d40c60e8..d53006b938ac 100644 > > --- a/Documentation/media/uapi/v4l/vidioc-reqbufs.rst > > +++ b/Documentation/media/uapi/v4l/vidioc-reqbufs.rst > > @@ -59,9 +59,12 @@ When the I/O method is not supported the ioctl returns an > > ``EINVAL`` error code. > > > > Applications can call :ref:`VIDIOC_REQBUFS` again to change the number of > > -buffers, however this cannot succeed when any buffers are still mapped. > > -A ``count`` value of zero frees all buffers, after aborting or finishing > > -any DMA in progress, an implicit > > +buffers. Note that if any buffers are still mapped or exported via DMABUF, > > +this can only succeed if the ``V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS`` flag > > +is set. In that case these buffers are orphaned and will be freed when they > > +are unmapped or when the exported DMABUF fds are closed. > > +A ``count`` value of zero frees or orphans all buffers, after aborting or > > +finishing any DMA in progress, an implicit > > > > :ref:`VIDIOC_STREAMOFF `. > > > > @@ -132,6 +135,12 @@ any DMA in progress, an implicit > > * - ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` > >- 0x0008 > >- This buffer type supports :ref:`requests `. > > +* - ``V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS`` > > + - 0x0010 > > + - The kernel allows calling :ref:`VIDIOC_REQBUFS` with a ``count`` > > value +of zero while buffers are still mapped or exported via > > DMABUF. These +orphaned buffers will be freed when they are &g
Re: [PATCH 2/5] v4l: controls: Add support for exponential bases, prefixes and units
Hi Sakari,
On Fri, Sep 28, 2018 at 11:00 PM Hans Verkuil wrote:
>
> On 09/25/2018 12:14 PM, Sakari Ailus wrote:
> > Add support for exponential bases, prefixes as well as units for V4L2
> > controls. This makes it possible to convey information on the relation
> > between the control value and the hardware feature being controlled.
> >
Sorry for being late to the party.
Thanks for the series. I think it has a potential to be very useful.
Please see my comments below.
> > Signed-off-by: Sakari Ailus
> > ---
> > include/uapi/linux/videodev2.h | 32 +++-
> > 1 file changed, 31 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> > index ae083978988f1..23b02f2db85a1 100644
> > --- a/include/uapi/linux/videodev2.h
> > +++ b/include/uapi/linux/videodev2.h
> > @@ -1652,6 +1652,32 @@ struct v4l2_queryctrl {
> > __u32reserved[2];
> > };
> >
> > +/* V4L2 control exponential bases */
> > +#define V4L2_CTRL_BASE_UNDEFINED 0
> > +#define V4L2_CTRL_BASE_LINEAR1
>
> I'm not really sure you need BASE_LINEAR. That is effectively the same
> as UNDEFINED since what else can you do? It's also weird to have this
> as 'base' if the EXPONENTIAL flag is set.
>
> I don't see why you need the EXPONENTIAL flag at all: if this is non-0,
> then you know the exponential base.
Or vice versa, we could remove UNDEFINED and LINEAR altogether and
have the EXPONENTIAL flag actually signify the presence of a valid
base? Besides that, "linear exponential base" just doesn't sound right
or am I missing some basic maths? ;)
Then we could actually have a LOGARITHMIC flag and it could reuse the
same bases enum.
>
> > +#define V4L2_CTRL_BASE_2 2
> > +#define V4L2_CTRL_BASE_1010
> > +
> > +/* V4L2 control unit prefixes */
> > +#define V4L2_CTRL_PREFIX_NANO-9
> > +#define V4L2_CTRL_PREFIX_MICRO -6
> > +#define V4L2_CTRL_PREFIX_MILLI -3
> > +#define V4L2_CTRL_PREFIX_1 0
>
> I would prefer PREFIX_NONE, since there is no prefix in this case.
>
> I assume this prefix is only valid if the unit is not UNDEFINED and not
> NONE?
>
> Is 'base' also dependent on a valid unit? (it doesn't appear to be)
>
> > +#define V4L2_CTRL_PREFIX_KILO3
> > +#define V4L2_CTRL_PREFIX_MEGA6
> > +#define V4L2_CTRL_PREFIX_GIGA9
> > +
> > +/* V4L2 control units */
> > +#define V4L2_CTRL_UNIT_UNDEFINED 0
> > +#define V4L2_CTRL_UNIT_NONE 1
Hmm, what's the meaning of NONE? How does it differ from UNDEFINED?
> > +#define V4L2_CTRL_UNIT_SECOND2
> > +#define V4L2_CTRL_UNIT_AMPERE3
> > +#define V4L2_CTRL_UNIT_LINE 4
> > +#define V4L2_CTRL_UNIT_PIXEL 5
> > +#define V4L2_CTRL_UNIT_PIXELS_PER_SEC6
> > +#define V4L2_CTRL_UNIT_HZ7
> > +
> > +
> > /* Used in the VIDIOC_QUERY_EXT_CTRL ioctl for querying extended controls
> > */
> > struct v4l2_query_ext_ctrl {
> > __u32id;
> > @@ -1666,7 +1692,10 @@ struct v4l2_query_ext_ctrl {
> > __u32elems;
> > __u32nr_of_dims;
> > __u32dims[V4L2_CTRL_MAX_DIMS];
> > - __u32reserved[32];
> > + __u8 base;
> > + __s8 prefix;
Should we make those bigger just in case, or leave some reserved
fields around so we can make them bigger when we need it?
Best regards,
Tomasz
Re: [RFC PATCH 0/3] Media Controller Properties
Hi Hans, On Fri, Aug 3, 2018 at 11:36 PM Hans Verkuil wrote: > > From: Hans Verkuil > > This RFC patch series implements properties for the media controller. > > This is not finished, but I wanted to post this so people can discuss > this further. > > No documentation yet (too early for that). > > An updated v4l2-ctl and v4l2-compliance that can report properties > is available here: > > https://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=props > > There is one main missing piece: currently the properties are effectively > laid out in random order. My plan is to change that so they are grouped > by object type and object owner. So first all properties for each entity, > then for each pad, etc. I started to work on that, but it's a bit more > work than expected and I wanted to post this before the weekend. > > While it is possible to have nested properties, this is not currently > implemented. Only properties for entities and pads are supported in this > code, but that's easy to extend to interfaces and links. > > I'm not sure about the G_TOPOLOGY ioctl handling: I went with the quickest > option by renaming the old ioctl and adding a new one with property support. > > I think this needs to change (at the very least the old and new should > share the same ioctl NR), but that's something for the future. > > Currently I support u64, s64 and const char * property types. But it > can be anything including binary data if needed. No array support (as we > have for controls), but there are enough reserved fields in media_v2_prop > to add this if needed. > > I added properties for entities and pads to vimc, so I could test this. I think I'm missing the background and the description doesn't mention it either. What's the use case for those and why not controls? Best regards, Tomasz
Re: [PATCH] media: vb2: Allow reqbufs(0) with "in use" MMAP buffers
On Wed, Nov 14, 2018 at 8:51 AM Nicolas Dufresne wrote:
>
> Le mercredi 14 novembre 2018 à 00:27 +0200, Sakari Ailus a écrit :
> > Hi Philipp,
> >
> > On Tue, Nov 13, 2018 at 04:06:21PM +0100, Philipp Zabel wrote:
> > > From: John Sheu
> > >
> > > Videobuf2 presently does not allow VIDIOC_REQBUFS to destroy outstanding
> > > buffers if the queue is of type V4L2_MEMORY_MMAP, and if the buffers are
> > > considered "in use". This is different behavior than for other memory
> > > types and prevents us from deallocating buffers in following two cases:
> > >
> > > 1) There are outstanding mmap()ed views on the buffer. However even if
> > >we put the buffer in reqbufs(0), there will be remaining references,
> > >due to vma .open/close() adjusting vb2 buffer refcount appropriately.
> > >This means that the buffer will be in fact freed only when the last
> > >mmap()ed view is unmapped.
> > >
> > > 2) Buffer has been exported as a DMABUF. Refcount of the vb2 buffer
> > >is managed properly by VB2 DMABUF ops, i.e. incremented on DMABUF
> > >get and decremented on DMABUF release. This means that the buffer
> > >will be alive until all importers release it.
> > >
> > > Considering both cases above, there does not seem to be any need to
> > > prevent reqbufs(0) operation, because buffer lifetime is already
> > > properly managed by both mmap() and DMABUF code paths. Let's remove it
> > > and allow userspace freeing the queue (and potentially allocating a new
> > > one) even though old buffers might be still in processing.
> > >
> > > To let userspace know that the kernel now supports orphaning buffers
> > > that are still in use, add a new V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS
> > > to be set by reqbufs and create_bufs.
> > >
> > > Signed-off-by: John Sheu
> > > Reviewed-by: Pawel Osciak
> > > Reviewed-by: Tomasz Figa
> > > Signed-off-by: Tomasz Figa
> > > [p.za...@pengutronix.de: moved __vb2_queue_cancel out of the mmap_lock
> > > and added V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS]
> > > Signed-off-by: Philipp Zabel
> >
> > This lets the user to allocate lots of mmap'ed buffers that are pinned in
> > physical memory. Considering that we don't really have a proper mechanism
> > to limit that anyway,
>
> It's currently limited to 32 buffers. It's not worst then DRM dumb
> buffers which will let you allocate as much as you want.
>
32 buffers for one mem2mem instance. One can open many of those and
allocate more anyway.
I think it's a part of the global problem of DMA memory not being
accounted to the process allocating...
Best regards,
Tomasz
> >
> > Acked-by: Sakari Ailus
> >
> > That said, the patch must be accompanied by the documentation change in
> > Documentation/media/uapi/v4l/vidioc-reqbufs.rst .
> >
> > I wonder what Hans thinks.
> >
> > > ---
> > > .../media/common/videobuf2/videobuf2-core.c | 26 +--
> > > .../media/common/videobuf2/videobuf2-v4l2.c | 2 +-
> > > include/uapi/linux/videodev2.h| 1 +
> > > 3 files changed, 3 insertions(+), 26 deletions(-)
> > >
> > > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> > > b/drivers/media/common/videobuf2/videobuf2-core.c
> > > index 975ff5669f72..608459450c1e 100644
> > > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > > @@ -553,20 +553,6 @@ bool vb2_buffer_in_use(struct vb2_queue *q, struct
> > > vb2_buffer *vb)
> > > }
> > > EXPORT_SYMBOL(vb2_buffer_in_use);
> > >
> > > -/*
> > > - * __buffers_in_use() - return true if any buffers on the queue are in
> > > use and
> > > - * the queue cannot be freed (by the means of REQBUFS(0)) call
> > > - */
> > > -static bool __buffers_in_use(struct vb2_queue *q)
> > > -{
> > > - unsigned int buffer;
> > > - for (buffer = 0; buffer < q->num_buffers; ++buffer) {
> > > - if (vb2_buffer_in_use(q, q->bufs[buffer]))
> > > - return true;
> > > - }
> > > - return false;
> > > -}
> > > -
> > > void vb2_core_querybuf(struct vb2_queue *q, unsigned int index, void *pb)
> > > {
> > > call_void_bufop(q, fill_user_buffer, q->bufs[index], pb);
> > > @@ -674,23 +660,13 @@ int vb2_core_r
Re: [RFP] Which V4L2 ioctls could be replaced by better versions?
On Sat, Nov 10, 2018 at 6:06 AM Nicolas Dufresne wrote: > > Le jeudi 08 novembre 2018 à 16:45 +0900, Tomasz Figa a écrit : > > > In this patch we should consider a way to tell userspace that this has > > > been opt in, otherwise existing userspace will have to remain using > > > sub-optimal copy based reclaiming in order to ensure that renegotiation > > > can work on older kernel tool. At worst someone could probably do trial > > > and error (reqbufs(1)/mmap/reqbufs(0)) but on CMA with large buffers > > > this introduces extra startup time. > > > > Would such REQBUFS dance be really needed? Couldn't one simply try > > reqbufs(0) when it's really needed and if it fails then do the copy, > > otherwise just proceed normally? > > In simple program, maybe, in modularized code, where the consumer of > these buffer (the one that is forced to make a copy) does not know the > origin of the DMABuf, it's a bit complicated. > > In GStreamer as an example, the producer is a plugin called > libgstvideo4linux2.so, while the common consumer would be libgstkms.so. > They don't know each other. The pipeline would be described as: > > v4l2src ! kmssink > > GStreamer does not have an explicit reclaiming mechanism. No one knew > about V4L2 restrictions when this was designed, DMABuf didn't exist and > GStreamer didn't have OMX support. > > What we ended up crafting, as a plaster, is that when upstream element > (v4l2src) query a new allocation from downstream (kmssink), we always > copy and return any ancient buffers by copying. kmssink holds on a > buffer because we can't remove the scannout buffer on the display. This > is slow and inefficient, and also totally unneeded if the dmabuf > originate from other kernel subsystems (like DRM). > > So what I'd like to be able to do, to support this in a more optimal > and generic way, is to mark the buffers that needs reclaiming before > letting them go. But for that, I would need a flag somewhere to tell me > this kernel allow this. Okay, got it. Thanks for explaining it. > > You got the context, maybe the conclusion is that I should simply do > kernel version check, though I'm sure a lot of people will backport > this, which means that check won't work so well. > > Let me know, I understand adding more API is not fun, but as nothing is > ever versionned in the linux-media world, it's really hard to detect > and use new behaviour while supporting what everyone currently run on > their systems. > > I would probably try and find a way to implement your suggestion, and > then introduce a flag in the query itself, but I would need to think > about it a little more. It's not as simple as it look like > unfortunately. It sounds like a good fit for a new capability in v4l2_requestbuffers and v4l2_create_buffers structs [1]. Perhaps something like V4L2_BUF_CAP_SUPPORTS_FREE_AFTER_EXPORT? Hans, what do you think? [1] https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/tree/include/uapi/linux/videodev2.h?h=next-20181109#n866 Best regards, Tomasz
Re: [RFP] Which V4L2 ioctls could be replaced by better versions?
Hi Nicolas, On Sat, Oct 27, 2018 at 6:38 PM Nicolas Dufresne wrote: > > Le lundi 22 octobre 2018 à 12:37 +0900, Tomasz Figa a écrit : > > Hi Philipp, > > > > On Mon, Oct 22, 2018 at 1:28 AM Philipp Zabel wrote: > > > > > > On Wed, Oct 03, 2018 at 05:24:39PM +0900, Tomasz Figa wrote: > > > [...] > > > > > Yes, but that would fall in a complete redesign I guess. The buffer > > > > > allocation scheme is very inflexible. You can't have buffers of two > > > > > dimensions allocated at the same time for the same queue. Worst, you > > > > > cannot leave even 1 buffer as your scannout buffer while reallocating > > > > > new buffers, this is not permitted by the framework (in software). As > > > > > a > > > > > side effect, there is no way to optimize the resolution changes, you > > > > > even have to copy your scannout buffer on the CPU, to free it in order > > > > > to proceed. Resolution changes are thus painfully slow, by design. > > > > > > [...] > > > > Also, I fail to understand the scanout issue. If one exports a vb2 > > > > buffer to a DMA-buf and import it to the scanout engine, it can keep > > > > scanning out from it as long as it want, because the DMA-buf will hold > > > > a reference on the buffer, even if it's removed from the vb2 queue. > > > > > > REQBUFS 0 fails if the vb2 buffer is still in use, including from dmabuf > > > attachments: vb2_buffer_in_use checks the num_users memop. The refcount > > > returned by num_users shared between the vmarea handler and dmabuf ops, > > > so any dmabuf attachment counts towards in_use. > > > > Ah, right. I've managed to completely forget about it, since we have a > > downstream patch that we attempted to upstream earlier [1], but didn't > > have a chance to follow up on the comments and there wasn't much > > interest in it in general. > > > > [1] https://lore.kernel.org/patchwork/patch/607853/ > > > > Perhaps it would be worth reviving? > > In this patch we should consider a way to tell userspace that this has > been opt in, otherwise existing userspace will have to remain using > sub-optimal copy based reclaiming in order to ensure that renegotiation > can work on older kernel tool. At worst someone could probably do trial > and error (reqbufs(1)/mmap/reqbufs(0)) but on CMA with large buffers > this introduces extra startup time. Would such REQBUFS dance be really needed? Couldn't one simply try reqbufs(0) when it's really needed and if it fails then do the copy, otherwise just proceed normally? Best regards, Tomasz
Re: [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
On Tue, Nov 6, 2018 at 2:50 PM Zhi, Yong wrote:
>
> Hi, Sakari,
>
> Thanks for the feedback.
>
> > -Original Message-
> > From: Sakari Ailus [mailto:sakari.ai...@linux.intel.com]
> > Sent: Monday, November 5, 2018 3:55 AM
> > To: Zhi, Yong
> > Cc: linux-media@vger.kernel.org; tf...@chromium.org;
> > mche...@kernel.org; hans.verk...@cisco.com;
> > laurent.pinch...@ideasonboard.com; Mani, Rajmohan
> > ; Zheng, Jian Xu ; Hu,
> > Jerry W ; Toivonen, Tuukka
> > ; Qiu, Tian Shu ; Cao,
> > Bingbu
> > Subject: Re: [PATCH v7 06/16] intel-ipu3: mmu: Implement driver
> >
> > Hi Yong,
> >
> > On Mon, Oct 29, 2018 at 03:23:00PM -0700, Yong Zhi wrote:
> > > From: Tomasz Figa
> > >
> > > This driver translates IO virtual address to physical address based on
> > > two levels page tables.
> > >
> > > Signed-off-by: Tomasz Figa
> > > Signed-off-by: Yong Zhi
> > > ---
> >
> > ...
> >
> > > +static void call_if_ipu3_is_powered(struct ipu3_mmu *mmu,
> > > + void (*func)(struct ipu3_mmu *mmu)) {
> > > + pm_runtime_get_noresume(mmu->dev);
> > > + if (pm_runtime_active(mmu->dev))
> > > + func(mmu);
> > > + pm_runtime_put(mmu->dev);
> >
> > How about:
> >
> > if (!pm_runtime_get_if_in_use(mmu->dev))
> > return;
> >
> > func(mmu);
> > pm_runtime_put(mmu->dev);
> >
>
> Ack, unless Tomasz has different opinion.
It's actually the proper way of doing it. Thanks for the suggestion.
Best regards,
Tomasz
Re: [PATCH v7 03/16] v4l: Add Intel IPU3 meta data uAPI
Hi Mauro,
On Fri, Nov 2, 2018 at 10:49 PM Mauro Carvalho Chehab
wrote:
>
> Em Mon, 29 Oct 2018 15:22:57 -0700
> Yong Zhi escreveu:
[snip]
> > +struct ipu3_uapi_awb_config_s {
> > + __u16 rgbs_thr_gr;
> > + __u16 rgbs_thr_r;
> > + __u16 rgbs_thr_gb;
> > + __u16 rgbs_thr_b;
> > + struct ipu3_uapi_grid_config grid;
> > +} __attribute__((aligned(32))) __packed;
>
> Hmm... Kernel defines a macro for aligned attribute:
>
> include/linux/compiler_types.h:#define __aligned(x)
> __attribute__((aligned(x)))
>
First, thanks for review!
Maybe I missed something, but last time I checked, it wasn't
accessible from UAPI headers in userspace.
> I'm not a gcc expert, but it sounds weird to first ask it to align
> with 32 bits and then have __packed (with means that pads should be
> removed).
>
> In other words, I *guess* is it should either be __packed
> or __aligned(32).
>
> Not that it would do any difference, in practice, as this
> specific struct has a size with is multiple of 32 bits, but
> let's do the right annotation here, not mixing two incompatible
> alignment requirements.
>
My understanding was that __packed makes the compiler not insert any
alignment between particular fields of the struct, while __aligned
makes the whole struct be aligned at given boundary, if placed in
another struct. If I didn't miss anything, having both should make
perfect sense here.
Best regards,
Tomasz
Re: VIVID/VIMC and media fuzzing
Hi Dmitry, On Wed, Oct 31, 2018 at 6:46 PM Hans Verkuil wrote: > > On 10/30/2018 03:02 PM, Dmitry Vyukov wrote: [snip] > > > > Do I understand it correctly that when a process opens /dev/video* or > > /dev/media* it gets a private instance of the device? In particular, > > if several processes test this in parallel, will they collide? Or they > > will stress separate objects? > > It actually depends on the driver. M2M devices will give you a private > instance whenever you open it. Others do not, but you can call most ioctls > in parallel. But after calling REQBUFS or CREATE_BUFS the filehandle that > called those ioctls becomes owner of the device until the buffers are > released. So other filehandles cannot do any streaming operations (EBUSY > will be returned). FWIW, you can query whether the device is M2M or not by VIDIOC_QUERYCAP [1]. The capabilities field would have V4L2_CAP_VIDEO_M2M or V4L2_CAP_VIDEO_M2M_MPLANE set for M2M devices. [1] https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-querycap.html Best regards, Tomasz
Re: [RFP] Which V4L2 ioctls could be replaced by better versions?
On Thu, Sep 20, 2018 at 11:42 PM Hans Verkuil wrote: > > Some parts of the V4L2 API are awkward to use and I think it would be > a good idea to look at possible candidates for that. > > Examples are the ioctls that use struct v4l2_buffer: the multiplanar support > is > really horrible, and writing code to support both single and multiplanar is > hard. > We are also running out of fields and the timeval isn't y2038 compliant. > > A proof-of-concept is here: > > https://git.linuxtv.org/hverkuil/media_tree.git/commit/?h=v4l2-buffer=a95549df06d9900f3559afdbb9da06bd4b22d1f3 > > It's a bit old, but it gives a good impression of what I have in mind. On a related, but slightly different note, I'm wondering how we should handle a case where we have an M format (e.g. NV12M with 2 memory planes), but only 1 DMA-buf with all planes to import. That generally means that we have to use the same DMA-buf FD with an offset for each plane. In theory, v4l2_plane::data_offset could be used for this, but the documentation says that it should be set by the application only for OUTPUT planes. Moreover, existing drivers tend to just ignore it... There is also the opposite problem. Sometimes the application is given 3 different FDs but pointing to the same buffer. If it has to work with a video device that only supports non-M formats, it can either fail, making it unusable, or blindly assume that they all point to the same buffer and just give the first FD to the video device (we do it in Chromium, since our allocator is guaranteed to keep all planes of planar formats in one buffer, if to be used with V4L2). Something that we could do is allowing the QBUF semantics of M formats for non-M formats, where the application would fill the planes[] array for all planes with all the FDs it has and the kernel could then figure out if they point to the same buffer (i.e. resolve to the same dma_buf struct) or fail if not. [...] > Do we have more ioctls that could use a refresh? S/G/TRY_FMT perhaps, again in > order to improve single vs multiplanar handling. I'd definitely be more than happy to see the plane handling unified between non-M and M formats, in general. The list of problems with current interface: 1) The userspace has to hardcode the computations of bytesperline for chroma planes of non-M formats (while they are reported for M formats). 2) Similarly, offsets of the planes in the buffer for non-M formats must be explicitly calculated in the application, 3) Drivers have to explicitly handle both non-M and M formats or otherwise they would suffer from issues with application compatibility or sharing buffers with other devices (one supporting only M and the other only non-M), 4) Inconsistency in the meaning of planes[0].sizeimage for non-M formats and M formats, making it impossible to use planes[0].sizeimage to set the luma plane size in the hardware for non-M formats (since it's the total size of all planes). I might have probably forgotten about something, but generally fixing the 4 above, would be a really big step forward. Best regards, Tomasz
Re: [PATCH 1/2] vicodec: Have decoder propagate changes to the CAPTURE queue
On Fri, Oct 19, 2018 at 10:00 PM Ezequiel Garcia wrote:
>
> On Fri, 2018-10-19 at 09:14 +0200, Hans Verkuil wrote:
> > On 10/18/2018 06:08 PM, Ezequiel Garcia wrote:
> > > The decoder interface (not yet merged) specifies that
> > > width and height values set on the OUTPUT queue, must
> > > be propagated to the CAPTURE queue.
> > >
> > > This is not enough to comply with the specification,
> > > which would require to properly support stream resolution
> > > changes detection and notification.
> > >
> > > However, it's a relatively small change, which fixes behavior
> > > required by some applications such as gstreamer.
> > >
> > > With this change, it's possible to run a simple T(T⁻¹) pipeline:
> > >
> > > gst-launch-1.0 videotestsrc ! v4l2fwhtenc ! v4l2fwhtdec ! fakevideosink
> > >
> > > Signed-off-by: Ezequiel Garcia
> > > ---
> > > drivers/media/platform/vicodec/vicodec-core.c | 15 +++
> > > 1 file changed, 15 insertions(+)
> > >
> > > diff --git a/drivers/media/platform/vicodec/vicodec-core.c
> > > b/drivers/media/platform/vicodec/vicodec-core.c
> > > index 1eb9132bfc85..a2c487b4b80d 100644
> > > --- a/drivers/media/platform/vicodec/vicodec-core.c
> > > +++ b/drivers/media/platform/vicodec/vicodec-core.c
> > > @@ -673,6 +673,13 @@ static int vidioc_s_fmt(struct vicodec_ctx *ctx,
> > > struct v4l2_format *f)
> > > q_data->width = pix->width;
> > > q_data->height = pix->height;
> > > q_data->sizeimage = pix->sizeimage;
> > > +
> > > + /* Propagate changes to CAPTURE queue */
> > > + if (!ctx->is_enc && V4L2_TYPE_IS_OUTPUT(f->type)) {
> >
> > Do we need !ctx->is_enc? Isn't this the same for both decoder and encoder?
> >
>
> Well, I wasn't really sure about this. The decoder document clearly
> says that changes has to be propagated to the capture queue, but that
> statement
> is not in the encoder spec.
>
> Since gstreamer didn't needs this, I decided not to add it.
>
> Perhaps it's something to correct in the encoder spec?
Hmm, in the v2 of the documentation I sent recently, the CAPTURE queue
of an encoder doesn't have width and height specified. For formats
that have the resolution embedded in bitstream metadata, this isn't
anything that the userspace should be concerned with. I forgot about
the formats that don't have the resolution in the metadata, so we
might need to bring them back. Then the propagation would have to be
there indeed.
> > > + ctx->q_data[V4L2_M2M_DST].width = pix->width;
> > > + ctx->q_data[V4L2_M2M_DST].height = pix->height;
> > > + ctx->q_data[V4L2_M2M_DST].sizeimage = pix->sizeimage;
> >
> > This is wrong: you are copying the sizeimage for the compressed format as
> > the
> > sizeimage for the raw format, which is quite different.
> >
>
> Doh, you are right.
>
> > I think you need to make a little helper function that can update the
> > width/height
> > of a particular queue and that can calculate the sizeimage correctly.
> >
I wish we had generic helpers to manage all the formats in one place,
rather than duplicating the handling in each driver. I found many
cases of drivers not reporting bytesperline correctly or not handling
some formats (other than default and so often not tested) correctly.
If we could just have the driver call
v4l2_fill_pixfmt_mp_for_format(_mp, pixelformat, width, height,
...), a lot of boilerplate and potential source of errors could be
removed. (Bonus points for helpers that can convert pixfmt_mp for a
non-M format, e.g. NV12, into a pixfmt_mp for the corresponding M
format, e.g. NV12M, so that all the drivers that can support M formats
can also handle non-M formats automatically.)
One thing to note, though, is that there might be driver specific
alignment constraints in the play, so care must be taken.
Best regards,
Tomasz
Re: [RFC] Informal meeting during ELCE to discuss userspace support for stateless codecs
On Tue, Oct 23, 2018 at 6:17 AM Hans Verkuil wrote: > > A quick update: > > As said in my previous email: we'll meet at 11 am at the registration desk. > From there we go to the Platform 5 Cafe. If that's too crowded/noisy, then > we'll try the Sheraton hotel. > > Tomasz, I'll ping you on irc when we found a good spot and we can setup the > Hangouts meeting. Thanks. In case of anyone else interested in joining, please use this link: http://bit.ly/elce-codecs Best regards, Tomasz > > Hope to see you all tomorrow, > > Regards, > > Hans > > On 10/10/2018 07:55 AM, Hans Verkuil wrote: > > On 10/08/2018 01:53 PM, Hans Verkuil wrote: > >> Hi all, > >> > >> I would like to meet up somewhere during the ELCE to discuss userspace > >> support > >> for stateless (and perhaps stateful as well?) codecs. > >> > >> It is also planned as a topic during the summit, but I would prefer to > >> prepare > >> for that in advance, esp. since I myself do not have any experience writing > >> userspace SW for such devices. > >> > >> Nicolas, it would be really great if you can participate in this meeting > >> since you probably have the most experience with this by far. > >> > >> Looking through the ELCE program I found two timeslots that are likely to > >> work > >> for most of us (because the topics in the program appear to be boring for > >> us > >> media types!): > >> > >> Tuesday from 10:50-15:50 > > > > Let's do this Tuesday. Let's meet at the Linux Foundation Registration > > Desk at 11:00. I'll try to figure out where we can sit the day before. > > Please check your email Tuesday morning for any last minute changes. > > > > Tomasz, it would be nice indeed if we can get you and Paul in as well > > using Hangouts on my laptop. > > > > I would very much appreciate it if those who have experience with the > > userspace support think about this beforehand and make some requirements > > list of what you would like to see. > > > > Regards, > > > > Hans > > > >> > >> or: > >> > >> Monday from 15:45 onward > >> > >> My guess is that we need 2-3 hours or so. Hard to predict. > >> > >> The basic question that I would like to have answered is what the userspace > >> component should look like? libv4l-like plugin or a library that userspace > >> can > >> link with? Do we want more general support for stateful codecs as well > >> that deals > >> with resolution changes and the more complex parts of the codec API? > >> > >> I've mailed this directly to those that I expect are most interested in > >> this, > >> but if someone want to join in let me know. > >> > >> I want to keep the group small though, so you need to bring relevant > >> experience > >> to the table. > >> > >> Regards, > >> > >> Hans > >> > > >
Re: [RFP] Which V4L2 ioctls could be replaced by better versions?
Hi Philipp, On Mon, Oct 22, 2018 at 1:28 AM Philipp Zabel wrote: > > On Wed, Oct 03, 2018 at 05:24:39PM +0900, Tomasz Figa wrote: > [...] > > > Yes, but that would fall in a complete redesign I guess. The buffer > > > allocation scheme is very inflexible. You can't have buffers of two > > > dimensions allocated at the same time for the same queue. Worst, you > > > cannot leave even 1 buffer as your scannout buffer while reallocating > > > new buffers, this is not permitted by the framework (in software). As a > > > side effect, there is no way to optimize the resolution changes, you > > > even have to copy your scannout buffer on the CPU, to free it in order > > > to proceed. Resolution changes are thus painfully slow, by design. > [...] > > Also, I fail to understand the scanout issue. If one exports a vb2 > > buffer to a DMA-buf and import it to the scanout engine, it can keep > > scanning out from it as long as it want, because the DMA-buf will hold > > a reference on the buffer, even if it's removed from the vb2 queue. > > REQBUFS 0 fails if the vb2 buffer is still in use, including from dmabuf > attachments: vb2_buffer_in_use checks the num_users memop. The refcount > returned by num_users shared between the vmarea handler and dmabuf ops, > so any dmabuf attachment counts towards in_use. Ah, right. I've managed to completely forget about it, since we have a downstream patch that we attempted to upstream earlier [1], but didn't have a chance to follow up on the comments and there wasn't much interest in it in general. [1] https://lore.kernel.org/patchwork/patch/607853/ Perhaps it would be worth reviving? Best regards, Tomasz
Re: [PATCH v7] media: add imx319 camera sensor driver
On Sun, Oct 21, 2018 at 6:53 AM Sakari Ailus wrote: > > Hi Tomasz, > > On Thu, Oct 18, 2018 at 12:24:31PM +0900, Tomasz Figa wrote: > > On Tue, Oct 16, 2018 at 8:50 PM Sakari Ailus > > wrote: > > > > > > Hi Tomasz, > > > > > > On Fri, Oct 12, 2018 at 05:06:56PM +0900, Tomasz Figa wrote: > > > > On Fri, Oct 12, 2018 at 4:58 PM Sakari Ailus > > > > wrote: > > > > > > > > > > Hi Tomasz, > > > > > > > > > > On Fri, Oct 12, 2018 at 01:51:10PM +0900, Tomasz Figa wrote: > > > > > > Hi Sakari, > > > > > > > > > > > > On Wed, Sep 26, 2018 at 11:38 AM wrote: > > > > > > > > > > > > > > From: Bingbu Cao > > > > > > > > > > > > > > Add a v4l2 sub-device driver for the Sony imx319 image sensor. > > > > > > > This is a camera sensor using the i2c bus for control and the > > > > > > > csi-2 bus for data. > > > > > > > > > > > > > > This driver supports following features: > > > > > > > - manual exposure and analog/digital gain control support > > > > > > > - vblank/hblank control support > > > > > > > - 4 test patterns control support > > > > > > > - vflip/hflip control support (will impact the output bayer order) > > > > > > > - support following resolutions: > > > > > > > - 3264x2448, 3280x2464 @ 30fps > > > > > > > - 1936x1096, 1920x1080 @ 60fps > > > > > > > - 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps > > > > > > > - support 4 bayer orders output (via change v/hflip) > > > > > > > - SRGGB10(default), SGRBG10, SGBRG10, SBGGR10 > > > > > > > > > > > > > > Cc: Tomasz Figa > > > > > > > Cc: Sakari Ailus > > > > > > > Signed-off-by: Bingbu Cao > > > > > > > Signed-off-by: Tianshu Qiu > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > This patch is based on sakari's media-tree git: > > > > > > > https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1 > > > > > > > > > > > > > > Changes from v5: > > > > > > > - add some comments for gain calculation > > > > > > > - use lock to protect the format > > > > > > > - fix some style issues > > > > > > > > > > > > > > Changes from v4 to v5: > > > > > > > - use single PLL for all internal clocks > > > > > > > - change link frequency to 482.4MHz > > > > > > > - adjust frame timing for 2x2 binning modes > > > > > > >and enlarge frame readout time > > > > > > > - get CSI-2 link frequencies and external clock > > > > > > >from firmware > > > > > > > > > > > > If I remember correctly, that was suggested by you. Why do we need > > > > > > to > > > > > > specify link frequency in firmware if it's fully configured by the > > > > > > driver, with the only external dependency being the external clock? > > > > > > > > > > The driver that's now in upstream supports, for now, a very limited > > > > > set of > > > > > configurations from what the sensor supports. These are more or less > > > > > tailored to the particular system where it is being used right now > > > > > (output > > > > > image size, external clock frequency, frame rates, link frequencies > > > > > etc.). > > > > > > > > As a side note, they're tailored to exactly the system I mentioned, > > > > with different link frequency hardcoded in the firmware, coming from > > > > earlier stage of development. > > > > > > > > > If the same sensor is needed elsewhere (quite likely), the > > > > > configuration > > > > > needed elsewhere is very likely to be different from what you're > > > > > using now. > > > > > > > > > > The link frequency in particular is important as using a different > > > > > link > > > > > frequency (which could be fine
Re: [PATCH v7] media: add imx319 camera sensor driver
On Tue, Oct 16, 2018 at 8:50 PM Sakari Ailus wrote: > > Hi Tomasz, > > On Fri, Oct 12, 2018 at 05:06:56PM +0900, Tomasz Figa wrote: > > On Fri, Oct 12, 2018 at 4:58 PM Sakari Ailus > > wrote: > > > > > > Hi Tomasz, > > > > > > On Fri, Oct 12, 2018 at 01:51:10PM +0900, Tomasz Figa wrote: > > > > Hi Sakari, > > > > > > > > On Wed, Sep 26, 2018 at 11:38 AM wrote: > > > > > > > > > > From: Bingbu Cao > > > > > > > > > > Add a v4l2 sub-device driver for the Sony imx319 image sensor. > > > > > This is a camera sensor using the i2c bus for control and the > > > > > csi-2 bus for data. > > > > > > > > > > This driver supports following features: > > > > > - manual exposure and analog/digital gain control support > > > > > - vblank/hblank control support > > > > > - 4 test patterns control support > > > > > - vflip/hflip control support (will impact the output bayer order) > > > > > - support following resolutions: > > > > > - 3264x2448, 3280x2464 @ 30fps > > > > > - 1936x1096, 1920x1080 @ 60fps > > > > > - 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps > > > > > - support 4 bayer orders output (via change v/hflip) > > > > > - SRGGB10(default), SGRBG10, SGBRG10, SBGGR10 > > > > > > > > > > Cc: Tomasz Figa > > > > > Cc: Sakari Ailus > > > > > Signed-off-by: Bingbu Cao > > > > > Signed-off-by: Tianshu Qiu > > > > > > > > > > --- > > > > > > > > > > This patch is based on sakari's media-tree git: > > > > > https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1 > > > > > > > > > > Changes from v5: > > > > > - add some comments for gain calculation > > > > > - use lock to protect the format > > > > > - fix some style issues > > > > > > > > > > Changes from v4 to v5: > > > > > - use single PLL for all internal clocks > > > > > - change link frequency to 482.4MHz > > > > > - adjust frame timing for 2x2 binning modes > > > > >and enlarge frame readout time > > > > > - get CSI-2 link frequencies and external clock > > > > >from firmware > > > > > > > > If I remember correctly, that was suggested by you. Why do we need to > > > > specify link frequency in firmware if it's fully configured by the > > > > driver, with the only external dependency being the external clock? > > > > > > The driver that's now in upstream supports, for now, a very limited set of > > > configurations from what the sensor supports. These are more or less > > > tailored to the particular system where it is being used right now (output > > > image size, external clock frequency, frame rates, link frequencies etc.). > > > > As a side note, they're tailored to exactly the system I mentioned, > > with different link frequency hardcoded in the firmware, coming from > > earlier stage of development. > > > > > If the same sensor is needed elsewhere (quite likely), the configuration > > > needed elsewhere is very likely to be different from what you're using > > > now. > > > > > > The link frequency in particular is important as using a different link > > > frequency (which could be fine elsewhere) could cause EMI issues, e.g. > > > rendering your GPS receiver inoperable during the time the camera sensor > > > is > > > streaming images. > > > > > > Should new configurations be added to this driver to support a different > > > system, the link frequencies used by those configurations may be > > > problematic to your system, and after a software update the driver could > > > as > > > well use those new frequencies. That's a big no-no. > > > > > > > Okay, those are some valid points indeed, thanks for clarifying. > > > > > > > > > > We're having problems with firmware listing the link frequency from v4 > > > > and we can't easily change it anymore to report the new one. I feel > > > > like this dependency on the firmware here is unnecessary, as long as > > > > the external clock frequency matches. > > > > > > This is information you really need to k
Re: [RFC] Informal meeting during ELCE to discuss userspace support for stateless codecs
+Alexandre Courbot On Wed, Oct 10, 2018 at 3:55 PM Hans Verkuil wrote: > > On 10/08/2018 01:53 PM, Hans Verkuil wrote: > > Hi all, > > > > I would like to meet up somewhere during the ELCE to discuss userspace > > support > > for stateless (and perhaps stateful as well?) codecs. > > > > It is also planned as a topic during the summit, but I would prefer to > > prepare > > for that in advance, esp. since I myself do not have any experience writing > > userspace SW for such devices. > > > > Nicolas, it would be really great if you can participate in this meeting > > since you probably have the most experience with this by far. > > > > Looking through the ELCE program I found two timeslots that are likely to > > work > > for most of us (because the topics in the program appear to be boring for us > > media types!): > > > > Tuesday from 10:50-15:50 > > Let's do this Tuesday. Let's meet at the Linux Foundation Registration > Desk at 11:00. I'll try to figure out where we can sit the day before. > Please check your email Tuesday morning for any last minute changes. > > Tomasz, it would be nice indeed if we can get you and Paul in as well > using Hangouts on my laptop. That would be great. Please let me know if you need any help with setup. Best regards, Tomasz > > I would very much appreciate it if those who have experience with the > userspace support think about this beforehand and make some requirements > list of what you would like to see. > > Regards, > > Hans > > > > > or: > > > > Monday from 15:45 onward > > > > My guess is that we need 2-3 hours or so. Hard to predict. > > > > The basic question that I would like to have answered is what the userspace > > component should look like? libv4l-like plugin or a library that userspace > > can > > link with? Do we want more general support for stateful codecs as well that > > deals > > with resolution changes and the more complex parts of the codec API? > > > > I've mailed this directly to those that I expect are most interested in > > this, > > but if someone want to join in let me know. > > > > I want to keep the group small though, so you need to bring relevant > > experience > > to the table. > > > > Regards, > > > > Hans > > >
Re: [PATCH v7] media: add imx319 camera sensor driver
On Fri, Oct 12, 2018 at 4:58 PM Sakari Ailus wrote: > > Hi Tomasz, > > On Fri, Oct 12, 2018 at 01:51:10PM +0900, Tomasz Figa wrote: > > Hi Sakari, > > > > On Wed, Sep 26, 2018 at 11:38 AM wrote: > > > > > > From: Bingbu Cao > > > > > > Add a v4l2 sub-device driver for the Sony imx319 image sensor. > > > This is a camera sensor using the i2c bus for control and the > > > csi-2 bus for data. > > > > > > This driver supports following features: > > > - manual exposure and analog/digital gain control support > > > - vblank/hblank control support > > > - 4 test patterns control support > > > - vflip/hflip control support (will impact the output bayer order) > > > - support following resolutions: > > > - 3264x2448, 3280x2464 @ 30fps > > > - 1936x1096, 1920x1080 @ 60fps > > > - 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps > > > - support 4 bayer orders output (via change v/hflip) > > > - SRGGB10(default), SGRBG10, SGBRG10, SBGGR10 > > > > > > Cc: Tomasz Figa > > > Cc: Sakari Ailus > > > Signed-off-by: Bingbu Cao > > > Signed-off-by: Tianshu Qiu > > > > > > --- > > > > > > This patch is based on sakari's media-tree git: > > > https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1 > > > > > > Changes from v5: > > > - add some comments for gain calculation > > > - use lock to protect the format > > > - fix some style issues > > > > > > Changes from v4 to v5: > > > - use single PLL for all internal clocks > > > - change link frequency to 482.4MHz > > > - adjust frame timing for 2x2 binning modes > > >and enlarge frame readout time > > > - get CSI-2 link frequencies and external clock > > >from firmware > > > > If I remember correctly, that was suggested by you. Why do we need to > > specify link frequency in firmware if it's fully configured by the > > driver, with the only external dependency being the external clock? > > The driver that's now in upstream supports, for now, a very limited set of > configurations from what the sensor supports. These are more or less > tailored to the particular system where it is being used right now (output > image size, external clock frequency, frame rates, link frequencies etc.). As a side note, they're tailored to exactly the system I mentioned, with different link frequency hardcoded in the firmware, coming from earlier stage of development. > If the same sensor is needed elsewhere (quite likely), the configuration > needed elsewhere is very likely to be different from what you're using now. > > The link frequency in particular is important as using a different link > frequency (which could be fine elsewhere) could cause EMI issues, e.g. > rendering your GPS receiver inoperable during the time the camera sensor is > streaming images. > > Should new configurations be added to this driver to support a different > system, the link frequencies used by those configurations may be > problematic to your system, and after a software update the driver could as > well use those new frequencies. That's a big no-no. > Okay, those are some valid points indeed, thanks for clarifying. > > > > We're having problems with firmware listing the link frequency from v4 > > and we can't easily change it anymore to report the new one. I feel > > like this dependency on the firmware here is unnecessary, as long as > > the external clock frequency matches. > > This is information you really need to know. > > A number of older drivers do not use the link frequency information from > the firmware but that comes with a risk. Really, it's better to change the > frequency now to something you can choose, rather than have it changed > later on to something someone else chose for you. I guess it means that we have to carry a local downstream patch that bypasses this check, since we cannot easily change the firmware anymore. An alternative would be to make the driver try to select a frequency that matches what's in the firmware, but issue a warning and fall back to a default one if a matching is not found. It might be actually better than nothing for some early testing on new systems, since it wouldn't require firmware changes. Best regards, Tomasz
Re: [PATCH v7] media: add imx319 camera sensor driver
Hi Sakari, On Wed, Sep 26, 2018 at 11:38 AM wrote: > > From: Bingbu Cao > > Add a v4l2 sub-device driver for the Sony imx319 image sensor. > This is a camera sensor using the i2c bus for control and the > csi-2 bus for data. > > This driver supports following features: > - manual exposure and analog/digital gain control support > - vblank/hblank control support > - 4 test patterns control support > - vflip/hflip control support (will impact the output bayer order) > - support following resolutions: > - 3264x2448, 3280x2464 @ 30fps > - 1936x1096, 1920x1080 @ 60fps > - 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps > - support 4 bayer orders output (via change v/hflip) > - SRGGB10(default), SGRBG10, SGBRG10, SBGGR10 > > Cc: Tomasz Figa > Cc: Sakari Ailus > Signed-off-by: Bingbu Cao > Signed-off-by: Tianshu Qiu > > --- > > This patch is based on sakari's media-tree git: > https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1 > > Changes from v5: > - add some comments for gain calculation > - use lock to protect the format > - fix some style issues > > Changes from v4 to v5: > - use single PLL for all internal clocks > - change link frequency to 482.4MHz > - adjust frame timing for 2x2 binning modes >and enlarge frame readout time > - get CSI-2 link frequencies and external clock >from firmware If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock? We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches. Best regards, Tomasz
Re: [RFC] Informal meeting during ELCE to discuss userspace support for stateless codecs
Hi Hans, On Mon, Oct 8, 2018 at 8:53 PM Hans Verkuil wrote: > > Hi all, > > I would like to meet up somewhere during the ELCE to discuss userspace support > for stateless (and perhaps stateful as well?) codecs. > > It is also planned as a topic during the summit, but I would prefer to prepare > for that in advance, esp. since I myself do not have any experience writing > userspace SW for such devices. > > Nicolas, it would be really great if you can participate in this meeting > since you probably have the most experience with this by far. > > Looking through the ELCE program I found two timeslots that are likely to work > for most of us (because the topics in the program appear to be boring for us > media types!): > > Tuesday from 10:50-15:50 If we end up with this or similar time slot (compatible with APAC time zones), do you think it would be possible to set up a video conference for me and/or Alex to join remotely? I can help figuring out any necessary infrastructure. Best regards, Tomasz
Re: s5p_mfc and H.264 frame cropping question
On Fri, Oct 5, 2018 at 5:38 PM Hans Verkuil wrote: > > On 10/05/2018 10:16 AM, Tomasz Figa wrote: > > On Fri, Oct 5, 2018 at 3:58 PM Hans Verkuil wrote: > >> > >> On 10/05/2018 05:12 AM, Tomasz Figa wrote: > >>> Hi Hans, > >>> > >>> On Fri, Oct 5, 2018 at 5:02 AM Hans Verkuil wrote: > >>>> > >>>> Hi all, > >>>> > >>>> I'm looking at removing the last users of vidioc_g/s_crop from the > >>>> driver and > >>>> I came across vidioc_g_crop in > >>>> drivers/media/platform/s5p-mfc/s5p_mfc_dec.c. > >>>> > >>>> What this really does AFAICS is return the H.264 frame crop as read from > >>>> the > >>>> bitstream. This has nothing to do with regular cropping/composing but it > >>>> might be > >>>> something that could be implemented as a new selection target. > >>> > >>> It has a lot to do, because the output frame buffer may contain (and > >>> on the hardware I worked with, s5p-mfc and mtk-vcodec, indeed does) > >>> the whole encoded stream and the frame crop from the bitstream > >>> specifies the rectangle within it that corresponds to the part that > >>> should be displayed. > >> > >> Yes, but is that part actually cropped? Or is the full uncropped image > >> DMAed > >> to the capture buffer? > > > > The latter. > > > >> > >> To take a practical example: a H.264 stream with a 1920x1088 image and a > >> frame > >> crop rectangle of 1920x1080. What is the G_FMT width/height for the decoder > >> capture stream: 1920x1088 or 1920x1080? > >> > >> If it is 1920x1088, then you have a compose rectangle. If it is 1920x1080 > >> then > >> you have a crop rectangle. > > > > 1920x1088 > > > >> > >> As far as I can tell from this driver it actually has a compose rectangle > >> and the use of g_crop is wrong and is there due to historical reasons (the > >> driver predates the selection API). > > > > Yes, it is there due to historical reasons. > > > >> > >>> > >>>> > >>>> I'm not really sure what to do with the existing code since it is an > >>>> abuse of > >>>> the crop API, but I guess the first step is to decide how this should be > >>>> handled > >>>> properly. > >>>> > >>>> Are there other decoders that can retrieve this information? Should this > >>>> be > >>>> mentioned in the stateful codec API? > >>> > >>> coda [1], mtk-vcodec [2] and venus [3] expose this using the > >>> V4L2_SEL_TGT_COMPOSE selection target. v1 of the specification defines > >>> the selection targets in a way, which is compatible with that: > >>> V4L2_SEL_TGT_COMPOSE defaults to V4L2_SEL_TGT_COMPOSE_DEFAULT, which > >>> equals to V4L2_SEL_TGT_CROP, which defaults to > >>> V4L2_SEL_TGT_CROP_DEFAULT, which is defined as follows: > >>> > >>> + ``V4L2_SEL_TGT_CROP_DEFAULT`` > >>> + a rectangle covering the part of the frame buffer that contains > >>> + meaningful picture data (visible area); width and height will > >>> be > >>> + equal to visible resolution of the stream > >> > >> Where do you get that from? That's the crop definition for an output > >> stream, > >> not a capture stream (assuming we have a codec). > > > > We're talking about a decoder here, right? > > > > In this case OUTPUT stream is just a sequence of bytes, not video > > frames, so there is no selection defined for OUTPUT queue. > > > > CAPTURE stream should be seen as a video grabber, so CROP targets > > relate to the encoded rectangle (1920x1088) and COMPOSE targets to the > > CAPTURE buffer. V4L2_SEL_TGT_COMPOSE would be the part of the CAPTURE > > buffer that is written with the image selected by V4L2_SEL_TGT_CROP. > > > > On a decoder that cannot do arbitrary crop and compose, like s5p-mfc, > > both targets would have identical rectangles, equal to the visible > > region (1920x1080). On hardware which can actually do fancier things, > > userspace could freely configure CAPTURE buffer width and height and > > V4L2_SEL_TGT_COMPOSE rectangle, for example to downscale the decoded > > video on the fly. > > > > Please check how I specified all the targets in last version of the > > specification (https://lore.kernel.org/patchwork/patch/966933/) and > > comment there, if there is anything that goes against the > > specification of the selection API. > > I think we all mean the same thing, but just got confused :-) > > > > >> > >> I kind of lost you with "which equals to V4L2_SEL_TGT_CROP". > >> > >> In any case, this particular driver should implement g_selection for > >> CAPTURE and implement the COMPOSE targets. That makes sense. > > Right. > > Please check my RFC series I just posted that hopefully fixes this. > > Specifically https://patchwork.linuxtv.org/patch/52393/ and > https://patchwork.linuxtv.org/patch/52388/ Yeah, I've noticed it, thanks! Added to my list. Best regards, Tomasz
Re: s5p_mfc and H.264 frame cropping question
On Fri, Oct 5, 2018 at 3:58 PM Hans Verkuil wrote: > > On 10/05/2018 05:12 AM, Tomasz Figa wrote: > > Hi Hans, > > > > On Fri, Oct 5, 2018 at 5:02 AM Hans Verkuil wrote: > >> > >> Hi all, > >> > >> I'm looking at removing the last users of vidioc_g/s_crop from the driver > >> and > >> I came across vidioc_g_crop in > >> drivers/media/platform/s5p-mfc/s5p_mfc_dec.c. > >> > >> What this really does AFAICS is return the H.264 frame crop as read from > >> the > >> bitstream. This has nothing to do with regular cropping/composing but it > >> might be > >> something that could be implemented as a new selection target. > > > > It has a lot to do, because the output frame buffer may contain (and > > on the hardware I worked with, s5p-mfc and mtk-vcodec, indeed does) > > the whole encoded stream and the frame crop from the bitstream > > specifies the rectangle within it that corresponds to the part that > > should be displayed. > > Yes, but is that part actually cropped? Or is the full uncropped image DMAed > to the capture buffer? The latter. > > To take a practical example: a H.264 stream with a 1920x1088 image and a frame > crop rectangle of 1920x1080. What is the G_FMT width/height for the decoder > capture stream: 1920x1088 or 1920x1080? > > If it is 1920x1088, then you have a compose rectangle. If it is 1920x1080 then > you have a crop rectangle. 1920x1088 > > As far as I can tell from this driver it actually has a compose rectangle > and the use of g_crop is wrong and is there due to historical reasons (the > driver predates the selection API). Yes, it is there due to historical reasons. > > > > >> > >> I'm not really sure what to do with the existing code since it is an abuse > >> of > >> the crop API, but I guess the first step is to decide how this should be > >> handled > >> properly. > >> > >> Are there other decoders that can retrieve this information? Should this be > >> mentioned in the stateful codec API? > > > > coda [1], mtk-vcodec [2] and venus [3] expose this using the > > V4L2_SEL_TGT_COMPOSE selection target. v1 of the specification defines > > the selection targets in a way, which is compatible with that: > > V4L2_SEL_TGT_COMPOSE defaults to V4L2_SEL_TGT_COMPOSE_DEFAULT, which > > equals to V4L2_SEL_TGT_CROP, which defaults to > > V4L2_SEL_TGT_CROP_DEFAULT, which is defined as follows: > > > > + ``V4L2_SEL_TGT_CROP_DEFAULT`` > > + a rectangle covering the part of the frame buffer that contains > > + meaningful picture data (visible area); width and height will be > > + equal to visible resolution of the stream > > Where do you get that from? That's the crop definition for an output stream, > not a capture stream (assuming we have a codec). We're talking about a decoder here, right? In this case OUTPUT stream is just a sequence of bytes, not video frames, so there is no selection defined for OUTPUT queue. CAPTURE stream should be seen as a video grabber, so CROP targets relate to the encoded rectangle (1920x1088) and COMPOSE targets to the CAPTURE buffer. V4L2_SEL_TGT_COMPOSE would be the part of the CAPTURE buffer that is written with the image selected by V4L2_SEL_TGT_CROP. On a decoder that cannot do arbitrary crop and compose, like s5p-mfc, both targets would have identical rectangles, equal to the visible region (1920x1080). On hardware which can actually do fancier things, userspace could freely configure CAPTURE buffer width and height and V4L2_SEL_TGT_COMPOSE rectangle, for example to downscale the decoded video on the fly. Please check how I specified all the targets in last version of the specification (https://lore.kernel.org/patchwork/patch/966933/) and comment there, if there is anything that goes against the specification of the selection API. > > I kind of lost you with "which equals to V4L2_SEL_TGT_CROP". > > In any case, this particular driver should implement g_selection for > CAPTURE and implement the COMPOSE targets. That makes sense. Right. Best regards, Tomasz
Re: [RFC PATCH] media: v4l2-ctrl: Add control for specific V4L2_EVENT_SRC_CH_RESOLUTION support
On Wed, Oct 3, 2018 at 7:02 PM Maxime Jourdan wrote: > > On Tue, Oct 2, 2018 at 1:43 PM Hans Verkuil wrote: > > > > On 10/02/18 13:31, Maxime Jourdan wrote: > > > For drivers that expose both an OUTPUT queue and > > > V4L2_EVENT_SRC_CH_RESOLUTION such as video decoders, it is > > > possible that support for this event is limited to a subset > > > of the enumerated OUTPUT formats. > > > > > > This adds V4L2_CID_SUPPORTS_CH_RESOLUTION that allows such a driver to > > > notify userspace of per-format support for this event. > > > > An alternative is a flag returned by ENUMFMT. > > > > I would definitely invert the flag/control since the default should be > > that this event is supported for these types of devices. Instead you > > want to signal the exception (not supported). > > > > I think a format flag is better since this is really tied to the format > > for this particular driver. > > > > This is a limitation of this hardware/firmware, right? It is not a > > limitation of the format itself? > > > > And basically the decoder hardware cannot report when the resolution > > changes midstream? What happens if userspace does not take this into > > account and just continue? DMA overflow? Hope not. > > > > I tested it: yep, DMA overflow if you're not careful. A solution is to > always allocate buffers that can hold the maximum decodable picture > size. > > Upon further investigation, the firmwares for older codecs (MPEG2 & > MPEG4) expose registers to fetch the parsed width/height. > There is one issue however: the decoder starts decoding right away, > and we can gather the width/height only on the first frame decoded, > which is the first interrupt we get. > > So I can send out a V4L2_EVENT_SRC_CH_RESOLUTION, the last remaining > problem is that these codecs don't support dynamically changing the > capture buffers, so you can't do a streamoff/reqbufs/streamon on the > capture queue without doing a full reset and losing the pending > capture/output frames. I believe that, if there was a resolution change, you need to do a full reset anyway, before you can continue decoding. So it doesn't sound like a problem. Or am I missing something? > > And then there's MJPEG where you just can't gather the width/height > and must rely on userspace setting it. Also need to allocate max size > capture buffers for that one. I guess that could be a case for the JPEG_RAW format, which moves the responsibility for parsing the headers and setting relevant parameters (format, controls) to userspace. > > I end up needing two ENUMFMT flags: > - the format doesn't support V4L2_EVENT_SRC_CH_RESOLUTION > - the format doesn't support dynamically changing the capture buffers > (but you can keep going with the original buffer set) > > Would this be okay ? With my replies above, would you still need any of those? Best regards, Tomasz > > Regards, > Maxime > > > Regards, > > > > Hans > > > > > > > > RFC notes: This patch is motivated by the work I'm doing on the Amlogic > > > video decoder where the firmwares allow me to support > > > V4L2_EVENT_SRC_CH_RESOLUTION for newer formats (H.264, HEVC..) but > > > can't support it for older ones (MPEG2, MPEG4, MJPEG..). > > > For the latter formats, userspace is expected to set the resolution via > > > S_FMT prior to decoding. > > > > > > Signed-off-by: Maxime Jourdan > > > --- > > > Documentation/media/uapi/v4l/control.rst | 7 +++ > > > drivers/media/v4l2-core/v4l2-ctrls.c | 3 +++ > > > include/uapi/linux/v4l2-controls.h | 4 +++- > > > 3 files changed, 13 insertions(+), 1 deletion(-) > > > > > > diff --git a/Documentation/media/uapi/v4l/control.rst > > > b/Documentation/media/uapi/v4l/control.rst > > > index c1e6adbe83d7..029a4e88bfd5 100644 > > > --- a/Documentation/media/uapi/v4l/control.rst > > > +++ b/Documentation/media/uapi/v4l/control.rst > > > @@ -297,6 +297,13 @@ Control IDs > > > set the alpha component value of all pixels for further processing > > > in the device. > > > > > > +``V4L2_CID_SUPPORTS_CH_RESOLUTION`` ``(boolean)`` > > > +This is a read-only control that can be read by the application when > > > +the driver exposes an OUTPUT queue and event > > > +``V4L2_EVENT_SRC_CH_RESOLUTION`` but doesn't support it for every > > > +OUTPUT format. It returns true if the currently selected OUTPUT > > > format > > > +supports this event. > > > + > > > ``V4L2_CID_LASTP1`` > > > End of the predefined control IDs (currently > > > ``V4L2_CID_ALPHA_COMPONENT`` + 1). > > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c > > > b/drivers/media/v4l2-core/v4l2-ctrls.c > > > index 599c1cbff3b9..a8037ff3935a 100644 > > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c > > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c > > > @@ -739,6 +739,7 @@ const char *v4l2_ctrl_get_name(u32 id) > > > case V4L2_CID_MIN_BUFFERS_FOR_OUTPUT: return "Min Number of > > > Output Buffers"; > > > case V4L2_CID_ALPHA_COMPONENT:
Re: [PATCH 0/5] Add units to controls
On Tue, Sep 25, 2018 at 9:30 PM Sakari Ailus wrote: [snip] > On Tue, Sep 25, 2018 at 01:48:02PM +0200, Helmut Grohne wrote: > > On Tue, Sep 25, 2018 at 12:14:29PM +0200, Sakari Ailus wrote: [snip] > > > Regarding Ricardo's suggestion --- I was thinking of adding a control flag > > > (yes, there are a few bits available) to tell how to round the value. The > > > user could use the TRY_EXT_CTRLS IOCTL to figure out the next (or > > > previous) control value by incrementing the current value and setting the > > > appropriate flag. This is out of the scope of this set though. > > > > This approach sounds really useful to me. Having control over the > > rounding would allow reading supported control values with reasonable > > effort. With such an approach, a very sparsely populated control becomes > > feasible and with integer64 controls that'd likely allow representing > > most exponential controls with linear values. If going this route, I > > don't see an application of V4L2_CTRL_FLAG_EXPONENTIAL. > > Yes, I think the flag can be dropped as I suggested. Wouldn't that be just a duplicate of menu controls? Integer controls are supposed to be described by min, max and step and any value matching those should be valid. Rather than introducing such tricky semantics, perhaps it would make more sense to allow controls to be of different type, depending on the driver? A driver that supports a contiguous range, would report the control as INTEGER, while one that doesn't, would report it as INTEGER_MENU. Putting that aside, V4L2_CTRL_FLAG_EXPONENTIAL would actually make it easier to enumerate the supported values to the userspace. Just one QUERYCTRL would be needed, instead of 1 ioctl for each possible value. (Although for the exponential case I wouldn't expect too many values indeed...) > > > > > Thus, I think that control over the rounding is tightly related to this > > patchset and needs to be discussed together. > > It addresses some of the same problem area but the implementation is > orthogonal to this. > > Providing that would probably make the base field less useful: the valid > control values could be enumerated by the user using TRY_EXT_CTRLS without > the need to tell the valid values are powers of e.g. two. > > I don't really have a strong opinion on that actually when it comes to the > API itself. The imx208 driver could proceed to use linear relation between > the control value and the digital gain. My worry is just the driver > implementation: this may not be entirely trivial. There's still no way to > address this problem in a generic way otherwise. What's not trivial for the imx208 driver? It just registers an integer control with a range from 0 to 4 and takes (1 << ctrl->val) as the value for the hardware. Best regards, Tomasz
Re: [RFP] Which V4L2 ioctls could be replaced by better versions?
On Fri, Sep 21, 2018 at 3:14 AM Nicolas Dufresne wrote: > > Le jeudi 20 septembre 2018 à 16:42 +0200, Hans Verkuil a écrit : > > Some parts of the V4L2 API are awkward to use and I think it would be > > a good idea to look at possible candidates for that. > > > > Examples are the ioctls that use struct v4l2_buffer: the multiplanar > > support is > > really horrible, and writing code to support both single and multiplanar is > > hard. > > We are also running out of fields and the timeval isn't y2038 compliant. > > > > A proof-of-concept is here: > > > > https://git.linuxtv.org/hverkuil/media_tree.git/commit/?h=v4l2-buffer=a95549df06d9900f3559afdbb9da06bd4b22d1f3 > > > > It's a bit old, but it gives a good impression of what I have in mind. > > > > Another candidate is > > VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL/VIDIOC_ENUM_FRAMEINTERVALS: > > expressing frame intervals as a fraction is really awkward and so is the > > fact > > that the subdev and 'normal' ioctls are not the same. > > > > Would using nanoseconds or something along those lines for intervals be > > better? > > This one is not a good idea, because you cannot represent well known > rates used a lot in the field. Like 6/1001 also known as 59.94 Hz. > You could endup with drift issues. > > For me, what is the most difficult with this API is the fact that it > uses frame internal (duration) instead of frame rate. > > > > > I have similar concerns with VIDIOC_SUBDEV_ENUM_FRAME_SIZE where there is no > > stepwise option, making it different from VIDIOC_ENUM_FRAMESIZES. But it > > should > > be possible to extend VIDIOC_SUBDEV_ENUM_FRAME_SIZE with stepwise support, I > > think. > > One of the thing to fix, maybe it's doable now, is the differentiation > between allocation size and display size. Pretty much all video capture > code assumes this is display size and ignores the selection API. This > should be documented explicitly. Technically, there is already a distinction between allocation and display size at format level - width and height could be interpreted as the rectangle containing the captured video, while bytesperline and sizeimage would match to the allocation size. The behavior between drivers seems to be extremely variable - some would just enforce bytesperline = bpp*width and sizeimage = bytesperline*height, while others would actually give control over them to the user space. It's a bit more complicated with video decoders, because the stream, in addition to the 2 sizes mentioned before, is also characterized by coded size, which corresponds to the actual number of pixels encoded in the stream, even if not all contain pixels to be displayed. This is something that needs to be specified explicitly (and is, in my documentation patches) in the Codec Interface. > > In fact, the display/allocation dimension isn't very nice, as both > information overlaps in structures. As an example, you call S_FMT with > a display size you want, and it will return you an allocation size > (which yes, can be smaller, because we always round to the middle). > > > > > Do we have more ioctls that could use a refresh? S/G/TRY_FMT perhaps, again > > in > > order to improve single vs multiplanar handling. > > Yes, but that would fall in a complete redesign I guess. The buffer > allocation scheme is very inflexible. You can't have buffers of two > dimensions allocated at the same time for the same queue. Worst, you > cannot leave even 1 buffer as your scannout buffer while reallocating > new buffers, this is not permitted by the framework (in software). As a > side effect, there is no way to optimize the resolution changes, you > even have to copy your scannout buffer on the CPU, to free it in order > to proceed. Resolution changes are thus painfully slow, by design. Hmm? There is VIDIOC_CREATEBUFS, which can allows you to allocate buffers for explicitly specified format, with other buffers already existing in the queue. Also, I fail to understand the scanout issue. If one exports a vb2 buffer to a DMA-buf and import it to the scanout engine, it can keep scanning out from it as long as it want, because the DMA-buf will hold a reference on the buffer, even if it's removed from the vb2 queue. > > You also cannot switch from internal buffers to importing buffers > easily (in some case you, like encoder, you cannot do that without > flushing the encoder state). Hmm, technically VIDIOC_CREATEBUFS accepts the "memory" type value, but I'm not sure what happens if the queue already has buffers requested with different one. Best regards, Tomasz
Re: [RFC PATCH] media: v4l2-ctrl: Add control for specific V4L2_EVENT_SRC_CH_RESOLUTION support
On Tue, Oct 2, 2018 at 8:44 PM Hans Verkuil wrote: > > On 10/02/18 13:31, Maxime Jourdan wrote: > > For drivers that expose both an OUTPUT queue and > > V4L2_EVENT_SRC_CH_RESOLUTION such as video decoders, it is > > possible that support for this event is limited to a subset > > of the enumerated OUTPUT formats. > > > > This adds V4L2_CID_SUPPORTS_CH_RESOLUTION that allows such a driver to > > notify userspace of per-format support for this event. > > An alternative is a flag returned by ENUMFMT. > > I would definitely invert the flag/control since the default should be > that this event is supported for these types of devices. Instead you > want to signal the exception (not supported). > > I think a format flag is better since this is really tied to the format > for this particular driver. +1 for format flag. It will also make it easier for userspace to enumerate the capabilities. For example, userspace that doesn't want to handle resolution changes on its own (e.g. Chromium), could just discard the formats for which the hardware/firmware doesn't handle them. Best regards, Tomasz
Re: [PATCH v5] media: add imx319 camera sensor driver
On Fri, Sep 21, 2018 at 11:28 AM Bing Bu Cao wrote:
>
>
> On 09/18/2018 05:49 PM, Tomasz Figa wrote:
> > Hi Bingbu,
> >
> > On Mon, Sep 17, 2018 at 2:53 PM wrote:
> >> From: Bingbu Cao
> >>
> >> Add a v4l2 sub-device driver for the Sony imx319 image sensor.
> >> This is a camera sensor using the i2c bus for control and the
> >> csi-2 bus for data.
> > Please see my comments inline. Also, I'd appreciate being CCed on
> > related work in the future.
> Ack.
> Sorry, will add you into the cc-list.
> >
> > [snip]
> >> +
> >> +static const char * const imx319_test_pattern_menu[] = {
> >> + "Disabled",
> >> + "100% color bars",
> >> + "Solid color",
> >> + "Fade to gray color bars",
> >> + "PN9"
> >> +};
> >> +
> >> +static const int imx319_test_pattern_val[] = {
> >> + IMX319_TEST_PATTERN_DISABLED,
> >> + IMX319_TEST_PATTERN_COLOR_BARS,
> >> + IMX319_TEST_PATTERN_SOLID_COLOR,
> >> + IMX319_TEST_PATTERN_GRAY_COLOR_BARS,
> >> + IMX319_TEST_PATTERN_PN9,
> >> +};
> > This array is not needed. All the entries are equal to corresponding
> > indices, i.e. the array is equivalent to { 0, 1, 2, 3, 4 }. We can use
> > ctrl->val directly.
> Ack.
> > [snip]
> >
> >> +/* Write a list of registers */
> >> +static int imx319_write_regs(struct imx319 *imx319,
> >> + const struct imx319_reg *regs, u32 len)
> >> +{
> >> + struct i2c_client *client = v4l2_get_subdevdata(>sd);
> >> + int ret;
> >> + u32 i;
> >> +
> >> + for (i = 0; i < len; i++) {
> >> + ret = imx319_write_reg(imx319, regs[i].address, 1,
> >> regs[i].val);
> >> + if (ret) {
> >> + dev_err_ratelimited(>dev,
> >> +
> > Hmm, the message is clipped here. Let me see if it's something with my
> > email client...
> The code here:
>
> 1827 for (i = 0; i < len; i++) {
> 1828 ret = imx319_write_reg(imx319, regs[i].address, 1, regs[i].val);
> 1829 if (ret) {
> 1830 dev_err_ratelimited(>dev,
> 1831 "write reg 0x%4.4x return err %d",
> 1832 regs[i].address, ret);
> 1833 return ret;
> 1834 }
> 1835 } Same as the code shown on your client?
That was an issue with my email client, which showed only lines until
1831. I've worked around it and reviewed rest of the code in next
reply. Sorry for the noise.
Best regards,
Tomasz
Re: [PATCH v5] media: imx208: Add imx208 camera sensor driver
[+Laurent and Sylwester]
On Wed, Aug 8, 2018 at 4:08 PM Ping-chung Chen
wrote:
[snip]
> +
> +/* Digital gain control */
> +#define IMX208_REG_GR_DIGITAL_GAIN 0x020e
> +#define IMX208_REG_R_DIGITAL_GAIN 0x0210
> +#define IMX208_REG_B_DIGITAL_GAIN 0x0212
> +#define IMX208_REG_GB_DIGITAL_GAIN 0x0214
> +#define IMX208_DGTL_GAIN_MIN 0
> +#define IMX208_DGTL_GAIN_MAX 4096
> +#define IMX208_DGTL_GAIN_DEFAULT 0x100
> +#define IMX208_DGTL_GAIN_STEP 1
> +
[snip]
> +/* Initialize control handlers */
> +static int imx208_init_controls(struct imx208 *imx208)
> +{
[snip]
> + v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops, V4L2_CID_DIGITAL_GAIN,
> + IMX208_DGTL_GAIN_MIN, IMX208_DGTL_GAIN_MAX,
> + IMX208_DGTL_GAIN_STEP,
> + IMX208_DGTL_GAIN_DEFAULT);
We have a problem here. The sensor supports only a discrete range of
values here - {1, 2, 4, 8, 16} (multiplied by 256, since the value is
fixed point). This makes it possible for the userspace to set values
that are not allowed by the sensor specification and also leaves no
way to enumerate the supported values.
I can see two solutions here:
1) Define the control range from 0 to 4 and treat it as an exponent of
2, so that the value for the sensor becomes (1 << val) * 256.
(Suggested by Sakari offline.)
This approach has the problem of losing the original unit (and scale)
of the value.
2) Use an integer menu control, which reports only the supported
discrete values - {1, 2, 4, 8, 16}.
With this approach, userspace can enumerate the real gain values, but
we would either need to introduce a new control (e.g.
V4L2_CID_DIGITAL_GAIN_DISCRETE) or abuse the specification and
register V4L2_CID_DIGITAL_GAIN as an integer menu.
Any opinions or better ideas?
Best regards,
Tomasz
Re: [PATCH v5 5/6] media: Add controls for JPEG quantization tables
On Thu, Sep 13, 2018 at 9:15 PM Paul Kocialkowski wrote: > > Hi, > > On Wed, 2018-09-05 at 19:00 -0300, Ezequiel Garcia wrote: > > From: Shunqian Zheng > > > > Add V4L2_CID_JPEG_QUANTIZATION compound control to allow userspace > > configure the JPEG quantization tables. > > > > Signed-off-by: Shunqian Zheng > > Signed-off-by: Ezequiel Garcia > > --- > > .../media/uapi/v4l/extended-controls.rst | 31 +++ > > .../media/videodev2.h.rst.exceptions | 1 + > > drivers/media/v4l2-core/v4l2-ctrls.c | 10 ++ > > include/uapi/linux/v4l2-controls.h| 12 +++ > > include/uapi/linux/videodev2.h| 1 + > > 5 files changed, 55 insertions(+) > > > > diff --git a/Documentation/media/uapi/v4l/extended-controls.rst > > b/Documentation/media/uapi/v4l/extended-controls.rst > > index 9f7312bf3365..1335d27d30f3 100644 > > --- a/Documentation/media/uapi/v4l/extended-controls.rst > > +++ b/Documentation/media/uapi/v4l/extended-controls.rst > > @@ -3354,7 +3354,38 @@ JPEG Control IDs > > Specify which JPEG markers are included in compressed stream. This > > control is valid only for encoders. > > > > +.. _jpeg-quant-tables-control: > > I just had a look at how the Allwinner VPU handles JPEG decoding and it > seems to require the following information (in addition to > quantization): I assume the hardware doesn't have the ability to parse those from the stream and so they need to be parsed by user space and given to the driver? > > * Horizontal and vertical sampling factors for each Y/U/V component: > > The number of components and sampling factors are coded separately in > the bitstream, but it's probably easier to use the already-existing > V4L2_CID_JPEG_CHROMA_SUBSAMPLING control for specifying that. > > However, this is potentially very much related to the destination > format. If we decide that this format should match the format resulting > from decompression, we don't need to specify it through an external > control. On the other hand, it's possible that the VPU has format > conversion block integrated in its pipeline so it would also make sense > to consider the destination format as independent. +1 for keeping it separate. > > * Custom Huffman tables (DC and AC), both for luma and chroma > > It seems that there is a default table when no Huffman table is provided > in the bitstream (I'm not too sure how standard that is, just started > learning about JPEG). We probably need a specific control for that. What happens if there is one in the bitstream? Would the hardware pick it automatically? I think it might make sense to just have a general control for Huffman table, which would be always provided by the user space, regardless of whether it's parsed from the stream or default, so that drivers don't have to care and could just always use it. Best regards, Tomasz
Re: [PATCH v5 5/6] media: Add controls for JPEG quantization tables
On Sun, Sep 16, 2018 at 1:48 AM Paul Kocialkowski wrote: > > Hi, > > On Mon, 2018-09-10 at 10:25 -0300, Ezequiel Garcia wrote: > > Hi Hans, > > > > Thanks for the review. > > > > On Mon, 2018-09-10 at 14:42 +0200, Hans Verkuil wrote: > > > On 09/06/2018 12:00 AM, Ezequiel Garcia wrote: > > > > From: Shunqian Zheng > > > > > > > > Add V4L2_CID_JPEG_QUANTIZATION compound control to allow userspace > > > > configure the JPEG quantization tables. > > > > > > > > Signed-off-by: Shunqian Zheng > > > > Signed-off-by: Ezequiel Garcia > > > > --- > > > > .../media/uapi/v4l/extended-controls.rst | 31 +++ > > > > .../media/videodev2.h.rst.exceptions | 1 + > > > > drivers/media/v4l2-core/v4l2-ctrls.c | 10 ++ > > > > include/uapi/linux/v4l2-controls.h| 12 +++ > > > > include/uapi/linux/videodev2.h| 1 + > > > > 5 files changed, 55 insertions(+) > > > > > > > > diff --git a/Documentation/media/uapi/v4l/extended-controls.rst > > > > b/Documentation/media/uapi/v4l/extended-controls.rst > > > > index 9f7312bf3365..1335d27d30f3 100644 > > > > --- a/Documentation/media/uapi/v4l/extended-controls.rst > > > > +++ b/Documentation/media/uapi/v4l/extended-controls.rst > > > > @@ -3354,7 +3354,38 @@ JPEG Control IDs > > > > Specify which JPEG markers are included in compressed stream. This > > > > control is valid only for encoders. > > > > > > > > +.. _jpeg-quant-tables-control: > > > > > > > > +``V4L2_CID_JPEG_QUANTIZATION (struct)`` > > > > +Specifies the luma and chroma quantization matrices for encoding > > > > +or decoding a V4L2_PIX_FMT_JPEG_RAW format buffer. The > > > > :ref:`itu-t81` > > > > +specification allows 8-bit quantization coefficients for > > > > +baseline profile images, and 8-bit or 16-bit for extended profile > > > > +images. Supporting or not 16-bit precision coefficients is > > > > driver-specific. > > > > +Coefficients must be set in JPEG zigzag scan order. > > > > + > > > > + > > > > +.. c:type:: struct v4l2_ctrl_jpeg_quantization > > > > + > > > > +.. cssclass:: longtable > > > > + > > > > +.. flat-table:: struct v4l2_ctrl_jpeg_quantization > > > > +:header-rows: 0 > > > > +:stub-columns: 0 > > > > +:widths: 1 1 2 > > > > + > > > > +* - __u8 > > > > + - ``precision`` > > > > + - Specifies the coefficient precision. User shall set 0 > > > > +for 8-bit, and 1 for 16-bit. > > > > > > So does specifying 1 here switch the HW encoder to use extended profile? > > > What if the HW only supports baseline? The rockchip driver doesn't appear > > > to check the precision field at all... > > > > > > > The driver is missing to check that, when the user sets this control. > > > > > I think this needs a bit more thought. > > > > > > I am not at all sure that this is the right place for the precision field. > > > This is really about JPEG profiles, so I would kind of expect a JPEG > > > PROFILE > > > control (just like other codec profiles), or possibly a new pixelformat > > > for > > > extended profiles. > > > > > > And based on that the driver would interpret these matrix values as 8 or > > > 16 bits. > > > > > > > Right, the JPEG profile control is definitely needed. I haven't add it > > because > > it wouldn't be used, since this VPU can only do baseline. > > Well, I suppose it would still be relevant that you add it for the > encoder and only report baseline there. > > > However, the problem is that some JPEGs in the wild have with 8-bit data and > > 16-bit quantization coefficients, as per [1] and [2]: > > > > [1] https://github.com/martinhath/jpeg-rust/issues/1 > > [2] https://github.com/libjpeg-turbo/libjpeg-turbo/pull/90 > > > > So, in order to support decoding of these images, I've added the precision > > field to the quantization control. The user would be able to set a baseline > > or extended profile thru a (future) profile control, and if 16-bit > > tables are found, and if the hardware supports them, the driver > > would be able to support them. > > > > Another option, which might be even better, is have explicit baseline > > and extended quantization tables controls, e.g.: V4L2_CID_JPEG_QUANT > > and V4L2_CID_JPEG_EXT_QUANT. > > I think this makes more sense than a common structure with an indication > bit on how to interpret the data. > > However, it seems problematic that userspace can't figure out whether > 16-bit quant tables are supported with a baseline profile and just has > to try and see. > > Hans, do you think this is an acceptable approach or should we rather > stick to the standard here, at the cost of not supporting these pictures > that were encoded with this common abuse of the standard? Perhaps we just need a control called V4L2_CID_JPEG_QUANT_PRECISION, where drivers can set the min/max (e.g. min = 8, max = 16, step = 8) to the range they support and user space can select the precision it wants, if the hardware gives a
Re: [PATCH v5] media: add imx319 camera sensor driver
Hi Bingbu,
On Mon, Sep 17, 2018 at 2:53 PM wrote:
[snip]
> +static int imx319_update_digital_gain(struct imx319 *imx319, u32 d_gain)
> +{
> + int ret;
> +
> + ret = imx319_write_reg(imx319, IMX319_REG_DPGA_USE_GLOBAL_GAIN, 1, 1);
> + if (ret)
> + return ret;
> +
> + /* Digital gain = (d_gain & 0xFF00) + (d_gain & 0xFF)/256 times */
What's the unit here?
Is the equation above really correct? The range, besides ~0, would be
from 256.0 to 65280 + 255/256, which sounds strange.
> + return imx319_write_reg(imx319, IMX319_REG_DIG_GAIN_GLOBAL, 2,
> d_gain);
> +}
> +
> +static int imx319_set_ctrl(struct v4l2_ctrl *ctrl)
> +{
> + struct imx319 *imx319 = container_of(ctrl->handler,
> +struct imx319, ctrl_handler);
> + struct i2c_client *client = v4l2_get_subdevdata(>sd);
> + s64 max;
> + int ret;
> +
> + /* Propagate change of current control to all related controls */
> + switch (ctrl->id) {
> + case V4L2_CID_VBLANK:
> + /* Update max exposure while meeting expected vblanking */
> + max = imx319->cur_mode->height + ctrl->val - 18;
> + __v4l2_ctrl_modify_range(imx319->exposure,
> +imx319->exposure->minimum,
> +max, imx319->exposure->step, max);
> + break;
> + }
> +
> + /*
> +* Applying V4L2 control value only happens
> +* when power is up for streaming
> +*/
> + if (pm_runtime_get_if_in_use(>dev) == 0)
nit: if (!pm_runtime_get_if_in_use(>dev)
> + return 0;
> +
[snip]
> +/* Initialize control handlers */
> +static int imx319_init_controls(struct imx319 *imx319)
> +{
> + struct i2c_client *client = v4l2_get_subdevdata(>sd);
> + struct v4l2_ctrl_handler *ctrl_hdlr;
> + s64 exposure_max;
> + s64 vblank_def;
> + s64 vblank_min;
> + s64 hblank;
> + s64 pixel_rate;
> + const struct imx319_mode *mode;
> + int ret;
> +
> + ctrl_hdlr = >ctrl_handler;
> + ret = v4l2_ctrl_handler_init(ctrl_hdlr, 10);
> + if (ret)
> + return ret;
> +
> + ctrl_hdlr->lock = >mutex;
> + imx319->link_freq = v4l2_ctrl_new_int_menu(ctrl_hdlr,
> _ctrl_ops,
> + V4L2_CID_LINK_FREQ, 0, 0,
> + imx319->pdata->link_freqs);
> + if (imx319->link_freq)
> + imx319->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> +
> + /* pixel_rate = link_freq * 2 * nr_of_lanes / bits_per_sample */
> + pixel_rate = (imx319->link_def_freq * 2 * 4) / 10;
> + /* By default, PIXEL_RATE is read only */
> + imx319->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops,
> + V4L2_CID_PIXEL_RATE,
> pixel_rate,
> + pixel_rate, 1, pixel_rate);
> +
> + /* Initialze vblank/hblank/exposure parameters based on current mode
> */
> + mode = imx319->cur_mode;
> + vblank_def = mode->fll_def - mode->height;
> + vblank_min = mode->fll_min - mode->height;
> + imx319->vblank = v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops,
> + V4L2_CID_VBLANK, vblank_min,
> + IMX319_FLL_MAX - mode->height,
> + 1, vblank_def);
> +
> + hblank = mode->llp - mode->width;
> + imx319->hblank = v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops,
> + V4L2_CID_HBLANK, hblank, hblank,
> + 1, hblank);
> + if (imx319->hblank)
> + imx319->hblank->flags |= V4L2_CTRL_FLAG_READ_ONLY;
> +
> + exposure_max = mode->fll_def - 18;
> + imx319->exposure = v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops,
> +V4L2_CID_EXPOSURE,
> +IMX319_EXPOSURE_MIN,
> exposure_max,
> +IMX319_EXPOSURE_STEP,
> +IMX319_EXPOSURE_DEFAULT);
Please explain how to interpret the exposure value in a comment.
> +
> + imx319->hflip = v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops,
> + V4L2_CID_HFLIP, 0, 1, 1, 0);
> + imx319->vflip = v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops,
> + V4L2_CID_VFLIP, 0, 1, 1, 0);
> +
> + v4l2_ctrl_new_std(ctrl_hdlr, _ctrl_ops, V4L2_CID_ANALOGUE_GAIN,
> + IMX319_ANA_GAIN_MIN, IMX319_ANA_GAIN_MAX,
> + IMX319_ANA_GAIN_STEP, IMX319_ANA_GAIN_DEFAULT);
Please explain how the gain value and in what units is calculated in a comment.
> +
> +
Re: [PATCH v6 12/12] intel-ipu3: Add imgu top level pci device driver
On Mon, Sep 17, 2018 at 5:20 AM Zhi, Yong wrote:
>
> Hi, Tomasz,
>
> Thanks for the code review.
>
> > -Original Message-
> > From: linux-media-ow...@vger.kernel.org [mailto:linux-media-
> > ow...@vger.kernel.org] On Behalf Of Tomasz Figa
> > Sent: Monday, July 2, 2018 3:08 AM
> > To: Zhi, Yong
> > Cc: Linux Media Mailing List ; Sakari Ailus
> > ; Mani, Rajmohan
> > ; Toivonen, Tuukka
> > ; Hu, Jerry W ; Zheng,
> > Jian Xu
> > Subject: Re: [PATCH v6 12/12] intel-ipu3: Add imgu top level pci device
> > driver
> >
> > Hi Yong,
> >
> > On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote:
> > > +/*
> > > + * Queue as many buffers to CSS as possible. If all buffers don't fit
> > > +into
> > > + * CSS buffer queues, they remain unqueued and will be queued later.
> > > + */
> > > +int imgu_queue_buffers(struct imgu_device *imgu, bool initial) {
> > > + unsigned int node;
> > > + int r = 0;
> > > + struct imgu_buffer *ibuf;
> > > +
> > > + if (!ipu3_css_is_streaming(>css))
> > > + return 0;
> > > +
> > > + mutex_lock(>lock);
> > > +
> > > + /* Buffer set is queued to FW only when input buffer is ready */
> > > + if (!imgu_queue_getbuf(imgu, IMGU_NODE_IN)) {
> > > + mutex_unlock(>lock);
> > > + return 0;
> > > + }
> > > + for (node = IMGU_NODE_IN + 1; 1; node = (node + 1) %
> > > + IMGU_NODE_NUM) {
> >
> > Shouldn't we make (node != IMGU_NODE_IN || imgu_queue_getbuf(imgu,
> > IMGU_NODE_IN)) the condition here, rather than 1?
> >
> > This would also let us remove the explicit call to imgu_queue_getbuf()
> > above the loop.
> >
>
> Ack, will make the suggested changes regarding the loop condition evaluation.
Just to make sure, the suggestion also includes starting from
IMGU_NODE_IN (not + 1), i.e.
for (node = IMGU_NODE_IN;
node != IMGU_NODE_IN || imgu_queue_getbuf(imgu, IMGU_NODE_IN);
node = (node + 1) % IMGU_NODE_NUM) {
// ...
}
> > > +static int __maybe_unused imgu_suspend(struct device *dev) {
> > > + struct pci_dev *pci_dev = to_pci_dev(dev);
> > > + struct imgu_device *imgu = pci_get_drvdata(pci_dev);
> > > + unsigned long expire;
> > > +
> > > + dev_dbg(dev, "enter %s\n", __func__);
> > > + imgu->suspend_in_stream = ipu3_css_is_streaming(>css);
> > > + if (!imgu->suspend_in_stream)
> > > + goto out;
> > > + /* Block new buffers to be queued to CSS. */
> > > + atomic_set(>qbuf_barrier, 1);
> > > + /*
> > > +* Wait for currently running irq handler to be done so that
> > > +* no new buffers will be queued to fw later.
> > > +*/
> > > + synchronize_irq(pci_dev->irq);
> > > + /* Wait until all buffers in CSS are done. */
> > > + expire = jiffies + msecs_to_jiffies(1000);
> > > + while (!ipu3_css_queue_empty(>css)) {
> > > + if (time_is_before_jiffies(expire)) {
> > > + dev_err(dev, "wait buffer drain timeout.\n");
> > > + break;
> > > + }
> > > + }
> >
> > Uhm. We struggle to save some power by suspending the device only to
> > end up with an ugly busy wait that could take even a second here. This
> > doesn't make any sense.
> >
> > We had a working solution using a wait queue in previous revision [1].
> > What happened to it?
> >
> > [1] https://chromium-
> > review.googlesource.com/c/chromiumos/third_party/kernel/+/1029594/2
> > /drivers/media/pci/intel/ipu3/ipu3.c#b913
> > (see the left side)
> >
>
> The code here was based on an old version of patch "ipu3-imgu: Avoid might
> sleep operations in suspend callback" at submission, so it did have
> buf_drain_wq, sorry for the confusion.
>
I guess that means that v7 is going to have the workqueue back? :)
Best regards,
Tomasz
Re: [PATCH v6 11/12] intel-ipu3: Add v4l2 driver based on media framework
+Hans Verkuil
(I think he commented on earlier revisions. Please keep anyone who
commented before on CC when sending further revisions.)
On Mon, Sep 17, 2018 at 5:04 AM Zhi, Yong wrote:
>
> Hi, Tomasz,
>
> Sorry for the delay in responding to your review.
>
> > -Original Message-
> > From: linux-media-ow...@vger.kernel.org [mailto:linux-media-
> > ow...@vger.kernel.org] On Behalf Of Tomasz Figa
> > Sent: Monday, July 2, 2018 2:50 AM
> > To: Zhi, Yong
> > Cc: Linux Media Mailing List ; Sakari Ailus
> > ; Mani, Rajmohan
> > ; Toivonen, Tuukka
> > ; Hu, Jerry W ; Zheng,
> > Jian Xu ; Vijaykumar, Ramya
> >
> > Subject: Re: [PATCH v6 11/12] intel-ipu3: Add v4l2 driver based on media
> > framework
> >
> > Hi Yong,
> >
> > On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote:
> > [snip]
> > > +static int ipu3_vidioc_enum_input(struct file *file, void *fh,
> > > + struct v4l2_input *input) {
> > > + if (input->index > 0)
> > > + return -EINVAL;
> > > + strlcpy(input->name, "camera", sizeof(input->name));
> > > + input->type = V4L2_INPUT_TYPE_CAMERA;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int ipu3_vidioc_g_input(struct file *file, void *fh, unsigned
> > > +int *input) {
> > > + *input = 0;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int ipu3_vidioc_s_input(struct file *file, void *fh, unsigned
> > > +int input) {
> > > + return input == 0 ? 0 : -EINVAL; }
> > > +
> > > +static int ipu3_vidioc_enum_output(struct file *file, void *fh,
> > > + struct v4l2_output *output) {
> > > + if (output->index > 0)
> > > + return -EINVAL;
> > > + strlcpy(output->name, "camera", sizeof(output->name));
> > > + output->type = V4L2_INPUT_TYPE_CAMERA;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int ipu3_vidioc_g_output(struct file *file, void *fh,
> > > + unsigned int *output) {
> > > + *output = 0;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static int ipu3_vidioc_s_output(struct file *file, void *fh,
> > > + unsigned int output) {
> > > + return output == 0 ? 0 : -EINVAL; }
> >
> > Do we really need to implement the 6 functions above? They don't seem to
> > be doing anything useful.
> >
>
> They are here to pass v4l2-compliance test. I can add a note in next update
> for their purpose. We can remove them in the future when defaults callbacks
> are available for those ops.
>
Strange.
Hans, is it really mandatory to implement dmmy output/input setting if
there is no output/input switching capability?
Best regards,
Tomasz
Re: [PATCH v5] media: add imx319 camera sensor driver
Hi Bingbu,
On Mon, Sep 17, 2018 at 2:53 PM wrote:
>
> From: Bingbu Cao
>
> Add a v4l2 sub-device driver for the Sony imx319 image sensor.
> This is a camera sensor using the i2c bus for control and the
> csi-2 bus for data.
Please see my comments inline. Also, I'd appreciate being CCed on
related work in the future.
[snip]
> +
> +static const char * const imx319_test_pattern_menu[] = {
> + "Disabled",
> + "100% color bars",
> + "Solid color",
> + "Fade to gray color bars",
> + "PN9"
> +};
> +
> +static const int imx319_test_pattern_val[] = {
> + IMX319_TEST_PATTERN_DISABLED,
> + IMX319_TEST_PATTERN_COLOR_BARS,
> + IMX319_TEST_PATTERN_SOLID_COLOR,
> + IMX319_TEST_PATTERN_GRAY_COLOR_BARS,
> + IMX319_TEST_PATTERN_PN9,
> +};
This array is not needed. All the entries are equal to corresponding
indices, i.e. the array is equivalent to { 0, 1, 2, 3, 4 }. We can use
ctrl->val directly.
[snip]
> +/* Write a list of registers */
> +static int imx319_write_regs(struct imx319 *imx319,
> + const struct imx319_reg *regs, u32 len)
> +{
> + struct i2c_client *client = v4l2_get_subdevdata(>sd);
> + int ret;
> + u32 i;
> +
> + for (i = 0; i < len; i++) {
> + ret = imx319_write_reg(imx319, regs[i].address, 1,
> regs[i].val);
> + if (ret) {
> + dev_err_ratelimited(>dev,
> +
Hmm, the message is clipped here. Let me see if it's something with my
email client...
Best regards,
Tomasz
Re: [PATCHv4 09/10] media-request: EPERM -> EACCES/EBUSY
On Tue, Sep 4, 2018 at 4:59 PM Hans Verkuil wrote: > > From: Hans Verkuil > > If requests are not supported by the driver, then return EACCES, not > EPERM. > > If you attempt to mix queueing buffers directly and using requests, > then EBUSY is returned instead of EPERM: once a specific queueing mode > has been chosen the queue is 'busy' if you attempt the other mode > (i.e. direct queueing vs via a request). > > Signed-off-by: Hans Verkuil > --- > .../uapi/mediactl/media-request-ioc-queue.rst | 9 - > .../media/uapi/mediactl/request-api.rst| 4 ++-- > Documentation/media/uapi/v4l/buffer.rst| 2 +- > .../media/uapi/v4l/vidioc-g-ext-ctrls.rst | 9 - > Documentation/media/uapi/v4l/vidioc-qbuf.rst | 18 ++ > .../media/common/videobuf2/videobuf2-core.c| 2 +- > .../media/common/videobuf2/videobuf2-v4l2.c| 9 ++--- > drivers/media/media-request.c | 4 ++-- > include/media/media-request.h | 6 +++--- > 9 files changed, 33 insertions(+), 30 deletions(-) [snip] > diff --git a/include/media/media-request.h b/include/media/media-request.h > index d8c8db89dbde..0ce75c35131f 100644 > --- a/include/media/media-request.h > +++ b/include/media/media-request.h > @@ -198,8 +198,8 @@ void media_request_put(struct media_request *req); > * Get the request represented by @request_fd that is owned > * by the media device. > * > - * Return a -EPERM error pointer if requests are not supported > - * by this driver. Return -ENOENT if the request was not found. > + * Return a -EACCES error pointer if requests are not supported > + * by this driver. Return -EINVAL if the request was not found. I think the bottom-most line belongs to patch 1/10. With that fixed (possibly when applying): Reviewed-by: Tomasz Figa Best regards, Tomasz
Re: [PATCHv2 00/10] Post-v18: Request API updates
On Tue, Aug 28, 2018 at 10:49 PM Hans Verkuil wrote:
>
> From: Hans Verkuil
>
> Hi all,
>
> This patch series sits on top of my v18 series for the Request API.
> It makes some final (?) changes as discussed in:
>
> https://www.mail-archive.com/linux-media@vger.kernel.org/msg134419.html
>
> and:
>
> https://www.spinics.net/lists/linux-media/msg138596.html
>
> The combined v18 patches + this series is available here:
>
> https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=reqv18-1
>
> Updated v4l-utils for this is available here:
>
> https://git.linuxtv.org/hverkuil/v4l-utils.git/log/?h=request
>
> Userspace visible changes:
>
> - Invalid request_fd values now return -EINVAL instead of -ENOENT.
> - It is no longer possible to use VIDIOC_G_EXT_CTRLS for requests
> that are not completed. -EACCES is returned in that case.
> - Attempting to use requests if requests are not supported by the driver
> will result in -EACCES instead of -EPERM.
>
> Driver visible changes (important for the cedrus driver!):
>
> Drivers should set the new vb2_queue 'supports_request' bitfield to 1
> if a vb2_queue can support requests. Otherwise the queue cannot be
> used with requests.
>
> This bitfield is also used to fill in the new capabilities field
> in struct v4l2_requestbuffers and v4l2_create_buffers.
>
> Changes since v1:
>
> - Updated patch 4/10 to explain how to query the capabilities
> with REQBUFS/CREATE_BUFS with a minimum of side-effects
> (requested by Tomasz).
> - Added patches 6-10:
> 6: Sakari found a corner case: when accessing a request the
> request has to be protected from being re-inited. New
> media_request_(un)lock_for_access helpers are added for this.
> 7: use these helpers in g_ext_ctrls.
> 8: make s/try_ext_ctrls more robust by keeping the request
> references until we're fully done setting/trying the controls.
> 9: Change two more EPERM's to EACCES. EPERM suggests that you can
> fix it by changing permissions somehow, but in this case the
> driver simply doesn't support requests at all.
> 10: Update the request documentation based on Laurent's comments:
> https://www.spinics.net/lists/linux-media/msg139152.html
> To do: split off the V4L2 specifics into a V4L2 specific
> rst file. But this will take more time and is for later.
For all the patches which still don't have my Reviewed-by, except
patch 9/10 ("media-request: EPERM -> EACCES"):
Reviewed-by: Tomasz Figa
Thanks a lot for this great work!
Best regards,
Tomasz
Re: [PATCHv2 09/10] media-request: EPERM -> EACCES
On Thu, Aug 30, 2018 at 10:04 PM Sakari Ailus wrote: > > On Thu, Aug 30, 2018 at 01:51:39PM +0200, Hans Verkuil wrote: > > On 08/30/2018 12:15 PM, Sakari Ailus wrote: > > > Hi Hans, > > > > > > Thanks a lot for working on this! > > > > > > On Tue, Aug 28, 2018 at 03:49:10PM +0200, Hans Verkuil wrote: > > >> From: Hans Verkuil > > >> > > >> If requests are not supported by the driver, then return EACCES, not > > >> EPERM. This is consistent with the error that an invalid request_fd will > > >> give, and if requests are not supported, then all request_fd values are > > >> invalid. > > >> > > >> Signed-off-by: Hans Verkuil > > >> --- > > >> Documentation/media/uapi/v4l/buffer.rst | 2 +- > > >> .../media/uapi/v4l/vidioc-g-ext-ctrls.rst | 9 - > > >> Documentation/media/uapi/v4l/vidioc-qbuf.rst | 15 +-- > > >> drivers/media/media-request.c | 4 ++-- > > >> 4 files changed, 16 insertions(+), 14 deletions(-) > > >> > > >> diff --git a/Documentation/media/uapi/v4l/buffer.rst > > >> b/Documentation/media/uapi/v4l/buffer.rst > > >> index 1865cd5b9d3c..58a6d7d336e6 100644 > > >> --- a/Documentation/media/uapi/v4l/buffer.rst > > >> +++ b/Documentation/media/uapi/v4l/buffer.rst > > >> @@ -314,7 +314,7 @@ struct v4l2_buffer > > >>:ref:`ioctl VIDIOC_QBUF ` and ignored by other ioctls. > > >>Applications should not set ``V4L2_BUF_FLAG_REQUEST_FD`` for any > > >> ioctls > > >>other than :ref:`VIDIOC_QBUF `. > > >> - If the device does not support requests, then ``EPERM`` will be > > >> returned. > > >> + If the device does not support requests, then ``EACCES`` will be > > >> returned. > > >>If requests are supported but an invalid request file descriptor is > > >>given, then ``EINVAL`` will be returned. > > >> > > >> diff --git a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst > > >> b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst > > >> index ad8908ce3095..54a999df5aec 100644 > > >> --- a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst > > >> +++ b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst > > >> @@ -100,7 +100,7 @@ file descriptor and ``which`` is set to > > >> ``V4L2_CTRL_WHICH_REQUEST_VAL``, > > >> then the controls are not applied immediately when calling > > >> :ref:`VIDIOC_S_EXT_CTRLS `, but instead are applied > > >> by > > >> the driver for the buffer associated with the same request. > > >> -If the device does not support requests, then ``EPERM`` will be > > >> returned. > > >> +If the device does not support requests, then ``EACCES`` will be > > >> returned. > > >> If requests are supported but an invalid request file descriptor is > > >> given, > > >> then ``EINVAL`` will be returned. > > >> > > >> @@ -233,7 +233,7 @@ still cause this situation. > > >>these controls have to be retrieved from a request or tried/set for > > >>a request. In the latter case the ``request_fd`` field contains the > > >>file descriptor of the request that should be used. If the device > > >> - does not support requests, then ``EPERM`` will be returned. > > >> + does not support requests, then ``EACCES`` will be returned. > > >> > > >>.. note:: > > >> > > >> @@ -299,7 +299,7 @@ still cause this situation. > > >>- ``request_fd`` > > >>- File descriptor of the request to be used by this operation. > > >> Only > > >>valid if ``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``. > > >> - If the device does not support requests, then ``EPERM`` will be > > >> returned. > > >> + If the device does not support requests, then ``EACCES`` will be > > >> returned. > > >>If requests are supported but an invalid request file descriptor is > > >>given, then ``EINVAL`` will be returned. > > >> * - __u32 > > >> @@ -408,6 +408,5 @@ EACCES > > >> control, or to get a control from a request that has not yet been > > >> completed. > > >> > > >> -EPERM > > > > > > -EACCES here, too? > > > > The '-' in -EPERM is from diff, meaning: delete this line :-) > > Oh, I missed that while reading the quoted message. X-) > > > > > > > > >> -The ``which`` field was set to ``V4L2_CTRL_WHICH_REQUEST_VAL`` but > > >> the > > >> +Or the ``which`` field was set to ``V4L2_CTRL_WHICH_REQUEST_VAL`` > > >> but the > > >> device does not support requests. > > >> diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst > > >> b/Documentation/media/uapi/v4l/vidioc-qbuf.rst > > >> index 7bff69c15452..a2f4ac0b0ba1 100644 > > >> --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst > > >> +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst > > >> @@ -104,7 +104,7 @@ in use. Setting it means that the buffer will not be > > >> passed to the driver > > >> until the request itself is queued. Also, the driver will apply any > > >> settings associated with the request for this buffer. This field will > > >> be ignored unless the ``V4L2_BUF_FLAG_REQUEST_FD`` flag is set. > > >>
Re: [PATCHv2 07/10] v4l2-ctrls: use media_request_(un)lock_for_access
On Fri, Aug 31, 2018 at 11:55 PM Tomasz Figa wrote:
>
> Hi Hans,
>
> On Tue, Aug 28, 2018 at 10:49 PM Hans Verkuil wrote:
> >
> > From: Hans Verkuil
> >
> > When getting control values from a completed request, we have
> > to protect the request against being re-inited why it is
> > being accessed by calling media_request_(un)lock_for_access.
> >
> > Signed-off-by: Hans Verkuil
> > ---
> > drivers/media/v4l2-core/v4l2-ctrls.c | 21 +++--
> > 1 file changed, 15 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> > b/drivers/media/v4l2-core/v4l2-ctrls.c
> > index ccaf3068de6d..cc266a4a6e88 100644
> > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > @@ -3289,11 +3289,10 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
> > struct media_device *mdev,
> > struct v4l2_ext_controls *cs)
> > {
> > struct media_request_object *obj = NULL;
> > + struct media_request *req = NULL;
> > int ret;
> >
> > if (cs->which == V4L2_CTRL_WHICH_REQUEST_VAL) {
> > - struct media_request *req;
> > -
> > if (!mdev || cs->request_fd < 0)
> > return -EINVAL;
> >
> > @@ -3306,11 +3305,18 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
> > struct media_device *mdev,
> > return -EACCES;
> > }
> >
> > + ret = media_request_lock_for_access(req);
> > + if (ret) {
> > + media_request_put(req);
> > + return ret;
> > + }
> > +
> > obj = v4l2_ctrls_find_req_obj(hdl, req, false);
> > - /* Reference to the request held through obj */
> > - media_request_put(req);
> > - if (IS_ERR(obj))
> > + if (IS_ERR(obj)) {
> > + media_request_unlock_for_access(req);
> > + media_request_put(req);
> > return PTR_ERR(obj);
> > + }
> >
> > hdl = container_of(obj, struct v4l2_ctrl_handler,
> >req_obj);
> > @@ -3318,8 +3324,11 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
> > struct media_device *mdev,
> >
> > ret = v4l2_g_ext_ctrls_common(hdl, cs);
> >
> > - if (obj)
> > + if (obj) {
> > + media_request_unlock_for_access(req);
>
> We called media_request_lock_for_access() before looking up obj. Don't
> we also need to call media_request_unlock_for_access() regardless of
> whether obj is non-NULL?
Aha, never mind. I checked the full context and we can't have !obj
here if we operated on a request. Sorry for the noise.
Best regards,
Tomasz
Re: [PATCHv2 07/10] v4l2-ctrls: use media_request_(un)lock_for_access
Hi Hans,
On Tue, Aug 28, 2018 at 10:49 PM Hans Verkuil wrote:
>
> From: Hans Verkuil
>
> When getting control values from a completed request, we have
> to protect the request against being re-inited why it is
> being accessed by calling media_request_(un)lock_for_access.
>
> Signed-off-by: Hans Verkuil
> ---
> drivers/media/v4l2-core/v4l2-ctrls.c | 21 +++--
> 1 file changed, 15 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> b/drivers/media/v4l2-core/v4l2-ctrls.c
> index ccaf3068de6d..cc266a4a6e88 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> @@ -3289,11 +3289,10 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
> struct media_device *mdev,
> struct v4l2_ext_controls *cs)
> {
> struct media_request_object *obj = NULL;
> + struct media_request *req = NULL;
> int ret;
>
> if (cs->which == V4L2_CTRL_WHICH_REQUEST_VAL) {
> - struct media_request *req;
> -
> if (!mdev || cs->request_fd < 0)
> return -EINVAL;
>
> @@ -3306,11 +3305,18 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
> struct media_device *mdev,
> return -EACCES;
> }
>
> + ret = media_request_lock_for_access(req);
> + if (ret) {
> + media_request_put(req);
> + return ret;
> + }
> +
> obj = v4l2_ctrls_find_req_obj(hdl, req, false);
> - /* Reference to the request held through obj */
> - media_request_put(req);
> - if (IS_ERR(obj))
> + if (IS_ERR(obj)) {
> + media_request_unlock_for_access(req);
> + media_request_put(req);
> return PTR_ERR(obj);
> + }
>
> hdl = container_of(obj, struct v4l2_ctrl_handler,
>req_obj);
> @@ -3318,8 +3324,11 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
> struct media_device *mdev,
>
> ret = v4l2_g_ext_ctrls_common(hdl, cs);
>
> - if (obj)
> + if (obj) {
> + media_request_unlock_for_access(req);
We called media_request_lock_for_access() before looking up obj. Don't
we also need to call media_request_unlock_for_access() regardless of
whether obj is non-NULL?
> media_request_object_put(obj);
> + media_request_put(req);
> + }
> return ret;
> }
Best regards,
Tomasz
Re: [PATCH 4/5] videodev2.h: add new capabilities for buffer types
On Tue, Aug 28, 2018 at 9:37 PM Hans Verkuil wrote: > > On 24/08/18 16:36, Tomasz Figa wrote: > > Hi Hans, > > > > On Fri, Aug 24, 2018 at 5:22 PM Hans Verkuil wrote: > >> > >> From: Hans Verkuil > >> > >> VIDIOC_REQBUFS and VIDIOC_CREATE_BUFFERS will return capabilities > >> telling userspace what the given buffer type is capable of. > >> > > > > Please see my comments below. > > > >> Signed-off-by: Hans Verkuil > >> --- > >> .../media/uapi/v4l/vidioc-create-bufs.rst | 10 +- > >> .../media/uapi/v4l/vidioc-reqbufs.rst | 36 ++- > >> include/uapi/linux/videodev2.h| 13 +-- > >> 3 files changed, 55 insertions(+), 4 deletions(-) > >> > >> diff --git a/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > >> b/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > >> index a39e18d69511..fd34d3f236c9 100644 > >> --- a/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > >> +++ b/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > >> @@ -102,7 +102,15 @@ than the number requested. > >>- ``format`` > >>- Filled in by the application, preserved by the driver. > >> * - __u32 > >> - - ``reserved``\ [8] > >> + - ``capabilities`` > >> + - Set by the driver. If 0, then the driver doesn't support > >> +capabilities. In that case all you know is that the driver is > >> + guaranteed to support ``V4L2_MEMORY_MMAP`` and *might* support > >> + other :c:type:`v4l2_memory` types. It will not support any others > >> + capabilities. See :ref:`here ` for a list > >> of the > >> + capabilities. > > > > Perhaps it would make sense to document how the application is > > expected to query for these capabilities? Right now, the application > > is expected to fill in the "memory" field in this struct (and reqbufs > > counterpart), but it sounds a bit strange that one needs to know what > > "memory" value to write there to query what set of "memory" values is > > supported. In theory, MMAP is expected to be always supported, but it > > sounds strange anyway. Also, is there a way to call REQBUFS without > > altering the buffer allocation? > > No, this is only possible with CREATE_BUFS. > > But it is reasonable to call REQBUFS with a count of 0, since you want to > start with a clean slate anyway. > > The only option I see would be to introduce a new memory type (e.g. > V4L2_MEMORY_CAPS) to just return the capabilities. But that's more ugly > then just requiring that you use MMAP when calling this. > > I'm inclined to just document that you need to set memory to MMAP if > you want to get the capabilities since MMAP is always guaranteed to > exist. I guess it's the least evil we can get without changing the API too much. Fair enough. Best regards, Tomasz
Re: [RFC] Request API and V4L2 capabilities
On Fri, Aug 24, 2018 at 4:30 PM Hans Verkuil wrote: > > On 08/23/2018 07:37 PM, Nicolas Dufresne wrote: > > Le jeudi 23 août 2018 à 16:31 +0200, Hans Verkuil a écrit : > >>> I propose adding these capabilities: > >>> > >>> #define V4L2_BUF_CAP_HAS_REQUESTS 0x0001 > >>> #define V4L2_BUF_CAP_REQUIRES_REQUESTS0x0002 > >>> #define V4L2_BUF_CAP_HAS_MMAP 0x0100 > >>> #define V4L2_BUF_CAP_HAS_USERPTR 0x0200 > >>> #define V4L2_BUF_CAP_HAS_DMABUF 0x0400 > >> > >> I substituted SUPPORTS for HAS and dropped the REQUIRES_REQUESTS > >> capability. > >> As Tomasz mentioned, technically (at least for stateless codecs) you could > >> handle just one frame at a time without using requests. It's very > >> inefficient, > >> but it would work. > > > > I thought the request was providing a data structure to refer back to > > the frames, so each codec don't have to implement one. Do you mean that > > the framework will implicitly request requests in that mode ? or simply > > that there is no such helper ? > > Yes, that's done through controls as well. > > The idea would be that you set the necessary controls, queue a buffer to > the output queue, dequeue a buffer from the output queue, read back any > new state information and repeat the process. > > That said, I'm not sure if the cedrus driver for example can handle this > at the moment. It is also inefficient and it won't work if codecs require > more than one buffer in the queue for whatever reason. > > Tomasz, Paul, please correct me if I am wrong. > > In any case, I think we can do without this proposed capability since it is > simply a requirement when implementing the pixelformat for the stateless > codec that the Request API will be available and it should be documented > as such in the spec. No correction needed. :) Best regards, Tomasz
Re: [RFC] Request API and V4L2 capabilities
On Fri, Aug 24, 2018 at 2:33 AM Nicolas Dufresne wrote: > > Le jeudi 23 août 2018 à 10:05 +0200, Paul Kocialkowski a écrit : > > Hi, > > > > On Wed, 2018-08-22 at 14:33 -0300, Ezequiel Garcia wrote: > > > On Wed, 2018-08-22 at 16:10 +0200, Paul Kocialkowski wrote: > > > > Hi, > > > > > > > > On Tue, 2018-08-21 at 17:52 +0900, Tomasz Figa wrote: > > > > > Hi Hans, Paul, > > > > > > > > > > On Mon, Aug 6, 2018 at 6:29 PM Paul Kocialkowski > > > > > wrote: > > > > > > > > > > > > On Mon, 2018-08-06 at 11:23 +0200, Hans Verkuil wrote: > > > > > > > On 08/06/2018 11:13 AM, Paul Kocialkowski wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > On Mon, 2018-08-06 at 10:32 +0200, Hans Verkuil wrote: > > > > > > > > > On 08/06/2018 10:16 AM, Paul Kocialkowski wrote: > > > > > > > > > > On Sat, 2018-08-04 at 15:50 +0200, Hans Verkuil wrote: > > > > > > > > > > > Regarding point 3: I think this should be documented next > > > > > > > > > > > to the pixel format. I.e. > > > > > > > > > > > the MPEG-2 Slice format used by the stateless cedrus > > > > > > > > > > > codec requires the request API > > > > > > > > > > > and that two MPEG-2 controls (slice params and > > > > > > > > > > > quantization matrices) must be present > > > > > > > > > > > in each request. > > > > > > > > > > > > > > > > > > > > > > I am not sure a control flag (e.g. > > > > > > > > > > > V4L2_CTRL_FLAG_REQUIRED_IN_REQ) is needed here. > > > > > > > > > > > It's really implied by the fact that you use a stateless > > > > > > > > > > > codec. It doesn't help > > > > > > > > > > > generic applications like v4l2-ctl or qv4l2 either since > > > > > > > > > > > in order to support > > > > > > > > > > > stateless codecs they will have to know about the details > > > > > > > > > > > of these controls anyway. > > > > > > > > > > > > > > > > > > > > > > So I am inclined to say that it is not necessary to > > > > > > > > > > > expose this information in > > > > > > > > > > > the API, but it has to be documented together with the > > > > > > > > > > > pixel format documentation. > > > > > > > > > > > > > > > > > > > > I think this is affected by considerations about codec > > > > > > > > > > profile/level > > > > > > > > > > support. More specifically, some controls will only be > > > > > > > > > > required for > > > > > > > > > > supporting advanced codec profiles/levels, so they can only > > > > > > > > > > be > > > > > > > > > > explicitly marked with appropriate flags by the driver when > > > > > > > > > > the target > > > > > > > > > > profile/level is known. And I don't think it would be sane > > > > > > > > > > for userspace > > > > > > > > > > to explicitly set what profile/level it's aiming at. As a > > > > > > > > > > result, I > > > > > > > > > > don't think we can explicitly mark controls as required or > > > > > > > > > > optional. > > > > > > > > > > I'm not sure this is entirely true. The hardware may need to be > > > > > explicitly told what profile the video is. It may even not be the > > > > > hardware, but the driver itself too, given that the profile may imply > > > > > the CAPTURE pixel format, e.g. for VP9 profiles: > > > > > > > > > > profile 0 > > > > > color depth: 8 bit/sample, chroma subsampling: 4:2:0 > > > > > profile 1 > > > > > color depth: 8 bit, chroma subsampling: 4:2:0, 4:2:2, 4:4:4 > > > > > profil
Re: [RFC] Request API and V4L2 capabilities
On Wed, Aug 22, 2018 at 11:10 PM Paul Kocialkowski wrote: > > Hi, > > On Tue, 2018-08-21 at 17:52 +0900, Tomasz Figa wrote: > > Hi Hans, Paul, > > > > On Mon, Aug 6, 2018 at 6:29 PM Paul Kocialkowski > > wrote: > > > > > > On Mon, 2018-08-06 at 11:23 +0200, Hans Verkuil wrote: > > > > On 08/06/2018 11:13 AM, Paul Kocialkowski wrote: > > > > > Hi, > > > > > > > > > > On Mon, 2018-08-06 at 10:32 +0200, Hans Verkuil wrote: > > > > > > On 08/06/2018 10:16 AM, Paul Kocialkowski wrote: > > > > > > > On Sat, 2018-08-04 at 15:50 +0200, Hans Verkuil wrote: > > > > > > > > Regarding point 3: I think this should be documented next to > > > > > > > > the pixel format. I.e. > > > > > > > > the MPEG-2 Slice format used by the stateless cedrus codec > > > > > > > > requires the request API > > > > > > > > and that two MPEG-2 controls (slice params and quantization > > > > > > > > matrices) must be present > > > > > > > > in each request. > > > > > > > > > > > > > > > > I am not sure a control flag (e.g. > > > > > > > > V4L2_CTRL_FLAG_REQUIRED_IN_REQ) is needed here. > > > > > > > > It's really implied by the fact that you use a stateless codec. > > > > > > > > It doesn't help > > > > > > > > generic applications like v4l2-ctl or qv4l2 either since in > > > > > > > > order to support > > > > > > > > stateless codecs they will have to know about the details of > > > > > > > > these controls anyway. > > > > > > > > > > > > > > > > So I am inclined to say that it is not necessary to expose this > > > > > > > > information in > > > > > > > > the API, but it has to be documented together with the pixel > > > > > > > > format documentation. > > > > > > > > > > > > > > I think this is affected by considerations about codec > > > > > > > profile/level > > > > > > > support. More specifically, some controls will only be required > > > > > > > for > > > > > > > supporting advanced codec profiles/levels, so they can only be > > > > > > > explicitly marked with appropriate flags by the driver when the > > > > > > > target > > > > > > > profile/level is known. And I don't think it would be sane for > > > > > > > userspace > > > > > > > to explicitly set what profile/level it's aiming at. As a result, > > > > > > > I > > > > > > > don't think we can explicitly mark controls as required or > > > > > > > optional. > > > > I'm not sure this is entirely true. The hardware may need to be > > explicitly told what profile the video is. It may even not be the > > hardware, but the driver itself too, given that the profile may imply > > the CAPTURE pixel format, e.g. for VP9 profiles: > > > > profile 0 > > color depth: 8 bit/sample, chroma subsampling: 4:2:0 > > profile 1 > > color depth: 8 bit, chroma subsampling: 4:2:0, 4:2:2, 4:4:4 > > profile 2 > > color depth: 10–12 bit, chroma subsampling: 4:2:0 > > profile 3 > > color depth: 10–12 bit, chroma subsampling: 4:2:0, 4:2:2, 4:4:4 > > > > (reference: https://en.wikipedia.org/wiki/VP9#Profiles) > > I think it would be fair to expect userspace to select the right > destination format (and maybe have the driver error if there's a > mismatch with the meta-data) instead of having the driver somewhat > expose what format should be used. There are many different memory representations of the same physical YUV format, just for YUV 4:2:0: NV12, NV12M, NV21, NV21M, YUV420, YUV420M, YVU420, YVU420M. It depends on hardware and driver which one would be available for given stream to decode. How is the user space expected to know which one is, without querying the driver first? > > But maybe this would be an API violation, since all the enumerated > formats are probably supposed to be selectable? Correct. > > We could also look at it the other way round and consider that selecting > an exposed format is always legit, but that it implies passing a > bitstream that matches it or the driver will error (because of an > invalid
Re: [PATCH v3 6/7] media: Add controls for JPEG quantization tables
On Wed, Aug 29, 2018 at 11:50 AM Ezequiel Garcia wrote:
>
> On Mon, 2018-08-27 at 09:47 +0200, Paul Kocialkowski wrote:
> > Hi,
> >
> > On Wed, 2018-08-22 at 13:59 -0300, Ezequiel Garcia wrote:
> > > From: Shunqian Zheng
> > >
> > > Add V4L2_CID_JPEG_LUMA/CHROMA_QUANTIZATION controls to allow userspace
> > > configure the JPEG quantization tables.
> >
> > How about having a single control for quantization?
> >
> > In MPEG-2/H.264/H.265, we have a single control exposed as a structure,
> > which contains the tables for both luma and chroma. In the case of JPEG,
> > it's not that big a deal, but for advanced video formats, it would be
> > too much hassle to have one control per table.
> >
> > In order to keep the interface consistent, I think it'd be best to merge
> > both matrices into a single control.
> >
> > What do you think?
> >
>
> I think it makes a lot of sense. I don't see the benefit in having luma
> and chroma separated, and consistency is good.
>
> I guess the more consistent solution would be to expose a compound
> control, similar to the video quantization one.
>
> struct v4l2_ctrl_jpeg_quantization {
>__u8luma_quantization_matrix[64];
>__u8chroma_quantization_matrix[64];
> };
Makes sense indeed. It also lets us avoid the hassle of setting
.min/.max/.dims and other array control stuff, since everything is
already defined by the C struct itself.
Best regards,
Tomasz
Re: [PATCH v1 2/2] v4l: Document Intel IPU3 meta data uAPI
On Tue, Aug 14, 2018 at 5:50 AM Mauro Carvalho Chehab
wrote:
>
> Em Mon, 13 Aug 2018 15:42:34 +0200
> Hans Verkuil escreveu:
>
> > On 15/06/18 05:29, Yong Zhi wrote:
> > > These meta formats are used on Intel IPU3 ImgU video queues
> > > to carry 3A statistics and ISP pipeline parameters.
> > >
> > > V4L2_META_FMT_IPU3_3A
> > > V4L2_META_FMT_IPU3_PARAMS
> > >
> > > Signed-off-by: Yong Zhi
> > > Signed-off-by: Chao C Li
> > > Signed-off-by: Rajmohan Mani
> > > ---
> > > Documentation/media/uapi/v4l/meta-formats.rst |1 +
> > > .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 174 ++
> > > include/uapi/linux/intel-ipu3.h| 2816
> > >
> > > 3 files changed, 2991 insertions(+)
> > > create mode 100644
> > > Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > > create mode 100644 include/uapi/linux/intel-ipu3.h
> > >
> > > diff --git a/Documentation/media/uapi/v4l/meta-formats.rst
> > > b/Documentation/media/uapi/v4l/meta-formats.rst
> > > index 0c4e1ec..b887fca 100644
> > > --- a/Documentation/media/uapi/v4l/meta-formats.rst
> > > +++ b/Documentation/media/uapi/v4l/meta-formats.rst
> > > @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata`
> > > interface only.
> > > .. toctree::
> > > :maxdepth: 1
> > >
> > > +pixfmt-meta-intel-ipu3
> > > pixfmt-meta-uvc
> > > pixfmt-meta-vsp1-hgo
> > > pixfmt-meta-vsp1-hgt
> > > diff --git a/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > > b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > > new file mode 100644
> > > index 000..5c050e6
> > > --- /dev/null
> > > +++ b/Documentation/media/uapi/v4l/pixfmt-meta-intel-ipu3.rst
> > > @@ -0,0 +1,174 @@
> > > +.. -*- coding: utf-8; mode: rst -*-
> > > +
> > > +.. _intel-ipu3:
> > > +
> > > +**
> > > +V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
> > > +**
> > > +
> > > +.. c:type:: ipu3_uapi_stats_3a
> > > +
> > > +3A statistics
> > > +=
> > > +
> > > +For IPU3 ImgU, the 3A statistics accelerators collect different
> > > statistics over
> > > +an input bayer frame. Those statistics, defined in data struct
> > > +:c:type:`ipu3_uapi_stats_3a`, are meta output obtained from "ipu3-imgu
> > > 3a stat"
> > > +video node, which are then passed to user space for statistics analysis
> > > +using :c:type:`v4l2_meta_format` interface.
> > > +
> > > +The statistics collected are AWB (Auto-white balance) RGBS cells, AWB
> > > filter
>
> Just like you did with AWB, AF and AE, please place the full name in
> parenthesis
> for RGBS and AWB.
>
> > > +response, AF (Auto-focus) filter response, and AE (Auto-exposure)
> > > histogram.
> > > +
> > > +struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for
> > > all above.
> > > +
> > > +
> > > +.. code-block:: c
> > > +
> > > +
> > > + struct ipu3_uapi_stats_3a {
> > > + IPU3_ALIGN struct ipu3_uapi_awb_raw_buffer awb_raw_buffer;
> >
> > IPU3_ALIGN? What's that?
> >
> > OK, after reading the header I see what it does, but I think you should
> > drop it in the documentation since it doesn't help the reader.
>
> Yeah, that IPU3_ALIGN is confusing.
>
> Yet, instead of just dropping, I would replace it by a comment
> to explain that the struct is 32-bytes aligned.
>
> On a separate (but related) comment, you're declaring it as:
>
> #define IPU3_ALIGN
> __attribute__((aligned(IPU3_UAPI_ISP_WORD_BYTES)))
>
> This is a gcc-specific dialect. Better to use, instead, __aligned(x)
> which is defined as:
>
> #define __aligned(x)__attribute__((aligned(x)))
>
Note that this is an uapi/ header. Is the __aligned() macro okay to
use in uapi headers? I couldn't find any header there using it and we
had problems with our user space compiling with it.
By the way, I wonder if this is the right approach for controlling the
layout of ABI structs. I don't see many headers using any alignment in
uapi/ in general. Perhaps explicit padding bytes would be more
appropriate? They are also less tricky when one structure needs to be
embedded inside two or more different structures with different
alignments, which can't be done easily if you specify __aligned() on
the child struct.
Best regards,
Tomasz
Re: [PATCH 1/2] media: docs-rst: Document memory-to-memory video decoder interface
On Tue, Aug 21, 2018 at 8:29 PM Stanimir Varbanov wrote: > > Hi Tomasz, > > On 08/08/2018 05:55 AM, Tomasz Figa wrote: > > On Tue, Aug 7, 2018 at 4:37 PM Hans Verkuil wrote: > > >>>>>>> +7. If all the following conditions are met, the client may resume > >>>>>>> the > >>>>>>> +decoding instantly, by using :c:func:`VIDIOC_DECODER_CMD` with > >>>>>>> +``V4L2_DEC_CMD_START`` command, as in case of resuming after the > >>>>>>> drain > >>>>>>> +sequence: > >>>>>>> + > >>>>>>> +* ``sizeimage`` of new format is less than or equal to the size > >>>>>>> of > >>>>>>> + currently allocated buffers, > >>>>>>> + > >>>>>>> +* the number of buffers currently allocated is greater than or > >>>>>>> equal to > >>>>>>> + the minimum number of buffers acquired in step 6. > >>>>>> > >>>>>> You might want to mention that if there are insufficient buffers, then > >>>>>> VIDIOC_CREATE_BUFS can be used to add more buffers. > >>>>>> > >>>>> > >>>>> This might be a bit tricky, since at least s5p-mfc and coda can only > >>>>> work on a fixed buffer set and one would need to fully reinitialize > >>>>> the decoding to add one more buffer, which would effectively be the > >>>>> full resolution change sequence, as below, just with REQBUFS(0), > >>>>> REQBUFS(N) replaced with CREATE_BUFS. > >>>> > >>>> What happens today in those drivers if you try to call CREATE_BUFS? > >>> > >>> s5p-mfc doesn't set the .vidioc_create_bufs pointer in its > >>> v4l2_ioctl_ops, so I suppose that would be -ENOTTY? > >> > >> Correct for s5p-mfc. > > > > As Philipp clarified, coda supports adding buffers on the fly. I > > briefly looked at venus and mtk-vcodec and they seem to use m2m > > implementation of CREATE_BUFS. Not sure if anyone tested that, though. > > So the only hardware I know for sure cannot support this is s5p-mfc. > > In Venus case CREATE_BUFS is tested with Gstreamer. Stanimir: Alright. Thanks for confirmation. Hans: Technically, we could still implement CREATE_BUFS for s5p-mfc, but it would need to be restricted to situations where it's possible to reinitialize the whole hardware buffer queue, i.e. - before initial STREAMON(CAPTURE) after header parsing, - after a resolution change and before following STREAMON(CAPTURE) or DECODER_CMD_START (to ack resolution change without buffer reallocation). Would that work for your original suggestion? Best regards, Tomasz
Re: [PATCH 1/2] media: docs-rst: Document memory-to-memory video decoder interface
Hi Philipp, On Tue, Aug 21, 2018 at 12:34 AM Philipp Zabel wrote: > > On Mon, 2018-08-20 at 23:27 +0900, Tomasz Figa wrote: > [...] > > +3. Start queuing buffers to ``OUTPUT`` queue containing stream data after > > > > > > + the seek until a suitable resume point is found. > > > > > > + > > > > > > + .. note:: > > > > > > + > > > > > > + There is no requirement to begin queuing stream starting > > > > > > exactly from > > > > > > + a resume point (e.g. SPS or a keyframe). The driver must > > > > > > handle any > > > > > > + data queued and must keep processing the queued buffers > > > > > > until it > > > > > > + finds a suitable resume point. While looking for a resume > > > > > > point, the > > > > > > > > > > I think the definition of a resume point is too vague in this place. > > > > > Can the driver decide whether or not a keyframe without SPS is a > > > > > suitable resume point? Or do drivers have to parse and store SPS/PPS > > > > > if > > > > > the hardware does not support resuming from a keyframe without sending > > > > > SPS/PPS again? > > > > > > > > The thing is that existing drivers implement and user space clients > > > > rely on the behavior described above, so we cannot really change it > > > > anymore. > > > > > > My point is that I'm not exactly sure what that behaviour is, given the > > > description. > > > > > > Must a driver be able to resume from a keyframe even if userspace never > > > pushes SPS/PPS again? > > > If so, I think it should be mentioned more explicitly than just via an > > > example in parentheses, to make it clear to all driver developers that > > > this is a requirement that userspace is going to rely on. > > > > > > Or, if that is not the case, is a driver free to define "SPS only" as > > > its "suitable resume point" and to discard all input including keyframes > > > until the next SPS/PPS is pushed? > > > > > > It would be better to clearly define what a "suitable resume point" has > > > to be per codec, and not let the drivers decide for themselves, if at > > > all possible. Otherwise we'd need a away to inform userspace about the > > > per-driver definition. > > > > The intention here is that there is exactly no requirement for the > > user space to seek to any kind of resume point > > No question about this. > > > and so there is no point in defining such. > > I don't agree. Let me give an example: > > Assume userspace wants to play back a simple h.264 stream that has > SPS/PPS exactly once, in the beginning. > > If drivers are allowed to resume from SPS/PPS only, and have no way to > communicate this to userspace, userspace always has to assume that > resuming from keyframes alone is not possible. So it has to store > SPS/PPS and resubmit them with every seek, even if a specific driver > wouldn't require it: Otherwise those drivers that don't store SPS/PPS > themselves (or in hardware) would be allowed to just drop everything > after the first seek. > This effectively would make resending SPS/PPS mandatory, which doesn't > fit well with the intention of letting userspace just seek anywhere and > start feeding data (or: NAL units) into the driver blindly. > I'd say that such video is broken by design, because you cannot play back any arbitrary later part of it without decoding it from the beginning. However, if the hardware keeps SPS/PPS across seeks (and that should normally be the case), the case could be handled by the user space letting the decoder initialize with the first frames and only then seeking, which would probably be the typical case of a user opening a video file and then moving the seek bar to desired position (or clicking a bookmark). If the hardware doesn't keep SPS/PPS across seeks, stateless API could arguably be a better candidate for it, since it mandates the user space to keep SPS/PPS around. > > The only requirement here is that the > > hardware/driver keeps processing the source stream until it finds a > > resume point suitable for it - if the hardware keeps SPS/PPS in its > > state then just a keyframe; if it doesn't then SPS/PPS. > > Yes, but the difference between those two might be very relevant to > userspace behaviour. > > > Note that this is a documentation of the user space API, not a driver > >
Re: [PATCH 0/5] Post-v18: Request API updates
Hi Hans, On Fri, Aug 24, 2018 at 5:22 PM Hans Verkuil wrote: > > From: Hans Verkuil > > Hi all, > > This patch series sits on top of my v18 series for the Request API. > It makes some final (?) changes as discussed in: > > https://www.mail-archive.com/linux-media@vger.kernel.org/msg134419.html > > and: > > https://www.spinics.net/lists/linux-media/msg138596.html > For all patches except "[PATCH 4/5] videodev2.h: add new capabilities for buffer types": Reviewed-by: Tomasz Figa Best regards, Tomasz
Re: [PATCH 4/5] videodev2.h: add new capabilities for buffer types
Hi Hans, On Fri, Aug 24, 2018 at 5:22 PM Hans Verkuil wrote: > > From: Hans Verkuil > > VIDIOC_REQBUFS and VIDIOC_CREATE_BUFFERS will return capabilities > telling userspace what the given buffer type is capable of. > Please see my comments below. > Signed-off-by: Hans Verkuil > --- > .../media/uapi/v4l/vidioc-create-bufs.rst | 10 +- > .../media/uapi/v4l/vidioc-reqbufs.rst | 36 ++- > include/uapi/linux/videodev2.h| 13 +-- > 3 files changed, 55 insertions(+), 4 deletions(-) > > diff --git a/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > b/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > index a39e18d69511..fd34d3f236c9 100644 > --- a/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > +++ b/Documentation/media/uapi/v4l/vidioc-create-bufs.rst > @@ -102,7 +102,15 @@ than the number requested. >- ``format`` >- Filled in by the application, preserved by the driver. > * - __u32 > - - ``reserved``\ [8] > + - ``capabilities`` > + - Set by the driver. If 0, then the driver doesn't support > +capabilities. In that case all you know is that the driver is > + guaranteed to support ``V4L2_MEMORY_MMAP`` and *might* support > + other :c:type:`v4l2_memory` types. It will not support any others > + capabilities. See :ref:`here ` for a list of > the > + capabilities. Perhaps it would make sense to document how the application is expected to query for these capabilities? Right now, the application is expected to fill in the "memory" field in this struct (and reqbufs counterpart), but it sounds a bit strange that one needs to know what "memory" value to write there to query what set of "memory" values is supported. In theory, MMAP is expected to be always supported, but it sounds strange anyway. Also, is there a way to call REQBUFS without altering the buffer allocation? Best regards, Tomasz
Re: [RFC] Request API and V4L2 capabilities
On Wed, Aug 22, 2018 at 10:21 PM Paul Kocialkowski wrote: > > Hi, > > [...] > > On Wed, 2018-08-15 at 14:51 +0200, Maxime Jourdan wrote: > > Hi Paul, I think we need to go deeper than just exposing the supported > > profiles/levels and also include a way to query the CAPTURE pixel > > formats that are supported for each profile. > > > > Maybe HEVC Main produces yuv420p but HEVC Main10 gives you > > yuv420p10le. Maybe H.264 HiP produces NV12 but H.264 Hi422P produces > > YUYV while also supporting down-sampling to NV12. > > Well, I think we're looking at this backwards. Userspace certainly known > what destination format is relevant for the video, so it shouldn't have > to query the driver about it except to check that the format is indeed > supported. Typically the profile itself only defines the sub-sampling and sample size, but not the exact set of formats supported by the hardware. VP9 profile 0 is expected to decode into YUV 4:2:0 8-bit, but whether that would be NV12, YUV420, NV21 or maybe some hw-specific tiled format (like NV12MT), is entirely up to the hardware/driver. Userspace will definitely need to know if the decoder can decode the video into a format, which it can later use (display). > > > I don't know the specifics of each platform and the only example I can > > think of is the Amlogic HEVC decoder that can produce NV12 for Main, > > but only outputs a compressed proprietary format for Main10. > > > > I unfortunately don't have an idea about how to implement that, but > > I'll think about it. > > On the first generations of Allwinner platforms, we also have a vendor- > specific format as output, that we expose with a dedicated format. > There's no particular interfacing issue with that. Only that userspace > has to be aware of the format and how to deal with it. > Typically a decode operates as a part of a pipeline with other components. If your display doesn't let you display format X on an overlay plane, you may want to use a post-processor hardware to convert it to format Y. Or maybe use GPU to blit the video into the primary plane? Or maybe you need to do both, because format X is a decoder-specific tiled format and the GPU can't deal with it and all the overlay planes are occupied with some other surfaces? Or maybe it's just cheaper to do software decode rather than the conversion+GPU blit? This is something that normally needs to be queried before the video playback is initialized. Best regards, Tomasz
Re: [RFP] Media Summit: Complex cameras
Hi Mauro, On Thu, Aug 16, 2018 at 10:58 PM Mauro Carvalho Chehab wrote: > > I expect that we could have something to discuss there about complex > cameras. So, I'd reserve a 50 mins slot for it. > > The idea is to discuss about the undergoing work with complex camera > development is happening. > > As we're working to merge request API, another topic for discussion is > how to add support for requests on it (or on a separate but related > library). I need to recheck, but I might not be able to attend in person. Do you think it would be possible to join remotely, by using Hangouts (or anything else)? Best regards, Tomasz
Re: [RFC] Request API questions
On Tue, Aug 21, 2018 at 7:00 PM Sakari Ailus wrote:
>
> On Fri, Aug 17, 2018 at 12:09:40PM +0200, Hans Verkuil wrote:
> > On 17/08/18 12:02, Tomasz Figa wrote:
> > > On Thu, Aug 16, 2018 at 8:15 PM Mauro Carvalho Chehab
> > > wrote:
> > >>
> > >> Em Thu, 16 Aug 2018 12:25:25 +0200
> > >> Hans Verkuil escreveu:
> > >>
> > >>> Laurent raised a few API issues/questions in his review of the
> > >>> documentation.
> > >>>
> > >>> I've consolidated those in this RFC. I would like to know what others
> > >>> think
> > >>> and if I should make changes.
> > >
> > > Thanks Hans for a nice summary and Mauro for initial input. :)
> > >
> > >>>
> > >>> 1) Should you be allowed to set controls directly if they are also used
> > >>> in
> > >>>requests? Right now this is allowed, although we warn in the spec
> > >>> that
> > >>>this can lead to undefined behavior.
> > >>>
> > >>>In my experience being able to do this is very useful while testing,
> > >>>and restricting this is not all that easy to implement. I also think
> > >>> it is
> > >>>not our job. It is not as if something will break when you do this.
> > >>>
> > >>>If there really is a good reason why you can't mix this for a
> > >>> specific
> > >>>control, then the driver can check this and return -EBUSY.
> > >>
> > >> IMHO, there's not much sense on preventing it. Just having a warning
> > >> at the spec is enough.
> > >>
> > >
> > > I tend to agree with Mauro on this.
> > >
> > > Besides testing, there are some legit use cases where a carefully
> > > programmed user space may want to choose between setting controls
> > > directly and via a request, depending on circumstances. For example,
> > > one may want to set focus position alone (potentially a big step,
> > > taking time), before even attempting to capture any frames and then,
> > > when the capture starts, move the position gradually (in small steps,
> > > not taking too much time) with subsequent requests, to obtain a set of
> > > frames with different focus position.
> > >
> > >> +.. caution::
> > >> +
> > >> + Setting the same control through a request and also directly can
> > >> lead to
> > >> + undefined behavior!
> > >>
> > >> It is already warned with a caution. Anyone that decides to ignore a
> > >> warning like that will deserve his faith if things stop work.
> > >>
> > >>>
> > >>> 2) If request_fd in QBUF or the control ioctls is not a request fd,
> > >>> then we
> > >>>now return ENOENT. Laurent suggests using EBADR ('Invalid request
> > >>> descriptor')
> > >>>instead. This seems like a good idea to me. Should I change this?
> > >>
> > >> I don't have a strong opinion, but EBADR value seems to be
> > >> arch-dependent:
> > >>
> > >> arch/alpha/include/uapi/asm/errno.h:#define EBADR 98
> > >> /* Invalid request descriptor */
> > >> arch/mips/include/uapi/asm/errno.h:#define EBADR51
> > >> /* Invalid request descriptor */
> > >> arch/parisc/include/uapi/asm/errno.h:#defineEBADR 161
> > >> /* Invalid request descriptor */
> > >> arch/sparc/include/uapi/asm/errno.h:#define EBADR 103
> > >> /* Invalid request descriptor */
> > >> include/uapi/asm-generic/errno.h:#defineEBADR 53
> > >> /* Invalid request descriptor */
> > >>
> > >> Also, just because its name says "invalid request", it doesn't mean that
> > >> it
> > >> is the right error code. In this specific case, we're talking about a
> > >> file
> > >> descriptor. Invalid file descriptors is something that the FS subsystem
> > >> has already a defined set of return codes. We should stick with whatever
> > >> FS uses when a file descriptor is invalid.
> > >>
> > >> Where the VFS code returns EBADR? Does it make sense for our use cases?
> > >>
> > >
> > > DMA-buf framework seems to return -EINVAL if a
Re: [RFC] Request API and V4L2 capabilities
On Mon, Aug 6, 2018 at 5:32 PM Hans Verkuil wrote: > > On 08/06/2018 10:16 AM, Paul Kocialkowski wrote: > > Hi Hans and all, > > > > On Sat, 2018-08-04 at 15:50 +0200, Hans Verkuil wrote: > >> Hi all, > >> > >> While the Request API patch series addresses all the core API issues, there > >> are some high-level considerations as well: > >> > >> 1) How can the application tell that the Request API is supported and for > >>which buffer types (capture/output) and pixel formats? > >> > >> 2) How can the application tell if the Request API is required as opposed > >> to being > >>optional? > >> > >> 3) Some controls may be required in each request, how to let userspace > >> know this? > >>Is it even necessary to inform userspace? > >> > >> 4) (For bonus points): How to let the application know which streaming I/O > >> modes > >>are available? That's never been possible before, but it would be very > >> nice > >>indeed if that's made explicit. > > > > Thanks for bringing up these considerations and questions, which perhaps > > cover the last missing bits for streamlined use of the request API by > > userspace. I would suggest another item, related to 3): > > > > 5) How can applications tell whether the driver supports a specific > > codec profile/level, not only for encoding but also for decoding? It's > > common for low-end embedded hardware to not support the most advanced > > profiles (e.g. H264 high profile). > > > >> Since the Request API associates data with frame buffers it makes sense to > >> expose > >> this as a new capability field in struct v4l2_requestbuffers and struct > >> v4l2_create_buffers. > >> > >> The first struct has 2 reserved fields, the second has 8, so it's not a > >> problem to > >> take one for a capability field. Both structs also have a buffer type, so > >> we know > >> if this is requested for a capture or output buffer type. The pixel format > >> is known > >> in the driver, so HAS/REQUIRES_REQUESTS can be set based on that. I doubt > >> we'll have > >> drivers where the request caps would actually depend on the pixel format, > >> but it > >> theoretically possible. For both ioctls you can call them with count=0 at > >> the start > >> of the application. REQBUFS has of course the side-effect of deleting all > >> buffers, > >> but at the start of your application you don't have any yet. CREATE_BUFS > >> has no > >> side-effects. > > > > My initial thoughts on this point were to have flags exposed in > > v4l2_capability, but now that you're saying it, it does make sense for > > the flag to be associated with a buffer rather than the global device. > > > > In addition, I've heard of cases (IIRC it was some Rockchip platforms) > > where the platform has both stateless and stateful VPUs (I think it was > > stateless up to H264 and stateful for H265). AFAIR, on Rockchip those are two separate hardware blocks. > > This would allow supporting > > these two hardware blocks under the same video device (if that makes > > sense anyway). And even if there's no immediate need, it's always good > > to have this level of granularity (with little drawbacks). > > > >> I propose adding these capabilities: > >> > >> #define V4L2_BUF_CAP_HAS_REQUESTS0x0001 > >> #define V4L2_BUF_CAP_REQUIRES_REQUESTS 0x0002 > >> #define V4L2_BUF_CAP_HAS_MMAP0x0100 > >> #define V4L2_BUF_CAP_HAS_USERPTR 0x0200 > >> #define V4L2_BUF_CAP_HAS_DMABUF 0x0400 > >> > >> If REQUIRES_REQUESTS is set, then HAS_REQUESTS is also set. > >> > >> At this time I think that REQUIRES_REQUESTS would only need to be set for > >> the > >> output queue of stateless codecs. > >> > >> If capabilities is 0, then it's from an old kernel and all you know is that > >> requests are certainly not supported, and that MMAP is supported. Whether > >> USERPTR > >> or DMABUF are supported isn't known in that case (just try it :-) ). > > > > Sounds good to me! SGTM. +/- the fact that I'm not convinced if something really "requires requests" at V4L2 interface level. Would that mean that we strictly prohibit the lame synchronous way for stateless codecs? (Possibly useful for debugging.) Best regards, Tomasz
Re: [RFC] Request API and V4L2 capabilities
On Wed, Aug 15, 2018 at 11:18 PM Hans Verkuil wrote: > > On 15/08/18 14:11, Mauro Carvalho Chehab wrote: > > Em Sat, 4 Aug 2018 15:50:04 +0200 > > Hans Verkuil escreveu: > > > >> Hi all, > >> > >> While the Request API patch series addresses all the core API issues, there > >> are some high-level considerations as well: > >> > >> 1) How can the application tell that the Request API is supported and for > >>which buffer types (capture/output) and pixel formats? > >> > >> 2) How can the application tell if the Request API is required as opposed > >> to being > >>optional? > > > > Huh? Why would it be mandatory? > > It is mandatory for stateless codecs: you can't use them without the Request > API since > each frame needs the state as well. If you could make a driver for a > stateless codec > without the Request API we wouldn't have had to spend ages on developing it > in the first > place, would we? :-) Technically, one could still do the synchronous S_EXT_CTRL, QBUF, DQBUF, S_EXT_CTRL... loop, but I'm not sure if this is something worth considering. [snip] > >> Regarding point 3: I think this should be documented next to the pixel > >> format. I.e. > >> the MPEG-2 Slice format used by the stateless cedrus codec requires the > >> request API > >> and that two MPEG-2 controls (slice params and quantization matrices) must > >> be present > >> in each request. > > > > Makes sense to document with the pixel format... > > > >> I am not sure a control flag (e.g. V4L2_CTRL_FLAG_REQUIRED_IN_REQ) is > >> needed here. > > > > but it sounds worth to also have a flag. > > I'll wait to get some more feedback. I don't have a very strong opinion on > this. > I don't see any value in a flag for codecs. Querying the controls for the flag, if it's already required as a part of the Stateless Codec Interface for given pixel format, would only mean some redundant code both in kernel and user space. For other use cases, I'm not sure if we can say that a control is really required in a request. I think it should be something for the user space to decide, depending on the synchronization (and other) needs of given use case. Best regards, Tomasz
Re: [RFC] Request API and V4L2 capabilities
Hi Hans, Paul, On Mon, Aug 6, 2018 at 6:29 PM Paul Kocialkowski wrote: > > On Mon, 2018-08-06 at 11:23 +0200, Hans Verkuil wrote: > > On 08/06/2018 11:13 AM, Paul Kocialkowski wrote: > > > Hi, > > > > > > On Mon, 2018-08-06 at 10:32 +0200, Hans Verkuil wrote: > > > > On 08/06/2018 10:16 AM, Paul Kocialkowski wrote: > > > > > On Sat, 2018-08-04 at 15:50 +0200, Hans Verkuil wrote: > > > > > > Regarding point 3: I think this should be documented next to the > > > > > > pixel format. I.e. > > > > > > the MPEG-2 Slice format used by the stateless cedrus codec requires > > > > > > the request API > > > > > > and that two MPEG-2 controls (slice params and quantization > > > > > > matrices) must be present > > > > > > in each request. > > > > > > > > > > > > I am not sure a control flag (e.g. V4L2_CTRL_FLAG_REQUIRED_IN_REQ) > > > > > > is needed here. > > > > > > It's really implied by the fact that you use a stateless codec. It > > > > > > doesn't help > > > > > > generic applications like v4l2-ctl or qv4l2 either since in order > > > > > > to support > > > > > > stateless codecs they will have to know about the details of these > > > > > > controls anyway. > > > > > > > > > > > > So I am inclined to say that it is not necessary to expose this > > > > > > information in > > > > > > the API, but it has to be documented together with the pixel format > > > > > > documentation. > > > > > > > > > > I think this is affected by considerations about codec profile/level > > > > > support. More specifically, some controls will only be required for > > > > > supporting advanced codec profiles/levels, so they can only be > > > > > explicitly marked with appropriate flags by the driver when the target > > > > > profile/level is known. And I don't think it would be sane for > > > > > userspace > > > > > to explicitly set what profile/level it's aiming at. As a result, I > > > > > don't think we can explicitly mark controls as required or optional. I'm not sure this is entirely true. The hardware may need to be explicitly told what profile the video is. It may even not be the hardware, but the driver itself too, given that the profile may imply the CAPTURE pixel format, e.g. for VP9 profiles: profile 0 color depth: 8 bit/sample, chroma subsampling: 4:2:0 profile 1 color depth: 8 bit, chroma subsampling: 4:2:0, 4:2:2, 4:4:4 profile 2 color depth: 10–12 bit, chroma subsampling: 4:2:0 profile 3 color depth: 10–12 bit, chroma subsampling: 4:2:0, 4:2:2, 4:4:4 (reference: https://en.wikipedia.org/wiki/VP9#Profiles) > > > > > > > > > > I also like the idea that it should instead be implicit and that the > > > > > documentation should detail which specific stateless metadata controls > > > > > are required for a given profile/level. > > > > > > > > > > As for controls validation, the approach followed in the Cedrus driver > > > > > is to check that the most basic controls are filled and allow having > > > > > missing controls for those that match advanced profiles. > > > > > > > > > > Since this approach feels somewhat generic enough to be applied to all > > > > > stateless VPU drivers, maybe this should be made a helper in the > > > > > framework? > > > > > > > > Sounds reasonable. Not sure if it will be in the first version, but it > > > > is > > > > easy to add later. > > > > > > Definitely, I don't think this is such a high priority for now either. > > > We may want to put strict requirements on what controls are provided for given codec+profile/level. Otherwise we might get some user space that doesn't provide some of them and works only by luck, e.g. because some hardware defaults on initial drivers luckily match the needed values. Even if we don't validate it in the code yet, we should put a big warning saying that not providing the required controls would result in undefined behavior. > > > > > In addition, I see a need for exposing the maximum profile/level that > > > > > the driver supports for decoding. I would suggest reusing the already- > > > > > existing dedicated controls used for encoding for this purpose. For > > > > > decoders, they would be used to expose the (read-only) maximum > > > > > profile/level that is supported by the hardware and keep using them > > > > > as a > > > > > settable value in a range (matching the level of support) for > > > > > encoders. > > > > > > > > > > This is necessary for userspace to determine whether a given video can > > > > > be decoded in hardware or not. Instead of half-way decoding the video > > > > > (ending up in funky results), this would easily allow skipping > > > > > hardware > > > > > decoding and e.g. falling back on software decoding. > > > > > > > > I think it might be better to expose this through new read-only bitmask > > > > controls: i.e. a bitmask containing the supported profiles and levels. > > > > > > It seems that this is more or less what the coda driver is doing for > > > decoding actually, although it uses a menu control between
Re: [RFC] Request API questions
Hi Hans,
On Fri, Aug 17, 2018 at 7:09 PM Hans Verkuil wrote:
>
> On 17/08/18 12:02, Tomasz Figa wrote:
> > On Thu, Aug 16, 2018 at 8:15 PM Mauro Carvalho Chehab
> > wrote:
> >>
> >> Em Thu, 16 Aug 2018 12:25:25 +0200
> >> Hans Verkuil escreveu:
> >>
[snip]
> >>> 3) Calling VIDIOC_G_EXT_CTRLS for a request that has not been queued yet
> >>> will
> >>>return either the value of the control you set earlier in the request,
> >>> or
> >>>the current HW control value if it was never set in the request.
> >>>
> >>>I believe it makes sense to return what was set in the request
> >>> previously
> >>>(if you set it, you should be able to get it), but it is an idea to
> >>> return
> >>>ENOENT when calling this for controls that are NOT in the request.
> >>>
> >>>I'm inclined to implement that. Opinions?
> >>
> >> Return the request "cached" value, IMO, doesn't make sense. If the
> >> application needs such cache, it can implement itself.
> >
> > Can we think about any specific use cases for a user space that first
> > sets a control value to a request and then needs to ask the kernel to
> > get the value back? After all, it was the user space which set the
> > value, so I'm not sure if there is any need for the kernel to be an
> > intermediary here.
> >
> >>
> >> Return an error code if the request has not yet completed makes
> >> sense. Not sure what would be the best error code here... if the
> >> request is queued already (but not processed), EBUSY seems to be the
> >> better choice, but, if it was not queued yet, I'm not sure. I guess
> >> ENOENT would work.
> >
> > IMHO, as far as we assign unique error codes for different conditions
> > and document them well, we should be okay with any not absurdly
> > mismatched code. After all, most of those codes are defined for file
> > system operations and don't really map directly to anything else.
> >
> > FYI, VIDIOC_G_(EXT_)CTRL returns EINVAL if an unsupported control is
> > queried, so if we decided to keep the "cache" functionality after all,
> > perhaps we should stay consistent with it?
> > Reference:
> > https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-ext-ctrls.html#return-value
> >
> > My suggestion would be:
> > - EINVAL: the control was not in the request, (if we keep the cache
> > functionality)
> > - EPERM: the value is not ready, (we selected this code for Decoder
> > Interface to mean that CAPTURE format is not ready, which is similar;
> > perhaps that could be consistent?)
> >
> > Note that EINVAL would only apply to writable controls, while EPERM
> > only to volatile controls, since the latter can only change due to
> > request completion (non-volatile controls can only change as an effect
> > of user space action).
> >
>
> I'm inclined to just always return EPERM when calling G_EXT_CTRLS for
> a request. We can always relax this in the future.
>
> So when a request is not yet queued G_EXT_CTRLS returns EPERM, when
> queued but not completed it returns EBUSY and once completed it will
> work as it does today.
Not sure I'm following. What do we differentiate between with EPERM
and EBUSY? In both cases the value is not available yet and for codecs
we decided to have that signaled by EPERM.
On top of that, I still think we should be able to tell the case of
querying for a control that can never show up in the request, even
after it completes, specifically for any non-volatile control. That
could be done by returning a different code and -EINVAL would match
the control API behavior without requests.
The general logic would be like the pseudo code below:
G_EXT_CTRLS()
{
if ( control is not volatile )
return -EINVAL;
if ( request is not complete )
return -EPERM;
return control from the request;
}
Best regards,
Tomasz
Re: [RFC] Request API questions
On Thu, Aug 16, 2018 at 8:15 PM Mauro Carvalho Chehab
wrote:
>
> Em Thu, 16 Aug 2018 12:25:25 +0200
> Hans Verkuil escreveu:
>
> > Laurent raised a few API issues/questions in his review of the
> > documentation.
> >
> > I've consolidated those in this RFC. I would like to know what others think
> > and if I should make changes.
Thanks Hans for a nice summary and Mauro for initial input. :)
> >
> > 1) Should you be allowed to set controls directly if they are also used in
> >requests? Right now this is allowed, although we warn in the spec that
> >this can lead to undefined behavior.
> >
> >In my experience being able to do this is very useful while testing,
> >and restricting this is not all that easy to implement. I also think it
> > is
> >not our job. It is not as if something will break when you do this.
> >
> >If there really is a good reason why you can't mix this for a specific
> >control, then the driver can check this and return -EBUSY.
>
> IMHO, there's not much sense on preventing it. Just having a warning
> at the spec is enough.
>
I tend to agree with Mauro on this.
Besides testing, there are some legit use cases where a carefully
programmed user space may want to choose between setting controls
directly and via a request, depending on circumstances. For example,
one may want to set focus position alone (potentially a big step,
taking time), before even attempting to capture any frames and then,
when the capture starts, move the position gradually (in small steps,
not taking too much time) with subsequent requests, to obtain a set of
frames with different focus position.
> +.. caution::
> +
> + Setting the same control through a request and also directly can lead to
> + undefined behavior!
>
> It is already warned with a caution. Anyone that decides to ignore a
> warning like that will deserve his faith if things stop work.
>
> >
> > 2) If request_fd in QBUF or the control ioctls is not a request fd, then we
> >now return ENOENT. Laurent suggests using EBADR ('Invalid request
> > descriptor')
> >instead. This seems like a good idea to me. Should I change this?
>
> I don't have a strong opinion, but EBADR value seems to be arch-dependent:
>
> arch/alpha/include/uapi/asm/errno.h:#define EBADR 98 /*
> Invalid request descriptor */
> arch/mips/include/uapi/asm/errno.h:#define EBADR51 /*
> Invalid request descriptor */
> arch/parisc/include/uapi/asm/errno.h:#defineEBADR 161 /*
> Invalid request descriptor */
> arch/sparc/include/uapi/asm/errno.h:#define EBADR 103 /*
> Invalid request descriptor */
> include/uapi/asm-generic/errno.h:#defineEBADR 53 /*
> Invalid request descriptor */
>
> Also, just because its name says "invalid request", it doesn't mean that it
> is the right error code. In this specific case, we're talking about a file
> descriptor. Invalid file descriptors is something that the FS subsystem
> has already a defined set of return codes. We should stick with whatever
> FS uses when a file descriptor is invalid.
>
> Where the VFS code returns EBADR? Does it make sense for our use cases?
>
DMA-buf framework seems to return -EINVAL if a non-DMA-buf FD is
passed to dma_buf_get():
https://elixir.bootlin.com/linux/v4.18.1/source/drivers/dma-buf/dma-buf.c#L497
> >
> > 3) Calling VIDIOC_G_EXT_CTRLS for a request that has not been queued yet
> > will
> >return either the value of the control you set earlier in the request, or
> >the current HW control value if it was never set in the request.
> >
> >I believe it makes sense to return what was set in the request previously
> >(if you set it, you should be able to get it), but it is an idea to
> > return
> >ENOENT when calling this for controls that are NOT in the request.
> >
> >I'm inclined to implement that. Opinions?
>
> Return the request "cached" value, IMO, doesn't make sense. If the
> application needs such cache, it can implement itself.
Can we think about any specific use cases for a user space that first
sets a control value to a request and then needs to ask the kernel to
get the value back? After all, it was the user space which set the
value, so I'm not sure if there is any need for the kernel to be an
intermediary here.
>
> Return an error code if the request has not yet completed makes
> sense. Not sure what would be the best error code here... if the
> request is queued already (but not processed), EBUSY seems to be the
> better choice, but, if it was not queued yet, I'm not sure. I guess
> ENOENT would work.
IMHO, as far as we assign unique error codes for different conditions
and document them well, we should be okay with any not absurdly
mismatched code. After all, most of those codes are defined for file
system operations and don't really map directly to anything else.
FYI, VIDIOC_G_(EXT_)CTRL returns EINVAL if an unsupported control is
Re: [PATCH v2 5/6] media: Add controls for jpeg quantization tables
Hi Ezequiel,
On Fri, Aug 3, 2018 at 5:00 AM Ezequiel Garcia wrote:
>
> From: Shunqian Zheng
>
> Add V4L2_CID_JPEG_LUMA/CHROMA_QUANTIZATION controls to allow userspace
> configure the JPEG quantization tables.
>
> Signed-off-by: Shunqian Zheng
> Signed-off-by: Ezequiel Garcia
> ---
> Documentation/media/uapi/v4l/extended-controls.rst | 9 +
> drivers/media/v4l2-core/v4l2-ctrls.c | 4
> include/uapi/linux/v4l2-controls.h | 3 +++
> 3 files changed, 16 insertions(+)
Thanks for this series and sorry for being late with review. Please
see my comments inline.
>
> diff --git a/Documentation/media/uapi/v4l/extended-controls.rst
> b/Documentation/media/uapi/v4l/extended-controls.rst
> index 9f7312bf3365..80e26f81900b 100644
> --- a/Documentation/media/uapi/v4l/extended-controls.rst
> +++ b/Documentation/media/uapi/v4l/extended-controls.rst
> @@ -3354,6 +3354,15 @@ JPEG Control IDs
> Specify which JPEG markers are included in compressed stream. This
> control is valid only for encoders.
>
> +.. _jpeg-quant-tables-control:
> +
> +``V4L2_CID_JPEG_LUMA_QUANTIZATION (__u8 matrix)``
> +Sets the luma quantization table to be used for encoding
> +or decoding a V4L2_PIX_FMT_JPEG_RAW format buffer. This table is
> +expected to be in JPEG zigzag order, as per the JPEG specification.
Should we also specify this to be 8x8?
> +
> +``V4L2_CID_JPEG_CHROMA_QUANTIZATION (__u8 matrix)``
> +Sets the chroma quantization table.
>
nit: I guess we aff something like
"See also V4L2_CID_JPEG_LUMA_QUANTIZATION for details."
to avoid repeating the V4L2_PIX_FMT_JPEG_RAW and zigzag order bits? Or
maybe just repeating is better?
>
> .. flat-table::
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c
> b/drivers/media/v4l2-core/v4l2-ctrls.c
> index 599c1cbff3b9..5c62c3101851 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> @@ -999,6 +999,8 @@ const char *v4l2_ctrl_get_name(u32 id)
> case V4L2_CID_JPEG_RESTART_INTERVAL:return "Restart Interval";
> case V4L2_CID_JPEG_COMPRESSION_QUALITY: return "Compression Quality";
> case V4L2_CID_JPEG_ACTIVE_MARKER: return "Active Markers";
> + case V4L2_CID_JPEG_LUMA_QUANTIZATION: return "Luminance
> Quantization Matrix";
> + case V4L2_CID_JPEG_CHROMA_QUANTIZATION: return "Chrominance
> Quantization Matrix";
>
> /* Image source controls */
> /* Keep the order of the 'case's the same as in v4l2-controls.h! */
> @@ -1284,6 +1286,8 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum
> v4l2_ctrl_type *type,
> *flags |= V4L2_CTRL_FLAG_READ_ONLY;
> break;
> case V4L2_CID_DETECT_MD_REGION_GRID:
> + case V4L2_CID_JPEG_LUMA_QUANTIZATION:
> + case V4L2_CID_JPEG_CHROMA_QUANTIZATION:
It looks like with this setup, the driver has to explicitly set dims
to { 8, 8 } and min/max to 0/255.
At least for min and max, we could set them here. For dims, i don't
see it handled in generic code, so I guess we can leave it to the
driver now and add move into generic code, if another driver shows up.
Hans, what do you think?
Best regards,
Tomasz
Re: [PATCHv16 01/34] Documentation: v4l: document request API
Hi Hans, On Fri, Jul 6, 2018 at 1:04 AM Hans Verkuil wrote: > > From: Alexandre Courbot > > Document the request API for V4L2 devices, and amend the documentation > of system calls influenced by it. > > Signed-off-by: Alexandre Courbot > Signed-off-by: Hans Verkuil > --- > .../media/uapi/mediactl/media-controller.rst | 1 + > .../media/uapi/mediactl/media-funcs.rst | 6 + > .../uapi/mediactl/media-ioc-request-alloc.rst | 77 ++ > .../uapi/mediactl/media-request-ioc-queue.rst | 81 ++ > .../mediactl/media-request-ioc-reinit.rst | 51 > .../media/uapi/mediactl/request-api.rst | 247 ++ > .../uapi/mediactl/request-func-close.rst | 49 > .../uapi/mediactl/request-func-ioctl.rst | 68 + > .../media/uapi/mediactl/request-func-poll.rst | 74 ++ > Documentation/media/uapi/v4l/buffer.rst | 21 +- > .../media/uapi/v4l/vidioc-g-ext-ctrls.rst | 94 --- > Documentation/media/uapi/v4l/vidioc-qbuf.rst | 32 ++- > .../media/videodev2.h.rst.exceptions | 1 + > 13 files changed, 766 insertions(+), 36 deletions(-) > create mode 100644 > Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst > create mode 100644 > Documentation/media/uapi/mediactl/media-request-ioc-queue.rst > create mode 100644 > Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst > create mode 100644 Documentation/media/uapi/mediactl/request-api.rst > create mode 100644 Documentation/media/uapi/mediactl/request-func-close.rst > create mode 100644 Documentation/media/uapi/mediactl/request-func-ioctl.rst > create mode 100644 Documentation/media/uapi/mediactl/request-func-poll.rst > Thanks a lot for working on this and sorry for being late to the party. Please see some comments inline. (Mostly wording nits, though.) > diff --git a/Documentation/media/uapi/mediactl/media-controller.rst > b/Documentation/media/uapi/mediactl/media-controller.rst > index 0eea4f9a07d5..66aff38cd499 100644 > --- a/Documentation/media/uapi/mediactl/media-controller.rst > +++ b/Documentation/media/uapi/mediactl/media-controller.rst > @@ -21,6 +21,7 @@ Part IV - Media Controller API > media-controller-intro > media-controller-model > media-types > +request-api > media-funcs > media-header > > diff --git a/Documentation/media/uapi/mediactl/media-funcs.rst > b/Documentation/media/uapi/mediactl/media-funcs.rst > index 076856501cdb..260f9dcadcde 100644 > --- a/Documentation/media/uapi/mediactl/media-funcs.rst > +++ b/Documentation/media/uapi/mediactl/media-funcs.rst > @@ -16,3 +16,9 @@ Function Reference > media-ioc-enum-entities > media-ioc-enum-links > media-ioc-setup-link > +media-ioc-request-alloc > +request-func-close > +request-func-ioctl > +request-func-poll > +media-request-ioc-queue > +media-request-ioc-reinit > diff --git a/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst > b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst > new file mode 100644 > index ..8f8ecf6c63b6 > --- /dev/null > +++ b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst > @@ -0,0 +1,77 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +.. _media_ioc_request_alloc: > + > +* > +ioctl MEDIA_IOC_REQUEST_ALLOC > +* > + > +Name > + > + > +MEDIA_IOC_REQUEST_ALLOC - Allocate a request > + > + > +Synopsis > + > + > +.. c:function:: int ioctl( int fd, MEDIA_IOC_REQUEST_ALLOC, struct > media_request_alloc *argp ) > +:name: MEDIA_IOC_REQUEST_ALLOC > + > + > +Arguments > += > + > +``fd`` > +File descriptor returned by :ref:`open() `. > + > +``argp`` > + Missing description. > + > +Description > +=== > + > +If the media device supports :ref:`requests `, then > +this ioctl can be used to allocate a request. If it is not supported, then > +``errno`` is set to ``ENOTTY``. A request is accessed through a file > descriptor > +that is returned in struct :c:type:`media_request_alloc`. > + > +If the request was successfully allocated, then the request file descriptor > +can be passed to the :ref:`VIDIOC_QBUF `, > +:ref:`VIDIOC_G_EXT_CTRLS `, > +:ref:`VIDIOC_S_EXT_CTRLS ` and > +:ref:`VIDIOC_TRY_EXT_CTRLS ` ioctls. > + > +In addition, the request can be queued by calling > +:ref:`MEDIA_REQUEST_IOC_QUEUE` and re-initialized by calling > +:ref:`MEDIA_REQUEST_IOC_REINIT`. > + > +Finally, the file descriptor can be :ref:`polled ` to wait > +for the request to complete. > + > +The request will remain allocated until the associated file descriptor is > +closed by :ref:`close() ` and the driver no longer uses > +the request internally. I suppose that would be "until all the file descriptors associated with it are closed and", since one could dup() the original file descriptor and even close it, keeping just the copy. > + > +.. c:type:: media_request_alloc > + > +.. tabularcolumns::
Re: [RESEND PATCH v4] media: imx208: Add imx208 camera sensor driver
On Fri, Aug 3, 2018 at 11:57 AM Ping-chung Chen
wrote:
>
> From: "Chen, Ping-chung"
>
> Add a V4L2 sub-device driver for the Sony IMX208 image sensor.
> This is a camera sensor using the I2C bus for control and the
> CSI-2 bus for data.
>
> Signed-off-by: Ping-Chung Chen
> ---
> since v1:
> -- Update the function media_entity_pads_init for upstreaming.
> -- Change the structure name mutex as imx208_mx.
> -- Refine the control flow of test pattern function.
> -- vflip/hflip control support (will impact the output bayer order)
> -- support 4 bayer orders output (via change v/hflip)
> - SRGGB10(default), SGRBG10, SGBRG10, SBGGR10
> -- Simplify error handling in the set_stream function.
> since v2:
> -- Refine coding style.
> -- Fix the if statement to use pm_runtime_get_if_in_use().
> -- Print more error log during error handling.
> -- Remove mutex_destroy() from imx208_free_controls().
> -- Add more comments.
> since v3:
> -- Set explicit indices to link frequencies.
>
[snip]
> +/* Menu items for LINK_FREQ V4L2 control */
> +static const s64 link_freq_menu_items[] = {
> + [IMX208_LINK_FREQ_384MHZ_INDEX] = IMX208_LINK_FREQ_384MHZ,
> + [IMX208_LINK_FREQ_384MHZ_INDEX] = IMX208_LINK_FREQ_96MHZ,
IMX208_LINK_FREQ_96MHZ_INDEX?
With this fixed (and if there are no other changes), feel free to add
Reviewed-by: Tomasz Figa
Best regards,
Tomasz
Re: [PATCH v3] media: imx208: Add imx208 camera sensor driver
Hi Ping-chung,
On Wed, Aug 1, 2018 at 11:31 AM Ping-chung Chen
wrote:
>
> From: "Chen, Ping-chung"
>
> Add a V4L2 sub-device driver for the Sony IMX208 image sensor.
> This is a camera sensor using the I2C bus for control and the
> CSI-2 bus for data.
>
> Signed-off-by: Ping-Chung Chen
> ---
> since v1:
> -- Update the function media_entity_pads_init for upstreaming.
> -- Change the structure name mutex as imx208_mx.
> -- Refine the control flow of test pattern function.
> -- vflip/hflip control support (will impact the output bayer order)
> -- support 4 bayer orders output (via change v/hflip)
> - SRGGB10(default), SGRBG10, SGBRG10, SBGGR10
> -- Simplify error handling in the set_stream function.
> since v2:
> -- Refine coding style.
> -- Fix the if statement to use pm_runtime_get_if_in_use().
> -- Print more error log during error handling.
> -- Remove mutex_destroy() from imx208_free_controls().
> -- Add more comments.
Thanks for addressing the comments. There are still few remaining,
though. Please see inline.
[snip]
> +enum {
> + IMX208_LINK_FREQ_384MHZ_INDEX,
> + IMX208_LINK_FREQ_96MHZ_INDEX,
> +};
> +
> +/*
> + * pixel_rate = link_freq * data-rate * nr_of_lanes / bits_per_sample
> + * data rate => double data rate; number of lanes => 2; bits per pixel => 10
> + */
> +static u64 link_freq_to_pixel_rate(u64 f)
> +{
> + f *= IMX208_DATA_RATE_DOUBLE * IMX208_NUM_OF_LANES;
> + do_div(f, IMX208_PIXEL_BITS);
> +
> + return f;
> +}
> +
> +/* Menu items for LINK_FREQ V4L2 control */
> +static const s64 link_freq_menu_items[] = {
> + IMX208_LINK_FREQ_384MHZ,
> + IMX208_LINK_FREQ_96MHZ,
I asked for having explicit indices using IMX208_LINK_FREQ_*_INDEX
enum used here too.
> +};
> +
> +/* Link frequency configs */
> +static const struct imx208_link_freq_config link_freq_configs[] = {
> + [IMX208_LINK_FREQ_384MHZ_INDEX] = {
> + .pixels_per_line = IMX208_PPL_384MHZ,
> + .reg_list = {
> + .num_of_regs = ARRAY_SIZE(pll_ctrl_reg),
> + .regs = pll_ctrl_reg,
> + }
> + },
> + [IMX208_LINK_FREQ_96MHZ_INDEX] = {
> + .pixels_per_line = IMX208_PPL_96MHZ,
> + .reg_list = {
> + .num_of_regs = ARRAY_SIZE(pll_ctrl_reg),
> + .regs = pll_ctrl_reg,
> + }
> + },
> +};
> +
> +/* Mode configs */
> +static const struct imx208_mode supported_modes[] = {
> + [IMX208_LINK_FREQ_384MHZ_INDEX] = {
This is not the right index for this array (even if numerically the
same. This array is just a list of available modes (resolutions) and
there is no other data structure referring to particular entries of
it, so it doesn't need explicit indexing.
> + .width = 1936,
> + .height = 1096,
> + .vts_def = IMX208_VTS_60FPS,
> + .vts_min = IMX208_VTS_60FPS_MIN,
> + .reg_list = {
> + .num_of_regs = ARRAY_SIZE(mode_1936x1096_60fps_regs),
> + .regs = mode_1936x1096_60fps_regs,
> + },
> + .link_freq_index = IMX208_LINK_FREQ_384MHZ_INDEX,
> + },
> + [IMX208_LINK_FREQ_96MHZ_INDEX] = {
Ditto.
Best regards,
Tomasz
Re: [PATCH v2] media: imx208: Add imx208 camera sensor driver
On Tue, Jul 31, 2018 at 12:54 PM Chen, Ping-chung
wrote:
>
> Hi Tomasz,
>
> >-Original Message-
> > +/* Get bayer order based on flip setting. */ static __u32
> > +imx208_get_format_code(struct imx208 *imx208)
>
> >Why not just "u32"?
>
> Its return value will be assigned to the variable code which belongs to the
> structure
> v4l2_subdev_mbus_code_enum, and the type of this variable is __u32.
> struct v4l2_subdev_mbus_code_enum {
> __u32 pad;
> __u32 index;
> __u32 code;
> __u32 which;
> __u32 reserved[8];
> };
>
> > +{
> > + /*
> > +* Only one bayer order is supported.
> > +* It depends on the flip settings.
> > +*/
> > + static const __u32 codes[2][2] = {
>
> >Ditto.
>
> > + { MEDIA_BUS_FMT_SRGGB10_1X10, MEDIA_BUS_FMT_SGRBG10_1X10, },
> > + { MEDIA_BUS_FMT_SGBRG10_1X10, MEDIA_BUS_FMT_SBGGR10_1X10, },
> > + };
> > +
> > + return codes[imx208->vflip->val][imx208->hflip->val];
> > +}
> > +
That's okay. __u32 is an equivalent of u32 defined to be used in UAPI
headers. Inside of kernel-only code, u32 should be used.
Best regards,
Tomasz
Re: [PATCH 1/1] i2c: Fix pm_runtime_get_if_in_use() usage in sensor drivers
On Mon, Jul 30, 2018 at 9:12 PM Sakari Ailus wrote: > > pm_runtime_get_if_in_use() returns -EINVAL if runtime PM is disabled. This > should not be considered an error. Generally the driver has enabled > runtime PM already so getting this error due to runtime PM being disabled > will not happen. > > Instead of checking for lesser or equal to zero, check for zero only. > Address this for drivers where this pattern exists. > > This patch has been produced using the following command: > > $ git grep -l pm_runtime_get_if_in_use -- drivers/media/i2c/ | \ > xargs perl -i -pe 's/(pm_runtime_get_if_in_use\(.*\)) \<\= 0/!$1/' > > Signed-off-by: Sakari Ailus > --- > drivers/media/i2c/ov13858.c | 2 +- > drivers/media/i2c/ov2685.c | 2 +- > drivers/media/i2c/ov5670.c | 2 +- > drivers/media/i2c/ov5695.c | 2 +- > drivers/media/i2c/ov7740.c | 2 +- > 5 files changed, 5 insertions(+), 5 deletions(-) Thanks! Reviewed-by: Tomasz Figa Best regards, Tomasz
Re: [PATCH v2] media: imx208: Add imx208 camera sensor driver
On Mon, Jul 30, 2018 at 8:39 PM Sakari Ailus
wrote:
>
> Hi Tomasz,
>
> On Mon, Jul 30, 2018 at 07:19:56PM +0900, Tomasz Figa wrote:
> ...
> > > +static int imx208_set_ctrl(struct v4l2_ctrl *ctrl)
> > > +{
> > > + struct imx208 *imx208 =
> > > + container_of(ctrl->handler, struct imx208, ctrl_handler);
> > > + struct i2c_client *client = v4l2_get_subdevdata(>sd);
> > > + int ret;
> > > +
> > > + /*
> > > +* Applying V4L2 control value only happens
> > > +* when power is up for streaming
> > > +*/
> > > + if (pm_runtime_get_if_in_use(>dev) <= 0)
> >
> > This is buggy, because it won't handle the case of runtime PM disabled
> > in kernel config. The check should be
> > (!pm_runtime_get_if_in_use(>dev)).
>
> Good find. This seems to be the case for most other sensor drivers that
> make use of the function. I can submit a fix for those as well.
>
> I suppose most people use these with runtime PM enabled as this hasn't been
> spotted previously.
Yeah, I spotted it first with imx258 and it took us few emails to get
to the right code. :)
These drivers probably don't have too many users yet in general, so I
guess this didn't manage to cause any problems yet, but it became a
good example of bug propagation via copy/paste. ;)
Fixing would be appreciated indeed!
Best regards,
Tomasz
Re: [PATCH v2] media: imx208: Add imx208 camera sensor driver
Hi Ping-chung,
On Mon, Jul 30, 2018 at 6:19 PM Ping-chung Chen
wrote:
>
> From: "Chen, Ping-chung"
>
> Add a V4L2 sub-device driver for the Sony IMX208 image sensor.
> This is a camera sensor using the I2C bus for control and the
> CSI-2 bus for data.
>
Please see my comments inline.
[snip]
> diff --git a/drivers/media/i2c/imx208.c b/drivers/media/i2c/imx208.c
> new file mode 100644
> index 000..5adfb79
> --- /dev/null
> +++ b/drivers/media/i2c/imx208.c
> @@ -0,0 +1,984 @@
> +// SPDX-License-Identifier: GPL-2.0
> +// Copyright (C) 2018 Intel Corporation
> +
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +#include
> +
> +#define IMX208_REG_VALUE_08BIT 1
> +#define IMX208_REG_VALUE_16BIT 2
We don't need to define these. It's obvious that 8 bits is 1 byte and
16 bits are 2 bytes.
> +
> +#define IMX208_REG_MODE_SELECT 0x0100
> +#define IMX208_MODE_STANDBY0x00
> +#define IMX208_MODE_STREAMING 0x01
[snip]
> +/* Test Pattern Control */
> +#define IMX208_REG_TEST_PATTERN_MODE 0x0600
> +#define IMX208_TEST_PATTERN_DISABLE0
> +#define IMX208_TEST_PATTERN_SOLID_COLOR1
> +#define IMX208_TEST_PATTERN_COLOR_BARS 2
> +#define IMX208_TEST_PATTERN_GREY_COLOR 3
> +#define IMX208_TEST_PATTERN_PN94
Please use hexadecimal notation for register values (as already done below).
> +#define IMX208_TEST_PATTERN_FIX_1 0x100
> +#define IMX208_TEST_PATTERN_FIX_2 0x101
> +#define IMX208_TEST_PATTERN_FIX_3 0x102
> +#define IMX208_TEST_PATTERN_FIX_4 0x103
> +#define IMX208_TEST_PATTERN_FIX_5 0x104
> +#define IMX208_TEST_PATTERN_FIX_6 0x105
[snip]
> +static const int imx208_test_pattern_val[] = {
> + IMX208_TEST_PATTERN_DISABLE,
> + IMX208_TEST_PATTERN_SOLID_COLOR,
> + IMX208_TEST_PATTERN_COLOR_BARS,
> + IMX208_TEST_PATTERN_GREY_COLOR,
> + IMX208_TEST_PATTERN_PN9,
> + IMX208_TEST_PATTERN_FIX_1,
> + IMX208_TEST_PATTERN_FIX_2,
> + IMX208_TEST_PATTERN_FIX_3,
> + IMX208_TEST_PATTERN_FIX_4,
> + IMX208_TEST_PATTERN_FIX_5,
> + IMX208_TEST_PATTERN_FIX_6,
> +};
> +
> +/* Configurations for supported link frequencies */
> +#define IMX208_LINK_FREQ_384MHZ38400ULL
> +#define IMX208_LINK_FREQ_96MHZ 9600ULL
nit: If we really need defines for these, then at least they should be
somehow useful, e.g.
#define MHZ (1000*1000ULL)
#define IMX208_LINK_FREQ_384MHZ (384ULL * MHZ)
This at least makes it easy to see that there are no mistakes in the
number, e.g. wrong number of zeroes.
> +
> +enum {
> + IMX208_LINK_FREQ_384MHZ_INDEX,
> + IMX208_LINK_FREQ_96MHZ_INDEX,
> +};
> +
> +/*
> + * pixel_rate = link_freq * data-rate * nr_of_lanes / bits_per_sample
> + * data rate => double data rate; number of lanes => 2; bits per pixel => 10
> + */
> +static u64 link_freq_to_pixel_rate(u64 f)
> +{
> + f *= 2 * 2;
> + do_div(f, 10);
Please add macros for those magic numbers.
> +
> + return f;
> +}
> +
> +/* Menu items for LINK_FREQ V4L2 control */
> +static const s64 link_freq_menu_items[] = {
> + IMX208_LINK_FREQ_384MHZ,
> + IMX208_LINK_FREQ_96MHZ,
Since we have an enum already, please use it for explicit indices, to
ensure things are consistent and actually easier to read, i.e.
[IMX208_LINK_FREQ_384MHZ_INDEX] = IMX208_LINK_FREQ_384MHZ,
> +};
> +
> +/* Link frequency configs */
> +static const struct imx208_link_freq_config link_freq_configs[] = {
> + {
Explicit indices, i.e.
[IMX208_LINK_FREQ_384MHZ_INDEX] = {
> + .pixels_per_line = IMX208_PPL_384MHZ,
> + .reg_list = {
> + .num_of_regs = ARRAY_SIZE(mipi_data_rate),
> + .regs = mipi_data_rate,
> + }
> + },
> + {
Ditto.
> + .pixels_per_line = IMX208_PPL_96MHZ,
> + .reg_list = {
> + .num_of_regs = ARRAY_SIZE(mipi_data_rate),
> + .regs = mipi_data_rate,
How comes that both link frequencies have the same register values for
MIPI data rate?
> + }
> + },
> +};
> +
> +/* Mode configs */
> +static const struct imx208_mode supported_modes[] = {
> + {
> + .width = 1936,
> + .height = 1096,
> + .vts_def = IMX208_VTS_60FPS,
> + .vts_min = IMX208_VTS_60FPS_MIN,
> + .reg_list = {
> + .num_of_regs = ARRAY_SIZE(mode_1936x1096_60fps_regs),
> + .regs = mode_1936x1096_60fps_regs,
> + },
> + .link_freq_index = 0,
Please use the index that was defined before - IMX208_LINK_FREQ_384MHZ_INDEX.
> + },
> + {
> + .width = 968,
> + .height = 548,
> + .vts_def = IMX208_VTS_BINNING,
> + .vts_min = IMX208_VTS_BINNING_MIN,
> +
Re: [PATCH 0/2] Document memory-to-memory video codec interfaces
Hi Philipp,
On Wed, Jul 25, 2018 at 10:28 PM Philipp Zabel wrote:
>
> Hi Tomasz,
>
> On Tue, 2018-07-24 at 23:06 +0900, Tomasz Figa wrote:
> > This series attempts to add the documentation of what was discussed
> > during Media Workshops at LinuxCon Europe 2012 in Barcelona and then
> > later Embedded Linux Conference Europe 2014 in Düsseldorf and then
> > eventually written down by Pawel Osciak and tweaked a bit by Chrome OS
> > video team (but mostly in a cosmetic way or making the document more
> > precise), during the several years of Chrome OS using the APIs in
> > production.
> >
> > Note that most, if not all, of the API is already implemented in
> > existing mainline drivers, such as s5p-mfc or mtk-vcodec. Intention of
> > this series is just to formalize what we already have.
> >
> > It is an initial conversion from Google Docs to RST, so formatting is
> > likely to need some further polishing. It is also the first time for me
> > to create such long RST documention. I could not find any other instance
> > of similar userspace sequence specifications among our Media documents,
> > so I mostly followed what was there in the source. Feel free to suggest
> > a better format.
> >
> > Much of credits should go to Pawel Osciak, for writing most of the
> > original text of the initial RFC.
> >
> > Changes since RFC:
> > (https://lore.kernel.org/patchwork/project/lkml/list/?series=348588)
> > - The number of changes is too big to list them all here. Thanks to
> >a huge number of very useful comments from everyone (Philipp, Hans,
> >Nicolas, Dave, Stanimir, Alexandre) we should have the interfaces much
> >more specified now. The issues collected since previous revisions and
> >answers leading to this revision are listed below.
>
> Thanks a lot for the update, and especially for the nice Q summary of
> the discussions so far.
>
> [...]
> > Decoder issues
> >
> [...]
> > How should ENUM_FRAMESIZES be affected by profiles and levels?
> >
> > Answer: Not in current specification - the logic is too complicated and
> > it might make more sense to actually handle this in user space. (In
> > theory, level implies supported frame sizes + other factors.)
>
> For decoding I think it makes more sense to let the hardware decode them
> from the stream and present them as read-only controls, such as:
>
> 42a68012e67c ("media: coda: add read-only h.264 decoder profile/level
> controls")
To clarify, this point is only about the effect on ENUM_FRAMESIZES.
Profile and level controls are mentioned in capabilities enumeration,
but it may make sense to add optional steps of querying them in
Initialization sequence.
>
> if possible. For encoding, the coda firmware determines level from
> bitrate and coded resolution, itself, so I agree not making this part of
> the spec is a good idea for now.
Encoder controls are driver-specific in general, since the encoding
capabilities vary a lot, so I decided to just briefly mention the
general idea of encoding parameters in "Encoding parameter changes"
section. It could be a good idea to add a reference to the MPEG
control documentation there, though.
Best regards,
Tomasz
Re: [PATCH] media: imx208: Add imx208 camera sensor driver
Hi Ping-chung, On Tue, Jul 17, 2018 at 3:53 PM Chen, Ping-chung wrote: > > Hi Sakari, > > Some of suggestions below has been added in to [PATCH v2] or [PATCH v3]. > We will fix others in PATCH v4 soon. I don't see v2 or v3 on linux-media (or my inbox). Where were they sent? I'd like to review the driver, but if there is already v3, it would make sense to review that one. Also, please refrain from top-posting on mailing lists, it's considered bad manner (and makes threads difficult to read). Best regards, Tomasz
Re: [PATCH v1 2/2] v4l: Document Intel IPU3 meta data uAPI
Hi Mauro, Hans, On Fri, Jun 15, 2018 at 12:30 PM Yong Zhi wrote: > > These meta formats are used on Intel IPU3 ImgU video queues > to carry 3A statistics and ISP pipeline parameters. > > V4L2_META_FMT_IPU3_3A > V4L2_META_FMT_IPU3_PARAMS > > Signed-off-by: Yong Zhi > Signed-off-by: Chao C Li > Signed-off-by: Rajmohan Mani > --- > Documentation/media/uapi/v4l/meta-formats.rst |1 + > .../media/uapi/v4l/pixfmt-meta-intel-ipu3.rst | 174 ++ > include/uapi/linux/intel-ipu3.h| 2816 > The documentation seems to be quite extensive in current version. Do you think it's more acceptable now? Would you be able to take a look? We obviously need to keep working on the user space framework (and we're in process of figuring out how we can proceed further), but having the driver bit-rotting downstream might not be a very encouraging factor. ;) Best regards, Tomasz
Re: [PATCHv15 07/35] v4l2-dev: lock req_queue_mutex
Hi Hans,
On Mon, Jun 4, 2018 at 8:48 PM Hans Verkuil wrote:
[snip]
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c
> b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 965fd301f617..27a893aa0664 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -2714,6 +2714,7 @@ static long __video_do_ioctl(struct file *file,
> unsigned int cmd, void *arg)
> {
> struct video_device *vfd = video_devdata(file);
> + struct mutex *req_queue_lock = NULL;
> struct mutex *lock; /* ioctl serialization mutex */
> const struct v4l2_ioctl_ops *ops = vfd->ioctl_ops;
> bool write_only = false;
> @@ -2733,10 +2734,27 @@ static long __video_do_ioctl(struct file *file,
> if (test_bit(V4L2_FL_USES_V4L2_FH, >flags))
> vfh = file->private_data;
>
> + /*
> +* We need to serialize streamon/off with queueing new requests.
> +* These ioctls may trigger the cancellation of a streaming
> +* operation, and that should not be mixed with queueing a new
> +* request at the same time.
> +*/
> + if (v4l2_device_supports_requests(vfd->v4l2_dev) &&
> + (cmd == VIDIOC_STREAMON || cmd == VIDIOC_STREAMOFF)) {
Are STREAMON and STREAMOFF the only ioctls we need to acquire
req_queue_lock for? How about a race with, for example, S_CTRL(control
X, request_fd = -1) and req_validate() on a request that depends on
the value of control X? Maybe we should just lock here for any ioctl?
> + req_queue_lock = >v4l2_dev->mdev->req_queue_mutex;
> +
> + if (req_queue_lock &&
> mutex_lock_interruptible(req_queue_lock))
I believe it isn't possible for req_queue_lock to be NULL here.
> + return -ERESTARTSYS;
I guess it isn't really possible for mutex_lock_interruptible() to
return anything non-zero other than this, but I'd still return what it
returns here just in case.
Best regards,
Tomasz
Re: [PATCHv15 05/35] media-request: add media_request_object_find
Hi Hans, On Mon, Jun 4, 2018 at 8:48 PM Hans Verkuil wrote: > > From: Hans Verkuil > > Add media_request_object_find to find a request object inside a > request based on ops and/or priv values. Current code seems to always find based on both ops and priv values. Best regards, Tomasz
Re: [PATCHv15 02/35] media-request: implement media requests
On Mon, Jul 2, 2018 at 6:11 PM Hans Verkuil wrote:
>
> On 02/07/18 10:58, Tomasz Figa wrote:
> > Hi Hans,
> >
> > On Fri, Jun 15, 2018 at 4:09 PM Hans Verkuil wrote:
> >>
> >> On 04/06/18 13:46, Hans Verkuil wrote:
> >>> From: Hans Verkuil
> >>>
> >>> Add initial media request support:
> >>>
> >>> 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
> >>> 2) Add struct media_request to store request objects.
> >>> 3) Add struct media_request_object to represent a request object.
> >>> 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
> >>>
> >>> Basic lifecycle: the application allocates a request, adds
> >>> objects to it, queues the request, polls until it is completed
> >>> and can then read the final values of the objects at the time
> >>> of completion. When it closes the file descriptor the request
> >>> memory will be freed (actually, when the last user of that request
> >>> releases the request).
> >>>
> >>> Drivers will bind an object to a request (the 'adds objects to it'
> >>> phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
> >>> validated (req_validate op), then queued (the req_queue op).
> >>>
> >>> When done with an object it can either be unbound from the request
> >>> (e.g. when the driver has finished with a vb2 buffer) or marked as
> >>> completed (e.g. for controls associated with a buffer). When all
> >>> objects in the request are completed (or unbound), then the request
> >>> fd will signal an exception (poll).
> >>>
> >>> Signed-off-by: Hans Verkuil
> >>> Co-developed-by: Sakari Ailus
> >>> Signed-off-by: Sakari Ailus
> >>> Co-developed-by: Laurent Pinchart
> >>> Co-developed-by: Alexandre Courbot
> >>> ---
> >>> drivers/media/Makefile| 3 +-
> >>> drivers/media/media-device.c | 14 ++
> >>> drivers/media/media-request.c | 421 ++
> >>> include/media/media-device.h | 24 ++
> >>> include/media/media-request.h | 326 ++
> >>> 5 files changed, 787 insertions(+), 1 deletion(-)
> >>> create mode 100644 drivers/media/media-request.c
> >>> create mode 100644 include/media/media-request.h
> >>>
> >>
> >>
> >>
> >>> diff --git a/include/media/media-request.h b/include/media/media-request.h
> >>> new file mode 100644
> >>> index ..8acd2627835c
> >>> --- /dev/null
> >>> +++ b/include/media/media-request.h
> >>> @@ -0,0 +1,326 @@
> >>
> >>
> >>
> >>> +/**
> >>> + * struct media_request - Media device request
> >>> + * @mdev: Media device this request belongs to
> >>> + * @kref: Reference count
> >>> + * @debug_str: Prefix for debug messages (process name:fd)
> >>> + * @state: The state of the request
> >>> + * @updating_count: count the number of request updates that are in
> >>> progress
> >>> + * @objects: List of @struct media_request_object request objects
> >>> + * @num_incomplete_objects: The number of incomplete objects in the
> >>> request
> >>> + * @poll_wait: Wait queue for poll
> >>> + * @lock: Serializes access to this struct
> >>> + */
> >>> +struct media_request {
> >>> + struct media_device *mdev;
> >>> + struct kref kref;
> >>> + char debug_str[TASK_COMM_LEN + 11];
> >>> + enum media_request_state state;
> >>> + refcount_t updating_count;
> >>
> >> This will be replaced by unsigned int: it turns out that if
> >> CONFIG_REFCOUNT_FULL is set,
> >> the refcount implementation will complain when you increase the refcount
> >> from 0 to 1
> >> since it expects that refcount_t it used as it is in kref. Since
> >> updating_count is
> >> protected by the spinlock anyway there is no need to use refcount_t, a
> >> simple
> >> unsigned int works just as well.
> >
> > It seems to be read in media_request_clean(), which can be called by
> > media_request_release() or media_request_ioctl_reinit() and neither
> > acquires the spinlock for the time of the call.
>
> The request object is in state CLEANING at that time, and that guarantees
> that nobody else will make changes to the request.
Makes sense. Thanks for clarification.
Best regards,
Tomasz
Re: [PATCHv15 02/35] media-request: implement media requests
Hi Hans,
On Mon, Jun 4, 2018 at 8:47 PM Hans Verkuil wrote:
[snip]
> +static void media_request_object_release(struct kref *kref)
> +{
> + struct media_request_object *obj =
> + container_of(kref, struct media_request_object, kref);
> + struct media_request *req = obj->req;
> +
> + if (req)
> + media_request_object_unbind(obj);
Is it possible and fine to have a request bound here?
media_request_clean() seems to explicitly unbind before releasing and
this function would be only called if last reference is put, so maybe
we should actually WARN_ON(req)?
> + obj->ops->release(obj);
> +}
[snip]
> @@ -87,7 +104,12 @@ struct media_device_ops {
> * @enable_source: Enable Source Handler function pointer
> * @disable_source: Disable Source Handler function pointer
> *
> + * @req_queue_mutex: Serialise validating and queueing requests in order to
> + * guarantee exclusive access to the state of the device on
> + * the tip of the request queue.
> * @ops: Operation handler callbacks
> + * @req_queue_mutex: Serialise the MEDIA_REQUEST_IOC_QUEUE ioctl w.r.t.
> + * other operations that stop or start streaming.
Merge conflict? req_queue_mutex is documented twice.
Best regards,
Tomasz
Re: [PATCHv15 02/35] media-request: implement media requests
Hi Hans,
On Fri, Jun 15, 2018 at 4:09 PM Hans Verkuil wrote:
>
> On 04/06/18 13:46, Hans Verkuil wrote:
> > From: Hans Verkuil
> >
> > Add initial media request support:
> >
> > 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
> > 2) Add struct media_request to store request objects.
> > 3) Add struct media_request_object to represent a request object.
> > 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
> >
> > Basic lifecycle: the application allocates a request, adds
> > objects to it, queues the request, polls until it is completed
> > and can then read the final values of the objects at the time
> > of completion. When it closes the file descriptor the request
> > memory will be freed (actually, when the last user of that request
> > releases the request).
> >
> > Drivers will bind an object to a request (the 'adds objects to it'
> > phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
> > validated (req_validate op), then queued (the req_queue op).
> >
> > When done with an object it can either be unbound from the request
> > (e.g. when the driver has finished with a vb2 buffer) or marked as
> > completed (e.g. for controls associated with a buffer). When all
> > objects in the request are completed (or unbound), then the request
> > fd will signal an exception (poll).
> >
> > Signed-off-by: Hans Verkuil
> > Co-developed-by: Sakari Ailus
> > Signed-off-by: Sakari Ailus
> > Co-developed-by: Laurent Pinchart
> > Co-developed-by: Alexandre Courbot
> > ---
> > drivers/media/Makefile| 3 +-
> > drivers/media/media-device.c | 14 ++
> > drivers/media/media-request.c | 421 ++
> > include/media/media-device.h | 24 ++
> > include/media/media-request.h | 326 ++
> > 5 files changed, 787 insertions(+), 1 deletion(-)
> > create mode 100644 drivers/media/media-request.c
> > create mode 100644 include/media/media-request.h
> >
>
>
>
> > diff --git a/include/media/media-request.h b/include/media/media-request.h
> > new file mode 100644
> > index ..8acd2627835c
> > --- /dev/null
> > +++ b/include/media/media-request.h
> > @@ -0,0 +1,326 @@
>
>
>
> > +/**
> > + * struct media_request - Media device request
> > + * @mdev: Media device this request belongs to
> > + * @kref: Reference count
> > + * @debug_str: Prefix for debug messages (process name:fd)
> > + * @state: The state of the request
> > + * @updating_count: count the number of request updates that are in
> > progress
> > + * @objects: List of @struct media_request_object request objects
> > + * @num_incomplete_objects: The number of incomplete objects in the request
> > + * @poll_wait: Wait queue for poll
> > + * @lock: Serializes access to this struct
> > + */
> > +struct media_request {
> > + struct media_device *mdev;
> > + struct kref kref;
> > + char debug_str[TASK_COMM_LEN + 11];
> > + enum media_request_state state;
> > + refcount_t updating_count;
>
> This will be replaced by unsigned int: it turns out that if
> CONFIG_REFCOUNT_FULL is set,
> the refcount implementation will complain when you increase the refcount from
> 0 to 1
> since it expects that refcount_t it used as it is in kref. Since
> updating_count is
> protected by the spinlock anyway there is no need to use refcount_t, a simple
> unsigned int works just as well.
It seems to be read in media_request_clean(), which can be called by
media_request_release() or media_request_ioctl_reinit() and neither
acquires the spinlock for the time of the call.
Agreed, though, that refcount_t doesn't really match what we need
updating_count here for.
Best regards,
Tomasz
Re: [PATCH v6 12/12] intel-ipu3: Add imgu top level pci device driver
Hi Yong,
On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote:
> +/*
> + * Queue as many buffers to CSS as possible. If all buffers don't fit into
> + * CSS buffer queues, they remain unqueued and will be queued later.
> + */
> +int imgu_queue_buffers(struct imgu_device *imgu, bool initial)
> +{
> + unsigned int node;
> + int r = 0;
> + struct imgu_buffer *ibuf;
> +
> + if (!ipu3_css_is_streaming(>css))
> + return 0;
> +
> + mutex_lock(>lock);
> +
> + /* Buffer set is queued to FW only when input buffer is ready */
> + if (!imgu_queue_getbuf(imgu, IMGU_NODE_IN)) {
> + mutex_unlock(>lock);
> + return 0;
> + }
> + for (node = IMGU_NODE_IN + 1; 1; node = (node + 1) % IMGU_NODE_NUM) {
Shouldn't we make (node != IMGU_NODE_IN || imgu_queue_getbuf(imgu,
IMGU_NODE_IN)) the condition here, rather than 1?
This would also let us remove the explicit call to imgu_queue_getbuf()
above the loop.
> + if (node == IMGU_NODE_VF &&
> + (imgu->css.pipe_id == IPU3_CSS_PIPE_ID_CAPTURE ||
> +!imgu->nodes[IMGU_NODE_VF].enabled)) {
> + continue;
> + } else if (node == IMGU_NODE_PV &&
> + (imgu->css.pipe_id == IPU3_CSS_PIPE_ID_VIDEO ||
> + !imgu->nodes[IMGU_NODE_PV].enabled)) {
> + continue;
> + } else if (imgu->queue_enabled[node]) {
> + struct ipu3_css_buffer *buf =
> + imgu_queue_getbuf(imgu, node);
> + int dummy;
> +
> + if (!buf)
> + break;
> +
> + r = ipu3_css_buf_queue(>css, buf);
> + if (r)
> + break;
> + dummy = imgu_dummybufs_check(imgu, buf);
> + if (!dummy)
> + ibuf = container_of(buf, struct imgu_buffer,
> + css_buf);
> + dev_dbg(>pci_dev->dev,
> + "queue %s %s buffer %d to css da: 0x%08x\n",
> + dummy ? "dummy" : "user",
> + imgu_node_map[node].name,
> + dummy ? 0 : ibuf->vid_buf.vbb.vb2_buf.index,
> + (u32)buf->daddr);
> + }
> + if (node == IMGU_NODE_IN &&
> + !imgu_queue_getbuf(imgu, IMGU_NODE_IN))
> + break;
My suggestion to the for loop condition is based on this.
> + }
> + mutex_unlock(>lock);
> +
> + if (r && r != -EBUSY)
> + goto failed;
> +
> + return 0;
> +
> +failed:
> + /*
> +* On error, mark all buffers as failed which are not
> +* yet queued to CSS
> +*/
> + dev_err(>pci_dev->dev,
> + "failed to queue buffer to CSS on queue %i (%d)\n",
> + node, r);
> +
> + if (initial)
> + /* If we were called from streamon(), no need to finish bufs
> */
> + return r;
> +
> + for (node = 0; node < IMGU_NODE_NUM; node++) {
> + struct imgu_buffer *buf, *buf0;
> +
> + if (!imgu->queue_enabled[node])
> + continue; /* Skip disabled queues */
> +
> + mutex_lock(>lock);
> + list_for_each_entry_safe(buf, buf0,
> >nodes[node].buffers,
> +vid_buf.list) {
> + if (ipu3_css_buf_state(>css_buf) ==
> + IPU3_CSS_BUFFER_QUEUED)
> + continue; /* Was already queued, skip */
> +
> + ipu3_v4l2_buffer_done(>vid_buf.vbb.vb2_buf,
> + VB2_BUF_STATE_ERROR);
> + }
> + mutex_unlock(>lock);
> + }
> +
> + return r;
> +}
[snip]
> +static irqreturn_t imgu_isr_threaded(int irq, void *imgu_ptr)
> +{
> + struct imgu_device *imgu = imgu_ptr;
> +
> + /* Dequeue / queue buffers */
> + do {
> + u64 ns = ktime_get_ns();
> + struct ipu3_css_buffer *b;
> + struct imgu_buffer *buf;
> + unsigned int node;
> + bool dummy;
> +
> + do {
> + mutex_lock(>lock);
> + b = ipu3_css_buf_dequeue(>css);
> + mutex_unlock(>lock);
> + } while (PTR_ERR(b) == -EAGAIN);
> +
> + if (IS_ERR_OR_NULL(b)) {
> + if (!b || PTR_ERR(b) == -EBUSY) /* All done */
> + break;
> + dev_err(>pci_dev->dev,
> +
Re: [PATCH v6 11/12] intel-ipu3: Add v4l2 driver based on media framework
Hi Yong,
On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote:
[snip]
> +static int ipu3_vidioc_enum_input(struct file *file, void *fh,
> + struct v4l2_input *input)
> +{
> + if (input->index > 0)
> + return -EINVAL;
> + strlcpy(input->name, "camera", sizeof(input->name));
> + input->type = V4L2_INPUT_TYPE_CAMERA;
> +
> + return 0;
> +}
> +
> +static int ipu3_vidioc_g_input(struct file *file, void *fh, unsigned int
> *input)
> +{
> + *input = 0;
> +
> + return 0;
> +}
> +
> +static int ipu3_vidioc_s_input(struct file *file, void *fh, unsigned int
> input)
> +{
> + return input == 0 ? 0 : -EINVAL;
> +}
> +
> +static int ipu3_vidioc_enum_output(struct file *file, void *fh,
> + struct v4l2_output *output)
> +{
> + if (output->index > 0)
> + return -EINVAL;
> + strlcpy(output->name, "camera", sizeof(output->name));
> + output->type = V4L2_INPUT_TYPE_CAMERA;
> +
> + return 0;
> +}
> +
> +static int ipu3_vidioc_g_output(struct file *file, void *fh,
> + unsigned int *output)
> +{
> + *output = 0;
> +
> + return 0;
> +}
> +
> +static int ipu3_vidioc_s_output(struct file *file, void *fh,
> + unsigned int output)
> +{
> + return output == 0 ? 0 : -EINVAL;
> +}
Do we really need to implement the 6 functions above? They don't seem
to be doing anything useful.
[snip]
> +int ipu3_v4l2_register(struct imgu_device *imgu)
> +{
> + struct v4l2_mbus_framefmt def_bus_fmt = { 0 };
> + struct v4l2_pix_format_mplane def_pix_fmt = { 0 };
> +
> + int i, r;
> +
> + /* Initialize miscellaneous variables */
> + imgu->streaming = false;
> +
> + /* Set up media device */
> + imgu->media_dev.dev = >pci_dev->dev;
> + strlcpy(imgu->media_dev.model, IMGU_NAME,
> + sizeof(imgu->media_dev.model));
> + snprintf(imgu->media_dev.bus_info, sizeof(imgu->media_dev.bus_info),
> +"%s", dev_name(>pci_dev->dev));
> + imgu->media_dev.hw_revision = 0;
> + media_device_init(>media_dev);
> + r = media_device_register(>media_dev);
> + if (r) {
> + dev_err(>pci_dev->dev,
> + "failed to register media device (%d)\n", r);
> + return r;
> + }
Shouldn't we register the media device at the end, after all video
nodes are registered below? Otherwise, since media_device_register()
exposes the media node to userspace, we risk a race, when userspace
opens the media device before all the entities are created and linked.
[snip]
> +int ipu3_v4l2_unregister(struct imgu_device *imgu)
> +{
> + unsigned int i;
> +
> + for (i = 0; i < IMGU_NODE_NUM; i++) {
> + video_unregister_device(>nodes[i].vdev);
> + media_entity_cleanup(>nodes[i].vdev.entity);
> + mutex_destroy(>nodes[i].lock);
> + }
> +
> + v4l2_device_unregister_subdev(>subdev);
> + media_entity_cleanup(>subdev.entity);
> + kfree(imgu->subdev_pads);
> + v4l2_device_unregister(>v4l2_dev);
> + media_device_unregister(>media_dev);
Should unregister media device at the beginning, so that it cannot be
used when we continue to clean up the entities.
> + media_device_cleanup(>media_dev);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(ipu3_v4l2_unregister);
Best regards,
Tomasz
Re: [PATCH v6 06/12] intel-ipu3: css: Add support for firmware management
Hi Yong,
Continuing my review. Sorry for the delay.
On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote:
>
> Introduce functions to load and install ImgU FW blobs.
>
> Signed-off-by: Yong Zhi
> ---
> drivers/media/pci/intel/ipu3/ipu3-abi.h| 1888
>
> drivers/media/pci/intel/ipu3/ipu3-css-fw.c | 261
> drivers/media/pci/intel/ipu3/ipu3-css-fw.h | 198 +++
> 3 files changed, 2347 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-abi.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-fw.h
>
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-abi.h
> b/drivers/media/pci/intel/ipu3/ipu3-abi.h
> new file mode 100644
> index ..24102647a89e
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-abi.h
[snip]
> +/* SYSTEM_REQ_0_5_0_IMGHMMADR */
> +#define IMGU_REG_SYSTEM_REQ0x18
> +#define IMGU_SYSTEM_REQ_FREQ_MASK 0x3f
> +#define IMGU_SYSTEM_REQ_FREQ_DIVIDER 25
> +#define IMGU_REG_INT_STATUS0x30
> +#define IMGU_REG_INT_ENABLE0x34
> +#define IMGU_REG_INT_CSS_IRQ (1 << 31)
BIT(31)
[snip]
> + IMGU_ABI_FRAME_FORMAT_CSI_MIPI_LEGACY_YUV420_8, /* Legacy YUV420.
> +* UY odd line;
> +* VY even line
> +*/
> + IMGU_ABI_FRAME_FORMAT_CSI_MIPI_YUV420_10,/* 10 bit per Y/U/V. Y odd
> + * line; UYVY interleaved
> + * even line
> + */
> + IMGU_ABI_FRAME_FORMAT_YCgCo444_16, /* Internal format for ISP2.7,
Macros and enums should be uppercase.
[snip]
> +struct imgu_abi_shd_intra_frame_operations_data {
> + struct imgu_abi_acc_operation
> + operation_list[IMGU_ABI_SHD_MAX_OPERATIONS] IPU3_ALIGN;
> + struct imgu_abi_acc_process_lines_cmd_data
> + process_lines_data[IMGU_ABI_SHD_MAX_PROCESS_LINES] IPU3_ALIGN;
> + struct imgu_abi_shd_transfer_luts_set_data
> + transfer_data[IMGU_ABI_SHD_MAX_TRANSFERS] IPU3_ALIGN;
> +} __packed;
> +
> +struct imgu_abi_shd_config {
> + struct ipu3_uapi_shd_config_static shd IMGU_ABI_PAD;
> + struct imgu_abi_shd_intra_frame_operations_data shd_ops IMGU_ABI_PAD;
> + struct ipu3_uapi_shd_lut shd_lut IMGU_ABI_PAD;
Definitions of both IPU3_ALIGN and IMGU_ABI_PAD seem to be equivalent.
Could we remove one and use the other everywhere?
[snip]
> +struct imgu_abi_osys_scaler_params {
> + __u32 inp_buf_y_st_addr;
> + __u32 inp_buf_y_line_stride;
> + __u32 inp_buf_y_buffer_stride;
> + __u32 inp_buf_u_st_addr;
> + __u32 inp_buf_v_st_addr;
> + __u32 inp_buf_uv_line_stride;
> + __u32 inp_buf_uv_buffer_stride;
> + __u32 inp_buf_chunk_width;
> + __u32 inp_buf_nr_buffers;
> + /* Output buffers */
> + __u32 out_buf_y_st_addr;
> + __u32 out_buf_y_line_stride;
> + __u32 out_buf_y_buffer_stride;
> + __u32 out_buf_u_st_addr;
> + __u32 out_buf_v_st_addr;
> + __u32 out_buf_uv_line_stride;
> + __u32 out_buf_uv_buffer_stride;
> + __u32 out_buf_nr_buffers;
> + /* Intermediate buffers */
> + __u32 int_buf_y_st_addr;
> + __u32 int_buf_y_line_stride;
> + __u32 int_buf_u_st_addr;
> + __u32 int_buf_v_st_addr;
> + __u32 int_buf_uv_line_stride;
> + __u32 int_buf_height;
> + __u32 int_buf_chunk_width;
> + __u32 int_buf_chunk_height;
> + /* Context buffers */
> + __u32 ctx_buf_hor_y_st_addr;
> + __u32 ctx_buf_hor_u_st_addr;
> + __u32 ctx_buf_hor_v_st_addr;
> + __u32 ctx_buf_ver_y_st_addr;
> + __u32 ctx_buf_ver_u_st_addr;
> + __u32 ctx_buf_ver_v_st_addr;
> + /* Addresses for release-input and process-output tokens */
> + __u32 release_inp_buf_addr;
> + __u32 release_inp_buf_en;
> + __u32 release_out_buf_en;
> + __u32 process_out_buf_addr;
> + /* Settings dimensions, padding, cropping */
> + __u32 input_image_y_width;
> + __u32 input_image_y_height;
> + __u32 input_image_y_start_column;
> + __u32 input_image_uv_start_column;
> + __u32 input_image_y_left_pad;
> + __u32 input_image_uv_left_pad;
> + __u32 input_image_y_right_pad;
> + __u32 input_image_uv_right_pad;
> + __u32 input_image_y_top_pad;
> + __u32 input_image_uv_top_pad;
> + __u32 input_image_y_bottom_pad;
> + __u32 input_image_uv_bottom_pad;
> + __u32 processing_mode;
> +#define IMGU_ABI_OSYS_PROCMODE_BYPASS 0
> +#define IMGU_ABI_OSYS_PROCMODE_UPSCALE 1
> +#define IMGU_ABI_OSYS_PROCMODE_DOWNSCALE 2
Re: Software-only image processing for Intel "complex" cameras
Hi Pavel, On Fri, Jun 29, 2018 at 6:18 PM Pavel Machek wrote: > > Hi! > > > > On Nokia N900, I have similar problems as Intel IPU3 hardware. > > > > > > Meeting notes say that pure software implementation is not fast > > > enough, but that it may be useful for debugging. It would be also > > > useful for me on N900, and probably useful for processing "raw" images > > > from digital cameras. > > > > > > There is sensor part, and memory-to-memory part, right? What is > > > the format of data from the sensor part? What operations would be > > > expensive on the CPU? If we did everthing on the CPU, what would be > > > maximum resolution where we could still manage it in real time? > > > > We can still use the memory-to-memory part (IMGU), even without 3A. It > > would just do demosaicing at default parameters and give us a YUV > > (NV12) frame. We could use some software component to analyze the YUV > > output and adjust sensor parameters accordingly. Possibly the part we > > already have in libv4l2 could just work? > > As soon as you get YUV, yes, libv4l2 should be able to work with that. > > OTOH using the memory-to-memory part is going to be tricky. Why? I don't see any reason why it would be tricky. You just feed the right CAPTURE node with YUV buffers and the right OUTPUT node with raw buffers and that should work. > What > format is the data before demosaicing? Something common like BGGR10? It's documented in detail in V4L2 documentation: https://www.kernel.org/doc/html/latest/media/uapi/v4l/pixfmt-srggb10-ipu3.html > > > The expensive operation would be analyzing the frame itself. I suppose > > you need to build some histogram representing brightness and white > > balance of the frame and then infer necessary sensor adjustments from > > that. > > That does not really have to be expensive. You can sample ... say > 1 pixels from the image, and get good-enough data for 3A. Yes, but you need to do it relatively frequently to react for scene changes and also even 1 pixels (depending on distribution ) might mean fetching 1*cacheline bytes of data. > > > Would it be possible to get access to machine with IPU3, or would > > > there be someone willing to test libv4l2 patches? > > > > I should be able to help with some basic testing, preferably limited > > to command line tools (but I might be able to create a test > > environment for X11 tools if really necessary). > > Could you just compile libv4l2 with sdlcam demo on the target, and > then ssh -X there from some sort of reasonable system? Yes, that should work I guess. That would be a Chrome OS device (which doesn't include X11), so that would mean compiling some X11 libs (and probably some more dependencies) as well, but that's not impossible. Best regards, Tomasz
Re: Software-only image processing for Intel "complex" cameras
Hi Pavel, On Thu, Jun 21, 2018 at 5:38 AM Pavel Machek wrote: > > Hi! > > On Nokia N900, I have similar problems as Intel IPU3 hardware. > > Meeting notes say that pure software implementation is not fast > enough, but that it may be useful for debugging. It would be also > useful for me on N900, and probably useful for processing "raw" images > from digital cameras. > > There is sensor part, and memory-to-memory part, right? What is > the format of data from the sensor part? What operations would be > expensive on the CPU? If we did everthing on the CPU, what would be > maximum resolution where we could still manage it in real time? We can still use the memory-to-memory part (IMGU), even without 3A. It would just do demosaicing at default parameters and give us a YUV (NV12) frame. We could use some software component to analyze the YUV output and adjust sensor parameters accordingly. Possibly the part we already have in libv4l2 could just work? The expensive operation would be analyzing the frame itself. I suppose you need to build some histogram representing brightness and white balance of the frame and then infer necessary sensor adjustments from that. > > Would it be possible to get access to machine with IPU3, or would > there be someone willing to test libv4l2 patches? I should be able to help with some basic testing, preferably limited to command line tools (but I might be able to create a test environment for X11 tools if really necessary). Best regards, Tomasz
Re: [PATCH v6 04/12] intel-ipu3: Implement DMA mapping functions
On Tue, Jun 19, 2018 at 12:42 AM Zhi, Yong wrote: > > Hi, Tomasz, > > Thanks for the review. > > > -Original Message----- > > From: Tomasz Figa [mailto:tf...@chromium.org] > > Sent: Monday, June 18, 2018 12:09 AM > > To: Zhi, Yong > > Cc: Linux Media Mailing List ; Sakari Ailus > > ; Mani, Rajmohan > > ; Toivonen, Tuukka > > ; Hu, Jerry W ; Zheng, > > Jian Xu > > Subject: Re: [PATCH v6 04/12] intel-ipu3: Implement DMA mapping > > functions > > > > On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote: [snip] > > > diff --git a/drivers/media/pci/intel/ipu3/ipu3.h > > > b/drivers/media/pci/intel/ipu3/ipu3.h > > > new file mode 100644 > > > index ..2ba6fa58e41c > > > --- /dev/null > > > +++ b/drivers/media/pci/intel/ipu3/ipu3.h > > > @@ -0,0 +1,151 @@ > > > +/* SPDX-License-Identifier: GPL-2.0 */ > > > +/* Copyright (C) 2018 Intel Corporation */ > > > + > > > +#ifndef __IPU3_H > > > +#define __IPU3_H > > > + > > > +#include > > > +#include > > > + > > > +#include > > > +#include > > > + > > > +#include "ipu3-css.h" > > > + > > > +#define IMGU_NAME "ipu3-imgu" > > > + > > > +/* > > > + * The semantics of the driver is that whenever there is a buffer > > > +available in > > > + * master queue, the driver queues a buffer also to all other active > > nodes. > > > + * If user space hasn't provided a buffer to all other video nodes > > > +first, > > > + * the driver gets an internal dummy buffer and queues it. > > > + */ > > > +#define IMGU_QUEUE_MASTER IPU3_CSS_QUEUE_IN > > > +#define IMGU_QUEUE_FIRST_INPUT IPU3_CSS_QUEUE_OUT > > > +#define IMGU_MAX_QUEUE_DEPTH (2 + 2) > > > + > > > +#define IMGU_NODE_IN 0 /* Input RAW image */ > > > +#define IMGU_NODE_PARAMS 1 /* Input parameters */ > > > +#define IMGU_NODE_OUT 2 /* Main output for still or > > > video > > */ > > > +#define IMGU_NODE_VF 3 /* Preview */ > > > +#define IMGU_NODE_PV 4 /* Postview for still capture */ > > > +#define IMGU_NODE_STAT_3A 5 /* 3A statistics */ > > > +#define IMGU_NODE_NUM 6 > > > > Does this file really belong to this patch? > > > > Included because ipu3-dmamap uses struct defined in this header. It sounds like we should either move this patch later in the series or have just the necessary minimum or only forward declarations added in this patch. Best regards, Tomasz
Re: [ANN v2] Complex Camera Workshop - Tokyo - Jun, 19
Hi Paul, On Mon, Jun 18, 2018 at 5:42 PM Paul Elder wrote: > > > > Hello all, > > On June 4, 2018 10:33:03 PM GMT+09:00, Mauro Carvalho Chehab > wrote: > >Hi all, > > > >I consolidated hopefully all comments I receive on the past > >announcement > >with regards to the complex camera workshop we're planning to happen in > >Tokyo, just before the Open Source Summit in Japan. > > > >The main focus of the workshop is to allow supporting devices with > >MC-based > >hardware connected to a camera. > > > >I'm enclosing a detailed description of the problem, in order to > >allow the interested parties to be at the same page. > > > >We need to work towards an agenda for the meeting. > > > >From my side, I think we should have at least the following topics at > >the agenda: > > > >- a quick review about what's currently at libv4l2; > >- a presentation about PipeWire solution; > >- a discussion with the requirements for the new solution; > >- a discussion about how we'll address - who will do what. > > > >Comments? Suggestions? > > > >Are there anyone else planning to either be there physically or via > >Google Hangouts? > > > My name is Paul Elder. I am a university student studying computer science, > and I am interested in complex camera support in Linux. > > If it's not too late, could I join this meeting as well please, as I am in > Tokyo? Done. You should have received 3 further emails with necessary invitations. Best regards, Tomasz
Re: [PATCH v6 04/12] intel-ipu3: Implement DMA mapping functions
On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote:
>
> From: Tomasz Figa
>
> This driver uses IOVA space for buffer mapping through IPU3 MMU
> to transfer data between imaging pipelines and system DDR.
>
> Signed-off-by: Tomasz Figa
> Signed-off-by: Yong Zhi
> ---
> drivers/media/pci/intel/ipu3/ipu3-css-pool.h | 36
> drivers/media/pci/intel/ipu3/ipu3-dmamap.c | 280
> +++
> drivers/media/pci/intel/ipu3/ipu3-dmamap.h | 22 +++
> drivers/media/pci/intel/ipu3/ipu3.h | 151 +++
> 4 files changed, 489 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-dmamap.h
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3.h
>
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> new file mode 100644
> index ..4b22e0856232
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-css-pool.h
> @@ -0,0 +1,36 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2018 Intel Corporation */
> +
> +#ifndef __IPU3_UTIL_H
> +#define __IPU3_UTIL_H
> +
> +struct device;
> +
> +#define IPU3_CSS_POOL_SIZE 4
> +
> +struct ipu3_css_map {
> + size_t size;
> + void *vaddr;
> + dma_addr_t daddr;
> + struct vm_struct *vma;
> +};
> +
> +struct ipu3_css_pool {
> + struct {
> + struct ipu3_css_map param;
> + long framenum;
> + } entry[IPU3_CSS_POOL_SIZE];
> + unsigned int last; /* Latest entry */
It's not clear what "Latest entry" means here. Since these structs are
a part of the interface exposed by this header, could you write proper
kerneldoc comments for all fields in both of them?
> +};
> +
> +int ipu3_css_dma_buffer_resize(struct device *dev, struct ipu3_css_map *map,
> + size_t size);
> +void ipu3_css_pool_cleanup(struct device *dev, struct ipu3_css_pool *pool);
> +int ipu3_css_pool_init(struct device *dev, struct ipu3_css_pool *pool,
> + size_t size);
> +int ipu3_css_pool_get(struct ipu3_css_pool *pool, long framenum);
> +void ipu3_css_pool_put(struct ipu3_css_pool *pool);
> +const struct ipu3_css_map *ipu3_css_pool_last(struct ipu3_css_pool *pool,
> + unsigned int last);
> +
> +#endif
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> b/drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> new file mode 100644
> index ..b2bc5d00debc
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-dmamap.c
> @@ -0,0 +1,280 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (C) 2018 Intel Corporation
> + * Copyright (C) 2018 Google, Inc.
Would you mind changing as below?
Copyright 2018 Google LLC.
> + *
> + * Author: Tomasz Figa
> + * Author: Yong Zhi
> + */
> +
> +#include
> +
> +#include "ipu3.h"
> +#include "ipu3-css-pool.h"
> +#include "ipu3-mmu.h"
> +
> +/*
> + * Free a buffer allocated by ipu3_dmamap_alloc_buffer()
> + */
> +static void ipu3_dmamap_free_buffer(struct page **pages,
> + size_t size)
> +{
> + int count = size >> PAGE_SHIFT;
> +
> + while (count--)
> + __free_page(pages[count]);
> + kvfree(pages);
> +}
> +
> +/*
> + * Based on the implementation of __iommu_dma_alloc_pages()
> + * defined in drivers/iommu/dma-iommu.c
> + */
> +static struct page **ipu3_dmamap_alloc_buffer(size_t size,
> + unsigned long order_mask,
> + gfp_t gfp)
> +{
> + struct page **pages;
> + unsigned int i = 0, count = size >> PAGE_SHIFT;
> + const gfp_t high_order_gfp = __GFP_NOWARN | __GFP_NORETRY;
> +
> + /* Allocate mem for array of page ptrs */
> + pages = kvmalloc_array(count, sizeof(struct page *), GFP_KERNEL);
sizeof(*pages) to ensure that the right type is used regardless of declaration.
> +
No need for this blank line.
> + if (!pages)
> + return NULL;
[snip]
> +/**
> + * ipu3_dmamap_alloc - allocate and map a buffer into KVA
> + * @dev: struct device pointer
> + * @map: struct to store mapping variables
> + * @len: size required
> + *
> + * Return KVA on success or NULL on failure
> + */
> +void *ipu3_dmamap_alloc(struct device *dev, struct ipu3_css_map *map,
> + size_t len)
> +{
> + str
Re: [PATCH v6 03/12] intel-ipu3: mmu: Implement driver
On Fri, Mar 30, 2018 at 11:15 AM Yong Zhi wrote:
>
> From: Tomasz Figa
>
> This driver translates IO virtual address to physical
> address based on two levels page tables.
>
> Signed-off-by: Tomasz Figa
> Signed-off-by: Yong Zhi
> ---
> drivers/media/pci/intel/ipu3/ipu3-mmu.c | 560
>
> drivers/media/pci/intel/ipu3/ipu3-mmu.h | 28 ++
> 2 files changed, 588 insertions(+)
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.c
> create mode 100644 drivers/media/pci/intel/ipu3/ipu3-mmu.h
>
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-mmu.c
> b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
> new file mode 100644
> index ..a4b3e1680bbb
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-mmu.c
> @@ -0,0 +1,560 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2018 Intel Corporation.
> + * Copyright (C) 2018 Google, Inc.
I followed wrong guide when adding this one. Could you fix it up to
the following?
Copyright 2018 Google LLC.
[snip]
> +/**
> + * ipu3_mmu_exit() - clean up IPU3 MMU block
> + * @mmu: IPU3 MMU private data
> + */
> +void ipu3_mmu_exit(struct ipu3_mmu_info *info)
> +{
> + struct ipu3_mmu *mmu = to_ipu3_mmu(info);
> +
> + /* We are going to free our page tables, no more memory access. */
> + ipu3_mmu_set_halt(mmu, true);
> + ipu3_mmu_tlb_invalidate(mmu);
> +
> + ipu3_mmu_free_page_table(mmu->l1pt);
> + vfree(mmu->l2pts);
> + ipu3_mmu_free_page_table(mmu->dummy_l2pt);
> + kfree(mmu->dummy_page);
Should be free_page(). (Might be already included in your tree as per
https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/1084522)
> + kfree(mmu);
> +}
> +
> +void ipu3_mmu_suspend(struct ipu3_mmu_info *info)
> +{
> + struct ipu3_mmu *mmu = to_ipu3_mmu(info);
> +
> + ipu3_mmu_set_halt(mmu, true);
> +}
> +
> +void ipu3_mmu_resume(struct ipu3_mmu_info *info)
> +{
> + struct ipu3_mmu *mmu = to_ipu3_mmu(info);
> + u32 pteval;
> +
> + ipu3_mmu_set_halt(mmu, true);
> +
> + pteval = IPU3_ADDR2PTE(virt_to_phys(mmu->l1pt));
> + writel(pteval, mmu->base + REG_L1_PHYS);
> +
> + ipu3_mmu_tlb_invalidate(mmu);
> + ipu3_mmu_set_halt(mmu, false);
> +}
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-mmu.h
> b/drivers/media/pci/intel/ipu3/ipu3-mmu.h
> new file mode 100644
> index ..4976187c18f6
> --- /dev/null
> +++ b/drivers/media/pci/intel/ipu3/ipu3-mmu.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/* Copyright (C) 2018 Intel Corporation */
> +/* Copyright (C) 2018 Google, Inc. */
> +
> +#ifndef __IPU3_MMU_H
> +#define __IPU3_MMU_H
> +
> +struct ipu3_mmu_info {
> + dma_addr_t aperture_start; /* First address that can be mapped*/
> + dma_addr_t aperture_end; /* Last address that can be mapped */
> + unsigned long pgsize_bitmap;/* Bitmap of page sizes in use */
If documenting the fields, why not use a kerneldoc comment above the
struct instead?
Best regards,
Tomasz
