Re: [PATCH v3 01/16] media: imx: add mem2mem device
(resending as plain text)
On 10/21/18 10:43 AM, Philipp Zabel wrote:
On Fri, Oct 19, 2018 at 01:19:10PM -0700, Steve Longerbeam wrote:
On 10/19/18 2:53 AM, Philipp Zabel wrote:
Hi Tim,
On Thu, 2018-10-18 at 15:53 -0700, Tim Harvey wrote:
[...]
Philipp,
Thanks for submitting this!
I'm hoping this lets us use non-IMX capture devices along with the IMX
media controller entities to so we can use hardware
CSC,scaling,pixel-format-conversions and ultimately coda based encode.
I've built this on top of linux-media and see that it registers as
/dev/video8 but I'm not clear how to use it? I don't see it within the
media controller graph.
It's a V4L2 mem2mem device that can be handled by the GstV4l2Transform
element, for example. GStreamer should create a v4l2video8convert
element of that type.
The mem2mem device is not part of the media controller graph on purpose.
There is no interaction with any of the entities in the media controller
graph apart from the fact that the IC PP task we are using for mem2mem
scaling is sharing hardware resources with the IC PRP tasks used for the
media controller scaler entitites.
It would be nice in the future to link mem2mem output-side to the ipu_vdic:1
pad, to make use of h/w VDIC de-interlace as part of mem2mem operations.
The progressive output from a new ipu_vdic:3 pad can then be sent to the
image_convert APIs by the mem2mem driver for further tiled scaling, CSC,
and rotation by the IC PP task. The ipu_vdic:1 pad makes use of pure
DMA-based de-interlace, that is, all input frames (N-1, N, N+1) to the
VDIC are sent from DMA buffers, and this VDIC mode of operation is
well understood and produces clean de-interlace output. The risk is
that this would require iDMAC channel 5 for ipu_vdic:3, which IFAIK is
not verified to work yet.
Tiled mem2mem deinterlacing support would be nice, I'm not sure yet how
though. I'd limit media controller links to marking VDIC as unavailable
for the capture pipeline. The V4L2 subdev API is too lowlevel for tiling
mem2mem purposes, as we'd need to change the subdev format multiple
times per frame.
I wasn't considering tiled deinterlacing, only deinterlacing at the
hardware limited input frame size to the VDIC.
Also I'd like to keep the option of scheduling tile jobs to both IPUs on
i.MX6Q, which will become difficult to describe via MC, as both IPUs'
ipu_vdics would have to be involved.
Agreed, it would be good to add to the mem2mem driver the ability to
schedule jobs to whichever IPU is least busy.
The other problem with that currently is that mem2mem would have to be split
into separate device nodes: a /dev/videoN for output-side (linked to
ipu_vdic:1), and a /dev/videoM for capture-side (linked from
ipu_vdic:3). And then it no longer presents to userspace as a mem2mem
device with a single device node for both output and capture sides.
I don't understand why we'd need separate video devices for output and
capture, deinterlacing is still single input single (double rate)
output. As soon as we begin tiling, we are one layer of abstraction
away from the hardware pads anyway. Now if we want to support combining
on the other hand...
Again I wasn't thinking of doing tiled deinterlace.
Or is there another way? I recall work to integrate mem2mem with media
control. There is v4l2_m2m_register_media_controller(), but that
create three
entities:
source, processing, and sink. The VDIC entity would be part of mem2mem
processing but this entity already exists for the current graph. This
function could however be used as a guide to incorporate the VDIC
entity into m2m device.
I'm not sure if this is the right abstraction. Without tiling or
multi-IPU scheduling, sure. But the mem2mem driver does not directly
describe hardware operation anyway.
What I'm thinking is separate mem2mem devices from this proposed
tiled-scaling post-processing mem2mem, that solely do motion compensated
deinterlace using the VDIC as the processing entity.
[1] is the dot graph that demonstrates the idea, on imx6q SabreSD. Two
new mem2mem devices are attached to VDIC sink and source IDMAC pads.
Also the PP tiled scaling mem2mem is shown in the graph.
As of now, the VDIC is only available for video capture pipelines, so
the advantage would be to allow the use of hardware deinterlace
gstreamer pipelines from other types of interlaced sources like file or
network streams.
So something like the following example which would do hardware
deinterlace, followed by tiled scaling/CSC/rotation, then h.264 encode,
the VDIC mem2mem device is at /dev/video0, and the PP mem2mem device is
at /dev/video12 in this example:
gst-launch-1.0 \
v4l2src ! \
v4l2video0convert output-io-mode=dmabuf-import ! \
v4l2video12convert output-io-mode=dmabuf-import ! \
v4l2h264enc output-io-mode=dmabuf-import ! \
h264parse ! \
matroskamux ! \
filesink
I'm probably missing some stream properties in that example, but you get
the idea.
Steve
[1]
digraph board {
Re: [PATCH v3 01/16] media: imx: add mem2mem device
On Fri, Oct 19, 2018 at 01:19:10PM -0700, Steve Longerbeam wrote: > > On 10/19/18 2:53 AM, Philipp Zabel wrote: > > Hi Tim, > > > > On Thu, 2018-10-18 at 15:53 -0700, Tim Harvey wrote: > > [...] > > > Philipp, > > > > > > Thanks for submitting this! > > > > > > I'm hoping this lets us use non-IMX capture devices along with the IMX > > > media controller entities to so we can use hardware > > > CSC,scaling,pixel-format-conversions and ultimately coda based encode. > > > > > > I've built this on top of linux-media and see that it registers as > > > /dev/video8 but I'm not clear how to use it? I don't see it within the > > > media controller graph. > > It's a V4L2 mem2mem device that can be handled by the GstV4l2Transform > > element, for example. GStreamer should create a v4l2video8convert > > element of that type. > > > > The mem2mem device is not part of the media controller graph on purpose. > > There is no interaction with any of the entities in the media controller > > graph apart from the fact that the IC PP task we are using for mem2mem > > scaling is sharing hardware resources with the IC PRP tasks used for the > > media controller scaler entitites. > > It would be nice in the future to link mem2mem output-side to the ipu_vdic:1 > pad, to make use of h/w VDIC de-interlace as part of mem2mem operations. > The progressive output from a new ipu_vdic:3 pad can then be sent to the > image_convert APIs by the mem2mem driver for further tiled scaling, CSC, > and rotation by the IC PP task. The ipu_vdic:1 pad makes use of pure > DMA-based de-interlace, that is, all input frames (N-1, N, N+1) to the > VDIC are sent from DMA buffers, and this VDIC mode of operation is > well understood and produces clean de-interlace output. The risk is > that this would require iDMAC channel 5 for ipu_vdic:3, which IFAIK is > not verified to work yet. Tiled mem2mem deinterlacing support would be nice, I'm not sure yet how though. I'd limit media controller links to marking VDIC as unavailable for the capture pipeline. The V4L2 subdev API is too lowlevel for tiling mem2mem purposes, as we'd need to change the subdev format multiple times per frame. Also I'd like to keep the option of scheduling tile jobs to both IPUs on i.MX6Q, which will become difficult to describe via MC, as both IPUs' ipu_vdics would have to be involved. > The other problem with that currently is that mem2mem would have to be split > into separate device nodes: a /dev/videoN for output-side (linked to > ipu_vdic:1), and a /dev/videoM for capture-side (linked from > ipu_vdic:3). And then it no longer presents to userspace as a mem2mem > device with a single device node for both output and capture sides. I don't understand why we'd need separate video devices for output and capture, deinterlacing is still single input single (double rate) output. As soon as we begin tiling, we are one layer of abstraction away from the hardware pads anyway. Now if we want to support combining on the other hand... > Or is there another way? I recall work to integrate mem2mem with media > control. There is v4l2_m2m_register_media_controller(), but that > create three > entities: > source, processing, and sink. The VDIC entity would be part of mem2mem > processing but this entity already exists for the current graph. This > function could however be used as a guide to incorporate the VDIC > entity into m2m device. I'm not sure if this is the right abstraction. Without tiling or multi-IPU scheduling, sure. But the mem2mem driver does not directly describe hardware operation anyway. regards Philipp
Re: [PATCH v3 01/16] media: imx: add mem2mem device
On Fri, Oct 19, 2018 at 1:19 PM Steve Longerbeam wrote: > > > On 10/19/18 2:53 AM, Philipp Zabel wrote: > > Hi Tim, > > > > On Thu, 2018-10-18 at 15:53 -0700, Tim Harvey wrote: > > [...] > >> Philipp, > >> > >> Thanks for submitting this! > >> > >> I'm hoping this lets us use non-IMX capture devices along with the IMX > >> media controller entities to so we can use hardware > >> CSC,scaling,pixel-format-conversions and ultimately coda based encode. > >> > >> I've built this on top of linux-media and see that it registers as > >> /dev/video8 but I'm not clear how to use it? I don't see it within the > >> media controller graph. > > It's a V4L2 mem2mem device that can be handled by the GstV4l2Transform > > element, for example. GStreamer should create a v4l2video8convert > > element of that type. > > > > The mem2mem device is not part of the media controller graph on purpose. > > There is no interaction with any of the entities in the media controller > > graph apart from the fact that the IC PP task we are using for mem2mem > > scaling is sharing hardware resources with the IC PRP tasks used for the > > media controller scaler entitites. > > > It would be nice in the future to link mem2mem output-side to the ipu_vdic:1 > pad, to make use of h/w VDIC de-interlace as part of mem2mem operations. > The progressive output from a new ipu_vdic:3 pad can then be sent to the > image_convert APIs by the mem2mem driver for further tiled scaling, CSC, > and rotation by the IC PP task. The ipu_vdic:1 pad makes use of pure > DMA-based > de-interlace, that is, all input frames (N-1, N, N+1) to the VDIC are sent > from DMA buffers, and this VDIC mode of operation is well understood > and produces clean de-interlace output. The risk is that this would require > iDMAC channel 5 for ipu_vdic:3, which IFAIK is not verified to work yet. > > > The other problem with that currently is that mem2mem would have to be > split into > separate device nodes: a /dev/videoN for output-side (linked to > ipu_vdic:1), and > a /dev/videoM for capture-side (linked from ipu_vdic:3). And then it no > longer > presents to userspace as a mem2mem device with a single device node for both > output and capture sides. > > > Or is there another way? I recall work to integrate mem2mem with media > control. > There is v4l2_m2m_register_media_controller(), but that create three > entities: > source, processing, and sink. The VDIC entity would be part of mem2mem > processing but this entity already exists for the current graph. This > function > could however be used as a guide to incorporate the VDIC entity into m2m > device. > I agree - without being able to utilize de-interlace,csc,scaling and rotation it seems fairly limited today (but a great start!). Also, if it were in the media graph wouldn't we be able to use the compose selection subdev API? I've got an AVC8000 minPCIe card here with a TW6869 with 8x analog capture inputs that I'm hoping to someday soon be able to capture, compose into a single frame, and encode. Regards, Tim
Re: [PATCH v3 01/16] media: imx: add mem2mem device
On 10/19/18 2:53 AM, Philipp Zabel wrote: Hi Tim, On Thu, 2018-10-18 at 15:53 -0700, Tim Harvey wrote: [...] Philipp, Thanks for submitting this! I'm hoping this lets us use non-IMX capture devices along with the IMX media controller entities to so we can use hardware CSC,scaling,pixel-format-conversions and ultimately coda based encode. I've built this on top of linux-media and see that it registers as /dev/video8 but I'm not clear how to use it? I don't see it within the media controller graph. It's a V4L2 mem2mem device that can be handled by the GstV4l2Transform element, for example. GStreamer should create a v4l2video8convert element of that type. The mem2mem device is not part of the media controller graph on purpose. There is no interaction with any of the entities in the media controller graph apart from the fact that the IC PP task we are using for mem2mem scaling is sharing hardware resources with the IC PRP tasks used for the media controller scaler entitites. It would be nice in the future to link mem2mem output-side to the ipu_vdic:1 pad, to make use of h/w VDIC de-interlace as part of mem2mem operations. The progressive output from a new ipu_vdic:3 pad can then be sent to the image_convert APIs by the mem2mem driver for further tiled scaling, CSC, and rotation by the IC PP task. The ipu_vdic:1 pad makes use of pure DMA-based de-interlace, that is, all input frames (N-1, N, N+1) to the VDIC are sent from DMA buffers, and this VDIC mode of operation is well understood and produces clean de-interlace output. The risk is that this would require iDMAC channel 5 for ipu_vdic:3, which IFAIK is not verified to work yet. The other problem with that currently is that mem2mem would have to be split into separate device nodes: a /dev/videoN for output-side (linked to ipu_vdic:1), and a /dev/videoM for capture-side (linked from ipu_vdic:3). And then it no longer presents to userspace as a mem2mem device with a single device node for both output and capture sides. Or is there another way? I recall work to integrate mem2mem with media control. There is v4l2_m2m_register_media_controller(), but that create three entities: source, processing, and sink. The VDIC entity would be part of mem2mem processing but this entity already exists for the current graph. This function could however be used as a guide to incorporate the VDIC entity into m2m device. Steve
Re: [PATCH v3 01/16] media: imx: add mem2mem device
Hi Tim, On Thu, 2018-10-18 at 15:53 -0700, Tim Harvey wrote: [...] > Philipp, > > Thanks for submitting this! > > I'm hoping this lets us use non-IMX capture devices along with the IMX > media controller entities to so we can use hardware > CSC,scaling,pixel-format-conversions and ultimately coda based encode. > > I've built this on top of linux-media and see that it registers as > /dev/video8 but I'm not clear how to use it? I don't see it within the > media controller graph. It's a V4L2 mem2mem device that can be handled by the GstV4l2Transform element, for example. GStreamer should create a v4l2video8convert element of that type. The mem2mem device is not part of the media controller graph on purpose. There is no interaction with any of the entities in the media controller graph apart from the fact that the IC PP task we are using for mem2mem scaling is sharing hardware resources with the IC PRP tasks used for the media controller scaler entitites. regards Philipp
Re: [PATCH v3 01/16] media: imx: add mem2mem device
On Tue, Sep 18, 2018 at 2:34 AM Philipp Zabel wrote: > > Add a single imx-media mem2mem video device that uses the IPU IC PP > (image converter post processing) task for scaling and colorspace > conversion. > On i.MX6Q/DL SoCs with two IPUs currently only the first IPU is used. > > The hardware only supports writing to destination buffers up to > 1024x1024 pixels in a single pass, arbitrary sizes can be achieved > by rendering multiple tiles per frame. > > Signed-off-by: Philipp Zabel > [steve_longerb...@mentor.com: use ipu_image_convert_adjust(), fix > device_run() error handling] > Signed-off-by: Steve Longerbeam > --- > Changes since v2: > - Rely on ipu_image_convert_adjust() in mem2mem_try_fmt() for format >adjustments. This makes the mem2mem driver mostly a V4L2 mem2mem API >wrapper around the IPU image converter, and independent of the >internal image converter implementation. > - Remove the source and destination buffers on error in device_run(). >Otherwise the conversion is re-attempted apparently over and over >again (with WARN() backtraces). > - Allow subscribing to control changes. > --- > drivers/staging/media/imx/Kconfig | 1 + > drivers/staging/media/imx/Makefile| 1 + > drivers/staging/media/imx/imx-media-dev.c | 11 + > drivers/staging/media/imx/imx-media-mem2mem.c | 873 ++ > drivers/staging/media/imx/imx-media.h | 10 + > 5 files changed, 896 insertions(+) > create mode 100644 drivers/staging/media/imx/imx-media-mem2mem.c > Philipp, Thanks for submitting this! I'm hoping this lets us use non-IMX capture devices along with the IMX media controller entities to so we can use hardware CSC,scaling,pixel-format-conversions and ultimately coda based encode. I've built this on top of linux-media and see that it registers as /dev/video8 but I'm not clear how to use it? I don't see it within the media controller graph. Regards, Tim
[PATCH v3 01/16] media: imx: add mem2mem device
Add a single imx-media mem2mem video device that uses the IPU IC PP
(image converter post processing) task for scaling and colorspace
conversion.
On i.MX6Q/DL SoCs with two IPUs currently only the first IPU is used.
The hardware only supports writing to destination buffers up to
1024x1024 pixels in a single pass, arbitrary sizes can be achieved
by rendering multiple tiles per frame.
Signed-off-by: Philipp Zabel
[steve_longerb...@mentor.com: use ipu_image_convert_adjust(), fix
device_run() error handling]
Signed-off-by: Steve Longerbeam
---
Changes since v2:
- Rely on ipu_image_convert_adjust() in mem2mem_try_fmt() for format
adjustments. This makes the mem2mem driver mostly a V4L2 mem2mem API
wrapper around the IPU image converter, and independent of the
internal image converter implementation.
- Remove the source and destination buffers on error in device_run().
Otherwise the conversion is re-attempted apparently over and over
again (with WARN() backtraces).
- Allow subscribing to control changes.
---
drivers/staging/media/imx/Kconfig | 1 +
drivers/staging/media/imx/Makefile| 1 +
drivers/staging/media/imx/imx-media-dev.c | 11 +
drivers/staging/media/imx/imx-media-mem2mem.c | 873 ++
drivers/staging/media/imx/imx-media.h | 10 +
5 files changed, 896 insertions(+)
create mode 100644 drivers/staging/media/imx/imx-media-mem2mem.c
diff --git a/drivers/staging/media/imx/Kconfig
b/drivers/staging/media/imx/Kconfig
index bfc17de56b17..07013cb3cb66 100644
--- a/drivers/staging/media/imx/Kconfig
+++ b/drivers/staging/media/imx/Kconfig
@@ -6,6 +6,7 @@ config VIDEO_IMX_MEDIA
depends on HAS_DMA
select VIDEOBUF2_DMA_CONTIG
select V4L2_FWNODE
+ select V4L2_MEM2MEM_DEV
---help---
Say yes here to enable support for video4linux media controller
driver for the i.MX5/6 SOC.
diff --git a/drivers/staging/media/imx/Makefile
b/drivers/staging/media/imx/Makefile
index 698a4210316e..f2e722d0fa19 100644
--- a/drivers/staging/media/imx/Makefile
+++ b/drivers/staging/media/imx/Makefile
@@ -6,6 +6,7 @@ imx-media-ic-objs := imx-ic-common.o imx-ic-prp.o
imx-ic-prpencvf.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-capture.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-mem2mem.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-vdic.o
obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-ic.o
diff --git a/drivers/staging/media/imx/imx-media-dev.c
b/drivers/staging/media/imx/imx-media-dev.c
index 1931d1b038dc..9c59687612a0 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -359,6 +359,17 @@ static int imx_media_probe_complete(struct
v4l2_async_notifier *notifier)
goto unlock;
ret = v4l2_device_register_subdev_nodes(&imxmd->v4l2_dev);
+ if (ret)
+ goto unlock;
+
+ /* TODO: check whether we have IC subdevices first */
+ imxmd->m2m_vdev = imx_media_mem2mem_device_init(imxmd);
+ if (IS_ERR(imxmd->m2m_vdev)) {
+ ret = PTR_ERR(imxmd->m2m_vdev);
+ goto unlock;
+ }
+
+ ret = imx_media_mem2mem_device_register(imxmd->m2m_vdev);
unlock:
mutex_unlock(&imxmd->mutex);
if (ret)
diff --git a/drivers/staging/media/imx/imx-media-mem2mem.c
b/drivers/staging/media/imx/imx-media-mem2mem.c
new file mode 100644
index ..a2a4dca017ce
--- /dev/null
+++ b/drivers/staging/media/imx/imx-media-mem2mem.c
@@ -0,0 +1,873 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * i.MX IPUv3 mem2mem Scaler/CSC driver
+ *
+ * Copyright (C) 2011 Pengutronix, Sascha Hauer
+ * Copyright (C) 2018 Pengutronix, Philipp Zabel
+ */
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "imx-media.h"
+
+#define fh_to_ctx(__fh)container_of(__fh, struct mem2mem_ctx, fh)
+
+enum {
+ V4L2_M2M_SRC = 0,
+ V4L2_M2M_DST = 1,
+};
+
+struct mem2mem_priv {
+ struct imx_media_video_dev vdev;
+
+ struct v4l2_m2m_dev *m2m_dev;
+ struct device *dev;
+
+ struct imx_media_dev *md;
+
+ struct mutex mutex; /* mem2mem device mutex */
+
+ atomic_t num_inst;
+};
+
+#define to_mem2mem_priv(v) container_of(v, struct mem2mem_priv, vdev)
+
+/* Per-queue, driver-specific private data */
+struct mem2mem_q_data {
+ struct v4l2_pix_format cur_fmt;
+ struct v4l2_rectrect;
+};
+
+struct mem2mem_ctx {
+ struct mem2mem_priv *priv;
+
+ struct v4l2_fh fh;
+ struct mem2mem_q_data q_data[2];
+ int error;
+ struct ipu_image_convert_ctx *icc;
+
+ struct v4l2_ctrl_handler ctrl_hdlr;
+ int rotate;
+ bool
