Re: [PATCHv3 11/12] media: docs-rst: Document m2m stateless video decoder interface
On Fri, Sep 27, 2019 at 12:40 AM Nicolas Dufresne wrote:
>
> Le jeudi 26 septembre 2019 à 13:33 +0200, Hans Verkuil a écrit :
> > Hi Alexandre, Tomasz,
> >
> > On 8/15/19 4:44 PM, Hans Verkuil wrote:
> > > From: Alexandre Courbot
> > >
> > > Documents the protocol that user-space should follow when
> > > communicating with stateless video decoders.
> > >
> > > The stateless video decoding API makes use of the new request and tags
> > > APIs. While it has been implemented with the Cedrus driver so far, it
> > > should probably still be considered staging for a short while.
> >
> > I noticed that this stateless decoder spec doesn't describe how to set up
> > crop/compose selection rectangles for CAPTURE. Was there a reason for that?
> > Would this work differently for stateless vs stateful decoders?
There is one big difference between stateless and stateful decoders -
the latter detect the rectangle from the decoded stream. The former
just write pixels to the area specified by the application.
Another aspect is that a stateless decoder always needs to write the
full macroblocks to the frame buffer, because it is not aware of the
visible part. That invisible area might be also required for decoding
further inter-frames.
> >
> > It's true that cedrus and hantro currently do not support this, but I
> > assume there is no reason why this can't be supported by other drivers.
>
> Most of the encoders upstream aren't fully production ready, so they
> might be missing some bits. What we minimally need to be able to do is
> encode let's say 15x15 image and see this crop reflected into the
> headers.
I might be confused, but I thought we were talking about decoders here.
Best regards,
Tomasz
>
> I wonder if instead of using the compose API, the drivers aren't
> assuming that the width/height on the CAPTURE queue format is the
> display width/height, and then the format on the OUTPUT queue format is
> the padded one. Works for top/left crop, which is likely what we will
> be using most of the time.
>
> For sure, to support all possibilities, we'd need the compose API, the
> SPS has top/left/right/bottom offset (in pixels) for cropping the
> image. Specially if you have an odd left or top offset, since cropping
> with sub-sampled formats require re-siting to avoid chroma shift. But
> I'm pretty sure, this is rarely supported, or just handle the naive way
> (with offset and stride modification).
>
> >
> > Regards,
> >
> > Hans
> >
> > > Signed-off-by: Alexandre Courbot
> > > ---
> > > Documentation/media/uapi/v4l/dev-mem2mem.rst | 1 +
> > > .../media/uapi/v4l/dev-stateless-decoder.rst | 424 ++
> > > 2 files changed, 425 insertions(+)
> > > create mode 100644 Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> > >
> > > diff --git a/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > > b/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > > index caa05f5f6380..70953958cee6 100644
> > > --- a/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > > +++ b/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > > @@ -46,3 +46,4 @@ devices are given in the following sections.
> > > :maxdepth: 1
> > >
> > > dev-decoder
> > > +dev-stateless-decoder
> > > diff --git a/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> > > b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> > > new file mode 100644
> > > index ..4a26646eeec5
> > > --- /dev/null
> > > +++ b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> > > @@ -0,0 +1,424 @@
> > > +.. SPDX-License-Identifier: GPL-2.0
> > > +
> > > +.. _stateless_decoder:
> > > +
> > > +**
> > > +Memory-to-memory Stateless Video Decoder Interface
> > > +**
> > > +
> > > +A stateless decoder is a decoder that works without retaining any kind
> > > of state
> > > +between processed frames. This means that each frame is decoded
> > > independently
> > > +of any previous and future frames, and that the client is responsible for
> > > +maintaining the decoding state and providing it to the decoder with each
> > > +decoding request. This is in contrast to the stateful video decoder
> > > interface,
> > > +where the hardware and driver maintain the decoding state and all the
> > > client
> > > +has to do is to provide the raw encoded stream and dequeue decoded
> > > frames in
> > > +display order.
> > > +
> > > +This section describes how user-space ("the client") is expected to
> > > communicate
> > > +with stateless decoders in order to successfully decode an encoded
> > > stream.
> > > +Compared to stateful codecs, the decoder/client sequence is simpler, but
> > > the
> > > +cost of this simplicity is extra complexity in the client which is
> > > responsible
> > > +for maintaining a consistent decoding state.
> > > +
> > > +Stateless decoders make use of the :ref:`media-request-api`. A stateless
> > > +decoder must expose th
Re: [PATCHv3 11/12] media: docs-rst: Document m2m stateless video decoder interface
Le jeudi 26 septembre 2019 à 13:33 +0200, Hans Verkuil a écrit :
> Hi Alexandre, Tomasz,
>
> On 8/15/19 4:44 PM, Hans Verkuil wrote:
> > From: Alexandre Courbot
> >
> > Documents the protocol that user-space should follow when
> > communicating with stateless video decoders.
> >
> > The stateless video decoding API makes use of the new request and tags
> > APIs. While it has been implemented with the Cedrus driver so far, it
> > should probably still be considered staging for a short while.
>
> I noticed that this stateless decoder spec doesn't describe how to set up
> crop/compose selection rectangles for CAPTURE. Was there a reason for that?
> Would this work differently for stateless vs stateful decoders?
>
> It's true that cedrus and hantro currently do not support this, but I
> assume there is no reason why this can't be supported by other drivers.
Most of the encoders upstream aren't fully production ready, so they
might be missing some bits. What we minimally need to be able to do is
encode let's say 15x15 image and see this crop reflected into the
headers.
I wonder if instead of using the compose API, the drivers aren't
assuming that the width/height on the CAPTURE queue format is the
display width/height, and then the format on the OUTPUT queue format is
the padded one. Works for top/left crop, which is likely what we will
be using most of the time.
For sure, to support all possibilities, we'd need the compose API, the
SPS has top/left/right/bottom offset (in pixels) for cropping the
image. Specially if you have an odd left or top offset, since cropping
with sub-sampled formats require re-siting to avoid chroma shift. But
I'm pretty sure, this is rarely supported, or just handle the naive way
(with offset and stride modification).
>
> Regards,
>
> Hans
>
> > Signed-off-by: Alexandre Courbot
> > ---
> > Documentation/media/uapi/v4l/dev-mem2mem.rst | 1 +
> > .../media/uapi/v4l/dev-stateless-decoder.rst | 424 ++
> > 2 files changed, 425 insertions(+)
> > create mode 100644 Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> >
> > diff --git a/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > b/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > index caa05f5f6380..70953958cee6 100644
> > --- a/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > +++ b/Documentation/media/uapi/v4l/dev-mem2mem.rst
> > @@ -46,3 +46,4 @@ devices are given in the following sections.
> > :maxdepth: 1
> >
> > dev-decoder
> > +dev-stateless-decoder
> > diff --git a/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> > b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> > new file mode 100644
> > index ..4a26646eeec5
> > --- /dev/null
> > +++ b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> > @@ -0,0 +1,424 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +.. _stateless_decoder:
> > +
> > +**
> > +Memory-to-memory Stateless Video Decoder Interface
> > +**
> > +
> > +A stateless decoder is a decoder that works without retaining any kind of
> > state
> > +between processed frames. This means that each frame is decoded
> > independently
> > +of any previous and future frames, and that the client is responsible for
> > +maintaining the decoding state and providing it to the decoder with each
> > +decoding request. This is in contrast to the stateful video decoder
> > interface,
> > +where the hardware and driver maintain the decoding state and all the
> > client
> > +has to do is to provide the raw encoded stream and dequeue decoded frames
> > in
> > +display order.
> > +
> > +This section describes how user-space ("the client") is expected to
> > communicate
> > +with stateless decoders in order to successfully decode an encoded stream.
> > +Compared to stateful codecs, the decoder/client sequence is simpler, but
> > the
> > +cost of this simplicity is extra complexity in the client which is
> > responsible
> > +for maintaining a consistent decoding state.
> > +
> > +Stateless decoders make use of the :ref:`media-request-api`. A stateless
> > +decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on
> > its
> > +``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or
> > :c:func:`VIDIOC_CREATE_BUFS`
> > +are invoked.
> > +
> > +Depending on the encoded formats supported by the decoder, a single decoded
> > +frame may be the result of several decode requests (for instance, H.264
> > streams
> > +with multiple slices per frame). Decoders that support such formats must
> > also
> > +expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on
> > their
> > +``OUTPUT`` queue.
> > +
> > +Querying capabilities
> > +=
> > +
> > +1. To enumerate the set of coded formats supported by the decoder, the
> > client
> > + calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue.
> > +
> > + * T
Re: [PATCHv3 11/12] media: docs-rst: Document m2m stateless video decoder interface
Hi Alexandre, Tomasz,
On 8/15/19 4:44 PM, Hans Verkuil wrote:
> From: Alexandre Courbot
>
> Documents the protocol that user-space should follow when
> communicating with stateless video decoders.
>
> The stateless video decoding API makes use of the new request and tags
> APIs. While it has been implemented with the Cedrus driver so far, it
> should probably still be considered staging for a short while.
I noticed that this stateless decoder spec doesn't describe how to set up
crop/compose selection rectangles for CAPTURE. Was there a reason for that?
Would this work differently for stateless vs stateful decoders?
It's true that cedrus and hantro currently do not support this, but I
assume there is no reason why this can't be supported by other drivers.
Regards,
Hans
>
> Signed-off-by: Alexandre Courbot
> ---
> Documentation/media/uapi/v4l/dev-mem2mem.rst | 1 +
> .../media/uapi/v4l/dev-stateless-decoder.rst | 424 ++
> 2 files changed, 425 insertions(+)
> create mode 100644 Documentation/media/uapi/v4l/dev-stateless-decoder.rst
>
> diff --git a/Documentation/media/uapi/v4l/dev-mem2mem.rst
> b/Documentation/media/uapi/v4l/dev-mem2mem.rst
> index caa05f5f6380..70953958cee6 100644
> --- a/Documentation/media/uapi/v4l/dev-mem2mem.rst
> +++ b/Documentation/media/uapi/v4l/dev-mem2mem.rst
> @@ -46,3 +46,4 @@ devices are given in the following sections.
> :maxdepth: 1
>
> dev-decoder
> +dev-stateless-decoder
> diff --git a/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> new file mode 100644
> index ..4a26646eeec5
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
> @@ -0,0 +1,424 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +.. _stateless_decoder:
> +
> +**
> +Memory-to-memory Stateless Video Decoder Interface
> +**
> +
> +A stateless decoder is a decoder that works without retaining any kind of
> state
> +between processed frames. This means that each frame is decoded independently
> +of any previous and future frames, and that the client is responsible for
> +maintaining the decoding state and providing it to the decoder with each
> +decoding request. This is in contrast to the stateful video decoder
> interface,
> +where the hardware and driver maintain the decoding state and all the client
> +has to do is to provide the raw encoded stream and dequeue decoded frames in
> +display order.
> +
> +This section describes how user-space ("the client") is expected to
> communicate
> +with stateless decoders in order to successfully decode an encoded stream.
> +Compared to stateful codecs, the decoder/client sequence is simpler, but the
> +cost of this simplicity is extra complexity in the client which is
> responsible
> +for maintaining a consistent decoding state.
> +
> +Stateless decoders make use of the :ref:`media-request-api`. A stateless
> +decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on its
> +``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or
> :c:func:`VIDIOC_CREATE_BUFS`
> +are invoked.
> +
> +Depending on the encoded formats supported by the decoder, a single decoded
> +frame may be the result of several decode requests (for instance, H.264
> streams
> +with multiple slices per frame). Decoders that support such formats must also
> +expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on their
> +``OUTPUT`` queue.
> +
> +Querying capabilities
> +=
> +
> +1. To enumerate the set of coded formats supported by the decoder, the client
> + calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue.
> +
> + * The driver must always return the full set of supported ``OUTPUT``
> formats,
> + irrespective of the format currently set on the ``CAPTURE`` queue.
> +
> + * Simultaneously, the driver must restrain the set of values returned by
> + codec-specific capability controls (such as H.264 profiles) to the set
> + actually supported by the hardware.
> +
> +2. To enumerate the set of supported raw formats, the client calls
> + :c:func:`VIDIOC_ENUM_FMT` on the ``CAPTURE`` queue.
> +
> + * The driver must return only the formats supported for the format
> currently
> + active on the ``OUTPUT`` queue.
> +
> + * Depending on the currently set ``OUTPUT`` format, the set of supported
> raw
> + formats may depend on the value of some codec-dependent controls.
> + The client is responsible for making sure that these controls are set
> + before querying the ``CAPTURE`` queue. Failure to do so will result in
> the
> + default values for these controls being used, and a returned set of
> formats
> + that may not be usable for the media the client is trying to decode.
> +
> +3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported
>
[PATCHv3 11/12] media: docs-rst: Document m2m stateless video decoder interface
From: Alexandre Courbot
Documents the protocol that user-space should follow when
communicating with stateless video decoders.
The stateless video decoding API makes use of the new request and tags
APIs. While it has been implemented with the Cedrus driver so far, it
should probably still be considered staging for a short while.
Signed-off-by: Alexandre Courbot
---
Documentation/media/uapi/v4l/dev-mem2mem.rst | 1 +
.../media/uapi/v4l/dev-stateless-decoder.rst | 424 ++
2 files changed, 425 insertions(+)
create mode 100644 Documentation/media/uapi/v4l/dev-stateless-decoder.rst
diff --git a/Documentation/media/uapi/v4l/dev-mem2mem.rst
b/Documentation/media/uapi/v4l/dev-mem2mem.rst
index caa05f5f6380..70953958cee6 100644
--- a/Documentation/media/uapi/v4l/dev-mem2mem.rst
+++ b/Documentation/media/uapi/v4l/dev-mem2mem.rst
@@ -46,3 +46,4 @@ devices are given in the following sections.
:maxdepth: 1
dev-decoder
+dev-stateless-decoder
diff --git a/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
new file mode 100644
index ..4a26646eeec5
--- /dev/null
+++ b/Documentation/media/uapi/v4l/dev-stateless-decoder.rst
@@ -0,0 +1,424 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _stateless_decoder:
+
+**
+Memory-to-memory Stateless Video Decoder Interface
+**
+
+A stateless decoder is a decoder that works without retaining any kind of state
+between processed frames. This means that each frame is decoded independently
+of any previous and future frames, and that the client is responsible for
+maintaining the decoding state and providing it to the decoder with each
+decoding request. This is in contrast to the stateful video decoder interface,
+where the hardware and driver maintain the decoding state and all the client
+has to do is to provide the raw encoded stream and dequeue decoded frames in
+display order.
+
+This section describes how user-space ("the client") is expected to communicate
+with stateless decoders in order to successfully decode an encoded stream.
+Compared to stateful codecs, the decoder/client sequence is simpler, but the
+cost of this simplicity is extra complexity in the client which is responsible
+for maintaining a consistent decoding state.
+
+Stateless decoders make use of the :ref:`media-request-api`. A stateless
+decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on its
+``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or :c:func:`VIDIOC_CREATE_BUFS`
+are invoked.
+
+Depending on the encoded formats supported by the decoder, a single decoded
+frame may be the result of several decode requests (for instance, H.264 streams
+with multiple slices per frame). Decoders that support such formats must also
+expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on their
+``OUTPUT`` queue.
+
+Querying capabilities
+=
+
+1. To enumerate the set of coded formats supported by the decoder, the client
+ calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue.
+
+ * The driver must always return the full set of supported ``OUTPUT``
formats,
+ irrespective of the format currently set on the ``CAPTURE`` queue.
+
+ * Simultaneously, the driver must restrain the set of values returned by
+ codec-specific capability controls (such as H.264 profiles) to the set
+ actually supported by the hardware.
+
+2. To enumerate the set of supported raw formats, the client calls
+ :c:func:`VIDIOC_ENUM_FMT` on the ``CAPTURE`` queue.
+
+ * The driver must return only the formats supported for the format currently
+ active on the ``OUTPUT`` queue.
+
+ * Depending on the currently set ``OUTPUT`` format, the set of supported raw
+ formats may depend on the value of some codec-dependent controls.
+ The client is responsible for making sure that these controls are set
+ before querying the ``CAPTURE`` queue. Failure to do so will result in the
+ default values for these controls being used, and a returned set of
formats
+ that may not be usable for the media the client is trying to decode.
+
+3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported
+ resolutions for a given format, passing desired pixel format in
+ :c:type:`v4l2_frmsizeenum`'s ``pixel_format``.
+
+4. Supported profiles and levels for the current ``OUTPUT`` format, if
+ applicable, may be queried using their respective controls via
+ :c:func:`VIDIOC_QUERYCTRL`.
+
+Initialization
+==
+
+1. Set the coded format on the ``OUTPUT`` queue via :c:func:`VIDIOC_S_FMT`.
+
+ * **Required fields:**
+
+ ``type``
+ a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
+
+ ``pixelformat``
+ a coded pixel format.
+
+ ``width``, ``height``
+ coded width and height parsed from the stream.
+
+
