Re: [PATCH] [RFC RESEND] vdec: Add virtio video decode device specification
I shared PDF version of this RFC on Google drive:
https://drive.google.com/drive/folders/1hed-mTVI7dj0M_iab4DTfx5kPMLoeX8R
On Thu, Sep 19, 2019 at 8:15 PM Keiichi Watanabe wrote:
>
> Hi Hans,
> Thank you for your feedback.
>
> On Thu, Sep 19, 2019 at 6:53 PM Hans Verkuil wrote:
> >
> > Hi Keiichi,
> >
> > On 9/19/19 11:34 AM, Keiichi Watanabe wrote:
> > > [Resending because of some issues with sending to virtio-dev. Sorry for
> > > the noise.]
> > >
> > > This patch proposes virtio specification for new virtio video decode
> > > device.
> > > This device provides the functionality of hardware accelerated video
> > > decoding from encoded video contents provided by the guest into frame
> > > buffers accessible by the guest.
> > >
> > > We have prototype implementation for VMs on Chrome OS:
> > > * virtio-vdec device in crosvm [1]
> > > * virtio-vdec driver in Linux kernel v4.19 [2]
> > > - This driver follows V4L2 stateful video decoder API [3].
> > >
> > > Our prototype implementation uses [4], which allows the virtio-vdec
> > > device to use buffers allocated by virtio-gpu device.
> > >
> > > Any feedback would be greatly appreciated. Thank you.
> >
> > I'm not a virtio expert, but as I understand it the virtio-vdec driver
> > looks like a regular v4l2 stateful decoder devices to the guest, while
> > on the host there is a driver (or something like that) that maps the
> > virtio-vdec requests to the actual decoder hardware, right?
> >
> > What concerns me a bit (but there may be good reasons for this) is that
> > this virtio driver is so specific for stateful decoders.
> >
>
> We aim to design a platform-independent interface. The virtio-vdec protocol
> should be designed to be workable regardless of APIs, OS, and platforms
> eventually.
>
> Our prototype virtio-vdec device translates the virtio-vdec protocol to a
> Chrome's video decode acceleration API instead of talking to hardware decoders
> directly. This Chrome's API is an abstract layer for multiple decoder APIs on
> Linux such as V4L2 stateful, V4L2 slice, Intel's VAAPI.
>
> That is to say the guest driver translates V4L2 stateful API to virtio-vdec
> and
> the host device translates virtio-vdec to Chrome's API. So, I could say that
> this is already more general than a mere V4L2 stateful API wrapper, at least.
>
> I'd appreciate if you could let me know some parts are still specific to V4L2.
>
>
> > How does this scale to stateful encoders? Stateless codecs? Other M2M
> > devices like deinterlacers or colorspace converters? What about webcams?
> >
>
> We're designing virtio protocol for encoder as well, but we are at an early
> stage. So, I'm not sure if we should/can handle decoder and encoder in one
> protocol. I don't have any plans for other media devices.
>
>
> > In other words, I would like to see a bigger picture here.
> >
> > Note that there is also an effort for Xen to expose webcams to a guest:
> >
> > https://www.spinics.net/lists/linux-media/msg148629.html
> >
>
> Good to know. Thanks.
>
> > This may or may not be of interest. This was an RFC only, and I haven't
> > seen any follow-up patches with actual code.
> >
> > There will be a half-day meeting of media developers during the ELCE
> > in October about codecs. I know Alexandre and Tomasz will be there.
> > It might be a good idea to discuss this in more detail if needed.
> >
>
> Sounds good. They are closely working with me.
>
> Regards,
> Keiichi
>
> > Regards,
> >
> > Hans
> >
> > >
> > > [1]
> > > https://chromium-review.googlesource.com/q/hashtag:%22virtio-vdec-device%22+(status:open%20OR%20status:merged)
> > > [2]
> > > https://chromium-review.googlesource.com/q/hashtag:%22virtio-vdec-driver%22+(status:open%20OR%20status:merged)
> > > [3] https://hverkuil.home.xs4all.nl/codec-api/uapi/v4l/dev-decoder.html
> > > (to be merged to Linux 5.4)
> > > [4] https://lkml.org/lkml/2019/9/12/157
> > >
> > > Signed-off-by: Keiichi Watanabe
> > > ---
> > > content.tex | 1 +
> > > virtio-vdec.tex | 750
> > > 2 files changed, 751 insertions(+)
> > > create mode 100644 virtio-vdec.tex
> > >
> > > diff --git a/content.tex b/content.tex
> > > index 37a2190..b57d4a9 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -5682,6 +5682,7 @@ \subsubsection{Legacy Interface: Framing
> > > Requirements}\label{sec:Device
> > > \input{virtio-input.tex}
> > > \input{virtio-crypto.tex}
> > > \input{virtio-vsock.tex}
> > > +\input{virtio-vdec.tex}
> > >
> > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > >
> > > diff --git a/virtio-vdec.tex b/virtio-vdec.tex
> > > new file mode 100644
> > > index 000..d117129
> > > --- /dev/null
> > > +++ b/virtio-vdec.tex
> > > @@ -0,0 +1,750 @@
> > > +\section{Video Decode Device}
> > > +\label{sec:Device Types / Video Decode Device}
> > > +
> > > +virtio-vdec is a virtio based video decoder. This device provides the
> > > +functionality of hardware acc
Re: [PATCH] [RFC RESEND] vdec: Add virtio video decode device specification
Hi Hans,
Thank you for your feedback.
On Thu, Sep 19, 2019 at 6:53 PM Hans Verkuil wrote:
>
> Hi Keiichi,
>
> On 9/19/19 11:34 AM, Keiichi Watanabe wrote:
> > [Resending because of some issues with sending to virtio-dev. Sorry for the
> > noise.]
> >
> > This patch proposes virtio specification for new virtio video decode
> > device.
> > This device provides the functionality of hardware accelerated video
> > decoding from encoded video contents provided by the guest into frame
> > buffers accessible by the guest.
> >
> > We have prototype implementation for VMs on Chrome OS:
> > * virtio-vdec device in crosvm [1]
> > * virtio-vdec driver in Linux kernel v4.19 [2]
> > - This driver follows V4L2 stateful video decoder API [3].
> >
> > Our prototype implementation uses [4], which allows the virtio-vdec
> > device to use buffers allocated by virtio-gpu device.
> >
> > Any feedback would be greatly appreciated. Thank you.
>
> I'm not a virtio expert, but as I understand it the virtio-vdec driver
> looks like a regular v4l2 stateful decoder devices to the guest, while
> on the host there is a driver (or something like that) that maps the
> virtio-vdec requests to the actual decoder hardware, right?
>
> What concerns me a bit (but there may be good reasons for this) is that
> this virtio driver is so specific for stateful decoders.
>
We aim to design a platform-independent interface. The virtio-vdec protocol
should be designed to be workable regardless of APIs, OS, and platforms
eventually.
Our prototype virtio-vdec device translates the virtio-vdec protocol to a
Chrome's video decode acceleration API instead of talking to hardware decoders
directly. This Chrome's API is an abstract layer for multiple decoder APIs on
Linux such as V4L2 stateful, V4L2 slice, Intel's VAAPI.
That is to say the guest driver translates V4L2 stateful API to virtio-vdec and
the host device translates virtio-vdec to Chrome's API. So, I could say that
this is already more general than a mere V4L2 stateful API wrapper, at least.
I'd appreciate if you could let me know some parts are still specific to V4L2.
> How does this scale to stateful encoders? Stateless codecs? Other M2M
> devices like deinterlacers or colorspace converters? What about webcams?
>
We're designing virtio protocol for encoder as well, but we are at an early
stage. So, I'm not sure if we should/can handle decoder and encoder in one
protocol. I don't have any plans for other media devices.
> In other words, I would like to see a bigger picture here.
>
> Note that there is also an effort for Xen to expose webcams to a guest:
>
> https://www.spinics.net/lists/linux-media/msg148629.html
>
Good to know. Thanks.
> This may or may not be of interest. This was an RFC only, and I haven't
> seen any follow-up patches with actual code.
>
> There will be a half-day meeting of media developers during the ELCE
> in October about codecs. I know Alexandre and Tomasz will be there.
> It might be a good idea to discuss this in more detail if needed.
>
Sounds good. They are closely working with me.
Regards,
Keiichi
> Regards,
>
> Hans
>
> >
> > [1]
> > https://chromium-review.googlesource.com/q/hashtag:%22virtio-vdec-device%22+(status:open%20OR%20status:merged)
> > [2]
> > https://chromium-review.googlesource.com/q/hashtag:%22virtio-vdec-driver%22+(status:open%20OR%20status:merged)
> > [3] https://hverkuil.home.xs4all.nl/codec-api/uapi/v4l/dev-decoder.html (to
> > be merged to Linux 5.4)
> > [4] https://lkml.org/lkml/2019/9/12/157
> >
> > Signed-off-by: Keiichi Watanabe
> > ---
> > content.tex | 1 +
> > virtio-vdec.tex | 750
> > 2 files changed, 751 insertions(+)
> > create mode 100644 virtio-vdec.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 37a2190..b57d4a9 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5682,6 +5682,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device
> > \input{virtio-input.tex}
> > \input{virtio-crypto.tex}
> > \input{virtio-vsock.tex}
> > +\input{virtio-vdec.tex}
> >
> > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-vdec.tex b/virtio-vdec.tex
> > new file mode 100644
> > index 000..d117129
> > --- /dev/null
> > +++ b/virtio-vdec.tex
> > @@ -0,0 +1,750 @@
> > +\section{Video Decode Device}
> > +\label{sec:Device Types / Video Decode Device}
> > +
> > +virtio-vdec is a virtio based video decoder. This device provides the
> > +functionality of hardware accelerated video decoding from encoded
> > +video contents provided by the guest into frame buffers accessible by
> > +the guest.
> > +
> > +\subsection{Device ID}
> > +\label{sec:Device Types / Video Decode Device / Device ID}
> > +
> > +28
> > +
> > +\subsection{Virtqueues}
> > +\label{sec:Device Types / Video Decode Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] outq - queue for sending requests from t
Re: [PATCH] [RFC RESEND] vdec: Add virtio video decode device specification
Hi Keiichi,
On 9/19/19 11:34 AM, Keiichi Watanabe wrote:
> [Resending because of some issues with sending to virtio-dev. Sorry for the
> noise.]
>
> This patch proposes virtio specification for new virtio video decode
> device.
> This device provides the functionality of hardware accelerated video
> decoding from encoded video contents provided by the guest into frame
> buffers accessible by the guest.
>
> We have prototype implementation for VMs on Chrome OS:
> * virtio-vdec device in crosvm [1]
> * virtio-vdec driver in Linux kernel v4.19 [2]
> - This driver follows V4L2 stateful video decoder API [3].
>
> Our prototype implementation uses [4], which allows the virtio-vdec
> device to use buffers allocated by virtio-gpu device.
>
> Any feedback would be greatly appreciated. Thank you.
I'm not a virtio expert, but as I understand it the virtio-vdec driver
looks like a regular v4l2 stateful decoder devices to the guest, while
on the host there is a driver (or something like that) that maps the
virtio-vdec requests to the actual decoder hardware, right?
What concerns me a bit (but there may be good reasons for this) is that
this virtio driver is so specific for stateful decoders.
How does this scale to stateful encoders? Stateless codecs? Other M2M
devices like deinterlacers or colorspace converters? What about webcams?
In other words, I would like to see a bigger picture here.
Note that there is also an effort for Xen to expose webcams to a guest:
https://www.spinics.net/lists/linux-media/msg148629.html
This may or may not be of interest. This was an RFC only, and I haven't
seen any follow-up patches with actual code.
There will be a half-day meeting of media developers during the ELCE
in October about codecs. I know Alexandre and Tomasz will be there.
It might be a good idea to discuss this in more detail if needed.
Regards,
Hans
>
> [1]
> https://chromium-review.googlesource.com/q/hashtag:%22virtio-vdec-device%22+(status:open%20OR%20status:merged)
> [2]
> https://chromium-review.googlesource.com/q/hashtag:%22virtio-vdec-driver%22+(status:open%20OR%20status:merged)
> [3] https://hverkuil.home.xs4all.nl/codec-api/uapi/v4l/dev-decoder.html (to
> be merged to Linux 5.4)
> [4] https://lkml.org/lkml/2019/9/12/157
>
> Signed-off-by: Keiichi Watanabe
> ---
> content.tex | 1 +
> virtio-vdec.tex | 750
> 2 files changed, 751 insertions(+)
> create mode 100644 virtio-vdec.tex
>
> diff --git a/content.tex b/content.tex
> index 37a2190..b57d4a9 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5682,6 +5682,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device
> \input{virtio-input.tex}
> \input{virtio-crypto.tex}
> \input{virtio-vsock.tex}
> +\input{virtio-vdec.tex}
>
> \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>
> diff --git a/virtio-vdec.tex b/virtio-vdec.tex
> new file mode 100644
> index 000..d117129
> --- /dev/null
> +++ b/virtio-vdec.tex
> @@ -0,0 +1,750 @@
> +\section{Video Decode Device}
> +\label{sec:Device Types / Video Decode Device}
> +
> +virtio-vdec is a virtio based video decoder. This device provides the
> +functionality of hardware accelerated video decoding from encoded
> +video contents provided by the guest into frame buffers accessible by
> +the guest.
> +
> +\subsection{Device ID}
> +\label{sec:Device Types / Video Decode Device / Device ID}
> +
> +28
> +
> +\subsection{Virtqueues}
> +\label{sec:Device Types / Video Decode Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] outq - queue for sending requests from the driver to the
> + device
> +\item[1] inq - queue for sending requests from the device to the
> + driver
> +\end{description}
> +
> +Each queue is used uni-directionally. outq is used to send requests
> +from the driver to the device (i.e., guest requests) and inq is used
> +to send requests in the other direction (i.e., host requests).
> +
> +\subsection{Feature bits}
> +\label{sec:Device Types / Video Decode Device / Feature bits}
> +
> +There are currently no feature bits defined for this device.
> +
> +\subsection{Device configuration layout}
> +\label{sec:Device Types / Video Decode Device / Device configuration layout}
> +
> +None.
> +
> +\subsection{Device Requirements: Device Initialization}
> +\label{sec:Device Types / Video Decode Device / Device Requirements: Device
> Initialization}
> +
> +The virtqueues are initialized.
> +
> +\subsection{Device Operation}
> +\label{sec:Device Types / Video Decode Device / Device Operation}
> +
> +\subsubsection{Video Buffers}
> +\label{sec:Device Types / Video Decode Device / Device Operation / Buffers}
> +
> +A virtio-vdec driver and a device use two types of video buffers:
> +\emph{bitstream buffer} and \emph{frame buffer}. A bitstream buffer
> +contains encoded video stream data. This buffer is similar to an
> +OUTPUT buffer for Video for Linux Two (V4L2) API. A frame buff
