Re: [RFC 24/24] WIP: rust/audio: add GStreamer backend

2025-12-01 Thread Marc-André Lureau 
Hi Markus

On Mon, Dec 1, 2025 at 5:13 PM Markus Armbruster  wrote:
>
> [email protected] writes:
>
> > From: Marc-André Lureau 
> >
> > This patch introduce a rust/audio crate that replaces QEMU audio/
> > mixing/resampling code with GStreamer and Rust. It could potentially
> > remove the need for all the system-specific audio API implementation,
> > since GStreamer has audio elements for
> > ALSA/Pipewire/PulseAudio/jack/OSX/WASAPI etc (removing ~10k loc).
> >
> > TODO:
> > - test on various system, with various configuration to see if this
> >   backend can replace the other QEMU audio backends
> > - add a spicesink/spicesrc to handle spice, or rewrite spice to use
> >   the capture approach used by VNC code. Or drop capture support, and
> >   use custom qemusrc/qemusink for both Spice and VNC, lowering the feature
> >   and behaviour disparity.
> > - build-sys: make gstreamer optional
> > - build-sys: loadable module support
> > - investigate dropping get_buffer_size_out()
> > - investigate improving emulated devices to not require regular
> >   timers (appsrc need-data is called once)
> > - add generic audio backend tests
> > - more tests for the mixing/liveadder behaviour (synchronization)
> > - other: replace audio/dbus with a rust implementation (not using gstreamer)
> >
> > Signed-off-by: Marc-André Lureau 
> > ---
> >  qapi/audio.json   |   29 +
> >  audio/audio-driver_template.h |2 +
> >  rust/audio/wrapper.h  |   27 +
> >  audio/audio.c |5 +
> >  Cargo.lock|  572 --
> >  Cargo.toml|6 +
> >  audio/trace-events|5 +
> >  rust/audio/Cargo.toml |   29 +
> >  rust/audio/build.rs   |   49 ++
> >  rust/audio/meson.build|   75 +++
> >  rust/audio/src/audio.rs   |  516 
> >  rust/audio/src/bindings.rs|   32 +
> >  rust/audio/src/gstreamer.rs   | 1070 +
> >  rust/audio/src/lib.rs |   99 +++
> >  rust/meson.build  |6 +
> >  15 files changed, 2467 insertions(+), 55 deletions(-)
> >  create mode 100644 rust/audio/wrapper.h
> >  create mode 100644 rust/audio/Cargo.toml
> >  create mode 100644 rust/audio/build.rs
> >  create mode 100644 rust/audio/meson.build
> >  create mode 100644 rust/audio/src/audio.rs
> >  create mode 100644 rust/audio/src/bindings.rs
> >  create mode 100644 rust/audio/src/gstreamer.rs
> >  create mode 100644 rust/audio/src/lib.rs
> >
> > diff --git a/qapi/audio.json b/qapi/audio.json
> > index 2df87b9710..76dc7cbfa6 100644
> > --- a/qapi/audio.json
> > +++ b/qapi/audio.json
> > @@ -128,6 +128,33 @@
> >  '*out':   'AudiodevAlsaPerDirectionOptions',
> >  '*threshold': 'uint32' } }
> >
> > +##
> > +# @AudiodevGStreamerOptions:
> > +#
> > +# Options of the GStreamer audio backend.
> > +#
> > +# @in: options of the capture stream
> > +#
> > +# @out: options of the playback stream
> > +#
> > +# @sink: the name of the GStreamer sink element to use
> > +#(default 'autoaudiosink')
> > +#
> > +# @source: the name of the GStreamer source element to use
> > +#(default 'autoaudiosrc')
>
> Are 'autoaudiosink' and 'autoaudiosrc' well-known GStreamer names, or
> arbitrary?

Yes, they are well-known elements from the base plugins.

>
> > +#
> > +# Since: 11.0
> > +##
> > +{ 'struct': 'AudiodevGStreamerOptions',
> > +  'data': {
> > +'*in':'AudiodevPerDirectionOptions',
> > +'*out':   'AudiodevPerDirectionOptions',
> > +'*sink':  'str',
> > +'*source':'str'
> > +  }
> > +}
>
> Unindent by four, and format the doc comment like this:
>
> ##
> # @AudiodevGStreamerOptions:
> #
> # Options of the GStreamer audio backend.
> #
> # @in: options of the capture stream
> #
> # @out: options of the playback stream
> #
> # @sink: the name of the GStreamer sink element to use
> # (default 'autoaudiosink')
> #
> # @source: the name of the GStreamer source element to use
> # (default 'autoaudiosrc')
> #
> # Since: 11.0
> ##
>
>
> > +
> > +
> >  ##
> >  # @AudiodevSndioOptions:
> >  #
> > @@ -484,6 +511,7 @@
> >  { 'name': 'sdl', 'if': 'CONFIG_AUDIO_SDL' },
> >  { 'name': 'sndio', 'if': 'CONFIG_AUDIO_SNDIO' },
> >  { 'name': 'spice', 'if': 'CONFIG_SPICE' },
> > +{ 'name': 'gstreamer' },
>
> Short form suffices:
>
>'gstreamer',
>
> >  'wav' ] }
> >
> >  ##
> > @@ -530,6 +558,7 @@
> > 'if': 'CONFIG_AUDIO_SNDIO' },
> >  'spice': { 'type': 'AudiodevGenericOptions',
> > 'if': 'CONFIG_SPICE' },
> > +'gstreamer': { 'type': 'AudiodevGStreamerOptions' },
>
> Short form suffices:
>
>'gstreamer': 'AudiodevGStreamerOptions',
>
> >  'wav':   'AudiodevWavOptions' } }
> >
> >  ##
>
> [...]
>
>

Th

Re: [RFC 24/24] WIP: rust/audio: add GStreamer backend

2025-12-01 Thread Markus Armbruster 
[email protected] writes:

> From: Marc-André Lureau 
>
> This patch introduce a rust/audio crate that replaces QEMU audio/
> mixing/resampling code with GStreamer and Rust. It could potentially
> remove the need for all the system-specific audio API implementation,
> since GStreamer has audio elements for
> ALSA/Pipewire/PulseAudio/jack/OSX/WASAPI etc (removing ~10k loc).
>
> TODO:
> - test on various system, with various configuration to see if this
>   backend can replace the other QEMU audio backends
> - add a spicesink/spicesrc to handle spice, or rewrite spice to use
>   the capture approach used by VNC code. Or drop capture support, and
>   use custom qemusrc/qemusink for both Spice and VNC, lowering the feature
>   and behaviour disparity.
> - build-sys: make gstreamer optional
> - build-sys: loadable module support
> - investigate dropping get_buffer_size_out()
> - investigate improving emulated devices to not require regular
>   timers (appsrc need-data is called once)
> - add generic audio backend tests
> - more tests for the mixing/liveadder behaviour (synchronization)
> - other: replace audio/dbus with a rust implementation (not using gstreamer)
>
> Signed-off-by: Marc-André Lureau 
> ---
>  qapi/audio.json   |   29 +
>  audio/audio-driver_template.h |2 +
>  rust/audio/wrapper.h  |   27 +
>  audio/audio.c |5 +
>  Cargo.lock|  572 --
>  Cargo.toml|6 +
>  audio/trace-events|5 +
>  rust/audio/Cargo.toml |   29 +
>  rust/audio/build.rs   |   49 ++
>  rust/audio/meson.build|   75 +++
>  rust/audio/src/audio.rs   |  516 
>  rust/audio/src/bindings.rs|   32 +
>  rust/audio/src/gstreamer.rs   | 1070 +
>  rust/audio/src/lib.rs |   99 +++
>  rust/meson.build  |6 +
>  15 files changed, 2467 insertions(+), 55 deletions(-)
>  create mode 100644 rust/audio/wrapper.h
>  create mode 100644 rust/audio/Cargo.toml
>  create mode 100644 rust/audio/build.rs
>  create mode 100644 rust/audio/meson.build
>  create mode 100644 rust/audio/src/audio.rs
>  create mode 100644 rust/audio/src/bindings.rs
>  create mode 100644 rust/audio/src/gstreamer.rs
>  create mode 100644 rust/audio/src/lib.rs
>
> diff --git a/qapi/audio.json b/qapi/audio.json
> index 2df87b9710..76dc7cbfa6 100644
> --- a/qapi/audio.json
> +++ b/qapi/audio.json
> @@ -128,6 +128,33 @@
>  '*out':   'AudiodevAlsaPerDirectionOptions',
>  '*threshold': 'uint32' } }
>  
> +##
> +# @AudiodevGStreamerOptions:
> +#
> +# Options of the GStreamer audio backend.
> +#
> +# @in: options of the capture stream
> +#
> +# @out: options of the playback stream
> +#
> +# @sink: the name of the GStreamer sink element to use
> +#(default 'autoaudiosink')
> +#
> +# @source: the name of the GStreamer source element to use
> +#(default 'autoaudiosrc')

Are 'autoaudiosink' and 'autoaudiosrc' well-known GStreamer names, or
arbitrary?

> +#
> +# Since: 11.0
> +##
> +{ 'struct': 'AudiodevGStreamerOptions',
> +  'data': {
> +'*in':'AudiodevPerDirectionOptions',
> +'*out':   'AudiodevPerDirectionOptions',
> +'*sink':  'str',
> +'*source':'str'
> +  }
> +}

Unindent by four, and format the doc comment like this:

##
# @AudiodevGStreamerOptions:
#
# Options of the GStreamer audio backend.
#
# @in: options of the capture stream
#
# @out: options of the playback stream
#
# @sink: the name of the GStreamer sink element to use
# (default 'autoaudiosink')
#
# @source: the name of the GStreamer source element to use
# (default 'autoaudiosrc')
#
# Since: 11.0
##


> +
> +
>  ##
>  # @AudiodevSndioOptions:
>  #
> @@ -484,6 +511,7 @@
>  { 'name': 'sdl', 'if': 'CONFIG_AUDIO_SDL' },
>  { 'name': 'sndio', 'if': 'CONFIG_AUDIO_SNDIO' },
>  { 'name': 'spice', 'if': 'CONFIG_SPICE' },
> +{ 'name': 'gstreamer' },

Short form suffices:

   'gstreamer',

>  'wav' ] }
>  
>  ##
> @@ -530,6 +558,7 @@
> 'if': 'CONFIG_AUDIO_SNDIO' },
>  'spice': { 'type': 'AudiodevGenericOptions',
> 'if': 'CONFIG_SPICE' },
> +'gstreamer': { 'type': 'AudiodevGStreamerOptions' },

Short form suffices:

   'gstreamer': 'AudiodevGStreamerOptions',

>  'wav':   'AudiodevWavOptions' } }
>  
>  ##

[...]

[RFC 24/24] WIP: rust/audio: add GStreamer backend

2025-12-01 Thread marcandre . lureau 
From: Marc-André Lureau 

This patch introduce a rust/audio crate that replaces QEMU audio/
mixing/resampling code with GStreamer and Rust. It could potentially
remove the need for all the system-specific audio API implementation,
since GStreamer has audio elements for
ALSA/Pipewire/PulseAudio/jack/OSX/WASAPI etc (removing ~10k loc).

TODO:
- test on various system, with various configuration to see if this
  backend can replace the other QEMU audio backends
- add a spicesink/spicesrc to handle spice, or rewrite spice to use
  the capture approach used by VNC code. Or drop capture support, and
  use custom qemusrc/qemusink for both Spice and VNC, lowering the feature
  and behaviour disparity.
- build-sys: make gstreamer optional
- build-sys: loadable module support
- investigate dropping get_buffer_size_out()
- investigate improving emulated devices to not require regular
  timers (appsrc need-data is called once)
- add generic audio backend tests
- more tests for the mixing/liveadder behaviour (synchronization)
- other: replace audio/dbus with a rust implementation (not using gstreamer)

Signed-off-by: Marc-André Lureau 
---
 qapi/audio.json   |   29 +
 audio/audio-driver_template.h |2 +
 rust/audio/wrapper.h  |   27 +
 audio/audio.c |5 +
 Cargo.lock|  572 --
 Cargo.toml|6 +
 audio/trace-events|5 +
 rust/audio/Cargo.toml |   29 +
 rust/audio/build.rs   |   49 ++
 rust/audio/meson.build|   75 +++
 rust/audio/src/audio.rs   |  516 
 rust/audio/src/bindings.rs|   32 +
 rust/audio/src/gstreamer.rs   | 1070 +
 rust/audio/src/lib.rs |   99 +++
 rust/meson.build  |6 +
 15 files changed, 2467 insertions(+), 55 deletions(-)
 create mode 100644 rust/audio/wrapper.h
 create mode 100644 rust/audio/Cargo.toml
 create mode 100644 rust/audio/build.rs
 create mode 100644 rust/audio/meson.build
 create mode 100644 rust/audio/src/audio.rs
 create mode 100644 rust/audio/src/bindings.rs
 create mode 100644 rust/audio/src/gstreamer.rs
 create mode 100644 rust/audio/src/lib.rs

diff --git a/qapi/audio.json b/qapi/audio.json
index 2df87b9710..76dc7cbfa6 100644
--- a/qapi/audio.json
+++ b/qapi/audio.json
@@ -128,6 +128,33 @@
 '*out':   'AudiodevAlsaPerDirectionOptions',
 '*threshold': 'uint32' } }
 
+##
+# @AudiodevGStreamerOptions:
+#
+# Options of the GStreamer audio backend.
+#
+# @in: options of the capture stream
+#
+# @out: options of the playback stream
+#
+# @sink: the name of the GStreamer sink element to use
+#(default 'autoaudiosink')
+#
+# @source: the name of the GStreamer source element to use
+#(default 'autoaudiosrc')
+#
+# Since: 11.0
+##
+{ 'struct': 'AudiodevGStreamerOptions',
+  'data': {
+'*in':'AudiodevPerDirectionOptions',
+'*out':   'AudiodevPerDirectionOptions',
+'*sink':  'str',
+'*source':'str'
+  }
+}
+
+
 ##
 # @AudiodevSndioOptions:
 #
@@ -484,6 +511,7 @@
 { 'name': 'sdl', 'if': 'CONFIG_AUDIO_SDL' },
 { 'name': 'sndio', 'if': 'CONFIG_AUDIO_SNDIO' },
 { 'name': 'spice', 'if': 'CONFIG_SPICE' },
+{ 'name': 'gstreamer' },
 'wav' ] }
 
 ##
@@ -530,6 +558,7 @@
'if': 'CONFIG_AUDIO_SNDIO' },
 'spice': { 'type': 'AudiodevGenericOptions',
'if': 'CONFIG_SPICE' },
+'gstreamer': { 'type': 'AudiodevGStreamerOptions' },
 'wav':   'AudiodevWavOptions' } }
 
 ##
diff --git a/audio/audio-driver_template.h b/audio/audio-driver_template.h
index 40d1ad9dea..aa2451ac7f 100644
--- a/audio/audio-driver_template.h
+++ b/audio/audio-driver_template.h
@@ -391,6 +391,8 @@ AudiodevPerDirectionOptions *glue(audio_get_pdo_, 
TYPE)(Audiodev *dev)
 case AUDIODEV_DRIVER_SPICE:
 return dev->u.spice.TYPE;
 #endif
+case AUDIODEV_DRIVER_GSTREAMER:
+abort();
 case AUDIODEV_DRIVER_WAV:
 return dev->u.wav.TYPE;
 
diff --git a/rust/audio/wrapper.h b/rust/audio/wrapper.h
new file mode 100644
index 00..a7960d0acc
--- /dev/null
+++ b/rust/audio/wrapper.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+/*
+ * This header file is meant to be used as input to the `bindgen` application
+ * in order to generate C FFI compatible Rust bindings.
+ */
+
+#ifndef __CLANG_STDATOMIC_H
+#define __CLANG_STDATOMIC_H
+/*
+ * Fix potential missing stdatomic.h error in case bindgen does not insert the
+ * correct libclang header paths on its own. We do not use stdatomic.h symbols
+ * in QEMU code, so it's fine to declare dummy types instead.
+ */
+typedef enum memory_order {
+  memory_order_relaxed,
+  memory_order_consume,
+  memory_order_acquire,
+  memory_order_release,
+  memory_order_acq_rel,