On Tue, Dec 9, 2025 at 12:23 PM John Snow <[email protected]> wrote:
>
>
>
> On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <[email protected]> wrote:
>>
>> On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
>> > Hello!
>> >
>> > This series drops the in-tree version of our python-qemu-qmp package
>> > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
>> > located at https://gitlab.com/qemu-project/python-qemu-qmp.
>> >
>> > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
>> > (FreeBSD isn't my fault...!)
>> >
>> > The major effects of this patch series are:
>> >
>> > 1. qemu.qmp will be installed from PyPI or vendored packages instead of
>> > being utilized directly from the qemu.git tree.
>>
>> This is not getting installed in enough scenarios IMHO.
>
>
> It's hard to trigger an install when you don't use the build system,
> though... If you bypass make/meson/ninja entirely I'm not really sure what I
> can do to bootstrap the test deps.
>
>>
>> My current workflow is commonly
>>
>> $ ./configure --target-list=x86_64-softmmu
>> $ make
>> $ ./scripts/qmp/qmp-shell-wrap
>> /var/home/berrange/src/virt/qemu/build
>> Traceback (most recent call last):
>> File "/var/home/berrange/src/virt/qemu/scripts/qmp/qmp-shell-wrap", line
>> 7, in <module>
>> from qemu.qmp import qmp_shell
>> ModuleNotFoundError: No module named 'qemu.qmp'
>>
>> Even if I add in a call to 'source build/pyvenv/bin/activate'
>> after 'make', I'm still missing the qemu.qmp python module.
>>
>> AFAICT, qemu.qmp only gets installed in the venv if you run
>> 'make check', and IMHO that should not be expected for the
>> above usage scenario.
>
>
> "make check-venv", to do just the minimum setup. You don't have to run check.
>
> We could always add a flag to configure to front-load the testing environment
> and scripts, but I thought it was best to avoid adding new dependencies by
> default.
>
> Like --with-python-tests or something? Then you'd always have it like you do
> with meson and sphinx (with --enable-docs)
>
>>
>> Likewise I can no longer run any of the test programs directly
>> without first having run 'make check'. eg what I would currently
>> do is:
>>
>> $ ./configure --target-list=x86_64-softmmu
>> $ make
>> $ cd build/tests/qemu-iotests
>> $ ./check 300
>>
>> I don't generally run 'make check' as frequently as I
>> 'configure && make' as it adds alot of time which is
>> not needed usually.
>>
>> In general I find it tedious having to remember to run
>> 'source build/pyvenv/bin/activate' in every terminal
>> window where I might want to run a QEMU script that relies
>> on python. It is also fairly undesirable too, as I use the
>> same terminal window for various purposes, that activating
>> the QEMU venv can interfere with non-QEMU python usage.
>
>
> You can run one-offs with build/pyvenv/bin/python3 without activating the
> environment,
>
> but you're right that this does require an environment setup step (make
> check-venv)
>
>>
>> I would find it helpful if we provided a "run" script in the
>> root dir of the build directory that contained the following
>>
>> $ cat build/run
>> #!/bin/sh
>>
>> set -e
>>
>> HERE=$(realpath $(dirname $0))
>> source $HERE/pyvenv/bin/activate
>> exec "$@"
>>
>> Which would be used as a wrapper to execute scripts with the
>> right environment. That would let us keep the simpler workflow,
>> and avoid polluting the global terminal environment with the
>> qemu venv.
>>
>> eg to be used as
>>
>> $ ./configure --target-list=x86_64-softmmu
>> $ make
>> $ cd build/tests/qemu-iotests
>> $ ../../run ./check 300
>>
>> or
>>
>> $ ./configure --target-list=x86_64-softmmu
>> $ make
>> $ ./build/run ./scripts/qmp/qmp-shell-wrap
>
>
> Shouldn't be too hard to add, honestly. Maybe adding in a "meson compile
> pyvenv/pyvenv_tests_group" line at the start to trigger the dep installation
> if it hasn't occurred already?
>
> Or maybe just directly invoking mkvenv ensuregroup to avoid ninja
> re-configuring things while you develop.
>
> Something like that.
How about something like this inside of your build root:
```
#!/usr/bin/env python3
import os
from pathlib import Path
import shlex
import subprocess
import sys
def main():
venvpath = Path(__file__).parent.joinpath('pyvenv')
groupfile = venvpath.joinpath('checktests.group')
if not groupfile.exists():
subprocess.run(["ninja", "pyvenv/checktests.group"])
env = os.environ.copy()
env['VIRTUAL_ENV'] = str(venvpath)
env['PATH'] = str(venvpath.joinpath('bin')) + ':' + env['PATH']
env['VIRTUAL_ENV_PROMPT'] = 'pyvenv'
if 'PYTHONHOME' in env:
del env['PYTHONHOME']
command = shlex.join(sys.argv[1:])
res = subprocess.run(command, env=env, shell=True)
sys.exit(res.returncode)
if __name__ == '__main__':
main()
```
Then you can run arbitrary things, like "./pyrun which python3" or
whatever you fancy. It only bothers calling ninja if the timestamp
file is not there to help save some startup time and avoid unnecessary
reconfigures. This should work for qmp-shell, qmp-shell-wrap,
qemu-ga-client, qmp-tui, qom, qom-fuse, qom-get, qom-list, qom-set,
qom-tree, device-crash-test, iotests, etc.
>
>
>>
>>
>>
>>
>> > 2. There are no new python dependencies checked or installed during
>> > configure. All test dependencies are installed post-hoc on an
>> > as-needed basis.
>> > 3. "make check-venv" is no longer required to be run manually before any
>> > test that is integrated with meson; this includes "make check" and
>> > "make check-functional".
>> > 4. "make check-venv" no longer installs functional test dependencies: it
>> > installs only the core suite of python test dependencies.
>> > 5. "make check-venv" is now required as a pre-requisite for running
>> > device-crash-test and manually executed iotests.
>> > 6. Unfortunately, python3-wheel and python3-setuptools are now required
>> > on the host system if tests are to be executed and >= Python 3.13 is
>> > used.
>> > 7. An awful lot of deleted lines of code, and a lot fewer headaches
>> > managing two nearly-identical copies of this source code. O:-)
>> >
>> > Patches 1-5 are build system focused; they set up new pythondeps.toml,
>> > mkvenv, and meson systems in preparation for relying on an external
>> > qemu.qmp library, but does not yet make the switch.
>> >
>> > Patches 6-9 are testing and CI focused; they add necessary preparation
>> > steps to keep tests running happily once the in-tree qemu.qmp library is
>> > removed.
>> >
>> > Patches 10-15 are build system focused again; they implement the actual
>> > switchover to the external qemu.qmp library.
>>
>> With regards,
>> Daniel
>> --
>> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange
>> :|
>> |: https://libvirt.org -o- https://fstop138.berrange.com
>> :|
>> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange
>> :|
>>
