On Tue, Dec 09, 2025 at 03:05:43PM -0500, John Snow wrote: > On Tue, Dec 9, 2025 at 2:46 PM Daniel P. Berrangé <[email protected]> wrote: > > > > On Tue, Dec 09, 2025 at 12:23:21PM -0500, John Snow wrote: > > > On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <[email protected]> > > > wrote: > > > > > > > On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote: > > > > > Hello! > > > > > > > > > > This series drops the in-tree version of our python-qemu-qmp package > > > > > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository > > > > > is > > > > > located at https://gitlab.com/qemu-project/python-qemu-qmp. > > > > > > > > > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036 > > > > > (FreeBSD isn't my fault...!) > > > > > > > > > > The major effects of this patch series are: > > > > > > > > > > 1. qemu.qmp will be installed from PyPI or vendored packages instead > > > > > of > > > > > being utilized directly from the qemu.git tree. > > > > > > > > This is not getting installed in enough scenarios IMHO. > > > > > > > > > > It's hard to trigger an install when you don't use the build system, > > > > Well I am using the build system in the same way that I've always > > used it with QEMU. IOW, the benchmark for this is the current way > > things work with the python stuff in tree. The ideal would be to > > match that behaviour with no workflow changes needed, if it is > > practical to achieve that without major downsides. > > I know, but if we add out-of-tree things, there's some fundamental > things that change - we need to load that dependency somewhere, > somewhen.
Yep, there's a tradeoff, so we have to figure out what's practical to achieve. > > > though... If you bypass make/meson/ninja entirely I'm not really sure what > > > I can do to bootstrap the test deps. > > > > Can we just do it unconditionally in configure / meson ? These aren't > > big files to download, and we're already dealing other python stuff > > for the build, and git submodules, and rust crates. Pulling in qemu.qmp > > doesn't feel like it is a burden to do by default ? > > I definitely could, and I think it would be rather convenient; I only > have some minor concerns about it: > > - We don't promise tests on all platforms that we promise builds on. > We have definitely broken this in the past. It might work currently, > but it's a line I want to be aware we are crossing. It may necessitate > python3-wheel and python3-setuptools just to build in this case. > That's probably not an issue on workstations, but it's more bricks on > the wall that are actually not truly needed until you run tests (or > prepare to run tests). We should focus on doing what's optimal for platforms where tests do work, since they're inherantly our most important platforms. If we proactively fetch python deps for tests that happen to be broken on a platform already, I'm not too fussed about that, as long as we don't break things more than they are already broken. > - Configure will get slower by default. I can install the core test > deps by default if people don't mind the additional delay time. It > might be something like 2-4 seconds, depending. If you don't care, I > don't. IIUC, 'pip' will have a local cache of downloads somewhere under $HOME, so presumably when we install deps, we're only going to have a download hit the very first time on a machine, or when the needed versions change ? If we're mostly just copying files out of the local cache, the time overhead should remain fairly minimal most of the time. We have precedent for downloading stuff at configure stage from the Rust deps and git submodules. Those Rust deps are used for built code rather than tests, but if we did need some rust deps only for tests, presumably they'd still be done in configure ? > - Things like functional tests are still going to require some kind of > environment prep for all the extra stuff they require, I don't think > it's reasonable to preload all of that stuff at configure time, and we > never have. "make check-functional" is sufficient to pull those deps > in, but if there are ways to engage the functional tests manually that > people are using, I think another preparation step is unavoidable > there. I'd we interested to know how much overhead loading the functional test deps actually is ? As a conceptual point beyond this series, I've always tended to expect that a "configure" script (or equivalent) is responsible for validating the host OS pre-requisites and/or fetching neede pre-requisites. Anything after that (whether builds or tests) would then (mostly) work from stuff that was already acquired by configure. This lets devs see upfront if there are any problems with their host, instead of the host problems being hit later where the errors are mixed in with build logs. With functional tests we of course don't want to download the assets upfront as that's GB's of data. I would tend towards suggesting that any python deps should be fetched upfront at the configure stage though, unless we see an unacceptable time hit from that. > So, in addition to the "pyrun" wrapper I mailed in a separate reply > (which I think is a good idea anyway, regardless of what direction we > go here), I see two main paths that address your issues in differing > amounts: I sent that as a formal proposal here: https://lists.nongnu.org/archive/html/qemu-devel/2025-12/msg01351.html For the functional test example illustrated there though, we would preferrably want the functional test deps fetched upfront. > (1) make configure prepare the test deps *by default*, adding a flag > like --without-test-deps to disable it, or Note, this is not just about test deps. The qemu python code is used by other misc QEMU developer tools, such as the qmp-shell command. I'm considering the python deps as more like "QEMU build env pre-requisites", which is what pushed me towards believeing they should be fetched by default > (2) Continue not prepping the test deps, but allow --with-test-deps to > pre-load them. > > More or less the same solution, just with different defaults. I'm > fairly ambivalent, my only personal "habit" here is "I am really > reluctant to touch configure unless there is a strong consensus around > it because I dislike having to make arguments at that level." With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
