On Fri, May 13, 2022 at 11:55:18AM -0400, John Snow wrote: > On Fri, May 13, 2022, 6:29 AM Daniel P. Berrangé <[email protected]> > wrote: > > > On Fri, May 13, 2022 at 09:35:23AM +0100, Daniel P. Berrangé wrote: > > > On Thu, May 12, 2022 at 08:06:00PM -0400, John Snow wrote: > > > > RFC: This is a very early, crude attempt at switching over to an > > > > external Python package dependency for QMP. This series does not > > > > actually make the switch in and of itself, but instead just switches to > > > > the paradigm of using a venv in general to install the QEMU python > > > > packages instead of using PYTHONPATH to load them from the source tree. > > > > > > > > (By installing the package, we can process dependencies.) > > > > > > > > I'm sending it to the list so I can show you some of what's ugly so far > > > > and my notes on how I might make it less ugly. > > > > > > > > (1) This doesn't trigger venv creation *from* iotests, it merely prints > > > > a friendly error message if "make check-venv" has not been run > > > > first. Not the greatest. > > > > > > So if we run the sequence > > > > > > mkdir build > > > cd build > > > ../configure > > > make > > > ./tests/qemu-iotests/check 001 > > > > > > It won't work anymore, until we 'make check-venv' (or simply > > > 'make check') ? > > > > > > I'm somewhat inclined to say that venv should be created > > > unconditionally by default. ie a plain 'make' should always > > > everything needed to be able to invoke the tests directly. > > > > > > > (2) This isn't acceptable for SRPM builds, because it uses PyPI to > > fetch > > > > packages just-in-time. My thought is to use an environment variable > > like > > > > QEMU_CHECK_NO_INTERNET that changes the behavior of the venv setup > > > > process. We can use "--system-site-packages" as an argument to venv > > > > creation and "--no-index" as an argument to pip installation to achieve > > > > good behavior in SRPM building scenarios. It'd be up to the spec-writer > > > > to opt into that behavior. > > > > > > I think I'd expect --system-site-packages to be the default behaviour. > > > We expect QEMU to be compatible with the packages available in the > > > distros that we're targetting. So if the dev has the python packages > > > installed from their distro, we should be using them preferentially. > > > > > > This is similar to how we bundle slirp/capstone/etc, but will > > > preferentially use the distro version if it is available. > > > > AFAICT from testing it, when '--system-site-packages' is set > > for the venv, then 'pip install' appears to end up being a > > no-op if the package is already present in the host, but > > installs it if missing. > > > > IOW, if we default to --system-site-packages, but still > > also run 'pip install', we should "do the right thing". > > It'll use any distro packages that are available, and > > augment with stuff from pip. In the no-op case, pip will > > still try to consult the pipy servers to fetch version > > info IIUC, so we need to be able to stop that. So I'd > > suggest a --python-env arg taking three values > > > > * "auto" (the default) - add --system-site-packages, but > > also run 'pip install'. Good for developers day-to-day > > > > Sounds like a decent balance... > > ...My only concern is that the system packages might be very old and it's > possible that the qemu packages will be "too new" or have conflicts with > the system deps. > > I'll just have to test this. > > e.g. what if I want to require mypy >= 0.900 for testing, but you have a > system package that requires mypy < 0.700?
I would expect us to not require packages that are not present in the distros implied by https://www.qemu.org/docs/master/about/build-platforms.html if that was absolutely a must have, then gracefully skip tests if the system version wasn't new enough. The user could always pass --python-env=pip if they want to force new enough > The python dep compatibility matrix I try to enforce and support for > testing is already feeling overly wide. This might force me to support an > even wider matrix, which I think is the precisely wrong direction for venvs > where we want tighter control as a rule. As above, from a QEMU POV we have clearly defined our targetted platform matrix. Splitting off python packages isn't a reason to change QEMU's platform matrix IMHO. > > * "system" - add --system-site-packages but never run > > 'pip install'. Good for formal package builds. > > > > We still have to install the in-tree qemu ns package, but we can use > --no-index to do it. It'll fail if the deps aren't met. > > * "pip" - don't add --system-site-packages, always run > > 'pip install'. Good for testing compatibility with > > bleeding edge python versions, or if explicit full > > independance from host python install is desired. > > > > as arguments to configure, this spread of options makes sense to me than > paolo's version, but I've still got some doubt on mixing system and venv > packages. > > I am also as of yet not sold on building the venv *from* configure, see my > response to Paolo on that topic. > > I'll keep plugging away for now, but the big picture is still a tad murky > in my head. > > --js With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
