Currently, we "implicitly" install the local 'qemu' python package for 'make check-venv' with some logic inside tests/Makefile.include. I would like to make this installation explicit in pythondeps.toml instead.
The version constraint specification that Python specifies does not support relative paths, so it is difficult (or impossible?) to specify a path within the source tree, and we will need a workaround to do so. By specifying a package name that starts with $SRCROOT, you can now specify a file path to a local package for installation. This is done to allow us to install the python packages hosted inside of the tree while also processing dependencies; i.e. so that our "qemu" package can specify that it needs "qemu.qmp", which soon will not be included in qemu.git. This also has the benefit of being able to specify in a declarative configuration file that our pyvenv environment *will* have our local python packages installed and available without any PYTHONPATH hacks, which should simplify iotests, device-crash-test and functional tests without needing to manage local inclusion paths in environment variables. On the downsides, installing packages through mkvenv/ensuregroup means that there are extra steps we need to take in order to install a local package *offline*; namely we must disable build isolation (so we have access to setuptools) and we must also include python3-wheel in QEMU's build dependencies in order for "make check" to run successfully when in an offline, isolated environment. Signed-off-by: John Snow <[email protected]> --- python/scripts/mkvenv.py | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/python/scripts/mkvenv.py b/python/scripts/mkvenv.py index a22e3ee3394..608b8faa9b2 100644 --- a/python/scripts/mkvenv.py +++ b/python/scripts/mkvenv.py @@ -662,6 +662,7 @@ def pip_install( args: Sequence[str], online: bool = False, wheels_dir: Optional[Union[str, Path]] = None, + env: Optional[Dict[str, str]] = None, ) -> None: """ Use pip to install a package or package(s) as specified in @args. @@ -687,6 +688,7 @@ def pip_install( full_args += list(args) subprocess.run( full_args, + env=env, check=True, ) @@ -733,9 +735,15 @@ def _do_ensure( :param wheels_dir: If specified, search this path for packages. """ absent = [] + local_packages = [] present = [] canary = None for name, info in group.items(): + if name.startswith("$SRCROOT/"): + srcroot = Path(__file__).parents[2] + pkgpath = name.replace("$SRCROOT/", f"file://{srcroot}/") + local_packages.append(pkgpath) + continue constraint = _make_version_constraint(info, False) matcher = Matcher(name + constraint) print(f"mkvenv: checking for {matcher}", file=sys.stderr) @@ -770,15 +778,33 @@ def _do_ensure( print(f"mkvenv: installing {', '.join(absent)}", file=sys.stderr) try: pip_install(args=absent, online=online, wheels_dir=wheels_dir) - return None + absent = [] except subprocess.CalledProcessError: pass - return diagnose( - absent[0], - online, - wheels_dir, - canary, + if absent: + return diagnose( + absent[0], + online, + wheels_dir, + canary, + ) + + # Handle local packages separately and last so we can use different + # installation arguments (-e), and so that any dependencies that may + # be covered above will be handled according to the depfile + # specifications. + if local_packages: + print(f"mkvenv: installing {', '.join(local_packages)}", + file=sys.stderr) + env = dict(os.environ) + env['PIP_CONFIG_SETTINGS'] = "editable_mode=compat" + pip_install( + args=["--no-build-isolation", + "-e"] + local_packages, + online=online, + wheels_dir=wheels_dir, + env=env, ) return None -- 2.51.1
