On 10/12/25 21:26, Piyush Raj wrote:
> This patch adds the bpf-vmtest-tool subdirectory under contrib which tests
> BPF programs under a live kernel using a QEMU VM. It automatically
> builds the specified kernel version with eBPF support enabled
> and stores it under "~/.bpf-vmtest-tool", which is reused for future
> invocations.
>
> It can also compile BPF C source files or BPF bytecode objects and
> test them against the kernel verifier for errors. When a BPF program
> is rejected by the kernel verifier, the verifier logs are displayed.
>
> $ python3 main.py -k 6.15 --bpf-src assets/ebpf-programs/fail.c
> BPF program failed to load
> Verifier logs:
> btf_vmlinux is malformed
> 0: R1=ctx() R10=fp0
> 0: (81) r0 = *(s32 *)(r10 +4)
> invalid read from stack R10 off=4 size=4
> processed 1 insns (limit 1000000) max_states_per_insn 0 total_states
> 0 peak_states 0 mark_read 0
>
> See the README for more examples.
>
> The script uses vmtest (https://github.com/danobi/vmtest) to boot
> the VM and run the program. By default, it uses the host's root
> ("/") as the VM rootfs via the 9p filesystem, so only the kernel is
> replaced during testing.
>
> Tested with Python 3.9 and above.
This looks good to me, but please wait for Jose or another reviewer.
I have some usability suggestions inline below but none are blockers,
they can all wait to be addressed in the future.
>
> contrib/ChangeLog:
>
> * bpf-vmtest-tool/README: New file.
> * bpf-vmtest-tool/bpf.py: New file.
> * bpf-vmtest-tool/config.py: New file.
> * bpf-vmtest-tool/kernel.py: New file.
> * bpf-vmtest-tool/main.py: New file.
> * bpf-vmtest-tool/pyproject.toml: New file.
> * bpf-vmtest-tool/tests/test_cli.py: New file.
> * bpf-vmtest-tool/utils.py: New file.
> * bpf-vmtest-tool/vm.py: New file.
>
> Signed-off-by: Piyush Raj <[email protected]>
> ---
> contrib/bpf-vmtest-tool/README | 78 ++++++++
> contrib/bpf-vmtest-tool/bpf.py | 199 ++++++++++++++++++++
> contrib/bpf-vmtest-tool/config.py | 18 ++
> contrib/bpf-vmtest-tool/kernel.py | 209 ++++++++++++++++++++++
> contrib/bpf-vmtest-tool/main.py | 103 +++++++++++
> contrib/bpf-vmtest-tool/pyproject.toml | 36 ++++
> contrib/bpf-vmtest-tool/tests/test_cli.py | 167 +++++++++++++++++
> contrib/bpf-vmtest-tool/utils.py | 27 +++
> contrib/bpf-vmtest-tool/vm.py | 154 ++++++++++++++++
> 9 files changed, 991 insertions(+)
> create mode 100644 contrib/bpf-vmtest-tool/README
> create mode 100644 contrib/bpf-vmtest-tool/bpf.py
> create mode 100644 contrib/bpf-vmtest-tool/config.py
> create mode 100644 contrib/bpf-vmtest-tool/kernel.py
> create mode 100644 contrib/bpf-vmtest-tool/main.py
> create mode 100644 contrib/bpf-vmtest-tool/pyproject.toml
> create mode 100644 contrib/bpf-vmtest-tool/tests/test_cli.py
> create mode 100644 contrib/bpf-vmtest-tool/utils.py
> create mode 100644 contrib/bpf-vmtest-tool/vm.py
>
> diff --git a/contrib/bpf-vmtest-tool/README b/contrib/bpf-vmtest-tool/README
> new file mode 100644
> index 00000000000..599e3529aa8
> --- /dev/null
> +++ b/contrib/bpf-vmtest-tool/README
> @@ -0,0 +1,78 @@
> +This directory contains a Python script to run BPF programs or shell commands
> +under a live Linux kernel using QEMU virtual machines.
> +
> +USAGE
> +=====
> +
> +To run a shell command inside a live kernel VM:
> +
> + python main.py -k 6.15 -r / -c "uname -a"
> +
> +To run a BPF source file in the VM:
> +
> + python main.py -k 6.15 --bpf-src fail.c
> +
> +To run a precompiled BPF object file:
> +
> + python main.py -k 6.15 --bpf-obj fail.bpf.o
> +
> +The tool will download and build the specified kernel version from:
> +
> + https://www.kernel.org/pub/linux/kernel
> +
> +A prebuilt `bzImage` can be supplied using the `--kernel-image` flag.
IMO it's easy to miss the -v/--log-level flag, and that by default it
will not print any informational messages while downloading and building
the kernel.
It might be better to set --log-level=INFO by default, and turn it off
when invoking from the testsuite. That way a user running the tool
directly has by default more indication of what's going on.
In general it would be good to describe the available options and their
defaults here.
> +
> +NOTE
> +====
> +- Only x86_64 is supported
> +- Only "/" (the root filesystem) is currently supported as the VM rootfs when
> +running or testing BPF programs using `--bpf-src` or `--bpf-obj`.
> +
> +DEPENDENCIES
> +============
> +
> +- Python >= 3.9
> +- vmtest >= v0.18.0 (https://github.com/danobi/vmtest)
> + - QEMU
> + - qemu-guest-agent
> +
> +For compiling kernel
> +- https://docs.kernel.org/process/changes.html#current-minimal-requirements
> +For compiling and loading BPF programs:
> +
> +- libbpf
> +- bpftool
> +- gcc-bpf-unknown-none
> + (https://gcc.gnu.org/wiki/BPFBackEnd#Where_to_find_GCC_BPF)
> +- vmlinux.h
> + Can be generated using:
> +
> + bpftool btf dump file /sys/kernel/btf/vmlinux format c > \
> + /usr/local/include/vmlinux.h
> +
> + Or downloaded from https://github.com/libbpf/vmlinux.h/tree/main
> +
> +BUILD FLAGS
> +===========
> +You can customize compiler settings using environment variables:
> +
> +- BPF_CC: Compiler for the BPF program (default:
> bpf-unknown-none-gcc)
> +- BPF_CFLAGS: Extra flags for BPF program compilation (default: "-O2")
> +- BPF_INCLUDES: Include paths for BPF (default: -I/usr/local/include
> -I/usr/include)
> +- VMTEST_CC: Compiler for the user-space loader (default: gcc)
> +- VMTEST_CFLAGS: Flags for compiling the loader (default: -g -Wall)
> +- VMTEST_LDFLAGS: Linker flags for the loader (default: -lelf -lz -lbpf)
> +
> +Example usage:
> +
> + BPF_CFLAGS="-O3 -g" CFLAGS="-O2" python main.py -k 6.15 --bpf-src fail.c
> +
> +DEVELOPMENT
> +===========
> +
> +Development dependencies are specified in `pyproject.toml`, which can be used
> +with any suitable Python virtual environment manager.
> +
> +To run the test suite:
> +
> + python3 -m pytest
--- 8< ---
> --- /dev/null
> +++ b/contrib/bpf-vmtest-tool/vm.py
> @@ -0,0 +1,154 @@
> +import logging
> +import subprocess
> +from typing import List
> +
> +from kernel import KernelImage
> +
> +logger = logging.getLogger(__name__)
> +
> +
> +class VMConfig:
> + """Configuration container for VM settings"""
> +
> + def __init__(
> + self, kernel_image: KernelImage, rootfs_path: str, command: str,
> **kwargs
> + ):
> + self.kernel = kernel_image
> + self.kernel_path = str(kernel_image.path)
> + self.rootfs_path = rootfs_path
> + self.command = command
> + self.memory_mb = kwargs.get("memory_mb", 512)
> + self.cpu_count = kwargs.get("cpu_count", 1)
> + self.extra_args = kwargs.get("extra_args", {})
> +
> +
> +def bpf_verifier_logs(output: str) -> str:
> + start_tag = "--- Verifier log start ---"
> + end_tag = "--- Verifier log end ---"
> +
> + start_idx = output.find(start_tag)
> + end_idx = output.find(end_tag)
> +
> + if start_idx != -1 and end_idx != -1:
> + # Extract between the tags (excluding the markers themselves)
> + log_body = output[start_idx + len(start_tag) : end_idx].strip()
> + return log_body
> + else:
> + return "No verifier log found in the output."
> +
> +
> +class Vmtest:
> + """vmtest backend implementation"""
> +
> + def __init__(self):
> + pass
> +
> + def _boot_command(self, vm_config: VMConfig):
> + vmtest_command = ["vmtest"]
> + vmtest_command.extend(["-k", vm_config.kernel_path])
> + vmtest_command.extend(["-r", vm_config.rootfs_path])
> + vmtest_command.append(vm_config.command)
> + return vmtest_command
> +
> + def _remove_boot_log(self, full_output: str) -> str:
> + """
> + Filters QEMU and kernel boot logs, returning only the output after
> the
> + `===> Running command` marker.
> + """
> + marker = "===> Running command"
> + lines = full_output.splitlines()
> +
> + try:
> + start_index = next(i for i, line in enumerate(lines) if marker
> in line)
> + # Return everything after that marker (excluding the marker
> itself)
> + return "\n".join(lines[start_index + 1 :]).strip()
> + except StopIteration:
> + return full_output.strip()
> +
> + def run_command(self, vm_config):
> + vm = None
> + try:
> + logger.info(f"Booting VM with kernel: {vm_config.kernel_path}")
> + logger.info(f"Using rootfs: {vm_config.rootfs_path}")
> + vm = subprocess.run(
> + self._boot_command(vm_config),
> + check=True,
> + text=True,
> + capture_output=True,
> + shell=False,
> + )
> + vm_stdout = vm.stdout
> + logger.debug(vm_stdout)
> + return VMCommandResult(
> + vm.returncode, self._remove_boot_log(vm_stdout), None
> + )
> + except subprocess.CalledProcessError as e:
> + out = e.stdout
> + err = e.stderr
> + # when the command in the vm fails we consider it as a
> succesfull boot
> + if "===> Running command" not in out:
> + raise BootFailedError("Boot failed", out, err, e.returncode)
> + logger.debug("STDOUT: \n%s", out)
> + logger.debug("STDERR: \n%s", err)
> + return VMCommandResult(e.returncode, self._remove_boot_log(out),
> err)
> +
> +
> +class VMCommandResult:
> + def __init__(self, returncode, stdout, stderr) -> None:
> + self.returncode = returncode
> + self.stdout = stdout
> + self.stderr = stderr
> +
> +
> +class VirtualMachine:
> + """Main VM class - simple interface for end users"""
> +
> + # Registry of available hypervisors
> + _hypervisors = {
> + "vmtest": Vmtest,
> + }
> +
> + def __init__(
> + self,
> + kernel_image: KernelImage,
> + rootfs_path: str,
> + command: str,
> + hypervisor_type: str = "vmtest",
> + **kwargs,
> + ):
> + self.config = VMConfig(kernel_image, rootfs_path, command, **kwargs)
> +
> + if hypervisor_type not in self._hypervisors:
> + raise ValueError(f"Unsupported hypervisor: {hypervisor_type}")
> +
> + self.hypervisor = self._hypervisors[hypervisor_type]()
> +
> + @classmethod
> + def list_hypervisors(cls) -> List[str]:
> + """List available hypervisors"""
> + return list(cls._hypervisors.keys())
> +
> + def execute(self):
> + """Execute command in VM"""
> + return self.hypervisor.run_command(self.config)
If vmtest is not installed (or not in the PATH) this results in an
ugly and un-useful traceback through to the python stdlib subprocess.py.
It is user (me :D) error of course, but still it would be nice to add
a little bit of handling here to give a more helpful error message
if possible.
> +
> +
> +class BootFailedError(Exception):
> + """Raised when VM fails to boot properly (before command execution)."""
> +
> + def __init__(
> + self, message: str, stdout: str = "", stderr: str = "", returncode:
> int = -1
> + ):
> + super().__init__(message)
> + self.stdout = stdout
> + self.stderr = stderr
> + self.returncode = returncode
> +
> + def __str__(self):
> + base = super().__str__()
> + return (
> + f"{base}\n"
> + f"Return code: {self.returncode}\n"
> + f"--- STDOUT ---\n{self.stdout}\n"
> + f"--- STDERR ---\n{self.stderr}"
> + )
