Hello, Thomas!
On 8/25/25 07:29, Thomas Huth wrote:
On 19/08/2025 16.39, Gustavo Romero wrote:
The goal of this series is to remove Avocado as a dependency for running
the reverse_debugging functional test.
This test, the last one I’m aware of that relies on Avocado, requires it
because of the need for GDB to test reverse stepping and continue.
Hi!
Please note that there are currently also some efforts going on to extract the
GDB part from avocado into a more self-contained python module called aautils,
which might help here, too:
https://github.com/avocado-framework/aautils/issues/82
Thanks, I was not aware of it. But note that the Avocado GDB API requires one
to deal with GDB packets directly, which is unnecessary for tests like reverse
debug. I think that in general, in QEMU, we should try to avoid using it as
much as possible because the tests became annoying to read without any strong
need for it.
In the commit message for 4/4 I mention that benefit of using the GDB Python
API instead.
I think we should aim to make the tests in QEMU _extremely_ easy to list, run,
and read. The original reverse_debugging test, using Avocado's GDB API, for
instance, is one example we should try to avoid. Say some developer is changing
something not related to RR but ends up breaking reverse_debugging. The
developer
should at least be able to promptly run the QEMU, connect GDB, and reproduce the
issue manually following the exact commands in the test, without having to
consult the GDB protocol manual and to find out which GDB command is mapped to
which GDB packet.
In that sense I confess I'm not 100% happy with the output of reverse_debugging
yet, but one problem at a time :)
In this series, we leveraged the run-test.py script used in the
check-tcg tests, making it a GDB runner capable of calling a test script
without spawning any VMs. In this configuration, the test script can
manage the VMs and also import gdb from the test script, making the
Python GDB API available inside the functional test.
The test is kept “skipped” for aarch64, ppc64, and x86_64, so it is
necessary to set QEMU_TEST_FLAKY_TESTS=1 in the test environment to
effectively run the test on these archs.
On aarch64, the test is flaky, but there is a fix that I’ve tested while
writing this series [0] that resolves it. On ppc64 and x86_64, the test
always fails: on ppc64, GDB gets a bogus PC, and on x86_64, the last
part of the test (reverse-continue) does not hit the last executed PC
(as it should) but instead jumps to the beginning of the code (fist PC
in forward order).
At least the x86 test used to work for me most of the time, it just fails
occasionally. Same for the ppc64 test if I disable the powernv subtest there.
ok, I'll double check the state on x86 in master. I understand we're not
testing reverse_debugging in our CI currently?
>> Thus, to run the reverse_debugging test effectively on aarch64:
$ export QEMU_TEST_FLAKY_TESTS=1
$ make check-functional
I gave it a try, but this did not work for me, the test was not run at all
anymore. Are there any patches needed on top?
hmm that's odd. I'm able to run it with 'make check-functional' and with 'meson
test'...
This is how I'm running it (let me know if I'm missing something):
gromero@gromero0:/mnt/git/qemu_$ b4 shazam
20250819143916.4138035-1-gustavo.rom...@linaro.org
Grabbing thread from
lore.kernel.org/all/20250819143916.4138035-1-gustavo.rom...@linaro.org/t.mbox.gz
Checking for newer revisions
Grabbing search results from lore.kernel.org
Analyzing 9 messages in the thread
Looking for additional code-review trailers on lore.kernel.org
Analyzing 0 code-review messages
Checking attestation on all messages, may take a moment...
---
✓ [PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py
✓ [PATCH 2/4] tests/functional: Support tests that require a runner
✓ [PATCH 3/4] tests/functional: Mark main in QemuBaseTest class as a static
method
+ Reviewed-by: Daniel P. Berrangé <berra...@redhat.com> (✓ DKIM/redhat.com)
✓ [PATCH 4/4] tests/functional: Adapt reverse_debugging to run w/o Avocado
---
✓ Signed: DKIM/linaro.org
---
Total patches: 4
---
Applying: tests/guest-debug: Make QEMU optional in run-test.py
Applying: tests/functional: Support tests that require a runner
Applying: tests/functional: Mark main in QemuBaseTest class as a static method
Applying: tests/functional: Adapt reverse_debugging to run w/o Avocado
gromero@gromero0:/mnt/git/qemu_$
gromero@gromero0:/mnt/git/qemu_/build$ ../configure
--target-list=aarch64-softmmu --disable-docs
gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15 check-functional
[1/2] Generating tests/functional/func-precache-aarch64-aarch64_reverse_debug
with a custom command (wrapped by meson to set env)
2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache
'/home/gromero/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset
/home/gromero/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7
for
https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm
off' -ex "py
sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x
/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
[0/1] Running external command precache-functional (wrapped by meson to set env)
make[1]: Entering directory '/mnt/git/qemu_/build'
[1/6] Generating qemu-version.h with a custom command (wrapped by meson to
capture output)
/mnt/git/qemu_/build/pyvenv/bin/meson test --no-rebuild -t 1 --setup thorough
--num-processes 10 --print-errorlogs --suite func --suite func-quick --suite
func-thorough
1/27 qemu:func-thorough+func-aarch64-thorough+thorough /
func-aarch64-aarch64_virt_gpu SKIP 1.95s 0
subtests passed
2/27 qemu:func-thorough+func-aarch64-thorough+thorough /
func-aarch64-aarch64_smmu SKIP 4.17s 0
subtests passed
3/27 qemu:func-thorough+func-aarch64-thorough+thorough /
func-aarch64-aarch64_raspi4 OK 28.11s 2
subtests passed
4/27 qemu:func-thorough+func-aarch64-thorough+thorough /
func-aarch64-aarch64_device_passthrough OK 31.34s 1
subtests passed
5/27 qemu:func-thorough+func-aarch64-thorough+thorough /
func-aarch64-aarch64_reverse_debug OK 11.73s
<------ 'exitcode', not TAP, so no subtests-like output
6/27 qemu:func-quick+func-aarch64 / func-aarch64-migration
OK 0.97s 2 subtests passed
7/27 qemu:func-quick+func-aarch64 / func-aarch64-empty_cpu_model
OK 0.17s 1 subtests passed
[...]
gromero@gromero0:/mnt/git/qemu_/build$ grep -A 58 "= 5/27 ="
/mnt/git/qemu_/build/meson-logs/testlog-thorough.txt
==================================== 5/27 ====================================
test: qemu:func-thorough+func-aarch64-thorough+thorough /
func-aarch64-aarch64_reverse_debug
start time: 13:44:21
duration: 11.73s
result: exit status 0
command:
LD_LIBRARY_PATH=/mnt/git/qemu_/build/contrib/plugins:/mnt/git/qemu_/build/tests/tcg/plugins
QEMU_BUILD_ROOT=/mnt/git/qemu_/build MALLOC_PERTURB_=138 G_TEST_SLOW=1
ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 RUST_BACKTRACE=1
QEMU_TEST_QEMU_IMG=/mnt/git/qemu_/build/qemu-img MESON_TEST_ITERATION=1
PYTHONPATH=/mnt/git/qemu_/python:/mnt/git/qemu_/tests/functional:/mnt/git/qemu_/build/pyvenv/lib/python3.10/site-packages
QEMU_TEST_QEMU_BINARY=/mnt/git/qemu_/build/qemu-system-aarch64
UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1
MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1
/mnt/git/qemu_/build/pyvenv/bin/python3
/mnt/git/qemu_/tests/guest-debug/run-test.py --gdb /usr/bin/gdb-multiarch
--test /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
----------------------------------- stdout -----------------------------------
Formatting
'/mnt/git/qemu_/build/tests/functional/aarch64/test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt/scratch/disk.qcow2',
fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib
size=134217728 lazy_refcounts=off refcount_bits=16
TAP version 13
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0000 in ?? ()
0x0000000040080004 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040000010 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000008 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
Breakpoint 1 at 0x40000000
Breakpoint 2 at 0x40000004
Breakpoint 3 at 0x40000008
Breakpoint 4 at 0x4000000c
Breakpoint 5 at 0x40000010
Breakpoint 6 at 0x40000014
Breakpoint 7 at 0x40080000
Breakpoint 8 at 0x40080004
Breakpoint 9 at 0x410d0000
Breakpoint 10 at 0x410d0020
Program received signal SIGINT, Interrupt.
0xffff3e25348a2050 in ?? ()
Breakpoint 10, 0x00000000410d0020 in ?? ()
ok 1 test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
1..1
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm
off' -ex "py
sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x
/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
==============================================================================
gromero@gromero0:/mnt/git/qemu_/build$
Also, using meson test:
gromero@gromero0:/mnt/git/qemu_/build$ cat
/mnt/git/qemu_/build/meson-logs/testlog-thorough.txt
Log of Meson test suite run on 2025-08-25T13:54:13.544507
Inherited environment: SHELL=/bin/bash PYENV_SHELL=bash LC_ADDRESS=en_US.UTF-8
LC_NAME=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 PWD=/mnt/git/qemu_/build
LOGNAME=gromero XDG_SESSION_TYPE=tty TVM_HOME=/home/gromero/git/tvm
MOTD_SHOWN=pam HOME=/home/gromero LANG=en_US.UTF-8 LC_PAPER=en_US.UTF-8
LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
SSH_CONNECTION='192.168.122.1 57300 192.168.122.158 22'
LESSCLOSE='/usr/bin/lesspipe %s %s' XDG_SESSION_CLASS=user
PYTHONPATH=/home/gromero/git/tvm/python: TERM=xterm-256color
LC_IDENTIFICATION=en_US.UTF-8 LESSOPEN='| /usr/bin/lesspipe %s'
QEMU_TEST_FLAKY_TESTS=1 USER=gromero SHLVL=1 LC_TELEPHONE=en_US.UTF-8
LC_MEASUREMENT=en_US.UTF-8 XDG_SESSION_ID=2738 XDG_RUNTIME_DIR=/run/user/1000
SSH_CLIENT='192.168.122.1 57300 22' PYENV_ROOT=/home/gromero/.pyenv
LC_TIME=en_US.UTF-8
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/home/gromero/.pyenv/shims:/home/gromero/.pyenv/bin:/home/gromero/.local/bin:/home/gromero/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus SSH_TTY=/dev/pts/0
LC_NUMERIC=en_US.UTF-8 OLDPWD=/mnt/git/qemu_
_=/mnt/git/qemu_rr_fix/build/pyvenv/bin/meson
==================================== 1/1 =====================================
test: qemu:func-thorough+func-aarch64-thorough+thorough /
func-aarch64-aarch64_reverse_debug
start time: 13:54:13
duration: 2.65s
result: exit status 0
command:
PYTHONPATH=/mnt/git/qemu_/python:/mnt/git/qemu_/tests/functional:/mnt/git/qemu_/build/pyvenv/lib/python3.10/site-packages
MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1
LD_LIBRARY_PATH=/mnt/git/qemu_/build/contrib/plugins:/mnt/git/qemu_/build/tests/tcg/plugins
RUST_BACKTRACE=1 MALLOC_PERTURB_=243
ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1
QEMU_TEST_QEMU_BINARY=/mnt/git/qemu_/build/qemu-system-aarch64
QEMU_BUILD_ROOT=/mnt/git/qemu_/build
UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1
G_TEST_SLOW=1 MESON_TEST_ITERATION=1
QEMU_TEST_QEMU_IMG=/mnt/git/qemu_/build/qemu-img SPEED=thorough
/mnt/git/qemu_/build/pyvenv/bin/python3
/mnt/git/qemu_/tests/guest-debug/run-test.py --gdb /usr/bin/gdb-multiarch
--test /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
----------------------------------- stdout -----------------------------------
Formatting
'/mnt/git/qemu_/build/tests/functional/aarch64/test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt/scratch/disk.qcow2',
fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib
size=134217728 lazy_refcounts=off refcount_bits=16
TAP version 13
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0000 in ?? ()
0x0000000040080004 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040000010 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000008 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
Breakpoint 1 at 0x40000000
Breakpoint 2 at 0x40000004
Breakpoint 3 at 0x40000008
Breakpoint 4 at 0x4000000c
Breakpoint 5 at 0x40000010
Breakpoint 6 at 0x40000014
Breakpoint 7 at 0x40080000
Breakpoint 8 at 0x40080004
Breakpoint 9 at 0x410d0000
Breakpoint 10 at 0x410d0020
Program received signal SIGINT, Interrupt.
0xffff47bcc84a2050 in ?? ()
Breakpoint 10, 0x00000000410d0020 in ?? ()
ok 1 test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
1..1
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm
off' -ex "py
sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x
/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
==============================================================================
Ok: 1
Fail: 0
gromero@gromero0:/mnt/git/qemu_/build$
Any chance you got confused by no subtests-like output caused by 'exitcode'
used instead of ´tap'?
Anyway, shifting to a different test harness here makes me wonder whether the
whole reverse_debug test should maybe be rather moved to tests/tcg instead,
where we already have the basic support for the stuff from tests/guest-debug/ ?
The aarch64 would require a different guest payload, of course, in that case,
so not sure whether it's feasible?
I think reverse_debugging is really a functional test. It requires GDB, yes,
but also QMP and booting a whole kernel and the feature itself makes me think
it's a functional test. I wouldn't move it to tcg-check just for the sake of
no adding a new way to run test in meson.build in functional tests.
Cheers,
Gustavo
