On Fri, Jan 21, 2022 at 10:22 AM Daniel P. Berrangé <berra...@redhat.com> wrote: > > On Fri, Jan 21, 2022 at 12:23:23PM +0000, Alex Bennée wrote: > > > > Peter Maydell <peter.mayd...@linaro.org> writes: > > > > > On Fri, 21 Jan 2022 at 10:50, Markus Armbruster <arm...@redhat.com> wrote: > > >> No objection, but it's no replacement for looking into why these tests > > >> are so slow. > > >> > > >> The #1 reason for things being slow is not giving a damn :) > > > > > > See previous messages in the thread -- the test starts a > > > full-fat guest OS including UEFI boot, and it takes forever to > > > get to the login prompt because systemd is starting everything > > > including the kitchen sink. > > > > There has to be a half-way house between booting a kernel until it fails > > to find a rootfs and running a full Ubuntu distro. Maybe just asking > > systemd to reach "rescue.target" would be enough to show the disks are > > up and userspace works. > > Booting up full OS distros is useful, but at the same time I feel it > is too much as something to expect developers to do on any kind of > regular basis. >
Agreed. The solution IMO can be as simple as having different "test job profiles". > Ideally some decent amount of acceptance testing could be a standard > part of the 'make check', but that's impossible as long as we're > downloading large disk images or booting things that are very slow, > especially so with TCG. > > IMHO the ideal scenario would be for us to have a kernel, initrd > containing just busybox tools for the key arch targets we care > about. Those could be used with direct kernel boot or stuffed > into a disk iamge. Either way, they would boot in ~1 second, > even with TCG, and would be able to execute simple shell scripts > to test a decent amount of QEMU functionality. > I see different use cases here: A) Testing that QEMU can boot a full distro For testing purposes, the more different subsystems the "boot" process depends on, the better. Currently the "boot_linux.py" tests require the entire guest boot to complete and have a networking configuration and interaction. B) Using something as a base OS for scripts (tests) to run on it Here's where there's the most benefit in having a more lightweight distro (or kernel + initrd). But, this requirement will also come in different "optimal" sizes for different people. Some of the existing tests require not only a Fedora system, but a given version that has given capabilities. For a sustainable, framework-like solution, tests should be able to determine the guest they need with minimal setup from test writers[1]. If a Fedora-like system is not needed, maybe a lightweight system like CirrOS[2] is enough. CirrOS, unfortunately, can not be used Today as the distro in most of the acceptance tests because the cloud-init mechanism used to configure the networking is not currently supported, although there have been discussions to consider implementing it[3]. > It wouldn't eliminate the need to test with full OS, but it > would let us have some acceptance testing run as standard with > 'make check' in a decently fast time. It would then be less > critical if the more thorough full OS tests were somewhat > slower than we'd like. We could just leave those as a scheduled > job to run overnight post-merge. If they do detect any problems > post-merge, then write a dedicated test scenario to replicate it > under the minimal kernel/initrd acceptance test so it'll be > caught pre-merge in future. > Assuming this is about "Testing that QEMU can boot a full distro", I wouldn't try to solve the problem by making the distro too slim to get to the point of becoming an unrealistic system. IMO the deal breaker with regards to test time can be solved more cheaply by having and using KVM where these tests will run, and not running them by default otherwise. With the tagging mechanism we should be able to set a condition such as: "If using TCG, exclude tests that boot a full blown distro. If using KVM, do not criticize what gets booted". Resulting in something like: $ avocado list -t accel:tcg,boots:-distro -t accel:kvm ~/src/qemu/tests/avocado/{boot_linux.py,boot_linux_console.py} avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux.py:BootLinuxAarch64.test_virt_kvm avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_virt avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_xlnx_versal_virt avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_virt avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_emcraft_sf2 avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_raspi2_uart0 avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_exynos4210_initrd avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_initrd avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_sata avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_quanta_gsj avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_quanta_gsj_initrd avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd avocado-instrumented /home/cleber/src/qemu/tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd Does that sound like something appropriate? BTW, on the topic of "Using something as a base OS for scripts (tests) to run on it", another possibility for using full blown OS would be to save their initialized state, and load it to memory for each test, saving the guest boot time. This should of course be done at the framework level and transparent to tests. Best, - Cleber. [1] https://avocado-framework.readthedocs.io/en/94.0/guides/writer/libs/vmimage.html#supported-images [2] https://launchpad.net/cirros [3] https://github.com/cirros-dev/cirros/issues/67