> Cross-posting qemu-devel + avocado-devel. > >> While previously working on a Super I/O refactor, I encountered some problems >> at runtime, after building the codebase successfully and running qtests. >> I had to manually start to boot different guests and check the bootlog. >> >> I wanted to give a try at Avocado which seems designed to simplify that kind >> of functional tests. >> >> I applied Amador Pahim work following Cleber Rosa documentation from >> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html, >> however I had to modify few things to parse the boot console. >> Since his work is not merged, I included it in this series. >> >> The tests simply expect to find a string reported by Linux printk when a >> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a >> 16550A" >> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such >> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and >> network >> device are correctly detected: >> >> >> self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog) >> self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog) >> self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog) >> self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog) >> >> Example of the tests output: >> >> $ avocado run test_linux-boot-console.py -m >> test_linux-boot-console.py.data/parameters.yaml >> JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910 >> JOB LOG : >> /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log >> (1/6) >> test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: >> PASS (4.76 s) >> (2/6) >> test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: >> PASS (0.00 s) >> (3/6) >> test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: >> PASS (0.00 s) >> (4/6) >> test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: >> PASS (3.92 s) >> (5/6) >> test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: >> PASS (0.00 s) >> (6/6) >> test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: >> PASS (4.08 s) >> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | >> CANCEL 0 >> JOB TIME : 13.31 s >> >> Please apologize my ugly Python, this series is just a proof-of-concept :) >> I couldn't figure out how to use the @skipUnless(correct arch) decorator. > > Eduardo asked me to share my first impressions after trying this > framework with QEMU. > > So far it seems more designed to run tests _inside_ a qemu instance > (think user-space), eventually testing packages/scripts.
Current examples use the same pattern, which I felt a bit restricting: [QemuTest] .setUp: prepare VM arguments, launch the VM .test_*: run tests in the running VM .tearDown: shutdown the VM and cleanup dangling files > My tests are targeting the machine model itself, if the devices are > correctly instantiated and so. > The fastest approach was to check the Linux kernel bootlog, but if you > look at the TestAlphaClipperBoot console you'll see a boot "firmware" is > executed before the kernel. Casually this firmware also send information > on the console, but what if the console is not accessible? > We can use a chardev for the ioport80 POST, but some firmwares post boot > events via I2C, SPI, CAN... > > To improve the testing, my idea is to use the Tracing framework. > The machine trace events would be logged in a db and avogado would > verify some of the trace events, did they occur? in the correct sequence > order? with the correct arguments? We can also check relative > timestamps, is this sequence timing fast enough? > > I plan to use the gdb API to insert breakpoints and follow part > execution flow, eventually injecting (faulty) events. > > The multi-arch/machines support is a bit weak yet. > > I'm personally more interested in automatically testing [real world] > firmware or full machine boot process, the kind of integration testing I > can not do with qtests. > > I liked the possibility to generate coredumps, or the replay function. > > I also liked to be able to write a test on how a machine boots, in ~20 > LOC (see TestAlphaClipperBoot2_6). > > The storage API is OK to fetch a full VM image, but not a single file > like a kernel or a flash image
signature.asc
Description: OpenPGP digital signature