On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <cr...@redhat.com> wrote: > TL;DR > ===== > > This is about how QEMU developers can get started with functional > tests that are built on top of the Avocado libraries (and meant to be > run with the Avocado test runner). > > The past > ======== > > The Avocado project[1] has been working, for quite some time now, on a > "set of tools and libraries" with the goal of making writing tests > easier. It is supposed to be a framework agnostic to the exact > software that will be under test. > > But, at the same time, the Avocado project cannot deny its inheritance > and influences. Those come from Autotest[2], which had "KVM Autotest" > as its largest and most developed "test". This large Autotest test > (KVM Autotest) became virt-test[3] and later got integrated into > Avocado and became Avocado-VT[4] which is quite relevant here, > together with its QEMU test provider[5]. > > Avocado-VT and the QEMU test provider attempt to provide coverage > across platform and QEMU versions, which increases its complexity. > Also, it's built on a legacy set of principles and tools that makes > some developers stir away from it. > > What's new? > =========== > > A few months ago, the Avocado developers returned to its > "virtualization origins", in an attempt to learn from the QEMU > project, and try to help with a way to have more functional tests in > the upstream QEMU repo. > > We believe it's possible to expand the test coverage for QEMU by > facilitating > the creation of more functional tests QEMU. This is no different than how > other types of tests are already included in the tree itself. > > How > === > > How we did it (so far) > ---------------------- > > We're aware that there's a dilemma here: to be able to easily write > more powerful tests, a lot of the complexity has to be moved > elsewhere. Here, it means moving complexity from the test itself to a > framework. The QEMU source tree itself has proofs of this approach, > being the "scripts" and "tests/qemu-iotests" some of the examples. > > Avocado itself[1] provides a lot of the code that should help to > absorb some of the complexities in writing tests, but not exactly > everything that is needed for QEMU. The approach we believe will have > the best balance is to reuse upstream Avocado libraries whenever they > are useful and generic enough, and on top of that, libraries that are > part of QEMU itself. > > How can you get started with it > ------------------------------- > > First of all, get Avocado installed. Besides the Avocado test runner > itself, this will give you the basic libraries on which the other part > of this work was built on. We want that to be simple and painless, so > here's our best bet for a one-liner installation: > > pip install --user avocado-framework > avocado-framework-plugin-varianter-yaml-to-mux aexpect > > That will install Avocado within the user's home directory. If you > give up on it, it can be uninstalled with another simple one-liner: > > pip uninstall -y avocado-framework > avocado-framework-plugin-varianter-yaml-to-mux aexpect > > Now, suppose you're working on a given feature, and want to try your > luck writing a test using this work. To avoid having you fetching and > rebasing from our currently in development fork[6] and branch[7], you > can just > add one commit to your tree with: > > curl > https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch | > git am - > > This will get a simple patch from a snapshot branch[8]. You can, of course, > do it "the git way", fetching from that repo[6] and using the > non-snapshotted branch. > > After that, we'd love for you to take a look at some of the existing > tests[9][10] and then attempt to create test for your own use case. > The basic README[11] file, and the Avocado documentation[12] are also > important resources not to be missed. > > What's next? > ============ > > Initially, feedback is what we're looking for. It would be greatly > appreciated to understand if/how this suits (or not) use cases out > there. > > After feedback, further refinements, and more tests are written, the > Avocado developers will follow up with an initial patch series for > upstream QEMU. In such a proposal, we intend to have further > integration. Ideally in way that "configure" can be given a > "--with-functional-[avocado-]tests" parameter of sorts, and a "make > [functional-]check" would seamlessly include them.
I have a few thoughts. We use pytest/pexpect internally to kick off QEMU runs and monitor the output (no interaction with the QEMU source tree) and I think it is extremely useful. So I am all for using Python to test things and this looks really well done! What I don't understand though is what this gives us compared to the existing QEMU test infrastructure? Besides being able to use Python and a better interface what are the main benefits? I think that is something worth documenting somewhere. Also, it looks like this will require images checked into git somewhere is that correct? Is there a good plan on how to handle that? Alistair > > Thanks! > > References > ========== > > [1] http://avocado-framework.github.io/ > [2] http://autotest.github.io/ > [3] https://github.com/autotest/virt-test > [4] https://github.com/avocado-framework/avocado-vt > [5] https://github.com/autotest/tp-qemu > [6] https://github.com/apahim/qemu > [7] https://github.com/apahim/qemu/tree/avocado_qemu > [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot > [9] > https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py > [10] > https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py > [11] > https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst > [12] http://avocado-framework.readthedocs.io/ > > -- > Cleber Rosa > [ Sr Software Engineer - Virtualization Team - Red Hat ] > [ Avocado Test Framework - avocado-framework.github.io ] > [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >