On Thu, Jan 18, 2018 at 1:25 AM, Lukáš Doktor <ldok...@redhat.com> wrote: > Dne 18.1.2018 v 00:41 Alistair Francis napsal(a): >> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <cr...@redhat.com> wrote: >>> TL;DR >>> ===== >>> >>> This is about how QEMU developers can get started with functional >>> tests that are built on top of the Avocado libraries (and meant to be >>> run with the Avocado test runner). >>> >>> The past >>> ======== >>> >>> The Avocado project[1] has been working, for quite some time now, on a >>> "set of tools and libraries" with the goal of making writing tests >>> easier. It is supposed to be a framework agnostic to the exact >>> software that will be under test. >>> >>> But, at the same time, the Avocado project cannot deny its inheritance >>> and influences. Those come from Autotest[2], which had "KVM Autotest" >>> as its largest and most developed "test". This large Autotest test >>> (KVM Autotest) became virt-test[3] and later got integrated into >>> Avocado and became Avocado-VT[4] which is quite relevant here, >>> together with its QEMU test provider[5]. >>> >>> Avocado-VT and the QEMU test provider attempt to provide coverage >>> across platform and QEMU versions, which increases its complexity. >>> Also, it's built on a legacy set of principles and tools that makes >>> some developers stir away from it. >>> >>> What's new? >>> =========== >>> >>> A few months ago, the Avocado developers returned to its >>> "virtualization origins", in an attempt to learn from the QEMU >>> project, and try to help with a way to have more functional tests in >>> the upstream QEMU repo. >>> >>> We believe it's possible to expand the test coverage for QEMU by >>> facilitating >>> the creation of more functional tests QEMU. This is no different than how >>> other types of tests are already included in the tree itself. >>> >>> How >>> === >>> >>> How we did it (so far) >>> ---------------------- >>> >>> We're aware that there's a dilemma here: to be able to easily write >>> more powerful tests, a lot of the complexity has to be moved >>> elsewhere. Here, it means moving complexity from the test itself to a >>> framework. The QEMU source tree itself has proofs of this approach, >>> being the "scripts" and "tests/qemu-iotests" some of the examples. >>> >>> Avocado itself[1] provides a lot of the code that should help to >>> absorb some of the complexities in writing tests, but not exactly >>> everything that is needed for QEMU. The approach we believe will have >>> the best balance is to reuse upstream Avocado libraries whenever they >>> are useful and generic enough, and on top of that, libraries that are >>> part of QEMU itself. >>> >>> How can you get started with it >>> ------------------------------- >>> >>> First of all, get Avocado installed. Besides the Avocado test runner >>> itself, this will give you the basic libraries on which the other part >>> of this work was built on. We want that to be simple and painless, so >>> here's our best bet for a one-liner installation: >>> >>> pip install --user avocado-framework >>> avocado-framework-plugin-varianter-yaml-to-mux aexpect >>> >>> That will install Avocado within the user's home directory. If you >>> give up on it, it can be uninstalled with another simple one-liner: >>> >>> pip uninstall -y avocado-framework >>> avocado-framework-plugin-varianter-yaml-to-mux aexpect >>> >>> Now, suppose you're working on a given feature, and want to try your >>> luck writing a test using this work. To avoid having you fetching and >>> rebasing from our currently in development fork[6] and branch[7], you >>> can just >>> add one commit to your tree with: >>> >>> curl >>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch | >>> git am - >>> >>> This will get a simple patch from a snapshot branch[8]. You can, of course, >>> do it "the git way", fetching from that repo[6] and using the >>> non-snapshotted branch. >>> >>> After that, we'd love for you to take a look at some of the existing >>> tests[9][10] and then attempt to create test for your own use case. >>> The basic README[11] file, and the Avocado documentation[12] are also >>> important resources not to be missed. >>> >>> What's next? >>> ============ >>> >>> Initially, feedback is what we're looking for. It would be greatly >>> appreciated to understand if/how this suits (or not) use cases out >>> there. >>> >>> After feedback, further refinements, and more tests are written, the >>> Avocado developers will follow up with an initial patch series for >>> upstream QEMU. In such a proposal, we intend to have further >>> integration. Ideally in way that "configure" can be given a >>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make >>> [functional-]check" would seamlessly include them. >> >> I have a few thoughts. >> >> We use pytest/pexpect internally to kick off QEMU runs and monitor the >> output (no interaction with the QEMU source tree) and I think it is >> extremely useful. So I am all for using Python to test things and this >> looks really well done! >> >> What I don't understand though is what this gives us compared to the >> existing QEMU test infrastructure? Besides being able to use Python >> and a better interface what are the main benefits? I think that is >> something worth documenting somewhere. >> > > Hello Alistar, > > It was just briefly mentioned by Cleber in the main email, but to be more > detailed there are 2 (3) benefits: > > 1. Using a test runner avoids boiler-plate code to create and provide > results. Avocado is a test runner and can unify the execution as well as > results. Avocado results are compatible with multiple results DBs/systems and > even allows diffing (you can see what changed between different executions). > > 2. Avocado is not just a test runner, but also set of utils. This can > simplify writing tests as one does not have to reinvent the wheel and can > simply say "I require this pkg, install it", "I want to start this service" > or run commands interactively without the need to worry about setting pty, > reading-out the pipes etc. > > 3. There are at least three Red Hat employees based in virt team who work on > Avocado development. This means there is a background to support you the > community to simplify writing tests (avocado_qemu "helpers" are one example). > In return we expect less regressions as more testing can be performed easily > before releases/merges/during development.
Cool! Sorry for the delay here. If the test doesn't have to integrate with QEMU then does it make sense to have a separate repo? The way we handle this kind of testing is we just point to a built QEMU tree and py.test runs tests on all of the binaries. That way they can be developed independently, which has advantages and disadvantages. I don't have an answer, I was just wondering if you have thought about that. Alistair > > Regards, > Lukáš > > >> Also, it looks like this will require images checked into git >> somewhere is that correct? Is there a good plan on how to handle that? >> >> Alistair >> >>> >>> Thanks! >>> >>> References >>> ========== >>> >>> [1] http://avocado-framework.github.io/ >>> [2] http://autotest.github.io/ >>> [3] https://github.com/autotest/virt-test >>> [4] https://github.com/avocado-framework/avocado-vt >>> [5] https://github.com/autotest/tp-qemu >>> [6] https://github.com/apahim/qemu >>> [7] https://github.com/apahim/qemu/tree/avocado_qemu >>> [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot >>> [9] >>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py >>> [10] >>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py >>> [11] >>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst >>> [12] http://avocado-framework.readthedocs.io/ >>> >>> -- >>> Cleber Rosa >>> [ Sr Software Engineer - Virtualization Team - Red Hat ] >>> [ Avocado Test Framework - avocado-framework.github.io ] >>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >>> > >
