On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote: > On 03/08/2012 08:49 AM, Ademar Reis wrote: > >On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote: > >>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote: > >>>Virt/qemu tests: Minimal guest images > >>>------------------------------------- > >>> > >>>In order to make development level test possible, we need the tests to run > >>>fast. > >>>In order to do that, a set of minimal guest images is being developed and > >>>we > >>>have a version for x86_64 ready and functional: > >>> > >>>https://github.com/autotest/buildroot-autotest > >> > >>I'm really not a fan of buildroot. Note that in order to ship > >>binaries, full source needs to be provided in order to comply with > >>the GPL. The FSF at least states that referring to another website > >>for source that's not under your control doesn't satisfy the > >>requirements of the GPL. > >> > >>Just out of curiosity, did you try to use qemu-test? Is there a > >>reason you created something different? > >> > >>I think it's good that you're thinking about how to make writing > >>tests easier, but we have a growing test infrastructure in QEMU and > >>that's what I'd prefer people focused on. > >> > > > >You probably remember the long thread we had back in December on > >qemu-devel on this topic. Back then our message was "we have a > >growing test infrastructure in s/QEMU/autotest/ and that's what > >we'd prefer people focused on". :-) > > > > From Dor: > > > >(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html) > > > >""" > >If you wish, you can challenge Lucas and Cleber w/ these type of > >requirements and we'll all improve as a result. > >""" > > > >Your response was: > > > >""" > >Well consider qemu-test the challenge. It's an existence proof > >that we can have a very easy to use framework for testing that > >runs extremely fast and is very easy to write tests for. > >""" > > > >http://knowyourmeme.com/memes/challenge-accepted ;-) > > > >I particularly agreed with basically everything you said on that > >discussion regarding test simplification (I had just joined the > >team back then). To me, autotest has been focusing on QE-level, > >leaving the developer-level test requirements out. Now we're > >attacking this new front, and a lot of the requirements are > >indeed from that discussion. > > If you want to talk about this in terms of "requirements", my > requirement is for "developer-level" tests to live in qemu.git and > be integrated into make check. > > Just as we've been discussing and working on since the previous set of > discussions. > > >By simplifying the design and bringing barriers down, we hope to > >reach a broader audience and help developers write and maintain > >tests, benefiting from all the instrumentation that autotest > >brings. It's not going to be just about qemu (check the new test > >examples). > > > >We have a team fully dedicated to autotest and it's used not only > >by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc > >(these all have code contributions in autotest) > > > >That said, the current qemu-tests will probably be easily > >integrated into (the new) autotest and we hope that, given enough > >time, autotest will be good enough to relieve qemu from the > >framework maintenance and code duplication with other projects. > > autotest should not be the focal point for integration. qemu.git should be. > > I'd be perfectly happy to review patches submitting the test > infrastructure from kvm-autotest into qemu.git (provided it didn't > have unreasonable external dependencies and fit into QEMU). > > Developer-level tests need to live where the developers live. The > developers live in qemu.git. See my other response on this thread > for the explanation of why this is so important. >
Excelent, we're in the same page then. This was my number 1 requirement when I was discussing the changes with Lucas and Cleber. For convenience, I'll repeat here what I wrote in a previous e-mail (no qemu-devel archive available yet to use as a reference). In summary, autotest is (or is going to be) a framework that provides: - A test runner, with grid/cluster support and advanced instrumentation - A devel library and set of utilities for test writers - A set of pre-built images (JeOS – Just Enough OS) for test writers (attached is a picture showing what we want to achieve) If a project has an internal library or set of utilities that can be of general use, they can be submitted to autotest.git for inclusion, thus reaching a broader audience. A short summary of the plans: - Tests can live anywhere and each devel team implements and maintains their own set of tests - Usage of the autotest library by test writers is optional - Tests are scripts returning 0 or error (any language) - Tests can be run individually or in sets - Tests should run fast, our target is seconds or a few minutes - The test runner is smart and “just works” by default - Trivial standard output (FAIL, PASS, SKIPPED) - Collect logs, OS data and other stuff (e.g. --record-video!) - Skip unsupported tests based on the environment they're run - Multiplex configurations / platforms when on the grid - Support to run tests “in the cloud” There are also lots of small changes and usability improvements in the pipeline (and feedback right now is very valuable). Thanks, - Ademar -- Ademar de Souza Reis Jr. Red Hat ^[:wq!