On 12/26/2011 08:00 PM, Dor Laor wrote:
On 12/26/2011 05:12 PM, Anthony Liguori wrote:
Hi Dor,
Merry Christmas Anthony,
On 12/25/2011 09:19 AM, Dor Laor wrote:
On 12/19/2011 07:13 PM, Anthony Liguori wrote:
Well, I'm still not convinced that a new standalone package should
handle these
cases instead of kvm autotest. I'll be happy to integrate the tests to
kvm
autotest anyway and the more the merrier but imho it's a duplicate.
I'm sure kvm autotest could be taught to do exactly what qemu-test is
doing. But why does kvm autotest have to do everything? I doubt there
would be much code reuse.
I think it's not a bad thing to have multiple test suites when there
isn't considerable overlap.
I think the main goal of qemu-tests (may be implicit) is to be quick and
simple. That is indeed great, but if one thinks that's all we'll ever
going to need, that thought is pretty naive. And it may be true that
there's room for both test suites... or that, as busy developers, we're
refusing to deal with the added complexity (qemu alone accounts for a
lot) and delaying to fix the fixable. I believe on the later.
One example: kvm-autotest has a complex configuration file format with a
steep learning curve, while a test such as
qemu-tests/tests/simple-ping.sh would have to be tweaked if somehow the
kernel detects the first ethernet interface as em1 (such as recent
Fedora systems do). Simple, but not scalable.
Other example: kvm-autotest can easily implement (copy?) the
functionality to build a busybox initramfs. We already have code to
build from sources such as git, local source directories, remote
tarballs, you named it. And the recipe to build a given package is also
configurable/extensible so we could even leverage that. qemu-tests, on
the other hand, would need lots of code to test the same range of guests
that kvm-autotest tests. Simple, but limited.
So, it looks like, if we add the missing features to both test suites,
we'll end up with two implementations, using different
languages/approaches, but with very little added value.
I agree but in this case, it loos to me that qemu-test is likely to do
a subset of what kvm autotest do in a potentially less elegant way but
due to the influence of a key maintainer may gain traction and dilute
kvm autotest..
It has the following characteristics:
1) It builds a custom kernel and initramfs based on busybox. This is
fairly important to ensure that we can run tests with no device
pre-requisites.
This can be done easily w/ autotest too.
The Python requirement inside the guest is true *if* we want to run
regular autotest tests inside the guest (see
autotest/client/virt/tests/autotest.py) and this accounts for very very
little of kvm autotest usage. All other tests interact with the monitor
directly and with the guest via ssh/telnet/serial.
So, I see no reason for not using a more expressive language, having an
API, etc, to do that's done in in_host(). In the guest side, what
changes is that we send one command at a time, checking/collecting
results, instead of pushing the complete tests files into the initramfs.
Okay, please demonstrate :-) The procedure to do this with qemu test is:
$ git clone git://git.qemu.org/qemu-test.git
$ cd qemu-test
$ git submodule update --init
$ make
How would one do this with kvm autotest?
git clone git://github.com/autotest/autotest.git
cd autotest/client/tests/kvm
sudo ./get_started.py
One command shorter and the above downloads F15 automatically and runs
it.
(from
https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)
2) Tests are scripts that are launched in the initramfs
3) The test controls exactly how QEMU is launched which allows easy
testing of various QEMU options
afaik kvm autotest do the same. It's true that kvm autotest might look
less
friendly but its may only be this way since it has lots of options.
Actually, kvm-autotest has various layers of abstraction in how QEMU
ends up being launched. As you mention below, those layers are there to
allow for things like using libvirt.
Indeed the qemu command line parameters gets generated depending on many
configuration parameters. It'd be *really* simple to add a configuration
parameters that overwrites the qemu command with an static one.
It goes beyond that, since it also related to the monitor interface as
well.
That's desirable when you're doing "virt testing", but not so desirably
when you're trying to write specific unit tests against QEMU.
True, one may not need it at all but it's nice that a test for
migration/stress/hotplug will be tested directly w/ qemu and libvirt
w/ the same effort.
5) The tests execute very quickly, can be run stand alone, and do not
require root privileges
ditto for kvm auotest. It's possible to configure it w/o root too
which is not a
huge issue.
When I say, "run quickly", I mean, they execute very quickly.
/me too
$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
tests/virtio-serial.sh
real 0m4.385s
user 0m1.460s
sys 0m1.860s
That's impressive but it's more of a function of the guest being used
- if instead of running a full Fedora install, you'll chose your
busybox image w/ -kernel/initrd you'll get a similar result.
I also think so. Maybe kvm-autotest would take a little more time
because of the different approach we take when communicating with the
guest, but I bet it'd be irrelevant.
I've used kvm-autotest a lot, there is no test in kvm-autotest that is
even close to completing in 4 seconds start to finish. I don't think
kvm-autotest can even run without installing a guest first which puts
puts a simple test more at the 30 minute mark. If we're talking about
TCG testing, then we're in hours territory.
Autotest today may be too verbose by default and will call
echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way
and Lucas can easily create a slim mode that does not even need to be
root.
The drop_caches thing has been configurable for ages.
If there's a way to run kvm-autotest as a non-privileged user, I've not
figured it out yet.
Again, there is no magic in a test suite so if required we can wave it
(maybe we need to do that regardless)
Some of the things we do in kvm autotest (and are not currently done in
qemu-tests) indeed need root privileges, but we could add granularity to
that.
Of course, kvm-autotest can test a lot more things than qemu-test can
since the tests are guest agnostic. That is the primary architectural
difference. By writing the tests to one specific guest (busybox initrd),
they can be extremely fast in execution, but the tests are less general
purpose.
That makes qemu-test very useful for development testing (such as long
automated bisect runs) but less useful for acceptance testing.
Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and
the tests will be slimmer or each test will have such a mode.
Please compare your own virtio-serial test w/ the autotest version
of it:
https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
This single file tests functionality, limits, console, live
migration and
performance. Of course one can add a very basic 'hello world' sanity
test too
that will run quickly and will identify basic breakage fast.
Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.
I bet that
virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be
less than 52 lines. It depends what you like to test. as noted, the
above test fully covers all aspects of virtio-serial. It does not have
to be this way and I do agree it is nicer to split it to multiple sub
files.
There is a lot of value in being able to write simple tests in 50 lines
of code. In fact, the largest qemu-test at the moment is only ~70 lines
of code. And besides finding regressions in my own code (which is the
primary use atm), I've found and fixed a few real bugs in upstream QEMU.
Now imagine what virtio_console.py buys you :)
For a couple hundred lines of bash script, how can it be anything but a
good thing :-)
Noways we abstract kvm autotest so libvirt will be optionally tested
too w/ the
same tests.
Again, I'm not advocating that kvm autotest is a solution for anything
but for
plain guest-host communication, monitor commands, etc it's a really
good tool.
I agree that kvm autotest may be less friendly for developer users
since it
carries allot of options for testing a huge matrix. Lucas and Cleber
are working
these days to add a make kvmautotest target to qemu so one would be
able to
quickly execute autotest and we can think of additional parameters like
sanity-set, migration-set, etc.
We (the QEMU project) need to get more serious about testing. We need to
Like
(quickly) get to a point where we can mandate that features come with
test cases. It is extremely hard to implement touch everything features
with no good way to test all of the different things we support.
kvm autotest cannot fill that role because it's too complicated and too
many layers removed from QEMU.
virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
about to mandate that someone writes 2.2k LOC in a python test framework
in order to get 1.4k of code merged in QEMU.
But 50 lines of bash seems like a more than reasonable requirement.
Hmm, what's the coverage of these 50 LOC?
Maybe when your maintainer hat is on, it is indeed a huge task to keep
all of qemu platforms/architectures to compile and run so that's
enough but when I wear my Red Hat manager role, I wish that it will
not only compile and boot but will test as much functionality as
possible and will be easily re-used by R&D, QA, libvirt, while all
supported guests are _potentially_ exercised, not just busybox.
Since kvm autotest will be there anyway, I wish that developers will
contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will
justify it.
I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of
requirements and we'll all improve as a result.
Yes, I believe it'd be a nice exercise for all of us.
The only thing I ask is that we bear at least with some of the
complexity that kvm-autotest inherently holds... the last requests we
had was to get rid of all the complexity, while retaining all the other
nice characteristics. Pretty hard, so I think we failed, or maybe only
half-succeeded at it.
Cheers,
Dor
Regards,
Anthony Liguori
6) They are random by nature with the ability to fix the seed in order
to be used in git-bisect.
I think Gerd had been looking at doing something similar with a custom
initrd.
I've tried to consider other architectures and had hoped that we could
commit the vmlinuz and initramfs into git so that it was easy to test
other architectures without having a full build environment.
Unfortunately, busybox doesn't link statically with glibc and I can't
see an obvious way to commit binaries while respecting the GPL
since we
need to pull glibc into the initramfs.
I know buildroot exists specifically to deal with this but in my
experience, buildroot is very unreliable and extremely heavy weight
since it rebuilds gcc multiple times in order to bootstrap a ulibc
environment.
Anyway, the code is available at:
http://git.qemu.org/qemu-test.git
See the README for instructions on how to use it.
Regards,
Anthony Liguori
