On 12/27/2011 11:37 PM, Anthony Liguori wrote:
On 12/27/2011 04:35 PM, Cleber Rosa wrote:
On 12/26/2011 08:00 PM, Dor Laor wrote:
On 12/26/2011 05:12 PM, Anthony Liguori wrote:
Hi Dor,
Merry Christmas Anthony,
On 12/25/2011 09:19 AM, Dor Laor wrote:
On 12/19/2011 07:13 PM, Anthony Liguori wrote:
Well, I'm still not convinced that a new standalone package should
handle these
cases instead of kvm autotest. I'll be happy to integrate the
tests to
kvm
autotest anyway and the more the merrier but imho it's a duplicate.
I'm sure kvm autotest could be taught to do exactly what qemu-test is
doing. But why does kvm autotest have to do everything? I doubt there
would be much code reuse.
I think it's not a bad thing to have multiple test suites when there
isn't considerable overlap.
I think the main goal of qemu-tests (may be implicit) is to be quick
and simple.
qemu-test doesn't have a main goal. My goal is to improve QEMU's
quality. qemu-test is just a tool to help achieve that goal.
Maybe I've used the wrong wording. Besides the different approach
(simpler requirements, initramfs with busybox, etc), it looks to me that
keeping it simple played an important role in your decision to write
qemu-tests, which is indeed a sign of good design. But read on...
That is indeed great, but if one thinks that's all we'll ever going
to need,
that thought is pretty naive.
I don't know who "we" is, but I can tell you that qemu-test is exactly
what *I* need. Consider that I spent a good portion of every single
day testing QEMU with either my own or other people's patches, making
that job easier and more automated is fairly important to me.
"We" is everyone that contributes one way or another to qemu. So if you
only concerned with your needs, you're definitely on the right track.
If not, I think (trying to think as a community) it'd be beneficial to
concentrate all efforts, unless it's not possible to do so. It's as
simple as that. All my reasoning here revolves around this.
I'm sharing it because I suspect that a lot of other developers have a
similar need.
And it may be true that there's room for both test
suites... or that, as busy developers, we're refusing to deal with
the added
complexity (qemu alone accounts for a lot) and delaying to fix the
fixable. I
believe on the later.
One example: kvm-autotest has a complex configuration file format
with a steep
learning curve, while a test such as qemu-tests/tests/simple-ping.sh
would have
to be tweaked if somehow the kernel detects the first ethernet
interface as em1
(such as recent Fedora systems do). Simple, but not scalable.
I can tell by this comment that you don't actually understand how
qemu-test works. Please take a look at it before jumping to
conclusions about whether it should or shouldn't be part of kvm-autotest.
I can tell you did not grasp my point: kvm autotest is more complex, but
more capable and flexible. And I did *not* come to a conclusion, I'm
giving examples in an attempt to enrich the discussion.
Hint: qemu-test always uses the same kernel because it builds it as
part of the test suite. The behavior of how a nic test will never
change unless someone explicitly changes the kernel.
Hopefully you understand by now that I'm giving some reasons of why
kvm-autotest does some things the way it does (usually more complex),
and how qemu-tests, because of its approach, does not have to deal with
that.
1) It builds a custom kernel and initramfs based on busybox. This is
fairly important to ensure that we can run tests with no device
pre-requisites.
This can be done easily w/ autotest too.
The Python requirement inside the guest is true *if* we want to run
regular
autotest tests inside the guest (see
autotest/client/virt/tests/autotest.py) and
this accounts for very very little of kvm autotest usage. All other
tests
interact with the monitor directly and with the guest via
ssh/telnet/serial.
qemu-test does not require any specific hardware to be used in the
guest which lets it test a wider variety of scenarios in QEMU. So you
cannot assume there is ssh/telnet/serial available.
I was assuming that we could count on at least a serial port in the
guest. If not, then current kvm-autotest can not absorb the same
functionality of qemu-tests without re-writing it. A
So, I see no reason for not using a more expressive language,
I seriously doubt you can build a useful initramfs that contains
python without doing something crazy like livecd-tools does....
Actually, kvm-autotest has various layers of abstraction in how QEMU
ends up being launched. As you mention below, those layers are
there to
allow for things like using libvirt.
Indeed the qemu command line parameters gets generated depending on many
configuration parameters. It'd be *really* simple to add a configuration
parameters that overwrites the qemu command with an static one.
But if you're a QEMU developer, you want to have as much control of
the command line as possible. For instance, one of the tests in
qemu-test makes sure to test invocations without -device as this
triggers a different code path (there was a recent regression in this
too). You can't just add arguments to reproduce this behavior.
It goes beyond that, since it also related to the monitor interface
as well.
That's desirable when you're doing "virt testing", but not so
desirably
when you're trying to write specific unit tests against QEMU.
True, one may not need it at all but it's nice that a test for
migration/stress/hotplug will be tested directly w/ qemu and libvirt
w/ the
same effort.
5) The tests execute very quickly, can be run stand alone, and do
not
require root privileges
ditto for kvm auotest. It's possible to configure it w/o root too
which is not a
huge issue.
When I say, "run quickly", I mean, they execute very quickly.
/me too
$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
tests/virtio-serial.sh
real 0m4.385s
user 0m1.460s
sys 0m1.860s
That's impressive but it's more of a function of the guest being
used - if
instead of running a full Fedora install, you'll chose your busybox
image w/
-kernel/initrd you'll get a similar result.
I also think so. Maybe kvm-autotest would take a little more time
because of the
different approach we take when communicating with the guest, but I
bet it'd be
irrelevant.
I don't see any reason why everything needs to live in kvm-autotest...
but if you really feel that way, please provide patches that
demonstrate how this would work. We could argue indefinitely about
how things could work, it's much better to compare how things actually
do work :-)
I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of
requirements
and we'll all improve as a result.
Yes, I believe it'd be a nice exercise for all of us.
The only thing I ask is that we bear at least with some of the
complexity that
kvm-autotest inherently holds...
I think there's something of a knee jerk reaction here. The existence
of qemu-test does not take anything away from kvm-autotest. It's just
another tool in our arsenal to achieve our joint goal of making QEMU
(and KVM) higher quality.
autotest is made to invoke third party tests so the two tools can
co-exist in a complimentary way.
Regards,
Anthony Liguori
the last requests we had was to get rid of all
the complexity, while retaining all the other nice characteristics.
Pretty hard,
so I think we failed, or maybe only half-succeeded at it.
Cheers,
Dor
Regards,
Anthony Liguori
6) They are random by nature with the ability to fix the seed in
order
to be used in git-bisect.
I think Gerd had been looking at doing something similar with a
custom
initrd.
I've tried to consider other architectures and had hoped that we
could
commit the vmlinuz and initramfs into git so that it was easy to
test
other architectures without having a full build environment.
Unfortunately, busybox doesn't link statically with glibc and I
can't
see an obvious way to commit binaries while respecting the GPL
since we
need to pull glibc into the initramfs.
I know buildroot exists specifically to deal with this but in my
experience, buildroot is very unreliable and extremely heavy weight
since it rebuilds gcc multiple times in order to bootstrap a ulibc
environment.
Anyway, the code is available at:
http://git.qemu.org/qemu-test.git
See the README for instructions on how to use it.
Regards,
Anthony Liguori