subject:"Re\: \[Qemu\-devel\] \[RFC\] Future goals for autotest and virtualization tests"


On 03/08/2012 08:21 PM, Anthony Liguori wrote:

On 03/08/2012 04:24 PM, Ademar Reis wrote:

On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:

On 03/08/2012 03:02 PM, Ademar Reis wrote:

On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:

On 03/08/2012 11:59 AM, Ademar Reis wrote:


snip




I expect QEMU to grow tests for anything that involves launching
QEMU directly.  Where I would not see QEMU growing tests for is
things like launching QEMU through libvirt.


Agree. For QEMU developers, libvirt should not be on the way, the
interaction should be minimal or non-existent.

That's an area which will require some work in libautotest,
because due to previous QE requirements, it now invokes libvirt
methods instead of QEMU directly for a lot of stuff. But it can
be done and is part of the plan.


If you're talking about libautotest as an abstraction layer for
launching QEMU, I don't think that's something we should use in
upstream QEMU.  Abstraction layers are okay when the things you are
abstracting are well defined and relatively static.  We're talking
about testing the tip of a development tree though and coordinating
changes with an abstraction layer would be counter productive.



Valid point. If I'm a qemu developer and I want to make huge
changes, ideally I shouldn't have to submit patches to
libautotest because the tests in qemu.org will break.

I don't know how likely such scenario is (Lucas is maitaining
kvm-autotest for years now and the integration hardly breaks)
But we do have a conflict of interests, so lets evaluate a
few options:


Because kvm-autotest doesn't have thousands of tests and isn't part
of 200+ developers fast paths :-)



My main interests are:
 - remove barriers for developers to write tests


This is my number one priority.


 - share the testing effort between multiple projects (qemu,
   libvirt, ovirt, spice, QE, etc...)


This is my number 99 priority.


You have the right to consider the interests of qemu only, but
qemu is used by several other teams who depend on it to implement
their own tests. Some collaboration would be extremely beneficial
for everybody.

Collaboration goes both ways: you get benefits from the
interaction and code reuse with different projects and teams, but
it may require a few tradeoffs.


I think the problem here effectively boils down to code sharing at 
any cost.


Let's look at it in purely practical terms.  How many lines of code 
would be saved here and what does that code do?


If the only sharing is some infrastructure bits in libautotest, then I 
suspect we're talking in the low 10ks of lines at the absolute maximum 
(and I don't think it's even close to that in reality).  Launching 
QEMU and connecting to QMP isn't that hard.  It's in the hundreds of 
lines of code.  What else is there that's need to write a test?


I don't think this is sufficiently compelling to add an unconditional 
autotest dependency to QEMU.



I think you could achieve practical code sharing by decomposing
kvm-autotest into a set of useful, independent tools.  For
instance


Now the options:

1. We keep the invocation stuff in libautotest and when writting
a complex test in qemu.git, the test writter can choose to use it
because of the goodies there.

Pros:
   - Lots of codepaths implemented both in python and as cmd-line
 utilities: less lines of code to write a test, smaller
 barrier for the developer.


I've got an existence proof that this is not true.  The qemu-test
tests are smaller than the corresponding autotest tests.


You're comparing developer-level tests with the existent QA-level
tests (much more complex).


Let's be specific then.  Look at device-add.sh in qemu-test.  It's 
71LOC. pci_hotplug.py in autotest is 204LOC.


pci_hotplug.py does much more than device-add.sh:

   * tests both pci_add and device_add commands
   * checks the monitor syntax for adding a new drive, that is, it 
works on HEAD and on other versions (such as the ones in some RHEL 
releases);

   * tests both nic and block hotplug
   * for block, tests with both virtio and scsi
   * also does device removal, both for pci_add and device_add syntaxes



What's the much more complex part of pci_hotplug.py that is enabled 
by having the autotest framework?



We're talking about an approach where you'll need just a few
lines of code to instantiate a VM and interact with it.


This isn't that hard of a problem.  It just takes a few lines to 
launch QEMU in qemu-test too.



   - Mature code in the test library.
   - Goodies such as video-recording and system info collection
 in a standard way for different projects.


Why does video-recording have to be part of libautotest?  Why can't
you just have a simply utility that connects to a VNC session and
records it?  Presumably that's all it's doing here.

Likewise, there are dozens of system info collection such as
sosreport..  why fold this all into libautotest?


For standardization:

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 05:14 AM, Kevin Wolf wrote:

Am 09.03.2012 00:51, schrieb Ademar Reis:

On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:

Plus it's not unconditional: the test runner will report tests
SKIPPED if a dependency is not present.


But then the tests aren't run so if most developers didn't have it
installed, and most tests were written with it, most developers
wouldn't be running most tests which defeats the purpose, no?


Part of a TDD approach is to have build and test bots frequently
running tests on multiple platforms with different
configurations.

You can't expect developers to run all tests all the time.


I think this is one of the most important points: Not all developers
must run all the tests all the time.

Actually, Anthony agreed with me when I said that developers should run
some sanity tests for all of qemu and maybe a few more tests for the
subsystems they're touching.


And a small number of randomly chosen test cases.

We don't want to have test cases that never run under normal circumstances or 
else they're prone to break.  That's why I've talked a lot about keeping 'make 
check' time bound.



I agree that it would be bad to have a
autotest dependency for those basic tests that everyone should run and
that should take a few minutes at most.

For the rest of test cases, however, not everyone needs to run them and
I think an external dependency (that is reasonably easy to satisfy) is
not a problem there.


I'd prefer to avoid external dependencies as it just encourages people not to 
test.  There may be exceptions for certain types of tests, but it should be an 
exception, not the rule.



A test bot would be great, but even if people just
run them occasionally by hand, that would already detect bugs that are
currently left in the code for months. If maintainers do it before
pushing code into master, you'll even catch everything before it goes
into master. This is as good as it gets.


We'll integrate make check into buildbot which currently does look at 
submaintainer trees.


Regards,

Anthony Liguori



The important thing is that tests exist in the first place, not who runs
them.

Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 05:20 AM, Cleber Rosa wrote:

You're comparing developer-level tests with the existent QA-level
tests (much more complex).


Let's be specific then. Look at device-add.sh in qemu-test. It's 71LOC.
pci_hotplug.py in autotest is 204LOC.


pci_hotplug.py does much more than device-add.sh:

* tests both pci_add and device_add commands
* checks the monitor syntax for adding a new drive, that is, it works on HEAD
and on other versions (such as the ones in some RHEL releases);
* tests both nic and block hotplug
* for block, tests with both virtio and scsi
* also does device removal, both for pci_add and device_add syntaxes


Ok, but clearly, there's no magic in autotest that makes this sufficiently 
easier.  It's just a matter of:


cmd=`named_choose command device_add pci_add`

if test $cmd = device_add; then
  qmp device_add --driver=virtio-blk-pci --drive=hd0
else
  hmp pci_add auto virtio-blk-pci,drive=hd0
fi

It's not there today because pci_add is deprecated.  There assertion was that 
autotest makes it easier to write tests.  How does it make it easier to write 
pci_hotplug?


Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Kevin Wolf

Am 09.03.2012 12:59, schrieb Anthony Liguori:
 On 03/09/2012 05:14 AM, Kevin Wolf wrote:
 Am 09.03.2012 00:51, schrieb Ademar Reis:
 On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
 Plus it's not unconditional: the test runner will report tests
 SKIPPED if a dependency is not present.

 But then the tests aren't run so if most developers didn't have it
 installed, and most tests were written with it, most developers
 wouldn't be running most tests which defeats the purpose, no?

 Part of a TDD approach is to have build and test bots frequently
 running tests on multiple platforms with different
 configurations.

 You can't expect developers to run all tests all the time.

 I think this is one of the most important points: Not all developers
 must run all the tests all the time.

 Actually, Anthony agreed with me when I said that developers should run
 some sanity tests for all of qemu and maybe a few more tests for the
 subsystems they're touching.
 
 And a small number of randomly chosen test cases.
 
 We don't want to have test cases that never run under normal circumstances or 
 else they're prone to break.  That's why I've talked a lot about keeping 
 'make 
 check' time bound.

This sounds horrible. make check results must be reproducible, not
depend on a randomly chosen set. If you do it like this, a passed make
check means exactly _nothing_.

 I agree that it would be bad to have a
 autotest dependency for those basic tests that everyone should run and
 that should take a few minutes at most.

 For the rest of test cases, however, not everyone needs to run them and
 I think an external dependency (that is reasonably easy to satisfy) is
 not a problem there.
 
 I'd prefer to avoid external dependencies as it just encourages people not to 
 test.  There may be exceptions for certain types of tests, but it should be 
 an 
 exception, not the rule.
 
 A test bot would be great, but even if people just
 run them occasionally by hand, that would already detect bugs that are
 currently left in the code for months. If maintainers do it before
 pushing code into master, you'll even catch everything before it goes
 into master. This is as good as it gets.
 
 We'll integrate make check into buildbot which currently does look at 
 submaintainer trees.

But make check will never be the full thing. And if you want to
integrate make check into automated testing, then choosing a random
subset of tests for each is an even worse idea than it is anyway.

I believe the only sane thing to do is to distinguish between quick
sanity tests that are run by make check, and a full test suite that is
not run by every developer, but ideally by some test bots.

Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:

On 03/08/2012 06:24 PM, Anthony Liguori wrote:

Cons:
- Lot of code will be duplicated to cover the main code paths:
writting tests will require writting/supporting considerable
ammount of code (that already exists in autotest).

Again, existence proof that this isn't true.

Case in point, the virtio test (that uses an auxiliary script to send data to
the host). Can you tell me if both tests cover even remotely the same amount of
functionality?

https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py

https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py

Here is the qemu-test version

http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

So virtio-serial is an exception in autotest:

2174 virtio_console.py
1875 cgroup.py
615 ksm_overcommit.py
439 qemu_img.py
407 qmp_basic.py
389 qmp_basic_rhel6.py
356 stepmaker.py
247 steps.py

// The next largest actual test of QEMU
203 pci_hotplug.py
190 cdrom.py
182 physical_resources_check.py
181 timedrift.py
170 enospc.py
150 balloon_check.py
138 multi_disk.py
121 unittest.py
117 migration.py
111 cpu_hotplug.py
107 migration_multi_host.py
104 nic_hotplug.py
103 timedrift_with_stop.py
96 timedrift_with_migration.py
...

So picking virtio-serial as your comparison point is not really representative
of kvm-autotest but at any rate...

What the qemu-test version covers:
* host starts qemu with one virtio console device backed by a file
* guest verifies if the name of the device is correct
* guest writes to the console device
* host verifies if guest wrote to the virtio console

What the virtio-console covers:

* Sends data between host and guest back and forth, validates the data being
sent, for both small and large amounts of data, both random or sequential.

This is an improvement. qemu-test has shared random seed so adding a mode like
this wouldn't be hard.

* Tests write/send in blocking, polling, selecting mode, with port mode
sync/async

This is a Linux regression test, not a QEMU test.

* Verifies if the maximum amount of ports was created and it's available in the
guest

This is an improvement and would be trivial to add.

* Tries lseek on the device

This is a Linux regression test.

* Verifies if concomitant access to a single port passes or fails

I think this is also a Linux test.

* Verifies data throughput and resource utilization

This is outside the scope of qemu-test.

* Repeats the data transfer with live migration to see if the port connections
survive

This is neat and an improvement. I haven't attempted to do migration testing
with qemu-test.

* Keeps an eye on the guest OS to see if we have kernel panics, and if it does,
it'll clean up and put the guest in a working state so other functionality can
be checked

This is a Linux test.

* Probably something else I'm forgetting right now.

Good luck implementing that in a shell script. I'd love to see how you implement
that amount of coverage in less lines in shell.

So, more coverage, more code. It's as simple as that. We don't write code just
for the sake of it.

I hope it's clear though that most of what you're doing here is an integration
test with Linux. This is exactly what autotest should be doing IMHO and exactly
not what we should be doing in qemu.git.

I really see this all as over architecting to be honest.

Can someone explain in clear terms (without appealing to maturity,
flexibility, or any other qualitative aspect) why it takes anything more
than a few dozen shell functions to write all of the tests that are in
kvm-autotest test today?

Clearly as your requirements are different than the ones we had when the project
was written, qemu-test doesn't need any of the stuff present there and you can
do it all with a handful of shell script functions.

But to do cover the same things covered in autotest today with the same
requirements, I really doubt you can do the same with the same handful of shell
script functions. See the example about the virtio test above, not to mention
other functionality we have with the tests, such as make possible to do tests
run with a migration thread going on the background, while it's
plugging/unplugging devices, running a stress test or ltp, or a benchmark, or
measuring the time drift experienced by the guest.

That's fine. Your requirements are different and that's why you need autotest.
But we don't have the same requirements in qemu.git.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 06:13 AM, Kevin Wolf wrote:

Am 09.03.2012 12:59, schrieb Anthony Liguori:

On 03/09/2012 05:14 AM, Kevin Wolf wrote:

Am 09.03.2012 00:51, schrieb Ademar Reis:

On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:

Plus it's not unconditional: the test runner will report tests
SKIPPED if a dependency is not present.


But then the tests aren't run so if most developers didn't have it
installed, and most tests were written with it, most developers
wouldn't be running most tests which defeats the purpose, no?


Part of a TDD approach is to have build and test bots frequently
running tests on multiple platforms with different
configurations.

You can't expect developers to run all tests all the time.


I think this is one of the most important points: Not all developers
must run all the tests all the time.

Actually, Anthony agreed with me when I said that developers should run
some sanity tests for all of qemu and maybe a few more tests for the
subsystems they're touching.


And a small number of randomly chosen test cases.

We don't want to have test cases that never run under normal circumstances or
else they're prone to break.  That's why I've talked a lot about keeping 'make
check' time bound.


This sounds horrible. make check results must be reproducible,


It is reproducible by fixing the random seed.  When you run qemu-test, you get 
output like:


$ ./qemu-test qemu-system-x86_64 tests/device-add.sh
Using RANDOM seed 56782
Formatting '.tmp-2398/disk.img', fmt=qcow2 size=10737418240 encryption=off 
cluster_size=65536

ls: cannot access .tmp-2398/logfile-2398.log: No such file or directory
/home/anthony/build/qemu/x86_64-softmmu/qemu-system-x86_64 -kernel 
bin/vmlinuz-3.0 -initrd .tmp-2398/initramfs-2398.img.gz -append console=ttyS0 
seed=56782 -nographic -enable-kvm -device virtio-balloon-pci,id=balloon0 
-pidfile .tmp-2398/pidfile-2398.pid -qmp 
unix:.tmp-2398/qmpsock-2398.sock,server,nowait

test: 70: =: unexpected operator
[0.00] Initializing cgroup subsys cpuset
[0.00] Initializing cgroup subsys cpu
[0.00] Linux version 3.0.0 (anthony@titi) (gcc version 4.5.2 
(Ubuntu/Linaro 4.5.2-8ubuntu4) ) #7 SMP Mon Dec 19 15:54:15 CST 2011


To run the exact same test again, you would do:

$ QEMU_TEST_SEED=56782 ./qemu-test qemu-system-x86_64 tests/devices-add.sh

This lets you design high coverage tests without having to mess around with 
changing configuration files.  In the latest qemu-test, you can even fix your 
random choices if you want to configure a very specific test.



not
depend on a randomly chosen set. If you do it like this, a passed make
check means exactly _nothing_.


You can have it both ways.  You can fix choices in order to have a deterministic 
test of a specific thing or you can allow choices to be made randomly (with 
weighting).


So you can make make check validate very precise things but chances are, there's 
a class of things that you would like people to test but maybe not every time 
make check runs.  Having a contingent of occasionally executed tests means 
you'll get better coverage for less commonly used features.



A test bot would be great, but even if people just
run them occasionally by hand, that would already detect bugs that are
currently left in the code for months. If maintainers do it before
pushing code into master, you'll even catch everything before it goes
into master. This is as good as it gets.


We'll integrate make check into buildbot which currently does look at
submaintainer trees.


But make check will never be the full thing. And if you want to
integrate make check into automated testing, then choosing a random
subset of tests for each is an even worse idea than it is anyway.


make check-full.


I believe the only sane thing to do is to distinguish between quick
sanity tests that are run by make check, and a full test suite that is
not run by every developer, but ideally by some test bots.


Yeah, I don't think we're disagreeing at all.

Regards,

Anthony Liguori



Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 09:04 AM, Anthony Liguori wrote:

On 03/09/2012 05:20 AM, Cleber Rosa wrote:

You're comparing developer-level tests with the existent QA-level
tests (much more complex).


Let's be specific then. Look at device-add.sh in qemu-test. It's 71LOC.
pci_hotplug.py in autotest is 204LOC.


pci_hotplug.py does much more than device-add.sh:

* tests both pci_add and device_add commands
* checks the monitor syntax for adding a new drive, that is, it works 
on HEAD

and on other versions (such as the ones in some RHEL releases);
* tests both nic and block hotplug
* for block, tests with both virtio and scsi
* also does device removal, both for pci_add and device_add syntaxes


Ok, but clearly, there's no magic in autotest that makes this 
sufficiently easier.  It's just a matter of:


cmd=`named_choose command device_add pci_add`

if test $cmd = device_add; then
  qmp device_add --driver=virtio-blk-pci --drive=hd0
else
  hmp pci_add auto virtio-blk-pci,drive=hd0
fi

It's not there today because pci_add is deprecated.  There assertion 
was that autotest makes it easier to write tests.  How does it make it 
easier to write pci_hotplug?




Sure. I agree that it's fair from QEMU's PoV alone to forget about 
legacy things such as pci_add.


On this particular example, the one thing that strikes me the most is 
that (kvm-)autotest allows either a very static test run (as 
device_add.sh does), or a very configurable test run (as pci_hotplug.py 
does and QE needs).


I believe that with little compromises (required by any kind of 
collaboration) we can have the best 
tests/tools/libraries/framework/whatever by combining these similar code 
bases.


Again, we're willing to work and try to make that happen, but IMHO there 
must be a more open attitude here.


Cheers,
CR.


Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 06:40 AM, Cleber Rosa wrote:

On 03/09/2012 09:04 AM, Anthony Liguori wrote:

On 03/09/2012 05:20 AM, Cleber Rosa wrote:

You're comparing developer-level tests with the existent QA-level
tests (much more complex).


Let's be specific then. Look at device-add.sh in qemu-test. It's 71LOC.
pci_hotplug.py in autotest is 204LOC.


pci_hotplug.py does much more than device-add.sh:

* tests both pci_add and device_add commands
* checks the monitor syntax for adding a new drive, that is, it works on HEAD
and on other versions (such as the ones in some RHEL releases);
* tests both nic and block hotplug
* for block, tests with both virtio and scsi
* also does device removal, both for pci_add and device_add syntaxes


Ok, but clearly, there's no magic in autotest that makes this sufficiently
easier. It's just a matter of:

cmd=`named_choose command device_add pci_add`

if test $cmd = device_add; then
qmp device_add --driver=virtio-blk-pci --drive=hd0
else
hmp pci_add auto virtio-blk-pci,drive=hd0
fi

It's not there today because pci_add is deprecated. There assertion was that
autotest makes it easier to write tests. How does it make it easier to write
pci_hotplug?



Sure. I agree that it's fair from QEMU's PoV alone to forget about legacy things
such as pci_add.

On this particular example, the one thing that strikes me the most is that
(kvm-)autotest allows either a very static test run (as device_add.sh does), or
a very configurable test run (as pci_hotplug.py does and QE needs).


With named_choose and profiles, you can actually configure it very specifically. 
 Most of the tests are not using named_choose right now, but it's easy enough 
to change that.


Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 09:42 AM, Anthony Liguori wrote:

On 03/09/2012 06:40 AM, Cleber Rosa wrote:

On 03/09/2012 09:04 AM, Anthony Liguori wrote:

On 03/09/2012 05:20 AM, Cleber Rosa wrote:

You're comparing developer-level tests with the existent QA-level
tests (much more complex).


Let's be specific then. Look at device-add.sh in qemu-test. It's 
71LOC.

pci_hotplug.py in autotest is 204LOC.


pci_hotplug.py does much more than device-add.sh:

* tests both pci_add and device_add commands
* checks the monitor syntax for adding a new drive, that is, it 
works on HEAD

and on other versions (such as the ones in some RHEL releases);
* tests both nic and block hotplug
* for block, tests with both virtio and scsi
* also does device removal, both for pci_add and device_add syntaxes


Ok, but clearly, there's no magic in autotest that makes this 
sufficiently

easier. It's just a matter of:

cmd=`named_choose command device_add pci_add`

if test $cmd = device_add; then
qmp device_add --driver=virtio-blk-pci --drive=hd0
else
hmp pci_add auto virtio-blk-pci,drive=hd0
fi

It's not there today because pci_add is deprecated. There assertion 
was that
autotest makes it easier to write tests. How does it make it easier 
to write

pci_hotplug?



Sure. I agree that it's fair from QEMU's PoV alone to forget about 
legacy things

such as pci_add.

On this particular example, the one thing that strikes me the most is 
that
(kvm-)autotest allows either a very static test run (as device_add.sh 
does), or

a very configurable test run (as pci_hotplug.py does and QE needs).


With named_choose and profiles, you can actually configure it very 
specifically.  Most of the tests are not using named_choose right now, 
but it's easy enough to change that.


And then the two code bases will end up having even more similar 
features, similar complexity, etc.




Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Lucas Meneghel Rodrigues


On 03/09/2012 09:13 AM, Anthony Liguori wrote:

On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:

Here is the qemu-test version

http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master



So virtio-serial is an exception in autotest:

2174 virtio_console.py
1875 cgroup.py
615 ksm_overcommit.py
439 qemu_img.py
407 qmp_basic.py
389 qmp_basic_rhel6.py
356 stepmaker.py
247 steps.py

// The next largest actual test of QEMU
203 pci_hotplug.py
190 cdrom.py
182 physical_resources_check.py
181 timedrift.py
170 enospc.py
150 balloon_check.py
138 multi_disk.py
121 unittest.py
117 migration.py
111 cpu_hotplug.py
107 migration_multi_host.py
104 nic_hotplug.py
103 timedrift_with_stop.py
96 timedrift_with_migration.py
...

So picking virtio-serial as your comparison point is not really
representative of kvm-autotest but at any rate...


We have a bunch of high level test functions in 
client/virt/virt_test_utils.py that contain some commonly used test 
functions such as migration and running autotest tests on vms, and other 
functions, that allow us to reuse those functions on the tests and save 
code, but we can reasonably assume that it doesn't change the order of 
magnitude of the actual qemu tests in size, so point taken.


You also have a point in the respect that a lot of the large tests are 
more linux-qemu integration tests, name cpuflags, cgroups, ksm_overcommit.


The point you tried to make and I replied to was 'qemu-test tests are 
all smaller than the equivalent kvm autotest tests'. Well, sure they 
tend to be, but in pretty much all cases more code means more 
functionality being covered, and making sure the same test works on 
RHEL5, RHEL6, upstream, on an Ubuntu, Fedora, RHEL or even Windows guests.


It is indeed a bit nerve wrecking to hear that all you can do with the 
stuff you have been working on the last 3 years can be done better with 
a dozen of shell script functions. It's similar to say that we just like 
to throw lines at a text editor just for the fun of it. I am sure you 
didn't mean it but that is how it sounded, and that's why I'd like to 
assure that the code there *does stuff*. It's just that this extra stuff 
is potentially not interesting to the goal of doing developer level 
regression testing of qemu alone.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 08/03/2012 18:59, Ademar Reis ha scritto:
 unit-test:
 void main()
 {
 my_function();
 }
 
 integration test (or validation test):
 void main()
 {
 exec(my-application);
 }
 
 But that's all semantics, not important for this discussion IMO.

We do have some real unit tests in QEMU, but because a large part of
it is not suitable to inclusion as a library, there's also something in
the middle for QEMU.

Both qtest and qemu-io-tests are in some sense integration tests because
they need working binaries (respectively qemu and qemu-io).  However,
the protocol spoken to the binary, and the kind of stimuli in the test
are designed to affect a particular function or module of QEMU; that
makes the tests more similar to a unit test, especially for qtest which
is written using gtest or PyUnit.

This is where we want to have most of test case growth in the
not-so-long term.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 08/03/2012 22:24, Anthony Liguori ha scritto:
 The qemu-test tests are smaller than the corresponding autotest tests.

They also do much less.

It's true that a combination of qemu-test + qtests could do 99% of the
job more simply than autotest.  But the last 1% (including migration)
would require a large effort, while it would be just there in autotest.

 Making autotest an unconditional dependency of qemu.git is a non-starter for 
 me.

If it were as simple as yum install py-autotest it would not be a
problem, would it?

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 09/03/2012 01:04, Anthony Liguori ha scritto:

 * Sends data between host and guest back and forth, validates the data
 being
 sent, for both small and large amounts of data, both random or
 sequential.
 * Tests write/send in blocking, polling, selecting mode, with port
 mode sync/async
 
 This bullet (and many of the bullets) below are not tests of QEMU.  They
 are tests of the Linux kernel.  This is an integration test, not a
 functional test of QEMU.

I think only these two other bullets are pure kernel tests:

  * Tries lseek on the device
  * Verifies if concomitant access to a single port passes or fails

In particular, migration and monitoring for panics are very useful for
QEMU, since QEMU bugs can crash the kernel.  Of course this is moot if
you have proper unit tests at the qtest level, but there's quite some
work to do before you can dismiss all the work in autotest.  And when
you have those, you can also dismiss some of the work in qemu-test.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 08/03/2012 22:03, Anthony Liguori ha scritto:


 Herein lies the problem. You forgot and it's your proposal :-)

 Ok, fair enough :) But still, qemu-jeos points out to external
 repositories,
 just as much as buildroot. It seems to me that the whole point about FSF
 requiring the source to be under your control is no longer valid here.
 
 There aren't qemu-jeos binaries on qemu.org.  There won't be until I
 mirror the git repos.
 
 It's the infrastructure that matters here.  Submodules provides a nice
 infrastructure to handle all of this and minimizing the external
 components makes the whole thing much more manageable.

How do you handle out-of-tree patches with submodules (as is the case
when working on new code)?

You want to require tests in order to commit to qemu, but this (for
tests where using qtest is not feasible for any reason) requires all
drivers to be upstream and accessible to qemu-jeos.

Perhaps for this it would make sense to associate a qemu-jeos commit id
with an upstream commit (from upstream) + a quilt patch queue.

But there's also the problem of embedded devices whose toolchain is not
available upstream at all, so you'd need to import those separately and
somehow add a different submodule.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 07:07 AM, Paolo Bonzini wrote:

Il 08/03/2012 22:24, Anthony Liguori ha scritto:

The qemu-test tests are smaller than the corresponding autotest tests.


They also do much less.

It's true that a combination of qemu-test + qtests could do 99% of the
job more simply than autotest.  But the last 1% (including migration)
would require a large effort, while it would be just there in autotest.


Can you clarify here?  When you say migration, what I think you mean is that 
kvm-autotest has the ability to run autotest client tests within a guest while 
migration.


I do not believe (with the exception of virtio-serial) that it has the ability 
to run tests like pci_hotplug while migrating.


That means libautotest would have nothing to do with the migration test in 
kvm-autotest.


IOW, if we integrate qemu-test with gtest, and autotest learned how to speak the 
gtest protocol, what features would not be available to gtest-based tests verses 
ones written against libautotest?





Making autotest an unconditional dependency of qemu.git is a non-starter for me.


If it were as simple as yum install py-autotest it would not be a
problem, would it?


It if existed today, had clear value, was widely available, and had a stable 
interface, it wouldn't be an insurmountable problem.


But none of those points are true right now.  And right now, the most important 
one is has clear value.  We really need to have a clear technical discussion 
about libautotest would actually do.


AFAICT, only two things have been suggested for what it would do:

1) it replaces gtest with its own protocol (but why is this protocol better than 
gtest?)


2) it adds a launching abstraction layer such that the same test can be launched 
with QEMU directly and with libvirt.


But (2) is an anti-feature from my point of view.  The last thing we need is yet 
another layer of abstraction for launching QEMU.


Regards,

Anthony Liguori



Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Ademar Reis

On Fri, Mar 09, 2012 at 09:41:05AM +, Stefan Hajnoczi wrote:
 On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis ar...@redhat.com wrote:
  On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
  On 03/08/2012 04:24 PM, Ademar Reis wrote:
  On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
  On 03/08/2012 03:02 PM, Ademar Reis wrote:
  On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
  On 03/08/2012 11:59 AM, Ademar Reis wrote:
     - QE will be alienated from the qemu test effort. There will be
       no integration between the QE efforts and the maintenance of
       the qemu developer-level tests.
  
  I think we're a pretty friendly and open community :-)  There is no
  reason that QE should be alienated unless folks are choosing not
  to participate upstream.
  
  For the exact same reasons you as a developer don't want to
  implement tests inside autotest, QE won't want to implement tests
  for qemu.git. It's out of their comfort zone, just put yourself
  on their shoes.
 
  This is a really, really poor argument and I hope I don't need to go
  into details of why.  If the primary reason for libautotest is so
  the people writing tests for QEMU can avoid actually working with
  the developers of QEMU...  we've got a problem.
 
  No, one of the benefits of having libautotest is to *collaborate*
  with QE. I'll explain again:
 
  - As a qemu developer, I don't want to spend my time learning and
   getting involved in autotest, which is a complex QE project
   (I heard this numerous times).
 
  - As a Quality Engineer, I don't want to invest my time learning
   and getting involved into upstream qemu to test HEAD.
 
 I think this is the key point of the whole discussion - most of the
 other topics have been distractions.  Both communities do testing but
 we test different things and have different priorities.
 
 For me this has been the big realization from this discussion.  I felt
 kvm-autotest and qemu should share tests.  I was pushing for that but
 after following this thread I don't think it makes sense, here's why:
 
 The Quality Engineer you describe is not a QEMU upstream QE, instead
 the QE has a broader and more downstream focus.  (This is why
 comparisons with WebKit or other upstream projects doing testing are
 not valid comparisons.)

Lucas, Cleber and the others red-hatters should remembers this
from my internal presentation, it was the first point I made:
QE and Developers have very different goals and interests.

Which is why we're pushing all these changes in autotest. We see
opportunities for collaboration, but we do realize the difference.

And look: Lucas and Cleber are not QE, they're developers working
on the autotest framework/library/whatever. We'll need similar
positions inside qemu as the test infra-structure grows.

 
 There is not enough in common between upstream QEMU testing and
 downstream KVM QE to make convergence a win-win.  libautotest sounds
 like a technical solution to a people problem - the problem is that we
 have different priorities.  We overlap but at the end of the day we do
 different things.  We can make a best effort to converge but I don't
 see incentives that will make this a success.  Creating an abstraction
 library will be sub-optimal for both communities.

This is the part I don't agree, but in the end it's a matter of opinion.

 
 I think what's much more valuable is for qemu.git tests to be easily
 hooked into autotest.  That way you get access to the testing that the
 qemu community is doing for free.

We get this by default in our plans: any application that can be
run and returns 0/error can be run as a test inside autotest.

What we're asking is the *possibility* of a developer using
libautotest when writting a complex test.

- We're not asking everybody to use autotest
- We're not saying autotest is better for everything
- We're not requiring you to install autotest on your machine
  (even though the process will be trivial)

- We're asking that, if a developer is going to write a test
  together with his patch to submit to qemu.git, he should be
  allowed to use libautotest at his own discretion...
- Ditto for using the test-runner: even if tests are simple
  scripts, there will be benefits in using our optional
  test-runner during development.

This is how I see the interaction with autotest-based tests:

$ make check-full
tests.d/my-test-using-qemu-tests.sh -- PASS
tests.d/trivial-test.sh -- PASS
tests.d/my-test-using-libautotest -- SKIP
...

$ yum install libautotest
# (or whatever the bootstrap procedure is)
$ make check-full
tests.d/my-test-using-qemu-tests.sh -- PASS
tests.d/trivial-test.sh -- PASS
tests.d/my-test-using-libautotest -- PASS
...

But for some reason that I still don't understand, the simple
acceptance of the optional dependency of libautotest in qemu is
seen as a bad thing.

If the ultimate requirement now is we don't want to use an

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 07:36 AM, Paolo Bonzini wrote:

Il 08/03/2012 22:03, Anthony Liguori ha scritto:



Herein lies the problem. You forgot and it's your proposal :-)


Ok, fair enough :) But still, qemu-jeos points out to external
repositories,
just as much as buildroot. It seems to me that the whole point about FSF
requiring the source to be under your control is no longer valid here.


There aren't qemu-jeos binaries on qemu.org.  There won't be until I
mirror the git repos.

It's the infrastructure that matters here.  Submodules provides a nice
infrastructure to handle all of this and minimizing the external
components makes the whole thing much more manageable.


How do you handle out-of-tree patches with submodules (as is the case
when working on new code)?


It's very easy to update .gitmodules to point to a different tree on your local 
system and then update the ref to a local commit.


So from a development perspective, it's simple.  The harder question is what to 
do about qemu-test HEAD on qemu.org


Maintaining downstreams or patches is just too much work IMHO.  I think for 
qemu.org, we simply have to use Linus or Avi's tree and simply live with the 
consequences.


We could open up another tree on qemu.org with patches if someone was willing to 
maintain it but I think that's a bad strategy to take.  But it's a possibility 
if this really becomes a problem.




You want to require tests in order to commit to qemu, but this (for
tests where using qtest is not feasible for any reason) requires all
drivers to be upstream and accessible to qemu-jeos.


We're really just talking about virtio here, no?  Maybe we can convince Rusty to 
have a proper virtio-next.git...



Perhaps for this it would make sense to associate a qemu-jeos commit id
with an upstream commit (from upstream) + a quilt patch queue.

But there's also the problem of embedded devices whose toolchain is not
available upstream at all, so you'd need to import those separately and
somehow add a different submodule.


I think this ends up being outside the scope of qemu-test.  Perfect is the enemy 
of good here.


Regards,

Anthony Liguori


Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

On 03/09/2012 06:48 AM, Lucas Meneghel Rodrigues wrote:

On 03/09/2012 09:13 AM, Anthony Liguori wrote:

On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:

Here is the qemu-test version

http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

So virtio-serial is an exception in autotest:

2174 virtio_console.py
1875 cgroup.py
615 ksm_overcommit.py
439 qemu_img.py
407 qmp_basic.py
389 qmp_basic_rhel6.py
356 stepmaker.py
247 steps.py

So picking virtio-serial as your comparison point is not really
representative of kvm-autotest but at any rate...

We have a bunch of high level test functions in client/virt/virt_test_utils.py
that contain some commonly used test functions such as migration and running
autotest tests on vms, and other functions, that allow us to reuse those
functions on the tests and save code, but we can reasonably assume that it
doesn't change the order of magnitude of the actual qemu tests in size, so point
taken.

You also have a point in the respect that a lot of the large tests are more
linux-qemu integration tests, name cpuflags, cgroups, ksm_overcommit.

And this is exactly the type of thing that autotest will always be the best tool
for.

The point you tried to make and I replied to was 'qemu-test tests are all
smaller than the equivalent kvm autotest tests'. Well, sure they tend to be, but
in pretty much all cases more code means more functionality being covered, and
making sure the same test works on RHEL5, RHEL6, upstream, on an Ubuntu, Fedora,
RHEL or even Windows guests.

But this is not the scope of qemu-test. And we may end up duplicating some
things here but does it really matter? We're talking about a few dozen python
scripts that are a ~100 lines.

It is indeed a bit nerve wrecking to hear that all you can do with the stuff you
have been working on the last 3 years can be done better with a dozen of shell
script functions. It's similar to say that we just like to throw lines at a text
editor just for the fun of it. I am sure you didn't mean it but that is how it
sounded, and that's why I'd like to assure that the code there *does stuff*.

Look at how this discussion started. We've been discussing testing on
qemu-devel at excruciating length and detail and have finally come to something
of a consensus. AFAIK, no one from autotest has participated in those
discussions which is fair as I'm sure ya'll don't read qemu-devel religiously.

Then we see this note that more or less declares, this is how QEMU should do all
of its testing. What reaction did you really expect there to be? :-)

It's just that this extra stuff is potentially not interesting to the goal of
doing developer level regression testing of qemu alone.

I think we need to focus this discussion on concrete technical proposals. If
the proposal is, QEMU should use libautotest, we need to start with an awful lot
more detail about libautotest does and what functions it provides.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 09/03/2012 15:01, Anthony Liguori ha scritto:
 How do you handle out-of-tree patches with submodules (as is the case
 when working on new code)?
 
 It's very easy to update .gitmodules to point to a different tree on
 your local system and then update the ref to a local commit.
 
 So from a development perspective, it's simple.  The harder question is
 what to do about qemu-test HEAD on qemu.org

Yep.  So you would commit patches to qemu.git with some kind of IOU for
tests, that will be committed as soon as kernel support hits the mainline?

 You want to require tests in order to commit to qemu, but this (for
 tests where using qtest is not feasible for any reason) requires all
 drivers to be upstream and accessible to qemu-jeos.
 
 We're really just talking about virtio here, no?  Maybe we can convince
 Rusty to have a proper virtio-next.git...
 
 I think this ends up being outside the scope of qemu-test.  Perfect is
 the enemy of good here.

No, this ends up being outside the scope of Anthony, who only cares
about testing KVM/x86.  We add one virtio device every couple years,
while new ARM boards are added every couple months.  It kind of works
because a large part of the QEMU development community is testing on
KVM/x86, but not entirely.

It is not a problem. of course: you're human, you're one person and one
would need a dozen to set up everything properly.  OTOH, it's too easy
to dismiss buildroot when you are catering to a much smaller objective.
The complete objective would have a much bigger overlap with
buildroot's, and likely the design would too.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Lucas Meneghel Rodrigues


On 03/09/2012 11:13 AM, Anthony Liguori wrote:

It is indeed a bit nerve wrecking to hear that all you can do with the
stuff you
have been working on the last 3 years can be done better with a dozen
of shell
script functions. It's similar to say that we just like to throw lines
at a text
editor just for the fun of it. I am sure you didn't mean it but that
is how it
sounded, and that's why I'd like to assure that the code there *does
stuff*.


Look at how this discussion started. We've been discussing testing on
qemu-devel at excruciating length and detail and have finally come to
something of a consensus. AFAIK, no one from autotest has participated
in those discussions which is fair as I'm sure ya'll don't read
qemu-devel religiously.


All right, point taken.


Then we see this note that more or less declares, this is how QEMU
should do all of its testing. What reaction did you really expect there
to be? :-)


It was an attempt to offer what we have rather than dictating how QEMU 
should do all its testing:



One of our main goals is to provide useful tools for the qemu community,
since we have a good number of tests and libraries written to perform
integration/QA testing for that tool, being successfuly used by a number
of QA teams that work on qemu.


I re-read the first message I sent, and certainly did not find where we 
declare that this is the way QEMU should do its testing.


If you judge that there's nothing interesting there for qemu, I'm fine 
with it. That all said, certainly I did not expect your repeated 
attempts to show that you can do it all better with a couple of shell 
script lines, oh boy, I did not.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Ademar Reis

On Fri, Mar 09, 2012 at 08:13:45AM -0600, Anthony Liguori wrote:
 On 03/09/2012 06:48 AM, Lucas Meneghel Rodrigues wrote:

 Look at how this discussion started.  We've been discussing testing
 on qemu-devel at excruciating length and detail and have finally
 come to something of a consensus.  AFAIK, no one from autotest has
 participated in those discussions which is fair as I'm sure ya'll
 don't read qemu-devel religiously.

The discussion started back in December and this is a follow-up
RFC based on what was said back then.

(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg02434.html)

 
 Then we see this note that more or less declares, this is how QEMU
 should do all of its testing.  What reaction did you really expect
 there to be? :-)

This is a clear misunderstanding: we've repeated numerous times
that libautotest and the test runner are optional. I don't know
how much I have to stress this out.

Technically speaking, the use of autotest would be enabled by
import autotest, source $AUTOTESTDIR/libautotest.sh or
something similar. This can be easily detected by the test
runner, which could skip such tests if autotest is not available.

We're also offering a global test-runner that we believe will
bring advantages to anybody running tests that follow the simple
requirement of being a script returning 0/error.

 
 It's just that this extra stuff is potentially not interesting to the goal of
 doing developer level regression testing of qemu alone.
 
 I think we need to focus this discussion on concrete technical
 proposals.  If the proposal is, QEMU should use libautotest, we need
 to start with an awful lot more detail about libautotest does and
 what functions it provides.

The discussion diverged long ago. So much that the initial test
examples, bootstrap procedure, directory structure, test runner
output, and our request for requiements was all kind of
forgotten.

Anyway, we'll keep working on autotest targeting the other
projects which will benefit from it. As I said in the other
e-mail, we'll probably revisit this topic in the KVM forum in
person, with more code to show and hopefully with some beers
around us. :-)

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 09/03/2012 14:56, Anthony Liguori ha scritto:
 On 03/09/2012 07:07 AM, Paolo Bonzini wrote:
 Il 08/03/2012 22:24, Anthony Liguori ha scritto:
 The qemu-test tests are smaller than the corresponding autotest 
 tests.
 
 They also do much less.
 
 It's true that a combination of qemu-test + qtests could do 99% of 
 the job more simply than autotest.  But the last 1% (including 
 migration) would require a large effort, while it would be just 
 there in autotest.
 
 Can you clarify here?  When you say migration, what I think you 
 mean is that kvm-autotest has the ability to run autotest client 
 tests within a guest while migration.

1) Migration is quite different from other tests in terms of VM
lifecycle, even if you use migrate-to-file + restore.  It requires
special infrastructure in the drivers.

2) Migration bugs are subtle and often nondeterministic, so they require
long running tests that do the same thing over and over again.  These
requirements are more similar to integration tests than unit tests.  So
integration tests cannot really be dismissed.

BTW, I don't see the problem with requiring a large make check time.
I probably already said this: when I was working on GCC, I ran tests
overnight every time I had piled up enough changes, and it was simply
not possible to skip that step because compiler changes often had
surprising ramifications.  Yes, in some cases I had a bad surprise the
following day, but it's unavoidable.

Even today, bootstrapping and testing GCC on embedded systems can be
done but it takes 4-5 *days*.  Of course you do it first in a
cross-compiler, but surprises still can happen.

But we're lucky: compilers and QEMU are pretty much the opposite, and a
bad surprise should be much more rare here.  QEMU tests and
infrastructure are harder, but the components are delimited much better
than in a compiler with 200 cascading passes, which is what makes qtest
possible at all.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 08:30 AM, Paolo Bonzini wrote:

Il 09/03/2012 15:01, Anthony Liguori ha scritto:

How do you handle out-of-tree patches with submodules (as is the case
when working on new code)?


It's very easy to update .gitmodules to point to a different tree on
your local system and then update the ref to a local commit.

So from a development perspective, it's simple.  The harder question is
what to do about qemu-test HEAD on qemu.org


Yep.  So you would commit patches to qemu.git with some kind of IOU for
tests, that will be committed as soon as kernel support hits the mainline?


Probably commit the test and mark it as an expected failure until support hits 
mainline.





You want to require tests in order to commit to qemu, but this (for
tests where using qtest is not feasible for any reason) requires all
drivers to be upstream and accessible to qemu-jeos.


We're really just talking about virtio here, no?  Maybe we can convince
Rusty to have a proper virtio-next.git...

I think this ends up being outside the scope of qemu-test.  Perfect is
the enemy of good here.


No, this ends up being outside the scope of Anthony, who only cares
about testing KVM/x86.  We add one virtio device every couple years,
while new ARM boards are added every couple months.  It kind of works
because a large part of the QEMU development community is testing on
KVM/x86, but not entirely.


qemu-test is a specific solution to a specific problem.  The problem is: how can 
you leverage an existing kernel to simplify writing device model tests without 
having a full blown libOS in qtest.


Linux is the only part that matters here.  The userspace in qemu-jeos is just 
there to give a small environment for Linux to function properly in.



It is not a problem. of course: you're human, you're one person and one
would need a dozen to set up everything properly.  OTOH, it's too easy
to dismiss buildroot when you are catering to a much smaller objective.
The complete objective would have a much bigger overlap with
buildroot's, and likely the design would too.


You have a different objective than I do.  You want to build a reproducible 
guest that runs on a wider variety of architectures than common distributions do 
with a good enough userspace to write integration tests with.  That's a good 
goal if you care about the full stack.


I'm skeptical of the value of this from a QEMU point of view.  Do we really care 
if the buildroot version of udev propagates hotplug events correct in buildroot 
for ARM in a beagleboard machine?


I'm sure someone cares about that, but it's ultimately not a QEMU problem.  I 
think we need to focus on getting our house in order (in terms of quality) 
before we start trying to tackle a larger scope.


Regards,

Anthony Liguori



Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 08:43 AM, Paolo Bonzini wrote:

Il 09/03/2012 14:56, Anthony Liguori ha scritto:

On 03/09/2012 07:07 AM, Paolo Bonzini wrote:

Il 08/03/2012 22:24, Anthony Liguori ha scritto:

The qemu-test tests are smaller than the corresponding autotest
tests.


They also do much less.

It's true that a combination of qemu-test + qtests could do 99% of
the job more simply than autotest.  But the last 1% (including
migration) would require a large effort, while it would be just
there in autotest.


Can you clarify here?  When you say migration, what I think you
mean is that kvm-autotest has the ability to run autotest client
tests within a guest while migration.


1) Migration is quite different from other tests in terms of VM
lifecycle, even if you use migrate-to-file + restore.  It requires
special infrastructure in the drivers.

2) Migration bugs are subtle and often nondeterministic, so they require
long running tests that do the same thing over and over again.  These
requirements are more similar to integration tests than unit tests.  So
integration tests cannot really be dismissed.


I think migration really requires special test support in QEMU.  I've long 
wanted to introduce a migration torture mode that would constantly tear down the 
device model and then restore it during normal operation.


I would then want to make -migration-torture an option to any test.


BTW, I don't see the problem with requiring a large make check time.
I probably already said this: when I was working on GCC, I ran tests
overnight every time I had piled up enough changes, and it was simply
not possible to skip that step because compiler changes often had
surprising ramifications.  Yes, in some cases I had a bad surprise the
following day, but it's unavoidable.

Even today, bootstrapping and testing GCC on embedded systems can be
done but it takes 4-5 *days*.  Of course you do it first in a
cross-compiler, but surprises still can happen.


I think it's fair to have multiple levels of check.  There should be a short 
sanity check one at least but I have no problem with make check-full taking a 
long period of time.


Regards,

Anthony Liguori


But we're lucky: compilers and QEMU are pretty much the opposite, and a
bad surprise should be much more rare here.  QEMU tests and
infrastructure are harder, but the components are delimited much better
than in a compiler with 200 cascading passes, which is what makes qtest
possible at all.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Stefan Hajnoczi

On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis ar...@redhat.com wrote:
 On Fri, Mar 09, 2012 at 09:41:05AM +, Stefan Hajnoczi wrote:
 On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis ar...@redhat.com wrote:
  On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
  On 03/08/2012 04:24 PM, Ademar Reis wrote:
  On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
  On 03/08/2012 03:02 PM, Ademar Reis wrote:
  On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
  On 03/08/2012 11:59 AM, Ademar Reis wrote:
     - QE will be alienated from the qemu test effort. There will be
       no integration between the QE efforts and the maintenance of
       the qemu developer-level tests.
  
  I think we're a pretty friendly and open community :-)  There is no
  reason that QE should be alienated unless folks are choosing not
  to participate upstream.
  
  For the exact same reasons you as a developer don't want to
  implement tests inside autotest, QE won't want to implement tests
  for qemu.git. It's out of their comfort zone, just put yourself
  on their shoes.
 
  This is a really, really poor argument and I hope I don't need to go
  into details of why.  If the primary reason for libautotest is so
  the people writing tests for QEMU can avoid actually working with
  the developers of QEMU...  we've got a problem.
 
  No, one of the benefits of having libautotest is to *collaborate*
  with QE. I'll explain again:
 
  - As a qemu developer, I don't want to spend my time learning and
   getting involved in autotest, which is a complex QE project
   (I heard this numerous times).
 
  - As a Quality Engineer, I don't want to invest my time learning
   and getting involved into upstream qemu to test HEAD.

 I think this is the key point of the whole discussion - most of the
 other topics have been distractions.  Both communities do testing but
 we test different things and have different priorities.

 For me this has been the big realization from this discussion.  I felt
 kvm-autotest and qemu should share tests.  I was pushing for that but
 after following this thread I don't think it makes sense, here's why:

 The Quality Engineer you describe is not a QEMU upstream QE, instead
 the QE has a broader and more downstream focus.  (This is why
 comparisons with WebKit or other upstream projects doing testing are
 not valid comparisons.)

 Lucas, Cleber and the others red-hatters should remembers this
 from my internal presentation, it was the first point I made:
 QE and Developers have very different goals and interests.

 Which is why we're pushing all these changes in autotest. We see
 opportunities for collaboration, but we do realize the difference.

 And look: Lucas and Cleber are not QE, they're developers working
 on the autotest framework/library/whatever. We'll need similar
 positions inside qemu as the test infra-structure grows.

I don't understand this last paragraph.  If qemu.git upstream was
doing full-scale QE it would work fine because the differences that
I've described and you also have pointed out would be absent.

 I think what's much more valuable is for qemu.git tests to be easily
 hooked into autotest.  That way you get access to the testing that the
 qemu community is doing for free.

 We get this by default in our plans: any application that can be
 run and returns 0/error can be run as a test inside autotest.

Yes, and I think that part of the plan makes sense.

 What we're asking is the *possibility* of a developer using
 libautotest when writting a complex test.

 - We're not asking everybody to use autotest
 - We're not saying autotest is better for everything
 - We're not requiring you to install autotest on your machine
  (even though the process will be trivial)

 - We're asking that, if a developer is going to write a test
  together with his patch to submit to qemu.git, he should be
  allowed to use libautotest at his own discretion...
 - Ditto for using the test-runner: even if tests are simple
  scripts, there will be benefits in using our optional
  test-runner during development.

 This is how I see the interaction with autotest-based tests:

    $ make check-full
    tests.d/my-test-using-qemu-tests.sh -- PASS
    tests.d/trivial-test.sh -- PASS
    tests.d/my-test-using-libautotest -- SKIP
    ...

    $ yum install libautotest
    # (or whatever the bootstrap procedure is)
    $ make check-full
    tests.d/my-test-using-qemu-tests.sh -- PASS
    tests.d/trivial-test.sh -- PASS
    tests.d/my-test-using-libautotest -- PASS
    ...

 But for some reason that I still don't understand, the simple
 acceptance of the optional dependency of libautotest in qemu is
 seen as a bad thing.

I don't mind a libvirttest but I've expressed concerns that that if
qemu.git has it's own test library then the incentive to use or
contribute to an external library is not there.  That's why I
suggested a less ambitious approach.

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 09/03/2012 15:43, Anthony Liguori ha scritto:
 Linux is the only part that matters here.  The userspace in qemu-jeos
 is just there to give a small environment for Linux to function
 properly in.

But again that's not okay for all testcases.  If I want to do SCSI
tests, I cannot write them in shell scripts because qemu-jeos does not
have sg3_utils.

Even libos would not help; relying on sg3_utils means that the client is
a completely clear reimplementation.  I don't care about the extra trips
to the SCSI specs that sg3_utils would save.  The problem is that since
qtest cannot run on real hardware, I don't have confirmation that I
didn't misinterpret the standards.  Writing all bits of the testsuite
makes it likely to have the same misinterpretations in both the tested
code and the tests, even if I try to write the tests from scratch with
only the spec in front of me.

 I'm skeptical of the value of this from a QEMU point of view.  Do we
 really care if the buildroot version of udev propagates hotplug
 events correct in buildroot for ARM in a beagleboard machine?

I don't care about udev.  But even just binutils+GCC+kernel+uclibc is
not as easy if you care about dozens of architectures.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Ademar Reis

On Fri, Mar 09, 2012 at 02:54:23PM +, Stefan Hajnoczi wrote:
 On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis ar...@redhat.com wrote:
  On Fri, Mar 09, 2012 at 09:41:05AM +, Stefan Hajnoczi wrote:
  On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis ar...@redhat.com wrote:
   On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
   On 03/08/2012 04:24 PM, Ademar Reis wrote:
   On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
   On 03/08/2012 03:02 PM, Ademar Reis wrote:
   On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
   On 03/08/2012 11:59 AM, Ademar Reis wrote:
  - QE will be alienated from the qemu test effort. There will be
    no integration between the QE efforts and the maintenance of
    the qemu developer-level tests.
   
   I think we're a pretty friendly and open community :-)  There is no
   reason that QE should be alienated unless folks are choosing not
   to participate upstream.
   
   For the exact same reasons you as a developer don't want to
   implement tests inside autotest, QE won't want to implement tests
   for qemu.git. It's out of their comfort zone, just put yourself
   on their shoes.
  
   This is a really, really poor argument and I hope I don't need to go
   into details of why.  If the primary reason for libautotest is so
   the people writing tests for QEMU can avoid actually working with
   the developers of QEMU...  we've got a problem.
  
   No, one of the benefits of having libautotest is to *collaborate*
   with QE. I'll explain again:
  
   - As a qemu developer, I don't want to spend my time learning and
    getting involved in autotest, which is a complex QE project
    (I heard this numerous times).
  
   - As a Quality Engineer, I don't want to invest my time learning
    and getting involved into upstream qemu to test HEAD.
 
  I think this is the key point of the whole discussion - most of the
  other topics have been distractions.  Both communities do testing but
  we test different things and have different priorities.
 
  For me this has been the big realization from this discussion.  I felt
  kvm-autotest and qemu should share tests.  I was pushing for that but
  after following this thread I don't think it makes sense, here's why:
 
  The Quality Engineer you describe is not a QEMU upstream QE, instead
  the QE has a broader and more downstream focus.  (This is why
  comparisons with WebKit or other upstream projects doing testing are
  not valid comparisons.)
 
  Lucas, Cleber and the others red-hatters should remembers this
  from my internal presentation, it was the first point I made:
  QE and Developers have very different goals and interests.
 
  Which is why we're pushing all these changes in autotest. We see
  opportunities for collaboration, but we do realize the difference.
 
  And look: Lucas and Cleber are not QE, they're developers working
  on the autotest framework/library/whatever. We'll need similar
  positions inside qemu as the test infra-structure grows.
 
 I don't understand this last paragraph.  If qemu.git upstream was
 doing full-scale QE it would work fine because the differences that
 I've described and you also have pointed out would be absent.
 

In order to have QEMU working in full TDD Mode (a current
goal), I predict developers assigned to the maintenance of the
in-house test infrastructure (qemu-test) will be needed, on
positions similar to what Lucas and Cleber currently do with
autotest. Only time will tell.

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 09:00 AM, Paolo Bonzini wrote:

Il 09/03/2012 15:43, Anthony Liguori ha scritto:

Linux is the only part that matters here.  The userspace in qemu-jeos
is just there to give a small environment for Linux to function
properly in.


But again that's not okay for all testcases.  If I want to do SCSI
tests, I cannot write them in shell scripts because qemu-jeos does not
have sg3_utils.


What SCSI tests are you trying to write?

Are these the sort of tests that would be interesting to also run on Fedora, 
Windows, and Ubuntu?


Regards,

Anthony Liguori


Even libos would not help; relying on sg3_utils means that the client is
a completely clear reimplementation.  I don't care about the extra trips
to the SCSI specs that sg3_utils would save.  The problem is that since
qtest cannot run on real hardware, I don't have confirmation that I
didn't misinterpret the standards.  Writing all bits of the testsuite
makes it likely to have the same misinterpretations in both the tested
code and the tests, even if I try to write the tests from scratch with
only the spec in front of me.


I'm skeptical of the value of this from a QEMU point of view.  Do we
really care if the buildroot version of udev propagates hotplug
events correct in buildroot for ARM in a beagleboard machine?


I don't care about udev.  But even just binutils+GCC+kernel+uclibc is
not as easy if you care about dozens of architectures.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

2012-03-09 Thread Stefan Hajnoczi

On Fri, Mar 9, 2012 at 3:01 PM, Ademar Reis ar...@redhat.com wrote:
 On Fri, Mar 09, 2012 at 02:54:23PM +, Stefan Hajnoczi wrote:
 On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis ar...@redhat.com wrote:
  On Fri, Mar 09, 2012 at 09:41:05AM +, Stefan Hajnoczi wrote:
  On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis ar...@redhat.com wrote:
   On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
   On 03/08/2012 04:24 PM, Ademar Reis wrote:
   On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
   On 03/08/2012 03:02 PM, Ademar Reis wrote:
   On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
   On 03/08/2012 11:59 AM, Ademar Reis wrote:
  - QE will be alienated from the qemu test effort. There will be
    no integration between the QE efforts and the maintenance of
    the qemu developer-level tests.
   
   I think we're a pretty friendly and open community :-)  There is no
   reason that QE should be alienated unless folks are choosing not
   to participate upstream.
   
   For the exact same reasons you as a developer don't want to
   implement tests inside autotest, QE won't want to implement tests
   for qemu.git. It's out of their comfort zone, just put yourself
   on their shoes.
  
   This is a really, really poor argument and I hope I don't need to go
   into details of why.  If the primary reason for libautotest is so
   the people writing tests for QEMU can avoid actually working with
   the developers of QEMU...  we've got a problem.
  
   No, one of the benefits of having libautotest is to *collaborate*
   with QE. I'll explain again:
  
   - As a qemu developer, I don't want to spend my time learning and
    getting involved in autotest, which is a complex QE project
    (I heard this numerous times).
  
   - As a Quality Engineer, I don't want to invest my time learning
    and getting involved into upstream qemu to test HEAD.
 
  I think this is the key point of the whole discussion - most of the
  other topics have been distractions.  Both communities do testing but
  we test different things and have different priorities.
 
  For me this has been the big realization from this discussion.  I felt
  kvm-autotest and qemu should share tests.  I was pushing for that but
  after following this thread I don't think it makes sense, here's why:
 
  The Quality Engineer you describe is not a QEMU upstream QE, instead
  the QE has a broader and more downstream focus.  (This is why
  comparisons with WebKit or other upstream projects doing testing are
  not valid comparisons.)
 
  Lucas, Cleber and the others red-hatters should remembers this
  from my internal presentation, it was the first point I made:
  QE and Developers have very different goals and interests.
 
  Which is why we're pushing all these changes in autotest. We see
  opportunities for collaboration, but we do realize the difference.
 
  And look: Lucas and Cleber are not QE, they're developers working
  on the autotest framework/library/whatever. We'll need similar
  positions inside qemu as the test infra-structure grows.

 I don't understand this last paragraph.  If qemu.git upstream was
 doing full-scale QE it would work fine because the differences that
 I've described and you also have pointed out would be absent.


 In order to have QEMU working in full TDD Mode (a current
 goal), I predict developers assigned to the maintenance of the
 in-house test infrastructure (qemu-test) will be needed, on
 positions similar to what Lucas and Cleber currently do with
 autotest. Only time will tell.

I agree that engineers are needed to work on testing as testing
increases upstream.

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 09/03/2012 16:02, Anthony Liguori ha scritto:

 But again that's not okay for all testcases.  If I want to do SCSI
 tests, I cannot write them in shell scripts because qemu-jeos does not
 have sg3_utils.
 
 What SCSI tests are you trying to write?

At the very least dump the inquiry pages, mode pages, etc. and see that
they make sense and correspond to the device properties.

 Are these the sort of tests that would be interesting to also run on
 Fedora, Windows, and Ubuntu?

They should give exactly the same output on any guest.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 09:17 AM, Paolo Bonzini wrote:

Il 09/03/2012 16:02, Anthony Liguori ha scritto:


But again that's not okay for all testcases.  If I want to do SCSI
tests, I cannot write them in shell scripts because qemu-jeos does not
have sg3_utils.


What SCSI tests are you trying to write?


At the very least dump the inquiry pages, mode pages, etc. and see that
they make sense and correspond to the device properties.


Is this not something that's reasonably easy to do in qtest?

Is it possible to write a C program that does the ioctl and dump the inquiry 
page in a text format conducive to shell parsing?



Are these the sort of tests that would be interesting to also run on
Fedora, Windows, and Ubuntu?


They should give exactly the same output on any guest.


Is it valuable to have a per-platform test or since this is mostly passthrough 
to the device (I assume), do you just need a single test?


Regards,

Anthony Liguori



Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

Il 09/03/2012 16:24, Anthony Liguori ha scritto:
 At the very least dump the inquiry pages, mode pages, etc. and see that
 they make sense and correspond to the device properties.
 
 Is this not something that's reasonably easy to do in qtest?

Yes (at least with virtio-scsi the libos bits are relatively small; just
think of what it would have been like when the only HBA was LSI), but
with one gotcha...

 Is it possible to write a C program that does the ioctl and dump the
 inquiry page in a text format conducive to shell parsing?

... sg_utils also parses the pages and dumps them in human-readable
format.  This is useful because it provides a completely separate
implementation and avoids problems with misinterpretation of the
standard.  Of course it would work just as well if someone wrote tests
instead of me.

 Are these the sort of tests that would be interesting to also run on
 Fedora, Windows, and Ubuntu?

 They should give exactly the same output on any guest.
 
 Is it valuable to have a per-platform test or since this is mostly
 passthrough to the device (I assume), do you just need a single test?

Ah, understood.  Yeah, a single test is enough for the purpose of
testing QEMU.  If you want to test the driver too, running under Windows
would be useful.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests


On 03/09/2012 09:34 AM, Paolo Bonzini wrote:

Il 09/03/2012 16:24, Anthony Liguori ha scritto:

At the very least dump the inquiry pages, mode pages, etc. and see that
they make sense and correspond to the device properties.


Is this not something that's reasonably easy to do in qtest?


Yes (at least with virtio-scsi the libos bits are relatively small; just
think of what it would have been like when the only HBA was LSI), but
with one gotcha...


Is it possible to write a C program that does the ioctl and dump the
inquiry page in a text format conducive to shell parsing?


... sg_utils also parses the pages and dumps them in human-readable
format.  This is useful because it provides a completely separate
implementation and avoids problems with misinterpretation of the
standard.  Of course it would work just as well if someone wrote tests
instead of me.


I don't recommend it in the general case, but it should be trivial to add 
additional packages to qemu-jeos and reuse the toolchain to build them.


I don't think it's all that valuable here.  I think you really want to test this 
via qtest.  You could easy copy/paste code from sg_utils to do the parsing if 
you were so inclined...



Are these the sort of tests that would be interesting to also run on
Fedora, Windows, and Ubuntu?


They should give exactly the same output on any guest.


Is it valuable to have a per-platform test or since this is mostly
passthrough to the device (I assume), do you just need a single test?


Ah, understood.  Yeah, a single test is enough for the purpose of
testing QEMU.  If you want to test the driver too, running under Windows
would be useful.


We should separate integration test (testing multiple components where we want 
to be able to use different versions/implementations of one component) from 
functional/unit tests that are strictly testing a single component.  There isn't 
always a clean separation and there may be overlap, but I don't think we should 
stress out about the overlap.


For instance, doing general purpose I/O testing is something where we want to 
test I/O with a Linux guest, a Windows guest, etc.  This is an integration test 
and we should focus on it.


But we probably want some sort of I/O test in qemu-test too since it provides a 
simple functional test.  Yes, there's overlap, but the functional form of the 
test is so simple that it really isn't that important.


But testing how QEMU handles malformatted virtio-blk requests is something 
that's clearly QEMU specific.  There's no value doing that with a Linux and 
Windows guest (even if it's somehow possible).  It's definitely a unit test 
that's strictly specific to QEMU.


I think autotest should be able to execute QEMU's unit and functional tests.  By 
using a test framework like gtest that exposes a test protocol, it should be 
trivial to add that support.


Regards,

Anthony Liguori


Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests