On Fri, May 16, 2014 at 11:44:39PM +0200, Paolo Bonzini wrote:
> Il 16/05/2014 23:28, Ademar Reis ha scritto:
> >
> >The leaves are:
> >
> > env->production
> > env->debug
> > tests->sync_test->standard
> > tests->sync_test->aggressive
> >
> >This will get multiplexed into 4 test scenarios, thus requiring 4
> >test runs:
> >
> > env->production + tests->sync_test->standard
> > env->production + tests->sync_test->aggressive
> > env->debug + tests->sync_test->standard
> > env->debug + tests->sync_test->aggressive
> >
> >It's easy to notice that the system grows exponentially with the
> >number of variants.
>
> So far so good. What would it be without the filter-out: "tests"?
>
The tree I've shown has the other tests chopped out already. But
imagining this new tree:
<ROOT>
* env
* production
- malloc_perturb = no
- gcc_flags = -O3
* debug
- malloc_pertub = yes
- gcc_flags = -g
* tests
* sync_test
* standard
- sync_timeout = 30
- sync_tries = 10
* aggressive
- sync_timeout = 10
- sync_tries = 20
* ping_test
* standard
- ping_tries = 10
- timeout = 20
* aggressive
- ping_flags = -f
- ping_tries = 100
- timeout - 5
The leaves are now:
env->production
env->debug
tests->sync_test->standard
tests->sync_test->aggressive
tests->ping_test->standard
tests->ping_test->aggressive
And now we have 8 combinations:
env->production + tests->sync_test->standard \
+ tests->ping_test->standard
env->production + tests->sync_test->standard \
+ tests->ping_test->aggressive
env->production + tests->sync_test->aggressive
+ tests->ping_test->standard
env->production + tests->sync_test->aggressive
+ tests->ping_test->aggressive
env->debug + tests->sync_test->standard
+ tests->ping_test->standard
env->debug + tests->sync_test->standard \
+ tests->ping_test->aggressive
env->debug + tests->sync_test->aggressive
+ tests->ping_test->standard
env->debug + tests->sync_test->aggressive
+ tests->ping_test->aggressive
The "filter-out: tests" for tests prevents this expansion,
because the multiplexing won't recurse into the tests branch.
> >## default values ##
> >
> >The multiplex system is optional. Tests are supposed to run with
> >default values in a supported system, just like virt-test does
> >today. Test writers declare the defaults (not part of the
> >multiplexing system).
>
> I'm not sure I understand the ramifications of this.
>
> Machine types should be part of the multiplexing system, so that the user
> can run tests with many machine types.
correct.
>
> However, (some) defaults should come from the machine types, not from the
> user, because different machine types have different defaults. Such
> defaults should be picked up automatically unless the user overrides them.
> A user should be able to just ask "run this test on Linux/x86 Windows and
> Linux/Cubieboard" and get virtio with Linux/x86 guests, IDE with Windows
> guests, and an SD card with the Linux/Cubieboard guests.
Excellent example! Let's try to explore it:
In the test code, the qemu command line will be constructed from
variables, returned by the framework. The algorithm that looks
for these variables should be something like:
- get it from the multiplexer
- OR get it from a default config file
- OR get it from the default declared in the test code
The code will probably look something like:
(ignore the details, this is pseudo-code)
if get_varialble("qemu_disk_type") == "ide":
qemu_cmdline.append("-device device ide-drive,bus=ide.0")
...
run \
qemu -M get_variable("qemu_machine_type") qemu_cmdline
...
OR simply:
run \
qemu -M get_variable("qemu_machine_type") \
get_variable("qemu_dash_device_command_line_options")
...
In the multiplexer, the tree to run the tests would have the
following branches:
using host->qemu_config->machine_types
* rhel-6.5.0
- filter-only: "guest->os->linux*"
- machine_type: "rhel-6.5.0"
* rhel-7.0.0
- filter-only: "guest->os->linux*"
- machine_type: "rhel-7.0.0"
* cubieboard-armv8
- filter-only: "guest->os->linux-cubieboard-armv8"
- machine_type: "cubieboard-armv8"
using guest->os
* linux-x86
- os: linux-x86
* linux-cubieboard-armv8
- os: linux-cubieboard-arm-v8
- filter-out "guest-hw->*->virtio"
- filter-only "guest->hw->disk->sdcard"
* windowsxp-x86
- os: winxp-x86
- filter-only "guest->hw->disk->ide
Now assuming we have the original tree and we're not filtering it
too much, we'll have many branches, with many many combinations
after the multiplexing.
For example:
... host->qemu_config_machine_types->rhel-6.4.0
+ guest->hw->disk->ide
+ guest->hw->network->e1000
+ ...
... host->qemu_config_machine_types->rhel-6.4.0
+ guest->hw->disk->scsi
+ guest->hw->network->virtio
+ ...
... host->qemu_config_machine_types->rhel-6.4.0
+ guest->hw->disk->virtio
+ guest->hw->network->rtl8139
+ ...
....
But given linux-cubieboard filters-only
"guest->hw->disk->sdcard", there won't be an option that combines
linux-cubieboard with IDE or SCSI. And also because of a
filter-only, winxp will be restricted to IDE options.
There won't be options that combine a rhel-6.5.0 machine-type
with a cubieboard, also because of filters.
And the list goes on...
>
> If the user says "default = IDE", the test framework should run the tests
> with IDE disks on all guests. If the user says "default = SCSI", the test
> framework should cut out the Cubieboard and only run tests (with SCSI) on
> Linux/x86 and Windows.
The user says "default = SCSI" this way:
$ avocado --mux \
--mux-filter "filter-only: guest->hw->disk->scsi" \
run 'test1 test2 test3'
With this filter, in the case of cubieboard we have the following
intersection:
- filter-only: "guest->hw->disk->scsi" # from the user
(will remove all other disk branches, including sdcard)
- filter-only: "guest->hw->disk->sdcard" # from the variant tree
(will remove all other disk branches, including scsi)
The result is an empty set... No cubieboard runs if using scsi.
>
> I hope that by enforcing structure and naming of variants, we can do this
> without the shortcomings of Cartesian config.
Me too! :-)
>
> Paolo
>
> >So when a branch is filtered out, the test will fallback to the
> >defaults provided by the test runner. Ditto for when tests are
> >run without the multiplexer.
>
--
Ademar de Souza Reis Jr.
Red Hat
^[:wq!
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
