Let me go over the proposal:
On Fri, Nov 21, 2014 at 10:56 AM, Lukáš Doktor <[email protected]
### Introduction
Our goal is to multiplex setting for various platforms, hw layouts
and tests, ...
As example I'm using mix of yaml file similarly to Ademar's RFC, the
only change is that multiplexation of leafs would happen only on
leafs of lists of dicts. This helps us solve problem with the need to
filter-out tests, oss, ... (will talk about it later).
The final tree would be compound from 3 places:
1) default tree (big tree with all the variants)
- hw:
foo: 123
bar: [1,2,3]
variants:
- cpu:
cflags: '-O2'
amd:
@intel:
arm:
filter-only:
- '/env/debug'
- '/os/linux'
filter-out: ['/hw']
- disk:
scsi:
@virtio:
filter-only: ['/os/linux']
- nic_model:
@virtio:
nic_model: virtio
rtl8139:
nic_model: rtl8139
e1000:
nic_modle: e1000
- os:
linux:
dev-tools: 'gcc'
@fedora:
package-manager: 'yum'
init: 'systemd'
mint:
package-manager: 'apt-get'
init: 'init'
win:
bsod: true
dev-tools: ['Cygwin', 'MinGW', 'Visual Studio']
win7:
win8:
bsod: false
- env:
debug:
@prod:
filter-only: ['/os/win']
The idea of 'default' variants (or branches, the name is not that
important) is something I like. You mentioned on IRC that they are
probably not needed - if you can elaborate on that, I got curious.
2) another files merged on cmdline (the same format (1)):
- overrides values and extends the existing nodes
- keyword to delete existing branch
Right, expanding configs with the values from another config file is a
very useful feature. We could decide on a keyword that deletes existing
branches.
3) each test has it's own small tree which will be merged into
/tests/$TESTNAME and list of key components to be multiplexed with
(either as file with the same name + .yaml or as variable inside the
python file __MUX__ = """..."""):
ping_test.yaml:
mux: ["/hw/nic_model"]
vms: "vm1 vm2"
# Other test default variables
variants:
- size:
@default:
long:
size = 65507
- interval:
@default:
short:
interval = 0
requires_root = true
=> would merge into /tests/ping_test node.
Sure, config snippets similar to what it's done in virt-test.
## Default variant
(test developer)
When nothing is specified, no multiplexation would happen, not even
the default mux tree is processed, default values will be used from
the global namespace or special hooks:
intel.virtio.virtio.fedora.prod.pingtest.default.default
Ok, again the idea of default variants, cool.
## Full test
(feature developer, basic check)
When --mux-test=true|false is specified, multiplexer would take into
account the `mux` variable inside each specified test. So basically
it means all variants of the tests on one hw. (again, default mux
tree is not taken into account)
intel.virtio.virtio.fedora.prod.pingtest.default.default
intel.virtio.virtio.fedora.prod.pingtest.default.short
intel.virtio.virtio.fedora.prod.pingtest.long.default
intel.virtio.virtio.fedora.prod.pingtest.long.default
The mux keyword is a nice idea. Nice way to reduce the problem space
and let people combine only the things they are interested into.
## All related components (Paolo, if I understood you correctly, this
one is for you ;-) )
(feature developer, check before submitting)
When --mux_related=true|false is specified, it multiplexes the nodes
specified in test's `mux:` node (so only parts of the tree specified
in `mux:` are processed and multiplexed):
intel.virtio.virtio.fedora.prod.pingtest.default.default
intel.virtio.rtl8139.fedora.prod.pingtest.default.default
intel.virtio.e1000.fedora.prod.pingtest.default.default
in case you use --mux-test and --mux-related it would run
combinations of above (12 variants).
Yep, sounds good.
## Multiplex all
(QA's who wants to test everything on everything unless it's disabled
by `filter-out`)
When --mux is specified, it process the full tree and multiplex
leafs, which are not in the same list item. This is a bit different
than in Ademar's proposal, but very similar to variants in autotest
(with some improvements). I'll talk about it in following chapter.
For now let's just say it'd execute 144 variants (If I'm not wrong).
This way you'd be able to specify tests in the old fashion, all
variants unless filter-out removes them.
OK
### Multiplexation problem
Take a look at the ping_test using config from previous RFC:
hw:...
tests:
filter-out: ["/tests"]
ping_test:
vms: "vm1 vm2"
variants:
size:
default:
long:
size = 65507
interval:
default:
short:
interval = 0
requires_root = true
as you can see it makes sense to have tests-variants. But when we
filter-out tests, multiplexation won't happen and result would be:
ping_test.default
ping_test.long
ping_test.default
ping_test.short
the only way to make this work is to move the test's variants to the
global space:
hw: ...
tests:
ping_test:
variants:
filter-only: ["/tests/ping_test"]
size:
default:
...
where the filter-out is not set.
Alternatively we can specify the multiplexation domains similarly as
we did in autotest using key-word `variants`. This is what I proposed
with the list of dicts in yaml file (the `-` in the example above).
The way it works is that it discovers leafs and when it reaches `list
of dicts` (eg. "/hw/variants") it knows, that it has to multiplex
each leaf from "/hw/variants/cpu" with all leafs from
"/hw/variants/disk" with all leafs of "/hw/variants/nic_model".
It's very similar to autotests:
variants:
- cpu:
...
variants:
- disk:
...
variants:
- nic_model
The only difference is that autotest's version used the items, this
yaml version uses leafs. So for example you can use:
- hw:
- disk_format:
qcow:
format: qcow2
2:
2v3:
image_extra_params = "compat=1.1"
raw:
format = raw
...
(in autotest it had to be linear list, now you can do as complex
structure as you want with the variable inheritance and only leafs
will be used, which is kind-of cool.
Also this means you define the multiplexation so instead of
multiplexing everything which doesn't share the parent or you
explicitly don't forbid by `filter-only/out`, you specify domains and
`filter-only/out` only the parts which doesn't work together.
See comment about the mux keyword. I like this.
### Default parameters
params.get() should allow hooks for keywords. One that comes to my
mind is:
params.get("/hw/cpu/arch") which would discover available
architecture (ppc vs. x86) and modify the result based on the hook.
The hook would be able to see the raw "/hw/cpu/arch" value so it
would be able to deal with problems (or None when not specified)
Similarly we'd be able to supply different versions, eg:
params.get("/sw/qemu/has_device") would return value based on system,
rather than cfg (well in fact we can see the raw value, so in case
it's set to false, we can return false even though the qemu supports
it).
There is one problem, imagine this config:
- qemu_version: # hook
0:
filter-only: /use_devices/no
1:
2:
- use_devices: # hook
yes:
no:
- tests:
my_test:
mux: ["/use_devices"]
1) no multiplex, no problem
2) multiplex_test, no problem
3) multiplex related - in case use_devices is not available, the test
would either be executed twice for the same environment (hook
overrides yes to no), or it would skip the "yes" version or it would
raise exception that use_devices is not supported.
4) all multiplex - no problem, filter-only takes care of that.
In my opinion in this case the hook should just raise TestNAError()
so the test is skipped and we have no problem. Sometimes the other
solution based on the hook should apply. I was unable to find real
problem, but perhaps someone else could.
Defining hooks (callables) is more complex, but introduces a new level
of flexibility on what gets executed and the params. I believe the
extra complexity is more than compensated by the extra flexibility.
### Scheduler
while thinking about the variants, it struck me. Why not to use
multiplexer for sheduler too. At least as hinter. Because when you go
through the mux tree and you reach the hooks, you just need to ask
the hook what it requires for the value of the first, second, ...
leaf. Then you can add this into machine filter and schedule this
particular variant on machine, which supports all conditional
branches.
So the only difference is, that the hooks should support booth ways.
1) get default value for this machine (and respecting the raw value)
2) return requirements for given leaf (eg. ppc64, arm, mem > 64G, ...)
The scheduler idea is basically a way to find/expand variables on the
fly, based on hooks. Yes, sure. I guess we'll know of limitations when
some prototype gets implemented.
### Summarize
There are 4 separate ideas described.
The first one is about different approaches of avocado users, which
should speed up the normal execution (for most users no tree would
need to be created or just very basic one).
The second is addressing the problem of multiplexation of abusing
filter-out to not multiplex non-parent leaves. By design it should
decrease number of multiplexations.
The third describes my view of special parameters handling. If we put
the hooks into single file (or split by parent namespaces) it should
be pretty readable and describes one potential problem.
The last one is a bit similar to the 3rd one, only from different
perspective. It tries to utilize the mux tree for scheduler.
The ideas proposed sound good, I feel like they bring enough to the
table to justify going for them. Thanks!
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
