пн, 6 авг. 2018 г. в 19:03, Lukáš Doktor <ldok...@redhat.com>:
> Dne 6.8.2018 v 09:52 Sergey Bronnikov napsal(a): > > пт, 3 авг. 2018 г. в 14:55, Lukáš Doktor <ldok...@redhat.com>: > > > >> Well the execution is actually simpler than you think, `avocado run > >> basic.yaml` should does the trick. Anyway the yaml-file is slightly > >> corrupted and even then some things won't probably result in what you > >> expect. Let me explain it a bit and provide better example) > >> > >> Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a): > >>> Lukas, thanks for explanation! But I still have a question. > >>> > >>> Imagine we have an YAML config with a list of tests and their > parameters: > >>> > >>> ```yaml > >>> basic: !mux > >>> > >>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader > >>> timeout: 360 > >>> test_reference_resolver_args: !!python/dict > >>> vt_extra_params: > >>> - nic_model=rtl8139 > >>> - extra_params=" -cpu host" > >>> - drive_format=ahci > >>> - mem=2048 > >>> vt-machine-type: q35 > >> > >> this should be "vt_machine_type" > >> > >>> > >>> shutdown: > >>> test_reference: io-github-autotest-qemu.shutdown > >> > >> Here `test_reference_resolver_extra:` is missing. > >> > >>> vt_extra_params: > >> > >> this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`) > >> > >>> - shutdown_method = shell > >>> > >>> reboot: > >>> test_reference: io-github-autotest-qemu.reboot > >> > >> Here `test_reference_resolver_extra:` is missing. > >> > >>> vt_extra_params: > >>> - param1=1 > >>> - param2=2 > >>> > >>> stress: > >>> test_reference: io-github-autotest-qemu.linux_stress > >>> > >>> iofuzz: > >>> test_reference: io-github-autotest-qemu.iofuzz > >> > >> Here `test_reference_resolver_extra:` is missing. > >> > >>> vt_extra_params: > >>> - param1=1 > >>> - param2=2 > >>> > >>> ``` > >>> > >>> How to execute tests from this testplan? > >> > >> as mentioned earlier, it's recognized as test format, therefor simply > >> `avocado run basic.yaml`. > >> > >> Also one thing, the vt discovery is quite slow (yeah we parse cartesian > >> config on each invocation as it might have different values). You can > put > >> multiple `test_reference` into the same line and separate them by space > in > >> avocado-vt loader. But beware, it works as filters, therefor the > discovered > >> tests will be re-oredered accordingly to cartesian config (rather than > how > >> you put them there) and they'll occur only once. Therefor: > >> > >> test_reference: shutdown boot boot > >> > >> will generate: > >> > >> avocado list basic.yaml > >> VT io-github-autotest-qemu.boot > >> VT io-github-autotest-qemu.shutdown > >> > >> Anyway the speed of that is incomparably faster, which is why I use it > >> when possible. > >> > > > > > > Finally with corrections I got a following testplan: > > Dear Sergey, > > actually you unveiled a bug; fix can be found here: > https://github.com/avocado-framework/avocado-vt/pull/1696 (in my CI I > always use `--vt-extra-params` so I forgot it might not be set. Sorry about > this). > > `avocado list plan.yaml` still fails. With --verbose it awlays shows error for any testplan: "Unexpected character / on pos 7. Special chars are allowed only in variable assignation statement: 'only ../avocado-vt/plans/lukas.yaml' (<string>:1)" Common error for any YAML testplan: ~$ avocado list --verbose ../avocado-vt/plans/lukas.yaml Type Test Tag(s) NOT_A_TEST ../avocado-vt/plans/lukas.yaml: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test !VT ../avocado-vt/plans/lukas.yaml: Unexpected character / on pos 7. Special chars are allowed only in variable assignation statement: 'only ../avocado-vt/plans/lukas.yaml' (<string>:1) TEST TYPES SUMMARY ... ~$ curl https://github.com/avocado-framework/avocado/blob/master/examples/yaml_to_mux_loader/basic.yaml > ../avocado-vt/plans/basic.yaml ~$ avocado list --verbose ../avocado-vt/plans/basic.yaml Type Test Tag(s) NOT_A_TEST ../avocado-vt/plans/basic.yaml: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test !VT ../avocado-vt/plans/basic-plain.yaml: Unexpected character / on pos 7. Special chars are allowed only in variable assignation statement: 'only ../avocado-vt/plans/basic-plain.yaml' (<string>:1) TEST TYPES SUMMARY ================== !VT: 1 ACCESS_DENIED: 0 BROKEN_SYMLINK: 0 EXTERNAL: 0 INSTRUMENTED: 0 MISSING: 0 NOT_A_TEST: 1 PYUNITTEST: 0 SIMPLE: 0 VT: 0 Sergey > With this there is still one little remaining issue in your `basic.yaml`, > see below... > > > > > basic: !mux > > test_reference_resolver_class: avocado_vt.loader.VirtTestLoader > > timeout: 360 > > test_reference_resolver_extra: !!python/dict > > This is supposed to be "test_reference_resolver_args" (you're modifying > the `self.args` of Avocado for the purpose of this loader. Note this only > affects things inside the loader, so you can't for example change the > job-results-dir or so...) > > > vt_extra_params: > > - mem=2048 > > - nic_model=rtl8139 > > - drive_format=ahci > > - disk_bus=ahci > > - extra_params=" -cpu host" > > vt_machine_type: q35 > > > > shutdown: > > test_reference: io-github-autotest-qemu.shutdown > > test_reference_resolver_extra: !!python/dict > > avocado_vt_extra_params: > > - shutdown_method = shell > > > > reboot: > > test_reference: io-github-autotest-qemu.reboot > > test_reference_resolver_extra: !!python/dict > > avocado_vt_extra_params: > > - param1 = 1 > > - param2 = 2 > > > > stress: > > test_reference: io-github-autotest-qemu.linux_stress > > > > iofuzz: > > test_reference: io-github-autotest-qemu.iofuzz > > test_reference_resolver_extra: !!python/dict > > avocado_vt_extra_params: > > - param1 = 1 > > - param2 = 2 > > > > ~$ avocado variants -m testplans/basic.yaml > > Multiplex variants (4): > > Variant shutdown-edec: /run/basic/shutdown > > Variant reboot-c8ba: /run/basic/reboot > > Variant stress-fd6a: /run/basic/stress > > Variant iofuzz-e8bc: /run/basic/iofuzz > > ~$ > > > > But it is still not accepted by 'avocado {list, run}': > > ~$ avocado list testplans/basic.yaml > > Unable to resolve reference(s) 'testplans/basic.yaml' with plugins(s) > > 'file', 'vt', 'external', try running 'avocado list -V > > testplans/basic.yaml' to see the details. > > > > Do you have any idea why it may happen? > > With that single change it works well for me (tried changing the > "test_reference_resolver_args" and first test's > "test_reference_resolver_extra" and it successfully propagated to test > params). > > Regards, > Lukáš > > >> autotest==0.16.4 > > avocado-framework==63.0 > > avocado-framework-plugin-varianter-yaml-to-mux==63.0 > > avocado-plugins-vt==63.0 > > > > Sergey > > > > > >> > >> Regards, > >> Lukáš > >> > >> PS: If this happens to work for you, would you please write an > Avocado-vt > >> documentation about this? I'd prefer someone external doing so because > >> (apparently) I take many things as given... (you might as well improve > the > >> Avocado documentation, but looking at this I think the Avocado-vt one is > >> missing more) > >> > >>> > >>> $ avocado run -m basic.yaml > >>> No test references provided nor any other arguments resolved into > tests. > >>> Please double check the executed command. > >>> > >>> > >>> Sergey > >>> > >> > >> > > > > > >>> > >>> чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldok...@redhat.com>: > >>> > >>>> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a): > >>>>> Hello! > >>>>> > >>>>> I'm trying to find a way to organize my tests for running without > >>>>> specifying all of them in command-line. Imagine we have a set of > >>>> different > >>>>> tests: sleep.py, passtest.py and fail.py. To run all of them with > >>>> required > >>>>> parameters (aka variants) we may execute a command: > >>>>> avocado run sleep.py fail.py passtest.py -m basic.yaml > >>>>> > >>>>> Is there a way to run the same tests without explicit specifying > them? > >>>>> For example: avocado run -m basic.yaml > >>>>> > >>>>> Sergey > >>>>> > >>>> > >>>> Dear Sergey, > >>>> > >>>> as a matter of fact, there are multiple ways. > >>>> > >>>> 1. filter-by-tag: > >>>> > >> > http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests > >>>> > >>>> you can tag your tests eg. `sanity`, `release`, ... and then run > >>>> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests > >> tagged > >>>> `sanity` (note the granularity is per-class or even per-test-method. > >>>> > >>>> 2. yaml loader optional plugin: > >>>> > >> > http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html > >>>> > >>>> which is probably more what you're looking for, but currently only > >>>> works on python2 (but should get py3 support, hopefully, soon). It > works > >>>> similarly to yaml_to_mux varianter plugin, but allows special keys > like > >>>> `test_reference` to specify what test will be used or even > >>>> `test_reference_resolver_class` when your tests are not the default > >> ones. > >>>> This is for example useful when you want to include multiple > >>>> external-runner tests with different external-runner. There are > examples > >>>> in: > >>>> > >> > https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader > >>>> but let me share my slighly advanced example here: > >>>> > >>>> ```yaml > >>>> # Generated s390x mux-suite > >>>> !mux > >>>> qemu_make: !mux > >>>> mux_suite_test_name_prefix: 'qemu_make/git ' > >>>> check: > >>>> test_reference: > >>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check > >>>> install: > >>>> test_reference: > >>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install > >>>> > >>>> unit: > >>>> variants: !mux > >>>> test_reference_resolver_class: > >> avocado.core.loader.ExternalLoader > >>>> timeout: 360 > >>>> rpm-tcg: > >>>> mux_suite_test_name_prefix: 'unit/rpm/tcg ' > >>>> test_reference_resolver_extra: !!python/dict > >>>> loader_options: > >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg > >>>> rpm-kvm: > >>>> mux_suite_test_name_prefix: 'unit/rpm/kvm ' > >>>> test_reference_resolver_extra: !!python/dict > >>>> loader_options: > >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm > >>>> git-tcg: > >>>> mux_suite_test_name_prefix: 'unit/git/tcg ' > >>>> test_reference_resolver_extra: !!python/dict > >>>> loader_options: > >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg > >>>> git-kvm: > >>>> mux_suite_test_name_prefix: 'unit/git/kvm ' > >>>> test_reference_resolver_extra: !!python/dict > >>>> loader_options: > >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm > >>>> tests: !mux > >>>> cmm: > >>>> test_reference: cmm > >>>> diag10: > >>>> test_reference: diag10 > >>>> emulator: > >>>> test_reference: emulator > >>>> gs: > >>>> test_reference: gs > >>>> iep: > >>>> test_reference: iep > >>>> intercept: > >>>> test_reference: intercept > >>>> pfmf: > >>>> test_reference: pfmf > >>>> selftest-setup: > >>>> test_reference: selftest-setup > >>>> sieve: > >>>> test_reference: sieve > >>>> skey: > >>>> test_reference: skey > >>>> sthyi: > >>>> test_reference: sthyi > >>>> vector: > >>>> test_reference: vector > >>>> functional: !mux > >>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader > >>>> s390x-rpm-libvirt/dvd: > >>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd ' > >>>> test_reference: > >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads > >>>> io-github-autotest-qemu.boot remove_guest.without_disk > >>>> test_reference_resolver_args: !!python/dict > >>>> vt_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_type: libvirt > >>>> vt_extra_params: > >>>> - automem=no > >>>> - url=$URL > >>>> - vga=none > >>>> s390x-rpm-libvirt/url: > >>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url ' > >>>> test_reference: > >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads > >>>> io-github-autotest-qemu.boot remove_guest.without_disk > >>>> test_reference_resolver_args: !!python/dict > >>>> vt_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_type: libvirt > >>>> vt_extra_params: > >>>> - automem=no > >>>> - url=$URL > >>>> - vga=none > >>>> s390x-rpm-dvd: > >>>> mux_suite_test_name_prefix: 'functional/rpm/dvd ' > >>>> test_reference: > >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads > boot > >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec > >>>> migrate.default.tcp.with_post_copy > >>>> test_reference_resolver_args: !!python/dict > >>>> vt_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_extra_params: > >>>> - automem=no > >>>> - url=$URL > >>>> - ping_pong=5 > >>>> - vga=none > >>>> s390x-rpm-url: > >>>> mux_suite_test_name_prefix: 'functional/rpm/url ' > >>>> test_reference: > >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot > >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec > >>>> migrate.default.tcp.with_post_copy > >>>> test_reference_resolver_args: !!python/dict > >>>> vt_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm > >>>> vt_extra_params: > >>>> - automem=no > >>>> - url=$URL > >>>> - ping_pong=5 > >>>> - vga=none > >>>> s390x-git-dvd: > >>>> mux_suite_test_name_prefix: 'functional/git/dvd ' > >>>> test_reference: > >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads > boot > >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec > >>>> migrate.default.tcp.with_post_copy > >>>> test_reference_resolver_args: !!python/dict > >>>> vt_qemu_bin: > >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x > >>>> vt_dst_qemu_bin: > >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x > >>>> vt_extra_params: > >>>> - automem=no > >>>> - url=$URL > >>>> - ping_pong=5 > >>>> - vga=none > >>>> s390x-git-url: > >>>> mux_suite_test_name_prefix: 'functional/git/url ' > >>>> test_reference: > >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot > >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec > >>>> migrate.default.tcp.with_post_copy > >>>> test_reference_resolver_args: !!python/dict > >>>> vt_qemu_bin: > >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x > >>>> vt_dst_qemu_bin: > >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x > >>>> vt_extra_params: > >>>> - automem=no > >>>> - url=$URL > >>>> - ping_pong=5 > >>>> - vga=none > >>>> ``` > >>>> > >>>> (feel free to ask for details, but basically it uses SIMPLE tests, > >>>> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always > >> receives > >>>> all the params so you can override the standard arguments. If you > >> specify > >>>> "test_reference_resolver_args" it also updates the loader parameters, > >> which > >>>> is what I use for Avocado-vt tests). > >>>> > >>>> Theoretically there is yet another way and that is to create directory > >>>> "sanity" and put symlinks to actual tests there, but the granularity > >> there > >>>> would be only per-test-class, which usually does not suffice. > >>>> > >>>> Anyway, hopefully at least one method suits you, but feel free to > >>>> elaborate more, we can always improve. > >>>> > >>>> Kind regards, > >>>> Lukáš > >>>> > >>>> > >>> > >> > >> > >> > > > > >