Dne 7.8.2018 v 11:16 Sergey Bronnikov napsal(a): > пн, 6 авг. 2018 г. в 19:03, Lukáš Doktor <ldok...@redhat.com>: > >> Dne 6.8.2018 v 09:52 Sergey Bronnikov napsal(a): >>> пт, 3 авг. 2018 г. в 14:55, Lukáš Doktor <ldok...@redhat.com>: >>> >>>> Well the execution is actually simpler than you think, `avocado run >>>> basic.yaml` should does the trick. Anyway the yaml-file is slightly >>>> corrupted and even then some things won't probably result in what you >>>> expect. Let me explain it a bit and provide better example) >>>> >>>> Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a): >>>>> Lukas, thanks for explanation! But I still have a question. >>>>> >>>>> Imagine we have an YAML config with a list of tests and their >> parameters: >>>>> >>>>> ```yaml >>>>> basic: !mux >>>>> >>>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader >>>>> timeout: 360 >>>>> test_reference_resolver_args: !!python/dict >>>>> vt_extra_params: >>>>> - nic_model=rtl8139 >>>>> - extra_params=" -cpu host" >>>>> - drive_format=ahci >>>>> - mem=2048 >>>>> vt-machine-type: q35 >>>> >>>> this should be "vt_machine_type" >>>> >>>>> >>>>> shutdown: >>>>> test_reference: io-github-autotest-qemu.shutdown >>>> >>>> Here `test_reference_resolver_extra:` is missing. >>>> >>>>> vt_extra_params: >>>> >>>> this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`) >>>> >>>>> - shutdown_method = shell >>>>> >>>>> reboot: >>>>> test_reference: io-github-autotest-qemu.reboot >>>> >>>> Here `test_reference_resolver_extra:` is missing. >>>> >>>>> vt_extra_params: >>>>> - param1=1 >>>>> - param2=2 >>>>> >>>>> stress: >>>>> test_reference: io-github-autotest-qemu.linux_stress >>>>> >>>>> iofuzz: >>>>> test_reference: io-github-autotest-qemu.iofuzz >>>> >>>> Here `test_reference_resolver_extra:` is missing. >>>> >>>>> vt_extra_params: >>>>> - param1=1 >>>>> - param2=2 >>>>> >>>>> ``` >>>>> >>>>> How to execute tests from this testplan? >>>> >>>> as mentioned earlier, it's recognized as test format, therefor simply >>>> `avocado run basic.yaml`. >>>> >>>> Also one thing, the vt discovery is quite slow (yeah we parse cartesian >>>> config on each invocation as it might have different values). You can >> put >>>> multiple `test_reference` into the same line and separate them by space >> in >>>> avocado-vt loader. But beware, it works as filters, therefor the >> discovered >>>> tests will be re-oredered accordingly to cartesian config (rather than >> how >>>> you put them there) and they'll occur only once. Therefor: >>>> >>>> test_reference: shutdown boot boot >>>> >>>> will generate: >>>> >>>> avocado list basic.yaml >>>> VT io-github-autotest-qemu.boot >>>> VT io-github-autotest-qemu.shutdown >>>> >>>> Anyway the speed of that is incomparably faster, which is why I use it >>>> when possible. >>>> >>> >>> >>> Finally with corrections I got a following testplan: >> >> Dear Sergey, >> >> actually you unveiled a bug; fix can be found here: >> https://github.com/avocado-framework/avocado-vt/pull/1696 (in my CI I >> always use `--vt-extra-params` so I forgot it might not be set. Sorry about >> this). >> >> > `avocado list plan.yaml` still fails. With --verbose it awlays shows error > for any testplan: > "Unexpected character / on pos 7. Special chars are allowed only in > variable assignation statement: 'only ../avocado-vt/plans/lukas.yaml' > (<string>:1)" > > Common error for any YAML testplan: > ~$ avocado list --verbose ../avocado-vt/plans/lukas.yaml > Type > Test > Tag(s) > NOT_A_TEST ../avocado-vt/plans/lukas.yaml: Not an INSTRUMENTED > (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE > (executable) test > !VT ../avocado-vt/plans/lukas.yaml: Unexpected character / on pos > 7. Special chars are allowed only in variable assignation statement: 'only > ../avocado-vt/plans/lukas.yaml' (<string>:1) > > TEST TYPES SUMMARY > ... > > ~$ curl > https://github.com/avocado-framework/avocado/blob/master/examples/yaml_to_mux_loader/basic.yaml
I assume you used `curl https://raw.githubusercontent.com/avocado-framework/avocado/master/examples/yaml_to_mux_loader/basic.yaml` >> ../avocado-vt/plans/basic.yaml > ~$ avocado list --verbose ../avocado-vt/plans/basic.yaml > Type > Test > Tag(s) > NOT_A_TEST ../avocado-vt/plans/basic.yaml: Not an INSTRUMENTED > (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE > (executable) test > !VT ../avocado-vt/plans/basic-plain.yaml: Unexpected character / on > pos 7. Special chars are allowed only in variable assignation statement: > 'only ../avocado-vt/plans/basic-plain.yaml' (<string>:1) > This is correct, it's NOT_A_TEST and !VT. Anyway there should be the "loader_yaml" installed which should understand that format and discover the tests defined by the varianter. What is the output of: $ avocado list --loaders help Unknown loader 'help'. Available plugins are: glib: glib.GLIB golang: golang.GOLANG yaml_testsuite: robot: robot.ROBOT vt: vt.VT file: file.SIMPLE, file.INSTRUMENTED, file.PyUNITTEST external: external.EXTERNAL If the "yaml_testsuite is not there you need to install it (cd optional_plugins/loader_yaml && python setup.py install). If you use `make link` it should be there but `make install` won't install the optional plugins. $ avocado list --verbose examples/mux-lukas.yaml Type Test Tag(s) NOT_A_TEST examples/mux-lukas.yaml: Not an INSTRUMENTED (avocado.Test based), PyUNITTEST (unittest.TestCase based) or SIMPLE (executable) test !GLIB examples/mux-lukas.yaml: No GLib-like tests found !GOLANG examples/mux-lukas.yaml: Go binary not found. INSTRUMENTED passtest.py:PassTest.test fast INSTRUMENTED sleeptest.py:SleepTest.test INSTRUMENTED failtest.py:FailTest.test failure_expected INSTRUMENTED failtest.py:FailTest.test failure_expected INSTRUMENTED failtest.py:FailTest.test failure_expected !ROBOT examples/mux-lukas.yaml: Unsupported file format 'yaml'. !VT examples/mux-lukas.yaml: Unexpected character / on pos 13. Special chars are allowed only in variable assignation statement: 'only examples/mux-lukas.yaml' (<string>:1) $ avocado list examples/mux-lukas.yaml INSTRUMENTED passtest.py:PassTest.test INSTRUMENTED sleeptest.py:SleepTest.test INSTRUMENTED failtest.py:FailTest.test INSTRUMENTED failtest.py:FailTest.test INSTRUMENTED failtest.py:FailTest.test Hopefully it's just the missing plugin, Lukáš > TEST TYPES SUMMARY > ================== > !VT: 1 > ACCESS_DENIED: 0 > BROKEN_SYMLINK: 0 > EXTERNAL: 0 > INSTRUMENTED: 0 > MISSING: 0 > NOT_A_TEST: 1 > PYUNITTEST: 0 > SIMPLE: 0 > VT: 0 > > Sergey > > >> With this there is still one little remaining issue in your `basic.yaml`, >> see below... >> >>> >>> basic: !mux >>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader >>> timeout: 360 >>> test_reference_resolver_extra: !!python/dict >> >> This is supposed to be "test_reference_resolver_args" (you're modifying >> the `self.args` of Avocado for the purpose of this loader. Note this only >> affects things inside the loader, so you can't for example change the >> job-results-dir or so...) >> >>> vt_extra_params: >>> - mem=2048 >>> - nic_model=rtl8139 >>> - drive_format=ahci >>> - disk_bus=ahci >>> - extra_params=" -cpu host" >>> vt_machine_type: q35 >>> >>> shutdown: >>> test_reference: io-github-autotest-qemu.shutdown >>> test_reference_resolver_extra: !!python/dict >>> avocado_vt_extra_params: >>> - shutdown_method = shell >>> >>> reboot: >>> test_reference: io-github-autotest-qemu.reboot >>> test_reference_resolver_extra: !!python/dict >>> avocado_vt_extra_params: >>> - param1 = 1 >>> - param2 = 2 >>> >>> stress: >>> test_reference: io-github-autotest-qemu.linux_stress >>> >>> iofuzz: >>> test_reference: io-github-autotest-qemu.iofuzz >>> test_reference_resolver_extra: !!python/dict >>> avocado_vt_extra_params: >>> - param1 = 1 >>> - param2 = 2 >>> >>> ~$ avocado variants -m testplans/basic.yaml >>> Multiplex variants (4): >>> Variant shutdown-edec: /run/basic/shutdown >>> Variant reboot-c8ba: /run/basic/reboot >>> Variant stress-fd6a: /run/basic/stress >>> Variant iofuzz-e8bc: /run/basic/iofuzz >>> ~$ >>> >>> But it is still not accepted by 'avocado {list, run}': >>> ~$ avocado list testplans/basic.yaml >>> Unable to resolve reference(s) 'testplans/basic.yaml' with plugins(s) >>> 'file', 'vt', 'external', try running 'avocado list -V >>> testplans/basic.yaml' to see the details. >>> >>> Do you have any idea why it may happen? >> >> With that single change it works well for me (tried changing the >> "test_reference_resolver_args" and first test's >> "test_reference_resolver_extra" and it successfully propagated to test >> params). >> > > > >> Regards, >> Lukáš >> >>>> autotest==0.16.4 >>> avocado-framework==63.0 >>> avocado-framework-plugin-varianter-yaml-to-mux==63.0 >>> avocado-plugins-vt==63.0 >>> >>> Sergey >>> >>> >>>> >>>> Regards, >>>> Lukáš >>>> >>>> PS: If this happens to work for you, would you please write an >> Avocado-vt >>>> documentation about this? I'd prefer someone external doing so because >>>> (apparently) I take many things as given... (you might as well improve >> the >>>> Avocado documentation, but looking at this I think the Avocado-vt one is >>>> missing more) >>>> >>>>> >>>>> $ avocado run -m basic.yaml >>>>> No test references provided nor any other arguments resolved into >> tests. >>>>> Please double check the executed command. >>>>> >>>>> >>>>> Sergey >>>>> >>>> >>>> >>> >>> >>>>> >>>>> чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldok...@redhat.com>: >>>>> >>>>>> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a): >>>>>>> Hello! >>>>>>> >>>>>>> I'm trying to find a way to organize my tests for running without >>>>>>> specifying all of them in command-line. Imagine we have a set of >>>>>> different >>>>>>> tests: sleep.py, passtest.py and fail.py. To run all of them with >>>>>> required >>>>>>> parameters (aka variants) we may execute a command: >>>>>>> avocado run sleep.py fail.py passtest.py -m basic.yaml >>>>>>> >>>>>>> Is there a way to run the same tests without explicit specifying >> them? >>>>>>> For example: avocado run -m basic.yaml >>>>>>> >>>>>>> Sergey >>>>>>> >>>>>> >>>>>> Dear Sergey, >>>>>> >>>>>> as a matter of fact, there are multiple ways. >>>>>> >>>>>> 1. filter-by-tag: >>>>>> >>>> >> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests >>>>>> >>>>>> you can tag your tests eg. `sanity`, `release`, ... and then run >>>>>> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests >>>> tagged >>>>>> `sanity` (note the granularity is per-class or even per-test-method. >>>>>> >>>>>> 2. yaml loader optional plugin: >>>>>> >>>> >> http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html >>>>>> >>>>>> which is probably more what you're looking for, but currently only >>>>>> works on python2 (but should get py3 support, hopefully, soon). It >> works >>>>>> similarly to yaml_to_mux varianter plugin, but allows special keys >> like >>>>>> `test_reference` to specify what test will be used or even >>>>>> `test_reference_resolver_class` when your tests are not the default >>>> ones. >>>>>> This is for example useful when you want to include multiple >>>>>> external-runner tests with different external-runner. There are >> examples >>>>>> in: >>>>>> >>>> >> https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader >>>>>> but let me share my slighly advanced example here: >>>>>> >>>>>> ```yaml >>>>>> # Generated s390x mux-suite >>>>>> !mux >>>>>> qemu_make: !mux >>>>>> mux_suite_test_name_prefix: 'qemu_make/git ' >>>>>> check: >>>>>> test_reference: >>>>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check >>>>>> install: >>>>>> test_reference: >>>>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install >>>>>> >>>>>> unit: >>>>>> variants: !mux >>>>>> test_reference_resolver_class: >>>> avocado.core.loader.ExternalLoader >>>>>> timeout: 360 >>>>>> rpm-tcg: >>>>>> mux_suite_test_name_prefix: 'unit/rpm/tcg ' >>>>>> test_reference_resolver_extra: !!python/dict >>>>>> loader_options: >>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg >>>>>> rpm-kvm: >>>>>> mux_suite_test_name_prefix: 'unit/rpm/kvm ' >>>>>> test_reference_resolver_extra: !!python/dict >>>>>> loader_options: >>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm >>>>>> git-tcg: >>>>>> mux_suite_test_name_prefix: 'unit/git/tcg ' >>>>>> test_reference_resolver_extra: !!python/dict >>>>>> loader_options: >>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg >>>>>> git-kvm: >>>>>> mux_suite_test_name_prefix: 'unit/git/kvm ' >>>>>> test_reference_resolver_extra: !!python/dict >>>>>> loader_options: >>>>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm >>>>>> tests: !mux >>>>>> cmm: >>>>>> test_reference: cmm >>>>>> diag10: >>>>>> test_reference: diag10 >>>>>> emulator: >>>>>> test_reference: emulator >>>>>> gs: >>>>>> test_reference: gs >>>>>> iep: >>>>>> test_reference: iep >>>>>> intercept: >>>>>> test_reference: intercept >>>>>> pfmf: >>>>>> test_reference: pfmf >>>>>> selftest-setup: >>>>>> test_reference: selftest-setup >>>>>> sieve: >>>>>> test_reference: sieve >>>>>> skey: >>>>>> test_reference: skey >>>>>> sthyi: >>>>>> test_reference: sthyi >>>>>> vector: >>>>>> test_reference: vector >>>>>> functional: !mux >>>>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader >>>>>> s390x-rpm-libvirt/dvd: >>>>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd ' >>>>>> test_reference: >>>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads >>>>>> io-github-autotest-qemu.boot remove_guest.without_disk >>>>>> test_reference_resolver_args: !!python/dict >>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_type: libvirt >>>>>> vt_extra_params: >>>>>> - automem=no >>>>>> - url=$URL >>>>>> - vga=none >>>>>> s390x-rpm-libvirt/url: >>>>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url ' >>>>>> test_reference: >>>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads >>>>>> io-github-autotest-qemu.boot remove_guest.without_disk >>>>>> test_reference_resolver_args: !!python/dict >>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_type: libvirt >>>>>> vt_extra_params: >>>>>> - automem=no >>>>>> - url=$URL >>>>>> - vga=none >>>>>> s390x-rpm-dvd: >>>>>> mux_suite_test_name_prefix: 'functional/rpm/dvd ' >>>>>> test_reference: >>>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads >> boot >>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>>>> migrate.default.tcp.with_post_copy >>>>>> test_reference_resolver_args: !!python/dict >>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_extra_params: >>>>>> - automem=no >>>>>> - url=$URL >>>>>> - ping_pong=5 >>>>>> - vga=none >>>>>> s390x-rpm-url: >>>>>> mux_suite_test_name_prefix: 'functional/rpm/url ' >>>>>> test_reference: >>>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot >>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>>>> migrate.default.tcp.with_post_copy >>>>>> test_reference_resolver_args: !!python/dict >>>>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>>>> vt_extra_params: >>>>>> - automem=no >>>>>> - url=$URL >>>>>> - ping_pong=5 >>>>>> - vga=none >>>>>> s390x-git-dvd: >>>>>> mux_suite_test_name_prefix: 'functional/git/dvd ' >>>>>> test_reference: >>>>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads >> boot >>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>>>> migrate.default.tcp.with_post_copy >>>>>> test_reference_resolver_args: !!python/dict >>>>>> vt_qemu_bin: >>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>>>> vt_dst_qemu_bin: >>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>>>> vt_extra_params: >>>>>> - automem=no >>>>>> - url=$URL >>>>>> - ping_pong=5 >>>>>> - vga=none >>>>>> s390x-git-url: >>>>>> mux_suite_test_name_prefix: 'functional/git/url ' >>>>>> test_reference: >>>>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot >>>>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>>>> migrate.default.tcp.with_post_copy >>>>>> test_reference_resolver_args: !!python/dict >>>>>> vt_qemu_bin: >>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>>>> vt_dst_qemu_bin: >>>>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>>>> vt_extra_params: >>>>>> - automem=no >>>>>> - url=$URL >>>>>> - ping_pong=5 >>>>>> - vga=none >>>>>> ``` >>>>>> >>>>>> (feel free to ask for details, but basically it uses SIMPLE tests, >>>>>> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always >>>> receives >>>>>> all the params so you can override the standard arguments. If you >>>> specify >>>>>> "test_reference_resolver_args" it also updates the loader parameters, >>>> which >>>>>> is what I use for Avocado-vt tests). >>>>>> >>>>>> Theoretically there is yet another way and that is to create directory >>>>>> "sanity" and put symlinks to actual tests there, but the granularity >>>> there >>>>>> would be only per-test-class, which usually does not suffice. >>>>>> >>>>>> Anyway, hopefully at least one method suits you, but feel free to >>>>>> elaborate more, we can always improve. >>>>>> >>>>>> Kind regards, >>>>>> Lukáš >>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>> >> >> >> >
signature.asc
Description: OpenPGP digital signature
