28.08.2019. 23.24, "Cleber Rosa" <cr...@redhat.com> је написао/ла: > > On Thu, Aug 22, 2019 at 07:59:07PM +0200, Aleksandar Markovic wrote: > > 22.08.2019. 05.15, "Aleksandar Markovic" <aleksandar.m.m...@gmail.com> је > > написао/ла: > > > > > > > > > 21.08.2019. 23.00, "Eduardo Habkost" <ehabk...@redhat.com> је написао/ла: > > > > > > > > On Wed, Aug 21, 2019 at 10:27:11PM +0200, Aleksandar Markovic wrote: > > > > > 02.08.2019. 17.37, "Aleksandar Markovic" < > > aleksandar.marko...@rt-rk.com> је > > > > > написао/ла: > > > > > > > > > > > > From: Aleksandar Markovic <amarko...@wavecomp.com> > > > > > > > > > > > > This little series improves linux_ssh_mips_malta.py, both in the > > sense > > > > > > of code organization and in the sense of quantity of executed tests. > > > > > > > > > > > > > > > > Hello, all. > > > > > > > > > > I am going to send a new version in few days, and I have a question > > for > > > > > test team: > > > > > > > > > > Currently, the outcome of the script execition is either PASS:1 > > FAIL:0 or > > > > > PASS:0 FAIL:1. But the test actually consists of several subtests. Is > > there > > > > > any way that this single Python script considers these subtests as > > separate > > > > > tests (test cases), reporting something like PASS:12 FAIL:7? If yes, > > what > > > > > would be the best way to achieve that? > > > > > > > > If you are talking about each test_*() method, they are already > > > > treated like separate tests. If you mean treating each > > > > ssh_command_output_contains() call as a separate test, this might > > > > be difficult. > > > > > > > > > > Yes, I meant the latter one, individual code segments involving an > > invocation of ssh_command_output_contains() instance being treated as > > separate tests. > > > > > > > Hello, Cleber, > > > > I am willing to rewamp python file structure if needed. > > > > The only thing I feel a little unconfortable is if I need to reboot the > > virtual machine for each case of ssh_command_output_contains(). > > > > Hi Aleksandar, > > The short answer is that Avocado provides no way to report "subtest" > statuses (as a formal concept), neither does the current > "avocado_qemu" infrastructure allow for management of VMs across > tests. The later is an Avocado-VT feature, and it to be honest it > brings a good deal of problems in itself, which we decided to avoid > here. > > About the lack of subtests, we (the autotest project, then the Avocado > project) found that this concept, to be well applied, need more than > we could deal with initially. For instance, Avocado has the concept > of "pre_test" and "post_test" hooks, with that, should those be > applied to subtests as well? Also, there's support for capturing > system information (a feature called sysinfo) before and after the > tests... again, should it be applied to subtests? Avocado also stores > a well defined results directory, and we'd have to deal with something > like that for subtests. With regards to the variants feature, should > they also be applied to subtests? The list of questions goes on and > on. > > The fact that one test should not be able (as much as possible) to > influence another test also comes into play in our initial decision > to avoid subtests. > > IMO, the best way to handle this is to either keep a separate logger > with the test progress: > > https://avocado-framework.readthedocs.io/en/71.0/WritingTests.html#advanced-logging-capabilities > > With a change similar to: > > --- > diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py > index 509ff929cf..0683586c35 100644 > --- a/tests/acceptance/linux_ssh_mips_malta.py > +++ b/tests/acceptance/linux_ssh_mips_malta.py > @@ -17,6 +17,7 @@ from avocado_qemu import Test > from avocado.utils import process > from avocado.utils import archive > > +progress_log = logging.getLogger("progress") > > class LinuxSSH(Test): > > @@ -149,6 +150,7 @@ class LinuxSSH(Test): > stdout, _ = self.ssh_command(cmd) > for line in stdout: > if exp in line: > + progress_log.info('Check successful for "%s"', cmd) > break > else: > self.fail('"%s" output does not contain "%s"' % (cmd, exp)) > --- > > You could run tests with: > > $ ./tests/venv/bin/avocado --show=console,progress run --store-logging-stream progress -- tests/acceptance/linux_ssh_mips_malta.py > > And at the same time: > > $ tail -f ~/avocado/job-results/latest/progress.INFO > 17:20:44 INFO | Check successful for "uname -a" > 17:20:44 INFO | Check successful for "cat /proc/cpuinfo" > ... > > I hope this helps somehow. > > Best regards, > - Cleber. >
Thanks, Cleber, for your detailed response. I'll use whatever is available, along the lines you highligted. I will most likely gradually modify this test until I find the sweet spot where I am satisfied with test behavior and reporting, but also everything fits well into Avocado framework. Thanks again, both to you and Eduardo, Aleksandar > > Grateful in advance, > > Aleksandar > > > > > > Cleber, is there something already available in the Avocado API > > > > that would help us report more fine-grained results inside each > > > > test case? > > > > > > > > > > Thanks, that would be a better way of expressing my question. > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > Aleksandar > > > > > > > > > > > Aleksandar Markovic (2): > > > > > > tests/acceptance: Refactor and improve reporting in > > > > > > linux_ssh_mips_malta.py > > > > > > tests/acceptance: Add new test cases in linux_ssh_mips_malta.py > > > > > > > > > > > > tests/acceptance/linux_ssh_mips_malta.py | 81 > > > > > ++++++++++++++++++++++++++------ > > > > > > 1 file changed, 66 insertions(+), 15 deletions(-) > > > > > > > > > > > > -- > > > > > > 2.7.4 > > > > > > > > > > > > > > > > > > > > -- > > > > Eduardo