Dne 11. 09. 19 v 19:14 Brian J. Murrell napsal(a): > On Wed, 2019-09-11 at 19:09 +0200, Lukáš Doktor wrote: >> >> Hello Brian, > > Hi Lukáš, > >> this is expected behavior. It's because the test hasn't actually >> started, therefor it can not fail. > > Ahh. But it has, if setUp() is running code from the module that is > being tested. In this case the code that is in the module that setUp() > is running is code that needs to run prior to every test. But it's > still code under test from the module being tested. > >> This is deliberate as setUp issues should not be related to the >> module under testing. > > Unless setUp() is using some of the code being tested to prepare for > the tests. >
Well it used to be an issue in several of our tests, but according to academic description, it shouldn't and adopting this mindset actually improved overlooking of the results. Basically when writing tests, one should focus on particular module-under-testing and the test itself. For example (I come from virtualization background) one wants to run "stress -c 10" inside a VM and see, whether it utilizes 10 pinned CPUs on host. In this case module under testing is Qemu and host OS. So in case of: 1. different CPU usage -> FAIL 2. non-zero stress exit code -> ERROR (setup/environment issue) 3. fail to boot VM -> ERROR (setup) 4. not enough CPUs -> SKIP (no setup should be performed, avocado exit status is 0 as no tests failed - this is usually controversial, but in academic world it works and after long discussion we used this approach...) 5. VM crashes -> ERROR (setup/environment issue) Only the (1) should be discussed with the Qemu developer, the rest are setup issues and should be tackled by QA/CI guys. In another test one wants to see that "stress -c 10" won't crash. Setup is similar, but this time: 1. different CPU usage -> PASS/ERROR (depends whether we care) 2. non-zero stress exit code -> FAIL/ERROR (depends on whether we detect SIGABRT or just generic failure) 3. fail to boot VM -> ERROR (setup) 4. not enough CPUs -> We don't care 5. VM crashes -> FAIL Anyway this is what we adopted and it worked for us. Honestly I have to say that it took some time, but I learned to like it. @Cleber, do you have anything to add? Regards, Lukáš > Cheers, > b. > >>>
signature.asc
Description: OpenPGP digital signature
