Jose, This is very good. Would it make sense to add this information to the wiki?
David On Fri, Jul 29, 2016 at 4:09 AM, Jose Lausuch <[email protected]> wrote: > Hi, > > > > In order to provide a better user experience and useful feedback from the > daily runs in Jenkins, these features have been implemented in Functest > recently: > > > > *“Blocking flag” for each test case:* > > For each test case executed in Functest, there is a flag called “blocking” > that can be set to True or False. If the test fails and the flag is set > ‘true’, Functest will stop the execution of the rest of the tests in the > daily loop exiting with error. If the test fails but the flag is set to > false, it will *mark the Job as failed*, but it will allow the execution > of the remaining test cases. > > https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml > > The purpose of this is to allow the critical test cases like the > healthcheck/vping be a “gate” or a filter i.e. if you can’t create VMs, why > should you bother in testing sfc or any other feature? Some other tests > like Tempest, will allow the execution of the preceding tests even though > if there is a failure. > > > > > > *Summary report:* > > Some of you gave us some feedback about the confusing output given by > Functest in Jekins if a job has failed. This has been improved a lot, > providing clear output messages to understand what is happening. But now > you don’t even need to go through all that output, you just need to scroll > down and at the bottom of the page you will see the “Functest report” to > check what has worked/failed along with some other useful information. > > Example here: > https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/261/console > > > > > > *Black listing tests in Tempest:* > > If you know that in a certain scenario/installer, a tempest test case > fails, you can now blacklist the test case in this yaml: > > > https://git.opnfv.org/cgit/functest/tree/testcases/OpenStack/tempest/custom_tests/blacklist.txt > > For example, for bgpvpn we have deactivated the tests that deal with > floating ips due to lacking support for floating IPs in ODL VPNService > (which will be fixed in the future). > > So, if you know that a known bug will make a test case fail, you can add > the test in this list and it will be skipped. > > NOTE: this is not meant to avoid running tests that fail without a reason. > It should be justified (known bug, feature lack of support, …). > > > > > > You’ll see a lot of red in Jenkins, and that’s because Functest has become > a bit more strict when giving the final status. It will *only return blue > if all the tests succeeded* according to the defined success criteria, > not only if the execution of the test script succeeded (even though the > test failed). > > For example: > > > https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/261/console > > > https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/260/console > > The exit status is -1 because there was >0 failures in Tempest-smoke, and > the success criteria for Tempest-smoke is 100%. > > > > This *strictness* has been changed like this after some feedback > collected from the community, in particular from the installer teams. > > > > > > > > Thanks, > > > > *JOSE LAUSUCH* > Senior Systems Designer > *Ericsson* > > > > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > -- *David McBride* Release Manager, OPNFV Mobile: +1.805.276.8018 Email/Google Talk: [email protected] Skype: davidjmcbride1 IRC: dmcbride
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
