Ok, somehow I did miss this email. We don't have a way to do what you want currently, although I started to formulate how to do so in autotest. I would lean towards doing the following:
1) Create a new job status EXPECTED_FAIL or similar on client/shared/base_job.py 1) Create an ExpectedFailure() exception class inside client/shared/error.py, possibly inheriting from TestWarning 2) Create a decorator called expected_failure or whatever in some library, that wraps the called function, and if it raises an exception, it traps it and throws the expected error failure. Now the tricky parts should be checking what is necessary to make web UI and the such to display the test in a non fatal way. In autotest, even TestWarnings are considered failures, since you can't really know on a quick glance if the test passed or simply had an expected failure just by glancing at happily green painted cells. Unfortunately I can't really say I'm going to commit time to work on it, as I can barely read email and do minimal maintenance in virt-test and autotest these days with all the rush coding our new project. In other news, your comment about the run_tests() function, that's an old bug that you hit. I did send a fix in: https://github.com/autotest/virt-test/pull/1697 On Tue, Apr 8, 2014 at 6:09 AM, Plamen Dimitrov <[email protected]> wrote: > Hi guys, > > I was wondering does Autotest have a feature whereby it can detect an > expected test failure? Say we have some tests that we have developed and > that for various reasons we want to temporarily disable. Is there a better > way to do this than commenting them out? I would prefer to have them in the > test results report generated in the end, but neither as "FAIL" nor as > "PASS". I would be happy even if I could have "WARN" although a better > status would be "EXPECT_FAIL" or possibly something shorter. Do you think > we have a way around this? > > On a side note, trying to use the boolean return value of > utils_misc.run_tests() it never seems to give False although the test has > practically failed. The reason for this are the lines: > > if not current_status: > failed = True > status_dict[param_dict.get("name")] = current_status > > return not failed > > Where if the test failed, the current status will be "FAIL" which will > still resolve to "failed = False". Is this intended behavior? > > Best, > Plamen Dimitrov > > > _______________________________________________ > Virt-test-devel mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/virt-test-devel > > -- Lucas
_______________________________________________ Virt-test-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/virt-test-devel
