It would be good to make this clearer. The definitions below are a great start but I am not sure they are complete. Perhaps there are three categories:
Failures: These indicate that the application under test has a bug which needs to be fixed (or the test has the incorrect assumption and it needs to be fixed). (Not should flip these) ExecutionException (Errors?): These indicate that the test environment has a problem (e.g. incorrect parameters to a step). No amount of "fixing" of your application under test will help resolve these. (Not should leave these alone). Unknown: These are errors that could indicate that the AUT has a bug or could indicate that the test is incorrect or could indicate a problem with the environment or a bug in some underlying 3rd party technology. To illustrate the point, missing a link on a page (which 99% of the time is probably a failure) could also occur if some unsupported JS was supposed to add the link to the page. There is nothing wrong with my application or my tests. Why should it fail? If a page request can't be executed correctly (ex: http error code received, or a socket timeout occurs, or a JS error occurs etc.) it could be an environmental issue or it could indicate a bug in my application - e.g. when I click a link it is trying to go to a secure page when I am not logged in. In an ideal world, these should sometimes fail - but there is typically no way to know. Perhaps they should be marked as unknown? Before we jump to using three types of errors we should look at how junit tackles this issue. In JUnit 3.x they have errors and failures. In JUnit 4 (actually I didn't check the final version but as of several months ago) they found it too hard to try to make this distinction and I believe now only have tests which either pass or fail. Personally, I think two categories can be made to work. We should probably just review our usage (on this list) when we want to make significant changes and for the unknown cases think through whether it is 80-99% likely to be one of error or failure and make it that way. Cheers, Paul. Marc Guillemot wrote:
Hi, we currently throw StepFailedException or StepExecutionException when a test doesn't pass but I'm not sure to know exactly what we want to differentiate with these 2 exceptions or if we do it consistently. For me a StepFailedException should be reserved to cases like: - verification steps: when the verification fails (this is what we do) - action steps: when the action can't be done, for instance no link can be found in a clickLink (this is what we do too) All (?) other cases should use a StepExecution exception, for instance when: - the parameters of a step are incorrect (this is already what is done) - a page request can't be executed correctly (ex: http error code received, or a socket timeout occurs, or a JS error occurs) => this is NOT what we currently do. Any thoughts? Marc. _______________________________________________ WebTest mailing list [email protected] http://lists.canoo.com/mailman/listinfo/webtest
_______________________________________________ WebTest mailing list [email protected] http://lists.canoo.com/mailman/listinfo/webtest
