I am not sure there is a fully deterministic answer but that doesn't
mean we can't try. I'll give two definitions:
[1] (close to Marc's original rules):
StepFailedException: Should indicate that the application under test (AUT)
worked with no unexpected errors but didn't meet the expectations outlined
in the test. (Either the AUT has a bug or AUT is working but while I have
correctly set up the test I have supplied the wrong expectations).
If it is not possible to be sure that this holds but we expect it
holds > 80% of the time then use this exception.
StepExecutionException: All other cases
<not> only applies to StepFailedException as per current behaviour.
[2] Return SFE whenever it might make sense to use <not> to test
that an AUT doesn't do something.
Application of these rules below.
Marc Guillemot wrote:
let's start making it clearer (Paul, your explaination is interesting but not
deterministic ;-()
What should be thrown when:
- step has invalid parameters
[1] and [2] indicate SEE: Test can't proceed until params fixed, nothing can be
said about whether test expectations are met or not
- step "context" is not present (ex: no current page for a step working on a
page)
[1] and [2] indicate SEE: usually indicates a bug in the test
- http error code is received
Using [2] we would go with SFE. Using [1] it could be SFE or SEE.
A normal scenario would be we are trying to test a page
and there is an environmental issue - we forget to start the test server.
This should be SEE. Another scenario is we are trying to ensure that we get
a permission denied when we visit a protected page without logging in.
Or perhaps we used to have a link to something on every page but that
link has changed and now we want to check that no page has the legacy link.
In both these case being able to use <not> would be handy.
We could make this SEE and the workaround for the other scenario would
be to set ThrowExceptionOnFailingStatusCode to false and check the content
on the error pages. I guess the other alternative is to have similar
booleans (like ThrowExceptionOnFailingStatusCode) ourselves which work
at the WebTest level rather than HtmlUnit level. We could set the flag
to get either SEE or SFE invoked.
- JavaScript error occured
If the JavaScript is caused by a parse error because my JavaScript
is not well formed then I would expect SEE. There possibly could be
some cases where SFE might be useful but I can't think of any.
I would probably always set this one to SEE. If we find a case where
SFE is desirable we could always make a flag which you could set to
get the SFE behaviour. So, [1] and [2] probably say SEE.
- in a verification, the actual value and the expected one differ
[1] and [2] indicate probably always SFE.
Which of these cases should be "catched" by a wrapping <not>?
only SFE
Socket timeout is a bit harder. Using [1] I would probably set it to SEE.
Most of the time, some part of my environment is probably not set
up if I receive this error but sometimes it means my AUT is running
too slow and needs to be fixed. <not> could be useful here but it
isn't real clear. [2] might suggest SFE but it is a weak case.
I think we need [3]:
Use rule [2] but in cases where it isn't always useful to have SFE
we supply a flag which lets the user decide between SFE and SEE
with the default probably being SEE.
I think that we should make at least a distinction between invalid parameters
and the other cases (invalid parameters can be compared to a compilation error
in junit). I don't have any problem to put all the other cases in the same
category.
Like Denis, I think that the invalid parameters case should not be catched by a
<not>.
Marc.
_______________________________________________
WebTest mailing list
[email protected]
http://lists.canoo.com/mailman/listinfo/webtest
