Good of you to bring up the topic...
Adrian Sutton wrote:
Howdy all,
We now have a license for clover to analyze our test cases and am now just starting to work through adding test cases to improve our code coverage. I've very quickly come to the realization that 100% code coverage may actually be a bad thing. I've gotten AuthChallengeParser to 100% coverage now so let me use it as an example:
I think, like all code metrics, that coverage is a useful metric, but one that can be just as misleading as any other code metric - for example, lines of comments, or lines of code. It is just as easy to have a false sense of security due to metrics such as this one. If you consider a simple function like:
void doSomething(...) {
if (a) { ... }
if (b) { ... }
if (c) { ... }
...
}Your "coverage" tool will tell you you've exercised 100% of the code when you've only covered three of the eight possible ways through the routine, and that is in a function without any looping. In any case, the above function may either need detailed test cases that track for all eight possibilities, or only three, depending on how the routine works, and the contract it exposes, and how critical the code is.
So I would agree with your premise that 100% coverage could be a bad thing, particularly if it lulls you into a false sense of security. Also, given scarce resources, we should focus our time on testing those areas that most need it, rather than simply improving the statistics. On the other hand, now that we have the statistic, it should never go _down_.
There are four test cases that I consider pedantic and 1 of those that I really don't like. The pedantic ones are:
[snip]
Now, I don't mind what happens with any of these decisions to be honest as none affect the actual behaviour of HttpClient - they are very much edge cases. I would however like to set up a policy on the types of test cases I should create (do we want to avoid testing things like the pedantic things above) as well as the best way to keep track of questionable or overly pedantic test cases. Currently I'm just adding a // pedantic above any test case that seems pedantic and a todo comment over anything that I think may require a change to the code but isn't clearly a bug.
I figure from time to time I can provide a list of issues that need to be considered as I work my way through the codebase.
I struggled with a project with reams of test code on "getters" and "setters", yet the value of those "get/set" functions and the tests that went with them wasn't properly evaluated until late in the project. My attitude would be to avoid adding such test cases, to the extent that doing so does effectively enlarge the contract of the API. On the other hand, where such tests already exists, I don't see much point in removing them until they are actually in the way, either wrong or testing deprecated functionality.
Personally, I'm hoping to achieve 100% test coverage firstly because I've discovered how dependent I am on having good test cases while working on HttpClient (most people don't have the detailed level of knowledge that Mike and Oleg do and thus aren't aware that a change will break some other section of code - NTLM is a regular victim of this). Also, aiming for 100% coverage makes a very clear-cut decision on when the job is done which make life easier as well and makes it much more noticeable when new test cases need to be added.
Any thoughts?
If there is one thing I'd like to see with the testing code of HttpClient, it is a better way to test without having to actually have a "connection" to either local or remote host or proxy server. I'd enhance SimpleConnection and SimpleConnectionManager so that we could simulate more cases without actually having to resort to connecting to actual servers. Putting in simulations of proxy handling, simulations of dropped connections, NTLM, and so on, would be very cool. And I'd like a way to do it so that you simply write a "configuration file" that specifies what should be received and what the response should be. Unfortunately, I've not had the time to tackle the problem, and this week I'm spending my scarce open source time on the client side Slide DAV library, to improve its compatibility with HttpClient.
Maybe someone else can run with the idea?
-Eric.
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
