On Saturday, 25 July 2015 at 08:58:26 UTC, Walter Bright wrote:
It's still unusual to have 100% coverage in Phobos, and this is not because it is hard. Most of the time, it is easy to do. It's just that nobody checks it.
Yeah. I thought there had been a change to make it so that the coverage got printed out as part of the build, but I don't see it right now, at least on FreeBSD. Folks would still have to pay attention to those numbers though. Writing the tests is easy, and if someone is conscientious about their tests, I'd expect them to typically hit 100% without having to check (though you can still miss the occasional branch even then - especially with templated code), but frequently folks just write a few tests to make sure that the most basic functionality works and then call it a day.
I know that I'm too often guilty of assuming that I hit 100%, because I was thorough with my testing (since I tend to be _very_ thorough with my tests), and I should do better at verifying that I didn't miss something. I did put an effort a while back in making sure that std.datetime was as close to 100% as was possible, but I haven't checked it in a while...
Well, that's embarrassing. Almost all of the uncovered lines are lines that should never be run - e.g. assert(0) - but it does look like there are some lines which aren't covered which should be (not many in comparison to the whole, but there shouldn't be any). Interestingly though, the coverage is worse than it should be because it was generated from the release build on the unit tests, and stuff like invariants doesn't get run. I'll have to figure out how to get it to give me the coverage for the debug run of the tests, since that would be more accurate - though std.datetime can never actually hit 100% thanks to all of the assert(0) lines in it and the scope(failure) lines for printing out extra information when a test does fail. I suspect that I'm all of a percentage point off of what the max is.
Interestingly enough, @disable this() {} counts as a 0 too, even though the compiler should know that it's impossible for that to run even if the code is wrong - unlike assert(0). It _would_ be nice though if the assert(0) lines at least weren't counted, and it is kind of weird that the unit test lines count (though aside from scope(failure) lines, those should all run, though since I tend to put scope(failure) lines in unit tests for better output on failures, that's going to hurt my code coverage). So, _actually_ hitting 100% is likely to be annoyingly elusive for a lot of code, even if it's actually fully tested. But while std.datetime is almost as close as it can get, it's still _almost_ as close as it can get rather than all the way. :(
Clearly, I have a PR or two to write...
Although we have succeeded in making unit tests part of the culture, the next step is 100% coverage.
Agreed.
I know that 100% unit test coverage hardly guarantees code correctness. However, since I started using code coverage analyzers in the 1980s, the results are surprising - code with 100% test coverage has at LEAST an order of magnitude fewer bugs showing up in the field. It's surprisingly effective.
Oh, definitely. But while 100% unit test coverage is a huge step forward, I also think that for truly solid code, you want to go beyond that and make sure that you test corner cases and the like, test with a large enough variety of types with templates to catch behavioral bugs, etc. So, I don't think that we want to stop at 100% code coverage, but we do need to make sure that we're at 100% first and foremost.
This is a huge reason why I want to switch to ddmd. I want to improve the quality of the compiler with unit tests.
That would definitely be cool.
The various unit tests schemes I've tried for C++ are all ugly, inconvenient, and simply a bitch. It's like trying to use a slide rule after you've been given a calculator.
Yeah. It's not that hard to write one, and the tests themselves generally end up being pretty much the same as what you'd have in D, but there's still an annoying amount of boilerplate in getting them declared and set up - which is annoying enough in and of itself, but yeah, once you're used to D's unit tests, it seems particularly onerous.
(I remember the calculator revolution. It happened my freshman year at college. September 1975 had $125 slide rules in the campus bookstore. December they were at $5 cutout prices, and were gone by January. I never saw anyone use a slide rule again. I've never seen a technological switchover happen so fast, before or since.)
If only folks thought that D's advantages over C++ were that obvious. ;)
- Jonathan M Davis
