On Wednesday, July 14, 2010 16:30:58 Walter Bright wrote: > Jonathan M Davis wrote: > > On the other hand, never halting on assertion failures makes for a lot > > of cruft in your results. > > I don't really understand why that's a problem, or causes extra work. > > But I can see that having to divide up the tests into multiple unittest > blocks adds extra work. > _______________________________________________ > phobos mailing list > [email protected] > http://lists.puremagic.com/mailman/listinfo/phobos
Well, I find it perfectly normal having to divide up the unit tests since every unit testing framework that I've used divides up unit tests into functions where each function is a test that succeeds or fails. There may be many assertions in a particular function, but once one fails, that function ceases to run (generally because an exception was thrown which is then caught by the framework). So, it actually makes more sense to me to divide up tests into unit test blocks. True, if you're testing a set of small but unrelated stuff, it could make sense to put them all together in one test, but if you're testing blocks as a whole, then you have the choice of whether to break them up or not just like in another framework, you'd have the choice of whether you have a single function or multiple functions. The cruft is a problem because it makes it harder to find the errors you actually care about. Unless you're checking a bunch of small, unrelated things, any assertion failure after the initial error is likely to be totally erroneous (since it was testing invalid state to begin with), so all of those errors just clutter the output, making it harder to find where one unittest block's tests end and where another begins (since it'll be the beginning of the next block where you'll start caring about errors again). It's definitely better to run all tests in a block regardless of assertion failures than it is to stop running all of the unittests in a program or module because one assertion failed. However, I do think that it's better to not continue to run a particular unittest block after an assertion fails because then it's likely to be testing invalid state and report a bunch of errors which are irrelevant. If you look at each unittest block as being a unit test testing a set of functionality rather than a particular assertion being the test, then it's just plain weird to continue to run the tests after the first assertion failed. The test already failed. However, if you're viewing each assertion as a test with the idea that there may be many unrelated assertions together, then I can see why it might be desirable to have the block continue to be executed (since splitting them up would mean a lot of unittest blocks). In any case, I do think that continuing to execute a unittest block after a failure is odd, and that it will result in a lot of unnecessary cruft, so I'd prefer that that didn't happen. However, it's definitely better to have the test continue than to stop them all after a single assertion failure. I suppose that the ideal solution would be to allow for both by having some way of indicating whether you wanted the test to continue after a particular assertion failure, but that would likely require multiple assertion types, which we don't have right now (at least not that the programmer can choose). Maybe if asserts threw normally (making it so that the first assertion failure in the unittest block resulted in that block ceasing execution, followed by the next unittest block being executed) but we had a library function of some kind which simply printed an error rather than throwing that would allow for the best of both worlds (like assert_nothrow() or something similar). - Jonathan M Davis _______________________________________________ phobos mailing list [email protected] http://lists.puremagic.com/mailman/listinfo/phobos
