* Michael G Schwern <[EMAIL PROTECTED]> [2008-05-14 08:50]: > As I understand it, you want to know when broken code breaks > differently.
Indeed. > I can sort of see the point as a regression test... sort of. > But if you're at the point of fussiness that broken code has to > break in a specific way... is it broken any more? Depends. > I agree with others on this list, if you expect it to return 3, > put in a test that says it should return 3. > > As a technique, paying attention to how broken code changes, > why does it matter that broken code breaks differently? What > does this information tell you that might fix code? It means there is a known internal dependency on some other part of the code that is not being tested directly, either itself or the interaction therewith. You want to be alerted when something changes the result of this interaction. This is very reasonable. > Is the opportunity cost of paying attention to and maintaining > specific tests for broken code worthwhile? It might be harder to test the depended-upon part of the code, or it might require reaching into the blackbox. One might argue, as chromatic apparently did in this thread, that this means one should just write the proper test that directly asserts the things one is interested in. Personally I see that sort of conviction as counterproductive. It’s always better to lower the bar, making it easy to get a little extra safety with a little extra effort, even if there is a Right Way To Do It that gives much more safety at the cost of much more effort. Cf. `no_plan`, `all_done` and the like. Some testing is always better than none. > It seems like a dodgey regression to me of the "something > changed but we don't know if its significant" variety that > generates so many false alarms and so much busy work keeping it > up to date. A classic example is a test that does a simple > string compare with a big woodge of HTML. Any inconsequential > change to the formatting is going to trip the test leading to > many false negatives and eventually ignoring the test. Again, that depends. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
