I'm not a big fan of 'known bugs' in unit tests. Unit tests should always pass. How practical is it that I'm going to invest the time to write a unit tests on a bug which I'm then not able to fix in the same merge. In many cases writing the test cases are actually harder than writing the code to fix the actual bug. If you really need to write a test case ahead of time (perhaps even to make your case that a bug exists) why not just create a launchpad bug and then attach the test case as a patch to the bug report? Seems like 'know bugs' also provides a mechanism for potentially stale unit tests code to hide out in the nova codebase? If code (including test code) is being actively used then I'd actually prefer to not have it be in the codebase.
Lastly, QA testers would probably focus more on the functional and integration types of testing rather than unit tests right? Dan -----Original Message----- From: "Justin Santa Barbara" <jus...@fathomdb.com> Sent: Monday, February 28, 2011 1:56pm To: openstack@lists.launchpad.net Subject: [Openstack] How to deal with 'tangential' bugs? Jay and I have been having an interesting discussion about how to deal with bugs that mean that unit tests _should_ fail. So, if I find a bug, I should write a failing unit test first, and fix it (in one merge). However, if I can't fix it, I can't get a failing unit test merged into the trunk (because it fails). It may be that I can't get what I'm actually working on merged with good unit tests until this 'tangential' bug is fixed. (The discussion is here: [https://code.launchpad.net/~justin-fathomdb/nova/bug724623/+merge/51227] https://code.launchpad.net/~justin-fathomdb/nova/bug724623/+merge/51227) I suggested that we introduce a "known_bugs" collection. It would have a set of values to indicate bugs that are known but not yet fixed. Ideally these would be linked to bug reports (we could mandate this). When a developer wants to write a test or behavior to work around a particular bug, they can control it based on testing this collection ("if 'bug12345' in known_bugs:") When someone is ready to fix the bug, they remove the bug from the collection, the unit tests then fail, they fix the code and commit with the known_bugs item removed. This would let people that find bugs but can't or don't want to fix them still contribute unit tests. This could be a QA person that can write tests but not necessarily code the fix. This could be a developer who simply isn't familiar with the particular system. Or it could be where the fix needs to go through the OpenStack discussion process. Or it could simply be a train-of-thought / 'flow' issue. Take, for example, my favorite OpenStack API authentication issue. To get a passing unit test with OpenStack authentication, my best bet is to set all three values (username, api_key, api_secret) to the same value. This, however, is a truly terrible test case. Having "known_bugs" marks my unit test as being suboptimal; it lets me provide better code in the same place (controlled by the known_bugs setting); and when the bug fixer comes to fix it they easily get a failing unit test that they can use for TDD. Jay (correctly) points out that this is complicated; can cause problems down the line when the bug is fixed in an unexpected way; that known_bugs should always be empty; and that the right thing to do is to fix the bug or get it fixed. I agree, but I don't think that getting the bug fixed before proceeding is realistic in a project with as many stakeholders as OpenStack has. Can we resolve the dilemma? How should we proceed when we find a bug but we're working on something different? Justin
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
