Hi Keith, Good write up, some comments below...
On 05/14/10 01:44 AM, Keith Mitchell wrote: > On 05/11/10 08:23 AM, Dave Miner wrote: >> >>> >>> Can you please add "Unit Test Requirements For Bug Fixes" as a topic for >>> discussion at our next group meeting? >>> >> >> I've reserved 30 minutes for it on the agenda. However, feel free to >> continue the email discussion before then. >> >> Dave >> _______________________________________________ >> caiman-discuss mailing list >> [email protected] >> http://mail.opensolaris.org/mailman/listinfo/caiman-discuss > > Hi all, > > Below is a summary of the discussion points and conclusions from the > meeting. Please let me know if I missed anything important, or if there are > any further questions! Thanks to everyone for being patient and taking the > time to work through this. > > The primary points of concern were: > 1) For bug fixes, the overhead of setting up for a test would place a > burden on the developer. > - In most cases, the overhead of setting up a unit test should be > minimal to non-existent. The tests being asked for as part of this > change are intended to target very specific functions (e.g. by > running some_module.some_function("args")), as opposed to verifying > the exact conditions of a bug (e.g. running "beadm create xyz" on a > system with exactly 415 zfs datasets). > > - In terms of infrastructure, the addition of unit tests for a bug fix > should generally only require adding a "test" sub-folder near the > source code, and creating a "test_XXX.py" file within that folder. > Running the test should be straightforward; the basic case involves > building slim_source, pointing PYTHONPATH to the proto area, and > executing the test_XXX.py file. I think that it's vital to be clear about the PYTHONPATH that people are expected to use. This especially is important during the development cycle where most people will be likely to be writing test-cases for the first time. It's *VERY* easy at this phase, to end up writing a test case that, for example, imports from the local directory, as opposed to a fully qualified import (import Z, as opposed to import X.Y.Z), the latter of which is required to run the tests using "make test" as opposed to just "python ./my_test.py". Consistency of the environment is vital here, and can be probably best enforced by always running the tests via the Makefiles. As such, when this make infrastructure is being implemented by RE, it would be worthwhile having a mechanism to be able to run specific tests like : make TESTS=X test - and if its not set then it would just run all tests in the dir... > > 2) Even assuming tests are written as outlined in (1), there is still a time > cost associated with writing the tests. Depending on the code in > question, that time cost may include refactoring "nearby" code to some > degree. > - The time cost has a high return on investment in the form of "bugs > not filed" > - For time critical fixes (P1 bugs, etc.), it's acceptable to > deliver the fix, and then add tests subsequently. However, the tests > should be added in a "near future" time scale. > > 3) Having a hard requirement of "all pushes require unit tests" and > "unit test coverage must hit 80%" is unreasonable. > - Yes it is. The requirement is not inflexible; the intent is that any > decision not to include tests is deliberate and upfront. > - The 80% coverage goal is an initial goal, and targeted more at new > projects. This goal may shift over time depending on the realities of > writing unit tests against installer code. As I understood it, the enforcement of the test requirement is being done by the CR reviewers, not just RE - is that correct? Thanks, Darren. _______________________________________________ caiman-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/caiman-discuss
