On 05/11/10 08:23 AM, Dave Miner wrote:
Can you please add "Unit Test Requirements For Bug Fixes" as a topic for
discussion at our next group meeting?
I've reserved 30 minutes for it on the agenda. However, feel free to
continue the email discussion before then.
Dave
_______________________________________________
caiman-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/caiman-discuss
Hi all,
Below is a summary of the discussion points and conclusions from the
meeting. Please let me know if I missed anything important, or if there
are any further questions! Thanks to everyone for being patient and
taking the time to work through this.
The primary points of concern were:
1) For bug fixes, the overhead of setting up for a test would place a
burden on the developer.
- In most cases, the overhead of setting up a unit test should be
minimal to non-existent. The tests being asked for as part of this
change are intended to target very specific functions (e.g. by running
some_module.some_function("args")), as opposed to verifying the exact
conditions of a bug (e.g. running "beadm create xyz" on a system with
exactly 415 zfs datasets).
- In terms of infrastructure, the addition of unit tests for a bug
fix should generally only require adding a "test" sub-folder near the
source code, and creating a "test_XXX.py" file within that folder.
Running the test should be straightforward; the basic case involves
building slim_source, pointing PYTHONPATH to the proto area, and
executing the test_XXX.py file.
2) Even assuming tests are written as outlined in (1), there is still a
time cost associated with writing the tests. Depending on the code in
question, that time cost may include refactoring "nearby" code to some
degree.
- The time cost has a high return on investment in the form of
"bugs not filed"
- For time critical fixes (P1 bugs, etc.), it's acceptable to
deliver the fix, and then add tests subsequently. However, the tests
should be added in a "near future" time scale.
3) Having a hard requirement of "all pushes require unit tests" and
"unit test coverage must hit 80%" is unreasonable.
- Yes it is. The requirement is not inflexible; the intent is that
any decision not to include tests is deliberate and upfront.
- The 80% coverage goal is an initial goal, and targeted more at
new projects. This goal may shift over time depending on the realities
of writing unit tests against installer code.
- Keith
_______________________________________________
caiman-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/caiman-discuss
