I wasn't trying to pick on you or anyone for that matter. It is something
I've been meaning to bring up for some months actually.
What I would hope to see is follows:
(1) if there is a bug where a given input FO file causes an unhandled
exception, and the bug fix is to fix the underlying problem causing the
exception, then that input FO file, or a subset thereof, should be added as
a test case, verified to fail prior to the fix, and verified not to fail
after the fix; a regression after the fix would cause the exception to occur
(2) similarly, if the bug is reporting a failure to produce expected results
for an existing feature; in this case, one or more test cases should be
added to verify the expected results;
(3) similarly, if a bug is requesting a new feature, and the bug fix is to
add the feature, then there should be some reasonable attempt to add tests
for that new feature; i'm not asking for 100% coverage of the feature, but
something more than nothing is a good thing;
(4) in the case of a performance fix, i would not say a test case
(demonstrating the improvement) is required; of course, if someone was
energetic enough to include a test case showing the performance delta, then
that would be welcome;
Of the above, I'm mostly concerned with the first and second cases.
On Mon, Jan 24, 2011 at 3:42 PM, Andreas Delmelle <
> On 24 Jan 2011, at 23:17, Glenn Adams wrote:
> Hi Glenn
> > Is there a reason why *all* bug fixes are not accompanied by one or more
> new test cases that demonstrate the presence and absence of the bug before
> and after the fix? Adding such test cases should be mandatory for all bug
> fix commits. I wouldn't quite go that far for performance and clean up
> fixes, however, but for legitimate bugs, there should be a reliable way to
> ensure that (1) the bug and its fix are testable and (2) that future
> regressions do not occur. I realize it takes more effort, but it is worth it
> in the long term, both in actual improvements in quality and the ability to
> demonstrate consistently good practice to maintain that quality.
> FWIW, in case you are referring to some of my recent commits...
> Any specific ones I should be taking a look at? Just following old habits,
> I'm afraid, and try to add tests wherever appropriate, even if the fix
> affects only a single line of code.
> Performance fixes may be rather difficult to test for, but even then one
> could conjure up a way to test whether adding nodes remains a linear
> operation, for example. We have no real framework set up for that type of
> thing, but I guess I could invest some time in that.
> > Many Apache projects require this process be followed; I would urge the
> FOP project to adopt a similar criterion for bug fix commits.
> Basically agreed with your viewpoint, so point out where exactly you feel
> something was missing and I'll see if I can accommodate you.