Re: [Zope3-dev] Re: The bug fixing problem

2006-07-11 Thread Martijn Faassen

Jim Fulton wrote:
[snip]

I would say that there are two bugs in the case you are describing:  the
one you meant to fix and the one which is the lack of any tests for the
module / class / whatever.  I would bet that spending your thirty
minutes adding minimal tests to such a module is a *higher* value
activity than fixing most bugs, because it makes it easier for you (or
someone else) to fix that bug and others in that module.


Good point.


The PyPy project actually works with many tests that are not working. 
They have an infrastructure where such tests can be in the code and 
explicitly disabled.


In some cases, the bug-reporter may be able to write a test and not fix 
it. Or, alternatively, the person who goes and tries to fix a bug can 
write tests but doesn't have time to fix them.


In such case it would be nice to be able to add tests that are 
explicitly disabled and thus does not show up in the normal test run. 
Only when turning a knob these buggy tests show up, and a bug fixer can 
then easily go and try to fix them.


One danger is that this can be used to temporarily disable tests that 
*used to work*. Then again, that's not hard to do now either.


Regards,

Martijn
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-08 Thread Marius Gedminas
On Fri, Jul 07, 2006 at 08:48:39PM +0200, [EMAIL PROTECTED] wrote:
 In many cases, I can convince myself that a fix does actually work without
 performing a test -- at least in the sense that it removes one bug.

In many cases I have made completely trivial bug fixes that I was
absolutely confident about.  I wrote unit tests for those bug fixes only
out of habit.

Quite often I was surprised when my superfluous tests caught bugs in
my trivial bugfixes.

Marius Gedminas
-- 
Look!  Before our very eyes, the future is becoming the past.


signature.asc
Description: Digital signature
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-07 Thread Jim Fulton


On Jul 6, 2006, at 12:29 PM, Philipp von Weitershausen wrote:


[EMAIL PROTECTED] wrote:

Christian Theune wrote at 2006-7-5 11:46 +0200:

...
Another thing are the rules about unit tests. Some bugs touch  
areas that
are poorly tested. When I fix a bug over there, do I have to work  
harder

to introduce the fix because I have to start introducing tests?
We should find and announce a reasonable answer for the procedure in
this case.


Although I have (so far) never fixed a bug in Zope 3 (but posted
several patches for Zope 2), I can confirm this:

   There are bugs that do not need a test once they are fixed.
   All kinds of NameError and AttributeError fall into this
   category.

   Requiring to write a unit test for these or similarly trivial
   bugs is silly -- especially if there is not yet a testing file
   for the module (such that a trivial test would suffice).


I disagree. How would you make sure that your fix for even a trivial
NameError actually works? Perhaps you introduced another typo in the
bugfix? Or perhaps another problem pops up in the same codepath.
Clearly, since the NameError didn't occur in any other tests, the
codepath hasn't been tested yet, so it should be no matter what.

There's another aspect to tests for bugs: reproduceability. Especially
when fixing bugs I tend to write tests first in order to be absolutely
sure that I can reproduce the problem in an automated manner. Then
fixing the bug is easy: Just make the test pass...


Well said. I agree 100%

Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-07 Thread Jim Fulton


On Jul 6, 2006, at 2:03 PM, Lennart Regebro wrote:


On 7/6/06, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote:

When I have introduced secondary bugs in my fixes (which occasionally
happened), then a unit test would not have helped. The reason was  
then

that the affected code was used in unanticipated ways -- and
because it was unanticipated, I would not have written a test for it.


Sure. The point of the tests is not to prevent this. The point of them
is mainly to make sure that the thing you fixed says fixed.


Or that you actually fixed the thing you set out to fix, because, as
part of the test writing process you has to actually reproduce the
problem.  I often learn quite a bit about bugs when trying to
reproduce them. Bugs that appear shallow often turn out to be
much deeper during this process.

The security problem we dealt with the other day turned out this
way.  It was only through testing that Tres found out that the problem
was much different than it appeared on the surface.

Also, I think the habit of testing is as important as anything else.
We should always feel a bit queasy about making a change without
tests.

Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-07 Thread Jim Fulton


On Jul 6, 2006, at 10:11 PM, Tres Seaver wrote:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Christian Theune wrote:

Hi,

Marius Gedminas wrote:

I do not think that the requirements to
4. Write unit tests
5. Merge bugfixes from trunk to the release branch
6. Wait for the incredibly slow updates on the collector

discourage me all that much.


Right. They don't discourage me either, but there is a special  
case in

4) which I hit several times lately, where the unit tests need very
special effort.

I think that if more bug reports had a solution outlined in  
English, I'd

be more likely to go fix them every now and them.


Indeed!


I would say that there are two bugs in the case you are  
describing:  the
one you meant to fix and the one which is the lack of any tests for  
the

module / class / whatever.  I would bet that spending your thirty
minutes adding minimal tests to such a module is a *higher* value
activity than fixing most bugs, because it makes it easier for you (or
someone else) to fix that bug and others in that module.


Good point.

OTOH, well I'll mention the other hand in a separate post. :)

Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-07 Thread Christian Theune

Hi,

Jim Fulton wrote:


On Jul 6, 2006, at 12:29 PM, Philipp von Weitershausen wrote:


[EMAIL PROTECTED] wrote:

Christian Theune wrote at 2006-7-5 11:46 +0200:

...
Another thing are the rules about unit tests. Some bugs touch areas 
that
are poorly tested. When I fix a bug over there, do I have to work 
harder

to introduce the fix because I have to start introducing tests?
We should find and announce a reasonable answer for the procedure in
this case.


Although I have (so far) never fixed a bug in Zope 3 (but posted
several patches for Zope 2), I can confirm this:

   There are bugs that do not need a test once they are fixed.
   All kinds of NameError and AttributeError fall into this
   category.

   Requiring to write a unit test for these or similarly trivial
   bugs is silly -- especially if there is not yet a testing file
   for the module (such that a trivial test would suffice).


I disagree. How would you make sure that your fix for even a trivial
NameError actually works? Perhaps you introduced another typo in the
bugfix? Or perhaps another problem pops up in the same codepath.
Clearly, since the NameError didn't occur in any other tests, the
codepath hasn't been tested yet, so it should be no matter what.

There's another aspect to tests for bugs: reproduceability. Especially
when fixing bugs I tend to write tests first in order to be absolutely
sure that I can reproduce the problem in an automated manner. Then
fixing the bug is easy: Just make the test pass...


Well said. I agree 100%


Just to make sure nobody got me wrong: I'm not arguing that tests are a 
bad thing. They belong to the best thing that came up in the last years. :)


I was merely pointing to a small edge-case in bugfixing, that might be 
even smaller then anticipated, where I got frustrated in the last time.


+1 for the tests-are-good-recap :)

Christian

--
gocept gmbh  co. kg - forsterstraße 29 - 06112 halle/saale - germany
www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 -
fax +49 345 122 9889 1 - zope and plone consulting and development

___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-07 Thread Jim Fulton


On Jul 7, 2006, at 7:52 AM, Christian Theune wrote:


Hi,

Jim Fulton wrote:

On Jul 6, 2006, at 12:29 PM, Philipp von Weitershausen wrote:

[EMAIL PROTECTED] wrote:

Christian Theune wrote at 2006-7-5 11:46 +0200:

...
Another thing are the rules about unit tests. Some bugs touch  
areas that
are poorly tested. When I fix a bug over there, do I have to  
work harder

to introduce the fix because I have to start introducing tests?
We should find and announce a reasonable answer for the  
procedure in

this case.


Although I have (so far) never fixed a bug in Zope 3 (but posted
several patches for Zope 2), I can confirm this:

   There are bugs that do not need a test once they are fixed.
   All kinds of NameError and AttributeError fall into this
   category.

   Requiring to write a unit test for these or similarly trivial
   bugs is silly -- especially if there is not yet a testing file
   for the module (such that a trivial test would suffice).


I disagree. How would you make sure that your fix for even a  
trivial

NameError actually works? Perhaps you introduced another typo in the
bugfix? Or perhaps another problem pops up in the same codepath.
Clearly, since the NameError didn't occur in any other tests, the
codepath hasn't been tested yet, so it should be no matter what.

There's another aspect to tests for bugs: reproduceability.  
Especially
when fixing bugs I tend to write tests first in order to be  
absolutely

sure that I can reproduce the problem in an automated manner. Then
fixing the bug is easy: Just make the test pass...

Well said. I agree 100%


Just to make sure nobody got me wrong: I'm not arguing that tests  
are a bad thing. They belong to the best thing that came up in the  
last years. :)


I was merely pointing to a small edge-case in bugfixing, that might  
be even smaller then anticipated, where I got frustrated in the  
last time.


+1 for the tests-are-good-recap :)


And just to make sure that I was clear, when trying to urgently go  
through

bugs for a release, I think it's OK to provide a test for just the fix.

If, in the course of a fix, you notice that a package is short of tests,
it would be great to add more then, but it would be OK to just turn that
lack of tests into a separate non-urgent bug entry.

Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-06 Thread Lennart Regebro

On 7/6/06, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote:

When I have introduced secondary bugs in my fixes (which occasionally
happened), then a unit test would not have helped. The reason was then
that the affected code was used in unanticipated ways -- and
because it was unanticipated, I would not have written a test for it.


Sure. The point of the tests is not to prevent this. The point of them
is mainly to make sure that the thing you fixed says fixed.

--
Lennart Regebro, Nuxeo http://www.nuxeo.com/
CPS Content Management http://www.cps-project.org/
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] Re: The bug fixing problem

2006-07-06 Thread dieter
Lennart Regebro wrote at 2006-7-6 20:03 +0200:
On 7/6/06, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote:
 When I have introduced secondary bugs in my fixes (which occasionally
 happened), then a unit test would not have helped. The reason was then
 that the affected code was used in unanticipated ways -- and
 because it was unanticipated, I would not have written a test for it.

Sure. The point of the tests is not to prevent this. The point of them
is mainly to make sure that the thing you fixed says fixed.

I hear (also) other sounds than unit tests ;-)
Thus, my fixes can say fixed differently (without a passing unit test)...



-- 
Dieter
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com