Re: [Zope-dev] Why I dislike narrative doctests
On Thu, Apr 24, 2008 at 10:07 PM, Jim Fulton [EMAIL PROTECTED] wrote: - If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation. - Test edge cases in separate tests. These are typically short-ish strings in test modules. - A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation. I'ö happy to here this from an authorative source. I've been saying it for years, and I have usually been contradicted by people saying no all tests should be doctests, Lets kill that sillyness once and for all now. :-) -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Thu, Apr 24, 2008 at 04:07:12PM -0400, Jim Fulton wrote: - If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation. - Test edge cases in separate tests. These are typically short-ish strings in test modules. - A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation. Thanks for summing that up. -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Apr 25, 2008, at 1:59 AM, Lennart Regebro wrote: On Thu, Apr 24, 2008 at 10:07 PM, Jim Fulton [EMAIL PROTECTED] wrote: - If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation. - Test edge cases in separate tests. These are typically short-ish strings in test modules. - A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation. I'ö happy to here this from an authorative source. I've been saying it for years, and I have usually been contradicted by people saying no all tests should be doctests, Lets kill that sillyness once and for all now. :-) Everything I said above refers to doctests. I only pointed out that long narrative doctests aren't always desirable. Having said that, I can live with regular unit tests that are easy to follow and documented. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Thu, Apr 24, 2008 at 3:07 PM, Jim Fulton [EMAIL PROTECTED] wrote: On Apr 24, 2008, at 1:12 AM, Christian Theune wrote: Hi, On Wed, Apr 23, 2008 at 04:47:59PM -0400, Jim Fulton wrote: On Apr 23, 2008, at 4:47 PM, Marius Gedminas wrote: ... The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything. Except that that is not the current fashion, which has been pointed out many times in many places. For my own record (I must have missed this many times in many places), is the current fashion something along the lines: Use the various test styles as reasonable, text-file narrative doctests are preferred. No. WRT doc tests: - If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation. - Test edge cases in separate tests. These are typically short-ish strings in test modules. - A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation. However, as Sphinx lowers the barrier to cross-referencing, they will become important as link destinations from 'real' documentation. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists -http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Apr 25, 2008, at 10:20 AM, Kent Tenney wrote: However, as Sphinx lowers the barrier to cross-referencing, Sounds like I need to check that out. :) they will become important as link destinations from 'real' documentation. Maaaybe. I like doctests for readability even if they aren't documentation. I'd be hesitant to link to d=non-documentation from documentation. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Friday 25 April 2008, Jim Fulton wrote: However, as Sphinx lowers the barrier to cross-referencing, Sounds like I need to check that out. :) Paul just diud that work the last two days for z3c.form: http://docs.carduner.net/z3c.form/ Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. Zope Stephan Richter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Fri, Apr 25, 2008 at 9:39 AM, Jim Fulton [EMAIL PROTECTED] wrote: On Apr 25, 2008, at 10:20 AM, Kent Tenney wrote: However, as Sphinx lowers the barrier to cross-referencing, Sounds like I need to check that out. :) they will become important as link destinations from 'real' documentation. Maaaybe. I like doctests for readability even if they aren't documentation. I'd be hesitant to link to d=non-documentation from documentation. I think even non-documentation doctests are documentation since they demonstrate correct usage. They need to be explained in the linking document, but they are invaluable as examples. Seems like a natural separation of concerns. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Fri, Apr 25, 2008 at 10:11 AM, Kent Tenney [EMAIL PROTECTED] wrote: On Fri, Apr 25, 2008 at 9:39 AM, Jim Fulton [EMAIL PROTECTED] wrote: On Apr 25, 2008, at 10:20 AM, Kent Tenney wrote: However, as Sphinx lowers the barrier to cross-referencing, Sounds like I need to check that out. :) they will become important as link destinations from 'real' documentation. Maaaybe. I like doctests for readability even if they aren't documentation. I'd be hesitant to link to d=non-documentation from documentation. I think even non-documentation doctests are documentation since they demonstrate correct usage. They need to be explained in the linking document, but they are invaluable as examples. Seems like a natural separation of concerns. To make this work well, the doctests would want to have some markup added to identify specific tests, so the link would be to the test, not the file. This would be a good entry level chore for folks wanting to contribute. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Apr 25, 2008, at 11:22 AM, Stephan Richter wrote: - A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation. Why is this not documentation? Because it's written primarily as a test, not as documentation. For a file to be useable as documentation, greater thought needs to be given to the user reading it. Sometimes, I can't afford the extra time. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Apr 24, 2008, at 1:12 AM, Christian Theune wrote: Hi, On Wed, Apr 23, 2008 at 04:47:59PM -0400, Jim Fulton wrote: On Apr 23, 2008, at 4:47 PM, Marius Gedminas wrote: ... The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything. Except that that is not the current fashion, which has been pointed out many times in many places. For my own record (I must have missed this many times in many places), is the current fashion something along the lines: Use the various test styles as reasonable, text-file narrative doctests are preferred. No. WRT doc tests: - If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation. - Test edge cases in separate tests. These are typically short-ish strings in test modules. - A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation. Jim -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Why I dislike narrative doctests
Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure
objects make unittest.TextTestRunner cry). Being a good developer I
want to start the bug fix with a unit test.
Now if zope.testing used old-style isolated unit tests, I could open the
tests.py (or tests/test_testrunner.py) in a text editor and start
writing:
class OptionsStub(object):
post_mortem = False
def doctest_StartUpFailure():
Test that StartUpFailure is a proper unittest.TestCase
StartUpFailure is a TestCase-lookalike that is inserted into the
test suite to indicate that a failed import of a test module
won't be unnoticed.
from zope.testing.testrunner import StartUpFailure
options = OptionsStub()
module_name = 'test_me'
exc_info = (ImportError, ImportError('zonkulator'), None)
test = StartUpFailure(options, module_name, exc_info)
unittest.TextTestRunner(verbosity=2).run(test)
This is a regression test for http://launchpad.net/bugs/221151
There, I'm almost done. Now I can fix the bug and paste the correct
output into the doctest.
But zope.testing uses narrative doctests instead of isolated unit tests.
There are 22 plain-text files that together comprise over 4500 lines of
text. Now instead of mechanically opening the appropriate test module
and grepping for the class/function name I have to figure out which of
the txt files is the appropriate place for the new test. After that I
have to figure out how to insert it into the narrative seamlessly. Or
decide to create a new file---but a hundred of .txt files each 20-lines
long is not a good idea, in my opinion.
Suddenly I almost don't want to fix the bug any more.
While I was writing this, I suddenly realised that almost all
zope.testing's txt files are actually functional tests. There are
nearly no unit tests there. Perhaps that's what causing me pain.
I still maintain that:
* adding a new unit test should be a trivial task
* extending a narrative doctest is not a trivial task
* the only way to solve this conflict is to explicitly separate unit
tests (test_*.py) from doctest-enhanced API documentation (*.txt).
Marius Gedminas
--
UNIX is user friendly. It's just selective about who its friends are.
signature.asc
Description: Digital signature
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
Can't you start a plain-old unittests module and stick your test in
there? Is there any rule that says you cannot create a plain-old test
module if the project you are fixing the bug in uses doctests?
On Wed, Apr 23, 2008 at 5:19 PM, Marius Gedminas [EMAIL PROTECTED] wrote:
Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure
objects make unittest.TextTestRunner cry). Being a good developer I
want to start the bug fix with a unit test.
Now if zope.testing used old-style isolated unit tests, I could open the
tests.py (or tests/test_testrunner.py) in a text editor and start
writing:
class OptionsStub(object):
post_mortem = False
def doctest_StartUpFailure():
Test that StartUpFailure is a proper unittest.TestCase
StartUpFailure is a TestCase-lookalike that is inserted into the
test suite to indicate that a failed import of a test module
won't be unnoticed.
from zope.testing.testrunner import StartUpFailure
options = OptionsStub()
module_name = 'test_me'
exc_info = (ImportError, ImportError('zonkulator'), None)
test = StartUpFailure(options, module_name, exc_info)
unittest.TextTestRunner(verbosity=2).run(test)
This is a regression test for http://launchpad.net/bugs/221151
There, I'm almost done. Now I can fix the bug and paste the correct
output into the doctest.
But zope.testing uses narrative doctests instead of isolated unit tests.
There are 22 plain-text files that together comprise over 4500 lines of
text. Now instead of mechanically opening the appropriate test module
and grepping for the class/function name I have to figure out which of
the txt files is the appropriate place for the new test. After that I
have to figure out how to insert it into the narrative seamlessly. Or
decide to create a new file---but a hundred of .txt files each 20-lines
long is not a good idea, in my opinion.
Suddenly I almost don't want to fix the bug any more.
While I was writing this, I suddenly realised that almost all
zope.testing's txt files are actually functional tests. There are
nearly no unit tests there. Perhaps that's what causing me pain.
I still maintain that:
* adding a new unit test should be a trivial task
* extending a narrative doctest is not a trivial task
* the only way to solve this conflict is to explicitly separate unit
tests (test_*.py) from doctest-enhanced API documentation (*.txt).
Marius Gedminas
--
UNIX is user friendly. It's just selective about who its friends are.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFID5m0kVdEXeem148RAqs5AJ4qUo9IiqMSsuFOIrZBTq+xyWv/FQCePM/Q
eTvxFfGfDPuGA1YzlM8CTks=
=J91E
-END PGP SIGNATURE-
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
--
Sidnei da Silva
Enfold Systemshttp://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Apr 23, 2008, at 4:19 PM, Marius Gedminas wrote:
Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure
objects make unittest.TextTestRunner cry). Being a good developer I
want to start the bug fix with a unit test.
Now if zope.testing used old-style isolated unit tests, I could open
the
tests.py (or tests/test_testrunner.py) in a text editor and start
writing:
class OptionsStub(object):
post_mortem = False
def doctest_StartUpFailure():
Test that StartUpFailure is a proper unittest.TestCase
StartUpFailure is a TestCase-lookalike that is inserted into
the
test suite to indicate that a failed import of a test module
won't be unnoticed.
from zope.testing.testrunner import StartUpFailure
options = OptionsStub()
module_name = 'test_me'
exc_info = (ImportError, ImportError('zonkulator'), None)
test = StartUpFailure(options, module_name, exc_info)
unittest.TextTestRunner(verbosity=2).run(test)
This is a regression test for http://launchpad.net/bugs/221151
There, I'm almost done. Now I can fix the bug and paste the correct
output into the doctest.
But zope.testing uses narrative doctests instead of isolated unit
tests.
There are 22 plain-text files that together comprise over 4500 lines
of
text. Now instead of mechanically opening the appropriate test module
and grepping for the class/function name I have to figure out which of
the txt files is the appropriate place for the new test. After that I
have to figure out how to insert it into the narrative seamlessly. Or
decide to create a new file---but a hundred of .txt files each 20-
lines
long is not a good idea, in my opinion.
Suddenly I almost don't want to fix the bug any more.
You don't have to modify any narrative tests. You can add isolated
tests to your heart's content. Just add tests to tests.py. These can
be isolated doctests or unit tests, although I prefer the former. I
haven't done this yet in zope.testing, but I have in lots of other
projects.
See, for example:
http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/tests.py?rev=85041view=auto
I like narrative tests when the tests are reasonably a part of a
story. Not all tests are, or many tests are there own stories unto
themselves. In cases like this, small isolated tests are a lot better.
Jim
--
Jim Fulton
Zope Corporation
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Wed, Apr 23, 2008 at 05:22:29PM -0300, Sidnei da Silva wrote: Can't you start a plain-old unittests module and stick your test in there? Is there any rule that says you cannot create a plain-old test module if the project you are fixing the bug in uses doctests? I'll probably do that (since I sort of took over the maintainership of zope.testing a while ago). The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything. These are very fine when you initially write them, but a pain when you come back a year later and need to modify them. Trying to kill two birds with one stone (i.e. have narrative doctests serve both as documentation and as the test suite) doesn't seem to work: * users complain about boring corner cases and irrelevant test fixtures in what they expected to be documentation * programmers complain about narratives being too hard to modify Marius Gedminas -- You can't spell evil without vi. signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
On Wed, Apr 23, 2008 at 10:19 PM, Marius Gedminas [EMAIL PROTECTED] wrote: Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure objects make unittest.TextTestRunner cry). Being a good developer I want to start the bug fix with a unit test. ... I still maintain that: * adding a new unit test should be a trivial task * extending a narrative doctest is not a trivial task Yup. And then, try to debug it when it fails. Oups, you can't pdb through it. I repeat, and will continue to repeat until everybody gets it: ;-) Doctests are great. For writing testable documentation. They are NOT good for writing tests. * the only way to solve this conflict is to explicitly separate unit tests (test_*.py) from doctest-enhanced API documentation (*.txt). Yup. -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Why I dislike narrative doctests
Hi, On Wed, Apr 23, 2008 at 04:47:59PM -0400, Jim Fulton wrote: On Apr 23, 2008, at 4:47 PM, Marius Gedminas wrote: ... The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything. Except that that is not the current fashion, which has been pointed out many times in many places. For my own record (I must have missed this many times in many places), is the current fashion something along the lines: Use the various test styles as reasonable, text-file narrative doctests are preferred. ? Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
