Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
Am 17.04.2010, 03:41 Uhr, schrieb Tres Seaver tsea...@palladion.com: The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation. I'm all for this but in the past I have struggled mightily (this could be an ironic use of the word ;-) with unit tests for views. Is this 1) Because I've overlooked something? 2) Because views depend on too many things, setting up test cases is a lot of work? 3) Because it's not been done a lot in the past? Charlie -- Charlie Clark Managing Director Clark Consulting Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
Am 22.04.2010, 05:48 Uhr, schrieb Christian Theune c...@gocept.com: Could get a lint template (or whatever it's called) to help here? Not sure what that means, can you elaborate a bit? Lint is a tool that analyses code for all kinds of languages, including Python, and scores it against various definable metrics. The idea is to encourage code readability and best practices. I imagine that for more predictable parts of code like unit tests it might be possible to use it to encourage a uniform style such as TestMock seems to promise for VisualStudio http://site.typemock.com/test-lint/ I'll admit to not using PyLint (from Nuxeo) myself on a regular basis but this is as much down to not knowing how to configure it to work with Zope projects as much as anything else. But maybe we can add it to anything the repository policy checker? Charlie -- Charlie Clark Managing Director Clark Consulting Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
Am 20.04.2010, 18:43 Uhr, schrieb Christian Theune c...@gocept.com: Let me raise another issue which is that we also need to figure out how to follow up with this once we start. Could get a lint template (or whatever it's called) to help here? Charlie -- Charlie Clark Managing Director Clark Consulting Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 04/21/2010 09:05 PM, Charlie Clark wrote: Am 20.04.2010, 18:43 Uhr, schrieb Christian Theunec...@gocept.com: Let me raise another issue which is that we also need to figure out how to follow up with this once we start. Could get a lint template (or whatever it's called) to help here? Not sure what that means, can you elaborate a bit? Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 04/17/2010 10:56 PM, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lennart Regebro wrote: On Sat, Apr 17, 2010 at 19:17, Tres Seavertsea...@palladion.com wrote: I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets. Well, unless they take of time, but they shouldn't, really. I think it's positive to get a quick feedback if you are breaking the documentation. Again, I don't mind that part, but I want to break the cycle of jamming crap (for documentation purposes) into the docs for purposes of getting test coverage. I think Lennart was arguing for a safety belt: if we're not supposed to break those tests *at all* within bug fixes then I want to know right away if I did so accidentally. I guess your coverage argument was that if the unit tests do the coverage anyway then we won't miss anything, but we won't be 100% anyway and as the tests in the docs do exist and should run quickly, the safety belt would be cheap. I'm +0.5 after some pondering. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 04/19/2010 03:56 PM, Wichert Akkerman wrote: On 4/19/10 15:48 , Marius Gedminas wrote: def doctest_MyClass_bar(): Test MyClass.bar y = MyClass() The bar method peforms a bar calculation that typically returns twenty-three: y.bar() 23 What is the advantage of that over: def test_something(self): # Test MyClass.bar y=MyClass() # The bar method peforms a bar calculation that typically # returns 23. self.assertEqual(y.bar(), 23) It reads the same, and as a bonus you can step through it with pdb and syntax highlighting works normally in most editors. I had some good experiences in the last 6 months or so when switching from doctests that were tests (not documentation) to this style of testing. Also the review process mentioned (or pairing) helps to put comments in places where the code isn't easily understood by itself (after that, refactor the code so you don't need comments and put the real comments in). Christian -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 04/19/2010 04:06 PM, Lennart Regebro wrote: On Mon, Apr 19, 2010 at 16:03, Lennart Regebrorege...@gmail.com wrote: On Mon, Apr 19, 2010 at 15:48, Marius Gedminasmar...@gedmin.as wrote: If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc. And from __future__ imports doesn't work. And.. I'm sure I'm gonna remember more reason why doctests are bad later, but I can't be bothered really. Try defining a (persistent) class and sticking it into a ZODB. -- Christian Theune · c...@gocept.com gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
Hello Tres, Saturday, April 17, 2010, 3:41:02 AM, you wrote: TS -BEGIN PGP SIGNED MESSAGE- TS Hash: SHA1 snip +lots on more docs +lots on 100% coverage TS The trickier testing bits we would re-write as super thorough, no TS shortcuts-taken unit tests: one testcase class per class (or API TS function) under test, at least one method per class-under-test method, TS with more preferred to get all code paths / preconditions covered. The TS goal here would be to comment out the doctests from the test_suites and TS get the code to 100% coverage using pure unit tests. Meanwhile, the TS doctests would be refactored into the Sphinx documentation, losing all TS the bits which obscure their intent as documentation. I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers). OTOH I think most unittests maybe have some comments, worst case they don't even use speaking variable names. I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones. -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: You look tired ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
Am 19.04.2010, 12:50 Uhr, schrieb Adam GROSZER agros...@gmail.com: Before I start: + lots on the general idea. Zope's narrative documentation is attrocious[1] and we have lost users and developers as a result. I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. All the more reason not to think of them as part of the documentation. Would it possible to come up with some metrics for the quality of tests? A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers). Narrative documentation forces you to explain yourself to someone else. Neither, however, will necessarily notice if you forget something: tests aren't documentation and docu OTOH I think most unittests maybe have some comments, worst case they don't even use speaking variable names. The key to any of this is making it easy for developers to want to go this way. Writing good, legible tests *will* encourage others to emulate them. I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones. What does Lint have to say about them? Charlie [1] http://www.stereoplex.com/2010/jan/14/migrating-django-mingus/ -- Charlie Clark Managing Director Clark Consulting Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Mon, Apr 19, 2010 at 7:21 AM, Charlie Clark charlie.cl...@clark-consulting.eu wrote: Narrative documentation forces you to explain yourself to someone else. Neither, however, will necessarily notice if you forget something: tests aren't documentation and docu [sic] Quite true. It is also true that tests should be well documented and documentation should be well tested. Some prefer doctests for both. -- Benji York ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Mon, Apr 19, 2010 at 12:50:23PM +0200, Adam GROSZER wrote: Hello Tres, Saturday, April 17, 2010, 3:41:02 AM, you wrote: TS -BEGIN PGP SIGNED MESSAGE- TS Hash: SHA1 snip +lots on more docs +lots on 100% coverage TS The trickier testing bits we would re-write as super thorough, no TS shortcuts-taken unit tests: one testcase class per class (or API TS function) under test, at least one method per class-under-test method, TS with more preferred to get all code paths / preconditions covered. The TS goal here would be to comment out the doctests from the test_suites and TS get the code to 100% coverage using pure unit tests. Meanwhile, the TS doctests would be refactored into the Sphinx documentation, losing all TS the bits which obscure their intent as documentation. I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers). That's true, but if the doctest gets too long, any readability advantages are negated. If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests -- and by short I mean 1-7 statements: import unittest import doctest from mypackage.mymodule import MyClass class TestMyClass(unittest.TestCase): def test_foo(): x = MyClass() self.assertEquals(x.foo(), 42) def doctest_MyClass_bar(): Test MyClass.bar y = MyClass() The bar method peforms a bar calculation that typically returns twenty-three: y.bar() 23 def doctest_MyClass_bar_raises_if_incorrect_setup(): Test MyClass.bar y = MyClass(setup='incorrect') If you haven't supplied the correct setup, you get an exception: y.bar() Traceback (most recent call last): ... IncorrectSetupError: the setup was incorrect def test_suite(): return unittest.TestSuite([ unittest.makeSuite(TestMyClass), doctest.DocTestSuite(), ]) The downside of this style is that it's difficult to refactor common bits from the doctests into shared methods, so you end up with a lot of duplicated code. OTOH I think most unittests maybe have some comments, worst case they don't even use speaking variable names. I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development signature.asc Description: Digital signature ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Mon, Apr 19, 2010 at 15:48, Marius Gedminas mar...@gedmin.as wrote: If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc. DocTests are for testing documentation, not documenting tests. Sorry guys, all other usage of doctests than making documentation testable is evil. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Mon, Apr 19, 2010 at 16:03, Lennart Regebro rege...@gmail.com wrote: On Mon, Apr 19, 2010 at 15:48, Marius Gedminas mar...@gedmin.as wrote: If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc. And from __future__ imports doesn't work. And.. I'm sure I'm gonna remember more reason why doctests are bad later, but I can't be bothered really. Just don't use them unless you are writing documentation. PLEASE! -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Mon, Apr 19, 2010 at 9:48 AM, Marius Gedminas mar...@gedmin.as wrote: On Mon, Apr 19, 2010 at 12:50:23PM +0200, Adam GROSZER wrote: I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers). That's true, but if the doctest gets too long, any readability advantages are negated. If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests -- and by short I mean 1-7 statements: You might be interested in Manuel's isolation mechanism: http://packages.python.org/manuel/#test-isolation The downside of this style is that it's difficult to refactor common bits from the doctests into shared methods, so you end up with a lot of duplicated code. If I understand you (which I'm not sure I do), I prefer to factor out common code into test helpers that are either referenced in footnotes (to keep all the code in one place) or in a separate module. -- Benji York ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Mon, Apr 19, 2010 at 3:03 PM, Lennart Regebro rege...@gmail.com wrote: On Mon, Apr 19, 2010 at 15:48, Marius Gedminas mar...@gedmin.as wrote: If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc. You may wish to read: http://bemusement.org/diary/2008/October/23/narrative-tests http://bemusement.org/diary/2008/October/24/more-doctest-problems which are an extended meditation on the whole topic. jml ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Adam GROSZER wrote: I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers). OTOH I think most unittests maybe have some comments, worst case they don't even use speaking variable names. I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones. I agree that the unit tests themselves often need cleaninng up: - - They should avoid importing the module-under-test at module scope, and minimize any other imports as well: import errors should cause tests to fail, not to be masked. - - They need to be broken up, so that each testcase method tests exactly one code path through each method (or function) under test. Each testcase method should be named to indicate which method/function it tests, and which set of preconditions. - - The local variables need to have *much* better names. - - They need to avoid using test fixtures from other modules: having to stop to look up exactly what the fixture means or does is a pipeline stall for the CPU of the programmer tryihg to understand the test. Doing this cleanup is part of the effort I am proposing we undertake: getting the tests comprehensible is as important as getting to 100% coverage, because the lines executed bit (syntax) that coverage analyzers report is still not sufficient to ensure that you cover all the semantics of the module. Analyzing completeness on the latter requires understanding what each method or function tested does, and how each test exercieses it. I have made a start on such cleanups for zope.interface: http://svn.zope.org/zope.interface/branches/tseaver-better_unit_tests/ Look particularly at the 'test_interface.py' changes: http://svn.zope.org/zope.interface/branches/tseaver-better_unit_tests/src/zope/interface/tests/test_interface.py?r1=texttr1=09r2=texttr2=10diff_format=h and those to 'test_verify.py': http://svn.zope.org/zope.interface/branches/tseaver-better_unit_tests/src/zope/interface/tests/test_verify.py?r1=texttr1=09r2=texttr2=14diff_format=h Note as well that I am deliberately leaving off the table the integration testing which is normally done via the zope.testbrowser layer support: as with doctests, such tests can be useful, but I would never rely on them for the coverage part of testing a package. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvMeB0ACgkQ+gerLs4ltQ7rXwCfVxQho7XLjsYTQITxuWHF3/3N 3hcAoLlL/LoaUlLSlaEpjANGPD3jR7rz =nqY1 -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Benji York wrote: On Mon, Apr 19, 2010 at 7:21 AM, Charlie Clark charlie.cl...@clark-consulting.eu wrote: Narrative documentation forces you to explain yourself to someone else. Neither, however, will necessarily notice if you forget something: tests aren't documentation and docu [sic] Quite true. It is also true that tests should be well documented and documentation should be well tested. Some prefer doctests for both. I would disagree that tests should be well documented, at least in a thread whose main purpose is to point up the very grave shortcomings in the narrative documentation for the ZTK package set as a whole. Writing clean, understandable code, with good class / method / function / variable names, comments where appropriate, etc., is not the same thing as documenting that code. Overloading the word well-documented to describe code which conforms to such (completely uncontroversial) standards muddies the point here. I am not arguing that writing good doctests is impossible, any more than writing good unit tests. What I am arguing is that we have lots of mediocre-to-poor tests (of both kinds), and that we have basically no documentation of the sort that most people mean when they use the term. I want us to take off the I'm testing the code hat when writing documentation, and focus on producing a much more useful documentation set, one which actually encourages folks who haven't already drunk our Kool-Aid to use the code. For instance, the ratio of lines of narrative text to lines of examples in the documentation ought to be *much* higher than the near-parity which is the rule in most ZTK doctests. I don't think we can get there by continuing to pretend that our existing doctests are working as documenation, overall, which was my rationale for moving the ones which might form the basis for real docs *out* of the code, and manage them separately as Sphinx documentation. As I have said elasewhere in the thread, I also believe that the snipptes in that documentation should be testable, where possible: I think that the Sphinx addons which allow running the snippets as part of building the tests are a really good fit for ensuring that the snippets don't bit-rot. Unfortunately, no tool is going to help ensure that the narrative itself hasn't bit-rotted: that takes a dedicated investment over time by the team maintaining the packages. I also believe that the goal of testability of document snippets remains subordinate to the goals of comprehensibility and of completness *as documentation*. Doumentation writers must work to remove any artifacts of testability which shows up in the rendered docs, where such artifacts don't actively aid comprehension. When we wear the testing hat, as opposed to the documenation hat, we should come to some kind of agreement about how to achieve the goal of getting our code thoroughly tested. I am not adamantly opposed to using doctests to aid in this goal, particularly for covering the main line of usage of an API. However, because they frequently test non-essential artifacts (hash ordering, exception rendering, etc.), they tend to be the most fragile of our tests across things like major Python version changes, platforms, etc. Also, because they take a narrative form, doctests alos encourage test writere to use shared state between logically-discrete tests, which leads to really ugly coupling and reduces the ability of the test writer to reason about semantic completeness of the tests. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvMjPAACgkQ+gerLs4ltQ5X0QCcDdeRhjUKLQtjcNZVXW7fGulv WLMAoLbNRu9SMximbOrqnsV2mqkWrbxz =SExz -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Mon, Apr 19, 2010 at 6:42 PM, Marius Gedminas mar...@gedmin.as wrote: On Mon, Apr 19, 2010 at 03:56:02PM +0200, Wichert Akkerman wrote: On 4/19/10 15:48 , Marius Gedminas wrote: def doctest_MyClass_bar(): Test MyClass.bar y = MyClass() The bar method peforms a bar calculation that typically returns twenty-three: y.bar() 23 What is the advantage of that over: def test_something(self): # Test MyClass.bar y=MyClass() *cringe* Sorry, I've this reflex to cringe every time I see a PEP-8 violation. # The bar method peforms a bar calculation that typically # returns 23. self.assertEqual(y.bar(), 23) It reads the same, and as a bonus you can step through it with pdb and syntax highlighting works normally in most editors. The advantage is that I've rarely seen comments in unit tests and personally I never felt compelled to write a comment when writing a unit test. Twisted and Launchpad both have policies that require new or modified unit tests to have comments, enforced as a part of our respective review processes. This seems to work well, since the reviewer can block silly comments. The documentation you get is more like a list of requirements / behaviours (e.g. We raise an error when we try to create a branch in a namespace where there is already a branch of the same name) than a story, but sometimes that's a good thing. It's surprising how easy it is to slip into the habit. It's no substitute for tutorial / howto documentation, but is often a great supplement to API docs. jml ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 17/04/10 23:20, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: snip - I can almost guarantee that any examples that aren't tested will be wrong. I tried to do a good job on the bobo docs. I made almost all of the examples executable, and the ones I didn't had an amazing tendency to have bugs. I'm not against having the snippets be executable, because I *do* want them to work. I just don't want to encourage anyone to think that they are testing the software when they write the snippets, or execute them. I second Martin: the sentence hereunder is worth its weight in gold. (I wonder if this is an appropriate use of the translation of French valoir son pesant d'or) Executing the snippets is testing the documentation, not the software. - I agree that tests should be separate from documentation. You can get some of your coverage from the docs, but you'll need tests for edge cases and details not addressed by the docs. I would actually prefer to measure coverage (objectively) without reference to the snippets in the docs. I think this would make a lot of sense. Having *all* the tests for a module in one place helps think more clearly about getting good coverage, both from the lines executed standpoint (which can be objectively measured) and from the semantics enforced standpoint, which can't. Tres. With both unit and doc testing, we can get full lines executed coverage. Because of the insulation enforced by our unit testing framework, this can lead to better semantic coverage than with the dependent tests we write when doc testing. One of the reason that full lines executed coverage does not translate to full semantics coverage is the presence of class attributes. In fact, those attributes are holding sub-global state that multiplies the number of combinations that should be tested. I slowly have come to the idea that class attributes can be as bad as global variables : they allow hidden dependencies between methods. This is an insidious problem as class attributes look better until you are told other. A good sign that this problem is creeping in is when you need to write complex setup in order to write method tests. In other words, unit testing can be seen as a way of being warned early of too many attributes in our classes. PS: Hereabove, I speak only of the insulated semantics of each class. This does not remove the need for integration tests which *try* to test all interactions between those classes. -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Sat, Apr 17, 2010 at 03:41, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3. I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible: - - Testing setup and teardown code obscures the narrative flow / intent of the documentation. - - Edge case testing *really* obscures the same. - - Test coverage suffers, because it is hard to write edge-case tests in doctest. - - Porting the doctests where people have tried to juggle both goals across Python versions is a mess. The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.). We can still leave the main line of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an intersphinx tree underneath the main ZTK docs, which could serve as the gateway into them. The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation. I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the edge-case tests. I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion: http://svn.zope.org/zope.event/branches/tseaver-new_style_docs I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback). The rendered docs from that branch are here: http://palladion.com/static/zope.event-docs/ I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view): https://code.launchpad.net/~tseaver/zope.event/new_style_docs I am attaching an annotated diff for those who would prefer to see the changes in that format. I would say that this cleanup effort should start at the bottom layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move upwards (in the depenency sense) over time. Thoughts? +lots. ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
(Although possible we could call it developing instead of hacking but that's not a big issue. ;) ) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 2010-4-17 03:41, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3. I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible: - - Testing setup and teardown code obscures the narrative flow / intent of the documentation. - - Edge case testing *really* obscures the same. - - Test coverage suffers, because it is hard to write edge-case tests in doctest. - - Porting the doctests where people have tried to juggle both goals across Python versions is a mess. The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.). We can still leave the main line of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an intersphinx tree underneath the main ZTK docs, which could serve as the gateway into them. The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation. I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the edge-case tests. I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion: http://svn.zope.org/zope.event/branches/tseaver-new_style_docs I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback). The rendered docs from that branch are here: http://palladion.com/static/zope.event-docs/ I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view): https://code.launchpad.net/~tseaver/zope.event/new_style_docs I am attaching an annotated diff for those who would prefer to see the changes in that format. I would say that this cleanup effort should start at the bottom layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move upwards (in the depenency sense) over time. Thoughts? +1 Wichert. -- Wichert Akkerman wich...@wiggy.net It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
Hi! Tres Seaver wrote: The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.). We can still leave the main line of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. [...] Thoughts? +1 in general, but I'm concerned about one thing: How can we make sure docs and code don't get out of sync? Do we have to run unittests *and* build the docs before each checkin? Will someone make sure buildbots and nightly tests report broken docs as well? Cheers, Yuppie ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Sat, Apr 17, 2010 at 11:18, yuppie y.2...@wcm-solutions.de wrote: How can we make sure docs and code don't get out of sync? Do we have to run unittests *and* build the docs before each checkin? Will someone make sure buildbots and nightly tests report broken docs as well? Hm... As long as you use the syntax and none of the ..test-block:: syntax of sphinx, the sphinx docs can be tested as a part of the standard testruns, but I guess part of this proposal was so we could easily use the testblock syntax as it has the possibility to hide setup etc easier. And then I don't know the best way to integrate it, but it would definitely be a drawback if you need two different testruns to run all the tests... Also I think it should be the responsibility of the one who does a release to make sure that we can build HTML docs and that the formatting isn't *too* broken. Preferably these docs should also be uploaded to packages.python.org, maybe there can be a makefile step for that in the docs? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 17 April 2010 10:41, Lennart Regebro rege...@gmail.com wrote: On Sat, Apr 17, 2010 at 11:18, yuppie y.2...@wcm-solutions.de wrote: How can we make sure docs and code don't get out of sync? Do we have to run unittests *and* build the docs before each checkin? Will someone make sure buildbots and nightly tests report broken docs as well? Hm... As long as you use the syntax and none of the ..test-block:: syntax of sphinx, the sphinx docs can be tested as a part of the standard testruns, but I guess part of this proposal was so we could easily use the testblock syntax as it has the possibility to hide setup etc easier. And then I don't know the best way to integrate it, but it would definitely be a drawback if you need two different testruns to run all the tests... Also I think it should be the responsibility of the one who does a release to make sure that we can build HTML docs and that the formatting isn't *too* broken. Preferably these docs should also be uploaded to packages.python.org, maybe there can be a makefile step for that in the docs? It's important that documentation is tested as part of the standard test run, that means when you change something you know to update the docs. repoze.bfg seemed to make an attempt at this, though it is currently disabled. http://svn.repoze.org/repoze.bfg/trunk/repoze/bfg/tests/test_docs.py Laurence ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Sat, Apr 17, 2010 at 16:05, Laurence Rowe l...@lrowe.co.uk wrote: It's important that documentation is tested as part of the standard test run, that means when you change something you know to update the docs. repoze.bfg seemed to make an attempt at this, though it is currently disabled. http://svn.repoze.org/repoze.bfg/trunk/repoze/bfg/tests/test_docs.py How could I forget about Manuel. *facepalm*. I've written Manuel extensions to test sphinx tests, but Benji hasn't had time to review it yet: http://svn.zope.org/manuel/branches/sphinx/ So it's perfectly possible to run all types of Sphinx tests even with codeblocks, as a part of the standard testrun. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lennart Regebro wrote: On Sat, Apr 17, 2010 at 16:05, Laurence Rowe l...@lrowe.co.uk wrote: It's important that documentation is tested as part of the standard test run, that means when you change something you know to update the docs. repoze.bfg seemed to make an attempt at this, though it is currently disabled. http://svn.repoze.org/repoze.bfg/trunk/repoze/bfg/tests/test_docs.py How could I forget about Manuel. *facepalm*. I've written Manuel extensions to test sphinx tests, but Benji hasn't had time to review it yet: http://svn.zope.org/manuel/branches/sphinx/ So it's perfectly possible to run all types of Sphinx tests even with codeblocks, as a part of the standard testrun. I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets. Basically, bugfix changeses shouldn't need to run the documentation tests, because the visible / main-line behavior isn't supposed to change at all. Feature changes should *start* by writing the narrative docs about the added feature: feature branches which don't include updates to the docs shouldn't be merged. Running the snippet tests as part of making a release (or even via some sort of '--all' argument for nightly runners) seems pretty adequate to me. If we upgrade our testing style to keep each testcase tightly focused, with meaningful names for the methods and local varioables, it should be much easier to figure out what, if anything, got overlooked in the documentaion. +0 on running them every time IFF we can find an easy way to run them during 'setup.py test' as well as buildout, and we continue to be mindful of the distinction between docs that work primarily as docs and tests that cover every code path. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvJ7Q0ACgkQ+gerLs4ltQ7QAwCfUZfetKf3ZBdCa4AZ/G+02jab w5AAn2ZkjA5eevc+clc4Jre/IHSk7SH0 =rdQf -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Sat, Apr 17, 2010 at 19:17, Tres Seaver tsea...@palladion.com wrote: I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets. Well, unless they take of time, but they shouldn't, really. I think it's positive to get a quick feedback if you are breaking the documentation. +0 on running them every time IFF we can find an easy way to run them during 'setup.py test' as well as buildout, and we continue to be mindful of the distinction between docs that work primarily as docs and tests that cover every code path. I think manuel should solve that, we'll see, I'll have to test. I need to get manuel ported to Python 3 then too. :) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lennart Regebro wrote: On Sat, Apr 17, 2010 at 19:17, Tres Seaver tsea...@palladion.com wrote: I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets. Well, unless they take of time, but they shouldn't, really. I think it's positive to get a quick feedback if you are breaking the documentation. Again, I don't mind that part, but I want to break the cycle of jamming crap (for documentation purposes) into the docs for purposes of getting test coverage. +0 on running them every time IFF we can find an easy way to run them during 'setup.py test' as well as buildout, and we continue to be mindful of the distinction between docs that work primarily as docs and tests that cover every code path. I think manuel should solve that, we'll see, I'll have to test. I need to get manuel ported to Python 3 then too. :) Turtles all the way down. ;) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvKIHgACgkQ+gerLs4ltQ7foQCfTzBh8IGTr8beH/H+vdSLrAI5 TdEAoJWNJ1w/GQaVoMITw1siOxKaRJZj =ns67 -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On Fri, Apr 16, 2010 at 9:41 PM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3. I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible: - - Testing setup and teardown code obscures the narrative flow / intent of the documentation. - - Edge case testing *really* obscures the same. - - Test coverage suffers, because it is hard to write edge-case tests in doctest. - - Porting the doctests where people have tried to juggle both goals across Python versions is a mess. The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.). We can still leave the main line of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an intersphinx tree underneath the main ZTK docs, which could serve as the gateway into them. The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation. I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the edge-case tests. I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion: http://svn.zope.org/zope.event/branches/tseaver-new_style_docs I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback). The rendered docs from that branch are here: http://palladion.com/static/zope.event-docs/ I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view): https://code.launchpad.net/~tseaver/zope.event/new_style_docs I am attaching an annotated diff for those who would prefer to see the changes in that format. I would say that this cleanup effort should start at the bottom layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move upwards (in the depenency sense) over time. Thoughts? I agree with the main thrust. (Of course, I disagree with some of your judgments of doctest, but that's mostly beside the point.) - Documentation should be written for documentation's sake. The emphasis should be on helping people understand what the software is for and how to use it, *not* on coverage. - Documentation should be executable. Manuel helps a lot for this. - I can almost guarantee that any examples that aren't tested will be wrong. I tried to do a good job on the bobo docs. I made almost all of the examples executable, and the ones I didn't had an amazing tendency to have bugs. - I agree that tests should be separate from documentation. You can get some of your coverage from the docs, but you'll need tests for edge cases and details not addressed by the docs. Jim -- Jim Fulton ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Fri, Apr 16, 2010 at 9:41 PM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3. I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible: - - Testing setup and teardown code obscures the narrative flow / intent of the documentation. - - Edge case testing *really* obscures the same. - - Test coverage suffers, because it is hard to write edge-case tests in doctest. - - Porting the doctests where people have tried to juggle both goals across Python versions is a mess. The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.). We can still leave the main line of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an intersphinx tree underneath the main ZTK docs, which could serve as the gateway into them. The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation. I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the edge-case tests. I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion: http://svn.zope.org/zope.event/branches/tseaver-new_style_docs I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback). The rendered docs from that branch are here: http://palladion.com/static/zope.event-docs/ I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view): https://code.launchpad.net/~tseaver/zope.event/new_style_docs I am attaching an annotated diff for those who would prefer to see the changes in that format. I would say that this cleanup effort should start at the bottom layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move upwards (in the depenency sense) over time. Thoughts? I agree with the main thrust. (Of course, I disagree with some of your judgments of doctest, but that's mostly beside the point.) I'm trying hard to keep my prejudices out of the discussion;). - Documentation should be written for documentation's sake. The emphasis should be on helping people understand what the software is for and how to use it, *not* on coverage. Amen. - Documentation should be executable. Manuel helps a lot for this. Amen. The Sphinx tools for this are pretty neat, too. - I can almost guarantee that any examples that aren't tested will be wrong. I tried to do a good job on the bobo docs. I made almost all of the examples executable, and the ones I didn't had an amazing tendency to have bugs. I'm not against having the snippets be executable, because I *do* want them to work. I just don't want to encourage anyone to think that they are testing the software when they write the snippets, or execute them. Executing the snippets is testing the documentation, not the software. - I agree that tests should be separate from documentation. You can get some of your coverage from the docs, but you'll need tests for edge cases and details not addressed by the docs. I would actually prefer to measure coverage (objectively) without reference to the snippets
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 4/17/10 5:20 PM, Tres Seaver wrote: - Documentation should be written for documentation's sake. The emphasis should be on helping people understand what the software is for and how to use it, *not* on coverage. Amen. - Documentation should be executable. Manuel helps a lot for this. Amen. The Sphinx tools for this are pretty neat, too. Documentation needn't be executable. Documentation should be right. But sometimes documentation is just documentation and automating the checking of its rightness isn't reasonable, especially if making it executable takes the slightest bit of meaning away from it as documentation. Sorry. No free lunch here. - C ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 18 April 2010 05:20, Tres Seaver tsea...@palladion.com wrote: I'm not against having the snippets be executable, because I *do* want them to work. I just don't want to encourage anyone to think that they are testing the software when they write the snippets, or execute them. Executing the snippets is testing the documentation, not the software. Everyone who's ever written a doctest, or is thinking about doing so: Please frame this and hang it above your monitor. :-) Martin ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages
On 17 April 2010 09:41, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3. I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. Thoughts? +1000, to both aim and approach. :) Martin ___ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
