[issue29428] Doctest documentation unclear about multi-line fixtures

[Carol Willing]

Carol Willing  added the comment:

I've closed the associated PR https://github.com/python/cpython/pull/45 with a 
recommendation to resubmit a smaller, more atomic PR if desired.

--
nosy: +willingc
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Bump. Could I get a few more eyes looking at the current state of 
https://github.com/python/cpython/pull/45/ ? 

* @bitdancer made some suggestions. I accepted some, and replied to others 
where I think we should keep looking for common ground. I'd like to see replies.

* I've made changes in response to other comments.

I think pull/45 represents a targeted set of changes to those parts of the 
module documentation which are particularly unclear. It is not a wholesale 
re-write; that's a task for a different time. I believe it is responsive to 
comments so far. I'd appreciate more eyes to help arrive at consensus on those 
improvements where we can. 

Thank you!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Pull Request https://github.com/python/cpython/pull/45 submitted to new Github 
repo. 

I would appreciate a review.

I attempted to balance all the different opinions in the discussion below: stay 
concise, but also improve the clarity.

--
pull_requests: +45

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

I've drafted some fairly restricted changes to the doctest documentation page. 
They are in my Github branch, 
https://github.com/JDLH/cpython/tree/Issue29428_doctest_docs . The diffs are at 
https://github.com/JDLH/cpython/commit/223ef8f8a6d2fbec6db774912028abb4d2ff88b6 
. 

There currently is no official Python github repot against which to make a pull 
request. In a few days, once it appears, I'll make a pull request.

In the meantime, I can take review comments and improve it. If you are 
interested, please review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Marco:

> To have a wide view, usually it is a good idea to start from the beginning:
> 
> https://groups.google.com/forum/#!msg/comp.lang.python/DfzH5Nrt05E/Yyd3s7fPVxwJ

Thank you, that is a very interesting thread. Clearly the creator of doctests, 
Tim Peters, is a well-regarded person in the Python world. 

On the other hand, I was surprised to see some of the examples used in the 
present-day documentation in Tim's original 1999 message. I find it remarkable 
that in 18 years since then, no rewrite or editing has replaced those examples 
with better ones.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Marco Buttu]

Marco Buttu added the comment:

> In parallel, I was thinking of some howto content that I would like to
> see documented.

Great :-)

> How to decide what to test with doctests and what with unittests.
> I have a feeling that the sweet spot of doctests is functionality
> which you can invoke in one line, and which produces one line of
> output which you can match against a reference string. If it takes
> many lines to set up, or invoke, or checking for correctness is
> more than a test for string equality, then maybe unittest is a
> better tool.

All your questions come up every time I speak about doctest, and these doubts 
are the main reason for my previous sentence: "there is more to say [about 
doctest] than only the API". That is why IMO an howto could be really helpful. 
What doctest checks is not one line of output, but the output as a string, and 
of course it could have multiple lines. Try to execute the doctest (with the -v 
option) for this docstring:

def foo():
""" 
>>> a = 33
>>> a
33
>>> for i in range(2):
...   print(i)
...
0
1
"""

Three tests will be executed: one for `a=33`, expecting nothing, one for `a`, 
expecting 33 as output, and one test for the whole `for` block, expecting one 
output composed of two lines.

For several reasons IMO doctest is not a substitute for unittest. They are 
different tools for different purposes, as thier name point up. To have a wide 
view, usually it is a good idea to start from the beginning:

https://groups.google.com/forum/#!msg/comp.lang.python/DfzH5Nrt05E/Yyd3s7fPVxwJ

Anyway, IMO this is not the right place to discuss and explain the objectives 
of doctest. I propose you to open a repository on github, work and discuss 
there, and when all is ready, we can ask for a review. Please write here the 
link to the github repo, so anyone else wants to contribute can find it. Thanks 
again for your time

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Marco, thank you for the suggestion of a howto. That is a good idea. 

In parallel, I was thinking of some howto content that I would like to see 
documented. 

• How to decide what to test with doctests and what with unittests. I have a 
feeling that the sweet spot of doctests is functionality which you can invoke 
in one line, and which produces one line of output which you can match against 
a reference string. If it takes many lines to set up, or invoke, or checking 
for correctness is more than a test for string equality, then maybe unittest is 
a better tool.

• An effective way to write doctests. I suspect that some people write doctests 
by exercising the functionality in Python's interactive mode, then copy and 
paste the transcript from that session to the doctests. I don't do it that way, 
I author in the docstring. Maybe I'm not doing it the best way.

• How to author doctests so that they both prove the module correct, and 
provide clear and readable documentation. I imagine some effective tests for 
edge cases and error conditions are important to have, but are not readable 
documentation. Maybe such tests belong in a unittest framework instead of as 
doctests.

• How to run doctests from a unittest harness (your earlier note about the 
unittest API would be a part of this).

A problem for me is that I think I don't have the experience and wisdom to give 
good advice in these areas. I would be happy to start such a howto, and to 
accept feedback, and to edit it into good prose. I would need wiser people to 
contribute good ideas for the howto.

Also, Marco, thank you for being willing to review a patch.  That is helpful.

My next step is to check out the documentation source, to be in a position to 
make a patch.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Marco Buttu]

Marco Buttu added the comment:

IMO if you would like to apply big changes to the current doc, as you wrote 
before, maybe is worth considering to propose a separate howto, instead of wide 
changes to the current page. I am suggesting this to you, because looking at 
the other modules of the standard library, I see they just explain the API, 
extending with examples when needed. When is worth adding extra documentation, 
in a different style, I see the usual solution is to extend with an howto. Look 
at the argparse module and related howto, for instance:

https://docs.python.org/3/library/argparse.html
https://docs.python.org/3/howto/argparse.html

I also think an howto in this case is appropriate, because there is more to say 
than only the API. If you want to follow the howto way, I would like to 
contribute. In any case, I am disposed to review your patch

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Marco and David, thank you again for your prompt replies. 

Let me respond to both of your comments on the doctest module documentation 
itself.


[Marco] > The example in section 26.3.3.2 [1], before the compound statement 
(the if/else), contains two simple statements. I also think the example is 
complete and straightforward for its purpose, and it does not have to be 
complicated adding avanced features.

You are correct, it contain two simple statements. I missed this when I first 
read the example.  Part of the problem is that the example is contrived. It is 
an actual test. I think it would be better to replace these two lines by a 
two-line test, one which sets up state and another which exercises the module 
under test.

You believe "the example is complete and straightforward for its purpose, and 
it does not have to be complicated adding avanced features." I am giving you 
feedback that it was not complete and straightforward for me, as an ordinary 
software developer trying to use the documentation. 


[Marco] > If you want a more complex control over your tests, for instance 
adding fixtures, you can find how to do it later...

No, control over tests is not what I am bringing up in this issue. I an 
concerned with documentation of doctest "Examples".


[Marco] > I am -1 on this [changing 26.3.3.2]. IMHO there is no need to add 
extra description to a such self-explaining, clear and concise example. But 
this I think is also clear from the beginning. Infact, the first line of the 
doc says: "The doctest module searches for pieces of text that look like 
interactive Python sessions"

[David] > As for the prompts, I agree with Marco: I don't think there is any 
need to add words, since it clearly says it is supposed to look like an 
interactive session and the reader gets reminded of the rules for that by the 
example provided.  IMO it is not appropriate to re-document those rules here.


My feedback to you is that for me, as an ordinary software developer trying to 
use the doctest module, this section is not "self-explaining, clear and 
concise", and that "How Docstring examples are recognised" is not "clear from 
the beginning". 

Perhaps we have a difference of opinion on how clear and effective this 
documentation is.  The best way to resolve that would be to gather data: having 
several ordinary software developers reading the document, then measuring their 
comprehension. That would be hard to arrange. At least this bug report will 
serve as a data point.

I believe that a library module documentation should at some point clearly 
specify how the module works. I believe section Section "26.3.3.2. How are 
Docstring Examples Recognized?" does poorly at that. To say, "In most cases a 
copy-and-paste of an interactive console session works fine, but doctest isn’t 
trying to do an exact emulation of any specific Python shell." is not a 
specification. It's a blurry generalisation. The list of "fine print" is at a 
better level of detail, but still doesn't mention how the parser treats 
statements, nor how prefixes relate to statement boundaries.

I am giving you feedback that for me, as a developer trying to use the doctest 
module, this documentation failed to give me the comprehension I needed to be 
effective.


[Marco] >Also this section [26.3.3.3, execution context] IMO is well written 
and clear. I do not know what to say... If someone else thinks it is not enough 
understandable, maybe the better think to do is to add a very small example, 
with two functions, that shows their docstrings have independent globals.

[David] > It does appear that "test" is being used ambiguously in the docs.  In 
most places it means a single statement, but in execution context it appears to 
be being used as a synonym for "docstring".  In that section it should be 
replaced by "docstring".  It would probably be worth checking the rest of the 
doc to make sure it is used consistently elsewhere as well.

[David] > The execution context section makes it clear that you can use names 
defined earlier, as does the fact that doctests emulate interactive sessions.  
Both of these make it clear you can do multi line examples that use shared 
state. 

Again, I am giving you feedback that this document was not "clear" for me. I am 
taking the time to write this issue because I think the documentation has 
weaknesses, and would like to help improve it.

I agree with David, that one problem in this section is that the word "test" is 
used with three different meanings. Some instances of the word "test" should be 
changed to "docstring" or similar. That would help. 


[David] > So in summary, I think we *need* a one word change in the execution 
context section, and we'll consider any other enhancements you want to propose.

All right, I'll work on a patch that makes that one-word change, and offers 
some other concise improvements. I expect it will be more that you think 
necessary, 

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Marco, thank you for the suggestion to watch Raymond Hettinger's talk. 

> I suggest you to watch this talk of Raymond Hettinger, before going on:
>
> https://www.youtube.com/watch?v=voXVTjwnn-U

That video is 63 minutes long, and a lot of those minutes are taken up by 
stories. I will watch the rest when I get time. I wish his insights were 
summarised in writing; I'll bet that would take much less than 63 minutes to 
read. I find that written text is a much faster way to deliver information, 
though maybe it takes the teacher more time to write well than to give a talk.

I understand your point about the need for a review, and reviews being the 
bottleneck. The "Lifecycle of a Patch" document[1] is clear about that.

[1] https://docs.python.org/devguide/patch.html#reviewing

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

Jim DeLaHunt added the comment:

Marco, thank you for your prompt comment. 

However, 
> Maybe you are running the doctest with Python 3. The StringIO module is no 
> more available in Python 3, so your doctest will fail on Python 3 

Please let me be clear. This issue is not about the StringIO module or about 
any of my doctests. It is about the documentation of the doctests module, and 
the ways in which that documentation is less helpful than it could be.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[R. David Murray]

R. David Murray added the comment:

And thanks for wanting to improve the docs!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[R. David Murray]

R. David Murray added the comment:

It does appear that "test" is being used ambiguously in the docs.  In most 
places it means a single statement, but in execution context it appears to be 
being used as a synonym for "docstring".  In that section it should be replaced 
by "docstring".  It would probably be worth checking the rest of the doc to 
make sure it is used consistently elsewhere as well.

As for the prompts, I agree with Marco: I don't think there is any need to add 
words, since it clearly says it is supposed to look like an interactive session 
and the reader gets reminded of the rules for that by the example provided.  
IMO it is not appropriate to re-document those rules here.  However, if you 
come up with a really concise addition that clarifies it, we'll consider it.

I don't think an additional example of using previous state is required, and 
certainly not an additional section.  The execution context section makes it 
clear that you can use names defined earlier, as does the fact that doctests 
emulate interactive sessions.  Both of these make it clear you can do multi 
line examples that use shared state.  However, if you come up with an example 
that adds value to the documentation, we'll certainly consider it.

So in summary, I think we *need* a one word change in the execution context 
section, and we'll consider any other enhancements you want to propose.

--
nosy: +r.david.murray
versions: +Python 2.7, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Marco Buttu]

Marco Buttu added the comment:

> I just had a problem with doctests. It manifested as an
> error that occurred when I did
> >>> import StringIO in my doctest.

Maybe you are running the doctest with Python 3. The StringIO module is no more 
available in Python 3, so your doctest will fail on Python 3 and will pass on 
Python 2.

> 1. There is no example of a multiple-line doctest fixture:
> specifically, a fixture which involves at least one line to
> set up state, and another line to be the example that tested.
> I suggest adding such an example as a new section, "26.3.2.
> Usage: multi-line tests".
> Note that the example in 26.3.3.2 doesn't count: that is a
> Compound Statement, and I'm talking about a test consisting
> of multiple interactive statements.

The example in section 26.3.3.2 [1], before the compound statement (the 
if/else), contains two simple statements. I also think the example is complete 
and straightforward for its purpose, and it does not have to be complicated 
adding avanced features. If you want a more complex control over your tests, 
for instance adding fixtures, you can find how to do it later, in [2].

> 2. Section "26.3.3.2. How are Docstring Examples Recognized?" does
> not talk about when to use a PS1 prefix (>>>) and when to use a
> PS2 prefix (...). It should say to use the PS1 at the start of each
> Python "interactive statement", that is, a statement list ending with
> a newline, or a Compound Statement. And, to use the PS2 to prefix each
> continuing line within a Compound Statement. 

I am -1 on this. IMHO there is no need to add extra description to a such 
self-explaining, clear and concise example. But this I think is also clear from 
the beginning. Infact, the first line of the doc says: "The doctest module 
searches for pieces of text that look like interactive Python sessions"

> 3. Section "26.3.3.3. What’s the Execution Context?" is not
> clear about the crucial difference between state that accumulates
> within a single docstring from Example to Example, and that is reset
> from one docstring to another. The phrase "so that one test in M can’t
> leave behind crumbs that accidentally allow another test to work"
> can be interpreted as applying within a single docstring. The phrase
> "names defined earlier in the docstring being run" is easy to miss.

Also this section IMO is well written and clear. I do not know what to say... 
If someone else thinks it is not enough understandable, maybe the better think 
to do is to add a very small example, with two functions, that shows their 
docstrings have independent globals.

> Unless someone tells me otherwise, I'll get around to filing similar
> issues against Python 3.6 and Python 2.7.
> Then it's my intention to write improvements to the documentation
> for Python 3.7, 3.6, and 2.7, and submit each of those.

Keep in mind that writing a patch is not enough. The patch has to be reviewed, 
and that means other people have to find the time to work on it. I suggest you 
to watch this talk of Raymond Hettinger, before going on:

https://www.youtube.com/watch?v=voXVTjwnn-U


[1] 
https://docs.python.org/3/library/doctest.html#how-are-docstring-examples-recognized
[2] https://docs.python.org/3/library/doctest.html#unittest-api

--
nosy: +marco.buttu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29428] Doctest documentation unclear about multi-line fixtures

[Jim DeLaHunt]

New submission from Jim DeLaHunt:

I just had a problem with doctests. It manifested as an error that occurred 
when I did
>>> import StringIO
in my doctest. 

After some investigation, I realised that my actual problem was that the 
doctest library documentation https://docs.python.org/3/library/doctest.html 
left me unclear about three aspects of multi-line doctest fixtures.

1. There is no example of a multiple-line doctest fixture: specifically, a 
fixture which involves at least one line to set up state, and another line to 
be the example that tested.  I suggest adding such an example as a new section, 
"26.3.2. Usage: multi-line tests".  Note that the example in 26.3.3.2 doesn't 
count: that is a Compound Statement, and I'm talking about a test consisting of 
multiple interactive statements.

2. Section "26.3.3.2. How are Docstring Examples Recognized?" does not talk 
about when to use a PS1 prefix (>>>) and when to use a PS2 prefix (...). It 
should say to use the PS1 at the start of each Python "interactive statement", 
that is, a statement list ending with a newline, or a Compound Statement. And, 
to use the PS2 to prefix each continuing line within a Compound Statement. 

3. Section "26.3.3.3. What’s the Execution Context?" is not clear about the 
crucial difference between state that accumulates within a single docstring 
from Example to Example, and that is reset from one docstring to another. The 
phrase "so that one test in M can’t leave behind crumbs that accidentally allow 
another test to work" can be interpreted as applying within a single docstring. 
The phrase "names defined earlier in the docstring being run" is easy to miss.

A StackOverflow Question and Answer describing my problem and my insight about 
what the library documentation didn't tell me is at 
http://stackoverflow.com/questions/41070547/why-is-importing-a-module-breaking-my-doctest-python-2-7
 . A blog post of mine, a source for documentation improvements, is at 
http://blog.jdlh.com/en/2017/01/31/multi-line-doctests/ .

I actually encountered this problem in a Python 2.7 context, with 
https://docs.python.org/2/library/doctest.html . But it also applies to Python 
3.6 https://docs.python.org/3/library/doctest.html and to the current dev 
https://docs.python.org/dev/library/doctest.html . In the spirit of fixing the 
new stuff first, I am filing this issue against Python 3.7. 

Unless someone tells me otherwise, I'll get around to filing similar issues 
against Python 3.6 and Python 2.7.
Then it's my intention to write improvements to the documentation for Python 
3.7, 3.6, and 2.7, and submit each of those. It looks like they will be pretty 
similar, because the library documentation is pretty similar in each version.

I'm new to Python issue filing and doc fixing, so I would appreciate correction 
if (when?) I start to blunder.

--
assignee: docs@python
components: Documentation
messages: 286843
nosy: JDLH, docs@python
priority: normal
severity: normal
status: open
title: Doctest documentation unclear about multi-line fixtures
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com