[issue34071] asyncio: repr(task) raises AssertionError for coros which loop.create_task accepts; complications ensue
Jim DeLaHunt added the comment: As the original reporter, I have no objection to closing this old report. It remains in the historical record. That was its purpose all along. Thank you to all the bug data maintainers! -- ___ Python tracker <https://bugs.python.org/issue34071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36675] Doctest directives and comments missing from code samples
Jim DeLaHunt added the comment: My goodness, things get complex sometimes. If we cannot make Sphinx preserve doctest directives and comments, perhaps we should go back to the historical bug discussion to look at workarounds which we considered earlier. For instance, maybe we should modify the text surrounding the examples to explain that doctests directives should appear there, but that our tool chain currently removes them. At the moment, I see doctest directives in the doctest source code at: https://github.com/python/cpython/blob/master/Doc/library/doctest.rst#directives which do not appear in the corresponding HTML output at: https://docs.python.org/3/library/doctest.html#directives How about rewording the text before each of those examples? Or maybe finding some way to show those examples as literal text which Sphinx won't modify, even if it is not formatted like Python code examples? By the way, the discussion of this same problem back in 2011-2012 is at #12947 . At that time, we applied a "monkey patch" to the Sphinx code. I haven't read the discussion closely enough to figure out if such a patch would help now. -- ___ Python tracker <https://bugs.python.org/issue36675> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36675] Doctest directives and comments missing from code samples
Jim DeLaHunt added the comment: We discovered and fixed this same problem back in 2011-2012 with #12947 . That was apparently the source of the monkeypatch that was removed as "obselete code" on 2019-09-12. That old issue commentary has some suggestions about other workarounds. This includes adding explanatory text about the fact that doctest directives are missing from the examples which should be showing them. Maybe some of those workarounds would be effective for us now. -- nosy: +JDLH ___ Python tracker <https://bugs.python.org/issue36675> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34176] Asyncio StreamReader fails to close Transport
New submission from Jim DeLaHunt : Asyncio's StreamReaderProtocol[1] often returns True from Protocol.eof_received(). This tells the Transport that "closing the transport is up to the protocol" [2]. However, StreamReaderProtocol does not call Transport.close(). More precisely, StreamReaderProtocol leaves the transport interaction to StreamReader. It calls StreamReader.feed_eof()[3]. But StreamReader does not call self._transport.close(). It only sets a self._eof flag, and awakens any of its functions which were waiting for data. The only occurrence of self._transport.close() occurs in StreamWriter[4]. There is none in StreamReader or StreamReaderProtocol. I believe that the fix should be for StreamReader to call self._transport.close(). However, it may need to avoid this if StreamReaderProtocol._over_ssl is True, and I don't immediately see how it can tell this. Also, it may need to wait for anything waiting for self._waiter to run, before closing the Transport. Or, maybe StreamReaderProtocol should not return True from Protocol.eof_received(). I discovered this when using a Transport of my own devising, which reads from a file instead of from a socket, with asyncio.streams. The file did not close in the unit test. My transport duly called Protocol.eof_received(), but received a True in response, so did not close the file. My workaround will probably be to not believe what Protocol.eof_received() tells me, and to close my transport regardless. [1] Lib/asyncio.streams.StreamReaderProtocol.eof_received, line 259-266. [2] 19.5.4.2.3. "Streaming protocols" <https://docs.python.org/3/library/asyncio-protocol.html> [3] Lib/Lib/asyncio.streams.StreamReader.feed_eof, lines 419-421. [4] Lib/Lib/asyncio.streams.StreamWriter.close, lines 316-317. All line numbers are as of tag v3.7.0, commit 1bf9cc509326bc42cd8cb1650eb9bf64550d817e . Bug observed in v3.7.0. Not yet investigated in HEAD revision. -- components: asyncio messages: 322047 nosy: JDLH, asvetlov, yselivanov priority: normal severity: normal status: open title: Asyncio StreamReader fails to close Transport type: behavior versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue34176> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34071] repr(task) raises AssertionError for coros which loop.create_task accepts; complications ensue
Jim DeLaHunt added the comment: This is what I observe when I run my original program with Python 3.7.0. Notice that the Task object instantiation fails with a clear error message: % python -c 'import sys; print(sys.version)' 3.7.0 (default, Jun 28 2018, 06:01:52) [Clang 8.0.0 (clang-800.0.42.1)] % python bug_task_repr_1.py type(ag()) is asyncio.iscoroutine(ag)? False repr(ag()) is Traceback (most recent call last): File "bug_task_repr_1.py", line 42, in task = asyncio.get_event_loop().create_task(ag()) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 386, in create_task task = tasks.Task(coro, loop=self) TypeError: a coroutine was expected, got Also, the unit test example fails due to asyncio.test_utils no longer being present in Python 3.7.0. I am comfortable with a workaround of upgrading to 3.7.0 or later for anyone who encounters this problem. I file this bug report mostly to let people who encounter this problem find this report in web searches, to help them understand the cause, and to see the workaround (patching Task._repr_info) if upgrading to 3.7 is not feasible. -- ___ Python tracker <https://bugs.python.org/issue34071> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34071] repr(task) raises AssertionError for coros which loop.create_task accepts; complications ensue
New submission from Jim DeLaHunt :
In Python 3.6.5, Task.__repr__() with raise an AssertionError for
certain, arguably incorrect, coroutines which the instantiation of Task()
accepts. repr(task) thus fails with an AssertionError, instead of
returning a string which describes the task object and why its coroutine
is wrong. Complications ensue.
In particular, if this Task is used in a unittest.TestCase, and appears in test
failure diagnostics, the test
diagnostics are discarded when the AssertionError occurs, masking the
problem in the code under test.
In Python 3.7.0, Task.__init__() checks the supplied coroutine argument,
using the same asyncio.iscoroutine() method which Task.__repr__() calls.
Thus, while repr(task) can still raise an AssertionError, it is much harder to
reach this situation. Task.__init__() is likely to raise an exception first,
and the diagnostics will be pretty clear.
To reproduce:
1. Here is a simple, perhaps misguided, attempt at a coroutine, and a Task
which uses it:
import asyncio
async def ag():
yield None
print("type(ag()) is ", type(ag()))
print("asyncio.iscoroutine(ag)? ", asyncio.iscoroutine(ag))
print("repr(ag()) is ", repr(ag()))
task = asyncio.get_event_loop().create_task(ag())
print("type(task) is ", type(task))
print("asyncio.iscoroutine(task._coro)? ", asyncio.iscoroutine(task._coro))
print("repr(task) is ")
print(repr(task))
The output from Python 3.6.5 is:
type(ag()) is
asyncio.iscoroutine(ag)? False
repr(ag()) is
type(task) is
asyncio.iscoroutine(task._coro)? False
repr(task) is
Traceback (most recent call last):
File "bug_task_repr_1.py", line 31, in
print(repr(task))
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_tasks.py",
line 15, in _task_repr_info
coro = coroutines._format_coroutine(task._coro)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py",
line 276, in _format_coroutine
assert iscoroutine(coro)
AssertionError
Observed behaviour:
Here is a unit test which attempts to exercise the above Task. The test fails,
but the AssertionError in Task.__repr__() discards the error diagnostics:
from asyncio import test_utils
class Test(test_utils.TestCase):
def setUp(self):
super().setUp()
self.loop = self.new_test_loop()
def test_wrongly_scheduled(self):
# this is a simple, and maybe mistaken, coroutine
async def ag():
yield None
_ = self.loop.create_task(ag())
test_utils.run_once(self.loop)
The output from Python 3.6.5 is:
% python -m unittest bug_task_repr_2.py
Exception in default exception handler
Traceback (most recent call last):
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
line 1291, in call_exception_handler
self.default_exception_handler(context)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
line 1263, in default_exception_handler
value = repr(value)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_tasks.py",
line 15, in _task_repr_info
coro = coroutines._format_coroutine(task._coro)
File
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/coroutines.py",
line 276, in _format_coroutine
assert iscoroutine(coro)
AssertionError
.
--
Ran 1 test in 0.001s
OK
Note that the error traceback is all about the exception from Task.__repr__(),
and the diagnostics about why the unit test failed is discarded.
Expected behaviour:
Here is the same unit test, but this time with a replacement method for
Task._task_repr_info() which avoids the AssertionError.
import asyncio
from asyncio import test_utils
import types
class Test(test_utils.TestCase):
def setUp(self):
super().setUp()
self.loop = self.new_test_loop()
def test_wrongly_scheduled(self):
# this is a simple, and maybe mistaken, coroutine
async def ag():
yield None
task = self.loop.create_task(ag())
task._repr_info = types.MethodType( task_safe_repr_info, task )
test_utils.run_once(self.loop)
def task_safe_repr_info(self):
'''task.task_safe_repr_info(): string list representation of task. Won't
raise AssertionError
If you have a Task object which is raising an AssertionError from
"assert iscoroutine(coro)", then assign to instantiated object:
task._repr_info = task_safe_repr_info
Patched task object should be able to survive the AssertionError and
deliver a useful representation.
'''
[issue33649] asyncio docs overhaul
Jim DeLaHunt added the comment: I'm a developer using Python in my application. I just spent the last couple of weeks learning about asyncio with the present documentation. I am very happy to see that work is underway for improved documentation. I would be glad to take on writing tasks or to review drafts, if there is a way to plug myself into that work. I like the general structure. Some specific topics which would have helped me in my recent learning: * As an application developer using an event loop as part of the application, how do I develop a Protocol specific to my app, and use with Streams? An important part of this is being clear what the interface is between Transport and Protocol. This interface resides in both classes; Protocol calls Transport methods, and Transport calls Protocol methods. * As an application developer using an event loop as part of the application, how do I develop a Transport specific to my app, and use it with Streams? (In my case, I have two transports in mind: a file, containing data archived from a TCP network port, and an RS232 serial connection, carrying data that might otherwise be sent via a TCP network port. It seems to me that I should be able at run-time to select from any of those three Transports, and apply my app-specific Protocol when creating my connection.) * As an application developer, how do I give a function which is not a coroutine to the event loop for scheduling and execution? (I think the answer is loop.call_soon(), but the docs don't say "to run your function call loop.call_soon(myfunc...)", they say "call_soon(): Arrange for a callback to be called as soon as possible". Not the same thing. And, is a "callback" different "an arbitrary function"? Not clear.) * As an application developer, how do I make an syncio-based streaming server and streaming client respond cleanly to interrupts, e.g. control-C in a command-line app? (Existing docs hints at catching signals, but don't show how to combine those with exception handlers in the coroutine to shut down connections cleanly.) * As a Transport developer, what interface must I provide between Transport and Protocol? What services does the EventLoop provide to help implement my Transport? As mentioned before, to me it looks like the Transport-Protocol interface resides in both classes; Protocol calls Transport methods, and Transport calls Protocol methods. The EventLoop services useful to Transports are not gathered in a discussion of Transports. There is only an enumeration of the EventLoop methods, regardless of purpose. Editorial checks I suggest making to the revision: Rewrite to describe the behaviour in the doc, instead of delegating to a PEP. No more "this class is almost like the thing in PEP xyz, except for these differences". Describe the asyncio class, and steal text from PEP xyz as needed to do that. Every section about a thing must start by saying what that thing is and what it is used for. There are some big counterexamples in the current text. e.g. "19.5.1.2. Calls" <https://docs.python.org/3/library/asyncio-eventloop.html#calls> starts out with: "Most asyncio functions don’t accept keywords. If you want to pass keywords to your callback, use functools.partial()." That does not describe what "calls" are, nor what they are used for. And, in this case, a better title might be, "Getting the event loop to call regular functions". e.g. "19.5.1.4. Futures" <https://docs.python.org/3/library/asyncio-eventloop.html#futures> has no starting text. This is the first heading mentioning Futures when reading sequentially in the asyncio doc, so I interpreted it as a place to learn about Futures. It is not. This section would be better titles "Eventloop methods for Futures". The proposed "Low-level APIs" section might solve this problem by explaining the nature and purpose of Futures before mentioning the EventLoop factory for Futures objects. Consider separating the explanation of the nature and purpose of a thing from the reference to the methods of the thing. The proposed structure talks about "High-Level APIs" and "Low-Level APIs", but not "Architecture" and "API Reference". I would suggest injecting an "Architecture" section which can give the nature and purpose of each of the public classes in Asyncio, without enumerating their methods. Then the sections labelled "High-Level APIs" and "Low-Level APIs" can be the API reference. I hope these suggestions are helpful. I'll be monitoring this issue to see how I can help. -- nosy: +JDLH ___ Python tracker <https://bugs.python.org/issue33649> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: Bump. Could I get a few more eyes looking at the current state of https://github.com/python/cpython/pull/45/ ? * @bitdancer made some suggestions. I accepted some, and replied to others where I think we should keep looking for common ground. I'd like to see replies. * I've made changes in response to other comments. I think pull/45 represents a targeted set of changes to those parts of the module documentation which are particularly unclear. It is not a wholesale re-write; that's a task for a different time. I believe it is responsive to comments so far. I'd appreciate more eyes to help arrive at consensus on those improvements where we can. Thank you! -- ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29562] test_getgroups of test_posix fails (on OS X 10.10)
Jim DeLaHunt added the comment: > Note that the result of getgroups(2) is fixed on login, while "id -G" > reflects the current state of the user database on macOS. Wow, that's interesting! Thank you for this information. The test code for test_getgroups does not mention this interaction. I can certainly see how it could affect the test. Maybe it should be added? Since I last tried that test, I've logged out and restarted several times, and changed OS to Mac OS X 10.11 El Capitan. Nothing like changing several independent variables at once while diagnosing! I will try the test again and report back. -- ___ Python tracker <http://bugs.python.org/issue29562> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29520] Documentation uses deprecated "defindex.html" Sphinx template
Jim DeLaHunt added the comment: Jaysinh, thank you for checking. From your log, I see you are using Sphinx version 1.3.6. I am seeing this problem with Sphinx version 1.5.2. I think you need Sphinx 1.5.2 or later to see the warning message. I notice my original bug description didn't specify a Sphinx version. Now we know that it needs to be a fairly recent version of Sphinx. >From the discussion in https://github.com/sphinx-doc/sphinx/issues/2986, it >looks like the Sphinx team added the warning message at or shortly after the >release of 1.5.0. -- ___ Python tracker <http://bugs.python.org/issue29520> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29562] test_getgroups of test_posix fails (on OS X 10.10)
Jim DeLaHunt added the comment: The Mac OS 10.10 man page for initgroups(3) says: "Processes should not use the group ID numbers from getgroups(2) to determine a user's group membership. The list obtained from getgroups() may only be a partial list of a user's group membership. Membership checks should use the mbr_gid_to_uuid(3), mbr_uid_to_uuid(3), and mbr_check_membership(3) functions." (http://www.manpagez.com/man/3/initgroups/ -- not official Apple page, but it matches what I see in my OS.) When the man page says, "The list obtained from getgroups() may only be a partial list of a user's group membership.", and the list from `id -G` is presumably a complete list, should we understand that Apple is saying their getgroups(2) implementation isn't POSIX-compliant? If so, maybe we should skip test_getgroups on Mac OS X systems? Or, should we consider rewriting os_getgroups_impl() to use a Mac-specific implementation on Mac OS X? -- ___ Python tracker <http://bugs.python.org/issue29562> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29562] test_getgroups of test_posix fails (on OS X 10.10)
Jim DeLaHunt added the comment:
I guess I didn't state the things I find odd about what the new test_getgroups
results.
1. `os.getgroups()` used to return group (395, 'com.apple.access_ftp'), but no
longer does. I don't see a reason why.
2. `os.getgroups()` is returning 2 fewer group id's than `id -G`, even as the
total number of groups is reduced. This is not the behaviour of an API limited
by {NGROUPS_MAX}.
--
___
Python tracker
<http://bugs.python.org/issue29562>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29562] test_getgroups of test_posix fails (on OS X 10.10)
Jim DeLaHunt added the comment:
Some diagnosis.
Group `com.apple.sharepoint.group.1` appears to be related to a certain kind of
file sharing, but I don't have hard evidence.
Its only member was a test user I created as part of screen sharing with Apple
Support.
```
% dscacheutil -q group -a name com.apple.sharepoint.group.1
name: com.apple.sharepoint.group.1
password: *
gid: 701
users: testuser
```
I removed File Sharing for this user's home directory.
1. Open System Preferences... Sharing.
2. Click on "File Sharing", which is checked. In the right pane, a list of
shared folders appears.
3. Click on the entry "Testuser Public Folder" in the Shared Folders list.
4. Click on the "-" button below the Shared Folders list. The "Testuser Public
Folder" entry disappears.
Having done that, the group `com.apple.sharepoint.group.1` no longer appeared.
```
% dscacheutil -q group -a name com.apple.sharepoint.group.1
%
```
Interestingly, `test_getgroups` still failed, and still had a discrepancy of
two groups from the output of `id -G`.
```
% ./python.exe -m unittest -v test.test_posix.PosixTester.test_getgroups
test_getgroups (test.test_posix.PosixTester) ... FAIL
==
FAIL: test_getgroups (test.test_posix.PosixTester)
--
Traceback (most recent call last):
File "/Users/jdlh/workspace/cpython/Lib/test/test_posix.py", line 841, in
test_getgroups
self.assertEqual(len(symdiff), 0, msg)
AssertionError: 2 != 0 : id -G and posix.groups() should have zero difference.
Groups in id -G but not posix.groups(): [(395, 'com.apple.access_ftp'), (398,
'com.apple.access_screensharing')]
Groups in posix.groups() but not id -G: []
(Effective GID (20) was disregarded.)
--
Ran 1 test in 0.013s
FAILED (failures=1)
```
Earlier, group `com.apple.access_ftp` was not part of the difference. Now it
is. The output of `id -G` didn't change. The implementation of
`posix.getgroups()` didn't change. It calls getgroups (2), I believe:
https://github.com/python/cpython/blob/master/Modules/posixmodule.c#L6078-L6103
That makes me think that the behaviour of getgroups (2) in Mac OS is behaving
differently than we expect.
`man 2 getgroups` gives documentation. (I can't find this page at an apple URL,
but http://www.manpagez.com/man/2/getgroups/ seems to have the same content.)
It says,
>>> "To provide compatibility with applications that use getgroups() in
>>> environments where users may be in more than {NGROUPS_MAX} groups, a
>>> variant of getgroups(), obtained when compiling with either the macros
>>> _DARWIN_UNLIMITED_GETGROUPS or _DARWIN_C_SOURCE defined, can be used that
>>> is not limited to {NGROUPS_MAX} groups. However, this variant only returns
>>> the user's default group access list and not the group list modified by a
>>> call to setgroups(2) (either in the current process or an ancestor
>>> process). Use of setgroups(2) is highly discouraged, and there is no
>>> foolproof way to determine if it has been previously called."
I don't know how to determine if my copy of Mac OS X 10.10 was complied with
either of these two macros.
On my system, I chased NGROUPS_MAX down to /usr/include/sys/syslimits.h:84,
where it is set to 16. That is more than the number of groups `id -G` is
reporting, so I don't see how that is relevant.
```% id -G
20 507 12 61 80 98 399 33 100 204 395 398
```
This is 12 groups, whereas before it was 13 groups (see my message from
2017-02-15 02:03). This is unsurprising. However, the number of groups
returned by posix.getgroups() has also shrunk by 1:
```% ./python.exe -c 'import grp,os; g={i: (n, p, i, mem) for (n, p, i, mem) in
grp.getgrall()}; print(sorted([(i, g[i][0]) for i in os.getgroups()]) )'
[(12, 'everyone'), (20, 'staff'), (33, '_appstore'), (61, 'localaccounts'),
(80, 'admin'), (98, '_lpadmin'), (100, '_lpoperator'), (204, '_developer'),
(399, 'com.apple.access_ssh'), (507, 'xampp')]
```
Notice that group (395, 'com.apple.access_ftp') is no longer being returned by
os.getgroups(). This is as a consequence of a different group being deleted.
The test_getgroups comment asserts: "# 'id -G' and 'os.getgroups()' should
return the same groups, ignoring order, duplicates, and the effective gid."
https://github.com/python/cpython/blob/master/Lib/test/test_posix.py#L819-L820
I'm getting skeptical about that claim. Does Mac OS X actually guarantee that
'id -G' and 'getgroups(2)' return the same groups?
--
___
Python tracker
<http://bugs.python.org/issue29562>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29561] Interactive mode gives sys.ps2 not sys.ps1 after comment-only line
Jim DeLaHunt added the comment: Thank you for the analysis, Eryk Sun. You wrote, "the interpreter is correctly conveying that it's still tokenizing the input; it hasn't compiled or executed any code." I think one important question for this issue is, what meaning should the choice of prompt convey? That breaks down into, 1. what meaning does the user believe it conveys, and 2. what meaning does the present behaviour of the code imply? "it's still tokenizing the input" translates to a meaning, sys.ps1 indicates the interpreter just finished executing and has no input, while sys.ps2 indicates the interpreter is tokenizing input and awaiting a chance to execute. I think my intuition is, and the bash shell's behaviour says, sys.ps1 indicates the interpreter is ready to start a new statement, and sys.ps2 indicates the interpreter is in the midst of a compound statement. -- ___ Python tracker <http://bugs.python.org/issue29561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29563] Update Devguide about building documentation.
Jim DeLaHunt added the comment: Per request from Brett Cannon, I've moved this issue to https://github.com/python/devguide/issues/116 . Please continue the discussion there. -- ___ Python tracker <http://bugs.python.org/issue29563> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29563] Update Devguide about building documentation.
New submission from Jim DeLaHunt: The Devguide section 7.5.1 "Building the documentation" / "Using make / make.bat" is out of date. The document lists 10 documentation targets for `make`. The Doc/Makefile lists 17. One important omission is `make check`, which looks for errors in reStructuredText syntax, using `rstlint.py`. It's important for contributors to know about and to run `make check`, because with the move to GitHub, the project is running make check after pulling in new code. If contributors haven't cleaned up their submissions, the build fails. I suggest that the following improvements be made: 1. rewrite the bullet list "Available make targets are:" in 7.5.1 https://cpython-devguide.readthedocs.io/documenting.html#using-make-make-bat to match the list of targets in Doc/Makefile:21-38. 2. add a comment that `make help` will give the most up-to-date list of targets from the makefile, which might be better than the list in the Devguide. 3. add `make help` to both Doc/Makefile:21-38 and Devguide 7.5.1. 4. add to Devguide 6.3. "Helping with the Developer’s Guide" an instruction to run `make check` before submitting a pull request to the Devguide. -- assignee: docs@python components: Documentation messages: 287808 nosy: JDLH, docs@python priority: normal severity: normal status: open title: Update Devguide about building documentation. type: enhancement versions: Python 3.7 ___ Python tracker <http://bugs.python.org/issue29563> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29562] test_getgroups of test_posix fails (on OS X 10.10)
Jim DeLaHunt added the comment: I have pushed a branch for this issue to my cpython fork: https://github.com/JDLH/cpython/tree/bpo-29562_failing_test_getgroups_on_os_x It modifies test_getgroups in test_posix.py to give better diagnostics in the event of a test failure. It says specifically which groups were in id -G, and posix.getgroups(), but not in the other. % ./python.exe -m unittest -v test.test_posix.PosixTester.test_getgroups test_getgroups (test.test_posix.PosixTester) ... FAIL == FAIL: test_getgroups (test.test_posix.PosixTester) -- Traceback (most recent call last): File "/Users/jdlh/workspace/cpython/Lib/test/test_posix.py", line 841, in test_getgroups self.assertEqual(len(symdiff), 0, msg) AssertionError: 2 != 0 : id -G and posix.groups() should have zero difference. Groups in id -G but not posix.groups(): [(701, 'com.apple.sharepoint.group.1'), (398, 'com.apple.access_screensharing')] Groups in posix.groups() but not id -G: [] (Effective GID (20) was disregarded.) -- Ran 1 test in 0.020s I don't think this branch is ready yet to submit to the main codebase, but it may help people diagnose the issue. -- ___ Python tracker <http://bugs.python.org/issue29562> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29562] test_getgroups of test_posix fails (on OS X 10.10)
New submission from Jim DeLaHunt:
When I run test.test_posix.PosixTester.test_getgroups on my Mac OS X system, it
fails:
% ./python.exe -m unittest -v test.test_posix.PosixTester.test_getgroups
test_getgroups (test.test_posix.PosixTester) ... FAIL
==
FAIL: test_getgroups (test.test_posix.PosixTester)
--
Traceback (most recent call last):
File "/Users/jdlh/workspace/cpython/Lib/test/test_posix.py", line 824, in
test_getgroups
self.assertTrue(not symdiff or symdiff == {posix.getegid()})
AssertionError: False is not true
--
Ran 1 test in 0.013s
FAILED (failures=1)
Details of my system:
% sw_vers
ProductName:Mac OS X
ProductVersion: 10.10.5
BuildVersion: 14F2109
% id -G
20 507 12 61 80
98 399 33 100
204 395 398
701
% id -G -n
staff xampp everyone localaccounts admin
_lpadmin com.apple.access_ssh _appstore _lpoperator
_developer com.apple.access_ftp com.apple.access_screensharing
com.apple.sharepoint.group.1
# I wrapped these lines similarly, to make the correspondence clearer
% ./python.exe -c 'import grp,os; g={i: (n, p, i, mem) for (n, p, i, mem) in
grp.getgrall()}; print(sorted([(i, g[i][0]) for i in os.getgroups()]) )'
[(12, 'everyone'), (20, 'staff'), (33, '_appstore'), (61, 'localaccounts'),
(80, 'admin'), (98, '_lpadmin'), (100, '_lpoperator'), (204, '_developer'),
(395, 'com.apple.access_ftp'), (399, 'com.apple.access_ssh'), (507, 'xampp')]
So the difference, which triggers the test failure, is that id -G is returning
groups (701, 'com.apple.sharepoint.group.1'), and (398,
'com.apple.access_screensharing'), while posix.getgroups() is not. I do not
yet understand why.
Others say this test works on their OS X 10.10 system, so maybe it's triggered
by something in my environment.
Also: python3.6 from MacPorts, and python2.7 from MacPorts, return the same set
of groupids as does the dev build of python3.7.
This bug affects the same test, and the same posix.getgroups() call, as
http://bugs.python.org/issue17557 "test_getgroups of test_posix can fail on OS
X 10.8 if more than 16 groups" (2013-2014, closed). But I think it is a
different problem: issue17557 is related to how posix.getgroups() deals with
large numbers of groups, and it is fixed.
I would appreciate help in getting this test to pass. Maybe my environment is
wrong, in which case I should fix my environment. But maybe the cpython code is
sensitive to some detail of my environment, in which case perhaps I should fix
the cpython code.
--
components: Tests
messages: 287806
nosy: JDLH
priority: normal
severity: normal
status: open
title: test_getgroups of test_posix fails (on OS X 10.10)
type: behavior
versions: Python 3.7
___
Python tracker
<http://bugs.python.org/issue29562>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29561] Interactive mode gives sys.ps2 not sys.ps1 after comment-only line
New submission from Jim DeLaHunt: When you run the Python interpreter in interactive mode, get a sys.ps1 prompt (`...`), and type a comment-only or white-space-only line, the interpreter responds with a sys.ps2 prompt (`...`), instead of a sys.ps1 prompt. This seems wrong. For example: % ./python.exe Python 3.7.0a0 (default, Feb 13 2017, 16:27:07) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> # comment-only<<-- line A ... <<-- line B >>> <<-- line C >>> <<-- line D Line A is a comment-only line entered by the user. The interpreter responds with the sys.ps2 prompt (`...`), in line B. Line C is a null line (user enters only the Enter key, nothing else). The interpreter responds with a sys.ps1 prompt (`...`). If user enters a white-space-only line at a sys.ps1 prompt, the interpreter responds the same as if it were a comment-only line, with a sys.ps2 prompt (line B). The expected expected interpreter behaviour in response to a comment-only line or a white-space-only line, not in the midst of a compound statement, is that it be the same as to a null line, with a sys.ps1 prompt. In the context of a compound statement, the expected interpreter behaviour is that a comment-only line or a white-space-only line continue the compound statement. Entering a null line in a compound statement ends the compound statement. Another way to phrase it: the expected interpreter behaviour is that a comment-only line or white-space-only line not change the interpreter prompt: if it was sys.ps1, it remains so; if it was sys.ps2, it remains so. I have reproduced this behaviour on my build of cpython 3.7.a0, python 3.6 from MacPorts, python 2.7 from MacPorts, all on Mac OS X. I see the same behaviour in the interactive shell at python.org/shell . This makes me suspect that the Python design says my expectations are wrong. In that case, the current behaviour certainly is surprising. It should be documented. I haven't found such documentation yet. -- components: Interpreter Core messages: 287799 nosy: JDLH priority: normal severity: normal status: open title: Interactive mode gives sys.ps2 not sys.ps1 after comment-only line type: behavior versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <http://bugs.python.org/issue29561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: Pull Request https://github.com/python/cpython/pull/45 submitted to new Github repo. I would appreciate a review. I attempted to balance all the different opinions in the discussion below: stay concise, but also improve the clarity. -- pull_requests: +45 ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29387] Tabs vs spaces FAQ out of date
Jim DeLaHunt added the comment: PR https://github.com/python/cpython/pull/41 to the new Github repo contains the following wording in Doc/faq/windows.rst: Python raises :exc:`IndentationError` or :exc:`TabError` if mixed tabs and spaces are causing problems in leading whitespace. You may also run the :mod:`tabnanny` module to check a directory tree in batch mode. This is parallel wording with the contents of Martin's patch. The PR may be enough to fix this issue. -- pull_requests: +44 ___ Python tracker <http://bugs.python.org/issue29387> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29521] Minor warning messages when compiling documentation
Jim DeLaHunt added the comment: I submitted a PR https://github.com/python/cpython/pull/41 to the new Github repo which finishes clearing the warnings in this bug report. I would appreciate review and committing. -- pull_requests: +43 ___ Python tracker <http://bugs.python.org/issue29521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29521] Minor warning messages when compiling documentation
Jim DeLaHunt added the comment: It looks like commit e7ffb99f842ebff97cffa0fc90b18be4e5abecf2 to the new GitHub python/cpython repo, by Ryan Gonzalez, fixed the problems in Doc/conf.py, Doc/Makefile, and Misc/NEWS . It did not fix the problems in Doc/make.bat or Doc/faq/windows.rst . I'll make a new Pull Request on the new repo to fix these two parts. That commit was related to https://github.com/python/cpython/pull/9. http://bugs.python.org/issue29527 talks about over 6000 warnings in the doc build, maybe including some of these. Another PR appeared to have fixed many of the warnings. -- ___ Python tracker <http://bugs.python.org/issue29521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: I've drafted some fairly restricted changes to the doctest documentation page. They are in my Github branch, https://github.com/JDLH/cpython/tree/Issue29428_doctest_docs . The diffs are at https://github.com/JDLH/cpython/commit/223ef8f8a6d2fbec6db774912028abb4d2ff88b6 . There currently is no official Python github repot against which to make a pull request. In a few days, once it appears, I'll make a pull request. In the meantime, I can take review comments and improve it. If you are interested, please review. -- ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29521] Minor warning messages when compiling documentation
Jim DeLaHunt added the comment: My Pull Request was closed, because apparently https://github.com/python/cpython/ will not be the new GitHub repo for Python. The actual repo will open on 11. Feb, I'm told. I will repeat the PR there. Please stand by. -- ___ Python tracker <http://bugs.python.org/issue29521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29387] Tabs vs spaces FAQ out of date
Jim DeLaHunt added the comment: I just created Pull Request 76 https://github.com/python/cpython/pull/76 to address http://bugs.python.org/issue29521 . In faq/windows.rst, I've used your wording from the discussion in of this bug. I may have address this issue completely, in fact. I did not do the more careful "testing all versions patched" which Terry suggested. -- nosy: +JDLH ___ Python tracker <http://bugs.python.org/issue29387> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29521] Minor warning messages when compiling documentation
Jim DeLaHunt added the comment: Thanks, Martin, for your suggestions. In Misc/NEWS, I've respelled the ``char *`` as you suggested. In faq/windows.rst, I've used your wording from the discussion in http://bugs.python.org/issue29387 . Pull Request 76 https://github.com/python/cpython/pull/76 ready for review. -- pull_requests: +27 ___ Python tracker <http://bugs.python.org/issue29521> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29520] Documentation uses deprecated "defindex.html" Sphinx template
Jim DeLaHunt added the comment: The other warnings in the "make html" output are the subject of http://bugs.python.org/issue29521 . -- ___ Python tracker <http://bugs.python.org/issue29520> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29521] Minor warning messages when compiling documentation
New submission from Jim DeLaHunt:
When I build the documentation on the current CPython code, there are various
error and warning messages on the console.
Here's what my build output looks like. I've marked the messages I'm concerned
about with a numbered >>0>> prefix.
= (beginning of output)
% make html
sphinx-build -b html -d build/doctrees -D latex_paper_size= . build/html
Running Sphinx v1.5.2
loading pickled environment... done
>>1>> WARNING: latex_preamble is deprecated. Use latex_elements['preamble']
>>instead.
>>2>> WARNING: latex_paper_size is deprecated. Use latex_elements['papersize']
>>instead.
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 466 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] whatsnew/changelog
>>3>> ../../Misc/NEWS:659: WARNING: Inline emphasis start-string without
>>end-string.
>>4>> ../../Misc/NEWS:659: WARNING: Inline emphasis start-string without
>>end-string.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] whatsnew/index
>>5>> /Users/jdlh/workspace/cpython_github/Doc/faq/windows.rst:303: WARNING:
>>unknown option: -t
generating indices... genindex py-modindex
writing additional pages... download index
>>6>> WARNING: Now base template defindex.html is deprecated.
search opensearch
copying images... [100%] using/win_installer.png
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 6 warnings.
Build finished. The HTML pages are in build/html.
= (end of output)
This is observed when building documentation from branch master, commit
b1dc6b6d5fa20f63f9651df2e7986a066c88ff7d .
The build command is "cd Doc; make html".
Warning >>6>> is the subject of http://bugs.python.org/issue29520 . It's harder
to fix, and I won't address it here.
The other five warnings are pretty easy to fix.
Warnings >>1>>, >>2>> are Sphinx warnings about names used in Doc/conf.py ,
namely `latex_preamble` and `latex_paper_size`. There are straightforward
changes to build a dict latex_elements{}, with keys 'preamble' and 'papersize'.
It turns out that makefiles Doc/Makefile and Doc/make.bat also referred to
`latex_paper_size`. Those references are rewritten as
`latex_elements.papersize`, per Sphinx syntax for external names.
Warnings >>3>>, >>4>> are Sphinx warnings about the text, in Misc/NEWS:661,
```is now of type "const char *" rather of "char *".```
Put a backslash in front of the '*', and the warning disappears.
Warning >>5>> is a Sphinx warning about this text, in
Doc/faq/windows.rst:303:
If you suspect mixed tabs and spaces are causing problems in leading
whitespace, run Python with the :option:`-t` switch or run
``Tools/Scripts/tabnanny.py`` to check a directory tree in batch mode.
The notation :option:`-t` seems to need a corresponding ``.. cmdoption:: -t ``
entry, perhaps in the same file. There is no such entry.
It turns out that the -t option has no function in Python 3.6, maybe in all of
3.x. Python swallows the option but does nothing. Thus, instead of trying to
make the reference to '-t' work, I decided to cut the whole phrase. This
paragraph now reads,
If you suspect mixed tabs and spaces are causing problems in leading
whitespace, run ``Tools/Scripts/tabnanny.py`` to check a directory tree in
batch mode.
I am making a Pull Request with these fixes. I will shortly link to it from
here.
--
assignee: docs@python
components: Documentation
messages: 287480
nosy: JDLH, docs@python
priority: normal
severity: normal
status: open
title: Minor warning messages when compiling documentation
type: compile error
versions: Python 3.7
___
Python tracker
<http://bugs.python.org/issue29521>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29520] Documentation uses deprecated "defindex.html" Sphinx template
New submission from Jim DeLaHunt:
When I build the documentation on the current CPython code, there is a
deprecation warning on the console.
= (beginning of output)
% make html
sphinx-build -b html -d build/doctrees -D latex_elements.papersize= .
build/html
Running Sphinx v1.5.2
...[omitted irrelevant output]...
generating indices... genindex py-modindex
writing additional pages... download index
WARNING: Now base template defindex.html is deprecated.
search opensearch
copying images... [100%] faq/python-video-icon.png
...[omitted irrelevant output]...
build succeeded, 1 warning.
Build finished. The HTML pages are in build/html.
= (end of output)
This is observed when building documentation from branch master, commit
b1dc6b6d5fa20f63f9651df2e7986a066c88ff7d .
The build command is "cd Doc; make html".
There are other warnings in the output, and I'm dealing with them in a
different issue (number to follow). They are easier to fix than this one.
Diagnosis:
Sphinx config file Doc/conf.py:72 invokes the building of template
'indexcontent.html'.
Doc/tools/templates/indexcontent.html:1 contains Sphinx directive
`{% extends "defindex.html" %}`.
This invokes file sphinx/sphinx/themes/basic/defindex.html
[See
https://github.com/sphinx-doc/sphinx/blob/8ecd7ff08249739bbc6d900527fe9306592456ab/sphinx/themes/basic/defindex.html
]. Sure enough, it issues a deprecation warning.
{{ warn('Now base template defindex.html is deprecated.') }}
There's a story behind this file.
Sphinx issue 2986 (https://github.com/sphinx-doc/sphinx/issues/2986) says that
this is a very old file, from about the 0.2 version of Sphinx. It wasn't HTML 5
compatible, so they declared it obsolete and threw it out. Well, that lasted
only about two weeks. It became apparent that not only Python's docs, but
thousands of other projects, seem to rely on it. So, defindex.html was
restored, but with the deprecation warning.
Then, on 1. January 2017, Sphinx deleted defindex.html again. (See
https://github.com/sphinx-doc/sphinx/commit/45d3f2e8b279efa9d42068d4109cd97eb3f2d899
). I can only imagine that, once this change makes it into the public release
of Sphinx, Python's documentation, and that of thousands of projects, will
break again.
So, it seems like a good idea to proactively remove the dependence on this
Sphinx file, before that new Sphinx release comes out.
Options:
1. Copy the Sphinx defindex.html file into our source tree, and keep using it.
Plus points: it's simple and easy. Minus points: the Sphinx licence terms may
not permit this. And, it is not HTML5 compatible, which we might care about.
2. Identify the template which Sphinx intends as a successor to defindex.html,
and switch to using that. I've done a bit of searching, and couldn't find out
which template that might be.
3. Reimplement our Doc/tools/templates/indexcontent.html to rely on supported
Sphinx template, and replace whatever intermediate content we were using from
defindex.html with freshly-written code.
I don't have a solution in mind for this issue. I just want to get it in the
bug list, so we know about it.
--
assignee: docs@python
components: Documentation
messages: 287478
nosy: JDLH, docs@python
priority: normal
severity: normal
status: open
title: Documentation uses deprecated "defindex.html" Sphinx template
type: compile error
versions: Python 3.7
___
Python tracker
<http://bugs.python.org/issue29520>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29510] gitignore settings files for Eclipse IDE
Jim DeLaHunt added the comment: I have set up a global gitignore, and it works for me. % git config --global core.excludesfile ~/.gitignore_global (Amusingly, I had _already- set a global gitignore, but I had forgotten it existed, and it didn't ignore these IDE files.) I agree that we should not put IDE entries in the CPython .gitignore. I also think it's a good idea to add the points Inada mentions in the developer doc somewhere. That should include the Git global ignore idea. I see Berker's point that the Devguide shouldn't get too big, but it also shouldn't leave out useful information. There's a balance to strike. I think that's the topic of a different issue, however. I'll open that when I get back to this. In the meantime, if someone searches the issue database for "gitignore", they will find this discussion. That's helpful. Thank you both for your responses, Inada and Berker. -- ___ Python tracker <http://bugs.python.org/issue29510> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29510] gitignore settings files for Eclipse IDE
Jim DeLaHunt added the comment: gitignore_global is a great idea. I had not heard of it before. But here it is: https://help.github.com/articles/ignoring-files/ . This instruction also has a link to a gist with a lot of helpful global ignores. I understand your reluctance to add entries for every IDE. Maybe a better solution would be to add something to the devguide about excluding IDE settings files, with a link to the Github instructions. -- ___ Python tracker <http://bugs.python.org/issue29510> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29510] gitignore settings files for Eclipse IDE
Jim DeLaHunt added the comment: I'm now looking at cpython as retrieved from Mercurial by Eclipse. It appears to be concerned by the presence of .project which is also an Eclipse settings file. It might be reasonable to extend the scope of this issue to include telling Mercurial to ignore settings files. My pull request does not do this, because I haven't yet learned the right Mercurial action. -- ___ Python tracker <http://bugs.python.org/issue29510> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29510] gitignore settings files for Eclipse IDE
Changes by Jim DeLaHunt : -- type: -> enhancement ___ Python tracker <http://bugs.python.org/issue29510> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29510] gitignore settings files for Eclipse IDE
Jim DeLaHunt added the comment: A fix is in GitHub cpython PR #75. -- pull_requests: +26 ___ Python tracker <http://bugs.python.org/issue29510> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29510] gitignore settings files for Eclipse IDE
New submission from Jim DeLaHunt: The Eclipse IDE, and its relatives pydev and LiClipse, store settings in the root of a repository. It would be nice for the master .gitignore file to ignore these files, so that individual developers don't have to do this. I am preparing a GitHub Pull Request that fixes this issue. It is a matter of adding 4 lines to the top-level .gitignore file. -- messages: 287388 nosy: JDLH priority: normal severity: normal status: open title: gitignore settings files for Eclipse IDE versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7 ___ Python tracker <http://bugs.python.org/issue29510> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: Marco: > To have a wide view, usually it is a good idea to start from the beginning: > > https://groups.google.com/forum/#!msg/comp.lang.python/DfzH5Nrt05E/Yyd3s7fPVxwJ Thank you, that is a very interesting thread. Clearly the creator of doctests, Tim Peters, is a well-regarded person in the Python world. On the other hand, I was surprised to see some of the examples used in the present-day documentation in Tim's original 1999 message. I find it remarkable that in 18 years since then, no rewrite or editing has replaced those examples with better ones. -- ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: Marco, thank you for the suggestion of a howto. That is a good idea. In parallel, I was thinking of some howto content that I would like to see documented. • How to decide what to test with doctests and what with unittests. I have a feeling that the sweet spot of doctests is functionality which you can invoke in one line, and which produces one line of output which you can match against a reference string. If it takes many lines to set up, or invoke, or checking for correctness is more than a test for string equality, then maybe unittest is a better tool. • An effective way to write doctests. I suspect that some people write doctests by exercising the functionality in Python's interactive mode, then copy and paste the transcript from that session to the doctests. I don't do it that way, I author in the docstring. Maybe I'm not doing it the best way. • How to author doctests so that they both prove the module correct, and provide clear and readable documentation. I imagine some effective tests for edge cases and error conditions are important to have, but are not readable documentation. Maybe such tests belong in a unittest framework instead of as doctests. • How to run doctests from a unittest harness (your earlier note about the unittest API would be a part of this). A problem for me is that I think I don't have the experience and wisdom to give good advice in these areas. I would be happy to start such a howto, and to accept feedback, and to edit it into good prose. I would need wiser people to contribute good ideas for the howto. Also, Marco, thank you for being willing to review a patch. That is helpful. My next step is to check out the documentation source, to be in a position to make a patch. -- ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: Marco and David, thank you again for your prompt replies. Let me respond to both of your comments on the doctest module documentation itself. [Marco] > The example in section 26.3.3.2 [1], before the compound statement (the if/else), contains two simple statements. I also think the example is complete and straightforward for its purpose, and it does not have to be complicated adding avanced features. You are correct, it contain two simple statements. I missed this when I first read the example. Part of the problem is that the example is contrived. It is an actual test. I think it would be better to replace these two lines by a two-line test, one which sets up state and another which exercises the module under test. You believe "the example is complete and straightforward for its purpose, and it does not have to be complicated adding avanced features." I am giving you feedback that it was not complete and straightforward for me, as an ordinary software developer trying to use the documentation. [Marco] > If you want a more complex control over your tests, for instance adding fixtures, you can find how to do it later... No, control over tests is not what I am bringing up in this issue. I an concerned with documentation of doctest "Examples". [Marco] > I am -1 on this [changing 26.3.3.2]. IMHO there is no need to add extra description to a such self-explaining, clear and concise example. But this I think is also clear from the beginning. Infact, the first line of the doc says: "The doctest module searches for pieces of text that look like interactive Python sessions" [David] > As for the prompts, I agree with Marco: I don't think there is any need to add words, since it clearly says it is supposed to look like an interactive session and the reader gets reminded of the rules for that by the example provided. IMO it is not appropriate to re-document those rules here. My feedback to you is that for me, as an ordinary software developer trying to use the doctest module, this section is not "self-explaining, clear and concise", and that "How Docstring examples are recognised" is not "clear from the beginning". Perhaps we have a difference of opinion on how clear and effective this documentation is. The best way to resolve that would be to gather data: having several ordinary software developers reading the document, then measuring their comprehension. That would be hard to arrange. At least this bug report will serve as a data point. I believe that a library module documentation should at some point clearly specify how the module works. I believe section Section "26.3.3.2. How are Docstring Examples Recognized?" does poorly at that. To say, "In most cases a copy-and-paste of an interactive console session works fine, but doctest isn’t trying to do an exact emulation of any specific Python shell." is not a specification. It's a blurry generalisation. The list of "fine print" is at a better level of detail, but still doesn't mention how the parser treats statements, nor how prefixes relate to statement boundaries. I am giving you feedback that for me, as a developer trying to use the doctest module, this documentation failed to give me the comprehension I needed to be effective. [Marco] >Also this section [26.3.3.3, execution context] IMO is well written and clear. I do not know what to say... If someone else thinks it is not enough understandable, maybe the better think to do is to add a very small example, with two functions, that shows their docstrings have independent globals. [David] > It does appear that "test" is being used ambiguously in the docs. In most places it means a single statement, but in execution context it appears to be being used as a synonym for "docstring". In that section it should be replaced by "docstring". It would probably be worth checking the rest of the doc to make sure it is used consistently elsewhere as well. [David] > The execution context section makes it clear that you can use names defined earlier, as does the fact that doctests emulate interactive sessions. Both of these make it clear you can do multi line examples that use shared state. Again, I am giving you feedback that this document was not "clear" for me. I am taking the time to write this issue because I think the documentation has weaknesses, and would like to help improve it. I agree with David, that one problem in this section is that the word "test" is used with three different meanings. Some instances of the word "test" should be changed to "docstring" or similar. That would help. [David] > So in summary, I think we *need* a one word change in the execution context section, and we'll consider a
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: Marco, thank you for the suggestion to watch Raymond Hettinger's talk. > I suggest you to watch this talk of Raymond Hettinger, before going on: > > https://www.youtube.com/watch?v=voXVTjwnn-U That video is 63 minutes long, and a lot of those minutes are taken up by stories. I will watch the rest when I get time. I wish his insights were summarised in writing; I'll bet that would take much less than 63 minutes to read. I find that written text is a much faster way to deliver information, though maybe it takes the teacher more time to write well than to give a talk. I understand your point about the need for a review, and reviews being the bottleneck. The "Lifecycle of a Patch" document[1] is clear about that. [1] https://docs.python.org/devguide/patch.html#reviewing -- ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
Jim DeLaHunt added the comment: Marco, thank you for your prompt comment. However, > Maybe you are running the doctest with Python 3. The StringIO module is no > more available in Python 3, so your doctest will fail on Python 3 Please let me be clear. This issue is not about the StringIO module or about any of my doctests. It is about the documentation of the doctests module, and the ways in which that documentation is less helpful than it could be. -- ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29428] Doctest documentation unclear about multi-line fixtures
New submission from Jim DeLaHunt: I just had a problem with doctests. It manifested as an error that occurred when I did >>> import StringIO in my doctest. After some investigation, I realised that my actual problem was that the doctest library documentation https://docs.python.org/3/library/doctest.html left me unclear about three aspects of multi-line doctest fixtures. 1. There is no example of a multiple-line doctest fixture: specifically, a fixture which involves at least one line to set up state, and another line to be the example that tested. I suggest adding such an example as a new section, "26.3.2. Usage: multi-line tests". Note that the example in 26.3.3.2 doesn't count: that is a Compound Statement, and I'm talking about a test consisting of multiple interactive statements. 2. Section "26.3.3.2. How are Docstring Examples Recognized?" does not talk about when to use a PS1 prefix (>>>) and when to use a PS2 prefix (...). It should say to use the PS1 at the start of each Python "interactive statement", that is, a statement list ending with a newline, or a Compound Statement. And, to use the PS2 to prefix each continuing line within a Compound Statement. 3. Section "26.3.3.3. What’s the Execution Context?" is not clear about the crucial difference between state that accumulates within a single docstring from Example to Example, and that is reset from one docstring to another. The phrase "so that one test in M can’t leave behind crumbs that accidentally allow another test to work" can be interpreted as applying within a single docstring. The phrase "names defined earlier in the docstring being run" is easy to miss. A StackOverflow Question and Answer describing my problem and my insight about what the library documentation didn't tell me is at http://stackoverflow.com/questions/41070547/why-is-importing-a-module-breaking-my-doctest-python-2-7 . A blog post of mine, a source for documentation improvements, is at http://blog.jdlh.com/en/2017/01/31/multi-line-doctests/ . I actually encountered this problem in a Python 2.7 context, with https://docs.python.org/2/library/doctest.html . But it also applies to Python 3.6 https://docs.python.org/3/library/doctest.html and to the current dev https://docs.python.org/dev/library/doctest.html . In the spirit of fixing the new stuff first, I am filing this issue against Python 3.7. Unless someone tells me otherwise, I'll get around to filing similar issues against Python 3.6 and Python 2.7. Then it's my intention to write improvements to the documentation for Python 3.7, 3.6, and 2.7, and submit each of those. It looks like they will be pretty similar, because the library documentation is pretty similar in each version. I'm new to Python issue filing and doc fixing, so I would appreciate correction if (when?) I start to blunder. -- assignee: docs@python components: Documentation messages: 286843 nosy: JDLH, docs@python priority: normal severity: normal status: open title: Doctest documentation unclear about multi-line fixtures type: enhancement versions: Python 3.7 ___ Python tracker <http://bugs.python.org/issue29428> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
