[issue33036] test_selectors.PollSelectorTestCase failing on macOS 10.13.3
Nathan Henrie added the comment:
I've continued looking into this.
If you have your limits configured higher than default, as I did (and which
seems to be working fine):
```
sudo launchctl limit maxfiles 64000 524288
ulimit -Sn 64000
```
then you'll find that having a soft limit (`NUM_FDS`, prior to the `-=32`)
greater than 10273 causes the error I was seeing.
First, I found this number by passing in NUM_FDS as an envvar to the unittest
and looping in a bash script, but I have since extracted some of the relevant
portions to a separate python script:
```python
import selectors
import socket
import sys
def main(NUM_FDS):
s = selectors.PollSelector()
NUM_FDS -= 32
sockets = []
try:
for i in range(NUM_FDS // 2):
try:
rd, wr = socket.socketpair()
sockets.extend([rd, wr])
except OSError as e:
print(f"err 2: {type(e)} {e}")
# too many FDs, skip - note that we should only catch EMFILE
# here, but apparently *BSD and Solaris can fail upon connect()
# or bind() with EADDRNOTAVAIL, so let's be safe
print("FD limit reached")
try:
s.register(rd, selectors.EVENT_READ)
s.register(wr, selectors.EVENT_WRITE)
except OSError as e:
print(f"err 3: {type(e)} {e}")
if e.errno == errno.ENOSPC:
# this can be raised by epoll if we go over
# fs.epoll.max_user_watches sysctl
print("FD limit reached")
raise
try:
slen = len(s.select())
except OSError as e:
print(f"err 4: {type(e)} {e}")
raise
print(f"asserting {NUM_FDS // 2} == {slen}")
assert NUM_FDS // 2 == slen
finally:
for sock in sockets:
sock.close()
s.close()
print("Closed")
if __name__ == "__main__":
main(int(sys.argv[1]))
```
```shellsession
$ ./python.exe tester.py 10273
asserting 5120 == 5120
Closed
$ ./python.exe tester.py 10274
err 4: [Errno 22] Invalid argument
Closed
Traceback (most recent call last):
File "tester.py", line 50, in
main(int(sys.argv[1]))
File "tester.py", line 36, in main
slen = len(s.select())
File "cpython/Lib/selectors.py", line 376, in select
fd_event_list = self._poll.poll(timeout)
OSError: [Errno 22] Invalid argument
```
Tested and confirmed that I can provoke the error with the above launchctl /
ulimit settings on my wife's Macbook Air.
I wonder if the 10273 limit I'm running up against has something to do with the
10240 number I keep running into as a default kern.maxfilesperproc
(https://unix.stackexchange.com/a/350824/77904), given the `NUM_FDS -= 32` in
the test.
--
___
Python tracker
<https://bugs.python.org/issue33036>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33036] test_selectors.PollSelectorTestCase failing on macOS 10.13.3
Nathan Henrie added the comment: Update -- I found the following plist at `/Library/LaunchDaemons/limit.maxfiles.plist`: ```xml http://www.apple.com/DTDs/PropertyList-1.0.dtd";> Label limit.maxfiles ProgramArguments launchctl limit maxfiles 64000 524288 RunAtLoad ServiceIPC ``` I think I made this file at some point to deal with an error about insufficient file descriptors while playing with asyncio / sockets. After `launchctl unload`ing it and rebooting **the test now passes**, and I now see a *much* higher hard limit (similar to that posted by Ned) with the `getrlimit()` command, and `unlimited` as the hard limit with `launchtl limit`. I can reproduce the failure by restoring my prior settings: ```bash $ sudo launchctl limit maxfiles 64000 524288 $ ulimit -n 64000 $ $ ./python.exe -m unittest -v test.test_selectors.PollSelectorTestCase.test_above_fd_setsize test_above_fd_setsize (test.test_selectors.PollSelectorTestCase) ... ERROR == ERROR: test_above_fd_setsize (test.test_selectors.PollSelectorTestCase) -- Traceback (most recent call last): File "/Users/n8henrie/git/cpython/Lib/test/support/__init__.py", line 600, in wrapper return func(*args, **kw) File "/Users/n8henrie/git/cpython/Lib/test/test_selectors.py", line 453, in test_above_fd_setsize self.assertEqual(NUM_FDS // 2, len(s.select())) File "/Users/n8henrie/git/cpython/Lib/selectors.py", line 376, in select fd_event_list = self._poll.poll(timeout) OSError: [Errno 22] Invalid argument -- Ran 1 test in 9.771s FAILED (errors=1) ``` -- resolution: works for me -> status: closed -> open ___ Python tracker <https://bugs.python.org/issue33036> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33036] test_selectors.PollSelectorTestCase failing on macOS 10.13.3
Nathan Henrie added the comment: Thanks for the response -- I'll keep looking, feel free to close since it's not being reproduced. ``` $ sysctl kern.maxfilesperproc kern.maxfilesperproc: 64000 $ ./python.exe -c 'import resource; print(resource.getrlimit(resource.RLIMIT_NOFILE))' (64000, 524288) ``` Nope, admin access. -- ___ Python tracker <https://bugs.python.org/issue33036> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33036] test_selectors.PollSelectorTestCase failing on macOS 10.13.3
Nathan Henrie added the comment: Hmmm, still failing for me. I wonder if it's something specific to my machine. ``` git reset --hard 3.6 && make clean && git pull && ./configure --with-pydebug && make -j && ./python.exe -m unittest -v test.test_selectors.PollSelectorTestCase.test_above_fd_setsize ``` Related: - https://bugs.python.org/issue18963 - https://bugs.python.org/issue21901 -- ___ Python tracker <https://bugs.python.org/issue33036> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32517] test_read_pty_output() of test_asyncio hangs on macOS 10.13.2 (darwin 17.3.0)
Nathan Henrie added the comment: Awesome, I'm really excited to have contributed something, no matter how small. -- ___ Python tracker <https://bugs.python.org/issue32517> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33036] test_selectors.PollSelectorTestCase failing on macOS 10.13.3
Nathan Henrie added the comment: Traceback: ``` File "cpython/Lib/test/test_selectors.py", line 453, in test_above_fd_setsize self.assertEqual(NUM_FDS // 2, len(s.select())) File "cpython/Lib/selectors.py", line 376, in select fd_event_list = self._poll.poll(timeout) OSError: [Errno 22] Invalid argument ``` -- ___ Python tracker <https://bugs.python.org/issue33036> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33036] test_selectors.PollSelectorTestCase failing on macOS 10.13.3
New submission from Nathan Henrie : Failing for me on latest 3.6, 3.6.1, 3.5.5, may be related to https://bugs.python.org/issue32517, presumably a change on macOS KQueue stuff. Can anyone else on macOS 10.13.3 see if they can reproduce? ``` make clean && ./configure --with-pydebug && make -j ./python.exe -m unittest -v test.test_selectors.PollSelectorTestCase ``` -- components: Tests messages: 313487 nosy: n8henrie priority: normal severity: normal status: open title: test_selectors.PollSelectorTestCase failing on macOS 10.13.3 versions: Python 3.6 ___ Python tracker <https://bugs.python.org/issue33036> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32517] test_read_pty_output() of test_asyncio hangs on macOS 10.13.2 (darwin 17.3.0)
Nathan Henrie added the comment: It seems to work if you close proto.transport (as is done in `test_write_pty()`). -- ___ Python tracker <https://bugs.python.org/issue32517> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32517] test_read_pty_output() of test_asyncio hangs on macOS 10.13.2 (darwin 17.3.0)
Change by Nathan Henrie : -- keywords: +patch pull_requests: +5799 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue32517> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32517] test_read_pty_output() of test_asyncio hangs on macOS 10.13.2 (darwin 17.3.0)
Nathan Henrie added the comment: I can reproduce on my local machine. MacOS 10.13.2, trying to build 3.6.4. Waited for up to 6 hours for it to fail or finish, never does, just hangs at `test_asyncio`. -- nosy: +n8henrie ___ Python tracker <https://bugs.python.org/issue32517> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32578] x86-64 Sierra 3.6: test_asyncio fails with timeout after 15 minutes
Nathan Henrie added the comment: Think I am also seeing this, MacOS 10.13.2, making 3.6.4 from source test_asyncio hangs indefinitely. I killed it after 2 hours this morning, last output was: `running: test_asyncio (9481 sec)` -- nosy: +n8henrie ___ Python tracker <https://bugs.python.org/issue32578> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31846] Error in 3.6.3 epub docs
New submission from Nathan Henrie : I routinely download the epub version of the docs to my computer and mobile devices as an offline copy. The 3.6.3 version reports a big error on the first (and many other pages): > This page contains the following errors: error on line 5176 at column 11: Entity 'copy' not defined Below is a rendering of the page up to the first error. Numerous similar errors reporting `Entity 'copy' not defined` scattered throughout the epub. Wonder if this was introduced by the change to `conf.py` that fixed the recent problem of the 404s with the HTML docs. -- messages: 304799 nosy: n8henrie priority: normal severity: normal status: open title: Error in 3.6.3 epub docs versions: Python 3.6 ___ Python tracker <https://bugs.python.org/issue31846> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31395] Docs Downloads are 404s
New submission from Nathan Henrie:
Please see my (closed) issue, I was told to resubmit here.
https://github.com/python/pythondotorg/issues/1140
Basically, I usually download a local copy of the epub and HTML docs so I can
reference offline (and faster using the "custom search engine" feature of
several modern browsers). Tried to download 3.6.2 and noticed I was getting a
404: https://docs.python.org/3/archives/python-3.6.2-docs-html.zip
Testing the other links on the page, looks like most are down:
```shell_session
$ curl -s https://docs.python.org/3/download.html | grep -o '"archives/.*"' |
xargs -I{} bash -c 'echo; url="https://docs.python.org/3/{}";; echo "${url}"; c
url -s -I "${url}" | head -n 1'
https://docs.python.org/3/archives/python-3.6.2-docs-pdf-letter.zip
HTTP/1.1 404 Not Found
https://docs.python.org/3/archives/python-3.6.2-docs-pdf-letter.tar.bz2
HTTP/1.1 200 OK
https://docs.python.org/3/archives/python-3.6.2-docs-pdf-a4.zip
HTTP/1.1 404 Not Found
https://docs.python.org/3/archives/python-3.6.2-docs-pdf-a4.tar.bz2
HTTP/1.1 404 Not Found
https://docs.python.org/3/archives/python-3.6.2-docs-html.zip
HTTP/1.1 404 Not Found
https://docs.python.org/3/archives/python-3.6.2-docs-html.tar.bz2
HTTP/1.1 404 Not Found
https://docs.python.org/3/archives/python-3.6.2-docs-text.zip
HTTP/1.1 404 Not Found
https://docs.python.org/3/archives/python-3.6.2-docs-text.tar.bz2
HTTP/1.1 404 Not Found
https://docs.python.org/3/archives/python-3.6.2-docs.epub
HTTP/1.1 404 Not Found
```
I looked to see if this was an existing issue, sorry if it's a duplicate.
--
assignee: docs@python
components: Documentation
messages: 301699
nosy: docs@python, n8henrie
priority: normal
severity: normal
status: open
title: Docs Downloads are 404s
___
Python tracker
<https://bugs.python.org/issue31395>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21888] plistlib.FMT_BINARY behavior doesn't send required dict parameter
New submission from Nathan Henrie: When using the new plistlib.load and the FMT_BINARY option, line 997: p = _FORMATS[fmt]['parser'](use_builtin_types=use_builtin_types) doesn't send the dict_type to _BinaryPlistParser.__init__ (line 601), which has dict_type as a required positional parameter, causing an error def __init__(self, use_builtin_types, dict_type): My first bugs.python.org report, hope I'm doing it right... -- components: Library (Lib) messages: 221969 nosy: n8henrie priority: normal severity: normal status: open title: plistlib.FMT_BINARY behavior doesn't send required dict parameter type: behavior versions: Python 3.4 ___ Python tracker <http://bugs.python.org/issue21888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
