[issue28022] SSL releated deprecation for 3.6
Nick Guenther added the comment: Hello everyone, and thank you as usual for all your hard work keeping the python ecosystem going. I saw that the start of this thread said it was going to > - make ftplib, imaplib, nntplib, pop3lib, smtplib etc. validate certs by > default. but this hasn't been done, at least not for imaplib. imaplib is still calling the undocumented "ssl._create_stdlib_context": https://github.com/python/cpython/blob/2b496e79293a8b80e8ba0e514e186b3b1467b64b/Lib/imaplib.py#L1320 which is actually "ssl._create_unverified_context": https://github.com/python/cpython/blob/2b496e79293a8b80e8ba0e514e186b3b1467b64b/Lib/ssl.py#L842 which is indeed unverified: despite defaulting to PROTOCOL_TLS_CLIENT, which "enables CERT_REQUIRED and check_hostname by default.", it overrides that by setting check_hostname=False: https://github.com/python/cpython/blob/2b496e79293a8b80e8ba0e514e186b3b1467b64b/Lib/ssl.py#L811 To demonstrate, check out this tester script: $ cat a.py import os, imaplib with imaplib.IMAP4_SSL(os.environ.get('HOSTNAME')) as S: print(S.login(os.environ.get('USERNAME'), os.environ.get('PASSWORD'))) $ HOSTNAME=46.23.90.174 USERNAME=test1 PASSWORD=test1test1 python3 a.py ('OK', [b'Logged in']) I don't have a cert for 46.23.90.174 (no one will give out certs for IPs!), so this is wrong! In order to actually enable verification you need to know the incantation. It's not that long but it is subtle and frighteningly easy to get wrong. Here it is: $ cat a.py import os, ssl, imaplib ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) ctx.load_default_certs() with imaplib.IMAP4_SSL(os.environ.get('HOSTNAME'), ssl_context=ctx) as S: print(S.login(os.environ.get('USERNAME'), os.environ.get('PASSWORD'))) $ HOSTNAME=46.23.90.174 USERNAME=test1 PASSWORD=test1test1 python3 a.py Traceback (most recent call last): File "a.py", line 6, in with imaplib.IMAP4_SSL(os.environ.get('HOSTNAME'), ssl_context=ctx) as S: File "/usr/lib/python3.6/imaplib.py", line 1288, in __init__ IMAP4.__init__(self, host, port) File "/usr/lib/python3.6/imaplib.py", line 198, in __init__ self.open(host, port) File "/usr/lib/python3.6/imaplib.py", line 1301, in open IMAP4.open(self, host, port) File "/usr/lib/python3.6/imaplib.py", line 299, in open self.sock = self._create_socket() File "/usr/lib/python3.6/imaplib.py", line 1293, in _create_socket server_hostname=self.host) File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket _context=self, _session=session) File "/usr/lib/python3.6/ssl.py", line 817, in __init__ self.do_handshake() File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake self._sslobj.do_handshake() File "/usr/lib/python3.6/ssl.py", line 694, in do_handshake match_hostname(self.getpeercert(), self.server_hostname) File "/usr/lib/python3.6/ssl.py", line 327, in match_hostname % (hostname, ', '.join(map(repr, dnsnames ssl.CertificateError: hostname '46.23.90.174' doesn't match either of 'comms.kousu.ca', 'comms3.kousu.ca' I can see from this thread there were some concerns about breaking people's self-signed certs back in 2016. But it's five years later now and letsencrypt is super common now, and most servers and clients are enforcing TLS, especially when credentials are involved. Could this be revisited? Thanks for any attention you have gifted to this :) -- nosy: +kousu ___ Python tracker <https://bugs.python.org/issue28022> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40110] multiprocessing.Pool.imap() should be lazy
Nick Guenther added the comment: Thank you for taking the time to consider my points! Yes, I think you understood exactly what I was getting at. I slept on it and thought about what I'd posted the day after and realized most of the points you raise, especially that serialized next() would mean serialized processing. So the naive approach is out. I am motivated by trying to introduce backpressure to my pipelines. The example you gave has potentially infinite memory usage; if I simply slow it down with sleep() I get a memory leak and the main python proc pinning my CPU, even though it "isn't" doing anything: with multiprocessing.Pool(4) as pool: for i, v in enumerate(pool.imap(worker, itertools.count(1)), 1): time.sleep(7) print(f"At {i}: {v}, memory usage is {sys.getallocatedblocks()}") At 1->1, memory usage is 230617 At 2->8, memory usage is 411053 At 3->27, memory usage is 581439 At 4->64, memory usage is 748584 At 5->125, memory usage is 909694 At 6->216, memory usage is 1074304 At 7->343, memory usage is 1238106 At 8->512, memory usage is 1389162 At 9->729, memory usage is 1537830 At 10->1000, memory usage is 1648502 At 11->1331, memory usage is 1759864 At 12->1728, memory usage is 1909807 At 13->2197, memory usage is 2005700 At 14->2744, memory usage is 2067407 At 15->3375, memory usage is 2156479 At 16->4096, memory usage is 2240936 At 17->4913, memory usage is 2328123 At 18->5832, memory usage is 2456865 At 19->6859, memory usage is 2614602 At 20->8000, memory usage is 2803736 At 21->9261, memory usage is 2999129 PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 11314 kousu 20 0 303308 40996 6340 S 91.4 2.1 0:21.68 python3.8 11317 kousu 20 0 54208 10264 4352 S 16.2 0.5 0:03.76 python3.8 11315 kousu 20 0 54208 10260 4352 S 15.8 0.5 0:03.74 python3.8 11316 kousu 20 0 54208 10260 4352 S 15.8 0.5 0:03.74 python3.8 11318 kousu 20 0 54208 10264 4352 S 15.5 0.5 0:03.72 python3.8 It seems to me like any usage of Pool.imap() either has the same issue lurking or is run on a small finite data set where you are better off using Pool.map(). I like generators because they keep constant-time constant-memory work, which seems like a natural fit for stream processing situations. Unix pipelines have backpressure built-in, because write() blocks when the pipe buffer is full. I did come up with one possibility after sleeping on it again: run the final iteration in parallel, perhaps by a special plist() method that makes as many parallel next() calls as possible. There's definitely details to work out but I plan to prototype when I have spare time in the next couple weeks. You're entirely right that it's a risky change to suggest, so maybe it would be best expressed as a package if I get it working. Can I keep this issue open to see if it draws in insights from anyone else in the meantime? Thanks again for taking a look! That's really cool of you! -- ___ Python tracker <https://bugs.python.org/issue40110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40110] multiprocessing.Pool.imap() should be lazy
New submission from Nick Guenther : multiprocessing.Pool.imap() is supposed to be a lazy version of map. But it's not: it submits work to its workers eagerly. As a consequence, in a pipeline, all the work from earlier steps is queued, performed, and finished first, before starting later steps. If you use python3's built-in map() -- aka the old itertools.imap() -- the operations are on-demand, so it surprised me that Pool.imap() doesn't. It's basically no better than using Pool.map(). Maybe it saves memory by not materializing large iterables in every worker process? But it still materializes the CPU time from the iterables even if unneeded. This can be partially worked around by giving each step of the pipeline its own Pool -- then, at least the earlier steps of the pipeline don't block the later steps -- but the jobs are still done eagerly instead of waiting for their results to actually be requested. -- files: multiprocessing-eager-imap.py messages: 365295 nosy: kousu priority: normal severity: normal status: open title: multiprocessing.Pool.imap() should be lazy versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file49010/multiprocessing-eager-imap.py ___ Python tracker <https://bugs.python.org/issue40110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19282] dbm is not a context manager
New submission from Nick Guenther: This code doesn't work. I think it should. import dbm with dbm.open("what is box.db", "c") as db: db["Bpoind"] = Boing Indeed, there is nothing supporting PEP 343 for dbm on my system: [kousu@galleon ~]$ grep -r __exit__ /usr/lib/python3.3/dbm [kousu@galleon ~]$ -- components: Library (Lib) messages: 200191 nosy: kousu priority: normal severity: normal status: open title: dbm is not a context manager versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker <http://bugs.python.org/issue19282> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10976] json.loads() raises TypeError on bytes object
Changes by Nick Guenther : -- nosy: +kousu ___ Python tracker <http://bugs.python.org/issue10976> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19250] Duplicate import documentation in py3k
Nick Guenther added the comment: Thank you. Sorry for the duplicates. I tried to search the bug tracker but nothing stood out to me. -- ___ Python tracker <http://bugs.python.org/issue19250> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19250] Duplicate import documentation in py3k
New submission from Nick Guenther: Python3's docs given by >> help("import") duplicate these two paragraphs: "The *public names* defined by a module are determined by checking the module's namespace for a variable named ``__all__``; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in ``__all__`` are all considered public and are required to exist. If ``__all__`` is not defined, the set of public names includes all names found in the module's namespace which do not begin with an underscore character (``'_'``). ``__all__`` should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module). The ``from`` form with ``*`` may only occur in a module scope. Attempting to use it in class or function definitions will raise a ``SyntaxError``." (full output attached for proof) Digging through the sources, I can't find where this is defined. The python2 on my system has the same two paragraphs but they are *not* duplicated there, and in the sources, which admittedly I don't know my way around, Doc/reference/simple_stmts.rst contains these paragraphs but only once each, and grep seems to imply this is the source of them. Did sphinx screw up somewhere between your hg and my distro mirror? Here's my version info (arch linux, 64 bit) [kousu@galleon ~]$ pacman -Qi python Name : python Version: 3.3.2-2 Description: Next generation of the python high-level scripting language Architecture : x86_64 URL: http://www.python.org/ Licenses : custom Groups : None Provides : python3 Depends On : expat bzip2 gdbm openssl libffi zlib Optional Deps : tk: for tkinter [installed] sqlite [installed] Required By: cython ipython ktoblzcheck libreoffice-common python-cairo python-dateutil python-dbus python-markupsafe python-numpy python-pyparsing python-pytz python-pyzmq python-setuptools python-sip python-six python-sympy python-tornado python-xdg ranger youtube-dl Optional For : systemd Conflicts With : None Replaces : python3 Installed Size : 86980.00 KiB Packager : Bart Build Date : Fri Sep 6 03:31:57 2013 Install Date : Tue Sep 17 10:29:36 2013 Install Reason : Installed as a dependency for another package Install Script : No Validated By : Signature -- assignee: docs@python components: Documentation files: The ``import`` statement - (2013-10-13 15:36:07) messages: 199775 nosy: docs@python, kousu priority: normal severity: normal status: open title: Duplicate import documentation in py3k versions: Python 3.3 Added file: http://bugs.python.org/file32096/The ``import`` statement - (2013-10-13 15:36:07) ___ Python tracker <http://bugs.python.org/issue19250> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19248] sphinx is not py3k compatible
Nick Guenther added the comment: I see that in Doc/tools/sphinx-build.py there is this check: if sys.version_info[:3] < (2, 4, 0): sys.stderr.write("""\ Error: Sphinx needs to be executed with Python 2.4 or newer (not 3.0 though). (If you run this from the Makefile, you can set the PYTHON variable to the path of an alternative interpreter executable, e.g., ``make html PYTHON=python2.5``). """) But I never saw this and had to debug it myself. I submit the attached patch as a solution. -- keywords: +patch resolution: duplicate -> status: closed -> open type: enhancement -> Added file: http://bugs.python.org/file32093/sphinx_version_check.patch ___ Python tracker <http://bugs.python.org/issue19248> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19248] sphinx is not py3k compatible
New submission from Nick Guenther: I'm running Arch. I just checked out python's hg and tried to build the docs, and found that I couldn't: $ cd cpython/doc $ make update [lots of output stripped] $ make pydoc-topics mkdir -p build/pydoc-topics build/doctrees python tools/sphinx-build.py -b pydoc-topics -d build/doctrees -D latex_paper_size= . build/pydoc-topics Traceback (most recent call last): File "tools/sphinx-build.py", line 27, in from sphinx import main File "/home/kousu/pro/cpython/Doc/tools/sphinx/__init__.py", line 44 except ImportError, err: ^ SyntaxError: invalid syntax make: *** [build] Error 1 I edited the Makefile to say PYTHON = python2 and then it worked. Is the assumption that python2 is still supposed to be the default python everywhere? Is python 3 not able to build itself yet? [kousu@galleon Doc]$ pacman -Qi python Name : python Version: 3.3.2-2 Description: Next generation of the python high-level scripting language Architecture : x86_64 URL: http://www.python.org/ Licenses : custom Groups : None Provides : python3 Depends On : expat bzip2 gdbm openssl libffi zlib Optional Deps : tk: for tkinter [installed] sqlite [installed] Required By: cython ipython ktoblzcheck libreoffice-common python-cairo python-dateutil python-dbus python-markupsafe python-numpy python-pyparsing python-pytz python-pyzmq python-setuptools python-sip python-six python-sympy python-tornado python-xdg ranger youtube-dl Optional For : systemd Conflicts With : None Replaces : python3 Installed Size : 86980.00 KiB Packager : Bart Build Date : Fri Sep 6 03:31:57 2013 Install Date : Tue Sep 17 10:29:36 2013 Install Reason : Installed as a dependency for another package Install Script : No Validated By : Signature -- assignee: docs@python components: Documentation messages: 199764 nosy: docs@python, kousu priority: normal severity: normal status: open title: sphinx is not py3k compatible versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker <http://bugs.python.org/issue19248> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2529] list/generator comprehension parser doesn't match spec
Nick Guenther <[EMAIL PROTECTED]> added the comment: Oh, okay. That's really confusing because I expect "in" to always return a bool, but in the spirit of python there's no reason it should I guess. __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2529> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2529] list/generator comprehension parser doesn't match spec
New submission from Nick Guenther <[EMAIL PROTECTED]>: I think I've found a bug in python's list comprehension parser. Observe: >>> [e for i in j in ['a','b','c']] Traceback (most recent call last): File "", line 1, in NameError: name 'j' is not defined Now, according to the grammar at http://docs.python.org/ref/lists.html, a list comprehension is (condensed for clarity): list_comprehension ::= expression list_for list_for::= "for" target_list "in" old_expression_list [list_for] So a list comprehension should always be [ for ... in for ... in ... for ... in ...] (that is, alternating 'for's and 'in's) but here I have a test case that python happily tries to run that looks like [... for ... in ... in ] -- components: Interpreter Core messages: 64818 nosy: kousu severity: normal status: open title: list/generator comprehension parser doesn't match spec type: behavior versions: Python 2.5 __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2529> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com