Re: [Python-Dev] The Tulip Has Landed
On 17 October 2013 23:40, Larry Hastings la...@hastings.org wrote: For those interested parties: Guido just checked asyncio, aka Tulip, aka PEP 3156, in to trunk. I expect it to be part of Python 3.4.0a4, hopefully to be released this weekend. Cool! How often do the online docs get built? There's nothing showing in there yet (http://docs.python.org/3.4/library/index.html)... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The Tulip Has Landed
On 10/18/2013 4:10 AM, Paul Moore wrote: On 17 October 2013 23:40, Larry Hastings la...@hastings.org wrote: For those interested parties: Guido just checked asyncio, aka Tulip, aka PEP 3156, in to trunk. I expect it to be part of Python 3.4.0a4, hopefully to be released this weekend. Cool! How often do the online docs get built? There's nothing showing in there yet (http://docs.python.org/3.4/library/index.html)... I believe once every 24 hours. The current page is dated Oct 17 (at bottom). It is now Oct 18 most everywhere. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The Tulip Has Landed
On 18 October 2013 09:56, Terry Reedy tjre...@udel.edu wrote: I believe once every 24 hours. The current page is dated Oct 17 (at bottom). It is now Oct 18 most everywhere. Thanks, I didn't know there was a generated date at the bottom. Useful to know for the future! I'll wait for the update, cheers. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #16129: Add `Py_SetStandardStreamEncoding`
On 18 Oct 2013 15:20, Georg Brandl g.bra...@gmx.net wrote: Am 17.10.2013 17:36, schrieb Antoine Pitrou: On Thu, 17 Oct 2013 15:22:03 +0200 (CEST) nick.coghlan python-check...@python.org wrote: +.. c:function:: int Py_SetStandardStreamEncoding(char *encoding, char *errors) + + .. index:: + single: Py_Initialize() + single: main() + triple: stdin; stdout; sdterr + + This function should be called before :c:func:`Py_Initialize`. It + specifies which encoding and error handling to use with standard io, + with the same meanings as in :func:`str.encode`. + + It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code + to control io encoding when the environment variable does not work. + + ``encoding`` and/or ``errors`` may be NULL to use + :envvar:`PYTHONIOENCODING` and/or default values (depending on other + settings). + + Note that :data:`sys.stderr` always uses the backslashreplace error + handler, regardless of this (or any other) setting. + + If :c:func:`Py_Finalize` is called, this function will need to be called + again in order to affect subsequent calls to :c:func:`Py_Initialize`. + + Returns 0 if successful. + + Needs a versionadded tag. Also, shouldn't it be excluded from the stable/limited API? Since it didn't expose any struct layouts, I just followed the precedent set by the other PySet_* APIs on that front. Moving it would be fine, too, though, since embedding the interpreter and using the limited API don't exactly mix :) Cheers, Nick. Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 454 (tracemalloc): new minimalist version
Hi,
I plan to push the new tracemalloc module this week-end, before the
next (and last) alpha version, except if someone complains :-) I
prefer to have one month before the first beta to have more time to
test the module. So if a major issue is raised, we may remove the
tracemalloc module before the final version.
Charles-François Natali reviewed my PEP 454 (tracemalloc). He
suggested me to split the module into two parts: keep only essential
parts in the tracemalloc Python (added to Python stdlib), and move
other tools to an external tool (hosted on PyPI). We may integrate
more functions (especially metrics) later (in Python 3.5), after they
are well tested externally. I updated the PEP and the implementation.
I will update the patch on the bug tracker later.
HTML version of the PEP:
http://www.python.org/dev/peps/pep-0454/
Since the previous version of the PEP, almost all changes are removed parts:
* Remove tasks: the tool now uses classic threads
* Remove functions to get metrics: the tool is now responsible to add
them to a snapshot
* Remove DisplayTop and DisplayTopTask class
* Remove TakeSnaphot class
PEP: 454
Title: Add a new tracemalloc module to trace Python memory allocations
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner victor.stin...@gmail.com
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-September-2013
Python-Version: 3.4
Abstract
This PEP proposes to add a new ``tracemalloc`` module to trace memory
blocks allocated by Python.
Rationale
=
Common debug tools tracing memory allocations record the C filename
and line number where the allocation occurs. Using such tools to
analyze Python memory allocations does not help because most memory
blocks are allocated in the same C function, in ``PyMem_Malloc()`` for
example.
There are debug tools dedicated to the Python language like ``Heapy``
and ``PySizer``. These tools analyze objects type and/or content.
They are useful when most memory leaks are instances of the same type
and this type is only instantiated in a few functions. Problems arise
when the object type is very common like ``str`` or ``tuple``, and it
is hard to identify where these objects are instantiated.
Finding reference cycles is also a difficult problem. There are
different tools to draw a diagram of all references. These tools
cannot be used on large applications with thousands of objects because
the diagram is too huge to be analyzed manually.
Proposal
Using the customized allocation API from PEP 445, it becomes easy to
set up a hook on Python memory allocators. A hook can inspect Python
internals to retrieve Python tracebacks.
This PEP proposes to add a new ``tracemalloc`` module, as a debug tool
to trace memory blocks allocated by Python. The module provides the
following information:
* Computed differences between two snapshots to detect memory leaks
* Statistics on allocated memory blocks per filename and per line
number: total size, number and average size of allocated memory blocks
* Traceback where a memory block was allocated
The API of the tracemalloc module is similar to the API of the
faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()``
functions, an environment variable (``PYTHONFAULTHANDLER`` and
``PYTHONTRACEMALLOC``), and a ``-X`` command line option (``-X
faulthandler`` and ``-X tracemalloc``). See the
`documentation of the faulthandler module
http://docs.python.org/3/library/faulthandler.html`_.
The tracemalloc module has been written for CPython. Other
implementations of Python may not be able to provide it.
API
===
Main Functions
--
``clear_traces()`` function:
Clear traces and statistics on Python memory allocations, and reset
the ``get_traced_memory()`` counter.
``disable()`` function:
Stop tracing Python memory allocations.
See also ``enable()`` and ``is_enabled()`` functions.
``enable()`` function:
Start tracing Python memory allocations.
At fork, the module is automatically disabled in the child process.
See also ``disable()`` and ``is_enabled()`` functions.
``get_stats()`` function:
Get statistics on traced Python memory blocks as a dictionary
``{filename (str): {line_number (int): stats}}`` where *stats* in a
``(size: int, count: int)`` tuple, *filename* and *line_number* can
be ``None``.
Return an empty dictionary if the ``tracemalloc`` module is
disabled.
See also the ``get_traces()`` function.
``get_traced_memory()`` function:
Get the current size and maximum size of memory blocks traced by the
``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``.
``get_tracemalloc_memory()`` function:
Get the memory usage in bytes of the ``tracemalloc`` module as a
tuple: ``(size: int, free: int)``.
* *size*: total size of bytes allocated by the module,
including *free* bytes
* *free*: number of free bytes available to
Re: [Python-Dev] PEP 454 (tracemalloc): new minimalist version
This looks really nice to me, and splitting it so the core functionality is in the standard library and there's a separate higher level tool on PyPI (allowing faster iteration on the analysis features) is a fine idea. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The Tulip Has Landed
On Fri, Oct 18, 2013 at 4:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 17 October 2013 23:40, Larry Hastings la...@hastings.org wrote: For those interested parties: Guido just checked asyncio, aka Tulip, aka PEP 3156, in to trunk. I expect it to be part of Python 3.4.0a4, hopefully to be released this weekend. Cool! How often do the online docs get built? There's nothing showing in there yet (http://docs.python.org/3.4/library/index.html)... It doesn't matter as Guido has not checked in any docs yet. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2013-10-11 - 2013-10-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open4184 (-56) closed 26872 (+115) total 31056 (+59) Open issues with patches: 1948 Issues opened (43) == #16203: Proposal: add re.fullmatch() method http://bugs.python.org/issue16203 reopened by georg.brandl #17294: compile-flag for single-execution to return value instead of p http://bugs.python.org/issue17294 reopened by Albert.Zeyer #19205: Don't import re and sysconfig in site.py http://bugs.python.org/issue19205 reopened by pitrou #19227: test_multiprocessing_xxx hangs under Gentoo buildbots http://bugs.python.org/issue19227 opened by pitrou #19229: operator.py: move the Python implementation in the else block http://bugs.python.org/issue19229 opened by haypo #19230: Reimplement the keyword module in C http://bugs.python.org/issue19230 opened by haypo #19232: Speed up _decimal import http://bugs.python.org/issue19232 opened by skrah #19233: test_io.test_interrupted_write_retry_text() hangs on Solaris 1 http://bugs.python.org/issue19233 opened by haypo #19234: socket.fileno() documentation http://bugs.python.org/issue19234 opened by georg.brandl #19235: Add a dedicated subclass for recursion errors http://bugs.python.org/issue19235 opened by elazar #19236: Add Tornado HTTP benchmark http://bugs.python.org/issue19236 opened by pitrou #19237: Proposal : LCM function to complement GCD http://bugs.python.org/issue19237 opened by CliffM #19238: Misleading explanation of fill and align in format_spec http://bugs.python.org/issue19238 opened by davidchambers #19239: add inspect functions to retrieve attributes from both old dir http://bugs.python.org/issue19239 opened by ethan.furman #19240: iglob should try to use `readdir` http://bugs.python.org/issue19240 opened by tebeka #19241: MINGW: install import library http://bugs.python.org/issue19241 opened by rpetrov #19242: MINGW: generalization of posix installation in distutils http://bugs.python.org/issue19242 opened by rpetrov #19243: MINGW: support stdcall without underscore http://bugs.python.org/issue19243 opened by rpetrov #19244: MINGW: use replace instead rename to avoid failure on windows http://bugs.python.org/issue19244 opened by rpetrov #19245: mingw-meta: installation http://bugs.python.org/issue19245 opened by rpetrov #19247: Describe surrogateescape algorithm in the Library Reference http://bugs.python.org/issue19247 opened by ncoghlan #19249: Enumeration.__eq__ http://bugs.python.org/issue19249 opened by CliffM #19251: bitwise ops for bytes of equal length http://bugs.python.org/issue19251 opened by christian.heimes #19253: PyArg_ParseTuple: wrong use of seterror() clips helpful type e http://bugs.python.org/issue19253 opened by georg.brandl #19254: Python implementation of PBKDF2_HMAC http://bugs.python.org/issue19254 opened by christian.heimes #19255: Don't wipe builtins at shutdown http://bugs.python.org/issue19255 opened by serhiy.storchaka #19256: Optimize marshal format and add version token. http://bugs.python.org/issue19256 opened by kristjan.jonsson #19257: Sub-optimal error message when importing a non-package http://bugs.python.org/issue19257 opened by alex #19259: Provide Python implementation of operator.compare_digest() http://bugs.python.org/issue19259 opened by christian.heimes #19261: Add support for 24-bit in the sunau module http://bugs.python.org/issue19261 opened by serhiy.storchaka #19262: Add asyncio (tulip, PEP 3156) to stdlib http://bugs.python.org/issue19262 opened by gvanrossum #19263: enum.py : Enum.__new__(). __objclass__ http://bugs.python.org/issue19263 opened by CliffM #19264: subprocess.Popen doesn't support unicode on Windows http://bugs.python.org/issue19264 opened by peter0 #19265: Increased test coverage for datetime pickling http://bugs.python.org/issue19265 opened by Colin.Williams #19270: sched.cancel() breaks events order http://bugs.python.org/issue19270 opened by serhiy.storchaka #19271: Update OrderedDict see also link http://bugs.python.org/issue19271 opened by BreamoreBoy #19273: Update PCbuild/readme.txt http://bugs.python.org/issue19273 opened by zach.ware #19274: make zipfile.PyZipFile more usable http://bugs.python.org/issue19274 opened by Christian.Tismer #19279: UTF-7 to UTF-8 decoding crash http://bugs.python.org/issue19279 opened by glebourgeois #19280: Add a datatype to represent mime types to the email module http://bugs.python.org/issue19280 opened by r.david.murray #19281: add __objclass__ to the docs http://bugs.python.org/issue19281 opened by ethan.furman #19282: dbm is not a context manager http://bugs.python.org/issue19282 opened by kousu #19283: Need support to avoid Windows CRT compatibility issue. http://bugs.python.org/issue19283 opened by
[Python-Dev] Looking for volunteers to test Tulip on Windows
I'm working from home today and my Windows laptop is in the office, so I won't be able to test my latest Tulip changes on Windows (I just renamed pause to pause_reading, and hope to commit pause_reading later today). Is anyone luckier? Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I lost courage, so I've not tested the SSL behavior on Windows at all. Is anyone lucky enough to have a Python 3.4 alpha for Windows setup with OpenSSL already configured? If so, could you run test_asyncio for me and let me know if there are any issues? (Actually, even a working Python 3.3 setup would be useful, although you'd have to check out the Tulip repo and run the tests from there.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] possibility of shaving a stat call from imports
importlib.machinery.FileFinder does a stat call to check if a path is a file if the package check failed. Now I'm willing to bet that the check is rather redundant as the file extension should be a dead give-away that something in a directory is a file and not some non-file type. The import would still fail even if this is the case in the loader when attempting to read from the file, but it happens a little later and it means finders would be more permissive in claiming they found a loader. Does anyone see a good reason not to take the more optimistic route in the finder? As I said, the only thing I see breaking user code is if they have a directory or something named spam.py and so the finder claims it found a module when in fact it didn't and thus stopping the search for the module from continuing. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
On 18/10/2013 5:31pm, Guido van Rossum wrote: I'm working from home today and my Windows laptop is in the office, so I won't be able to test my latest Tulip changes on Windows (I just renamed pause to pause_reading, and hope to commit pause_reading later today). Is anyone luckier? $ hg id 97f6f35b02e5 (asyncio) tip $ python-release runtests.py Skipping 'test_unix_events': UNIX only s.s..ss....ssss..sss.ss...ss -- Ran 544 tests in 12.790s OK (skipped=39) Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I lost courage, so I've not tested the SSL behavior on Windows at all. Is anyone lucky enough to have a Python 3.4 alpha for Windows setup with OpenSSL already configured? If so, could you run test_asyncio for me and let me know if there are any issues? (Actually, even a working Python 3.3 setup would be useful, although you'd have to check out the Tulip repo and run the tests from there.) $ python-release -c 'import sys; print(sys.version)' 3.4.0a3+ (default:7172135d60f6, Oct 18 2013, 17:54:18) [MSC v.1600 32 bit (Intel)] $ python-release -m test test_asyncio [1/1] test_asyncio 1 test OK. BTW, pcbuild.sln was not building _overlapped automatically -- I have pushed a fix. -- Richard ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
Thanks! There are some new changes (I fixed a race with sockets closing) and I hope to land flow control (finally) later today. Do you know what those skips are? I suspect they might be due to ssl not working for you either. :-( On Fri, Oct 18, 2013 at 10:04 AM, Richard Oudkerk shibt...@gmail.comwrote: On 18/10/2013 5:31pm, Guido van Rossum wrote: I'm working from home today and my Windows laptop is in the office, so I won't be able to test my latest Tulip changes on Windows (I just renamed pause to pause_reading, and hope to commit pause_reading later today). Is anyone luckier? $ hg id 97f6f35b02e5 (asyncio) tip $ python-release runtests.py Skipping 'test_unix_events': UNIX only ..**..s...** ..s..ss...**.ss...**.ss..sss.ss...** ss**..** ..**..** ..**..** ..**..** ..**..** ..**..** ..** --**--**-- Ran 544 tests in 12.790s OK (skipped=39) Also, reading through the Windows OpenSSL setup in PCbuild/readme.txt I lost courage, so I've not tested the SSL behavior on Windows at all. Is anyone lucky enough to have a Python 3.4 alpha for Windows setup with OpenSSL already configured? If so, could you run test_asyncio for me and let me know if there are any issues? (Actually, even a working Python 3.3 setup would be useful, although you'd have to check out the Tulip repo and run the tests from there.) $ python-release -c 'import sys; print(sys.version)' 3.4.0a3+ (default:7172135d60f6, Oct 18 2013, 17:54:18) [MSC v.1600 32 bit (Intel)] $ python-release -m test test_asyncio [1/1] test_asyncio 1 test OK. BTW, pcbuild.sln was not building _overlapped automatically -- I have pushed a fix. -- Richard -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756
Is anyone looking into those leaks? I suspect they might have to do with Serhiy's latest re changes in add40e9f7cbe. (just a barely educated guess, though) Regards Antoine. On Fri, 18 Oct 2013 09:31:49 +0200 solip...@pitrou.net wrote: results for 30f33e6a04c1 on branch default test_unittest leaked [317, 317, 317] references, sum=951 test_unittest leaked [196, 197, 197] memory blocks, sum=590 test_doctest leaked [2232, 2232, 2232] references, sum=6696 test_doctest leaked [1667, 1669, 1669] memory blocks, sum=5005 test_argparse leaked [70935, 70935, 70935] references, sum=212805 test_argparse leaked [43489, 43491, 43491] memory blocks, sum=130471 test_asyncio leaked [2577, -2577, 0] references, sum=0 test_asyncio leaked [786, -1103, 2] memory blocks, sum=-315 test_cgitb leaked [90, 90, 90] references, sum=270 test_cgitb leaked [65, 65, 65] memory blocks, sum=195 test_cmd leaked [41, 41, 41] references, sum=123 test_cmd leaked [20, 20, 20] memory blocks, sum=60 test_cmd_line leaked [34, 34, 34] references, sum=102 test_cmd_line leaked [34, 34, 34] memory blocks, sum=102 test_cmd_line_script leaked [24, 24, 24] references, sum=72 test_cmd_line_script leaked [19, 19, 19] memory blocks, sum=57 test_compileall leaked [6, 6, 6] references, sum=18 test_compileall leaked [6, 6, 6] memory blocks, sum=18 test_configparser leaked [34560, 34560, 34560] references, sum=103680 test_configparser leaked [16778, 16780, 16780] memory blocks, sum=50338 test_contextlib leaked [10, 10, 10] references, sum=30 test_contextlib leaked [5, 5, 5] memory blocks, sum=15 test_ctypes leaked [137, 137, 137] references, sum=411 test_ctypes leaked [105, 105, 105] memory blocks, sum=315 test_decimal leaked [10, 10, 10] references, sum=30 test_decimal leaked [8, 8, 8] memory blocks, sum=24 test_difflib leaked [401, 401, 401] references, sum=1203 test_difflib leaked [267, 269, 269] memory blocks, sum=805 test_distutils leaked [533, 533, 533] references, sum=1599 test_distutils leaked [389, 391, 391] memory blocks, sum=1171 test_docxmlrpc leaked [60, 60, 60] references, sum=180 test_docxmlrpc leaked [60, 60, 60] memory blocks, sum=180 test_email leaked [4307, 4307, 4307] references, sum=12921 test_email leaked [2419, 2421, 2421] memory blocks, sum=7261 test_faulthandler leaked [111, 111, 111] references, sum=333 test_faulthandler leaked [98, 98, 98] memory blocks, sum=294 test_gc leaked [4, 4, 4] references, sum=12 test_gc leaked [4, 4, 4] memory blocks, sum=12 test_generators leaked [5, 5, 5] references, sum=15 test_generators leaked [4, 4, 4] memory blocks, sum=12 test_genexps leaked [5, 5, 5] references, sum=15 test_genexps leaked [4, 4, 4] memory blocks, sum=12 test_gettext leaked [134, 134, 134] references, sum=402 test_gettext leaked [128, 128, 128] memory blocks, sum=384 test_htmlparser leaked [2177, 2177, 2177] references, sum=6531 test_htmlparser leaked [40, 41, 41] memory blocks, sum=122 test_http_cookiejar leaked [2805, 2805, 2805] references, sum=8415 test_http_cookiejar leaked [1808, 1810, 1810] memory blocks, sum=5428 test_imp leaked [1, -1, 0] references, sum=0 test_import leaked [16, 16, 16] references, sum=48 test_import leaked [10, 10, 10] memory blocks, sum=30 test_inspect leaked [2, 2, 2] references, sum=6 test_inspect leaked [2, 2, 2] memory blocks, sum=6 test_json leaked [2553, 2553, 2553] references, sum=7659 test_json leaked [1323, 1325, 1325] memory blocks, sum=3973 test_keyword leaked [39, 39, 39] references, sum=117 test_keyword leaked [20, 20, 20] memory blocks, sum=60 test_lib2to3 leaked [57052, 57052, 57052] references, sum=171156 test_lib2to3 leaked [46298, 46300, 46300] memory blocks, sum=138898 test_listcomps leaked [10, 10, 10] references, sum=30 test_listcomps leaked [8, 8, 8] memory blocks, sum=24 test_locale leaked [43, 43, 43] references, sum=129 test_locale leaked [19, 19, 19] memory blocks, sum=57 test_logging leaked [706, 706, 706] references, sum=2118 test_logging leaked [362, 364, 364] memory blocks, sum=1090 test_mailbox leaked [46, 46, 46] references, sum=138 test_mailbox leaked [41, 41, 41] memory blocks, sum=123 test_multiprocessing_forkserver leaked [2, 2, 2] references, sum=6 test_multiprocessing_forkserver leaked [2, 2, 2] memory blocks, sum=6 test_multiprocessing_spawn leaked [2, 2, 2] references, sum=6 test_multiprocessing_spawn leaked [2, 2, 2] memory blocks, sum=6 test_netrc leaked [96, 96, 96] references, sum=288 test_netrc leaked [59, 59, 59] memory blocks, sum=177 test_nntplib leaked [1774, 1774, 1774] references, sum=5322 test_nntplib leaked [902, 904, 904] memory blocks, sum=2710 test_parser leaked [2, 2, 2] references, sum=6 test_parser leaked [2, 2, 2] memory blocks, sum=6 test_pep292 leaked [174, 174, 174] references, sum=522 test_pep292 leaked [107, 107, 107] memory blocks, sum=321 test_pkg leaked [57, 57, 57] references, sum=171 test_pkg leaked
Re: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756
On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou solip...@pitrou.net wrote: Is anyone looking into those leaks? I suspect they might have to do with Serhiy's latest re changes in add40e9f7cbe. (just a barely educated guess, though) I have confirmed that and was working towards getting my findings posted when this message came in :). Specifically, the leak is somewhere in sub (I believe pattern_subx() in _sre.c), but I currently lack the skill to figure out where. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
On 18/10/2013 6:15pm, Guido van Rossum wrote: Thanks! There are some new changes (I fixed a race with sockets closing) and I hope to land flow control (finally) later today. Do you know what those skips are? I suspect they might be due to ssl not working for you either. :-( Lack of support for subprocess, signals, ssl (with iocp), pipes, add_*() (with iocp): $ python-release runtests.py -v 21 | grep skipped test_add_signal_handler (test_events.SelectEventLoopTests) ... skipped 'No SIGKILL' test_read_pipe (test_events.SelectEventLoopTests) ... skipped Don't support pipes for Windows test_signal_handling_args (test_events.SelectEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_selecting (test_events.SelectEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_finish (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_close_client_stream (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exec (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exitcode (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_interactive (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_kill (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_send_signal (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_shell (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr_redirect_to_stdout (test_events.SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_write_pipe (test_events.SelectEventLoopTests) ... skipped Don't support pipes for Windows test_write_pipe_disconnect_on_close (test_events.SelectEventLoopTests) ... skipped Don't support pipes for Windows test_add_signal_handler (test_events.ProactorEventLoopTests) ... skipped 'No SIGKILL' test_create_datagram_endpoint (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have create_datagram_endpoint()' test_create_server_ssl (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_create_ssl_connection (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_read_pipe (test_events.ProactorEventLoopTests) ... skipped Don't support pipes for Windows test_reader_callback (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_reader_callback_cancel (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_signal_handling_args (test_events.ProactorEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_selecting (test_events.ProactorEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_finish (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_close_client_stream (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exec (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exitcode (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_interactive (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_kill (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_send_signal (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_shell (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr_redirect_to_stdout (test_events.ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_write_pipe (test_events.ProactorEventLoopTests) ... skipped Don't support pipes for Windows test_write_pipe_disconnect_on_close (test_events.ProactorEventLoopTests) ... skipped Don't support pipes for Windows test_writer_callback (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' test_writer_callback_cancel (test_events.ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_writer()' OK (skipped=39) -- Richard ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
[Python-Dev] Current start-up times
tl;dr: 2.7 - 3.3 = 1.92x slower 2.7 - 3.4 = 1.36x slower 3.3 - 3.4 = 1.40x faster IOW the work people have been putting in to speed up interpreter startup are definitely paying off. ./perf.py -b normal_startup,startup_nosite ../cpython/py3.3/python.exe ../cpython/default/python.exe Running normal_startup... INFO:root:Running `../cpython/default/python.exe -c ` 1000 times INFO:root:Running `../cpython/py3.3/python.exe -c ` 1000 times Running startup_nosite... INFO:root:Running `../cpython/default/python.exe -S -c ` 2000 times INFO:root:Running `../cpython/py3.3/python.exe -S -c ` 2000 times Report on Darwin bcannon-macbookpro2.roam.corp.google.com 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### normal_startup ### Min: 0.679865 - 0.484924: 1.40x faster Avg: 0.696644 - 0.497123: 1.40x faster Significant (t=47.03) Stddev: 0.02796 - 0.01087: 2.5731x smaller ### startup_nosite ### Min: 0.297685 - 0.285106: 1.04x faster Avg: 0.319151 - 0.309291: 1.03x faster Significant (t=5.64) Stddev: 0.01219 - 0.01252: 1.0272x larger ./perf.py -b normal_startup,startup_nosite ../cpython/py2.7/python.exe ../cpython/default/python.exe Running normal_startup... INFO:root:Running `../cpython/default/python.exe -c ` 1000 times INFO:root:Running `../cpython/py2.7/python.exe -c ` 1000 times Running startup_nosite... INFO:root:Running `../cpython/default/python.exe -S -c ` 2000 times INFO:root:Running `../cpython/py2.7/python.exe -S -c ` 2000 times Report on Darwin bcannon-macbookpro2.roam.corp.google.com 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### normal_startup ### Min: 0.350003 - 0.484200: 1.38x slower Avg: 0.362935 - 0.492708: 1.36x slower Significant (t=-44.29) Stddev: 0.02021 - 0.00459: 4.3999x smaller ### startup_nosite ### Min: 0.165302 - 0.282765: 1.71x slower Avg: 0.169915 - 0.293589: 1.73x slower Significant (t=-222.46) Stddev: 0.00273 - 0.00484: 1.7705x larger ./perf.py -b normal_startup,startup_nosite ../cpython/py2.7/python.exe ../cpython/py3.3/python.exe Running normal_startup... INFO:root:Running `../cpython/py3.3/python.exe -c ` 1000 times INFO:root:Running `../cpython/py2.7/python.exe -c ` 1000 times Running startup_nosite... INFO:root:Running `../cpython/py3.3/python.exe -S -c ` 2000 times INFO:root:Running `../cpython/py2.7/python.exe -S -c ` 2000 times Report on Darwin bcannon-macbookpro2.roam.corp.google.com 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### normal_startup ### Min: 0.350545 - 0.675245: 1.93x slower Avg: 0.363317 - 0.696069: 1.92x slower Significant (t=-120.97) Stddev: 0.01682 - 0.00977: 1.7222x smaller ### startup_nosite ### Min: 0.166232 - 0.289506: 1.74x slower Avg: 0.172511 - 0.300979: 1.74x slower Significant (t=-209.92) Stddev: 0.00299 - 0.00534: 1.7862x larger ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756
On Fri, Oct 18, 2013 at 1:34 PM, Zachary Ware zachary.ware+py...@gmail.comwrote: On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou solip...@pitrou.net wrote: Is anyone looking into those leaks? I suspect they might have to do with Serhiy's latest re changes in add40e9f7cbe. (just a barely educated guess, though) I have confirmed that and was working towards getting my findings posted when this message came in :). Specifically, the leak is somewhere in sub (I believe pattern_subx() in _sre.c), but I currently lack the skill to figure out where. I think there is a Py_DECREF(list) call missing at line 2434. Running the test suite now to verify. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756
On Fri, Oct 18, 2013 at 1:51 PM, Brett Cannon br...@python.org wrote: On Fri, Oct 18, 2013 at 1:34 PM, Zachary Ware zachary.ware+py...@gmail.com wrote: On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou solip...@pitrou.net wrote: Is anyone looking into those leaks? I suspect they might have to do with Serhiy's latest re changes in add40e9f7cbe. (just a barely educated guess, though) I have confirmed that and was working towards getting my findings posted when this message came in :). Specifically, the leak is somewhere in sub (I believe pattern_subx() in _sre.c), but I currently lack the skill to figure out where. I think there is a Py_DECREF(list) call missing at line 2434. Running the test suite now to verify. Make that line 2430. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc): new minimalist version
Hi,
I'm happy to see this move forward!
API
===
Main Functions
--
``clear_traces()`` function:
Clear traces and statistics on Python memory allocations, and reset
the ``get_traced_memory()`` counter.
That's nitpicking, but how about just ``reset()`` (I'm probably biased
by oprofile's opcontrol --reset)?
``get_stats()`` function:
Get statistics on traced Python memory blocks as a dictionary
``{filename (str): {line_number (int): stats}}`` where *stats* in a
``(size: int, count: int)`` tuple, *filename* and *line_number* can
be ``None``.
It's probably obvious, but you might want to say once what *size* and
*count* represent (and the unit for *size*).
``get_tracemalloc_memory()`` function:
Get the memory usage in bytes of the ``tracemalloc`` module as a
tuple: ``(size: int, free: int)``.
* *size*: total size of bytes allocated by the module,
including *free* bytes
* *free*: number of free bytes available to store data
What's *free* exactly? I assume it's linked to the internal storage
area used by tracemalloc itself, but that's not clear at all.
Also, is the tracemalloc overhead included in the above stats (I'm
mainly thinking about get_stats() and get_traced_memory()?
If yes, I find it somewhat confusing: for example, AFAICT, valgrind's
memcheck doesn't report the memory overhead, although it can be quite
large, simply because it's not interesting.
Trace Functions
---
``get_traceback_limit()`` function:
Get the maximum number of frames stored in the traceback of a trace
of a memory block.
Use the ``set_traceback_limit()`` function to change the limit.
I didn't see anywhere the default value for this setting: it would be
nice to write it somewhere, and also explain the rationale (memory/CPU
overhead...).
``get_object_address(obj)`` function:
Get the address of the main memory block of the specified Python object.
A Python object can be composed by multiple memory blocks, the
function only returns the address of the main memory block.
IOW, this should return the same as id() on CPython? If yes, it could
be an interesting note.
``get_object_trace(obj)`` function:
Get the trace of a Python object *obj* as a ``(size: int,
traceback)`` tuple where *traceback* is a tuple of ``(filename: str,
lineno: int)`` tuples, *filename* and *lineno* can be ``None``.
I find the trace word confusing, so it might be interesting to add a
note somewhere explaining what it is (callstack leading to the object
allocation, or whatever).
Also, this function leaves me a mixed feeling: it's called
get_object_trace(), but you also return the object size - well, a
vague estimate thereof. I wonder if the size really belongs here,
especially if the information returned isn't really accurate: it will
be for an integer, but not for e.g. a list, right? How about just
using sys.getsizeof(), which would give a more accurate result?
``get_trace(address)`` function:
Get the trace of a memory block as a ``(size: int, traceback)``
tuple where *traceback* is a tuple of ``(filename: str, lineno:
int)`` tuples, *filename* and *lineno* can be ``None``.
Return ``None`` if the ``tracemalloc`` module did not trace the
allocation of the memory block.
See also ``get_object_trace()``, ``get_stats()`` and
``get_traces()`` functions.
Do you have example use cases where you want to work with a raw addresses?
Filter
--
``Filter(include: bool, pattern: str, lineno: int=None, traceback:
bool=False)`` class:
Filter to select which memory allocations are traced. Filters can be
used to reduce the memory usage of the ``tracemalloc`` module, which
can be read using the ``get_tracemalloc_memory()`` function.
``match(filename: str, lineno: int)`` method:
Return ``True`` if the filter matchs the filename and line number,
``False`` otherwise.
``match_filename(filename: str)`` method:
Return ``True`` if the filter matchs the filename, ``False`` otherwise.
``match_lineno(lineno: int)`` method:
Return ``True`` if the filter matchs the line number, ``False``
otherwise.
``match_traceback(traceback)`` method:
Return ``True`` if the filter matchs the *traceback*, ``False``
otherwise.
*traceback* is a tuple of ``(filename: str, lineno: int)`` tuples.
Are those ``match`` methods really necessary for the end user, i.e.
are they worth being exposed as part of the public API?
StatsDiff
-
``StatsDiff(differences, old_stats, new_stats)`` class:
Differences between two ``GroupedStats`` instances.
The ``GroupedStats.compare_to()`` method creates a ``StatsDiff``
instance.
``sort()`` method:
Sort the ``differences`` list from the biggest difference to the
smallest difference. Sort by ``abs(size_diff)``, *size*,
``abs(count_diff)``, *count* and then by *key*.
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
Thanks! Those are all expected (though contributions are always welcome -- not looking at you specifically :-). Does examples/fetch3.py work for you with an https URL? (Try http://dropbox.com, i.e. without 's' -- you get two redirects to https URLs. :-) On Fri, Oct 18, 2013 at 10:36 AM, Richard Oudkerk shibt...@gmail.comwrote: On 18/10/2013 6:15pm, Guido van Rossum wrote: Thanks! There are some new changes (I fixed a race with sockets closing) and I hope to land flow control (finally) later today. Do you know what those skips are? I suspect they might be due to ssl not working for you either. :-( Lack of support for subprocess, signals, ssl (with iocp), pipes, add_*() (with iocp): $ python-release runtests.py -v 21 | grep skipped test_add_signal_handler (test_events.**SelectEventLoopTests) ... skipped 'No SIGKILL' test_read_pipe (test_events.**SelectEventLoopTests) ... skipped Don't support pipes for Windows test_signal_handling_args (test_events.**SelectEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_**selecting (test_events.**SelectEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_**finish (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_close_client_**stream (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exec (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exitcode (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_interactive (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_kill (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_send_signal (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_shell (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr_**redirect_to_stdout (test_events.**SelectEventLoopTests) ... skipped Don't support subprocess for Windows yet test_write_pipe (test_events.**SelectEventLoopTests) ... skipped Don't support pipes for Windows test_write_pipe_disconnect_on_**close (test_events.**SelectEventLoopTests) ... skipped Don't support pipes for Windows test_add_signal_handler (test_events.**ProactorEventLoopTests) ... skipped 'No SIGKILL' test_create_datagram_endpoint (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have create_datagram_endpoint()' test_create_server_ssl (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_create_ssl_connection (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop imcompatible with SSL' test_read_pipe (test_events.**ProactorEventLoopTests) ... skipped Don't support pipes for Windows test_reader_callback (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_reader_callback_cancel (test_events.**ProactorEventLoopTests) ... skipped 'IocpEventLoop does not have add_reader()' test_signal_handling_args (test_events.**ProactorEventLoopTests) ... skipped 'No SIGALRM' test_signal_handling_while_**selecting (test_events.**ProactorEventLoopTests) ... skipped 'No SIGALRM' test_subprocess_close_after_**finish (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_close_client_**stream (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exec (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_exitcode (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_interactive (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_kill (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_send_signal (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_shell (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_subprocess_stderr_**redirect_to_stdout (test_events.**ProactorEventLoopTests) ... skipped Don't support subprocess for Windows yet test_write_pipe (test_events.**ProactorEventLoopTests) ... skipped Don't support pipes for Windows test_write_pipe_disconnect_on_**close (test_events.**ProactorEventLoopTests) ... skipped Don't
Re: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756
On Fri, Oct 18, 2013 at 1:53 PM, Brett Cannon br...@python.org wrote: On Fri, Oct 18, 2013 at 1:51 PM, Brett Cannon br...@python.org wrote: On Fri, Oct 18, 2013 at 1:34 PM, Zachary Ware zachary.ware+py...@gmail.com wrote: On Fri, Oct 18, 2013 at 12:30 PM, Antoine Pitrou solip...@pitrou.net wrote: Is anyone looking into those leaks? I suspect they might have to do with Serhiy's latest re changes in add40e9f7cbe. (just a barely educated guess, though) I have confirmed that and was working towards getting my findings posted when this message came in :). Specifically, the leak is somewhere in sub (I believe pattern_subx() in _sre.c), but I currently lack the skill to figure out where. I think there is a Py_DECREF(list) call missing at line 2434. Running the test suite now to verify. Make that line 2430. That was it, so Antoine and Zach were right about the location. Should be fixed now. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
On 18/10/2013 6:57pm, Guido van Rossum wrote: Thanks! Those are all expected (though contributions are always welcome -- not looking at you specifically :-). Does examples/fetch3.py work for you with an https URL? (Try http://dropbox.com, i.e. without 's' -- you get two redirects to https URLs. :-) It fails -- not sure why... $ python-release examples/fetch3.py http://dropbox.com redirect to https://dropbox.com/ Traceback (most recent call last): File examples/fetch3.py, line 211, in module main() File examples/fetch3.py, line 206, in main body = loop.run_until_complete(fetch(sys.argv[1], '-v' in sys.argv)) File C:\Repos\cpython-dirty\lib\asyncio\base_events.py, line 172, in run_until_complete self.run_forever() File C:\Repos\cpython-dirty\lib\asyncio\base_events.py, line 153, in run_forever self._run_once() File C:\Repos\cpython-dirty\lib\asyncio\base_events.py, line 576, in _run_once event_list = self._selector.select(timeout) File C:\Repos\cpython-dirty\lib\selectors.py, line 219, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File C:\Repos\cpython-dirty\lib\selectors.py, line 210, in _select r, w, x = select.select(r, w, w, timeout) OSError: [WinError 10038] An operation was attempted on something that is not a socket -- Richard ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily reference leaks (30f33e6a04c1): sum=975756
18.10.13 21:04, Brett Cannon написав(ла): That was it, so Antoine and Zach were right about the location. Should be fixed now. Thank you Brett. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
Maybe the dummy socket returned by wrap_socket() is not acceptable for select? --Guido van Rossum (sent from Android phone) On Oct 18, 2013 11:26 AM, Richard Oudkerk shibt...@gmail.com wrote: On 18/10/2013 6:57pm, Guido van Rossum wrote: Thanks! Those are all expected (though contributions are always welcome -- not looking at you specifically :-). Does examples/fetch3.py work for you with an https URL? (Try http://dropbox.com, i.e. without 's' -- you get two redirects to https URLs. :-) It fails -- not sure why... $ python-release examples/fetch3.py http://dropbox.com redirect to https://dropbox.com/ Traceback (most recent call last): File examples/fetch3.py, line 211, in module main() File examples/fetch3.py, line 206, in main body = loop.run_until_complete(fetch(**sys.argv[1], '-v' in sys.argv)) File C:\Repos\cpython-dirty\lib\**asyncio\base_events.py, line 172, in run_until_complete self.run_forever() File C:\Repos\cpython-dirty\lib\**asyncio\base_events.py, line 153, in run_forever self._run_once() File C:\Repos\cpython-dirty\lib\**asyncio\base_events.py, line 576, in _run_once event_list = self._selector.select(timeout) File C:\Repos\cpython-dirty\lib\**selectors.py, line 219, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File C:\Repos\cpython-dirty\lib\**selectors.py, line 210, in _select r, w, x = select.select(r, w, w, timeout) OSError: [WinError 10038] An operation was attempted on something that is not a socket -- Richard ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] possibility of shaving a stat call from imports
On Fri, 18 Oct 2013 12:53:55 -0400 Brett Cannon br...@python.org wrote: importlib.machinery.FileFinder does a stat call to check if a path is a file if the package check failed. Now I'm willing to bet that the check is rather redundant as the file extension should be a dead give-away that something in a directory is a file and not some non-file type. The import would still fail even if this is the case in the loader when attempting to read from the file, but it happens a little later and it means finders would be more permissive in claiming they found a loader. Is it one stat() call per successful import? Or one stat() call per sys.path entry? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
On 18/10/2013 9:19pm, Guido van Rossum wrote: Maybe the dummy socket returned by wrap_socket() is not acceptable for select? An error SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553)') is being raised in _on_handshake(). This seems to result in the socket being closed without being unregistered from the selector. select() fails before the SSLError gets reported, so it does not appear in the traceback. -- Richard ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] possibility of shaving a stat call from imports
On 19 October 2013 03:53, Brett Cannon br...@python.org wrote: importlib.machinery.FileFinder does a stat call to check if a path is a file if the package check failed. Now I'm willing to bet that the check is rather redundant as the file extension should be a dead give-away that something in a directory is a file and not some non-file type. The import would still fail even if this is the case in the loader when attempting to read from the file, but it happens a little later and it means finders would be more permissive in claiming they found a loader. Does anyone see a good reason not to take the more optimistic route in the finder? As I said, the only thing I see breaking user code is if they have a directory or something named spam.py and so the finder claims it found a module when in fact it didn't and thus stopping the search for the module from continuing. Whilst directories with extensions are unusual on Windows, they're fairly common on UNIX-based systems. For example, blah.rc directories. And I personally often create directories with extensions - usually a timestamp of some kind. If the extension is specifically an extension that Python uses (e.g. .py[cod]) then I think it would be reasonable to make the assumption and let the import fail at the loader instead. Would the extension check be faster or slower than another stat() call? As an alternative, is there another stat call later that could be bypassed if you temporarily cached the result of this stat call? And if so, when should the cached value be cleared? Tim Delaney ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
Good sleuthing! Does the attached patch fix it? (Off-topic: the code is pretty inconsistent about catching BaseException. Maybe it shouldn't be caught at all?) On Fri, Oct 18, 2013 at 2:04 PM, Richard Oudkerk shibt...@gmail.com wrote: On 18/10/2013 9:19pm, Guido van Rossum wrote: Maybe the dummy socket returned by wrap_socket() is not acceptable for select? An error SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:553)') is being raised in _on_handshake(). This seems to result in the socket being closed without being unregistered from the selector. select() fails before the SSLError gets reported, so it does not appear in the traceback. -- Richard -- --Guido van Rossum (python.org/~guido) quickfix.diff Description: Binary data ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
On 18/10/2013 10:37pm, Guido van Rossum wrote:
Good sleuthing! Does the attached patch fix it?
(Off-topic: the code is pretty inconsistent about catching
BaseException. Maybe it shouldn't be caught at all?)
It fixes it in the sense of printing a sensible traceback;-)
$ PYTHONPATH='c:/Repos/tulip' /c/Repos/cpython-33/PCbuild/python
fetch3.py http://dropbox.com -v
* Connecting to dropbox.com:80 using tcp
* dropbox.com resolves to 108.160.165.62, 108.160.166.62,
199.47.216.179, 199.47.217.179
* New connection ('108.160.165.62', 80, False)
* Connected to ('108.160.165.62', 80)
GET / HTTP/1.1
Host: dropbox.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 18 Oct 2013 22:40:13 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://dropbox.com/
redirect to https://dropbox.com/
* Connecting to dropbox.com:443 using ssl
* dropbox.com resolves to 108.160.165.62, 108.160.166.62,
199.47.216.179, 199.47.217.179
Traceback (most recent call last):
File fetch3.py, line 211, in module
main()
File fetch3.py, line 206, in main
body = loop.run_until_complete(fetch(sys.argv[1], '-v' in sys.argv))
File c:\Repos\tulip\asyncio\base_events.py, line 177, in
run_until_complete
return future.result()
File c:\Repos\tulip\asyncio\futures.py, line 221, in result
raise self._exception
File c:\Repos\tulip\asyncio\tasks.py, line 257, in _step
result = coro.throw(exc)
File fetch3.py, line 192, in fetch
yield from request.connect(pool)
File fetch3.py, line 80, in connect
ssl=self.ssl)
File fetch3.py, line 36, in open_connection
reader, writer = yield from open_connection(host, port, ssl=ssl)
File c:\Repos\tulip\asyncio\streams.py, line 41, in open_connection
lambda: protocol, host, port, **kwds)
File c:\Repos\tulip\asyncio\base_events.py, line 356, in
create_connection
yield from waiter
File c:\Repos\tulip\asyncio\futures.py, line 318, in __iter__
yield self # This tells Task to wait for completion.
File c:\Repos\tulip\asyncio\tasks.py, line 308, in _wakeup
value = future.result()
File c:\Repos\tulip\asyncio\futures.py, line 221, in result
raise self._exception
File c:\Repos\tulip\asyncio\selector_events.py, line 579, in
_on_handshake
self._sock.do_handshake()
File C:\Repos\cpython-33\lib\ssl.py, line 520, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:553)
--
Richard
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] possibility of shaving a stat call from imports
On Fri, Oct 18, 2013 at 2:59 PM, Antoine Pitrou solip...@pitrou.net wrote: Is it one stat() call per successful import? Or one stat() call per sys.path entry? It's one per finder (i.e. path entry) where a matching name is in the directory (per the finder's cache). So it's a pretty uncommon case with not much overhead when the name is not in the cache. The same thing happens earlier with directories, checking to see if the name was in the cache and *then* checking if it is a directory (_path_isdir). Again, uncommon and otherwise not all that impactful on performance. In the directory case removing the directory check might be inadvisable since it it would result in a _path_isfile call for each registered suffix before moving on (plus it would be incorrectly considered a portion of a potential namespace package). Of course, this is what happens already with directories that have no __init__.suffix file in them, which is also an unlikely case that doesn't make much of a difference on performance normally. In the file case I agree that it's much more unlikely that there will be a directory in just the right place with a .py (etc.) suffix in the name. In either case I don't think it saves us much performance-wise, but it may be nice to make that code a simpler if it won't realistically cause anyone headaches. I'm fine with just leaving it as-is. (Also, FTR Brett's email relates to http://bugs.python.org/issue18810.) -eric p.s. I brought up the same point about dir/file checks in http://bugs.python.org/issue19216. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for volunteers to test Tulip on Windows
Thanks! That's probably fine for now -- it means the standard library
doesn't know where the root certificates are. We had a huge discussion
about this over on python-tulip:
https://groups.google.com/forum/#!topic/python-tulip/c_lqdFjPEbE
TL;DR: The stdlib openssl wrapper ought to know where each platform stores
its root certificates and automatically use them, but it currently doesn't
always. Users who really don't care but still want to use SSL must create
an SSL context with verify_mode set to ssl.CERT_NONE (and live with the
risk, obviously). This stuff passes on OS X only because there's a system
openssl library that always uses the system root certificates.
If anyone can help fixing the ssl.py module (or the _ssl extension) so that
sslcontext.set_default_verify_paths() uses the system root certs on Windows
that would be a huge help. (I have tried this on an Ubuntu box too, and
there it actually works.)
On Fri, Oct 18, 2013 at 3:42 PM, Richard Oudkerk shibt...@gmail.com wrote:
On 18/10/2013 10:37pm, Guido van Rossum wrote:
Good sleuthing! Does the attached patch fix it?
(Off-topic: the code is pretty inconsistent about catching
BaseException. Maybe it shouldn't be caught at all?)
It fixes it in the sense of printing a sensible traceback;-)
$ PYTHONPATH='c:/Repos/tulip' /c/Repos/cpython-33/PCbuild/**python
fetch3.py http://dropbox.com -v
* Connecting to dropbox.com:80 using tcp
* dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179,
199.47.217.179
* New connection ('108.160.165.62', 80, False)
* Connected to ('108.160.165.62', 80)
GET / HTTP/1.1
Host: dropbox.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 18 Oct 2013 22:40:13 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://dropbox.com/
redirect to https://dropbox.com/
* Connecting to dropbox.com:443 using ssl
* dropbox.com resolves to 108.160.165.62, 108.160.166.62, 199.47.216.179,
199.47.217.179
Traceback (most recent call last):
File fetch3.py, line 211, in module
main()
File fetch3.py, line 206, in main
body = loop.run_until_complete(fetch(**sys.argv[1], '-v' in sys.argv))
File c:\Repos\tulip\asyncio\base_**events.py, line 177, in
run_until_complete
return future.result()
File c:\Repos\tulip\asyncio\**futures.py, line 221, in result
raise self._exception
File c:\Repos\tulip\asyncio\tasks.**py, line 257, in _step
result = coro.throw(exc)
File fetch3.py, line 192, in fetch
yield from request.connect(pool)
File fetch3.py, line 80, in connect
ssl=self.ssl)
File fetch3.py, line 36, in open_connection
reader, writer = yield from open_connection(host, port, ssl=ssl)
File c:\Repos\tulip\asyncio\**streams.py, line 41, in open_connection
lambda: protocol, host, port, **kwds)
File c:\Repos\tulip\asyncio\base_**events.py, line 356, in
create_connection
yield from waiter
File c:\Repos\tulip\asyncio\**futures.py, line 318, in __iter__
yield self # This tells Task to wait for completion.
File c:\Repos\tulip\asyncio\tasks.**py, line 308, in _wakeup
value = future.result()
File c:\Repos\tulip\asyncio\**futures.py, line 221, in result
raise self._exception
File c:\Repos\tulip\asyncio\**selector_events.py, line 579, in
_on_handshake
self._sock.do_handshake()
File C:\Repos\cpython-33\lib\ssl.**py, line 520, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
(_ssl.c:553)
--
Richard
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc): new minimalist version
2013/10/18 Charles-François Natali cf.nat...@gmail.com:
I'm happy to see this move forward!
Thanks for your reviews. I had some time in my train travel to improve
the implementation.
I removed the call to pthread_atfork(): tasks have been removed, it
now makes sense to keep tracemalloc enabled in the child process. Call
disable() explicitly in the child process to disable tracemalloc.
I modified the implementation according to your remarks, here is the
updated doc:
http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html
I will update the PEP if you like the new doc.
API
===
Main Functions
--
``clear_traces()`` function:
Clear traces and statistics on Python memory allocations, and reset
the ``get_traced_memory()`` counter.
That's nitpicking, but how about just ``reset()`` (I'm probably biased
by oprofile's opcontrol --reset)?
= done
Well, I already hesitated to rename it to reset() because
clear_traces sounds too specific. The function clears more than
traces.
``get_stats()`` function:
Get statistics on traced Python memory blocks as a dictionary
``{filename (str): {line_number (int): stats}}`` where *stats* in a
``(size: int, count: int)`` tuple, *filename* and *line_number* can
be ``None``.
It's probably obvious, but you might want to say once what *size* and
*count* represent (and the unit for *size*).
= done
``get_tracemalloc_memory()`` function:
Get the memory usage in bytes of the ``tracemalloc`` module as a
tuple: ``(size: int, free: int)``.
* *size*: total size of bytes allocated by the module,
including *free* bytes
* *free*: number of free bytes available to store data
What's *free* exactly? I assume it's linked to the internal storage
area used by tracemalloc itself, but that's not clear at all.
Also, is the tracemalloc overhead included in the above stats (I'm
mainly thinking about get_stats() and get_traced_memory()?
If yes, I find it somewhat confusing: for example, AFAICT, valgrind's
memcheck doesn't report the memory overhead, although it can be quite
large, simply because it's not interesting.
My goal is to able to explain how *every* byte is allocated in Python.
If you enable tracemalloc, your RSS memory will double, or something
like that. You can use get_tracemalloc_memory() to add metrics to a
snapshot. It helps to understand how the RSS memory evolves.
Basically, get_tracemalloc_size() is the memory used to store traces.
It's something internal to the C module (_tracemalloc). This memory is
not traced because it *is* the traces... and so is not counted in
get_traced_memory().
The issue is probably the name (or maybe also the doc): would you
prefer get_python_memory() / get_traces_memory() names, instead of
get_traced_memory() / get_tracemalloc_memory()?
FYI Objects allocated in tracemalloc.py (real objects, not traces) are
not counted in get_traced_memory() because of a filter set up by
default (it was not the case in previous versions of the PEP). You can
remove the filter using tracemalloc.clear_filters() to see this
memory. There are two exceptions: Python objects created for the
result of get_traces() and get_stats() are never traced for
efficiency. It *is* possible to trace these objects, but it's really
too slow. get_traces() and get_stats() may be called outside
tracemalloc.py, so another filter would be needed. Well, it's easier
to never trace these objects. Anyway, they are not interesting to
understand where your application leaks memory.
Trace Functions
---
``get_traceback_limit()`` function:
Get the maximum number of frames stored in the traceback of a trace
of a memory block.
Use the ``set_traceback_limit()`` function to change the limit.
I didn't see anywhere the default value for this setting: it would be
nice to write it somewhere,
= done
and also explain the rationale (memory/CPU
overhead...).
I already explained this partially in set_traceback_limit() doc. I
added something to get_traceback_limit() to explain the default (1 is
enough to get statistics).
(For information, it's possible to set the limit to 0, but it's not
really useful, so it's not mentionned in the doc.)
``get_object_address(obj)`` function:
Get the address of the main memory block of the specified Python object.
A Python object can be composed by multiple memory blocks, the
function only returns the address of the main memory block.
IOW, this should return the same as id() on CPython? If yes, it could
be an interesting note.
= done
I modified the doc to mention id(). id() is only the same if the
object is not tracked by the garbage collector. Otherwise, there is a
difference of sizeof(PyGC_Head) (12 bytes on x86).
``get_object_trace(obj)`` function:
Get the trace of a Python object *obj* as a ``(size: int,
traceback)`` tuple where *traceback* is a tuple of ``(filename: str,
lineno: int)``
Re: [Python-Dev] possibility of shaving a stat call from imports
On 19 Oct 2013 02:56, Brett Cannon br...@python.org wrote: importlib.machinery.FileFinder does a stat call to check if a path is a file if the package check failed. Now I'm willing to bet that the check is rather redundant as the file extension should be a dead give-away that something in a directory is a file and not some non-file type. The import would still fail even if this is the case in the loader when attempting to read from the file, but it happens a little later and it means finders would be more permissive in claiming they found a loader. Does anyone see a good reason not to take the more optimistic route in the finder? As I said, the only thing I see breaking user code is if they have a directory or something named spam.py and so the finder claims it found a module when in fact it didn't and thus stopping the search for the module from continuing. I suspect anything that makes weird import edge case failures harder to debug than they already are won't be worth it. My opinion would likely differ if the speed up on offer was dramatic, but it sounds like that isn't the case. (Also, customised Python builds may have additional extensions that are at higher risk of being used on directory names) Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 (tracemalloc): new minimalist version
On 19 Oct 2013 03:57, Charles-François Natali cf.nat...@gmail.com wrote:
Hi,
I'm happy to see this move forward!
Speaking of which... Charles-François, would you be willing to act as
BDFL-Delegate for this PEP? This will be a very useful new analysis tool,
and between yourself and Victor it looks like you'll be able to come up
with a solid API.
I just suggested that approach to Guido and he also liked the idea :)
Cheers,
Nick.
API
===
Main Functions
--
``clear_traces()`` function:
Clear traces and statistics on Python memory allocations, and reset
the ``get_traced_memory()`` counter.
That's nitpicking, but how about just ``reset()`` (I'm probably biased
by oprofile's opcontrol --reset)?
``get_stats()`` function:
Get statistics on traced Python memory blocks as a dictionary
``{filename (str): {line_number (int): stats}}`` where *stats* in a
``(size: int, count: int)`` tuple, *filename* and *line_number* can
be ``None``.
It's probably obvious, but you might want to say once what *size* and
*count* represent (and the unit for *size*).
``get_tracemalloc_memory()`` function:
Get the memory usage in bytes of the ``tracemalloc`` module as a
tuple: ``(size: int, free: int)``.
* *size*: total size of bytes allocated by the module,
including *free* bytes
* *free*: number of free bytes available to store data
What's *free* exactly? I assume it's linked to the internal storage
area used by tracemalloc itself, but that's not clear at all.
Also, is the tracemalloc overhead included in the above stats (I'm
mainly thinking about get_stats() and get_traced_memory()?
If yes, I find it somewhat confusing: for example, AFAICT, valgrind's
memcheck doesn't report the memory overhead, although it can be quite
large, simply because it's not interesting.
Trace Functions
---
``get_traceback_limit()`` function:
Get the maximum number of frames stored in the traceback of a trace
of a memory block.
Use the ``set_traceback_limit()`` function to change the limit.
I didn't see anywhere the default value for this setting: it would be
nice to write it somewhere, and also explain the rationale (memory/CPU
overhead...).
``get_object_address(obj)`` function:
Get the address of the main memory block of the specified Python
object.
A Python object can be composed by multiple memory blocks, the
function only returns the address of the main memory block.
IOW, this should return the same as id() on CPython? If yes, it could
be an interesting note.
``get_object_trace(obj)`` function:
Get the trace of a Python object *obj* as a ``(size: int,
traceback)`` tuple where *traceback* is a tuple of ``(filename: str,
lineno: int)`` tuples, *filename* and *lineno* can be ``None``.
I find the trace word confusing, so it might be interesting to add a
note somewhere explaining what it is (callstack leading to the object
allocation, or whatever).
Also, this function leaves me a mixed feeling: it's called
get_object_trace(), but you also return the object size - well, a
vague estimate thereof. I wonder if the size really belongs here,
especially if the information returned isn't really accurate: it will
be for an integer, but not for e.g. a list, right? How about just
using sys.getsizeof(), which would give a more accurate result?
``get_trace(address)`` function:
Get the trace of a memory block as a ``(size: int, traceback)``
tuple where *traceback* is a tuple of ``(filename: str, lineno:
int)`` tuples, *filename* and *lineno* can be ``None``.
Return ``None`` if the ``tracemalloc`` module did not trace the
allocation of the memory block.
See also ``get_object_trace()``, ``get_stats()`` and
``get_traces()`` functions.
Do you have example use cases where you want to work with a raw addresses?
Filter
--
``Filter(include: bool, pattern: str, lineno: int=None, traceback:
bool=False)`` class:
Filter to select which memory allocations are traced. Filters can be
used to reduce the memory usage of the ``tracemalloc`` module, which
can be read using the ``get_tracemalloc_memory()`` function.
``match(filename: str, lineno: int)`` method:
Return ``True`` if the filter matchs the filename and line number,
``False`` otherwise.
``match_filename(filename: str)`` method:
Return ``True`` if the filter matchs the filename, ``False``
otherwise.
``match_lineno(lineno: int)`` method:
Return ``True`` if the filter matchs the line number, ``False``
otherwise.
``match_traceback(traceback)`` method:
Return ``True`` if the filter matchs the *traceback*, ``False``
otherwise.
*traceback* is a tuple of ``(filename: str, lineno: int)`` tuples.
Are those ``match`` methods really necessary for the
Re: [Python-Dev] [Python-checkins] cpython: Issue #18416: Fix various os calls in importlib.machinery.FileFinder
On 19 Oct 2013 02:01, brett.cannon python-check...@python.org wrote:
http://hg.python.org/cpython/rev/33844153cd02
changeset: 86438:33844153cd02
user:Brett Cannon br...@python.org
date:Fri Oct 18 12:01:06 2013 -0400
summary:
Issue #18416: Fix various os calls in importlib.machinery.FileFinder
now that self.path is no longer forced to '.'.
Hmm, could this break subclasses in a similar way? It may be safer to keep
the empty string - period conversion, even though PathFinder itself
doesn't rely on it any more.
Cheers,
Nick.
files:
Lib/importlib/_bootstrap.py | 4 +-
Python/importlib.h | 1554 +++---
2 files changed, 780 insertions(+), 778 deletions(-)
diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py
+++ b/Lib/importlib/_bootstrap.py
@@ -1390,7 +1390,7 @@
is_namespace = False
tail_module = fullname.rpartition('.')[2]
try:
-mtime = _os.stat(self.path).st_mtime
+mtime = _os.stat(self.path or _os.getcwd()).st_mtime
except OSError:
mtime = -1
if mtime != self._path_mtime:
@@ -1432,7 +1432,7 @@
Fill the cache of potential modules and packages for this
directory.
path = self.path
try:
-contents = _os.listdir(path)
+contents = _os.listdir(path or _os.getcwd())
except (FileNotFoundError, PermissionError, NotADirectoryError):
# Directory has either been removed, turned into a file, or
made
# unreadable.
diff --git a/Python/importlib.h b/Python/importlib.h
--- a/Python/importlib.h
+++ b/Python/importlib.h
[stripped]
--
Repository URL: http://hg.python.org/cpython
___
Python-checkins mailing list
python-check...@python.org
https://mail.python.org/mailman/listinfo/python-checkins
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #18810: Be optimistic with stat calls when seeing if a directory
On 19 Oct 2013 03:24, brett.cannon python-check...@python.org wrote: http://hg.python.org/cpython/rev/11f2f4af1979 changeset: 86444:11f2f4af1979 user:Brett Cannon br...@python.org date:Fri Oct 18 13:24:13 2013 -0400 summary: Issue #18810: Be optimistic with stat calls when seeing if a directory exists when checking for a package. Before there was an isdir check and then various isfile checks for possible __init__ files when looking for a package. This change drops the isdir check by leaning on the assumption that a directory will not contain something named after the module being imported which is not a directory. If the module is a package then it saves a stat call. If there is nothing in the directory with the potential package name it also saves a stat call. Only if there is something in the directory named the same thing as the potential package will the number of stat calls increase (due to more wasteful __init__ checks). I don't follow this logic. There's now not even an existence check for the base name, so it reads to me like we will look for all the possible init file extensions even if there's *no* directory with an appropriate name. What am I missing? Cheers, Nick. Semantically there is no change as the isdir check moved down so that namespace packages continue to have no chance of accidentally collecting non-existent directories. files: Lib/importlib/_bootstrap.py |19 +- Python/importlib.h | 1537 +++--- 2 files changed, 777 insertions(+), 779 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1406,16 +1406,15 @@ # Check if the module is the name of a directory (and thus a package). if cache_module in cache: base_path = _path_join(self.path, tail_module) -if _path_isdir(base_path): -for suffix, loader in self._loaders: -init_filename = '__init__' + suffix -full_path = _path_join(base_path, init_filename) -if _path_isfile(full_path): -return (loader(fullname, full_path), [base_path]) -else: -# A namespace package, return the path if we don't also -# find a module in the next section. -is_namespace = True +for suffix, loader in self._loaders: +init_filename = '__init__' + suffix +full_path = _path_join(base_path, init_filename) +if _path_isfile(full_path): +return (loader(fullname, full_path), [base_path]) +else: +# If a namespace package, return the path if we don't +# find a module in the next section. +is_namespace = _path_isdir(base_path) # Check for a file w/ a proper suffix exists. for suffix, loader in self._loaders: full_path = _path_join(self.path, tail_module + suffix) diff --git a/Python/importlib.h b/Python/importlib.h --- a/Python/importlib.h +++ b/Python/importlib.h [stripped] -- Repository URL: http://hg.python.org/cpython ___ Python-checkins mailing list python-check...@python.org https://mail.python.org/mailman/listinfo/python-checkins ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore().
First, thanks for the education. What you wrote is extremely edifying about more than just context managers, and I really appreciate the visionary understanding you reported from BrisPy and further elucidated on, regarding the educational pattern of using things before you learn how they work... that applies strongly in arenas other than programming as well: - you learn how to walk before you understand the musculoskeletal physics - you learn how to turn on/off the lights before you understand how electricity works - you learn how to drive before you learn how/why a vehicle works - you learn how to speak before you understand how grammar works - you learn how to locate the constellations before you understand interplanetary gravitational forces - many, many, many, many more things And of course, many people never reach the understanding of how or why for many things they commonly use, do, or observe. That's why some people make things happen, some people watch what happens, and some people wonder What happened? What it doesn't do, though is address the dubious part of the whole construct, which is composition. On 10/17/2013 8:26 AM, Nick Coghlan wrote: And even a two line version: with suppress(FileNotFoundError): os.remove(somefile.tmp) with suppress(FileNotFoundError): os.remove(someotherfile.tmp) The above example, especially if extended beyond two files, begs to used in a loop, like your 5 line version: for name in (somefile.tmp, someotherfile.tmp): with suppress(FileNotFoundError): os.remove(name) which would be fine, of course. But to some with less education about the how and why, it is not clear why it couldn't be written like: with suppress(FileNotFoundError): for name in (somefile.tmp, someotherfile.tmp): os.remove(name) yet to the cognoscenti, it is obvious there are seriously different semantics. In my own code, I have a safe_delete function to bundle the exception handling and the os.remove, and when factored that way, the temptation to nest the loop inside the suppress is gone. With suppress available, though, and if used, the temptation to factor it, either correctly or incorrectly, appears. How many cut-n-paste programmers will get it right and how many will get it wrong, is the serious question here, I think, and while suppress is a slightly better term than ignore, it still hides the implications to the control flow when an exception is actually raised within the block. I'm still dubious that the benefits of this simpler construct, while an interesting composition of powerful underlying constructs, has sufficient benefit to outweigh the naïve user's potential for misusing it (exacerbated by a name that doesn't imply control flow), or even the extra cost in performance per the microbenchmark someone published. Your more complex examples for future versions may have greater merit because they provide a significantly greater reduction in complexity to offset the significantly greater learning curve required to use and understand them. But even those look like an expensive form of goto (of course, goto is considered harmful, and I generally agree with the reasons why, but have coded them in situations where they are more useful than harmful in languages which support them). I imagine that everyone on python-dev is aware that most of the control flow constructs in structured programming (which is a subset of OO) are to control the context of the CPUs instruction pointer without the use of goto. The real problem with goto is not that the instruction pointer is changed non-sequentially, but that arbitrary changes can easily violate poorly documented preconditions of the target location. Hence, structured programming is really an attempt to avoid writing documentation, a laudable goal as the documentation is seldom sufficient at that level of detail... or if sufficient, is repetitive and overwhelming to create, maintain, and comprehend. It achieves that by making control flow constructs that are higher level than goto, that have meanings that can be understood and explained in educational texts, which then are implicit documentation for those control flow aspects of a particular program. OO builds on structured programming to make neat packages of state and control flow, to isolate state into understandable chunks so that larger programs can be comprehended, as the BrisPy presenter enlightened us, without understanding all the details of how each object and function within it works. Programmers raised on OO and GUI toolkits are building more and more systems out of more complex parts, which increases productivity, and that is good, although when they fail to fully understand the parts, some interesting performance characteristics can result. ignore/suppress seems to me to be a sledge hammer solution for driving a tack. The tack may be driven
