[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Jelle Zijlstra added the comment: I'm also concerned that the slowness of namedtuple creation is causing people to avoid using it. I can see why we wouldn't want a complicated solution like using Argument Clinic, but it's not clear to me why Serhiy's approach in namedtuple-no-compile.patch

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Raymond Hettinger added the comment: Sorry INADA but I think this is all a foolish and inconsequential optimization that complicates the code in a way that isn't worth it (saving only a 1/2 millisecond in a single import. Also, we don't want the argument clinic code to start invading the pure

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: I think external cache system introduces more complexity and startup overhead than AC. I think functools is the only "very common" module using namedtuple, because `functools.wraps()` is used to create decorator functions. But if general solution for all

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Serhiy Storchaka added the comment: Argument Clinic is used just for running the generating code and inlining the result. This is the simplest part of Argument Clinic and using it looks an overhead. Argument Clinic has other disadvantages: * In any case you need a rule in Makefile, otherwise

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: > That leads into my main comment on the AC patch: the files that are > explicitly listed as triggering a new clinic run should be factored out into > a named variable and that list commented accordingly. done. -- Added file:

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Nick Coghlan added the comment: The concern with using the "generate a private module that can be cached" approach is that it doesn't generalise well - any time you want to micro-optimise a new module that way, you have to add a custom Makefile rule. By contrast, Argument Clinic is a general

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Serhiy Storchaka added the comment: Argument Clinic is not needed, since we can use Makefile. -- Added file: http://bugs.python.org/file45724/functools-CacheInfo-Makefile.patch ___ Python tracker

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: (reopen the issue to discuss about using Argument Clinic) -- resolution: rejected -> status: closed -> open ___ Python tracker

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Nick Coghlan added the comment: Ah, I had forgotten that Larry had already included Python support in Argument Clinic. With the inline code auto-generated from the pure Python implementation, that addresses the main maintenance concerns I had. I did briefly wonder about the difficulties of

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: Updated patch: fixed small issue in argument clinic, and added comment why we use code generation. -- Added file: http://bugs.python.org/file45580/namedtuple-clinic2.diff ___ Python tracker

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: > If someone comes forward with more fully formed idea for code generation or > overall structural enchancement, that can be put in another tracker item. I noticed argument clinic supports Python [1]. So there is one way to code generation already. Attached

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Raymond Hettinger added the comment: Thanks Nick. I'm going to mark this as closed, as the proposal to microscopic to warrant incurring technical debt. If someone comes forward with more fully formed idea for code generation or overall structural enchancement, that can be put in another

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Nick Coghlan added the comment: I'll echo Raymond's concerns here, as we simply don't have the collective maintenance capacity to sustain a plethora of special case micro-optimisations aimed at avoiding importing common standard library modules. I will note however, that there has been

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Raymond Hettinger added the comment: > half a millisecond is reduced. I would like to caution against any significant changes to save microscopic amounts of time. Twisting the code into knots for minor time savings is rarely worth it and it not what Python is all about. > Half milliseconds

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Eric V. Smith added the comment: This file is derived from my namedlist project on PyPI. I've stripped out the namedlist stuff, and just left namedtuple. I've also removed the Python 2 support, and I've removed support for default parameters. After that surgery, I have not tested it very

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Serhiy Storchaka added the comment: One of problems with this patch is that it make instantiating a namedtuple much slower (due to parsing arguments by Python code). This can be solved by using eval() for creating only the __new__ method (see commented out line "result.__new__ = eval(...)").

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: (tip) $ ~/local/py37/bin/python3 -m perf timeit -s 'import importlib, functools' -- 'importlib.reload(functools)' . Median +- std dev: 1.21 ms +- 0.01 ms (namedtuple-no-compile.patch) $ ~/local/py37/bin/python3 -m perf timeit -s 'import

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Serhiy Storchaka added the comment: Here is a sample patch that make namedtuple() not using dynamic compilation. It has rough the same performance effect as inlining the named tuple source, but affects all named tuples. -- Added file:

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: > What is the main culprit, importing the collections module or compiling a > named tuple? In this time, later. But collections module takes 1+ ms to import too. I'll try to optimize it. > Using namedtuple is not new in 3.6, thus this is not a regression that

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Serhiy Storchaka added the comment: Using namedtuple is not new in 3.6, thus this is not a regression that can be fixed at beta stage. Inlining the source of a named tuple class looks ugly solution. It would be better to write the source in separate file and import it. Makefile can have a

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Serhiy Storchaka added the comment: What is the main culprit, importing the collections module or compiling a named tuple? -- nosy: +serhiy.storchaka ___ Python tracker

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: > The lazy import system could benefit many libs so the result could be > impressive. But here only functools is enhanced, half a millisecond is > reduced. On the other hand, implementing lazy import makes application complex. This patch only enhance functools,

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Xiang Zhang added the comment: > Yes. But first import time is also important for CLI applications. That's why mercurial and Bazaar has lazy import system. The lazy import system could benefit many libs so the result could be impressive. But here only functools is enhanced, half a millisecond

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: > The slow import is the case only the first time functools is imported. Later > imports will just use the cache (sys.modules). Yes. But first import time is also important for CLI applications. That's why mercurial and Bazaar has lazy import system. Since many

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Changes by Xiang Zhang : -- nosy: +ncoghlan, rhettinger ___ Python tracker ___ ___

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Xiang Zhang added the comment: I doubt this deserves a change. The slow import is the case only the first time functools is imported. Later imports will just use the cache (sys.modules). And if this is gonna change, maybe we don't have to copy the entire namedtuple structure? --

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

INADA Naoki added the comment: I feel this patch is safe enough to be landed in 3.6. -- keywords: +patch Added file: http://bugs.python.org/file45386/28638-functools-no-namedtuple.patch ___ Python tracker

[issue28638] Creating namedtuple is too slow to be used in common stdlib (e.g. functools)

Changes by INADA Naoki : -- components: +Library (Lib) title: Creating namedtuple is too slow -> Creating namedtuple is too slow to be used in common stdlib (e.g. functools) versions: +Python 3.6, Python 3.7 ___ Python