Re: [Python-Dev] Impact of Namedtuple on startup time

On 07/17/2017 07:25 PM, Nathaniel Smith wrote: I think the intention is to allow users with a certain kind of security requirement to opt in to a restricted version of the language that doesn't support exec. This is difficult if the stdlib is calling exec all over the place. But nobody is

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, 17 Jul 2017 at 19:26 Nathaniel Smith wrote: > On Jul 17, 2017 5:28 PM, "Steven D'Aprano" wrote: > > On Mon, Jul 17, 2017 at 09:31:20PM +, Brett Cannon wrote: > > > As for removing exec() as a goal, I'll back up Christian's point and the > > one

Re: [Python-Dev] Impact of Namedtuple on startup time

On 07/18/2017 08:12 AM, Guido van Rossum wrote: There are some weighty things being said in this subthread that shouldn't be hidden under the heading of improving NamedTuple. For continued discussion of our development philosophy let's open a new thread. (I have an opinion but I expect I'm not

Re: [Python-Dev] Impact of Namedtuple on startup time

There are some weighty things being said in this subthread that shouldn't be hidden under the heading of improving NamedTuple. For continued discussion of our development philosophy let's open a new thread. (I have an opinion but I expect I'm not the only one with that opinion, so I'll let others

Re: [Python-Dev] Impact of Namedtuple on startup time

On Tue, 18 Jul 2017 22:13:01 +1000 Nick Coghlan wrote: > > As another example of this: while trading the global import lock for > per-module locks eliminated most of the old import deadlocks, it turns > out that it *also* left us with some fairly messy race conditions and >

Re: [Python-Dev] Impact of Namedtuple on startup time

On 18 July 2017 at 05:42, Raymond Hettinger wrote: > One minor grumble: I think we need to give careful cost/benefit > considerations to optimizations that complicate the implementation. Over the > last several years, the source for Python has grown increasingly

Re: [Python-Dev] Impact of Namedtuple on startup time

> > I think that the right solution of this issue is generalizing the import > machinery and allowing it to cache not just files, but arbitrary chunks of > code. We already use precompiled bytecode files for exactly same goal -- > speed up the startup by avoiding compilation. This solution could

Re: [Python-Dev] Impact of Namedtuple on startup time

17.07.17 15:43, Antoine Pitrou пише: Cost of creating a namedtuple has been identified as a contributor to Python startup time. Not only Python core and the stdlib, but any third-party library creating namedtuple classes (there are many of them). An issue was created for this:

Re: [Python-Dev] Impact of Namedtuple on startup time

On 07/17/2017 04:45 PM, Guido van Rossum wrote: Raymond agreed to reopen the issue. Everyone who's eager to redesign namedtuple, please go to python-ideas. Python Ideas thread started. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] Impact of Namedtuple on startup time

On Jul 17, 2017 5:28 PM, "Steven D'Aprano" wrote: On Mon, Jul 17, 2017 at 09:31:20PM +, Brett Cannon wrote: > As for removing exec() as a goal, I'll back up Christian's point and the > one Steve made at the language summit that removing the use of exec() from > the

Re: [Python-Dev] Impact of Namedtuple on startup time

On Tue, Jul 18, 2017 at 01:17:24AM +0200, Giampaolo Rodola' wrote: > The extra memory overhead is a price I would be happy to pay considering > that collections.namedtuple is considerably slower than a plain tuple. > Other than the additional overhead on startup / import time, instantiation > is

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 8:16 PM, Barry Warsaw wrote: > .. > > class Foo(NamedTuple, fields = 'x,y,z'): > > ... > > > > Then the name is explicit and you get to add methods > > etc. if you want. > > Yes, I like how that reads. > > I would prefer class

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 09:31:20PM +, Brett Cannon wrote: > As for removing exec() as a goal, I'll back up Christian's point and the > one Steve made at the language summit that removing the use of exec() from > the critical path in Python is a laudable goal from a security perspective. I'm

Re: [Python-Dev] Impact of Namedtuple on startup time

On Jul 17, 2017, at 18:27, Greg Ewing wrote: > > Barry Warsaw wrote: >> namedtuple is great and clever, but it’s also a bit clunky. It has a weird >> signature and requires a made up type name. > > Maybe a metaclass could be used to make something > like this

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 6:27 PM, Greg Ewing wrote: > > > Maybe a metaclass could be used to make something > like this possible: > > >class Foo(NamedTuple, fields = 'x,y,z'): > ... > > If you think of it, collection.namedtuple *is* a metaclass. A simple

Re: [Python-Dev] Impact of Namedtuple on startup time

Raymond agreed to reopen the issue. Everyone who's eager to redesign namedtuple, please go to python-ideas. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 11:24 PM, Tim Peters wrote: > [Giampaolo Rodola' ] > > > > To be entirely honest, I'm not even sure why they need to be forcefully > > declared upfront in the first place, instead of just having a first-class > > function

Re: [Python-Dev] Impact of Namedtuple on startup time

On 07/17/2017 03:27 PM, Greg Ewing wrote: Barry Warsaw wrote: namedtuple is great and clever, but it’s also a bit clunky. It has a weird signature and requires a made up type name. Maybe a metaclass could be used to make something like this possible: class Foo(NamedTuple, fields =

Re: [Python-Dev] Impact of Namedtuple on startup time

On 07/17/2017 02:31 PM, Brett Cannon wrote: I vaguely remember some years ago someone proposing a patch that used metaclasses to avoid using exec() (I think it was to benefit PyPy or one of the JIT-backed interpreters). Would that work to remove the need for exec() while keeping the code in

Re: [Python-Dev] Impact of Namedtuple on startup time

On 07/17/2017 02:26 PM, Barry Warsaw wrote: namedtuple is great and clever, but it’s also a bit clunky. It has a weird > signature and requires a made up type name. It’s also rather unPythonic if > you want to support default arguments when creating namedtuple instances. > Maybe as you say,

Re: [Python-Dev] Impact of Namedtuple on startup time

Barry Warsaw wrote: namedtuple is great and clever, but it’s also a bit clunky. It has a weird signature and requires a made up type name. Maybe a metaclass could be used to make something like this possible: class Foo(NamedTuple, fields = 'x,y,z'): ... Then the name is explicit

Re: [Python-Dev] Impact of Namedtuple on startup time

Just for sake of completeness since people are talking about a namedtuple overhaul, I have a couple implementations here - https://github.com/jsbueno/extradict/blob/master/extradict/extratuple.py If any idea there can help inspiring someone, I will be happy. js -><- On 17 July 2017 at

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, 17 Jul 2017 at 13:28 Raymond Hettinger wrote: > > > On Jul 17, 2017, at 8:49 AM, Guido van Rossum wrote: > > > > One of the reasons to be wary of exec()/eval() other than the usual > security concerns is that in some Python implementations

Re: [Python-Dev] Impact of Namedtuple on startup time

[Giampaolo Rodola' ] > > To be entirely honest, I'm not even sure why they need to be forcefully > declared upfront in the first place, instead of just having a first-class > function (builtin?) written in C: > > >>> ntuple(x=1, y=0) > (x=1, y=0) > > ...or even a literal

Re: [Python-Dev] Impact of Namedtuple on startup time

namedtuple is great and clever, but it’s also a bit clunky. It has a weird signature and requires a made up type name. It’s also rather unPythonic if you want to support default arguments when creating namedtuple instances. Maybe as you say, a lot of the typical use cases for namedtuples

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 11:07 PM, Petr Viktorin wrote: > On 07/17/2017 10:31 PM, Giampaolo Rodola' wrote: > >> I completely agree. I love namedtuples but I've never been too happy >> about the additional overhead vs. plain tuples (both for creation and >> attribute access

Re: [Python-Dev] Impact of Namedtuple on startup time

On 07/17/2017 10:31 PM, Giampaolo Rodola' wrote: I completely agree. I love namedtuples but I've never been too happy about the additional overhead vs. plain tuples (both for creation and attribute access times), to the point that I explicitly avoid to use them in certain circumstances (e.g. a

Re: [Python-Dev] Impact of Namedtuple on startup time

On 2017-07-17 21:46, MRAB wrote: On 2017-07-17 21:31, Giampaolo Rodola' wrote: I completely agree. I love namedtuples but I've never been too happy about the additional overhead vs. plain tuples (both for creation and attribute access times), to the point that I explicitly avoid to use them

Re: [Python-Dev] Impact of Namedtuple on startup time

On 2017-07-17 21:31, Giampaolo Rodola' wrote: I completely agree. I love namedtuples but I've never been too happy about the additional overhead vs. plain tuples (both for creation and attribute access times), to the point that I explicitly avoid to use them in certain circumstances (e.g. a

Re: [Python-Dev] Impact of Namedtuple on startup time

My apologies, I misunderstood what had been proposed (and rejected). So it sounds like the _source is a pre-requisite for the current exec-based implementation, but the proposal is to replace with a non-exec-based implementation, meaning _source would no longer be needed for the module to work

Re: [Python-Dev] Impact of Namedtuple on startup time

I completely agree. I love namedtuples but I've never been too happy about the additional overhead vs. plain tuples (both for creation and attribute access times), to the point that I explicitly avoid to use them in certain circumstances (e.g. a busy loop) and only for public end-user APIs

Re: [Python-Dev] Impact of Namedtuple on startup time

> On Jul 17, 2017, at 8:49 AM, Guido van Rossum wrote: > > One of the reasons to be wary of exec()/eval() other than the usual security > concerns is that in some Python implementations they have a high overhead to > initialize the parser and compiler. (Even in CPython it's

Re: [Python-Dev] Impact of Namedtuple on startup time

> On Jul 17, 2017, at 8:49 AM, Guido van Rossum wrote: > > The approach of generating source code and exec()ing it, is a cool > demonstration of Python's expressive power, but it's always been my sense > that whenever we encounter a popular idiom that uses exec() and

Re: [Python-Dev] Impact of Namedtuple on startup time

2017-07-17 9:45 GMT-07:00 Steven D'Aprano : > On Mon, Jul 17, 2017 at 02:43:19PM +0200, Antoine Pitrou wrote: > > > > Hello, > > > > Cost of creating a namedtuple has been identified as a contributor to > > Python startup time. Not only Python core and the stdlib, but any >

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 02:43:19PM +0200, Antoine Pitrou wrote: > > Hello, > > Cost of creating a namedtuple has been identified as a contributor to > Python startup time. Not only Python core and the stdlib, but any > third-party library creating namedtuple classes (there are many of > them).

Re: [Python-Dev] Impact of Namedtuple on startup time

2017-07-17 18:13 GMT+02:00 Gregory P. Smith : > I get that namedtuple ._source is a public API. We may need to keep it. If > so, that just means revisiting lazily generating it as a property - > issue19640. I agree. Technically speaking, optimizing namedtuple doesn't have to mean

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 8:00 AM Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > > > On Jul 17, 2017, at 6:31 AM, Antoine Pitrou wrote: > > > >> I think I understand well enough to say something intelligent… > >> > >> While actual references to _source are likely

Re: [Python-Dev] Impact of Namedtuple on startup time

I am firmly with Antoine here. The cumulative startup time of large Python programs is a serious problem and namedtuple is one of the major contributors -- especially because it is so convenient that it is ubiquitous. The approach of generating source code and exec()ing it, is a cool demonstration

Re: [Python-Dev] Impact of Namedtuple on startup time

Makes sense. Thanks. S Steve Holden On Mon, Jul 17, 2017 at 4:29 PM, Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > > > On Jul 17, 2017, at 8:22 AM, Steve Holden wrote: > > > > My only question is "what's a variable called _source doing in the > public API?" >

Re: [Python-Dev] Impact of Namedtuple on startup time

> On Jul 17, 2017, at 8:22 AM, Steve Holden wrote: > > My only question is "what's a variable called _source doing in the public > API?" The convention for named tuple hnas been for all the methods and attributes to be prefixed with an underscore so that the names won't

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 3:59 PM, Raymond Hettinger < raymond.hettin...@gmail.com> wrote: > I really don't want to throw away these benefits to save a couple of > milliseconds. As Nick Coghlan recently posted, "Speed isn't everything, > and it certainly isn't adequate justification for breaking

Re: [Python-Dev] Impact of Namedtuple on startup time

On 2017-07-17 14:43, Antoine Pitrou wrote: > So my take is: > > 1) Usage of "_source" in open source code (as per the search above) > seems non-existent. > > 2) If the primary intent of "_source" is to show-case how to write a > tuple subclass, well, why not write a recipe or tutorial somewhere?

Re: [Python-Dev] Impact of Namedtuple on startup time

2017-07-17 16:56 GMT+02:00 Facundo Batista : > My experience inside Canonical is that golang stole a lot of "codebase > share" from Python, and (others and mine) talks hit two walls, mainly: > one is memory consumption, and the other is startup time. > > So yes, startup

Re: [Python-Dev] Impact of Namedtuple on startup time

On Jul 17, 2017, at 10:59, Raymond Hettinger wrote: > > ISTM this issue is being pressed by micro-optimizers who are being very > aggressive and not responding to actual user needs (it is more an invented > issue than a real one). Named tuple has been around for

Re: [Python-Dev] Impact of Namedtuple on startup time

> On Jul 17, 2017, at 6:31 AM, Antoine Pitrou wrote: > >> I think I understand well enough to say something intelligent… >> >> While actual references to _source are likely rare (certainly I’ve never >> used it), my understanding is that the way namedtuple works is to >>

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, Jul 17, 2017 at 9:43 AM, Antoine Pitrou wrote: > As for 2), startup time is actually a very important consideration > nowadays, both for small scripts *and* for interactive use with the > now very wide-spread use of Jupyter Notebooks. A 1 ms. cost when > importing a

Re: [Python-Dev] Impact of Namedtuple on startup time

Le 17/07/2017 à 15:26, Isaac Morland a écrit : > > I think I understand well enough to say something intelligent… > > While actual references to _source are likely rare (certainly I’ve never > used it), my understanding is that the way namedtuple works is to > construct _source, and then exec

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, 17 Jul 2017 15:03:26 +0200 Ivan Levkivskyi wrote: > Interesting coincidence, just two days ago I have heard that a team at one > large company completely abandoned namedtuple because of the creation time > problem. > > Concerning _source, why it is not possible to

Re: [Python-Dev] Impact of Namedtuple on startup time

On 17 July 2017 at 08:43, Antoine Pitrou wrote: > > Hello, > > Cost of creating a namedtuple has been identified as a contributor to > Python startup time. Not only Python core and the stdlib, but any > third-party library creating namedtuple classes (there are many of >

Re: [Python-Dev] Impact of Namedtuple on startup time

Interesting coincidence, just two days ago I have heard that a team at one large company completely abandoned namedtuple because of the creation time problem. Concerning _source, why it is not possible to make it a property so that all the string formatting will happen on request, thus saving

Re: [Python-Dev] Impact of Namedtuple on startup time

On Mon, 17 Jul 2017 14:43:19 +0200 Antoine Pitrou wrote: > Hello, > > Cost of creating a namedtuple has been identified as a contributor to > Python startup time. Imprecise wording: that's the cost of creating a namedtuple *class*, i.e. anytime someone writes `MyClass =

[Python-Dev] Impact of Namedtuple on startup time

Hello, Cost of creating a namedtuple has been identified as a contributor to Python startup time. Not only Python core and the stdlib, but any third-party library creating namedtuple classes (there are many of them). An issue was created for this: https://bugs.python.org/issue28638 Raymond