Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc

2014-04-15 Thread Charles-François Natali
Indeed, that's very reasonable. Please open an issue on the tracker! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Chris Angelico
On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon bcan...@gmail.com wrote: In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: Sure, but not everyone uses Boost or has long

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Nick Coghlan
On 14 Apr 2014 18:37, Glenn Linderman v+pyt...@g.nevcal.com wrote: On 4/14/2014 2:51 PM, Brett Cannon wrote: Freezing everything except encodings.__init__, os, and _sysconfigdata, I suppose these are omitted because they can vary in different environments? But isn't Python built for a

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread M.-A. Lemburg
On 15.04.2014 09:45, Chris Angelico wrote: On Tue, Apr 15, 2014 at 8:21 AM, Brett Cannon bcan...@gmail.com wrote: In my work environment (Python 2.7.2, all the heavy lifting done in C++), startup costs are dominated by dynamic linking of all our C++ libraries and their Boost wrappers: Sure,

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Eric Snow
On Mon, Apr 14, 2014 at 3:51 PM, Brett Cannon bcan...@gmail.com wrote: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Eric Snow
On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote: Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; migrating hg to Py3 will (if I understand the above figures correctly) widen that gap, so any

Re: [Python-Dev] Python 2migr8

2014-04-15 Thread Nick Coghlan
On 14 Apr 2014 17:17, Donald Stufft don...@stufft.io wrote: Now I will admit I personally have probably had a harder time than some others because of the nature of the things I was trying to work on, and lately it’s gotten better (although I think that’s partially because I’m more known now, and

Re: [Python-Dev] Appeal for reviews

2014-04-15 Thread Nick Coghlan
On 14 Apr 2014 08:42, R. David Murray rdmur...@bitdance.com wrote: On Mon, 14 Apr 2014 08:18:13 -0400, Nick Coghlan ncogh...@gmail.com wrote: On 14 Apr 2014 01:56, Stephen J. Turnbull step...@xemacs.org wrote: mar...@v.loewis.de writes: For gaining commit access, it's really more

Re: [Python-Dev] Appeal for reviews

2014-04-15 Thread Guido van Rossum
On Tue, Apr 15, 2014 at 10:41 AM, Nick Coghlan ncogh...@gmail.com wrote: On 14 Apr 2014 08:42, R. David Murray rdmur...@bitdance.com wrote: When considering who we give commit access to, I think we would be well served to start giving more weight to the quality of the code reviews that

Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc

2014-04-15 Thread Victor Stinner
Hi, 2014-04-14 1:39 GMT-04:00 Nathaniel Smith n...@pobox.com: The new tracemalloc infrastructure in python 3.4 is super-interesting to numerical folks, because we really like memory profiling. Cool, thanks :-) calloc() is more awesome than malloc()+memset() (...) I had a discussion with

Re: [Python-Dev] [numpy wishlist] PyMem_*Calloc

2014-04-15 Thread Nathaniel Smith
On Tue, Apr 15, 2014 at 9:31 AM, Charles-François Natali cf.nat...@gmail.com wrote: Indeed, that's very reasonable. Please open an issue on the tracker! Done! http://bugs.python.org/issue21233 I'll ping numpy-discussion and see if I can convince someone to do the work ;-). -n -- Nathaniel

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Daniel Holth
IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? On Tue, Apr 15, 2014 at 10:34 AM, Eric Snow ericsnowcurren...@gmail.com wrote: On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico ros...@gmail.com wrote: Specific

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Brett Cannon
On Tue, Apr 15, 2014 at 11:19 AM, Daniel Holth dho...@gmail.com wrote: IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? It's definitely minimized since Python 3.3 and the caching of stat results at the

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Chris Angelico
On Wed, Apr 16, 2014 at 1:34 AM, Skip Montanaro s...@pobox.com wrote: I find it hard to believe that freezing the stdlib is going to lower the barrier enough for the Mercurial folks, if, in fact, import slowness is their main reason for not moving to 3.x. I've no idea whether that's the case

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/15/2014 11:34 AM, Skip Montanaro wrote: I find it hard to believe that freezing the stdlib is going to lower the barrier enough for the Mercurial folks, if, in fact, import slowness is their main reason for not moving to 3.x. My

[Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Skip Montanaro
Eric wrote: Perhaps not so much a very real advantage as less of a distraction. It's still significantly slower than 2.7. :) I'm still confused. I peeked in /usr/bin/hg. The only system modules it imports directly are os and system (maybe I'm using an ancient version?). After that, it

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Skip Montanaro
On Tue, Apr 15, 2014 at 10:40 AM, Chris Angelico ros...@gmail.com wrote: I've no idea whether that's the case or not. All I know is, every time I need to work with a Mercurial repo it feels a lot slower than doing similar work on a similar size git repo [1], so any improvement (or reduction of

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Skip Montanaro
On Tue, Apr 15, 2014 at 10:42 AM, Daniel Holth dho...@gmail.com wrote: I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. Instant for me is the blink of an eye, which Wikipedia reports as typically between 100ms and 400ms.

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Chris Angelico
On Wed, Apr 16, 2014 at 1:56 AM, Skip Montanaro s...@pobox.com wrote: On Tue, Apr 15, 2014 at 10:42 AM, Daniel Holth dho...@gmail.com wrote: I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. Instant for me is the blink of

Re: [Python-Dev] Appeal for reviews

2014-04-15 Thread Stephen J. Turnbull
On 14 Apr 2014 08:42, R. David Murray rdmur...@bitdance.com wrote: Or to put it another way, I'd like to encourage contributors who want to get commit access to focus just as much on doing good reviews as they do on writing new patches. Currently the focus is all on getting patches

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Daniel Holth
I find Python's startup time to be very sluggish. I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. On my machine 'python -c ' takes about 0.05 seconds but 'python3 -c ' takes 0.125 seconds. I will be very happy to see any

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Antoine Pitrou
Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually rely on large repositories and/or non-trivial operations, so startup time is

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Antoine Pitrou
Le 15/04/2014 17:42, Daniel Holth a écrit : I find Python's startup time to be very sluggish. I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for instant. On my machine 'python -c ' takes about 0.05 seconds but 'python3 -c ' takes 0.125

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Antoine Pitrou
Le 14/04/2014 23:51, Brett Cannon a écrit : It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Chris Angelico
On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, git will usually win on performance, and hg is using Py2; Keep in mind those shoot-outs usually

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Antoine Pitrou
Le 15/04/2014 17:34, Skip Montanaro a écrit : This suggests to me that Mercurial's import slowness is mostly in its own code (I counted 104 Python modules and six shared objects in its mercurial package, which isn't going to be affected (much?) by freezing the Python standard modules. Skip is

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Stephen J. Turnbull
Skip Montanaro writes: Instant for me is the blink of an eye, which Wikipedia reports as typically between 100ms and 400ms. http://en.wikipedia.org/wiki/Blink If you blink, you've missed Python 2.7 startup on a relatively modern machine. This is what I see for Mac OS X Mavericks on a

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Daniel Holth
In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help option or any initial user feedback from Mercurial or git is a big driver in perceived speed as opposed to how long the entire operation might take. But for me any

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Antoine Pitrou
Le 15/04/2014 19:09, Daniel Holth a écrit : In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help option or any initial user feedback from Mercurial or git is a big driver in perceived speed as opposed to how long the

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Paul Moore
On 15 April 2014 18:09, Daniel Holth dho...@gmail.com wrote: For me Python's startup time (warm) takes about 1/4 of the hg startup time in the worst case. I expect to both notice and appreciate any speedups and encourage all optimizers to optimize. My experience is essentially irrelevant (as a

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Daniel Holth
On Tue, Apr 15, 2014 at 1:29 PM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 19:09, Daniel Holth a écrit : In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help option or any initial user feedback from

Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

2014-04-15 Thread Mark Lawrence
On 15/04/2014 18:32, Daniel Holth wrote: On Tue, Apr 15, 2014 at 1:29 PM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 19:09, Daniel Holth a écrit : In case you were wondering, I'm using Ubuntu's 2.7.5+ and 3.3.2+. My feeling has long been that the speed of getting at the --help

Re: [Python-Dev] Appeal for reviews

2014-04-15 Thread Terry Reedy
On 4/15/2014 12:15 PM, Stephen J. Turnbull wrote: I've always really liked MvL's 5-reviews-to-get-1 approach. The only thing I don't like about it[3] is that it puts an explicit price on core developer time (my time is worth 5x as much as yours). Not really true since any of the 5 could be

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Guido van Rossum
Can we please stop the argument about Hg vs. Git? On Tue, Apr 15, 2014 at 12:54 PM, Chris Angelico ros...@gmail.com wrote: On Wed, Apr 16, 2014 at 2:40 AM, Antoine Pitrou solip...@pitrou.net wrote: Le 15/04/2014 09:45, Chris Angelico a écrit : Specific use-case that I can see:

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Chris Angelico
On Wed, Apr 16, 2014 at 4:54 AM, Guido van Rossum gu...@python.org wrote: Can we please stop the argument about Hg vs. Git? My apologies. All I was saying was that hg is a use case where startup performance really does matter, as opposed to the ones presented earlier in the thread where a

Re: [Python-Dev] Appeal for reviews

2014-04-15 Thread Stephen J. Turnbull
Terry Reedy writes: On 4/15/2014 12:15 PM, Stephen J. Turnbull wrote: The only thing I don't like about it[3] is that it puts an explicit price on core developer time (my time is worth 5x as much as yours). Not really true But that is *not* your call! It's for the would-be

[Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-15 Thread Brett Cannon
To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL: 10 _Py_ReadyTypes(): 930 more types: 44

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-15 Thread Guido van Rossum
Are you going to post your code (or a link to it)? On Tue, Apr 15, 2014 at 2:26 PM, Brett Cannon bcan...@gmail.com wrote: To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT:

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-15 Thread Brett Cannon
On Tue, Apr 15, 2014 at 5:40 PM, Guido van Rossum gu...@python.org wrote: Are you going to post your code (or a link to it)? I basically wrote a function that uses gettimeofday() and just calculates the delta between the calls. Greg Smith verified that I wasn't doing anything stupid. =)

Re: [Python-Dev] Python 2migr8

2014-04-15 Thread Steve Dower
Apologies for the slow reply, I was travelling back from PyCon. From Guido: - I'd prefer a name that plays on 2 and 3, not 2 and 8. :-) I didn't really expect the name to stick, but Nick and I had such a good laugh that it would have been unfair not to share it :-) (though I laughed even more

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-15 Thread Terry Reedy
On 4/15/2014 5:26 PM, Brett Cannon wrote: To finish my timing work I decided to see where Py_InitializeEx_Private() spends its time. The following is a breakdown measured in microseconds running using -E: INIT: setlocale: 11 envvar: 2 random init: 2 interp creation: 15 thread creation: 6 GIL:

Re: [Python-Dev] Python 2migr8

2014-04-15 Thread Barry Warsaw
On Apr 15, 2014, at 10:55 PM, Steve Dower wrote: Really, I'd expect the name to be pretty boring - Python Straddle, Python 2 Strict or Python Migration Edition (if anyone can live with Python ME ;-) ) We often call code that works in both Python 2 and 3 from a single source bi-lingual. Maybe

Re: [Python-Dev] Python 2migr8

2014-04-15 Thread Nick Coghlan
On 15 Apr 2014 18:56, Steve Dower steve.do...@microsoft.com wrote: From Eric: This should be doable with an import hook. For that matter it would be pretty straight-forward to provide a script around the Python binary that installs the import hook. All this could be bundled up and distributed

Re: [Python-Dev] Timing breakdown of Py_InitializeEx_Private()

2014-04-15 Thread Guido van Rossum
Well, that's the part that does import site. Anything that speeds up the code in Lib/site.py might help. :-) On Tue, Apr 15, 2014 at 5:23 PM, Terry Reedy tjre...@udel.edu wrote: On 4/15/2014 5:26 PM, Brett Cannon wrote: To finish my timing work I decided to see where

Re: [Python-Dev] this is what happens if you freeze all the modules required for startup

2014-04-15 Thread Stefan Behnel
Brett Cannon, 14.04.2014 23:51: It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of