Re: [Python-Dev] Python startup time

2017-07-20 Thread Nick Coghlan
On 21 July 2017 at 15:30, Cesare Di Mauro wrote: > > > 2017-07-21 4:52 GMT+02:00 Nick Coghlan : > >> On 21 July 2017 at 12:44, Nick Coghlan wrote: >> > We can separately measure the cost of unmarshalling the code object: >> > >> > $ python3 -m perf timeit -s "import typing; from marshal import l

Re: [Python-Dev] Python startup time

2017-07-20 Thread Cesare Di Mauro
2017-07-21 4:52 GMT+02:00 Nick Coghlan : > On 21 July 2017 at 12:44, Nick Coghlan wrote: > > We can separately measure the cost of unmarshalling the code object: > > > > $ python3 -m perf timeit -s "import typing; from marshal import loads; > from > > importlib.util import cache_from_source; cach

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Chris Jerdonek
On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan wrote: > ... > * Lazy loading can have a significant impact on startup time, as it > means you don't have to pay for the cost of finding and loading > modules that you don't actually end up using on that particular run > > We've historically resisted a

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Chris Angelico
On Fri, Jul 21, 2017 at 2:09 PM, Nick Coghlan wrote: > On 21 July 2017 at 13:55, Chris Angelico wrote: >> On Fri, Jul 21, 2017 at 1:49 PM, Nick Coghlan wrote: >>> The are some genuine downsides in increasing the complexity of >>> bootstrapping CPython when all you're starting with is a VCS clone

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Nick Coghlan
On 21 July 2017 at 13:55, Chris Angelico wrote: > On Fri, Jul 21, 2017 at 1:49 PM, Nick Coghlan wrote: >> The are some genuine downsides in increasing the complexity of >> bootstrapping CPython when all you're starting with is a VCS clone and >> a C compiler, but those complications are ultimatel

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Chris Angelico
On Fri, Jul 21, 2017 at 1:49 PM, Nick Coghlan wrote: > The are some genuine downsides in increasing the complexity of > bootstrapping CPython when all you're starting with is a VCS clone and > a C compiler, but those complications are ultimately no worse than > those we already have with Argument

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Nick Coghlan
On 21 July 2017 at 10:19, Nathaniel Smith wrote: > I'm not sure either of these make much sense when python startup is already > in the single digit milliseconds. While it's certainly great if we can lower > that further, my impression is that for any real application, startup time > is overwhelmi

Re: [Python-Dev] Python startup time

2017-07-20 Thread Nick Coghlan
On 21 July 2017 at 12:44, Nick Coghlan wrote: > We can separately measure the cost of unmarshalling the code object: > > $ python3 -m perf timeit -s "import typing; from marshal import loads; from > importlib.util import cache_from_source; cache = > cache_from_source(typing.__file__); data = open(

Re: [Python-Dev] Python startup time

2017-07-20 Thread Nick Coghlan
On 21 July 2017 at 05:38, Cesare Di Mauro wrote: > > > 2017-07-20 19:23 GMT+02:00 Victor Stinner : > >> 2017-07-20 19:09 GMT+02:00 Cesare Di Mauro : >> > I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib. >> That's something that also influences the startup time (compiling so

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Gregory P. Smith
On Thu, Jul 20, 2017 at 10:56 AM Jim J. Jewett wrote: > I agree that startup time is a problem, but I wonder if some of the pain > could be mitigated by using a persistent process. > This is one strategy that works under some situations, but not all. There are downsides to daemons: * They only

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Nathaniel Smith
On Jul 20, 2017 14:18, "Eric Snow" wrote: On Thu, Jul 20, 2017 at 11:53 AM, Jim J. Jewett wrote: > I agree that startup time is a problem, but I wonder if some of the pain > could be mitigated by using a persistent process. > > [snip] > > Is it too hard to create a daemon server? > Is the commun

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Eric Snow
On Thu, Jul 20, 2017 at 11:53 AM, Jim J. Jewett wrote: > I agree that startup time is a problem, but I wonder if some of the pain > could be mitigated by using a persistent process. > > [snip] > > Is it too hard to create a daemon server? > Is the communication and context switch slower than a new

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Paul Moore
On 20 July 2017 at 18:53, Jim J. Jewett wrote: > Is it too hard to create a daemon server? > Is the communication and context switch slower than a new startup? > Is the pattern just not well-enough advertised? Managing a daemon (including things like stopping it when it's been idle for "too long"

Re: [Python-Dev] Python startup time

2017-07-20 Thread Cesare Di Mauro
2017-07-20 19:23 GMT+02:00 Victor Stinner : > 2017-07-20 19:09 GMT+02:00 Cesare Di Mauro : > > I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib. > That's something that also influences the startup time (compiling source vs > loading pre-compiled modules). > > My benchmark was

Re: [Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Oleg Broytman
On Thu, Jul 20, 2017 at 01:53:52PM -0400, "Jim J. Jewett" wrote: > I agree that startup time is a problem, but I wonder if some of the pain > could be mitigated by using a persistent process. > > For example, in > https://mail.python.org/pipermail/python-dev/2017-July/148664.html Ben Hoyt > ment

[Python-Dev] startup time repeated? why not daemon

2017-07-20 Thread Jim J. Jewett
I agree that startup time is a problem, but I wonder if some of the pain could be mitigated by using a persistent process. For example, in https://mail.python.org/pipermail/python-dev/2017-July/148664.html Ben Hoyt mentions that the Google Cloud SDK (CLI) team has found it "especially problematic

Re: [Python-Dev] Python startup time

2017-07-20 Thread Victor Stinner
2017-07-20 19:09 GMT+02:00 Cesare Di Mauro : > I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib. > That's something that also influences the startup time (compiling source vs > loading pre-compiled modules). My benchmark was "python3 -m perf command -- python3 -c pass": I d

Re: [Python-Dev] Python startup time

2017-07-20 Thread Cesare Di Mauro
2017-07-19 16:26 GMT+02:00 Victor Stinner : > 2017-07-19 15:22 GMT+02:00 Oleg Broytman : > > On Wed, Jul 19, 2017 at 02:59:52PM +0200, Victor Stinner < > victor.stin...@gmail.com> wrote: > >> "Python is very slow to start on Windows 7" > >> https://stackoverflow.com/questions/29997274/python-is- >

Re: [Python-Dev] Python startup time

2017-07-20 Thread Nick Coghlan
On 20 July 2017 at 23:32, Stefan Behnel wrote: > So, before considering to write an accelerator module in C that replaces > some existing Python module, and thus duplicating its entire source code > with highly increased complexity, I'd like to remind you that simply > compiling the Python module

Re: [Python-Dev] Python startup time

2017-07-20 Thread Stefan Behnel
Ivan Levkivskyi schrieb am 20.07.2017 um 13:24: > I agree the start-up time is important. There is something that is related. > ABCMeta is currently implemented in Python. > This makes it slow, creation of an ABC is 2x slower than creation of a > normal class. > However, ABCs are used by many mediu

Re: [Python-Dev] Python startup time

2017-07-20 Thread Antoine Pitrou
On Thu, 20 Jul 2017 21:29:18 +0900 INADA Naoki wrote: > > WeakSet should be cared specially. > Maybe, ABCMeta can be optimized first. > > Currently, ABCMeta use three WeakSets. But it can be delayed until > `register` or > `issubclass` is called. > So even if WeakSet is implemented in Python, I

Re: [Python-Dev] Python startup time

2017-07-20 Thread INADA Naoki
Hi, Ivan. First of all, Yes, please do it! On Thu, Jul 20, 2017 at 8:24 PM, Ivan Levkivskyi wrote: > I agree the start-up time is important. There is something that is related. > ABCMeta is currently implemented in Python. > This makes it slow, creation of an ABC is 2x slower than creation of a

Re: [Python-Dev] Python startup time

2017-07-20 Thread Ivan Levkivskyi
I agree the start-up time is important. There is something that is related. ABCMeta is currently implemented in Python. This makes it slow, creation of an ABC is 2x slower than creation of a normal class. However, ABCs are used by many medium and large size projects. Also, both abc and _collections

Re: [Python-Dev] Python startup time

2017-07-20 Thread Victor Stinner
Hi, I applied the patch above to count the number of times that Python is run. Running the Python test suite with "./python -m test -j0 -rW" runs Python 2,256 times. Honestly, I expected more. I'm running tests with Python compiled in debug mode. And in debug mode, Python startup time is much wor