On 21 July 2017 at 05:38, Cesare Di Mauro <cesare.di.ma...@gmail.com> wrote:
>
>
> 2017-07-20 19:23 GMT+02:00 Victor Stinner <victor.stin...@gmail.com>:
>
>> 2017-07-20 19:09 GMT+02:00 Cesare Di Mauro <cesare.di.ma...@gmail.com>:
>> > I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib.
>> That's something that also influences the startup time (compiling source vs
>> loading pre-compiled modules).
>>
>> My benchmark was "python3 -m perf command -- python3 -c pass": I don't
>> explicitly remove .pyc files, I expect that Python uses prebuilt .pyc
>> files from __pycache__.
>>
>> Victor
>>
>
> OK, that should be the best case.
>
> An idea to improve the situation might be to find an alternative structure
> for .pyc/pyo files, which allows to (partially) "parallelize" their loading
> (not execution, of course), or at least speed-up the process. Maybe a GSoC
> project for some student, if no core dev has time to investigate it.
>
Unmarshalling the code object from disk generally isn't the slow part -
it's the module level execution that takes time.
Using the typing module as an example, a full reload cycle takes almost 10
milliseconds:
$ python3 -m perf timeit -s "import typing; from importlib import reload"
"reload(typing)"
.....................
Mean +- std dev: 9.89 ms +- 0.46 ms
(Don't try timing "import typing" directly - the sys.modules cache
amortises the cost down to being measured in nanoseconds, since you're
effectively just measuring the speed of a dict lookup)
We can separately measure the cost of unmarshalling the code object:
$ python3 -m perf timeit -s "import typing; from marshal import loads; from
importlib.util import cache_from_source; cache =
cache_from_source(typing.__file__); data = open(cache, 'rb').read()[12:]"
"loads(data)"
.....................
Mean +- std dev: 286 us +- 4 us
Finding the module spec:
$ python3 -m perf timeit -s "from importlib.util import find_spec"
"find_spec('typing')"
.....................
Mean +- std dev: 69.2 us +- 2.3 us
And actually running the module's code (this includes unmarshalling the
code object, but *not* calculating the import spec):
$ python3 -m perf timeit -s "import typing; loader_exec =
typing.__spec__.loader.exec_module" "loader_exec(typing)"
.....................
Mean +- std dev: 9.68 ms +- 0.43 ms
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
