On Tue, Sep 18, 2018 at 2:57 PM, Carl Shapiro <carl.shap...@gmail.com> wrote:
> On Tue, Sep 18, 2018 at 5:55 AM, Fabio Zadrozny <fabi...@gmail.com> wrote: > >> During the import process, Python can already deal with folders and .zip >> files in sys.path... now, instead of having special handling for a new >> concept with a custom command line, etc, why not just say that this is a >> special file (e.g.: files with a .pyfrozen extension) and make importlib be >> able to deal with it when it's on sys.path (that way there could be >> multiple of those and there should be no need to turn it on/off, custom >> command line, etc)? >> > > That is an interesting idea but it might not be easy to work into this > design. The improvement in start-up time comes from eliminating the > overheads of filesystem I/O, memory allocation, and un-marshaling > bytecode. Having this data on the filesystem would reintroduce the cost of > filesystem I/O and it would add a load-time relocation to the equation so > the overall performance benefits would be greatly lessened. > > >> Another question: doesn't importlib already provide hooks for external >> contributors which could address that use case? (so, this could initially >> be available as a third party library for maturing outside of CPython and >> then when it's deemed to be mature it could be integrated into CPython -- >> not that this can't happen on Python 3.8 timeframe, but it'd be useful >> checking its use against the current Python version and measuring benefits >> with real world code). >> > > This may be possible but, for the same reasons I outline above, it would > certainly come at the expense of performance. > > I think many people are interested in a better .pyc format but our goals > are much more modest. We are actually trying to not introduce a whole new > way to externalize .py data in CPython. Rather, we think of this as just > making the existing frozen module capability much faster so its use can be > broadened to making start-up performance better. The user visible part, > the command line interface to bypass the frozen module, would be a > nice-to-have for developers but is something we could live without. > Just to make sure we're in the same page, the approach I'm talking about would still be having a dll, not a better .pyc format, so, during the import a custom importer would open that dll once and provide modules from it -- do you think this would be much more overhead than what's proposed now? I guess it may be a bit slower because it'd have to obey the existing import capabilities, but that shouldn't mean more time is spent on IO, memory allocation nor un-marshaling bytecode (although it may be that I misunderstood the approach or the current import capabilities don't provide the proper api for that).
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
