Thanks for all the replies, everyone! I'll reply to a few comments
individually. But I first wanted to address the common theme around
zipimport.

First, one idea that nobody mentioned (and came to me after reading the
replies) was to possibly leverage zipimport for freezing the standard
library instead of extending the frozen importer. I strongly feel this
option is preferable to extending the frozen importer with additional
functionality. I suspect the Python core developers would prefer to close
importer feature gaps / bugs with zipimport over the frozen importer. And
since zipimport is actually usable without having to build your own
binaries, improvements to zipimport would more significantly benefit the
larger Python ecosystem. If zipimporter gained the ability to open a zip
archive residing in a memory address or a PyObject implementing the buffer
protocol, [parts of] the standard library could be persisted as a zip file
in libpython and the frozen importer would be limited to bootstrapping the
import subsystem, just like it is today. This avoids adding additional
complexity (like supporting __file__ and __cached__) to the frozen
importer. And it keeps the standard library using a commonly-used importer
in the standard library, just like it is today with PathFinder.

Onto the bigger question that can be summarized as "why not use zipimport:
why do we need something different?" I sympathize with this reasoning.
zipimport exists, it is functional, and it is widely used and has
demonstrated value.

Performance is a major reason to build something better than zipimport.

I response to your replies, I implemented a handful of benchmarks for
oxidized_importer and also implemented a pure Rust implementation of a zip
file importer to collect some hard data. You can reproduce my results by
cloning https://github.com/indygreg/PyOxidizer.git and running `cargo bench
-p pyembed-bench`. At time of writing, the benchmarks materialize the full
standard library on the filesystem, in a zip archive (with no compression),
and in the "Python packed resources" format. It then fires up a Python
interpreter and imports ~450 modules comprising the standard library. I
encourage you to obtain your own numbers and look at the benchmark code to
better understand testing methodology. But here are some results from Linux
on my Ryzen 5950x.

* zipimporter is slower than PathFinder to import the entirety of the
standard library when the disk cache is hot. 201.81ms for zipimporter vs
174.06ms for PathFinder.
* My pure Rust zip importer is faster than zipimporter and PathFinder for
the same operation. 161.67ms when reading zip data from memory; 164.45ms
when using buffered filesystem I/O (8kb read operations).
* OxidizedFinder + Python packed resources are the fastest of all. 121.07ms
loading from memory.
* Parsing/indexing the container formats is fast in Rust. Python packed
resources parses in 107.69us and indexes in 200.52us (0.2ms). A zip archive
table of contents is parsed in 809.61us and indexes in 1.205ms. If that
same zip archive is read/seeked using filesystem I/O, the numbers go up to
4.6768ms and 5.1591ms.
* Starting and finalizing a Python interpreter takes 11.930ms with
PathFinder and 4.8887ms with OxidizedFinder.

I won't post the full set of numbers for Windows, but they are generally
higher, especially if filesystem I/O is involved. PathFinder is still
faster than zipimporter, however. And zipimporter's relative slowness
compared to OxidizedFinder is more pronounced.

There are many interesting takeaways from these numbers. But here are what
I think are the most important:

* The Rust implementation of a zip importer trouncing performance of
zipimporter probably means zipimporter could be made a lot faster (I didn't
profile to measure why zipimporter is so slow. But I suspect its
performance is hindered by being implemented in Python.)
* OxidizedFinder + Python packed resources are still significantly faster
than the next fastest solution (Rust implemented zip importer).
* The overhead of reading and parsing the container format can matter.
PyOxidizer built binaries can start and finalize a Python interpreter in
<5ms (this ignores new process overhead). ~1.2ms for the Rust zip importer
to index the zip file is a significant percentage!

Succinctly, today zipimporter is somewhat slow when you aren't I/O
constrained. The existence proof of a faster Rust implementation implies it
could be made significantly faster. Is that "good enough" to forego
standard library inclusion of a yet more efficient solution? That's a
healthy debate to have. You know which side I'm on :) But it would probably
be prudent to optimize zipimporter before investing in something more
esoteric.

Onto the individual replies.

On Fri, Sep 3, 2021 at 12:42 AM Paul Moore <p.f.mo...@gmail.com> wrote:

> My quick reaction was somewhat different - it would be a great idea, but
> it’s entirely possible to implement this outside the stdlib as a 3rd party
> module. So the fact that no-one has yet done so means there’s less general
> interest than the OP is suggesting.
>

Let me slightly push back on the "less general interest" assertion. While
oxidized_importer is an existence proof that this is possible today, its
upside today is limited because there is still a heavy dependence on a
Python install being present and in a usable and well-defined state. This
is difficult to achieve in practice and is why many distributed Python
applications include their own Python distribution: it's the only way to be
sure.

Even if you bundle your own unmodified Python distribution, the upside of
something like oxidized_importer by itself is limited because you have to
accommodate the modules in the standard library that are imported during
interpreter initialization. Today, in order to import the entirety of the
standard library from something other than .py files, you need to rely on
zipimporter or a custom built binary that injects a meta path importer
during interpreter startup. The latter is what PyOxidizer built executables
do.

I think the current limitations preventing 3rd party meta path finders from
being used exclusively constrain the upside of these tools. If we get to a
point where a subset of the stdlib is "frozen" into the binary and
PathFinder isn't used at all during startup before your __main__ code runs,
then I think we'll finally be at a place where alternative 3rd party
finders are viable and start seeing wider adoption. A potential feature
request here would be a way to inject a sys.meta_path or sys.path_hooks
entry during interpreter initialization, before any non-builtin extension
modules are imported. If you could do this via environment variables,
command line arguments, shebang tricks, or likewise, that opens up a lot of
possibilities for enabling 3rd party meta path importers.

Something else to factor in here is that many people don't realize things
like oxidized_importer are even possible! The importing mechanism is
complex and implementing a conformant meta path importer is hard. But I do
believe there is a latent market need here. I suspect if I spent the time
to polish oxidized_importer a bit and actually spent effort to "market" it,
it would probably see adoption in some of the larger Python projects out
there where the performance/simplicity benefits would matter to end-users.
But, that's all speculation: I understand there's a bar that needs to be
cleared to justify complexity. I have more work to do here.

On Fri, Sep 3, 2021 at 4:29 AM Paul Moore <p.f.mo...@gmail.com> wrote:

> But would the downside of it not being possible to manage the format
> with existing standard tools outweigh that?
>

This is a fair call out. I agree that the ubiquity of zip files is a major
selling point. There would likely be a high hurdle to clear to justify
introducing a non-standard format versus reusing something like zip files.

On Fri, Sep 3, 2021 at 12:37 PM Eric Snow <ericsnowcurren...@gmail.com>
wrote:

> At the (relative) extreme is to throw out the existing frozen module
> approach (or even the "unmarshal + exec" approach of source-based
> modules) and replace it with something more efficient and/or more
> compatible (and cross-platform).  From what I understood, this is the
> main focus of this thread.
>

Just to be clear, oxidized_importer + Python packed resources still retain
the "unmarshal + exec" solution: it's the file container format that's
different. (From my benchmarking and proof of existence in
Facebook/Instagram land, we know that there are more efficient solutions
for "unmarshal + exec" and I'm excited to see people poking around here!)


> > a) backwards incompatible changes to the C API to support additional
> metadata on frozen modules (or at the very least a supplementary API that
> fragments what a "frozen" module is).
>
> What part of the C-API, specifically?
>

The interpreter configuration and initialization APIs. If you extend the
frozen struct to capture more metadata, that's an API break.


> >  You end up slowly reimplementing the importing mechanism in C (remember
> Python 2?) or disappoint users.
>
> I'm not sure I follow.  What part of the import system would be
> reimplemented in C?  The frozen importer is written in pure Python
> with a few small helpers written in C.  I expect that nearly all
> necessary changes would happen in Lib/importlib/_bootstrap.py and not
> Python/import.c.
>

I think I overspoke here, not realizing how much of the import machinery is
in fact implemented in Python. (I even thought aspects of the zip importer
were still implemented in C.)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ASIY2SSDIBXWVQ6OHWZ3VCV7URB3Z44G/
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to