[Python-Dev] Re: A better way to freeze modules

[Gregory Szorc]

Thanks for all the replies, everyone! I'll reply to a few comments
individually. But I first wanted to address the common theme around
zipimport.

First, one idea that nobody mentioned (and came to me after reading the
replies) was to possibly leverage zipimport for freezing the standard
library instead of extending the frozen importer. I strongly feel this
option is preferable to extending the frozen importer with additional
functionality. I suspect the Python core developers would prefer to close
importer feature gaps / bugs with zipimport over the frozen importer. And
since zipimport is actually usable without having to build your own
binaries, improvements to zipimport would more significantly benefit the
larger Python ecosystem. If zipimporter gained the ability to open a zip
archive residing in a memory address or a PyObject implementing the buffer
protocol, [parts of] the standard library could be persisted as a zip file
in libpython and the frozen importer would be limited to bootstrapping the
import subsystem, just like it is today. This avoids adding additional
complexity (like supporting __file__ and __cached__) to the frozen
importer. And it keeps the standard library using a commonly-used importer
in the standard library, just like it is today with PathFinder.

Onto the bigger question that can be summarized as "why not use zipimport:
why do we need something different?" I sympathize with this reasoning.
zipimport exists, it is functional, and it is widely used and has
demonstrated value.

Performance is a major reason to build something better than zipimport.

I response to your replies, I implemented a handful of benchmarks for
oxidized_importer and also implemented a pure Rust implementation of a zip
file importer to collect some hard data. You can reproduce my results by
cloning https://github.com/indygreg/PyOxidizer.git and running `cargo bench
-p pyembed-bench`. At time of writing, the benchmarks materialize the full
standard library on the filesystem, in a zip archive (with no compression),
and in the "Python packed resources" format. It then fires up a Python
interpreter and imports ~450 modules comprising the standard library. I
encourage you to obtain your own numbers and look at the benchmark code to
better understand testing methodology. But here are some results from Linux
on my Ryzen 5950x.

* zipimporter is slower than PathFinder to import the entirety of the
standard library when the disk cache is hot. 201.81ms for zipimporter vs
174.06ms for PathFinder.
* My pure Rust zip importer is faster than zipimporter and PathFinder for
the same operation. 161.67ms when reading zip data from memory; 164.45ms
when using buffered filesystem I/O (8kb read operations).
* OxidizedFinder + Python packed resources are the fastest of all. 121.07ms
loading from memory.
* Parsing/indexing the container formats is fast in Rust. Python packed
resources parses in 107.69us and indexes in 200.52us (0.2ms). A zip archive
table of contents is parsed in 809.61us and indexes in 1.205ms. If that
same zip archive is read/seeked using filesystem I/O, the numbers go up to
4.6768ms and 5.1591ms.
* Starting and finalizing a Python interpreter takes 11.930ms with
PathFinder and 4.8887ms with OxidizedFinder.

I won't post the full set of numbers for Windows, but they are generally
higher, especially if filesystem I/O is involved. PathFinder is still
faster than zipimporter, however. And zipimporter's relative slowness
compared to OxidizedFinder is more pronounced.

There are many interesting takeaways from these numbers. But here are what
I think are the most important:

* The Rust implementation of a zip importer trouncing performance of
zipimporter probably means zipimporter could be made a lot faster (I didn't
profile to measure why zipimporter is so slow. But I suspect its
performance is hindered by being implemented in Python.)
* OxidizedFinder + Python packed resources are still significantly faster
than the next fastest solution (Rust implemented zip importer).
* The overhead of reading and parsing the container format can matter.
PyOxidizer built binaries can start and finalize a Python interpreter in
<5ms (this ignores new process overhead). ~1.2ms for the Rust zip importer
to index the zip file is a significant percentage!

Succinctly, today zipimporter is somewhat slow when you aren't I/O
constrained. The existence proof of a faster Rust implementation implies it
could be made significantly faster. Is that "good enough" to forego
standard library inclusion of a yet more efficient solution? That's a
healthy debate to have. You know which side I'm on :) But it would probably
be prudent to optimize zipimporter before investing in something more
esoteric.

Onto the individual replies.

On Fri, Sep 3, 2021 at 12:42 AM Paul Moore  wrote:

> My quick reaction was somewhat different - it would be a great idea, but
> it’s entirely possible to implement this outside the stdlib as a 3rd party
> module. So the 

[Python-Dev] A better way to freeze modules

[Gregory Szorc]

Over in https://bugs.python.org/issue45020 there is some exciting work
around expanding the use of the frozen importer to speed up Python
interpreter startup. I wholeheartedly support the effort and don't want to
discourage progress in this area.

Simultaneously, I've been down this path before with PyOxidizer and feel
like I have some insight to share.

I don't think I'll be offending anyone by saying the existing CPython
frozen importer is quite primitive in terms of functionality: it does the
minimum it needs to do to support importing module bytecode embedded in the
interpreter binary [for purposes of bootstrapping the Python-based
importlib modules]. The C struct representing frozen modules is literally
just the module name and a pointer to a sized buffer containing bytecode.

In issue45020 there is talk of enhancing the functionality of the frozen
importer to support its potential broader use. For example, setting
__file__ or exposing .__loader__.get_source(). I support the overall
initiative.

However, introducing enhanced functionality of the frozen importer will at
the C level require either:

a) backwards incompatible changes to the C API to support additional
metadata on frozen modules (or at the very least a supplementary API that
fragments what a "frozen" module is).
b) CPython only hacks to support additional functionality for "freezing"
the standard library for purposes of speeding up startup.

I'm not a CPython core developer, but neither "a" nor "b" seem ideal to me.
"a" is backwards incompatible. "b" seems like a stop-gap solution until a
more generic version is available outside the CPython standard library.

>From my experience with PyOxidizer and software in general, here is what I
think is going to happen:

1. CPython enhances the frozen importer to be usable in more situations.
2. Python programmers realize this solution has performance and
ease-of-distribution wins and want to use it more.
3. Limitations in the frozen importer are found. Bugs are reported. Feature
requests are made.
4. The frozen importer keeps getting incrementally extended or Python
developers grow frustrated that its enhancements are only available to the
standard library. You end up slowly reimplementing the importing mechanism
in C (remember Python 2?) or disappoint users.

Rather than extending the frozen importer, I would suggest considering an
alternative solution that is far more useful to the long-term success of
Python: I would consider building a fully-featured, generic importer that
is capable of importing modules and resource data from a well-defined and
portable serialization format / data structure that isn't defined by C
structs and APIs.

Instead of defining module bytecode (and possible additional minimal
metadata) in C structs in a frozen modules array (or an equivalent C API),
what if we instead defined a serialization format for representing the
contents of loadable Python data (module source, module bytecode, resource
files, extension module library data, etc)? We could then point the Python
interpreter at instances of this data structure (in memory or in files) so
it could import/load the resources within using a meta path importer.

What if this serialization format were designed so that it was extremely
efficient to parse and imports could be serviced with the same trivially
minimal overhead that the frozen importer currently has? We could embed
these data structures in produced binaries and achieve the same desirable
results we'll be getting in issue45020 all while delivering a more generic
solution.

What if this serialization format were portable across machines? The entire
Python ecosystem could leverage it as a container format for distributing
Python resources. Rather than splatting dozens or hundreds of files on the
filesystem, you could write a single file with all of a package's
resources. Bugs around filesystem implementation details such as case
(in)sensitivity and Unicode normalization go away. Package installs are
quicker. Run-time performance is better due to faster imports.

(OK, maybe that last point brings back bad memories of eggs and you
instinctively reject the idea. Or you have concerns about development
ergonomics when module source code isn't in standalone editable files.
These are fair points!)

What if the Python interpreter gains an "app mode" where it is capable of
being paired with a single "resources file" and running the application
within? Think running zip applications today, but a bit faster, more
tailored to Python, and more fully featured.

What if an efficient binary serialization format could be leveraged as a
cache to speed up subsequent interpreter startups?

These were all considerations on my mind in the early days of PyOxidizer
when I realized that the frozen importer and zip importers were lacking the
features I desired and I would need to find an alternative solution.

One thing led to another and I have incrementally developed the "Python
packed 

[Python-Dev] Re: macOS issues with 3.8.7rc1

[Gregory Szorc]

On Wed, Dec 9, 2020 at 10:10 AM Gregory P. Smith  wrote:

>
>
> As a meta question: Is there a good reason to support binaries running on
> macOS earlier than ~ $latest_version-1?
>
> Aren't systems running those old releases rather than upgrading
> unsupported by Apple, never to be patched, and thus not wise to even have
> on a network?
>
> Yes, that means some very old hardware becomes useless as Apple drops
> support. But that is what people signed up for when they bought it. Why
> should that be our problem?
>
> (It sounds like y'all will make it work, that's great! I'm really just
> wondering where the motivation comes from)
>

FWIW The patches are necessary to target 10.15 when building on 11.0. So
the value of the patches is very real to target audiences today, especially
since many developers have already adopted 11.0 out of necessity.

While I think there are merits to having a discussion around macOS version
support policy, may I ask that it take place in a different thread so it
doesn't derail the more immediate conversation centering around the 3.8.7
release?


>
> On Wed, Dec 9, 2020, 9:25 AM Gregory Szorc 
> wrote:
>
>> On Wed, Dec 9, 2020 at 4:13 AM Ronald Oussoren 
>> wrote:
>>
>>>
>>>
>>> On 8 Dec 2020, at 19:59, Gregory Szorc  wrote:
>>>
>>> Regarding the 3.8.7rc1 release, I wanted to raise some issues regarding
>>> macOS.
>>>
>>> Without the changes from https://github.com/python/cpython/pull/22855
>>> backported, attempting to build a portable binary on macOS 11 (e.g. by
>>> setting `MACOSX_DEPLOYMENT_TARGET=10.9`) results in a myriad of `warning:
>>> 'XXX' is only available on macOS 10.13 or newer
>>> [-Wunguarded-availability-new]` warnings during the build. This warning
>>> could be innocuous if there is run-time probing in place (the symbols in
>>> question are weakly linked, which is good). But if I'm reading the code
>>> correctly, run-time probing was introduced by commits like eee543722 and
>>> isn't present in 3.8.7rc1.
>>>
>>> I don't have a machine with older macOS sitting around to test, but I'm
>>> fairly certain the lack of these patches means binaries built on macOS 11
>>> will blow up at run-time when run on older macOS versions.
>>>
>>> These same patches also taught CPython to build and run properly on
>>> Apple ARM hardware. I suspect some people will care about these being
>>> backported to 3.8.
>>>
>>> We know. Backporting the relevant changes to 3.8 is taking more time
>>> than I had hoped. It doesn’t help that I’ve been busy at work and don’t
>>> have as much energy during the weekend as I’d like.
>>>
>>> The backport to 3.9 was fairly easy because there were few changes
>>> between master and the 3.9 branch at the time. Sadly there have been
>>> conflicting changes since 3.8 was forked (in particular in posixmodule.c).
>>>
>>> The current best practice for building binaries that work on macOS 10.9
>>> is to build on that release (or rather, with that SDK).  That doesn’t help
>>> if you want to build Universal 2 binaries though.
>>>
>>
>> Thank you for your hard work devising the patches and working to backport
>> them.
>>
>> I personally care a lot about these patches and I have the technical
>> competency to perform the backport. If you need help, I could potentially
>> find time to hack on it. Just email me privately (or ping @indygreg on
>> GitHub) and let me know. Even if they don't get into 3.8.7, I'll likely
>> cherry pick the patches for
>> https://github.com/indygreg/python-build-standalone. And I'm sure other
>> downstream packagers will want them as well. So having them in an
>> unreleased 3.8 branch is better than not having them at all.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/5AWFX2POTPNVW72VUPCPTJIOA6AOSVWY/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BVZ6YZVMHRICX5WLATCBJLSL3XA55QKQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: macOS issues with 3.8.7rc1

[Gregory Szorc]

On Wed, Dec 9, 2020 at 4:13 AM Ronald Oussoren 
wrote:

>
>
> On 8 Dec 2020, at 19:59, Gregory Szorc  wrote:
>
> Regarding the 3.8.7rc1 release, I wanted to raise some issues regarding
> macOS.
>
> Without the changes from https://github.com/python/cpython/pull/22855
> backported, attempting to build a portable binary on macOS 11 (e.g. by
> setting `MACOSX_DEPLOYMENT_TARGET=10.9`) results in a myriad of `warning:
> 'XXX' is only available on macOS 10.13 or newer
> [-Wunguarded-availability-new]` warnings during the build. This warning
> could be innocuous if there is run-time probing in place (the symbols in
> question are weakly linked, which is good). But if I'm reading the code
> correctly, run-time probing was introduced by commits like eee543722 and
> isn't present in 3.8.7rc1.
>
> I don't have a machine with older macOS sitting around to test, but I'm
> fairly certain the lack of these patches means binaries built on macOS 11
> will blow up at run-time when run on older macOS versions.
>
> These same patches also taught CPython to build and run properly on Apple
> ARM hardware. I suspect some people will care about these being backported
> to 3.8.
>
> We know. Backporting the relevant changes to 3.8 is taking more time than
> I had hoped. It doesn’t help that I’ve been busy at work and don’t have as
> much energy during the weekend as I’d like.
>
> The backport to 3.9 was fairly easy because there were few changes between
> master and the 3.9 branch at the time. Sadly there have been conflicting
> changes since 3.8 was forked (in particular in posixmodule.c).
>
> The current best practice for building binaries that work on macOS 10.9 is
> to build on that release (or rather, with that SDK).  That doesn’t help if
> you want to build Universal 2 binaries though.
>

Thank you for your hard work devising the patches and working to backport
them.

I personally care a lot about these patches and I have the technical
competency to perform the backport. If you need help, I could potentially
find time to hack on it. Just email me privately (or ping @indygreg on
GitHub) and let me know. Even if they don't get into 3.8.7, I'll likely
cherry pick the patches for
https://github.com/indygreg/python-build-standalone. And I'm sure other
downstream packagers will want them as well. So having them in an
unreleased 3.8 branch is better than not having them at all.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5AWFX2POTPNVW72VUPCPTJIOA6AOSVWY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] macOS issues with 3.8.7rc1

[Gregory Szorc]

Regarding the 3.8.7rc1 release, I wanted to raise some issues regarding
macOS.

Without the changes from https://github.com/python/cpython/pull/22855
backported, attempting to build a portable binary on macOS 11 (e.g. by
setting `MACOSX_DEPLOYMENT_TARGET=10.9`) results in a myriad of `warning:
'XXX' is only available on macOS 10.13 or newer
[-Wunguarded-availability-new]` warnings during the build. This warning
could be innocuous if there is run-time probing in place (the symbols in
question are weakly linked, which is good). But if I'm reading the code
correctly, run-time probing was introduced by commits like eee543722 and
isn't present in 3.8.7rc1.

I don't have a machine with older macOS sitting around to test, but I'm
fairly certain the lack of these patches means binaries built on macOS 11
will blow up at run-time when run on older macOS versions.

These same patches also taught CPython to build and run properly on Apple
ARM hardware. I suspect some people will care about these being backported
to 3.8.

I suspect people in the Python application packaging/distribution space
will be significantly affected by this (I know I am with PyOxidizer). Is it
worth making the backport of these patches a 3.8.7 release blocker or a
trigger for a special 3.8.8 release shortly thereafter?

Gregory
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WX262O2BT2OCTOO5AGBP6JGUODLRV55D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Building Standalone Python Applications with PyOxidizer

[Gregory Szorc]

Hey Python developers,

I just published the initial release of PyOxidizer - a utility for
producing self-contained, potentially single file executable Python
applications. You can read more about it at
https://gregoryszorc.com/blog/2019/06/24/building-standalone-python-applications-with-pyoxidizer/

I wanted to draw this list's attention to it because I think PyOxidizer
could have significant implications for the Python community. And the 2nd
to final section in the blog post is of particular relevance to CPython's
development team.

I want to state explicitly that I would prefer to work with the Python
community in whatever appropriate capacity is needed to advance the state
of Python application distribution. (I have already participated in
discussions about PEP 587.) PyOxidizer is still young and there's a long
way to go. But I would love to e.g. implement PyOxidizer with an eye
towards incorporation in official Python packaging tools a few years down
the road. I'm not too familiar with the various day-to-day happenings of
Python and Python Packaging. But if you steer me in the right direction, I
could start getting involved...

I hope you like PyOxidizer!

Gregory Szorc
gregory.sz...@gmail.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DF5Q2GIXHXQGU2FZRR3M6QZDV74X6DFL/

Re: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version

[Gregory Szorc]

On 5/20/2019 4:09 AM, Victor Stinner wrote:
> Hi Gregory,
> 
> IMHO your remarks are not directly related to the PEP 587 and can be
> addressed in parallel.
> 
>> It sounds like PyOxidizer and Hermetic Python are on the same page and
>> we're working towards a more official solution. But I want to make sure
>> by explicitly stating what PyOxidizer is doing.
>>
>> Essentially, to facilitate in-memory import, we need to register a
>> custom sys.meta_path importer *before* any file-based imports are
>> attempted. (...)
>> I /think/ providing a 2-phase
>> initialization that stops between _Py_InitializeCore() and
>> _Py_InitializeMainInterpreter() would get the job done for PyOxidizer
>> today. (...)
> 
> Extract of PEP 587: "This extracts a subset of the API design from the
> PEP 432 development and refactoring work that is now considered
> sufficiently stable to make public (allowing 3rd party embedding
> applications access to the same configuration APIs that the native
> CPython CLI is now using)."
> 
> We know that my PEP 587 is incomplete, but the work will continue in
> Python 3.9 to support your use case.
> 
> The PEP 587 introduces an experimental separation between "core" and
> "main" initialization phases. PyConfig._init_main=0 stops at the
> "core" phase, then you are free to run C and Python,
> _Py_InitializeMain() finishes the Python initialization ("main"
> phase).
> 
> 
>>> In the "Isolate Python" section, I suggest to set the "isolated"
>>> parameter to 1 which imply setting user_site_directory to 0. So
>>> sys.path isn't modified afterwards. What you pass to PyConfig is what
>>> you get in sys.path in this case.
>>
>> Regarding site.py, I agree it is problematic for embedding scenarios.
>> Some features of site.py can be useful. Others aren't. It would be
>> useful to have more granular control over which bits of site.run are
>> run. My naive suggestion would be to add individual flags to control
>> which functions site.py:main() runs. That way embedders can cherry-pick
>> site.py features without having to manually import the module and call
>> functions within. That feels much more robust for long-term maintainability.
> 
> I agree that more work can be done on the site module. IMHO core
> features which are needed by everybody should be done before calling
> site. Maybe using a frozen "presite" module or whatever. I would be
> interested to make possible to use Python for most cases without the
> site module.
> 
> 
>> Regarding Python calling exit(), this is problematic for embedding
>> scenarios.
> 
> I am working on that. I fixed dozens of functions. For example,
> Py_RunMain() should not longer exit if there is an uncaught SystemExit
> when calling PyErr_Print(). SystemExit is now handled separately
> before calling PyErr_Print(). The work is not done, but it should be
> way better than Python 3.6 and 3.7 state.
> 
> 
>> This thread called attention to exit() during interpreter
>> initialization. But it is also a problem elsewhere. For example,
>> PyErr_PrintEx() will call Py_Exit() if the exception is a SystemExit.
>> There's definitely room to improve the exception handling mechanism to
>> give embedders better control when SystemExit is raised. As it stands,
>> we need to check for SystemExit manually and reimplement
>> _Py_HandleSystemExit() to emulate its behavior for e.g. exception value
>> handling (fun fact: you can pass non-None, non-integer values to
>> sys.exit/SystemExit).
> 
> I don't know well these functions, maybe new functions are needed. It
> can be done without/outside the PEP 587.
> 
> 
>> Having a more efficient member lookup for BuiltinImporter and
>> FrozenImporter might shave off a millisecond or two from startup. This
>> would require some kind of derived data structure. (...)
> 
> I don't think that the structures/API to define frozen/builtin modules
> has to change. We can convert these lists into an hash table during
> the initialization of the importlib module.
> 
> I'm not saying that the current API is perfect, just that IMHO it can
> be solved without the API.
> 
> 
>> Unfortunately, as long as there is a global data structure that can be 
>> mutated any time
>> (the API contract doesn't prohibit modifying these global arrays after
>> initialization), you would need to check for "cache invalidation" on
>> every lookup, undermining performance benefits.
> 
> Do you really expect an application modifying these lists dynamically?

At this time, not really. But the mark of good API design IMO is that
its flexibility empowers new and novel ideas and ways of doing things.
Bad APIs (including the use of global variables) inhibit flexibility and
constrain creativity and advanced usage.

For this particular item, I could see some potential uses in processes
hosting multiple, independent interpreters. Maybe you want to give each
interpreter its own set of modules. That's not something you see today
because of the GIL and all the other global state in CPython. But 

Re: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version

[Gregory Szorc]

On 5/16/2019 10:25 AM, Victor Stinner wrote:
> Le jeu. 16 mai 2019 à 16:10, Thomas Wouters  a écrit :
>>> Would you be ok with a "PyConfig_Init(PyConfig *config);" function
>>> which would initialize all fields to theire default values? Maybe
>>> PyConfig_INIT should be renamed to PyConfig_STATIC_INIT.
>>>
>>> You can find a similar API for pthread mutex, there is a init function
>>> *and* a macro for static initialization:
>>>
>>>int pthread_mutex_init(pthread_mutex_t *restrict mutex,
>>>const pthread_mutexattr_t *restrict attr);
>>>
>>>pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
>>
>>
>> This was going to be my suggestion as well: for any non-trivial macro, we 
>> should have a function for it instead.
> 
> Ok, I will do that.
> 
> 
>> I would also point out that PEP 587 has a code example that uses 
>> PyWideStringList_INIT, but that macro isn't mention anywhere else.
> 
> Oh, I forgot to better document it. Well, the macro is trivial:
> 
> #define _PyWstrList_INIT (_PyWstrList){.length = 0, .items = NULL}
> 
> For consistency, I prefer to not initialize manually these fields, but
> use a macro instead.
> 
> (Variables are allocated on the stack and so *must* be initialized.)
> 
> 
>> The PEP is a bit unclear as to the semantics of PyWideStringList as a whole: 
>> the example uses a static array with length, but doesn't explain what would 
>> happen with statically allocated data like that if you call the Append or 
>> Extend functions. It also doesn't cover how e.g. argv parsing would remove 
>> items from the list. (I would also suggest the PEP shouldn't use the term 
>> 'list', at least not unqualified, if it isn't an actual Python list.)
> 
> Calling PyWideStringList_Append() or PyWideStringList_Insert() on a
> "constant" list will crash: don't do that :-)
> 
> I tried to explain the subtle details of "constant" vs "dynamic"
> configurations in "Initialization with constant PyConfig" and "Memory
> allocations and Py_DecodeLocale()" functions.
> 
> A "constant" PyWideStringList must not be used with a "dynamic"
> PyConfig: otherwise, PyConfig_Clear() will crash as well.
> 
> I would prefer to have separated "const PyWideStringList" and "const
> PyConfig" types, but the C language doesn't convert "wchat_*" to
> "const wchar_t*" when you do that. We would need duplicated
> PyConstantWideStringList and PyConstantConfig structures, which would
> require to be "casted" to PyWideStringList and PyConfig internally to
> reuse the same code for constant and dynamic configuration.
> 
> If you consider that the specific case of "constant configuration"
> adds too much burden / complexity, we mgiht remove it and always
> require to use dynamic configuration.
> 
> Right now, Programs/_testembed.c almost uses only "constant"
> configuration. Using dynamic memory would make the code longer: need
> to handle memory allocation failures.
> 
> 
>> I understand the desire to make static allocation and initialisation 
>> possible, but since you only need PyWideStringList for PyConfig, not 
>> PyPreConfig (which sets the allocator), perhaps having a 
>> PyWideStringList_Init(), which copies memory, and PyWideStringList_Clear() 
>> to clear it, would be better?
> 
> Do you mean to always require to build dynamic lists? Said
> differently, not allow to write something like the following code?
> 
> static wchar_t* argv[] = {
> L"python3",
> L"-c",
> L"pass",
> L"arg2",
> };
> 
> _PyCoreConfig config = _PyCoreConfig_INIT;
> config.argv.length = Py_ARRAY_LENGTH(argv);
> config.argv.items = argv;
> 
> 
>>> If the private field "_init_main" of the PEP 587 is set to 0,
>>> Py_InitializeFromConfig() stops at the "core" phase (in fact, it's
>>> already implemented!). But I didn't implement yet a
>>> _Py_InitializeMain() function to "finish" the initialization. Let's
>>> say that it exists, we would get:
>>>
>>> ---
>>> PyConfig config = PyConfig_INIT;
>>> config._init_main = 0;
>>> PyInitError err = Py_InitializeFromConfig();
>>> if (Py_INIT_FAILED(err)) {
>>> Py_ExitInitError(err);
>>> }
>>>
>>> /* add your code to customize Python here */
>>> /* calling PyRun_SimpleString() here is safe */
>>>
>>> /* finish Python initialization */
>>> PyInitError err = _Py_InitializeMain();
>>> if (Py_INIT_FAILED(err)) {
>>> Py_ExitInitError(err);
>>> }
>>> ---
>>>
>>> Would it solve your use case?
>>
>>
>> FWIW, I understand the need here: for Hermetic Python, we solved it by 
>> adding a new API similar to PyImport_AppendInittab, but instead registering 
>> a generic callback hook to be called *during* the initialisation process: 
>> after the base runtime and the import mechanism are initialised (at which 
>> point you can create Python objects), but before *any* modules are imported. 
>> We use that callback to insert a meta-importer that satisfies all stdlib 
>> imports from an embedded archive. (Using a meta-importer allows us to bypass 
>> the 

Re: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version

[Gregory Szorc]

On 5/15/2019 4:10 PM, Victor Stinner wrote:
> Hi,
> 
> Thanks to the constructive discussions, I enhanced my PEP 587. I don't
> plan any further change, the PEP is now ready to review (and maybe
> even for pronouncement, hi Thomas! :-)).
> 
> The Rationale now better explains all challenges and the complexity of
> the Python Initialization Configuration.
> 
> The "Isolate Python" section is a short guide explaining how configure
> Python to embed it into an application.
> 
> The "Path Configuration" section elaborates the most interesting part
> of the configuration: configure where Python looks for modules
> (sys.path). I added PyWideStringList_Insert() to allow to prepend a
> path in module_search_paths.
> 
> The "Python Issues" section give a long list of issues solved directly
> or indirectly by this PEP.
> 
> I'm open for bikeshedding on PyConfig fields names and added functions
> names ;-) I hesitate for "use_module_search_paths": maybe
> "module_search_paths_set" is a better name, as in "is
> module_search_paths set?". The purpose of this field is to allow to
> have an empty sys.path (ask PyConfig_Read() to not override it). IMHO
> an empty sys.path makes sense for some specific use cases, like
> executing Pyhon code without any external module.
> 
> My PEP 587 proposes better names: Py_FrozenFlag becomes
> PyConfig.pathconfig_warnings and Py_DebugFlag becomes
> PyConfig.parser_debug. I also avoided double negation. For example,
> Py_DontWriteBytecodeFlag becomes write_bytecode.
> 
> Changes between version 3 and version 2:
> 
> * PyConfig: Add configure_c_stdio and parse_argv; rename _frozen to
> pathconfig_warnings.
> * Rename functions using bytes strings and wide strings. For example,
> Py_PreInitializeFromWideArgs() becomes Py_PreInitializeFromArgs(), and
> PyConfig_SetArgv() becomes PyConfig_SetBytesArgv().
> * Add PyWideStringList_Insert() function.
> * New "Path configuration", "Isolate Python", "Python Issues" and
> "Version History" sections.
> * PyConfig_SetString() and PyConfig_SetBytesString() now requires the
> configuration as the first argument.
> * Rename Py_UnixMain() to Py_BytesMain()
> 
> 
> HTML version:
> https://www.python.org/dev/peps/pep-0587/
> 
> Full PEP text below.
> 
> I know that the PEP is long, but well, it's a complex topic, and I
> chose to add many examples to make the API easier to understand.

I saw your request for feedback on Twitter a few days back and found
this thread.

This PEP is of interest to me because I'm the maintainer of PyOxidizer -
a project for creating single file executables embedding Python. As part
of hacking on PyOxidizer, I will admit to grumbling about the current
state of the configuration and initialization mechanisms. The reliance
on global variables and the haphazard way in which you must call certain
functions before others was definitely a bit frustrating to deal with.

I don't want to wade into too much bikeshedding in my review. I'll let
the professionals deal with things like naming :) Also, I haven't read
previous posts about this PEP. Apologies if my comments bring up old topics.

Let's get on with the review...

My most important piece of feedback is: thank you for tackling this!
Your work to shore up the inner workings of interpreter state and
management is a big deal on multiple dimensions. I send my sincere
gratitude.

Overall, I'm very happy with the state of the proposal. Better than what
we currently have on nearly every dimension. When reading my feedback,
please keep in mind that I'm in like 95% agreement with the proposal as is.

The following paragraphs detail points of feedback.

PyPreConfig_INIT and PyConfig_INIT as macros that return a struct feel
weird to me. Specifically, the `PyPreConfig preconfig =
PyPreConfig_INIT;` pattern doesn't feel right. I'm sort of OK with these
being implemented as macros. But I think they should look like function
calls so the door is open to converting them to function calls in the
future. An argument to make them actual function calls today is to
facilitate better FFI interop. As it stands, non-C/C++ bindings to the
API will need to reimplement the macro's logic. That might be simple
today. But who knows what complexity may be added in years ahead. An
opaque function implementation future proofs the API.

PyPreConfig.allocator being a char* seems a bit weird. Does this imply
having to use strcmp() to determine which allocator to use? Perhaps the
allocator setting should be an int mapping to a constant instead?
Relatedly, how are custom allocators registered? e.g. from Rust, I want
to use Rust's allocator. How would I do that in this API? Do I still
need to call PyMem_SetAllocator()? I thought a point of this proposal
was to consolidate per-interpreter config settings?

I'm a little confused about the pre-initialization functions that take
command arguments. Is this intended to only be used for parsing the
arguments that `python` recognizes? Presumably a custom application
embedding 

Re: [Python-Dev] Python startup time

[Gregory Szorc]

On 5/1/2018 8:26 PM, Gregory Szorc wrote:
> On 7/19/2017 12:15 PM, Larry Hastings wrote:
>>
>>
>> On 07/19/2017 05:59 AM, Victor Stinner wrote:
>>> Mercurial startup time is already 45.8x slower than Git whereas tested
>>> Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial
>>> developers, with a startup time 2x - 3x slower...
>>
>> When Matt Mackall spoke at the Python Language Summit some years back, I
>> recall that he specifically complained about Python startup time.  He
>> said Python 3 "didn't solve any problems for [them]"--they'd already
>> solved their Unicode hygiene problems--and that Python's slow startup
>> time was already a big problem for them.  Python 3 being /even slower/
>> to start was absolutely one of the reasons why they didn't want to upgrade.
>>
>> You might think "what's a few milliseconds matter".  But if you run
>> hundreds of commands in a shell script it adds up.  git's speed is one
>> of the few bright spots in its UX, and hg's comparative slowness here is
>> a palpable disadvantage.
>>
>>
>>> So please continue efforts for make Python startup even faster to beat
>>> all other programming languages, and finally convince Mercurial to
>>> upgrade ;-)
>>
>> I believe Mercurial is, finally, slowly porting to Python 3.
>>
>> https://www.mercurial-scm.org/wiki/Python3
>>
>> Nevertheless, I can't really be annoyed or upset at them moving slowly
>> to adopt Python 3, as Matt's objections were entirely legitimate.
> 
> I just now found found this thread when searching the archive for
> threads about startup time. And I was searching for threads about
> startup time because Mercurial's startup time has been getting slower
> over the past few months and this is causing substantial pain.
> 
> As I posted back in 2014 [1], CPython's startup overhead was >10% of the
> total CPU time in Mercurial's test suite. And when you factor in the
> time to import modules that get Mercurial to a point where it can run
> commands, it was more like 30%!
> 
> Mercurial's full test suite currently runs `hg` ~25,000 times. Using
> Victor's startup time numbers of 6.4ms for 2.7 and 14.5ms for
> 3.7/master, Python startup overhead contributes ~160s on 2.7 and ~360s
> on 3.7/master. Even if you divide this by the number of available CPU
> cores, we're talking dozens of seconds of wall time just waiting for
> CPython to get to a place where Mercurial's first bytecode can execute.
> 
> And the problem is worse when you factor in the time it takes to import
> Mercurial's own modules.
> 
> As a concrete example, I recently landed a Mercurial patch [2] that
> stubs out zope.interface to prevent the import of 9 modules on every
> `hg` invocation. This "only" saved ~6.94ms for a typical `hg`
> invocation. But this decreased the CPU time required to run the test
> suite on my i7-6700K from ~4450s to ~3980s (~89.5% of original) - a
> reduction of almost 8 minutes of CPU time (and over 1 minute of wall time)!
> 
> By the time CPython gets Mercurial to a point where we can run useful
> code, we've already blown most of or past the time budget where humans
> perceive an action/command as instantaneous. If you ignore startup
> overhead, Mercurial's performance compares quite well to Git's for many
> operations. But the reality is that CPython startup overhead makes it
> look like Mercurial is non-instantaneous before Mercurial even has the
> opportunity to execute meaningful code!
> 
> Mercurial provides a `chg` program that essentially spins up a daemon
> `hg` process running a "command server" so the `chg` program [written in
> C - no startup overhead] can dispatch commands to an already-running
> Python/`hg` process and avoid paying the startup overhead cost. When you
> run Mercurial's test suite using `chg`, it completes *minutes* faster.
> `chg` exists mainly as a workaround for slow startup overhead.
> 
> Changing gears, my day job is maintaining Firefox's build system. We use
> Python heavily in the build system. And again, Python startup overhead
> is problematic. I don't have numbers offhand, but we invoke likely a few
> hundred Python processes as part of building Firefox. It should be
> several thousand. But, we've had to "hack" parts of the build system to
> "batch" certain build actions in single process invocations in order to
> avoid Python startup overhead. This undermines the ability of some build
> tools to formulate a reasonable understanding of the DAG and it causes a
> bit of pain for build system developers and makes it difficult to
> achieve "no-op&quo

Re: [Python-Dev] Python startup time

[Gregory Szorc]

On Wed, May 2, 2018 at 8:26 PM, Benjamin Peterson <benja...@python.org>
wrote:

>
>
> On Wed, May 2, 2018, at 09:42, Gregory Szorc wrote:
> > The direction Mercurial is going in is that `hg` will likely become a
> Rust
> > binary (instead of a #!python script) that will use an embedded Python
> > interpreter. So we will have low-level control over the interpreter via
> the
> > C API. I'd also like to see us distribute a copy of Python in our
> official
> > builds. This will allow us to take various shortcuts, such as not having
> to
> > probe various sys.path entries since certain packages can only exist in
> one
> > place. I'd love to get to the state Google is at where they have
> > self-contained binaries with ELF sections containing Python modules. But
> > that requires a bit of very low-level hacking. We'll likely have a Rust
> > binary (that possibly static links libpython) and a separate JAR/zip-like
> > file containing resources.
>
> I'm curious about the rust binary. I can see that would give you startup
> time benefits similar to the ones you could get hacking the interpreter
> directly; e.g., you can use a zipfile for everything and not have site.py.
> But it seems like the Python-side wins would stop there. Is this all a
> prelude to incrementally rewriting hg in rust? (Mercuric oxide?)
>

The plans are recorded at https://www.mercurial-scm.org/wiki/OxidationPlan.
tl;dr we want to write some low-level bits in Rust but we anticipate the
bulk of the application logic remaining in Python.

Nobody in the project is seriously talking about a complete rewrite in
Rust. Contributors to the project have varying opinions on how aggressively
Rust should be utilized. People who contribute to the C code, low-level
primitives (like storage, deltas, etc), and those who care about
performance tend to want more Rust. One thing we almost universally agree
on is that we want to rewrite all of Mercurial's C code in Rust. I
anticipate that figuring out the balance between Rust and Python in
Mercurial will be an ongoing conversation/process for the next few years.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Gregory Szorc]

On 5/2/18 2:24 PM, Barry Warsaw wrote:
> On May 2, 2018, at 09:42, Gregory Szorc <gregory.sz...@gmail.com> wrote:
> 
>> As for things Python could do to make things better, one idea is for 
>> "package bundles." Instead of using .py, .pyc, .so, etc files as separate 
>> files on the filesystem, allow Python packages to be distributed as 
>> standalone "archive" files.
> 
> Of course, .so files have to be extracted to the file system, because we have 
> to live with dlopen()’s API.  In our first release of shiv, we had a loader 
> that did exactly that for just .so files.  We ended up just doing .pyz file 
> unpacking unconditionally, ignoring zip-safe, mostly because too many 
> packages still use __file__, which doesn’t work in a zipapp.

FWIW, Google has a patched glibc that implements dlopen_with_offset().
It allows you to do things like memory map the current binary and then
dlopen() a shared library embedded in an ELF section.

I've seen the code in the branch at
https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/google/grte/v4-2.19/master.
It likely exists elsewhere. An attempt to upstream it occurred at
https://sourceware.org/bugzilla/show_bug.cgi?id=11767. It is probably
well worth someone's time to pick up the torch and get this landed in
glibc so everyone can be a massive step closer to self-contained, single
binary applications. Of course, it will take years before you can rely
on a glibc version with this API being deployed universally. But the
sooner this lands...

> 
> I’ll plug shiv and importlib.resources (and the standalone 
> importlib_resources) again here. :)
> 
>> If you go this route, please don't require the use of zlib for file 
>> compression, as zlib is painfully slow compared to alternatives like lz4 and 
>> zstandard.
> 
> shiv works in a similar manner to pex, although it’s a completely new 
> implementation that doesn’t suffer from huge sys.paths or the use of 
> pkg_resources.  shiv + importlib.resources saves us 25-50% of warm cache 
> startup time.  That makes things better but still not ideal.  Ultimately 
> though that means we don’t suffer from the slowness of zlib since we don’t 
> count cold cache times (i.e. before the initial pyz unpacking operation).
> 
> Cheers,
> -Barry
> 
> 
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Gregory Szorc]

 On Tue, May 1, 2018 at 11:55 PM, Ray Donnelly 
wrote:

> Is your Python interpreter statically linked? The Python 3 ones from the
anaconda distribution (use Miniconda!) are for Linux and macOS and that
roughly halved our startup times.

My Python interpreters use a shared library. I'll definitely investigate
the performance of a statically-linked interpreter.

Correct me if I'm wrong, but aren't there downsides with regards to C
extension compatibility to not having a shared libpython? Or does all the
packaging tooling "just work" without a libpython? (It's possible I have my
wires crossed up with something else regarding a statically linked Python.)

On Wed, May 2, 2018 at 2:26 AM, Victor Stinner  wrote:

> What do you propose to make Python startup faster?
>

That's a very good question. I'm not sure I'm able to answer it because I
haven't dug too much into CPython's internals much farther than what is
required to implement C extensions. But I can share insight from what the
Mercurial project has collectively learned.


>
> As I wrote in my previous emails, many Python core developers care of
> the startup time and we are working on making it faster.
>
> INADA Naoki added -X importtime to identify slow imports and
> understand where Python spent its startup time.
>

-X importtime is a great start! For a follow-up enhancement, it would be
useful to see what aspects of import are slow. Is it finding modules
(involves filesystem I/O)? Is it unmarshaling pyc files? Is it executing
the module code? If executing code, what part is slow? Inline
statements/expressions? Compiling types? Printing the microseconds it takes
to import a module is useful. But it only gives me a general direction: I
want to know what parts of the import made it slow so I know if I should be
focusing on code running during module import, slimming down the size of a
module, eliminating the module import from fast paths, pursuing alternative
module importers, etc.


>
> Recent example: Barry Warsaw identified that pkg_resources is slow and
> added importlib.resources to Python 3.7:
> https://docs.python.org/dev/library/importlib.html#module-
> importlib.resources
>
> Brett Cannon is also working on a standard solution for lazy imports
> since many years:
> https://pypi.org/project/modutil/
> https://snarky.ca/lazy-importing-in-python-3-7/
>

Mercurial has used lazy module imports for years. On 2.7.14, it reduces `hg
version` from ~160ms to ~55ms (~34% of original). On Python 3, we're using
`importlib.util.LazyLoader` and it reduces `hg version` on 3.7 from ~245ms
to ~120ms (~49% of original). I'm not sure why Python 3's built-in module
importer doesn't yield the speedup that our custom Python 2 importer does.
One explanation is our custom importer is more advanced than importlib.
Another is that Python 3's import mechanism is slower (possibly due to
being written in Python instead of C). We haven't yet spent much time
optimizing Mercurial for Python 3: our immediate goal is to get it working
first. Given the startup performance problem on Python 3, it is only a
matter of time before we dig into this further.

It's worth noting that lazy module importing can be undone via common
patterns. Most commonly, `from foo import X`. It's *really* difficult to
implement a proper object proxy. Mercurial's lazy importer gives up in this
case and imports the module and exports the symbol. (But if the imported
module is a package, we detect that and make the module exports proxies to
a lazy module.)

Another common undermining of the lazy importer is code that runs during
import time module exec that accesses an attribute. e.g.

```
import foo

class myobject(foo.Foo):
pass
```

Mercurial goes out of its way to avoid these patterns so modules can be
delay imported as much as possible. As long as import times are
problematic, it would be helpful if the standard library adopted similar
patterns. Although I recognize there are backwards compatibility concerns
that tie your hands a bit.


> Nick Coghlan is working on the C API to configure Python startup: PEP
> 432. When it will be ready, maybe Mercurial could use a custom Python
> optimized for its use case.
>

That looks great!

The direction Mercurial is going in is that `hg` will likely become a Rust
binary (instead of a #!python script) that will use an embedded Python
interpreter. So we will have low-level control over the interpreter via the
C API. I'd also like to see us distribute a copy of Python in our official
builds. This will allow us to take various shortcuts, such as not having to
probe various sys.path entries since certain packages can only exist in one
place. I'd love to get to the state Google is at where they have
self-contained binaries with ELF sections containing Python modules. But
that requires a bit of very low-level hacking. We'll likely have a Rust
binary (that possibly static links libpython) and a separate JAR/zip-like
file 

Re: [Python-Dev] Python startup time

[Gregory Szorc]

On 7/19/2017 12:15 PM, Larry Hastings wrote:
> 
> 
> On 07/19/2017 05:59 AM, Victor Stinner wrote:
>> Mercurial startup time is already 45.8x slower than Git whereas tested
>> Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial
>> developers, with a startup time 2x - 3x slower...
> 
> When Matt Mackall spoke at the Python Language Summit some years back, I
> recall that he specifically complained about Python startup time.  He
> said Python 3 "didn't solve any problems for [them]"--they'd already
> solved their Unicode hygiene problems--and that Python's slow startup
> time was already a big problem for them.  Python 3 being /even slower/
> to start was absolutely one of the reasons why they didn't want to upgrade.
> 
> You might think "what's a few milliseconds matter".  But if you run
> hundreds of commands in a shell script it adds up.  git's speed is one
> of the few bright spots in its UX, and hg's comparative slowness here is
> a palpable disadvantage.
> 
> 
>> So please continue efforts for make Python startup even faster to beat
>> all other programming languages, and finally convince Mercurial to
>> upgrade ;-)
> 
> I believe Mercurial is, finally, slowly porting to Python 3.
> 
> https://www.mercurial-scm.org/wiki/Python3
> 
> Nevertheless, I can't really be annoyed or upset at them moving slowly
> to adopt Python 3, as Matt's objections were entirely legitimate.

I just now found found this thread when searching the archive for
threads about startup time. And I was searching for threads about
startup time because Mercurial's startup time has been getting slower
over the past few months and this is causing substantial pain.

As I posted back in 2014 [1], CPython's startup overhead was >10% of the
total CPU time in Mercurial's test suite. And when you factor in the
time to import modules that get Mercurial to a point where it can run
commands, it was more like 30%!

Mercurial's full test suite currently runs `hg` ~25,000 times. Using
Victor's startup time numbers of 6.4ms for 2.7 and 14.5ms for
3.7/master, Python startup overhead contributes ~160s on 2.7 and ~360s
on 3.7/master. Even if you divide this by the number of available CPU
cores, we're talking dozens of seconds of wall time just waiting for
CPython to get to a place where Mercurial's first bytecode can execute.

And the problem is worse when you factor in the time it takes to import
Mercurial's own modules.

As a concrete example, I recently landed a Mercurial patch [2] that
stubs out zope.interface to prevent the import of 9 modules on every
`hg` invocation. This "only" saved ~6.94ms for a typical `hg`
invocation. But this decreased the CPU time required to run the test
suite on my i7-6700K from ~4450s to ~3980s (~89.5% of original) - a
reduction of almost 8 minutes of CPU time (and over 1 minute of wall time)!

By the time CPython gets Mercurial to a point where we can run useful
code, we've already blown most of or past the time budget where humans
perceive an action/command as instantaneous. If you ignore startup
overhead, Mercurial's performance compares quite well to Git's for many
operations. But the reality is that CPython startup overhead makes it
look like Mercurial is non-instantaneous before Mercurial even has the
opportunity to execute meaningful code!

Mercurial provides a `chg` program that essentially spins up a daemon
`hg` process running a "command server" so the `chg` program [written in
C - no startup overhead] can dispatch commands to an already-running
Python/`hg` process and avoid paying the startup overhead cost. When you
run Mercurial's test suite using `chg`, it completes *minutes* faster.
`chg` exists mainly as a workaround for slow startup overhead.

Changing gears, my day job is maintaining Firefox's build system. We use
Python heavily in the build system. And again, Python startup overhead
is problematic. I don't have numbers offhand, but we invoke likely a few
hundred Python processes as part of building Firefox. It should be
several thousand. But, we've had to "hack" parts of the build system to
"batch" certain build actions in single process invocations in order to
avoid Python startup overhead. This undermines the ability of some build
tools to formulate a reasonable understanding of the DAG and it causes a
bit of pain for build system developers and makes it difficult to
achieve "no-op" and fast incremental builds because we're always
invoking certain Python processes because we've had to move DAG
awareness out of the build backend and into Python. At some point, we'll
likely replace Python code with Rust so the build system is more "pure"
and easier to maintain and reason about.

I've seen posts in this thread and elsewhere in the CPython development
universe that challenge whether milliseconds in startup time matter.
Speaking as a Mercurial and Firefox build system developer,
*milliseconds absolutely matter*. Going further, *fractions of
milliseconds matter*. For Mercurial's 

Re: [Python-Dev] Better support for consuming vendored packages

[Gregory Szorc]

On 3/22/2018 10:48 AM, Oleg Broytman wrote:
> Hi!
> 
> On Thu, Mar 22, 2018 at 09:58:07AM -0700, Gregory Szorc 
> <gregory.sz...@gmail.com> wrote:
>> Not all consumers of Python packages wish to consume Python packages in the
>> common `pip install `
> 
> IMO `pip` is for developers. To package and distribute end-user
> applications there are rpm, dpkg/deb, PyInstaller, cx_Freeze, py2exe
> (+ installer like NSIS or InnoSetup), py2app, etc...
> 
> Most of them pack a copy of Python interpreter and necessary parts
> of stdlib, so there is no problem with `sys.path` and wrong imports.

Yes, there are tools to create standalone packages. Some even bundle a
Python install so the execution environment is more deterministic. These
are great ways to distribute Python applications!

However, if you are a Python application that is maintained by your
distribution's package manager, you pretty much must use a Python
installed by the system package manager. And that leaves us with the
original problem of an undefined execution environment. So packaging
tools for standalone Python applications only work if you control the
package distribution channel. If you are a successful Python application
that is packaged by your distro, you lose the ability to control your
own destiny and must confront these problems for users who installed
your application through their distro's package manager. i.e. the cost
of success for your Python application is a lot of pain inflicted by
policies of downstream packagers.

Also, not vendoring dependencies puts the onus on downstream packagers
to deal with those dependencies. There can be package version conflicts
between various packaged Python applications ("dependency hell").
Vendoring dependencies under application-local package names removes the
potential for version conflicts.

It's worth noting that some downstream packagers do insist on unbundling
dependencies. So they may get stuck with work regardless. But if you
vendor dependencies, a downstream packager is at least capable of
packaging a Python application without having to deal with "dependency
hell."

Maybe what I'm asking for here is import machinery where an application
can forcefully limit or influence import mechanisms for modules in a
certain package. But this seems difficult to achieve given the
constraint of a single, global modules namespace (`sys.modules`) per
interpreter.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Better support for consuming vendored packages

[Gregory Szorc]

 I'd like to start a discussion around practices for vendoring package
dependencies. I'm not sure python-dev is the appropriate venue for this
discussion. If not, please point me to one and I'll gladly take it there.

I'll start with a problem statement.

Not all consumers of Python packages wish to consume Python packages in the
common `pip install ` + `import ` manner. Some Python
applications may wish to vendor Python package dependencies such that known
compatible versions are always available.

For example, a Python application targeting a general audience may not wish
to expose the existence of Python nor want its users to be concerned about
Python packaging. This is good for the application because it reduces
complexity and the surface area of things that can go wrong.

But at the same time, Python applications need to be aware that the Python
environment may contain more than just the Python standard library and
whatever Python packages are provided by that application. If using the
system Python executable, other system packages may have installed Python
packages in the system site-packages and those packages would be visible to
your application. A user could `pip install` a package and that would be in
the Python environment used by your application. In short, unless your
application distributes its own copy of Python, all bets are off with
regards to what packages are installed. (And even then advanced users could
muck with the bundled Python, but let's ignore that edge case.)

In short, `import X` is often the wild west. For applications that want to
"just work" without requiring end users to manage Python packages, `import
X` is dangerous because `X` could come from anywhere and be anything -
possibly even a separate code base providing the same package name!

Since Python applications may not want to burden users with Python
packaging, they may vendor Python package dependencies such that a known
compatible version is always available. In most cases, a Python application
can insert itself into `sys.path` to ensure its copies of packages are
picked up first. This works a lot of the time. But the strategy can fall
apart.

Some Python applications support loading plugins or extensions. When
user-provided code can be executed, that code could have dependencies on
additional Python packages. Or that custom code could perform `sys.path`
modifications to provide its own package dependencies. What this means is
that `import X` from the perspective of the main application becomes
dangerous again. You want to pick up the packages that you provided. But
you just aren't sure that those packages will actually be picked up. And to
complicate matters even more, an extension may wish to use a *different*
version of a package from what you distribute. e.g. they may want to adopt
the latest version that you haven't ported to you or they may want to use
an old versions because they haven't ported yet. So now you have the
requirements that multiple versions of packages be available. In Python's
shared module namespace, that means having separate package names.

A partial solution to this quagmire is using relative - not absolute -
imports. e.g. say you have a package named "knights." It has a dependency
on a 3rd party package named "shrubbery." Let's assume you distribute your
application with a copy of "shrubbery" which is installed at some packages
root, alongside "knights:"

  /
  /knights/__init__.py
  /knights/ni.py
  /shrubbery/__init__.py

If from `knights.ni` you `import shrubbery`, you /could/ get the copy of
"shrubbery" distributed by your application. Or you could pick up some
other random copy that is also installed somewhere in `sys.path`.

Whereas if you vendor "shrubbery" into your package. e.g.

  /
  /knights/__init__.py
  /knights/ni.py
  /knights/vendored/__init__.py
  /knights/vendored/shrubbery/__init__.py

Then from `knights.ni` you `from .vendored import shrubbery`, you are
*guaranteed* to get your local copy of the "shrubbery" package.

This reliable behavior is highly desired by Python applications.

But there are problems.

What we've done is effectively rename the "shrubbery" package to
"knights.vendored.shrubbery." If a module inside that package attempts an
`import shrubbery.x`, this could fail because "shrubbery" is no longer the
package name. Or worse, it could pick up a separate copy of "shrubbery"
somewhere else in `sys.path` and you could have a Frankenstein package
pulling its code from multiple installs. So for this to work, all
package-local imports must be using relative imports. e.g. `from . import
x`.

The takeaway is that packages using relative imports for their own modules
are much more flexible and therefore friendly to downstream consumers that
may wish to vendor them under different names. Packages using relative
imports can be dropped in and used, often without source modifications.
This is a big deal, as downstream consumers don't want to be
modifying/forking 

[Python-Dev] Binary CPython distribution for Linux

[Gregory Szorc]

I'm an advocate of getting users and projects to move to modern Python 
versions. I believe dropping support for end-of-lifed Python versions is 
important for the health of the Python community. If you've done any 
amount of Python 3 porting work, you know things get much harder the 
more 2.x legacy versions you need to support.


I led the successful charge to drop support for Python 2.6 and below 
from Firefox's build system. I failed to win the argument that Mercurial 
should drop 2.4 and 2.5 [1]. A few years ago, I started a similar 
conversation with the LLVM project [2]. I wrote a blog post on the 
subject [3] that even got Slashdotted [4] (although I don't think that's 
the honor it was a decade ago).


While much of the opposition to dropping Python 2.7 stems from the RHEL 
community (they still have 2.4 in extended support and 2.7 wasn't in a 
release until a few weeks ago), a common objection from the users is I 
can't install a different Python or it's too difficult to install a 
different Python. The former is a legit complaint - if you are on 
shared hosting and don't have root, as easy as it is to add an alternate 
package repository that provides 2.7 (or newer), you don't have the 
permissions so you can't do it.


This leaves users with attempting a userland install of Python. 
Personally, I think installing Python in userland is relatively simple. 
Tools like pyenv make this turnkey. Worst case you fall back to 
configure + make. But I'm an experienced developer and have a compiler 
toolchain and library dependencies on my machine. What about less 
experienced users or people that don't have the necessary build 
dependencies? And, even if they do manage to find or build a Python 
distribution, we all know that there's enough finicky behavior with 
things like site-packages default paths to cause many headaches, even 
for experienced Python hackers.


I'd like to propose a solution to this problem: a pre-built distribution 
of CPython for Linux available via www.python.org in the list of 
downloads for a particular release [5]. This distribution could be 
downloaded and unarchived into the user's home directory and users could 
start running it immediately by setting an environment variable or two, 
creating a symlink, or even running a basic installer script. This would 
hopefully remove the hurdles of obtaining a (sane) Python distribution 
on Linux. This would allow projects to more easily drop end-of-life 
Python versions and would speed adoption of modern Python, including 
Python 3 (because porting is much easier if you only have to target 2.7).


I understand there may be technical challenges with doing this for some 
distributions and with producing a universal binary distribution. I 
would settle for a binary distribution that was targeted towards RHEL 
users and variant distros, as that is the user population that I 
perceive to be the most conservative and responsible for holding modern 
Python adoption back.


[1] 
http://permalink.gmane.org/gmane.comp.version-control.mercurial.devel/68902

[2] http://lists.cs.uiuc.edu/pipermail/llvmdev/2012-December/056545.html
[3] 
http://gregoryszorc.com/blog/2014/01/08/why-do-projects-support-old-python-releases/
[4] 
http://developers.slashdot.org/story/14/01/09/1940232/why-do-projects-continue-to-support-old-python-releases

[5] https://www.python.org/download/releases/2.7.7/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python process creation overhead

[Gregory Szorc]

On 5/12/2014 8:43 PM, Nick Coghlan wrote:
 On 13 May 2014 10:19,  dw+python-...@hmmz.org wrote:
 On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote:
 
 Why can't Python start as quickly as Perl or Ruby?
 
 On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops
 from 81ms startup to 20ms by specifying -S, which disables
 site.py.
 
 Oblitering the .pth files immediately knocks 10ms off regular
 startup. I guess the question isn't why Python is slower than
 perl, but what aspects of site.py could be cached, reimplemented,
 or stripped out entirely.  I'd personally love to see .pth
 support removed.
 
 The startup code is currently underspecified and underdocumented,
 and quite fragile as a result. It represents 20+ years of organic
 growth without any systematic refactoring to simplify and
 streamline things.
 
 That's what PEP 432 aims to address, and is something I now expect
 to have time to get back to for Python 3.5. And yes, one thing
 those changes should enable is the creation of system Python
 runtimes on Linux that initialise faster than the current
 implementation.

This is terrific news and something I greatly anticipate taking
advantage of!

But the great many of us still on 2.7 likely won't see a benefit,
correct? How insane would it be for people to do things like pass -S
in the shebang and manually implement the parts of site.py that are
actually needed?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] python process creation overhead

[Gregory Szorc]

On 5/10/2014 2:46 PM, Victor Stinner wrote:
 Le 10 mai 2014 22:51, Gregory Szorc gregory.sz...@gmail.com 
 mailto:gregory.sz...@gmail.com a écrit :
 Furthermore, Python 3 appears to be 50% slower than Python 2.
 
 Please mention the minor version. It looks like you compared 2.7
 and 3.3. Please test 3.4, we made interesting progress on the
 startup time.
 
 There is still something to do, especially on OS X. Depending on
 the OS, different modules are loaded and some functions are
 implemented differently.

3.4.0 does appear to be faster than 3.3.5 on Linux - `python -c ''` is
taking ~50ms (as opposed to ~60ms) on my i7-2600K. Great to see!

But 3.4.0 is still slower than 2.7.6. And all versions of CPython are
over 3x slower than Perl 5.18.2. This difference amounts to minutes of
CPU time when thousands of processes are involved. That seems
excessive to me.

Why can't Python start as quickly as Perl or Ruby?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] python process creation overhead

[Gregory Szorc]

I was investigating speeding up Mercurial's test suite (it runs ~13,000
Python processes) and I believe I've identified CPython
process/interpreter creation and destruction as sources of significant
overhead and thus a concern for any CPython user.

Full details are at [1]. tl;dr 10-18% of CPU time in test suite
execution was spent doing the equivalent of `python -c 1` and 30-38% of
CPU time was spent doing the equivalent of `hg version` (I suspect this
is mostly dominated by module importing - and Mercurial has lazy module
importing). My measurements show CPython is significantly slower than
Perl and Ruby when it comes to process/interpreter creation/destruction.
Furthermore, Python 3 appears to be 50% slower than Python 2.

Any system spawning many Python processes will likely feel this
overhead. I'm also a contributor to Firefox's build and testing
infrastructure. Those systems collectively spawn thousands of Python
processes. While I haven't yet measured, I fear that CPython process
overhead is also contributing to significant overhead there.

While more science and knowledge should be added before definite
conclusions are reached, I'd like to ring the figurative bell about this
problem. This apparent inefficiency has me concerned about the use of
CPython in scenarios where process spawning is a common operation and
decent latency is important. This includes CLI tools, build systems, and
testing. I'm curious if others feel this is a problem and what steps can
be taken to mitigate or correct it.

[1] http://www.selenic.com/pipermail/mercurial-devel/2014-May/058854.html

Gregory Szorc
gregory.sz...@gmail.com

P.S. I'm not subscribed, so please CC me on responses.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com