date:20210928

Hi,

I would like to change the Python C API. I failed to write a single
document listing all constraints and proposing all changes that I
would to do. For example, my previous PEP 620 contains too many
changes and is too long.

Here is my attempt to focus on the bare minimum and (what I consider
as) the least controversial part: list current usages of C API and
constraints of these usages. This *informal* PEP should be the base of
future PEP changing the C API.

The current draft lives at:
https://github.com/vstinner/misc/blob/main/cpython/pep-c-api-next-level.rst

My PEP is based on HPy Next Level Manifesto written by Simon Cross:
https://github.com/hpyproject/hpy/wiki/c-api-next-level-manifesto

To reach most users of the C API, I cross-posted this email to
python-dev, capi-sig and hpy-dev.


+
Taking the Python C API to the Next Level
+

::

PEP: xxx
Title: Taking the Python C API to the Next Level
Author: Victor Stinner 
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 28-Sep-2021
Python-Version: 3.11

While the C API is a key of the Python popularity, it causes multiple
subtle and complex issues. There are different ways to use the C API,
each usage has its own constraints, and some constraints are exclusive.
This document lists constraints but doesn't propose changes, it only
gives vague ideas how to solve some issues. More concrete C API changes
will require writing separated PEPs.

C extensions are a key component of the Python popularity
=

The Python popularity comes from its great programming language and from
its wide collection of modules freely available on PyPI. Many of the
most popular Python modules rely directly or indirectly on C extensions
written with the C API. The Python C API is a key component of the
Python popularity.

For example, the numpy project is now a common dependency on many
scientific projects and a large part of the project is written by hand
with the C API.

**Abandoning or removing the C API** is out of question. Years ago, the
incomplete C API support was the main drawback of PyPy, since PyPy only
supported a minority of C extensions.

Today, CPython still have a similar issue. **When Cython or numpy don't
support a new Python version** (because of incompatible C API changes),
many Python projects depending on them are cannot be installed,
especially during the development phase of the next Python version.


Backward compatibility and unmaintained C extensions


One important property of the C API is the backward compatibility.
Developers expect that if their C extension works on Python 3.10, it
will work unmodified in Python 3.11: building the C extension with
Python 3.11 should be enough.

This property is even more important for unmaintained C extensions.
Sometimes, unmaintained just means that the only maintainer is busy or
overwhelmed for a few months. Sometimes, the project has no activity for
longer than 5 years.

When an incompatible change is introduced in the C API, like removing a
function or changing a function behavior, there is a **risk of breaking
an unknown number of C extensions**.

One option can be to update old C extensions when they are built on
recent Python versions, to adapt them to incompatible changes. This
conversion is non trivial and cannot handle all kinds of incompatible
changes.


Migration plan for incompatible changes
===

There should be a **sensible migration path** for large C extensions
(e.g.  numpy) when incompatible changes are introduced. Whenever
possible, it should be possible to write a **single code base** compatible
with old and new Python versions.

A **compatibility layer** can be maintained externally.  Cython and
numpy have their own internal compatibility layer.

There should be a way to easily pick up common errors introduced by
migrating.

One practical way to **minimize the number of broken projects** is to
attempt to check in advance if an incompatible change is going to break
popular C extensions. For broken C extensions, propose a fix and wait
until a new release includes the fix, before introducing the change in
Python. Obviously, it doesn't solve the problem of less popular C
extensions and private C extensions.


Obtain the best possible performance


There are two main reasons for writing a C extension: implement a
function which cannot be written in pure Python, or write a **C
accelerator**: rewrite the 10% of an application in C where 90% of the
CPU time is spent. About the former use case, the intent is to obtain
the best possible performance. Tradeoffs are made with portability: it
is acceptable to only support a limited number of Python versions and to
only support a limited number of Python impl

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Filipe Laíns

On Tue, 2021-09-28 at 10:22 +0200, Marc-Andre Lemburg wrote:
> On 27.09.2021 18:51, Eric Snow wrote:
> > We've frozen most of the stdlib modules imported during "python -c
> > pass" [1][2], to make startup a bit faster.  Import of those modules
> > is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> > to "off" but we'd like to default to "on".  The blocker is the impact
> > on contributors.  I expect many will make changes to a stdlib module
> > and then puzzle over why those changes aren't getting used.  That's an
> > annoyance we can avoid, which is the point of this thread.
> > 
> > Possible solutions:
> > 
> > 1. always default to "on" (the annoyance for contributors isn't big enough?)
> > 2. default to "on" if it's a PGO build (and "off" otherwise)
> > 3. default to "on" unless running from the source tree
> > 
> > Thoughts?
> 
> #3 sounds like a good solution, but how would you detect "running
> from the source tree" ? This sounds like you need another stat call
> somewhere, which is what the frozen modules try to avoid.

It does. FYI, here's the sysconfig implementation
https://github.com/python/cpython/blob/main/Lib/sysconfig.py#L146-L181

But a more efficient way to do this could be added.

> I'd like to suggest adding an environment variable to enable /
> disable the setting instead. This makes it easy to customize the
> behavior without introducing complicated logic.
> 

From your followup reply, it seems like you are suggesting that it should be
enabled by default, and use a env var to disable it. That would have the same
problem regarding the annoyance of contributors.

Is there any reason why you would prefer that over #2? That seems like the best
option to me if #3 is not feasible.

Cheers :)
Filipe Laíns


signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZU7L6GFVSWLVYAG2MOWJ4YU4FRPD6G5I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Paul Moore

On Tue, 28 Sept 2021 at 15:33, Eric Snow  wrote:

> > It means that the site module module can no longer be "customized" by
> > modifying directly the site.py file (inject a path in PYTHONPATH env
> > var where the customized site.py lives). But there is already a
> > supported way to customize the site module: create a module named
> > "sitecustomize" or "usercustomizer". I recall that virtualenv likes to
> > override stdlib site.py with its own code. tox uses virtualenv by
> > default. Someone should check if freezing site doesn't break
> > virtualenv and tox, since they seem to be popular in Python. The venv
> > doesn't need to override site.py and tox can use venv if I recall
> > correctly.
> >
> > If site.py customization is too popular, I would suggest to not freeze
> > this one, until the community stops doing that.
>
> Good point.  I'll look into that.

I don't believe virtualenv ships its own site.py these days. That was
a historical thing, and was always a pain point, but when virtualenv
got rewritten I'm almost certain we stopped doing it.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C5J763VQV423T4KIA4WRDIEJUQ3423FA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:55 AM Eric V. Smith  wrote:
> As a compromise, how about go with #1, but print a warning if python
> detects that it's not built with optimizations or is run from a source
> tree (the conditions in #2 and #3)? The warning could suggest running
> with "-X frozen_modules=off". I realize that it will probably be ignored
> over time, but maybe it will provide enough of a reminder if someone is
> debugging and sees the warning.

Yeah, that would probably be sufficient (and much simpler).  I'll try it out.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EFNVXABK36DTKO6IDFC2PTP6P4OHM46B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:47 AM Pablo Galindo Salgado
 wrote:
> One interesting consequence of what Eric mentioned (They have a different 
> loader and repr.  Also, frozen modules do not
> have __file__ set (and __path__ is always []).) is that frozen modules don't 
> have a `__file__` attribute IIRC and therefore
>  tracebacks won't include the source.

FYI, we are planning on setting __file__ on the frozen stdlib modules,
whenever possible.  (We can do that whenever we can determine the
stdlib dir during startup. See https://bugs.python.org/issue45211.)
Regardless, for tracebacks we would need to set co_filename on the
module's code objects, right?

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W6F2V3H3KHGLOL5CJDLTO7DGO37LYIG5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:36 AM Victor Stinner  wrote:
> Honestly, for me, #1: always on, is the most reasonable choice.
>
> I dislike when Python behaves differently depending on subtle things
> like "was it built with optimizations" or "is Python started from its
> source tree"?
>
> When I built Python without optimization and/or from its source tree,
> I do that to debug an issue. If the bug goes away in this case, it can
> waste my time.
>
> So I prefer to teach everybody how to use "-X frozen_modules=off" if
> they want to hack the stdlib for their greatest pleasure. I prefer
> that such special use case requires an opt-in option, the special use
> case is not special enough to be the default.

Agreed.  I just don't want to discourage potential contributors nor
waste anyone's time.  I suppose that's the fundamental question I
originally posted: would it be too annoying for contributors if we
made the default "on" always?  I expect most non-docs contributions
are made against the stdlib so that factors in.

> It means that the site module module can no longer be "customized" by
> modifying directly the site.py file (inject a path in PYTHONPATH env
> var where the customized site.py lives). But there is already a
> supported way to customize the site module: create a module named
> "sitecustomize" or "usercustomizer". I recall that virtualenv likes to
> override stdlib site.py with its own code. tox uses virtualenv by
> default. Someone should check if freezing site doesn't break
> virtualenv and tox, since they seem to be popular in Python. The venv
> doesn't need to override site.py and tox can use venv if I recall
> correctly.
>
> If site.py customization is too popular, I would suggest to not freeze
> this one, until the community stops doing that.

Good point.  I'll look into that.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M53U66ZP7QUSHDBYK2HONALLKW2EKSFQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 2:22 AM Marc-Andre Lemburg  wrote:
> #3 sounds like a good solution, but how would you detect "running
> from the source tree" ? This sounds like you need another stat call
> somewhere, which is what the frozen modules try to avoid.

We already look for the stdlib dir in Modules/getpath.c.  We can use
that information without an extra stat.  (See
https://bugs.python.org/issue45211.)

> I'd like to suggest adding an environment variable to enable /
> disable the setting instead. This makes it easy to customize the
> behavior without introducing complicated logic.

That's essentially what "-X frozen_modules=..." provides, though with
an env var you don't have to adjust your CLI invocation each time.
That said, there are a couple reasons why an env var might not be
suitable.  For one, I expect use of the -X option to be very uncommon,
especially outside of core development, so more of a one-off feature.
In contrast, to me environment variables imply repeated usage.  Also,
if we use an env var to override the default (of "on"), contributors
will still get bitten by the problem I described originally.  To me,
it's important that the default in that case be "off" without any
other intervention.

FWIW, I consider the "complicated logic" part as the negative side of
going with running-in-source-tree.  So, at this point I'm leaning more
toward Brett's suggestion of using "configure --with-pydebug" (AKA
Py_DEBUG) to determine the default.  That should be a suitable
approximation of running-in-source-tree.  We can circle back if it
proves inadequate.

On Tue, Sep 28, 2021 at 2:26 AM Marc-Andre Lemburg  wrote:
> Just to clarify: the modules would still always be frozen with
> the env var setting, but Python would simply not import them
> as frozen modules, but instead go and look on the PYTHONPATH
> for the modules.
>
> This could be achieved by special casing the frozen module
> finder function to only trigger on importlib modules and
> return NULL for all other possibly frozen modules.

Right.  That is essentially what we're doing.  (See find_frozen() in
Python/import.c.)

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BVXMNZYHBPTKYC4QEVHGWUKQMLR2XGSZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:02 AM Ronald Oussoren via Python-Dev
 wrote:
> Of course. I mentioned it because the proposal is to add a new option that’s 
> enabled after installation, and basically not when the testsuite is run.  
> That’s not a problem, we could just enable the option in most CI jobs.

FYI, I already added the CLI option (-X frozen_modules=[on|off]) a
couple weeks ago, with the default always "off", and have frozen about
10 of the stdlib modules (see _imp._frozen_module_names()).  This
thread is about a satisfactory approach to changing the default to
"on".

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HD7GBPNT74GPY6COVQ6W4V7MTJ4NIHUT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 2:54 AM Ronald Oussoren via Python-Dev
 wrote:
> I agree, but… Most CPython tests are run while running from the source tree, 
> that means that there will have to be testrunner configurations that run with 
> “-X frozen_modules=on”.

If the build option that determines the default is covered by existing
builtbots then we will be running the test suite in both modes without
any extra work.  The alternative is that we do for other modules what
we do with importlib: run the relevant tests one in each mode.
However, it's better to run the whole suite in both modes, so I'd
favor relying on the build-option-specific buildbots to get us
coverage.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M2LMJYETM7KXVAWQ6UY7DMAZUXO6H33K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

The Python Debug Build document lists changes compared to a release build:
https://docs.python.org/dev/using/configure.html#python-debug-build

Sometimes, I'm confused that "./python" (Python built locally in debug
mode) displays warnings, whereas "python" (Fedora package) doesn't.


See also the Python Development Mode:
https://docs.python.org/dev/library/devmode.html#devmode

The documentation starts with: "The Python Development Mode introduces
additional runtime checks that are too expensive to be enabled by
default."

Victor

On Tue, Sep 28, 2021 at 3:33 PM Eric V. Smith  wrote:
>
> On 9/28/2021 9:17 AM, Antoine Pitrou wrote:
> > On Tue, 28 Sep 2021 09:14:38 -0400
> > "Eric V. Smith"  wrote:
> >> On 9/28/2021 9:10 AM, Antoine Pitrou wrote:
> >>> On Tue, 28 Sep 2021 08:55:05 -0400
> >>> "Eric V. Smith"  wrote:
> > So I prefer to teach everybody how to use "-X frozen_modules=off" if
> > they want to hack the stdlib for their greatest pleasure. I prefer
> > that such special use case requires an opt-in option, the special use
> > case is not special enough to be the default.
>  I agree with Victor here: I'd rather have #1.
> 
>  As a compromise, how about go with #1, but print a warning if python
>  detects that it's not built with optimizations or is run from a source
>  tree (the conditions in #2 and #3)? The warning could suggest running
>  with "-X frozen_modules=off". I realize that it will probably be ignored
>  over time, but maybe it will provide enough of a reminder if someone is
>  debugging and sees the warning.
> >>> What would be the point of printing a warning instead of doing just
> >>> what the user is expecting?
> >> To me, the point would be to get the same behavior no matter which
> >> python executable I run, and without regard to where I run it from.
> > But why do you care about this?  What does it change *concretely*?
>
> It reduces the number of things I have to remember which are different
> based on where I'm running python (or which executable I'm running).
>
> Eric
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/2FN2OCYGMLU5XST44N7SFDUAI426VRB3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F6SHNS7EU2JCBV4FMYMFPC2JZZJV2L6Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Eric V. Smith

On 9/28/2021 9:17 AM, Antoine Pitrou wrote:

On Tue, 28 Sep 2021 09:14:38 -0400
"Eric V. Smith" wrote:

On 9/28/2021 9:10 AM, Antoine Pitrou wrote:

On Tue, 28 Sep 2021 08:55:05 -0400
"Eric V. Smith" wrote:

So I prefer to teach everybody how to use "-X frozen_modules=off" if
they want to hack the stdlib for their greatest pleasure. I prefer
that such special use case requires an opt-in option, the special use
case is not special enough to be the default.

I agree with Victor here: I'd rather have #1.

As a compromise, how about go with #1, but print a warning if python
detects that it's not built with optimizations or is run from a source
tree (the conditions in #2 and #3)? The warning could suggest running
with "-X frozen_modules=off". I realize that it will probably be ignored
over time, but maybe it will provide enough of a reminder if someone is
debugging and sees the warning.

What would be the point of printing a warning instead of doing just
what the user is expecting?

To me, the point would be to get the same behavior no matter which
python executable I run, and without regard to where I run it from.

But why do you care about this? What does it change *concretely*?

It reduces the number of things I have to remember which are different
based on where I'm running python (or which executable I'm running).

Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/2FN2OCYGMLU5XST44N7SFDUAI426VRB3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 28 Sep 2021 09:14:38 -0400
"Eric V. Smith"  wrote:
> On 9/28/2021 9:10 AM, Antoine Pitrou wrote:
> > On Tue, 28 Sep 2021 08:55:05 -0400
> > "Eric V. Smith"  wrote:  
> >>> So I prefer to teach everybody how to use "-X frozen_modules=off" if
> >>> they want to hack the stdlib for their greatest pleasure. I prefer
> >>> that such special use case requires an opt-in option, the special use
> >>> case is not special enough to be the default.  
> >> I agree with Victor here: I'd rather have #1.
> >>
> >> As a compromise, how about go with #1, but print a warning if python
> >> detects that it's not built with optimizations or is run from a source
> >> tree (the conditions in #2 and #3)? The warning could suggest running
> >> with "-X frozen_modules=off". I realize that it will probably be ignored
> >> over time, but maybe it will provide enough of a reminder if someone is
> >> debugging and sees the warning.  
> > What would be the point of printing a warning instead of doing just
> > what the user is expecting?  
> 
> To me, the point would be to get the same behavior no matter which 
> python executable I run, and without regard to where I run it from.

But why do you care about this?  What does it change *concretely*?


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7OAQ6IE3KX5GLVF2EUWSJR25BYEARNFA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Marc-Andre Lemburg

On 28.09.2021 14:26, Filipe Laíns wrote:
> On Tue, 2021-09-28 at 10:22 +0200, Marc-Andre Lemburg wrote:
>> On 27.09.2021 18:51, Eric Snow wrote:
>>> We've frozen most of the stdlib modules imported during "python -c
>>> pass" [1][2], to make startup a bit faster.  Import of those modules
>>> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
>>> to "off" but we'd like to default to "on".  The blocker is the impact
>>> on contributors.  I expect many will make changes to a stdlib module
>>> and then puzzle over why those changes aren't getting used.  That's an
>>> annoyance we can avoid, which is the point of this thread.
>>>
>>> Possible solutions:
>>>
>>> 1. always default to "on" (the annoyance for contributors isn't big enough?)
>>> 2. default to "on" if it's a PGO build (and "off" otherwise)
>>> 3. default to "on" unless running from the source tree
>>>
>>> Thoughts?
>>
>> #3 sounds like a good solution, but how would you detect "running
>> from the source tree" ? This sounds like you need another stat call
>> somewhere, which is what the frozen modules try to avoid.
> 
> It does. FYI, here's the sysconfig implementation
> https://github.com/python/cpython/blob/main/Lib/sysconfig.py#L146-L181
> 
> But a more efficient way to do this could be added.

Thanks. So the Setup files are used as landmarks to determine
whether Python is from a Python source tree or not.

>> I'd like to suggest adding an environment variable to enable /
>> disable the setting instead. This makes it easy to customize the
>> behavior without introducing complicated logic.
>>
> 
> From your followup reply, it seems like you are suggesting that it should be
> enabled by default, and use a env var to disable it. That would have the same
> problem regarding the annoyance of contributors.

Setting an env var in a dev environment is not really all that hard
(I always have such environments set up for all the projects I'm working
on), so it's only a mild annoyance, while not affecting all other
times Python is run by others.

The env var would also have the nice side effect of not cluttering
up the command line and making it easy to test drive frozen vs.
source code based imports.

> Is there any reason why you would prefer that over #2? That seems like the 
> best
> option to me if #3 is not feasible.

PGO optimizations have to be enabled with ./configure and are not
enabled per default, so it's rather likely that many Python binaries
out there do not have PGO enabled and would then not benefit from
the frozen modules -- even though PGO is not required for supporting
frozen modules.

I don't think combining the two features is a good idea.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Sep 28 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SKDBB3XN2FNWTIVDTOBFLEJXUFAWG6IO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Eric V. Smith

On 9/28/2021 9:10 AM, Antoine Pitrou wrote:

On Tue, 28 Sep 2021 08:55:05 -0400
"Eric V. Smith" wrote:

I agree with Victor here: I'd rather have #1.

What would be the point of printing a warning instead of doing just
what the user is expecting?

To me, the point would be to get the same behavior no matter which
python executable I run, and without regard to where I run it from.

Eric

Freezing the stdlib is a startup performance optimization. It doesn't
need to be turned on when hacking on the Python source code... And
having to type "-X frozen_modules=off" is much more of a nuisance than
losing 10 ms in startup time (which you won't even notice in most
cases).

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/IJEP37TCGR4SVRPGAHIILA42BSUKZZ2O/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/CEKYHC2Z66DAASLQEHCT4PEQ35LWUE7K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 28 Sep 2021 08:55:05 -0400
"Eric V. Smith"  wrote:
> >
> > So I prefer to teach everybody how to use "-X frozen_modules=off" if
> > they want to hack the stdlib for their greatest pleasure. I prefer
> > that such special use case requires an opt-in option, the special use
> > case is not special enough to be the default.  
> 
> I agree with Victor here: I'd rather have #1.
> 
> As a compromise, how about go with #1, but print a warning if python 
> detects that it's not built with optimizations or is run from a source 
> tree (the conditions in #2 and #3)? The warning could suggest running 
> with "-X frozen_modules=off". I realize that it will probably be ignored 
> over time, but maybe it will provide enough of a reminder if someone is 
> debugging and sees the warning.

What would be the point of printing a warning instead of doing just
what the user is expecting?

Freezing the stdlib is a startup performance optimization. It doesn't
need to be turned on when hacking on the Python source code... And
having to type "-X frozen_modules=off" is much more of a nuisance than
losing 10 ms in startup time (which you won't even notice in most
cases).

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IJEP37TCGR4SVRPGAHIILA42BSUKZZ2O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Eric V. Smith

On 9/28/2021 8:36 AM, Victor Stinner wrote:

On Mon, Sep 27, 2021 at 6:58 PM Eric Snow wrote:

We've frozen most of the stdlib modules imported during "python -c
pass" [1][2], to make startup a bit faster. Import of those modules
is controlled by "-X frozen_modules=[on|off]". Currently it defaults
to "off" but we'd like to default to "on". The blocker is the impact
on contributors. I expect many will make changes to a stdlib module
and then puzzle over why those changes aren't getting used. That's an
annoyance we can avoid, which is the point of this thread.

Possible solutions:

1. always default to "on" (the annoyance for contributors isn't big enough?)
2. default to "on" if it's a PGO build (and "off" otherwise)
3. default to "on" unless running from the source tree

Thoughts?

Honestly, for me, #1: always on, is the most reasonable choice.

I dislike when Python behaves differently depending on subtle things
like "was it built with optimizations" or "is Python started from its
source tree"?

When I built Python without optimization and/or from its source tree,
I do that to debug an issue. If the bug goes away in this case, it can
waste my time.

I agree with Victor here: I'd rather have #1.

Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/5QKEUKJSUQON6G3WOT2GS2WGLHLSQSKS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

Would it be possible to add a __file__ attribute?

Victor

On Tue, Sep 28, 2021 at 2:47 PM Pablo Galindo Salgado
 wrote:
>
> > What is the annoyance? What is different between frozen and not frozen?
>
> One interesting consequence of what Eric mentioned (They have a different 
> loader and repr.  Also, frozen modules do not
> have __file__ set (and __path__ is always []).) is that frozen modules don't 
> have a `__file__` attribute IIRC and therefore
>  tracebacks won't include the source.
>
>
> On Mon, 27 Sept 2021 at 22:31, Victor Stinner  wrote:
>>
>> Hi Eric,
>>
>> Which stdlib modules are currently frozen? If I really want to hack
>> site.py or os.py for whatever reason, I just have to use "python3 -X
>> frozen_modules=off"?
>>
>> > 1. always default to "on" (the annoyance for contributors isn't big 
>> > enough?)
>>
>> What is the annoyance? What is different between frozen and not frozen?
>>
>> Victor
>>
>> On Mon, Sep 27, 2021 at 6:58 PM Eric Snow  
>> wrote:
>> >
>> > We've frozen most of the stdlib modules imported during "python -c
>> > pass" [1][2], to make startup a bit faster.  Import of those modules
>> > is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
>> > to "off" but we'd like to default to "on".  The blocker is the impact
>> > on contributors.  I expect many will make changes to a stdlib module
>> > and then puzzle over why those changes aren't getting used.  That's an
>> > annoyance we can avoid, which is the point of this thread.
>> >
>> > Possible solutions:
>> >
>> > 1. always default to "on" (the annoyance for contributors isn't big 
>> > enough?)
>> > 2. default to "on" if it's a PGO build (and "off" otherwise)
>> > 3. default to "on" unless running from the source tree
>> >
>> > Thoughts?
>> >
>> > -eric
>> >
>> >
>> > [1] https://bugs.python.org/issue45020
>> > [2] FWIW, we may end up also freezing the modules imported for "python
>> > -m ...", along with some other commonly used modules (like argparse).
>> > That is a separate discussion.
>> > ___
>> > Python-Dev mailing list -- python-dev@python.org
>> > To unsubscribe send an email to python-dev-le...@python.org
>> > https://mail.python.org/mailman3/lists/python-dev.python.org/
>> > Message archived at 
>> > https://mail.python.org/archives/list/python-dev@python.org/message/4ESW3NNOX43DRFKLEW3IMDXDKPDMNRGR/
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>>
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/CLODS7B5Z3UEZTQ7QIALG2DWB4H37EWP/
>> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CRL7C7I2VHGI56TFJMK6U53LCKRRHDA3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Pablo Galindo Salgado

> What is the annoyance? What is different between frozen and not frozen?

One interesting consequence of what Eric mentioned (They have a different
loader and repr.  Also, frozen modules do not
have __file__ set (and __path__ is always []).) is that frozen modules
don't have a `__file__` attribute IIRC and therefore
 tracebacks won't include the source.


On Mon, 27 Sept 2021 at 22:31, Victor Stinner  wrote:

> Hi Eric,
>
> Which stdlib modules are currently frozen? If I really want to hack
> site.py or os.py for whatever reason, I just have to use "python3 -X
> frozen_modules=off"?
>
> > 1. always default to "on" (the annoyance for contributors isn't big
> enough?)
>
> What is the annoyance? What is different between frozen and not frozen?
>
> Victor
>
> On Mon, Sep 27, 2021 at 6:58 PM Eric Snow 
> wrote:
> >
> > We've frozen most of the stdlib modules imported during "python -c
> > pass" [1][2], to make startup a bit faster.  Import of those modules
> > is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> > to "off" but we'd like to default to "on".  The blocker is the impact
> > on contributors.  I expect many will make changes to a stdlib module
> > and then puzzle over why those changes aren't getting used.  That's an
> > annoyance we can avoid, which is the point of this thread.
> >
> > Possible solutions:
> >
> > 1. always default to "on" (the annoyance for contributors isn't big
> enough?)
> > 2. default to "on" if it's a PGO build (and "off" otherwise)
> > 3. default to "on" unless running from the source tree
> >
> > Thoughts?
> >
> > -eric
> >
> >
> > [1] https://bugs.python.org/issue45020
> > [2] FWIW, we may end up also freezing the modules imported for "python
> > -m ...", along with some other commonly used modules (like argparse).
> > That is a separate discussion.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4ESW3NNOX43DRFKLEW3IMDXDKPDMNRGR/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CLODS7B5Z3UEZTQ7QIALG2DWB4H37EWP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TDPS44OPEXK7MX3U2PMKTYLXNMQMA3MB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 6:58 PM Eric Snow  wrote:
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
>
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
>
> Thoughts?

Honestly, for me, #1: always on, is the most reasonable choice.

I dislike when Python behaves differently depending on subtle things
like "was it built with optimizations" or "is Python started from its
source tree"?

When I built Python without optimization and/or from its source tree,
I do that to debug an issue. If the bug goes away in this case, it can
waste my time.

So I prefer to teach everybody how to use "-X frozen_modules=off" if
they want to hack the stdlib for their greatest pleasure. I prefer
that such special use case requires an opt-in option, the special use
case is not special enough to be the default.

--

It means that the site module module can no longer be "customized" by
modifying directly the site.py file (inject a path in PYTHONPATH env
var where the customized site.py lives). But there is already a
supported way to customize the site module: create a module named
"sitecustomize" or "usercustomizer". I recall that virtualenv likes to
override stdlib site.py with its own code. tox uses virtualenv by
default. Someone should check if freezing site doesn't break
virtualenv and tox, since they seem to be popular in Python. The venv
doesn't need to override site.py and tox can use venv if I recall
correctly.

If site.py customization is too popular, I would suggest to not freeze
this one, until the community stops doing that.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FWAYUPG5DRRHFULNPXQOZF3NR6QZ37K2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Ronald Oussoren via Python-Dev



> On 28 Sep 2021, at 10:54, Antoine Pitrou  wrote:
> 
> On Tue, 28 Sep 2021 10:51:53 +0200
> Ronald Oussoren via Python-Dev  wrote:
>>> On 28 Sep 2021, at 10:05, Antoine Pitrou  wrote:
>>> 
>>> On Mon, 27 Sep 2021 10:51:43 -0600
>>> Eric Snow >> > wrote:  
 We've frozen most of the stdlib modules imported during "python -c
 pass" [1][2], to make startup a bit faster.  Import of those modules
 is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
 to "off" but we'd like to default to "on".  The blocker is the impact
 on contributors.  I expect many will make changes to a stdlib module
 and then puzzle over why those changes aren't getting used.  That's an
 annoyance we can avoid, which is the point of this thread.
 
 Possible solutions:
 
 1. always default to "on" (the annoyance for contributors isn't big 
 enough?)
 2. default to "on" if it's a PGO build (and "off" otherwise)
 3. default to "on" unless running from the source tree
 
 Thoughts?  
>>> 
>>> My vote is on #3 to minimize contributor annoyance and
>>> eventual puzzlement.  
>> 
>> I agree, but… Most CPython tests are run while running from the source tree, 
>> that means that there will have to be testrunner configurations that run 
>> with “-X frozen_modules=on”. 
> 
> Well, multiplying CI configurations is the price of adding options in
> general.


Of course. I mentioned it because the proposal is to add a new option that’s 
enabled after installation, and basically not when the testsuite is run.  
That’s not a problem, we could just enable the option in most CI jobs.

Ronald

> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/R6C5XNYBV7LGUWRAL4X4OQDABRKQW42C/
> Code of Conduct: http://python.org/psf/codeofconduct/

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6YXGAXLK76MQECMBP32ZBZ5GFDQKCHPW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

2021-09-28 Thread Phil Thompson via Python-Dev

On 27/09/2021 21:53, Brett Cannon wrote:

On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev <
python-dev@python.org> wrote:

On 26/09/2021 05:21, Steven D'Aprano wrote:

[snip]

> These are not rhetorical questions, I genuinely do not know. I *think*
> that there was an attempt to make a stable C API back in 3.2 days:
>
> https://www.python.org/dev/peps/pep-0384/
>
> but I don't know what difference it has made to extension writers in
> practice. From your description, it sounds like perhaps not as big a
> difference as we would have liked.
>
> Maybe extension writers are not using the stable C API? Is that even
> possible? Please excuse my ignorance.

PyQt has used the stable ABI for many years. The main reason for using
it is to reduce the number of wheels. The PyQt ecosystem currently
contains 15 PyPI projects across 4 platforms supporting 5 Python
versions (including v3.10). Without the stable ABI a new release would
require 300 wheels. With the stable ABI it is a more manageable 60
wheels.

However the stable ABI is still a second class citizen as it is still
not possible (AFAIK) to specify a wheel name that doesn't need to
explicitly include each supported Python version (rather than a 
minimum

stable ABI version).

Actually you can do this. The list of compatible wheels for a platform
starts at CPython 3.2 when the stable ABI was introduced and goes 
forward

to the version of Python you are running. So you can build a wheel file
that targets the oldest version of CPython that you are targeting and 
its

version of the stable ABI and it is considered forward compatible. See
`python -m pip debug --verbose` for the complete list of wheel tags 
that

are supported for an interpreter.

Logical and it works.

Many thanks,
Phil
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/22MU4QKR46SMFQWQFPWUIWM76JYJMJ3L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Ronald Oussoren via Python-Dev



> On 28 Sep 2021, at 10:05, Antoine Pitrou  wrote:
> 
> On Mon, 27 Sep 2021 10:51:43 -0600
> Eric Snow mailto:ericsnowcurren...@gmail.com>> 
> wrote:
>> We've frozen most of the stdlib modules imported during "python -c
>> pass" [1][2], to make startup a bit faster.  Import of those modules
>> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
>> to "off" but we'd like to default to "on".  The blocker is the impact
>> on contributors.  I expect many will make changes to a stdlib module
>> and then puzzle over why those changes aren't getting used.  That's an
>> annoyance we can avoid, which is the point of this thread.
>> 
>> Possible solutions:
>> 
>> 1. always default to "on" (the annoyance for contributors isn't big enough?)
>> 2. default to "on" if it's a PGO build (and "off" otherwise)
>> 3. default to "on" unless running from the source tree
>> 
>> Thoughts?
> 
> My vote is on #3 to minimize contributor annoyance and
> eventual puzzlement.

I agree, but… Most CPython tests are run while running from the source tree, 
that means that there will have to be testrunner configurations that run with 
“-X frozen_modules=on”. 

Ronald

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NPID5HAZAYOZ6J4OYHN7O4UNEWXMR7WS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 28 Sep 2021 10:51:53 +0200
Ronald Oussoren via Python-Dev  wrote:
> > On 28 Sep 2021, at 10:05, Antoine Pitrou  wrote:
> > 
> > On Mon, 27 Sep 2021 10:51:43 -0600
> > Eric Snow  > > wrote:  
> >> We've frozen most of the stdlib modules imported during "python -c
> >> pass" [1][2], to make startup a bit faster.  Import of those modules
> >> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> >> to "off" but we'd like to default to "on".  The blocker is the impact
> >> on contributors.  I expect many will make changes to a stdlib module
> >> and then puzzle over why those changes aren't getting used.  That's an
> >> annoyance we can avoid, which is the point of this thread.
> >> 
> >> Possible solutions:
> >> 
> >> 1. always default to "on" (the annoyance for contributors isn't big 
> >> enough?)
> >> 2. default to "on" if it's a PGO build (and "off" otherwise)
> >> 3. default to "on" unless running from the source tree
> >> 
> >> Thoughts?  
> > 
> > My vote is on #3 to minimize contributor annoyance and
> > eventual puzzlement.  
> 
> I agree, but… Most CPython tests are run while running from the source tree, 
> that means that there will have to be testrunner configurations that run with 
> “-X frozen_modules=on”. 

Well, multiplying CI configurations is the price of adding options in
general.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R6C5XNYBV7LGUWRAL4X4OQDABRKQW42C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Marc-Andre Lemburg

On 27.09.2021 18:51, Eric Snow wrote:
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
> 
> Possible solutions:
> 
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
> 
> Thoughts?

#3 sounds like a good solution, but how would you detect "running
from the source tree" ? This sounds like you need another stat call
somewhere, which is what the frozen modules try to avoid.

I'd like to suggest adding an environment variable to enable /
disable the setting instead. This makes it easy to customize the
behavior without introducing complicated logic.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Sep 28 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FLFBNODDQOGFNKXOMFBQKTE5AQHLA7LS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Marc-Andre Lemburg

On 28.09.2021 10:22, Marc-Andre Lemburg wrote:
> On 27.09.2021 18:51, Eric Snow wrote:
>> We've frozen most of the stdlib modules imported during "python -c
>> pass" [1][2], to make startup a bit faster.  Import of those modules
>> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
>> to "off" but we'd like to default to "on".  The blocker is the impact
>> on contributors.  I expect many will make changes to a stdlib module
>> and then puzzle over why those changes aren't getting used.  That's an
>> annoyance we can avoid, which is the point of this thread.
>>
>> Possible solutions:
>>
>> 1. always default to "on" (the annoyance for contributors isn't big enough?)
>> 2. default to "on" if it's a PGO build (and "off" otherwise)
>> 3. default to "on" unless running from the source tree
>>
>> Thoughts?
> 
> #3 sounds like a good solution, but how would you detect "running
> from the source tree" ? This sounds like you need another stat call
> somewhere, which is what the frozen modules try to avoid.
> 
> I'd like to suggest adding an environment variable to enable /
> disable the setting instead. This makes it easy to customize the
> behavior without introducing complicated logic.

Just to clarify: the modules would still always be frozen with
the env var setting, but Python would simply not import them
as frozen modules, but instead go and look on the PYTHONPATH
for the modules.

This could be achieved by special casing the frozen module
finder function to only trigger on importlib modules and
return NULL for all other possibly frozen modules.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Sep 28 2021)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XVEVVUFUXNDM7LQPWPPDOWITW2HXMPPK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

2021-09-28 Thread Nathaniel Smith

On Tue, Sep 28, 2021 at 12:40 AM Guido van Rossum  wrote:
>
> What I have heard repeatedly, from people who are paid to know, is that most 
> users don’t care about the latest features, and would rather stick to a 
> release until it becomes unsupported. (Extreme example: Python 2.)
>
> Numpy isn’t random, it’s at the bottom of the food chain for a large 
> ecosystem or two — if it doesn’t support a new Python release, none of its 
> dependent packages can even start porting. (I guess only Cython is even 
> lower, but it’s a build-time tool. And indeed it has supported 3.10 for a 
> long time.)

Well, no, it wasn't entirely random :-).

Being on the bottom of the food chain is important, but I don't think
it's the full story -- Tensorflow is also at the bottom of a huge
ecosystem. I think it's also related to NumPy being mostly
volunteer-run, which means they're sensitive to feedback from
individual enthusiasts, and enthusiasts are the most aggressive early
adopters. OTOH Tensorflow is a huge commercial collaboration, and
companies *hate* upgrading.

Either way though, it doesn't seem to be anything to do with CPython's
ABI stability or release cadence.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UNE7TY7ZFGBOOOG3H3YNB5MFGX5IWZ76/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.