[Python-Dev] Re: Python release announcement format

[Gregory P. Smith]

On Tue, Dec 14, 2021 at 9:06 AM Yann Droneaud  wrote:

> Hi,
>
> I'm not familiar with the Python release process, but looking at the latest 
> release
> https://www.python.org/downloads/release/python-3101/
>
> we can see MD5 is still used ... which doesn't sound right in 2021 ...
> especially since we proved it's possible to build different .tar.gz that have
> the same MD5
>
> https://twitter.com/ydroneaud/status/1448659749604446211https://twitter.com/angealbertini/status/1449736035110461443
>
> You would reply there's OpenPGP / GnuPG signature. But then I would like to 
> raise
> another issue regarding the release process:
>
> As the announcement on comp.lang.python.announce / 
> python-announce-l...@python.org
> doesn't record the release digest / release signature, the operator 
> behindhttps://www.python.org/downloads/release/python-3101/ are free to 
> change the release
> content at any time, provided there's a valid signature. And there will no 
> way for
> us to check the release wasn't modified after the announcement.
>
>
For source archives, one can diff the contents of the source download vs
those of the equivalent tag in the git repository. For binaries, well,
there's already a ton of trust involved in accepting a binary from anyone.
But agreed having the currently secure hashes in the announcement email
would be good.


> It would be great if https://www.python.org/dev/peps/pep-0101/ would be 
> improved
> from the naive:
>
>  "Write the announcement for the mailing lists.  This is the fuzzy bit 
> because not
>   much can be automated.  You can use an earlier announcement as a template, 
> but
>   edit it for content!"
>
> to require the release announcement to record release archives digests as 
> SHA-2 256
> (added point if the announcement is signed), or the armored OpenPGP 
> signatures (but's
> that a lot of base64 characters).
>
> Should I open a bug for this issue ?
>
>
Makes sense, it is a pretty small change to make to the announcement
format. Filed. https://bugs.python.org/issue46077

-gps

Regards.
>
> --
> Yann Droneaud
> OPTEYA
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6NI6V7DHTXCTUTNC2C5YSGOB6UJRFUDR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OXS2TK43QKH2M54R5HHECOZ6HYCQGJON/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Nathaniel Smith]

Whoops, never mind, I see that you started the "immortal objects"
thread to discuss this.

On Tue, Dec 14, 2021 at 4:54 PM Nathaniel Smith  wrote:
>
> How did you end up solving the issue where Py_None is a static global
> that's exposed as part of the stable C ABI?
>
> On Tue, Dec 14, 2021 at 9:13 AM Eric Snow  wrote:
> >
> > Hi all,
> >
> > I'm still hoping to land a per-interpreter GIL for 3.11.  There is
> > still a decent amount of work to be done but little of it will require
> > solving any big problems:
> >
> > * pull remaining static globals into _PyRuntimeState and PyInterpreterState
> > * minor updates to PEP 554
> > * finish up the last couple pieces of the PEP 554 implementation
> > * maybe publish a companion PEP about per-interpreter GIL
> >
> > There are also a few decisions to be made.  I'll open a couple of
> > other threads to get feedback on those.  Here I'd like your thoughts
> > on the following:
> >
> > Do we need a PEP about per-interpreter GIL?
> >
> > I haven't thought there would be much value in such a PEP.  There
> > doesn't seem to be any decision that needs to be made.  At best the
> > PEP would be an explanation of the project, where:
> >
> > * the objective has gotten a lot of support (and we're working on
> > addressing the concerns of the few objectors)
> > * most of the required work is worth doing regardless (e.g. improve
> > runtime init/fini, eliminate static globals)
> > * the performance impact is likely to be a net improvement
> > * it is fully backward compatible and the C-API is essentially unaffected
> >
> > So the value of a PEP would be in consolidating an explanation of the
> > project into a single document.  It seems like a poor fit for a PEP.
> >
> > (You might wonder, "what about PEP 554?"  I purposefully avoided any
> > discussion of the GIL in PEP 554.  It's purpose is to expose
> > subinterpreters to Python code.)
> >
> > However, perhaps I'm too close to it all.  I'd like your thoughts on the 
> > matter.
> >
> > Thanks!
> >
> > -eric
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-dev@python.org/message/PNLBJBNIQDMG2YYGPBCTGOKOAVXRBJWY/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QDXHLDVHIW4B3GN7PNEL4LPZRJVGGW2R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Nathaniel Smith]

How did you end up solving the issue where Py_None is a static global
that's exposed as part of the stable C ABI?

On Tue, Dec 14, 2021 at 9:13 AM Eric Snow  wrote:
>
> Hi all,
>
> I'm still hoping to land a per-interpreter GIL for 3.11.  There is
> still a decent amount of work to be done but little of it will require
> solving any big problems:
>
> * pull remaining static globals into _PyRuntimeState and PyInterpreterState
> * minor updates to PEP 554
> * finish up the last couple pieces of the PEP 554 implementation
> * maybe publish a companion PEP about per-interpreter GIL
>
> There are also a few decisions to be made.  I'll open a couple of
> other threads to get feedback on those.  Here I'd like your thoughts
> on the following:
>
> Do we need a PEP about per-interpreter GIL?
>
> I haven't thought there would be much value in such a PEP.  There
> doesn't seem to be any decision that needs to be made.  At best the
> PEP would be an explanation of the project, where:
>
> * the objective has gotten a lot of support (and we're working on
> addressing the concerns of the few objectors)
> * most of the required work is worth doing regardless (e.g. improve
> runtime init/fini, eliminate static globals)
> * the performance impact is likely to be a net improvement
> * it is fully backward compatible and the C-API is essentially unaffected
>
> So the value of a PEP would be in consolidating an explanation of the
> project into a single document.  It seems like a poor fit for a PEP.
>
> (You might wonder, "what about PEP 554?"  I purposefully avoided any
> discussion of the GIL in PEP 554.  It's purpose is to expose
> subinterpreters to Python code.)
>
> However, perhaps I'm too close to it all.  I'd like your thoughts on the 
> matter.
>
> Thanks!
>
> -eric
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/PNLBJBNIQDMG2YYGPBCTGOKOAVXRBJWY/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KRGRUHLHKBN5QQUY4VBO33DUS2Z2H5A4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

[Eric Snow]

On Tue, Dec 14, 2021 at 4:09 PM Brett Cannon  wrote:

> There's also the concern of memory usage if these immortal objects are never 
> collected.
>
> But which objects are immortal? You only listed None, True, and False. 
> Otherwise assume/remember I'm management and provide a list and/or link of 
> what would get marked as immortal so we can have an idea of the memory impact.

Pretty much we would mark any object as immortal which would exist for
the lifetype of the runtime (or the respective interpreter in some
cases).  So currently that would include the global singletons (None,
True, False, small ints, empty tuple, etc.) and the static types.  We
would likely also include cached strings (_Py_Identifier, interned,
etc.).

>From another angle: I'm working on static allocation for nearly all
the objects currently dynamically allocated during runtime/interpreter
init.  All of them would be marked immortal.  This is similar to the
approach taken by Eddie with walking the heap and marking all objects
found.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JKRY6FQYZIFFYQ64BSKLFGWUKX74NZ7M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

[Eric Snow]

Yeah, no (mutable) global state at the C level.  It would also be good
to implement multi-phase init (PEP 489), but I don't expect that to
require much work itself.

-eric

On Tue, Dec 14, 2021 at 4:04 PM Brett Cannon  wrote:
>
>
>
> On Tue, Dec 14, 2021 at 9:41 AM Eric Snow  wrote:
>>
>> One of the open questions relative to subinterpreters is: how to
>> reduce the amount of work required for extension modules to support
>> them?  Thanks to Petr Viktorin for a lot of work he's done in this
>> area (e.g. PEP 489)!  Extensions also have the option to opt out of
>> subinterpreter support.
>>
>> However, that's only one part of the story.  A while back Nathaniel
>> expressed concerns with how making subinterpreters more accessible
>> will have a negative side effect affecting projects that publish large
>> extensions, e.g. numpy.  Not all extensions support subinterpreters
>> due to global state (incl. in library dependencies).  The amount of
>> work to get there may be large.  As subinterpreters increase in usage
>> in the community, so will demand increase for subinterpreter support
>> in those extensions.  Consequently, such projects be pressured to do
>> the extra work (which is made even more stressful by the short-handed
>> nature of most open source projects) .
>>
>> So we (the core devs) would effectively be requiring those extensions
>> to support subinterpreters, regardless of letting them opt out.  This
>> situation has been weighing heavily on my mind since Nathaniel brought
>> this up.  Here are some ideas I've had or heard of about what we could
>> do to help:
>>
>> * add a page to the C-API documentation about how to support subinterpreters
>> * identify the extensions most likely to be impacted and offer to help
>> * add more helpers to the C-API to make adding subinterpreter support
>> less painful
>> * fall back to loading the extension in its own namespace (e.g. use 
>> ldm_open())
>> * fall back to copying the extension's file and loading from the copied file
>> * ...
>>
>>  I'd appreciate your thoughts on what we can do to help.  Thanks!
>
>
> What are the requirements put upon an extension in order to support 
> subinterpreters? you hint at global state at the C level, but nothing else is 
> mentioned. Is that it?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BQU3PVN6MHR2P24RAUPJSWFS547W7FPM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python release announcement format

[Steven D'Aprano]

On Tue, Dec 14, 2021 at 11:56:09AM +0100, Yann Droneaud wrote:
> Hi,
> 
> I'm not familiar with the Python release process, but looking at the latest 
> release
> 
> https://www.python.org/downloads/release/python-3101/
> 
> we can see MD5 is still used ... which doesn't sound right in 2021 ...
> especially since we proved it's possible to build different .tar.gz that 
> have the same MD5

Please ignore my ignorant question, but what is the actual threat you 
are concerned about here?

Anyone who gains access to the Python web site sufficient to smuggle in 
a different .tar.gz file will surely also be able to just change the 
checksum on the website to whatever they want. It doesn't matter what 
the checksum is or how strong it is, MD5 or SHA256 or any other 
checksum. An attacker who can change the file can change the checksum to 
match it.

And both the download and the webpage listing the checksum are over 
https. If we don't trust https, the whole internet is broken and 
changing to a stronger checksum won't help. A hypothetical MITM attacker 
capable of breaking https and injecting new content into the download 
file can likewise change the checksum.

Checksums are, as far as I understand it, only good for verifying data 
integrity against download errors, not malicious attack. And for that, 
MD5 is more than sufficient.

Have I missed something?


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IFBRCJ3LKLV3MINIFJF5WO6QHN54ERVC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: "immortal" objects and how they would help per-interpreter GIL

[Brett Cannon]

On Tue, Dec 14, 2021 at 10:23 AM Eric Snow 
wrote:

> Most of the work toward interpreter isolation and a per-interpreter
> GIL involves moving static global variables to _PyRuntimeState or
> PyInterpreterState (or module state).  Through the effort of quite a
> few people, we've made good progress.  However, many globals still
> remain, with the majority being objects and most of those being static
> strings (e.g. _Py_Identifier), static types (incl. exceptions), and
> singletons.
>
> On top of that, a number of those objects are exposed in the public
> C-API and even in the limited API. :(  Dealing with this specifically
> is probably the trickiest thing I've had to work through in this
> project.
>
> There is one solution that would help both of the above in a nice way:
> "immortal" objects.
>
> The idea of objects that never get deallocated isn't new and has been
> explored here several times.  Not that long ago I tried it out by
> setting the refcount really high.  That worked.  Around the same time
> Eddie Elizondo at Facebook did something similar but modified
> Py_INCREF() and Py_DECREF() to keep the refcount from changing.  Our
> solutions were similar but with different goals in mind.  (Facebook
> wants to avoid copy-on-write in their pre-fork model.)
>
> A while back I concluded that neither approach would work for us.  The
> approach I had taken would have significant cache performance
> penalties in a per-interpreter GIL world.  The approach that modifies
> Py_INCREF() has a significant performance penalty due to the extra
> branch on such a frequent operation.
>
> Recently I've come back to the idea of immortal objects because it's
> much simpler than the alternate (working) solution I found.  So how do
> we get around that performance penalty?  Let's say it makes CPython 5%
> slower.  We have some options:
>
> * live with the full penalty
> * make other changes to reduce the penalty to a more acceptable
> threshold than 5%
> * eliminate the penalty (e.g. claw back 5% elsewhere)
> * abandon all hope
>
> Mark Shannon suggested to me some things we can do.  Also, from a
> recent conversation with Dino Viehland it sounds like Eddie was able
> to reach performance-neutral with a few techniques.  So here are some
> things we can do to reduce or eliminate that penalty:
>
> * reduce refcount operations on high-activity objects (e.g. None, True,
> False)
> * reduce refcount operations in general
> * walk the heap at the end of runtime initialization and mark all
> objects as immortal
> * mark all global objects as immortal (statics or in _PyRuntimeState;
> for PyInterpreterState not needed)
>
> What do you think?  Does this sound realistic?  Are there additional
> things we can do to counter that penalty?
>

There's also the concern of memory usage if these immortal objects are
never collected.

But *which *objects are immortal? You only listed None, True, and False.
Otherwise assume/remember I'm management and provide a list and/or link of
what would get marked as immortal so we can have an idea of the memory
impact.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XSN7EFFQHGG4PS4JVXRXT7Q626ZGIKHF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: subinterpreters and their possible impact on large extension projects

[Brett Cannon]

On Tue, Dec 14, 2021 at 9:41 AM Eric Snow 
wrote:

> One of the open questions relative to subinterpreters is: how to
> reduce the amount of work required for extension modules to support
> them?  Thanks to Petr Viktorin for a lot of work he's done in this
> area (e.g. PEP 489)!  Extensions also have the option to opt out of
> subinterpreter support.
>
> However, that's only one part of the story.  A while back Nathaniel
> expressed concerns with how making subinterpreters more accessible
> will have a negative side effect affecting projects that publish large
> extensions, e.g. numpy.  Not all extensions support subinterpreters
> due to global state (incl. in library dependencies).  The amount of
> work to get there may be large.  As subinterpreters increase in usage
> in the community, so will demand increase for subinterpreter support
> in those extensions.  Consequently, such projects be pressured to do
> the extra work (which is made even more stressful by the short-handed
> nature of most open source projects) .
>
> So we (the core devs) would effectively be requiring those extensions
> to support subinterpreters, regardless of letting them opt out.  This
> situation has been weighing heavily on my mind since Nathaniel brought
> this up.  Here are some ideas I've had or heard of about what we could
> do to help:
>
> * add a page to the C-API documentation about how to support
> subinterpreters
> * identify the extensions most likely to be impacted and offer to help
> * add more helpers to the C-API to make adding subinterpreter support
> less painful
> * fall back to loading the extension in its own namespace (e.g. use
> ldm_open())
> * fall back to copying the extension's file and loading from the copied
> file
> * ...
>
>  I'd appreciate your thoughts on what we can do to help.  Thanks!
>

What *are* the requirements put upon an extension in order to support
subinterpreters? you hint at global state at the C level, but nothing else
is mentioned. Is that it?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S4QV6SYRGEN7IZZX6YBLS3DQRNLRGCKH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python release announcement format

[Christian Heimes]

On 14/12/2021 11.56, Yann Droneaud wrote:

Hi,

I'm not familiar with the Python release process, but looking at the latest 
release

https://www.python.org/downloads/release/python-3101/

we can see MD5 is still used ... which doesn't sound right in 2021 ...
especially since we proved it's possible to build different .tar.gz that have
the same MD5

https://twitter.com/ydroneaud/status/1448659749604446211
https://twitter.com/angealbertini/status/1449736035110461443

You would reply there's OpenPGP / GnuPG signature. But then I would like to 
raise
another issue regarding the release process:

As the announcement on comp.lang.python.announce 
/python-announce-l...@python.org
doesn't record the release digest / release signature, the operator behind
https://www.python.org/downloads/release/python-3101/  are free to change the 
release
content at any time, provided there's a valid signature. And there will no way 
for
us to check the release wasn't modified after the announcement.

It would be great ifhttps://www.python.org/dev/peps/pep-0101/  would be improved
from the naive:

  "Write the announcement for the mailing lists.  This is the fuzzy bit because 
not
   much can be automated.  You can use an earlier announcement as a template, 
but
   edit it for content!"

to require the release announcement to record release archives digests as SHA-2 
256
(added point if the announcement is signed), or the armored OpenPGP signatures 
(but's
that a lot of base64 characters).



I would also argue that OpenPGP signatures are a bad solution in 2021. 
PGP has not aged well and GnuPG tool has flaws. Better, more modern 
options like sigstore are still under development, though.


We could (and maybe should) provide a SHA256 tag file (sha256sum --tag) 
and sign it with OpenGPG. The signature of a sha256 checksum file is as 
good as signing the files directly.


Christian
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FEQAD752SIWTOBMLVOP2JJV3RFPRJBD4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Sad news from Zurich

[Ivan Van Laningham]

Fredrik was a lovely man who was also essential to the development of
Python as it is today.  I'm sorry that I only met him once, and I will miss
him greatly.

Metta,
Ivan
-- 
Ivan Van Laningham
God N Locomotive Works
https://legacy.python.org/workshops/1998-11/proceedings/papers/laningham/laningham.html

Army Signal Corps:  Cu Chi, Class of '70
Author:  Teach Yourself Python in 24 Hours
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MPU2I4RPWKVYZGSN37V7Y3QYYUZDD46I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should dataclasses add__set__ (and possibly __get __) descriptors ?

[Vioshim]

Oh certainly that's the approach I was intending, basically wanted to make an 
enum that shares the methods that the original class can do, but this is quite 
helpful, I've been testing this approach and seems to work very well despite 
only working with *args, thanks a lot for the reply and everyone's feedback as 
now I have a full understanding on how this works.

Have a great day.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NB3HGBWY5KDBDZ4S2Z3WO4BHZOLZQHOP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] "immortal" objects and how they would help per-interpreter GIL

[Eric Snow]

Most of the work toward interpreter isolation and a per-interpreter
GIL involves moving static global variables to _PyRuntimeState or
PyInterpreterState (or module state).  Through the effort of quite a
few people, we've made good progress.  However, many globals still
remain, with the majority being objects and most of those being static
strings (e.g. _Py_Identifier), static types (incl. exceptions), and
singletons.

On top of that, a number of those objects are exposed in the public
C-API and even in the limited API. :(  Dealing with this specifically
is probably the trickiest thing I've had to work through in this
project.

There is one solution that would help both of the above in a nice way:
"immortal" objects.

The idea of objects that never get deallocated isn't new and has been
explored here several times.  Not that long ago I tried it out by
setting the refcount really high.  That worked.  Around the same time
Eddie Elizondo at Facebook did something similar but modified
Py_INCREF() and Py_DECREF() to keep the refcount from changing.  Our
solutions were similar but with different goals in mind.  (Facebook
wants to avoid copy-on-write in their pre-fork model.)

A while back I concluded that neither approach would work for us.  The
approach I had taken would have significant cache performance
penalties in a per-interpreter GIL world.  The approach that modifies
Py_INCREF() has a significant performance penalty due to the extra
branch on such a frequent operation.

Recently I've come back to the idea of immortal objects because it's
much simpler than the alternate (working) solution I found.  So how do
we get around that performance penalty?  Let's say it makes CPython 5%
slower.  We have some options:

* live with the full penalty
* make other changes to reduce the penalty to a more acceptable
threshold than 5%
* eliminate the penalty (e.g. claw back 5% elsewhere)
* abandon all hope

Mark Shannon suggested to me some things we can do.  Also, from a
recent conversation with Dino Viehland it sounds like Eddie was able
to reach performance-neutral with a few techniques.  So here are some
things we can do to reduce or eliminate that penalty:

* reduce refcount operations on high-activity objects (e.g. None, True, False)
* reduce refcount operations in general
* walk the heap at the end of runtime initialization and mark all
objects as immortal
* mark all global objects as immortal (statics or in _PyRuntimeState;
for PyInterpreterState not needed)

What do you think?  Does this sound realistic?  Are there additional
things we can do to counter that penalty?

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7O3FUA52QGTVDC6MDAV5WXKNFEDRK5D6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] subinterpreters and their possible impact on large extension projects

[Eric Snow]

One of the open questions relative to subinterpreters is: how to
reduce the amount of work required for extension modules to support
them?  Thanks to Petr Viktorin for a lot of work he's done in this
area (e.g. PEP 489)!  Extensions also have the option to opt out of
subinterpreter support.

However, that's only one part of the story.  A while back Nathaniel
expressed concerns with how making subinterpreters more accessible
will have a negative side effect affecting projects that publish large
extensions, e.g. numpy.  Not all extensions support subinterpreters
due to global state (incl. in library dependencies).  The amount of
work to get there may be large.  As subinterpreters increase in usage
in the community, so will demand increase for subinterpreter support
in those extensions.  Consequently, such projects be pressured to do
the extra work (which is made even more stressful by the short-handed
nature of most open source projects) .

So we (the core devs) would effectively be requiring those extensions
to support subinterpreters, regardless of letting them opt out.  This
situation has been weighing heavily on my mind since Nathaniel brought
this up.  Here are some ideas I've had or heard of about what we could
do to help:

* add a page to the C-API documentation about how to support subinterpreters
* identify the extensions most likely to be impacted and offer to help
* add more helpers to the C-API to make adding subinterpreter support
less painful
* fall back to loading the extension in its own namespace (e.g. use ldm_open())
* fall back to copying the extension's file and loading from the copied file
* ...

 I'd appreciate your thoughts on what we can do to help.  Thanks!

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X3ZOSP2A4RTSKTBZ4XYHROSJBONCEDID/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] my plans for subinterpreters (and a per-interpreter GIL)

[Eric Snow]

Hi all,

I'm still hoping to land a per-interpreter GIL for 3.11.  There is
still a decent amount of work to be done but little of it will require
solving any big problems:

* pull remaining static globals into _PyRuntimeState and PyInterpreterState
* minor updates to PEP 554
* finish up the last couple pieces of the PEP 554 implementation
* maybe publish a companion PEP about per-interpreter GIL

There are also a few decisions to be made.  I'll open a couple of
other threads to get feedback on those.  Here I'd like your thoughts
on the following:

Do we need a PEP about per-interpreter GIL?

I haven't thought there would be much value in such a PEP.  There
doesn't seem to be any decision that needs to be made.  At best the
PEP would be an explanation of the project, where:

* the objective has gotten a lot of support (and we're working on
addressing the concerns of the few objectors)
* most of the required work is worth doing regardless (e.g. improve
runtime init/fini, eliminate static globals)
* the performance impact is likely to be a net improvement
* it is fully backward compatible and the C-API is essentially unaffected

So the value of a PEP would be in consolidating an explanation of the
project into a single document.  It seems like a poor fit for a PEP.

(You might wonder, "what about PEP 554?"  I purposefully avoided any
discussion of the GIL in PEP 554.  It's purpose is to expose
subinterpreters to Python code.)

However, perhaps I'm too close to it all.  I'd like your thoughts on the matter.

Thanks!

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PNLBJBNIQDMG2YYGPBCTGOKOAVXRBJWY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python release announcement format

[Yann Droneaud]

Hi,

I'm not familiar with the Python release process, but looking at the latest 
release

https://www.python.org/downloads/release/python-3101/

we can see MD5 is still used ... which doesn't sound right in 2021 ...
especially since we proved it's possible to build different .tar.gz that have
the same MD5

https://twitter.com/ydroneaud/status/1448659749604446211
https://twitter.com/angealbertini/status/1449736035110461443

You would reply there's OpenPGP / GnuPG signature. But then I would like to 
raise
another issue regarding the release process:

As the announcement on comp.lang.python.announce 
/python-announce-l...@python.org
doesn't record the release digest / release signature, the operator behind
https://www.python.org/downloads/release/python-3101/  are free to change the 
release
content at any time, provided there's a valid signature. And there will no way 
for
us to check the release wasn't modified after the announcement.

It would be great ifhttps://www.python.org/dev/peps/pep-0101/  would be improved
from the naive:

 "Write the announcement for the mailing lists.  This is the fuzzy bit because 
not
  much can be automated.  You can use an earlier announcement as a template, but
  edit it for content!"

to require the release announcement to record release archives digests as SHA-2 
256
(added point if the announcement is signed), or the armored OpenPGP signatures 
(but's
that a lot of base64 characters).

Should I open a bug for this issue ?

Regards.

--
Yann Droneaud
OPTEYA

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6NI6V7DHTXCTUTNC2C5YSGOB6UJRFUDR/
Code of Conduct: http://python.org/psf/codeofconduct/