[Python-Dev] Re: Python Software Foundation Board of Directors Election 2022

[Nick Coghlan]

Thanks for that heads up, Tim.

Gmail hadn't pushed mine to Spam, but it had squirrelled it away in
the Inbox "Updates" tab, so I could easily have missed it.

Cheers,
Nick.

On Mon, 20 Jun 2022 at 03:32, Tim Peters  wrote:
>
> Heads up! I found my PSF Board voting info in my gmail spam folder
> today; looks like it was mailed out this morning.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/UDX2JHQZHICWWWMUGD3LBH32ABHWVCZC/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OPVWQGEARJNG4M5XMQDHILO2ESOHREAX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 689 – Unstable C API tier (was: Semi-stable C API tier)

[Nick Coghlan]

On Fri, 6 May 2022 at 22:50, Petr Viktorin  wrote:
> IMO it's fine to delay formalizing this until 3.12, so we can have a
> proper discussion.
> However, if someone has the time, it would be nice to ensure the use
> cases from PEP 523 are possible with the 3.11 API.
> Specifically, AFAIK, struct _PyInterpreterFrame needs to be exposed
> (as an incomplete, opaque struct) to make _PyFrameEvalFunction usable,
> _PyFrame_GetFrameObject needs to be exposed, and a
> PyEval_EvalFrameDefault that takes PyFrameObject should be added.
> Also, the What's New currently says “See PEP 523 for more details of
> how to use [_PyFrameEvalFunction]”, which isn't very helpful.

Given the proximity of beta1, I think affected projects are going to
need to define Py_BUILD_CORE for the Python 3.11 series - at least
some of them have already been updated to declare that during the
alpha cycle to handle the frame API changes.

PEP 689 can then be informed by feedback from those projects regarding
what breaks for them if they *don't* declare Py_BUILD_CORE.

That said, one slightly awkward consequence of this approach is
affected projects would end up needing to include CPython's
"patchlevel.h" [1] directly, so they can query PY_VERSION_HEX *before*
including "Python.h". Something like:

#include "patchlevel.h"  # CPython's version declaration header
#if Py_VERSION_HEX+0 >= 0x030c  # CPython 3.12+
#define Py_USING_UNSTABLE_API
#elif Py_VERSION_HEX+0 >= 0x030b # CPython 3.11.x
#define Py_BUILD_CORE
#endif
#include "Python.h"

So maybe it would be worth asking the SC for an exception to the beta
freeze, and allow the unstable API tier to land during the beta
period?

Cheers,
Nick.

[1] https://github.com/python/cpython/blob/main/Include/patchlevel.h

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KG56QXYVEPDUSPIGDDUDEDZLMYXOXPQG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 689 – Semi-stable C API tier

[Nick Coghlan]

On Sat, 30 Apr 2022, 3:02 am Guido van Rossum,  wrote:

>
> I think picking "semi-stable" would be giving in to the OCD nerd in all of
> us. :-) While perhaps technically less precise, "unstable" is the catchy
> name with the right association. (And yes, we should keep it stable within
> bugfix releases, but the name doesn't need to reflect that detail.) The
> "internal API" isn't an API at all (except for CPython core developers and
> contributors). The "unstable API" would definitely be an *API* for users
> outside the core.
>
> So let's please go with "unstable".
>

While I've advocated for semi-stable in previous threads, I now agree the
pragmatic arguments for "unstable" hold up well enough to make the simpler
term the better choice:

* no question around using a hyphen or not
* "unstable public C API" is sufficient to distinguish the new tier from
Py_BUILD_CORE's completely unstable internal API

The risks of misinterpretation are also low:

* external users that need one of these APIs will presumably be invested
enough to actually check the stability expectations in the docs
* core devs will have regression tests to remind us that the published
unstable APIs aren't allowed to change after beta 1

Cheers,
Nick.




> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6TVQ6MEVKNLYNQAHMQHCHDMJES7RIRXL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 3: Closing thoughts on "forward class", etc.

[Nick Coghlan]

On Sat, 23 Apr 2022, 11:17 am Larry Hastings,  wrote:

>
>
> Just a quick note from me on the proto-PEP and the two proposed
> implementations.  When I started exploring this approach, I didn't suspect
> it'd require such sweeping changes to be feasible.  Specifically, I didn't
> think I was going to propose changing the fundamental mechanism used to
> create class objects.  That's an enormous change, and it makes me
> uncomfortable; I suspect I won't be alone in having that reaction.
>
> The alternate implementation with proxy objects was borne of my reaction,
> but it's worrisome too.  It's a hack--though whether it's a "big" hack or a
> "small" hack is debatable.  Anyway, I'm specifically worried about the
> underlying class object escaping the proxy and becoming visible inside
> Python somehow.  If that happened, we'd have two objects representing the
> same "type" at runtime, a situation that could quickly become confusing.
>
> Also, as I hopefully made clear in the "alternate implementation" approach
> using a class proxy object, I'm not 100% certain that the proxy will work
> in all cases.  I ran out of time to investigate it more--I wanted to post
> this idea with some lead time before the 2022 Language Summit, so that
> folks had time to read and digest it and discuss it before the Summit.  I
> have some implementation ideas--the "class proxy" class may need its own
> exotic metaclass.
>
> Ultimately I'm posting this proto-PEP to foster discussion.  I'm confident
> that "forward class" / "continue class" could solve all our
> forward-reference and circular-reference problems; the questions we need to
> collectively answer are:
>
>- how should the implementation work, and
>- is the cost of the implementation worth it?
>
>
Something worth considering: whether forward references need to be
*transparent* at runtime. If they can just be regular ForwardRef objects
then much of the runtime complexity will go away (at the cost of some
potentially confusing identity check failures between forward references
and the actual class objects).

ForwardRef's constructor could also potentially be enhanced to accept a
"resolve" callable, and a "resolve()" method added to its public API,
although the extra complexity that would bring may not be worth it.

Cheers,
Nick.




>
>
> Best wishes,
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HD7YPONSPL5ZFZISKCOUWVUXMIJTQG2M/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DMITVTUIQKJW6RYVOPQXHD54VSYE7QHA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 687 – Isolating modules in the standard library

[Nick Coghlan]

On Tue, 12 Apr 2022, 3:59 am Guido van Rossum,  wrote:

> In the sentence starting with
>
> Types whose methods need access to their module instance will be converted
> to heap types[...]
>
> please emphasize (bold!) "whose methods need access to their module
> instance".
>
> Also emphasize this paragraph:
>
> "Static types that do not need module state access, and have no other
> reason to be converted, should stay static."
>
> I hadn't noticed the qualification in the first sentence and had assumed
> all types were to be converted, until I came across the second at the very
> end of the section (where it is easily overlooked by lazy readers :-).
>


Making that final paragraph the first paragraph in the section should help
on both points (reminds me of flipping an if/else in code so the one-liner
branch is the first one).

Definite +1 from me on the overall PEP.

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GY7KAR7QIZTE4URQPXY2ALTULTIG2WKJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Importing a submodule doesn't always set an attribute on its parent

[Nick Coghlan]

On Sun, 10 Apr 2022, 8:44 am Eric V. Smith,  wrote:

> On 4/9/2022 4:28 PM, Terry Reedy wrote:
> > On 4/9/2022 5:09 AM, Arfrever Frehtes Taifersar Arahesis wrote:
> >>
> >> Not only deletion, but also random assignments:
> >
> > Ok.  Change "Manual deletion of entries from sys.modules" to "Direct
> > manipulation of sys.modules"
>
> I'm not sure it's worth the hassle to document this. There are no doubt
> hundreds of ways to break things. We can't enumerate all of them.
>


I thought we already explicitly said that if you directly alter the import
system's data stores then it's possible to break things in non-obvious ways.

If we don't, then it seems reasonable to make it explicit that the runtime
*isn't* implicitly enforcing the invariants the import system expects to
see, it is instead assuming that all code manipulating the data stores is
cooperating to ensure those invariants hold.

As the OP noted, in the context of programming languages in general, rather
than Python in particular, it's a relatively unusual design choice to allow
developers the ability to inadvertently (or intentionally!) corrupt the
internal consistency of the import system's runtime state to the point
where legal code won't run anymore (even though it's aligned with Python's
generally permissive philosophy that allows even destructive actions like
clearing the builtin namespace).

Cheers,
Nick.


> Eric
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WGP7SLEIEIDPJEOU3ANSRPGHWKLMM7QE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DSQJVXMVAC3JXQ6H3NKLXBNPBCQHRWTC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin,  wrote:

>
> This applies to:
>
> - Functions added in PEP 523
> - PyCode_New, PyCode_NewWithPosOnlyArgs
> - Ideally anything documented as subject to change between minor
> releases. (To be kind to users, if something is added later we should
> again have one release of compiler warnings before requiring the opt-in.
> Unless that API just changed and users would get errors anyway.)
>


Other candidate items for this tier:


* non-opaque access to frame structs and any other key APIs needed to
implement alternate eval loops with comparable performance to the default
eval loop (unless & until we can figure out stable public APIs that can
deliver equivalent performance)

> * C APIs that provide access to compiled code whether in AST or opcode
form (the API itself may be stable, but the compiled code isn't, so this is
kinda covered by your last point)

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CW543RWJU6EWAPTWCDDHEUS37UHSAU5I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin,  wrote:

> So here's my proposal:
>
> - This API stays with the regular public API (Include/cpython/), but to
> use it you'll need to #define Py_USING_UNSTABLE_API (name up for
> bikeshedding).
>

I'm fine with the rest of what you suggest, but I don't think this is the
right mechanical approach:

* "unstable" is the wrong term. We already have an unstable API tier: the
internal API, which can change even in maintenance releases. The value of
the new tier is that it is "semi stable": stable in maintenance releases,
unstable in feature releases.
* the lesson I take from our stable ABI experience is that mixing two tiers
of the API in a single header file is hard to maintain, as it's too easy to
add a new API to the wrong section. A separate file that gets included
automatically from the relevant header file(s) when the new definition is
used makes the split much clearer.

Cheers,
Nick.



>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV5R2A2RRN5DM32UQFNC42DDGJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Wed, 6 Apr 2022, 7:05 am Victor Stinner,  wrote:

> On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan  wrote:
> > The changes you've made have been excellent, and the existing 3
> categories (stable public ABI, stable public API, unstable internal API)
> cover the vast majority of cases.
> >
> > The final case that isn't quite covered yet is to offer a "semi-stable"
> API category for use cases that are intrinsically coupled to implementation
> details that may change between feature releases, but should remain stable
> within a release series.
> >
> > The concrete motivating example for the new category is the extra APIs
> you need in order to provide an alternative eval loop implementation.
> >
> > The internal API category doesn't properly cover that case, as the APIs
> there are free to change even in maintenance releases, and setting
> Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
> need.
> >
> > Regular public functions may work in some cases, but aren't necessarily
> practical in others (such as exposing the internal frame details for use in
> alternative eval loops).
> >
> > From an implementation PoV, my own suggestion would be to define a new
> API tier with an opt-in macro rather than relying solely on documentation
> or naming conventions.
> >
> > For example, define "Py_SEMI_STABLE_API" to opt in, with the headers
> under "Include/cpython/semi_stable/" (I don't like "unstable" as potential
> terminology here, since the internal API is already unstable - we're
> splitting the difference between that and the long term stability of the
> full public API)
>
> For me an API is either stable (remains the same forever) or unstable
> (change time to time).
>
> Public API means: stable, documented, tested.
>
> Internal API means: unstable, not documented, not tested.
>
> I'm not convinced that it's worth it to create something in the
> middle. If you want to add doc and tests, it should become a public
> stable API.
>

The middle semi-stable tier formalises a concept that we already have: no
guarantees across feature releases, but both API and ABI stable within a
release series.

It's useful for tightly coupled projects like Cython, as it means they can
get core level performance without the risk of API compatibility breaks in
maintenance releases.

Without defining this tier, effectively the *entire* internal API becomes
semi-stable, as any changes made will risk breaking the third party
projects that we've told to define Py_BUILD_CORE when compiling.

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7P3ENW56DFR5BOUFSOQUSXPFQNQ5MF56/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Nick Coghlan]

On Sun, 3 Apr 2022, 12:58 pm Steven D'Aprano,  wrote:

>
> I haven't considered pickling, or deep-copying. I don't think there is
> any way to get access to the underlying dict and modify it, except
> perhaps via ctypes, so I think it is as immutable as it is possible to
> get from Python code. Feel free to take that as a challenge to break it.
>


MappingProxyType does expose the underlying mapping to the other operand
during type coercion for binary operators (and trying to prevent that
causes enough problems that we decided the problem wasn't worth fixing):
https://bugs.python.org/issue43838


For a lot of use cases, MappingProxyType is already a "good enough"
immutable dict, but it does have a definite discoverability problem.

Cheers,
Nick.



>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3DN5BIOAZSG7YFVVHVRVF66MJC5YNJON/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Fri, 1 Apr 2022, 6:47 pm Victor Stinner,  wrote:

> On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum  wrote:
>
> I'm not convinced that advertising an API as being Unstable (in the
> documentation?) is going to solve any kind of problem. People love to
> use private APIs, and they do it simply because it's technically
> possible :-) At the end of the day, we have to help them updating
> their code, otherwise we (at least my Red Hat team) cannot update
> Python.
>
> I designed the internal C API to be more annoying to be used (need to
> define a macro, need more specific #include) in the hope that people
> will think twice before using it :-)
>


The changes you've made have been excellent, and the existing 3 categories
(stable public ABI, stable public API, unstable internal API) cover the
vast majority of cases.

The final case that isn't quite covered yet is to offer a "semi-stable" API
category for use cases that are intrinsically coupled to implementation
details that may change between feature releases, but should remain stable
within a release series.

The concrete motivating example for the new category is the extra APIs you
need in order to provide an alternative eval loop implementation.

The internal API category doesn't properly cover that case, as the APIs
there are free to change even in maintenance releases, and setting
Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
need.

Regular public functions may work in some cases, but aren't necessarily
practical in others (such as exposing the internal frame details for use in
alternative eval loops).

>From an implementation PoV, my own suggestion would be to define a new API
tier with an opt-in macro rather than relying solely on documentation or
naming conventions.

For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under
"Include/cpython/semi_stable/" (I don't like "unstable" as potential
terminology here, since the internal API is already unstable - we're
splitting the difference between that and the long term stability of the
full public API)

Cheers,
Nick.



> Victor
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZBNJTAXS6TWQY7QH5H5XZS4CP64ZQAUU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Nick Coghlan]

On Tue, 29 Mar 2022, 9:44 am Toshio Kuratomi,  wrote:

> One thing about talking about "make urllib more like requests" that is
> different than any of the other libs, though, is that requests aims to be
> easier to use than anything else (which I note Chris Barker called out as
> why he wanted urllib to be more like it).  I think that's important to
> think about because i think ease of use is also the number one thing that
> the audience you talk about is also looking for.
>
> Of course, figuring out whether an api like request's is actually easier
> to use than urllib or merely more featureful is open to debate.
>

There's a concrete scope difference between the two:

* requests was written as a way to make HTTP(S) requests from Python,
including authenticated JSON based API requests
* urllib was designed to work with a variety of URL schemas which may or
may not support authentication and encryption (before the rise of the web
and restrictive corporate firewalls helped HTTP become the de facto
standard for data exchange over the public internet)


I wrote more on that topic back in 2016 [1], and the gist of that article
still holds true: urllib is OK for the basic HTTP GET use case, but having
to tell urllib that you want to use HTTP-only features exposes a lot of low
level plumbing that end users ideally wouldn't have to care about.

Unfortunately, it isn't clear where a useful subset of a full-fledged HTTP
client library lies. "Make it easy to send a TLS protected JSON API POST
request with Basic authentication and check the status code of the
response" *might* qualify, but even that isn't particularly easy to define
or maintain (What about proxies? What about redirects? What about other
authentication methods?).

Thus the status quo persists - adding a fully featured HTTP client library
to the existing stdlib maintenance burden doesn't (or at least shouldn't)
strike *anyone* as a good idea, but it also isn't clear how much of the
ease of use of the existing 3rd party libraries comes from the sheer amount
of functionality they have built-in so users don't have to worry about it
themselves, which means defining something "lighter and easier to maintain
than a fully featured HTTP client library, but still easier to use than
calling in to urllib directly" isn't a viable option either.

At least with urllib present in the stdlib, it's *possible* to share "No
dependency" recipes for crafting particular HTTP requests, even though the
default recommendation remains to grab a higher level 3rd party library
instead of rolling your own solution.

Cheers,
Nick.


[1]
https://www.curiousefficiency.org/posts/2016/08/what-problem-does-it-solve.html






> -toshio
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4ZQC4H7HD3UXFT3CONU64YPOQBSPUTVY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QENBMBYL4YABJKRPX7UR65DOGRAZUJ26/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Enhancing generic type documentation in the standard library

[Nick Coghlan]

On Wed, 30 Mar 2022, 3:49 am Brett Cannon,  wrote:

>
>
> On Mon, Mar 28, 2022 at 3:58 PM Luciano Ramalho 
> wrote:
>
>>
>> Documenting the generic types in the standard library is a much
>> smaller task than turning the typing PEPs into specs. It seems like a
>> good next step on the way to better typing docs.
>>
>
> Since the typing-sig will likely be doing the work and driving such an
> effort I would ask over there. I personally would love for a
> typing.python.org or equivalent subsection of docs.python.org to exist.
>

The import system gained its own subsection of the language reference to
get info out of the historical PEPs.

Perhaps gradual typing could go the same way, with the initial content
being:

* the type hinting docs for the builtin generics
* pointers to the PEPs that provide the interim docs until the reference
section is fleshed out

(we used the latter trick to bootstrap the PyPA specs page without having
to rewrite all the content: some of the pages are just pointers to the
relevant PEP)

Cheers,
Nick.


> -Brett
>
>
>>
>> Cheers,
>>
>> Luciano
>>
>>
>>
>> >
>> > -Brett
>> >
>> >>
>> >>
>> >> We now have lots of generic types in the standard library, but their
>> >> formal type parameters are poorly documented—or not documented at
>> >> all—in the standard library documentation.
>> >>
>> >> More importantly: the documentation we have about specific
>> >> covariant/contravariant types is now in entries in the `typing` module
>> >> that are all deprecated since PEP 585 was implemented in Python 3.9.
>> >>
>> >> Below I present two of many examples where the documentation of a
>> >> generic type is not great.
>> >>
>> >> However, if people agree this is a problem, we need to discuss where
>> >> and how to put the documentation in a way that is not too disruptive
>> >> to users of Python who don't know or don't care about type hints, for
>> >> many reasons that we should not judge.
>> >>
>> >> For example, where do we document the fact that `dict` accepts two
>> >> invariant formal type parameters, and that `frozenset` accepts one
>> >> contravariant type parameter?
>> >>
>> >> What do you think?
>> >>
>> >> Cheers,
>> >>
>> >> Luciano
>> >>
>> >> _
>> >> EXAMPLE 1: `Callable` variance is not documented
>> >>
>> >> For example, in the `Callable` type, the `ReturnType` parameter is
>> >> covariant, and the `ParameterType` parameters are all contravariant.
>> >>
>> >> But there is no mention of variance in this entry:
>> >>
>> https://docs.python.org/3/library/typing.html?highlight=typing#typing.Callable
>> >>
>> >> Also, no mention of the fact that `collections.abc.Callable` is
>> generic here:
>> >>
>> https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable
>> >>
>> >> PEP 483—The Theory of Type Hints—is the only official Python doc where
>> >> I found the information about the variance of the formal type
>> >> parameters of `Callable`. The explanation there is brilliant [0].
>> >>
>> >> [0] https://peps.python.org/pep-0483/#covariance-and-contravariance
>> >>
>> >> Regardless, the intended audience of PEPs is "core developers"—which
>> >> is neither a superset nor a subset of "Python devs now using type
>> >> hints". We should not rely on PEPs to document features for Python
>> >> users in general.
>> >>
>> >> _
>> >> EXAMPLE 2: `Generator` variance could be better documented
>> >>
>> >> The entry for `typing.Generator` [1] has this heading:
>> >>
>> >> class typing.Generator(Iterator[T_co], Generic[T_co, T_contra, V_co])
>> >>
>> >> Answer quickly: how many formal type parameters does `Generator`
>> >> require? Which are covariant? Which are contravariant?
>> >>
>> >> [1]
>> https://docs.python.org/3/library/typing.html?highlight=typing#typing.Generator
>> >>
>> >> Nowhere in that page [1] there's an explanation of the `*_co` and
>> >> `*_contra` naming convention, much less their semantics.
>> >>
>> >> Fortunately, the text of the `typing.Generator` entry says: "A
>> >> generator can be annotated by the generic type `Generator[YieldType,
>> >> SendType, ReturnType]".
>> >>
>> >> Unfortunately, `typing.Generator` is deprecated and will be gone in 5
>> >> years or so...
>> >>
>> >> The same issue applies to all the other generic types: builtins
>> >> (`dict`, `frozenset`), ABCs, etc.
>> >> Their formal type parameters they accept as generics are either
>> >> undocumented, or documented in parts of the `typing` module that are
>> >> already deprecated.
>> >>
>> >> Thoughts?
>> >>
>> >> --
>> >> Luciano Ramalho
>> >> |  Author of Fluent Python (O'Reilly, 2015)
>> >> | http://shop.oreilly.com/product/0636920032519.do
>> >> |  Technical Principal at ThoughtWorks
>> >> |  Twitter: @ramalhoorg
>> >> ___
>> >> Python-Dev mailing list -- python-dev@python.org
>> >> To unsubscribe send an email to python-dev-le...@python.org
>> >> 

[Python-Dev] Re: A memory map based data persistence and startup speedup approach

[Nick Coghlan]

(belated follow-up as I noticed there hadn't been a reply on list yet, just
the previous feedback on the faster-cpython ticket)

On Mon, 21 Feb 2022, 6:53 pm Yichen Yan via Python-Dev, <
python-dev@python.org> wrote:

>
> Hi folks, as illustrated in faster-cpython#150 [1], we have implemented a
> mechanism that supports data persistence of a subset of python date types
> with mmap, therefore can reduce package import time by caching code object.
> This could be seen as a more eager pyc format, as they are for the same
> purpose, but our approach try to avoid [de]serialization. Therefore, we get
> a speedup in overall python startup by ~15%.
>

This certainly sounds interesting!


> Currently, we’ve made it a third-party library and have been working on
> open-sourcing.
>
>
> Our implementation (whose non-official name is “pycds”) mainly contains
> two parts:
>
>- importlib hooks, this implements the mechanism to dump code objects
>to an archive and a `Finder` that supports loading code object from mapped
>memory.
>- Dumping and loading (subset of) python types with mmap. In this
>part, we deal with 1) ASLR by patching `ob_type` fields; 2) hash seed
>randomization by supporting only basic types who don’t have hash-based
>layout (i.e. dict is not supported); 3) interned string by re-interning
>strings while loading mmap archive and so on.
>
> I assume the files wouldn't be portable across architectures, so does the
cache file naming scheme take that into account?

(The idea is interesting regardless of whether it produces arch-specific
files - kind of a middle ground between portable serialisation based pycs
and fully frozen modules)

Cheers,
Nick.


>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O4A4CHQZW6SSTXP7D5ZB5NOYWDMATE3W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New PEP website is horrible to read on mobile device

[Nick Coghlan]

On Thu, 17 Mar 2022, 10:29 am Inada Naoki,  wrote:

> I can reproduce it. I reported it on
> https://github.com/python/peps/issues/2437


Summarising the issues identified there for the thread: the gist is that
some elements in some PEPs (mainly tables and code blocks) could result in
the initial rendering width being wider than the initial view port size,
hence having to zoom out to see the full text.

The styling of affected elements is being adjusted so that mobile browsers
are better able to shrink them to fit on a phone screen.

Cheers,
Nick.


> Cheers
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NBZWV43XXS67LEQ44SAHUUTQK36KP3IY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 684: A Per-Interpreter GIL

[Nick Coghlan]

On Thu, 10 Mar 2022, 12:12 pm Eric Snow, 
wrote:

>
> On Wed, Mar 9, 2022 at 10:58 AM Petr Viktorin  wrote:
> > This PEP definitely makes per-interpreter GIL sound possible :)
>
>
> >
> > “Main interpreter” should be defined.
>
> +1
>
> > (Or maybe the term should be
> > avoided instead -- always having to spell out “interpreter started by
> > Py_Initialize rather than Py_NewInterpreter” might push us toward
> > finding ways to avoid the special case...)
>
> We (me, Nick, Victor, others) have considered this in the past and
> have concluded that having a distinct main interpreter is valuable.
> That topic is a bit out of scope for this PEP though.
>


The PEP can mostly link to
https://docs.python.org/3/c-api/init.html#sub-interpreter-support for the
explanation of the main interpreter, and just include a partial paraphrase
of those docs to give the gist of the idea. We added that info the last
time we considered whether the main interpreter's "first among equals"
status was necessary and decided it was.

For example, something based on the first and third sentences out of the
docs explanation:

The “main” interpreter is the first one created when the runtime
initializes. It continues to manage unique process-global responsibilities
like signal handling even when other subinterpreters have been started.

Cheers,
Nick.



> > >
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M4IWPEJVKS6VRKLZGAFYZSXMFK6CMBD6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improvements to the sys.path initialization documentation

[Nick Coghlan]

On Mon, 7 Mar 2022, 7:42 pm Victor Stinner,  wrote:

> On Fri, Mar 4, 2022 at 1:37 PM Eryk Sun  wrote:
> > I don't understand. The site packages directories, including virtual
> > environments, are a site extension.
>
> I propose to change that. Move all code related to sys.path from the
> site module to the Modules/getpath.py module.
>

Certain entries not being on sys.path when the site module isn't loaded is
a feature (the "-S" CLI option), not a bug to be fixed.

Those site entries haven't historically been the really confusing part of
path inititialisation - it's been finding out where all the *other* default
entries are coming from, so you can get a clearer picture of what's always
there vs what's being added by the site module.

Cheers,
Nick.



> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/HXD2JIDHBX7BFIU3FA7R3LWRFEXQMVYW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ICOGZEUBU7BAL25OUMMTFSJ2VSVVYN43/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move the pythoncapi_compat project under the GitHub Python or PSF organization?

[Nick Coghlan]

On Tue, 15 Feb 2022, 2:57 am Petr Viktorin,  wrote:

> >>
> >> Yes.
> >> On older Python versions, where the public API wasn't yet available,
> >> those backports use private API. If we change the private API in a
> >> point release, the backport will break.
> >
> > Do you have an example of this? On first glance the pythoncapi_compat.h
> > header only uses public APIs, other than (maybe) accessing fields of the
> > thread state directly.
>
> That's my example. Those fields are documented as "subject to change at
> any time."
>
> But I wouldn't be afraid to do this more generally -- if we add a public
> API for something that needed private API before, freeze the old way in
> previous versions. Not only add it to pythoncapi_compat, but also to
> CPython CI, and maybe to the docs.
>


Adopting pythoncapi_compat would offer a relatively clean way to test that:
if a maintenance branch change breaks pythoncapi_compat, then it's the
maintenance branch that's considered broken.

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KDWC6S66DKC7FYAP7GQ43Q7UZLHPFKXR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments

[Nick Coghlan]

On Thu, 13 Jan 2022, 2:24 pm Dennis Sweeney, 
wrote:

> Like others expressed, I don't like the idea of the typing and non-typing
> parts of Python separating.
>
> Has anyone considered adding a new special method like `__arrow__` or
> something, that would be user-definable, but also defined for tuples and
> types as returning a Callable? For example `int -> str` could mean
> Callable[[int], str], and (int, str) -> bool could mean Callable[[int,
> str], bool]. I would find that sort of semantics more agreeable since
> Python already has operators that dispatch to dunder methods, and anyone
> who knows how that bit of Python works would automatically mostly know how
> the new operator works.
>

If such a protocol were to be proposed, then int.__arrow__ could be defined
such that "1 -> 100" was equivalent to "range(1, 100)" (in addition to the
typing use cases).


The existing section on runtime behaviour in the PEP doesn't look ready for
SC consideration, since it essentially says "this hasn't been defined yet".

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ENTIEFGCG7O7APREWRLKAX72DISHXSNT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

[Nick Coghlan]

On Wed, 15 Dec 2021, 3:18 am Eric Snow,  wrote:

> Hi all,
>
> I'm still hoping to land a per-interpreter GIL for 3.11.  There is
> still a decent amount of work to be done but little of it will require
> solving any big problems:
>
> * pull remaining static globals into _PyRuntimeState and PyInterpreterState
> * minor updates to PEP 554
> * finish up the last couple pieces of the PEP 554 implementation
> * maybe publish a companion PEP about per-interpreter GIL
>
> There are also a few decisions to be made.  I'll open a couple of
> other threads to get feedback on those.  Here I'd like your thoughts
> on the following:
>
> Do we need a PEP about per-interpreter GIL?
>
> I haven't thought there would be much value in such a PEP.  There
> doesn't seem to be any decision that needs to be made.  At best the
> PEP would be an explanation of the project, where:
>
> * the objective has gotten a lot of support (and we're working on
> addressing the concerns of the few objectors)
> * most of the required work is worth doing regardless (e.g. improve
> runtime init/fini, eliminate static globals)
> * the performance impact is likely to be a net improvement
> * it is fully backward compatible and the C-API is essentially unaffected
>
> So the value of a PEP would be in consolidating an explanation of the
> project into a single document.  It seems like a poor fit for a PEP.
>

I think PEP 630 (Petr's summary of the improvements to extension module
reloading) is a good example of such a PEP being valuable.

Writing such a PEP also provides a place to summarise key design decisions
and known limitations (e.g. do the threading module primitives work for
cross-interpreter synchronisation? If not, what should be used instead?
multiprocessing? Something new that is still to be defined?).

Cheers,
Nick.


> (



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WVNNIGDQPGJHQO4LBJWF66BYGLY2WK4Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nick Coghlan]

On Tue, 28 Sep 2021, 6:55 am Brett Cannon,  wrote:

>
>
> On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev <
> python-dev@python.org> wrote:
>
>>
>> However the stable ABI is still a second class citizen as it is still
>> not possible (AFAIK) to specify a wheel name that doesn't need to
>> explicitly include each supported Python version (rather than a minimum
>> stable ABI version).
>>
>
> Actually you can do this. The list of compatible wheels for a platform
> starts at CPython 3.2 when the stable ABI was introduced and goes forward
> to the version of Python you are running. So you can build a wheel file
> that targets the oldest version of CPython that you are targeting and its
> version of the stable ABI and it is considered forward compatible. See
> `python -m pip debug --verbose` for the complete list of wheel tags that
> are supported for an interpreter.
>

I think Phil's point is a build side one: as far as I know, the process for
getting one of those more generic file names is still to build a wheel with
an overly precise name for the stable ABI declarations used, and then
rename it.

The correspondence between "I used these stable ABI declarations in my
module build" and "I can use this more broadly accepted wheel name" is
currently obscure enough that I couldn't tell you off the top of my head
how to do it, and I contributed to the design of both sides of the equation.

Actually improving the build ergonomics would be hard (and outside
CPython's own scope), but offering a table in the stable ABI docs giving
suggested wheel tags for different stable ABI declarations should be
feasible, and would be useful to both folks renaming already built wheels
and anyone working on improving the build automation tools.

Cheers,
Nick.


___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/E4EA6GW6N5NREEBM4QOA7GTLBGTGD4AV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NZRLQUMOJB5VD3CW5OQF6AKSNBJ2DW5L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Does ensurepip still have to include a copy of setuptools?

[Nick Coghlan]

On Mon, 4 Oct 2021, 5:23 pm Antoine Pitrou,  wrote:

> On Sat, 2 Oct 2021 13:27:03 +0100
> Paul Moore  wrote:
> > On Sat, 2 Oct 2021 at 12:20, Thomas Grainger  wrote:
> > >
> > > I raised an issue about this: https://github.com/pypa/pip/issues/10530
>
> >
> > I agree with the comment made on that issue - this isn't the right way
> > to handle the problem. We need to encourage projects to opt into the
> > new approach and remove the legacy path once it's no longer needed. We
> > should *not* maintain the "old style" approach indefinitely, hiding
> > the fact that it's no longer the correct approach by having some sort
> > of "auto convert" logic in the tools.
>
> Can you explain what the "old style" approach is here?  I would hope
> for the "old style" approach to be deprecated (with a *visible*
> warning message) for at least 2 years before it is removed.
>
> It is nice that well-maintained packages with lots of contributors get
> frequent releases and keep up with the pace of changes in the packaging
> ecosystem, but please don't forget that there's a long tail of packages
> that are updated infrequently and but still work properly and perform
> an important function for some parts of the user base.
>


That's the real milestone intended in the setuptools bundling wording: we
can drop the bundling when pip doesn't need it itself, *and* can also
transparently bootstrap it for all other projects that need it, whether
those projects have been updated to use the modern packaging standards or
not.

The fact those were different requirements wasn't clear when we wrote the
PEP though, so the concrete example given only covers the first criterion
and not the second one, rendering the section as a whole ambiguous.

Paul's mail covered some of the details still pending before automatic
installation of build time dependencies into an isolated build environment
becomes a universal default on new pip installations. We wouldn't drop
setuptools from the bundling until that default changes.

Cheers,
Nick.





> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OKLQED7AYUBA4MEO6OGBLPCE6UT2F4MJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NHT76LC2HTBZM6I5NY76YRPDJJCTNZ7E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Nick Coghlan]

On Mon, 20 Sep 2021, 9:19 pm Pablo Galindo Salgado, 
wrote:

> Hi,
>
> I have started a project to move the parsing off-strings to the parser and
> the grammar. Appart
> from some maintenance improvements (we can drop a considerable amount of
> hand-written code),
> there are some interesting things we **could** (emphasis on could) get out
> of this and I wanted
> to discuss what people think about them.
>

The change seems like a good idea, but the consequences should be
summarised in a PEP (either the existing
https://www.python.org/dev/peps/pep-0536/ or a replacement for it).

Cheers,
Nick.

>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6JCSTZNIKWX7HMJO5RIAH3QARXWJSWFZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 467 feedback from the Steering Council

[Nick Coghlan]

On Fri, 10 Sep 2021, 12:32 am Arnaud Delobelle,  wrote:

> It probably won't fly but why not bytes.frombyte?
>
> There's no such thing as a byte type in Python, only bytes, so I want
> to argue it makes it clear the argument is a number in the range
> 0..255 and the result is a bytes object containing this single byte
> value.
>

The getbyte() and iterbytes() methods added by the PEP make a length 1
bytes object the pseudo "byte" type, just as a length 1 string is the
pseudo "character" (technically "code point") type, so "frombyte()" would
have the type implications backwards (it produces a "byte", it doesn't
really read one).

I think it's OK for "int.from_bytes", "int.to_bytes", "bytes.fromint" and
"byte array.fromint" to have closely related names, since they're closely
related operations. The OverflowError raised for out of bounds values in
the latter two methods could even mention int.to_bytes explicitly.

While the SC already accepted "fromint", the range limitation could be made
more explicit by appending "byte" to give "bytes.fromintbyte" and
"bytearray.fromintbyte" (thus naming a pseudo "int byte" type for integers
in the range 0-255 inclusive). An advantage of this approach is to give a
specific name to the kinds of values that regular indexing and iteration of
bytes objects produce.

The SC has already rejected "fromord" as too obscure.

I'd be OK with bytes.bchr, but bytearray.bchr would look odd to me, so I
don't like that option as a whole.

For "bytes.byte", I have 3 objections:
* I think Greg's Smith's API stuttering concerns are valid
* I think it's potentially ambiguous as to whether it is an alternate
constructor or an indexed access method (i.e. doing what getbyte() does in
the PEP)
* I don't like it as a bytearray method name

Cheers,
Nick.





> Tentatively,
>
> Arnaud
>
> PS. But truly I feel like this method is superfluous.
>
> On Thu, 9 Sept 2021 at 11:11, Victor Stinner  wrote:
> >
> > I proposed bytes.byte earlier in this thread:
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/KBVVBJL2PHI55Y26Z4FMSCJPER242LFA/
> >
> > Gregory dislikes the name: "I don't *like* to argue over names (the
> > last stage of anything) but I do need to point out how that sounds to
> > read".
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/DGJWM3VMNMDBUTGYG72H5WLKDWBYFSUV/
> >
> > That's why I proposed: bytes.fromchar(). I still like bytes.byte() :-)
> >
> > Victor
> >
> > On Thu, Sep 9, 2021 at 11:07 AM Antoine Pitrou 
> wrote:
> > >
> > > On Thu, 9 Sep 2021 18:55:04 +1000
> > > Nick Coghlan  wrote:
> > > >
> > > > P.S. The fact that it *didn't* look like the inverse operation for
> > > > `int.from_bytes` was one advantage of calling the method
> > > > `bytes.fromord` instead of `bytes.fromint`, but I'm still happy the
> SC
> > > > is right that `bytes.fromint` is a more comprehensible method name
> > > > overall.
> > >
> > > Perhaps we can call it `bytes.byte` to make it unambiguous?
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > > ___
> > > Python-Dev mailing list -- python-dev@python.org
> > > To unsubscribe send an email to python-dev-le...@python.org
> > > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WZUPBP4UASRCJLAKP6FMQJLLMYJY22CL/
> > > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> >
> >
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6W4G32NOBXAQ73VESVE4UL7AZIWUAD6A/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/P7XG5CLLBXZTS6UE72KSGWLVYXRDXKT4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OJBHNWKKCWUN25GMONL3BGGW6I3TGYUV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 467 feedback from the Steering Council

[Nick Coghlan]

On Thu, 9 Sept 2021 at 01:46, Ethan Furman  wrote:
>
> On 9/7/21 10:39 PM, Steven D'Aprano wrote:
>  > On Tue, Sep 07, 2021 at 08:09:33PM -0700, Barry Warsaw wrote:
>  >
>  >> I think Nick is on board with bytes.fromint() and no bchr(), and my
>  >> sense of the sentiment here is that this would be an acceptable
>  >> resolution for most folks.  Ethan, can you reconsider?
>  >
>  > I haven't been completely keeping up with the entire thread, so
>  > apologies if this has already been covered. I assume that the idea is
>  > that bytes.fromint should return a single byte, equivalent to chr()
>  > returning a single character.
>  >
>  > To me, it sounds like should be the opposite of int.from_bytes.
>  >
>  >  >>> int.from_bytes(b'Hello world', 'little')
>  >  121404708502361365413651784
>  >  >>> bytes.from_int(121404708502361365413651784, 'little')
>  >  # should return b'Hello world'
>
> That certainly makes sense to me.  At this point, the only reason that would 
> not work is an arbitrary limit of 255 on
> the input, and the only reason that limit is there is to have `bchr` be the 
> inverse of `ord`.  Since `bchr` isn't going
> to happen, I see no reason to have the 255 limit.  `byteorder` can default to 
> None with a requirement of being set when
> the integer is over 255.

I've posted a PR removing bchr from the proposal:
https://github.com/python/peps/pull/2068/files

`bytes.fromint` is still the inverse of `ord` for bytes objects, even
without the `bchr` builtin alias. The spelling of the trio is just
`ord`/`bytes.fromint`/`chr` rather than `ord`/`bchr`/`chr`. The fact
the method throws an exception for integers that won't fit in a single
byte is an input data validation feature, not an undesirable
limitation.

As Brandt already noted, we don't need a new general purpose int to
bytes converter as `int.to_bytes` already has that covered.

Cheers,
Nick.

P.S. The fact that it *didn't* look like the inverse operation for
`int.from_bytes` was one advantage of calling the method
`bytes.fromord` instead of `bytes.fromint`, but I'm still happy the SC
is right that `bytes.fromint` is a more comprehensible method name
overall.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2QO35ROTATWQWR6DMHY5BOS25QK2YLER/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP-535 (Rich comparison chaining) Discussion?

[Nick Coghlan]

On Wed, 1 Sep 2021, 7:18 am Senthil Kumaran,  wrote:

> On Tue, Aug 31, 2021 at 8:57 AM Angus Hollands  wrote:
>
>
>> Should look more like
>> ```python
>> x = np.array(...)
>> y = x[2 < x < 8]
>> ```
>> Is there any interest in pushing this PEP along?
>>
>
> My personal opinion is, it is worth discussing again. I could see the
> benefits this brings to Numpy Arrays.
> The PEP itself was deferred based on boolean overload dislike than the
> short-circuit or chaining operator preference.
>

While I don't plan to reactivate these PEPs myself, I'd be happy to welcome
a co-author that was keen to champion them (535 doesn't stand alone - it
depends on the "binary if" and "binary else" circuit breaking protocol
proposal in 532).

While I believe 535 is mostly OK, 532 needs some updates to account for:

* the new language grammar (the PEP was written before the switch to pegen)
* the walrus operator (some of the possible future extensions mentioned are
irrelevant now)
* fixing the diagram rendering and ReST mark up

Cheers,
Nick.



> --
> Senthil
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ENJYFXADZXDU7RZBD5RJTPS6ZE7YFNMN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QY2FL7PMW6OBU24UFDQVPEJLOHZVQUPO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Nick Coghlan]

On Thu, 2 Sep 2021, 7:08 am Victor Stinner,  wrote:

> I saw Python projects injecting fake frames for XML and JSON parsers,
> maybe also configuration file (.ini?) parsers. So text files which
> have line numbers ;-)
>
> On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado
>  wrote:
> > I don't think we should think on those terms. We certainly don't want to
> be on a case
> > where yet again we cannot change the internals because we have an
> official C-API exposed.
>
> PyCode_New() is annoying since it requires to provide *all* arguments.
> I'm thinking of an Frame API which only allows to set 2 values:
> filename and line number. Nothing else.
>
> Something like: frame = PyFrame_New(filename, lineno).
>


Perhaps "PyCode_FromLocation(file name, lineno)" for the new "good enough
to satisfy trace hooks and exception tracebacks" API? These are needed when
creating function-like (et al) objects from extension module code, not just
for runtime frames.

And then explicitly define which code object fields are expected to always
be populated and which are expected to be None on emulated code objects
that don't contain Python byte code? (e.g. some fields should be zero or
the empty tuple, rather than being set to None)

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AXGJAQ5MMRAKAKHHQMYVF5XS7PHALH4W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Discrepancy between what aiter() and `async for` requires on purpose?

[Nick Coghlan]

On Tue, 31 Aug 2021, 2:52 am Brett Cannon,  wrote:

>
> On Sun, Aug 29, 2021 at 2:01 PM Serhiy Storchaka 
> wrote:
>
>>
>> > So my question is whether the discrepancy between what `async for`
>> > expects and what `aiter()` expects on purpose?
>> > https://bugs.python.org/issue31861 
>> > was the issue for creating aiter() and I didn't notice a discussion of
>> > this difference. The key reason I'm asking is this does cause a
>> > deviation compared to the relationship between `for` and `iter()` (which
>> > does not require `__iter__` to be defined on the iterator, although
>> > collections.abc.Iterator does). It also makes the glossary definition
>> > being linked from
>> >
>> https://docs.python.org/3.10/reference/compound_stmts.html#the-async-for-statement
>> > <
>> https://docs.python.org/3.10/reference/compound_stmts.html#the-async-for-statement
>> >
>> > incorrect.
>>
>> PyIter_Check() only checks existence of __next__, not __iter__ (perhaps
>> for performance reasons).
>>
>
> Or maybe no one thought to require __iter__ for iterators?
>

I don't think PyIter_Check is testing the formal definition of an iterator,
I think it's just testing if calling __iter__ can be skipped (as you say,
for performance reasons).

I'm surprised iter() would skip calling __iter__ just because an object
defines __next__, though. Even though "__iter__ is defined and returns
self" is part of the iterator definition, it still feels like a leap from
there to "if __next__ is defined, skip calling __iter__ in iter()".

The optimisation that bypasses the __[a]iter__ method call feels more
legitimate in the actual for loop syntax, it just feels odd to me if the
builtin isn't forcing the call.

Cheers,
Nick.

>



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5UMLDQ4CANKY4WM6RNG67AEJMQI44X42/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Fri, 27 Aug 2021, 2:33 am Guido van Rossum,  wrote:

> On Thu, Aug 26, 2021 at 2:31 AM Nick Coghlan  wrote:
>
>> On Thu, 26 Aug 2021 at 18:29, Nick Coghlan  wrote:
>> [Guido wrote]
>> > > PS. The mapping from varname to position should be on the code
>> object, not on the frame. This is how Mark does it (though his
>> implementation would need to be extended to take cells into account).
>> >
>> > It's taking cells into account that forces the lookup mapping to be on
>> > the frame: different executions of the same code object may reference
>> > different cell objects.
>>
>> Technically, if you design the fast refs mapping so that even cell
>> references have to do an indirection through the fast locals array,
>> then you can put the name-to-fast-locals-offset mapping on the code
>> object. Mine doesn't do that though, it references the cells directly
>> instead.
>>
>> I don't think it makes much difference in practice though, and I don't
>> like the idea of storing a lazily initialised cache on nominally
>> immutable code objects.
>>
>
> Only the part visible to the user is immutable. The specializing
> interpreter stores all sorts of internal mutable data on there -- and at
> least since 3.9 we've had the inline cache stored there. So as long as this
> mapping is invisible to the user, please let's put it on the code object --
> we have a lot more frame objects than code objects, so saving a pointer on
> the frame object and adding it to the code object is advantageous. The cost
> of the extra indirection is irrelevant, this is always going to be a slow
> interface meant for occasional use in a debugger.
>

OK, that makes sense - I'll add a todo note to the implementation PR.

For the main topic of PEP 667 (reducing the cache consistency question to
purely a matter of PyEval_GetLocals() compatibility), I think I can see a
way to make the extra code complexity of the 5 new custom accessory types
(iterator, reversed iterator, keys set, values multi set, items set)
worthwhile to my mind: write the latter 3 in terms of the first two and the
generic mapping API, and expose them for use in other custom mapping
implementations (either directly in the types module, or as optional C
accelerators for the collections module). (I wouldn't make exposing them
part of the PEP, I'd just aim to write them so only the forward and reverse
iterators were specific to proxy objects)

With that done, popitem() is trivial to rewrite to depend on the iterator
code instead of the cache.

That would leave len() and value comparison as the only proxy operations
that don't adhere to the expected algorithmic complexity of mapping
objects, and writing up the comparison with PEP 667 finally convinced me
that those are quirks that API users could manage more easily than 558's
current lazy caching semantics.

I'd still keep the full value cache under the hood, though, as I don't see
enough benefit in getting rid of it when the cost is an unnecessary API
compatibility break.

However, the PEP 667 write up and your own points *have* persuaded me that
the extra C code needed to offer assuredly consistent views through all the
proxy mapping methods is worthwhile.

That will then leave the proposed C APIs as the only differences between
the PEPs - the Python level behaviour will be aligned with Mark's proposal.

Cheers,
Nick.

P.S. I don't want to rely on Python code anywhere in the fast locals proxy
implementation, as that could cause weird interactions if a trace hook is
enabled during the initial start up of the interpreter and tries to trace
the proxy implementation code.



> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QUFA4YUZJILA5OUHKYR6BYTOQJ4ZOGFG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Tue, 24 Aug 2021 at 01:57, Guido van Rossum  wrote:
> On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:
>> Can we avoid describing the C structs in any of these PEPs?
>>
>> It confuses readers having Python attributes and "C-level attributes"
>> (C struct fields?).
>> It also restricts the implementation unnecessarily.
>>
>> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7)
>
>
> I'd be happy to. Nick's PEP still references it (and indeed it is very 
> confusing) and I took it from him. And honestly it would be nice to have a 
> specific short name for it, rather than circumscribing it with "an internal 
> dynamic snapshot stored on the frame object " :-)

https://github.com/python/peps/pull/2060/files switches over to
calling it the "frame value cache" in PEP 558, since that's
essentially what it is on optimised frames (even today).

An unrelated issue that came up while working on that update is
something that affects both PEPs: calling "proxy.clear()" is *super
weird* if we make it work the same way as PyEval_LocalsToFast() works
today. Specifically, it can reach out and clear cells in outer frames,
including the __class__ cell used by zero-arg super(). I can't see
anyone being super upset if we decide to change that and say that
proxy.clear() leaves free variables alone, and only clears local
variables and cells owned by that particular frame.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FJYJJWZBZNAZBW2UDWGSNFCVB26JINNU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Thu, 26 Aug 2021 at 18:29, Nick Coghlan  wrote:
[Guido wrote]
> > PS. The mapping from varname to position should be on the code object, not 
> > on the frame. This is how Mark does it (though his implementation would 
> > need to be extended to take cells into account).
>
> It's taking cells into account that forces the lookup mapping to be on
> the frame: different executions of the same code object may reference
> different cell objects.

Technically, if you design the fast refs mapping so that even cell
references have to do an indirection through the fast locals array,
then you can put the name-to-fast-locals-offset mapping on the code
object. Mine doesn't do that though, it references the cells directly
instead.

I don't think it makes much difference in practice though, and I don't
like the idea of storing a lazily initialised cache on nominally
immutable code objects.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A6CLBN6EUA26BAIJID7UB3KXSF3MSP5D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Mon, 23 Aug 2021 at 13:07, Guido van Rossum  wrote:
> But... I also care about backwards compatibility, and I have a crazy idea for 
> making PyEval_GetLocals() work in a useful manner without compromising the 
> behavior of the f_locals proxy:
>
> - Let's start your idea of using the C-level f_locals field to store the 
> "extra" variables.
> - The Python-level f_locals proxy looks in the actual frame "fast" locals and 
> cells, and uses the C-level f_locals field only for extras
> - However, PyEval_GetLocals() doesn't return the proxy.
> - What PyEval_GetLocals() does: it calls PyEval_FastToLocals(), which makes a 
> pass over the frame locals and adds them to the C-level f_locals field; then 
> it returns that field.
> - So the borrowed reference is owned by the frame, which is the same as 
> currently.
> - The proxy only uses the f_locals field for extra variables. If a variable 
> is deleted in the frame but exists in the f_locals field, the proxy reports 
> it as deleted. (This requires some care but can be done, since we have the 
> mapping from proper variable names to frame locals or cells.)
> - We'll still deprecate PyEval_GetLocals() and PyEval_FastToLocals(), but 
> unless the user turns the deprecation warning into an error, they will work 
> for another few releases. Eventually we'll make PyEval_GetLocals() always 
> return an error (similar to Mark's proposal), since it's in the stable ABI.
> - For PyEval_LocalsToFast() I don't care too much whether we keep it (per 
> Mark's proposal) or make it return an error (per yours).

I don't think any of this is crazy, as it's how the PEP 558 reference
implementation already works for individual keys (aside from only
having a documented deprecation of PyEval_GetLocals(), not a
programmatic one)

You don't need to eliminate the cache (and hence break compatibility
with PyEval_GetLocals()) to ensure it can never get out of sync - you
just have to resync it every time you use it, rather than allowing the
frame API consumer to make that call.

The reason I don't like the proxy semantics proposed in PEP 667 is
because it either makes the common case (analysing a frame from a
tracing function while no application code is running) slower in order
to avoid having to worry about cache consistency when trying to
analyse a running frame, or else we have to write and maintain a whole
lot more fast locals proxy specific code to implement the 5 different
dict iteration APIs.

> PS. The mapping from varname to position should be on the code object, not on 
> the frame. This is how Mark does it (though his implementation would need to 
> be extended to take cells into account).

It's taking cells into account that forces the lookup mapping to be on
the frame: different executions of the same code object may reference
different cell objects.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S66QCDWYP3B73HF7PK6S55NGNHYVC2S2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Sun, 22 Aug 2021, 10:47 am Guido van Rossum,  wrote:

> Hopefully anyone is still reading python-dev.
>
> I'm going to try to summarize the differences between the two proposals,
> even though Mark already did so in his PEP. But I'd like to start by
> calling out the key point of contention.
>
> Everything here is about locals() and f_locals in *function scope*. (I use
> f_locals to refer to the f_locals field of frame objects as seen from
> Python code.) And in particular, it is about what I'll call "extra
> variables": the current CPython feature that you can add *new* variables to
> f_locals that don't exist in the frame, for example:
>
> def foo():
> x = 1
> locals()["y"] = 2  # or sys._getframe()["y"] = 2
>
> My first reaction was to propose to drop this feature, but I realize it's
> kind of important for debuggers to be able to execute arbitrary code in
> function code -- assignments to locals should affect the frame, but it
> should also be possible to create new variables (e.g. temporaries). So I
> agree we should keep this.
>

I actually tried taking this feature out in one of the PEP 558 drafts, but
actually doing so breaks the pdb test suite.



>
> So apparently the key difference of opinion between Mark and Nick is about
> f_locals, and what to do with extras. In Nick's proposal when you reference
> f.f_locals twice in a row (for the same frame object f), you get the same
> proxy object, whereas in Mark's proposal you get a different object each
> time, but it doesn't matter, because the proxy has no state other than a
> reference to the frame.
>

If PEP 558 is still giving that impression, I need to fix the wording - the
proxy objects are ephemeral in both PEPs (the 558 text is slightly behind
the implementation on that point, as the fast refs mapping is now stored on
the frame object, so it only needs to be built once)

In Mark's proposal, if you assign a value to an extra variable, it gets
> stored in a hidden dict field on the frame, and when you read the proxy,
> the contents of that hidden dict field gets included. This hidden dict
> lazily created on the first store to an extra variable. (Mark shows
> pseudo-code to clarify this; the hidden dict is stored as _extra_locals on
> the frame.)
>

PEP 558 works essentially the same way, the difference is that it uses the
existing locals dict storage rather than adding new storage just for
optimised frames.

In Nick's proposal, there's a cache on the frame that stores both the
> extras and the proper variables. This cache can get out of sync with the
> contents of the proper variables when some bytecode is executed (for
> performance reasons we don't want the bytecode to keep the cache up to date
> on every store), so there's an operation to sync the frame cache
> (sync_frame_cache(), it's not defined in which namespace this exists -- is
> it a builtin or in sys?).
>

It's an extra method on the proxy objects. You only need it if you keep an
old proxy object around - if you always retrieve a new proxy object after
executing Python code, that proxy will refresh the cache when it needs to.



> Frankly the description in Nick's PEP is hard to follow -- I am not 100%
> sure what is meant by "the dynamic snapshot", and it's not quite clear
> whether proper variables are copied into the cache (and if so, why).
>

Aye, Mark was a bit quicker with his PEP than I anticipated, so I've
incorporated the implementation improvements arising from his last round of
comments, but the PEP text hasn't been updated yet.


Personally, I find Mark's proposed semantics for f_locals simpler --
> there's no cache, only storage for extras, so there's nothing that can get
> out of sync.
>

The wording in PEP 667 undersells the cost of that simplification:

"Code that uses PyEval_GetLocals() will continue to operate safely, but
will need to be changed to use PyEval_Locals() to restore functionality."


Code that uses PyEval_GetLocals() will NOT continue to operate safely under
PEP 667: all such code will raise an exception at runtime, and need to be
rewritten to use a new API with different refcounting semantics. That's
essentially all code that accesses the frame locals from C, since we don't
offer supported APIs for that other than PyEval_GetLocals() (directly
accessing the f_locals field on the frame object is only "supported" in a
very loose sense of the word, although PEP 558 mostly keeps that working,
too)

This means the real key difference between the two PEPs is that Mark is
proposing a gratuitous compatibility break for PyEval_GetLocals() that also
means that the algorithmic complexity characteristics of the proxy
implementation will be completely off from those of a regular dict (e.g.
len(proxy) will be O(n) in the number of variables defined on the frame
rather than being O(1) after the proxy's initial cache update the way it is
in PEP 558)

If Mark's claim that PyEval_GetLocals() could not be fixed was true then I
would be more sympathetic to 

[Python-Dev] Re: Making code object APIs unstable

[Nick Coghlan]

On Tue, 17 Aug 2021, 4:30 am Guido van Rossum,  wrote:

> On Mon, Aug 16, 2021 at 9:30 AM Steve Dower 
> wrote:
>
>> On 8/16/2021 12:47 AM, Guido van Rossum wrote:
>> > My current proposal is to issue a DeprecationWarning in PyCode_New()
>> and
>> > PyCode_NewWithPosArgs(), which can be turned into an error using a
>> > command-line flag. If it's made an error, we effectively have (B); by
>> > default, we have (A).
>> >
>> > Then in 3.13 we can drop them completely.
>>
>> We definitely had legitimate use cases come up when adding positional
>> arguments (hence the new API, rather than breaking the existing one,
>> which was the first attempt at adding the feature).
>>
>> I don't recall exactly what they are (perhaps Pablo does, or they may be
>> in email/issue archives), but since they exist, presumably they are
>> useful and viable _despite_ the bytecode varying between releases. This
>> suggests there's probably a better API we should add at the same time -
>> possibly some kind of unmarshalling or cloning-with-updates function?
>>
>
> I presume the use cases are essentially some variant of the .replace() API
> that exists at the Python level. At the C level you would get all fields
> from an existing code object and pass them to PyCode_New[WithPosArgs]
> except for e.g. the co_filename field. Unfortunately those use cases will
> still break if there are any try blocks in the code or if the
> endline/column info is needed. Also you can't access any of the code
> object's fields without the internal API (this wasn't always so). So I
> think it's different now.
>

A cloning-with-replacement API that accepted the base code object and the
"safe to modify" fields could be a good complement to the API deprecation
proposal.

Moving actual "from scratch" code object creation behind the Py_BUILD_CORE
guard with an underscore prefix on the name would also make sense, since it
defines a key piece of the compiler/interpreter boundary.

Cheers,
Nick.

P.S. Noting an idea that won't work, in case anyone else reading the thread
was thinking the same thing: a "PyType_FromSpec" style API won't help here,
as the issue is that the compiler is now doing more work up front and
recording that extra info in the code object for the interpreter to use.
There is no way to synthesise that info if it isn't passed to the
constructor, as it isn't intrinsically recorded in the opcode sequence.


>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/K2PUYK6MHXILWCBDWZZIUORIALI6SP4A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YIKXHLNR4FYP7ZU5PUTRIBNHENIXEM7A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558, the simplest thing I could come up with

[Nick Coghlan]

Bringing the public record up to date with a brief off-list discussion
between Mark, Nathaniel and I:

* Mark hasn't convinced me that getting rid of the frame value cache
entirely for optimised frames is a good idea, so he's going to write that
proposal up as a competing PEP. Once it has been drafted and is ready for
review, he will request the SC assign a PEP delegate.
* On the PEP 558 front, Mark's proposal has highlighted a few
inefficiencies in my reference implementation, where the code still
implicitly updates the frame value cache in cases where the cache being up
to date may not matter to the proxy API client. So I'll be working on
another iteration of the implementation that ensures each caching proxy
instance (at worst) only pays the O(N) cache refresh price on the first
less than O(N) operation that relies on the cache being up to date, rather
than paying it every time "f_locals" is retrieved from the frame object.

We still have plenty of time before 3.11b1, so we expect it will be a month
or two before the two proposals are in a position to be compared directly.

Cheers,
Nick.

On Fri, 30 Jul 2021, 5:25 pm Nick Coghlan,  wrote:

>
>
> On Fri, 30 Jul 2021, 2:30 pm Nathaniel Smith,  wrote:
>
>>
>> >
>> > For [proxy] versus [snapshot], a lot depends on what we think of
>> changing the semantics of exec(). [proxy] is definitely more consistent and
>> elegant, and if we could go back in time I think it's
>> what we'd have done from the start. Its compatibility is maybe a bit
>> worse than [snapshot] on non-exec() cases, but this seems pretty minor
>> overall (it often doesn't matter, and if it does just write
>> dict(locals()) instead of locals(), like you would in non-function
>> scope). But the change in exec() semantics is an actual language
>> change, even though it may not affect much real code, so that's what
>> stands out for me.
>>
>> I *think* (please correct me if I'm wrong) that what that calls
>> [PEP-minus-tracing] is now corresponds to the current PEP draft, and
>> [proxy] corresponds to Mark's draft at the beginning of this thread?
>>
>
> At the locals() level, PEP 558 is now [snapshot] (due to your original
> analysis showing that it was strictly better than what I had at the time),
> while Mark's draft is indeed [proxy].
>
> Cheers,
> Nick.
>
>
>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EU2TU6R2HDV6NLZQ56MSDIRMG6EHSOJO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558, the simplest thing I could come up with

[Nick Coghlan]

On Fri, 30 Jul 2021, 2:30 pm Nathaniel Smith,  wrote:

>
> >
> > For [proxy] versus [snapshot], a lot depends on what we think of
> changing the semantics of exec(). [proxy] is definitely more consistent and
> elegant, and if we could go back in time I think it's
> what we'd have done from the start. Its compatibility is maybe a bit
> worse than [snapshot] on non-exec() cases, but this seems pretty minor
> overall (it often doesn't matter, and if it does just write
> dict(locals()) instead of locals(), like you would in non-function
> scope). But the change in exec() semantics is an actual language
> change, even though it may not affect much real code, so that's what
> stands out for me.
>
> I *think* (please correct me if I'm wrong) that what that calls
> [PEP-minus-tracing] is now corresponds to the current PEP draft, and
> [proxy] corresponds to Mark's draft at the beginning of this thread?
>

At the locals() level, PEP 558 is now [snapshot] (due to your original
analysis showing that it was strictly better than what I had at the time),
while Mark's draft is indeed [proxy].

Cheers,
Nick.



> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZQM44JMPRJ6ISZ2Z6BBYMPDRP3JBEMYV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558, the simplest thing I could come up with

[Nick Coghlan]

On Fri, 30 Jul 2021, 6:05 am Mark Shannon,  wrote:

> Hi Nick,
>
> Our discussion on PEP 558 got me thinking
> "What is the simplest thing that would work?".
>
> This is what I came up (in the form of a draft PEP):
> https://github.com/markshannon/peps/blob/pep-locals/pep-06xx.rst
>
> It doesn't have O(1) len(f_locals), and it does break
> `PyEval_GetLocals()` but I think the that is a small price to pay for
> simplicity and consistency.
>

I don't think it is OK to break PyEval_GetLocals() when we really don't
need to,
and the proposal also discards all the feedback that I received on earlier
iterations of PEP 558. (I particularly recommend reading Nathaniel's
analysis of why returning the proxy from locals() would be more likely to
cause bugs in existing code than it would be to eliminate any).

As it looks like we have some fairly fundamental disagreements regarding
the level of behaviour change and API breakage that is acceptable for the
sake of a simpler implementation, I think we're going to need competing
PEPs for the SC to evaluate (or delegate to Nathaniel for evaluation) if
you want to pursue that path further.


(I'm only OK with explicitly breaking PyEval_LocalsToFast because the
write-back strategy is already intrinsically broken. By contrast,
maintaining a frame cache for optimised frames isn't fundamentally flawed,
it's just a matter of figuring out when and how to pay the cost of updating
it to avoid wasting those CPU cycles in code that doesn't care if the cache
is up to date or not)

Cheers,
Nick.


>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/57BOZNO3GCVUDJ6FPF2OCKMTHE4U36Q2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 467 feedback from the Steering Council

[Nick Coghlan]

On Fri, 30 Jul 2021, 8:47 am Barry Warsaw,  wrote:

>
> Hello Nick, Ethan,
>
> The Python Steering Council reviewed PEP 467 -- Minor API improvements for
> binary sequences at our 2021-07-26 meeting.
>
> Thank you for work on this PEP.  We’re generally very favorable for adding
> to Python 3.11 the features and APIs described in the PEP.


Thank you!


>
> Let us know what you think about making these changes.  We aren’t making
> acceptance contingent on these changes, but we do think they make the PEP
> and the new APIs better.
>


Those changes all sound reasonable to me, so if Ethan is also amenable, I
think we should incorporate them.

Cheers,
Nick.


> -Barry (on behalf of the Python Steering Council)
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5DW4PRS3W4Q5KLARYKS3Y2QP2RNFZSFA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Wed, 28 Jul 2021, 8:13 pm Mark Shannon,  wrote:

>
>
> On 28/07/2021 1:03 am, Nick Coghlan wrote:
> >
> >
> > On Wed, 28 Jul 2021, 1:50 am Mark Shannon,  > <mailto:m...@hotpy.org>> wrote:
> >
> >     Hi Nick,
> >
> > On 27/07/2021 2:29 pm, Nick Coghlan wrote:
> >  >
> >  >
> >  > The reference documentation should be precise as well, since that
> is
> >  > what other implementations will be following.
> >  >
> >  > What semantics do you feel are left unspecified?
> >  >
> >  >> The documentation needs to explain the behavior to the majority
> of
> >  >> users, but the PEP should be worded so that it can be implemented
> >  >> correctly from just reading the PEP.
> >  >>
> >  >> There is no reason to remove the sugested documentation changes,
> but
> >  >> they should be secondary to the more formal specification.
> >  >
> >  > I don't know what you want to see on this front.
> >  >
> >  > The current PEP seems complete to me, except in relation to the
> >  > specifics of the proxy behaviour, which it deliberately leaves
> >  > underspecified (mostly because my current implementation sucks in
> a
> >  > few areas, and I really don't want to enshrine those limitations
> in
> >  > the language spec).
> >
> > All the above were mostly just suggestions, the format of the PEP is
> up
> > to you.
> >
> > But please, do not underspecify anything. I need to maintain this,
> > as do
> > the PyPy and MicroPython developers. We need this to be precise.
> >
> >
> > You still haven't said what you consider underspecified.
>
> You said it was underspecified, not me. Quoting you from a few lines up
> "which it deliberately leaves underspecified"
>

I said the proxy was under specified. The entire frame API is a CPython
implementation detail that some other implementations emulate, so I'm OK
with that.

You were claiming the *locals()* behaviour was under specified, so I want
to know what is unclear.


> Perhaps read the implementation and tell me which parts of what the
> > tests are covering you want to see replicated in the PEP text?
> >
> >
> >  >
> >  >> Resolving the issues with tracing mode behaviour
> >  >> 
> >  >>
> >  >> According to this section the `f_locals` object
> > (_PyFastLocalsProxy_Type
> >  >> in C) will have *two* fields.
> >  >> It should only have *one*, the pointer to the underlying frame
> > object.
> >  >
> >  > That's a good point - fast_refs doesn't reference the frame
> > object, so
> >  > it's safe to store it on the frame object and reference it from
> the
> >  > individual proxies.
> >  >
> >  > Switching to that approach also has the advantage that every proxy
> >  > after the first is much cheaper, since they don't need to build
> the
> >  > fast refs map again.
> >  >
> >  >> In order to maximize backwards compatibility and, more
> importantly,
> >  >> avoid the synchronization issues that lead to obscure bugs, all
> the
> >  >> state of the `f_locals` object should be kept on the frame.
> >  >>
> >  >> Any changes to `f_locals` should be instantly visible to any
> > thread both
> >  >> through other `f_locals` objects (for the same frame) and the
> > underlying
> >  >> local variable (if it exists).
> >  >
> >  > [snip proxy method implementation sketch] >
> >  > This gets the gist of how the implementation works, but it isn't
> > quite
> >  > that simple for a few reasons:
> >  >
> >  > 1. The proxy needs to offer the same algorithmic complexity as
> dict
> >  > APIs, so doing O(n) linear searches of C arrays and Python tuples
> >  > inside O(1) lookup operations isn't OK (hence the fastrefs
> mapping to
> >  > allow for O(1) resolution of variable names to cell references and
> >  > local variable array offsets)
> >
> > Linear searches are faster for small arrays, and there is nothing
> > stopping you adding a helper map of names->indexes for large
> functions.
> >

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Wed, 28 Jul 2021, 1:50 am Mark Shannon,  wrote:

> Hi Nick,
>
> On 27/07/2021 2:29 pm, Nick Coghlan wrote:
> >
> >
> > The reference documentation should be precise as well, since that is
> > what other implementations will be following.
> >
> > What semantics do you feel are left unspecified?
> >
> >> The documentation needs to explain the behavior to the majority of
> >> users, but the PEP should be worded so that it can be implemented
> >> correctly from just reading the PEP.
> >>
> >> There is no reason to remove the sugested documentation changes, but
> >> they should be secondary to the more formal specification.
> >
> > I don't know what you want to see on this front.
> >
> > The current PEP seems complete to me, except in relation to the
> > specifics of the proxy behaviour, which it deliberately leaves
> > underspecified (mostly because my current implementation sucks in a
> > few areas, and I really don't want to enshrine those limitations in
> > the language spec).
>
> All the above were mostly just suggestions, the format of the PEP is up
> to you.
>
> But please, do not underspecify anything. I need to maintain this, as do
> the PyPy and MicroPython developers. We need this to be precise.
>

You still haven't said what you consider underspecified.

Perhaps read the implementation and tell me which parts of what the tests
are covering you want to see replicated in the PEP text?


> >
> >> Resolving the issues with tracing mode behaviour
> >> 
> >>
> >> According to this section the `f_locals` object (_PyFastLocalsProxy_Type
> >> in C) will have *two* fields.
> >> It should only have *one*, the pointer to the underlying frame object.
> >
> > That's a good point - fast_refs doesn't reference the frame object, so
> > it's safe to store it on the frame object and reference it from the
> > individual proxies.
> >
> > Switching to that approach also has the advantage that every proxy
> > after the first is much cheaper, since they don't need to build the
> > fast refs map again.
> >
> >> In order to maximize backwards compatibility and, more importantly,
> >> avoid the synchronization issues that lead to obscure bugs, all the
> >> state of the `f_locals` object should be kept on the frame.
> >>
> >> Any changes to `f_locals` should be instantly visible to any thread both
> >> through other `f_locals` objects (for the same frame) and the underlying
> >> local variable (if it exists).
> >
> > [snip proxy method implementation sketch] >
> > This gets the gist of how the implementation works, but it isn't quite
> > that simple for a few reasons:
> >
> > 1. The proxy needs to offer the same algorithmic complexity as dict
> > APIs, so doing O(n) linear searches of C arrays and Python tuples
> > inside O(1) lookup operations isn't OK (hence the fastrefs mapping to
> > allow for O(1) resolution of variable names to cell references and
> > local variable array offsets)
>
> Linear searches are faster for small arrays, and there is nothing
> stopping you adding a helper map of names->indexes for large functions.
> Provided the mapping from name to local index is O(1), then the whole
> operation is O(1).
>

That helper map already exists in the fast refs mapping. Skipping it for
small functions would be a future optimisation opportunity that doesn't
seem necessary in the initial implementation.


> You really don't need much extra machinery to maintain O(1) behavior
> (which no one cares about, they just care about overall performance).
>

Yes, the only extra machinery needed for working with individual keys is
the fast refs mapping.

It's other aspects of the mapping API that benefit from the continuing
presence of the  frame cache (which is maintained on the frame, just as it
is today, NOT separately in each proxy).

If you can give me a reliable O(1) implementation of "len(frame.f_locals)"
that doesn't rely on up to date cache of:
* which local and cell variable names are currently bound to values
* which extra variables have been added via either proxies or
PyEval_GetLocals()



> 2. The proxy needs to deal with *SIX* kinds of possible key lookup, not
> two:
> >
> > * unbound local variable (numeric index in fast refs map, NULL in fast
> > locals slot)
> > * bound local variable (numeric index in fast refs map, object pointer
> > in fast locals slot)
> > * unbound cell variable (cell reference in fast refs map, cell contains
> NULL)
> > * bound cell variable (

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

 return locals;
>  return PyDict_Copy(locals); // This leaks, need to decref locals

The proposed API does offer that functionality directly:
PyLocals_GetCopy() and PyFrame_GetLocalsCopy(f).

> Reducing the runtime overhead of trace hooks
> 
>
> I'm confused by this part.
> Since `_PyFrame_FastToLocals` and friends do not alter the logical state
> of the frame object (or anything else), they should no-ops.
> It is `PyFrame_GetLocals()` that creates the proxy object, surely.

_PyFrame_FastToLocals() still refreshes the frame cache that is used
as the return value from PyEval_GetLocals(), the same way it always
has.

The frame cache is implicitly kept in sync by operations on frame
proxy objects, but there are still two ways for it to get out of sync:
* code execution inside the frame (this updates the fast locals array,
not the key/value cache stored in the C level f_locals attribute)
* direct manipulation of the frame cache by callers of
PyEval_GetLocals (the frame will ignore any attempts to bind or unbind
variables that way, but it can still throw off proxy operations that
rely on the cache)

The reason the frame cache still exists in the first place is because
there are a number of mapping APIs where the easiest way to meet the
algorithmic complexity expectations of the dict API is to use an
actual dict:

= Length checking =

The fast_refs mapping (and the frame metadata in general), includes
all variables defined for the frame, regardless of whether they're
currently bound or not.
The correct length for the fast locals proxy mapping only includes the
*currently bound* fast local and cell variables, together with any
extra keys that have been added.
The locals proxy keeps this an O(1) operation by reporting the size of
the frame cache, so it doesn't have to check which names are currently
bound every time.

= Mapping comparisons =

Mapping comparisons have an O(1) early exit if their lengths don't
match, before falling back to a full O(N) key value comparison. The
locals proxy takes advantage of that by delegating comparison
operations to the frame cache rather than reimplementing them.

= Iteration =

Rather than reimplementing forward and reverse iteration and the
keys(), values(), and items() methods, the mapping proxy just
delegates these operations to the frame cache.

Operations on the proxy other than these ones either don't rely on the
cache being up to date at all (because they can use the fast_refs
mapping instead, implicitly updating any affected cache entries along
the way), or else implicitly refresh the cache before relying on it
(e.g. copying the proxy mapping works by refreshing the frame cache,
then copying the cache).

I assume there will be ways to improve the implementation to make
explicit frame cache refreshes less necessary over time (hence the
caveat along those lines in the final paragraph of
https://www.python.org/dev/peps/pep-0558/#continuing-to-support-storing-additional-data-on-optimised-frames
), but the existence of the PyEval_GetLocals() API means that we're
highly unlikely to ever get rid of the frame cache entirely.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H2YVJXADIXCZBHML7WOSQYNJQ6BTRPVC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: types.Union or types.UnionType?

[Nick Coghlan]

On Sun, 25 Jul 2021, 9:26 pm Serhiy Storchaka,  wrote:

> In 3.10 the union type (the type of the result of the | operator for
> types) was added (https://www.python.org/dev/peps/pep-0604/). It is
> exposed as types.Union. There are differences between typing.Union and
> types.Union:
>
> * typing.Union is indexable, types.Union is not.
> * types.Union is a class, typing.Union is not.
>
> types.Union corresponds to private class typing._UnionGenericAlias, not
> typing.Union. It is confusing that typing.Union and types.Union have the
> same name but are so different. Note also that most classes in the types
> module have the "Type" suffix: FunctionType, MethodType, ModuleType,
> etc. I think that it would be better to rename types.Union to
> types.UnionType.
>

If we wanted to be completely explicit, the most exact name would be
"TypeUnionType": it is the type of object you get specifically when
unioning types, not when unioning arbitrary objects.

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3O6STJNH6NCNWEBOGV3OB3DC6M2K47RN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: enum in the stable ABI (Was: PEP 558: Defined semantics for locals)

[Nick Coghlan]

On Sat, 24 Jul 2021, 9:37 am Larry Hastings,  wrote:

>
> On 7/23/21 7:38 AM, Petr Viktorin wrote:
>
> (In both C & C++, the size of an `enum` is implementation-defined. That's
> unlikely to be a problem in practice, but one more point against enum.)
>
>
> True, but there's always the old trick of sticking in a value that forces
> it to be at least 32-bit:
>
> typedef enum {
> INVALID = 0,
> RED = 1,
> BLUE = 2,
> GREEN = 3,
>
> UNUSED = 1073741824
> } color_t;
>
>
> */arry*
>

My current inclination is to define the enum as "_PyLocals_KindValues", and
then typedef "PyLocals_Kind" itself as an int. The frame API would then
return the former, while the stable query API would return the latter.

However, I'll make a full survey of the enums currently in the stable ABI
before making a decision, as there may be an existing approach that I like
better.

I do agree Petr's right to be cautious about this, as compilers can get up
to some arcane shenanigans in the presence of formally undefined code:
https://queue.acm.org/detail.cfm?id=3468263

The fact that the behaviour in this case is likely to be well-defined at
the time of compilation would protect us from the weirder potential
outcomes, but it still makes sense for us to define the query API in a way
that tells both compilers and humans not to assume that the values returned
by the current version of Python are the only values that will ever be
returned by all future versions of Python.

Cheers,
Nick.


> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/3C5ANX5ONLE6OWZ4N24ENDOX2H3R5UC2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RNR6AVDT45PEKPY6TLQGF7ZPAB4NRBCD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bug report

[Nick Coghlan]

On Fri, 23 Jul 2021, 2:22 am Ee Durbin,  wrote:

> On Wed, Jul 21, 2021 at 7:45 PM Kyle Stanley  wrote:
>
>> On Wed, Jul 21, 2021 at 12:19 PM Nguyen Do Minh Duc <
>> gd.minhduc1...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When I find what's new in 3.10 beta in
>>> https://docs.python.org/whatsnew/3.10.html
>>> It redirected me to https://docs.python.org/3/whatsnew/3.10.html which
>>> shows 404 error not found nginx. Can you fix this?
>>>
>>
>> Thanks for the report! I don't believe the code and/or config for the web
>> server hosting docs.python.org is a public repo (may be mistaken), so
>> I'll CC the director of infrastructure, Ee W. Durbin. I suspect they will
>> know who to forward this to (or be able to fix it).
>>
>
> Config that’s responsible for redirects lives at
>
> https://github.com/python/psf-salt/blob/master/salt/docs/config/nginx.docs-backend.conf
> if anyone is able to take a look before I’m able to ascertain what’s going
> on.
>
> Do we know what the expected behavior is?
>

The expected behaviour is what is currently happening:
https://github.com/python/psf-salt/blob/6cdc61db0868d81514fde95ab5dfe11914e49754/salt/docs/config/nginx.docs-backend.conf#L19

3.10 is still in its pre-release cycle, so pages that don't exist in the
3.9 documentation (like the 3.10 What's New doc) need a "dev" or "3.10" in
their URL to be reachable.

Only when 3.10 is released and becomes the default version for the Python 3
docs would we expect the OP's link to work.

That said, if nginx config allows for rewrites that only happen when a 404
is about to be emitted otherwise, it would make sense to have "/3/" URLs
fall back to "/dev/" URLs before giving up and emitting the 404.

Cheers,
Nick.


___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/3M4KRVNWXSC56VJK2E7O45DX3UJLBYM5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6IXBHB5V6ROSOZPWDJN3CZ6WHCUX44TW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bug report

[Nick Coghlan]

On Thu, 22 Jul 2021, 9:47 am Kyle Stanley,  wrote:

> On Wed, Jul 21, 2021 at 12:19 PM Nguyen Do Minh Duc <
> gd.minhduc1...@gmail.com> wrote:
>
>> Hi,
>>
>> When I find what's new in 3.10 beta in
>> https://docs.python.org/whatsnew/3.10.html
>> It redirected me to https://docs.python.org/3/whatsnew/3.10.html which
>> shows 404 error not found nginx. Can you fix this?
>>
>
> Thanks for the report! I don't believe the code and/or config for the web
> server hosting docs.python.org is a public repo (may be mistaken), so
> I'll CC the director of infrastructure, Ee W. Durbin. I suspect they will
> know who to forward this to (or be able to fix it).
>

As far as I know, it's a bulk redirect of otherwise unmatched deep links to
the Python 3 docs, so the OPs link won't work until 3.10 is officially
released.

Until then, only the version specific and dev URLs will work.

Cheers,
Nick.



> Best Regards,
> --
> --Kyle R. Stanley, Python Core Developer (what is a core dev?
> )
> *Pronouns: they/them **(why is my pronoun here?*
> 
> )
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WAJM6PFMX2DXOGPEC6I2QATAUT2GNJXC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EMZUUJMCAHH5A3NFVOIFFWXVCNY77LJN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558: Defined semantics for locals()

[Nick Coghlan]

On Thu, 22 Jul 2021, 6:01 pm Petr Viktorin,  wrote:

>
>
> On 21. 07. 21 14:18, Nick Coghlan wrote:
> > On Mon, 19 Jul 2021 at 21:32, Petr Viktorin  wrote:
> >> The proposal assumes that in the future, ``PyLocals_Get``, and thus
> >> ``locals()``, will never gain another kind of return value, however
> >> unlikely that is.
> >> AFAICS, code that uses this will usually check for a single special case
> >> and fall back (or error) for the other(s), so I think it'd be reasonable
> >> to make this an "enum" with two values. e.g.:
> >>
> >> int PyLocals_GetReturnBehavior();  # better name?
> >> #define PyLocals_DIRECT_REFERENCE 0
> >> #define PyLocals_SHALLOW_COPY 1
> >
> > After looking at PyUnicode_Kind, PySendResult, and other already
> > public enums for inspiration, my proposed spelling is as follows:
> >
> > 
> > typedef enum {
> >  PyLocals_UNDEFINED = -1;
> >  PyLocals_DIRECT_REFERENCE = 0,
> >  PyLocals_SHALLOW_COPY = 1
> > } PyLocals_Kind;
> >
> > PyLocals_Kind PyLocals_GetKind(void);
> > PyLocals_Kind PyFrame_GetLocalsKind(PyFrameObject *);
> > 
> >
> > The PyLocals_UNDEFINED case comes from PyLocals_GetKind() needing an
> > error value to return when the query API is called with no active
> > thread state.
> >
> > I've updated the draft reference implementation to use this API, and
> > added the associated PEP changes to the review PR at
> > https://github.com/python/peps/pull/2038/files
>
> Please don't put the enum in the stable ABI. If we would add another
> value and then an older extension would receive it, we'd get undefined
> behavior.
>

Hmm, I was copying an example that is already in the stable ABI
(PySendResult).

I think it's new in 3.10, though, so it should still be possible to fix
that.

Cheers,
Nick.

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IEZADZNWFHVDIYVWTITW7ZSS27NIQDSB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC for PEP 663: Improving and Standardizing Enum str(), repr(), and format() behaviors

[Nick Coghlan]

Thanks for this Ethan.

I don't have any substantive comments on what you're proposing (aside
from "Yes, that sounds reasonable to me"), so my comments below are
just some minor suggested clarifications for the PEP text.

On Wed, 21 Jul 2021 at 14:17, Ethan Furman  wrote:
> IntEnum, IntFlag, and StrEnum should be as close to a drop-in replacement of
> existing integer and string constants as is possible.  Towards that goal, the
> str() output of each should be its inherent value; i.e.::
>
>  >>> Color.RED
>  
>  >>> str(Color.RED)
>  1
>  >>> format(Color.RED)
>  '1'

The suggested output here looks like the second line was
`print(Color.RED)` rather than `str(Color.RED)` (since the quotes on
the string are missing).

The text should also state explicitly that `Color` in the example is
an `IntEnum` subclass.

> Note that format() already produces the correct output, only str() needs
> updating.

This paragraph may be a little clearer if it said "... correct output
in Python 3.10, only str() ...".


> Using the above enumerations, the following table shows the old and new
> behavior, while the last shows the final result:

Since the enum output changes were reverted for 3.10, replacing "3.9"
with "3.10" in the tables may make it a little clearer that there was
no intermediate step between 3.9 and 3.11.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YF5QU23VJRVJINKXEPXHUI35WH5ZLDIF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558: Defined semantics for locals()

[Nick Coghlan]

On Mon, 19 Jul 2021 at 21:32, Petr Viktorin  wrote:
> The proposal assumes that in the future, ``PyLocals_Get``, and thus
> ``locals()``, will never gain another kind of return value, however
> unlikely that is.
> AFAICS, code that uses this will usually check for a single special case
> and fall back (or error) for the other(s), so I think it'd be reasonable
> to make this an "enum" with two values. e.g.:
>
> int PyLocals_GetReturnBehavior();  # better name?
> #define PyLocals_DIRECT_REFERENCE 0
> #define PyLocals_SHALLOW_COPY 1

After looking at PyUnicode_Kind, PySendResult, and other already
public enums for inspiration, my proposed spelling is as follows:


typedef enum {
PyLocals_UNDEFINED = -1;
PyLocals_DIRECT_REFERENCE = 0,
PyLocals_SHALLOW_COPY = 1
} PyLocals_Kind;

PyLocals_Kind PyLocals_GetKind(void);
PyLocals_Kind PyFrame_GetLocalsKind(PyFrameObject *);


The PyLocals_UNDEFINED case comes from PyLocals_GetKind() needing an
error value to return when the query API is called with no active
thread state.

I've updated the draft reference implementation to use this API, and
added the associated PEP changes to the review PR at
https://github.com/python/peps/pull/2038/files

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QLPJWPVNDZ3ODMS275VDW4M2UW6PUW5N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558: Defined semantics for locals()

[Nick Coghlan]

On Wed, 21 Jul 2021, 10:30 am Guido van Rossum,  wrote:

> On Tue, Jul 20, 2021 at 4:52 AM Nick Coghlan  wrote:
> [Petr]
>
>> >> int PyLocals_GetReturnBehavior();  # better name?
>>
> [Nick]
>
>> > We've used "Kind" for similar APIs elsewhere, so calling this API
>> "PyLocals_Kind()" would make sense to me.
>> >
>> > However, there's a potential point of confusion here, as there's
>> already an implementation level "locals kind" that the runtime uses. This
>> public kind is related to that internal kind, but they're not the same.
>>
>> [Nick, responding to himself]
>
>> Now that I'm on my actual computer rather than my phone, some further
>> details on the new-in-Python-3.11 internal-only API that already
>> refers to "locals kind":
>>
>> Typedef name: _PyLocals_Kind
>> Value flags: CO_FAST_LOCAL, CO_FAST_CELL, CO_FAST_FREE (with more
>> expected to be defined in the future)
>> Frame local variable lookup API: _PyLocals_GetKind, _PyLocals_SetKind
>>
>> Rather than relating to the frame as a whole, these flags relate to
>> individual variables on the frame. CO_FAST_FREE can even be used on
>> non-optimised frames, since classes defined inside functions have
>> access to variables defined in the function scope.
>>
>> Due to this, I'd definitely want to hear Mark Shannon's opinion before
>> we went down the path of using "PyLocals_Kind" (or variations on that
>> theme) as a public API, since we'd need to rename the internal APIs to
>> avoid confusion if we did that.
>>
>> I don't think a rename would be too bad though - since the existing
>> flags relate to individual local variables, my suggestion would be to
>> use `LocalVar` instead of `Locals`, giving the revised enum name
>> _PyLocalVar_Kind, and _PyLocalVar_GetKind and _PyLocalVar_SetKind as
>> the access and update methods. (There are only 27 hits on the
>> _PyLocals prefix across the current 3.11 code base, all relating to
>> this internal API)
>>
>
> My name isn't Mark Shannon, but I was involved in picking this name. I
> think it's fine to rename this concept to "local variable kind" and spell
> the typedef as you propose, though I think I'd like it slightly better to
> switch from "Locals" to "Local", and to put the underscore after that:
> `_PyLocal_VarKind`, `_PyLocal_GetVarKind()`, `_PyLocal_SetVarKind()`. (The
> API for this is very much in flux -- we'll settle by 3.11 beta 1 though. I
> expect we'll also have some Python-level public (but unstable) API to
> access these flags, so people can play around with this stuff.)
>

Thanks - that works for me. I'll update the draft implementation
accordingly, and add this API tweak to the pending changes for the PEP (I
won't merge any changes until Nathaniel has had a chance to comment on the
posted version, though)

Cheers,
Nick.


> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IWZ33GSLBJ6DWVOEVITSAPUEQIHNSPM4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558: Defined semantics for locals()

[Nick Coghlan]

On Mon, 19 Jul 2021 at 22:08, Nick Coghlan  wrote:
> On Mon, 19 Jul 2021, 9:32 pm Petr Viktorin,  wrote:
>> The proposal assumes that in the future, ``PyLocals_Get``, and thus
>> ``locals()``, will never gain another kind of return value, however
>> unlikely that is.
>> AFAICS, code that uses this will usually check for a single special case
>> and fall back (or error) for the other(s), so I think it'd be reasonable
>> to make this an "enum" with two values. e.g.:
>>
>> int PyLocals_GetReturnBehavior();  # better name?
>
>
> We've used "Kind" for similar APIs elsewhere, so calling this API 
> "PyLocals_Kind()" would make sense to me.
>
> However, there's a potential point of confusion here, as there's already an 
> implementation level "locals kind" that the runtime uses. This public kind is 
> related to that internal kind, but they're not the same.

Now that I'm on my actual computer rather than my phone, some further
details on the new-in-Python-3.11 internal-only API that already
refers to "locals kind":

Typedef name: _PyLocals_Kind
Value flags: CO_FAST_LOCAL, CO_FAST_CELL, CO_FAST_FREE (with more
expected to be defined in the future)
Frame local variable lookup API: _PyLocals_GetKind, _PyLocals_SetKind

Rather than relating to the frame as a whole, these flags relate to
individual variables on the frame. CO_FAST_FREE can even be used on
non-optimised frames, since classes defined inside functions have
access to variables defined in the function scope.

Due to this, I'd definitely want to hear Mark Shannon's opinion before
we went down the path of using "PyLocals_Kind" (or variations on that
theme) as a public API, since we'd need to rename the internal APIs to
avoid confusion if we did that.

I don't think a rename would be too bad though - since the existing
flags relate to individual local variables, my suggestion would be to
use `LocalVar` instead of `Locals`, giving the revised enum name
_PyLocalVar_Kind, and _PyLocalVar_GetKind and _PyLocalVar_SetKind as
the access and update methods. (There are only 27 hits on the
_PyLocals prefix across the current 3.11 code base, all relating to
this internal API)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4KYGTJKJ7USCKJ7IDHXGTQPSFWV46IPA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 558: Defined semantics for locals()

[Nick Coghlan]

On Mon, 19 Jul 2021, 9:32 pm Petr Viktorin,  wrote:

> Thanks, Nick! This looks wonderful.
> I do have a nitpick, below:
>
> On 18. 07. 21 7:59, Nick Coghlan wrote:
> [...]>
> > Changes to the stable C API/ABI
> > ---
> >
> > Unlike Python code, extension module functions that call in to the
> Python C API
> > can be called from any kind of Python scope. This means it isn't obvious
> from
> > the context whether ``locals()`` will return a snapshot or not, as it
> depends
> > on the scope of the calling Python code, not the C code itself.
> >
> > This means it is desirable to offer C APIs that give predictable, scope
> > independent, behaviour. However, it is also desirable to allow C code to
> > exactly mimic the behaviour of Python code at the same scope.
> >
> > To enable mimicking the behaviour of Python code, the stable C ABI would
> gain
> > the following new functions::
> >
> >  PyObject * PyLocals_Get();
> >  int PyLocals_GetReturnsCopy();
> >
> > ``PyLocals_Get()`` is directly equivalent to the Python ``locals()``
> builtin.
> > It returns a new reference to the local namespace mapping for the active
> > Python frame at module and class scope, and when using ``exec()`` or
> ``eval()``.
> > It returns a shallow copy of the active namespace at
> > function/coroutine/generator scope.
> >
> > ``PyLocals_GetReturnsCopy()`` returns zero if ``PyLocals_Get()`` returns
> a
> > direct reference to the local namespace mapping, and a non-zero value if
> it
> > returns a shallow copy. This allows extension module code to determine
> the
> > potential impact of mutating the mapping returned by ``PyLocals_Get()``
> without
> > needing access to the details of the running frame object.
>
> Since this goes in the stable ABI, I'm thinking about how extensible
> this will be in the future.
>

This boolean API bothered me a bit, too, and I think you've captured below
exactly what was bothering me about it.


> The proposal assumes that in the future, ``PyLocals_Get``, and thus
> ``locals()``, will never gain another kind of return value, however
> unlikely that is.
> AFAICS, code that uses this will usually check for a single special case
> and fall back (or error) for the other(s), so I think it'd be reasonable
> to make this an "enum" with two values. e.g.:
>
> int PyLocals_GetReturnBehavior();  # better name?
>

We've used "Kind" for similar APIs elsewhere, so calling this API
"PyLocals_Kind()" would make sense to me.

However, there's a potential point of confusion here, as there's already an
implementation level "locals kind" that the runtime uses. This public kind
is related to that internal kind, but they're not the same.

#define PyLocals_DIRECT_REFERENCE 0
> #define PyLocals_SHALLOW_COPY 1
>

I like those names for the two behaviours.

Other values may be added in future versions of Python, if/when the
> Python ``locals()`` builtin is changed to return a different kind of value.
>
> (and same for PyFrame_GetLocalsReturnsCopy)
>

This would become "PyFrame_GetLocalsKind(f)"

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TKFXHZXI7BLSUX5L3BDY63V4M3UVOGHN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 558: Defined semantics for locals()

[Nick Coghlan]

Hi folks,

It's been a long time coming, but I've finally made enough progress on
the reference implementation that I think it's time to ask Nathaniel
to pronounce on the current iteration of PEP 558 (Defined semantics
for locals()).

The rendered version is up at
https://www.python.org/dev/peps/pep-0558/, and I've included the plain
text version below.

For those that are reading the PEP for the first time, the gist is:

* standardise on Python 3.10 behaviour *except* that locals() at
function scope returns a fresh snapshot every time instead of a
reference to the frame level state cache
* make the Python level frame.f_locals on optimised frames a
write-through proxy that keeps both the real fast locals storage and
the C level f_locals state cache up to date
* add new C APIs that allow C code to explicitly request the semantics
the client code actually wants ("behave like the Python locals()
builtin", "always make a copy", "always provide a read-only view")
* soft-deprecate the legacy PyEval_GetLocals() API (while ensuring it
still works)
* use the new features to significantly improve the performance of
code execution tracing hooks implemented in Python

For those that remember reading older versions of the PEP, the key
changes relative to the last discourse thread (back in late 2019/early
2020) are:

* incorporating the C API design improvements from the 2019/20 Discourse thread
* incorporating Mark Shannon's feedback from earlier this year (most
notably, changing the proxy design to only create a reference cycle
from the frames back to the fast locals proxies that reference them if
you store a reference to the proxy as a local variable on the frame)
* trying (and failing) to remove the fast locals proxy dependency on
the C level f_locals cache on optimised frame objects. Instead, the
fast locals proxy more explicitly uses that dictionary as a state
cache to speed up certain operations (e.g. dict equality comparisons,
iteration, and rendering the proxy contents as a string), and to store
keys that don't correspond to frame level variables, while also
exposing a new ``sync_frame_cache()`` method to sync the state cache
on the underlying frame with changes made via other mechanisms.

Cheers,
Nick.

P.S. The PEP text in the email already incorporates the text review
edits in https://github.com/python/peps/pull/2038/files that haven't
been merged to the web version yet.

Suggested clarifications to the text of the PEP that don't affect the
overall design can be added as comments on that PR rather than being
added to the mailing list thread.

=
PEP: 558
Title: Defined semantics for locals()
Author: Nick Coghlan 
BDFL-Delegate: Nathaniel J. Smith
Discussions-To: 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 08-Sep-2017
Python-Version: 3.11
Post-History: 2017-09-08, 2019-05-22, 2019-05-30, 2019-12-30, 2021-07-18


Abstract


The semantics of the ``locals()`` builtin have historically been underspecified
and hence implementation dependent.

This PEP proposes formally standardising on the behaviour of the CPython 3.10
reference implementation for most execution scopes, with some adjustments to the
behaviour at function scope to make it more predictable and independent of the
presence or absence of tracing functions.

In addition, it proposes that the following functions be added to the stable
Python C API/ABI::

PyObject * PyLocals_Get();
int PyLocals_GetReturnsCopy();
PyObject * PyLocals_GetCopy();
PyObject * PyLocals_GetView();

It also proposes the addition of several supporting functions and type
definitions to the CPython C API.


Rationale
=

While the precise semantics of the ``locals()`` builtin are nominally undefined,
in practice, many Python programs depend on it behaving exactly as it behaves in
CPython (at least when no tracing functions are installed).

Other implementations such as PyPy are currently replicating that behaviour,
up to and including replication of local variable mutation bugs that
can arise when a trace hook is installed [1]_.

While this PEP considers CPython's current behaviour when no trace hooks are
installed to be largely acceptable, it considers the current
behaviour when trace hooks are installed to be problematic, as it causes bugs
like [1]_ *without* even reliably enabling the desired functionality of allowing
debuggers like ``pdb`` to mutate local variables [3]_.

Review of the initial PEP and the draft implementation then identified an
opportunity for simplification of both the documentation and implementation
of the function level ``locals()`` behaviour by updating it to return an
independent snapshot of the function locals and closure variables on each
call, rather than continuing to return the semi-dynamic intermittently updated
shared copy that it has historically returned in CPython.


Proposal


The expected semantics of the ``locals()`` built

[Python-Dev] Re: Enum -- last call for comments on 3.10 changes

[Nick Coghlan]

On Tue, 29 Jun 2021, 12:45 am Ethan Furman,  wrote:

> On 6/28/21 6:54 AM, Nick Coghlan wrote:
>
>  > * Enum repr() changing back to the historical behaviour, unless you opt
> in to the
>  >   new behaviour with the global enum decorator: definite +1 here
>
> Question for Nick: this behavior is currently in place for stdlib
> enumerations, and has been since beta 0; are you
> saying this change should also be put off to 3.11 ?
>


I'm not sure what you mean by "this change". I don't think the change to
make repr() less informative for IntEnum instances should be done in any
version.

>
I think repr() should continue to show both the symbolic name and the
associated numeric value, as it did in 3.9 (I'm not worried about the exact
spelling, though, I just don't want to have to find and look up a
translation table when debugging).

Did I misunderstand the proposed change for the next beta?

Cheers,
Nick.



> --
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/AVNMYCH5PZ7CI6COY6NEZKSXO4VV56CA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XKQKPTZBWDXM7DS6RMBT4ADKCLHWWRCT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Enum -- last call for comments on 3.10 changes

[Nick Coghlan]

On Mon, 28 Jun 2021, 11:54 pm Nick Coghlan,  wrote:

> So my vote would be to revert both the repr() and str() changes on IntEnum
> (et al) for 3.10, and then switch to the new str() approach for those
> classes in 3.11.
>


Sorry, I missed including the requested rationale:

* if the str() change is postponed to 3.11, then existing code will upgrade
seamlessly to 3.10, and there is a full year for notification of the coming
change to filter out. The cost is another year of the "drop in replacement"
promise not being kept
* code that was already updated to cope with the module name disappearing
from str presumably still handles it being present for compatibility with
older versions, so shouldn't be hard to adapt to the prefix coming back

(My opinion would likely be different if we were earlier in the beta cycle).

Cheers,
Nick.


>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GTABR3Z2AWI7SPI4F7QX25JRELGPKYJ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Enum -- last call for comments on 3.10 changes

[Nick Coghlan]

On Mon, 28 Jun 2021, 6:02 pm Ethan Furman,  wrote:

> I have spoken with Pablo (3.10 RM), and he agrees that a change to Enum
> str() in 3.10 and another in 3.11 is less than
> ideal, so this new thread is to collect comments about Enum and it's str()
> and repr() and whether the changes take
> effect in 3.10, 3.11, or both.
>
> TL;DR -- sorry, there isn't one.
>

I'll have a go at one:

* Enum str() continuing as "module.NAME", while format() omits the module
* Enum repr() changing back to the historical behaviour, unless you opt in
to the new behaviour with the global enum decorator: definite +1 here
* Making str() consistent with format() and the non-Enum base class for
IntEnum et al (it is already consistent for StrEnum) rather than displaying
the symbolic name

I know I said in the other thread that the last change was problematic, but
I think you have a strong argument that the current behaviour breaks the
"drop in replacement" promise of those concrete types.

So my vote would be to revert both the repr() and str() changes on IntEnum
(et al) for 3.10, and then switch to the new str() approach for those
classes in 3.11.

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5WHRHS3DKJF44DY5MSRUD7YCHDOC5B5L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: IntEnum, IntFlag, and the stdlib

[Nick Coghlan]

On Mon, 28 Jun 2021, 12:44 am Nick Coghlan,  wrote:

>
> For the other proposal, I think you're stuck from a backwards
> compatibility point of view - the default needs to be the current
> behaviour, with some way to opt out of using the symbolic str
> representation.
>

Although, looking at that problem from a different perspective: the
contracts on "str()" and "repr()" under subclassing are *super* weak, since
they're primarily designed for consumption by humans.

If consuming code accepts subclasses, but requires them to behave
*identically* to the base class, it should either be coercing them to the
base class first, or using the better defined string formatting options.

Cheers,
Nick.



> Cheers,
> Nick.
>
>
>
>
>
>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WGL6EFHZAYKQ7RCYINCO6M7PJ4Y5HK7V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: IntEnum, IntFlag, and the stdlib

[Nick Coghlan]

On Thu, 24 Jun 2021, 12:58 pm Ethan Furman,  wrote:

> TL;DR  I am considering changing IntEnum and IntFlag's `__str__` to be
> `int.__str__`
>
> IntEnum and IntFlag are becoming more common in the stdlib.  They
> currently show up in
>
> * http
> * re
> * signal
> * ssl
> * socket
>
> to name just a few.
>
> 3.10 already has some changes to the str() and repr() of enums in general:
>
> HTTPStatus ->  OK  and  HTTPStatus.OK  instead of HTTPStatus.OK and
> 
>


That's already a major loss in enum repr() usability - the whole point of
having both forms in the output was so that repr() gave you both the
symbolic name and the raw number, allowing them to be looked up in external
systems without needing access to the original enum definition to translate
between forms.

For the other proposal, I think you're stuck from a backwards compatibility
point of view - the default needs to be the current behaviour, with some
way to opt out of using the symbolic str representation.

Cheers,
Nick.





>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HXE3XTMYKSGDTV53AQ5TKDI5F5IOAS72/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proposal: declare "unstable APIs"

[Nick Coghlan]

On Fri, 4 Jun 2021, 4:17 am Gregory P. Smith,  wrote:

> Overall agreement. Your list of ast and code objects and bytecode
> instructions are things that I'd _hope_ people already consider unstable so
> declaring them as such just makes sense if we're not doing that already.
>

Another example of a public API that explicitly declares itself unstable is
"ssl.get_default_context()" (along with any TLS-enabled API that uses it).
It was made that way so the default TLS settings could evolve with the
times, even on maintenance branches. The secrets module has a similar
caveat on its default token lengths (i.e. making them longer is considered
an acceptable API change - if consuming code can't handle that for some
reason, it should set an explicit length).

The metaprogramming APIs for customisation of class creation also arguably
qualify - we've previously imposed new obligations on metaclass developers
as a consequence of adding features like zero-arg super() and the
descriptor naming hooks.

It occurs to me that PEP 387 (the backwards compatibility policy) should
probably mention that these formally unstable APIs exist, and link to a
page in the standard library docs that:

* references the formally unstable APIs
* specifies the points where potentially incompatible changes are allowed
(normally new feature releases, but the default SSL/TLS context definition
may change in maintenance releases if necessary)

Cheers,
Nick.



>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQFAP6PVIO7HPN6JFBASECH6R67DHL37/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: name for new Enum decorator

[Nick Coghlan]

On Sat, 29 May 2021, 7:27 am Ethan Furman,  wrote:

> On 5/28/21 12:43 AM, Petr Viktorin wrote:
>  > On 28. 05. 21 5:24, Ethan Furman wrote:
>
>  >>  class FlagWithMasks(IntFlag):
>  >>  DEFAULT = 0x0
>  >>
>  >>  FIRST_MASK = 0xF
>  >>  FIRST_ROUND = 0x0
>  >>  FIRST_CEIL = 0x1
>  >>  FIRST_TRUNC = 0x2
>  >>
>  >>  SECOND_MASK = 0xF0
>  >>  SECOND_RECALC = 0x00
>  >>  SECOND_NO_RECALC = 0x10
>  >>
>  >>  THIRD_MASK = 0xF00
>  >>  THIRD_DISCARD = 0x000
>  >>  THIRD_KEEP = 0x100
>  >>
>  >> Here we have three flags (FIRST_MASK, SECOND_MASK, THIRD_MASK) that
> are aliasing values
>  >> that don't exist, but it seems intentional and not an error.
>  >
>  > Are you looking for a decorator for the whole Enum, or a way to mark
> individual *values* as masks?
>
> The decorator is for whole enum.  The issue is not that some values are
> masks, but whether the absence of named bits
> covered by the mask is an error.
>


If you were prepared to incur a deprecation period, then you could make the
decorator "@allow_masks" and flip the default behaviour to emit Deprecation
Warning for masks, eventually upgrading it to Value Error.

Otherwise, maybe something like "@no_masks"?

Cheers,
Nick.





> --
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OM5M774MP5QPLFXZ7OVGBPR7ZFB6X35A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C6ZYW7PFTPUETPCKLD57O7SWV6GK4M63/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: March Steering Council update.

[Nick Coghlan]

Thanks for the update!

I know this request is a little ironic coming from me, but would it be
possible to state the PEP titles the first time they're mentioned each
month?

Cross referencing is a little awkward when reading the summary on a phone
rather than a full computer.

Cheers,
Nick.

On Tue, 18 May 2021, 11:12 pm Thomas Wouters,  wrote:

>
> The SC has just published the community update for March:
>
>
> https://github.com/python/steering-council/blob/main/updates/2021-03-steering-council-update.md
>
> We're still trying to get these done every month, but between the rush of
> PEPs and other issues before the 3.10b1 deadline, and PyCon US, we're
> delayed a little. (The April update hopefully won't be too long.) In the
> meantime, the SC keynote from PyCon US covers our longer-term plans and our
> points of view on a few other subjects, and that will hopefully be up on
> YouTube (
> https://www.youtube.com/playlist?list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K)
> soon. (Since it's mentioned in the notes below, I just want to point out
> that we didn't record on April 19th; it was pushed back to May 3rd, the
> same day 3.10b1 was cut.)
>
> March 1
>
>- Steering Council synced up on the rejection draft for PEP 651. Group
>discussed the importance of conveying that all aspects of the proposal were
>reviewed and the decision is a holistic one. Group will continue working on
>the draft and will sync up via Slack during the week so it can be sent out.
>- The Steering Council discussed PEP 648. Barry created a doc and the
>group will consolidate feedback+questions into this document. Then the
>Steering Council will post this on Discourse as well as to python-dev@.
>- The Steering Council extensively discussed PEP 637. The group
>decided that it would reject the PEP based on the PEPs costs not being
>worth the benefits.
>- The Steering Council also discussed typing in general and who should
>own it.
>- Next week the SC will vote on the Documentation Work Group's charter.
>
>
> March
> 8
>
>- The Steering Council discussed Mark's response to the rejection of
>PEP 651.
>- The group also discussed the notification for PEP 637.
>- The group approved PEP 624 & PEP 597.
>- The Steering Council discussed PEP 644 and decided further
>communication was needed with Christian.
>- The Steering Council voted on and approved the Documentation Work
>Group.
>- The group discussed moving master to main and decided it needed to
>be done. The SC discussed communication around this change.
>
>
> March
> 15
>
>- Thomas updated the notifications for PEP 597, PEP 624 and PEP 637,
>and sent out the notifications March 15th.
>- Group checked in on the draft for PEP 648. Thomas has more text to
>add to it.
>- The Steering Council discussed what kind of presentation they will
>give at PyCon US. The group decided on a combination of presentation and
>Q Team is working on an outline. SC members were also reminded to
>register for PyCon US, and encourage others to do so.
>- Barry proposed to the SC that they create a Work Group with a subset
>of the Python Security Response Team members and that group can help scope
>the future of the PSRT group. Everyone is fine with this.
>- Steering Council discussed code of conduct situations around
>changing master to main. The group decided on a warning to S.D. that
>will be sent as a warning to all. Thomas is working on the initial draft.
>
>
> March
> 22
>
>- Barry and Thomas checked-in on PEP 648's draft response, they will
>be sending it out soon.
>- The SC reviewed PEP 644 more. Pablo raised a concern around users
>that can be in an env without OpenSSL 1.1.1 or newer and then not being
>able to use wheels. Pablo will email Christian to get clarification and
>will keep the SC informed.
>- Steering Council discussed their keynote at PyCon US 2021. They
>would like to gather questions from the community. Ewa will create a Slido,
>which will be live from April 5 to 11th. SC will review questions on the
>12th. The current plan is to record on April 19th.
>- Steering Council discussed the response to Debian. Carol's done with
>her draft. Pablo is going to take a look at it to see if anything else
>needs to be added.
>- SC discussed the behavior on python-dev@ pertaining to switching git
>from master to main.
>
>
> March
> 29
>
>- The Steering Council accepted PEP 644 (Require 

[Python-Dev] Re: Using FutureWarning for last version before deletion.

[Nick Coghlan]

On Tue, 11 May 2021, 7:57 pm Petr Viktorin,  wrote:

>
>
> On 11. 05. 21 11:08, Inada Naoki wrote:
> > On Tue, May 11, 2021 at 5:30 PM Petr Viktorin  wrote:
> >>
> >> Test tools should treat DeprecationWarning as error by default [0][1].
> >> So even if end users don't really see it, I don't consider it "hidden".
> >>
> >
> > *should* is not *do*. For example, nosetests don't show
> DeprecationWarning.
> > And there are many scripts without tests.
> >
> > So it is hidden for some people.
> >
>
> Sadly, there's not much we can do for users of nose. Nose itself is only
> tested with Python 3.5 and below.
>
> I'm aware that there are scripts without tests. But maybe letting them
> suddenly break is the right balance between letting people know and
> annyoing everyone with unactionable warnings.
>

Scripts (with or without tests) have gone back to showing deprecation
warnings by default since 3.7: https://www.python.org/dev/peps/pep-0565/

If DeprecationWarning is not enough, then we should be having a wider
> discussion, and PEP 387 should change. This particular issue should not
> be an exception to the process.
>


The limitations section of PEP 565 covers several known problems even with
the 3.7+ status quo:
https://www.python.org/dev/peps/pep-0565/#limitations-on-pep-scope

Most notably, unit test doesn't catch import time or compile time
deprecation warnings, and it doesn't catch deprecations in subprocesses.

Another case that came up in this thread was support libraries that are
tested with warnings treated as errors, but:

* the test runner still doesn't enable all warnings (e.g. nose projects
that never migrated to nose2)
* the project isn't covered as a dependency by any *other* project's test
suites that are stricter about avoiding deprecation warnings in their code
and the code they use

And, of course, projects that don't treat warnings as errors in their tests
will get a green tick from CI systems no matter how many warnings they emit.

If we wanted to go back to enabling DeprecationWarning by default across
the board, I think we could only reasonably do that if we provided an
easier to use "hide all warnings except the ones I specifically opt in to
seeing" API like the one mentioned at the end of PEP 565:
https://bugs.python.org/issue32229

As currently written, that API proposal would flip the ergonomics problem
though - while ensuring the warnings system was configured early enough to
capture compile time and import time warnings would be up to the
application, being able to easily propagate the config change to
subprocesses would need to be part of the new API design.

Cheers,
Nick.





>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NEXGBG6ERDVK6WMDZM6IHPDAQSB4ANIS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Nick Coghlan]

On Fri, 14 May 2021, 1:47 am Stéfane Fermigier,  wrote:

>
>
> On Thu, May 13, 2021 at 8:42 AM Abdur-Rahmaan Janhangeer <
> arj.pyt...@gmail.com> wrote:
>
>> Actual quote by "a Python Software Foundation fellow and contrib-
>> utor to Python infrastructure projects"
>>
>
> Ah, this is what you were referring to. The document was published 5 years
> ago, so this may or may not reflect the current situation.
>
>
>> What frustrates me most is that we have an all-time high of
>> Python developers and an all-time low on high quality contri-
>> butions.[...] As soon as pivotal developers like Armin Ronacher
>> slow down their churn, the whole community feels it immedi-
>> ately.
>>
>
> That's true, but, AFAIK, Armin was never a direct contributor to CPython
> (confirmed by looking at
> https://github.com/python/cpython/graphs/contributors ) so I guess that's
> another issue.
>


The problems mentioned in the Ford Foundation report definitely aren't
solved, but one of the contributing factors that the PSF identified at the
time was that if core projects like CPython and pypi.org are underfunded,
that poor precedent severely hurts fund raising and other sustainability
efforts in the wider Python ecosystem.

Thus efforts like the "developer in residence" role that the PSF are
currently recruiting for, as well as the increased focus on (and investment
in) ecosystem sustainability from commercial users and redistributors.

Cheers,
Nick.



>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VOKWM5ZJ45BERKIPURSSIXGNBWPS5IBI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

[Nick Coghlan]

On Sat, 15 May 2021, 5:39 am Tal Einat,  wrote:

(snip useful feature summary)

The common `SENTINEL = object()` idiom fails #3, #4 and #5. This is
> what I've been using for years, and I now think that it isn't good
> enough. This not having a nice repr is what started this thread.
>
> I'd also personally prefer something simple, ideally without a new
> class or module.
>

The key advantage of a new base class is that it offers a way to
communicate a shift in the recommended development idiom. "sentinel =
object()" is entrenched enough that I would expect that only a
"types.Sentinel" base class would stand a good chance of displacing it.

Why the "types" module? I don't think this is important enough to be a
builtin type, and the types module is cheap to import, doesn't bring in
many transitive dependencies, and the purpose of the new base class would
be defining custom sentinel types with various desirable properties.

Cheers,
Nick.

>


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V67I4A6WX6INEAGGSJWYFX4WKQHA73NW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Can't sync cpython main to my fork

[Nick Coghlan]

On Fri, 7 May 2021, 8:13 am Ethan Furman,  wrote:

> On 5/6/21 7:14 AM, Jelle Zijlstra wrote:
>
>  > Maybe others have different workflows, but I don't see much of a need
>  > for keeping your fork's main branch up to date.
>
> I will occasionally do a `git push origin main` just to shut up the
> messages about being behind/ahead; other than that,
> I have no idea why I would need origin to be up to date.
>

I sync mine occasionally so I can make draft PRs in my repo before
submitting them to the main repo.

That's the only use case I have found for it though.

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R5GX6JWKSKSSXA3JSU7QQOTGVE6IQEAC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Nick Coghlan]

On Sat, 8 May 2021, 8:53 am Pablo Galindo Salgado, 
wrote:

> > One thought: could the stored column position not include the
> indentation? Would that help?
>
> The compiler doesn't have access easy access to the source unfortunately
> so we don't know how much is the indentation. This can make life
> a bit harder for other tools, although it can make it easier for reporting
> the exception as the current traceback display removes indentation.
>


If the lnotab format (or a new data structure on the code object) could
store a line indent offset for each line, each instruction within a line
would only need to record the offset from the end of the indentation.

If we assume "deeply indented code" is the most likely source of
excessively long lines rather than "long expressions and other one line
statements produced by code generators" it may be worth it, but I'm not
sure that's actually true.

If we instead assume long lines are likely to come from code generators,
then we can impose the 255 column limit, and breaking lines at 255 code
points to improve tracebacks would become a quality of implementation issue
for code generators.

The latter assumption seems more likely to be true to me, and if the deep
indentation case does come up, the line offset idea could be pursued later.

Cheers,
Nick.


>
> On Fri, 7 May 2021 at 23:37, MRAB  wrote:
>
>> On 2021-05-07 22:45, Pablo Galindo Salgado wrote:
>> > Hi there,
>> >
>> > We are preparing a PEP and we would like to start some early discussion
>> > about one of the main aspects of the PEP.
>> >
>> > The work we are preparing is to allow the interpreter to produce more
>> > fine-grained error messages, pointing to
>> > the source associated to the instructions that are failing. For example:
>> >
>> > Traceback (most recent call last):
>> >
>> >File "test.py", line 14, in 
>> >
>> >  lel3(x)
>> >
>> >  ^^^
>> >
>> >File "test.py", line 12, in lel3
>> >
>> >  return lel2(x) / 23
>> >
>> > ^^^
>> >
>> >File "test.py", line 9, in lel2
>> >
>> >  return 25 + lel(x) + lel(x)
>> >
>> >  ^^
>> >
>> >File "test.py", line 6, in lel
>> >
>> >  return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
>> >
>> >   ^
>> >
>> > TypeError: 'NoneType' object is not subscriptable
>> >
>> >
>> > The cost of this is having the start column number and end
>> column number
>> > information for every bytecode instruction
>> > and this is what we want to discuss (there is also some stack cost to
>> > re-raise exceptions but that's not a big problem in
>> > any case). Given that column numbers are not very big compared with
>> line
>> > numbers, we plan to store these as unsigned chars
>> > or unsigned shorts. We ran some experiments over the standard library
>> > and we found that the overhead of all pyc files is:
>> >
>> > * If we use shorts, the total overhead is ~3% (total size 28MB and the
>> > extra size is 0.88 MB).
>> > * If we use chars. the total overhead is ~1.5% (total size 28 MB and
>> the
>> > extra size is 0.44MB).
>> >
>> > One of the disadvantages of using chars is that we can only report
>> > columns from 1 to 255 so if an error happens in a column
>> > bigger than that then we would have to exclude it (and not show the
>> > highlighting) for that frame. Unsigned short will allow
>> > the values to go from 0 to 65535.
>> >
>> [snip]How common are lines are longer than 255 characters, anyway?
>>
>> One thought: could the stored column position not include the
>> indentation? Would that help?
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/MHF3PMCJOR6VK765OSA7NSO66NY3QU3V/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OKQYNAI2B2BRCFMYJPLYPG2HHHUB5QR6/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BA4UQ36VQGZ52BN56XPKFRPVO2TWD6BN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: On the migration from master to main

[Nick Coghlan]

On Tue, 4 May 2021, 10:50 am Łukasz Langa,  wrote:

>
> On 4 May 2021, at 02:04, Łukasz Langa  wrote:
>
> Having renamed the branch in my fork first, the exact sequence I used on
> my own clone was:
>
> ❯ git checkout master
> ❯ git branch -m master main
> ❯ git fetch origin
> ❯ git branch -u origin/main main
> ❯ git remote set-head origin -a
> ❯ git fetch upstream
> ❯ git remote set-head upstream -a
> ❯ git pull upstream main
>
> This worked, I can successfully pull from upstream's main and push to my
> upstream's main. The `set-head` ones were suggested by the GitHub UI and
> ended up with heads explicitly being listed in `git log` which I guess is
> harmless.
>
>
> Seeing Ned's and Mariatta's comments on the matter I guess I should
> clarify that my commands above are for a *symmetrical* setup, i.e. where
> I expect the default `git pull` to *pull from origin*, and the default
> `git push` to *push to origin*.
>
> I might not be your typical user here, since I actually push to upstream
> too every now and again as part of release management, in particular
> pushing out signed release tags (which are full git objects with the same
> metadata as commits).
>


The dev guide (and Github) have long recommended this arrangement, so I
believe it's those of us that define a separate "pr" remote for our fork
and have "origin" referring to the main CPython repo that are the odd ones
out.

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5KXVPXAJYFRJCT3RGQ4SC22NEM6IUKBE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Let's Fix Class Annotations -- And Maybe Annotations Generally

[Nick Coghlan]

On Sat, 24 Apr 2021, 5:53 pm Larry Hastings,  wrote:

>
> So I now suspect that my knee-jerk answer is wrong.  Am I going too far
> down the rabbit hole?  Should I *just* make the change for user classes
> and leave builtin classes untouched?  What do you think?
>

I'd suggest kicking the can down the road: leave builtin classes alone for
now, but file a ticket to reconsider the question for 3.11.

In the meantime, inspect.get_annotations can help hide the discrepancy.

Cheers,
Nick.


>
> Cheers,
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/IK7IWUCTESD5OZE47J45EY3FRVM7GEKM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2OPILFSQYQP4ERZB6OGABL5ENEWFTRQN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Nick Coghlan]

On Fri, 23 Apr 2021, 12:34 pm Inada Naoki,  wrote:

>
> I think using ABC to distinguish sequence or mapping is a bad idea.
>
> There are three policies:
>
> a) Use duck-typing; just us it as sequence. No type check at all.
> b) Use strict type checking; isinstance(x, list) / isinstance(x, (list,
> tuple)).
> c) Use ABC.
>
> But (c) is broken by design. It is not fixable.
> IMHO, We should chose (a) or (b) and reject any idea relying on Sequence
> ABC.
>


That ship sailed long ago, since distinguishing sequences from mappings was
one of the original motivating use cases for ABCs (see the last sentence in
https://www.python.org/dev/peps/pep-3119/#abcs-vs-duck-typing ).

One of the important things to remember about ABCs is that passing a
*runtime* isinstance check is a matter of calling
"the_abc.register(my_type)". Hence the comment earlier in the thread that
with ABCs, passing "isinstance(obj, the_abc)" becomes just another
criterion for quacking like a duck.

Cheers,
Nick.




>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6G5EBNTHXMQJAD6FRVRPNDHNZBPVAOIK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Nick Coghlan]

On Sat, 24 Apr 2021, 10:02 am Skip Montanaro, 
wrote:

>
> Practically speaking, one issue I have is how easy it is to write
>> isinstance or issubclass checks. It has historically been much more
>> difficult to write and maintain a check that something looks like a duck.
>>
>>  `if hasattr(foo, 'close') and hasattr(foo, 'seek') and hasattr(foo,
>> 'read'):`
>>
>> Just does not roll off the figurative tongue and that is a relatively
>> simple example of what is required for a duck check.
>>
>> To prevent isinstance use when a duck check would be better,
>>
>
> I'm going to chime in briefly then return to lurking on this topic, trying
> to figure out all the changes to typing while I wasn't paying attention.
> Back in ancient times I recall "look before you leap" as the description of
> either of the above styles of checks, no matter which was easier to type.
> At the time, I thought the general recommendation was to document what
> attributes you expected objects to provide and just make the relevant
> unguarded references. I no longer recall what the tongue-in-cheek
> description of that style was (just "leap"?) Is that more simple usage more
> akin to classic "duck typing" than always guarding accesses? I assume that
> will still have a place in the pantheon of Python type variants.
>

LBYL: Look before you leap (check then use)
EAFP: Easier to ask forgiveness than permission (use, then handle any
exceptions or let them escape)

(for accessing external resources rather than satisfying internal type
consistency, you usually need the latter, as the external state may change
between checking and usage)

Duck typing usually falls squarely into the EAFP category.

Cheers,
Nick.


> Skip
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QW7NQ5KTOQLV27MKZO5B3TMSTXIR5MC5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZA6IHKR3DOICMTEILXCBTX3LJXEVCFMX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Nick Coghlan]

On Mon, 12 Apr 2021, 1:48 pm Guido van Rossum,  wrote:

>
> At the very least I recommend that the SC take this into account when they
> consider PEP 649. Accepting it has some nice benefits when it comes to the
> scoping rules for annotations -- but it would forever close the door for
> the "relaxed annotation syntax" idea you brought up. (Isn't it fun to be on
> the SC. :-)
>

I may have missed someone else mentioning this, but I don't think this
concern is necessarily true, as even if PEP 649 were accepted, the only
pre-PEP-563 constraints it would reintroduce would be that all future type
annotation syntax:

* have a defined runtime effect;
* that runtime effect be consistent with normal expressions when reusing
existing syntax; and
* be explicitly quoted when using type hinting syntax from later Python
versions in code that needs to run on earlier versions

Any PEPs adding new type hinting specific syntax would be free to define
the runtime effect of the new syntax as "produces a string containing the
text of the part of the annotation using the new syntax, as if the new
syntax were explicitly quoted", even if we decided not to go ahead with the
idea of applying those "produces a string" semantics to *all* annotations.

Cheers,
Nick.


>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DLCZHSXCRBVGCXBAKG7SXI6LBFQH23X7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Nick Coghlan]

On Sun, 18 Apr 2021, 2:47 am Antoine Pitrou,  wrote:

> On Sun, 18 Apr 2021 02:13:57 +1000
> Nick Coghlan  wrote:
> >
> > If
> > they want automatic resource management, then we want them working out
> how
> > to lift the affected code out of C and into Python (for example, the
> import
> > system rewrite).
>
> There's a significant amount of wishful thinking here.  When CPython
> code gets written in C, it's often because it's presumed to be
> performance-critical.  The import system was indeed rewritten in
> Python... but some parts were then written back in C (see
> Python/import.c) to fix performance regressions.
>


Aye, I know, but the overall organisation of the import system still
changed to "Python with C acceleration of key parts" rather than the
previous "almost entirely C" approach.

Other implementations are now free to reuse the Python parts, and only have
to implement the accelerators separately.


> > In a lot of ways, CPython's C API can be viewed as one of the many
> > competing approaches to enabling "C-with-objects" programming, just like
> > C++.
>
> It's a clumsy API to use *from C*.  Witness the inconsistent behaviour
> of those APIs wrt. borrowed references, for example, or the fact that
> forgetting an INCREF or DECREF can lead to occasional leaks or crashes,
> or the delicate task of teaching the cyclic GC about a particular type,
> or all the special cases where those APIs deviate slightly in semantics
> from their pure Python equivalents, or the fact that APIs are added in
> adhoc manner, leading to an inconsistent set of primitives...
>

Right, but that clumsiness isn't any easier to manage from C++ than it is
from C. Things like RAII rely on consistent behaviour from the resources
being managed, and the current C API doesn't offer that.

For example, putting a borrowed or stolen reference into an RAII based
reference manager will result in an early free or a resource leak, just as
happens when you decref a borrowed reference or incref a stolen one in C.

Hence the need for wrapper APIs and tools that make the C API more C++
friendly rather than it being straightforward to use directly.

CPython's C API is probably fine for consumption by intermediate layers
> such as Cython. It's a maze to navigate for direct use.
>

That I also agree with, and hence I'd definitely be a fan if we ever
figured out a way to bootstrap Cython into the CPython build process so
that accelerated standard library modules could be written in Cython.

Unlike C++, Cython uses the same data structures and object model as Python
does, and it already smooths over the irregularities in the C API to allow
for fully automated resource management. The Python based syntax can even
help lower the barriers to entry for folks that don't already know C
(although you do still need to learn the underlying C memory model).

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N53JDKLNBQRGPSGLOK3HUIMJGD3FLSCV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Nick Coghlan]

On Sun, 18 Apr 2021, 1:59 am Jelle Zijlstra, 
wrote:

> El sáb, 17 abr 2021 a las 8:30, Nick Coghlan ()
> escribió:.
>
>>
>> Metaclass __prepare__ methods can inject names into the class namespace
>> that the compiler doesn't know about, so yeah, it unfortunately has to be
>> conservative and use LOAD_NAME in class level code.
>>
>> But of course, most metaclasses don't. I wonder if there are cases where
> the compiler can statically figure out that there are no metaclass
> shenanigans going on, and emit LOAD_GLOBAL anyway. It seems safe at least
> when the class has no base classes and no metaclass=.
>

Aye, that particular case is one the symtable pass could at least
theoretically identify.

As soon as there is a name to resolve in the class header, though, it's no
longer safe for the compiler to make assumptions :(

Cheers,
Nick.

>


>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NDYKJTWUN4HT2BPNJVBUKYVD6HOTIBMY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Nick Coghlan]

On Sat, 17 Apr 2021, 3:51 am ,  wrote:

> Guys, the issue is that I most of the time see that somebody used C++ for
> one or two times, did not understand it and left with bad taste ...
>

I've got more than a decade and a half of experience with both C++ (dating
back to the relatively low quality VC6 C++ runtime) and CPython (getting on
towards 20 years now), and the value of adding a C++ runtime to the CPython
runtime would be vastly smaller than the value of adding it to an arbitrary
C project.

If folks want rich data structures in the CPython implementation, we want
them using the Python builtin types, not the C++ STL containers. If they
want a type hierarchy, we want them using Python types, not C++ types. If
they want automatic resource management, then we want them working out how
to lift the affected code out of C and into Python (for example, the import
system rewrite).

In a lot of ways, CPython's C API can be viewed as one of the many
competing approaches to enabling "C-with-objects" programming, just like
C++.

There are certainly nice features in modern C++ that make it easier to
scale to large projects than vanilla C code, but the bulk of the CPython
code base has never really been vanilla C code - it has always been able to
rely on the builtin Python containers and type hierarchy for a lot of the
heavy lifting.

And it is already a bigger barrier to entry than we would like to ask
potential contributors to the lower level code to learn two object models
(the Python one and the much simpler C one), let alone if we were to start
asking them to learn a 3rd one (C++) that has a deserved reputation as one
of the most complex and error prone object models in widespread use.

Cheers,
Nick.





>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SQKSB4OYXL5FPS72BNUJQO2ALRPW6YSX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Nick Coghlan]

On Fri, 16 Apr 2021, 3:14 pm Larry Hastings,  wrote:

>
> Anyway I assume it wasn't "fixable".  The compiler would presumably
> already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
> would be cheaper every time for a global or builtin.  The fact that it
> already doesn't do so implies that it can't.
>

Metaclass __prepare__ methods can inject names into the class namespace
that the compiler doesn't know about, so yeah, it unfortunately has to be
conservative and use LOAD_NAME in class level code.

Cheers,
Nick.

>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZJYDHWJNMMMICUE32M3O7DGMSMVIOQ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP-0467: Minor API improvements for binary sequences

[Nick Coghlan]

Thanks for picking this back up!

The deferral section can go away now that you're actively working on it
again, and +1 from me on the resolution of the previously open questions
(although I wouldn't be particularly upset if the SC considered bchr
redundant, given that "bchr = bytes.fromord" is a trivial alias in cases
where the shorter spelling is easier to read).

Cheers,
Nick.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6YQOPMNXINMFFU572EYXTV6SVWKBT2SQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: SC feedback: PEP 648 -- Extensible customizations of the interpreter at startup

[Nick Coghlan]

On Wed, 31 Mar 2021 at 11:01, Barry Warsaw  wrote:
>
> Kind of :)
>
> PEP 648 would definitely allow us to deprecate the executable part of pth 
> files.  I let my own biases leak in to my response because I would like to 
> find a way to replace the sys.path feature of pth with something much more 
> auditable and discoverable.  To me that means deprecating pth files and 
> finding something better, but maybe not.

Adding pth file auditing to the output of "python -m site" should be
entirely feasible, it just hasn't been done yet.

Even if it just listed the files found, it would make them easier to
audit than they are today.

Declaring the feature impossible to audit when we haven't even really
tried to make it auditable seems premature (the existing site output
doesn't even indicate which paths in sys.path will be considered when
looking for pth files, let alone indicate which of those directories
actually contain any).

Cheers,
Nick.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L5ZDEXRTR3NJT65BHV4YLEMEYFEQGJ6M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: SC feedback: PEP 648 -- Extensible customizations of the interpreter at startup

[Nick Coghlan]

On Wed, 31 Mar 2021, 3:15 am Barry Warsaw,  wrote:

> .  We would like to eventually go farther, including deprecation of pth
> files entirely, but that is outside the scope of this PEP.
>

Please don't, since that would force everyone to start using PEP 648 just
to extend sys.path, which would be just as bad as the status quo.

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MSOV7NKDCO7L3X46SDDLVTUEN7ER2EDB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Nick Coghlan]

On Mon, 29 Mar 2021, 7:47 pm Mark Shannon,  wrote:

[Guido wrote]

> >
> > Also, I think that we should probably separate this out in two separate
> > flag sets, one for subjects and one for class patterns -- it is pretty
> > confusing to merge the flag sets into a single value when their
> > applicability (subject or class pattern) is so different.
>
> That would require two different special attributes, which adds bulk
> without adding any value.
>
> __match_kind__ = MATCH_SEQUENCE | MATCH_DEFAULT
>
> should be clear to anyone familiar with integer flags.
>

The combined flags might be clearer if the class matching flags were
"MATCH_CLS_DEFAULT" and "MATCH_CLS_SELF"

Without that, it isn't obvious that they're modifying the way class
matching works.

Alternatively, given Guido's suggestion of two attributes, they could be
"__match_container__" and "__match_class__".

The value of splitting them is that they should compose better under
inheritance - the container ABCs could set "__match_container__"
appropriately without affecting the way "__match_class__" is set.

An implementation might flatten them out at class definition time for
optimisation reasons, but it wouldn't need to be part of the public API.

Cheers,
Nick.

>


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I6TJQQV5H7VYQYKTBPP7TFELEEQLQLEY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]

[Nick Coghlan]

On Sun, 28 Mar 2021, 12:34 am Irit Katriel via Python-Dev, <
python-dev@python.org> wrote:

>
> Hi Paul,
>
> IIUC, you are saying that exception group should not be a builtin type
> because it is (1) complex (2) special-purposed.   Instead, you propose that
> we make exception handling pluggable.
>


Note that (2) would be the *exact opposite* of the direction that we have
taken exception handling in general: it used to be far more permissive than
it is, and then got more restrictive over time to make structured exception
handling more consistent.

In particular, features like exception chaining and use case independent
trace back printing only work because exception consumers can rely on all
exceptions being instances of BaseException and having particular
double-underscore attributes like "cause", "context", and "traceback"

The lesson we've learned from Trio's experience is:

* regular exceptions are still fine for most purposes
* when they're not fine, they're not fine in a consistent way across a
variety of use cases where the existing exception machinery currently loses
or hides information, the problem isn't specific to Trio's nursery API

I understand the temptation to generalise, but a general proposal would
fail for the same reason the "user defined infix operators" PEPs failed:
despite decades of experience, there just weren't compelling use cases for
a general purpose feature. Instead, the proposal that succeeded was narrow:
it targeted the feature that people actually wanted (an infix operator for
matrix multiplication), rather than proposing more general machinery that
could be used to build that feature.

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PHCPGXZOMCT7X27QGWRTSYD6RSN2WOKL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]

[Nick Coghlan]

On Sat, 27 Mar 2021, 9:24 am Guido van Rossum,  wrote:

> Everyone,
>
> Given the resounding silence I'm inclined to submit this to the Steering
> Council. While I'm technically a co-author, Irit has done almost all the
> work, and she's done a great job. If there are no further issues I'll send
> this SC-wards on Monday.
>

+1 from me. I remember talking to Nathaniel (and Yury IIRC) about early
iterations of the MultiError handling in trio at PyCon years ago, so it's
great to see the way the concept has matured in the intervening time.

The one thing I did think was that it might be good to mention
concurrent.futures and contextlib in the PEP as modules that could
potentially take advantage of exception groups. I'm not sure it's really
necessary though, as the presentation is already compelling as it is.

Cheers,
Nick.

P.S. The potential use case I see in contextlib: the way ExitStack nests
dynamic resource handling *works*, but the tracebacks when multiple clean
up steps fail can be spectacularly cryptic. With exception groups
available, it should be possible to define a new "context lib.ExitGroup"
API that flattens out the resource clean up and collects any exceptions
that occur into a more comprehensible exception group, rather than building
a nested exception context tree the way ExitStack does.




> --Guido
>
> On Sat, Mar 20, 2021 at 10:05 AM Irit Katriel 
> wrote:
>
>>
>> We would like to present for feedback a new version of PEP 654, which
>> incorporates the feedback we received in the discussions so far:
>> https://www.python.org/dev/peps/pep-0654/
>> The reference implementation has also been updated along with the PEP.
>>
>> The changes we made since the first post are:
>>
>> 1. Instead of ExceptionGroup(BaseException), we will have two new builtin
>> types: BaseExceptionGroup(BaseException) and
>> ExceptionGroup(BaseExceptionGroup, Exception).
>> This is so that "except Exception" catches ExceptionGroups (but not
>> BaseExceptionGroups). BaseExceptionGroup.__new__ inspects the wrapped
>> exceptions, and if they are all Exception subclasses, it creates an
>> ExceptionGroup instead of a BaseExceptionGroup.
>>
>> 2. The exception group classes are not final - they can be subclassed and
>> split()/subgroup() work correctly if the subclass overrides the derive()
>> instance method as described here:
>> https://www.python.org/dev/peps/pep-0654/#subclassing-exception-groups
>>
>> 3. We had some good suggestions on formatting exception groups, which we
>> have implemented as you can see in the output shown for the examples in the
>> PEP.
>>
>> 4. We expanded the section on handling Exception Groups, to show how
>> subgroup can be used (with side effects) to do something for each leaf
>> exception, and how to iterate correctly when the tracebacks of leaf
>> exceptions are needed:
>> https://www.python.org/dev/peps/pep-0654/#handling-exception-groups
>>
>> 5. We expanded the sections on rationale and backwards compatibility to
>> explain our premise and expectations regarding how exception groups will be
>> used and how the transition to using them will be managed.
>>
>> 6. We added several items to the rejected ideas section.
>>
>> We did not receive any comments (or make any changes) to the proposed
>> semantics of except*, hopefully this is because everyone thought they are
>> sensible.
>>
>> Irit, Yury and Guido
>>
>>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/AQGDIZDZOJDR4HNRYCJVFA2XJ7YOOXS5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GEERSVTZECXPM6KIRCJS7Z2VO3XJCOL5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python history: origin of the arrow annotation

[Nick Coghlan]

On Fri, 5 Mar 2021, 9:10 pm Steven D'Aprano,  wrote:

> I was curious how and why return annotations use the arrow `->` symbol,
> so I went spelunking into the depths of the Python-Ideas and Python-Dev
> mailing lists.
>
> Much to my surprise, I couldn't find any discussion or debate about it.
>

If you haven't already, also try to track down any PEP 3107 discussions on
the python-3000 mailing list.

I don't actually recall any major debate over the syntax of function
annotations though - the only controversy I can recall was over whether or
not it was a good idea to add the syntax before we had a solid plan for the
standard associated semantics.

Cheers,
Nick.




>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5TGDHTV57GYJQDUOBW4WNZBMGXOQNCCQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Nick Coghlan]

On Wed, 24 Feb 2021 at 10:49, Random832  wrote:
>
> I was reading a discussion thread 
> <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about 
> various issues with the Debian packaged version of Python, and the following 
> statement stood out for me as shocking:
>
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we 
> > don't want users to break their operating system with sudo pip install.
>
> I don't think sudo pip install should break the operating system. And I think 
> if it does, that problem should be solved rather than merely advising users 
> against using it. And why is it, anyway, that distributions whose package 
> managers can't coexist with pip-installed packages don't ever seem to get the 
> same amount of flak for "damaging python's brand" as Debian is getting from 
> some of the people in the discussion thread? Why is it that this community is 
> resigned to recommending a workaround when distributions decide the 
> site-packages directory belongs to their package manager rather than pip, 
> instead of bringing the same amount of fiery condemnation of that practice as 
> we apparently have for *checks notes* splitting parts of the stdlib into 
> optional packages? Why demand that pip be present if we're not going to 
> demand that it works properly?

The reason venv is promoted as heavily as it is is because it's the
only advice that can be given that is consistently correct regardless
of the operating system the user is running locally, whereas safely
using a system-wide Python installation varies a lot depending on
whether you're on Windows, Mac OS X, or Linux (let alone some other
platform outside the big 3 desktop clients).

conda is also popular for the same reason: while the instructions for
installing conda in the first place are OS-dependent, once it is up
and running you can use consistent platform independent conda commands
rather than having to caveat all your documentation with
platform-specific instructions.

Apple moved all of their dynamic language interpreter implementations
to inaccessible-by-default locations so Mac OS X users would stop
using them to run their own code.

Alongside that, we *have* worked with the Linux distro vendors to help
make "sudo pip install" safe (e.g [1]), but that only helps if a user
is running a new enough version of a distro that has participated in
that work.

However, while the option of running "platform native" environments
will never go away, and work will continue to make it less error
prone, the level of knowledge of your specific OS's idiosyncrasies
that it requires is almost certainly going to remain too high for it
to ever again become the default recommendation that it used to be.

Cheers,
Nick.

[1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note:
this change mitigated some aspects of the problem in a way similar to
what Debian does, but still doesn't solve it completely, as custom
Python builds may still make arbitrary changes)

P.S. "But what about user site-packages?" you ask. Until relatively
recently, Debian didn't put the user's local bin directory on the
system path by default, so commands provided by user level package
installs didn't work without the user adjusting their PATH. The
CPython Windows installer also doesn't adjust PATH by default (for
good reasons). And unlike a venv, "python -m" doesn't let you ensure
that the code executed is the version installed in user site-packages
- it could be coming from a directory earlier in sys.path.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RDLEH6DUF57UB6U4HNL2QRVAJY4KDSSJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [python-committers] Acceptance of Pattern Matching PEPs 634, 635, 636, Rejection of PEPs 640 and 642

[Nick Coghlan]

On Tue, 9 Feb 2021, 6:21 am Python Steering Council, <
steering-coun...@python.org> wrote:

> After much deliberation, the Python Steering Council is happy to announce
> that we have chosen to accept PEP 634, and its companion PEPs 635 and 636,
> collectively known as the Pattern Matching PEPs. We acknowledge that
> Pattern Matching is an extensive change to Python and that reaching
> consensus across the entire community is close to impossible.  Different
> people have reservations or concerns around different aspects of the
> semantics and the syntax (as does the Steering Council). In spite of this,
> after much deliberation, reviewing all conversations around these PEPs, as
> well as competing proposals and existing poll results, and after several
> in-person discussions with the PEP authors, we are confident that Pattern
> Matching as specified in PEP 634, et al, will be a great addition to the
> Python language.


Thank you for resolving this discussion - I'm sure it wasn't an easy
decision.

Obviously I think we're going to find things to regret in the way name
binding works in class and mapping patterns, but I also think using "_ as
name" as an explanatory tool is likely to cover most of them.

I'll also consider submitting a PEP *adding* explicit patterns to the
implicit ones, rather than proposing to replace them, so the handling of
different kinds of bare expressions can be readily explained as syntactic
shorthand for more explicit operations rather than the shorthand being the
only available spelling.

I won't do that until after helping with the review of the main PEP 634
implementation, though.

Cheers,
Nick.


>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OMXOT2NCMJBJBLFLMHRZDSIHN6Z7CEDT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Wed, 3 Feb 2021, 10:16 pm Sven R. Kunze,  wrote:

> Hi Mark,
>
> I've been working on a project heavily relying on frame.f_locals.
>
> Are you planning to remove it?
>

No, PEP 558 doesn't remove it, it enhances it to be a live view of the
frame state instead of an inconsistently updated snapshot.

The potential incompatibility Mark is referring to is the fact that even
optimised frames currently allow writing to arbitrary keys in
frame.f_locals and making the bound values visible to the locals() builtin
and other consumers of frame.f_locals.

For PEP 558, it's an open question as to whether that behaviour will become
limited to the PyEval_GetLocals() backwards compatible C API, with the
updated Python frame API instead throwing KeyError for attempts to write to
unknown keys on optimised frames.

Regards,
Nick.




>
> On 30.01.21 13:18, Mark Shannon wrote:
> >
> > Given that f_locals is broken, why is keeping compatibility for this
> > obscure, and probably unused case worthwhile?
> >
> > The break in compatibility with locals() seems much more intrusive,
> > yet you are OK with that (as am I).
>
>
> Best,
> Sven
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YXW2EU5YBFLG3MSNDXW3RMTZ36GG5ZH2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Sun, 31 Jan 2021 at 14:09, Nick Coghlan  wrote:
> On Sat, 30 Jan 2021, 10:55 pm Nick Coghlan,  wrote:
>> On Sat, 30 Jan 2021, 10:18 pm Mark Shannon,  wrote:
>>> The break in compatibility with locals() seems much more intrusive, yet
>>> you are OK with that (as am I).
>> PyEval_GetLocals() is part of the stable ABI and returns a borrowed 
>> reference. That means there are a lot of implementation restrictions around 
>> keeping that API working. A follow-up PEP could propose deprecating and 
>> removing the API as intrinsically broken, but I don't want to go that far in 
>> PEP 558.
> After sleeping on this, I'm now convinced that you're right, and we can 
> reasonably drop the "stash extra info in the frame locals snapshot" feature 
> from the *new* optimised frame locals C API.

The PR at https://github.com/python/peps/pull/1787 has been updated to
accept your suggestion into the PEP rather than listing it as an open
question. It also cleans up a bunch more references to the old "store
the proxy on the frame" idea that I missed when moving away from the
idea.

I haven't updated the reference implementation at
https://github.com/python/cpython/pull/3640/files to match though, and
don't know when I might get to doing that myself. If you're interested
in tackling that, the offer of PEP co-authorship definitely still
stands :)

While I haven't listed it in the PR as an open issue, there is one
potential point for simplification that I haven't resolved yet:
whether the new APIs should *completely* ignore the shared dynamic
snapshot that ``PyEval_GetLocals()`` uses. At the moment, most of the
APIs (including the Python ``locals()`` builtin) will still see extra
values written via that backwards compatibility interface, as they
implicitly update it, and then use it to generate their final return
value. The only APIs that will ignore it completely are the Python
level frame.f_locals, and the C level PyLocals_GetView() and
PyFrame_GetLocalsView(), as those rely on the new fast locals proxy,
and bypass the dynamic snapshot.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6XED5D7UAXSILC2NSM6C6D6H62Z6GRFL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Sat, 30 Jan 2021, 10:55 pm Nick Coghlan,  wrote:

>
>
> On Sat, 30 Jan 2021, 10:18 pm Mark Shannon,  wrote:
>
>>
>>
>
>> The break in compatibility with locals() seems much more intrusive, yet
>> you are OK with that (as am I).
>>
>
> PyEval_GetLocals() is part of the stable ABI and returns a borrowed
> reference. That means there are a lot of implementation restrictions around
> keeping that API working. A follow-up PEP could propose deprecating and
> removing the API as intrinsically broken, but I don't want to go that far
> in PEP 558.
>

After sleeping on this, I'm now convinced that you're right, and we can
reasonably drop the "stash extra info in the frame locals snapshot" feature
from the *new* optimised frame locals C API.

What I realised is that even if we don't offer that feature on the new
frame locals proxy type, the *old* C API can still support it, unless &
until that API is deprecated, as PyEval_GetLocals() bypasses the new API
and accesses the frame state directly.

Cheers,
Nick.






>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VED6ZQSXVXEEDBB4M3UBICZNONEHBASH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Sat, 30 Jan 2021, 10:18 pm Mark Shannon,  wrote:

> Hi Nick,
>
> On 30/01/2021 4:44 am, Nick Coghlan wrote:
> > On Sat, 30 Jan 2021 at 10:30, Nick Coghlan  wrote:
> >> On Sat, 30 Jan 2021, 12:13 am Mark Shannon,  wrote:
> >>> With a direct proxy coherency is not an issue.
> >>
> >> For things in the frame, it *is* a direct proxy - reads pull from the
> frame object, and writes go to both the frame object and the mapping:
> https://github.com/python/cpython/pull/3640/files#diff-7b8cef249e5cca077d30de4e428a6bde6b9b803464e790e9cffa7e052e19efddR1315
> >
> > Reviewing the code again, I'd misremembered how the draft
> > implementation works - right now, reads are relying on the snapshot
> > being up to date.
> >
> >> Given the resulting C API compatibility break and the loss of code
> sharing, I don't think it's a net win to drop the extra storage.
>
> I'm not sure what you mean by "loss of code sharing", but I don't see
> how a mapping that is both a cache and a proxy can be easier to
> implement than a mapping that is just a proxy.
>

The read side is currently completely handled by the existing
MappingProxyType, so the frame locals proxy subclass only implements the
extra MutableMapping bits.


"reads are relying on the snapshot being up to date", doesn't that just
> mean that reads can be wrong?
>
> There is no way to keep a snapshot reliably up to date in the presence
> of threads.
>

We don't support running the same *frame* from multiple threads, so only
one thread can be updating the frame directly, and the proxy writes to both
the frame and the snapshot.

But avoiding the need to update the entire snapshot to read specific
variables from the frame would definitely be a compelling technical benefit
regardless.


> >
> > For now, I have a PR adding this as an open question:
> > https://github.com/python/peps/pull/1787/files
> >
> > Given the performance benefit of being able to more reasonably drop
> > the implicit call to `PyFrame_LocalsToFast()`, I'm mostly convinced
> > that switching reads to pull from the frame if possible is the right
> > thing to do, even if it reduces the amount of code that can be
> > inherited without modification from MappingProxyType.
> >
> > The API compatibility concerns would mean the extra mapping store
> > still needed to stick around, but it would only be used for: >
> > * providing backwards compatibility for the `PyEval_GetLocals()` and
> > C-level `f_locals` interfaces
> > * reading and writing names that don't have entries in the `fast_refs`
> mapping
> > * writing shadow updates for names in the `fast_refs` mapping
>
> Given that f_locals is broken, why is keeping compatibility for this
> obscure, and probably unused case worthwhile?
>

f_locals *isn't* broken a lot of the time. The PEP aims to keep that code
working unchanged rather than forcing people to modify their code to handle
problems they may not have.


> The break in compatibility with locals() seems much more intrusive, yet
> you are OK with that (as am I).
>

PyEval_GetLocals() is part of the stable ABI and returns a borrowed
reference. That means there are a lot of implementation restrictions around
keeping that API working. A follow-up PEP could propose deprecating and
removing the API as intrinsically broken, but I don't want to go that far
in PEP 558.

By contrast, for the Python level API, Nathaniel was able to make a
compelling case that most locals() usage would at worst suffer a
performance degradation with the semantic change, and at least some use
cases would see latent defects transparently fixed.

Cheers,
Nick.



> Cheers,
> Mark.
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HLWVTPUF5GPSUDU322AHYRPCEPFZEYKM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: More comments on PEP 558

[Nick Coghlan]

On Sat, 30 Jan 2021, 10:22 pm Mark Shannon,  wrote:

> Hi Nick,
>
> A couple more issues with PEP 558, as I see it.
>
>
> Lack of specification
> -
>
> There is no specification section in PEP 558.
>
> Much of PEP 558 describes the differences between the current behavior
> and what PEP 558 proposes. These differences are often intermingled with
> a discussion of the flaws with the current behavior, which makes it
> difficult to understand what the proposed behavior is.
>
> I could not implement PEP 558 from the PEP alone.
>

That's probably true, but as I wasn't trying to fully specify the
implementation of the new C API, just describe the usage of it.



>
> Cycles and the cost of maintaining the f_locals cache
> -
>
> PEP 558 proposes that f_locals acts as a proxy mapping to the locals in
> a frame *and* that the frame caches the proxy. This obviously creates a
> cycle, which needs to be broken by the cycle GC.
>
> Although the cycle only exists if f_locals is actually used, there is
> still some overhead for *every* call and return, as the f_locals field
> needs to be initialized on call and checked for non-NULL on return.
>


Where does it still propose caching the proxy? You convinced me to take
that out ages ago, and just store a plain dict instead, the same way Python
3.9 (et al) do.

There was one remaining reference to cycle breaking, but the latest PR
removes that. (That PR is still open, as it may yet move the new "Open
Issue" to an actual design change).

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D2C5UE47XJN2GU3HT22WR77WPDCDNYHX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Sat, 30 Jan 2021 at 10:30, Nick Coghlan  wrote:
> On Sat, 30 Jan 2021, 12:13 am Mark Shannon,  wrote:
>> With a direct proxy coherency is not an issue.
>
> For things in the frame, it *is* a direct proxy - reads pull from the frame 
> object, and writes go to both the frame object and the mapping: 
> https://github.com/python/cpython/pull/3640/files#diff-7b8cef249e5cca077d30de4e428a6bde6b9b803464e790e9cffa7e052e19efddR1315

Reviewing the code again, I'd misremembered how the draft
implementation works - right now, reads are relying on the snapshot
being up to date.

> Given the resulting C API compatibility break and the loss of code sharing, I 
> don't think it's a net win to drop the extra storage.

For now, I have a PR adding this as an open question:
https://github.com/python/peps/pull/1787/files

Given the performance benefit of being able to more reasonably drop
the implicit call to `PyFrame_LocalsToFast()`, I'm mostly convinced
that switching reads to pull from the frame if possible is the right
thing to do, even if it reduces the amount of code that can be
inherited without modification from MappingProxyType.

The API compatibility concerns would mean the extra mapping store
still needed to stick around, but it would only be used for:

* providing backwards compatibility for the `PyEval_GetLocals()` and
C-level `f_locals` interfaces
* reading and writing names that don't have entries in the `fast_refs` mapping
* writing shadow updates for names in the `fast_refs` mapping

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZKS4NG6HNOSNHJVUYHKHNSWT747FFPFP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Improve CPython tracing performance

[Nick Coghlan]

On Wed, 20 Jan 2021 at 19:22, Victor Stinner  wrote:
>
> Reply to an old thread.
>
> On Sat, Oct 31, 2020 at 8:02 AM Nick Coghlan  wrote:
> > > Debuggers and profilers usually only care of specific frames or
> > > function calls (ex: 10% of function calls or even a single function
> > > call in a whole application). The problem is how to make them as
> > > efficient as possible for "no operation" calls, when they don't care
> > > about the current frame. Avoiding PyFrame_FastToLocalsWithError() to
> > > enter the debugger/profile and avoiding PyFrame_LocalsToFast() on exit
> > > sounds a simple and practical solution.
> >
> > Aye, I agree. I just don't think we can remove those implicit calls
> > without preparing a replacement API first.
>
> Again, I don't think that it's incompatible. We can enforce calling
> PyFrame_FastToLocalsWithError() at enter and PyFrame_LocalsToFast() at
> exit for now, and enhance the API later.

PEP 558 makes `PyFrame_LocalsToFast()` raise an exception, so the two
approaches definitely aren't compatible :)

> To be clear: currently, PyFrame_FastToLocalsWithError() is called at
> enter and PyFrame_LocalsToFast() is called at exit. So asking
> debuggers/profilers to call them explicitly doesn't make the situation
> worse (nor better ;-)) for generators/coroutines, it would be exactly
> the same behavior. It's just an optimization.
>
> The PEP 558 is being discussed for 5 years and still a draft. I don't
> think that it should hold bpo-42197 optimization.

No, what should hold up the bpo-42197 PR is the fact that it's an API
compatibility break that shouldn't be done without a PEP.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PDHPZ3DQLT2Z4S5PTUKLB6FUJ3R676NI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Sat, 30 Jan 2021, 12:13 am Mark Shannon,  wrote:

> Hi
>
> It is a lot more complex, because you need to worry about coherency.
> With a direct proxy coherency is not an issue.
>

For things in the frame, it *is* a direct proxy - reads pull from the frame
object, and writes go to both the frame object and the mapping:
https://github.com/python/cpython/pull/3640/files#diff-7b8cef249e5cca077d30de4e428a6bde6b9b803464e790e9cffa7e052e19efddR1315

The technical pay-off from having the mapping there is that most read-only
dict operations are handled by the existing MappingProxyType.

If you drop the mapping, then you lose any code sharing, and have to
implement the full mapping API from scratch, instead of just the mutation
and single key lookup parts.

You also reduce compatibility with the status quo, since writes to unknown
keys will fail rather than being stored solely in the mapping section, and
there's also no way to preserve the "PyEval_GetLocals()" C API, which is
expected to return a borrowed reference to a plain Python dict stored in
the C level "f_locals" field on the frame struct.

The cache coherency issues on the bulk mapping APIs this perpetuates also
aren't any worse than those on the locals() built-in itself: they'll see
the state as of the last snapshot update rather than the live state on the
frame (which may not exist any more if the proxy outlives the underlying
frame).

Given the resulting C API compatibility break and the loss of code sharing,
I don't think it's a net win to drop the extra storage.

Cheers,
Nick.






>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BZAMC7NXQGYY7WGDAOUT3TYXBDP7OZCT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Thu, 28 Jan 2021, 11:18 pm Mark Shannon,  wrote:

>
> Hi Nick,
>
> Regarding `f_locals` PEP 558 states:
>
> """
> Instead of being a direct reference to the internal dynamic snapshot
> used to populate the independent snapshots returned by locals(),
> frame.f_locals will be updated to instead return a dedicated proxy type
> (implemented as a private subclass of the existing
> types.MappingProxyType) that has two internal attributes not exposed as
> part of the Python runtime API:
>
> *mapping: an implicitly updated snapshot of the function local
> variables and closure references, as well as any arbitrary items that
> have been set via the mapping API, even if they don't have storage
> allocated for them on the underlying frame
> *frame: the underlying frame that the snapshot is for
>
> """
>
> This seems rather complex, and consequently fragile.
> I fear that this is just going to result in different bugs, rather than
> fixing the bugs it proposes to fix.
>
> Why not just make `f_local` a direct view on the underlying frame?
> It would be simpler to understand, more robust, and should perform better.
>

The concern I have with the simplification is that I don't know what would
break if trace hooks lost the ability to stash additional state that the
compiler doesn't know anything about in f_locals.

Rather than trying to assess how common such usage is, and whether we care
about breaking any use cases that people have for it, I instead elected to
just keep it working.

The extra implementation complexity beyond what's already needed to cope
with closure cells also isn't that much.

Cheers,
Nick.



> Cheers,
> Mark.
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MCRXLN5O5NVOY4WGYLTOSQYSGD2255WB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

[Nick Coghlan]

On Tue, 26 Jan 2021 at 01:05, Mark Shannon  wrote:
> The PEP mentions "tracing mode" and changes behavior according to
> whether a program is in "tracing mode" or not. I'd like to remove this
> distinction.

OK, I've checked the status of things from an actual computer rather
than my phone, and it turns out this is just confusing wording on my
part: the PEP still makes it sound like the "tracing mode" distinction
will persist into the future, even though the actual goal now is to
eliminate that distinction.

I've made a new PR to the PEP to try to clear that up:
https://github.com/python/peps/pull/1783/files

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4KATOFB2YMC7VUETL24FG2OMUKR6Q57V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

[Nick Coghlan]

Hmm, I need to review the state of my PEP PRs, as what's in the main repo
definitely isn't up to date. The tracing mode distinction should be long
gone, but is still mentioned in the text on python.org.

Cheers,
Nick.

On Wed, 27 Jan 2021, 10:03 pm Nick Coghlan,  wrote:

> Note that PEP 558 already doesn't change behaviour in tracing mode any
> more though - that idea didn't survive the first round of review.
>
> Cheers,
> Nick.
>
> On Wed, 27 Jan 2021, 9:58 pm Nick Coghlan,  wrote:
>
>> As far as I'm aware, the design is in a potentially acceptable state, I
>> just stalled out completely on the boring bits of finishing the
>> implementation of the write-through proxy:
>>
>> * implement & test the rest of the mutable mapping methods
>> * refactor to properly share code with the odict implementation instead
>> of copying & pasting it
>>
>> So a co-author would definitely be most welcome, given I've been
>> procrastinating on that part for literally years at this point.
>>
>> Cheers,
>> Nick.
>>
>>
>> On Tue, 26 Jan 2021, 1:21 am Guido van Rossum,  wrote:
>>
>>> Sounds good to me. Have you talked to Nick?
>>>
>>> On Mon, Jan 25, 2021 at 07:07 Mark Shannon  wrote:
>>>
>>>> Hi,
>>>>
>>>> PEP 558 seems to be dormant, if not abandoned.
>>>>
>>>> There are at least two open issues for bugs resulting from the
>>>> currently
>>>> weird and inefficient behavior of `f_locals` and `locals()`.
>>>> See https://bugs.python.org/issue30744 for an example of undesirable
>>>> behaviour.
>>>>
>>>> PEP 588, or something like it, would fix those.
>>>>
>>>> I'd be happy to take over the PEP, or write a new one.
>>>> I like PEP 588, although I would propose a simplification.
>>>>
>>>> The PEP mentions "tracing mode" and changes behavior according to
>>>> whether a program is in "tracing mode" or not. I'd like to remove this
>>>> distinction.
>>>>
>>>> Cheers,
>>>> Mark.
>>>> ___
>>>> Python-Dev mailing list -- python-dev@python.org
>>>> To unsubscribe send an email to python-dev-le...@python.org
>>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>>> Message archived at
>>>> https://mail.python.org/archives/list/python-dev@python.org/message/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>
>>> --
>>> --Guido (mobile)
>>>
>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G6K3VT33NR2XWZRAPN3XHZJ5R24H7IUV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

[Nick Coghlan]

Note that PEP 558 already doesn't change behaviour in tracing mode any more
though - that idea didn't survive the first round of review.

Cheers,
Nick.

On Wed, 27 Jan 2021, 9:58 pm Nick Coghlan,  wrote:

> As far as I'm aware, the design is in a potentially acceptable state, I
> just stalled out completely on the boring bits of finishing the
> implementation of the write-through proxy:
>
> * implement & test the rest of the mutable mapping methods
> * refactor to properly share code with the odict implementation instead of
> copying & pasting it
>
> So a co-author would definitely be most welcome, given I've been
> procrastinating on that part for literally years at this point.
>
> Cheers,
> Nick.
>
>
> On Tue, 26 Jan 2021, 1:21 am Guido van Rossum,  wrote:
>
>> Sounds good to me. Have you talked to Nick?
>>
>> On Mon, Jan 25, 2021 at 07:07 Mark Shannon  wrote:
>>
>>> Hi,
>>>
>>> PEP 558 seems to be dormant, if not abandoned.
>>>
>>> There are at least two open issues for bugs resulting from the currently
>>> weird and inefficient behavior of `f_locals` and `locals()`.
>>> See https://bugs.python.org/issue30744 for an example of undesirable
>>> behaviour.
>>>
>>> PEP 588, or something like it, would fix those.
>>>
>>> I'd be happy to take over the PEP, or write a new one.
>>> I like PEP 588, although I would propose a simplification.
>>>
>>> The PEP mentions "tracing mode" and changes behavior according to
>>> whether a program is in "tracing mode" or not. I'd like to remove this
>>> distinction.
>>>
>>> Cheers,
>>> Mark.
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> --
>> --Guido (mobile)
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GJSORMCNKKZUAXHJQOBBOQUKIHE2RX22/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Resurrecting PEP 558 (defined semantics for locals())

[Nick Coghlan]

As far as I'm aware, the design is in a potentially acceptable state, I
just stalled out completely on the boring bits of finishing the
implementation of the write-through proxy:

* implement & test the rest of the mutable mapping methods
* refactor to properly share code with the odict implementation instead of
copying & pasting it

So a co-author would definitely be most welcome, given I've been
procrastinating on that part for literally years at this point.

Cheers,
Nick.


On Tue, 26 Jan 2021, 1:21 am Guido van Rossum,  wrote:

> Sounds good to me. Have you talked to Nick?
>
> On Mon, Jan 25, 2021 at 07:07 Mark Shannon  wrote:
>
>> Hi,
>>
>> PEP 558 seems to be dormant, if not abandoned.
>>
>> There are at least two open issues for bugs resulting from the currently
>> weird and inefficient behavior of `f_locals` and `locals()`.
>> See https://bugs.python.org/issue30744 for an example of undesirable
>> behaviour.
>>
>> PEP 588, or something like it, would fix those.
>>
>> I'd be happy to take over the PEP, or write a new one.
>> I like PEP 588, although I would propose a simplification.
>>
>> The PEP mentions "tracing mode" and changes behavior according to
>> whether a program is in "tracing mode" or not. I'd like to remove this
>> distinction.
>>
>> Cheers,
>> Mark.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> --
> --Guido (mobile)
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J2DQDKEYVH3IQBEXEJLBUEQ4GYHU6X4F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Let's Fix Class Annotations -- And Maybe Annotations Generally

[Nick Coghlan]

On Wed, 13 Jan 2021, 12:35 pm Larry Hastings,  wrote:

>
> On 1/12/21 5:28 PM, Brett Cannon wrote:
>
> The other thing to keep in mind is we are talking about every module,
> class, and function getting 64 bytes ... which I bet isn't that much.
>
> Actually it's only every module and class.  Functions don't have this
> problem because they've always stored __annotations__ internally--meaning,
> peeking in their __dict__ doesn't work, and they don't support inheritance
> anyway.  So the number is even smaller than that.
>
> If we can just make __annotations__ default to an empty dict on classes
> and modules, and not worry about the memory consumption, that goes a long
> way to cleaning up the semantics.
>

Could you get the best of both worlds by making __annotations__ an
auto-populating descriptor on "type", the way it is on functions?

Continue to add a non-empty annotations dict to the class dict eagerly, but
only add the empty dict when "cls.__annotations__" is accessed.

Then your co_annotations PEP would only be changing the way the non-empty
case was handled, rather than introducing the descriptor in the first place.

Cheers,
Nick.

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BMBFLLQZY4WZP5RQ32OFQYR2N3UMHTFF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Nick Coghlan]

On Tue, 12 Jan 2021, 11:55 am Ethan Furman,  wrote:

> On 1/7/21 4:56 AM, Nick Coghlan wrote:
>
>  > Both EnumMeta and ABCMeta should probably be relying on `__set_name__`
>  > for their per-member set up work these days, rather than deferring
>  > that work until after __new__ returns.
>
> And here I was thinking that `__set_name__` was for, well, setting the
> name.  ;-)
>


Yeah, __set_owner__ was also discussed as a possible name, since the method
notifies the descriptor of both the defining class and the attribute name
within that class.

We decided we preferred __set_name__ out of the two short options.



> But it does work.
>

Good to hear :)

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RY7FBQFJ4T64CYILI5B3BOL2KYX65X62/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 642 v3: Explicit patterns for structured pattern matching

[Nick Coghlan]

On Sun, 10 Jan 2021, 7:37 pm Paul Sokolovsky,  wrote:

> And I patiently continue this thread, hoping that people whose argument
> would be along the lines of "I teach Python, and I don't want to teach
> my students 2 ways of doing the same thing, and which way use when. Why,
> if PEP634 offers just one way?"
>

They don't do the same thing, though. One does traditional duck typing
(checking for the presence of a specified set of attributes on an object),
while the other matches a sequence of attributes specified by the class.

PEP 634 just conflates the two tasks into a single call-like syntax that
may or may not bare any resemblance to the type's constructor signature.

Cheers,
Nick.

>


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2XYLWGHUSDADOUWDDHPLL2NNMJZM4VO2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 642 v3: Explicit patterns for structured pattern matching

[Nick Coghlan]

On Sun, 10 Jan 2021, 2:55 am Paul Moore,  wrote:

> On Sat, 9 Jan 2021 at 14:54, Nick Coghlan  wrote:
>
> [...]
> > And I've already said I'd be fine with making the colon mandatory if the
> SC share that view.
>
> So the response to my comment that omitting the colon makes it hard to
> see that it's a dictionary unpacking is either "don't use that form if
> you don't like it" (which as we all know doesn't account for the
> problem of dealing with other people's code) or "the SC can make the
> colon mandatory" which seems to me to be extending the SC's role to
> designing the syntax rather than just approving the proposal. And
> means that the PEP we're debating might not be what gets accepted.
> Which isn't how I understood the process to work, but OK.
>

It's the same process any PEP goes through: comments from the PEP delegate
carry more weight than general review comments, as ignoring them risks
getting the PEP rejected.

>
The SC haven't provided any feedback on the submitted version of 642 yet,
so I don't know at this point if they share the concern that allowing
omitting the key/value separating colon to be omitted pushes mapping &
instance attribute patterns too close to looking like a variation on set
syntax instead of mapping syntax.

I do accept that it's a legitimate concern, it's just also a feature that's
easy to drop if the SC agrees it is problematic.


> And if the colon is mandatory, the syntax is {"text": as message,
> "color": as c}. Which is essentially the same as PEP 634 with an
> additional "as". Which basically makes this a somewhat disguised
> version of "add a marker to indicate the variable to be bound".


It isn't disguised at all, as I state it in the PEP's Design Discussion
section:

"""... instead requiring that all uses of bare simple names for anything
other than a variable lookup be qualified by a preceding sigil or keyword"""

(from
https://www.python.org/dev/peps/pep-0642/#requiring-explicit-qualification-of-simple-names-in-match-patterns
)

Meh.
> We've been down this route, some people want bindings to be explicitly
> marked, other people don't. I don't feel compelled to have explicit
> markers, I like the "match pattern looks like what's being matched"
> feature of PEP 634,


PEP 634 doesn't have that feature for class patterns in general, only for
classes like data classes, where the constructor signature is carefully
aligned with the match arguments.

For positional args, PEP 642 would also align in those carefully
constructed cases, so it's only the new syntax for attribute matching where
things differ.

For mapping patterns, I think you've made a reasonable case that the colon
should be required even when unconditionally binding names, so it wouldn't
surprise me at all if the shorthand to omit them gets rejected by the SC
(or at least moved to the "Deferred Ideas" section).

> What other syntax do we have that binds to the right without "as"?
> Allowing it at all is only as old as with statements, but that's still more
> precedence than ":" and "=" have for being used that way.
>
> None at the moment, but that's not my point. My point is that patterns
> are more like expressions than statements, and all current uses of
> "as" for binding are in statements. Expressions typically use
> punctuation rather than keywords, so there's no compelling argument
> that the syntax for "binding the RHS" *in expressions* has to be "as".
>

But PEP 634 uses "PTRN as NAME" as well.

The only difference in this regard between the two PEPs is that 634 allows
"_ as NAME" to be shortened to just "NAME" (omitting both the pattern and
the keyword), while 642 only allows "__ as NAME" to be shortened to "as
NAME" (omitting only the pattern).

You say "we use as in statements, so let's re-use that", I say "we use
> punctuation in expressions, and a keyword looks clumsy and verbose". I
> don't believe that point has ever been addressed directly, which is
> why I get frustrated that arguments about "as" seem to be getting
> redirected into arguments about binding to the right all the time.
>

I didn't realise it was an argument you were trying to make.

Cheers,
Nick.


> >
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4YGNYB44PXEHD3WFX6DKQI3LHOAYUC5S/
Code of Conduct: http://python.org/psf/codeofconduct/