[Python-Dev] Re: enum in the stable ABI (Was: PEP 558: Defined semantics for locals)

[Petr Viktorin]

On 24. 07. 21 1:58, Nick Coghlan wrote:



On Sat, 24 Jul 2021, 9:37 am Larry Hastings, > wrote:



On 7/23/21 7:38 AM, Petr Viktorin wrote:

(In both C & C++, the size of an `enum` is implementation-defined.
That's unlikely to be a problem in practice, but one more point
against enum.)



True, but there's always the old trick of sticking in a value that
forces it to be at least 32-bit:

typedef enum {
     INVALID = 0,
     RED = 1,
     BLUE = 2,
     GREEN = 3,

     UNUSED = 1073741824
} color_t;


//arry/


My current inclination is to define the enum as "_PyLocals_KindValues", 
and then typedef "PyLocals_Kind" itself as an int. The frame API would 
then return the former, while the stable query API would return the latter.


However, I'll make a full survey of the enums currently in the stable 
ABI before making a decision, as there may be an existing approach that 
I like better.


Here's a survey: https://bugs.python.org/issue44727#msg398071

I do agree Petr's right to be cautious about this, as compilers can get 
up to some arcane shenanigans in the presence of formally undefined 
code: https://queue.acm.org/detail.cfm?id=3468263 



The fact that the behaviour in this case is likely to be well-defined at 
the time of compilation would protect us from the weirder potential 
outcomes, but it still makes sense for us to define the query API in a 
way that tells both compilers and humans not to assume that the values 
returned by the current version of Python are the only values that will 
ever be returned by all future versions of Python.


If you ask me, I don't think C provides that much type safety to make 
enum worth it, even for the version-specific API.
But I like Larry's "old trick" better than having two different APIs. 
Thanks for that! It's a new trick for me!

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JIBBPKHA5URUBKF43XWMUANP5ESQB4JM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

(I'm not sure Mailman will get the threading right when I've received
the original email directly, so apologies in advance if this reply
creates a 2nd thread)
On Mon, 26 Jul 2021 at 23:57, Mark Shannon  wrote:
>
> Hi,
>
> First of all let me say that I agree with the aims of PEP 558 and most
> of the design.
> I do find the wording of the PEP itself a bit confusing in a couple of
> places.
>
> This critique is based on my understanding of the PEP.
> If I am mistaken in my misunderstanding, then treat that as an implied
> request for clarification of the PEP :)
>
>
> Critique of PEP 558
> ===
>
> Layout of the PEP
> -
>
> [This is largely to help the reader, it doesn't change the nature of the
> PEP]
>
> Could we replace the section "CPython Implementation Changes" with a
> section that states what the behavior will be, not how it changes.
> Having to mentally apply the suggested changes to the existing (and
> convoluted) implementation makes it hard to work out what the proposed
> behavior will be.

Just stating what the new behaviour is isn't sufficient to assess the
impact of the change, though - it needs to be compared to the old
(arcane) behaviour, and I don't want to assume readers already
understand what that behaviour is.

That said, the summary section is intended to describe how things will
work post-change. It just leaves out the details of the new proxy
implementation (which I admit is the most complicated part of the
updated implementation).

> Could the design discussion be moved to an appendix or another document?
> The key parts of it should be moved to the Rationale or Motivation.

It's effectively an Appendix already (the only things after it are the
pointer to the implementation and the acknoweldgements).

That said, I'll note that several of the questions you've asked in
this email are answered directly in the Design Discussions section
that you're suggesting removing from the PEP.

> There should be a Motivation section explaining why this PEP is
> necessary, for those not familiar with the weirdness of `locals()`.
> Much of what is in the Rationale should perhaps be in the Motivation.

Yes, the Rationale section could be retitled Motivation - it's the
rationale for the PEP existing, not the rationale for the design
decisions (those are in the Design Discussion section).

The historical arcane behaviour at function scope is covered in
https://www.python.org/dev/peps/pep-0558/#historical-semantics-at-function-scope
but the bug references seemed more important for the Motivation
section (if the status quo *worked*, I'd never have tried to work out
how to change it, but with the status quo both arcane *and* broken, it
makes sense to try to figure out a better alternative).

>
> Proposal
> 
>
> [Ditto; this is largely to help the reader, it doesn't change the nature
> of the PEP]
>
> Drop the definitions of the type of scope. They are (at least they
> should be) clearly defined in existing documentation.

They're not, unfortunately. As far as I am aware,
https://docs.python.org/3/reference/executionmodel.html#naming-and-binding
is as good as we've currently got, so

> Why "largely" eliminate the concept of a separate tracing mode? Wasn't
> the plan to eliminate it entirely?

The "largely" there just relates to the fact that even though the PEP
eliminates the side effects that tracing mode has historically had on
the behaviour of locals(), the core eval loop still has the notion of
"tracing or not" that turns off some of the execution shortcuts it can
otherwise take.

> Rather than specifying what the documentation will be, could you specify
> the semantics. The language here should be precise.

The reference documentation should be precise as well, since that is
what other implementations will be following.

What semantics do you feel are left unspecified?

> The documentation needs to explain the behavior to the majority of
> users, but the PEP should be worded so that it can be implemented
> correctly from just reading the PEP.
>
> There is no reason to remove the sugested documentation changes, but
> they should be secondary to the more formal specification.

I don't know what you want to see on this front.

The current PEP seems complete to me, except in relation to the
specifics of the proxy behaviour, which it deliberately leaves
underspecified (mostly because my current implementation sucks in a
few areas, and I really don't want to enshrine those limitations in
the language spec).

> Resolving the issues with tracing mode behaviour
> 
>
> According to this section the `f_locals` object (_PyFastLocalsProxy_Type
> in C) will have *two* fields.
> It should only have *one*, the pointer to the underlying frame object.

That's a good point - fast_refs doesn't reference the frame object, so
it's safe to store it on the frame object and reference it from the
individual proxies.

Switching to that approac

[Python-Dev] Re: Comments on PEP 558

[Mark Shannon]

Hi Nick,

On 27/07/2021 2:29 pm, Nick Coghlan wrote:

(I'm not sure Mailman will get the threading right when I've received
the original email directly, so apologies in advance if this reply
creates a 2nd thread)
On Mon, 26 Jul 2021 at 23:57, Mark Shannon  wrote:


Hi,

First of all let me say that I agree with the aims of PEP 558 and most
of the design.
I do find the wording of the PEP itself a bit confusing in a couple of
places.

This critique is based on my understanding of the PEP.
If I am mistaken in my misunderstanding, then treat that as an implied
request for clarification of the PEP :)


Critique of PEP 558
===

Layout of the PEP
-

[This is largely to help the reader, it doesn't change the nature of the
PEP]

Could we replace the section "CPython Implementation Changes" with a
section that states what the behavior will be, not how it changes.
Having to mentally apply the suggested changes to the existing (and
convoluted) implementation makes it hard to work out what the proposed
behavior will be.


Just stating what the new behaviour is isn't sufficient to assess the
impact of the change, though - it needs to be compared to the old
(arcane) behaviour, and I don't want to assume readers already
understand what that behaviour is.

That said, the summary section is intended to describe how things will
work post-change. It just leaves out the details of the new proxy
implementation (which I admit is the most complicated part of the
updated implementation).


Could the design discussion be moved to an appendix or another document?
The key parts of it should be moved to the Rationale or Motivation.


It's effectively an Appendix already (the only things after it are the
pointer to the implementation and the acknoweldgements).

That said, I'll note that several of the questions you've asked in
this email are answered directly in the Design Discussions section
that you're suggesting removing from the PEP.


There should be a Motivation section explaining why this PEP is
necessary, for those not familiar with the weirdness of `locals()`.
Much of what is in the Rationale should perhaps be in the Motivation.


Yes, the Rationale section could be retitled Motivation - it's the
rationale for the PEP existing, not the rationale for the design
decisions (those are in the Design Discussion section).

The historical arcane behaviour at function scope is covered in
https://www.python.org/dev/peps/pep-0558/#historical-semantics-at-function-scope
but the bug references seemed more important for the Motivation
section (if the status quo *worked*, I'd never have tried to work out
how to change it, but with the status quo both arcane *and* broken, it
makes sense to try to figure out a better alternative).



Proposal


[Ditto; this is largely to help the reader, it doesn't change the nature
of the PEP]

Drop the definitions of the type of scope. They are (at least they
should be) clearly defined in existing documentation.


They're not, unfortunately. As far as I am aware,
https://docs.python.org/3/reference/executionmodel.html#naming-and-binding
is as good as we've currently got, so


Why "largely" eliminate the concept of a separate tracing mode? Wasn't
the plan to eliminate it entirely?


The "largely" there just relates to the fact that even though the PEP
eliminates the side effects that tracing mode has historically had on
the behaviour of locals(), the core eval loop still has the notion of
"tracing or not" that turns off some of the execution shortcuts it can
otherwise take.


Rather than specifying what the documentation will be, could you specify
the semantics. The language here should be precise.


The reference documentation should be precise as well, since that is
what other implementations will be following.

What semantics do you feel are left unspecified?


The documentation needs to explain the behavior to the majority of
users, but the PEP should be worded so that it can be implemented
correctly from just reading the PEP.

There is no reason to remove the sugested documentation changes, but
they should be secondary to the more formal specification.


I don't know what you want to see on this front.

The current PEP seems complete to me, except in relation to the
specifics of the proxy behaviour, which it deliberately leaves
underspecified (mostly because my current implementation sucks in a
few areas, and I really don't want to enshrine those limitations in
the language spec).


All the above were mostly just suggestions, the format of the PEP is up 
to you.


But please, do not underspecify anything. I need to maintain this, as do 
the PyPy and MicroPython developers. We need this to be precise.





Resolving the issues with tracing mode behaviour


According to this section the `f_locals` object (_PyFastLocalsProxy_Type
in C) will have *two* fields.
It should only have *one*, the pointer to the underlying fram

[Python-Dev] Re: Comments on PEP 558

[Nick Coghlan]

On Wed, 28 Jul 2021, 1:50 am Mark Shannon,  wrote:

> Hi Nick,
>
> On 27/07/2021 2:29 pm, Nick Coghlan wrote:
> >
> >
> > The reference documentation should be precise as well, since that is
> > what other implementations will be following.
> >
> > What semantics do you feel are left unspecified?
> >
> >> The documentation needs to explain the behavior to the majority of
> >> users, but the PEP should be worded so that it can be implemented
> >> correctly from just reading the PEP.
> >>
> >> There is no reason to remove the sugested documentation changes, but
> >> they should be secondary to the more formal specification.
> >
> > I don't know what you want to see on this front.
> >
> > The current PEP seems complete to me, except in relation to the
> > specifics of the proxy behaviour, which it deliberately leaves
> > underspecified (mostly because my current implementation sucks in a
> > few areas, and I really don't want to enshrine those limitations in
> > the language spec).
>
> All the above were mostly just suggestions, the format of the PEP is up
> to you.
>
> But please, do not underspecify anything. I need to maintain this, as do
> the PyPy and MicroPython developers. We need this to be precise.
>

You still haven't said what you consider underspecified.

Perhaps read the implementation and tell me which parts of what the tests
are covering you want to see replicated in the PEP text?


> >
> >> Resolving the issues with tracing mode behaviour
> >> 
> >>
> >> According to this section the `f_locals` object (_PyFastLocalsProxy_Type
> >> in C) will have *two* fields.
> >> It should only have *one*, the pointer to the underlying frame object.
> >
> > That's a good point - fast_refs doesn't reference the frame object, so
> > it's safe to store it on the frame object and reference it from the
> > individual proxies.
> >
> > Switching to that approach also has the advantage that every proxy
> > after the first is much cheaper, since they don't need to build the
> > fast refs map again.
> >
> >> In order to maximize backwards compatibility and, more importantly,
> >> avoid the synchronization issues that lead to obscure bugs, all the
> >> state of the `f_locals` object should be kept on the frame.
> >>
> >> Any changes to `f_locals` should be instantly visible to any thread both
> >> through other `f_locals` objects (for the same frame) and the underlying
> >> local variable (if it exists).
> >
> > [snip proxy method implementation sketch] >
> > This gets the gist of how the implementation works, but it isn't quite
> > that simple for a few reasons:
> >
> > 1. The proxy needs to offer the same algorithmic complexity as dict
> > APIs, so doing O(n) linear searches of C arrays and Python tuples
> > inside O(1) lookup operations isn't OK (hence the fastrefs mapping to
> > allow for O(1) resolution of variable names to cell references and
> > local variable array offsets)
>
> Linear searches are faster for small arrays, and there is nothing
> stopping you adding a helper map of names->indexes for large functions.
> Provided the mapping from name to local index is O(1), then the whole
> operation is O(1).
>

That helper map already exists in the fast refs mapping. Skipping it for
small functions would be a future optimisation opportunity that doesn't
seem necessary in the initial implementation.


> You really don't need much extra machinery to maintain O(1) behavior
> (which no one cares about, they just care about overall performance).
>

Yes, the only extra machinery needed for working with individual keys is
the fast refs mapping.

It's other aspects of the mapping API that benefit from the continuing
presence of the  frame cache (which is maintained on the frame, just as it
is today, NOT separately in each proxy).

If you can give me a reliable O(1) implementation of "len(frame.f_locals)"
that doesn't rely on up to date cache of:
* which local and cell variable names are currently bound to values
* which extra variables have been added via either proxies or
PyEval_GetLocals()



> 2. The proxy needs to deal with *SIX* kinds of possible key lookup, not
> two:
> >
> > * unbound local variable (numeric index in fast refs map, NULL in fast
> > locals slot)
> > * bound local variable (numeric index in fast refs map, object pointer
> > in fast locals slot)
> > * unbound cell variable (cell reference in fast refs map, cell contains
> NULL)
> > * bound cell variable (cell reference in fast refs map, cell contains
> > object pointer)
> > * not yet converted cell variables (numeric index in fast refs map,
> > possibly NULL initial value in fast locals slot)
> > * extra keys injected via a fast locals proxy or PyEval_GetLocals()
>
> There could be a million different cases internally, but that doesn't
> need to reflected in the API.
>

And it isn't, *except* in the form of potential frame cache
desynchronisation.

Execution of the frame code can move local variables be

[Python-Dev] [IMPORTANT] [Release communication] Python 3.10.0rc1 next week: get ready!

[Pablo Galindo Salgado]

Hi everyone,

This is a friendly reminder from the release management team that the first
release candidate of Python 3.10 is
next Monday. Now is a fantastic time you make sure that:

## If you are a user or library developer

* If you filed a bug for something not working in any of the betas for
3.10, check that the bug is properly fixed.
* Ensure that your library/application works as expected with Python 3.10.
* Ensure that if your library needs to interact with the Python syntax or
type system, it works with the new additions in 3.10

.
* [Optional] Check that there are no performance regressions in your
application/library with Python 3.10

## If you are a core developer or a member of the triage team

* Merge or review any urgent bugfixes that you are interested in
* Your changes are properly documented.
* There isn't any critical bug in the tracker regarding a feature you
implemented/merged.
* If your change is relevant enough, it appears in the What's new document
 (if you have doubts, you
can ask me ;) ).

Some technical details of the release candidate:

Once the 3.10 branch reaches RC status, it only can have bug fixes applied
that have been reviewed by
other core developers (so you cannot merge your own PR without review even
if you are a core dev). Generally, these issues
must be severe enough (e.g. crashes) that they deserve fixing before the
final release. All other issues should be deferred to
the next development cycle (Python 3.10.1) since stability is the strongest
concern at this point. Also bear in mind that
once we reach the RC, the *ABI is frozen* and cannot change even for bug
fixes.

While the goal is to have no code changes between an RC and a final
release, there may be a need for final documentation o
test fixes. Any such proposed changes should be discussed first with the
release manager.

*You cannot skip the peer review during an RC*, no matter how small! Even
if it is a simple copy-and-paste change,
everything requires peer review from a core developer.

(You can find these instructions and details in the devguide
).

Thank you all for your help!

Regards from rainy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RLO5Y6FPOPUJAJVN6MHRIKUSMSRUXUT6/
Code of Conduct: http://python.org/psf/codeofconduct/