Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Random832]

On Wed, May 29, 2019, at 01:25, Nick Coghlan wrote:
> Having a single locals() call de-optimize an entire function would be 
> far from ideal.

What if there were a way to explicitly de-optimize a function, rather than 
guessing the user's intent based on looking for locals and exec calls (both of 
which are builtins which could be shadowed or assigned to other variables)?

Also, regardless of anything else, maybe in an optimized function locals should 
return a read-only mapping?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[MRAB]

On 2019-06-02 13:51, Steven D'Aprano wrote:

On Sun, Jun 02, 2019 at 11:52:02PM +1200, Greg Ewing wrote:

Armin Rigo wrote:
>You have the occasional big function that benefits a lot from being
>JIT-compiled but which contains ``.format(**locals())``.

There should be a lot less need for that now that we have f-strings.


I think you're forgetting that a lot of code (especially libraries)
either have to support older versions of Python, and so cannot use
f-strings at all, or was written using **locals before f-strings came
along, and hasn't been touched since.

Another case where f-strings don't help is when the template is
dynamically generated.

It may be that there will be less new code written using **locals() but
I don't think that the **locals() trick will disappear any time before
Python 5000.

We've had .format_map since Python 3.2, so why use 
``.format(**locals())`` instead of ``.format_map(locals())``?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Steven D'Aprano]

On Sun, Jun 02, 2019 at 11:52:02PM +1200, Greg Ewing wrote:
> Armin Rigo wrote:
> >You have the occasional big function that benefits a lot from being
> >JIT-compiled but which contains ``.format(**locals())``.
> 
> There should be a lot less need for that now that we have f-strings.

I think you're forgetting that a lot of code (especially libraries) 
either have to support older versions of Python, and so cannot use 
f-strings at all, or was written using **locals before f-strings came 
along, and hasn't been touched since.

Another case where f-strings don't help is when the template is 
dynamically generated.

It may be that there will be less new code written using **locals() but 
I don't think that the **locals() trick will disappear any time before 
Python 5000.


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Greg Ewing]

Armin Rigo wrote:

You have the occasional big function that benefits a lot from being
JIT-compiled but which contains ``.format(**locals())``.


There should be a lot less need for that now that we have f-strings.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Armin Rigo]

Hi,

On Wed, 29 May 2019 at 08:07, Greg Ewing  wrote:
> Nick Coghlan wrote:
> > Having a single locals() call de-optimize an entire function would be
> > far from ideal.
>
> I don't see what would be so bad about that. The vast majority
> of functions have no need for locals().

You have the occasional big function that benefits a lot from being
JIT-compiled but which contains ``.format(**locals())``.  That occurs
in practice, and that's why PyPy is happy that there is a difference
between ``locals()`` and ``sys._getframe().f_locals``.  PyPy could be
made to support the full mutable view, but that's extra work that
isn't done so far and is a bit unlikely to occur at this point.  It
also raises the significantly the efforts for other JIT
implementations of Python if they have to support a full-featured
``locals()``; supporting ``_getframe().f_locals`` is to some extent
optional, but supporting ``locals()`` is not.


A bientôt,

Armin.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Guido van Rossum]

On Thu, May 30, 2019 at 4:28 PM Greg Ewing 
wrote:

> Nick Coghlan wrote:
> > So for me, getting rid of write backs via exec and "import *" was a
> > matter of "Yay, we finally closed those unfortunate loopholes" rather
> > than being any kind of regrettable necessity.
>
> If that were the reasoning, the principled thing to do would be
> to raise an exception if an eval or exec tries to write to a
> local, rather than mostly ignore it.
>
> In any case, I don't really agree with that philosophy. Python
> is at its essence a dynamic language. Things like JIT and static
> type analysis are only possible to the extent that you refrain
> from using some of its dynamic features. Removing features
> entirely just because they *can* interfere with these things goes
> against the spirit of the language, IMO.
>

Right. And static analysis should also be able to detect most uses of
locals() in a frame. I believe I've heard of some alternate Python
implementations that detect usage of some functions to disable some
optimizations (IIRC IronPython did this to sys._getframe()).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Greg Ewing]

Nick Coghlan wrote:

So for me, getting rid of write backs via exec and "import *" was a
matter of "Yay, we finally closed those unfortunate loopholes" rather
than being any kind of regrettable necessity.


If that were the reasoning, the principled thing to do would be
to raise an exception if an eval or exec tries to write to a
local, rather than mostly ignore it.

In any case, I don't really agree with that philosophy. Python
is at its essence a dynamic language. Things like JIT and static
type analysis are only possible to the extent that you refrain
from using some of its dynamic features. Removing features
entirely just because they *can* interfere with these things goes
against the spirit of the language, IMO.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nick Coghlan]

On Thu, 30 May 2019 at 09:12, Greg Ewing  wrote:
>
> Nick Coghlan wrote:
> > If there was a compelling use case for letting "a = 1; exec(src);
> > print(a)" print something other than "1" at function scope, then I'd
> > be more amenable to the idea of the associated compatibility break and
> > potential performance regression in other implementations.
> >
> > However, there isn't any such use case - if there were, we wouldn't
> > have deliberately changed the semantics from the old Python 2 ones to
> > the current Python 3 ones in PEP 3100 [1].
>
> I get the impression that was done because everyone involved
> thought it wasn't worth the ugliness of maintaining all the
> fast/locals swapping stuff, not because of any principle that
> the current behaviour is right or better in any way.

You may have felt that way, but I certainly don't - routinely hiding
function local rebinding from the compiler (and type checkers, and
human readers) is awful, and a privilege that should be reserved to
debuggers and other tools operating at a similar level of "able to
interfere with the normal runtime execution of a program". (Module and
class namespaces are different - they're inherently mutable shared
namespaces as far as the Python runtime is occurred, so locals()
providing one more way of getting a reference to them isn't that big
of a deal - if it's a problem, you can just push the code where you
need to avoid those semantics matters down into a helper function)

So for me, getting rid of write backs via exec and "import *" was a
matter of "Yay, we finally closed those unfortunate loopholes" rather
than being any kind of regrettable necessity.

It's likely also a factor that Python 2.2.2 was the first version of
Python that I ever used extensively, so the existing snapshot-like
behaviour of locals() is the behaviour that feels normal to me, and
I'm somewhat mystified by the notion that anyone might actually *want*
it to behave differently (except to close the remaining loopholes that
still allowed mysterious implicit mutation of previously created
snapshots).

The only rationale for considering that possibility seems to be "It
would make function namespaces behave more like class and module
namespaces", to which my response is "Why would you even want that,
and what practical benefit would it bring to justify the otherwise
gratuitous compatibility break?"

Cheers,
Nick.


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Greg Ewing]

Nick Coghlan wrote:

If there was a compelling use case for letting "a = 1; exec(src);
print(a)" print something other than "1" at function scope, then I'd
be more amenable to the idea of the associated compatibility break and
potential performance regression in other implementations.

However, there isn't any such use case - if there were, we wouldn't
have deliberately changed the semantics from the old Python 2 ones to
the current Python 3 ones in PEP 3100 [1].


I get the impression that was done because everyone involved
thought it wasn't worth the ugliness of maintaining all the
fast/locals swapping stuff, not because of any principle that
the current behaviour is right or better in any way.

Given a locals proxy object, it would be much easier to support
the old behaviour (which seems obvious and correct to me) without
eval or exec having to be anything special.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nick Coghlan]

On Wed, 29 May 2019 at 16:08, Greg Ewing  wrote:
>
> Nick Coghlan wrote:
> > Having a single locals() call de-optimize an entire function would be
> > far from ideal.
>
> I don't see what would be so bad about that. The vast majority
> of functions have no need for locals().

If there was a compelling use case for letting "a = 1; exec(src);
print(a)" print something other than "1" at function scope, then I'd
be more amenable to the idea of the associated compatibility break and
potential performance regression in other implementations.

However, there isn't any such use case - if there were, we wouldn't
have deliberately changed the semantics from the old Python 2 ones to
the current Python 3 ones in PEP 3100 [1].

It's also worth noting that the "no backwards compatibility
guarantees" wording is only in the help() text, and not in
https://docs.python.org/3/library/functions.html#locals - the latter
just notes that writing back to it may not work, not that the
semantics may arbitrarily change between CPython versions.

I think the [snapshot] approach is a solid improvement over my initial
proposal, though, since removing the "locals() must always return the
same mapping object" requirement also makes it possible to remove some
oddities in the fastlocalsproxy implementation, and Nathaniel makes a
compelling case that in the areas where the status quo and the
snapshot proposal differ, those differences mostly either don't
matter, or else they will serve make code otherwise subject to subtle
bugs in tracing mode more correct.

Cheers,
Nick.

[1] "exec as a statement is not worth it -- make it a function" in
https://www.python.org/dev/peps/pep-3100/#core-language

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Guido van Rossum]

Indeed.

On Tue, May 28, 2019 at 11:07 PM Greg Ewing 
wrote:

> Nick Coghlan wrote:
> > Having a single locals() call de-optimize an entire function would be
> > far from ideal.
>
> I don't see what would be so bad about that. The vast majority
> of functions have no need for locals().
>
> --
> Greg
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-- 
--Guido (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Greg Ewing]

Nick Coghlan wrote:
Having a single locals() call de-optimize an entire function would be 
far from ideal.


I don't see what would be so bad about that. The vast majority
of functions have no need for locals().

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nick Coghlan]

On Wed., 29 May 2019, 2:29 pm Guido van Rossum,  wrote:

> So why is it “hellish” for JITs if locals() returns a proxy, while
> frame.f_locals being a proxy is okay?
>

As I understand it, they already drop out of compiled mode if they detect
that the code is tinkering with frame objects.

Having a single locals() call de-optimize an entire function would be far
from ideal.

Cheers,
Nick.

)
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Guido van Rossum]

So why is it “hellish” for JITs if locals() returns a proxy, while
frame.f_locals being a proxy is okay?

On Tue, May 28, 2019 at 9:12 PM Nick Coghlan  wrote:

> (I'll likely write a more detailed reply once I'm back on an actual
> computer, but wanted to send an initial response while folks in the US are
> still awake, as the detailed reply may not be needed)
>
> Thanks for this write-up Nathaniel - I think you've done a good job of
> capturing the available design alternatives.
>
> The one implicit function locals() update case that you missed is in the
> accessor for "frame.f_locals", so I think dropping CPython's implicit
> update for all Python trace functions would be a clear win (I actually
> almost changed the PEP to do that once I realized it was already pretty
> redundant given the frame accessor behaviour).
>
> I'm OK with either [PEP-minus-tracing] or [snapshot], with a slight
> preference for [snapshot] (since it's easier to explain).
>
> The only design option I wouldn't be OK with is [proxy], as I think that
> poses a significant potential backwards compatibility problem, and I trust
> Armin Rigo's perspective that it would be hellish for JIT-compiled Python
> implementations to handle without taking the same kind of performance hit
> they do when they need to emulate the frame API.
>
> By contrast, true snapshot semantics will hopefully make life *easier* for
> JIT compilers, and folks that actually want the update() behaviour can
> either rely on frame.f_locals, or do an explicit update.
>
> This would also create a possible opportunity to simplify the fast locals
> proxy semantics: if it doesn't need to emulate the current behaviour of
> allowing arbitrary keys to be added and preserved between calls to
> locals(), then it could dispense with its internal dict cache entirely, and
> instead reject any operations that try to add new keys that aren't defined
> on the underlying code object (removing keys and then adding them back
> would still be permitted, and handled like a code level del statement).
>
> Cheers,
> Nick.
>
>
> --
--Guido (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nick Coghlan]

(I'll likely write a more detailed reply once I'm back on an actual
computer, but wanted to send an initial response while folks in the US are
still awake, as the detailed reply may not be needed)

Thanks for this write-up Nathaniel - I think you've done a good job of
capturing the available design alternatives.

The one implicit function locals() update case that you missed is in the
accessor for "frame.f_locals", so I think dropping CPython's implicit
update for all Python trace functions would be a clear win (I actually
almost changed the PEP to do that once I realized it was already pretty
redundant given the frame accessor behaviour).

I'm OK with either [PEP-minus-tracing] or [snapshot], with a slight
preference for [snapshot] (since it's easier to explain).

The only design option I wouldn't be OK with is [proxy], as I think that
poses a significant potential backwards compatibility problem, and I trust
Armin Rigo's perspective that it would be hellish for JIT-compiled Python
implementations to handle without taking the same kind of performance hit
they do when they need to emulate the frame API.

By contrast, true snapshot semantics will hopefully make life *easier* for
JIT compilers, and folks that actually want the update() behaviour can
either rely on frame.f_locals, or do an explicit update.

This would also create a possible opportunity to simplify the fast locals
proxy semantics: if it doesn't need to emulate the current behaviour of
allowing arbitrary keys to be added and preserved between calls to
locals(), then it could dispense with its internal dict cache entirely, and
instead reject any operations that try to add new keys that aren't defined
on the underlying code object (removing keys and then adding them back
would still be permitted, and handled like a code level del statement).

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nathaniel Smith]

On Tue, May 28, 2019 at 6:48 PM Greg Ewing  wrote:
>
> Nathaniel Smith wrote:
> > - [proxy]: Simply return the .f_locals object, so in all contexts
> > locals() returns a live mutable view of the actual environment:
> >
> >   def locals():
> >   return get_caller_frame().f_locals
>
> Not sure I quite follow this --  as far as I can see, f_locals
> currently has the same snapshot behaviour as locals().
>
> I'm assuming you mean to change things so that locals() returns a
> mutable view tracking the environment in both directions. That
> sounds like a much better idea all round to me. No weird
> shared-snapshot behaviour, and no need for anything to behave
> differently when tracing.

Yeah, I made the classic mistake and forgot that my audience isn't as
immersed in this as I am :-). Throughout the email I'm assuming we're
going to adopt PEP 558's proposal about replacing f_locals with a new
kind of mutable view object, and then given that, asking what we
should do about locals().

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Greg Ewing]

Nathaniel Smith wrote:

- [proxy]: Simply return the .f_locals object, so in all contexts
locals() returns a live mutable view of the actual environment:

  def locals():
  return get_caller_frame().f_locals


Not sure I quite follow this --  as far as I can see, f_locals
currently has the same snapshot behaviour as locals().

I'm assuming you mean to change things so that locals() returns a
mutable view tracking the environment in both directions. That
sounds like a much better idea all round to me. No weird
shared-snapshot behaviour, and no need for anything to behave
differently when tracing.

If the change to the behaviour of exec() and eval() is a concern,
then perhaps there should be a new localsview() function that
returns a mutable view, with locals() redefined as dict(localsview()).

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Steven D'Aprano]

On Tue, May 28, 2019 at 08:37:17AM +0100, Paul Moore wrote:

> Of course, all of this is only if we have decided to formalise the
> semantics and change CPython to conform. I've never personally been
> affected by any of the edge cases with locals(), so on a purely
> personal basis, I'm equally happy with "do nothing" :-)

I don't think "Do Nothing" is a good option, because (as I understand 
it) the status quo has some weird bugs when you have tracing functions 
written in Python (but not in C?). So something has to change, one way 
or another. Hence Nick's PEP.


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Paul Moore]

On Tue, 28 May 2019 at 06:00, Nathaniel Smith  wrote:

> - There's the "justified" action-at-a-distance that currently happens
> at module scope, where locals().__setitem__ affects variable lookup,
> and variable mutation affects locals().__getitem__. This can produce
> surprising results if you pass locals() into something that's
> expecting a regular dict, but it's also arguably the point of an
> environment introspection API, and like you say, it's unavoidable and
> expected at module scope and when using globals().

If I understand you, this "justified" action at a distance is exactly
similar to how os.environ works, making it very reasonable that locals
and globals work the same (indeed, explaining that locals *doesn't* do
this at the moment is why (IMO) people consider locals() as "a bit
weird").

> - And then there's the "spooky" action-at-a-distance that currently
> happens at function scope, where calling locals() has the side-effect
> of mutating the return value from previous calls to locals(), and the
> objects returned from locals may or may not spontaneously mutate
> themselves depending on whether some other code registered a trace
> function. This is traditional, but extremely surprising if you aren't
> deeply familiar with internals of CPython's implementation.

Yep, this is the one that most non-experts commenting on this thread
(including me) seem to have been surprised by.

> Of the four designs:
>
> [PEP] and [PEP-minus-tracing] both have "spooky" action-at-a-distance
> (worse in [PEP]), but they don't have "justified"
> action-at-a-distance.
>
> [proxy] adds "justified" action-at-a-distance, and removes "spooky"
> action at a distance.
>
> [snapshot] gets rid of both kinds of action-at-a-distance (at least in
> function scope).

This summary leaves me very strongly feeling that I prefer [proxy]
first, then [snapshot], with the two [PEP] variants a distant third
and fourth.

Of course, all of this is only if we have decided to formalise the
semantics and change CPython to conform. I've never personally been
affected by any of the edge cases with locals(), so on a purely
personal basis, I'm equally happy with "do nothing" :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nathaniel Smith]

On Mon, May 27, 2019 at 9:18 PM Guido van Rossum  wrote:
>
> Note that the weird, Action At A Distance behavior is also visible for 
> locals() called at module scope (since there, locals() is globals(), which 
> returns the actual dict that's the module's __dict__, i.e. the Source Of 
> Truth. So I think it's unavoidable in general, and we would do wise not to 
> try and "fix" it just for function locals. (And I certainly don't want to 
> mess with globals().)

I think it's worth distinguishing between two different types of weird
Action At A Distance here:

- There's the "justified" action-at-a-distance that currently happens
at module scope, where locals().__setitem__ affects variable lookup,
and variable mutation affects locals().__getitem__. This can produce
surprising results if you pass locals() into something that's
expecting a regular dict, but it's also arguably the point of an
environment introspection API, and like you say, it's unavoidable and
expected at module scope and when using globals().

- And then there's the "spooky" action-at-a-distance that currently
happens at function scope, where calling locals() has the side-effect
of mutating the return value from previous calls to locals(), and the
objects returned from locals may or may not spontaneously mutate
themselves depending on whether some other code registered a trace
function. This is traditional, but extremely surprising if you aren't
deeply familiar with internals of CPython's implementation.

Of the four designs:

[PEP] and [PEP-minus-tracing] both have "spooky" action-at-a-distance
(worse in [PEP]), but they don't have "justified"
action-at-a-distance.

[proxy] adds "justified" action-at-a-distance, and removes "spooky"
action at a distance.

[snapshot] gets rid of both kinds of action-at-a-distance (at least in
function scope).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Guido van Rossum]

Note that the weird, Action At A Distance behavior is also visible for
locals() called at module scope (since there, locals() is globals(), which
returns the actual dict that's the module's __dict__, i.e. the Source Of
Truth. So I think it's unavoidable in general, and we would do wise not to
try and "fix" it just for function locals. (And I certainly don't want to
mess with globals().)

This is another case for the proposed [proxy] semantics, assuming we can
get over our worry about backwards incompatibility.

On Mon, May 27, 2019 at 7:31 PM Steven D'Aprano  wrote:

> On Mon, May 27, 2019 at 08:15:01AM -0700, Nathaniel Smith wrote:
> [...]
> > I'm not as sure about the locals() parts of the proposal. It might be
> > fine, but there are some complex trade-offs here that I'm still trying
> > to wrap my head around. The rest of this document is me thinking out
> > loud to try to clarify these issues.
>
> Wow. Thanks for the detail on this, I think the PEP should link to this
> thread, you've done some great work here.
>
>
> [...]
> > In function scopes, things are more complicated. The *local
> > environment* is conceptually well-defined, and includes:
> > - local variables (current source of truth: "fast locals" array)
> > - closed-over variables (current source of truth: cell objects)
>
> I don't think closed-over variables are *local* variables. They're
> "nonlocal", and you need a special keyword to write to them.
>
>
> > - any arbitrary key/values written to frame.f_locals that don't
> > correspond to local or closed-over variables, e.g. you can do
> > frame.f_locals[object()] = 10, and then later read it out again.
>
> Today I learned something new.
>
>
> > However, the mapping returned by locals() does not directly reflect
> > this local environment. Instead, each function frame has a dict
> > associated with it. locals() returns this dict. The dict always holds
> > any non-local/non-closed-over variables, and also, in certain
> > circumstances, we write a snapshot of local and closed-over variables
> > back into the dict.
>
> I'm going to try to make a case for your "snapshot" scenario.
>
> The locals dict inside a function is rather weird:
>
> - unlike in the global or class scope, writing to the dict does not
>   update the variables
>
> - writing to it is discouraged, but its not a read-only proxy
>
> - it seems to be a snapshot of the state of variables when you
>   called locals():
>
>   # inside a function
>   x = 1
>   d = locals()
>   x = 2
>   assert d['x'] == 1
>
> - but it's not a proper, static, snapshot, because sometimes it
>   will mutate without you touching it.
>
> That last point is Action At A Distance, and while it is explicable
> ("there's only one locals dict, and calling locals() updates it") its
> also rather unintuitive and surprising and violates the Principle Of
> Least Surprise.
>
> [Usual disclaimers about "surprising to whom?" applies.]
>
> Unless I missed something, it doesn't seem that any of the code you
> (Nathan) analysed is making use of this AAAD behaviour, at least not
> deliberately. At least one of the examples took steps to avoid it by
> making an explicit copy after calling locals(), but missed one leaving
> that function possibly buggy.
>
> Given how weirdly the locals dict behaves, and how tricky it is to
> explain all the corner cases, I'm going to +1 your "snapshot" idea:
>
> - we keep the current behaviour for locals() in the global and class
>   scopes;
>
> - we keep the PEP's behaviour for writebacks when locals() or exec()
>   (and eval with walrus operator) are called, for the frame dict;
>
> - but we change locals() to return a copy of that dict, rather than
>   the dict itself.
>
> (I think I've got the details right... please correct me if I've
> misunderstood anything.)
>
> Being a backwards-incompatible change, that means that folks who were
> relying on that automagical refresh of the snapshot will need to change
> their code to explicitly refresh:
>
> # update in place:
> d.update(locals())
>
> # or get a new snapshot
> d = locals()
>
>
> Or they explicitly grab a reference to the frame dict instead of
> calling locals(). Either way is likely to be less surprising than the
> status quo and less likely to lead to accidental, unexpected updates of
> the local dictionary without your knowledge.
>
>
> [...]
> > Of course, many of these edge cases are pretty obscure, so it's not
> > clear how much they matter. But I think we can at least agree that
> > this isn't the one obvious way to do it :-).
>
> Indeed. I thought I was doing well to know that writing to locals()
> inside a function didn't necessarily update the variable, but I had no
> idea of the levels of complexity actually involved!
>
>
> > I can think of a lot of criteria that all-else-being-equal we would
> > like Python to meet. (Of course, in practice they conflict.)
> >
> > Consistency across APIs: it's surprising if locals() and
> > frame.f_locals 

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Glenn Linderman]

On 5/27/2019 7:28 PM, Steven D'Aprano wrote:

On the other hand, locals() currently returns a dict everywhere. It
might be surprising for it to start returning a proxy object inside
functions instead of a dict.
I thought the proxy object sounded more useful... how different is it in 
use from a dict? "proxy" sounds like it should quack like a dict, as a 
general term, but maybe a more specific "proxy" is meant here, that 
doesn't quite quack like a dict?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Steven D'Aprano]

On Mon, May 27, 2019 at 08:15:01AM -0700, Nathaniel Smith wrote:
[...]
> I'm not as sure about the locals() parts of the proposal. It might be
> fine, but there are some complex trade-offs here that I'm still trying
> to wrap my head around. The rest of this document is me thinking out
> loud to try to clarify these issues.

Wow. Thanks for the detail on this, I think the PEP should link to this 
thread, you've done some great work here.


[...]
> In function scopes, things are more complicated. The *local
> environment* is conceptually well-defined, and includes:
> - local variables (current source of truth: "fast locals" array)
> - closed-over variables (current source of truth: cell objects)

I don't think closed-over variables are *local* variables. They're 
"nonlocal", and you need a special keyword to write to them.


> - any arbitrary key/values written to frame.f_locals that don't
> correspond to local or closed-over variables, e.g. you can do
> frame.f_locals[object()] = 10, and then later read it out again.

Today I learned something new.


> However, the mapping returned by locals() does not directly reflect
> this local environment. Instead, each function frame has a dict
> associated with it. locals() returns this dict. The dict always holds
> any non-local/non-closed-over variables, and also, in certain
> circumstances, we write a snapshot of local and closed-over variables
> back into the dict.

I'm going to try to make a case for your "snapshot" scenario.

The locals dict inside a function is rather weird:

- unlike in the global or class scope, writing to the dict does not 
  update the variables

- writing to it is discouraged, but its not a read-only proxy

- it seems to be a snapshot of the state of variables when you 
  called locals():

  # inside a function
  x = 1
  d = locals()
  x = 2
  assert d['x'] == 1

- but it's not a proper, static, snapshot, because sometimes it
  will mutate without you touching it.

That last point is Action At A Distance, and while it is explicable 
("there's only one locals dict, and calling locals() updates it") its 
also rather unintuitive and surprising and violates the Principle Of 
Least Surprise.

[Usual disclaimers about "surprising to whom?" applies.]

Unless I missed something, it doesn't seem that any of the code you 
(Nathan) analysed is making use of this AAAD behaviour, at least not 
deliberately. At least one of the examples took steps to avoid it by 
making an explicit copy after calling locals(), but missed one leaving 
that function possibly buggy.

Given how weirdly the locals dict behaves, and how tricky it is to 
explain all the corner cases, I'm going to +1 your "snapshot" idea: 

- we keep the current behaviour for locals() in the global and class 
  scopes;

- we keep the PEP's behaviour for writebacks when locals() or exec()
  (and eval with walrus operator) are called, for the frame dict;

- but we change locals() to return a copy of that dict, rather than
  the dict itself.

(I think I've got the details right... please correct me if I've 
misunderstood anything.)

Being a backwards-incompatible change, that means that folks who were 
relying on that automagical refresh of the snapshot will need to change 
their code to explicitly refresh:

# update in place:
d.update(locals())

# or get a new snapshot
d = locals()


Or they explicitly grab a reference to the frame dict instead of 
calling locals(). Either way is likely to be less surprising than the 
status quo and less likely to lead to accidental, unexpected updates of 
the local dictionary without your knowledge.


[...]
> Of course, many of these edge cases are pretty obscure, so it's not
> clear how much they matter. But I think we can at least agree that
> this isn't the one obvious way to do it :-).

Indeed. I thought I was doing well to know that writing to locals() 
inside a function didn't necessarily update the variable, but I had no 
idea of the levels of complexity actually involved!


> I can think of a lot of criteria that all-else-being-equal we would
> like Python to meet. (Of course, in practice they conflict.)
> 
> Consistency across APIs: it's surprising if locals() and
> frame.f_locals do different things. This argues for [proxy].

I don't think it is that surprising, since frame.f_locals is kinda 
obscure (the average Python coder wouldn't know a frame if one fell 
on them) and locals() has been documented as weird for decades.

In any case, at least "its a copy of ..." is simple and understandable.


> Consistency across contexts: it's surprising if locals() has acts
> differently in module/class scope versus function scope. This argues
> for [proxy].

True as far as it goes, but its also true that for the longest time, in 
most implementations, locals() has acted differently. So no change there.

On the other hand, locals() currently returns a dict everywhere. It 
might be surprising for it to start returning a proxy 

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Guido van Rossum]

OK, I apologize for not catching on to the changed semantics of f_locals
(which in the proposal is always a proxy for the fast locals and the
cells). I don't know if I just skimmed that part of the PEP or that it
needs calling out more.

I'm assuming that there are backwards compatibility concerns, and the PEP
is worried that more code will break if one of the simpler options is
chosen. In particular I guess that the choice of returning the same object
is meant to make locals() in a function frame more similar to locals() in a
module frame. And the choice of returning a plain dict rather than a proxy
is meant to make life easier for code that assumes it's getting a plain
dict.

Other than that I agree that returning a proxy (i.e. just a reference to
f_locals) seems to be the most attractive option...

On Mon, May 27, 2019 at 9:41 AM Nathaniel Smith  wrote:

> On Mon, May 27, 2019 at 9:16 AM Guido van Rossum  wrote:
> >
> > I re-ran your examples and found that some of them fail.
> >
> > On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith  wrote:
> [...]
> >> The interaction between f_locals and and locals() is also subtle:
> >>
> >>   def f():
> >>   a = 1
> >>   loc = locals()
> >>   assert "loc" not in loc
> >>   # Regular variable updates don't affect 'loc'
> >>   a = 2
> >>   assert loc["a"] == 1
> >>   # But debugging updates do:
> >>   sys._getframe().f_locals["a"] = 3
> >>   assert a == 3
> >
> >
> > That assert fails; `a` is still 2 here for me.
>
> I think you're running on current Python, and I'm talking about the
> semantics in the current PEP 558 draft, which redefines f_locals so
> that the assert passes. Nick has a branch here if you want to try it:
> https://github.com/python/cpython/pull/3640
>
> (Though I admit I was lazy, and haven't tried running my examples at
> all -- they're just based on the text.)
>
> >>
> >>   assert loc["a"] == 3
> >>   # But it's not a full writeback
> >>   assert "loc" not in loc
> >>   # Mutating 'loc' doesn't affect f_locals:
> >>   loc["a"] = 1
> >>   assert sys._getframe().f_locals["a"] == 1
> >>   # Except when it does:
> >>   loc["b"] = 3
> >>   assert sys._getframe().f_locals["b"] == 3
> >
> >
> > All of this can be explained by realizing `loc is
> sys._getframe().f_locals`. IOW locals() always returns the dict in f_locals.
>
> That's not true in the PEP version of things. locals() and
> frame.f_locals become radically different. locals() is still a dict
> stored in the frame object, but f_locals is a magic proxy object that
> reads/writes to the fast locals array directly.
>
> >>
> >> Again, the results here are totally different if a Python-level
> >> tracing/profiling function is installed.
> >>
> >> And you can also hit these subtleties via 'exec' and 'eval':
> >>
> >>   def f():
> >>   a = 1
> >>   loc = locals()
> >>   assert "loc" not in loc
> >>   # exec() triggers writeback, and then mutates the locals dict
> >>   exec("a = 2; b = 3")
> >>   # So now the current environment has been reflected into 'loc'
> >>   assert "loc" in loc
> >>   # Also loc["a"] has been changed to reflect the exec'ed
> assignments
> >>   assert loc["a"] == 2
> >>   # But if we look at the actual environment, directly or via
> >>   # f_locals, we can see that 'a' has not changed:
> >>   assert a == 1
> >>   assert sys._getframe().f_locals["a"] == 1
> >>   # loc["b"] changed as well:
> >>   assert loc["b"] == 3
> >>   # And this *does* show up in f_locals:
> >>   assert sys._getframe().f_locals["b"] == 3
> >
> >
> > This works indeed. My understanding is that the bytecode interpreter,
> when accessing the value of a local variable, ignores f_locals and always
> uses the "fast" array. But exec() and eval() don't use fast locals, their
> code is always compiled as if it appears in a module-level scope.
> >
> > While the interpreter is running and no debugger is active, in a
> function scope f_locals is not used at all, the interpreter only interacts
> with the fast array and the cells. It is initialized by the first locals()
> call for a function scope, and locals() copies the fast array and the cells
> into it. Subsequent calls in the same function scope keep the same value
> for f_locals and re-copy fast and cells into it. This also clears out
> deleted local variables and emptied cells, but leaves "strange" keys (like
> "b" in the examples) unchanged.
> >
> > The truly weird case happen when Python-level tracers are present, then
> the contents of f_locals is written back to the fast array and cells at
> certain points. This is intended for use by pdb (am I the only user of pdb
> left in the world?), so one can step through a function and mutate local
> variables. I find this essential in some cases.
>
> Right, the original goal for the PEP was to remove the "truly weird
> case" but keep pdb working
>
> >>
> >> Of course, many of these 

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nathaniel Smith]

On Mon, May 27, 2019 at 9:16 AM Guido van Rossum  wrote:
>
> I re-ran your examples and found that some of them fail.
>
> On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith  wrote:
[...]
>> The interaction between f_locals and and locals() is also subtle:
>>
>>   def f():
>>   a = 1
>>   loc = locals()
>>   assert "loc" not in loc
>>   # Regular variable updates don't affect 'loc'
>>   a = 2
>>   assert loc["a"] == 1
>>   # But debugging updates do:
>>   sys._getframe().f_locals["a"] = 3
>>   assert a == 3
>
>
> That assert fails; `a` is still 2 here for me.

I think you're running on current Python, and I'm talking about the
semantics in the current PEP 558 draft, which redefines f_locals so
that the assert passes. Nick has a branch here if you want to try it:
https://github.com/python/cpython/pull/3640

(Though I admit I was lazy, and haven't tried running my examples at
all -- they're just based on the text.)

>>
>>   assert loc["a"] == 3
>>   # But it's not a full writeback
>>   assert "loc" not in loc
>>   # Mutating 'loc' doesn't affect f_locals:
>>   loc["a"] = 1
>>   assert sys._getframe().f_locals["a"] == 1
>>   # Except when it does:
>>   loc["b"] = 3
>>   assert sys._getframe().f_locals["b"] == 3
>
>
> All of this can be explained by realizing `loc is sys._getframe().f_locals`. 
> IOW locals() always returns the dict in f_locals.

That's not true in the PEP version of things. locals() and
frame.f_locals become radically different. locals() is still a dict
stored in the frame object, but f_locals is a magic proxy object that
reads/writes to the fast locals array directly.

>>
>> Again, the results here are totally different if a Python-level
>> tracing/profiling function is installed.
>>
>> And you can also hit these subtleties via 'exec' and 'eval':
>>
>>   def f():
>>   a = 1
>>   loc = locals()
>>   assert "loc" not in loc
>>   # exec() triggers writeback, and then mutates the locals dict
>>   exec("a = 2; b = 3")
>>   # So now the current environment has been reflected into 'loc'
>>   assert "loc" in loc
>>   # Also loc["a"] has been changed to reflect the exec'ed assignments
>>   assert loc["a"] == 2
>>   # But if we look at the actual environment, directly or via
>>   # f_locals, we can see that 'a' has not changed:
>>   assert a == 1
>>   assert sys._getframe().f_locals["a"] == 1
>>   # loc["b"] changed as well:
>>   assert loc["b"] == 3
>>   # And this *does* show up in f_locals:
>>   assert sys._getframe().f_locals["b"] == 3
>
>
> This works indeed. My understanding is that the bytecode interpreter, when 
> accessing the value of a local variable, ignores f_locals and always uses the 
> "fast" array. But exec() and eval() don't use fast locals, their code is 
> always compiled as if it appears in a module-level scope.
>
> While the interpreter is running and no debugger is active, in a function 
> scope f_locals is not used at all, the interpreter only interacts with the 
> fast array and the cells. It is initialized by the first locals() call for a 
> function scope, and locals() copies the fast array and the cells into it. 
> Subsequent calls in the same function scope keep the same value for f_locals 
> and re-copy fast and cells into it. This also clears out deleted local 
> variables and emptied cells, but leaves "strange" keys (like "b" in the 
> examples) unchanged.
>
> The truly weird case happen when Python-level tracers are present, then the 
> contents of f_locals is written back to the fast array and cells at certain 
> points. This is intended for use by pdb (am I the only user of pdb left in 
> the world?), so one can step through a function and mutate local variables. I 
> find this essential in some cases.

Right, the original goal for the PEP was to remove the "truly weird
case" but keep pdb working

>>
>> Of course, many of these edge cases are pretty obscure, so it's not
>> clear how much they matter. But I think we can at least agree that
>> this isn't the one obvious way to do it :-).
>>
>>
>> # What's the landscape of possible semantics?
>>
>> I did some brainstorming, and came up with 4 sets of semantics that
>> seem plausible enough to at least consider:
>>
>> - [PEP]: the semantics in the current PEP draft.
>
>
> To be absolutely clear this copies the fast array and cells to f_locals when 
> locals() is called, but never copies back, except when Python-level 
> tracing/profiling is on.

In the PEP draft, it never copies back at all, under any circumstance.

>>
>> - [PEP-minus-tracing]: same as [PEP], except dropping the writeback on
>> Python-level trace/profile events.
>
>
> But this still copies the fast array and cells to f_locals when a Python 
> trace function is called, right? It just doesn't write back.

No, when I say "writeback" in this email I always mean
PyFrame_FastToLocals. The PEP removes PyFrame_LocalsToFast 

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Guido van Rossum]

I re-ran your examples and found that some of them fail.

On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith  wrote:

> First, I want to say: I'm very happy with PEP 558's changes to
> f_locals. It solves the weird threading bugs, and exposes the
> fundamental operations you need for debugging in a simple and clean
> way, while leaving a lot of implementation flexibility for future
> Python VMs. It's a huge improvement over what we had before.
>
> I'm not as sure about the locals() parts of the proposal. It might be
> fine, but there are some complex trade-offs here that I'm still trying
> to wrap my head around. The rest of this document is me thinking out
> loud to try to clarify these issues.
>
>
> # What are we trying to solve?
>
> There are two major questions, which are somewhat distinct:
> - What should the behavior of locals() be in CPython?
> - How much of that should be part of the language definition, vs
> CPython implementation details?
>
> The status quo is that for locals() inside function scope, the
> behavior is quite complex and subtle, and it's entirely implementation
> defined. In the current PEP draft, there are some small changes to the
> semantics, and also it promotes them becoming part of the official
> language semantics.
>
> I think the first question, about semantics, is the more important
> one. If we're promoting them to the language definition, the main
> effect is just to make it more important we get the semantics right.
>
>
> # What are the PEP's proposed semantics for locals()?
>
> They're kinda subtle. [Nick: please double-check this section, both
> for errors and because I think it includes some edge cases that the
> PEP currently doesn't mention.]
>
> For module/class scopes, locals() has always returned a mapping object
> which acts as a "source of truth" for the actual local environment –
> mutating the environment directly changes the mapping object, and
> vice-versa. That's not going to change.
>
> In function scopes, things are more complicated. The *local
> environment* is conceptually well-defined, and includes:
> - local variables (current source of truth: "fast locals" array)
> - closed-over variables (current source of truth: cell objects)
> - any arbitrary key/values written to frame.f_locals that don't
> correspond to local or closed-over variables, e.g. you can do
> frame.f_locals[object()] = 10, and then later read it out again.
>
> However, the mapping returned by locals() does not directly reflect
> this local environment. Instead, each function frame has a dict
> associated with it. locals() returns this dict. The dict always holds
> any non-local/non-closed-over variables, and also, in certain
> circumstances, we write a snapshot of local and closed-over variables
> back into the dict.
>
> Specifically, we write back:
>
> - Whenever locals() is called
> - Whenever exec() or eval() is called without passing an explicit
> locals argument
> - After every trace/profile event, if a Python-level tracing/profiling
> function is registered.
>
> (Note: in CPython, the use of Python-level tracing/profiling functions
> is extremely rare. It's more common in alternative implementations
> like PyPy. For example, the coverage package uses a C-level tracing
> function on CPython, which does not trigger locals updates, but on
> PyPy it uses a Python-level tracing function, which does trigger
> updates.)
>
> In addition, the PEP doesn't say, but I think that any writes to
> f_locals immediately update both the environment and the locals dict.
>
> These semantics have some surprising consequences. Most obviously, in
> function scope (unlike other scopes), mutating locals() does not
> affect the actual local environment:
>
>   def f():
>   a = 1
>   locals()["a"] = 2
>   assert a == 1
>
> The writeback rules can also produce surprising results:
>
>   def f():
>   loc1 = locals()
>   # Since it's a snapshot created at the time of the call
>   # to locals(), it doesn't contain 'loc1':
>   assert "loc1" not in loc1
>   loc2 = locals()
>   # Now loc1 has changed:
>   assert "loc1" in loc1
>
> However, the results here are totally different if a Python-level
> tracing/profiling function is installed – in particular, the first
> assertion fails.
>
> The interaction between f_locals and and locals() is also subtle:
>
>   def f():
>   a = 1
>   loc = locals()
>   assert "loc" not in loc
>   # Regular variable updates don't affect 'loc'
>   a = 2
>   assert loc["a"] == 1
>   # But debugging updates do:
>   sys._getframe().f_locals["a"] = 3
>   assert a == 3
>

That assert fails; `a` is still 2 here for me.


>   assert loc["a"] == 3
>   # But it's not a full writeback
>   assert "loc" not in loc
>   # Mutating 'loc' doesn't affect f_locals:
>   loc["a"] = 1
>   assert sys._getframe().f_locals["a"] == 1
>   # Except when it does:
>   loc["b"] = 3
>   assert 

[Python-Dev] [PEP 558] thinking through locals() semantics

[Nathaniel Smith]

First, I want to say: I'm very happy with PEP 558's changes to
f_locals. It solves the weird threading bugs, and exposes the
fundamental operations you need for debugging in a simple and clean
way, while leaving a lot of implementation flexibility for future
Python VMs. It's a huge improvement over what we had before.

I'm not as sure about the locals() parts of the proposal. It might be
fine, but there are some complex trade-offs here that I'm still trying
to wrap my head around. The rest of this document is me thinking out
loud to try to clarify these issues.


# What are we trying to solve?

There are two major questions, which are somewhat distinct:
- What should the behavior of locals() be in CPython?
- How much of that should be part of the language definition, vs
CPython implementation details?

The status quo is that for locals() inside function scope, the
behavior is quite complex and subtle, and it's entirely implementation
defined. In the current PEP draft, there are some small changes to the
semantics, and also it promotes them becoming part of the official
language semantics.

I think the first question, about semantics, is the more important
one. If we're promoting them to the language definition, the main
effect is just to make it more important we get the semantics right.


# What are the PEP's proposed semantics for locals()?

They're kinda subtle. [Nick: please double-check this section, both
for errors and because I think it includes some edge cases that the
PEP currently doesn't mention.]

For module/class scopes, locals() has always returned a mapping object
which acts as a "source of truth" for the actual local environment –
mutating the environment directly changes the mapping object, and
vice-versa. That's not going to change.

In function scopes, things are more complicated. The *local
environment* is conceptually well-defined, and includes:
- local variables (current source of truth: "fast locals" array)
- closed-over variables (current source of truth: cell objects)
- any arbitrary key/values written to frame.f_locals that don't
correspond to local or closed-over variables, e.g. you can do
frame.f_locals[object()] = 10, and then later read it out again.

However, the mapping returned by locals() does not directly reflect
this local environment. Instead, each function frame has a dict
associated with it. locals() returns this dict. The dict always holds
any non-local/non-closed-over variables, and also, in certain
circumstances, we write a snapshot of local and closed-over variables
back into the dict.

Specifically, we write back:

- Whenever locals() is called
- Whenever exec() or eval() is called without passing an explicit
locals argument
- After every trace/profile event, if a Python-level tracing/profiling
function is registered.

(Note: in CPython, the use of Python-level tracing/profiling functions
is extremely rare. It's more common in alternative implementations
like PyPy. For example, the coverage package uses a C-level tracing
function on CPython, which does not trigger locals updates, but on
PyPy it uses a Python-level tracing function, which does trigger
updates.)

In addition, the PEP doesn't say, but I think that any writes to
f_locals immediately update both the environment and the locals dict.

These semantics have some surprising consequences. Most obviously, in
function scope (unlike other scopes), mutating locals() does not
affect the actual local environment:

  def f():
  a = 1
  locals()["a"] = 2
  assert a == 1

The writeback rules can also produce surprising results:

  def f():
  loc1 = locals()
  # Since it's a snapshot created at the time of the call
  # to locals(), it doesn't contain 'loc1':
  assert "loc1" not in loc1
  loc2 = locals()
  # Now loc1 has changed:
  assert "loc1" in loc1

However, the results here are totally different if a Python-level
tracing/profiling function is installed – in particular, the first
assertion fails.

The interaction between f_locals and and locals() is also subtle:

  def f():
  a = 1
  loc = locals()
  assert "loc" not in loc
  # Regular variable updates don't affect 'loc'
  a = 2
  assert loc["a"] == 1
  # But debugging updates do:
  sys._getframe().f_locals["a"] = 3
  assert a == 3
  assert loc["a"] == 3
  # But it's not a full writeback
  assert "loc" not in loc
  # Mutating 'loc' doesn't affect f_locals:
  loc["a"] = 1
  assert sys._getframe().f_locals["a"] == 1
  # Except when it does:
  loc["b"] = 3
  assert sys._getframe().f_locals["b"] == 3

Again, the results here are totally different if a Python-level
tracing/profiling function is installed.

And you can also hit these subtleties via 'exec' and 'eval':

  def f():
  a = 1
  loc = locals()
  assert "loc" not in loc
  # exec() triggers writeback, and then mutates the locals dict
  exec("a = 2; b = 3")
  # So now