[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Thu, Aug 26, 2021 at 5:29 PM Nick Coghlan  wrote:

> [snip]
> For the main topic of PEP 667 (reducing the cache consistency question to
> purely a matter of PyEval_GetLocals() compatibility), I think I can see a
> way to make the extra code complexity of the 5 new custom accessory types
> (iterator, reversed iterator, keys set, values multi set, items set)
> worthwhile to my mind: write the latter 3 in terms of the first two and the
> generic mapping API, and expose them for use in other custom mapping
> implementations (either directly in the types module, or as optional C
> accelerators for the collections module). (I wouldn't make exposing them
> part of the PEP, I'd just aim to write them so only the forward and reverse
> iterators were specific to proxy objects)
>

Yes, that makes sense. I wouldn't sweat it though.


> With that done, popitem() is trivial to rewrite to depend on the iterator
> code instead of the cache.
>
> That would leave len() and value comparison as the only proxy operations
> that don't adhere to the expected algorithmic complexity of mapping
> objects, and writing up the comparison with PEP 667 finally convinced me
> that those are quirks that API users could manage more easily than 558's
> current lazy caching semantics.
>

That's what we've been saying. :-)

I'd still keep the full value cache under the hood, though, as I don't see
> enough benefit in getting rid of it when the cost is an unnecessary API
> compatibility break.
>

That's for PyEval_GetLocals(), right? Mark updated PEP 667 to also keep a
reference to whatever it returned on the frame. So then I think the two
PEPs have converged.


> However, the PEP 667 write up and your own points *have* persuaded me that
> the extra C code needed to offer assuredly consistent views through all the
> proxy mapping methods is worthwhile.
>

Right.

That will then leave the proposed C APIs as the only differences between
> the PEPs - the Python level behaviour will be aligned with Mark's proposal.
>

Yeah, I presume that the *new* APIs proposed are just a matter of a little
bikeshedding, nothing major.


> Cheers,
> Nick.
>

Thanks for your flexibility!


> P.S. I don't want to rely on Python code anywhere in the fast locals proxy
> implementation, as that could cause weird interactions if a trace hook is
> enabled during the initial start up of the interpreter and tries to trace
> the proxy implementation code.
>

Of course. But it would still be interesting to have pseudo-code in your
PEP showing the semantics you intend to implement -- that way we can
compare 558 and 667 more easily.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P37GSW6BHHZGRQNZECMHJEWG3WDEGFDR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Fri, 27 Aug 2021, 2:33 am Guido van Rossum,  wrote:

> On Thu, Aug 26, 2021 at 2:31 AM Nick Coghlan  wrote:
>
>> On Thu, 26 Aug 2021 at 18:29, Nick Coghlan  wrote:
>> [Guido wrote]
>> > > PS. The mapping from varname to position should be on the code
>> object, not on the frame. This is how Mark does it (though his
>> implementation would need to be extended to take cells into account).
>> >
>> > It's taking cells into account that forces the lookup mapping to be on
>> > the frame: different executions of the same code object may reference
>> > different cell objects.
>>
>> Technically, if you design the fast refs mapping so that even cell
>> references have to do an indirection through the fast locals array,
>> then you can put the name-to-fast-locals-offset mapping on the code
>> object. Mine doesn't do that though, it references the cells directly
>> instead.
>>
>> I don't think it makes much difference in practice though, and I don't
>> like the idea of storing a lazily initialised cache on nominally
>> immutable code objects.
>>
>
> Only the part visible to the user is immutable. The specializing
> interpreter stores all sorts of internal mutable data on there -- and at
> least since 3.9 we've had the inline cache stored there. So as long as this
> mapping is invisible to the user, please let's put it on the code object --
> we have a lot more frame objects than code objects, so saving a pointer on
> the frame object and adding it to the code object is advantageous. The cost
> of the extra indirection is irrelevant, this is always going to be a slow
> interface meant for occasional use in a debugger.
>

OK, that makes sense - I'll add a todo note to the implementation PR.

For the main topic of PEP 667 (reducing the cache consistency question to
purely a matter of PyEval_GetLocals() compatibility), I think I can see a
way to make the extra code complexity of the 5 new custom accessory types
(iterator, reversed iterator, keys set, values multi set, items set)
worthwhile to my mind: write the latter 3 in terms of the first two and the
generic mapping API, and expose them for use in other custom mapping
implementations (either directly in the types module, or as optional C
accelerators for the collections module). (I wouldn't make exposing them
part of the PEP, I'd just aim to write them so only the forward and reverse
iterators were specific to proxy objects)

With that done, popitem() is trivial to rewrite to depend on the iterator
code instead of the cache.

That would leave len() and value comparison as the only proxy operations
that don't adhere to the expected algorithmic complexity of mapping
objects, and writing up the comparison with PEP 667 finally convinced me
that those are quirks that API users could manage more easily than 558's
current lazy caching semantics.

I'd still keep the full value cache under the hood, though, as I don't see
enough benefit in getting rid of it when the cost is an unnecessary API
compatibility break.

However, the PEP 667 write up and your own points *have* persuaded me that
the extra C code needed to offer assuredly consistent views through all the
proxy mapping methods is worthwhile.

That will then leave the proposed C APIs as the only differences between
the PEPs - the Python level behaviour will be aligned with Mark's proposal.

Cheers,
Nick.

P.S. I don't want to rely on Python code anywhere in the fast locals proxy
implementation, as that could cause weird interactions if a trace hook is
enabled during the initial start up of the interpreter and tries to trace
the proxy implementation code.



> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QUFA4YUZJILA5OUHKYR6BYTOQJ4ZOGFG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Thu, Aug 26, 2021 at 2:31 AM Nick Coghlan  wrote:

> On Thu, 26 Aug 2021 at 18:29, Nick Coghlan  wrote:
> [Guido wrote]
> > > PS. The mapping from varname to position should be on the code object,
> not on the frame. This is how Mark does it (though his implementation would
> need to be extended to take cells into account).
> >
> > It's taking cells into account that forces the lookup mapping to be on
> > the frame: different executions of the same code object may reference
> > different cell objects.
>
> Technically, if you design the fast refs mapping so that even cell
> references have to do an indirection through the fast locals array,
> then you can put the name-to-fast-locals-offset mapping on the code
> object. Mine doesn't do that though, it references the cells directly
> instead.
>
> I don't think it makes much difference in practice though, and I don't
> like the idea of storing a lazily initialised cache on nominally
> immutable code objects.
>

Only the part visible to the user is immutable. The specializing
interpreter stores all sorts of internal mutable data on there -- and at
least since 3.9 we've had the inline cache stored there. So as long as this
mapping is invisible to the user, please let's put it on the code object --
we have a lot more frame objects than code objects, so saving a pointer on
the frame object and adding it to the code object is advantageous. The cost
of the extra indirection is irrelevant, this is always going to be a slow
interface meant for occasional use in a debugger.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CXGSRPRQMERDHE6KRDQJVTR46ZV5ED4P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Thu, Aug 26, 2021 at 2:40 AM Nick Coghlan  wrote:

> [snip]
> An unrelated issue that came up while working on that update is
> something that affects both PEPs: calling "proxy.clear()" is *super
> weird* if we make it work the same way as PyEval_LocalsToFast() works
> today. Specifically, it can reach out and clear cells in outer frames,
> including the __class__ cell used by zero-arg super(). I can't see
> anyone being super upset if we decide to change that and say that
> proxy.clear() leaves free variables alone, and only clears local
> variables and cells owned by that particular frame.
>

That would be another weird corner case. The only exception I would make
would be for __class__, which is only a cell for implementation
convenience. But why would anyone write proxy.clear()? That would be like
deleting all local variables -- what would be the use case for that? I
guess to start over with a computation. But there are better ways to do
that. So let's not worry too much about preventing the user from shooting
themselves in the foot -- surely at the global level, "globals().clear()"
will do weird shit too. :-)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OTTSADTLKEC4LTYRULLY7P46QK36IMVG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Thu, Aug 26, 2021 at 1:29 AM Nick Coghlan  wrote:

> On Mon, 23 Aug 2021 at 13:07, Guido van Rossum  wrote:
> > But... I also care about backwards compatibility, and I have a crazy
> idea for making PyEval_GetLocals() work in a useful manner without
> compromising the behavior of the f_locals proxy:
> >
> > [snip]
>
> I don't think any of this is crazy, as it's how the PEP 558 reference
> implementation already works for individual keys
>

I'm not sure I understand what it means to be like my idea "for individual
keys". But I'm not sure I care.

(aside from only
> having a documented deprecation of PyEval_GetLocals(), not a
> programmatic one)
>

Yeah, that's a detail. The deprecation could start out silent.


> You don't need to eliminate the cache (and hence break compatibility
> with PyEval_GetLocals()) to ensure it can never get out of sync - you
> just have to resync it every time you use it, rather than allowing the
> frame API consumer to make that call.
>

I don't think eliminating the cache part is breaking compatibility.


> The reason I don't like the proxy semantics proposed in PEP 667 is
> because it either makes the common case (analysing a frame from a
> tracing function while no application code is running) slower in order
> to avoid having to worry about cache consistency when trying to
> analyse a running frame, or else we have to write and maintain a whole
> lot more fast locals proxy specific code to implement the 5 different
> dict iteration APIs.
>

I just think it's weird to write a long PEP to clean up the semantics and
then replace it with such a complicated solution.


> > PS. The mapping from varname to position should be on the code object,
> not on the frame. This is how Mark does it (though his implementation would
> need to be extended to take cells into account).
>
> It's taking cells into account that forces the lookup mapping to be on
> the frame: different executions of the same code object may reference
> different cell objects.
>

But the frame stores the cells at known positions (after all, the index is
embedded in the bytecode). YOu just need to interpret the mapping a bit
differently.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7TNHSW6QJMDHZHFFGU5ZGIHITMUFWSP7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Tue, 24 Aug 2021 at 01:57, Guido van Rossum  wrote:
> On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:
>> Can we avoid describing the C structs in any of these PEPs?
>>
>> It confuses readers having Python attributes and "C-level attributes"
>> (C struct fields?).
>> It also restricts the implementation unnecessarily.
>>
>> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7)
>
>
> I'd be happy to. Nick's PEP still references it (and indeed it is very 
> confusing) and I took it from him. And honestly it would be nice to have a 
> specific short name for it, rather than circumscribing it with "an internal 
> dynamic snapshot stored on the frame object " :-)

https://github.com/python/peps/pull/2060/files switches over to
calling it the "frame value cache" in PEP 558, since that's
essentially what it is on optimised frames (even today).

An unrelated issue that came up while working on that update is
something that affects both PEPs: calling "proxy.clear()" is *super
weird* if we make it work the same way as PyEval_LocalsToFast() works
today. Specifically, it can reach out and clear cells in outer frames,
including the __class__ cell used by zero-arg super(). I can't see
anyone being super upset if we decide to change that and say that
proxy.clear() leaves free variables alone, and only clears local
variables and cells owned by that particular frame.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FJYJJWZBZNAZBW2UDWGSNFCVB26JINNU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Thu, 26 Aug 2021 at 18:29, Nick Coghlan  wrote:
[Guido wrote]
> > PS. The mapping from varname to position should be on the code object, not 
> > on the frame. This is how Mark does it (though his implementation would 
> > need to be extended to take cells into account).
>
> It's taking cells into account that forces the lookup mapping to be on
> the frame: different executions of the same code object may reference
> different cell objects.

Technically, if you design the fast refs mapping so that even cell
references have to do an indirection through the fast locals array,
then you can put the name-to-fast-locals-offset mapping on the code
object. Mine doesn't do that though, it references the cells directly
instead.

I don't think it makes much difference in practice though, and I don't
like the idea of storing a lazily initialised cache on nominally
immutable code objects.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A6CLBN6EUA26BAIJID7UB3KXSF3MSP5D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Mon, 23 Aug 2021 at 13:07, Guido van Rossum  wrote:
> But... I also care about backwards compatibility, and I have a crazy idea for 
> making PyEval_GetLocals() work in a useful manner without compromising the 
> behavior of the f_locals proxy:
>
> - Let's start your idea of using the C-level f_locals field to store the 
> "extra" variables.
> - The Python-level f_locals proxy looks in the actual frame "fast" locals and 
> cells, and uses the C-level f_locals field only for extras
> - However, PyEval_GetLocals() doesn't return the proxy.
> - What PyEval_GetLocals() does: it calls PyEval_FastToLocals(), which makes a 
> pass over the frame locals and adds them to the C-level f_locals field; then 
> it returns that field.
> - So the borrowed reference is owned by the frame, which is the same as 
> currently.
> - The proxy only uses the f_locals field for extra variables. If a variable 
> is deleted in the frame but exists in the f_locals field, the proxy reports 
> it as deleted. (This requires some care but can be done, since we have the 
> mapping from proper variable names to frame locals or cells.)
> - We'll still deprecate PyEval_GetLocals() and PyEval_FastToLocals(), but 
> unless the user turns the deprecation warning into an error, they will work 
> for another few releases. Eventually we'll make PyEval_GetLocals() always 
> return an error (similar to Mark's proposal), since it's in the stable ABI.
> - For PyEval_LocalsToFast() I don't care too much whether we keep it (per 
> Mark's proposal) or make it return an error (per yours).

I don't think any of this is crazy, as it's how the PEP 558 reference
implementation already works for individual keys (aside from only
having a documented deprecation of PyEval_GetLocals(), not a
programmatic one)

You don't need to eliminate the cache (and hence break compatibility
with PyEval_GetLocals()) to ensure it can never get out of sync - you
just have to resync it every time you use it, rather than allowing the
frame API consumer to make that call.

The reason I don't like the proxy semantics proposed in PEP 667 is
because it either makes the common case (analysing a frame from a
tracing function while no application code is running) slower in order
to avoid having to worry about cache consistency when trying to
analyse a running frame, or else we have to write and maintain a whole
lot more fast locals proxy specific code to implement the 5 different
dict iteration APIs.

> PS. The mapping from varname to position should be on the code object, not on 
> the frame. This is how Mark does it (though his implementation would need to 
> be extended to take cells into account).

It's taking cells into account that forces the lookup mapping to be on
the frame: different executions of the same code object may reference
different cell objects.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S66QCDWYP3B73HF7PK6S55NGNHYVC2S2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[MRAB]

On 2021-08-24 17:55, Patrick Reader wrote:

On 24/08/2021 06:27, Steven D'Aprano wrote:

Wouldn't that attempt to resolve global y, rather than local y? Unless
there is a change to the current behaviour of the compiler, I think you 
need to fool the compiler:


   if False: y = 0  # anywhere inside the function is okay

Time to add a `nonnonlocal` statement ;)


Or perhaps use "not nonlocal"? :-)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/USZ5FOQ6WF365VC7C4ZH4K4TR5AW2X2Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Patrick Reader]

On 24/08/2021 06:27, Steven D'Aprano wrote:
> Wouldn't that attempt to resolve global y, rather than local y? Unless
> there is a change to the current behaviour of the compiler, I think you 
> need to fool the compiler:
>
>if False: y = 0  # anywhere inside the function is okay
Time to add a `nonnonlocal` statement ;)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WMUK3XLP4GOR25ELCF3LUTAWT6YIPDSI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

I apologize. This does indeed not work. What does work, however, is setting
a variable from a tracing function. There is magic in the tracing machinery
that writes back the variables to the frame when the tracing function
returns. This causes the bug referenced as [1] in both PEPs, and both
propose to fix the bug by  not writing things back that way, instead
writing back whenever a key in the proxy is set. The discussion is about
subtler differences between the proposals.

—Guido

On Mon, Aug 23, 2021 at 22:19 Steven D'Aprano  wrote:

> On Sat, Aug 21, 2021 at 05:46:52PM -0700, Guido van Rossum wrote:
> > Hopefully anyone is still reading python-dev.
>
> I am :-)
>
> [...]
> > Everything here is about locals() and f_locals in *function scope*. (I
> use
> > f_locals to refer to the f_locals field of frame objects as seen from
> > Python code.) And in particular, it is about what I'll call "extra
> > variables": the current CPython feature that you can add *new* variables
> to
> > f_locals that don't exist in the frame, for example:
> >
> > def foo():
> > x = 1
> > locals()["y"] = 2  # or sys._getframe()["y"] = 2
>
> I'm confused. I don't think it currently works, at least not in the
> sense that I understand "works" to mean. Sure, you can add a new key to
> the dict, but that doesn't add a new local variable:
>
>
> >>> def spam():
> ... if False: y = 0  # Fool the compiler into treating y as a
> local.
> ... x = 0
> ... locals()['y'] = 1
> ... print(sys._getframe().f_locals)
> ... print(y)
> ...
> >>> spam()
> {'x': 0}
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 6, in spam
> UnboundLocalError: local variable 'y' referenced before assignment
>
>
> Am I missing something? The above is in 3.9, has something changed in
> 3.10 or am I just misunderstanding what you mean by "works"?
>
> Using f_locals instead of locals() doesn't change the UnboundLocalError
> in my testing.
>
> If you are not referring to new local variables, then I don't understand
> what these "extra variables" are or where they live in the current
> implementation.
>
> A recent thread on Discuss is maybe relevant:
>
>
> https://discuss.python.org/t/how-can-i-use-exec-in-functions-in-python3-6-similar-to-python2-7/10191
>
> Would either of these two proposals re-enable exec to work inside
> functions as it used to?
>
> I think Nick's PEP 558 does not, it wants to make it explicit that
> changes to locals will not be reflected in the local variables. Although
> Nick does refer to a "write back" strategy that apparently works *now*.
> That suggests:
>
> - there currently is a trick to writing new local variables in the
> function namespace;
>
> - and Nick's PEP will break it.
>
> Am I correct?
>
> Mark's PEP 667 says "However f.f_locals == f.f_locals will be True, and
> all changes to the underlying variables, by any means, will be always be
> visible" so I think that means it will allow exec to work as in
> Python2.x, at least if you pass sys._getframe().f_locals to exec instead
> of locals().
>
> Perhaps we need an informational PEP to explain how local variables
> inside functions work now :-)
>
>
>
> --
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CREZXKZWVSICQY4WHKUPXGJU67UUPQZ2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WHK32PFNLULIRQMHPFIC3SETT2O6PMKV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Steven D'Aprano]

On Fri, Aug 20, 2021 at 04:22:56PM +0100, Mark Shannon wrote:
> Hi all,
> 
> I have submitted PEP 667 as an alternative to PEP 558.
> https://www.python.org/dev/peps/pep-0667


Specification has a code snippet:

def test():
x = 1
l()['x'] = 2
l()['y'] = 4
l()['z'] = 5
y
print(locals(), x)

https://www.python.org/dev/peps/pep-0667/#id10

Wouldn't that attempt to resolve global y, rather than local y? Unless 
there is a change to the current behaviour of the compiler, I think you 
need to fool the compiler:

   if False: y = 0  # anywhere inside the function is okay


Open Issues says:

"there would be backwards compatibility issues when locals is assigned 
to a local variable or when passed to eval."

https://www.python.org/dev/peps/pep-0667/#id24

Is that eval meant to be exec? Or both eval and exec?


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DZMWW3ROSXSNWONOHAARPDHWLBLVLEFH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Steven D'Aprano]

On Sat, Aug 21, 2021 at 05:46:52PM -0700, Guido van Rossum wrote:
> Hopefully anyone is still reading python-dev.

I am :-)

[...]
> Everything here is about locals() and f_locals in *function scope*. (I use
> f_locals to refer to the f_locals field of frame objects as seen from
> Python code.) And in particular, it is about what I'll call "extra
> variables": the current CPython feature that you can add *new* variables to
> f_locals that don't exist in the frame, for example:
> 
> def foo():
> x = 1
> locals()["y"] = 2  # or sys._getframe()["y"] = 2

I'm confused. I don't think it currently works, at least not in the 
sense that I understand "works" to mean. Sure, you can add a new key to 
the dict, but that doesn't add a new local variable:


>>> def spam():
... if False: y = 0  # Fool the compiler into treating y as a local.
... x = 0
... locals()['y'] = 1
... print(sys._getframe().f_locals)
... print(y)
... 
>>> spam()
{'x': 0}
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 6, in spam
UnboundLocalError: local variable 'y' referenced before assignment


Am I missing something? The above is in 3.9, has something changed in 
3.10 or am I just misunderstanding what you mean by "works"?

Using f_locals instead of locals() doesn't change the UnboundLocalError 
in my testing.

If you are not referring to new local variables, then I don't understand 
what these "extra variables" are or where they live in the current 
implementation.

A recent thread on Discuss is maybe relevant:

https://discuss.python.org/t/how-can-i-use-exec-in-functions-in-python3-6-similar-to-python2-7/10191

Would either of these two proposals re-enable exec to work inside 
functions as it used to?

I think Nick's PEP 558 does not, it wants to make it explicit that 
changes to locals will not be reflected in the local variables. Although 
Nick does refer to a "write back" strategy that apparently works *now*. 
That suggests:

- there currently is a trick to writing new local variables in the 
function namespace;

- and Nick's PEP will break it.

Am I correct?

Mark's PEP 667 says "However f.f_locals == f.f_locals will be True, and 
all changes to the underlying variables, by any means, will be always be 
visible" so I think that means it will allow exec to work as in 
Python2.x, at least if you pass sys._getframe().f_locals to exec instead 
of locals().

Perhaps we need an informational PEP to explain how local variables 
inside functions work now :-)



-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CREZXKZWVSICQY4WHKUPXGJU67UUPQZ2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Gregory P. Smith]

Just adding a datapoint, searching our internal codebase at work, I only
found two things that I'd consider to be uses of PyEval_GetLocals() outside
of CPython itself:

https://github.com/earwig/mwparserfromhell/blob/develop/src/mwparserfromhell/parser/ctokenizer/tokenizer.c#L220
https://github.com/numba/numba/blob/master/numba/_dispatcher.cpp#L664

The bulk of uses are naturally with CPython's own ceval.c and similar.

-gps

On Mon, Aug 23, 2021 at 9:02 AM Guido van Rossum  wrote:

> On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:
>
>> Hi Guido,
>>
>> On 23/08/2021 3:53 pm, Guido van Rossum wrote:
>> > On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon > > > wrote:
>> >
>> > Hi Nick,
>> >
>> > On 22/08/2021 4:51 am, Nick Coghlan wrote:
>> >
>> >  > If Mark's claim that PyEval_GetLocals() could not be fixed was
>> > true then
>> >  > I would be more sympathetic to his proposal, but I know it isn't
>> > true,
>> >  > because it still works fine in the PEP 558 implementation (it
>> even
>> >  > immediately sees changes made via proxies, and proxies see
>> > changes to
>> >  > extra variables). The only truly unfixable public API is
>> >  > PyFrame_LocalsToFast().
>> >
>> > You are making claims that seem inconsistent with each other.
>> > Namely, you are claiming that:
>> >
>> > 1. That the result of locals() is ephemeral.
>> > 2. That PyEval_GetLocals() returns a borrowed reference.
>> >
>> > This seems impossible, as you can't return a borrowed reference to
>> > an emphemeral object. That's just a pointer to freed memory.
>> >
>> > Do `locals()` and `PyEval_GetLocals()` behave differently?
>> >
>> >
>> > That is my understanding, yes. in PEP 558 locals() returns a snapshot
>> > dict, the Python-level f_locals property returns a fresh proxy that has
>> > no state except a pointer to the frame, and PyEval_GetLocals() returns
>> a
>> > borrowed reference to the dict that's stored on the frame's C-level
>> > f_locals attribute
>>
>> Can we avoid describing the C structs in any of these PEPs?
>>
>> It confuses readers having Python attributes and "C-level attributes"
>> (C struct fields?).
>> It also restricts the implementation unnecessarily.
>>
>> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>>
>> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7
>> )
>>
>
> I'd be happy to. Nick's PEP still references it (and indeed it is very
> confusing) and I took it from him. And honestly it would be nice to have a
> specific short name for it, rather than circumscribing it with "an internal
> dynamic snapshot stored on the frame object " :-)
>
>
>> >
>> > (In my "crazy" proposal all that is the same.)
>>
>> >
>> > Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
>> >
>> >
>> > I wouldn't call it a cache -- deleting it would affect the semantics,
>> > not just the performance. But yes, it returns a reference to an object
>> > that is owned by the frame, just as it does in 3.10 and before.
>> >
>> > If that were the case, then it is a bit confusing, but could work.
>> >
>> >
>> > Yes, see my "crazy" proposal.
>> >
>> > Would PyEval_GetLocals() be defined as something like this?
>> >
>> > (add _locals_cache attribute to the frame which is initialized to
>> NULL).
>> >
>> > def PyEval_GetLocals():
>> >   frame._locals_cache attribute = locals()
>> >   return borrow(frame._locals_cache attribute)
>> >
>> >
>> > Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
>> > C-level f_locals attribute, which is consulted by the Python-level
>> > f_locals proxy -- primarily to store "extra" variables, but IIUC in
>> > Nick's latest version it is also still used to cache by that proxy.
>> > Nick's locals() just returns dict(sys._getframe().f_locals).
>>
>> The "extra" variables must be distinct from the result of locals() as
>> that includes both extras and "proper" variables.
>> If we want to cache the locals(), it needs to be distinct from the extra
>> variables.
>>
>
> I don't care that much about caching locals(), but it seems we're bound to
> cache any non-NULL result from PyEval_GetLocals(), since it returns a
> borrowed reference. So they may be different things, with different
> semantics, if we don't cache locals().
>
>
>> A debugger setting extra variables in a function that that is also
>> accessed by a C call to PyEval_GetLocals() is going to be incredibly
>> rare. Let's not worry about efficiency here.
>>
>
> Agreed.
>
> >
>> > None of this is clear (at least not to me) from PEP 558.
>> >
>> >
>> > One problem with PEP 558 is that it's got too many words, and it's
>> > lacking a section that crisply describes the semantics of the proposed
>> > implementation. I've suggested to Nick that he add a section with
>> > pseudo-code for the implementation, like you did in yours.
>> >
>> > (PS, did you read my PS 

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Mon, Aug 23, 2021 at 8:46 AM Mark Shannon  wrote:

> Hi Guido,
>
> On 23/08/2021 3:53 pm, Guido van Rossum wrote:
> > On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon  > > wrote:
> >
> > Hi Nick,
> >
> > On 22/08/2021 4:51 am, Nick Coghlan wrote:
> >
> >  > If Mark's claim that PyEval_GetLocals() could not be fixed was
> > true then
> >  > I would be more sympathetic to his proposal, but I know it isn't
> > true,
> >  > because it still works fine in the PEP 558 implementation (it even
> >  > immediately sees changes made via proxies, and proxies see
> > changes to
> >  > extra variables). The only truly unfixable public API is
> >  > PyFrame_LocalsToFast().
> >
> > You are making claims that seem inconsistent with each other.
> > Namely, you are claiming that:
> >
> > 1. That the result of locals() is ephemeral.
> > 2. That PyEval_GetLocals() returns a borrowed reference.
> >
> > This seems impossible, as you can't return a borrowed reference to
> > an emphemeral object. That's just a pointer to freed memory.
> >
> > Do `locals()` and `PyEval_GetLocals()` behave differently?
> >
> >
> > That is my understanding, yes. in PEP 558 locals() returns a snapshot
> > dict, the Python-level f_locals property returns a fresh proxy that has
> > no state except a pointer to the frame, and PyEval_GetLocals() returns a
> > borrowed reference to the dict that's stored on the frame's C-level
> > f_locals attribute
>
> Can we avoid describing the C structs in any of these PEPs?
>
> It confuses readers having Python attributes and "C-level attributes"
> (C struct fields?).
> It also restricts the implementation unnecessarily.
>
> (E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11:
>
> https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7
> )
>

I'd be happy to. Nick's PEP still references it (and indeed it is very
confusing) and I took it from him. And honestly it would be nice to have a
specific short name for it, rather than circumscribing it with "an internal
dynamic snapshot stored on the frame object " :-)


> >
> > (In my "crazy" proposal all that is the same.)
>
> >
> > Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
> >
> >
> > I wouldn't call it a cache -- deleting it would affect the semantics,
> > not just the performance. But yes, it returns a reference to an object
> > that is owned by the frame, just as it does in 3.10 and before.
> >
> > If that were the case, then it is a bit confusing, but could work.
> >
> >
> > Yes, see my "crazy" proposal.
> >
> > Would PyEval_GetLocals() be defined as something like this?
> >
> > (add _locals_cache attribute to the frame which is initialized to
> NULL).
> >
> > def PyEval_GetLocals():
> >   frame._locals_cache attribute = locals()
> >   return borrow(frame._locals_cache attribute)
> >
> >
> > Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
> > C-level f_locals attribute, which is consulted by the Python-level
> > f_locals proxy -- primarily to store "extra" variables, but IIUC in
> > Nick's latest version it is also still used to cache by that proxy.
> > Nick's locals() just returns dict(sys._getframe().f_locals).
>
> The "extra" variables must be distinct from the result of locals() as
> that includes both extras and "proper" variables.
> If we want to cache the locals(), it needs to be distinct from the extra
> variables.
>

I don't care that much about caching locals(), but it seems we're bound to
cache any non-NULL result from PyEval_GetLocals(), since it returns a
borrowed reference. So they may be different things, with different
semantics, if we don't cache locals().


> A debugger setting extra variables in a function that that is also
> accessed by a C call to PyEval_GetLocals() is going to be incredibly
> rare. Let's not worry about efficiency here.
>

Agreed.

>
> > None of this is clear (at least not to me) from PEP 558.
> >
> >
> > One problem with PEP 558 is that it's got too many words, and it's
> > lacking a section that crisply describes the semantics of the proposed
> > implementation. I've suggested to Nick that he add a section with
> > pseudo-code for the implementation, like you did in yours.
> >
> > (PS, did you read my PS about what locals() should do in class scope
> > when __prepare__ returns a non-dict?)
>
> Yes, but no harm in a reminder :)
> I'll update my PEP to fix the semantics of locals().
>

I'm in suspense as to what semantics you chose. :-)

PS. Another point where you seem to have missed some detail is in the
mapping from (proper) variable names to "frame locals" -- you use
co_varnames, but that (currently, at least) doesn't include cells.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Mark Shannon]

Hi Guido,

On 23/08/2021 3:53 pm, Guido van Rossum wrote:
On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon > wrote:


Hi Nick,

On 22/08/2021 4:51 am, Nick Coghlan wrote:

 > If Mark's claim that PyEval_GetLocals() could not be fixed was
true then
 > I would be more sympathetic to his proposal, but I know it isn't
true,
 > because it still works fine in the PEP 558 implementation (it even
 > immediately sees changes made via proxies, and proxies see
changes to
 > extra variables). The only truly unfixable public API is
 > PyFrame_LocalsToFast().

You are making claims that seem inconsistent with each other.
Namely, you are claiming that:

1. That the result of locals() is ephemeral.
2. That PyEval_GetLocals() returns a borrowed reference.

This seems impossible, as you can't return a borrowed reference to
an emphemeral object. That's just a pointer to freed memory.

Do `locals()` and `PyEval_GetLocals()` behave differently?


That is my understanding, yes. in PEP 558 locals() returns a snapshot 
dict, the Python-level f_locals property returns a fresh proxy that has 
no state except a pointer to the frame, and PyEval_GetLocals() returns a 
borrowed reference to the dict that's stored on the frame's C-level 
f_locals attribute


Can we avoid describing the C structs in any of these PEPs?

It confuses readers having Python attributes and "C-level attributes"
(C struct fields?).
It also restricts the implementation unnecessarily.

(E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11: 
https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7)




(In my "crazy" proposal all that is the same.)




Is the result of `PyEval_GetLocals()` cached, but `locals()` not?


I wouldn't call it a cache -- deleting it would affect the semantics, 
not just the performance. But yes, it returns a reference to an object 
that is owned by the frame, just as it does in 3.10 and before.


If that were the case, then it is a bit confusing, but could work.


Yes, see my "crazy" proposal.

Would PyEval_GetLocals() be defined as something like this?

(add _locals_cache attribute to the frame which is initialized to NULL).

def PyEval_GetLocals():
      frame._locals_cache attribute = locals()
      return borrow(frame._locals_cache attribute)


Nah, the dict returned by PyEval_GetLocals() is stored in the frame's 
C-level f_locals attribute, which is consulted by the Python-level 
f_locals proxy -- primarily to store "extra" variables, but IIUC in 
Nick's latest version it is also still used to cache by that proxy. 
Nick's locals() just returns dict(sys._getframe().f_locals).


The "extra" variables must be distinct from the result of locals() as
that includes both extras and "proper" variables.
If we want to cache the locals(), it needs to be distinct from the extra 
variables.


A debugger setting extra variables in a function that that is also 
accessed by a C call to PyEval_GetLocals() is going to be incredibly 
rare. Let's not worry about efficiency here.




None of this is clear (at least not to me) from PEP 558.


One problem with PEP 558 is that it's got too many words, and it's 
lacking a section that crisply describes the semantics of the proposed 
implementation. I've suggested to Nick that he add a section with 
pseudo-code for the implementation, like you did in yours.


(PS, did you read my PS about what locals() should do in class scope 
when __prepare__ returns a non-dict?)


Yes, but no harm in a reminder :)
I'll update my PEP to fix the semantics of locals().

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PA3ELKA4QLUV5VKEKUHZHRGT4P64EHGC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon  wrote:

> Hi Nick,
>
> On 22/08/2021 4:51 am, Nick Coghlan wrote:
>
> > If Mark's claim that PyEval_GetLocals() could not be fixed was true then
> > I would be more sympathetic to his proposal, but I know it isn't true,
> > because it still works fine in the PEP 558 implementation (it even
> > immediately sees changes made via proxies, and proxies see changes to
> > extra variables). The only truly unfixable public API is
> > PyFrame_LocalsToFast().
>
> You are making claims that seem inconsistent with each other.
> Namely, you are claiming that:
>
> 1. That the result of locals() is ephemeral.
> 2. That PyEval_GetLocals() returns a borrowed reference.
>
> This seems impossible, as you can't return a borrowed reference to
> an emphemeral object. That's just a pointer to freed memory.
>
> Do `locals()` and `PyEval_GetLocals()` behave differently?
>

That is my understanding, yes. in PEP 558 locals() returns a snapshot dict,
the Python-level f_locals property returns a fresh proxy that has no state
except a pointer to the frame, and PyEval_GetLocals() returns a borrowed
reference to the dict that's stored on the frame's C-level f_locals
attribute.

(In my "crazy" proposal all that is the same.)


> Is the result of `PyEval_GetLocals()` cached, but `locals()` not?
>

I wouldn't call it a cache -- deleting it would affect the semantics, not
just the performance. But yes, it returns a reference to an object that is
owned by the frame, just as it does in 3.10 and before.


> If that were the case, then it is a bit confusing, but could work.
>

Yes, see my "crazy" proposal.


> Would PyEval_GetLocals() be defined as something like this?
>
> (add _locals_cache attribute to the frame which is initialized to NULL).
>
> def PyEval_GetLocals():
>  frame._locals_cache attribute = locals()
>  return borrow(frame._locals_cache attribute)
>

Nah, the dict returned by PyEval_GetLocals() is stored in the frame's
C-level f_locals attribute, which is consulted by the Python-level f_locals
proxy -- primarily to store "extra" variables, but IIUC in Nick's latest
version it is also still used to cache by that proxy. Nick's locals() just
returns dict(sys._getframe().f_locals).


> None of this is clear (at least not to me) from PEP 558.
>

One problem with PEP 558 is that it's got too many words, and it's lacking
a section that crisply describes the semantics of the proposed
implementation. I've suggested to Nick that he add a section with
pseudo-code for the implementation, like you did in yours.

(PS, did you read my PS about what locals() should do in class scope when
__prepare__ returns a non-dict?)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/54HKYGCFC4M76ZCWYVMG56BOGJNVACFM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Mark Shannon]

Hi Nick,

On 22/08/2021 4:51 am, Nick Coghlan wrote:



On Sun, 22 Aug 2021, 10:47 am Guido van Rossum, > wrote:


Hopefully anyone is still reading python-dev.

I'm going to try to summarize the differences between the two
proposals, even though Mark already did so in his PEP. But I'd like
to start by calling out the key point of contention.

Everything here is about locals() and f_locals in *function scope*.
(I use f_locals to refer to the f_locals field of frame objects as
seen from Python code.) And in particular, it is about what I'll
call "extra variables": the current CPython feature that you can add
*new* variables to f_locals that don't exist in the frame, for example:

def foo():
     x = 1
     locals()["y"] = 2  # or sys._getframe()["y"] = 2

My first reaction was to propose to drop this feature, but I realize
it's kind of important for debuggers to be able to execute arbitrary
code in function code -- assignments to locals should affect the
frame, but it should also be possible to create new variables (e.g.
temporaries). So I agree we should keep this.


I actually tried taking this feature out in one of the PEP 558 drafts, 
but actually doing so breaks the pdb test suite.





So apparently the key difference of opinion between Mark and Nick is
about f_locals, and what to do with extras. In Nick's proposal when
you reference f.f_locals twice in a row (for the same frame object
f), you get the same proxy object, whereas in Mark's proposal you
get a different object each time, but it doesn't matter, because the
proxy has no state other than a reference to the frame.


If PEP 558 is still giving that impression, I need to fix the wording - 
the proxy objects are ephemeral in both PEPs (the 558 text is slightly 
behind the implementation on that point, as the fast refs mapping is now 
stored on the frame object, so it only needs to be built once)


In Mark's proposal, if you assign a value to an extra variable, it
gets stored in a hidden dict field on the frame, and when you read
the proxy, the contents of that hidden dict field gets included.
This hidden dict lazily created on the first store to an extra
variable. (Mark shows pseudo-code to clarify this; the hidden dict
is stored as _extra_locals on the frame.)


PEP 558 works essentially the same way, the difference is that it uses 
the existing locals dict storage rather than adding new storage just for 
optimised frames.


In Nick's proposal, there's a cache on the frame that stores both
the extras and the proper variables. This cache can get out of sync
with the contents of the proper variables when some bytecode is
executed (for performance reasons we don't want the bytecode to keep
the cache up to date on every store), so there's an operation to
sync the frame cache (sync_frame_cache(), it's not defined in which
namespace this exists -- is it a builtin or in sys?).


It's an extra method on the proxy objects. You only need it if you keep 
an old proxy object around - if you always retrieve a new proxy object 
after executing Python code, that proxy will refresh the cache when it 
needs to.




Frankly the description in Nick's PEP is hard to follow -- I am not
100% sure what is meant by "the dynamic snapshot", and it's not
quite clear whether proper variables are copied into the cache (and
if so, why).


Aye, Mark was a bit quicker with his PEP than I anticipated, so I've 
incorporated the implementation improvements arising from his last round 
of comments, but the PEP text hasn't been updated yet.



Personally, I find Mark's proposed semantics for f_locals simpler --
there's no cache, only storage for extras, so there's nothing that
can get out of sync.


The wording in PEP 667 undersells the cost of that simplification:

"Code that uses PyEval_GetLocals() will continue to operate safely, but 
will need to be changed to use PyEval_Locals() to restore functionality."



Code that uses PyEval_GetLocals() will NOT continue to operate safely 
under PEP 667: all such code will raise an exception at runtime, and 
need to be rewritten to use a new API with different refcounting 
semantics. That's essentially all code that accesses the frame locals 
from C, since we don't offer supported APIs for that other than 
PyEval_GetLocals() (directly accessing the f_locals field on the frame 
object is only "supported" in a very loose sense of the word, although 
PEP 558 mostly keeps that working, too)


This means the real key difference between the two PEPs is that Mark is 
proposing a gratuitous compatibility break for PyEval_GetLocals() that 
also means that the algorithmic complexity characteristics of the proxy 
implementation will be completely off from those of a regular dict (e.g. 
len(proxy) will be O(n) in the number of variables defined on the 

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

On Sat, Aug 21, 2021 at 8:52 PM Nick Coghlan  wrote:

>
> On Sun, 22 Aug 2021, 10:47 am Guido van Rossum,  wrote:
>
>>
>> Everything here is about locals() and f_locals in *function scope*. (I
>> use f_locals to refer to the f_locals field of frame objects as seen from
>> Python code.) And in particular, it is about what I'll call "extra
>> variables": the current CPython feature that you can add *new* variables to
>> f_locals that don't exist in the frame, for example:
>>
>> def foo():
>> x = 1
>> locals()["y"] = 2  # or sys._getframe()["y"] = 2
>>
>> My first reaction was to propose to drop this feature, but I realize it's
>> kind of important for debuggers to be able to execute arbitrary code in
>> function code -- assignments to locals should affect the frame, but it
>> should also be possible to create new variables (e.g. temporaries). So I
>> agree we should keep this.
>>
>
> I actually tried taking this feature out in one of the PEP 558 drafts, but
> actually doing so breaks the pdb test suite.
>

I wonder if we should reconsider this, given that so much of the complexity
of the competing PEPs is due to this issue of "extra" variables. We can fix
pdb, and other debuggers probably will be happy to make some changes
(debuggers often are closely tied to the implementation anyway -- e.g.
PyDev currently seems broken with 3.11). One way to fix it would be to have
the debugger use a mapping implementation that acts as a proxy for
f_locals, but stores extra variables in its own storage. Maybe this just
moves the problem, but I feel support for extra variables will always be a
bit of a wart, and many uses of f_locals don't need them.

Another thing I feel we should at least have a good second look at is the
"locals() returns a snapshot" behavior. This is the same in both PEPs but
it is inconsistent with module and class scopes, as well as different from
3.10.
I wonder if we're valuing "does it return the same type" too much over
"does it exhibit the same high-level (conceptual) behavior" here. Yes, a
snapshot is a dict, just like what you get from locals() in class and
module scopes. But no, that dict is not an alias for the actual contents of
the scope. If we made locals() return the same proxy that f_locals gives,
it's no longer a dict, but it has the same *conceptual* behavior (maybe:
"meaning") as for the other types of scopes.


> So apparently the key difference of opinion between Mark and Nick is about
>> f_locals, and what to do with extras. In Nick's proposal when you reference
>> f.f_locals twice in a row (for the same frame object f), you get the same
>> proxy object, whereas in Mark's proposal you get a different object each
>> time, but it doesn't matter, because the proxy has no state other than a
>> reference to the frame.
>>
>
> If PEP 558 is still giving that impression, I need to fix the wording -
> the proxy objects are ephemeral in both PEPs (the 558 text is slightly
> behind the implementation on that point, as the fast refs mapping is now
> stored on the frame object, so it only needs to be built once)
>

Ah, I didn't actually find a clear indication one way or another.  If your
proposal *also* makes f.f_locals return a new object on each use, the
difference between the two proposals really is entirely in the C API, as
you bring up below.

>
> In Mark's proposal, if you assign a value to an extra variable, it gets
>> stored in a hidden dict field on the frame, and when you read the proxy,
>> the contents of that hidden dict field gets included. This hidden dict
>> lazily created on the first store to an extra variable. (Mark shows
>> pseudo-code to clarify this; the hidden dict is stored as _extra_locals on
>> the frame.)
>>
>
> PEP 558 works essentially the same way, the difference is that it uses the
> existing locals dict storage rather than adding new storage just for
> optimised frames.
>

Oh, you're right. So then this doesn't really matter -- if there's no other
use for the C-level field f_locals in a function scope, then we might as
well use that to store the extras (assuming its NULL-ness isn't used as a
flag for some other purpose). To be clear, I *think* that for a function
scope where the f_locals property has never been used and locals() has
never been called, the C-level f_locals field is NULL.


> In Nick's proposal, there's a cache on the frame that stores both the
>> extras and the proper variables. This cache can get out of sync with the
>> contents of the proper variables when some bytecode is executed (for
>> performance reasons we don't want the bytecode to keep the cache up to date
>> on every store), so there's an operation to sync the frame cache
>> (sync_frame_cache(), it's not defined in which namespace this exists -- is
>> it a builtin or in sys?).
>>
>
> It's an extra method on the proxy objects. You only need it if you keep an
> old proxy object around - if you always retrieve a new proxy object after
> executing Python code, that proxy will 

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Nick Coghlan]

On Sun, 22 Aug 2021, 10:47 am Guido van Rossum,  wrote:

> Hopefully anyone is still reading python-dev.
>
> I'm going to try to summarize the differences between the two proposals,
> even though Mark already did so in his PEP. But I'd like to start by
> calling out the key point of contention.
>
> Everything here is about locals() and f_locals in *function scope*. (I use
> f_locals to refer to the f_locals field of frame objects as seen from
> Python code.) And in particular, it is about what I'll call "extra
> variables": the current CPython feature that you can add *new* variables to
> f_locals that don't exist in the frame, for example:
>
> def foo():
> x = 1
> locals()["y"] = 2  # or sys._getframe()["y"] = 2
>
> My first reaction was to propose to drop this feature, but I realize it's
> kind of important for debuggers to be able to execute arbitrary code in
> function code -- assignments to locals should affect the frame, but it
> should also be possible to create new variables (e.g. temporaries). So I
> agree we should keep this.
>

I actually tried taking this feature out in one of the PEP 558 drafts, but
actually doing so breaks the pdb test suite.



>
> So apparently the key difference of opinion between Mark and Nick is about
> f_locals, and what to do with extras. In Nick's proposal when you reference
> f.f_locals twice in a row (for the same frame object f), you get the same
> proxy object, whereas in Mark's proposal you get a different object each
> time, but it doesn't matter, because the proxy has no state other than a
> reference to the frame.
>

If PEP 558 is still giving that impression, I need to fix the wording - the
proxy objects are ephemeral in both PEPs (the 558 text is slightly behind
the implementation on that point, as the fast refs mapping is now stored on
the frame object, so it only needs to be built once)

In Mark's proposal, if you assign a value to an extra variable, it gets
> stored in a hidden dict field on the frame, and when you read the proxy,
> the contents of that hidden dict field gets included. This hidden dict
> lazily created on the first store to an extra variable. (Mark shows
> pseudo-code to clarify this; the hidden dict is stored as _extra_locals on
> the frame.)
>

PEP 558 works essentially the same way, the difference is that it uses the
existing locals dict storage rather than adding new storage just for
optimised frames.

In Nick's proposal, there's a cache on the frame that stores both the
> extras and the proper variables. This cache can get out of sync with the
> contents of the proper variables when some bytecode is executed (for
> performance reasons we don't want the bytecode to keep the cache up to date
> on every store), so there's an operation to sync the frame cache
> (sync_frame_cache(), it's not defined in which namespace this exists -- is
> it a builtin or in sys?).
>

It's an extra method on the proxy objects. You only need it if you keep an
old proxy object around - if you always retrieve a new proxy object after
executing Python code, that proxy will refresh the cache when it needs to.



> Frankly the description in Nick's PEP is hard to follow -- I am not 100%
> sure what is meant by "the dynamic snapshot", and it's not quite clear
> whether proper variables are copied into the cache (and if so, why).
>

Aye, Mark was a bit quicker with his PEP than I anticipated, so I've
incorporated the implementation improvements arising from his last round of
comments, but the PEP text hasn't been updated yet.


Personally, I find Mark's proposed semantics for f_locals simpler --
> there's no cache, only storage for extras, so there's nothing that can get
> out of sync.
>

The wording in PEP 667 undersells the cost of that simplification:

"Code that uses PyEval_GetLocals() will continue to operate safely, but
will need to be changed to use PyEval_Locals() to restore functionality."


Code that uses PyEval_GetLocals() will NOT continue to operate safely under
PEP 667: all such code will raise an exception at runtime, and need to be
rewritten to use a new API with different refcounting semantics. That's
essentially all code that accesses the frame locals from C, since we don't
offer supported APIs for that other than PyEval_GetLocals() (directly
accessing the f_locals field on the frame object is only "supported" in a
very loose sense of the word, although PEP 558 mostly keeps that working,
too)

This means the real key difference between the two PEPs is that Mark is
proposing a gratuitous compatibility break for PyEval_GetLocals() that also
means that the algorithmic complexity characteristics of the proxy
implementation will be completely off from those of a regular dict (e.g.
len(proxy) will be O(n) in the number of variables defined on the frame
rather than being O(1) after the proxy's initial cache update the way it is
in PEP 558)

If Mark's claim that PyEval_GetLocals() could not be fixed was true then I
would be more sympathetic to 

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Guido van Rossum]

Hopefully anyone is still reading python-dev.

I'm going to try to summarize the differences between the two proposals,
even though Mark already did so in his PEP. But I'd like to start by
calling out the key point of contention.

Everything here is about locals() and f_locals in *function scope*. (I use
f_locals to refer to the f_locals field of frame objects as seen from
Python code.) And in particular, it is about what I'll call "extra
variables": the current CPython feature that you can add *new* variables to
f_locals that don't exist in the frame, for example:

def foo():
x = 1
locals()["y"] = 2  # or sys._getframe()["y"] = 2

My first reaction was to propose to drop this feature, but I realize it's
kind of important for debuggers to be able to execute arbitrary code in
function code -- assignments to locals should affect the frame, but it
should also be possible to create new variables (e.g. temporaries). So I
agree we should keep this.

Terminology-wise, I will refer to variables that are allocated in the frame
(like "x" above, and including nonlocals/cells) as "proper" variables.

Both PEPs give up when it comes to locals(), declaring it to return a
snapshot in this case. This is mostly to ensure better backwards
compatibility, since existing code calling locals() may well assume it's a
dict. Both PEPs make f_locals some kind of proxy that gives a direct
read-write view on the variables in the frame (including cells used for
nonlocal references), but they differ in the precise semantics.

So apparently the key difference of opinion between Mark and Nick is about
f_locals, and what to do with extras. In Nick's proposal when you reference
f.f_locals twice in a row (for the same frame object f), you get the same
proxy object, whereas in Mark's proposal you get a different object each
time, but it doesn't matter, because the proxy has no state other than a
reference to the frame. In Mark's proposal, if you assign a value to an
extra variable, it gets stored in a hidden dict field on the frame, and
when you read the proxy, the contents of that hidden dict field gets
included. This hidden dict lazily created on the first store to an extra
variable. (Mark shows pseudo-code to clarify this; the hidden dict is
stored as _extra_locals on the frame.)

In Nick's proposal, there's a cache on the frame that stores both the
extras and the proper variables. This cache can get out of sync with the
contents of the proper variables when some bytecode is executed (for
performance reasons we don't want the bytecode to keep the cache up to date
on every store), so there's an operation to sync the frame cache
(sync_frame_cache(), it's not defined in which namespace this exists -- is
it a builtin or in sys?).

Frankly the description in Nick's PEP is hard to follow -- I am not 100%
sure what is meant by "the dynamic snapshot", and it's not quite clear
whether proper variables are copied into the cache (and if so, why).

There are also differences in the proposed C API changes, but the
differences there are solvable once we choose the semantics for f_locals.

Personally, I find Mark's proposed semantics for f_locals simpler --
there's no cache, only storage for extras, so there's nothing that can get
out of sync.

I would even consider making locals() return the same proxy -- this is
simpler and more consistent with module and class scopes, but it is less
backwards compatible, and locals() is used orders of magnitude more than
f_locals. (Also, we'd have to modify exec() and eval() to allow using a
non-dict as globals, which would require some deep changes in the
interpreter.)

--Guido

PS. In Mark's PEP, there's a pseudo-code version of locals() that can give
a different result in class scope than the current CPython implementation:
Using __prepare__, a metaclass can provide a namespace to execute the class
body that's not a dict (subclass) instance. The current CPython behavior
and AFAICT Nick's PEP return that namespace from locals(), but Mark's
pseudo-code would return a snapshot copy. I think it's better to stick to
the current semantics (and I suspect Mark overlooked this edge case).

On Fri, Aug 20, 2021 at 8:23 AM Mark Shannon  wrote:

> Hi all,
>
> I have submitted PEP 667 as an alternative to PEP 558.
> https://www.python.org/dev/peps/pep-0667/
>
> Nick and I have agreed to disagree on the way to fix locals() and
> f_locals. We are both in agreement that it needs fixing.
>
> In summary, PEP 667 has roughly the same surface behavior as PEP 558 but
> is simpler and more consistent internally, at the expense of some minor
> C API backwards incompatibility issues.
>
> PEP 558 also has backwards incompatibility issues, but claims to be more
> compatible at the C API level.
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
>