Re: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug.

[Tim Peters]

The PR for this looks good to go:

https://github.com/python/cpython/pull/13612

But, I still have no idea how it works for the OP's original test
case.  So, if you have at least 80 GB of RAM to try it, I added
`arena.py` to the BPO report:

https://bugs.python.org/issue37029

That adds code to the OP's test case to display the times needed to
build the tree and to tear it down (& to display some obmalloc stats).
So there's no need for you to think about anything ;-)

I'm keen to get feedback on this before merging the PR, because this
case is so very much larger than anything I've ever tried that I'm
wary that there may be more than one "surprise" lurking here.  The PR
certainly addresses "an obvious" (with hindsight) problem - but is
that the _only_ gross design flaw here?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Guido van Rossum]

On Thu, May 30, 2019 at 4:28 PM Greg Ewing 
wrote:

> Nick Coghlan wrote:
> > So for me, getting rid of write backs via exec and "import *" was a
> > matter of "Yay, we finally closed those unfortunate loopholes" rather
> > than being any kind of regrettable necessity.
>
> If that were the reasoning, the principled thing to do would be
> to raise an exception if an eval or exec tries to write to a
> local, rather than mostly ignore it.
>
> In any case, I don't really agree with that philosophy. Python
> is at its essence a dynamic language. Things like JIT and static
> type analysis are only possible to the extent that you refrain
> from using some of its dynamic features. Removing features
> entirely just because they *can* interfere with these things goes
> against the spirit of the language, IMO.
>

Right. And static analysis should also be able to detect most uses of
locals() in a frame. I believe I've heard of some alternate Python
implementations that detect usage of some functions to disable some
optimizations (IIRC IronPython did this to sys._getframe()).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Should I postpone PEP 558 (locals() semantics) to Python 3.9?

[Guido van Rossum]

Given the flurry of discussion that happened this week, and that the PR
switching the PEP to use independent snapshots hasn't even been landed, I'm
skeptical that it's wise to attempt to get this in before beta 1 even if
you were to resolve the segfaults.

On Thu, May 30, 2019 at 4:24 PM Nick Coghlan  wrote:

> Hi folks,
>
> The reference implementation for PEP 558 (my attempt to fix the
> interaction between tracing functions and closure variables) is currently
> segfaulting somewhere deep in the garbage collector, and I've found that
> there's an issue with the PyEval_GetLocals() API returning a borrowed
> reference that means I need to tweak the proposed C API a bit such that
> PyEval_GetLocals() returns the proxy at function scope, and we add a new
> PyEval_GetPyLocals() that matches the locals() builtin.
>
> I don't *want* to postpone this to Python 3.9, but there turned out to be
> more remaining work than I thought there was to get this ready for
> inclusion in beta 1.
>
> I'll try to get the C API design details sorted today, but the segfault is
> mystifying me, and prevents the option of putting the core implementation
> in place for b1, and tidying up the documentation and comments for b2.
>
> Cheers,
> Nick.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Greg Ewing]

Nick Coghlan wrote:

So for me, getting rid of write backs via exec and "import *" was a
matter of "Yay, we finally closed those unfortunate loopholes" rather
than being any kind of regrettable necessity.


If that were the reasoning, the principled thing to do would be
to raise an exception if an eval or exec tries to write to a
local, rather than mostly ignore it.

In any case, I don't really agree with that philosophy. Python
is at its essence a dynamic language. Things like JIT and static
type analysis are only possible to the extent that you refrain
from using some of its dynamic features. Removing features
entirely just because they *can* interfere with these things goes
against the spirit of the language, IMO.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Should I postpone PEP 558 (locals() semantics) to Python 3.9?

[Nick Coghlan]

Hi folks,

The reference implementation for PEP 558 (my attempt to fix the interaction
between tracing functions and closure variables) is currently segfaulting
somewhere deep in the garbage collector, and I've found that there's an
issue with the PyEval_GetLocals() API returning a borrowed reference that
means I need to tweak the proposed C API a bit such that PyEval_GetLocals()
returns the proxy at function scope, and we add a new PyEval_GetPyLocals()
that matches the locals() builtin.

I don't *want* to postpone this to Python 3.9, but there turned out to be
more remaining work than I thought there was to get this ready for
inclusion in beta 1.

I'll try to get the C API design details sorted today, but the segfault is
mystifying me, and prevents the option of putting the core implementation
in place for b1, and tidying up the documentation and comments for b2.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 558: Defined semantics for locals()

[Nick Coghlan]

On Fri., 31 May 2019, 5:20 am Xavier de Gaye,  wrote:

> Currently f_locals is documented as readonly [1].
>

Read-only in the sense that you can't rebind it to point to a different
object - the dict it points to is mutable.


> The PEP says:
> > * "Don't change what isn't broken": the current tracing mode problems
> are caused
> >   by a requirement that's specific to tracing mode (support for external
> >   rebinding of function local variable references), so it made sense to
> also
> >   restrict any related fixes to tracing mode
> >
> > However, actually attempting to implement and document that dynamic
> approach
> > highlighted the fact that it makes for a really subtle runtime state
> dependent
> > behaviour distinction in how ``frame.f_locals`` works, and creates
> several
> > new edge cases around how ``f_locals`` behaves as trace functions are
> added
> > and removed.
> >
> > Accordingly, the design was switched to the current one, where
> > ``frame.f_locals`` is always a write-through proxy, and ``locals()`` is
> always
> > a dynamic snapshot, which is both simpler to implement and easier to
> explain.
>
> Do these edge cases still exist when f_locals write access is restricted
> to code executed by the tracing function (which is more restrictive than
> 'tracing mode') ?


We can use the condition frame->f_trace not NULL and tstate->tracing true
> (tstate being a pointer to the PyThreadState structure) to know when code
> is executed by the tracing function [2]:
> * The condition on tstate->tracing allows to figure out if we are running
> a frame executed by the trace function as opposed to a frame that is being
> traced or a frame executed in 'regular operation'.
> * The condition on frame->f_trace removes the ambiguity whether
> tstate->tracing is set by a tracing function or by a profiling function.
>

Always creating the proxy and sometimes bypassing it and returning the
snapshot instead would indeed have fewer edge cases than sometimes storing
the snapshot directly on the frame object without creating the proxy at all.

It's still significantly harder to document than "frame.f_locals references
a proxy, locals() creates a snapshot", though.

Cheers,
Nick.


>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 558: Defined semantics for locals()

[Xavier de Gaye]

Currently f_locals is documented as readonly [1].

The PEP says:

* "Don't change what isn't broken": the current tracing mode problems are caused
  by a requirement that's specific to tracing mode (support for external
  rebinding of function local variable references), so it made sense to also
  restrict any related fixes to tracing mode

However, actually attempting to implement and document that dynamic approach
highlighted the fact that it makes for a really subtle runtime state dependent
behaviour distinction in how ``frame.f_locals`` works, and creates several
new edge cases around how ``f_locals`` behaves as trace functions are added
and removed.

Accordingly, the design was switched to the current one, where
``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always
a dynamic snapshot, which is both simpler to implement and easier to explain.


Do these edge cases still exist when f_locals write access is restricted to 
code executed by the tracing function (which is more restrictive than 'tracing 
mode') ?
We can use the condition frame->f_trace not NULL and tstate->tracing true 
(tstate being a pointer to the PyThreadState structure) to know when code is executed 
by the tracing function [2]:
* The condition on tstate->tracing allows to figure out if we are running a 
frame executed by the trace function as opposed to a frame that is being traced or 
a frame executed in 'regular operation'.
* The condition on frame->f_trace removes the ambiguity whether tstate->tracing 
is set by a tracing function or by a profiling function.

[1] In section 'Frame objects' at 
https://github.com/python/cpython/blob/master/Doc/reference/datamodel.rst#the-standard-type-hierarchy
[2] Except that frame->f_trace is NULL in a 'PyTrace_CALL' trace event, so 
f_locals would remain readonly in that case. But this event is always followed by 
a 'PyTrace_LINE' event anyway so this limitation is not important IMO.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [PEP 558] thinking through locals() semantics

[Nick Coghlan]

On Thu, 30 May 2019 at 09:12, Greg Ewing  wrote:
>
> Nick Coghlan wrote:
> > If there was a compelling use case for letting "a = 1; exec(src);
> > print(a)" print something other than "1" at function scope, then I'd
> > be more amenable to the idea of the associated compatibility break and
> > potential performance regression in other implementations.
> >
> > However, there isn't any such use case - if there were, we wouldn't
> > have deliberately changed the semantics from the old Python 2 ones to
> > the current Python 3 ones in PEP 3100 [1].
>
> I get the impression that was done because everyone involved
> thought it wasn't worth the ugliness of maintaining all the
> fast/locals swapping stuff, not because of any principle that
> the current behaviour is right or better in any way.

You may have felt that way, but I certainly don't - routinely hiding
function local rebinding from the compiler (and type checkers, and
human readers) is awful, and a privilege that should be reserved to
debuggers and other tools operating at a similar level of "able to
interfere with the normal runtime execution of a program". (Module and
class namespaces are different - they're inherently mutable shared
namespaces as far as the Python runtime is occurred, so locals()
providing one more way of getting a reference to them isn't that big
of a deal - if it's a problem, you can just push the code where you
need to avoid those semantics matters down into a helper function)

So for me, getting rid of write backs via exec and "import *" was a
matter of "Yay, we finally closed those unfortunate loopholes" rather
than being any kind of regrettable necessity.

It's likely also a factor that Python 2.2.2 was the first version of
Python that I ever used extensively, so the existing snapshot-like
behaviour of locals() is the behaviour that feels normal to me, and
I'm somewhat mystified by the notion that anyone might actually *want*
it to behave differently (except to close the remaining loopholes that
still allowed mysterious implicit mutation of previously created
snapshots).

The only rationale for considering that possibility seems to be "It
would make function namespaces behave more like class and module
namespaces", to which my response is "Why would you even want that,
and what practical benefit would it bring to justify the otherwise
gratuitous compatibility break?"

Cheers,
Nick.


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com