[Python-Dev] Latest version of PEP 669 -- Low Impact Monitoring for CPython

[Mark Shannon]

Hi everyone,

The latest version of PEP 669 is up.

The discussion is happening at 
https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018,
but I am following email, if you want to respond here.

I'd like to get this into 3.12, so I'll probably submit the PEP to the steering 
council in a month or so.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XWLJ7JXODYRKAPKQCXO7IQ2AME6JM7HQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: code.replace() and Python 3.11 exception table

[Mark Shannon]

Hi Gabriele,

On 01/04/2022 4:50 pm, Gabriele wrote:

Does this mean that this line in the bytecode library is likely to fail with 
3.11, with no way to fix it?



You can pass the exception table the same way you pass all the other arguments.
The exception table depends on the code, but that is nothing new. The bytecode 
library already recomputes the consts, names, etc.

TBH, calling `types.CodeType` didn't work for earlier versions either.
It just sort of worked, some of the time.

Cheers,
Mark.



https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
 


On Fri, 1 Apr 2022, 10:40 Victor Stinner, mailto:vstin...@python.org>> wrote:

I created https://bugs.python.org/issue47185 
 to discuss this issue:
either recompute automatically co_exceptiontable, or at least document
the change.

Victor

On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner mailto:vstin...@python.org>> wrote:
 >
 > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
 > private C API to the internal C API")
 >
 > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico mailto:ros...@gmail.com>> wrote:
 > >
 > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner mailto:vstin...@python.org>> wrote:
 > > > In Python, sadly the types.CodeType type also has a public 
constructor
 > > > and many projects break at each Python release because the API
 > > > changes. Hopefully, it seems like the new CodeType.replace() method
 > > > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
 > > > better abstraction and closer to what developers need in practice.
 > >
 > > It certainly has been for me. When I want to do bytecode hackery, I
 > > usually start by creating a function with def/lambda, then construct a
 > > modified function using f.__code__.replace(). It's the easiest way to
 > > ensure that all the little details are correct.
 >
 > Python 3.11 added the concept of "exception table"
 > (code.co_exceptiontable). You have to build this table, otherwise
 > Python can no longer catch exceptions :-)
 >
 > I don't know how to build this exception table. It seems like
 > currently there is no Python function in the stdlib to build this
 > table.
 >
 > Example:
 > ---
 > def f():
 >     try:
 >         print("raise")
 >         raise ValueError
 >     except ValueError:
 >         print("except")
 >     else:
 >         print("else")
 >     print("exit func")
 >
 > def g(): pass
 >
 > if 1:
 >     code = f.__code__
 >     g.__code__ = g.__code__.replace(
 >         co_code=code.co_code,
 >         co_consts=code.co_consts,
 >         co_names=code.co_names,
 >         co_flags=code.co_flags,
 >         co_stacksize=code.co_stacksize)
 > else:
 >     g.__code__ = f.__code__  # this code path works on Python 3.11
 >
 > g()
 > ---
 >
 > Output with Python 3.10 (ok):
 > ---
 > raise
 > except
 > exit func
 > ---
 >
 > Output with Python 3.11 (oops):
 > ---
 > raise
 > Traceback (most recent call last):
 >   ...
 > ValueError
 > ---
 >
 > By the way, this change is not documented at all:
 >
 > * https://docs.python.org/dev/library/types.html#types.CodeType 

 > * https://docs.python.org/dev/whatsnew/3.11.html 

 >
 > I understand that these changes come from the "Zero cost exception
 > handling" change:
 > https://bugs.python.org/issue40222 
 >
 > Victor
 > --
 > Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.

___
Python-Dev mailing list -- python-dev@python.org 

To unsubscribe send an email to python-dev-le...@python.org 

https://mail.python.org/mailman3/lists/python-dev.python.org/ 

Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT4XQ7LOGCYM7WJCI3RYVW2VGV/
 

Code of Conduct: http://python.org/psf/codeofconduct/ 



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to pytho

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Mark Shannon]

Hi,

On 22/03/2022 6:07 pm, Victor Stinner wrote:

Hi,

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):

* https://github.com/python/cpython/pull/32052
* https://github.com/python/cpython/pull/32054


I just want to say that it is important the API is easy to use and access.

Otherwise there is a temptation to directly set the underlying function 
pointer, and that means we don't get to see when PEP 523 is used.
We might then mis-optimize on the assumption that PEP 523 isn't used.

Cheers,
Mark.



API:

* _PyFrameEvalFunction type
* _PyInterpreterState_GetEvalFrameFunc()
* _PyInterpreterState_SetEvalFrameFunc()
* (undocumented) _PyEval_EvalFrameDefault()

The private API to get/set the eval function *is* documented at:
https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc

I added the Get/Set functions so debuggers don't have to access
directly to the PyInterpreterState structure which has been moved to
the internal C API in Python 3.8.

This API causes me multiple issues:

* It's a private API and I'm trying to remove the private API from the
public C API header files.
* The _PyFrameEvalFunction type is not stable: it got a new "tstate"
parameter in Python 3.9 and the type of the second parameter changed
from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
* These functions use the _PyInterpreterFrame type which is part of
the internal C API.

While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.

I propose to move theses API to the internal header files
(Include/internals/) to clarify that it's not part of the public C API
and that there is no backward compatibility warranty.

The change is being discussed at:
https://bugs.python.org/issue46850

--

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex()
* _PyCode_GetExtra()
* _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the
pydevd debugger. The two others seem to be unused in the wild. I'm not
sure if these ones should be moved to the internal C API. They can be
left unchanged, since they don't use a type only defined by the
internal C API.

Victor

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L2E2HLUT757ROREKWIAD7M2HIHNUJ3VE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Compiling of ast.Module in Python 3.10 and co_firstlineno behavior

[Mark Shannon]

Hi Fabio,

On 17/02/2022 7:30 pm, Fabio Zadrozny wrote:


Em qui., 17 de fev. de 2022 às 16:05, Mark Shannon mailto:m...@hotpy.org>> escreveu:

Hi Fabio,

This happened as part of implementing PEP 626.
The previous behavior isn't very robust w.r.t doc strings and
compiler optimizations.

OOI, why would you want to revert to the old behavior?


Hi Mark,

The issue I'm facing is that ipython uses an approach of obtaining the ast for 
a function to be executed and then it goes on node by node executing it.

When running in the debugger, the debugger caches some information based on 
(co_firstlineno, co_name, co_filename) to have information saved across multiple calls to 
the same function, which works in general because each function in a given python file 
would have its own co_firstlineno, but in this specific case here it gets a single function 
and then recompiles it expression by expression -- so, it'll have the same co_filename 
() and the same co_name (), but then the co_firstlineno would be 
different (because the statement resides in a different line), but with Python 3.10 this 
assumption fails as even the co_firstlineno will be the same...


A bit off topic, but why not use a different name for each cell?



You can see the actual issues at: https://github.com/microsoft/vscode-jupyter/issues/8803 
<https://github.com/microsoft/vscode-jupyter/issues/8803> / 
https://github.com/ipython/ipykernel/issues/841/ 
<https://github.com/ipython/ipykernel/issues/841/> 
https://github.com/microsoft/debugpy/issues/844 
<https://github.com/microsoft/debugpy/issues/844>

After thinkering a bit it seems it's possible to create a new code object based 
on an existing code object with `code.replace` (re-assembling the 
co_lnotab/co_firstlineno), so, I'm going to propose that as a fix to ipython, 
but I found it really strange that this did change in Python 3.10 in the first 
place as the old behavior seemed reasonable for me (i.e.: with the new behavior 
it's a bit strange that the user is compiling something with a single statement 
on line 99 and yet the resulting code object will have the co_firstlineno == 1).


That's the behavior for functions. If I define a function on line 10, but the 
first line of code in that function is on line 100, then 
`func.__code__.co_firstlineno == 10`, not 100. Modules start on line 1, by 
definition.

You can find the first line of actual code using the `co_lines()` iterator.

firstline = next(mod.__code__.co_lines())[2]

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4JWS4QUENUSBWVXUFPNR5IWYFMC7AV53/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Compiling of ast.Module in Python 3.10 and co_firstlineno behavior

[Mark Shannon]

Hi Fabio,

This happened as part of implementing PEP 626.
The previous behavior isn't very robust w.r.t doc strings and
compiler optimizations.

OOI, why would you want to revert to the old behavior?

Cheers,
Mark.

On 17/02/2022 5:52 pm, Fabio Zadrozny wrote:

Hi all,

I'm stumbling with an issue where the co_firstlineno behavior changed from 
Python 3.9 to Python 3.10 and I was wondering if this was intentional or not.

i.e.: Whenever a code is compiled in Python 3.10, the `code.co_firstlineno` is 
now always 1, whereas previously it was equal to the first statement.

Also, does anyone know if there is any way to restore the old behavior in 
Python 3.10? I tried setting the `module.lineno` but it didn't really make any 
difference...

As an example, given the code below:

|import dis source = ''' print(1) print(2) ''' initial_module = compile(source, 
'', 'exec', PyCF_ONLY_AST, 1) import sys print(sys.version) for i in range(2): 
module = Module([initial_module.body[i]], []) module_code = compile(module, '', 'exec') print(' --> First lineno:', module_code.co_firstlineno) print(' --> 
Line starts :', list(lineno for offset, lineno in dis.findlinestarts(module_code))) print(' 
dis ---') dis.dis(module_code)|



I have the following outputs for Pyhon 3.9/Python 3.10:

|3.9.6 (default, Jul 30 2021, 11:42:22) [MSC v.1916 64 bit (AMD64)] --> First lineno: 2 
--> Line starts : [2]  dis --- 2 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (1) 4 
CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE --> First lineno: 4 
--> Line starts : [4]  dis --- 4 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (2) 4 
CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE|



|3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] --> 
First lineno: 1 --> Line starts : [2]  dis --- 2 0 LOAD_NAME 0 (print) 2 LOAD_CONST 
0 (1) 4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE --> First 
lineno: 1 --> Line starts : [4]  dis --- 4 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (2) 
4 CALL_FUNCTION 1 6 POP_TOP 8 LOAD_CONST 1 (None) 10 RETURN_VALUE |

Thanks,

Fabio


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VXW3TVHVYOMXDQIQBJNZ4BTLXFT4EPQZ/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2YHNQVGQEDDDKF7MVZIQA4GBIMYC2CJD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Steering-council] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Mark Shannon]

On 31/01/2022 5:23 am, Gregory P. Smith wrote:

-cc: python-steering-council

On Fri, Mar 5, 2021 at 4:26 PM Guido van Rossum mailto:gu...@python.org>> wrote:

On Fri, Mar 5, 2021 at 11:11 AM Brett Cannon mailto:br...@python.org>> wrote:

Speaking for myself ...


Ditto ...

On Fri, Mar 5, 2021 at 7:04 AM Mark Shannon mailto:m...@hotpy.org>> wrote:
[...]

In some cases, the PEP would have improved the situation.

For example:
sys.setrecursionlimit(5000)
def f():
      f()

Currently, it raises a RecursionError on linux, but crashes the
interpreter on Windows.
With PEP 651 it would have raised a RecursionError on both 
platforms.

Am I missing something here?


So your example shows a user already comfortable in raising their 
recursion limit to work around needing more stack space to reach completion. 
What is stopping the user from continuing to raise the limit until they still 
reach their memory limit even with PEP 651? If you're worried about runaway 
recursion you will very likely hit that with the default stack depth already, 
so I personally don't see how a decoupled stack counter from the C stack 
specifically makes it any easier/better to detect runaway recursion. And if I 
need more recursion than the default, you're going to bump the recursion depth 
anyway, which weakens the protection in either the C or decoupled counter 
scenarios. Sure, it's currently platform-specific, but plenty of people want to 
push that limit based on their machine anyway and don't need consistency on 
platforms they will never run on, i.e. I don't see a huge benefit to being able 
to say that an algorithm consistently won't go past 5000 calls on
all platforms compared to what the C stack protection already gives us (not to 
say there's zero benefit, but it isn't massive or widespread either IMO). I personally 
just don't see many people saying, "I really want to limit my program to an exact 
call stack depth of 5000 on all platforms which is beyond the default, but anything under 
is fine and anything over -- regardless of what the system can actually handle -- is 
unacceptable".

Tack on the amount of changes required to give a cross-platform stack 
count and limit check compared to the benefit being proposed, and to me that 
pushes what the PEP is proposing into net-negative payoff.


To me, the point of that example is as a reminder that currently fiddling 
with the recursion limit can cause segfaults.

Mark's PEP proposes two, somewhat independent, changes: (1) don't consume C 
stack on pure Python-to-Python (pp2p) function calls; (2) implement fool-proof 
C stack overflow checks.

Change (2) makes it safe for users to mess with the stack overflow limit 
however they see fit. Despite (1), the limit for pp2p calls remains at 1000 so 
that users who unintentionally write some naively recursive code don't have to 
wait until they fill up all of memory before they get a traceback. (Of course 
they could also write a while-True loop that keeps adding an item to a list and 
they'd end up in the same situation. But in my experience that situation is 
less painful to deal with than accidental stack overflow, and I'd shudder at 
the thought of a traceback of a million lines.)

Given that we have (1), why is (2) still needed? Because there are ways to 
recursively call Python code that aren't pp2p calls. By a pp2p (pure 
Python-to-Python) call, I mean any direct call, e.g. a method call or a 
function call. But what about other operations that can invoke Python code? 
E.g. if we have a dict d and a class C, we could create an instance of C and 
use it to index d, e.g. d[C()]. This operation is not a p2pp call -- the 
BINARY_SUBSCR opcode calls the dict's `__getitem__` method, and that calls the 
key's `__hash__` method. Here's a silly demonstration:
```
def f(c):
     d = {}
     return d[c]

class C:
     def __hash__(self):
         return f(self)

f(C())
```
Note that the "traceback compression" implemented for simple recursive 
calls fails here -- I just ran this code and got 2000 lines of output.

The way I imagine Mark wants to implement pp2p calls means that in this 
case each recursion step *does* add several other C stack frames, and this 
would be caught by the limit implemented in (2). I see no easy way around this 
-- after all the C code involved in the recursion could be a piece of 3rd party 
C code that itself is not at fault.

So we could choose to implement only (2), robust C stack overflow checks. 
This would require a bunch of platform-specific code, and there might be 
platforms where we don't know how to implement this (I vaguely recall 

[Python-Dev] Re: Python 3.11.0a4 is blocked

[Mark Shannon]

Hi Pablo,

Issue 43683 should not have been marked as a release blocker, as it is a 
performance issue, not a bug.

The related bug (which was a release blocker) is 
https://bugs.python.org/issue46009, and was fixed a while ago.

Cheers,
Mark.

On 04/01/2022 11:12 pm, Pablo Galindo Salgado wrote:

Hi everyone,

I am writing this to notify you that unfortunately the release of 3.11.0a4 is 
blocked as there are a bunch of release blockers
(some of them affect Python 3.10):

https://bugs.python.org/issue46263 
https://bugs.python.org/issue46208 
https://bugs.python.org/issue46006 
https://bugs.python.org/issue43683 

If this was a single release blocker I would think about moving forward but 
unfortunately, there are several of them and one of
them is that Python fails to compile FreeBSD, so I am halting the release until 
these are fixed.

Regards from rainy London,
Pablo Galindo Salgado

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7OWGAL4UL5HNKKQFUGH6N6L4JDVPEN7R/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HHKP32L4VE6UCQKQRLV5POC7CAMHSNIT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Mark Shannon]

OOI, of those 1577 Callable type hints, how many distinct Callable types?


On 20/12/2021 7:52 am, Andrew Svetlov wrote:

At my job, we have 1577 Callable type hints scattered in 1063 Python files.
For comparison, this codebase also has 2754 dict annotations and 1835 list ones.

On Mon, Dec 20, 2021 at 8:11 AM Christopher Barker mailto:python...@gmail.com>> wrote:

note: I wasn't thinking -- typeshed, of course, has a lot more than the 
standard lib.  But it's still a collection of widely used somewhat general 
purpose libraries. So I think my hypothesis is still valid.

-CHB


On Sun, Dec 19, 2021 at 8:54 PM Christopher Barker mailto:python...@gmail.com>> wrote:

A question that came up for me is:

How common is it to need to use Callable for type hints? particularly 
complex versions, specifying what parameters the Callable takes? A more compact 
and easier to read syntax is nice, but not very important if it isn't used much.

My first thought on this was that I can't remember a single time that I 
wrote production code that took a Callable as a function parameter -- or 
returned one -- OK maybe a few times, but it's certainly rare in my production 
code.

So I looked in the PEP to see if that issue was addressed, and indeed 
it is:

"The Callable type is widely used. For example, as of October 2021 it was 
the fifth most common complex type in typeshed,"

That did surprise me, but on thinking about it, maybe not so much. It 
strikes me that Callable is most likely to be used in fairly low level, general 
purpose functions, like map(), sort(), various functions in itertools, etc. 
Just the sort of functions that are common in the standard library, but may not 
so much in production code.

I have no idea how to evaluate how common it is in production code -- 
maybe type hinting is common enough now  that PyPi could be searched -- but 
even PyPi is a biased sample, as it is full of, by definition, libraries for 
others' use -- i.e. general purpose tools (less general that the stad lib, but 
still not specialty production code, which I suspect is the majority of Python 
code out there).

Perhaps some folks that have been type=hinting their production code 
bases could provide anecdotal evidence.

Anyway, if my hypothesis is correct, then it's not so bad that 
not-so-nice syntax is required to type hint general purpose utilities.

-CHB

-- 
Christopher Barker, PhD (Chris)


Python Language Consulting
   - Teaching
   - Scientific Software Development
   - Desktop GUI and Web Development
   - wxPython, numpy, scipy, Cython



-- 
Christopher Barker, PhD (Chris)


Python Language Consulting
   - Teaching
   - Scientific Software Development
   - Desktop GUI and Web Development
   - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org 

To unsubscribe send an email to python-dev-le...@python.org 

https://mail.python.org/mailman3/lists/python-dev.python.org/ 

Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LWIXFDUGRM6Z3KHI3YGV65HWXRD2S4H5/
 

Code of Conduct: http://python.org/psf/codeofconduct/ 




--
Thanks,
Andrew Svetlov

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JYZGIDEBV4R5E7XXT3KFS2O545TDTAGT/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RQ32VDXY7KUXHCUM6DASPVZ434PN5GLM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Mark Shannon]

Hi,

Why not make Callable usable as a function decorator?



The motivating example in the PEP is this:


def flat_map(
l: list[int],
func: Callable[[int], list[int]]
) -> list[int]:



Since, as the PEP claims, `Callable[[int], list[int]]` is hard to read, then 
give it a name and use regular function definition syntax.


@Callable
def IntToIntFunc(a:int)->int:
pass


def flat_map(
l: list[int],
func: IntToIntFunc
) -> list[int]:



To me, this seems much clearer than the proposed syntax and is more general.


It is a little longer, but unless you are playing code golf, that shouldn't 
matter.

Cheers,
Mark.



On 16/12/2021 5:57 pm, Steven Troxler wrote:

Hello all,

Thanks everyone for comments on our earlier thread [1] about callable type 
syntax. We now  have a draft PEP [2] proposing an arrow-based syntax for 
callable types, for example:

```
(int, str) -> bool # equivalent to Callable[[int, str], bool]
```

In support of the PEP we also have:
- a reference implementation of the parser [3] to ensure the grammar is correct 
 (tests [5], [6], [7])
- a detailed specification of planned runtime behavior [4], which is not yet in 
the reference implementation

We'd like to get your feedback about the PEP in general, and especially details 
and edge cases we need to consider regarding runtime behavior.

Cheers,
Steven Troxler

-
[1] Earlier python-dev thread 
https://mail.python.org/archives/list/python-dev@python.org/thread/VBHJOS3LOXGVU6I4FABM6DKHH65GGCUB/
[2] PEP 677: https://www.python.org/dev/peps/pep-0677/
[3] Reference implementation of Parser: 
https://github.com/stroxler/cpython/tree/callable-type-syntax--shorthand
[4] Details on the runtime behavior:  
https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG71Lk/edit

[5] Ast tests for parser changes:
https://github.com/stroxler/cpython/blob/20eb59fdca0d6d8dbe4efa3b04038c7c22024654/Lib/test/test_ast.py#L359-L392
[6] Easy-read tests of examples from the PEP: 
https://github.com/stroxler/cpython/blob/callable-type-syntax--shorthand/Lib/test/test_callable_type_examples_for_pep.py
[7] Test sanity checking hundreds of examples pulled from typeshed:
https://github.com/stroxler/cpython/blob/callable-type-syntax--shorthand/Lib/test/test_callable_type_examples_for_pep.py
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OGACYN2X7RX2GHAUP2AKRPT6DP432VCN/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2BKD5YBU7WJMUY3TSX34HX5IICT5UFRQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [RELEASE] Python 3.11.0a3 is available

[Mark Shannon]

On 08/12/2021 11:51 pm, Pablo Galindo Salgado wrote:


* The [Faster Cpython Project](https://github.com/faster-cpython 
) is already yielding some exciting results: this 
version of CPython 3.11 is ~12% faster on the geometric mean of the [PyPerformance 
benchmarks](speed.python.org ), compared to 3.10.0.


Actually, it is quite a lot better than that at 19% on the standard benchmark 
suite :)

https://gist.github.com/markshannon/0ddfb0b705d23b863477d7f7f9f00ef1


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BDJFYAC5IL77MBYQMH56D7ST67GDBMN2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 669: Low Impact Monitoring for CPython

[Mark Shannon]

Hi,

I would like to announce latest PEP, PEP 669: Low Impact Monitoring for CPython.

The aim of this PEP is to provide an API for profilers, debuggers and other 
tools to avoid the punitive overhead of using sys.settrace.

If you have any interest in profilers, debuggers, coverage tools or anything of 
that ilk, then do please take a look.

There is no change to the language and it adds 7 functions to the sys module, 
so shouldn't be too intrusive for those of who aren't planning on implementing 
any of those tools.

As always, all feedback and comments are welcome.

You can read the PEP here:
https://python.github.io/peps/pep-0669/

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VNSD4TSAM2BM64FJNIQPAOPNEGNX4MDX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Optimizing literal comparisons and contains

[Mark Shannon]

Hi,

I am surprised by the insistence on this thread for excluding comparisons from 
constant folding.
Why should we special case comparisons? Am I missing something here?

We already constant fold a variety of expressions

0 * 7
'' * 7
True - True
True * False

(All the above are falsey)

Excluding  1 < 2 seems inconsistent.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BRQHDHZKNIMMXCA5SSAW2N3D57V2ZDZJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Optimizing literal comparisons and contains

[Mark Shannon]

On 28/11/2021 6:28 am, raymond.hettin...@gmail.com wrote:

For the benefit of the audience on python-dev, you should also mention that 
this proposal and associated PR has been twice discussed and rejected on the 
tracker:

https://bugs.python.org/issue45907
https://bugs.python.org/issue45843

The response just given by Skip pretty much matches the comments already given 
by Batuhan, Pablo, and Serhiy.  So far, no one who has looked at this thinks 
this should be done.


That is not entirely true: 
https://github.com/python/cpython/pull/29639#issuecomment-974146979
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YDNHIZQOW2IWPGKLFQ6Y5LLXD42TBQMN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 except* formatting

[Mark Shannon]

Another +1 for `except group` from me.

On 04/10/2021 2:57 pm, Ammar Askar wrote:

Throwing in another +1 for `except group`.

It's explicit, doesn't introduce new punctuation and avoids confusion 
with unpacking.


Regards,
Ammar

On Mon, Oct 4, 2021, 3:31 AM Antoine Pitrou > wrote:


On Sun, 3 Oct 2021 19:42:29 +0200
Łukasz Langa mailto:luk...@langa.pl>> wrote:
 >
 > -1
 >
 > If I could read the vertical line as a pipe character, the
expression would read "except or E as e".
 > But I can't read it that way anyway. Instead, all I see is a
lowercase EXCEPTL.
 >
 > My idea is this:
 >
 > try:
 >     ...
 > except group E as e:
 >     ...
 > except group E1, T2 as e:
 >     ...
 >
 > Should be doable given the magical match-case contextual keywords
precedent. This looks nice and is explicit, since you will always
get an ExceptionGroup instance under `e`.

+1.  This is much more helpful to the reader than the cryptic
asterisk.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/INK6TSOGGODA4NZ3CI5MOXIAI4Z4CZ53/


Code of Conduct: http://python.org/psf/codeofconduct/



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4ZUBUDQ4CGXYJAIYKMJMJBGUGGTODECF/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OFSIDJNKCXPXRJJNFDUG3JKNLPJUQGLD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] How to decipher Windows errors on builds?

[Mark Shannon]

Hi,

I'm getting build errors on Windows for 
https://github.com/python/cpython/pull/28386


The error message is:

C:\Program Files (x86)\Windows 
Kits\10\Include\10.0.22000.0\um\winnt.h(253): error RC2188: 
D:\a\cpython\cpython\PCbuild\obj\311win32_Release\pythoncore\RCa01688(47) 
: fatal error RC1116: RC terminating after preprocessor errors 
[D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]


Can anyone translate that into English for me?

It suggests that there are errors in the preprocessor. But how do I tell 
what the error is?


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SOCIYBNX5VZYPZFQYKWRCCPPN7TNVUMJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Mark Shannon]

Hi Guido,

On 23/08/2021 3:53 pm, Guido van Rossum wrote:
On Mon, Aug 23, 2021 at 4:38 AM Mark Shannon <mailto:m...@hotpy.org>> wrote:


Hi Nick,

On 22/08/2021 4:51 am, Nick Coghlan wrote:

 > If Mark's claim that PyEval_GetLocals() could not be fixed was
true then
 > I would be more sympathetic to his proposal, but I know it isn't
true,
 > because it still works fine in the PEP 558 implementation (it even
 > immediately sees changes made via proxies, and proxies see
changes to
 > extra variables). The only truly unfixable public API is
 > PyFrame_LocalsToFast().

You are making claims that seem inconsistent with each other.
Namely, you are claiming that:

1. That the result of locals() is ephemeral.
2. That PyEval_GetLocals() returns a borrowed reference.

This seems impossible, as you can't return a borrowed reference to
an emphemeral object. That's just a pointer to freed memory.

Do `locals()` and `PyEval_GetLocals()` behave differently?


That is my understanding, yes. in PEP 558 locals() returns a snapshot 
dict, the Python-level f_locals property returns a fresh proxy that has 
no state except a pointer to the frame, and PyEval_GetLocals() returns a 
borrowed reference to the dict that's stored on the frame's C-level 
f_locals attribute


Can we avoid describing the C structs in any of these PEPs?

It confuses readers having Python attributes and "C-level attributes"
(C struct fields?).
It also restricts the implementation unnecessarily.

(E.g. the PyFrameObject doesn't have a `f_locals` field in 3.11: 
https://github.com/python/cpython/blob/main/Include/cpython/frameobject.h#L7)




(In my "crazy" proposal all that is the same.)




Is the result of `PyEval_GetLocals()` cached, but `locals()` not?


I wouldn't call it a cache -- deleting it would affect the semantics, 
not just the performance. But yes, it returns a reference to an object 
that is owned by the frame, just as it does in 3.10 and before.


If that were the case, then it is a bit confusing, but could work.


Yes, see my "crazy" proposal.

Would PyEval_GetLocals() be defined as something like this?

(add _locals_cache attribute to the frame which is initialized to NULL).

def PyEval_GetLocals():
      frame._locals_cache attribute = locals()
      return borrow(frame._locals_cache attribute)


Nah, the dict returned by PyEval_GetLocals() is stored in the frame's 
C-level f_locals attribute, which is consulted by the Python-level 
f_locals proxy -- primarily to store "extra" variables, but IIUC in 
Nick's latest version it is also still used to cache by that proxy. 
Nick's locals() just returns dict(sys._getframe().f_locals).


The "extra" variables must be distinct from the result of locals() as
that includes both extras and "proper" variables.
If we want to cache the locals(), it needs to be distinct from the extra 
variables.


A debugger setting extra variables in a function that that is also 
accessed by a C call to PyEval_GetLocals() is going to be incredibly 
rare. Let's not worry about efficiency here.




None of this is clear (at least not to me) from PEP 558.


One problem with PEP 558 is that it's got too many words, and it's 
lacking a section that crisply describes the semantics of the proposed 
implementation. I've suggested to Nick that he add a section with 
pseudo-code for the implementation, like you did in yours.


(PS, did you read my PS about what locals() should do in class scope 
when __prepare__ returns a non-dict?)


Yes, but no harm in a reminder :)
I'll update my PEP to fix the semantics of locals().

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PA3ELKA4QLUV5VKEKUHZHRGT4P64EHGC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 667: Consistent views of namespaces

[Mark Shannon]

Hi Nick,

On 22/08/2021 4:51 am, Nick Coghlan wrote:



On Sun, 22 Aug 2021, 10:47 am Guido van Rossum, > wrote:


Hopefully anyone is still reading python-dev.

I'm going to try to summarize the differences between the two
proposals, even though Mark already did so in his PEP. But I'd like
to start by calling out the key point of contention.

Everything here is about locals() and f_locals in *function scope*.
(I use f_locals to refer to the f_locals field of frame objects as
seen from Python code.) And in particular, it is about what I'll
call "extra variables": the current CPython feature that you can add
*new* variables to f_locals that don't exist in the frame, for example:

def foo():
     x = 1
     locals()["y"] = 2  # or sys._getframe()["y"] = 2

My first reaction was to propose to drop this feature, but I realize
it's kind of important for debuggers to be able to execute arbitrary
code in function code -- assignments to locals should affect the
frame, but it should also be possible to create new variables (e.g.
temporaries). So I agree we should keep this.


I actually tried taking this feature out in one of the PEP 558 drafts, 
but actually doing so breaks the pdb test suite.





So apparently the key difference of opinion between Mark and Nick is
about f_locals, and what to do with extras. In Nick's proposal when
you reference f.f_locals twice in a row (for the same frame object
f), you get the same proxy object, whereas in Mark's proposal you
get a different object each time, but it doesn't matter, because the
proxy has no state other than a reference to the frame.


If PEP 558 is still giving that impression, I need to fix the wording - 
the proxy objects are ephemeral in both PEPs (the 558 text is slightly 
behind the implementation on that point, as the fast refs mapping is now 
stored on the frame object, so it only needs to be built once)


In Mark's proposal, if you assign a value to an extra variable, it
gets stored in a hidden dict field on the frame, and when you read
the proxy, the contents of that hidden dict field gets included.
This hidden dict lazily created on the first store to an extra
variable. (Mark shows pseudo-code to clarify this; the hidden dict
is stored as _extra_locals on the frame.)


PEP 558 works essentially the same way, the difference is that it uses 
the existing locals dict storage rather than adding new storage just for 
optimised frames.


In Nick's proposal, there's a cache on the frame that stores both
the extras and the proper variables. This cache can get out of sync
with the contents of the proper variables when some bytecode is
executed (for performance reasons we don't want the bytecode to keep
the cache up to date on every store), so there's an operation to
sync the frame cache (sync_frame_cache(), it's not defined in which
namespace this exists -- is it a builtin or in sys?).


It's an extra method on the proxy objects. You only need it if you keep 
an old proxy object around - if you always retrieve a new proxy object 
after executing Python code, that proxy will refresh the cache when it 
needs to.




Frankly the description in Nick's PEP is hard to follow -- I am not
100% sure what is meant by "the dynamic snapshot", and it's not
quite clear whether proper variables are copied into the cache (and
if so, why).


Aye, Mark was a bit quicker with his PEP than I anticipated, so I've 
incorporated the implementation improvements arising from his last round 
of comments, but the PEP text hasn't been updated yet.



Personally, I find Mark's proposed semantics for f_locals simpler --
there's no cache, only storage for extras, so there's nothing that
can get out of sync.


The wording in PEP 667 undersells the cost of that simplification:

"Code that uses PyEval_GetLocals() will continue to operate safely, but 
will need to be changed to use PyEval_Locals() to restore functionality."



Code that uses PyEval_GetLocals() will NOT continue to operate safely 
under PEP 667: all such code will raise an exception at runtime, and 
need to be rewritten to use a new API with different refcounting 
semantics. That's essentially all code that accesses the frame locals 
from C, since we don't offer supported APIs for that other than 
PyEval_GetLocals() (directly accessing the f_locals field on the frame 
object is only "supported" in a very loose sense of the word, although 
PEP 558 mostly keeps that working, too)


This means the real key difference between the two PEPs is that Mark is 
proposing a gratuitous compatibility break for PyEval_GetLocals() that 
also means that the algorithmic complexity characteristics of the proxy 
implementation will be completely off from those of a regular dict (e.g. 
len(proxy) will be O(n) in the number of variables defined on the fra

[Python-Dev] PEP 667: Consistent views of namespaces

[Mark Shannon]

Hi all,

I have submitted PEP 667 as an alternative to PEP 558.
https://www.python.org/dev/peps/pep-0667/

Nick and I have agreed to disagree on the way to fix locals() and 
f_locals. We are both in agreement that it needs fixing.


In summary, PEP 667 has roughly the same surface behavior as PEP 558 but 
is simpler and more consistent internally, at the expense of some minor 
C API backwards incompatibility issues.


PEP 558 also has backwards incompatibility issues, but claims to be more 
compatible at the C API level.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4RH5YCXIHIP6MRVTCOKOOO4GKCIMH4GJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

[Mark Shannon]

On 10/08/2021 4:25 am, Inada Naoki wrote:

On Tue, Aug 10, 2021 at 12:30 AM Eric V. Smith  wrote:


Personally, I'd like to see PEP 649 accepted. There are a number of
issues on bpo where people expect dataclass's field.type to be an actual
Python object representing a type, and not a string. This field is
copied directly from __annotations__. If we stick with PEP 563's string
annotations, I'll probably just close these issues as "won't fix", and
tell the caller they need to convert the strings to Python objects
themselves. If 649 is accepted, then they'll all get their wish, and in
addition I can remove some ugly logic from dataclasses.



I don't think there is much difference.

PEP 563 is not default. And PEP 563 is not the only source of
stringified annotation.
So Accepting PEP 649 doesn't mean "they'll all get their wish". We
need to say "won't fix" anyway.



Do we need to do anything here to move forward on this issue? I've
chatted with Larry and Mark Shannon, who have some additional thoughts
and I'm sure will chime in.


My only comment concerned performance.

I won't claim that the cost of PEP 649 will be zero, but it won't be 
significantly more than PEP 563 if annotations are unused.
The cost of unmarshalling a code object will be greater than a string, 
but it won't be significant. Not only that, but we are actively looking 
to reduce startup in 3.11 which will reduce the overhead further.


If annotations *are* used, then PEP 649 should be cheaper as it relies 
on the interpreter to do the evaluation in an efficient fashion.
For users of dataclasses and Pydantic, I expect PEP 649 to outperform 
PEP 563.






Currently, reference implementation of PEP 649 has been suspended.
We need to revive it and measure performance/memory impact.

As far as I remember, the reference implementation created a function
object for each methods.


No function object is created under normal circumstances.
__annotations__ is a property that calls the underlying 
__co_annotations__ property, which lazily creates a callable.


I'll leave it to Larry to explain why __co_annotations__ isn't just a 
code object.



It means doubles function objects. It has major impact to memory
usage, startup time, and GC time.


Only if __annotations__ are widely used, in which case PEP 563 is 
probably worse.


Cheers,
Mark.



There was an idea to avoid creating function objects for most cases.
But it was not implemented.

Regards,


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BIM5KHILDVFVTIL6SUIY6TGGV5SHG5MQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 558, the simplest thing I could come up with

[Mark Shannon]

Hi Nick,

Our discussion on PEP 558 got me thinking
"What is the simplest thing that would work?".

This is what I came up (in the form of a draft PEP):
https://github.com/markshannon/peps/blob/pep-locals/pep-06xx.rst

It doesn't have O(1) len(f_locals), and it does break 
`PyEval_GetLocals()` but I think the that is a small price to pay for 
simplicity and consistency.


What do you think?

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FCWMWQ7TCSHE7LIXRMMR3Z37LJKIOO5J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Repealing PEP 509 (Add a private version to dict)

[Mark Shannon]

Hi Steve,

On 29/07/2021 1:55 pm, Steve Dower wrote:

On 7/29/2021 11:41 AM, Mark Shannon wrote:
The dictionary version number is currently unused in CPython and just 
wastes memory. I am not claiming that we will never need it, just that
we shouldn't be required to have it. It should be an internal 
implementation detail that we can add or remove depending on 
requirements.


Sounds reasonable.

Maybe we should have a "Type" other than Standards Track for PEPs that 
are documenting implementation designs, rather than requirements for 
standardisation?


That makes sense to me.

There a few PEPs that fall into that category, but not many.
E.g. 393, 412, 509, 590, 659.




If/when we ever get to a point where other implementations want to claim 
to implement "standard" Python, having an explicit distinction here 
would be helpful.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3UZZCTFF2PPLS4FCZSRHUDSD6F3YBTVJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Repealing PEP 509 (Add a private version to dict)

[Mark Shannon]

Hi everyone,

I would like to repeal PEP 509. We don't really have a process for 
repealing a PEP. Presumably I would just write another PEP.


Before I do so, I would like to know if anyone thinks we should keep
PEP 509.

The dictionary version number is currently unused in CPython and just 
wastes memory. I am not claiming that we will never need it, just that
we shouldn't be required to have it. It should be an internal 
implementation detail that we can add or remove depending on requirements.


Thoughts?

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GV4CW3T7SUTJOYSCP6IJMV4AHDNNZIPV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Mark Shannon]

On 28/07/2021 1:03 am, Nick Coghlan wrote:



On Wed, 28 Jul 2021, 1:50 am Mark Shannon, <mailto:m...@hotpy.org>> wrote:


Hi Nick,

On 27/07/2021 2:29 pm, Nick Coghlan wrote:
 >
 >
 > The reference documentation should be precise as well, since that is
 > what other implementations will be following.
 >
 > What semantics do you feel are left unspecified?
 >
 >> The documentation needs to explain the behavior to the majority of
 >> users, but the PEP should be worded so that it can be implemented
 >> correctly from just reading the PEP.
 >>
 >> There is no reason to remove the sugested documentation changes, but
 >> they should be secondary to the more formal specification.
 >
 > I don't know what you want to see on this front.
 >
 > The current PEP seems complete to me, except in relation to the
 > specifics of the proxy behaviour, which it deliberately leaves
 > underspecified (mostly because my current implementation sucks in a
 > few areas, and I really don't want to enshrine those limitations in
 > the language spec).

All the above were mostly just suggestions, the format of the PEP is up
to you.

But please, do not underspecify anything. I need to maintain this,
as do
the PyPy and MicroPython developers. We need this to be precise.


You still haven't said what you consider underspecified.


You said it was underspecified, not me. Quoting you from a few lines up
"which it deliberately leaves underspecified"



Perhaps read the implementation and tell me which parts of what the 
tests are covering you want to see replicated in the PEP text?



 >
 >> Resolving the issues with tracing mode behaviour
 >> 
 >>
 >> According to this section the `f_locals` object
(_PyFastLocalsProxy_Type
 >> in C) will have *two* fields.
 >> It should only have *one*, the pointer to the underlying frame
object.
 >
 > That's a good point - fast_refs doesn't reference the frame
object, so
 > it's safe to store it on the frame object and reference it from the
 > individual proxies.
 >
 > Switching to that approach also has the advantage that every proxy
 > after the first is much cheaper, since they don't need to build the
 > fast refs map again.
 >
 >> In order to maximize backwards compatibility and, more importantly,
 >> avoid the synchronization issues that lead to obscure bugs, all the
 >> state of the `f_locals` object should be kept on the frame.
 >>
 >> Any changes to `f_locals` should be instantly visible to any
thread both
 >> through other `f_locals` objects (for the same frame) and the
underlying
 >> local variable (if it exists).
 >
 > [snip proxy method implementation sketch] >
 > This gets the gist of how the implementation works, but it isn't
quite
 > that simple for a few reasons:
 >
 > 1. The proxy needs to offer the same algorithmic complexity as dict
 > APIs, so doing O(n) linear searches of C arrays and Python tuples
 > inside O(1) lookup operations isn't OK (hence the fastrefs mapping to
 > allow for O(1) resolution of variable names to cell references and
 > local variable array offsets)

Linear searches are faster for small arrays, and there is nothing
stopping you adding a helper map of names->indexes for large functions.
Provided the mapping from name to local index is O(1), then the whole
operation is O(1).


That helper map already exists in the fast refs mapping. Skipping it for 
small functions would be a future optimisation opportunity that doesn't 
seem necessary in the initial implementation.


fastref is a mapping from names to values. I am suggesting a mapping 
from names to indexes in the locals array. That is not the same thing.
The mapping from names to indexes is fixed at compile time and cannot 
get out of sync.





You really don't need much extra machinery to maintain O(1) behavior
(which no one cares about, they just care about overall performance).


Yes, the only extra machinery needed for working with individual keys is 
the fast refs mapping.


It's other aspects of the mapping API that benefit from the continuing 
presence of the  frame cache (which is maintained on the frame, just as 
it is today, NOT separately in each proxy).


If you can give me a reliable O(1) implementation of 
"len(frame.f_locals)" that doesn't rely on up to date cache of:

* which local and cell variable names are currently bound to values
* which

[Python-Dev] Re: Comments on PEP 558

[Mark Shannon]

Hi Nick,

On 27/07/2021 2:29 pm, Nick Coghlan wrote:

(I'm not sure Mailman will get the threading right when I've received
the original email directly, so apologies in advance if this reply
creates a 2nd thread)
On Mon, 26 Jul 2021 at 23:57, Mark Shannon  wrote:


Hi,

First of all let me say that I agree with the aims of PEP 558 and most
of the design.
I do find the wording of the PEP itself a bit confusing in a couple of
places.

This critique is based on my understanding of the PEP.
If I am mistaken in my misunderstanding, then treat that as an implied
request for clarification of the PEP :)


Critique of PEP 558
===

Layout of the PEP
-

[This is largely to help the reader, it doesn't change the nature of the
PEP]

Could we replace the section "CPython Implementation Changes" with a
section that states what the behavior will be, not how it changes.
Having to mentally apply the suggested changes to the existing (and
convoluted) implementation makes it hard to work out what the proposed
behavior will be.


Just stating what the new behaviour is isn't sufficient to assess the
impact of the change, though - it needs to be compared to the old
(arcane) behaviour, and I don't want to assume readers already
understand what that behaviour is.

That said, the summary section is intended to describe how things will
work post-change. It just leaves out the details of the new proxy
implementation (which I admit is the most complicated part of the
updated implementation).


Could the design discussion be moved to an appendix or another document?
The key parts of it should be moved to the Rationale or Motivation.


It's effectively an Appendix already (the only things after it are the
pointer to the implementation and the acknoweldgements).

That said, I'll note that several of the questions you've asked in
this email are answered directly in the Design Discussions section
that you're suggesting removing from the PEP.


There should be a Motivation section explaining why this PEP is
necessary, for those not familiar with the weirdness of `locals()`.
Much of what is in the Rationale should perhaps be in the Motivation.


Yes, the Rationale section could be retitled Motivation - it's the
rationale for the PEP existing, not the rationale for the design
decisions (those are in the Design Discussion section).

The historical arcane behaviour at function scope is covered in
https://www.python.org/dev/peps/pep-0558/#historical-semantics-at-function-scope
but the bug references seemed more important for the Motivation
section (if the status quo *worked*, I'd never have tried to work out
how to change it, but with the status quo both arcane *and* broken, it
makes sense to try to figure out a better alternative).



Proposal


[Ditto; this is largely to help the reader, it doesn't change the nature
of the PEP]

Drop the definitions of the type of scope. They are (at least they
should be) clearly defined in existing documentation.


They're not, unfortunately. As far as I am aware,
https://docs.python.org/3/reference/executionmodel.html#naming-and-binding
is as good as we've currently got, so


Why "largely" eliminate the concept of a separate tracing mode? Wasn't
the plan to eliminate it entirely?


The "largely" there just relates to the fact that even though the PEP
eliminates the side effects that tracing mode has historically had on
the behaviour of locals(), the core eval loop still has the notion of
"tracing or not" that turns off some of the execution shortcuts it can
otherwise take.


Rather than specifying what the documentation will be, could you specify
the semantics. The language here should be precise.


The reference documentation should be precise as well, since that is
what other implementations will be following.

What semantics do you feel are left unspecified?


The documentation needs to explain the behavior to the majority of
users, but the PEP should be worded so that it can be implemented
correctly from just reading the PEP.

There is no reason to remove the sugested documentation changes, but
they should be secondary to the more formal specification.


I don't know what you want to see on this front.

The current PEP seems complete to me, except in relation to the
specifics of the proxy behaviour, which it deliberately leaves
underspecified (mostly because my current implementation sucks in a
few areas, and I really don't want to enshrine those limitations in
the language spec).


All the above were mostly just suggestions, the format of the PEP is up 
to you.


But please, do not underspecify anything. I need to maintain this, as do 
the PyPy and MicroPython developers. We need this to be precise.





Resolving the issues with tracing mode behaviour


According to this section the `f_l

[Python-Dev] Comments on PEP 558

[Mark Shannon]

Hi,

First of all let me say that I agree with the aims of PEP 558 and most 
of the design.
I do find the wording of the PEP itself a bit confusing in a couple of 
places.


This critique is based on my understanding of the PEP.
If I am mistaken in my misunderstanding, then treat that as an implied 
request for clarification of the PEP :)



Critique of PEP 558
===

Layout of the PEP
-

[This is largely to help the reader, it doesn't change the nature of the 
PEP]


Could we replace the section "CPython Implementation Changes" with a 
section that states what the behavior will be, not how it changes.
Having to mentally apply the suggested changes to the existing (and 
convoluted) implementation makes it hard to work out what the proposed 
behavior will be.


Could the design discussion be moved to an appendix or another document? 
The key parts of it should be moved to the Rationale or Motivation.
There should be a Motivation section explaining why this PEP is 
necessary, for those not familiar with the weirdness of `locals()`.

Much of what is in the Rationale should perhaps be in the Motivation.

Proposal


[Ditto; this is largely to help the reader, it doesn't change the nature 
of the PEP]


Drop the definitions of the type of scope. They are (at least they 
should be) clearly defined in existing documentation.


Why "largely" eliminate the concept of a separate tracing mode? Wasn't 
the plan to eliminate it entirely?


Rather than specifying what the documentation will be, could you specify 
the semantics. The language here should be precise.
The documentation needs to explain the behavior to the majority of 
users, but the PEP should be worded so that it can be implemented 
correctly from just reading the PEP.


There is no reason to remove the sugested documentation changes, but 
they should be secondary to the more formal specification.



Resolving the issues with tracing mode behaviour


According to this section the `f_locals` object (_PyFastLocalsProxy_Type 
in C) will have *two* fields.

It should only have *one*, the pointer to the underlying frame object.

In order to maximize backwards compatibility and, more importantly, 
avoid the synchronization issues that lead to obscure bugs, all the 
state of the `f_locals` object should be kept on the frame.


Any changes to `f_locals` should be instantly visible to any thread both 
through other `f_locals` objects (for the same frame) and the underlying 
local variable (if it exists).


E.g.

def __getitem__(self, name):
f = self.frame
if name in f.f_code.co_varnames:
index = f.f_code.co_varnames.index(name)
obj = f.locals[index]  # f.locals refers to the fast array of 
variables

if obj is NULL:
raise KeyError(name)
return obj
else:
return f._extra_locals[name]

def __setitem__(self, name, value):
f = self.frame
if name in f.f_code.co_varnames:
index = f.f_code.co_varnames.index(name)
CLEAR(f.locals[index])
f.locals[index] = value
else:
f._extra_locals[name] = value

def items(self):
f = self.frame
for index, name in enumerate(f.f_code.co_varnames)
obj = f.locals[index]
if obj is not NULL:
yield name, obj
yield from f._extra_locals.items()

Where `_extra_locals` is a normal dictionary that is not visible to 
either Python or the C API.


C API changes
-

The PEP suggests adding four new functions to the stable API.
Then in the "Changes to the public CPython C API" section, another five 
functions are added for a total of nine new functions!


At the Python level, there are two ways to access a locals mapping.
1. locals()
2. frame.f_locals

We only need two C functions, one for each of the above.

`PyEval_GetLocals()` should be equivalent to `locals()`.
It will have to return a new reference. It cannot return a borrowed 
reference as it would be returning freed memory.

There is nothing to borrow the reference from, for a function scope.

`PyFrame_GetLocals(PyFrameObject *)` should be equivalent to 
`frame.f_locals`.


The PEP doesn't explicitly state why `PyLocals_GetKind()` is needed, but 
I believe it is avoid unnecessary copying?
Why is creating an extra copy an issue? It takes a microsecond or so to 
create the copy.


If a function that returns a copy of the local namespace is really 
needed, then why not offer that functionality directly?

E.g. `PyFrame_GetLocalsCopy()` which is roughly:

PyObject *locals = PyEval_GetLocals();
if (current_scope_is_function())
return locals;
return PyDict_Copy(locals); // This leaks, need to decref locals



Reducing the runtime overhead of trace hooks


I'm confused by this part.
Since `_PyFrame_FastToLocals` and friends do not alter the logical state 
of the frame object (or anything else), they shou

[Python-Dev] Critique of PEP 657

[Mark Shannon]

Hi,

Apologies for my tardiness in doing this, but no one explicitly said it 
was too late to critique PEP 657...



Critique of PEP 657 (Include Fine Grained Error Locations in Tracebacks)


First of all I want to say that I support the goal of improving error 
messages. IMO the PEP should be "accepted in principle". I think all of 
the issues below can be fixed while still supporting the general aims of 
the PEP.



The change from points to ranges as locations
-

Because Python is a procedural language, there is an expectation that 
code executes in a certain order. PEP 626 (Precise line numbers for 
debugging and other tools) seeks to guarantee that that expectation is met.


PEP 657 describes how locations for exceptions are to be handled, but is 
vague on the treatment of locations for tracing, profiling and debugging.


PEP 657 proposes that locations for exceptions be treated as ranges, 
whereas tracing, profiling and debugging currently treat locations as 
points.


Either this will end in contradictions and confusion should those 
locations disagree, or the locations for tracing, profiling and 
debugging must change.


Using the start of a range as the point location for tracing may be 
misleading when the operation that causes an exception is on a different 
line within that range.


Consider this example:
https://github.com/python/cpython/blob/main/Lib/test/test_compile.py#L861

This might seem like a contrived case, but it is based on a real bug 
report https://bugs.python.org/issue39316


1.  def load_method():
2.  return (
3.  o.
4.  m(
5.  0
6.  )

Currently the call is traced on line 4.

PEP 657 would change the location of the call from line 4 to the range 
3-6, which would mean that the line of call is no longer traced 
separately (or traced several times). PEP 657 makes no mention of this 
change.


The PEP claims that these changes are improvements. Maybe they are, but 
they are quite impactful changes which the PEP glosses over. The impact 
on tools like coverage.py and debuggers should be made clearer. For 
example, how would one set a breakpoint on line 4 above?


There are other languages (e.g. jinja templates) that compile to Python 
AST and bytecode. These *might* produce locations that overlap, but are 
not nested. The behavior of tracing and debuggers needs to be described 
for those locations.


Backwards Compatibility
---

PEP 657 claims it is fully backwards compatible, but it cannot be both 
backwards compatible and consistent.
There are fundamental differences between using ranges and points as 
locations.


Impact on startup time
--

The PEP 657 suggests the impact on startup would be negligible. That is 
not quite true. The impact on startup is probably acceptable, but a 
proper analysis needs to be made.


The increase in size of pyc files ~20% puts an upper bound on the 
increase of startup time, but I would expect it to be much less than 
that as loading files from disk is only a fraction of startup.


Currently, startup is dominated by inefficiencies in interpreter 
creation, unmarshalling and module loading.
We plan to reduce these a lot for 3.11, so that the impact of PEP 657 on 
startup will be larger (as a ratio) than experiments with 3.10 suggest.


The API
---

The C API adds three new functions, one each for the end line, start 
column and end column.
This is either slow, as any compressed table needs to be parsed four 
times, or space inefficient using an uncompressed table.


Opt-out
---

Allowing opt-out prevents consistent compression of location data, 
resulting in larger pyc files for those that do not opt-out.
The exact semantics, in terms of error formatting, tracing, etc is not 
described should the user opt-out.


Summary
---

Overall, there is nothing that blocks acceptance of the PEP in 
principle, but there are quite a few issues that need resolving.



Suggestions
---

1. Clarify, in detail, the impact on line-based tools like profilers, 
coverage.py and debuggers. This should include help on how to use the 
new APIs and where using the old APIs might result in behavioral changes.


2. Change the C API to a single function:
int PyCode_Addr2Location(PyCodeObject *co, int addr, int *startline, int 
*startcolumn, int *endline, int *endcolumn)


3. Drop the opt-out option.
If the extra information is optional, then the compression scheme must 
allow for that; making the code more complex and potentially less 
efficient. Does opting out use the start of the range, or the old line, 
as the location?


4. Drop the limitation on column offsets.
The data needs to be compressed anyway, so allowing arbitrary column 
offsets is effectively free.


6. Store all location information in a single table (this applies more 
to the implementation than 

[Python-Dev] Is it too late to critique PEP 657? (Include Fine Grained Error Locations in Tracebacks)

[Mark Shannon]

Hi,

I was expected the announcement of a BDFL delegate for PEP 657, as the 
author is a steering council member.


It seems that a PEP submitted by a SC member has been accepted by the SC 
with seemingly no external review.


PEP 657 is likely to cause significant worsening of start up time and 
interact poorly with debugging and profiling.



Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OPTIO7SWLTEZCWMAHVDKF7WUX2LSQ46X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

[Mark Shannon]

Hi Brett,

On 26/05/2021 3:56 am, Brett Cannon wrote:



On Tue., May 25, 2021, 12:58 Guido van Rossum, > wrote:


On Tue, May 25, 2021 at 12:34 PM Brett Cannon mailto:br...@python.org>> wrote:


I personally think it should be a Standards Track PEP. This PEP
isn't documenting some detail like PEP 13 or some release
schedule, but is instead proposing a rather major change to the
interpreter which a lot of us will need to understand in order
to support the code (and I do realize the entire area of "what
requires a PEP and what doesn't" is very hazy).


Does that also mean you think the design should be completely hashed
out and approved by the SC ahead of merging the implementation?
Given the amount of work, that would run into another issue -- many
of the details of the design can't be fixed until the implementation
has proceeded, and we'd end up with a long-living fork of the
implementation followed by a giant merge. My preference (and my
promise at the Language Summit) is to avoid mega-PRs and instead
work on this incrementally.

Now, we've done similar things before (for example, the pattern
matching implementation was a long-living branch), but the
difference is that for pattern matching, the implementation followed
the design, whereas for the changes to the bytecode interpreter that
we're undertaking here, much of the architecture will be designed as
the implementation proceeds, based on what we learn during the
implementation.

Or do you think the "Standards Track" PEP should just codify general
agreement that we're going to implement a specializing adaptive
interpreter, with the level of detail that's currently in the PEP?


This. Having this as an informational PEP that's already marked as 
Active seems off somehow to me. I guess it feels more "we're doing this" 
(which I know isn't intended) rather than "this is our plan, what do you 
all think? All good?"


The PEP is a "we're doing this" document. Maybe it shouldn't be a PEP at 
all? I've changed its status to "draft" for now.


I want to document what we are doing as publicly as possible and a PEP 
seems like a good way to do that.


I also want to reiterate that the PEP doesn't propose changing the 
language, libraries, Python API or C API in any way. It is just 
information about how we plan to speed up the interpreter.





I don't recall other standards track PEPs that don't also spell out
the specification of the proposal in detail.


I also am not aware of a PEP that's proposed restructuring the eval loop 
like this either. 😉 I'm personally fine with the detail and saying 
details may shift as things move forward and lessons are learned based 
on the scope and updating the PEP accordingly. But that's just me and I 
don't know if others agree (hence the reason I'm suggesting this be 
Standards Track).


Suppose it were a standards PEP, what would that mean if it were rejected?
Rejection of a PEP is a choice in favor of an alternative, but what is 
that alternative?
You can't simply say the "status quo" as that would implicitly prevent 
any development at all on the bytecode interpreter.



Cheers,
Mark.


p.s.

For those not at the language summit, here's my grand plan for CPython:
https://docs.google.com/presentation/d/1_cvQUwO2WWsaySyCmIy9nj9by4JKnkbiPCqtluLP3Mg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3FTLC7G5A4FA3MZDWX5W2MWDFCPBXSCA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Mark Shannon]

Hi,

On 17/05/2021 5:22 pm, Ammar Askar wrote:
>> While nicer locations for errors is great, it won't be popular if it has
>> a negative impact on performance.
>> Locations need to tracked through the compiler.
>
> In performance sensitive contexts won't most code be pre-compiled into
> pyc files anyway? I feel like the performance cost of accurate column
> tracking in the compiler isn't too big of a concern unless I'm missing
> something.
>

The cost I'm concerned about is the runtime cost of worse code, because 
the compiler can't perform some optimizations due the constraints of 
providing the extended debug information.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IXXOBTZJQEVF6EZP5ACQNKTN7RVDQ7SI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Mark Shannon]

Hi everyone,

I fully agree with the rationale for the PEP, better error messages 
always help.
However, I think the proposed implementation could be misleading in some 
cases and wastes memory.



Use one position, not one-and-a-half positions.
---

The main problem with PEP 657, IMO, is that it wants to present the 
location of an error as a pair of positions, but without the line number 
of the second position.

Consequently it ends up with one and a half positions, not two.
This does not work well for a number of reasons.

1.  Errors spanning multiple lines.

Consider:

(i1 + i2 +
 s1
 )

Where i1, i2 are integers and s1 is a string.

With a single location, the error points to the second `+`. PEP 657 
would highlight the whole line, `i1 + i2 +`, making it unclear where the 
error occurred.



2. Repeated binary operations on the same line.

A single location can also be clearer when all the code is on one line.

i1 + i2 + s1

PEP 657:

i1 + i2 + s1


Using a single location:

i1 + i2 + s1
^

3. Tracking locations in the compiler.

While nicer locations for errors is great, it won't be popular if it has 
a negative impact on performance.
Locations need to tracked through the compiler. The simpler the 
location, the easier this is to do correctly without a negative 
performance impact.
It is already tricky to do this correctly with just line numbers because 
we have both source that compiles to no bytecodes (try, pass) and 
bytecodes that have no source (implicit return None and except cleanups).



A single location can still be presented as a whole token, as tokenizing 
a single line of code is easy and fast. So when presenting an error, the 
whole token can be highlighted.


E.g:

NameError
name


Compression
---

PEP 657 proposes that no compression be used, but also mandates the use 
of a lossy compression scheme (truncating all offsets over 255).
I think it would be better to provide an API like PEP 626 and not 
restrict the internal format used. In fact, extending or replacing the 
API of PEP 626 seems the best approach to me.


I wouldn't worry about memory consumption.
The code object layout and unmarshalling process needs to be 
re-implemented for reduced memory use and faster startup: 
https://github.com/markshannon/faster-cpython/blob/master/tiers.md#tier-0

A few bytes more or less in the location table(s) is inconsequential.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

[Mark Shannon]

Hi Fabio,

On 13/05/2021 7:11 pm, Fabio Zadrozny wrote:


Em qua., 12 de mai. de 2021 às 14:45, Mark Shannon <mailto:m...@hotpy.org>> escreveu:


Hi everyone,

I would like to present PEP 659.

This is an informational PEP about a key part of our plan to improve
CPython performance for 3.11 and beyond.

For those of you aware of the recent releases of Cinder and Pyston,
PEP 659 might look similar.
It is similar, but I believe PEP 659 offers better interpreter
performance and is more suitable to a collaborative, open-source
development model.

As always, comments and suggestions are welcome.


Hi Mark,

I think this seems like a nice proposal... I do have some questions 
related to the PEP though (from the point of view of implementing a 
debugger over it some things are kind of vague to me):


1. When will the specialization happen? (i.e.: is bytecode expected to 
be changed while the code is running inside a frame or must it all be 
done prior to entering a frame on a subsequent call?)


The specialization is adaptive, so it can happen at anytime during 
execution.




2. When the adaptive specialization happens, will that be reflected on 
the actual bytecode seen externally in the frame or is that all 
internal? Will clients be able to make sense of that? -- i.e.: In the 
debugger right now I have a need on some occasions to detect the 
structure of the code from the bytecode itself (for instance to detect 
whether some exception would be handled or unhandled at raise time just 
given the bytecode).


The bytecode, as externally visible, will be unchanged. All 
specializations will be internal and should be invisible to all Python 
tools.




3. Another example: I'm working right now on a feature to step into a 
method. To do that right now my approach is:
     - Compute the function call names and bytecode offsets in a given 
frame.
     - When a frame is called (during a frame.f_trace call), check the 
parent frame bytecode offset (frame.f_lasti) to detect if the last thing 
was the expected call (and if it was, break the execution).


This seems reasonable given the current implementation, where bytecodes 
are all fixed and there's a mapping from the frame.f_lasti ... Will that 
still work with the specializing adaptive interpreter?


If you are implementing this in Python, then everything should work as 
it does now.


OOI, would inserting a breakpoint at offset 0 in the callee function
work?



4. Will it still be possible to change the frame.f_code prior to 
execution from a callback set in `PyThreadState.interp.eval_frame` 
(which will change the code to add a breakpoint to the bytecode and 
later call `_PyEval_EvalFrameDefault`)? Note: this is done in the 
debugger so that Python can run without any tracing until the breakpoint 
is hit (tracing is set afterwards to actually pause the execution as 
well as doing step operations).


Since frame.f_code is read-only in Python, I assume you mean in C.

I can make no guarantees about the layout or meaning of fields in the C 
frame struct, I'm afraid.

But I'm sure we can get something to work for you.

Cheers,
Mark.



Best regards,

Fabio




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JLD3PDRU6YXPIGXVUDE3JP4EEBI2PWJ7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

[Mark Shannon]

Hi Terry,

On 13/05/2021 8:20 am, Terry Reedy wrote:

On 5/12/2021 1:40 PM, Mark Shannon wrote:

This is an informational PEP about a key part of our plan to improve 
CPython performance for 3.11 and beyond.


What is the purpose of this PEP?  It seems in part to be like a 
Standards Track PEP in that it proposes a new (revised) implementation 
idea for the CPython bycode interpreter.  Do you not intend this to not 
constitute approval of even the principle?


I will make it a standards PEP if anyone feels that would be better.
We can implement PEP 659 incrementally, without any large changes to the 
implementation or any to the language or API/ABI, so a standards PEP 
didn't seem necessary to us.


However, because it is a large change to the implementation, it seemed 
worth documenting and doing so in a clearly public fashion. Hence the 
informational PEP.




One of the issues in the new project gave formulas for the cost versus 
benefit calculations underlying specialization.  Depending on your 
purpose, it might be good to include them.  They certainly gave some 
clarity to me.




Which ones in particular? I can add something like them to the PEP.

Cheers,
Mark.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VY3JB3XO4D2E65ZR5IZUDP7MFQJ3JXIF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

[Mark Shannon]

Hi Terry,

On 13/05/2021 5:32 am, Terry Reedy wrote:

On 5/12/2021 1:40 PM, Mark Shannon wrote:

This is an informational PEP about a key part of our plan to improve 
CPython performance for 3.11 and beyond.



As always, comments and suggestions are welcome.


The claim that starts the Motivation section, "Python is widely 
acknowledged as slow.", has multiple problems. While some people 
believe, or at least claim to believe "Python is slow", other know that 
as stated, the latter is false.  Languages do not have a speed, only 
implementations running code for particular applications have a speed, 
or a speed relative to equivalent code in another language with a 
different runtime.


I broadly agree, but CPython is largely synonymous with Python and 
CPython is slower than it could be.


The phrase was not meant to upset anyone.
How would you rephrase it, bearing in mind that needs to be short?



I reason I am picking on this is that the meme 'Python is slow' is being 
morphed into 'Python is destroying the earth' (and should be abandoned, 
if not banned).  Last fall, a science news journal (Nature News?) quoted 
a 'concerned scientist' saying just this.  An internet troll repeated it 
last week on comp.lang.python (from where it leaked onto python-list).


It is a legitimate concern that CPython is bad for the environment, and 
one that I hope we can address by speeding up CPython.


Since, faster == less energy for the same amount of work, making CPython 
faster will reduce the amount of CO2 produced to do that work and 
hopefully make it less of a concern.


Of course, compared to the environmental disaster that is BitCoin, it's 
not a big deal.




It is true that Python has characteristics that make it *relatively* 
difficult to write interpreters that are *relatively* fast in certain 
applications.  But the opposite is also true.  The language does *not* 
mandate that objects, their methods, and modules be written in the 
language.


Hence, CPython implements builtin objects and function and some stdlib 
modules in C and allows 3rd party modules written in C or C++ or 
Fortran. I believe the first killer app for Python, in the mid 1990s, 
numerical computing with NumericalPython.  Rather than being 'slow', 
CPython *enabled* people, with a few percent of added time, to access 
fast, heavily optimized C and Fortran libraries and do things they could 
not do in Fortran and that would have been much more difficult in C.  My 
daughter's PhD thesis work is a recent example of using Python to access 
C libraries.


Yes, one of the great things about Python is that almost every library 
of any size has Python bindings.


But there is a difference between making code that is already written in 
C/Fortran available to Python and telling people to write code in 
C/Fortran because their Python code is too slow.


We want people to be able to write code in Python and have it perform at 
the level they would get from a good Javascript or lua implementation.




The concerned scientist mentioned above noted, more or less correctly, 
that numerical code, such as neuro-network code, is, say, 80x slower in 
pure python than in compiled C.  But he did not mention that serious 
numerical and scientific work in Python is not done with such code.

I have seen this sort of bogus comparison before.


It is still important to speed up Python though.

If a program does 95% of its work in a C++ library and 5% in Python, it 
can easily spend the majority of its time in Python because CPython is a 
lot slower than C++ (in general).


Cheers,
Mark.




https://www.python.org/dev/peps/pep-0659/



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QAWW7I733TPNU4VE2I4J2FCPCXXVM7MG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 659: Specializing Adaptive Interpreter

[Mark Shannon]

Hi everyone,

I would like to present PEP 659.

This is an informational PEP about a key part of our plan to improve 
CPython performance for 3.11 and beyond.


For those of you aware of the recent releases of Cinder and Pyston,
PEP 659 might look similar.
It is similar, but I believe PEP 659 offers better interpreter 
performance and is more suitable to a collaborative, open-source 
development model.


As always, comments and suggestions are welcome.

Cheers,
Mark.

Links:

https://www.python.org/dev/peps/pep-0659/
https://github.com/facebookincubator/cinder
https://github.com/pyston/pyston
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VKBV4X6ZEMRBALW7JNOZYI22KETR4F3R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Mark Shannon]

Hi Luciano,

On 20/04/2021 11:35 pm, Luciano Ramalho wrote:

I am not taking sides now, but I want to share with you a useful
diagram to reason about typing support in Python.

I struggled to explain what Python offers until I came up with this diagram:

https://standupdev.com/wiki/doku.php?id=python_protocols#the_typing_map


That's really nice, thanks.



The Typing Map has two orthogonal axis:

- when are types checked:
-- runtime checking
-- static checking

- how are type checked:
-- structural types
-- nominal types

The quadrants are informally labeled with the terms in ALL CAPS below.

Traditionally, mainstream languages supported one of two diagonally
opposite quadrants: STATIC TYPING and DUCK TYPING.

Now the situation is more complicated.

- Java supports only STATIC TYPING: static checking of nominal types;
Python started supporting nominal types with PEP 484

- Before ABCs, Python supported only DUCK TYPING: runtime checking of
structural types;

- With ABCs, Python started supporting GOOSE TYPING (a term invented
by Alex Martelli, in cc because I just quoted him): runtime checking
of nominal types (with subclass hook which is a backdoor to support
explicit checks on structural types as well);

- With PEP 544, Python started supporting STATIC DUCK TYPING: static
checking of structural types;

There are languages that support multiple quadrants:

- TypeScript, like Python, supports all four quadrants.

- Go supports STATIC TYPING, but it also famously popularized STATIC
DUCK TYPING, and even supports GOOSE TYPING with features like type
assertions and type switches [1] designed for explicit runtime
checking of nominal or structural types.

[1] https://tour.golang.org/methods/16

The Typing Map will be featured in my upcoming PyCon US talk [2]

[2] https://us.pycon.org/2021/schedule/presentation/80/

Cheers,

Luciano


PS. If you are aware of other languages that support more than one of
these quadrants, please let me know!




On Tue, Apr 20, 2021 at 6:53 PM Eric Casteleijn  wrote:




On Tue, Apr 20, 2021 at 10:03 AM Mark Shannon  wrote:


...
PEP 544 supports structural typing, but to declare a structural type you
must inherit from Protocol.
That smells a lot like nominal typing to me.



Note that to implement a protocol you do not have to inherit from anything. You 
create a structural type that subclasses Protocol, but then any object that 
satisfies that protocol can be passed where that type is expected, without 
having to inherit anything, so I would argue that this really is structural 
typing.


--
- eric casteleijn (he/him)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R3VP4KORAWI6KK4CNFL6JNYCATWR47EV/
Code of Conduct: http://python.org/psf/codeofconduct/





___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UXBP7AII3Z4Z7COK7P4OETAR7J5F5XP5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Keeping Python a Duck Typed Language.

[Mark Shannon]

Hi everyone,

Once upon a time Python was a purely duck typed language.

Then came along abstract based classes, and some nominal typing starting 
to creep into the language.


If you guarded your code with `isinstance(foo, Sequence)` then I could 
not use it with my `Foo` even if my `Foo` quacked like a sequence. I was 
forced to use nominal typing; inheriting from Sequence, or explicitly 
registering as a Sequence.



Then came type hints. PEP 484 explicitly said that type hints were 
optional and would *always* be optional.


Then came along many typing PEPs that assumed that type hints would only 
used for static typing, making static typing a bit less optional.
Not only that, but the type system proposed by many of these PEPs was 
clearly nominal, not structural.


PEP 544 supports structural typing, but to declare a structural type you 
must inherit from Protocol.

That smells a lot like nominal typing to me.

Then came PEP 563 and said that if you wanted to access the annotations 
of an object, you needed to call typing.get_type_hints() to get 
annotations in a meaningful form.

This smells a bit like enforced static typing to me.

Then came PEP 634 (structural pattern matching). Despite having the word 
'structural' in the name, PEP 634 insists on nominal typing to 
distinguish between sequences and mappings.



Nominal typing in a dynamically typed language makes little sense. It 
gains little or no safety, but restricts the programs you can write.
Because a class can claim to be a nominal type, but not match it 
structurally, it can appear to be type safe but fail at runtime. 
Conversely nominal typing errors can result in failures where 
structurally typed programs would work.


An extreme example of that is this:

# Some magic code to mess with collections.abc.Sequence
>>> match {}:
... case []:
...print("WTF!")
...
WTF!

With duck typing this would be impossible (unless you use ctypes to mess 
with the dict object).


To be fair, nominal typing is not always a problem. All exceptions must 
inherit from BaseException, and it doesn't seem to be a problem in practice.



So, lets stick to our promise that type hints will always be optional, 
and restore duck typing.



I'm not suggesting that we get rid type hints and abstract base classes.
They are popular for a reason.
But let's treat them as useful tools, not warp the rest of the language 
to fit them.


Cheers,
Mark.


Quick summaries of type systems:

https://en.wikipedia.org/wiki/Nominal_type_system

https://en.wikipedia.org/wiki/Structural_type_system

https://en.wikipedia.org/wiki/Duck_typing

Or, if you're really keen:

https://www.cis.upenn.edu/~bcpierce/tapl/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5KUUL5P5QQMIYSDO5DNK5DK5I6NRE3GA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Memory use of PEP 563 vs PEP 649

[Mark Shannon]

Hi,

There has been some discussion on this mailing list about the memory use 
of PEP 563 vs PEP 649.


It doesn't matter.

The memory use of either is small, and any measurements are merely 
measuring artifacts of the current implementations, and the current 
on-disk representation of code objects.
In an ideal implementation, of either PEP, the underlying data for 
`__annotations__` will sit on disk at zero cost in memory and load time.


There are much more important differences between the two PEPs.

I support keeping the 3.9 behavior for 3.10 to give us time to resolve 
the discussion and maybe for us to implement some of the above "ideal 
implementation".


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3DF5P2SRAYIYSDWG7GXPUUSRAXCXGJ4M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making staticmethod callable, any oposite?

[Mark Shannon]

On 14/04/2021 2:20 am, Inada Naoki wrote:

Hi, all.

I am implementing PEP 597. During review, Victor suggested to
deprecate `OpenWrapper`. `OpenWrapper` is defined only for
compatibility between C function and Python function:

```
from _pyio import open as py_open
from _io import open as c_open

class C:
 py_open = py_open
 c_open = c_open

C().c_open("README.rst")  # works
C().py_open("README.rst")  # TypeError: expected str, bytes or
os.PathLike object, not C
```

So builtin open is not io.open, but io.OpenWrapper in Python 3.9.
Making staticfunction callable fixes this issue.

```
@staticfunction
def open(...): ...
```

Now open defined in Python behaves like C function. We don't need
OpenWrapper anymore.
This has already been committed by Guido's approval. staticmethod is
callable, and OpenWrapper is just an alias of open and deprecated in
master branch.

But Mark Shannon said we shouldn't make such a change without
discussing at python-dev.
I don't know we *should*, but I agree that it is *ideal*.

Then, does anyone oppose this change?


I do (although not strongly).

I think we are changing the wrong thing.

Sometimes code gets moved from C to Python and vice-versa.
The differences in descriptor behavior between builtin function and 
Python functions trips people up. We agree on that.


However I don't think changing the behavior of static methods is the way 
to fix that.


A staticmethod is not a function, builtin or otherwise. It is a method 
that, when called, ignores the object it is attached to.


If we want Python functions to behave like a builtin function, then 
marking them `@staticmethod` is misleading, IMO.


I'm also worried about corner cases where this change in behavior will 
break code.


I'm all in favor of replacing C code with Python and don't want to make 
it difficult.

So, why not add a new descriptor, that clearly describes the intent:

`@non_method` or just `@function`?

The decorator would make a new object that behaves like a 
builtin-function, even though it is implemented in Python.


Cheers,
Mark.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WRHFP4FKWCCJWOU2JTRVFXB6LSFRATKG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Brandt,

On 04/04/2021 1:34 am, Brandt Bucher wrote:

Mark Shannon said:

I was relying on the "reference" implementation, which is also in the PEP.


Can you please stop putting scare quotes around "reference implementation"? 
You've done it twice now, and it's been a weekend-ruiner for me each time.


I'm sorry for ruining your weekends.

My intention, and I apologize for not making this clearer, was not to 
denigrate your work, but to question the implications of the term 
"reference".


Calling something a "reference" implementation suggests that it is 
something that people can refer to, that is near perfectly correct and 
fills in the gaps in the specification.


That is a high standard, and one that is very difficult to attain.
It is why I use the term "implementation", and not "reference 
implementation" in my PEPs.




I've put months of work into writing and improving CPython's current pattern 
matching implementation, mostly on nights and weekends. I don't know whether 
it's intentional or not, but when you say things like that it instantly 
devalues all of my hard work in front of everyone on the list.

It definitely wasn't my intention.



For such a huge feature, I'm honestly quite amazed that this is the only issue 
we've found since it was merged over a month ago (and both authors have agreed 
that it needs to be fixed in the PEP, not the implementation). The PR 
introducing this behavior was reviewed by at least a half-dozen people, 
including you.


Indeed, I reviewed the implementation.
I thought it was good enough to merge.
I still think that.



The last time you said something like this, I just muted the thread. Let's 
please keep this respectful; we're all obviously committing a lot of our own 
time and energy to this, and we need to work well together for it to be 
successful in the long term.
Please don't take my criticisms of PEP 634 as criticisms of you or your 
efforts. I know it can often sound like that, but that really isn't my 
intent.


Pattern matching is a *big* new feature, and to get it right takes a lot 
of discussion.

Having your ideas continually battered is no fun, I know.
So, I'd like to apologize again for any hurt caused.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TAINF7X3JMX74FBORYIZSYPT2ZOVV6C6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Guido,

On 02/04/2021 10:05 pm, Guido van Rossum wrote:
On Fri, Apr 2, 2021 at 12:43 PM Brandt Bucher <mailto:brandtbuc...@gmail.com>> wrote:


Mark Shannon wrote:
 > On 02/04/2021 7:19 am, Brandt Bucher wrote:
 > > I agree that self-matching classes should absolutely allow
keyword matches. I had no idea the PEP forbade it.
 > PEP 634 allows it.

PEP 634 says:

 > For a number of built-in types (specified below), a single
positional subpattern is accepted which will match the entire
subject; for these types no keyword patterns are accepted.

(https://www.python.org/dev/peps/pep-0634/#class-patterns
<https://www.python.org/dev/peps/pep-0634/#class-patterns>)


But that's not what the implementation does. It still supports keyword 
patterns for these types -- and (as I've said earlier in this thread) I 
think the implementation is correct.


 > Most checks are cheap though.
 > Checking for duplicates in `__match_args__` can be done at class
creation time, and checking for duplicates in the pattern can be
done at compile time.

I assume the compile-time check only works for named keyword
attributes. The current implementation already does this.

-1 on checking `__match_args__` anywhere other than the match block
itself.


Agreed.


Why? (I also asked Brandt this)

It is far more efficient to check `__match_args__` at class creation (or 
class attribute assignment) time.


The most efficient way to check in the match block is to check at class 
creation time anyway and store a flag whether  `__match_args__` is 
legal. In the match block we would check this flag, then proceed.


It seems silly to know that there will be a runtime error, but not act 
on that information, allowing latent bugs could have been reported.


Cheers,
Mark.



--
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)
/Pronouns: he/him //(why is my pronoun here?)/ 
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S45E7DCDXU2LTBSNYYXSIDH6CL7CVDIK/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EN25PHRQO55AORYWL5SB63JMLVFT6V2L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Brandt,

On 02/04/2021 8:41 pm, Brandt Bucher wrote:

Mark Shannon wrote:

On 02/04/2021 7:19 am, Brandt Bucher wrote:

I agree that self-matching classes should absolutely allow keyword matches. I 
had no idea the PEP forbade it.

PEP 634 allows it.


PEP 634 says:


For a number of built-in types (specified below), a single positional 
subpattern is accepted which will match the entire subject; for these types no 
keyword patterns are accepted.


(https://www.python.org/dev/peps/pep-0634/#class-patterns)


I was relying on the "reference" implementation, which is also in the PEP.

>>> match 0:
... case int(imag=0):
...print ("Experimentally, int supports keyword matching.")
...
Experimentally, int supports keyword matching.

I take this as +1 for having more precisely defined semantics for 
pattern matching :)





Most checks are cheap though.
Checking for duplicates in `__match_args__` can be done at class creation time, 
and checking for duplicates in the pattern can be done at compile time.


I assume the compile-time check only works for named keyword attributes. The 
current implementation already does this.

-1 on checking `__match_args__` anywhere other than the match block itself.


I'm curious, why?
It is much faster *and* gives better error messages to check 
`__match_args__` at class creation time.




Guido van Rossum wrote:

On Fri, Apr 2, 2021 at 3:38 AM Mark Shannon m...@hotpy.org wrote:

Are there are any use-cases?
The test-case `int(real=0+0j, imag=0-0j)` is contrived, but I'm struggling to 
come up with less contrived examples for any of float, list, dict, tuple, str.

There could be a subclass that adds an attribute. That's still contrived  
though.


I could see the case for something like `case defaultdict({"Spam": s}, 
default_factory=f)`. I certainly don't think it should be forbidden.


It is forbidden in the PEP, as written, correct?
OOI, have you changed your mind, or was that an oversight in the original?

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R7ZIAAQNL2CN3K3M3DMR2IQNC6DLD7FF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Brandt,

On 02/04/2021 7:19 am, Brandt Bucher wrote:

Guido van Rossum wrote:

Well, now I have egg on my face, because the current implementation does reject multiple 
occurrences of the same identifier in __match_args__. We generate an error like 
"TypeError: C() got multiple sub-patterns for attribute 'a'". However, I cannot 
find this uniqueness requirement in PEP 634, so I think it was a mistake to implement it.

Researching this led me to find another issue where PEP 634 and the implementation 
differ, but this time it's the other way around: PEP 634 says about types which accept a 
single positional subpattern (int(x), str(x) etc.) "for these types no keyword 
patterns are accepted." Mark's example `case int(real=0, imag=0):` makes me think 
this requirement is wrong and I would like to amend PEP 634 to strike this requirement. 
Fortunately, this is not what is implemented. E.g. `case int(1, real=1):` is accepted and 
works, as does `case int(real=0):`.

Calling out Brandt to get his opinion. And thanks to Mark for finding these!


The current implementation will reject any attribute being looked up more than 
once, by position *or* keyword. It's actually a bit tricky to do, which is why 
the `MATCH_CLASS` op is such a beast... it needs to look up positional and 
keyword attributes all in one go, keeping track of everything it's seen and 
checking for duplicates.

I believe this behavior is a holdover from PEP 622:


The interpreter will check that two match items are not targeting the same 
attribute, for example `Point2d(1, 2, y=3)` is an error.


(https://www.python.org/dev/peps/pep-0622/#overlapping-sub-patterns)

PEP 634 explicitly disallows duplicate keywords, but as far as I can tell it says 
nothing about duplicate `__match_args__` or keywords that also appear in 
`__match_args__`. It looks like an accidental omission during the 622 -> 634 
rewrite.

(I guess I figured that if somebody matches `Spam(foo, y=bar)`, where `Spam.__match_args__` is 
`("y",)`, that's probably a bug in the user's code. Ditto for `Spam(y=foo, y=bar)` and `Spam(foo, 
bar)` where `Spam.__match_args__` is `("y", "y")` But it's not a hill I'm willing to die 
on.)


Repeated keywords do seem likely to be a bug.
Most checks are cheap though.
Checking for duplicates in `__match_args__` can be done at class 
creation time, and checking for duplicates in the pattern can be done at 
compile time.


So how about explicitly disallowing those, but not checking that the 
intersection of `__match_args__` and keywords is empty?

We would get most of the error checking without the performance impact.



I agree that self-matching classes should absolutely allow keyword matches. I 
had no idea the PEP forbade it.


PEP 634 allows it. PEP 653 currently forbids it, mainly for consistency 
reasons.
The purpose of self-matching is to prevent deconstruction, so it seems 
inconsistent to allow it for keyword arguments.


Are there are any use-cases?
The test-case `int(real=0+0j, imag=0-0j)` is contrived,
but I'm struggling to come up with less contrived examples for any of 
float, list, dict, tuple, str.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L73HXM3QZGJWVPTYRUNF4JZ2ETAMJSGK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Guido,

On 02/04/2021 2:38 am, Guido van Rossum wrote:
On Thu, Apr 1, 2021 at 2:18 PM Mark Shannon <mailto:m...@hotpy.org>> wrote:


On 31/03/2021 9:53 pm, Guido van Rossum wrote:
 > On Wed, Mar 31, 2021 at 12:08 PM Mark Shannon mailto:m...@hotpy.org>
 > <mailto:m...@hotpy.org <mailto:m...@hotpy.org>>> wrote:

[snip]

 >     Apart from that, I think the semantics are so similar once
you've added
 >     __match_seq__/__match_map__  to PEP 634 that is hard to
 >     claim one is better than the other.
 >     My (unfinished) implementation of PEP 653 makes almost no
changes to
 >     the test suite.
 >
 > I'd like to see where those differences are -- then we can talk
about
 > which is better. :-)

Almost all the changes come from requiring __match_args__ to be a tuple
of unique strings.


Ah, *unique* strings. Not sure I care about that. Explicitly checking 
for that seems extra work, and I don't see anything semantically suspect 
in allowing that.


Checking for uniqueness is almost free because __match_args__ is a 
tuple, and therefore immutable, so the check can be done at class 
creation time.




The only other change is that

case int(real=0+0j, imag=0-0j):

fails to match 0, because `int` is `MATCH_SELF` so won't match
attributes.


Oh, but that would be a problem. The intention wasn't that "self" mode 
prevents keyword/attribute matches. (FWIW real and imag should 
attributes should not be complex numbers, so that testcase is weird, but 
it should work.)


I thought matching `int(real=0+0j, imag=0-0j)` was a bit weird too.

The change required to make it work is trivial, but the code seems more 
consistent if `int(real=0+0j, imag=0-0j)` is disallowed, which is why I 
went for that.






https://github.com/python/cpython/compare/master...markshannon:pep-653-implementation?expand=1#diff-490b4f3b911cb4ca281e9ca6ff814bc10d331f0421f6c6971b08d9f29020620b

<https://github.com/python/cpython/compare/master...markshannon:pep-653-implementation?expand=1#diff-490b4f3b911cb4ca281e9ca6ff814bc10d331f0421f6c6971b08d9f29020620b>



--
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)
/Pronouns: he/him //(why is my pronoun here?)/ 
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2ACKZY2J7OVP4WELBIFLLGR7343AEQA7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Guido,

On 31/03/2021 9:53 pm, Guido van Rossum wrote:
On Wed, Mar 31, 2021 at 12:08 PM Mark Shannon <mailto:m...@hotpy.org>> wrote:




[snip]


Apart from that, I think the semantics are so similar once you've added
__match_seq__/__match_map__  to PEP 634 that is hard to
claim one is better than the other.
My (unfinished) implementation of PEP 653 makes almost no changes to
the
test suite.


I'd like to see where those differences are -- then we can talk about 
which is better. :-)


Almost all the changes come from requiring __match_args__ to be a tuple 
of unique strings.


The only other change is that

case int(real=0+0j, imag=0-0j):

fails to match 0, because `int` is `MATCH_SELF` so won't match attributes.

https://github.com/python/cpython/compare/master...markshannon:pep-653-implementation?expand=1#diff-490b4f3b911cb4ca281e9ca6ff814bc10d331f0421f6c6971b08d9f29020620b

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IKJXKRO5MFUKKEB347X7TSXFVWWLDJYB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

On 31/03/2021 11:31 pm, Brandt Bucher wrote:

Guido van Rossum wrote:

On Wed, Mar 31, 2021 at 2:14 PM Brandt Bucher brandtbuc...@gmail.com
wrote:

(One change from my last email: it doesn't allow `__match_map__` /
`__match_seq__` to be set to `False`... only `True`. This prevents some
otherwise tricky multiple-inheritance edge-cases present in both of our
flagging systems that I discovered during testing. I don't think there are
actual use-cases for unsetting the flags in subclasses, but we can revisit
that later if needed.)

That's surprising to me. Just like we can have a class that inherits from
int but isn't hashable, and make that explicit by setting `__hash__ =
None`, why couldn't I have a class that inherits from something else that
happens to inherit from Sequence, and say "but I don't want it to match
like a sequence" by adding `__match_sequence__ = False`? AFAIK all Mark's
versions would support this by setting `__match_kind__ = 0`.


The issue isn't when *I* set `__match_seq__ = False` or `__match_container__ = 
0`. It's when *one of my parents* does it that things become difficult.


Maybe you can show an example edge case where this would be undesirable?


Good idea. I've probably been staring at this stuff for too long to figure it 
out myself. :)

As far as I can tell, these surprising cases arise because a bit flag can only be either 
0 or 1. For us, "not specified" is equivalent to 0, which can lead to ambiguity.

Consider this case:

```
class Seq:
 __match_seq__ = True
 # or __match_container__ = MATCH_SEQUENCE

class Parent:
 pass

class Child(Parent, Seq):
 pass
```

Okay, cool. `Child` will match as a sequence, which seems correct. But what 
about this similar case?

```
class Seq:
 __match_seq__ = True
 # or __match_container__ = MATCH_SEQUENCE

class Parent:
 __match_seq__ = False
 # or __match_container__ = 0

class Child(Parent, Seq):
 pass
```

Here, `Child` will *not* match as a sequence, even though it probably should. The only 
workarounds I've found (like allowing `None` to mean "this is unset, don't inherit 
me if another parent sets this flag", ditching tp_flags entirely, or not inheriting 
these attributes) feel a bit extreme just to allow some users to do the moral equivalent 
of un-subclassing `collections.abc.Sequence`.


This is just a weird case, so I don't think we should worry about it too 
much.




So, my current solution (seen on the branch linked in my earlier email) is:

- Set the flag if the corresponding magic attribute is set to True in the class 
definition
- Raise at class definition time if it's set to anything other than True
- Otherwise, set the flag if any of the parents set have the flag set

As far as I can tell, this leads to the expected (and current, as of 3.10.0a6) 
behavior in all cases. Plus, it doesn't break my mental model of how 
inheritance works.


Inheritance in Python is based on the MRO (using the C3 linearization 
algorithm) so my mental model is that Child.__match_container__ == 0.


Welcome the wonderful world of multiple inheritance :)

If Parent.__match_container__ == 0 (rather than just inheriting it) then 
it is explicitly stating that it is *not* a container.

Seq explicitly states that it *is* a sequence.

So Child is just broken. That it is broken for pattern matching is 
consistent with it being broken in general.


Cheers,
Mark.



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R3BGDN6NINJMLUWBVMVYIGORSLPJOMJP/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z7TK37K25DOV6LSWZDXA7HDXOTKGZTUP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Guido,

On 31/03/2021 6:21 pm, Guido van Rossum wrote:
On Wed, Mar 31, 2021 at 2:30 AM Mark Shannon <mailto:m...@hotpy.org>> wrote:


 > - Add new `__match_seq__` and `__match_map__` special attributes,
corresponding to new public `Py_TPFLAGS_MATCH_SEQ` and
`Py_TPFLAGS_MATCH_MAP` flags for use in `tp_flags`. When Python
classes are defined with one or both of these attributes set to a
boolean value, `type.__new__` will update the flags on the type to
reflect the change (using a similar mechanism as `__slots__`
definitions). They will be inherited otherwise. For convenience,
`collections.abc.Sequence` will define `__match_seq__ = True`, and
`collections.abc.Mapping` will define `__match_map__ = True`.
 >
 > Using this in Python would look like:
 >
 > ```
 > class MySeq:
 >      __match_seq__ = True
 >      ...
 >
 > class MyMap:
 >      __match_map__ = True
 >      ...
 > ```

I don't like the way this need special inheritance rules, where
inheriting one attribute mutates the value of another.
It seems convoluted.

Consider:

class WhatIsIt(MySeq, MyMap):
      pass

With __match_container__ it works as expected with no special
inheritance rules.


Wait a minute, do you expect WhatIsIt to be a sequence but not a map? 
*I* would expect that it is both, and that's exactly what Brandt's 
proposal does. So I see this as a plus.



Earlier you said:

Classes that are both mappings and sequences are ill-conceived.
Let's not compromise semantics or optimizability to support these.
   (IOW I agree with Mark here.)

PEP 653 requires that:
(__match_container__ & (MATCH_SEQUENCE | MATCH_MAPPING)) != 
(MATCH_SEQUENCE | MATCH_MAPPING)


Would you require that (__match_seq__ and __match_map__) is always false?
If so, then what is the mechanism for handling the `WhatIsIt` class?
If not, then you loose the ability to make a single test to determine
which patterns can apply.



I think we are close to agreement on the mechanism for selecting which
pattern to match, but I still want the better defined semantics of
PEP 653.


I don't know that PEP 653's semantics are better. Have you analyzed any 
*differences* besides the proposal above? I've personally found reading 
your pseudo-code very difficult, so I simply don't know.


PEP 653 semantics are more precise. I think that is better :)

Apart from that, I think the semantics are so similar once you've added 
__match_seq__/__match_map__  to PEP 634 that is hard to

claim one is better than the other.
My (unfinished) implementation of PEP 653 makes almost no changes to the 
test suite.


The code in the examples is Python, not pseudo-code.
That might be easier to follow.

Cheers,
Mark.


--
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)
/Pronouns: he/him //(why is my pronoun here?)/ 
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TC46W5FKYMKDSWRT77ESVJCSWRHSLCNF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Brandt,

On 30/03/2021 11:49 pm, Brandt Bucher wrote:

Hi Mark.

I've spoken with Guido, and we are willing to propose the following amendments 
to PEP 634:

- Require `__match_args__` to be a tuple.


I think we're all in agreement on this one.
Let's just do it.


- Add new `__match_seq__` and `__match_map__` special attributes, corresponding 
to new public `Py_TPFLAGS_MATCH_SEQ` and `Py_TPFLAGS_MATCH_MAP` flags for use 
in `tp_flags`. When Python classes are defined with one or both of these 
attributes set to a boolean value, `type.__new__` will update the flags on the 
type to reflect the change (using a similar mechanism as `__slots__` 
definitions). They will be inherited otherwise. For convenience, 
`collections.abc.Sequence` will define `__match_seq__ = True`, and 
`collections.abc.Mapping` will define `__match_map__ = True`.

Using this in Python would look like:

```
class MySeq:
 __match_seq__ = True
 ...

class MyMap:
 __match_map__ = True
 ...
```


I don't like the way this need special inheritance rules, where 
inheriting one attribute mutates the value of another.

It seems convoluted.

Consider:

class WhatIsIt(MySeq, MyMap):
pass

With __match_container__ it works as expected with no special 
inheritance rules.


This was why you convinced me to split __match_kind__; it works better 
with inheritance.


Anther reason for preferring __match_container__ is that it provides a 
better option for extensibility, IMO.
Suppose we wanted to add a "set" pattern in the future, with 
__match_container__ we just need to add a new constant.

With your proposed approach, we would need another special attribute.



Using this in C would look like:

```
PyTypeObject PyMySeq_Type = {
 ...
 .tp_flags = Py_TPFLAGS_MATCH_SEQ | ...,
 ...
}

PyTypeObject PyMyMap_Type = {
 ...
 .tp_flags = Py_TPFLAGS_MATCH_MAP | ...,
 ...
}
```


I'm wary of using up tp_flags, as they are a precious resource, but this 
does provide a more declarative way to specific the behavior than 
setting the attribute via the C-API.




We believe that these changes will result in the best possible outcome:
- The new mechanism should faster than either PEP.


The naive implementation of the boolean version might be a tiny bit 
faster (it would hard to measure a difference).
However, once specialized by type version (as we do for LOAD_ATTR) both 
forms become a no-op.



- The new mechanism should provide a better user experience than either PEP 
when defining types in either Python *or C*.


The inheritance rules make __match_container__ a better user experience 
in Python, IMO.
As for C, there no reason why the it would make any difference, 
__match_container__ could be

(tp_flags & (Py_TPFLAGS_MATCH_SEQ|Py_TPFLAGS_MATCH_MAP))
shifted to line up the bits.



If these amendments were made, would you be comfortable withdrawing PEP 653? We 
think that if we're in agreement here, a compromise incorporating these 
promising changes into the current design would be preferable to submitting yet 
another large pattern matching PEP for a very busy SC to review and pronounce 
before the feature freeze. I am also willing, able, and eager to implement 
these changes promptly (perhaps even before the next alpha) if so.


I think we are close to agreement on the mechanism for selecting which 
pattern to match, but I still want the better defined semantics of PEP 653.




Thanks for pushing us to make this better.


And thank you for the feedback.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VXSUTM6CNEMOBZUJZXVP42OYYZWCBE2C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Brandt,


On 30/03/2021 5:25 pm, Brandt Bucher wrote:

Overall, I am still uncomfortable with PEP 653, and would probably not support 
its acceptance.

Although it has thankfully become a much less radical proposal than it was a 
few weeks ago (thanks, Mark, for your attention to our feedback), I feel that 
the rules it binds implementations to are *very* premature, and that the new 
mechanisms it introduces to do so only modestly improve potential performance 
at great expense to the ease of learning, using, and maintaining code using 
structural pattern matching.

A few notes follow:


For example, using `sympy`, we might want to write:

```
# a*a == a**2
case Mul(args=[a, b]) if a == b:
 return Pow(a, 2)
```

Which requires the sympy class `Symbol` to "self" match. For `sympy` to support 
this pattern with PEP 634 is possible, but a bit tricky. With this PEP it can be 
implemented very easily.


Maybe I'm missing something, but I don't understand at all how the provided 
code snippet relies on the self-matching behavior.


No I'm missing something.
That should read
 case Mul(args=[Symbol(a), Symbol(b)) if a == b:
 ...

I'll fix that in the PEP thanks.



Have the maintainers of SymPy (or any large library supposedly benefitting here) come out 
in support of the PEP? Are they at least aware of it? Have they indicated that the 
proposed idiom for implementing self-matching behavior using a property is truly too 
"tricky" for them?

Have you identified any stdlib classes that would benefit greatly from this?

For me, `__match_class__` feels like a feature without demonstrated need. Even 
if there is a great demand for this, I certainly think that there are far 
better options than the proposed flagging system:


The distinction between those classes that have the default behavior and 
those that match "self" is from PEP 634. I didn't introduce it.

I'm just proposing a more principled way to make that distinction.



- A `@match_self` class decorator (someone's bound to put one on PyPI, at any 
rate).
- Allowing `__match_args__ = None` to signal this case (an option we previously 
considered, and my personal preference).

...both of which can be added later, if needed.

Further, PEP 634 makes it very easy for libraries to support Python versions 
with *and* without pattern matching (something I consider to be an important 
requirement). The following class works with both 3.9 and 3.10:

```
class C(collections.abc.Sequence):
 ...
```

While something like this is required for PEP 653:

```
class C:
 if sys.version_info >= (3, 10):
 from somewhere import MATCH_SEQUENCE
 __match_container__ = MATCH_SEQUENCE
 ...
```


Or

class C:
__match_container__ = 1 # MATCH_SEQUENCE

Which is one reason the PEP states that the values of MATCH_SEQUENCE, 
etc. will never change.





PEP 634 relies on the `collections.abc` module when determining which patterns 
a value can match, implicitly importing it if necessary. This PEP will 
eliminate surprising import errors and misleading audit events from those 
imports.


I think that a broken `_collections_abc` module *should* be surprising. Is 
there any reasonable scenario where it's expected to not exist, or be not be 
fit for this purpose?


No reasonable scenario, but unreasonable scenarios happen all too often.



And I'm not sure how an audit event for an import that is happening could be considered 
"misleading"... I certainly wouldn't want it suppressed.


It's misleading because a match statement doesn't include any explicit 
imports.





Looking up a special attribute is much faster than performing a subclass test 
on an abstract base class.


How much faster? A quick benchmark on my machine suggests less than half a 
microsecond. PEP 634 (like PEP 653) already allows us to cache this information 
for the subject of a match statement, so I doubt that this is actually a real 
issue in practice. An indeed, with the current implementation, this test isn't 
even performed on the most common types, such as lists, tuples, and 
dictionaries.


Half a microsecond is thousands of instructions on a modern CPU.
That is a long time for a single VM operation.



At the very least, PEP 653's confusing new flag system seems to be a *very* 
premature optimization, seriously hurting usability for a modest performance 
increase. (Using them wrongly also seems to introduce a fair amount of 
undefined behavior, which seems to go against the PEP's own motivation.)


Why do you say it is a premature optimization?
It's primary purpose is reliability and precise semantics.
It is more optimizable, I agree, but that is hardly premature.

You also say it is confusing, but I think it is simpler than the 
workarounds to match "self" that you propose.

This is very subjective though. Evidently we think differently.




If the value of `__match_args__` is not as specified, then the implementation 
may raise any exception, or match the wrong pattern.

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Guido,

Thanks for the feedback.

On 27/03/2021 10:15 pm, Guido van Rossum wrote:

Hi Mark,

Reading that spec will take some time. Can you please summarize the 
differences in English, in a way that is about as precise as PEP 634? I 
have some comments inline below as well.


On Sat, Mar 27, 2021 at 10:16 AM Mark Shannon <mailto:m...@hotpy.org>> wrote:


Hi Oscar,

Thanks for the feedback.

On 27/03/2021 4:19 pm, Oscar Benjamin wrote:
 > On Sat, 27 Mar 2021 at 13:40, Mark Shannon mailto:m...@hotpy.org>> wrote:
 >>
 >
 > Hi Mark,
 >
 > Thanks for putting this together.
 >
 >> As the 3.10 beta is not so far away, I've cut down PEP 653 down
to the
 >> minimum needed for 3.10. The extensions will have to wait for 3.11.
 >>
 >> The essence of the PEP is now that:
 >>
 >> 1. The semantics of pattern matching, although basically
unchanged, are
 >> more precisely defined.
 >>
 >> 2. The __match_kind__ special attribute will be used to
determine which
 >> patterns to match, rather than relying on the collections.abc
module.
 >>
 >> Everything else has been removed or deferred.
 >
 > It would take me some time to compare exactly how this differs from
 > the current state after PEP 634 but I certainly prefer the
 > object-model based approach. It does seem that there are a lot of
 > permutations of how matching works but I guess that's just trying to
 > tie up all the different cases introduced in PEP 634.


It would be simpler if this was simply an informational PEP without 
proposing new features -- then we wouldn't have to rush.


It is about to close to that as I can get it. The change to using 
__match_kind__ requires some small changes to behaviour.




You could then propose the new __match_kind__ attribute in a separate 
PEP, written more in the style of PEP 634, without pseudo code.


I find it difficult to wrap my head around the semantics of 
__match_kind__ because it really represents a few independent flags 
(with some constraints) but all the text is written using explicit, 
hard-to-read bitwise and/or operations. Let me give it a try.


- Let's call the four flag bits by short names: SEQUENCE, MAPPING, 
DEFAULT, SELF.


SEQUENCE and MAPPING are for use when an instance of a class appears in 
the subject position (i.e., for `match x`, we look for these bits in 
`type(x).__match_kind__`). Neither of these is set by default. At most 
one of them should be set.


- If SEQUENCE is set, the subject is treated like a sequence (this is 
set for list, tuple and other sequences, but not for str, bytes and 
bytearray).


- Similarly, MAPPING means the subject should be treated as a mapping, 
and is set for dict and other mapping types.


The DEFAULT and SELF flags are for use when a class is used in a class 
pattern (i.e., for `case cls(...)` we look for these bits in 
`cls.__match_kind__`). At most one of these should be set. DEFAULT is 
set on class `object` and anything that doesn't explicitly clear it.


- If DEFAULT is set, semantics of PEP 634 apply except for the special 
behavior enabled by the SELF flag.


- If SELF is set, `case cls(x)` binds the subject to x, and no other 
forms of `case cls(...)` are allowed.


`case cls():` is always allowed, regardless of flags.



- If neither DEFAULT nor SELF is set, `case cls(...)` does not take 
arguments at all.


Please correct any misunderstandings I expressed here! (And please 
include some kind of summary like this in your PEP.)


I think you expressed it well. I'll add a more informal overview section 
to the PEP.




Also, I think that we should probably separate this out in two separate 
flag sets, one for subjects and one for class patterns -- it is pretty 
confusing to merge the flag sets into a single value when their 
applicability (subject or class pattern) is so different.


That would require two different special attributes, which adds bulk 
without adding any value.


   __match_kind__ = MATCH_SEQUENCE | MATCH_DEFAULT

should be clear to anyone familiar with integer flags.




 >> The PEP now has only the slightest changes to semantics, which
should be
 >> undetectable in normal use. For those corner cases where there is a
 >> difference, it is to make pattern matching more robust.
 >
 > Maybe I misunderstood but it looks to me as if this (PEP 653) changes
 > the behaviour of a mapping pattern in relation to extra keys. In PEP
 > 634 extra keys in the target are ignored e.g.:
 >
 > obj = {'a': 1, 'b': 2}
 > match(obj):
 >      case {'a': 1}:
 >           # matches obj because key 'b' is ignored
 >
 >

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi Oscar,

Thanks for the feedback.

On 27/03/2021 4:19 pm, Oscar Benjamin wrote:

On Sat, 27 Mar 2021 at 13:40, Mark Shannon  wrote:




Hi Mark,

Thanks for putting this together.


As the 3.10 beta is not so far away, I've cut down PEP 653 down to the
minimum needed for 3.10. The extensions will have to wait for 3.11.

The essence of the PEP is now that:

1. The semantics of pattern matching, although basically unchanged, are
more precisely defined.

2. The __match_kind__ special attribute will be used to determine which
patterns to match, rather than relying on the collections.abc module.

Everything else has been removed or deferred.


It would take me some time to compare exactly how this differs from
the current state after PEP 634 but I certainly prefer the
object-model based approach. It does seem that there are a lot of
permutations of how matching works but I guess that's just trying to
tie up all the different cases introduced in PEP 634.


The PEP now has only the slightest changes to semantics, which should be
undetectable in normal use. For those corner cases where there is a
difference, it is to make pattern matching more robust.


Maybe I misunderstood but it looks to me as if this (PEP 653) changes
the behaviour of a mapping pattern in relation to extra keys. In PEP
634 extra keys in the target are ignored e.g.:

obj = {'a': 1, 'b': 2}
match(obj):
 case {'a': 1}:
  # matches obj because key 'b' is ignored

In PEP 634 the use of **rest is optional if it is desired to catch the
other keys but does not affect matching. Here in PEP 653 there is the
pseudocode:

# A pattern not including a double-star pattern:
if $kind & MATCH_MAPPING == 0:
 FAIL
if $value.keys() != $KEYWORD_PATTERNS.keys():
 FAIL


I missed that when updating the PEP, thanks for pointing it out.
It should be the same as for double-star pattern:

if not $value.keys() >= $KEYWORD_PATTERNS.keys():
FAIL

I'll update the PEP.



My reading of that is that all keys would need to be match unless
**rest is used to absorb the others.

Is that an intended difference?

Personally I prefer extra keys not to be ignored by default so to me
that seems an improvement. If intentional then it should be listed as
another semantic difference though.


I don't have a strong enough opinion either way.
I can see advantages to both ways of doing it.




E.g. With PEP 653, pattern matching will work in the collections.abc
module. With PEP 634 it does not.


As I understood it this proposes that match obj: should use the class
attribute type(obj).__match_kind__ to indicate whether the object
being matched should be considered a sequence or a mapping or
something else rather than using isinstance(obj, Sequence) and
isinstance(obj, Mapping). Is there a corner case here where an object
can be both a Sequence and a Mapping? (How does PEP 634 handle that?)


If you define a class as a subclass of both collections.abc.Sequence and 
collections.abc.Mapping, then PEP 634 will treat it as both sequence and 
mapping, meaning it has to try every pattern. That prevents the 
important (IMO) optimization of checking the kind only once.


Cheers,
Mark.



Not using the Sequence and Mapping ABCs is good IMO. I'm not aware of
other core language features requiring the use of ABCs. In SymPy we
have specifically avoided them because they slow down isinstance
checking (this is measurable in the time taken to run the whole test
suite). Using the ABCs in PEP 634 seems surprising given that the
original pattern matching PEP actually listed the performance impact
of isinstance checks as part of the opening motivation. Maybe the ABCs
can be made faster but either way using them like this seems not in
keeping with the rest of the language.


Oscar


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WCN3P4XYVWGL4UTEHNYO2LJO5ULZY3GI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Mark Shannon]

Hi everyone,

As the 3.10 beta is not so far away, I've cut down PEP 653 down to the 
minimum needed for 3.10. The extensions will have to wait for 3.11.


The essence of the PEP is now that:

1. The semantics of pattern matching, although basically unchanged, are 
more precisely defined.


2. The __match_kind__ special attribute will be used to determine which 
patterns to match, rather than relying on the collections.abc module.


Everything else has been removed or deferred.

The PEP now has only the slightest changes to semantics, which should be
undetectable in normal use. For those corner cases where there is a 
difference, it is to make pattern matching more robust.
E.g. With PEP 653, pattern matching will work in the collections.abc 
module. With PEP 634 it does not.



As always, all thoughts and comments are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YYIT3QXMLPNLXQAQ5BCXE4LLJ57EE7JV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New public C API functions must not steal references or return borrowed references

[Mark Shannon]

Hi Victor,

I'm with you 100% on not returning borrowed references, doing so is just 
plain dangerous.


However, is a blanket ban on stealing references the right thing?

Maybe the problem is the term "stealing".
The caller is transferring the reference to the callee.
In some circumstances it can make a lot of sense to do so, since the 
caller has probably finished with the reference and the callee needs a 
new one.


Cheers,
Mark.


On 25/03/2021 4:27 pm, Victor Stinner wrote:

Hi,

A new Include/README.rst file was just added to document the 3 C API
provided by CPython:

* Include/: Limited C API
* Include/cpython/: CPython implementation details
* Include/internal/: The internal API

I would like to note that *new* public C API functions must no longer
steal references or return borrowed references.

Don't worry, there is no plan to deprecate or remove existing
functions which do that, like PyModule_AddObject() (streal a
reference) or PyDict_GetItem() (return a borrowed reference). The
policy is only to *add* new functions.

IMO for the *internal* C API, it's fine to continue doing that for
best performances.

Moreover, the limited C API must not expose "implementation details".
For example, structure members must not be accessed directly, because
most structures are excluded from the limited C API. A function call
hiding implementation details is usually better.

Here is a copy of the current Include/README.rst file:

The Python C API


The C API is divided into three sections:

1. ``Include/``
2. ``Include/cpython/``
3. ``Include/internal/``


Include: Limited API


``Include/``, excluding the ``cpython`` and ``internal`` subdirectories,
contains the public Limited API (Application Programming Interface).
The Limited API is a subset of the C API, designed to guarantee ABI
stability across Python 3 versions, and is defined in :pep:`384`.

Guidelines for expanding the Limited API:

- Functions *must not* steal references
- Functions *must not* return borrowed references
- Functions returning references *must* return a strong reference
- Macros should not expose implementation details
- Please start a public discussion before expanding the API
- Functions or macros with a ``_Py`` prefix do not belong in ``Include/``.

It is possible to add a function or macro to the Limited API from a
given Python version.  For example, to add a function to the Limited API
from Python 3.10 and onwards, wrap it with
``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A``.


Include/cpython: CPython implementation details
===

``Include/cpython/`` contains the public API that is excluded from the
Limited API and the Stable ABI.

Guidelines for expanding the public API:

- Functions *must not* steal references
- Functions *must not* return borrowed references
- Functions returning references *must* return a strong reference


Include/internal: The internal API
==


With PyAPI_FUNC or PyAPI_DATA
-

Functions or structures in ``Include/internal/`` defined with
``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are
exposed only for specific use cases like debuggers and profilers.


With the extern keyword
---

Functions in ``Include/internal/`` defined with the ``extern`` keyword
*must not and can not* be used outside the CPython code base.  Only
built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN``
macro defined) can use such functions.

When in doubt, new internal C functions should be defined in
``Include/internal`` using the ``extern`` keyword.

Victor


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5UURMDSQUGSNZEUDUSQNHWRZIUKDIZJH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] New name for the development branch [was Steering Council update for February]

[Mark Shannon]

Hi,

Why choose "main" as a replacement for "master"?
It's the development branch, shouldn't we call it "development" or "dev"?

We give release branches meaningful names, so why give the development 
branch the not-so-meaningful name "main".
From a user's perspective the "main" branch is whatever their version 
of Python is built from.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HRHPZSNZ7INSK2NLE6LNYMMP767AIVRT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Mark Shannon]

Hi Antoine,

On 05/03/2021 4:07 pm, Antoine Pitrou wrote:

On Fri, 5 Mar 2021 15:03:59 +
Mark Shannon  wrote:


There are two issues here. Portability and changes to behaviour.

Regarding portability, I have to admit that PEP is rather vague.
That's my fault; I should have done more implementation first :(
FWIW, I have an implementation that should be portable.
https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation


I looked through this diff and I'm not sure how this works robustly.

This seems to assume:
- each thread's stack size is known and is a compile-time constant
We just need to know that the stack size is at least the compile-time 
constant, we don't need to know exactly what it is.


If the stack size is less than the compile-time constant, then a crash 
is a possibility. However, since PEP 651 would reduce stack consumption, 
a crash would have been less likely than it currently is.


For all supported platforms, there is a default stack size and it is 
considerably larger than the chosen value of 512k.



- Python owns the thread and can compute the stack limit reliably from
   the current stack frame
There are two points of entry into Python code. From the high level API 
and thread_run().
Since thread_run is called when a thread is started, it is guaranteed to 
have the full stack available.
The high level API (PyRun_SimpleStringFlags and friends) is a bit more 
problematic. If there isn't sufficient stack space, then a crash is a 
possibility.
But, this is no worse than the status quo, and probably better due to 
reduced C stack use.



- the OS allocates stack pages in units of BLOCK_SIZE or more (currently
   8kiB)
The BLOCK_SIZE is just a number. A larger numbers means fewer slow 
checks, but wastes more stack. It doesn't matter what the underlying 
platform's granularity is.



- the OS doesn't use a segmentation scheme that limits stack accesses
   with a finer granularity than page (or "block") size

I don't think that would matter.


- it's ok to write memory beyond the current stack top (I can imagine
   verification tools such as Valgrind complaining about that, though
   this can be worked around using suppressions)

It's not much of a stack if you can't grow it :)
I wouldn't be surprised if valgrind complains, though.



I would be curious if you could elaborate on these points.


The above scheme does require some care from users of the high-level 
C-API and should be tested thoroughly for new platforms.

But it is a *lot* more robust than what we have now.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WT6I75DELQM43VP5QYAYQGMG3RFPHA3F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice

[Mark Shannon]

Hi,

Thanks for taking the time to consider the PEP.

Although the PEP was rejected, I still believe that the safety 
guarantees in PEP 651 are worth adding to Python in the future.


To do that (maybe for 3.11), I need to understand your concerns better.

Would you clarify a few points for me?

On 03/03/2021 7:19 pm, Python Steering Council wrote:

Hi Mark,

Thank you for submitting PEP 651. The Steering Council has spent the past two 
weeks reviewing PEP 651. After careful consideration, we have decided to reject 
the PEP. The following were the key points that led us to this decision:

* The benefits are not compelling enough. Deep recursion is not a common tool in
   Python, and even with PEP 651 it would not be efficient enough to make it a 
common
   tool.

* The benefit of PEP 651 is negated as soon as a non-Python function is 
involved in the
   recursion, making the likelihood of it being useful even smaller. It also 
creates
   easy pitfalls for users who do end up relying on recursion.


Could you give an example pitfall?



* We believe the PEP understates the disruption created by the technical 
solution of
   multiple Python stack frames per C call. Although this may be solvable, it 
will
   certainly cause substantial disruption to existing debuggers, tracers, and 
state
   inspection tools as they need to adapt to this change (which may not be 
trivial).


This is presumably the key objection.
Is there a particular tool that you feel would be problematic?
I have only looked at gdb and py-spy.



* As the way to approach this will be platform-specific (as some parts of the 
proposal
   are not portable), this can cause generic Python code to behave differently 
on
   different platforms, making this kind of code less portable and less 
predictable.



There are two issues here. Portability and changes to behaviour.

Regarding portability, I have to admit that PEP is rather vague.
That's my fault; I should have done more implementation first :(
FWIW, I have an implementation that should be portable.
https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation

Regarding changes to behaviour, I don't see how "generic" Python code 
would behave differently on different platforms, except for cases where 
it already does.


In some cases, the PEP would have improved the situation.

For example:
sys.setrecursionlimit(5000)
def f():
f()

Currently, it raises a RecursionError on linux, but crashes the 
interpreter on Windows.

With PEP 651 it would have raised a RecursionError on both platforms.

Am I missing something here?


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L3U7FRGRFAXOVLA6YEAG3TSQB7EDQ2TU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Mark Shannon]

Hi Tobias,

On 19/02/2021 5:57 pm, Tobias Kohn wrote:
> Hi Mark,
>
> Quoting Mark Shannon mailto:m...@hotpy.org>>:
>
>> [...]
>>
>> If you had published these "more complicated, powerful protocols", you
>> might be able to claim that this is a "rehash".
>> But you didn't.
>>
> I would say that these ideas have been quite prominently published:
> https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol

But they are not referenced in PEP 634. I shouldn't have to trawl the 
internet to find the rejected ideas section.


>
> https://dl.acm.org/doi/10.1145/3426422.3426983

That paper describes a `__match__` method, which is absent from PEP 634.
Why?


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L7REZUBVMDRPRXX2TMWYMOWGISRGDHJR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Mark Shannon]

Hi Brandt,

On 18/02/2021 5:32 pm, Brandt Bucher wrote:

Thanks for taking the time to work on this, Mark.

Overall, I'm skeptical of this proposal. It seems like it takes a lot of "simple" things 
and makes them quite complex, and takes many "static" things and makes them quite 
dynamic. I feel that it also misrepresents certain aspects of PEP 634.

Here's a more detailed commentary:


Pattern matching will be more usable for complex classes, by allowing classes 
more control over which patterns they match.


I fear this is at the expense of most simple classes, which currently "just 
work" with PEP 634. I'm not convinced that making it easier for very complicated 
classes (like those in `sympy`) to participate at the expense of everyday classes is a 
win.


In my experience, nothing "just works".
It means "mostly works most of the time, and occasionally goes horribly 
wrong"





For comparison, here is what (as I understand it) each PEP requires for a class 
to be used in a pattern such as `C(x, y, z)`:

```py
class C:
 """A PEP 634 matchable class."""
 __match_args__ = ...

class C:
 """A PEP 653 matchable class."""
 __match_kind__ = MATCH_CLASS
 __attributes__ = ...
 def __deconstruct__(self):
 ...
```


True, but most pure-Python classes don't have a natural order of 
attributes, so the pattern is likely to be `C(x=x,y=y,z=z)`.


In which case no extra work is required for either PEP to pattern match.



For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, 
y=y, z=z)`:

```py
class C:
 """A PEP 634 matchable class (all classes work)."""

class C:
 """A PEP 653 matchable class."""
 __match_kind__ = MATCH_CLASS
 __attributes__ = ...
 def __deconstruct__(self):
 ...
```


This is not true. MATCH_DEFAULT does what you want.



It also appears that we lose a lot of expressive "idioms" by requiring 
`__attributes__` to be complete. For example, the following attribute extractions work 
with PEP 634, but not PEP 653.

```py
match subject:
 case object(a=_):
 # Match any object with an "a" attribute.
 ...
 case object(x=x, y=y, z=z):
 # Match any object with "x", "y", and "z" attributes, extracting them.
 ...
```


You need to define what you mean by "work" :)

This behaves the same for both PEPs:

class C:
__init__(self, x, y, z):
...

match C(1,2,3):
case object(x=x, y=y, z=z):
...

This also works the same for both PEPs, albeit with different mechanisms:

match namedtuple("n", "x, y, z")(1,2,3):
case object(x=x, y=y, z=z):
...


But, with PEP 634:

>>> match(sympy.cos(x)):
>>>case object(n):
>>>...
>>> n


Which seem like "not working" to me.



This also means that matching classes like `types.SimpleNamespace` are much less powerful 
under PEP 653, since the class must know which attributes are "allowed" to be 
looked up.


Why do you say this?

This should work the same for PEP 634 and 653:

>>> match types.SimpleNamespace(a = 1):
>>> case object(a):
>>>print(a)
1



Further, this whole specification shifts most of the matching logic from the 
class in the pattern to the subject itself:

```py
match ChildA(), ChildB():
 case Parent(a, b), Parent(x, y):
 ...
```

The above pattern could have completely different rules for extracting `a` and 
`b` vs `x` and `y` if either child overrides `__deconstruct__`.  I think that 
is a mistake.


Inheritance allows sub-classes to change the behavior of their 
super-classes. Yes, it can produce weird effects if they ignore Liskov's 
substitution principle. That why the modern advice is "Favour 
composition over inheritance".


You can't prevent people writing bad code.




PEP 634 also privileges some builtin classes with a special form of matching, the 
"self" match. For example the pattern `list(x)` matches a list and assigns the 
list to `x`. By allowing classes to choose which kinds of pattern they match, other 
classes can use this form as well.


This is already fairly trivial to implement:

```py
class C:
 __match_args__ = ("_self",)
 _self = property(lambda s: s)
```


That's pretty obscure, IMO. You can't claim that it is trivial compared to:

class C:
__match_kind__ = MATCH_SELF




You can even avoid adding a `_self` attribute if you do some magic with 
descriptors... ;). We could consider provide a decorator for this somewhere in 
the stdlib if there's demonstrated need (but I'm not sure there will be).


All classes should ensure that the the value of `__match_kind__` follows the 
specification. Therefore, implementations can assume, without checking, that 
all the following are false:
`(__match_kind__ & (MATCH_SEQUENCE | MATCH_MAPPING)) == (MATCH_SEQUENCE | 
MATCH_MAPPING)`
`(__match_kind__ & (MATCH_SELF | MATCH_CLASS)) == (MATCH_SELF | MATCH_CLASS)`
`(__match_kind__ & (MATCH_SELF | MATCH_DEFAULT)) == (MATCH_SELF | 
MATCH_DEFAULT)`
`(__match_kind__ 

[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching

[Mark Shannon]

Hi,

I wish I'd read this before replaying your last email

On 18/02/2021 6:49 pm, Brandt Bucher wrote:

Brandt Bucher wrote:

For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, 
y=y, z=z)`: ...



It also appears that we lose a lot of expressive "idioms" by requiring 
`__attributes__` to be complete.



This also means that matching classes like `types.SimpleNamespace` are much less powerful 
under PEP 653, since the class must know which attributes are "allowed" to be 
looked up.


Never mind these three points... I *think* setting `__match_kind__ = 
MATCH_DEFAULT` allows for arbitrary attribute extraction like this. Perhaps 
make it bit clearer?


Not quite. It uses the instance dictionary to avoid capturing bound 
methods and other computed attributes.


I'll make this more prominent.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QNASJRQ63QV5DML3LL4WIV34NKRHLWGL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 653: Precise Semantics for Pattern Matching

[Mark Shannon]

Hi everyone,

I'd like to announce a new PEP.
https://www.python.org/dev/peps/pep-0653/

It builds on PEP 634 (Structural Pattern Matching: Specification), adding:

More precise semantics.
A bit more customization for complex classes.

Its also a bit more robust and should be faster (eventually).

The syntax is unchanged, and in most cases the semantics are the same.

As always, comments and suggestions are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O4I345VHIQUXPDJWPDA54RWVXMYEEBRM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 647 (type guards) -- final call for comments

[Mark Shannon]

Hi,

First of all, sorry for not commenting on this earlier.
I only became aware of this PEP yesterday.


I like the general idea of adding a marker to show that a boolean 
function narrows the type of one (or more?) of its arguments.

However, the suggested approach seems clunky and impairs readability.

It impairs readability, because it muddles the return type.
The function in the example returns a bool.
The annotation is also misleading as the annotation is on the return 
type, not on the parameter that is narrowed.


At a glance, most programmers should be able to work out what

def is_str_list(val: List[object]) -> bool:

returns.

But,

def is_str_list(val: List[object]) -> TypeGuard[List[str]]:

is likely to confuse and require careful reading.
Type hints are for humans as well as type checkers.



Technical review.
-

For an annotation of this kind to be useful to a checker, that checker 
must perform both flow-sensitive and call-graph analysis. Therefore it 
is theoretically possible to remove the annotation altogether, using the 
following approach:


1. Scan the code looking for functions that return boolean and 
potentially narrow the type of their arguments.

2. Inline those functions in the analysis
3. Rely on pre-existing flow-senstive analysis to determine the correct 
types.


However, explicit is better and implicit. So some sort of annotation 
seems sensible.


I would contend that the minimal:

@narrows
def is_str_list(val: List[object]) -> bool:

is sufficient for a checker, as the checker can inline anything marked 
@narrows.

Plus, it does not mislead the reader by mangling the return type.


An alternative, and more explicit, approach would be to use variable 
annotations.

So:

def is_str_list(val: List[object]) -> bool:
"""Determines whether all objects in the list are strings"""
return all(isinstance(x, str) for x in val)

might become:


def is_str_list(val: List[object]) -> bool:
"""Determines whether all objects in the list are strings"""
val: NarrowsTo[List[str]]
return all(isinstance(x, str) for x in val)

Although the above lacks flow control and is thus ambiguous without the 
convention that `NarrowsTo` only applies if the result is True.



An alternative formulation would require the annotation to dominate the 
function exit:


def is_str_list(val: List[object]) -> bool:
"""Determines whether all objects in the list are strings"""
if all(isinstance(x, str) for x in val):
val: NarrowsTo[List[str]]
return True
return False

This is unambiguous.


Finally, I would ask for a change of name.

The "Type" part is redundant, since it is a type annotation, and the 
"Guard" part is incorrect. It is only a guard when used, the function 
itself is a predicate that narrows the type of an argument. "Narrows" or 
"NarrowsTo" would be better.



Cheers,
Mark.


On 09/02/2021 4:21 pm, Guido van Rossum wrote:
I think we have reached consensus on PEP 647 in typing-sig. We have 
implementations for mypy and pyright, not sure about the rest. This PEP 
does not affect CPython directly except for the addition of one special 
item (TypeGuard) to typing.py -- it would be nice to get that in the 
3.10 stdlib.


I'm CC'ing python-dev here to see if there are any further comments; if 
not, we can initiate the approval process by creating an issue at 
https://github.com/python/steering-council.


--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NOLCFYLYAQQHXISNMPYCEOAZ7ZPFCGUW/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4LKI6IGCT7JHH5C3YQPA4SXHDQWJC5Q2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Concerns about PEP 634

[Mark Shannon]

Hi Daniel,

On 06/02/2021 7:47 pm, Daniel Moisset wrote:

Hi Mark,

I think some of these issues have already been raised and replied (even 
if no agreement has been reached). but this is a good summary, so let me 
reply with a summary of responses for this. >
On Sat, 6 Feb 2021 at 15:51, Mark Shannon <mailto:m...@hotpy.org>> wrote:


Hi,

Since a decision on PEP 634 is imminent, I'd like to reiterate some
concerns that I voiced last year.

I am worried about the semantics and implementation of PEP 634.
I don't want to comment on the merits of pattern matching in
general, or
the proposed syntax in PEP 634 (or PEP 640 or PEP 642).

Semantics
-

1. PEP 634 eschews the object model, in favour of adhoc instance
checks,
length checks and attribute accesses.

This is in contrast to almost all of the the rest of the language,
which
uses special (dunder) methods:
    All operators,
    subscripting,
    attribute lookup,
    iteration,
    calls,
    tests,
    object creation,
    conversions,
    and the with statement

AFAICT, no justification is given for this.
Why can't pattern matching use the object model?


No one has said that "we can't". It's just that "we don't have to". The 
underlying mechanisms used by pattern matching (instance check, length, 
attribute access) already have their defined protocols and support 
within the object model. It's analogous as the way in which  iterable 
unpacking didn't need to define it's own object model special methods, 
because the existing iteration mechanism used in for loops was sufficient.


You seem to be jumping on my use of the word "can't".
I should have said
"Why *doesn't* PEP 634 use the object model?"

As for unpacking, it builds on iteration in a way that is clear and precise.

For example, I can desugar:

a, b = t

into:

try:
__tmp = iter(t)
except TypeError:
raise TypeError("cannot unpack non-iterable ...")
try:
__tmp_a = next(__tmp)
__tmp_b = next(__tmp)
except StopIteration:
raise ValueError("not enough values ...")
try:
next(__tmp)
except StopIteration:
pass
else:
raise raise ValueError("too many values ...")
a = __tmp_a; b = __tmp_b

Noting that variables starting "__tmp" are invisible.

Why not do something similar for PEP 634?




This does not exclude possible future extensions to the object model to 
include a richer protocol like described in 
https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol 
(where it also describes why we're not proposing that *now*, why it can 
be done later, and why we think it's best to do it later)


I don't see how this is relevant. It is the semantics of PEP 634 as 
proposed that concerns me.





PEP 343 (the "with" statement) added the __enter__ and __exit__ methods
to the object model, and that works very well.


2. PEP 634 deliberately introduces a large area of undefined behaviour
into Python.


https://www.python.org/dev/peps/pep-0634/#side-effects-and-undefined-behavior

Python has, in general, been strict about not having undefined
behaviour.
Having defined semantics means that you can reason about programs, even
non-idiomatic ones.
[This is not unique to Python, it is really only C and C++ that have
areas of undefined behaviour]


The C standard uses a very peculiar definition of "undefined behaviour" 
(I'm not familiar with the C++ one to assert anything, I'll assume it's 
the same), where for certain set of programs, any resulting behaviour is 
valid, even at compile time (so a compiler that deletes all your files 
when trying to compile "void f() {int a[10]; a[10]=0;}" is standard 
compliant). Comparing that with the use of the term "undefined 
behaviour" in the PEP is not very useful, because even if they are the 
same words, they don't have the same meaning


If you want to compare it with the C standards, the term we'd use would 
be "implementation defined behaviour". Python has a lot of those. For 
example, the output of all these python programs can change between 
implementations (And some of these even change between versions of cpython):


  * print({3,2,1})
  * print(hash("hello"))
  * if id(1) == id(1): print("hello")
  * import sys; print(sys.getsizeof([]))

Some order of operations is also implementation dependent for example in
def foo():
    print(open("/etc/passwd")).read()
foo()

The moment where file closing happens is implementation dependent.


The existence of a few cases of undefined behaviour (e.g. race 
conditions) does not excuse adding a whole lot more. Especially when 
there is no good reason.




The sectio

[Python-Dev] Concerns about PEP 634

[Mark Shannon]

Hi,

Since a decision on PEP 634 is imminent, I'd like to reiterate some 
concerns that I voiced last year.


I am worried about the semantics and implementation of PEP 634.
I don't want to comment on the merits of pattern matching in general, or 
the proposed syntax in PEP 634 (or PEP 640 or PEP 642).


Semantics
-

1. PEP 634 eschews the object model, in favour of adhoc instance checks, 
length checks and attribute accesses.


This is in contrast to almost all of the the rest of the language, which 
uses special (dunder) methods:

  All operators,
  subscripting,
  attribute lookup,
  iteration,
  calls,
  tests,
  object creation,
  conversions,
  and the with statement

AFAICT, no justification is given for this.
Why can't pattern matching use the object model?

PEP 343 (the "with" statement) added the __enter__ and __exit__ methods 
to the object model, and that works very well.



2. PEP 634 deliberately introduces a large area of undefined behaviour 
into Python.


https://www.python.org/dev/peps/pep-0634/#side-effects-and-undefined-behavior

Python has, in general, been strict about not having undefined behaviour.
Having defined semantics means that you can reason about programs, even 
non-idiomatic ones.
[This is not unique to Python, it is really only C and C++ that have 
areas of undefined behaviour]


I can see no good reason for adding undefined behaviour. It doesn't help 
anyone.


The lack of precise semantics makes programs harder to understand, and 
it makes the language harder to implement.
If the semantics aren't specified, then the implementation becomes the 
specification.

This bakes bugs into the language and makes it harder to maintain,
as bug-for-bug compatibility must be maintained.


3. Performance

PEP 634 says that each pattern must be checked in turn.
That means that multiple redundant checks must be performed on (for 
example) a sequence if there are several mapping patterns.

This is unnecessarily slow.


Implementation
--

My main concern with the implementation is that it does too much work 
into the interpreter.

Much of that work can and should be done in the compiler.
For example, deep in the implementation of the MATCH_CLASS instruction 
is the following comment:

https://github.com/brandtbucher/cpython/blob/patma/Python/ceval.c#L981

Such complex control flow should be handled during compilation, rather 
than in the interpreter.

Giant instructions like MATCH_CLASS are likely to have odd corner cases,
and may well have a negative impact on the performance of the rest of 
the language.
It is a lot easier to reason about a sequence of simple bytecodes, than 
one giant one with context-dependent behaviour.


We have spent quite a lot of effort over the last few years streamlining 
the interpreter.

Adding these extremely complex instructions would be a big backward step.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HC6XDUASX2EELTA4L5R73BSYNJPTAYNL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Mark Shannon]

Hi Sven,

On 04/02/2021 9:06 am, Sven R. Kunze wrote:

On 03.02.21 23:37, Nick Coghlan wrote:


No, PEP 558 doesn't remove it, it enhances it to be a live view of the 
frame state instead of an inconsistently updated snapshot.



As long as it is possible to **write** to existing keys to **add new 
keys** to frame.f_locals, I am actually quite happy.


Out of interest, why would you want to add new keys to the locals of a 
function frame?

The function will never be able to use those values.




The potential incompatibility Mark is referring to is the fact that 
even optimised frames currently allow writing to arbitrary keys in 
frame.f_locals and making the bound values visible to the locals() 
builtin and other consumers of frame.f_locals.


For PEP 558, it's an open question as to whether that behaviour will 
become limited to the PyEval_GetLocals() backwards compatible C API, 
with the updated Python frame API instead throwing KeyError for 
attempts to write to unknown keys on optimised frames.


So, it seems the restricting behavior is for special cases only (not 
including with-statements which I would care about).



I just wanted to mention that accessing problematic parts of the 
interpreter's internals can be quite helpful when doing concept-proofing 
or working the way through until a "batteries-included" solution exist.


Thanks,
Sven



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M7N3SNYPNPAAL5TF2BFMUCYWAT6WVARU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Constructing expected_opinfo_* lists in test_dis.py

[Mark Shannon]

Hi Skip,

On 01/02/2021 9:50 pm, Skip Montanaro wrote:

Guido> Maybe these lines in test_dis.py?
...
Skip> Thanks, I'll take a look. I was expecting there'd be a standalone
Skip> script somewhere. Hadn't considered that comments would be hiding
Skip> code.

Indeed, that did the trick, however... I'm a bit uncomfortable with
the methodology. It seems test_dis is using the same method
(dis.get_instructions) to both generate the expected output and verify
that dis.get_instructions works as expected. For the most part, you
see the test case fails, rerun the code to generate the list,
substitute, et voila! The test (magically) passes. Somewhere along the
way, it seems there should be a way to alert the user that perhaps
dis.get_instructions is broken and its output is not to be trusted
completely.


The problem is not that dis.get_instructions can't be trusted, but that 
the test isn't testing the dis module at all. It is testing whether the 
output from the compiler has changed.

A lot of the tests in test_dis do that.

Cheers,
Mark.



Skip
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FAI7XYMYO3CGKJDU3WBD2AJ6Z6SEDPYD/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DDLDPU4SYEGM3QACYEEU6BN6HZGMIDM2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] More comments on PEP 558

[Mark Shannon]

Hi Nick,

A couple more issues with PEP 558, as I see it.


Lack of specification
-

There is no specification section in PEP 558.

Much of PEP 558 describes the differences between the current behavior 
and what PEP 558 proposes. These differences are often intermingled with 
a discussion of the flaws with the current behavior, which makes it 
difficult to understand what the proposed behavior is.


I could not implement PEP 558 from the PEP alone.


Cycles and the cost of maintaining the f_locals cache
-

PEP 558 proposes that f_locals acts as a proxy mapping to the locals in 
a frame *and* that the frame caches the proxy. This obviously creates a 
cycle, which needs to be broken by the cycle GC.


Although the cycle only exists if f_locals is actually used, there is 
still some overhead for *every* call and return, as the f_locals field 
needs to be initialized on call and checked for non-NULL on return.



Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7TKPMD5LHCBXGFUIMKDAUZELRH6EX76S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Mark Shannon]

Hi Nick,

On 30/01/2021 4:44 am, Nick Coghlan wrote:

On Sat, 30 Jan 2021 at 10:30, Nick Coghlan  wrote:

On Sat, 30 Jan 2021, 12:13 am Mark Shannon,  wrote:

With a direct proxy coherency is not an issue.


For things in the frame, it *is* a direct proxy - reads pull from the frame 
object, and writes go to both the frame object and the mapping: 
https://github.com/python/cpython/pull/3640/files#diff-7b8cef249e5cca077d30de4e428a6bde6b9b803464e790e9cffa7e052e19efddR1315


Reviewing the code again, I'd misremembered how the draft
implementation works - right now, reads are relying on the snapshot
being up to date.


Given the resulting C API compatibility break and the loss of code sharing, I 
don't think it's a net win to drop the extra storage.


I'm not sure what you mean by "loss of code sharing", but I don't see 
how a mapping that is both a cache and a proxy can be easier to 
implement than a mapping that is just a proxy.


"reads are relying on the snapshot being up to date", doesn't that just 
mean that reads can be wrong?


There is no way to keep a snapshot reliably up to date in the presence 
of threads.




For now, I have a PR adding this as an open question:
https://github.com/python/peps/pull/1787/files

Given the performance benefit of being able to more reasonably drop
the implicit call to `PyFrame_LocalsToFast()`, I'm mostly convinced
that switching reads to pull from the frame if possible is the right
thing to do, even if it reduces the amount of code that can be
inherited without modification from MappingProxyType.

The API compatibility concerns would mean the extra mapping store
still needed to stick around, but it would only be used for: >
* providing backwards compatibility for the `PyEval_GetLocals()` and
C-level `f_locals` interfaces
* reading and writing names that don't have entries in the `fast_refs` mapping
* writing shadow updates for names in the `fast_refs` mapping


Given that f_locals is broken, why is keeping compatibility for this 
obscure, and probably unused case worthwhile?


The break in compatibility with locals() seems much more intrusive, yet 
you are OK with that (as am I).


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OEJKE6JB6V4KPFUQI7MP4Q35UTBKGTXF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Comments on PEP 558

[Mark Shannon]

Hi Nick,

On 29/01/2021 1:21 pm, Nick Coghlan wrote:
On Thu, 28 Jan 2021, 11:18 pm Mark Shannon, <mailto:m...@hotpy.org>> wrote:



Hi Nick,

Regarding `f_locals` PEP 558 states:

"""
Instead of being a direct reference to the internal dynamic snapshot
used to populate the independent snapshots returned by locals(),
frame.f_locals will be updated to instead return a dedicated proxy type
(implemented as a private subclass of the existing
types.MappingProxyType) that has two internal attributes not exposed as
part of the Python runtime API:

*    mapping: an implicitly updated snapshot of the function local
variables and closure references, as well as any arbitrary items that
have been set via the mapping API, even if they don't have storage
allocated for them on the underlying frame
*    frame: the underlying frame that the snapshot is for

"""

This seems rather complex, and consequently fragile.
I fear that this is just going to result in different bugs, rather than
fixing the bugs it proposes to fix.

Why not just make `f_local` a direct view on the underlying frame?
It would be simpler to understand, more robust, and should perform
better.


The concern I have with the simplification is that I don't know what 
would break if trace hooks lost the ability to stash additional state 
that the compiler doesn't know anything about in f_locals.


Do you know of any tools do that? It seems highly unlikely to me.
In fact tool authors are asking for less state, not more: 
https://bugs.python.org/issue42197

(for performance, rather than correctness reasons, I should note)



Rather than trying to assess how common such usage is, and whether we 
care about breaking any use cases that people have for it, I instead 
elected to just keep it working.


"keep it working" implies that it works now. It doesn't.
https://bugs.python.org/issue30744



The extra implementation complexity beyond what's already needed to cope 
with closure cells also isn't that much.


It is a lot more complex, because you need to worry about coherency. 
With a direct proxy coherency is not an issue.


https://bugs.python.org/issue30744 shows that the interactions between
stateful local caches, nonlocals, and concurrency is complex and likely 
to be buggy. If you are going to substitute one stateful construct for 
another, then you need to provide evidence that it won't introduce new bugs.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y7MAAIETRGYRJQ3GI6V75BCUCZXXEVGE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Why aren't we allowing the use of C11?

[Mark Shannon]

Hi everyone,

PEP 7 says that C code should conform to C89 with a subset of C99 allowed.
It's 2021 and all the major compilers support C11 (ignoring the optional 
parts).


C11 has support for thread locals, static asserts, and anonymous structs 
and unions. All useful features.


Is there a good reason not to start using C11 now?

Cheers,
Mark.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE7PRFXBXN2QY6VNPKUTM6I7OD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Comments on PEP 558

[Mark Shannon]

Hi Nick,

Regarding `f_locals` PEP 558 states:

"""
Instead of being a direct reference to the internal dynamic snapshot 
used to populate the independent snapshots returned by locals(), 
frame.f_locals will be updated to instead return a dedicated proxy type 
(implemented as a private subclass of the existing 
types.MappingProxyType) that has two internal attributes not exposed as 
part of the Python runtime API:


*mapping: an implicitly updated snapshot of the function local 
variables and closure references, as well as any arbitrary items that 
have been set via the mapping API, even if they don't have storage 
allocated for them on the underlying frame

*frame: the underlying frame that the snapshot is for

"""

This seems rather complex, and consequently fragile.
I fear that this is just going to result in different bugs, rather than 
fixing the bugs it proposes to fix.


Why not just make `f_local` a direct view on the underlying frame?
It would be simpler to understand, more robust, and should perform better.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A3UN4DGBCOB45STE6AQBITJFW6UZE43O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Does the C-API include C structs not explicitly documented?

[Mark Shannon]

Hi everyone,

Are C structs part of the C-API if the struct is only included in a 
header, but not documented?


I have always assumed not, but it isn't clear.

This question arose in https://github.com/python/cpython/pull/24298

Obviously, any structs listed in 
https://docs.python.org/3/c-api/structures.html are part of the API


Thoughts?

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S45H4YOOHCFRGLDIU4XRTPRANP5D2KSD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Resurrecting PEP 558 (defined semantics for locals())

[Mark Shannon]

Hi,

PEP 558 seems to be dormant, if not abandoned.

There are at least two open issues for bugs resulting from the currently 
weird and inefficient behavior of `f_locals` and `locals()`.
See https://bugs.python.org/issue30744 for an example of undesirable 
behaviour.


PEP 588, or something like it, would fix those.

I'd be happy to take over the PEP, or write a new one.
I like PEP 588, although I would propose a simplification.

The PEP mentions "tracing mode" and changes behavior according to 
whether a program is in "tracing mode" or not. I'd like to remove this

distinction.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651: Robust Overflow Handling, version 2

[Mark Shannon]

Hi Guido,

On 22/01/2021 5:20 am, Guido van Rossum wrote:
On Wed, Jan 20, 2021 at 9:16 AM Mark Shannon <mailto:m...@hotpy.org>> wrote:


I've updated the PEP in light of my experiments and feedback.
The new API is simpler and a bit more backwards compatible.

https://www.python.org/dev/peps/pep-0651
<https://www.python.org/dev/peps/pep-0651/>


Minor question: what's the `where` argument to `Py_CheckStackDepth()` for?


The same as for Py_EnterRecursiveCall(where); it gets incorporated into 
the exception message.




The implementation section doesn't mention how you intend to avoid the C 
stack for Python-to-Python calls. I have some idea, but it would be nice 
if you explained this a bit more. (I'm guessing you are planning to move 
some state from local variables into the frame object, store a "return 
location" (really frame object + bytecode program counter) somewhere, 
and then jump to the top of `_PyEval_EvalFrameDefault()`, or at least 
somewhere near the top. Of course this requires adjusting the various 
`CALL_*` opcodes to check whether the target is a Python function or 
not. On `RETURN_VALUE` you reverse the process, but only if the call 
came from the previous mechanism. Do I get a lollipop? :-)


Yes you do. 🍭

I'll expand the PEP a bit to include that information.



I'm guessing you needn't do anything for generators -- IIRC recursion 
through `next()` is not possible, since an active generator cannot be 
entered recursively.


Yes.
It might be nice to handle generators as well, but it is not necessary.

Cheers,
Mark.



--
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)
/Pronouns: he/him //(why is my pronoun here?)/ 
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HK7RWACEOBQDFGTXP3XPZVLBKRNU4KPT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 651: Robust Overflow Handling, version 2

[Mark Shannon]

Hi,

I've updated the PEP in light of my experiments and feedback.
The new API is simpler and a bit more backwards compatible.

https://www.python.org/dev/peps/pep-0651/

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PZWV3ENNDOSJLSE2C7CJNX5ZZVSZFGNT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

Hi Pablo,

On 20/01/2021 3:18 pm, Pablo Galindo Salgado wrote:

 >It depends on what you mean by "similar tools".

Any 3rd party tool or debugger that is printing merged stacks. (There 
are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)



For in-process tools, then the API will continue to work.
For out-of-process debuggers, then the author's are on their own.
But that's always been the case.
The source code is public, and it won't be any more opaque than it is at
the moment :)


I agree with that, but we need to have in mind that this will break
these tools and therefore we need to be aware on how difficult will
be to overcome the new situation. Is not the same that a small or
medium change is required than the case where is virtually impossible
to overcome. If we were designing the interpreter from scratch that would
not matter, but with an existing ecosystem of tools, it does. Breaking tools
in an established ecosystem has always been a major complaint of users 
when we do major releases.


Do you have any specific examples?

I feel the responsibility for this belongs with the tool authors.
If they aren't using the C-API, or requesting an alternative interface 
should the API be insufficient, then they can't really complain.




Notice as well that the gdb helpers *are* an out-of-process tool.


True, but they are in the standard library, so they are our 
responsibility to fix.


Cheers,
Mark.



Of course, I don't imply that this should be a show stopper or similar, 
but it should be taken into consideration along all the other pros and cons.


Regards from cloudy London,
Pablo Galindo Salgado

On Wed, 20 Jan 2021 at 15:12, Mark Shannon <mailto:m...@hotpy.org>> wrote:


Hi Pablo,

On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
 > Hi Mark,
 >
 > Thanks for gathering this proposal! Looks very interesting. I
have some preliminary questions: how is this going to affect
 > the "py-bt" command of the gdb helpers

(https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
 > and other similar tools that produce a unified Python-C
backtrace? There are several debuggers that
 > rely on the fact that they can merge the C stack and the Python
stack by substituting every call to "_PyEval_EvalFrameDefault"
 > and friends for the respective Python frame obtained separately
or by inspecting the frame object in the stack (like gdb does).

I don't see any problem fixing up the gdb helpers. The code will
need to
be aware that C frames will be one-to-many to Python frames, but
there's
nothing tricky there.

 >
 > Is this change going to affect these tools or they will continue
working as before? In case this change will affect this tools,
 > is there any workaround to produce the unified C/Python call
stack given the Python stack and the C stack?

It depends on what you mean by "similar tools".
For in-process tools, then the API will continue to work.
For out-of-process debuggers, then the author's are on their own.
But that's always been the case.
The source code is public, and it won't be any more opaque than it
is at
the moment :)

Cheers,
Mark.

 >
 > Kind regards,
 > Pablo Galindo Salgado
 > ___
 > Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
 > To unsubscribe send an email to python-dev-le...@python.org
<mailto:python-dev-le...@python.org>
 > https://mail.python.org/mailman3/lists/python-dev.python.org/
 > Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
 > Code of Conduct: http://python.org/psf/codeofconduct/
 >


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HHEKGU66A5LZRTQDZWO5627MK5CL37NC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

Hi Pablo,

On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:

Hi Mark,

Thanks for gathering this proposal! Looks very interesting. I have some 
preliminary questions: how is this going to affect
the "py-bt" command of the gdb helpers 
(https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
and other similar tools that produce a unified Python-C backtrace? There are 
several debuggers that
rely on the fact that they can merge the C stack and the Python stack by substituting 
every call to "_PyEval_EvalFrameDefault"
and friends for the respective Python frame obtained separately or by 
inspecting the frame object in the stack (like gdb does).


I don't see any problem fixing up the gdb helpers. The code will need to 
be aware that C frames will be one-to-many to Python frames, but there's 
nothing tricky there.




Is this change going to affect these tools or they will continue working as 
before? In case this change will affect this tools,
is there any workaround to produce the unified C/Python call stack given the 
Python stack and the C stack?


It depends on what you mean by "similar tools".
For in-process tools, then the API will continue to work.
For out-of-process debuggers, then the author's are on their own.
But that's always been the case.
The source code is public, and it won't be any more opaque than it is at 
the moment :)


Cheers,
Mark.



Kind regards,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y3NKLDE4ZL7SQQ6UWPHR73KA5RUY4BQI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

On 19/01/2021 5:48 pm, Guido van Rossum wrote:

I'm not clear on how you plan to implement this in CPython.

I can totally see that if a Python function calls another Python 
function, you can avoid the C stack frame and hence you can have as many 
Python call levels as you want.


However, there are many scenarios where a Python function calls a C 
function (e.g. `filter()`, or `dict.__setitem__()`) and that C function 
at some point calls a Python function (e.g. the `__hash__()` method of 
the key, or even the `__del__()` method of the value being replaced). 
Then that Python function can recursively do a similar thing.


Indeed, that is the second case below, where a Python __add__
function recursively performs addition. Most likely, the C stack will 
get exhausted before the recursion limit is hit, so you'll get a 
StackOverflow exception.




Are you proposing to also support that kind of thing to go on for a 
million levels of C stack frames?


No, most likely 10k to 20k calls before a StackOverflow exception.



(Do we even have a cross-platform way of avoiding segfaults due to C 
stack overflow?)


Arithmetic and comparisons on pointers from within the C stack may not 
strictly conform to the C standard, but it works just fine.
It's pretty standard VM implementation stuff. All the JVMs do this sort 
of thing.


IMO practically beats purity in this case, and crashing less has to be a 
good thing.



A rather hacky proof of concept for the stack overflow handling is here:

https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation


Cheers,
Mark.



On Tue, Jan 19, 2021 at 5:38 AM Mark Shannon <mailto:m...@hotpy.org>> wrote:


Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently
from runaway recursion. This would allow programs to set the maximum
recursion depth to fit their needs and provide additional safety
guarantees.

The following program will run safely to completion:

      sys.setrecursionlimit(1_000_000)

      def f(n):
          if n:
              f(n-1)

      f(500_000)

The following program will raise a StackOverflow, without causing a VM
crash:

      sys.setrecursionlimit(1_000_000)

      class X:
          def __add__(self, other):
              return self + other

      X() + 1

---

The full PEP can be found here:
https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-le...@python.org
<mailto:python-dev-le...@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
Code of Conduct: http://python.org/psf/codeofconduct/



--
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)
/Pronouns: he/him //(why is my pronoun here?)/ 
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PYJHVAA63AWOL72OJMNFDY7VODIT5KM7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

On 19/01/2021 4:15 pm, Antoine Pitrou wrote:

On Tue, 19 Jan 2021 15:54:39 +
Mark Shannon  wrote:

On 19/01/2021 3:40 pm, Antoine Pitrou wrote:

On Tue, 19 Jan 2021 13:31:45 +
Mark Shannon  wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.


On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth()
would look like?


It would depend on the platform, but a portable-ish implementation is here:

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71


This doesn't tell me how `stack_limit_pointer` is computed or estimated
:-)


It's nothing clever, and the numbers I've chosen are just off the top of 
my head.


https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Modules/_threadmodule.c#L1071






___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5HELIHBQATVIDWT53MJZTPFUEG5CKSOQ/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PXVS2I3UWTG2CDQUZ57IQ6NNCJ2JAT23/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

On 19/01/2021 3:43 pm, Sebastian Berg wrote:

On Tue, 2021-01-19 at 13:31 +, Mark Shannon wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently
from runaway recursion. This would allow programs to set the maximum
recursion depth to fit their needs and provide additional safety
guarantees.

The following program will run safely to completion:

  sys.setrecursionlimit(1_000_000)

  def f(n):
  if n:
  f(n-1)

  f(500_000)

The following program will raise a StackOverflow, without causing a
VM
crash:

  sys.setrecursionlimit(1_000_000)

  class X:
  def __add__(self, other):
  return self + other

  X() + 1




This is appreciated! I recently spend quite a bit of time trying to
solve a StackOverflow like this in NumPy (and was unable to fully
resolve it).  Of course the code triggering it was bordering on
malicious, but it would be nice if it was clear how to not segfault.

Just some questions/notes:

* We currently mostly use `Py_EnterRecursiveCall()` in situations where
we need to safe-guard against "almost python" recursions. For example
an attribute lookup that returns `self`, or a list containing itself.
In those cases the python recursion limit seems a bit nicer (lower and
easier to understand).
I am not sure it actually matters much, but my question is: Are we sure
we want to replace all (or even many) C recursion checks?


Would it help if you had the ability to increase and decrease the 
recursion depth, as `Py_EnterRecursiveCall()` currently does?


I'm reluctant to expose it, as it might encourage C code authors to use 
it, rather than `Py_CheckStackDepth()` resulting in crashes.


To be robust, C code must make a call to `Py_CheckStackDepth()`.
To check the recursion limit as well would be extra overhead.



* Assuming we swap `Py_EnterRecursiveCall()` logic, I am wondering if a
new `StackOverflow` exception name is useful. It may create two names
for almost identical Python code:  If you unpack a list containing
itself compared to a mapping implementing `__getitem__` in Python you
would get different exceptions.


True, but they are different. One is a soft limit that can be increased, 
the other is a hard limit that cannot (at least not easily).




* `Py_CheckStackDepthWithHeadRoom()` is usually not necessary, because
`Py_CheckStackDepth()` would leave plenty of headroom for typical
clean-up?


What is "typical" clean up? I would hope that typical cleanup is to 
return immediately.



Can we assume that DECREF's (i.e. list, tuple), will never check the
depth, so head-room is usually not necessary?  This is all good, but I
am not immediately sure when `Py_CheckStackDepthWithHeadRoom()` would
be necessary (There are probably many cases where it clearly is, but is
it ever for fairly simple code?).


Ideally, Dealloc should call `Py_CheckStackDepth()`, but it will need
to be very cheap for that to be practical.

If C code is consuming the stack, its responsibility is to not overflow.
We can't make you call `Py_CheckStackDepth()`, but we can provide it, so 
you that will have no excuse for blowing the stack :)



What happens if the maximum stack depth is reached while a
`StackOverflow` exception is already set?  Will the current "watermark"
mechanism remain, or could there be a simple rule that an uncleared
`StackOverflow` exception ensures some additional head-room?


When an exception is "set", the C code should be unwinding stack,
so those states shouldn't be possible.

We can't give you extra headroom. The C stack is a fixed size.
That's why `Py_CheckStackDepthWithHeadRoom()` is provided, if 
`Py_CheckStackDepth()` fails then it is too late to do much.


Cheers,
Mark.



Cheers,

Sebastian




---

The full PEP can be found here:
https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
Code of Conduct: http://python.org/psf/codeofconduct/




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N456CVKWZ3E3VKPOE2DZMFLVSMOK5BSF/
Code of Conduct: http://python.org/psf/codeofconduct/


___

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

On 19/01/2021 3:40 pm, Antoine Pitrou wrote:

On Tue, 19 Jan 2021 13:31:45 +
Mark Shannon  wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.


On the principle, no objection.

In practice, can you show how an implementation of Py_CheckStackDepth()
would look like?


It would depend on the platform, but a portable-ish implementation is here:

https://github.com/markshannon/cpython/blob/pep-overflow-implementation/Include/internal/pycore_ceval.h#L71



Also, what is the `headroom` argument in
Py_CheckStackDepthWithHeadroom() supposed to represent? Bytes? Stack
frames?


Bytes. I'll update the PEP.



Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XG6IU6A7ZGXVMF2TXZXOZ32SIKMAHB5X/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G5HQG6NMIU3XSI5TMPDBMHM623WY2YPV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Mark Shannon]

On 19/01/2021 2:38 pm, Terry Reedy wrote:

On 1/19/2021 8:31 AM, Mark Shannon wrote:

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently 
from runaway recursion. This would allow programs to set the maximum 
recursion depth to fit their needs and provide additional safety 
guarantees.


The following program will run safely to completion:

 sys.setrecursionlimit(1_000_000)

 def f(n):
 if n:
 f(n-1)

 f(500_000)


Are you sure?  On Windows, after adding the import
and a line at the top of f
     if not n % 1000: print(n)
I get with Command Prompt

C:\Users\Terry>py -m a.tem4
50
499000
498000

C:\Users\Terry>

with a pause of after 1 to multiple seconds.  Clearly did not run to 
completion, but no exception or Windows crash box to indicate such 
without the print.


In IDLE, I get nearly the same:
= RESTART: F:\Python\a\tem4.py
50
499000
498000

 RESTART: Shell
 >>>
The Shell restart indicates that the user code subprocess crashed and 
was restarted.  I checked that sys.getrecursionlimit() really returns 
1_000_000.


To show completion, do something like add global m and m+=1 in f and m=0 
and print(m) after the f call.





I'm not sure whether you are saying that this doesn't work now, that it 
can't work, or that it shouldn't work.


If that it doesn't work now, then I agree. That's why I've written the 
PEP; it should work.


If either of the other two, why?

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3F5S7G57GXRZ2C4E7OI5LJSTVGC6NZOI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 651 -- Robust Overflow Handling

[Mark Shannon]

Hi everyone,

It's time for yet another PEP :)

Fortunately, this one is a small one that doesn't change much.
It's aim is to make the VM more robust.

Abstract


This PEP proposes that machine stack overflow is treated differently 
from runaway recursion. This would allow programs to set the maximum 
recursion depth to fit their needs and provide additional safety guarantees.


The following program will run safely to completion:

sys.setrecursionlimit(1_000_000)

def f(n):
if n:
f(n-1)

f(500_000)

The following program will raise a StackOverflow, without causing a VM 
crash:


sys.setrecursionlimit(1_000_000)

class X:
def __add__(self, other):
return self + other

X() + 1

---

The full PEP can be found here:
https://www.python.org/dev/peps/pep-0651

As always, comments are welcome.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZY32N43YZJM3WYXSVD7OCGVNDGPR6DUM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.10 change (?) for __bool__

[Mark Shannon]

Hi Rob,

On 13/01/2021 12:18 pm, Rob Cliffe wrote:



On 12/01/2021 15:53, Mark Shannon wrote:


In master we convert `if x: pass` to `pass` which is equivalent, 
unless bool(x) has side effects the first time it is called. This is a 
recent change.
Can you please confirm that this optimisation ONLY applies to bare 
names, i.e. NOT


     if x(): pass
     if x.y: pass
     if x+y: pass



The optimization doesn't apply to the expression, but the test.
The optimizer (might) transform

if x+y: pass

to

x+y

But the expression is still evaluated.
Sorry for the confusion, I should have been clearer in my example.
It is the call to `bool()` that *might* be eliminated.

Cheers,
Mark.





Thanks
Rob Cliffe

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YZMA3ABVUKJ3PDFO4BS56QRSL4YCC3DJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.10 change (?) for __bool__

[Mark Shannon]

Hi everyone,

Should the optimizer eliminate tests that it can prove have no effect on 
the control flow of the program, even if that may eliminate some side 
effects in __bool__()?


For several years we have converted

if a and b:
...

to

if a:
if b:
...

which are equivalent, unless bool(a) has side effects the second time it 
is called.


In master we convert `if x: pass` to `pass` which is equivalent, unless 
bool(x) has side effects the first time it is called. This is a recent 
change.


This is one of those "easy to fix, if we can decide on the semantics" bugs.


Submit your thoughts to https://bugs.python.org/issue42899, please.

Cheers,
Mark.


On 12/01/2021 12:45 am, Guido van Rossum wrote:

Ah never mind. Seems to be a real bug -- thanks for reporting!

On Mon, Jan 11, 2021 at 2:57 PM Mats Wichmann > wrote:


On 1/11/21 1:00 PM, Guido van Rossum wrote:
 > All that said (I agree it's surprising that 3.10 seems backwards
 > incompatible here) I would personally not raise AttributeError but
 > TypeError in the `__bool__()` method.

eh, that was just me picking a cheap something to demo it.  the program
raises an application-specific error that I didn't feel like
defining to
keep the repro as short as possible.
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/SVGFN4DCDN462QVVMHY45IKH2XL4GVRD/
Code of Conduct: http://python.org/psf/codeofconduct/



--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BK4IUDXCZDDQCRSX3QGY7XUHOKMIDPG4/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DFX7HMZ7RFUQJMJI7MABHKEK4EOYHR4A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Advantages of pattern matching - a simple comparative analysis

[Mark Shannon]

Hi Eric,

On 23/11/2020 9:32 pm, Eric V. Smith wrote:


On 11/23/2020 3:44 PM, David Mertz wrote:
I have a little bit of skepticism about the pattern matching syntax, 
for similar reasons to those Larry expresses, and that Steve Dower 
mentioned on Discourse.


Basically, I agree matching/destructuring is a powerful idea.  But I 
also wonder how much genuinely better it is than a library that does 
not require a language change.  For example, I could create a library 
to allow this:


m = Matcher(arbitrary_expression)
if m.case("StringNode(s)"):
process_string(m.val)
elif m.case("[a, 5, 6, b]"):
process_two_free_vars(*m.values)
elif m.case("PairNone(a, b)"):
    a, b = m.values
    process_pair(a, b)
elif m.case("DictNode"):
    foo = {key, process_node(child_node) for key, child_node in 
m.values.items()}


I don't disagree that the pattern mini-language looks nice as syntax.  
But there's nothing about that mini-language that couldn't be put in a 
library (with the caveat that patterns would need to be quoted in some 
way).


I just commented on Steve's post over on Discourse. The problem with 
this is that the called function (m.case, here) needs to have access to 
the caller's namespace in order to resolve the expressions, such as 
StringNode and PairNone. This is one of the reasons f-strings weren't 
implemented as a function, and is also the source of many headaches with 
string type annotations.


My conclusion is that if you want something that operates on DSLs 
(especially ones that can't be evaluated as expressions), the compiler 
is going to need to know about it somehow so it can help you with it. I 
wish there were a general-purpose mechanism for this. Maybe it's PEP 
638, although I haven't really investigated it much, and pattern 
matching might be a bad fit for it.


Hygienic macros (PEP 638) solve two problems with a string based library 
(in my obviously biased opinion).


1. The pattern is parsed by the normal parser, so must have correct 
syntax, and the contents are visible to IDEs and editors.


if m.case("StringNode(s)"): the pattern is just a string.

case!(StringNode(s)):   the pattern is validated Python syntax.


2. The transformation is done at compile time, so the generated code 
will execute in the correct context. Basically, the macro generates the 
correct series of if/elifs for you.



Cheers,
Mark.



Eric



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CA6FJTECBEOZFKH3OC4YHT2QJWYKCTW5/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CSZHGSJ46KZF554AFCLD4FJDW42M7KH7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The semantics of pattern matching for Python

[Mark Shannon]

Short version: no!


Yes! ;)



Class patterns are an extension of instance checks.  Leaving out the 
meta-classes at this point, it is basically the class that is 
responsible for determining if an object is an instance of it.  Pattern 
matching follows the same logic, whereas Mark suggests to put that 
upside-down.  Since you certainly do not want to define the machinery in 
each instance, you end up delegating the entire thing to the class, anyway.


"Mark suggests to put that upside-down"
I have no idea what you mean by that, or the the rest of this paragraph.



I find this suggestion also somewhat strange in light of the history of 
our PEPs.  We started with a more complex protocol that would allow for 
customised patterns, which was then ditched because it was felt as being 
too complicated.  There is still a possibility to add it later on, of 
course.  But here we are with Mark proposing to introduce a complex 
protocol again.  It would obviously also mean that we could not rely as 
much on Python's existing infrastructure, which makes efficient pattern 
matching harder, again.  I completely fail to see what should be gained 
by this.


"But here we are with Mark proposing to introduce a complex protocol 
again". Again?
Please list the actual faults you see with whatever it is you are seeing 
faults with. I really don't know what you are getting at here.


If you are criticizing my proposal in 
https://github.com/markshannon/pattern-matching/blob/master/precise_semantics.rst#additions-to-the-object-model 
then address that, please.





*5. It should distinguish between a failed match and an erroneous pattern*

This seems like a reasonable idea.  However, I do not think it is 
compatible to Python's existing culture.  Let's pick up Mark's example 
of an object ``RemoteCount`` with two attributes ``success`` and 
``total``.  You can then execute the following line in Python without 
getting any issues from the interpreter::


Python has loads of runtime type-checking. If something is clearly an 
error, then why not report it?


>>> "" + 0
Traceback (most recent call last):
  File "", line 1, in 
TypeError: can only concatenate str (not "int") to str



   my_remote_count.count = 3

Python does not discover that the attribute should have been ``total`` 
rather than ``count`` here.  From a software engineering perspective, 
this is unfortunate and I would be surprised if there was anyone on this 
list who was never bitten by this.  But this is one of the prices we 
have to pay for Python's elegance in other aspects.  Requiring that 
pattern matching suddenly solves this is just not realistic.



*6. Syntax and Semantics*

There are some rather strange elements in this, such as the idea that 
the OR-pattern should be avoided.  In the matching process, you are also 
talking about matching an expression (under point 2), for instance; you 
might not really be aware of the issues of allowing expressions in 
patterns in the first place.


Please don't assume that people who disagree with you don't understand 
something or are ignorant.





---

It is most certainly a good idea to start with guiding principles to 
then design and build a new feature like pattern matching.  
Incidentally, this is what we actually did before going into the details 
of our proposal.  As evidenced by extensive (!) documentation on our 
part, there is also a vision behind our proposal for pattern matching.


In my view, Mark's proposal completely fails to provide a vision or any 
rationale for the guiding principles, other than reference to some 
mysterious "user" who "should" or "should not" do certain things.  
Furthermore, there are various obvious holes and imprecisions that would 
have to be addressed.


You've just addressed my guiding principles, but here they again:

* The semantics of pattern matching must be precisely defined.
* It must be implemented efficiently. That is, it should perform at 
least as well as an equivalent sequence of if, elif statements.

* Failed matches should not pollute the enclosing namespace.
* Objects should be able determine which patterns they match.
* It should distinguish, at much as possible, between a failed match and 
an erroneous pattern.


Where the guiding principles for PEP 634?

Cheers,
Mark.



Kind regards,
Tobias


[1]  
https://github.com/markshannon/pattern-matching/blob/master/precise_semantics.rst




Quoting Daniel Moisset mailto:dfmois...@gmail.com>>:


[sorry for the duplicate, meant to reply-all]
Thank you for this approach, I find it really helpful to put the 
conversation in these terms (semantics and guiding principles).
This is not an answer to the proposal (which I've read and helps me 
contextualize) but to your points below and how they apply to PEP-634. 
I'm also answering personally, with a reasonable guess ab

[Python-Dev] Preconditions for accepting any pattern matching PEP

[Mark Shannon]

Hi,

I'd like to request that any pattern matching PEP and its implementation
meet certain standards before acceptance.

As one of the maintainers of the AST-to-bytecode part of the compiler 
and the bytecode interpreter, I don't want to be in the situation where 
we are forced to accept a sub-standard implementation because a PEP has 
been accepted and the PEP authors are unable, or unwilling, to produce 
an implementation that is of a sufficiently high standard.


Therefore, I would ask the steering committee to require the following 
as a minimum before formal acceptance of any pattern matching PEP.


1. The semantics must be well defined.
2. There should be no global side-effects in the bytecode.
3. Each bytecode should perform a single, reasonably limited, operation.
4. There should be a clear path, using known compiler optimization 
techniques to making it efficient, if it is not initially so.


1.

We want to be able to change the implementation.

The current (3.9+) compiler produces quite clean bytecode for 
"try-finally" and "with" statements.
Earlier implementations of the compiler were simplistic and required 
quite convoluted bytecodes in the interpreter.


We were able to make these improvements because the "try-finally" and 
"with" statements are well specified, so we could reason about changes 
to the implementation.
Should the old and new implementations have differed, it was possible to 
refer to the language documentation to determine which was correct.


Without well defined semantics, the first implementation of pattern 
matching becomes the de-facto semantics, with all of its corner cases. 
Reasoning about changes becomes almost impossible.


2.

The implementation of PEP 634 can import "abc.collections" mid bytecode. 
I don't look forward to the bug reports when the module can't be 
imported for some unrelated reason and pattern matching fails.


We recently added the `LOAD_ASSERTION_ERROR` bytecode to ensure that 
asserts work even after `del builtins.AssertionError`. We should 
maintain this level of robustness.


3.

This comes down to reasoning about whether the compiler is correct, and 
interpreter performance.


Admittedly, the current bytecodes don't always adhere to the above rule, 
but they mostly do and I don't want the situation to deteriorate.


The implementation of PEP 634 includes a number of bytecodes that 
implement their own mini-interpreters within. It is the job of the 
compiler, not the interpreter, to handle such control flow.


4.

If pattern matching is added to the language, and it becomes popular, we 
don't want it to be slow. Knowing that there exists an efficient 
implementation, and how to achieve it, is important. Ideally the initial 
implementation should be efficient, but knowing that it could be is 
sufficient for acceptance.



Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RUMVGTKTMSNDOT3LTWN4ZLVFN2F6E4YO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The semantics of pattern matching for Python

[Mark Shannon]

e.


You'll just have to let go of your C-based preconceptions ;)


 6. Given the previous points, added to not seeing the gains of not
putting the or patterns into the desugared version, I'd prefer it to
be included in the desugaring.
A key point of the syntax is that if something looks like a Python 
expression, then it is a Python expression. That is hard to do with a 
"pattern or" operator embedded in the patterns.
That `0|1` means something completely different to `0+1` in PEP 634 
seems like an unnecessary trap.



 7. I think there's some surprising behaviour in the assignments being
done after a successful match but before the guard is evaluated. In
your proposal the guard has no access to the variables, so it has to
be compiled differently (using $0, $1, ... rather than the actual
names that appear in the expression). And if this guard calls a
function which exposes those variables in any way (for example if
the variable is in a closure) I think the behaviour may be
unexpected /surprising; same if I stop a debugger inside that
function and try to inspect the frame where the matching statement is.


All the "$0, $1" variables have a very limited scope. They're not 
visible outside of that scope, so they won't be visible to the debugger 
at all.


Modifying variables during matching, as you describe, is a serious flaw 
in PEP 634/642. You don't need a debugger for them to have surprising 
behavior, failed matches can change global state in an unspecified way.


Worse than that, PEP 634 comes close to blaming the user for any 
unwanted side-effects.

https://www.python.org/dev/peps/pep-0634/#side-effects-and-undefined-behavior


 8. I like your implementation approach to capture on the stack and then
assign. I was curious if you considered, rather than using a
variable number of stack cells using a single object/dict to store
those values. The compiler and the generated bytecode could end up
being simpler, and you need less stack juggling and possibly no PEEK
operation. a small list/array would suffice, but a dict may provide
further debugging opportunities (and it's likely that a split table
dict could make the representation quite compact). I know this is
less performant but I'm also thinking of simplicity.


Implement it how you please, as long as it's correct, maintainable, and 
not too slow :)



 9. I think your optimisation approaches are great, the spec was made
lax expecting for people like you to come up with a proposal of this
kind :) I don't think the first implementation of this should be
required to optimize/implement things in a certain way, but if the
spec is turned into implementation dependent and then fixed, it
shouldn't break anything (it's like the change in dictionary order
moving to "undefined/arbitrary" to "preserving insertion order") and
can be done later one


I think it is important that *all* implementations, including the first,
respect the exact semantics as defined. The first implementation should 
be reasonably efficient, but doesn't have to be super quick.


Cheers,
Mark.



Thanks again,

Daniel

On Mon, 16 Nov 2020 at 14:44, Mark Shannon <mailto:m...@hotpy.org>> wrote:


Hi everyone,

There has been much discussion on the syntax of pattern matching for
Python (in case you hadn't noticed ;)

Unfortunately the semantics seem to have been somewhat overlooked.
What pattern matching actually does seems at least as important as the
syntax.


I believe that a pattern matching implementation must have the
following
properties:

* The semantics must be precisely defined.
* It must be implemented efficiently.
* Failed matches must not pollute the enclosing namespace.
* Objects should be able determine which patterns they match.
* It should be able to handle erroneous patterns, beyond just syntax
errors.

PEP 634 and PEP 642 don't have *any* of these properties.


I've written up a document to specify a possible semantics of pattern
matching for Python that has the above properties, and includes reasons
why they are necessary.


https://github.com/markshannon/pattern-matching/blob/master/precise_semantics.rst

It's in the format of a PEP, but it isn't a complete PEP as it lacks
surface syntax.

Please, let me know what you think.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-le...@python.org
<mailto:python-dev-le...@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@pytho

[Python-Dev] Re: The semantics of pattern matching for Python

[Mark Shannon]

Hi Guido,

On 16/11/2020 4:41 pm, Guido van Rossum wrote:

Thanks Mark, this is a helpful and valuable contribution.

I will try to understand and review it in the coming weeks (there is no 
hurry since the decision is up to the next SC) but I encourage you to 
just put it in PEP form and check it into the PEP repo.


Because I only skimmed very briefly, I don't have an answer to one 
question: does your PEP also define a precise mapping from the PEP 634 
syntax to your "desugared" syntax? I think that ought to be part of your 
PEP.


No it doesn't define a precise mapping, and I don't think it will.
I'm not familiar enough with ever corner of PEP 634 to do that.
I could add a general "how to" guide though. It fairly straightforward 
conceptually but, as you know, the devil is in the details.


Cheers,
Mark.



--Guido

On Mon, Nov 16, 2020 at 6:41 AM Mark Shannon <mailto:m...@hotpy.org>> wrote:


Hi everyone,

There has been much discussion on the syntax of pattern matching for
Python (in case you hadn't noticed ;)

Unfortunately the semantics seem to have been somewhat overlooked.
What pattern matching actually does seems at least as important as the
syntax.


I believe that a pattern matching implementation must have the
following
properties:

* The semantics must be precisely defined.
* It must be implemented efficiently.
* Failed matches must not pollute the enclosing namespace.
* Objects should be able determine which patterns they match.
* It should be able to handle erroneous patterns, beyond just syntax
errors.

PEP 634 and PEP 642 don't have *any* of these properties.


I've written up a document to specify a possible semantics of pattern
matching for Python that has the above properties, and includes reasons
why they are necessary.


https://github.com/markshannon/pattern-matching/blob/master/precise_semantics.rst

It's in the format of a PEP, but it isn't a complete PEP as it lacks
surface syntax.

Please, let me know what you think.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-le...@python.org
<mailto:python-dev-le...@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/BTPODVYLPKY5IHWFKYQJICONTNTRNDB2/
Code of Conduct: http://python.org/psf/codeofconduct/



--
--Guido van Rossum (python.org/~guido <http://python.org/~guido>)
/Pronouns: he/him //(why is my pronoun here?)/ 
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EDHCIYYAGW4YRLWD6BKLTQY7FRNNTZH7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why does "except Ex as x" not restore the previous value of x?

[Mark Shannon]

Hi,

It turns out that implementing the save and restore semantics in the 
example I gave is not that difficult.


I was motivated to find out by the DLS 2020 paper on pattern matching.
It claims that introducing small scopes for variables would have to be 
implemented as a function preventing the use of normal control flow.


Since restoring the variable after an except block is a similar problem, 
I thought I'd see how difficult it was.


If anyone's interested, here's a prototype:

https://github.com/python/cpython/compare/master...markshannon:fix-exception-scoping

(This only saves globals and function-locals, class-locals and 
non-locals are unchanged. I'd probably want to emit a syntax warning for 
non-locals, as the semantics are a bit weird).


Cheers,
Mark.

On 17/11/2020 9:55 am, Mark Shannon wrote:

Hi,

I'm wondering why
```
x = "value"
try:
     1/0
except Exception as x:
     pass
```

does not restore "value" to x after
the `except` block.

There doesn't seem to be an explanation for this behavior in the docs or 
PEPs, that I can find.

Nor does there seem to be any good technical reason for doing it this way.

Anyone know why, or is just one of those unintended consequences like 
`True == 1`?



Here's an example of restoring the value of the variable after the 
`except` block:


 >>> def f(x):
... try:
... 1/0
... except Exception as x:
... pass
... return x
...
 >>> f("hi")
'hi'


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KGYRLITEPB22ZZO4N7DD4A7QP7FQS6JO/ 


Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IAY4XHLOOA572INPMP34WYXZPOSORBYU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why does "except Ex as x" not restore the previous value of x?

[Mark Shannon]

On 17/11/2020 10:22 am, Cameron Simpson wrote:

On 17Nov2020 09:55, Mark Shannon  wrote:

I'm wondering why
```
x = "value"
try:
1/0
except Exception as x:
pass
```

does not restore "value" to x after
the `except` block.


Because the except is not a new scope. So it is the same "x".

Here:

 https://docs.python.org/3/reference/compound_stmts.html#try

it says:

 When an exception has been assigned using as target, it is cleared
 at the end of the except clause. This is as if

 except E as N:
 foo

 was translated to

 except E as N:
 try:
 foo
 finally:
 del N

 This means the exception must be assigned to a different name to be
 able to refer to it after the except clause. Exceptions are cleared
 because with the traceback attached to them, they form a reference
 cycle with the stack frame, keeping all locals in that frame alive
 until the next garbage collection occurs.



Sorry, I should have made it clearer.

I'm not asking what are the semantics of the current version of Python.
I'm asking why they are that way.



Here's an example of restoring the value of the variable after the
`except` block:


def f(x):

... try:
... 1/0
... except Exception as x:
... pass
... return x
...

f("hi")

'hi'


In the Python 3.8.5 I don't see this:

 Python 3.8.5 (default, Jul 21 2020, 10:48:26)
 [Clang 11.0.3 (clang-1103.0.32.62)] on darwin
 Type "help", "copyright", "credits" or "license" for more information.
 >>> def f(x):
 ...   try:
 ... 1/0
 ...   except Exception as x:
 ... pass
 ...   return x
 ...
 >>> f(3)
 Traceback (most recent call last):
   File "", line 1, in 
   File "", line 6, in f
 UnboundLocalError: local variable 'x' referenced before assignment

and the same outside a function.



But why have we chosen for it do this?
Wouldn't restoring the value of x be a superior option?

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3JMEHLXJ7ZF2FN5ZFUIDMZHODNJYTE6A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why does "except Ex as x" not restore the previous value of x?

[Mark Shannon]

Hi Chris,

On 17/11/2020 11:03 am, Chris Angelico wrote:

On Tue, Nov 17, 2020 at 9:44 PM Steven D'Aprano  wrote:

`try...except` is no different.
...
The only wrinkle in the case of `try...except` is that the error
variable is deleted, even if it wasn't actually used. If you look at the
byte-code generated, each compound try...except with an exception
variable is followed by the equivalent of:

 err = None
 del err


There really ought to be a FAQ about this, but it has something to do
with the exception object forming a long-lasting reference cycle. To
avoid that, the error variable is nuked on leaving the compound block.


That's a much bigger wrinkle than it might seem at first, though, and
I agree, this is a quite literal frequently-asked-question and should
be made clear somewhere. The except clause is special in that, if you
want the exception afterwards, you have to reassign it to another
variable; but it doesn't ACTUALLY introduce a subscope, despite kinda
looking like it does.

Interestingly, Python 3.10 has a very odd disassembly:


def f():

... try: g()
... except Exception as e:
... print(e)
...

import dis
dis.dis(f)

   2   0 SETUP_FINALLY   10 (to 12)
   2 LOAD_GLOBAL  0 (g)
   4 CALL_FUNCTION0
   6 POP_TOP
   8 POP_BLOCK
  10 JUMP_FORWARD44 (to 56)

   3 >>   12 DUP_TOP
  14 LOAD_GLOBAL  1 (Exception)
  16 JUMP_IF_NOT_EXC_MATCH54
  18 POP_TOP
  20 STORE_FAST   0 (e)
  22 POP_TOP
  24 SETUP_FINALLY   20 (to 46)

   4  26 LOAD_GLOBAL  2 (print)
  28 LOAD_FAST0 (e)
  30 CALL_FUNCTION1
  32 POP_TOP
  34 POP_BLOCK
  36 POP_EXCEPT
  38 LOAD_CONST   0 (None)
  40 STORE_FAST   0 (e)
  42 DELETE_FAST  0 (e)
  44 JUMP_FORWARD10 (to 56)
 >>   46 LOAD_CONST   0 (None)
  48 STORE_FAST   0 (e)
  50 DELETE_FAST  0 (e)
  52 RERAISE
 >>   54 RERAISE
 >>   56 LOAD_CONST   0 (None)
  58 RETURN_VALUE




Reconstructing approximately equivalent Python code, this would mean
it looks something like this:

def f():
 try: g()
 except Exception as e:
 try:
 print(e)
 e = None
 del e
 raise
 finally:
 e = None
 del e
 except:
 raise
 return None



The equivalent Python is closer to this:

def f():
try:
g()
except Exception as e:
try:
print(e)
finally:
e = None
del e





I don't understand why (a) the "e = None; del e" part is duplicated,
nor (b) why the RERAISE opcodes are there in two branches, but I guess
it works out best to be explicit in there?



The reason for the seeming verbosity of the bytecode is that

try:
body
finally:
final

compiles to roughly:

try:
body:
except:
final
raise
else:
final

Which is why you see the duplicated sequences.

Cheers,
Mark.



Anyhow. You say that this can't come up very often because people can
go a long time without asking, but the trouble is that there are two
false interpretations that are both extremely close - either that
"except E as e:" is similar to "with E as e:", or that the except
clause creates its own scope. It's entirely possible to see supporting
evidence for your own wrong assumption and never actually know the
truth. Maybe this is going to be the next "Python has call-by-value"
vs "Python has call-by-reference" debate?

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6NKGXWLRX3SD4JQDFCOR43TAXREC33GD/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FNRZWHGCXOYL7QL5QUDR5NJS76RRTY3H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Why does "except Ex as x" not restore the previous value of x?

[Mark Shannon]

Hi,

I'm wondering why
```
x = "value"
try:
1/0
except Exception as x:
pass
```

does not restore "value" to x after
the `except` block.

There doesn't seem to be an explanation for this behavior in the docs or 
PEPs, that I can find.

Nor does there seem to be any good technical reason for doing it this way.

Anyone know why, or is just one of those unintended consequences like 
`True == 1`?



Here's an example of restoring the value of the variable after the 
`except` block:


>>> def f(x):
... try:
... 1/0
... except Exception as x:
... pass
... return x
...
>>> f("hi")
'hi'


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KGYRLITEPB22ZZO4N7DD4A7QP7FQS6JO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] The semantics of pattern matching for Python

[Mark Shannon]

Hi everyone,

There has been much discussion on the syntax of pattern matching for 
Python (in case you hadn't noticed ;)


Unfortunately the semantics seem to have been somewhat overlooked.
What pattern matching actually does seems at least as important as the 
syntax.



I believe that a pattern matching implementation must have the following 
properties:


* The semantics must be precisely defined.
* It must be implemented efficiently.
* Failed matches must not pollute the enclosing namespace.
* Objects should be able determine which patterns they match.
* It should be able to handle erroneous patterns, beyond just syntax errors.

PEP 634 and PEP 642 don't have *any* of these properties.


I've written up a document to specify a possible semantics of pattern 
matching for Python that has the above properties, and includes reasons 
why they are necessary.


https://github.com/markshannon/pattern-matching/blob/master/precise_semantics.rst

It's in the format of a PEP, but it isn't a complete PEP as it lacks 
surface syntax.


Please, let me know what you think.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BTPODVYLPKY5IHWFKYQJICONTNTRNDB2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Questions about about the DLS 2020

[Mark Shannon]

Hi Tobias,

A couple of questions about the DLS 2020 paper.

1. Why do you use the term "validate" rather than "test" for the process 
of selecting a match?


It seems to me, that this is a test, not a validation, as no exception 
is raised if a case doesn't match.



2. Is the error in the ast matching example, an intentional 
"simplification" or just an oversight?


The example:

```
def simplify(node):
match node:
case BinOp(Num(left), '+', Num(right)):
return Num(left + right)
case BinOp(left, '+' | '-', Num(0)):
return simplify(left)
case UnaryOp('-', UnaryOp('-', item)):
return simplify(item)
case _:
return node

```

is wrong.

The correct version is

```
def simplify(node):
match node:
case BinOp(Num(left), Add(), Num(right)):
return Num(left + right)
case BinOp(left, Add() | Sub(), Num(0)):
return simplify(left)
case UnaryOp(USub(), UnaryOp(USub(), item)):
return simplify(item)
case _:
return node
```

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ETZGYRCF4DR6RJXTHGXIRZXINXJ76J2D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Speeding up CPython

[Mark Shannon]

Hi Thomas,

I have to assume that this isn't a rejection of my proposal, since I 
haven't actually made a proposal to the SC yet :)


Thanks for the feedback though, it's very valuable to know the SC's 
thinking on this matter.


I have a few comments inline below.


On 04/11/2020 12:27 pm, Thomas Wouters wrote:


(For the record, I’m not replying as a PSF Director in this; I haven’t 
discussed this with the rest of the Board yet. This just comes from the 
Steering Council.)



The Steering Council discussed this proposal in our weekly meeting, last 
week. It's a complicated subject with a lot of different facets to 
consider. First of all, though, we want to thank you, Mark, for bringing 
this to the table. The Steering Council and the PSF have been looking 
for these kinds of proposals for spending money on CPython development. 
We need ideas like this to have something to spend money on that we 
might collect (e.g. via the new GitHub sponsors page), and also to have 
a good story to potential (corporate) sponsors.



That said, we do have a number of things to consider here.


For background, funding comes in a variety of flavours. Most donations 
to the PSF are general fund donations; the foundation is free to use it 
for whatever purpose it deems necessary (within its non-profit mission). 
The PSF Board and staff decide where this money has the biggest impact, 
as there are a lotof things the PSF could spend it on.



Funds can also be earmarked for a specific purpose. Donations to PyPI 
(donate.pypi.org ) work this way, for example. 
The donations go to the PSF, but are set aside specifically for PyPI 
expenses and development. Fiscal sponsorship 
(https://www.python.org/psf/fiscal-sponsorees/) is similar, but even 
more firmly restricted (and the fiscal sponsorees, not the PSF, decides +++

what to spend the money on).


A third way of handling funding is more targeted donations: sponsors 
donate for a specific program. For example, GitHub donated money 
specifically for the PSF to hire a project manager to handle the 
migration from bugs.python.org  to GitHub 
Issues. Ezio Melotti was contracted by the PSF for this job, not GitHub, 
even though the funds are entirely donated by GitHub. Similar to such 
targeted donations are grant requests, like the several grants PyPI 
received and the CZI grant request for CPython that was recently 
rejected (https://github.com/python/steering-council/issues/26). The 
mechanics are a little different, but the end result is the same: the 
PSF receives funds to achieve very specific goals.


I really don't want to take money away from the PSF. Ideally I would 
like the PSF to have more money.





Regarding donations to CPython development (as earmarked donations, or 
from the PSF's general fund), the SC drew up a plan for investment that 
is centered around maintenance: reducing the maintenance burden, easing 
the load on volunteers where desired, working through our bug and PR 
backlog. (The COVID-19 impact on PyCon and PSF funds put a damper on our 
plans, but we used much of the original plan for the CZI grant request, 
for example. Since that, too, fell through, we're hoping to collect 
funds for a reduced version of the plan through the PSF, which is 
looking to add it as a separate track in the sponsorship program.) 
Speeding up pure-Python programs is not something we consider a priority 
at this point, at least not until we can address the larger maintenance 
issues.


I too think we should improve the maintenance story.
But maintenance doesn't get anyone excited. Performance does.
By allocating part of the budget to maintenance we get performance *and* 
a better maintenance story. That's my goal anyway.


I think it is a lot easier to say to corporations, give us X dollars to 
speed up Python and you save Y dollars, than give us X dollars to 
improve maintenance with no quantifiable benefit to them.





And it may not be immediately obvious from Mark's plans, but as far as 
we can tell, the proposal is for speeding up pure-Python code. It will 
do little for code that is hampered, speed-wise, by CPython's object 
model, or threading model, or the C API. We have no idea how much this 
will actually matter to users. Making pure-Python code execution faster 
is always welcome, but it depends on the price. It may not be a good 
place to spend $500k or more, and it may even not be considered worth 
the implementation complexity.


I'll elaborate:

1. There will be a large total diff, but not that large an increase in 
code size; less than 1% of the current size of the C code base.


There would be an increase in the conceptual complexity of the interpreter,
but I'm hoping to largely offset that with better code organization.

It is perfectly possible to *improve* code quality,
if not necessarily size, while increasing performance.
Simpler code is often faster and better algorithms do not make worse code.

2. Th

[Python-Dev] Re: Thoughts on PEP 634 (Structural Pattern Matching)

[Mark Shannon]

Hi Brandt,

On 30/10/2020 4:09 pm, Brandt Bucher wrote:

Can we discuss whether we want pattern matching in Python and the broader 
semantics first, before dealing with low level details?


This is a huge step backward. These discussions have already taken place, over 
the last 10 years.


So what you are saying is that I'm not allowed to voice my opinions, 
because it is outside a time frame of your choosing?




Here's just a sampling:

- 
https://mail.python.org/archives/list/python-id...@python.org/thread/IEJFUSFC5GBDKFIPCAGS7JYXV5WGVAXP/
- 
https://mail.python.org/archives/list/python-id...@python.org/thread/GTRRJHUG4W2LXGDH4AU46SI3DLWXJF6A/
- 
https://mail.python.org/archives/list/python-id...@python.org/thread/EURSG3MYEFHXDDL2474PQNQZFJ3CUIOX/
- 
https://mail.python.org/archives/list/python-id...@python.org/thread/NTQEL3HRUJMULQYI6RDBTXQ2H3KHBBRO/
- 
https://mail.python.org/archives/list/python-id...@python.org/thread/NEC54II2RB3JRGHDP6PX3NOEALRAK6BV/
- 
https://mail.python.org/archives/list/python-id...@python.org/thread/T3VBUFECTLZMB424MBBGUHCI24YA4FPT/

We read all of these and more back way in March, before we even started 
brainstorming syntax and semantics.


Do we want a fancy switch statement, or a powerful expression?


It's right here that you lose me. Anyone who reduces pattern matching to "a fancy switch 
statement" probably isn't the right person to be discussing its semantics and usefulness with. 
It seems that some people just can't separate the two ideas in their mind. It's like calling a 
class a "fancy module".


Pattern matching is a fancy switch statement, if you define "fancy" 
appropriately ;)


Reducing pattern matching to some sort of switch statement is exactly 
what a good implementation should do. It's what compilers are for.

The comparison seems entirely reasonable to me.

OOI, what is the reasoning for choosing a statement, not an expression?



It's okay that you feel that way, but hopefully you'll understand if people 
start to tune out messages that contain these sorts of comments.



What does "these sorts of comments" mean?
Ones that you disagree with?

If I am wrong, please explain why in an as objective a fashion as possible.



What special method(s) should be added?


None. PEP 622 originally added one, but even that is more than we need right 
now. Some people may need to register their mappings or sequences as Mappings 
or Sequences, but otherwise that's it.


Much of the language uses special methods. Why should pattern matching 
be so different?


Why make this opt-out, rather than opt-in, given the potential for 
unwanted side effects?





I would ask anyone who wants pattern matching adding to Python, to not support 
PEP 634.


Seriously?


Yes. Absolutely.
PEP 634 is seriously flawed.



I would ask anyone who wants pattern matching added to Python to carefully 
consider the PEPs for themselves (particularly PEP 636, which is much less dry 
and contains more examples and commentary). We've written four of the largest, 
most detailed PEPs of any new feature I've seen, complete with a working 
implementation that we've made available from any browser. Of course it's not 
the *only* way of getting pattern matching... but if you want it, this is 
probably your *best* shot at getting it.
Given the size of the proposed change to the language, it really isn't 
that detailed.


The browser based implementation is nice, though :)

Cheers,
Mark.




Brandt
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DEUZMFMTDBYUD3OHB5HNN7MWWNP237VV/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UANZ3FB4C6AXX4Q4VX7FROWXRJOUQLL5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Thoughts on PEP 634 (Structural Pattern Matching)

[Mark Shannon]

Hi everyone,

PEP 634/5/6 presents a possible implementation of pattern matching for 
Python.


Much of the discussion around PEP 634, and PEP 622 before it, seems to 
imply that PEP 634 is synonymous with pattern matching; that if you 
reject PEP 634 then you are rejecting pattern matching.


That simply isn't true.

Can we discuss whether we want pattern matching in Python and
the broader semantics first, before dealing with low level details?



Do we want pattern matching in Python at all?
-

Pattern matching works really well in statically typed, functional 
languages.


The lack of mutability, constrained scope and the ability of the 
compiler to distinguish let variables from constants means that pattern 
matching code has fewer errors, and can be compiled efficiently.


Pattern matching works less well in dynamically-typed, functional 
languages and statically-typed, procedural languages.
Nevertheless, it works well enough for it to be a popular feature in 
both erlang and rust.


In dynamically-typed, procedural languages, however, it is not clear (at 
least not to me) that it works well enough to be worthwhile.


That is not say that pattern matching could never be of value in Python, 
but PEP 635 fails to demonstrate that it can (although it does a better 
job than PEP 622).



Should match be an expression, or a statement?
--

Do we want a fancy switch statement, or a powerful expression?
Expressions have the advantage of not leaking (like comprehensions in 
Python 3), but statements are easier to work with.



Can pattern matching make it clear what is assigned?


Embedding the variables to be assigned into a pattern, makes the pattern 
concise, but requires discarding normal Python syntax and inventing a 
new sub-language. Could we make patterns fit Python better?


Is it possible to make assignment to variables clear, and unambiguous, 
and allow the use of symbolic constants at the same time?

I think it is, but PEP 634 fails to do this.


How should pattern matching be integrated with the object model?


What special method(s) should be added? How and when should they be called?
PEP 634 largely disregards the object model, meaning it has many special 
cases, and is inefficient.



The semantics must be well defined.
---

Language extensions PEPs should define the semantics of those 
extensions. For example, PEP 343 and PEP 380 both did.

https://www.python.org/dev/peps/pep-0343/#specification-the-with-statement
https://www.python.org/dev/peps/pep-0380/#formal-semantics

PEP 634 just waves its hands and talks about undefined behavior, which 
horrifies me.



In summary,
I would ask anyone who wants pattern matching adding to Python, to not 
support PEP 634.

PEP 634 just isn't a good fit for Python, and we deserve something better.

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MJ4RQ3R5RGNJAZGRPZUSMFDM6QT26VR6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 626

[Mark Shannon]

Hi Pablo,

On 29/10/2020 11:08 am, Pablo Galindo Salgado wrote:

 > The new semantics may well result in some slowdowns. That's stated in
the PEP.I don't think I can reliably isolate the effects of the (very 
slight)

change in the behavior of f_lineno.

Ok, then let's make at least we measure the general slowdowns.


Except that we can't measure the performance of a specification.
We can only measure the performance of entire implementations.

I can make an implementation that conforms to PEP 626 that is slower 
than master, or I can make one that's faster :)

It doesn't change the value of the PEP itself.

Let me give you a toy example.

def f():
while 1:
body()

3.9 compiles this to:

(The trailing, implicit return has been stripped for clarity)

  3 >>0 LOAD_GLOBAL  0 (body)
  2 CALL_FUNCTION0
  4 POP_TOP
  6 JUMP_ABSOLUTE0

A naive implementation that conforms to PEP 626 would this compile to:

  2 >>0 NOP
  3   2 LOAD_GLOBAL  0 (body)
  4 CALL_FUNCTION0
  6 POP_TOP
  8 JUMP_ABSOLUTE0

But a better implementation could produce this:

  2   0 NOP
  3 >>2 LOAD_GLOBAL  0 (body)
  4 CALL_FUNCTION0
  6 POP_TOP
  2   8 JUMP_ABSOLUTE2

Which has the same bytecodes as 3.9 in the loop, and has the correct 
line numbers.


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A6RRMIGXVHV7I7QMG42BCD6K4AJBKVST/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 626

[Mark Shannon]

Hi Pablo,

On 29/10/2020 9:56 am, Pablo Galindo Salgado wrote:

 > Performance compared to what?

Compared before the patch. The comparison that I mentioned is before and 
after the PR with the PEP implementation.


PEP 626 changes the line number table and the compiler.
Such a comparison would not test the performance impact of the change to 
`f_lineno`, as it will likely be swamped by the other changes.




 > The current behavior of `f_lineno` is ill-defined, so mimicking it would
be tricky

Maybe I failed to express myself: that's fine, we don't need to mimick 
the current behaviour of f_lineno or change anything in the PEP 
regarding that. I just want to check that the new semantics do not slow 
down anything in a subtle way.


The new semantics may well result in some slowdowns. That's stated in 
the PEP.
I don't think I can reliably isolate the effects of the (very slight) 
change in the behavior of f_lineno.




 > What's the reason for supposing that it will be slower?

There is no real concern, but as there were some conversations about 
performance and the pep mentions that "the "f_lineno" attribute of the 
code object will be updated to point the current line being executed" I 
just want to make sure that updating that field on every bytecode line 
change does not affect anything. Again, I am pretty sure it will be 
negligible impact and the performance check should be just a routine 
confirmation.


When you say updating "field", are you thinking of a C struct?
That's an implementation detail.
The PEP states that the f_lineno *attribute* of the code object will be 
updated.


Note that the behavior of 3.9 is weird in some cases:

test.py:
import sys

def print_line():
print(sys._getframe(1).f_lineno)

def test():
print_line()
sys._getframe(0).f_trace = True
print_line()
print_line()

test()


$ python3.9 ~/test/test.py
7
8
8

With PEP 626 this is required to print:
7
9
10


Cheers,
Mark.



Cheers,
Pablo

On Thu, 29 Oct 2020, 09:47 Mark Shannon, <mailto:m...@hotpy.org>> wrote:


Hi,

That's great. Thanks Pablo.

On 29/10/2020 1:32 am, Pablo Galindo Salgado wrote:
 > On behalf of the steering council, I am happy to announce that as
 > BDFL-Delegate I am
 > accepting PEP 626 -- Precise line numbers for debugging and other
tools.
 > I am confident this PEP will result in a better experience for
 > debuggers, profilers and tools
 > that rely on tracing functions. All the existing concerns regarding
 > out-of-process debuggers
 > and profilers have been addressed by Mark in the latest version
of the
 > PEP. The acceptance of
 > the PEP comes with the following requests:
 >
 > * The PEP must be updated to explicitly state that the API functions
 > described in the
 >     "Out of process debuggers and profilers" must remain
self-contained
 > in any potential
 >      future modifications or enhancements.
 > * The PEP states that the "f_lineno" attribute of the code object
will
 > be updated to point to
 >     the current line being executed even if tracing is off. Also,
there
 > were some folks concerned with
 >     possible performance implications. Although in my view there
is no
 > reason to think this will impact
 >     performance negatively, I would like us to confirm that
indeed this
 > is the case before merging the
 >     implementation (with the pyperformance test suite, for example).

Performance compared to what?
The current behavior of `f_lineno` is ill-defined, so mimicking it
would
be tricky.

What's the reason for supposing that it will be slower?

Cheers,
Mark.

 >
 > Congratulations Mark Shannon!
 >
 > Thanks also toeveryone else who provided feedback on this PEP!
 >
 > Regards from rainy London,
 > Pablo Galindo Salgado


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WHBHMT5DD3AR3PKO6IQ6XDZBWTCWF3O7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 626

[Mark Shannon]

Hi,

That's great. Thanks Pablo.

On 29/10/2020 1:32 am, Pablo Galindo Salgado wrote:
On behalf of the steering council, I am happy to announce that as 
BDFL-Delegate I am

accepting PEP 626 -- Precise line numbers for debugging and other tools.
I am confident this PEP will result in a better experience for 
debuggers, profilers and tools
that rely on tracing functions. All the existing concerns regarding 
out-of-process debuggers
and profilers have been addressed by Mark in the latest version of the 
PEP. The acceptance of

the PEP comes with the following requests:

* The PEP must be updated to explicitly state that the API functions 
described in the
    "Out of process debuggers and profilers" must remain self-contained 
in any potential

     future modifications or enhancements.
* The PEP states that the "f_lineno" attribute of the code object will 
be updated to point to
    the current line being executed even if tracing is off. Also, there 
were some folks concerned with
    possible performance implications. Although in my view there is no 
reason to think this will impact
    performance negatively, I would like us to confirm that indeed this 
is the case before merging the

    implementation (with the pyperformance test suite, for example).


Performance compared to what?
The current behavior of `f_lineno` is ill-defined, so mimicking it would 
be tricky.


What's the reason for supposing that it will be slower?

Cheers,
Mark.



Congratulations Mark Shannon!

Thanks also toeveryone else who provided feedback on this PEP!

Regards from rainy London,
Pablo Galindo Salgado

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R3DGNMJ7WIJWBEVNK5274FXPYEMPZFJE/
Code of Conduct: http://python.org/psf/codeofconduct/