date:20200623


On 06/23/2020 10:31 PM, Chris Angelico wrote:

On Wed, Jun 24, 2020 at 3:21 PM Ethan Furman wrote:

On 06/23/2020 09:01 AM, PEP 622 wrote:


from enum import Enum

class Color(Enum):
 BLACK = 1
 RED = 2

BLACK = 1
RED = 2

match color:
 case .BLACK | Color.BLACK:
 print("Black suits every color")
 case BLACK:  # This will just assign a new value to BLACK.
 ...


As others have noted, the leading dot to disambiguate between a name assignment 
and a value check is going to be a problem.  I think it's also unnecessary 
because instead of

  case BLACK:
  blahblah()

we can do

  case _:
  # look ma! BLACK is just "color"!
  BLACK = color  # if you really want it bound to another name

In other words, the PEP is currently building in two ways to do the same thing -- make a 
default case.  One of those ways is going to be a pain; the other, once renamed to 
"else", will be perfect!  :-)  As a bonus, no special casing for leading dots.



But what if that's composed into something else?

class Room(Enum):
 LIBRARY = 1
 BILLIARD_ROOM = 2
 ...

match accusation:
 case (Color.SCARLETT, Room.BILLIARD_ROOM):
 print("Correct")
 case (Color.SCARLETT, _):
 print("Not there!")
 case (_, Room.BILLIARD_ROOM):
 print("Wrong person!")
 case (_, _):
 print("Nope. Just nope.")

Without the dots, there's no way to tell whether you're matching
specific values in the tuple, or matching by length alone and then
capturing. You can't use the 'else' keyword for a partial match.


Well, your example isn't using leading dots like `.BLACK` is, and your example isn't 
using `case _` as a final catch-all, and your example isn't using "case 
some_name_here" as an always True match.  In other words, your example isn't talking 
about what I'm talking about.  ;-)

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SX745WU7YFCTHIQSYLA4YIV4RUTLMAK4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Wed, Jun 24, 2020 at 3:21 PM Ethan Furman  wrote:
>
> On 06/23/2020 09:01 AM, PEP 622 wrote:
>
> > from enum import Enum
> >
> > class Color(Enum):
> > BLACK = 1
> > RED = 2
> >
> > BLACK = 1
> > RED = 2
> >
> > match color:
> > case .BLACK | Color.BLACK:
> > print("Black suits every color")
> > case BLACK:  # This will just assign a new value to BLACK.
> > ...
>
> As others have noted, the leading dot to disambiguate between a name 
> assignment and a value check is going to be a problem.  I think it's also 
> unnecessary because instead of
>
>  case BLACK:
>  blahblah()
>
> we can do
>
>  case _:
>  # look ma! BLACK is just "color"!
>  BLACK = color  # if you really want it bound to another name
>
> In other words, the PEP is currently building in two ways to do the same 
> thing -- make a default case.  One of those ways is going to be a pain; the 
> other, once renamed to "else", will be perfect!  :-)  As a bonus, no special 
> casing for leading dots.
>

But what if that's composed into something else?

class Room(Enum):
LIBRARY = 1
BILLIARD_ROOM = 2
...

match accusation:
case (Color.SCARLETT, Room.BILLIARD_ROOM):
print("Correct")
case (Color.SCARLETT, _):
print("Not there!")
case (_, Room.BILLIARD_ROOM):
print("Wrong person!")
case (_, _):
print("Nope. Just nope.")

Without the dots, there's no way to tell whether you're matching
specific values in the tuple, or matching by length alone and then
capturing. You can't use the 'else' keyword for a partial match.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TYKZU5HALAN3I72NBKE4QC3ZJI355DY7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 06/23/2020 09:01 AM, PEP 622 wrote:


from enum import Enum

class Color(Enum):
BLACK = 1
RED = 2

BLACK = 1
RED = 2

match color:
case .BLACK | Color.BLACK:
print("Black suits every color")
case BLACK:  # This will just assign a new value to BLACK.
...


As others have noted, the leading dot to disambiguate between a name assignment 
and a value check is going to be a problem.  I think it's also unnecessary 
because instead of

case BLACK:
blahblah()

we can do

case _:
# look ma! BLACK is just "color"!
BLACK = color  # if you really want it bound to another name

In other words, the PEP is currently building in two ways to do the same thing -- make a 
default case.  One of those ways is going to be a pain; the other, once renamed to 
"else", will be perfect!  :-)  As a bonus, no special casing for leading dots.

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CMA3U3CBIPONZPSPXWFCDZWNUZKIESPH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Nick Coghlan

On Wed., 24 Jun. 2020, 2:07 am Guido van Rossum,  wrote:

> I'm happy to present a new PEP for the python-dev community to review.
> This is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and
> Talin.
>

Very nice!

As with some others, the main thing that gives me pause is the elevation of
"_" to pseudo keyword status by making it the wildcard character for
pattern matching.

Users already find the existing common usages at least somewhat confusing
[1], so adding a 5th use case will definitely make that situation worse.

The first alternative I can see would be to adopt regex wildcard notation
for match patterns as well:

* exactly one arbitrary element: .
* any number of arbitrary elements: .*
* one or more arbitrary elements: .+
* zero or 1 arbitrary elements: .?

And then frame the ".name" reference syntax as another form of element
constraint (matching a value lookup rather than being completely arbitrary).

Alternatively, the file globbing notation, "?", could be used as a new
non-identifier symbol to indicate items that are present, but neither bound
nor constrained (exactly as "_" is currently used in the PEP).

This syntax would have the added bonus of potentially being added to
iterable unpacking, such that "a, b, *? = iterable" would mean "retrieve
the first two items without trying to retrieve more than that" (whereas the
existing throwaway variable convention still exhausts the RHS even when you
only care about the first few items).

That said, I don't think the extra confusion generated by using "_" would
be intolerable - I just think the possibility of using regex or globbing
inspired wildcard notations is worth considering, and unless I missed
something, the PEP doesn't currently cover any possible alternatives to
using "_".

Cheers,
Nick.

[1]
https://stackoverflow.com/questions/5893163/what-is-the-purpose-of-the-single-underscore-variable-in-python

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3L6ME43RYONBJES4EO34XQJJFKWYGRG3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Terry Reedy


On 6/23/2020 5:56 PM, Barry Warsaw wrote:


 match:
 expression
 case arm1:
 pass
 case arm2:
 pass
 else:
 pass

nicely mirrors try/except and if/elif/else constructs so it looks quite natural.


Agreed as to the look of the form.

Some observations:

1. 'else' is equivalent of 'elif True', so two alternative keywords are 
not strictly needed.  The proposal uses 'case _' as the parallel 
catchall to 'elif True'.  If '_' were not otherwise being used as a 
wildcard, I might suggest that 'case' by itself be used to always match.


2. 'expression' is *not* a general statement, let alone a suite of such, 
and it is not natural in non-interactive code that the object resulting 
from an expression be magically captured to be *implicitly* referenced 
in later code (the patterns).


Even though match may be thought of as a type of multiple elifs, this 
has no parallel in if/elif/else statements.  However, in try/except 
statements, raised exceptions *are* magically captured, to be 
*explicitly* referenced in 'except' tuples.  'Try' and 'match' both 
consist of a setup, multiple test:action pairs, and a default action. 
So a similar indent structure can easily seem appropriate.


3. A bonus of 'match' by itself is that re-based syntax colorizers, as 
in IDLE, could pretend that the keyword is 'match:' and avoid colorizing 
re.match, etc.  Or a colorizer could check that 'match' is followed by 
':'.  Always colorizing 'case' seems like less of an issue as I think it 
is less common as a name.


--
Terry Jan Reedy
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YTNXXREZIKBQ3TVTPRLVRJVSGAMD2DNB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread David Mertz

def whereis(point):
> match point:
> case MovingPoint(0, 0):
> print("Origin")
> case MovingPoint(0, y):
> print(f"Y={y}")
> case MovingPoint(x, 0):
> print(f"X={x}")
> case MovingPoint(1, 1):
> print("Diagonal at units")
>
case MovingPoint():
> print("Somewhere else")
> case _:
> print("Not a point")
>

What is the expected/intended behavior or this kind of point.  Both "What
matches?" and "What is the value of point.x and point.y afterwards?"

class MovingPoint:
def __init__(self, x, y):
self._x = x
self._y = y

@property
def x(self):
x = self._x
self._x +=1
return x

@property
def y(self):
y = self._y
self._y += 1
return y

point = MovingPoint(0, 0)
whereis(point)

-- 
The dead increasingly dominate and strangle both the living and the
not-yet born.  Vampiric capital and undead corporate persons abuse
the lives and control the thoughts of homo faber. Ideas, once born,
become abortifacients against new conceptions.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N4ADISQDR6NUWA37BYIQHZ3EPBYMVODP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Neil Girdhar

On Tue, Jun 23, 2020 at 9:45 PM Neil Girdhar  wrote:

> This is awesome!
>
> What I love about this is that it strongly encourages people not to do
> EAFP with types (which I've seen many times), which causes problems when
> doing type annotations.  Instead, if they use pattern matching, they're
> essentially forced to do isinstance without even realizing it.  I love
> features that encourage good coding practices by design.
>
> My question is how does this work with polymorphic types?  Typically, you
> might not want to fix everywhere the exact order of the attributes.  It
> would be a shame for you to be dissuaded from adding an attribute to a
> dataclass because it would mess up every one of your case statements.  Do
> case statements support extracting by attribute name somehow?
>
> case Point(x, z):
>
> means extract into x and y positionally.  What if I want to extract by
> keyword somehow?  Can something like this work?
>
> case Point(x=x, z=z):
>
> That way, if I add an attribute y, my case statement is just fine.
>

Ah, never mind!  You guys thought of everything.


>
>

> I like the design choices. After reading a variety of comments, I'm
> looking forward to seeing the updated PEP with discussion regarding:
> case _: vs else:
> _ vs ...
> case x | y: vs case x or y:
>
> Best,
> Neil
>
> On Tue, Jun 23, 2020 at 12:07 PM Guido van Rossum 
> wrote:
>
>> I'm happy to present a new PEP for the python-dev community to review.
>> This is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and
>> Talin.
>>
>> Many people have thought about extending Python with a form of pattern
>> matching similar to that found in Scala, Rust, F#, Haskell and other
>> languages with a functional flavor. The topic has come up regularly on
>> python-ideas (most recently yesterday :-).
>>
>> I'll mostly let the PEP speak for itself:
>> - Published: https://www.python.org/dev/peps/pep-0622/ (*)
>> - Source: https://github.com/python/peps/blob/master/pep-0622.rst
>>
>> (*) The published version will hopefully be available soon.
>>
>> I want to clarify that the design space for such a match statement is
>> enormous. For many key decisions the authors have clashed, in some cases we
>> have gone back and forth several times, and a few uncomfortable compromises
>> were struck. It is quite possible that some major design decisions will
>> have to be revisited before this PEP can be accepted. Nevertheless, we're
>> happy with the current proposal, and we have provided ample discussion in
>> the PEP under the headings of Rejected Ideas and Deferred Ideas. Please
>> read those before proposing changes!
>>
>> I'd like to end with the contents of the README of the repo where we've
>> worked on the draft, which is shorter and gives a gentler introduction than
>> the PEP itself:
>>
>>
>> # Pattern Matching
>>
>> This repo contains a draft PEP proposing a `match` statement.
>>
>> Origins
>> ---
>>
>> The work has several origins:
>>
>> - Many statically compiled languages (especially functional ones) have
>>   a `match` expression, for example
>>   [Scala](
>> http://www.scala-lang.org/files/archive/spec/2.11/08-pattern-matching.html
>> ),
>>   [Rust](https://doc.rust-lang.org/reference/expressions/match-expr.html
>> ),
>>   [F#](
>> https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching
>> );
>> - Several extensive discussions on python-ideas, culminating in a
>>   summarizing
>>   [blog post](
>> https://tobiaskohn.ch/index.php/2018/09/18/pattern-matching-syntax-in-python/
>> )
>>   by Tobias Kohn;
>> - An independently developed [draft
>>   PEP](
>> https://github.com/ilevkivskyi/peps/blob/pattern-matching/pep-.rst)
>>   by Ivan Levkivskyi.
>>
>> Implementation
>> --
>>
>> A full reference implementation written by Brandt Bucher is available
>> as a [fork]((https://github.com/brandtbucher/cpython/tree/patma)) of
>> the CPython repo.  This is readily converted to a [pull
>> request](https://github.com/brandtbucher/cpython/pull/2)).
>>
>> Examples
>> 
>>
>> Some [example code](
>> https://github.com/gvanrossum/patma/tree/master/examples/) is available
>> from this repo.
>>
>> Tutorial
>> 
>>
>> A `match` statement takes an expression and compares it to successive
>> patterns given as one or more `case` blocks.  This is superficially
>> similar to a `switch` statement in C, Java or JavaScript (an many
>> other languages), but much more powerful.
>>
>> The simplest form compares a target value against one or more literals:
>>
>> ```py
>> def http_error(status):
>> match status:
>> case 400:
>> return "Bad request"
>> case 401:
>> return "Unauthorized"
>> case 403:
>> return "Forbidden"
>> case 404:
>> return "Not found"
>> case 418:
>> return "I'm a teapot"
>> case _:
>> return "Something else"
>> ```
>>
>> Note the last block: the "variable

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Neil Girdhar

This is awesome!

What I love about this is that it strongly encourages people not to do EAFP
with types (which I've seen many times), which causes problems when doing
type annotations.  Instead, if they use pattern matching, they're
essentially forced to do isinstance without even realizing it.  I love
features that encourage good coding practices by design.

My question is how does this work with polymorphic types?  Typically, you
might not want to fix everywhere the exact order of the attributes.  It
would be a shame for you to be dissuaded from adding an attribute to a
dataclass because it would mess up every one of your case statements.  Do
case statements support extracting by attribute name somehow?

case Point(x, z):

means extract into x and y positionally.  What if I want to extract by
keyword somehow?  Can something like this work?

case Point(x=x, z=z):

That way, if I add an attribute y, my case statement is just fine.

I like the design choices. After reading a variety of comments, I'm looking
forward to seeing the updated PEP with discussion regarding:
case _: vs else:
_ vs ...
case x | y: vs case x or y:

Best,
Neil

On Tue, Jun 23, 2020 at 12:07 PM Guido van Rossum  wrote:

> I'm happy to present a new PEP for the python-dev community to review.
> This is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and
> Talin.
>
> Many people have thought about extending Python with a form of pattern
> matching similar to that found in Scala, Rust, F#, Haskell and other
> languages with a functional flavor. The topic has come up regularly on
> python-ideas (most recently yesterday :-).
>
> I'll mostly let the PEP speak for itself:
> - Published: https://www.python.org/dev/peps/pep-0622/ (*)
> - Source: https://github.com/python/peps/blob/master/pep-0622.rst
>
> (*) The published version will hopefully be available soon.
>
> I want to clarify that the design space for such a match statement is
> enormous. For many key decisions the authors have clashed, in some cases we
> have gone back and forth several times, and a few uncomfortable compromises
> were struck. It is quite possible that some major design decisions will
> have to be revisited before this PEP can be accepted. Nevertheless, we're
> happy with the current proposal, and we have provided ample discussion in
> the PEP under the headings of Rejected Ideas and Deferred Ideas. Please
> read those before proposing changes!
>
> I'd like to end with the contents of the README of the repo where we've
> worked on the draft, which is shorter and gives a gentler introduction than
> the PEP itself:
>
>
> # Pattern Matching
>
> This repo contains a draft PEP proposing a `match` statement.
>
> Origins
> ---
>
> The work has several origins:
>
> - Many statically compiled languages (especially functional ones) have
>   a `match` expression, for example
>   [Scala](
> http://www.scala-lang.org/files/archive/spec/2.11/08-pattern-matching.html
> ),
>   [Rust](https://doc.rust-lang.org/reference/expressions/match-expr.html),
>   [F#](
> https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching
> );
> - Several extensive discussions on python-ideas, culminating in a
>   summarizing
>   [blog post](
> https://tobiaskohn.ch/index.php/2018/09/18/pattern-matching-syntax-in-python/
> )
>   by Tobias Kohn;
> - An independently developed [draft
>   PEP](
> https://github.com/ilevkivskyi/peps/blob/pattern-matching/pep-.rst)
>   by Ivan Levkivskyi.
>
> Implementation
> --
>
> A full reference implementation written by Brandt Bucher is available
> as a [fork]((https://github.com/brandtbucher/cpython/tree/patma)) of
> the CPython repo.  This is readily converted to a [pull
> request](https://github.com/brandtbucher/cpython/pull/2)).
>
> Examples
> 
>
> Some [example code](
> https://github.com/gvanrossum/patma/tree/master/examples/) is available
> from this repo.
>
> Tutorial
> 
>
> A `match` statement takes an expression and compares it to successive
> patterns given as one or more `case` blocks.  This is superficially
> similar to a `switch` statement in C, Java or JavaScript (an many
> other languages), but much more powerful.
>
> The simplest form compares a target value against one or more literals:
>
> ```py
> def http_error(status):
> match status:
> case 400:
> return "Bad request"
> case 401:
> return "Unauthorized"
> case 403:
> return "Forbidden"
> case 404:
> return "Not found"
> case 418:
> return "I'm a teapot"
> case _:
> return "Something else"
> ```
>
> Note the last block: the "variable name" `_` acts as a *wildcard* and
> never fails to match.
>
> You can combine several literals in a single pattern using `|` ("or"):
>
> ```py
> case 401|403|404:
> return "Not allowed"
> ```
>
> Patterns can look like unpacking assignments, and can be used

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On 2020-06-24 01:09, Guido van Rossum wrote:
On Tue, Jun 23, 2020 at 4:41 PM MRAB > wrote:

[snip]

However, what if you wanted to match Ellipsis?

This could lead to bugs:

 >>> ...
Ellipsis
 >>> Ellipsis = 0
 >>> Ellipsis
0
 >>> ...
Ellipsis

Now you're just being silly. If you want to use Ellipsis as a variable 
you can't also use it to refer to the "..." token.

[snip]

The point here is that printing ... shows "Ellipsis" (not "..."), 
printing None shows "None", etc.

Printing Ellipsis also shows "Ellipsis", but you can bind to it. You 
can't bind to None, etc.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UZCECU4MCYBCI7EOZQM522ZCOZEKVHIX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Wed, Jun 24, 2020 at 10:11 AM Guido van Rossum  wrote:
>
> On Tue, Jun 23, 2020 at 4:41 PM MRAB  wrote:
>>
>> On 2020-06-23 22:50, Barry Warsaw wrote:
>> > On Jun 23, 2020, at 14:31, Chris Angelico  wrote:
>> >
>> >> I can't find it among the rejected alternatives, but was it considered
>> >> to use "..." as the wildcard, rather than "_"? It carries similar
>> >> meaning but its special case of "this will never be bound" is simply
>> >> preventing an error, rather than making one otherwise-valid name
>> >> special.
>> >
>> > I thought of that too as I was reading the PEP, but forgot to add it to my 
>> > notes.  I do like ellipsis more than underscore here.
>> >
>> +1
>
>
> The problem is that ellipsis already has a number of other meanings, *and* is 
> easily confused in examples and documentation with leaving things out that 
> should be obvious or uninteresting. Also, if I saw [a, ..., z] in a pattern I 
> would probably guess that it meant "any sequence of length > 2, and capture 
> the first and last element" rather than "a sequence of length three, and 
> capture the first and third elements". (The first meaning is currently 
> spelled as [a, *_, z].)
>

Ah, yes, very good point. Agreed.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QTB6GLWLB3BZU3I2PWF7TML73HF4LYRZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Tue, Jun 23, 2020 at 5:21 PM Ethan Furman  wrote:

> On 06/23/2020 04:26 PM, Guido van Rossum wrote:
> > On Tue, Jun 23, 2020 at 11:41 AM Ethan Furman wrote:
> >
> > Testing my understanding -- the following snippet from the PEP
> >
> >   match group_shapes():
> >   case [], [point := Point(x, y), *other]:
> >
> > will succeed if group_shapes() returns two lists, the first one
> being empty and the second one starting with a Point() ?
> >
> >
> > Correct. And it binds four variables: point, x, y, and other.
>
> Okay, so following that example some more:
>
> `other` is all the other items in `group_shape`'s second list
>
> `x` and `y` are the x,y values from the second list's first element (which
> is a Point)
>
> `point` is... the first element of the second list?  Or a __match__ object?
>

The former.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XQ7ULAM3FTEKFWYNITZA4Q46HYYBBNMT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

Le mar. 23 juin 2020 à 16:56, Stefan Behnel  a écrit :
> > Adding a new member breaks the stable ABI (PEP 384), especially for
> > types declared statically (e.g. ``static PyTypeObject MyType =
> > {...};``). In Python 3.4, the PEP 442 "Safe object finalization" added
> > the ``tp_finalize`` member at the end of the ``PyTypeObject`` structure.
> > For ABI backward compatibility, a new ``Py_TPFLAGS_HAVE_FINALIZE`` type
> > flag was required to announce if the type structure contains the
> > ``tp_finalize`` member. The flag was removed in Python 3.8 (`bpo-32388
> > `_).
>
> Probably not the best example. I think this is pretty much normal API
> evolution. Changing the deallocation protocol for objects is going to
> impact any public API in one way or another. PyTypeObject is also not
> exposed with its struct fields in the limited API, so your point regarding
> "tp_print" is also not a strong one.

The PEP 442 doesn't break backward compatibility. C extensions using
tp_dealloc continue to work.

But adding a new member to PyTypeObject caused practical
implementation issues. I'm not sure why you are mentioning the limited
C API. Most C extensions don't use it and declare their type as
"static types".

I'm not trying to describe the Py_TPFLAGS_HAVE_FINALIZE story as a
major blocker issue in the CPython history. It's just one of the many
examples of issues to evolve CPython internals.


> > Same CPython design since 1990: structures and reference counting
> > -
> > Members of ``PyObject`` and ``PyTupleObject`` structures have not
> > changed since the "Initial revision" commit (1990)
>
> While I see an advantage in hiding the details of PyObject (specifically
> memory management internals), I would argue that there simply isn't much to
> improve in PyTupleObject, so these two don't fly at the same level for me.

There are different reasons to make PyTupleObject opaque:

* Prevent access to members of its "PyObject ob_base" member (disallow
accessing directly "tuple->ob_base.ob_refcnt")

* Prevent C extensions to make assumptions on how a Python
implementation stores a tuple. Currently, C extensions are designed to
have best performances with CPython, but it makes them run slower on
PyPy.

* It becomes possible to experiment with a more efficient PyTypeObject
layout, in terms of memory footprint or runtime performance, depending
on the use case. For example, storing directly numbers as numbers
rather than PyObject. Or maybe use a different layout to make
PyList_AsTuple() an O(1) operation. I had a similar idea about
converting a bytearray into a bytes without having to copy memory. It
also requires to modify PyBytesObject to experiment such idea. An
array of PyObject* is the most efficient storage for all use cases.


> My feeling is that PyPy specifically is better served with the HPy API,
> which is different enough to consider it a mostly separate API, or an
> evolution of the limited API, if you want. Suggesting that extension
> authors support two different APIs is much, but forcing them to support the
> existing CPython C-API (for legacy reasons) and the changed CPython C-API
> (for future compatibility), and then asking them to support a separate
> C-API in addition (for platform independence, with performance penalties)
> seems stretching it a lot.

The PEP 620 changes the C API to make it converge to the limited C
API, but it also prepares C extensions to ease their migration to HPy.

For example, by design, HPy doesn't give a direct access to the
PyTupleObject.ob_item member. Enforce usage of PyTuple_GetItem()
function or PyTuple_GET_ITEM() maro should ease migration to
HPy_GetItem_i().


I disagree that extension authors have to support two C APIs. Many PEP
620 incompatible C changes are already completed, and I was surprised
by the very low numbers of extensions affected by these changes. In
practice, most extensions use simple and regular C code, they don't
"abuse" the C API. Cython itself is affected by most chances since
Cython basically uses all C API features :-) But in practice, only a
minority of extensions written with Cython are affected, since they
(indirectly, via Cython) only use a subset of the C API.

Also, once an extension is updated for incompatible changes, it
remains compatible with old Python versions. When a new function is
used, pythoncapi_compat.h can be used to support old Python versions.
It is not like code has to be duplicated to support two unrelated
APIs.


> > * (**Completed**) Add new functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` 
> > and
> >   ``Py_SET_SIZE()``. The ``Py_TYPE()``, ``Py_REFCNT()`` and
> >   ``Py_SIZE()`` macros become functions which cannot be used as l-value.
> > * (**Completed**) New C API functions must not return borrowed
> >   references.
> > * (**In Progress**) Provide ``pythoncapi_compat.h`` header file.
> > * (**In Progress**) Make structures

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 06/23/2020 04:26 PM, Guido van Rossum wrote:

On Tue, Jun 23, 2020 at 11:41 AM Ethan Furman wrote:

Testing my understanding -- the following snippet from the PEP

      match group_shapes():
          case [], [point := Point(x, y), *other]:

will succeed if group_shapes() returns two lists, the first one being empty 
and the second one starting with a Point() ?


Correct. And it binds four variables: point, x, y, and other.


Okay, so following that example some more:

`other` is all the other items in `group_shape`'s second list

`x` and `y` are the x,y values from the second list's first element (which is a 
Point)

`point` is... the first element of the second list?  Or a __match__ object?

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YO3TFM3Z65G6KL47JS457ZOCB54DWUEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Tue, Jun 23, 2020 at 4:41 PM MRAB  wrote:

> On 2020-06-23 22:50, Barry Warsaw wrote:
> > On Jun 23, 2020, at 14:31, Chris Angelico  wrote:
> >
> >> I can't find it among the rejected alternatives, but was it considered
> >> to use "..." as the wildcard, rather than "_"? It carries similar
> >> meaning but its special case of "this will never be bound" is simply
> >> preventing an error, rather than making one otherwise-valid name
> >> special.
> >
> > I thought of that too as I was reading the PEP, but forgot to add it to
> my notes.  I do like ellipsis more than underscore here.
> >
> +1
>

The problem is that ellipsis already has a number of other meanings, *and*
is easily confused in examples and documentation with leaving things out
that should be obvious or uninteresting. Also, if I saw [a, ..., z] in a
pattern I would probably guess that it meant "any sequence of length > 2,
and capture the first and last element" rather than "a sequence of length
three, and capture the first and third elements". (The first meaning is
currently spelled as [a, *_, z].)

So I'm not a fan. _ is what all other languages with pattern matching seem
to use, and I like that case (x, _, _) resembles the use of (x, _, _) in
assignment targets.

> However, what if you wanted to match Ellipsis?
>
> This could lead to bugs:
>
>  >>> ...
> Ellipsis
>  >>> Ellipsis = 0
>  >>> Ellipsis
> 0
>  >>> ...
> Ellipsis
>

Now you're just being silly. If you want to use Ellipsis as a variable you
can't also use it to refer to the "..." token.

> If you can have "case False:" and "case True:", should 'Ellipsis' become
> a keyword so that you could have "case Ellipsis:"? Or do they have to be
> "case .False:", "case .True:", in which case it could remain "case
> .Ellipsis:"?
>

I don't think this problem is at all bad enough to make "Ellipsis" a
keyword. It is much, much less used than True, False or None.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IO7KFDG3UWN5OWAKEV4BSYCECV4RYH42/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Tue, Jun 23, 2020 at 3:06 PM Emily Bowman 
wrote:

> Can you have case (x,x): ? I haven't tried the implementation, but it's
> not addressed in the PEP that I see, and if that's legal, then _ is
> effectively just a style choice, rather than a functional one, and there's
> no reason it shouldn't also be a named match.
>

Good question. It's explicitly forbidden by the PEP, in the "Name pattern"
section:

While matching against each case clause, a name may be bound at most
once, having two name patterns with coinciding names is an error. An
exception is made for the special single underscore (``_``) name; in
patterns, it's a wildcard that *never* binds::

  match data:
  case [x, x]:  # Error!
  ...
  case [_, _]:
  print("Some pair")
  print(_)  # Error!

Note: one can still match on a collection with equal items using `guards`_.
Also, ``[x, y] | Point(x, y)`` is a legal pattern because the two
alternatives are never matched at the same time.

I should add that if you want to check for two values in different
positions being equal, you need to use a guard:

match data:
case [x, y] if x == y:
print("Two equal values")

On Tue, Jun 23, 2020 at 12:11 PM Brett Cannon  wrote:

> I will say that trying to follow
> https://github.com/python/peps/blob/master/pep-0622.rst#runtime-specification
> was really hard. Any chance of getting some pseudo-code that shows how a
> match is performed? Otherwise all of that wording tries so hard to be a
> spec that I found it hard to follow in my head in how things function.
>

Sorry about that. This section was subject to heavy editing recently and
lost clarity. I will try to make it better! Writing it as pseudo code will
take a little time, but I will give it a try.

> For instance, "When __match_args__ is missing (as is the default) or None,
> a single positional sub-pattern is allowed to be passed to the call" is
> really misleading as it seems that a "sub-pattern" in this case is just
> going to be a constant like `[1, 2, 3]`. Otherwise how does `["<"|">"]` or
> `[1, 2, *_]` get represented as a "single positional sub-pattern" (if
> either of those examples is possible)? The use of the term "sub-pattern"
> feels misleading because while you may consider even constant patterns a
> "pattern", going that generic feels like any pattern should fit in that
> definition when in fact it seems to only be an object where a direct
> equality check is done.
>
> It seems the way things work is basically:
>
> 1. `__match__(obj)` returns a proxy object to have Python match against;
> it is passed in the thing that `match` is running against, returning `None`
> if it know there's no chance a match will work
> 2. If `__match_args__` is present, then it is used to map positional
> arguments in the pattern to attributes on the proxy object
> 3. From there the `match` functionality does a bunch of comparisons
> against attributes on the proxy object to see if the match works
>
> Is that right? That suggests all the work in implementing this for objects
> is coming up with a way to serialize an object to a proxy that makes
> pattern matching possible.
>

Yes, that's right, and the protocol was defined carefully so that the
author of __match__ doesn't have to do any pattern matching -- all they
have to do is produce an object that has the right attributes, and the
interpreter does the rest. Note that it is __match__'s responsibility to
check isinstance()! This is because __match__ may not want to use
isinstance() but instead check for the presence of certain attributes --
IOW, the class pattern supports duck typing! (This was a little easter egg.
:-)

> One thing I see mentioned in examples but not in the `__match__`
> definitions is how mappings work. Are you using `__match_args__` to map
> keys to attributes? Or are you using `__getitem__` and that just isn't
> directly mentioned? Otherwise the section on how `__match__` is used only
> mentioned attributes and never talks about keys.
>

Oh, __match__ is *only* used for class patterns. Mapping patterns are done
differently. They don't use __getitem__ exactly -- the PEP says

Matched key-value pairs must already be present in the mapping, and not
created
on-the-fly by ``__missing__`` or ``__getitem__``.  For example,
``collections.defaultdict`` instances will only match patterns with keys
that
were already present when the ``match`` block was entered.

You shouldn't try to depend on exactly what methods will be called -- you
should just faithfully implement the Mapping protocol.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On 2020-06-23 22:50, Barry Warsaw wrote:

On Jun 23, 2020, at 14:31, Chris Angelico  wrote:

I can't find it among the rejected alternatives, but was it considered
to use "..." as the wildcard, rather than "_"? It carries similar
meaning but its special case of "this will never be bound" is simply
preventing an error, rather than making one otherwise-valid name
special.

I thought of that too as I was reading the PEP, but forgot to add it to my 
notes.  I do like ellipsis more than underscore here.

+1

However, what if you wanted to match Ellipsis?

This could lead to bugs:

>>> ...
Ellipsis
>>> Ellipsis = 0
>>> Ellipsis
0
>>> ...
Ellipsis

If you can have "case False:" and "case True:", should 'Ellipsis' become 
a keyword so that you could have "case Ellipsis:"? Or do they have to be 
"case .False:", "case .True:", in which case it could remain "case 
.Ellipsis:"?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZM6YWTIHNBJ6JJPTFOQNWIPIV4CO3UPX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 2020-06-23 22:53, Emily Bowman wrote:
On Tue, Jun 23, 2020 at 2:10 PM MRAB > wrote:


I think that's done for consistency. '_' is a wildcard and you can
have:

     case (_, _):

to match any 2-tuple, so:

     case _:

would match any value, and can thus already serve as the default.

I wouldn't object to 'else', though.


Can you have case (x,x): ? I haven't tried the implementation, but 
it's not addressed in the PEP that I see, and if that's legal, then _ 
is effectively just a style choice, rather than a functional one, and 
there's no reason it shouldn't also be a named match.



You cannot bind to the same name twice, so "case (x,x):"is an error.
+1 for including "else:" for consistency even if that's just a shim 
for "case _:"

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6KXFMJ3M372GJO6UI5GFSAKETTNUF5HO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

(I'm replying to several messages in one reply. But there is too much to
respond to so this is merely the first batch.)

On Tue, Jun 23, 2020 at 10:10 AM MRAB  wrote:

> Why are:
>
>  case ._:
>
> not OK?
>
> In this:
>
>  case .BLACK:
>  ...
>  case BLACK:
>  ...
>
> the first matches the value against 'BLACK' and the second succeeds and
> binds the value to 'BLACK'.
>
> Replacing 'BLACK' with '_':
>
>  case ._:
>  ...
>  case _:
>  ...
>
> I'd expect something similar, except for the binding part.
>
> I think the same could be said for:
>
>  case Color.BLACK:
>
> and:
>
>  case _.BLACK:
>

We disallow ._ and _.xxx because when plain _ is used as a pattern it is a
*wildcard*, not an identifier. The pattern compiler must special-case _ as
a target because [x, x] is an invalid pattern (you can't bind the same
variable twice) but [_, _] is valid (it means "any non-string sequence of
two elements"). There are some other edge cases where _ is special-cased
too. But basically we want to prevent users doing a double take when they
see _ used as a wildcard on one line and as a variable on the next.

> I also wonder whether "or" would be clearer than "|":
>
>  case .BLACK or Color.BLACK:
>

That's just bikeshedding. I personally don't think it would be clearer.
Other languages with pattern matching tend to use "|" (as do regular
expressions :-).

On Tue, Jun 23, 2020 at 10:28 AM Antoine Pitrou  wrote:

> * What you call "Constant Value Patterns" can really refer to any local
>   or non-local name, regardless of how complex the referred object is,
>   right?  Also, __eq__ is called in that case, not __match__?
>

Yeah, we're considering renaming these to "Value Patterns". There's nothing
particularly constant about them. And yes, they use __eq__. Basically these
and literals are processed exactly the same once the value is obtained.

> * If I understand correctly, these:
>
> case b"":
> print("it's an empty bytes object")
>
> and
>
> case bytes():
> print("it's a bytes object")
>
> have entirely different meanings.  Am I right?  This sounds like I have
> to switch contexts when reading code, based on whether I am reading
> regular code or a match clause, given that semantics are quite
> different.
>

Yes, you're right. The first is a literal pattern, the second a class
pattern.

Switching how you interpret code when reading it based on context is common
-- seeing "x, y" on the LHS of an assignment is different than on the RHS,
and seeing "x=5" in a "def" line is completely different from seeing it in
a call.

> Instead, it seems like the latter would be more explicitly spelled out
> as, e.g.:
>
> case instanceof(bytes):
> print("it's a bytes object")
>

But that's arbitrary syntax. The beauty of using SomeClass() is that the
pattern (if you squint :-) looks like a constructor for an object.
SomeClass() is just an edge case of the general form SomeClass(subpattern1,
subpattern2, ..., arg5=subp5, arg6=subp6, ...).

> * The acronym "ADT" is never defined.
>

Yes it is, you missed this sentence:

A design pattern where a group of record-like classes is
combined into a union is popular in other languages that support pattern
matching and is known under a name of algebraic data types [2]_ or ADTs.

> * """If there are more positional items than the length of
>   __match_args__, an ImpossibleMatchError is raised."""
>
> What if there are less positional items than ``len(__match_args__)``?
> Can the match succeed rather than raise ImpossibleMatchError?  This
> seems potentially error-prone.
>

Yes, it can succeed. Honestly, we could have  gone the other way on this
one, but we figured that there are plenty of functions and class
constructors that can be called with a variable number of positional
arguments, since some arguments have default values. We even invented an
optional attribute __match_args_required__ (an int) that would have given
the minimal number of positional arguments required, but it was deemed too
obscure (and the name is ugly). Also we definitely wanted to be able to
write `case Point():` for the isinstance check.

> Overall, my main concern with this PEP is that the matching semantics
> and pragmatics are different from everything else in the language.
> When reading and understanding a match clause, there's a cognitive
> overhead because suddently `Point(x, 0)` means something entirely
> different (it doesn't call Point.__new__, it doesn't lookup `x` in the
> locals or globals...).  Obviously, there are cases where this is
> worthwhile, but still.
>

Quite a few other languages have done this and survived. And Python already
has LVALUES and RVALUES that look the same but have different meanings. (In
fact the pattern syntax for sequences was derived from those.)

> It may be useful to think about different notations for these new
> things, rather than re-use the object construction notation.
>
>

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

Le mar. 23 juin 2020 à 20:37, Neil Schemenauer
 a écrit :
> My thinking is that, eventually, we would like to allow CPython to
> use something other than reference counting for internal PyObject
> memory management.  In other systems with garbage collection, the
> memory allocator is typically tightly integrated with the garbage
> collector.  To get good efficiency, they need to cooperate.  E.g.
> newly allocated objects are allocated in nursery memory arenas.

The PEP 620 only mentions replacing CPython GC with a tracing GC at one place:

"It becomes possible to experiment with more advanced optimizations in
CPython than just micro-optimizations. For example, tagged pointers,
and replace the garbage collector with a tracing garbage collector
which can move objects."

I chose to not make it a requirement of the PEP 620 on purpose. IMHO
such a change will very likely require a whole PEP by itself, since
there are likely many small things which will have to be changed to
prepare such a major change.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PQOR7WB6DW3DO7IQZWPAZCGMOSKV3VCL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 06/23/2020 02:48 PM, Barry Warsaw wrote:

On Jun 23, 2020, at 13:50, Ethan Furman  wrote:


On 06/23/2020 12:49 PM, Barry Warsaw wrote:


Maybe the PEP could support this syntax:
 case in 401, 403, 404:
That seems like it would be both unambiguous and semantically obvious.


Unfortunately, that means:

  Is the object a three-tuple with the values 401, 403, and 404?


It would mean: is the object *in* a 3-tuple with values 401, 403, and 404, 
right?


If I understand the PEP correctly, then

  some_obj == (401, 403, 404)
  match some_obj:
  case 401, 403, 404:
  # this matches

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UBF7DNPS2TMKWPPB32DB6GA6YQKRCHKW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Jun 23, 2020, at 14:34, Richard Levasseur  wrote:

> I agree with not having flat indentation. I think having "case" indented from 
> "match" makes it more readable overall.

I don’t know whether my suggestion to use `match:` and putting the expression 
inside this stanza can be accomplished, but I do want to add another point 
about that suggestion that I’ve heard from several folks I’ve already chatted 
with about this PEP.  Using something like

match:
expression
case arm1:
pass
case arm2:
pass
else:
pass

nicely mirrors try/except and if/elif/else constructs so it looks quite natural.

Cheers,
-Barry

signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3OZDTDFYGVO4OIWR334E62MAH5EGOFEE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Emily Bowman

On Tue, Jun 23, 2020 at 2:10 PM MRAB  wrote:

> I think that's done for consistency. '_' is a wildcard and you can have:
>
>  case (_, _):
>
> to match any 2-tuple, so:
>
>  case _:
>
> would match any value, and can thus already serve as the default.
>
> I wouldn't object to 'else', though.
>

Can you have case (x,x): ? I haven't tried the implementation, but it's not
addressed in the PEP that I see, and if that's legal, then _ is effectively
just a style choice, rather than a functional one, and there's no reason it
shouldn't also be a named match.

+1 for including "else:" for consistency even if that's just a shim for
"case _:"
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SBFOEILU56E5QCM6344QNGEMKKT5MWKE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Jun 23, 2020, at 13:50, Ethan Furman  wrote:
> 
> On 06/23/2020 12:49 PM, Barry Warsaw wrote:
> 
>> Maybe the PEP could support this syntax:
>> case in 401, 403, 404:
>> That seems like it would be both unambiguous and semantically obvious.
> 
> Unfortunately, that means:
> 
>  Is the object a three-tuple with the values 401, 403, and 404?

It would mean: is the object *in* a 3-tuple with values 401, 403, and 404, 
right?

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LN6I356JLWAEW7GBTVUJRQS7RP4WR4TF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Jun 23, 2020, at 14:31, Chris Angelico  wrote:

> I can't find it among the rejected alternatives, but was it considered
> to use "..." as the wildcard, rather than "_"? It carries similar
> meaning but its special case of "this will never be bound" is simply
> preventing an error, rather than making one otherwise-valid name
> special.

I thought of that too as I was reading the PEP, but forgot to add it to my 
notes.  I do like ellipsis more than underscore here.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JDNQ55FLBWRWM3UKONBG2KXJBJSMVT5N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Richard Levasseur

Hi,

I read the PEP, and a few thoughts:

-

I think one of the examples is some lib2to3 code? I think the matcher
syntax is really great for that case (parse trees). The matcher syntax is
definitely an improvement over the litany of helper functions and
conditionals otherwise needed.

That said, I have a hard time seeing a particular use of this complicated
pattern matching outside "hetergenous trees" (for lack of a better term) of
objects? I've only really dealt with that problem with parse trees, but
perhaps that just an artifact of the domains I've ended up working in.

In any case, it might be useful to include some/more examples or use cases
that aren't as parser-centric.

-

Question: How are True, False, None, ..., etc handled? What does this do?

case whatever:
  case True: ...
  case False: ...
  case None: ...
  case ...:

I would expect they would be treated as literals the same as e.g.
numbers/strings, yes? Sorry if I missed this in the PEP.

-

I, too, had trouble understanding the __match__ protocol from the PEP text.
Brett's comments largely capture my thoughts about this.

-

The need to use "." to indicate "look up name" to avoid "match anything"
seems like a big foot gun. Simple examples such as:

FOO = 1
match get_case():
  case FOO:
print("you chose one")

clearly illustrate this, but the problem is present in any case expression:
a missing dot changes the meaning from "match this specific value" to
almost the opposite: "match any value". And all you really need to do is
miss a single leading dot anywhere in the case expression to trigger this.
I agree with Barry (I think he said this) that it seems like an easy cause
of mysterious bugs.

I think the foot-gun aspect derives directly from the change in how a
symbol is interpreted. i.e., Everywhere (predominantly? everything I can
think of atm) else in the language when you see "foo", you know it means
some sort of lookup of the name "foo" is occurring. The exception to this
is fairly simple: when there is some "assignment cue", e.g. "as", :=, =,
import, etc, and those assignment cues are always very close by (pretty
much always the leading/following token?). Anyways, my point is assignment
has a cue close by.

The proposed syntax flips that and mixes it, so it's very confusing.
Sometimes a symbol is a lookup, sometimes it's an assignment.

The PEP talks a bit about this in the "alternatives for constant value
pattern" section. I don't find the rationale in that section particularly
convincing. It basically says using "$FOO" to act as "look up value named
FOO" is rejected because "it is new syntax for a narrow use case" and "name
patterns are common in typical code ... so special syntax for the common
case would be weird".

I don't find that convincing because it seems *more weird* to change the
(otherwise consistent) lookup/assignment behavior of the language for a
specific sub-syntax.

Anyways, when I rewrite the examples and use a token to indicate "matcher",
I personally find them easier to read. I feel this is because it makes the
matcher syntax feel more like templates or string interpolation (or things
of that nature) that have some "placeholder" that gets "bound" to a value
after being given some "input".

It also sort of honors the "assignment only happens with a localized cue"
behavior that already exists.

ORIGIN = 0
case get_point():
  case Point(ORIGIN, $end):
...
  case $default:
print(default)

I will admit this gives me PHP flashbacks, but it's also very clear where
assignments are happening, and I can just use the usual name-lookup rules.
I just used $ since the PEP did.

As a bonus, I also think this largely mediates the foot gun problem because
there's now a cue a binding is happening, so it's easy to trigger our "is
that name already taken, is it safe to assign?" check we mentally perform.

In any case, this seems like a pretty fundamental either/or design decision
*someone* will have to make:

Either:
  names mean assignment, and the rules of what is a lookup vs assignment
are different with some special case support (i.e. leading dot).
Or:
  use some character to indicate assignment, and the lookup rules are the
same.

-

Related to the above: I also raise this because, in my usage, I doubt I'll
be using it as much more than a switch statement. I rarely have to match
complicated patterns, but very often have a set of values that I need to
test against. The combination of Literal and exhaustive-case checking is
very appealing.

So I'm very often going to want to type, e.g.

ValidModes = Union[Literal[A], Literal[B], etc etc]
def foo(mode: ValidModes):
  match mode:
case A: ...
case B: ...
case etc etc

And eventually I'm going to foot-gun myself with a missing dot.

-

Related to the above, I *don't* find that e.g. "case Point(...)" *not*
initializing
a Point particularly confusing. This feels like it might be inconsistent
with my whole thing above, but :shrug:. FWIW,

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Wed, Jun 24, 2020 at 2:04 AM Guido van Rossum  wrote:
> def http_error(status):
> match status:
> case 404:
> return "Not found"
> case 418:
> return "I'm a teapot"
> case _:
> return "Something else"
>
> Note the last block: the "variable name" `_` acts as a *wildcard* and
> never fails to match.

I can't find it among the rejected alternatives, but was it considered
to use "..." as the wildcard, rather than "_"? It carries similar
meaning but its special case of "this will never be bound" is simply
preventing an error, rather than making one otherwise-valid name
special.

> Raw strings and byte strings are supported. F-strings are not allowed (since 
> in general they are not really literals).

It won't come up often, but are triple-quoted strings allowed?

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OH4UXRFTPXG2JQFW44GZ2E2FGHYAC4LK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 2020-06-23 18:20, Antoine Pitrou wrote:


Some comments:

* What you call "Constant Value Patterns" can really refer to any local
   or non-local name, regardless of how complex the referred object is,
   right?  Also, __eq__ is called in that case, not __match__?

* If I understand correctly, these:

 case b"":
 print("it's an empty bytes object")

and

 case bytes():
 print("it's a bytes object")

have entirely different meanings.  Am I right?  This sounds like I have
to switch contexts when reading code, based on whether I am reading
regular code or a match clause, given that semantics are quite
different.

Instead, it seems like the latter would be more explicitly spelled out
as, e.g.:

 case instanceof(bytes):
 print("it's a bytes object")

* The acronym "ADT" is never defined.

* """If there are more positional items than the length of
   __match_args__, an ImpossibleMatchError is raised."""

What if there are less positional items than ``len(__match_args__)``?
Can the match succeed rather than raise ImpossibleMatchError?  This
seems potentially error-prone.


Overall, my main concern with this PEP is that the matching semantics
and pragmatics are different from everything else in the language.
When reading and understanding a match clause, there's a cognitive
overhead because suddently `Point(x, 0)` means something entirely
different (it doesn't call Point.__new__, it doesn't lookup `x` in the
locals or globals...).  Obviously, there are cases where this is
worthwhile, but still.

It may be useful to think about different notations for these new
things, rather than re-use the object construction notation.

For example:

 case Point with (x, y):
  print(f"Got a point with x={x}, y={y}")

or:

 case Point @ (x, y):
  print(f"Got a point with x={x}, y={y}")

(yes, "@" is the matrix multiplication operator... but it's probably
much less likely to appear in common code and especially with a class
object at the left)


Or:

case Point as (x, y):
print(f"Got a point with x={x}, y={y}")

perhaps as 'as' is already used for binding.

The disadvantage there is with nested patterns:

case Coordinate(Point(x1, y1), Point(x2, y2)):

where you're matching a coordinate and binding to x1, y1, x2 and y2.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IMKBETFAH3CS6TLBT3JNQX2ULQ4QKQLK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On 2020-06-23 20:27, Rob Cliffe via Python-Dev wrote:

On 23/06/2020 17:01, Guido van Rossum wrote:

You can combine several literals in a single pattern using `|` ("or"):

```py
case 401|403|404:
return "Not allowed"
The PEP is great, but this strikes me as horribly confusing, given that
401|403|404 is already legal syntax.
IIUC any legal expression can come between `case` and `:`, but
expressions that contain `|` at their outermost level are *interpreted
differently* than from in other contexts.

The grammar shows that 'case' is followed by a series of alternatives,
separated by '|', but the alternatives aren't expressions, basically
only literals and instances of a given class. You can't say "case 1 +
2:", for example.

Presumably adding parentheses:
case (401|403|404):
would make it equivalent to
case 407:

I'm not sure whether that's legal, but it's not equivalent.

Is a separator (other than whitespace) actually needed? Can the parser
cope with

case 401 403 404:

Failing that IMO preferable, albeit not ideal, possibilities would be
1) Use colon as the separator.
2) Use comma as the separator - this is already legal syntax too, but
IMO it reads more naturally.
(And IIRC there are already contexts where brackets are necessary
to indicate a tuple.)

Perhaps someone can think of something better.

I also (with others) prefer `else:` or perhaps `case else:` to using
the`_` variable.
The latter is obscure, and woudn't sit well with code that already uses
that variable for its own purposes.

I think that's done for consistency. '_' is a wildcard and you can have:

case (_, _):

to match any 2-tuple, so:

case _:

would match any value, and can thus already serve as the default.

I wouldn't object to 'else', though.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/SACSWYRDR5Q6LUPAMJCX455RKZ6ZFSY2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 06/23/2020 12:49 PM, Barry Warsaw wrote:


Maybe the PEP could support this syntax:

 case in 401, 403, 404:

That seems like it would be both unambiguous and semantically obvious.


Unfortunately, that means:

  Is the object a three-tuple with the values 401, 403, and 404?

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/US5CINA3LTPC5BWWWJTLUTIQNS5UMGML/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Jun 23, 2020, at 12:27, Rob Cliffe via Python-Dev  
wrote:
> 
> On 23/06/2020 17:01, Guido van Rossum wrote:
>> 
>> You can combine several literals in a single pattern using `|` ("or"):
>> 
>> ```py
>> case 401|403|404:
>> return "Not allowed"
> The PEP is great, but this strikes me as horribly confusing, given that 
> 401|403|404 is already legal syntax.
> IIUC any legal expression can come between `case` and `:`, but expressions 
> that contain `|` at their outermost level are interpreted differently than 
> from in other contexts.
> Presumably adding parentheses:
> case (401|403|404):
> would make it equivalent to
> case 407:
> 
> Is a separator (other than whitespace) actually needed?  Can the parser cope 
> with
> case 401 403 404:

Maybe the PEP could support this syntax:

case in 401, 403, 404:

That seems like it would be both unambiguous and semantically obvious.

Cheers,
-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OT6Y4DHTJNB7QSOLS3577AGQFIQPNKSP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Wed, Jun 24, 2020 at 5:30 AM Rob Cliffe via Python-Dev
 wrote:
>
> The PEP is great, but this strikes me as horribly confusing, given that 
> 401|403|404 is already legal syntax.
> IIUC any legal expression can come between `case` and `:`, but expressions 
> that contain `|` at their outermost level are interpreted differently than 
> from in other contexts.
> Presumably adding parentheses:
> case (401|403|404):
> would make it equivalent to
> case 407:
>
> Is a separator (other than whitespace) actually needed?  Can the parser cope 
> with
> case 401 403 404:
>
> Failing that IMO preferable, albeit not ideal, possibilities would be
>   1) Use colon as the separator.
>   2) Use comma as the separator - this is already legal syntax too, but IMO 
> it reads more naturally.
>   (And IIRC there are already contexts where brackets are necessary to 
> indicate a tuple.)
> Perhaps someone can think of something better.
>
> I also (with others) prefer `else:` or perhaps `case else:` to using the`_` 
> variable.
> The latter is obscure, and woudn't sit well with code that already uses that 
> variable for its own purposes.
>

It's not really arbitrary expressions, though. It's more like an
assignment target list, but with some handling of constants.

case (x, y):

is very similar to

(x, y) = ...

There is the definite risk of confusion with 'if' statements, because
you can say "case 401|403|404:" but you can't say "if x ==
401|403|404:", and people already try to do that (usually with the
'or' keyword). I think the risk is worth it, given the expressiveness
gained, but I can see the line of argument that it's going to cause
confusion.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GEMFLQCNZG3WPZYTBNC54S4A3XLIUMPR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Rob Cliffe via Python-Dev




On 23/06/2020 17:01, Guido van Rossum wrote:


You can combine several literals in a single pattern using `|` ("or"):

```py
        case 401|403|404:
            return "Not allowed"
The PEP is great, but this strikes me as horribly confusing, given that 
401|403|404 is already legal syntax.
IIUC any legal expression can come between `case` and `:`, but 
expressions that contain `|` at their outermost level are *interpreted 
differently* than from in other contexts.

Presumably adding parentheses:
    case (401|403|404):
would make it equivalent to
    case 407:

Is a separator (other than whitespace) actually needed?  Can the parser 
cope with

    case 401 403 404:

Failing that IMO preferable, albeit not ideal, possibilities would be
  1) Use colon as the separator.
  2) Use comma as the separator - this is already legal syntax too, but 
IMO it reads more naturally.
  (And IIRC there are already contexts where brackets are necessary 
to indicate a tuple.)

Perhaps someone can think of something better.

I also (with others) prefer `else:` or perhaps `case else:` to using 
the`_` variable.
The latter is obscure, and woudn't sit well with code that already uses 
that variable for its own purposes.


Rob Cliffe
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T4WY7QNHVODO227R5IFSK67Z6X6W2KI6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re%3A Re%3A New optional dependency - TkDND=<45fe0089-6c26-4238-9e7c-ec38f719d781%40gmail.com>

2020-06-23 Thread Elisha Paine

> What's the advantage of having TkDND bindings in the standard library

1. I think there is enough demand for the feature to make it worth
adding to the stdlib (and the actual bindings are not that large so
won’t require much maintenance). There are several threads on Stack
Overflow where using TkDND with tkinter is discussed and
https://stackoverflow.com/questions/14267900, for example, appears to
have attracted a fair amount of attention (being viewed 27k times).
2. tkinter is often one of the first Python GUI toolkits learnt and (on
most platforms) installing TkDND is not a one-click process. As a
result, there are at least a couple of SO threads regarding installing
TkDND (I keep mentioning SO as I consider it a good way of understanding
the ‘typical’ user).
3. To make the bindings work correctly, they required changes to
existing tkinter methods and I try very hard to avoid monkey-patching
where possible.
4. This is a small point, but I thought it worth mentioning that the
tkinter.dnd docs have said this has been coming for a long time. Even if
this change doesn’t go ahead I think it would be worth changing the docs
to stop saying that this feature is coming…

Thank you, both of you, for the reassurance you have given and I will
look into raising third party issues if I notice a distribution hasn’t
updated the package information (I won’t raise issues pre-emptively,
though).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PNPK7QQ6IVIMH7O6OLNU5NCN3BV3PDI4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

On Jun 23, 2020, at 09:01, Guido van Rossum  wrote:

> I'm happy to present a new PEP for the python-dev community to review. This 
> is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and Talin.

Congratulations to the authors.  This is a well written, complex PEP for a 
powerful feature.  Here are some of my initial comments/questions.

Couldn’t you adopt a flat indentation scheme with the minor change of moving 
the expression into a `match:` clause?  E.g.

match:
expression
case a:
 foo()
case b:
bar()
else:
baz()

I didn’t see that in the rejected alternatives.

I’m with others who think that `else:` would be a better choice than `case _:`. 
 Given my background in i18n (see the flufl.i18n library, etc.), my red flags 
go up when I see bare underscores being given syntactic meaning.

For example, the restriction against `case _.a:` *could* interact badly with my 
library.  There, _ is the most common name binding for an object that 
implements the translation semantics.  Therefore, it has attributes.  I can’t 
think of a concrete example right now, but e.g. what if I wanted to match 
against `_.code`?  That wouldn’t be legal if I’m understanding correctly 
(`code` being an attribute of the object typically bound to _).

I’m also concerned about the .BLACK vs BLACK example.  I get why the 
distinction is there, and I get that the PEP proposes that static analyzers 
help the ambiguity, but it strikes me as a potential gotcha and a source of 
mysterious errors.

Why not just bare `*` instead of `*_` in patterns?

The PEP is unclear about what kind of method __match__() is.  As I was reading 
along, I suspected it must be a static or class method, explicitly cannot be an 
instance method because the class in the case statement is never instantiated, 
but it’s not until I got to the object.__match__() discussion that this 
distinction was made clear.  I think the PEP should just be explicit upfront 
about that.

As a side note, I’m also concerned that `case Point(x, y)` *looks* like it 
instantiates `Point` and that it’s jarring to Python developers that they have 
to mentally switch models when reading that code.

I was also unclear about that __match__() had to return an object until much 
later in the PEP.  My running notes asked:

# Is returning None better than raising an exception? This won’t work:
class C:
def __init__(self, x=None):
self.x = x
@staticmethod
def __match__(obj):
   # Should just return obj?
return obj.x

match C():
case x:
print(x)

But once I read on, I realized that __match__() should return `obj` in this 
case.  Still, the fact that returning None to signal a case arm not matching 
feels like there’s a gotcha lurking in there somewhere.

Should @sealed be in a separate PEP?  It’s relevant to the discussion in 622, 
but seems like it would have use outside of the match feature so should 
possibly be proposed separately.

The PEP is unclear whether `case` is also a soft keyword.  I’m guessing it must 
be.

Those are my current thoughts on first read through.

Cheers,
-Barry

signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X47ORE7XBKEBVALZYXH4J6HKLR3IJ23C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Brett Cannon

I will say that trying to follow
https://github.com/python/peps/blob/master/pep-0622.rst#runtime-specification
was really hard. Any chance of getting some pseudo-code that shows how a match
is performed? Otherwise all of that wording tries so hard to be a spec that I
found it hard to follow in my head in how things function.

For instance, "When __match_args__ is missing (as is the default) or None, a
single positional sub-pattern is allowed to be passed to the call" is really
misleading as it seems that a "sub-pattern" in this case is just going to be a
constant like `[1, 2, 3]`. Otherwise how does `["<"|">"]` or `[1, 2, *_]` get
represented as a "single positional sub-pattern" (if either of those examples
is possible)? The use of the term "sub-pattern" feels misleading because while
you may consider even constant patterns a "pattern", going that generic feels
like any pattern should fit in that definition when in fact it seems to only be
an object where a direct equality check is done.

It seems the way things work is basically:

1. `__match__(obj)` returns a proxy object to have Python match against; it is
passed in the thing that `match` is running against, returning `None` if it
know there's no chance a match will work
2. If `__match_args__` is present, then it is used to map positional arguments
in the pattern to attributes on the proxy object
3. From there the `match` functionality does a bunch of comparisons against
attributes on the proxy object to see if the match works

Is that right? That suggests all the work in implementing this for objects is
coming up with a way to serialize an object to a proxy that makes pattern
matching possible.

One thing I see mentioned in examples but not in the `__match__` definitions is
how mappings work. Are you using `__match_args__` to map keys to attributes? Or
are you using `__getitem__` and that just isn't directly mentioned? Otherwise
the section on how `__match__` is used only mentioned attributes and never
talks about keys.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/AOKEZGM2UJJWVA7BW2CKAXI6QXI6ZBM5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Paul Moore

On Tue, 23 Jun 2020 at 17:07, Guido van Rossum  wrote:
>
> I'm happy to present a new PEP for the python-dev community to review. This 
> is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and Talin.
>
> Many people have thought about extending Python with a form of pattern 
> matching similar to that found in Scala, Rust, F#, Haskell and other 
> languages with a functional flavor. The topic has come up regularly on 
> python-ideas (most recently yesterday :-).
>
> I'll mostly let the PEP speak for itself:
> - Published: https://www.python.org/dev/peps/pep-0622/ (*)
> - Source: https://github.com/python/peps/blob/master/pep-0622.rst
>
> (*) The published version will hopefully be available soon.
>
> I want to clarify that the design space for such a match statement is 
> enormous. For many key decisions the authors have clashed, in some cases we 
> have gone back and forth several times, and a few uncomfortable compromises 
> were struck. It is quite possible that some major design decisions will have 
> to be revisited before this PEP can be accepted. Nevertheless, we're happy 
> with the current proposal, and we have provided ample discussion in the PEP 
> under the headings of Rejected Ideas and Deferred Ideas. Please read those 
> before proposing changes!
>
> I'd like to end with the contents of the README of the repo where we've 
> worked on the draft, which is shorter and gives a gentler introduction than 
> the PEP itself:

I'm just going to say that I *really* like the look of this proposal.
I suspect there's going to be a fairly extensive, and detailed,
discussion over some of the edge cases, but they can be worked out.
Overall, though, I love the general idea.

I'll hold off on saying anything more until I've read the PEP
properly. In particular, the "Deferred Ideas" section seems to cover a
lot of my initial "what about X" questions.
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PZE2VU5KZOMZEBYON7L6ODQ7GQWAQFAK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 06/23/2020 09:01 AM, Guido van Rossum wrote:

Very nice!  I am totally in favor (with some bike-shedding, of course).

First, a formatting comment:

A new sentence immediately following formatted code needs more space -- it 
looks like the same sentence otherwise.  Will putting two spaces after the 
period help in this case?


Testing my understanding -- the following snippet from the PEP

match group_shapes():
case [], [point := Point(x, y), *other]:

will succeed if group_shapes() returns two lists, the first one being empty and 
the second one starting with a Point() ?


---
Runtime Specifications

The __match__ protocol
---
Suffers from several indentation errors (the nested lists are not).

-

My biggest complaint is the use of

  case _:

Unless I'm missing something, every other control flow statement in Python that can have an 
"else" type branch uses "else" to denote it:

if/else

for/else

while/else

try/except/else

Since "else" perfectly sums up what's happening, why "case _" ?

match something:
case 0 | 1 | 2:
print("Small number")
case [] | [_]:
print("A short sequence")
case str() | bytes():
print("Something string-like")
else:
print("Something else")

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BEOJ5S4YCTQM27A2B5NDEC6Q7BCYPPYY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Neil Schemenauer

On 2020-06-23, Thomas Wouters wrote:
> I think the ability for per-type allocation/deallocation routines isn't
> really about efficiency, but more about giving more control to embedding
> systems (or libraries wrapped by extension modules) about how *their*
> objects are allocated. It doesn't make much sense, however, because Python
> wouldn't allocate their objects anyway, just the Python objects wrapping
> theirs. Allocating CPython objects should be CPython's job.

My thinking is that, eventually, we would like to allow CPython to
use something other than reference counting for internal PyObject
memory management.  In other systems with garbage collection, the
memory allocator is typically tightly integrated with the garbage
collector.  To get good efficiency, they need to cooperate.  E.g.
newly allocated objects are allocated in nursery memory arenas.  

The current API doesn't allow that because you can allocate memory
via some custom allocator and then pass that memory to be
initialized and treated as a PyObject.  That's one thing locking
us into reference counting.

This relates to the sub-interpreter discussion.  I think the
sub-interpreter cleanup work is worth doing, if only because it will
make embedding CPython cleaner.  I have some doubts that
sub-interpreters will help much in terms of multi-core utilization.
Efficiently sharing data between interpreters seems like a huge
challenge.  I think we should also pursue Java style multi-threading
and complete the "gilectomy".  To me, that means killing reference
counting for internal PyObject management.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FQS5TB6G77EE35QW3JHRU7ISZ4ASDCTQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching

2020-06-23 Thread Antoine Pitrou

Some comments:

* What you call "Constant Value Patterns" can really refer to any local
  or non-local name, regardless of how complex the referred object is,
  right?  Also, __eq__ is called in that case, not __match__?

* If I understand correctly, these:

case b"":
print("it's an empty bytes object")

and

case bytes():
print("it's a bytes object")

have entirely different meanings.  Am I right?  This sounds like I have
to switch contexts when reading code, based on whether I am reading
regular code or a match clause, given that semantics are quite
different.

Instead, it seems like the latter would be more explicitly spelled out
as, e.g.:

case instanceof(bytes):
print("it's a bytes object")

* The acronym "ADT" is never defined.

* """If there are more positional items than the length of
  __match_args__, an ImpossibleMatchError is raised."""

What if there are less positional items than ``len(__match_args__)``?
Can the match succeed rather than raise ImpossibleMatchError?  This
seems potentially error-prone.

Overall, my main concern with this PEP is that the matching semantics
and pragmatics are different from everything else in the language.
When reading and understanding a match clause, there's a cognitive
overhead because suddently `Point(x, 0)` means something entirely
different (it doesn't call Point.__new__, it doesn't lookup `x` in the
locals or globals...).  Obviously, there are cases where this is
worthwhile, but still.

It may be useful to think about different notations for these new
things, rather than re-use the object construction notation.

For example:

case Point with (x, y):
 print(f"Got a point with x={x}, y={y}")

or:

case Point @ (x, y):
 print(f"Got a point with x={x}, y={y}")

(yes, "@" is the matrix multiplication operator... but it's probably
much less likely to appear in common code and especially with a class
object at the left)

Regards

Antoine.

On Tue, 23 Jun 2020 09:01:11 -0700
Guido van Rossum  wrote:
> I'm happy to present a new PEP for the python-dev community to review. This
> is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and Talin.
> 
> Many people have thought about extending Python with a form of pattern
> matching similar to that found in Scala, Rust, F#, Haskell and other
> languages with a functional flavor. The topic has come up regularly on
> python-ideas (most recently yesterday :-).
> 
> I'll mostly let the PEP speak for itself:
> - Published: https://www.python.org/dev/peps/pep-0622/ (*)
> - Source: https://github.com/python/peps/blob/master/pep-0622.rst
> 
> (*) The published version will hopefully be available soon.
> 
> I want to clarify that the design space for such a match statement is
> enormous. For many key decisions the authors have clashed, in some cases we
> have gone back and forth several times, and a few uncomfortable compromises
> were struck. It is quite possible that some major design decisions will
> have to be revisited before this PEP can be accepted. Nevertheless, we're
> happy with the current proposal, and we have provided ample discussion in
> the PEP under the headings of Rejected Ideas and Deferred Ideas. Please
> read those before proposing changes!

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MVQJA7LDO4NNQEU27Q6ERUWC5GRBQ7SG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 622: Structural Pattern Matching


On 2020-06-23 17:01, Guido van Rossum wrote:
I'm happy to present a new PEP for the python-dev community to review. 
This is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and 
Talin.


Many people have thought about extending Python with a form of pattern 
matching similar to that found in Scala, Rust, F#, Haskell and other 
languages with a functional flavor. The topic has come up regularly on 
python-ideas (most recently yesterday :-).


I'll mostly let the PEP speak for itself:
- Published: https://www.python.org/dev/peps/pep-0622/ (*)
- Source: https://github.com/python/peps/blob/master/pep-0622.rst

(*) The published version will hopefully be available soon.

I want to clarify that the design space for such a match statement is 
enormous. For many key decisions the authors have clashed, in some cases 
we have gone back and forth several times, and a few uncomfortable 
compromises were struck. It is quite possible that some major design 
decisions will have to be revisited before this PEP can be accepted. 
Nevertheless, we're happy with the current proposal, and we have 
provided ample discussion in the PEP under the headings of Rejected 
Ideas and Deferred Ideas. Please read those before proposing changes!


I'd like to end with the contents of the README of the repo where we've 
worked on the draft, which is shorter and gives a gentler introduction 
than the PEP itself:



# Pattern Matching

This repo contains a draft PEP proposing a `match` statement.

Origins
---

The work has several origins:

- Many statically compiled languages (especially functional ones) have
   a `match` expression, for example
   
[Scala](http://www.scala-lang.org/files/archive/spec/2.11/08-pattern-matching.html),

   [Rust](https://doc.rust-lang.org/reference/expressions/match-expr.html),
   
[F#](https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching);

- Several extensive discussions on python-ideas, culminating in a
   summarizing
   [blog 
post](https://tobiaskohn.ch/index.php/2018/09/18/pattern-matching-syntax-in-python/)

   by Tobias Kohn;
- An independently developed [draft
   
PEP](https://github.com/ilevkivskyi/peps/blob/pattern-matching/pep-.rst)

   by Ivan Levkivskyi.

Implementation
--

A full reference implementation written by Brandt Bucher is available
as a [fork]((https://github.com/brandtbucher/cpython/tree/patma)) of
the CPython repo.  This is readily converted to a [pull
request](https://github.com/brandtbucher/cpython/pull/2)).

Examples


Some [example 
code](https://github.com/gvanrossum/patma/tree/master/examples/) is 
available from this repo.


Tutorial


A `match` statement takes an expression and compares it to successive
patterns given as one or more `case` blocks.  This is superficially
similar to a `switch` statement in C, Java or JavaScript (an many
other languages), but much more powerful.

The simplest form compares a target value against one or more literals:

```py
def http_error(status):
     match status:
         case 400:
             return "Bad request"
         case 401:
             return "Unauthorized"
         case 403:
             return "Forbidden"
         case 404:
             return "Not found"
         case 418:
             return "I'm a teapot"
         case _:
             return "Something else"
```

Note the last block: the "variable name" `_` acts as a *wildcard* and
never fails to match.

You can combine several literals in a single pattern using `|` ("or"):

```py
         case 401|403|404:
             return "Not allowed"
```

Patterns can look like unpacking assignments, and can be used to bind
variables:

```py
# The target is an (x, y) tuple
match point:
     case (0, 0):
         print("Origin")
     case (0, y):
         print(f"Y={y}")
     case (x, 0):
         print(f"X={x}")
     case (x, y):
         print(f"X={x}, Y={y}")
     case _:
         raise ValueError("Not a point")
```

Study that one carefully!  The first pattern has two literals, and can
be thought of as an extension of the literal pattern shown above.  But
the next two patterns combine a literal and a variable, and the
variable is *extracted* from the target value (`point`).  The fourth
pattern is a double extraction, which makes it conceptually similar to
the unpacking assignment `(x, y) = point`.

If you are using classes to structure your data (e.g. data classes)
you can use the class name followed by an argument list resembling a
constructor, but with the ability to extract variables:

```py
from dataclasses import dataclass

@dataclass
class Point:
     x: int
     y: int

def whereis(point):
     match point:
         case Point(0, 0):
             print("Origin")
         case Point(0, y):
             print(f"Y={y}")
         case Point(x, 0):
             print(f"X={x}")
         case Point():
             print("Somewhere else")
         case _:
             print("Not a point")
```

We can use

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Antoine Pitrou

On Tue, 23 Jun 2020 16:34:28 +0200
Victor Stinner  wrote:
> Le mar. 23 juin 2020 à 15:56, Petr Viktorin  a écrit :
> > > Adding or removing members of C structures is causing multiple backward
> > > compatibility issues.
> > >
> > > Adding a new member breaks the stable ABI (PEP 384), especially for
> > > types declared statically (e.g. ``static PyTypeObject MyType =
> > > {...};``).  
> >
> > PyTypeObject is explicitly not part of the stable ABI, see PEP 384:
> > https://www.python.org/dev/peps/pep-0384/#structures
> > I don't know why Py_TPFLAGS_HAVE_FINALIZE was added, but it wasn't for
> > the PEP 384 stable ABI.  
> 
> Maybe Antoine Pitrou knows the rationale why Py_TPFLAGS_HAVE_FINALIZE
> flag was added. Removing the flag was discussed at:

Mainly because some packagers (how many I don't know) had the habit of
building a single extension DLL and distributing it for different
Python versions on Windows.

It was much easier to add the flag than to pollute the PEP 442
discussion with a sub-discussion about the robustness of such ABI
stability claims (which officially didn't exist, but were still relied
on by some users / packagers).

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EJ5QPTZZDZVNQNEUHENPPNKIUERJ2V4J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: New optional dependency

2020-06-23 Thread Gregory P. Smith

On Tue, Jun 23, 2020 at 7:41 AM Elisha Paine  wrote:

> Hi all,
>
> I am looking at getting TkDND support into tkinter, and have created
> issue40893. However, we are currently considering the practicalities of
> adding a new optional dependency to Python and I was hoping someone
> could help answer the question of: is there a procedure for this?
>
> The problem is that third-party distributors need to be notified of the
> change and edit the package details accordingly. The current idea is
> just to make it very clear on the “what’s new” page, however this would
> not guarantee it would be seen, so I am very open to ideas/opinions.
>
> Thanks,
> Elisha
>

What's New documentation combined with a configure.ac check to
conditionally compile it only when available is sufficient.  (once that
works, also ensuring that the dependency is added to the windows and mac
build setups so our binary builds include it)

It is on the third party distributors to pay attention.  If that worries
you, file your own issues in the debian and fedora trackers and that should
do it.

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DE7NIJATCRMQEQOUNWGSEIAL3T5GSWO5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 622: Structural Pattern Matching

I'm happy to present a new PEP for the python-dev community to review. This
is joint work with Brandt Bucher, Tobias Kohn, Ivan Levkivskyi and Talin.

Many people have thought about extending Python with a form of pattern
matching similar to that found in Scala, Rust, F#, Haskell and other
languages with a functional flavor. The topic has come up regularly on
python-ideas (most recently yesterday :-).

I'll mostly let the PEP speak for itself:
- Published: https://www.python.org/dev/peps/pep-0622/ (*)
- Source: https://github.com/python/peps/blob/master/pep-0622.rst

(*) The published version will hopefully be available soon.

I want to clarify that the design space for such a match statement is
enormous. For many key decisions the authors have clashed, in some cases we
have gone back and forth several times, and a few uncomfortable compromises
were struck. It is quite possible that some major design decisions will
have to be revisited before this PEP can be accepted. Nevertheless, we're
happy with the current proposal, and we have provided ample discussion in
the PEP under the headings of Rejected Ideas and Deferred Ideas. Please
read those before proposing changes!

I'd like to end with the contents of the README of the repo where we've
worked on the draft, which is shorter and gives a gentler introduction than
the PEP itself:


# Pattern Matching

This repo contains a draft PEP proposing a `match` statement.

Origins
---

The work has several origins:

- Many statically compiled languages (especially functional ones) have
  a `match` expression, for example
  [Scala](
http://www.scala-lang.org/files/archive/spec/2.11/08-pattern-matching.html),
  [Rust](https://doc.rust-lang.org/reference/expressions/match-expr.html),
  [F#](
https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching
);
- Several extensive discussions on python-ideas, culminating in a
  summarizing
  [blog post](
https://tobiaskohn.ch/index.php/2018/09/18/pattern-matching-syntax-in-python/
)
  by Tobias Kohn;
- An independently developed [draft
  PEP](
https://github.com/ilevkivskyi/peps/blob/pattern-matching/pep-.rst)
  by Ivan Levkivskyi.

Implementation
--

A full reference implementation written by Brandt Bucher is available
as a [fork]((https://github.com/brandtbucher/cpython/tree/patma)) of
the CPython repo.  This is readily converted to a [pull
request](https://github.com/brandtbucher/cpython/pull/2)).

Examples


Some [example code](
https://github.com/gvanrossum/patma/tree/master/examples/) is available
from this repo.

Tutorial


A `match` statement takes an expression and compares it to successive
patterns given as one or more `case` blocks.  This is superficially
similar to a `switch` statement in C, Java or JavaScript (an many
other languages), but much more powerful.

The simplest form compares a target value against one or more literals:

```py
def http_error(status):
match status:
case 400:
return "Bad request"
case 401:
return "Unauthorized"
case 403:
return "Forbidden"
case 404:
return "Not found"
case 418:
return "I'm a teapot"
case _:
return "Something else"
```

Note the last block: the "variable name" `_` acts as a *wildcard* and
never fails to match.

You can combine several literals in a single pattern using `|` ("or"):

```py
case 401|403|404:
return "Not allowed"
```

Patterns can look like unpacking assignments, and can be used to bind
variables:

```py
# The target is an (x, y) tuple
match point:
case (0, 0):
print("Origin")
case (0, y):
print(f"Y={y}")
case (x, 0):
print(f"X={x}")
case (x, y):
print(f"X={x}, Y={y}")
case _:
raise ValueError("Not a point")
```

Study that one carefully!  The first pattern has two literals, and can
be thought of as an extension of the literal pattern shown above.  But
the next two patterns combine a literal and a variable, and the
variable is *extracted* from the target value (`point`).  The fourth
pattern is a double extraction, which makes it conceptually similar to
the unpacking assignment `(x, y) = point`.

If you are using classes to structure your data (e.g. data classes)
you can use the class name followed by an argument list resembling a
constructor, but with the ability to extract variables:

```py
from dataclasses import dataclass

@dataclass
class Point:
x: int
y: int

def whereis(point):
match point:
case Point(0, 0):
print("Origin")
case Point(0, y):
print(f"Y={y}")
case Point(x, 0):
print(f"X={x}")
case Point():
print("Somewhere else")
case _:
print("Not a point")
```

We can use keyword parameters too.  The following patterns are all
equivalent (and all bind the `y` attribute to the `var`

[Python-Dev] Re: New optional dependency - TkDND

2020-06-23 Thread Petr Viktorin


On 2020-06-23 12:14, Elisha Paine wrote:

Hi all,

I am looking at getting TkDND support into tkinter, and have created
issue40893. However, we are currently considering the practicalities of
adding a new optional dependency to Python and I was hoping someone
could help answer the question of: is there a procedure for this?

The problem is that third-party distributors need to be notified of the
change and edit the package details accordingly. The current idea is
just to make it very clear on the “what’s new” page, however this would
not guarantee it would be seen, so I am very open to ideas/opinions.


Making it clear on “what’s new” would work best for me (as a Fedora 
packager). I don't think there are many other choices.


Obligatory question: What's the advantage of having TkDND bindings in 
the standard library, as opposed to making it a third-party package?
(And perhaps, in the stdlib, making it easier to wrap Tk packages in 
general?)



(FWIW, I read linux-sig but didn't get to respond before the python-dev 
post. So I'm replying here.)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YA2QQMAKGREPJYPPMRDD3V7Y5JMVHA2G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Stefan Behnel

Hi Victor,

thanks for your continued work on improving the C-API.

I'll comment on the PEP inline.

Victor Stinner schrieb am 22.06.20 um 14:10:
> PEP available at: https://www.python.org/dev/peps/pep-0620/
> [...]
> Motivation
> ==
> 
> The C API blocks CPython evolutions
> ---
> 
> Adding or removing members of C structures is causing multiple backward
> compatibility issues.
> 
> Adding a new member breaks the stable ABI (PEP 384), especially for
> types declared statically (e.g. ``static PyTypeObject MyType =
> {...};``). In Python 3.4, the PEP 442 "Safe object finalization" added
> the ``tp_finalize`` member at the end of the ``PyTypeObject`` structure.
> For ABI backward compatibility, a new ``Py_TPFLAGS_HAVE_FINALIZE`` type
> flag was required to announce if the type structure contains the
> ``tp_finalize`` member. The flag was removed in Python 3.8 (`bpo-32388
> `_).

Probably not the best example. I think this is pretty much normal API
evolution. Changing the deallocation protocol for objects is going to
impact any public API in one way or another. PyTypeObject is also not
exposed with its struct fields in the limited API, so your point regarding
"tp_print" is also not a strong one.


> Same CPython design since 1990: structures and reference counting
> -
> Members of ``PyObject`` and ``PyTupleObject`` structures have not
> changed since the "Initial revision" commit (1990)

While I see an advantage in hiding the details of PyObject (specifically
memory management internals), I would argue that there simply isn't much to
improve in PyTupleObject, so these two don't fly at the same level for me.


> Why is PyPy more efficient than CPython?
> 
> 
> The PyPy project is a Python implementation which is 4.2x faster than
> CPython on average. PyPy developers chose to not fork CPython, but start
> from scratch to have more freedom in terms of optimization choices.
> 
> PyPy does not use reference counting, but a tracing garbage collector
> which moves objects. Objects can be allocated on the stack (or even not
> at all), rather than always having to be allocated on the heap.
> 
> Objects layouts are designed with performance in mind. For example, a
> list strategy stores integers directly as integers, rather than objects.
> 
> Moreover, PyPy also has a JIT compiler which emits fast code thanks to
> the efficient PyPy design.

I would be careful with presenting examples of PyPy optimisations here.
Whichever you choose could easily give the impression that they are the
most important changes that made PyPy faster and should therefore be
followed in CPython. I doubt that there are any "top changes" that made the
biggest difference for PyPy. Even large breakthroughs on their side stand
on the shoulders of other important changes that may not have been visible
by themselves in the performance graphs.

CPython will not be rewritten from scratch, will continue to have its own
infrastructure, and will therefore have its own specific tweaks that it
will benefit from. Trying things out is fine, but there is no guarantee
that following a specific change in PyPy will make a similar difference in
CPython and its own ecosystem.


> PyPy bottleneck: the Python C API
> -
> While PyPy is way more efficient than CPython to run pure Python code,
> it is as efficient or slower than CPython to run C extensions.
> [...]
> Hide implementation details
> ---
> 
> Hiding implementation details from the C API has multiple advantages:
> 
> * It becomes possible to experiment with more advanced optimizations in
>   CPython than just micro-optimizations. For example, tagged pointers,
>   and replace the garbage collector with a tracing garbage collector
>   which can move objects.
> * Adding new features in CPython becomes easier.
> * PyPy should be able to avoid conversions to CPython objects in more
>   cases: keep efficient PyPy objects.
> * It becomes easier to implement the C API for a new Python
>   implementation.
> * More C extensions will be compatible with Python implementations other
>   than CPython.

I understand the goal of experimenting with new optimisations and larger
changes internally.

If, however, the goal is to make it easier for other implementations to
support (existing?) C extensions, then breaking all existing C extensions
in CPython first does not strike me as a good way to get there. :)

My feeling is that PyPy specifically is better served with the HPy API,
which is different enough to consider it a mostly separate API, or an
evolution of the limited API, if you want. Suggesting that extension
authors support two different APIs is much, but forcing them to support the
existing CPython C-API (for legacy reasons) and the changed CPython C-API
(for future compatibility), and

[Python-Dev] Re: (PEP 620) C API for efficient loop iterating on a sequence of PyObject** or other C types

2020-06-23 Thread Stefan Behnel

Victor Stinner schrieb am 23.06.20 um 11:18:
> Maybe an object can
> generate a temporary PyObject** view which requires to allocate
> resources (like memory) and the release function would release these
> resources.

I agree that this is more explicit when it comes to resource management,
but there is nothing that beats direct native data structure access when it
comes to speed. If a "PyObject*[]" is not what the runtime uses internally
as data structure, then why hand it out as an interface to users who
require performance? There's PyIter_Next() already for those who don't.

If the intention is to switch to a more efficient internal data structure
inside of CPython (or expose in PyPy whatever that uses), then I would look
more at PEP-393 for a good interface here, or "array.array". It's perfectly
fine to have 20 different internal array types, as long as they are
explicitly and safely exposed to users.

Stefan
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YWN2NXAHAHN4FAZTVCSP4ZVLDE2CGVXQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] New optional dependency

2020-06-23 Thread Elisha Paine

Hi all,

I am looking at getting TkDND support into tkinter, and have created
issue40893. However, we are currently considering the practicalities of
adding a new optional dependency to Python and I was hoping someone
could help answer the question of: is there a procedure for this?

The problem is that third-party distributors need to be notified of the
change and edit the package details accordingly. The current idea is
just to make it very clear on the “what’s new” page, however this would
not guarantee it would be seen, so I am very open to ideas/opinions.

Thanks,
Elisha
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UMESQSJCOV3Q4UNYPWHZPY327UD2MYYT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

Le mar. 23 juin 2020 à 15:56, Petr Viktorin  a écrit :
> > Adding or removing members of C structures is causing multiple backward
> > compatibility issues.
> >
> > Adding a new member breaks the stable ABI (PEP 384), especially for
> > types declared statically (e.g. ``static PyTypeObject MyType =
> > {...};``).
>
> PyTypeObject is explicitly not part of the stable ABI, see PEP 384:
> https://www.python.org/dev/peps/pep-0384/#structures
> I don't know why Py_TPFLAGS_HAVE_FINALIZE was added, but it wasn't for
> the PEP 384 stable ABI.

Maybe Antoine Pitrou knows the rationale why Py_TPFLAGS_HAVE_FINALIZE
flag was added. Removing the flag was discussed at:

* https://bugs.python.org/issue32388
* https://mail.python.org/pipermail/python-dev/2017-December/151328.html


> > Summary
> > ---
> >
> > * (**Completed**) Reorganize the C API header files: create
> > ``Include/cpython/`` and
> >``Include/internal/`` subdirectories.
> > * (**Completed**) Move private functions exposing implementation
> > details to the internal
> >C API.
> > * (**Completed**) Convert macros to static inline functions.
> > * (**Completed**) Add new functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` 
> > and
> >``Py_SET_SIZE()``. The ``Py_TYPE()``, ``Py_REFCNT()`` and
> >``Py_SIZE()`` macros become functions which cannot be used as l-value.
> > * (**Completed**) New C API functions must not return borrowed
> >references.
> > * (**In Progress**) Provide ``pythoncapi_compat.h`` header file.
> > * (**In Progress**) Make structures opaque, add getter and setter
> >functions.
> > * (**Not Started**) Deprecate ``PySequence_Fast_ITEMS()``.
> > * (**Not Started**) Convert ``PyTuple_GET_ITEM()`` and
> >``PyList_GET_ITEM()`` macros to static inline functions.
>
> What does **Completed** mean? All work in that category is done, and any
> new changes in the category will require a new PEP (or a change to this
> PEP)?

The purpose of the PEP is to show an overview of the work already done
towards the same goal. Changes were scattered in multiple issues and
so I don't think that it was easy to follow it.

The PEP cannot be exhaustive, hiding implementation details is a
work-in-progress.

Usually, once a PEP is approved, it should not be modified.

I don't think that minor incompatible changes should require a new whole PEP.

The purpose of the PEP is to define an overall process for past and
future changes in this area, and also announces  our willingness for
this (hide implementation details).

If tomorrow, an incompatible C API change becomes very controversial,
maybe another PEP will be justified.

So far, incompatible C API changes were done with no PEP. I decided to
write a PEP for the Py_TYPE() change: disallow to use it as an
l-value, that's one of the "most" incompatible changes since I started
to work on this.


> It seems that this PEP is largely asking for forgiveness, so trying to
> find better ways to solve the problems would be a waste of time at this
> point.

Most completed changes were already discussed during physical
meetings, like Pycons and Core dev sprints. From what I recall,
everybody (involved in the discussion) agreed on the solution.

But there is always room for enhancements! The purpose of the PEP is
also to make the work already done more visible, so it's easier to
reason about the overall picture.

Feel free to propose your ideas, it's exactly the role of a discussion on a PEP!


> > Make structures opaque
> > --
> >
> > All structures of the C API should become opaque: C extensions must
> > use getter or setter functions to get or set structure members. For
> > example, ``tuple->ob_item[0]`` must be replaced with
> > ``PyTuple_GET_ITEM(tuple, 0)``.
>
> All structures?
> I don't think we can make PyModuleDef opaque, for example.
> PEP 384 lists more structures, some of which I don't think should be
> opaque: https://www.python.org/dev/peps/pep-0384/#structures

Oh, no, PyModuleDef is fine. I would like to make the following
structures opaque:

* PyInterpreterState
* PyThreadState
* PyGC_Head
* PyTypeObject
* PyObject and PyVarObject
* PyTypeObject
* All types which inherit from PyObject or PyVarObject

But I'm not sure about the exact list. Let's start with this list :-)


> > Process to reduce the number of broken C extensions
> > ===
> >
> > Process to reduce the number of broken C extensions when introducing C
> > API incompatible changes listed in this PEP:
> >
> > * Estimate how many popular C extensions are affected by the
> >incompatible change.
> > * Coordinate with maintainers of broken C extensions to prepare their
> >code for the future incompatible change.
> > * Introduce the incompatible changes in Python. The documentation must
> >explain how to port existing code. It is recommended to merge such
> >changes at the beginning of a development cycle to have more time for
> >tests.
> > *

[Python-Dev] Re: Cython and incompatible C API changes

2020-06-23 Thread Baptiste Carvello

Hi,

Le 22/06/2020 à 11:21, Stefan Behnel a écrit :
> [...]
> 
> Basically, for maintained packages, I consider shipping the generated C
> code the right way. Less hassle, easier debugging, better user experience.
> For unmaintained packages, regenerating the C code at build time *can*
> extend the lifetime of the package to newer environments for as long as it
> does not run into failures due to Cython compiler changes (so you trade one
> compatibility level for another one).
> 
> The question is whether the point at which a package becomes unmaintained
> can ever be clear enough to make the switch. Regardless of which way you
> choose, at some point in the future someone will have to do something,
> either to your code or to your build setup, in order to prevent fatal bitrot.

This makes me wonder: could there be a (non-default, use at your own
risk) pip option to rerun Cython before compile? This would allow users
to try and workaround compilation failures on their side.

Not that the unpack/run Cython/bdist_wheel dance is really difficult,
but users have to find out/remember how to do it.

Cheers,
Baptiste
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6GTTUMLAT3MMMOTNHLQFRH2C2PZBFWOB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Petr Viktorin





On 2020-06-22 14:10, Victor Stinner wrote:

Hi,

PEP available at: https://www.python.org/dev/peps/pep-0620/


This PEP is the result of 4 years of research work on the C API:
https://pythoncapi.readthedocs.io/

It's the third version. The first version (2017) proposed to add a
"new C API" and advised C extensions maintainers to opt-in for it: it
was basically the same idea as PEP 384 limited C API but in a
different color. Well, I had no idea of what I was doing :-) The
second version (April 2020) proposed to add a new Python runtime built
from the same code base as the regular Python runtime but in a
different build mode, the regular Python would continue to be fully
compatible.

I wrote the third version, the PEP 620, from scratch. It now gives an
explicit and concrete list of incompatible C API changes, and has
better motivation and rationale sections. The main PEP novelty is the
new pythoncapi_compat.h header file distributed with Python to provide
new C API functions to old Python versions, the second novelty is the
process to reduce the number of broken C extensions.

Whereas PEPs are usually implemented in a single Python version, the
implementation of this PEP is expected to be done carefully over
multiple Python versions. The PEP lists many changes which are already
implemented in Python 3.7, 3.8 and 3.9. It defines a process to reduce
the number of broken C extensions when introducing the incompatible C
API changes listed in the PEP. The process dictates the rhythm of
these changes.



PEP: 620
Title: Hide implementation details from the C API
Author: Victor Stinner 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 19-June-2020
Python-Version: 3.10

Abstract


Introduce C API incompatible changes to hide implementation details.

Once most implementation details will be hidden, evolution of CPython
internals would be less limited by C API backward compatibility issues.
It will be way easier to add new features.

It becomes possible to experiment with more advanced optimizations in CPython
than just micro-optimizations, like tagged pointers.

Define a process to reduce the number of broken C extensions.

The implementation of this PEP is expected to be done carefully over
multiple Python versions. It already started in Python 3.7 and most
changes are already completed. The `Process to reduce the number of
broken C extensions`_ dictates the rhythm.


Motivation
==

The C API blocks CPython evolutions
---

Adding or removing members of C structures is causing multiple backward
compatibility issues.

Adding a new member breaks the stable ABI (PEP 384), especially for
types declared statically (e.g. ``static PyTypeObject MyType =
{...};``).


PyTypeObject is explicitly not part of the stable ABI, see PEP 384: 
https://www.python.org/dev/peps/pep-0384/#structures
I don't know why Py_TPFLAGS_HAVE_FINALIZE was added, but it wasn't for 
the PEP 384 stable ABI.


Can you find a different example, so users are not misled?


In Python 3.4, the PEP 442 "Safe object finalization" added
the ``tp_finalize`` member at the end of the ``PyTypeObject`` structure.
For ABI backward compatibility, a new ``Py_TPFLAGS_HAVE_FINALIZE`` type
flag was required to announce if the type structure contains the
``tp_finalize`` member. The flag was removed in Python 3.8 (`bpo-32388
`_).

The ``PyTypeObject.tp_print`` member, deprecated since Python 3.0
released in 2009, has been removed in the Python 3.8 development cycle.
But the change broke too many C extensions and had to be reverted before
3.8 final release. Finally, the member was removed again in Python 3.9.

C extensions rely on the ability to access directly structure members,
indirectly through the C API, or even directly.


I think you want to remove a "directly" from that sentence.


Modifying structures
like ``PyListObject`` cannot be even considered.

The ``PyTypeObject`` structure is the one which evolved the most, simply
because there was no other way to evolve CPython than modifying it.

In the C API, all Python objects are passed as ``PyObject*``: a pointer
to a ``PyObject`` structure. Experimenting tagged pointers in CPython is
blocked by the fact that a C extension can technically dereference a
``PyObject*`` pointer and access ``PyObject`` members. Small "objects"
can be stored as a tagged pointer with no concrete ``PyObject``
structure.


I think this would be confusing to people who don't already know what 
you mean. May I suggest:
A C extension can technically dereference a ``PyObject*`` pointer and 
access ``PyObject`` members. This prevents experiments like tagged 
pointers (storing small values as ``PyObject*`` which does not point to 
a valid ``PyObject`` structure).



Replacing Python garbage collector with a tracing garbage collector
would also need to remove ``PyObject.ob_refcnt`` reference counter,
whereas currently ``Py_INCREF()`` and

[Python-Dev] Re: (PEP 620) C API for efficient loop iterating on a sequence of PyObject** or other C types

2020-06-23 Thread Antoine Pitrou

On Tue, 23 Jun 2020 11:18:37 +0200
Victor Stinner  wrote:
> 
> Pseudo-code:
> 
> void iterate(PyObject *obj)
> {
> PyObjectPP_View view;
> 
> if (PyObjectPP_View_Get(, obj)) {
>   // fast-path: the object provides a PyObject** view
>   for (Py_ssize_t i=0; i < view.len; i++ {
> PyObject *item = view.array[i];
> ...
>   }
>   PyObjectPP_View_Release();
> }
> else {
>   // slow code path using PySequence_GetItem() or anything else
>   ...
> }

It is quite cumbersome for extension code to have to re-implement all
this by hand.  Instead, it would be nice to have a "Visit" primitive so
that one can write e.g.:

void iterate(PyObject* obj)
{
  Py_VisitObjectSequence([&](PyObject* item) {
// ...
  });
}

The above is a C++ lambda function (a closure, actually).  The C
spelling would be less nice, and you'd have to ensure that it is still
performant (i.e., that the visitor is inlined inside the iteration
loop - at least in release builds).

(I called it Py_VisitObjectSequence so that you can also have
Py_VisitIntSequence, Py_VisitFloatSequence, etc.)

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4SWMGP54OWHE3CBW23NUP5KF4WK5R4UV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-23 Thread Thomas Wouters

On Tue, Jun 23, 2020 at 11:33 AM Victor Stinner  wrote:

> Hi Neil,
>
> Le mar. 23 juin 2020 à 03:47, Neil Schemenauer
>
> > One aspect of the API that could be improved is memory management
> > for PyObjects.  The current API is quite a mess and for no good
> > reason except legacy, IMHO.  The original API design allowed
> > extension types to use their own memory allocator.  E.g. they could
> > call their own malloc()/free() implemention and the rest of the
> > CPython runtime would handle that.  One consequence is that
> > Py_DECREF() cannot call PyObject_Free() but instead has to call
> > tp_dealloc().  There was supposed to be multiple layers of
> > allocators, PyMem vs PyObject, but since the layering was not
> > enforced, we ended up with a bunch of aliases to the same underlying
> > function.
>
> I vaguely recall someone explaining that Python memory allocator
> created high memory fragmentation, and using a dedicated memory
> allocator was way more efficient. But I concur that the majority of
> people never override default tp_new and tp_free functions.

Not so much Python's memory allocator (it does better than most), but just
plain malloc. However, the answer in these cases isn't to replace the
allocator for a few extension types, since that wouldn't affect any of
Python's own allocations. The better answer is to replace malloc
altogether. At Google we use tcmalloc for everything, by linking it into
the binaries we build. However, the effect on Python's allocations isn't
very big (but it's still measurable) because obmalloc does a pretty good
job; we do it more for the C/C++ libraries we end up wrapping, where it can
matter *a lot*. I don't think we ever set tp_new/tp_free to anything other
than the defaults, and we could certainly live with it going away. We also
experimented with disabling obmalloc when using tcmalloc, but obmalloc
still does measurably better than tcmalloc.

There's another reason not to have different allocators, at least not ones
that don't trickle down to 'malloc': AddressSanitizer and ThreadSanitizer
rely on intercepting all allocations, and they are *very* useful tools for
any C/C++ codebase. They don't (at the moment) particularly benefit Python
code, but they certainly do benefit CPython extensions and the C/C++
libraries they wrap.

I think the ability for per-type allocation/deallocation routines isn't
really about efficiency, but more about giving more control to embedding
systems (or libraries wrapped by extension modules) about how *their*
objects are allocated. It doesn't make much sense, however, because Python
wouldn't allocate their objects anyway, just the Python objects wrapping
theirs. Allocating CPython objects should be CPython's job.

FWIW, I suspect the biggest problem with getting rid of tp_new/tp_free is
code that does *more* than just allocate in those functions, only because
the authors didn't realise they should be doing it in tp_alloc/tp_dealloc
instead.

-- 
Thomas Wouters 

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P4M57U4XPQYN2Y7RKTATU46C6KR7H3UM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Draft PEP: Remove wstr from Unicode

Le mar. 23 juin 2020 à 04:02, Inada Naoki  a écrit :
> Legacy unicode representation is using wstr so legacy unicode support
> is removed with wstr.
> PyUnicode_READY() will be no-op when wstr is removed.  We can remove
> calling of PyUnicode_READY() since then.
>
> I think we can deprecate PyUnicode_READY() when wstr is removed.

Would it be possible to rewrite the plan differently (merge
Specification sections) to list changes per Python version? Something
like:

== Python 3.9 ==

* Deprecate xxx in the documentation and add Py_DEPRECATED()
* Remove xxx

== Python 3.10 ==

* Deprecate xxx in the documentation and add Py_DEPRECATED()
* Add DeprecationWarning at runtime in xxx
* Remove xxx

== Python 3.11 ==

* Remove wstr member
* Remove xxx functions
* PyUnicode_READY() is kept for backward compatibility but it
deprecated and becomes as no-opt
* ...

== Python 3.12 ==

* Remove PyUnicode_READY()
* ...


Also, some functions are already deprecated. Would you mind to list
them in the PEP? I fail to track the status of each function.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3VMF2CB2K5NTFUUUNOFJ6R52I2OMTF55/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

Hi Neil,

Le mar. 23 juin 2020 à 03:47, Neil Schemenauer
 a écrit :
> Thanks for putting work into this.

You're welcome, I took some ideas from your tagged pointer proof of
concept ;-) I recall that we met the same C API issues in our
experiments ;-)


> Changes must be made for
> well founded reasons and not just because we think it makes a
> "cleaner" API.  I believe you are following those principles.

I mostly used the tagged pointer as a concrete goal to decide which
changes are required or not. PyPy and HPy developers also gave me API
that they would like to see disappearing :-)


> One aspect of the API that could be improved is memory management
> for PyObjects.  The current API is quite a mess and for no good
> reason except legacy, IMHO.  The original API design allowed
> extension types to use their own memory allocator.  E.g. they could
> call their own malloc()/free() implemention and the rest of the
> CPython runtime would handle that.  One consequence is that
> Py_DECREF() cannot call PyObject_Free() but instead has to call
> tp_dealloc().  There was supposed to be multiple layers of
> allocators, PyMem vs PyObject, but since the layering was not
> enforced, we ended up with a bunch of aliases to the same underlying
> function.

I vaguely recall someone explaining that Python memory allocator
created high memory fragmentation, and using a dedicated memory
allocator was way more efficient. But I concur that the majority of
people never override default tp_new and tp_free functions.

By the way, in Python 3.8, heap types started to increase their
reference counter when an instance is created, but decrementing the
type reference counter is the responsibility of the tp_dealloc
function and we failed to find a way to automate it.

More info on this issue:

* https://bugs.python.org/issue35810
* https://bugs.python.org/issue40217
* https://docs.python.org/dev/whatsnew/3.9.html#changes-in-the-c-api

C extensions maintainers now have to update their tp_dealloc method,
or their application will never be able to destroy their heap types.


> Perhaps there are a few cases when the flexibility to use a custom
> object allocator is useful.  I think in practice it is very rare
> than an extension needs to manage memory itself.  To achieve
> something similar, allow a PyObject to have a reference to some
> externally managed resource and then the tp_del method would take
> care of freeing it.  IMHO, the Python runtime should be in charge of
> allocating and freeing PyObject memory.

Do you think that it should be in PEP 620 or can it be done
independently? I don't know how to implement it, I have no idea how
many C extensions would be broken, etc.

I don't see an obvious relationship with interoperability with other
Python implementations or the stable ABI and hiding tp_del/tp_free.

While making object allocation and deallocation simpler would be nice,
it doesn't seem "required" in PEP 620 for now. What do you think?


> Another place for improvement is that the C API is unnecessarily
> large.  E.g. we don't really need PyList_GetItem(),
> PyTuple_GetItem(), and PyObject_GetItem().  Every extra API is a
> potential leak of implementation details and a burden for
> alternative VMs.  Maybe we should introduce something like
> WIN32_LEAN_AND_MEAN that hides all the extra stuff.  The
> Py_LIMITED_API define doesn't really mean the same thing since it
> tries to give ABI compatibility.  It would make sense to cooperate
> with the HPy project on deciding what parts are unnecessary.  Things
> like Cython might still want to use the larger API, to extract every
> bit of performance.  The vast majority of C extensions don't require
> that.

At the beginning, I had a plan to remove all functions and only keep
"abstract" functions like PyObject_GetItem().

Then someone asked what is the performance overhead of only using
abstract functions. I couldn't reply. Also, I didn't see a need to
only use abstract functions for now, so I abandoned this idea.

PyTuple_GetItem() returns a borrowed reference which is bad, whereas
PyObject_GetItem() returns a strong reference. Since PyPy cpyext
already solved this problem, I chose the leave the borrowed references
problem aside for now. Trying to fix all issues at once doesn't work
:-)

One issue of calling PyTuple_GetItem() or PyDict_GetItem() is that it
doesn't take in account the ability to override __getitem__() in a
subclass. Few developers write the correct code like:

if (PyDict_CheckExact(ns))
err = PyDict_SetItem(ns, name, v);
else
err = PyObject_SetItem(ns, name, v);

The PEP 620 is already quite long and introduces many incompatible
changes. I tried to make the PEP as short as possible and minimize the
number of incompatible C API changes.

Using Py_LIMITED_API provides a stable ABI, but it doesn't reduce the
Python maintenance burden, and other Python implementations must
continue to implement the full C API

[Python-Dev] (PEP 620) C API for efficient loop iterating on a sequence of PyObject** or other C types