[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Hi Oscar Quoting Oscar Benjamin : On Fri, 19 Feb 2021 at 15:41, Tobias Kohn wrote: [...] It's not easy now to look back over the history of all of this. My recollection of the original version of PEP 622 (if that's the right one...) was that it had an overcomplicated protocol for __match__. It needed to be simplified but in the end it was dropped. Without further context, I would agree with you: it is difficult to look back over the entire history of the pattern matching PEPs. However, on the one hand PEP 622 is directly linked in the current PEP 634. On the other hand, for those who actively particiated in the discussion of both PEP 622 as well as the DLS paper, I find it a bit too easy and 'convenient' to call those resources now 'hard to find'. That being said, I think it is part of your homework to first research about the history and what others have done before proposing your innovation. Granted, this is hard and very laborious work that often takes a long time and can be frustrating. But if we want to make progress and move forward, we have to stop running in circles.—To be absolutely clear here: I do not think that Mark's proposal is running in circles and I think it is fair enough to bring up these ideas. But I equally think it is well possible to acknowledge that one part of it is a discussion of existing ideas, and to have a look at why these ideas have not made it into PEP 634. The question now is if it will be straight-forward to retrofit a protocol to PEP 634 after it is released and when backward compatibility constraints kick in. PEP 653 (as discussed here) is precisely an attempt to retrofit a protocol to PEP 634. I think the difficulties involved in achieving that will become harder rather than easier in future. -- Oscar There are actually two protocols in PEP 653. One is about changing the fundamental design of pattern matching as outlined in PEP 634. This is a part that I reject for reasons presented in one of my last posts. The second one is the customisation part using `__deconstruct__` or `__match__`. This is something that I like a lot and would certainly like to see in pattern matching—alas, just being in favour of something is not a strong enough argument for it. Similar to my votum above: if we are going to discuss this, I think we need new arguments or evidence, something to show why the previous decision to drop it (for now) was wrong. For that it might be worth reading the respective section in deferred ideas of PEP 622. Some part of our decision was based on the notion that adding such a custom protocol later one will be relatively painless, but if we were wrong, please speak up and show us what we have overlooked. If you can present, for instance, some real-world use cases where this feature would enormously benefit sympy (or any other popular package), that would be helpful and a good starting point. Kind regards, Tobias ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7EK4K36F6NKD67VTBS2XWRXAN4E737AD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Hi Mark, Quoting Mark Shannon : Hi Tobias, [...] But they are not referenced in PEP 634. I shouldn't have to trawl the internet to find the rejected ideas section. https://dl.acm.org/doi/10.1145/3426422.3426983 That paper describes a `__match__` method, which is absent from PEP 634. Why? Cheers, Mark. You are right: you should not have to trawl the internet to get these information. However, given the history of the Pattern Matching PEPs and that PEP 622 is linked directly from PEP 634, I would think it is not that far a journey—even though there have been so many discussions that it has definitely become unwieldy to retain an overview... Anyway, the answer to both your questions lies in that the Pattern Matching feature itself is rather complex and the PEPs ended up being huge and hard to read and understand as it is. We therefore refrained from long lists of rejected ideas in PEP 634, since that has already been done in PEP 622. Moreover, the `__match__` method and protocol were removed from the PEPs to focus on a core infrastructure and keep it as simple as possible. I think such a customisable protocol will eventually add great value to pattern matching. However, one of the main arguments was to wait until we have more experience with pattern matching in Python and can give specific use cases for this extended protocol, which allow us to guide the design of the protocol. At the end of the day, I guess we quite agree that we would like to have `__match__`, `__deconstruct__` or whatever you want to name it. But the slight variations in design demonstrate that it might also be a good idea to carefully lay it out first, before adding it to the pattern matching engine. Hence, separating this from the core implementation seems like a good idea that I still fully support (despite also looking forward to having it some day in the future ^_^). Cheers, Tobias ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T6Y7DCKIXFYV3UQY7BHTARUBPUQ6BI2C/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Hi Mark, Quoting Mark Shannon : [...] If you had published these "more complicated, powerful protocols", you might be able to claim that this is a "rehash". But you didn't. I would say that these ideas have been quite prominently published: https://www.python.org/dev/peps/pep-0622/#custom-matching-protocol https://dl.acm.org/doi/10.1145/3426422.3426983 Moreover, much of the discussion is publicly available, for instance here: https://github.com/gvanrossum/patma/issues/8 Cheers, Tobias ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WS2DFBMBRTF5YXZS6IG42QNJNDJHPBYN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Hi Oscar, Quoting Oscar Benjamin : Yes, thanks Mark. I'm not sure I've fully understood the PEP yet but I can see some parts that I definitely like. [...] As I have just noted in my response to Mark, the aspect with the "deconstructor" (or `__match__` protocol as we called it) is definitely something that I like, too. Moreover, I think that packages like sympy might make a strong argument for it. That being said, perhaps we would have to start thinking about concrete use cases first and then consider how to best provide for those with an extended match protocol. Even though I still like the idea of a flexible match method, there might be aspects that I/we overlook without knowing exactly where we want to go with this. I'm not entirely sure but I think that with PEP 653 you can implement this like: def __deconstruct__(obj): if obj.step != 1: return obj.start, obj.stop, obj.step elif obj.start != 0: return obj.start, obj.stop else: return obj.stop I think that would then mean you could use a pattern range(10) to unambiguously match range(10) without matching range(10, 20) etc. This is certainly more of an anti-pattern (no pun intended) than an argument for the match protocol. Shuffling positional arguments around like this seems like a rather bad idea, particularly in the context of pattern matching. I'd rather write `case range(_, stop, _)` or `case range(0, stop, 1)` and be explicit here. Kind regards, Tobias ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PWJSQIQ7XHJRD7ZN7QWT6HF5MGMG47GN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Hi Mark, Thank you for your proposal to try and have more precise semantics for pattern matching. Of course, the proposal primarily introduces a new and extended protocol for pattern matching, upon which the 'semantics' is then based. I think it is important to recognise and discuss your work on this basis that it proposes a different protocol of how pattern matching works, before we consier whether your semantics really is 'more precise'. As I understand it, your PEP proposes two major differences: control over the matching process moves from patterns to the objects and a custom protocol is introduced (via the `__deconstruct__` method). There is also the matter of the `__match_kind__` with the intention of improving performance, which I find of minor importance here. If you go back and look at the history and development of PEP 634, you will find that we started off with a fully customisable method that we called `__match__` instead of `__deconstruct__`. The idea of more flexible and customisable pattern matching is thus certainly in the spirit of our proposal as well. However, we finally removed this part from our proposal due to strong concerns from the community that the match protocol is too complex, particularly for an initial version of pattern matching. Should the need arise, it would still be possible to later add such a customisation protocol in an upcoming version of Python. Given these concerns with respect to complexity and our concession to remove the `__match__` method, I am wondering: do you have strong arguments that would speak for inclusion of this feature in the first version of pattern matching after all? I would love to have this feature and am truly interested in any arguments in its favour. When pattern matching meets OOP, there is indeed some discussion that can be had on where the matching itself is to happen. Simply put: should an object recognise that it is an instance of a given class, or should the class recognise that an object is an instance of it? In the spirit of Python's `instancecheck` method, we opted for classes/patterns recognising whether an object matches, where the object itself is 'passive'. In other words, `case C()` would match any object that is an instance of a subclass of `C`, in line with usual OOP principles (where any instance of a subclass of `C` is still a valid instance of `C` itself). Your design turns this around and has the object take on the responsibility of checking whether it wants to match a specific pattern. This has the advantage that the object itself can morph into anything it wants to—as long as it is aware of the pattern. And therein lies the problem that IMHO fully negates one of the major advantages of pattern matching: that the patterns and the objects that are to be matched are de-coupled and fully independent. In other words: in your design it is not she who writes the patterns that controls what objects match, but he who designs the class hierarchy of the objects used. One of the great traits of pattern matching is that it works out-of-the-box even with objects from ancient or alien code bases. Pattern matching according to PEP 634 is isolated, it happens only where pattern matching is explicitly used. When writing classes, you do not have to worry about pattern matching at all, unless you want your classes to be used as patterns. And should we decide some time in the future that we want to introduce new kinds of patterns, we still do not have to change a single object because the matching happens in the patterns and not in the objects, anyway. That his would not be the case with your design is already obvious with the list of `__match_kind__` for common builtin classes, for instance. Although there are some other programming languages that chose to follow this route, I think it goes against the spirit of what is already there in Python, it violates the principle of separated concerns and is thus a rather ill-advised modification of the original protocol. Kind regards, Tobias ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7UIKPFTOU6APJGHW35A3P4JV4N6PL7K7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Questions about about the DLS 2020
Hi Koos, Yes, our three PEPs (634/635/636) are up to date. PEP 634 is the technical specification, so you probably want to start with the tutorial (PEP 636) or the rationale (PEP 635). https://www.python.org/dev/peps/pep-0636/ https://www.python.org/dev/peps/pep-0635/ Perhaps the following link to a discussion and vote might be a good starting point to get an overview of the other proposals that are around: https://discuss.python.org/t/gauging-sentiment-on-pattern-matching/5770 Kind regards, Tobias Quoting Koos Zevenhoven : I've had some things going on, and I'm still trying to catch up with the discussions here. Can someone tell me what would be the best place to look at the most recent proposal? Is one of the PEPs up to date? On Mon, Nov 16, 2020 at 7:02 PM Tobias Kohn wrote: _Hi Mark, Thank you for your interest and the questions. 1. This really comes down to how you look at it, or how you define pattern matching. The issue here is that the concept of pattern matching has grown into a large and somewhat diverse flock of interpretations and implementations (as a side note: interestingly enough, some of the only universally agreed-upon standards are to use `_` as a wildcard and not to mark names that capture/bind values---which are quite exactly the points most fiercely debatted here). Anyway, the paper presents the pattern matching structure we are proposing as one of three major variants of pattern matching: (a) Matching arguments to parameters in a function call, (b) Matching elements to elements in iterable unpacking, (c) Matching tree-like data to general patterns in a conditional pattern matching structure. The last one is the subject of the PEP and the paper. Nonetheless, in the first two cases (a) and (b), we find that indeed the computer will validate that the data matched the pattern and raise an exception if this fails. This is where this way of looking at it comes from. 2. Yes, that is indeed a deliberate simplification. The idea is to abstract away from the details of how exactly Python implements abstract syntax trees (which I honestly believe are irrelevant for the sake of the entire narrative). Moreover, using strings here allows us to exemplify the literal patterns, rather only showcasing only the constructor/class pattern. Essentially, this is a question of making the most out of the little space available. Since you have addressed this email to me directly, I would like to take this opportunity and briefly stress that this paper really grew out of a team effort. While I might have been the one pushing for an academic publication, the DLS'20 paper represents the input and ideas of all the authors, as well as the long discussions we had. Of course, I am happy to answer any questions about the paper, but it would be wrong to see me as the one person behind it. Cheers, Tobias Quoting Mark Shannon :_ _Hi Tobias, A couple of questions about the DLS 2020 paper. 1. Why do you use the term "validate" rather than "test" for the process of selecting a match? It seems to me, that this is a test, not a validation, as no exception is raised if a case doesn't match. 2. Is the error in the ast matching example, an intentional "simplification" or just an oversight? The example: ``` def simplify(node): match node: case BinOp(Num(left), '+', Num(right)): return Num(left + right) case BinOp(left, '+' | '-', Num(0)): return simplify(left) case UnaryOp('-', UnaryOp('-', item)): return simplify(item) case _: return node ``` is wrong. The correct version is ``` def simplify(node): match node: case BinOp(Num(left), Add(), Num(right)): return Num(left + right) case BinOp(left, Add() | Sub(), Num(0)): return simplify(left) case UnaryOp(USub(), UnaryOp(USub(), item)): return simplify(item) case _: return node ``` Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ETZGYRCF4DR6RJXTHGXIRZXINXJ76J2D/Code of Conduct: http://python.org/psf/codeofconduct/_ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CUNO57W7KSIM2WRROC5R43ZT7HUQZCZ6/ Code of Conduct: http://python.org/psf/codeofconduct/_ ___ Python-Dev mailing
[Python-Dev] Re: The semantics of pattern matching for Python
Hi David and Steve, There is hardly anything that needs to be added to your comments, of course. However, I consider these explicitly named attributes in the class pattern to be one of the most difficult aspects of our pattern matching proposal, which is something I just want to briefly acknowledge. Although there are good reasons for using the syntax as proposed, understanding that ``Spam(a=x)`` assigns the attribute ``a`` to the variable ``x`` is not quite as intuitive and straight forward. Using the curly braces for that purpose might help in that we instantly think differently and more in line with dictionaries. This small clue could potentially have quite an impact on readability. This idea has thus a huge advantage over the square brackets not because of PEP 637, but because it might change the way we look at the code at hand. The reason why I would not favour this specific syntax is because we expect that the predominant case would be to just write ``Spam(1, 2)`` without the attribute names. It is even possible to mix and match the two, which would then lead to something like ``Spam{1, b=2}``. In that case, the advantage of the dictionary-like notation might just evaporate I assume, leaving behind a notation that turns out to be unusual and somewhat annoying in the overwhelming majority of cases. There are a few other concepts and ideas behind it all, which I do not want to go into unless there is demand for it, but part of it is the idea that we want to be as much as possible in line with the syntax already used in iterable unpacking, say. Kind regards, Tobias Quoting David Mertz : On Sat, Nov 21, 2020 at 12:23 PM Steven D'Aprano wrote: Clearly Spam(a=1, b=2) does not necessarily result in an instance with attributes a and b. But the pattern `Spam(a=1, b=2)` is intended to be equivalent to (roughly): if (instance(obj, Spam) and getattr(obj, a) == 1 and getattr(obj, b) == 2) it doesn't imply that obj was *literally* created by a call to the constructor `Spam(a=1, b=2)`, or even that this call would be possible. I think this explanation makes me not worry about the fact that `Spam(a=1, b=2)` in a pattern looks a lot like a constructor. Like some other commenters, I was vaguely bothered that the identical spelling might have these different meanings in different contexts. But I think a match case just clearly enough IS a different context that using slightly different intuitions is no real conceptual stretch for remembering or teaching it. As a strawman, we could use different syntax for "match the thing of class Spam that has attributes with these values: match eggs: case Spam[a=1, b=2]: ... Or: match eggs: case Spam{a=1, b=2}: ... Well, the square brackets COULD mean something different if PEP 637 is adopted. But even supposing the curly braces could be fit into the grammar. Yes, it sort of suggests the connection between dictionaries and Spam.__dict__. But it still reads as "this is something special that I have to think about a little differently." Even where there are capture variables, I think I'd be completely comfortable thinking about the different context for: match eggs: case Spam(a=x, b=2): ... -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/J4GVSQN4UPOCN3EXZASQOIWN7MO723A4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The semantics of pattern matching for Python
Hi Steve, Thank you very much for your comments here. This is certainly not the first time I feel that you not only have an excellent insight into a topic, but also manage to make your points very clearly and succinctly. Your car example highlights the background of the proposed syntax very nicely, indeed, and the for-in example strikes me as quite astute. Kind regards, Tobias Quoting Steven D'Aprano : On Fri, Nov 20, 2020 at 02:23:45PM +, Mark Shannon wrote: Why force pattern matching onto library code that was not designed for pattern matching? It seems risky. Can you give a concrete example of how this will "force" pattern matching onto library code? I don't think that anyone has suggested that we go around to third-party libraries and insert pattern matching in them, so I'm having trouble understanding your fear here. Fishing arbitrary attributes out of an object and assuming that the values returned by attribute lookup are equivalent to the internal structure breaks abstraction and data-hiding. Again, can we have a concrete example of what you fear? Python is not really big on data-hiding. It's pretty close to impossible to hide data in anything written in pure Python. An object's API may consist of methods only. Pulling arbitrary attributes out of that object may have all sorts of unintended side-effects. Well, sure, but merely calling print() on an object might have all sorts of unintended side-effects. I think that almost the only operation guaranteed to be provably side-effect free in Python is the `is` operator. So I'm not sure what you fear here? If I have a case like: match obj: case Spam(eggs=x): I presume any sensible implementation is going to short-cut the attempted pattern match for x if obj is *not* an instance of Spam. So it's not going to be attempting to pull out arbitrary attributes of arbitrary objects, but only specific attributes of Spam objects. To the degree that your objection here has any validity at all, surely it has been true since Python 1.5 or older that we can pull arbitrary attributes out of unknown objects? That's what duck-typing does, whether you guard it with a LBYL call to hasattr or an EAFP try...except block. if hasattr(obj, 'eggs'): result = obj.eggs + 1 Not only could obj.eggs have side-effects, but so could the call to hasattr. Can you explain how pattern matching is worse than what we already do? PEP 634 and the DLS paper assert that deconstruction, by accessing attributes of an object, is the opposite of construction. This assertion seems false in OOP. Okay. Does it matter? Clearly Spam(a=1, b=2) does not necessarily result in an instance with attributes a and b. But the pattern `Spam(a=1, b=2)` is intended to be equivalent to (roughly): if (instance(obj, Spam) and getattr(obj, a) == 1 and getattr(obj, b) == 2) it doesn't imply that obj was *literally* created by a call to the constructor `Spam(a=1, b=2)`, or even that this call would be possible. I think that it will certainly be true that for many objects, there is a very close (possibly even exact) correspondence between the constructor parameters and the instance attributes, i.e. deconstruction via attribute access is the opposite of construction. But for the exceptions, why does it matter that they are exceptions? Let me be concrete for the sake of those who may not be following these abstract arguments. Suppose I have a class: class Car: def __init__(self, brand, model): self.brand = brand self.model = model and an instance: obj = Car("Suzuki", "Swift") For this class, deconstruction by attribute access is exactly the opposite of construction, and I can match any Suzuki like this: match obj: case Car(brand="Suzuki", model) which is roughly equivalent to: if isinstance(obj, Car) and getattr(obj, "brand") == "Suzuki": model = getattr(obj, "model") It's not actually asserting that the instance *was* constructed with a call to `Car(brand="Suzuki", model="Swift")`, only that for the purposes of deconstruction it might as well have been. If the constructor changes, leaving the internal structure the same: class Car: def __init__(self, manufacturer, variant): self.brand = manufacturer self.model = variant the case statement need not change. Remember that non-underscore attributes are public in Python, so a change to the internal structure: class Car: def __init__(self, brand, model): self.brand_name = brand self.model_id = model is already a breaking change, whether we have pattern matching or not. When we added the "with" statement, there was no attempt to force existing code to support it. We made the standard library support it, and let the community add support as and when it suited them. We should do the same with pattern matching. That's a terrible analogy. Pattern matching is
[Python-Dev] Re: Questions about about the DLS 2020
Hi Brett, Thanks for your replies. _> But you can write `123 .bit_length()`. That's a parser limitation more than human understanding._ Touché. I took this ambiguity of the dot so much for granted that I would not have thought of trying that. _> Yep, but PEP 634 not only changes the convention, it special-cases "_" to be different in how it can be used anywhere else in Python. [...]_ I feel this is a bit of a misunderstanding, though: the underscore is really only special as a pattern. Nothing in the PEP is changing anything outside of pattern matching. I think it is quite important that such a complex feature is kind of separate from the rest of the language. _> Welcome to worrying about every little detail in language design. _ Cheers ^_^. To be fair, I learned a lot in this regard during the past months and I feel extremely lucky to have benefitted so much from working with these very smart and highly experienced team mates. At the same token, we researched and approached the topic from so many different angles that I feel quite comfortable with the result and think this turned out to be a solid proposal. _> [...] Now obviously "practicality beats purity" as well, but the argument "other languages do it this way" doesn't hold well for a language that doesn't use curly braces for scoping delineation. _ I completely get that I fully agree with the intent of it. We obviously have to concentrate much more on OCaml, Pascal, and Basic, which do not use curly braces for scoping delineation, either ;-) ;-P. Honestly, though, I guess there is a certain grey area here. I am thinking, for instance, of things like a for-loop, which indicates iteration in about every language known(*), although there is research indicating that the keyword ``for`` itself is a rather bad choice. At some point, it just becomes so much a standard that it transcendents into some kind of "meta-language". I would argue that in this case, the burden of proof kind of reverses: you then need a very good argument why /_not_/ to stick to a universal standard. And somehow, "Python only sticks superficially and by convention to the standard but we want to be able to break the rules even in this case whenever we fancy" strikes me as particularly convincing... :) _> My question about adding pattern matching later was more to comment on the fact that the languages that use "_" for a wildcard pattern did it from early on, not later on; it had nothing to do with the proposal proposing pattern matching late in Python's history._ Hmm, yes I understand. Still, it is probably hard to make a solid argument either way, because only few languages really added pattern matching later, with most of them being statically typed, functional, and so on. So, there is not really much to go for I guess. It is probably Python's burden now to embark on adventures to seek out new worlds, new possibilities and boldly go where no snake has gone before... :) Kind regards, Tobias (*) And yes, I am fully aware of assembly, purely functional languages, brainf*, etc. Quoting Brett Cannon : On Tue, Nov 17, 2020 at 1:16 PM Tobias Kohn wrote: _Hi Brett, Without having really looked at the history of all the languages we mention in the PEPs, I have a hunch that most of them had pattern matching from quite the beginning or an early stage on, indeed. That being said, I think the question itself does not really make much sense, though. Programming languages are rarely if ever just invented out of the blue these days, but evolve from other languages that have gone before them. So, sure C++ and Objective-C both had objects and classes from the very beginning on, but if we take into consideration that they strongly build on C, we could argue just as well that OOP was a later addition to C. It really depends on your point of view. But since we are talking about different languages here, there is one example I could bring up. In C#, the underscore is a perfectly legal variable name. And yet, it is also used as a 'discard' [1] in newer versions. Moreover, F#, which certainly uses the underscore as a wildcard, runs on the same platform as C# and thus has to deal with this, too. Somehow, Microsoft and its developers seem able to cope with it. If I may widen the topic here a bit and seize the opportunity to go beyond just answering your email: I must admit that I find this entire discussion about the wildcard pattern rather mind-boggling. We seem all to accept the reality that ``match`` and ``case`` are context-sensitive keywords (i.e. they are both keywords and legal names)_ _Yes, because that's new syntax that's unambiguously new even if you have been programming in Python for decades._ _ _ _and that yo
[Python-Dev] Re: The semantics of pattern matching for Python
Hi Daniel and Mark, Sorry for being slightly late to the party, but please let me add a few remarks of my own to the discussion here. 1. MUST HAVE PRECISELY DEFINED SEMANTICS Yes, there are some aspects that we left open intentionally. Most prominently the question of how often the pattern matching engine will check whether the subject is an instance of a particular class. Take the following trivial example:: match some_data: case Pair(12, 34): ... case Triple(12, 34, z): ... case Pair(12, y): ... case Pair(x, y): ... In a perfect world, the compiler discovers that it must check whether ``some_data`` is an instance of ``Pair`` exactly once and not three times. This, of course, plays right into Mark's second point on efficiency and seems obvious enough. Yet, as soon as we are considering nested patterns, it turns much less obvious whether the compiler is supposed to cache repeated isinstance-checks. Can we really expect that the compiler must discover that in both case clauses the first element is checked against the same class? Or would it make more sense to simply expect the compiler to potentially perform this ``Num`` instance check twice:: match some_data: case [ Num(), 12 ]: ... case [ Num(), y, *z ]: ... It is easy to think of cases where we accidentally end up calling an isinstance check more than once because the compiler could not prove that they are equal. Still, whenever possible we want to give the compiler the freedom to optimise the pattern matching statement by caching. In a static language, all of this would not be an issue at all, of course. In Python, however, we end up being caught between its dynamic features and the desire to make pattern matching reasonably efficient. So, we ended up leaving the question open as how often the pattern matching engine is allowed or supposed to check instances. Naturally, if you go and write some isinstance-check on a class with side-effects, you can break it. 2. USERS SHOULD NOT HAVE TO PAY AN UNNECESSARY PERFORMANCE PENALTY TO USE PATTERN MATCHING To quote Mark [1] here: /> Users should not have to pay an unnecessary performance penalty to use pattern matching./ Alright, what does this even mean? What is an unnecessary performance penalty? How should that be measured or compared? Pattern matching is not just fancy syntax for an if-elif-statement, but a new way of writing and expressing structure. There is currently nothing in Python that fully compares to pattern matching (which is obviously why we propose to add in the first place). So, do you want to compare a pattern matching structure to an if-elif-chain or rather an implementation using reflection and/or the visitor pattern? When implementing pattern matching, would we be allowed to trade off a little speed handling the first pattern for moving faster to patterns further down? Do our PEPs really read to you like we went out of our ways to make it slow or inefficient? Sure, we said let's start with an implementation that is correct and worry about optimising it later. But I thought this is 101 of software engineering, anyway, and am thus rather surprised to find this item on the list. 3. FAILED MATCHES SHOULD NOT POLLUTE THE ENCLOSING NAMESPACE This is a slightly wider issue that has obviously sparked an entire discussion on this mailing list on scopes. If there is a good solution that only assigns variables once the entire pattern matched, I would be very happy with that. However, I think that variables should be assigned in full before evaluating any guards---even at the risk of the guard failing and variables being assigned that are not used later on. Anything else would obviously introduce a mini-scope and lead to shadowing, which hardly improves anything with respect to legibility. 4. OBJECTS SHOULD BE ABLE DETERMINE WHICH PATTERNS THEY MATCH Short version: no! Class patterns are an extension of instance checks. Leaving out the meta-classes at this point, it is basically the class that is responsible for determining if an object is an instance of it. Pattern matching follows the same logic, whereas Mark suggests to put that upside-down. Since you certainly do not want to define the machinery in each instance, you end up delegating the entire thing to the class, anyway. I find this suggestion also somewhat strange in light of the history of our PEPs. We started with a more complex protocol that would allow for customised patterns, which was then ditched because it was felt as being too complicated. There is still a possibility to add it later on, of course. But here we are with Mark proposing to introduce a complex protocol again. It would obviously also mean that we could not rely as much on Python's existing
[Python-Dev] Re: Questions about about the DLS 2020
Hi Brett, Without having really looked at the history of all the languages we mention in the PEPs, I have a hunch that most of them had pattern matching from quite the beginning or an early stage on, indeed. That being said, I think the question itself does not really make much sense, though. Programming languages are rarely if ever just invented out of the blue these days, but evolve from other languages that have gone before them. So, sure C++ and Objective-C both had objects and classes from the very beginning on, but if we take into consideration that they strongly build on C, we could argue just as well that OOP was a later addition to C. It really depends on your point of view. But since we are talking about different languages here, there is one example I could bring up. In C#, the underscore is a perfectly legal variable name. And yet, it is also used as a 'discard' [1] in newer versions. Moreover, F#, which certainly uses the underscore as a wildcard, runs on the same platform as C# and thus has to deal with this, too. Somehow, Microsoft and its developers seem able to cope with it. If I may widen the topic here a bit and seize the opportunity to go beyond just answering your email: I must admit that I find this entire discussion about the wildcard pattern rather mind-boggling. We seem all to accept the reality that ``match`` and ``case`` are context-sensitive keywords (i.e. they are both keywords and legal names) and that you cannot write ``123.bit_length()`` because the dot has several different meanings. But when it comes to the underscore and the idea that *as a pattern* it might simply not store a value, it is considered an insurmountable obstacle to learning and understanding pattern matching? Particularly because, after all, the underscore is in Python only a wildcard *by convention*? I think pattern matching has a lot in common with 'recursion': if you see it for the first time, it can be very hard to wrap you head around it and understand what is going on or why anyone would want to do such a thing. But once you understood it, it can be an extremely elegant solution to many problems. So, I really can fully understand a lot of the reservations brought forward about this feature and various aspects of it. And there were enough good and valid points brought up by the community, which we then integrated into and respected in our overhauled design---but that the wildcard pattern causes so much grief really is entirely beyond me... So, yes, most languages had pattern matching quite from the outset on (or not at all). But then, again, Python is about the only language I know to have introduced 'type' annotations so late in life rather than from its very beginning. It came at the cost of using the colon for something different than compound statements or dictionaries. If we consider how much annotations have been expanded in more recent versions of Python and how many tools make good use of it, I would call that a pretty successful move, though---even though all other languages might have had type annotations from their very birth. Kind regards, Tobias P.S. Sorry, I guess this turned out to be not so much a reply to your comment alone, as much more a reply to many a message and discussion that has been posted here over time. [1] https://docs.microsoft.com/en-us/dotnet/csharp/discards Quoting Brett Cannon : On Mon, Nov 16, 2020 at 9:03 AM Tobias Kohn wrote: _Hi Mark, Thank you for your interest and the questions. 1. This really comes down to how you look at it, or how you define pattern matching. The issue here is that the concept of pattern matching has grown into a large and somewhat diverse flock of interpretations and implementations (as a side note: interestingly enough, some of the only universally agreed-upon standards are to use `_` as a wildcard and not to mark names that capture/bind values---which are quite exactly the points most fiercely debatted here)._ _How many of those languages added pattern matching /later/ and not at the earliest stages of the language (if not from the beginning)? And for those that added it later, how many of those didn't already have a convention surrounding "_"? My suspicion is "not many" and "not many". _ _ _ _-Brett_ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KP2TVEAPDT553VXL4QEUOTYBHHMDUXSK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Questions about about the DLS 2020
Hi Mark, Thank you for your interest and the questions. 1. This really comes down to how you look at it, or how you define pattern matching. The issue here is that the concept of pattern matching has grown into a large and somewhat diverse flock of interpretations and implementations (as a side note: interestingly enough, some of the only universally agreed-upon standards are to use `_` as a wildcard and not to mark names that capture/bind values---which are quite exactly the points most fiercely debatted here). Anyway, the paper presents the pattern matching structure we are proposing as one of three major variants of pattern matching: (a) Matching arguments to parameters in a function call, (b) Matching elements to elements in iterable unpacking, (c) Matching tree-like data to general patterns in a conditional pattern matching structure. The last one is the subject of the PEP and the paper. Nonetheless, in the first two cases (a) and (b), we find that indeed the computer will validate that the data matched the pattern and raise an exception if this fails. This is where this way of looking at it comes from. 2. Yes, that is indeed a deliberate simplification. The idea is to abstract away from the details of how exactly Python implements abstract syntax trees (which I honestly believe are irrelevant for the sake of the entire narrative). Moreover, using strings here allows us to exemplify the literal patterns, rather only showcasing only the constructor/class pattern. Essentially, this is a question of making the most out of the little space available. Since you have addressed this email to me directly, I would like to take this opportunity and briefly stress that this paper really grew out of a team effort. While I might have been the one pushing for an academic publication, the DLS'20 paper represents the input and ideas of all the authors, as well as the long discussions we had. Of course, I am happy to answer any questions about the paper, but it would be wrong to see me as the one person behind it. Cheers, Tobias Quoting Mark Shannon : Hi Tobias, A couple of questions about the DLS 2020 paper. 1. Why do you use the term "validate" rather than "test" for the process of selecting a match? It seems to me, that this is a test, not a validation, as no exception is raised if a case doesn't match. 2. Is the error in the ast matching example, an intentional "simplification" or just an oversight? The example: ``` def simplify(node): match node: case BinOp(Num(left), '+', Num(right)): return Num(left + right) case BinOp(left, '+' | '-', Num(0)): return simplify(left) case UnaryOp('-', UnaryOp('-', item)): return simplify(item) case _: return node ``` is wrong. The correct version is ``` def simplify(node): match node: case BinOp(Num(left), Add(), Num(right)): return Num(left + right) case BinOp(left, Add() | Sub(), Num(0)): return simplify(left) case UnaryOp(USub(), UnaryOp(USub(), item)): return simplify(item) case _: return node ``` Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ETZGYRCF4DR6RJXTHGXIRZXINXJ76J2D/Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CUNO57W7KSIM2WRROC5R43ZT7HUQZCZ6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Pattern Matching controversy: Don't read PEP 635, read DLS'20 paper instead
Hi Paul, Thank you for your comments on the DLS'20 paper. I am glad to hear that it helps paint a clear(er) picture of pattern matching in Python. However, please let me set the record straight in a few regards. First, there is no 'shoehorning' or 'deception' in our pattern matching PEPs. This notion of our team standing against the Python community or trying to outsmart it is quite absurd and unnecessary, really. At the end of the day, you might find that the PEPs are a boiled-down version of months of intense work and discussions (as Brandt has already pointed out elsewhere) and thus necessarily terse in some parts. Pattern matching is a complex subject and the PEPs have been written with a sincere effort to convey the big picture and give a tour of this feature in an accessible manner. Our entire discussion is all openly available to anyone interested. Second, our work on pattern matching is based on a very simple premise/idea: *bring pattern matching to Python*. In particular, it is *not* to 'bring Python to pattern matching'. We want to introduce pattern matching as a new feature to Python as Python stands right now, without the need to first modify Python and make it more static, C-like, functional or whatever. As I have mentioned before, pattern matching is an opt-in feature, something with absolutely no influence on the behaviour of any Python program you have ever written before. Not introducing constants or a completely new kind of scopes to Python first is certainly not a lack of academic rigour, oversight on our part because we are not smart enough, or trick to coerce people into accepting pattern matching. It just means that we propose to introduce pattern matching to Python as it is right now. Third, the academic paper has a different scope than the PEPs. In case you missed it: after the first version of PEP 622, we listened to the reactions and suggestions from the community and went back to overhaul the entire design and build as much as possible on the raised concerns and cummulative experience of the community. One major outcome of this was to reduce the overall proposal to the absolute core features of what we need for pattern matching (another was to put more effort into writing an accessible account of what we propose and why we think the various design features are a good choice, leading to three PEPs with different aims). The academic paper outlines a much larger scope of possibilities than the PEPs, whereas the PEPs are more grounded in the pragmatic and taking one step at a time. In short, the academic paper lays out an entire journey, a vision, whereas the PEPs propose a first step forward. Finally, the reason why the academic paper has not been referred to before is simply that it will officially be published coming (!) week. Since the process of peer review is now complete, we can already provide a preprint. It was clear from the outset that the PEPs will contain a link to the paper as soon as it becomes publicly available. Of course, we encourage anyone interested in it to read the academic paper. Since its focus is somewhat complementary to the PEPs, it might highlight some ideas behind our design that might not be made clear enough in the PEPs. But while doing so, please keep in mind that the paper is a vision of a bigger picture and as such distinct from the PEPs! Kind regards, Tobias Quoting Paul Sokolovsky : Hello, As was mentioned many times on the list, PEP634-PEP636 are thoroughly prepared and good materials, many thanks to their authors. PEP635 "Motivation and Rationale" (https://www.python.org/dev/peps/pep-0635/) stands out among the 3 however: while reading it, chances that you'll get a feeling of "residue", accumulating a section over section. By the end of reading, you may get a well-formed feeling that you've read a half-technical, half-marketing material, which is intended to "sell" a particular idea among many other very viable ideas, by shoehorning some concepts, downplaying other ideas, and at the same time, light-heartedly presenting drawbacks of its subject one. Just to give one example, literally at the very beginning, at the "Pattern Matching and OO" section (3rd heading) it says: Pattern matching is complimentary to the object-oriented paradigm. It's not until the very end of document, in the "History and Context" it tells the whole truth: With its emphasis on abstraction and encapsulation, object-oriented programming posed a serious challenge to pattern matching. You may wonder how "complimentary" and "posed a serious challenge" relate to each other. While they're definitely not contradictory, starting the document with light-hearted "complimentary" can be seen as trying to set the stage where readers don't pay enough attention to the problem. And it kinda worked: only now [1] wider community discovers the
[Python-Dev] Re: My thoughts on Pattern Matching.
Hi Thomas, Thank you very much for your carefully worded and thoughtful email. I feel, however, that many of your concerns are based on an idealised picture of a future Python language that will never actually materialise. As I understand it your main point is that the concept of patterns might---or even should---be retro-fitted to general assignments. Just as we have borrowed from and expanded on the idea of iterable unpacking in assignments, so should assignments then pick up the concepts introduced in pattern matching. Indeed, assignments of the form ``Viking(spam, ham) = ...`` are not only appealing but can be found in various languages with pattern matching. So, why would we not want to have consistent concepts and symmetric (orthogonal) structures across all of Python? Unfortunately, such consistency or symmetry comes at a high cost---too high as far as I am concerned. One of the simplest patterns is without doubt the literal pattern that essential only matches itself (e.g., ``case 123:`` or ``case 'abc':``). Any future unification of patterns and assignments would therefore necessarily allow for statement such as:: 1 = x This is essentially equivalent to ``assert(x == 1)``. Indeed, _Elixir_ [1] uses such syntax for exactly this semantics. However, knowing how much novice programmers struggle with understanding that assignments are right-to-left (i.e. ``x = 1`` and not ``1 = x``), including such syntax would immediately raise the learning curve significantly. In short, a very common students' mistake will take them to error messages and concepts they could not possibly understand without a basic comprehension of pattern matching. And this is probably where it becomes most obvious how our views of pattern matching differ. The pattern matching as we propose it in PEPs 634/635/636 is guarded by a keyword needed to activate these features. Unless you start your statement with ``match my_data:``, you can easily pretend as if pattern matching did not exist: it will not, cannot affect your code. This encapsulation is intentional and by design. As far as I am aware, those languages that support syntax like ``Viking(spam, ham) = ...`` only allow this in combination with variable _declaration_, i.e. you actually have to write ``let Viking(spam, ham) = `` or ``var Viking(spam, ham) = ...`` or similar. Without such a marker, this syntax quickly descends into unreadable gibberish. As noted in the original section of rejected ideas in PEP 622, we had originally considered adding 'one-off pattern matching': pattern matching with only a single case that must succeed, very much like normal assignments do. But our approach was always guarded by a keyword, be that ``case`` or ``if``---in line with the ``var`` or ``let`` found in other languages. Even in that case, patterns would not leak into the language outside pattern matching. Finally, there is already a necessary inconsistency between iterable unpacking and pattern matching. By their very nature, patterns express a _possible_ structure, whereas iterable unpacking imposes a _necessary_ structure. So, when dealing with iterators, it is safe to 'unpack' the iterator in iterable unpacking. If the expected and actual structures differ, it is an error, anyway. In pattern matching, however, we have to be more conservative and careful, exploring options rather than certanties. Hence, even if all other concerns were wiped away, the closest we could come to an entirely symmetric and consistent language is one with some subtle differences and thus prone for bugs and errors. PEPs 634/635/636 are the result of a long and careful design process where we sought to appeal to the intuition of the Python programmer as much as possible, without betraying the differences and new concepts that pattern matching necessarily introduces. Readability was always one of our main concerns and we believe that having a clear context where patterns occur greatly helps writing readable and consistent code. So, long story short, I am afraid I would question the very premise upon which your concerns are founded: that it would ever be a good idea to expand patterns to assignments in general. Although such a unification was in principle possible, it would rob Python of one of its greatest and strongest assets: its simplicity and learnability. Kind regards, Tobias [1] https://elixir-lang.org/getting-started/pattern-matching.html ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KOYU2FAY3DZ4EHYWR3FXBQMTDP72EEQQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 642: Constraint Pattern Syntax for Structural Pattern Matching
Hi Nick and Everyone, We had actually considered a similar idea (i.e. load sigils) during the design phase of pattern matching. In the interest of having a rule that is as simple as possible, we had proposed to use a leading dot as a universal marker. Tin's example would thus have been written as:: match r: case (src, None): ... case (.c, msg): ... case (.s, msg): ... However, this idea was met with some resistance. After briefly looking at various alternatives again, we eventually decided to defer this discussion entirely, allowing for the community to perhaps gain some experience with the basic pattern matching infrastructure and have a more in-depth discussion later on. Paul also wrote [1]: Nice to hear that there're (high-hierarchy) people who want to do 2nd round on intent-explicitizing sigils, thanks. While we from the PEP-622/634/635/636 team are quite adamant that stores should *not* be marked, having a second round of discussion about load sigils is quite exactly what we aimed for! However, we should consider this to be a discussion about an *extension* of the existing PEPs (634-636), rather than about modifying them: *The introduction of a load sigil (be it the dot or a question mark or anything else) can actually be discussed quite independently of the rest of pattern matching.* You might have noticed that the original PEP 622 contained a lot more than the current PEPs 634-636. This is intentional: with the current pattern matching PEPs, we boiled down the entire concept to the basic infrastructure that we need in order to get it going; a basic "starter kit" if you will. There are a lot of ideas around for extending this basic pattern matching and make it much more powerful and versatile, including load sigils as proposed by PEP 642. But let us perhaps just start with pattern matching---hopefully in 3.10 :)---and then gradually build on that. Otherwise, I am afraid we will just keep running in circles and never get it to lift off. Cheers, Tobias [[1]] https://mail.python.org/archives/list/python-dev@python.org/message/QPYBAPOSMXK7XDETO5XB5GNFITI6JPTN/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FYGTU4RMZGKTQUANDSTRUSTC5GTZ5WUY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)
Hi Larry, You are right that just dismissing intuition is wrong. I should have been more careful with my wording or explain them better, and I would like to apologise if my response came across as too strong in this regard. The actual problem that I see is that we have different cultures/intuitions fundamentally clashing here. In particular, so many programmers welcome pattern matching as an "extended switch statement" and find it therefore strange that names are binding and not expressions for comparison. Others argue that it is at odds with current assignment statements, say, and question why dotted names are _/not/_ binding. What all groups seem to have in common, though, is that they refer to _/their/_ understanding and interpretation of the new match statement as 'consistent' or 'intuitive'---naturally pointing out where we as PEP authors went wrong with our design. But here is the catch: at least in the Python world, pattern matching as proposed by this PEP is an unprecedented and new way of approaching a common problem. It is not simply an extension of something already there. Even worse: while designing the PEP we found that no matter from which angle you approach it, you will run into issues of seeming 'inconsistencies' (which is to say that pattern matching cannot be reduced to a 'linear' extension of existing features in a meaningful way): there is always something that goes fundamentally beyond what is already there in Python. That's why I argue that arguments based on what is 'intuitive' or 'consistent' just do not make sense _/in this case/_. I think the discussion on this mailing list with the often contradictory views, proposals, and counter-proposals more than makes my point. As for your argument that it looks like calling a function or creating an object: I tried to explain a little while ago that you'd be well advised to rather approach it as something similar to a function _/definition/_. After all, the part after `def` in `def foo(a, b):` also looks like a function call! But nobody seems to mind this similarity in syntax there! And the target in `(a, b) = c` looks like a tuple constructor, although it actually is the exact opposite. Finally, I completely agree that intuition is informed by experience and serving us very well. The first part of this, however, is also to say that intuition is malleable thing! And experience from other programming languages who took the leap to having pattern matching shows that it quickly becomes a quite intuitive and easy to use feature. Cheers, Tobias P.S. Please excuse my late reply; I am currently on vacation. Quoting Larry Hastings : On 7/31/20 12:36 AM, Tobias Kohn wrote: And since pattern matching is really a new feature to be introduced to Python, a feature that can be seen in different lights, there is no 'Python-Programmer intuition' that would apply in this case. It's not fair to say "intuition doesn't apply because it's new syntax". There are plenty of examples of intuition serving a Python programmer well when encountering new syntax. A Python programmer's intuition is informed by existing syntax and conventions in the language. When they see a new construct, its similarity to existing constructs can make understanding the new syntax quite intuitive indeed. Take for example list comprehensions. Python 1 programmers hadn't seen a = [x for x in y] But they knew what square brackets meant in that context, it meant "creates a new list". And they knew what "for x in y" meant, that meant iteration. Understanding those separate two concepts, a Python 1 programmer would be well on their way to guessing what the new syntax meant--and they'd likely be right. And once they understood list comprehensions, the first time they saw generator expressions and set and dict comprehensions they'd surely intuit what those did immediately. The non-intuitiveness of PEP 622, as I see it, is that it repurposes what looks like existing Python syntax but frequently has wholly different semantics. For example, a "class pattern" looks like it's calling a function--perhaps instantiating an object?--but the actual semantics and behavior is very different. Similarly, a "mapping pattern" looks like it's instantiating a dict, but it does something very different, and has unfamiliar and seemingly arbitrary rules about what is permitted, e.g. you can't use full expressions or undotted-identifiers when defining a key. Add the "capture pattern" to both of these, and a Python programmer's intuition about what this syntax traditionally does will be of little help when encountering a PEP 622 match statement for the first time. Cheers, //arry/ ___ Python-Dev mailing list
[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)
Hi Caleb, I will only answer to the second part, as the wildcard issue has been brought up and discussed time and again, and the `np` analogue is quite a stretch and far-fetched, really. One thing that stood out a bit to me as I feel to have seen it a couple of times is the question of intuition, so I will add a few more general thoughts to that... > [...] but it seems quite unintuitive to me [...] [...] don't necessarily make it intuitively clear [...] Intuition (or lack thereof) has already been brought forward as an argument a couple of times. I would just like to briefly point out that there is no such thing as universal intuition in the field of programming. We all have different training, skills, preferences and experiences, which make up what we call 'intuition'. But what is intuitive is usually something completely different to C-programmer than to a Haskell- or Lisp-Programmer, say. And since pattern matching is really a new feature to be introduced to Python, a feature that can be seen in different lights, there is no 'Python-Programmer intuition' that would apply in this case. As for beginners, virtually every part of programming is unintuitive at first. Even something innocuous-looking like assignment is often reason for confusion because `3 + 4 = x` would probably be more 'intuitive'. But there is good reason with regards to the bigger picture to stick to `x = 3 + 4`. A Python-programmer (at any level) not familiar with pattern matching will most likely not understand all subtlety of the syntax---but this is alos true of features like `async` or the `/` in parameters, say. I would argue, though, that the clear choice of keywords allow anyone to quickly look pattern matching up and get informed on what it does. So, we do not need to come with something that is entirely 'intuitive' and 'self-evident'. But by sticking to common convention like `_` as wildcard, we can help quickly build the correct intuition. In your examples, for instance, it is perfectly obvious to me that you cannot directly assign to attributes and it would in fact look very weird to my eyes if you could. Your use case is quite similar to initialisers and you are arguing that you would like being able to write: ``` CLASS Point: DEF __init__(self, self.x, self.y): PASS ``` rather than the more verbose: ``` CLASS Point: DEF __init__(self, x, y): self.x, self.y = x, y ``` I do not think that this would be a good idea for either parameters or patterns. After all, pattern matching is */not/* assignment, even though it is related to it, of course. Kind regards, Tobias Quoting Caleb Donovick : Adding this feature would be a giant quality of life improvement for me and I really hope it succeeds. So I have been trying to keep up on the debate in this and related thread. While I do want this feature, I agree with a lot of the issues people are raising. First I agree that _ should not be the wildcard symbol. Or rather the hobgoblins in my mind think that if _ is to be the wildcard symbol it would be more consistent with assignment if it was bound to the last value it was matched with (as should other repeated identifiers) e.g., match pt:case (x, _, _):assert _ == pt[2] I understand the authors rationalize the decision based on conventions with the gettext module. I find these arguments very unconvincing. It's like saying the identifier np should be forbidden from appearing in cases because it's frequently used as the name of numpy. If there is to be a wildcard symbol (that does not bind and is repeatable) it should not be a valid identifier. Second, the distinction between a capture and constant value pattern should be more explicit. I don't have any great insight into the color of the shed beyond the numerous suggestions already made (name=, ?name, etc...), but it seems quite unintuitive to me that I can't capture into a namespace nor match a constant without a namespace. It is also unclear to me why it would be so terrible to add a new token or abuse an existing one. > Honestly, it doesn't help the case for `?` that it's been proposed as a mark for both capture patterns and value patterns (by different people, obviously :-). I agree that most of the proposed sheds don't necessarily make it intuitively clear what is a capture variable vs what is a constant. However they do give the programmer the ability to choose. For example if I want to modify the motivating example from the PEP slightly to copy attributes from one point to another I can't express it concisely: def update_point_3d(p: Point3d, pt):match pt: case (x, y): p.x, p.y = x, y case Point2d(x, y): p.x, p.y = x, y ... (Okay I could have just called the original make_point_3d and unpacked the results but it would require the creation of an
[Python-Dev] Re: PEP 622: Structural Pattern Matching -- followup
Hi Rob, You are right: the grammar should probably read `suite` rather than `block` (i.e. the `pass` is necessary). Thanks for catching this! As for the second question, I assume there might be a slight oversight on your part. The last line in the example replaces the string `"_"` rather than the variable `_`. The not-binding of `_` thus has no influence on the last line. I think I will leave it for Mark himself to name the two bugs rather than start a guessing game. However, in an earlier version we had left out the `if value` for the first case, accidentally translating the `len(value) > 1` as a `len(value) >= 1` instead. Kind regards, Tobias Quoting Rob Cliffe via Python-Dev : [...] _First question_: Sometimes no action is needed after a case clause. If the Django example had been written if ( isinstance(value, (list, tuple)) and len(value) > 1 and isinstance(value[-1], (Promise, str)) ): *value, label = value else: label = key.replace('_', ' ').title() the replacement code would/could be match value: case [*value, label := (Promise() | str())] if value: pass case _: label = key.replace('_', ' ').title() AFAICS the PEP does not *explicitly* state that the 'pass' line is necessary (is it?), i.e. that the block following `case` cannot (or can?) be empty. The term `block` is not defined in the PEP, or in https://docs.python.org/3/reference/grammar.html. But an empty block following a line ending in `:` would AFAIK be unprecedented in Python. I think it is worth clarifiying this. _Second question_: in the above example replacement, if `case _:` does not bind to `_`, does that mean that the following line will not work? Is this one of the "two bugs" that Mark Shannon alluded to? (I have read every message in the threads and I don't remember them being spelt out.) And I'm curious what the other one is (is it binding to a variable `v`?). Best wishes Rob Cliffe ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4MNGTBKIXNMMVAIFOLR2W62SLK637OY5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Another take on PEP 622
Hi Terry, Thank you: I really like your wave/particules analogy. I think that pattern matching is indeed uniting different concepts to build a stronger, more versatile structure. I also like your concept of a general "binding structure" with different forms, such as assignment, parameter passing, and (according to PEP 622) patterns. I feel that this categorisation puts pattern matching quite in the correct place. Of course, there is also the second aspect of "bind these variables---/if you can/" or "analyse and compare the structure", which is the other part of this particule/wave duality. Concerning the guards (optional conditions), I also think that you summarised very nearly. I tend to think of the patterns a bit like the grammar of a programming language. Something that is supposed to be static and declarative (as far as this is possible in Python). Yet, some constraints in programming languages cannot be expressed by a (context-free) grammar in a meaningful way. For instance, the grammar itself does not keep you from having two parameters in a function sharing the same name. This is a dynamic aspect, a relationship between two otherwise independent parts of the overall structure. And that is best caught by the guards. So, yes: as I see it, the guards really add something that goes beyond the declarative side of patterns. Hence, I completely agree with your characterisation. Kind regards, Tobias Quoting Terry Reedy : On 7/16/2020 9:51 PM, Tobias Kohn wrote: Hi Everyone, I feel there are still quite a few misconceptions around concerning PEP 622 and the new pattern matching feature it proposes. Please allow me therefore to take another attempt at explaining the ideas behind PEP 622 with a different approach. Bear in mind that I naturally cannot cover everything, though, and that some parts presented here are still slightly simplified. Thank you Tobias. I am a fan of looking at things from multiple viewpoint. For 200 years, physicists argued about whether light is waves or particles, before discovering that 'neither' and 'both' were more correct. 1. Function Overloading 2. Visitor Pattern and Dispatch > 3. Shape and Structure [snip, snip, snip] In an assignment statement, the code to the left of '=' is a 'target list' of 'target's, with some idiosyncratic rules. Even though it might, misleadingly, be called a 'target expression', it is not an expression to be evaluated. Similarly, the code between parentheses in a def statement is a 'parameter list' of 'defparameter' and special symbols, with other idiosyncratic rules. Both could be called 'binding lists' or more generally, 'binding structures'. To me, the important point of your point is that 'case' is somewhat analogous to 'def', and that the stuff between 'case' and ':' is a binding structure. We should call this structure a 'match structure'. It is misleading and confusing to call it a 'match expression'. A match structure consists of a 'match list' of 'match items' or 'matcher's and an optional "'if' ". Matchers have a 3rd, new, and larger set of idiosyncratic rules. The optional condition is an escape hatch for when expressing the intended match constraint and corresponding match set is difficult or worse using the match rules. As with target and parameter items, untagged simple name matchers are (and I now see, should be) binding targets. (The parameter list tags are ':' for types and '=' for default values.) Unlike assignment targets, dotted names and subscriptings are not binding targets. Like parameter lists, match lists include literals. Match lists also include syntactic structure not seen in the other binding structures. -- Terry Jan Reedy ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NNZ67OMAV2CDV7GSX64SOLUAERJSF5HP/Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6XZXGOHDBRN5ZXREZWB74DYHUGIEPLYI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 aspects
Hi Koos, Let me try and address some of the concerns and questions you are rising. I am replying here to two emails of yours so as to keep traffic down. Quoting Koos Zevenhoven : > (1) Class pattern that does isinstance and nothing else. If I understand the proposed semantics correctly, `Class()` is equivalent to checking `isinstance(obj, Class)`, also when `__match_args__` is not present. However, if a future match protocol is allowed to override this behavior to mean something else, for example `Class() == obj`, then the plain isinstance checks won't work anymore! I do find `Class() == obj` to be a more intuitive and consistent meaning for `Class()` than plain `isinstance` is. Instead, the plain isinstance check would seem to be well described by a pattern like `Class(...)`. This would allow isinstance checks for any class, and there is even a workaround if you really want to refer to the Ellipsis object. This is also related to the following point. (2) The meaning of e.g. `Class(x=1, y=_)` versus `Class(x=1)` In the proposed semantics, cases like this are equivalent. I can see why that is desirable in many cases, although Class(x=1, ...)` would make it more clear. A possible improvement might be to add an optional element to `__match_args__` that separates optional arguments from required ones (although "optional" is not the same as "don't care"). Please let me answer these two questions in reverse order, as I think it makes more sense to tackle the second one first. **2. ATTRIBUTES** There actually is an important difference between `Class(x=1, y=_)` and `Class(x=1)` and it won't do to just write `Class(x=1,...)` instead. The form `Class(x=1, y=_)` ensures that the object has an attribute `y`. In a way, this is where the "duck typing" is coming in. The class of an object and its actual shape (i.e. the set of attributes it has) are rather loosely coupled in Python: there is usually nothing in the class itself that specifies what attributes an object has (other than the good sense to add these attributes in `__init__`). Conceptually, it therefore makes sense to not only support `isinstance` but also `hasattr`/`getattr` as a means to specify the shape/structure of an object. Let me give a very simple example from Python's `AST` module. We know that compound statements have a field `body` (for the suite) and possibly even a field `orelse` (for the `else` part). But there is no common superclass for compound statements. Hence, although it is shared by several objects, you cannot detect this structure through `isinstance` alone. By allowing you to explicitly specify attributes in patterns, you can still use pattern matching notwithstanding: ``` MATCH node: CASE ast.stmt(body=suite, orelse=else_suite) if else_suite: # a statement with a non-empty else-part ... CASE ast.stmt(body=suite): # a compound statement without else-part ... CASE ast.stmt(): # a simple statement ... ``` The very basic form of class patterns could be described as `C(a_1=P_1, a_2=P_2, ...)`, where `C` is a class to be checked through `isinstance`, and the `a_i` are attribute names to be extracted by means of `getattr` to then be matched against the subpatterns `P_i`. In short: you specify the structure not only by class, but also by its actual structure in form of required attributes. Particularly for very simple objects, it becomes annoying to specify the attribute names each time. Take, for instance, the `Num`-expression from the AST. It has just a single field `n` to hold the actual number. But the AST objects also contain an attribute `_fields = ('n',)` that not only lists the *relevant* attributes, but also specifies an order. It thus makes sense to introduce a convention that in `Num(x)` without argument name, the `x` corresponds to the first field `n`. Likewise, you write `UnarOp('+', item)` without the attribute names because `_fields=('op', 'operand')` already tells you what attributes are meant. That is essentially the principle we adopted through introduction of `__match_args__`. **1. MATCH PROTOCOL** I am not entirely sure what you mean by `C() == obj`. In most cases you could not actually create an instance of `C` without some meaningful arguments for the constructor. The idea of the match-protocol is very similar to how you can already override the behaviour of `isinstance`. It is not meant to completely change the semantics of what is already there, but to allow you to customise it (in some exciting ways ^_^). Of course, as with everything customisable, you could go off and do something funny with it, but if it then breaks, that's quite on you. On the caveat that this is **NOT PART OF THIS PEP (!)**, let me try and explain why we would consider a match protocol in the first place. The
[Python-Dev] Another take on PEP 622
Hi Everyone, I feel there are still quite a few misconceptions around concerning PEP 622 and the new pattern matching feature it proposes. Please allow me therefore to take another attempt at explaining the ideas behind PEP 622 with a different approach. Bear in mind that I naturally cannot cover everything, though, and that some parts presented here are still slightly simplified. Let's start with perhaps the most crucial part: PEP 622 does **NOT** propose to introduce a `switch`-statement! Indeed, pattern matching is much more closely related to concepts like regular expressions, sequence unpacking, visitor patterns, or function overloading. In particular, the patterns themselves share more similarities with formal parameters than with expressions. So, to start with I would like to invite you to think of PEP 622 not so much as proposing a new control structure in the sense of `if`-`elif`-`else`, `try`-`except`-`finally`, or even `switch`, but much rather as addressing the question: what would function overloading look like in Python? Of course, this does not fully capture pattern matching or do it justice, either, but it might offer a better basis to start with. 1. Function Overloading --- In a statically typed language, you might define a function `product` that either accepts two numbers or an iterable something like the following (I am sticking to Python syntax for simplicity): ``` def product(a: float, b: float) -> float: return a * b def product(it: Iterable[float]) -> float: result = 1.0 for x in it: result *= x return result ``` In Python, however, this needs to be done differently and the dispatch logic has to be put inside the function itself: ``` def product(*args): if len(args) == 2 and isinstance(args[0], float) and isinstance(args[1], float): return args[0] * args[1] elif len(args) == 1 and isinstance(args[0], Iterable): result = 1.0 for x in args[0]: result *= x return result ``` It is this use case to begin with that pattern matching is addressing by introducing a more declarative way. Each `case` represents one possibility of what the parameter list might look like: ``` def product(*args): match args: case (float(a), float(b)): return a * b case (Iterable(it),): result = 1.0 for x in it: result *= x return result ``` And if you squint a little, you might even see that these parameter lists could almost be written in C: `(float a, float b)`. In the context of more functional languages, you might also have seen some wilder stuff, where function overloading allows you to include literals. One of the most prevalent examples is the factorial function, usually defined a bit like this: ``` def fact(0): return 1 def fact(n): return n * fact(n-1) ``` Again, when doing the same thing in Python, we put the dispatch logic inside the function: ``` def fact(n): if n == 0: return 1 else: return n * fact(n - 1) ``` It is only natural to also allow pattern matching to express this use case: ``` def fact(arg): match arg: case 0: return 1 case n: return n * fact(n - 1) ``` And this here is where you probably start to see good old `switch` coming in. Indeed, pattern matching is powerful and versatile enough to act as a `switch` replacement in many cases. But given how we arrived here, you might start to understand that this is a happy accident and not by design of the structure itself. There is one big elephant in the room, of course: why do we need the `match`-line in the first place? If all we want is to somehow express function overloading, couldn't we just put the cases directly into the function body? In principle, yes, we could do that. But! As it turns out, there is something to be gained by a clear separation. When actually using pattern matching, you might discover that you do not always want to define a function. Sometimes, it is quite convenient being able to have a pattern like this, for instance: ``` def foo(*args): for arg in args: match arg: case ... ``` In all of these examples, it looks as if pattern matching is replacing an `if`-`elif`-`else` chain. However, this is not entirely accurate. What we really want to express is the function overloading in the first place. The `if`s are only needed to express the same idea in Python for the time being. In other words: the individual cases here express independent implementations and we leave it up to the Python interpreter to choose the right one. 2. Visitor Pattern and Dispatch --- If you wanted to implement something like function overloading based on the type of the argument, you might use the _visitor pattern_ for that purpose. Thanks to
[Python-Dev] Re: PEP 622 constant value syntax idea
Hi Mohammad, In addition to what Rhodri James has already well pointed out, here are two additional thoughts on this. At the moment, the keyword `global` is a marker to say that the respective variable is /modified/ by a function. Your suggestion would invert that meaning and might therefore add more confusion than what it would solve. Moreover, to provide yet another alternative, you could also define something along the lines of: this = sys.modules[__name__] and then write: MATCH value: CASE this.x: ... Of course, this is a bit of a hack and not that much different to using SimpleNamespace. But it demonstrates that any such keyword would not really add that much power or save any keystrokes. In the end, I can well imagine that a future version of Python might add something like a "read and compare" marker to patten matching. But it is probably a good idea to start with a set of fully working but rather minimalistic core features, and then let it grow and evolve with time, guided by real use cases and issues that we run into. Kind regards, Tobias ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MJGSHTSPUXYCWEYCF5S6QGIMNAFP6ZSY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 and variadic positional-only args
Hi Oscar On Wed, Jul 15, 2020 at 4:41 PM Oscar Benjamin wrote: I've taken a look through PEP 622 and I've been thinking about how it could be used with sympy. Thank you very much for taking the time to carefully elaborate an interesting possible use case. I find this very helpful and a great basis for further/future discussions on the design of pattern matching. A deliberate part of the current design was to address the /structure/shape/ of objects rather than the constructor directly (which could be an arbitrary complex function after all). Writing `Add(args=(Mul(...), Mul(...))` for instance is therefore consistent as it reflects the /actual structure/ of your objects. The `__match_args__` is primarily intended for rather simple object shapes, where it is quite obvious what attributes constitute the object and in which order (similar to the `_fields` attribute in AST nodes). From this perspective, your use case makes an argument for something I would call '/variadic shape/' (in lack of a better word). Structurally, your objects behave like sequences or tuples, adorned with a specific type/class---which, again, is currently expressed as the "class(tuple)" pattern such as `Add((Mul(), Mul()))`. There are two possibilities to approach this issue. We could introduce patterns that extract "sequential elements" via `__getitem__` rather than attributes. Or we could have a special method `__match__` that might return a representation of the object's data in sequential form. The `__getitem__` approach turned out to come with quite a few complications. In short: it is very hard to assess an object's possibly sequential structure in a non-destructive way. Because of the multiple cases in the new pattern matching structure, we cannot just use an iterator as in unpacking. And `__getitem__` itself is extremely versatile, being used, e.g., for both sequences as well as mappings. We therefore ended up supporting only built-in structures like tuples, list, and dicts for now, for which the interpreter can easily determine how to handle `__getitem__`. The `__match__` protocol, on the other hand, is something that we deferred so that we can make sure it really is powerful and well designed enough to handle a wide range of use cases. One of the more interesting use cases, e.g., I had in mind was to destructure data that comes as byte strings, say (something that Rhodri James [1] has brought up, too). And I think you have just added another very interesting use case to take into consideration. But probably the best course of action is really to gain some experience and collect some additional use cases. Kind regards, Tobias [1] https://mail.python.org/archives/list/python-dev@python.org/message/WD2E3K5TWR4E6PZBM4TKGHTJ7VDERTDG/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PDZZGP6TY5G2SJBY2SII4W5GCO3B64PQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622: Structural Pattern Matching (version 2)
Hi Mark, Thank you for your message. I might be able to answer some of the questions and also address some issues with the underlying assumptions in your email---after all, we would most certainly want to avoid discussing and reasoning about straw men, as you yourself have repeatedly pointed out. Why the use of "shape" in "scare quotes"? Because the `shape' of an object is not something that is a well-defined term, but rather addresses the intuitive understanding to explain what we are talking about. It is usually the intention of the abstract to give a rough overview, before delving into the details and more formal descriptions. Using a contrived example like this seems like a straw man. It feels like it is constructed to favour the PEP, whilst being unlike any real code. It is a trait of many (good) tutorials to present a new structure with an example that highlights its feature in an approachable way rather than coming up with `real code'---particularly given that real code is always surrounded by a larger context, which rather clouds any understanding. And it would only be a straw man had we constructed a contrived example of unrealistic code that we then showed to be simplified by pattern matching. That's not the case: it really is just a simple educational example to present the idea. BTW: as a matter of course, it is constructed in favour of the PEP! There seem to be three things you want to enhance here: Unpacking; to avoid calls to `len` Type checking; to avoid calls to `isinstance`. To avoiding nesting by using complex lvalues. Actually, no. Pattern matching is not about _avoiding_ anything, it is about a more concise syntax. The suggestion that we wanted to avoid calls to `len` or `isinstance` is similar to the idea of using a lambda to avoid a function. The objective of pattern matching is to introduce a more succinct and readable way to express structure. It's all still there but we want to focus on the structure rather than `isinstance` calls. Pattern matching already exists in some limited way in Python. Even today you can write: `a, b = value` instead of: `a = value[0] b = value[1]` The idea of this is, of course, not so much about avoiding item access, but about having a more concise syntax for it. Moreover, saying this only saves a single line and is therefore rather useless is quite beside the point. It is about a better representation of what the code is supposed to do and not about saving lines. Saves two lines of code, but introduces two bugs! (Assuming that the original behavior should be preserved) Thank you for pointing out the bug in our example. This highlights, however, rather the difficulty of refactoring (did I already mention the issue of the 'context' that real code is embedded in?), than any shortcoming of pattern matching as a tool. And by the way: constructive critisism would perhaps include explicitly naming the two bugs. For example, by adding a simple "matches" method to Node and Leaf, `is_tuple` can be rewritten as something like: This assumes that you have full control over the entire code. But if you are using some third-party library, you cannot "simply add a `matches` method" (without some substantial trickery). Moreover, there is quite some work needed to define such a `matches` function as you propose it (including support for the ellipsis as wildcard). In effect, you would have to implement large parts of the pattern matching for just this simple example. Whereas we believe that it has merit enough to have the compiler do it for a wide range of possible use cases. I really don't see you how you can claim that `case 406:` is more readable than `elif response == HTTP_UPGRADE_REQUIRED:` ? Preventing the use of symbolic constants or other complex rvalues is a impairment to usability. First, let me quote what the PEP has to say on that exact example: Although this will work, it's not necessarily what the proposal is focused on. Moreover, it is usually a good idea to put constants into a separate namespace that further describes their meaning and intended use. And that is fully supported by the syntax as proposed in the PEP. ``` match response.status: case HTTP_RESPONSE.UPGRADE_REQUIRED: ... ``` For a PEP to succeed it needs to show two things. 1. Exactly what problem is being solved, or need is to be fulfilled, and that is a sufficiently large problem, or need, to merit the proposed change. 2. That the proposed change is the best known solution for the problem being addressed. IMO, PEP 622 fails on both counts. This seems fair enough as far as the two issues are concerned. It might seem indeed as if pattern matching does not add anything that could not be done already in Python. Part of the problem is that pattern matching starts to really shine when the objects and data