[Python-Dev] Re: About the new CFrame structure
Just to clear up a quick point I made: > - PyFrameObject.f_back just gives you a dummy wrapper around the previous > frame object. > - It's not really useful for unwinding anything. That should read "previous InterpreterFrame", rather than "previous frame object". Also, everything I wrote above is in the context of 3.11. InterpreterFrames don't exist in 3.10 and below, so in those versions PyFrameObject.f_back is indeed what you probably want. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TKPWWP33QJJEVEIP63C4SIEMVBY44LCW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: About the new CFrame structure
Hi Gabriele! > I hope you would indulge me in asking for some details about the new CFrame > structure, even in the form of existing literature (e.g. PEP) where the idea > behind it is explained. There isn't too much documentation on this, unfortunately (since these are all very unstable, low-level interpreter details), but a good place to start would be https://bugs.python.org/issue46090. Based on my own understanding (from reading the source code): - There are three relevant structures: CFrame, InterpreterFrame, and PyFrameObject. - PyFrameObjects are just PyObject wrappers around an InterpreterFrame, where all of the *actual* frame state for the Python stack is maintained. - They are created lazily (for example, when sys._getframe() is called). - See https://github.com/python/cpython/pull/27077. - InterpreterFrames live in a "datastack" for fast allocation and deallocation. - This "datastack" lives on the PyThreadState. - Because of how it is designed, InterpreterFrames must be allocated/deallocated "in order". - If an InterpreterFrame is cleared, but still has a live PyFrameObject that points to it, it will copy itself *into* the PyFrameObject first (to guarantee that the PyFrameObject keeps working). - See https://github.com/python/cpython/pull/26076. - A single CFrame is statically allocated inside of each _PyEval_EvalFrameDefault call, so it corresponds to the C stack, not the Python stack. - It links to a chain of one or more InterpreterFrames. - Multiple InterpreterFrames can correspond to a single CFrame! - This is a performance optimization in 3.11: rather than enter a new call to _PyEval_EvalFrameDefault, calls to pure-Python code just create a new InterpreterFrame, set it as the current one, and continue execution. - You can see how many InterpreterFrames correspond to the current CFrame by reading the "depth" member of the current InterpreterFrame. - A value of 0 indicates that this is the only InterpreterFrame for this CFrame. - A value of 42 means that this optimization has been performed 42 times (and there are currently 43 InterpreterFrames executing in this CFrame). > Also, I'd like to a quick question, if I may. There now appear to be two ways > of unwinding the frame stack: either iterate over CFrame.previous, or the > more traditional PyFrameObject.f_back. I suspect there are reasons why these > are perhaps not actually equivalent, and indeed this is mainly what I'd like > to read in the literature I've requested above. The above outline probably makes the differences clear: - PyFrameObject.f_back just gives you a dummy wrapper around the previous frame object. - It's not really useful for unwinding anything. - InterpreterFrame.previous gives you the previous interpreter frame (duh!). - This is probably what you want. - CFrame.previous gives you the previous call to _PyEval_EvalFrameDefault. - It's not really useful for unwinding anything. - This is only really useful to maintain the current tracing state when returning. Hopefully this helps! Somebody (Pablo or Mark?) will probably jump in here if I got anything wrong. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GDCPNESE2BWJUNPYFANCZVZK4EZNTKAF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?
Guido van Rossum wrote: > What's exasperating to me about this whole discussion is that nobody has > shown any reason why this kind of code would be occurring in user code. > STORE_FAST x LOAD_FAST x seems that it must come from a user writing > x = x I think that would be LOAD_FAST(x) STORE_FAST(x). STORE_FAST(x) LOAD_FAST(x) could be caused by something like this: def f(): x = … y = x Presumably this could be replaced with DUP_TOP() STORE_FAST(x)? But I don’t know if that’s any faster. It does require more stack space… > Are we just a solution (a clever peephole > trick) looking for a problem? It seems so. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XGFBCVDIEXEBKFGUOHSXJD3GKO7IAYLM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?
I can think of two reasons. The first reason is that this operation *does* have a side-effect: if a fast local is unbound, the load will raise a NameError! def f(): x # This should always raise. x = None # This makes x a fast local. The second reason is one that Guido already alluded to: the peephole optimizer shouldn’t be tasked with “fixing” poorly-written or uncommon code… just improving common code. If anything, we would probably just warn here. But even that seems like too much. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TJKKOVORTSM5QVRV5WXMUUKBUDYN2IQ7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 except* formatting
Łukasz Langa wrote: > Joking aside, since we allow any expression after 'except' 'group' then this > is indeed ambiguous. In theory! The ambiguity with function calls, though, is probably a dealbreaker: except group (E1, E2) as e: … except group(E1, E2) as e: … See my other message for an alternative (putting “group” after the expression). Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BUK2YFLH4TPIRTPG5JAYKNEWOBJTHC2B/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 except* formatting
Łukasz Langa wrote: > Joking aside, since we allow any expression after 'except' 'group' then this > is indeed ambiguous. In theory! Another option (to remove the ambiguity) could be to move the “group” after the expression. Bonus points for reading more clearly: except MemoryError group as e: … except (KeyError, IndexError) group as e: … except some + expression group as e: … And edge-cases like this still work normally: except some + group as e: … ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TW5I4Z3XKCSZC6IRXHNFVPZVLHEKI7O3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 except* formatting
Łukasz Langa wrote: > My idea is this: > try: > ... > except group E as e: > ... > except group E1, T2 as e: > ... > Should be doable given the magical match-case contextual keywords precedent. > This looks nice and is explicit, since you will always get an ExceptionGroup > instance under `e`. Heh, we crossed posts with the soft keywords. I like your idea (“except group”) better than mine (“except each”). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4FPTSD6VAIJD2WSP63KQUOQLDAOI3EWR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 except* formatting
Irit Katriel wrote: > It is also not too late to opt for a completely different syntax if a better > one is suggested. Honestly, I’ve never been a fan of the PEP’s proposed star syntax. If we’re okay adding a soft keyword, though, something like “except each” could help communicate the meaning of the blocks a bit more explicitly. I’m pretty sure that grammar would be unambiguous in all cases. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/44TWMI3PV3TKRL6ZJ4YU3GMQ6W43EHU5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should the definition of an "(async) iterator" include __iter__?
Guido van Rossum wrote: > TBH I don't think there is an *actual* problem here. I think it's just > about choosing the right wording for the glossary (which IMO does not have > status as a source of truth anyway). Good point. I'm probably approaching this from the wrong angle (by trying to "fix" the language, rather than the docs). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZUESGSZ2BIBZZI42ZUMCQVWUX3STIO6V/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should the definition of an "(async) iterator" include __iter__?
Guido van Rossum wrote: > On Tue, Sep 14, 2021 at 3:49 PM Brandt Bucher brandtbuc...@gmail.com > wrote: > > I think it's also worth noting that a missing "`__iter__` that returns > > self" is trivial to recover from... just use a new reference to the > > iterator instead. The overhead of a method call for this convention almost > > seems silly. > The use case is this: Yeah, I understand that. But what I'm hinting that is that the `GET_ITER` opcode and `iter` builtin *could* gracefully handle this situation when called on something that doesn't define `__iter__` but does define `__next__`. Pseudocode: def iter(o): if hasattr(o, "__iter__"): return o.__iter__() elif hasattr(o, "__next__"): # Oh well, o.__iter__() would have just returned o anyways... return o raise TypeError This would be implemented at the lowest possible level, in `PyObject_GetIter`. > > What worries me most about changing the current "requirement" is that it > > may create either confusion or backward compatibility issues for > > `collections.abc.Iterator` (which is a subtype of `Iterable`, and thus > > requires `__iter__`). > If you explicitly inherit from Iterator, you inherit a default > implementation of __iter__ (that returns self, of course). If you merely > register, it's up to you to comply. And sometimes people register things > that don't follow the letter of the protocol, just to get things going. > (This is common for complex protocols like Mapping, where some function you > have no control over insists on a Mapping but only calls one or two common > methods. Yeah, I was thinking about cases like `isinstance(o, Iterator)`, where `o` defines `__iter__` but not `__next__`. Even though this code might start returning the "right" answer, it's still a backward-compatibility break. Not sure what the true severity would be, though... ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SDZDMAF4MJDZHKIIWO2UUNRG6ZV2EU55/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should the definition of an "(async) iterator" include __iter__?
I think it's also worth noting that a missing "`__iter__` that returns self" is trivial to recover from... just use a new reference to the iterator instead. The overhead of a method call for this convention almost seems silly. What worries me most about changing the current "requirement" is that it may create either confusion or backward compatibility issues for `collections.abc.Iterator` (which is a subtype of `Iterable`, and thus requires `__iter__`). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RSV6MOIBVNEFKL4NDHTKDVSGVABVY65Q/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 467 feedback from the Steering Council
Brandt Bucher wrote: > You can even get creative and use the dedicated “pistol” operator… Ah, wait, ignore this example. I got the chr and ord behavior flipped in my head. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6JWFGVLOYIIDGATFNO3YUJTZPELXOK4E/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 467 feedback from the Steering Council
Steven D'Aprano wrote: > TIL :-) > How have I never noticed to_bytes until now? o_O I’m going to go out on a limb here: because it’s rarely ever needed? I mean, the proposed bchr() functionality is crazy simple to implement yourself if you actually *do* need it. You can even get creative and use the dedicated “pistol” operator: >>> b = b"*" >>> i ,= b >>> i 42 ;) Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AO5IOMBK2I5TSUBSRHEM75F6I3KC7AKP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 467 feedback from the Steering Council
Steven D'Aprano wrote: > To me, it sounds like should be the opposite of int.from_bytes. > >>> int.from_bytes(b'Hello world', 'little') > 121404708502361365413651784 > >>> bytes.from_int(121404708502361365413651784, 'little') > # should return b'Hello world' > If that's not the API being suggested, that's going to be confusing. I'm a bit lost here... why are we convinced at all that we need a new way to do this? Hasn't this functionality already existed for years? >>> x = int.from_bytes(b"*", "little") >>> x 42 >>> x.to_bytes(1, "little") b'*' Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FMG5K4BOX5GSUR2KU3G5ZLBBUIC3EQKD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Problems with dict subclassing performance
Marco Sulla wrote: > I remember the BDFL said in a post […] Really trying not to get involved, but for anybody still reading: Marco is seriously misquoting somebody here. The actual quote is “too many cooks”. https://mail.python.org/archives/list/python-dev@python.org/message/7QNWFKBLJPS5KY3UVFT426TDIHAF6WCX/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WLTXOABUP3X3HVTGBHYC7ZZHKOMTPWR5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks
Ethan Furman wrote:
> On 5/17/2021 6:13 AM, Mark Shannon wrote:
> > Where i1, i2 are integers and s1 is a string.
> > > i1 + i2 + s1
> >
> Wouldn't the carets just be under the i2 + s1 portion?
I don't think so, since this is executed as `((i1 + i2) + s1)`.
Mark's carets look correct to me, since the second (outer) addition's LHS is
the result of adding `i1` and `i2`:
```
Python 3.11.0a0 (heads/main:a42d98ed91, May 16 2021, 14:02:36) [GCC 7.5.0] on
linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> op = ast.parse("i1 + i2 + s1", mode="eval").body
>>> op
>>> op.col_offset
0
>>> op.end_col_offset
12
```
Brandt
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/YCMHFU4I7TEYDQP7OH4AX2YOD4KPLNFX/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: str() vs format(): trivia question
It has always bugged me that for Enums mixed in with int or str (a common
pattern in my code), `f"{MyEnum.X}"` is not the same as `str(MyEnum.X)`.
I'd be happy to see it changed!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/FAQ7QYJMUBMNM4PZSMKFV3NHF26WKVT3/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Hi Mark. Thanks for your reply, I really appreciate it. Mark Shannon said: > My intention, and I apologize for not making this clearer, was not to > denigrate your work, but to question the implications of the term "reference". > > Calling something a "reference" implementation suggests that it is something > that people can refer to, that is near perfectly correct and fills in the > gaps in the specification. > > That is a high standard, and one that is very difficult to attain. It is why > I use the term "implementation", and not "reference implementation" in my > PEPs. Interesting. The reason I typically include a "Reference Implementation" section in my PEPs is because they almost always start out as a copy-paste of the template in PEP 12 (which also appears in PEP 1): https://www.python.org/dev/peps/pep-0001/#what-belongs-in-a-successful-pep https://www.python.org/dev/peps/pep-0012/#suggested-sections Funny enough, PEP 635 has a "Reference Implementation" section, which itself refers to the implementation as simply a "feature-complete CPython implementation": https://www.python.org/dev/peps/pep-0635/#reference-implementation (PEP 634 and PEP 636 don't mention the existence of an implementation at all, as far as I can tell.) It's not a huge deal, but we might consider updating those templates if the term "Reference Implementation" implies a higher standard than "we've put in the work to make this happen, and you can try it out here" (which is what I've usually used the section to communicate). Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T2RRSWLTBEQKZVHROFQFUYVWCLZQ3MBB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Mark Shannon said: > I was relying on the "reference" implementation, which is also in the PEP. Can you please stop putting scare quotes around "reference implementation"? You've done it twice now, and it's been a weekend-ruiner for me each time. I've put months of work into writing and improving CPython's current pattern matching implementation, mostly on nights and weekends. I don't know whether it's intentional or not, but when you say things like that it instantly devalues all of my hard work in front of everyone on the list. For such a huge feature, I'm honestly quite amazed that this is the only issue we've found since it was merged over a month ago (and both authors have agreed that it needs to be fixed in the PEP, not the implementation). The PR introducing this behavior was reviewed by at least a half-dozen people, including you. The last time you said something like this, I just muted the thread. Let's please keep this respectful; we're all obviously committing a lot of our own time and energy to this, and we need to work well together for it to be successful in the long term. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HJHZPEALHSFWBBBZNF4XLGDHR5CXY7RE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Mark Shannon wrote:
> On 02/04/2021 7:19 am, Brandt Bucher wrote:
> > I agree that self-matching classes should absolutely allow keyword matches.
> > I had no idea the PEP forbade it.
> PEP 634 allows it.
PEP 634 says:
> For a number of built-in types (specified below), a single positional
> subpattern is accepted which will match the entire subject; for these types
> no keyword patterns are accepted.
(https://www.python.org/dev/peps/pep-0634/#class-patterns)
> Most checks are cheap though.
> Checking for duplicates in `__match_args__` can be done at class creation
> time, and checking for duplicates in the pattern can be done at compile time.
I assume the compile-time check only works for named keyword attributes. The
current implementation already does this.
-1 on checking `__match_args__` anywhere other than the match block itself.
Guido van Rossum wrote:
> On Fri, Apr 2, 2021 at 3:38 AM Mark Shannon m...@hotpy.org wrote:
> > Are there are any use-cases?
> > The test-case `int(real=0+0j, imag=0-0j)` is contrived, but I'm struggling
> > to come up with less contrived examples for any of float, list, dict,
> > tuple, str.
> There could be a subclass that adds an attribute. That's still contrived
> though.
I could see the case for something like `case defaultdict({"Spam": s},
default_factory=f)`. I certainly don't think it should be forbidden.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/L2RB6WYAOSLZECG4N2JTYMAAQX3U64GA/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Guido van Rossum wrote:
> Well, now I have egg on my face, because the current implementation does
> reject multiple occurrences of the same identifier in __match_args__. We
> generate an error like "TypeError: C() got multiple sub-patterns for
> attribute 'a'". However, I cannot find this uniqueness requirement in PEP
> 634, so I think it was a mistake to implement it.
>
> Researching this led me to find another issue where PEP 634 and the
> implementation differ, but this time it's the other way around: PEP 634 says
> about types which accept a single positional subpattern (int(x), str(x) etc.)
> "for these types no keyword patterns are accepted." Mark's example `case
> int(real=0, imag=0):` makes me think this requirement is wrong and I would
> like to amend PEP 634 to strike this requirement. Fortunately, this is not
> what is implemented. E.g. `case int(1, real=1):` is accepted and works, as
> does `case int(real=0):`.
>
> Calling out Brandt to get his opinion. And thanks to Mark for finding these!
The current implementation will reject any attribute being looked up more than
once, by position *or* keyword. It's actually a bit tricky to do, which is why
the `MATCH_CLASS` op is such a beast... it needs to look up positional and
keyword attributes all in one go, keeping track of everything it's seen and
checking for duplicates.
I believe this behavior is a holdover from PEP 622:
> The interpreter will check that two match items are not targeting the same
> attribute, for example `Point2d(1, 2, y=3)` is an error.
(https://www.python.org/dev/peps/pep-0622/#overlapping-sub-patterns)
PEP 634 explicitly disallows duplicate keywords, but as far as I can tell it
says nothing about duplicate `__match_args__` or keywords that also appear in
`__match_args__`. It looks like an accidental omission during the 622 -> 634
rewrite.
(I guess I figured that if somebody matches `Spam(foo, y=bar)`, where
`Spam.__match_args__` is `("y",)`, that's probably a bug in the user's code.
Ditto for `Spam(y=foo, y=bar)` and `Spam(foo, bar)` where `Spam.__match_args__`
is `("y", "y")` But it's not a hill I'm willing to die on.)
I agree that self-matching classes should absolutely allow keyword matches. I
had no idea the PEP forbade it.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/EKXKRZBJAUWVJVMF3MCNJA6I7HLLL26B/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Guido van Rossum wrote: > On Wed, Mar 31, 2021 at 2:14 PM Brandt Bucher brandtbuc...@gmail.com > wrote: > > (One change from my last email: it doesn't allow `__match_map__` / > > `__match_seq__` to be set to `False`... only `True`. This prevents some > > otherwise tricky multiple-inheritance edge-cases present in both of our > > flagging systems that I discovered during testing. I don't think there are > > actual use-cases for unsetting the flags in subclasses, but we can revisit > > that later if needed.) > That's surprising to me. Just like we can have a class that inherits from > int but isn't hashable, and make that explicit by setting `__hash__ = > None`, why couldn't I have a class that inherits from something else that > happens to inherit from Sequence, and say "but I don't want it to match > like a sequence" by adding `__match_sequence__ = False`? AFAIK all Mark's > versions would support this by setting `__match_kind__ = 0`. The issue isn't when *I* set `__match_seq__ = False` or `__match_container__ = 0`. It's when *one of my parents* does it that things become difficult. > Maybe you can show an example edge case where this would be undesirable? Good idea. I've probably been staring at this stuff for too long to figure it out myself. :) As far as I can tell, these surprising cases arise because a bit flag can only be either 0 or 1. For us, "not specified" is equivalent to 0, which can lead to ambiguity. Consider this case: ``` class Seq: __match_seq__ = True # or __match_container__ = MATCH_SEQUENCE class Parent: pass class Child(Parent, Seq): pass ``` Okay, cool. `Child` will match as a sequence, which seems correct. But what about this similar case? ``` class Seq: __match_seq__ = True # or __match_container__ = MATCH_SEQUENCE class Parent: __match_seq__ = False # or __match_container__ = 0 class Child(Parent, Seq): pass ``` Here, `Child` will *not* match as a sequence, even though it probably should. The only workarounds I've found (like allowing `None` to mean "this is unset, don't inherit me if another parent sets this flag", ditching tp_flags entirely, or not inheriting these attributes) feel a bit extreme just to allow some users to do the moral equivalent of un-subclassing `collections.abc.Sequence`. So, my current solution (seen on the branch linked in my earlier email) is: - Set the flag if the corresponding magic attribute is set to True in the class definition - Raise at class definition time if it's set to anything other than True - Otherwise, set the flag if any of the parents set have the flag set As far as I can tell, this leads to the expected (and current, as of 3.10.0a6) behavior in all cases. Plus, it doesn't break my mental model of how inheritance works. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R3BGDN6NINJMLUWBVMVYIGORSLPJOMJP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
>> - Add new `__match_seq__` and `__match_map__` special attributes, >> corresponding to new public `Py_TPFLAGS_MATCH_SEQ` and >> `Py_TPFLAGS_MATCH_MAP` flags for use in `tp_flags`. When Python classes are >> defined with one or both of these attributes set to a boolean value, >> `type.__new__` will update the flags on the type to reflect the change >> (using a similar mechanism as `__slots__` definitions). They will be >> inherited otherwise. For convenience, `collections.abc.Sequence` will define >> `__match_seq__ = True`, and `collections.abc.Mapping` will define >> `__match_map__ = True`. >> >> Using this in Python would look like: >> >> ``` >> class MySeq: >> __match_seq__ = True >> ... >> >> class MyMap: >> __match_map__ = True >> ... >> ``` > > I don't like the way this need special inheritance rules, where inheriting > one attribute mutates the value of another. It seems convoluted. Let me clarify: these two attributes do not interact with one another; each attribute only interacts with its own flag on the type. It is perfectly possible to do: ``` class WhatIsIt: __match_map__ = True __match_seq__ = True ``` This will set both flags, and this `WhatIsIt` will match as a mapping *and* a sequence. This is allowed and works in PEP 634, but like Guido I'm not entirely opposed to making the matching behavior of such a class undefined against sequence or mapping patterns. > Consider: > > class WhatIsIt(MySeq, MyMap): pass > > With __match_container__ it works as expected with no special inheritance > rules. What *is* the expected behavior of this? Based on the current behavior of PEP 634, I would expect the `__match_container__` of each base to be or'ed, and something like this to match as both a mapping and a sequence (which PEP 653 says leads to undefined behavior). The actual behavior seems more like it will just be a sequence and not a mapping, since `__match_container__` would be inherited from `MySeq` and `MyMap` would be ignored. In the interest of precision, here is an implementation of *exactly* what I am thinking: `typeobject.c`: https://github.com/python/cpython/compare/master...brandtbucher:patma-flags#diff-1decebeef15f4e0b0ce106c665751ec55068d4d1d1825847925ad4f528b5b872 `ceval.c`: https://github.com/python/cpython/compare/master...brandtbucher:patma-flags#diff-c22186367cbe20233e843261998dc027ae5f1f8c0d2e778abfa454ae74cc59de (One change from my last email: it doesn't allow `__match_map__` / `__match_seq__` to be set to `False`... only `True`. This prevents some otherwise tricky multiple-inheritance edge-cases present in both of our flagging systems that I discovered during testing. I don't think there are actual use-cases for unsetting the flags in subclasses, but we can revisit that later if needed.) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S4KLVTEA4HYDCAER25DZMOUB6LN6K63P/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Hi Mark.
I've spoken with Guido, and we are willing to propose the following amendments
to PEP 634:
- Require `__match_args__` to be a tuple.
- Add new `__match_seq__` and `__match_map__` special attributes, corresponding
to new public `Py_TPFLAGS_MATCH_SEQ` and `Py_TPFLAGS_MATCH_MAP` flags for use
in `tp_flags`. When Python classes are defined with one or both of these
attributes set to a boolean value, `type.__new__` will update the flags on the
type to reflect the change (using a similar mechanism as `__slots__`
definitions). They will be inherited otherwise. For convenience,
`collections.abc.Sequence` will define `__match_seq__ = True`, and
`collections.abc.Mapping` will define `__match_map__ = True`.
Using this in Python would look like:
```
class MySeq:
__match_seq__ = True
...
class MyMap:
__match_map__ = True
...
```
Using this in C would look like:
```
PyTypeObject PyMySeq_Type = {
...
.tp_flags = Py_TPFLAGS_MATCH_SEQ | ...,
...
}
PyTypeObject PyMyMap_Type = {
...
.tp_flags = Py_TPFLAGS_MATCH_MAP | ...,
...
}
```
We believe that these changes will result in the best possible outcome:
- The new mechanism should faster than either PEP.
- The new mechanism should provide a better user experience than either PEP
when defining types in either Python *or C*.
If these amendments were made, would you be comfortable withdrawing PEP 653? We
think that if we're in agreement here, a compromise incorporating these
promising changes into the current design would be preferable to submitting yet
another large pattern matching PEP for a very busy SC to review and pronounce
before the feature freeze. I am also willing, able, and eager to implement
these changes promptly (perhaps even before the next alpha) if so.
Thanks for pushing us to make this better.
Brandt
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/QJR5TT5BW55PZFXJJFEYUCNZ5XQF256O/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)
Overall, I am still uncomfortable with PEP 653, and would probably not support its acceptance. Although it has thankfully become a much less radical proposal than it was a few weeks ago (thanks, Mark, for your attention to our feedback), I feel that the rules it binds implementations to are *very* premature, and that the new mechanisms it introduces to do so only modestly improve potential performance at great expense to the ease of learning, using, and maintaining code using structural pattern matching. A few notes follow: > For example, using `sympy`, we might want to write: > > ``` > # a*a == a**2 > case Mul(args=[a, b]) if a == b: > return Pow(a, 2) > ``` > > Which requires the sympy class `Symbol` to "self" match. For `sympy` to > support this pattern with PEP 634 is possible, but a bit tricky. With this > PEP it can be implemented very easily. Maybe I'm missing something, but I don't understand at all how the provided code snippet relies on the self-matching behavior. Have the maintainers of SymPy (or any large library supposedly benefitting here) come out in support of the PEP? Are they at least aware of it? Have they indicated that the proposed idiom for implementing self-matching behavior using a property is truly too "tricky" for them? Have you identified any stdlib classes that would benefit greatly from this? For me, `__match_class__` feels like a feature without demonstrated need. Even if there is a great demand for this, I certainly think that there are far better options than the proposed flagging system: - A `@match_self` class decorator (someone's bound to put one on PyPI, at any rate). - Allowing `__match_args__ = None` to signal this case (an option we previously considered, and my personal preference). ...both of which can be added later, if needed. Further, PEP 634 makes it very easy for libraries to support Python versions with *and* without pattern matching (something I consider to be an important requirement). The following class works with both 3.9 and 3.10: ``` class C(collections.abc.Sequence): ... ``` While something like this is required for PEP 653: ``` class C: if sys.version_info >= (3, 10): from somewhere import MATCH_SEQUENCE __match_container__ = MATCH_SEQUENCE ... ``` > PEP 634 relies on the `collections.abc` module when determining which > patterns a value can match, implicitly importing it if necessary. This PEP > will eliminate surprising import errors and misleading audit events from > those imports. I think that a broken `_collections_abc` module *should* be surprising. Is there any reasonable scenario where it's expected to not exist, or be not be fit for this purpose? And I'm not sure how an audit event for an import that is happening could be considered "misleading"... I certainly wouldn't want it suppressed. > Looking up a special attribute is much faster than performing a subclass test > on an abstract base class. How much faster? A quick benchmark on my machine suggests less than half a microsecond. PEP 634 (like PEP 653) already allows us to cache this information for the subject of a match statement, so I doubt that this is actually a real issue in practice. An indeed, with the current implementation, this test isn't even performed on the most common types, such as lists, tuples, and dictionaries. At the very least, PEP 653's confusing new flag system seems to be a *very* premature optimization, seriously hurting usability for a modest performance increase. (Using them wrongly also seems to introduce a fair amount of undefined behavior, which seems to go against the PEP's own motivation.) > If the value of `__match_args__` is not as specified, then the implementation > may raise any exception, or match the wrong pattern. I think there's a name for this sort of behavior... ;) A couple of other, more technical notes: - PEP 653 requires mappings to have a `keys()` method that returns an object supporting set inequality operations. It is not really that common to find this sort of support in user code (in my experience, it is more likely that user-defined `keys()` methods will return iterables). It's not even clear to me if this is an interface requirement for mappings in general. For example, `weakref.WeakKeyDictionary` and `weakref.WeakValueDictionary` presently do not work with PEP 653's requirements for mapping patterns, since their `keys()` methods return iterators. - Treating `__getitem__` as pure is problematic for some common classes (such as `defaultdict`). That's why we use two-argument `get()` instead. As well-fleshed out as the pseudocode for the matching operations in this PEP may be, examples like this suggest that perhaps we should wait until 3.11 or later to figure out what actually works in practice and what doesn't. PEP 634 took a full year of work, and the ideas it proposed changed substantially during that time (in no small part because we
[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Oscar Benjamin wrote: > Under PEP 634 in general, for any class C, a pattern C(x) matches an object > C(x, y) and there's no way for C to override that. To me that is sufficiently > unintuitive in the abstract that no example is really needed to see where > there is room for improvement. We originally kicked around (and at one point even implemented) a `__match_args_required__` attribute, which is an integer specifying a minimum required number of positional sub-patterns. For reasons I can't recall, though, it was eventually dropped. It would take care of this, and I imagine it could be quite painless to add it back if there was enough support. Here's a query for related discussions: https://github.com/gvanrossum/patma/issues?q=__match_args_required__ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JKZUE2BUDZQQHQLZK3IYBUNHYQWYMVA6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Brandt Bucher wrote: > For a pattern with no positional subpatterns, such as like `C()` or `C(x=x, > y=y, z=z)`: ... > It also appears that we lose a lot of expressive "idioms" by requiring > `__attributes__` to be complete. > This also means that matching classes like `types.SimpleNamespace` are much > less powerful under PEP 653, since the class must know which attributes are > "allowed" to be looked up. Never mind these three points... I *think* setting `__match_kind__ = MATCH_DEFAULT` allows for arbitrary attribute extraction like this. Perhaps make it bit clearer? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HCXNNE3B2SS3PFTDVMAA4EWNVXD6TH7W/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 653: Precise Semantics for Pattern Matching
Thanks for taking the time to work on this, Mark.
Overall, I'm skeptical of this proposal. It seems like it takes a lot of
"simple" things and makes them quite complex, and takes many "static" things
and makes them quite dynamic. I feel that it also misrepresents certain aspects
of PEP 634.
Here's a more detailed commentary:
> Pattern matching will be more usable for complex classes, by allowing classes
> more control over which patterns they match.
I fear this is at the expense of most simple classes, which currently "just
work" with PEP 634. I'm not convinced that making it easier for very
complicated classes (like those in `sympy`) to participate at the expense of
everyday classes is a win.
For comparison, here is what (as I understand it) each PEP requires for a class
to be used in a pattern such as `C(x, y, z)`:
```py
class C:
"""A PEP 634 matchable class."""
__match_args__ = ...
class C:
"""A PEP 653 matchable class."""
__match_kind__ = MATCH_CLASS
__attributes__ = ...
def __deconstruct__(self):
...
```
For a pattern with no positional subpatterns, such as like `C()` or `C(x=x,
y=y, z=z)`:
```py
class C:
"""A PEP 634 matchable class (all classes work)."""
class C:
"""A PEP 653 matchable class."""
__match_kind__ = MATCH_CLASS
__attributes__ = ...
def __deconstruct__(self):
...
```
It also appears that we lose a lot of expressive "idioms" by requiring
`__attributes__` to be complete. For example, the following attribute
extractions work with PEP 634, but not PEP 653.
```py
match subject:
case object(a=_):
# Match any object with an "a" attribute.
...
case object(x=x, y=y, z=z):
# Match any object with "x", "y", and "z" attributes, extracting them.
...
```
This also means that matching classes like `types.SimpleNamespace` are much
less powerful under PEP 653, since the class must know which attributes are
"allowed" to be looked up.
Further, this whole specification shifts most of the matching logic from the
class in the pattern to the subject itself:
```py
match ChildA(), ChildB():
case Parent(a, b), Parent(x, y):
...
```
The above pattern could have completely different rules for extracting `a` and
`b` vs `x` and `y` if either child overrides `__deconstruct__`. I think that
is a mistake.
> PEP 634 also privileges some builtin classes with a special form of matching,
> the "self" match. For example the pattern `list(x)` matches a list and
> assigns the list to `x`. By allowing classes to choose which kinds of pattern
> they match, other classes can use this form as well.
This is already fairly trivial to implement:
```py
class C:
__match_args__ = ("_self",)
_self = property(lambda s: s)
```
You can even avoid adding a `_self` attribute if you do some magic with
descriptors... ;). We could consider provide a decorator for this somewhere in
the stdlib if there's demonstrated need (but I'm not sure there will be).
> All classes should ensure that the the value of `__match_kind__` follows the
> specification. Therefore, implementations can assume, without checking, that
> all the following are false:
> `(__match_kind__ & (MATCH_SEQUENCE | MATCH_MAPPING)) == (MATCH_SEQUENCE |
> MATCH_MAPPING)`
> `(__match_kind__ & (MATCH_SELF | MATCH_CLASS)) == (MATCH_SELF | MATCH_CLASS)`
> `(__match_kind__ & (MATCH_SELF | MATCH_DEFAULT)) == (MATCH_SELF |
> MATCH_DEFAULT)`
> `(__match_kind__ & (MATCH_DEFAULT | MATCH_CLASS)) == (MATCH_DEFAULT |
> MATCH_CLASS)`
Oof. I was scratching my head for way too long at this before I noticed the
previous sentence said "false". Maybe reword this section to indicate the
conditions that hold *true*? :)
> Security Implications
> Preventing the possible registering or unregistering of classes as sequences
> or a mappings, that PEP 634 allows, should improve security. However, the
> advantage is slight and is not a motivation for this PEP.
I would say the advantage is nonexistent. How is this any different than
tweaking the flags in a class's `__match_kind__`?
> Efficient Implementation / Implementation
I won't get too deep into this section, but on the surface most flavors of
implementation/optimization present here are also possible with PEP 634. I
understand that you feel there are benefits to having "rules" for what
optimizations are legal, but we don't need to completely change the mechanics
of the match statement in order to do so.
> Because the object model is a core part of Python, implementations already
> handle special attribute lookup efficiently. Looking up a special attribute
> is much faster than performing a subclass test on an abstract base class.
But calling a `__deconstruct__` method whenever positional arguments are
present will slow down normal class matches, right? I see it as mostly a wash.
> The changes to the semantics can be summarized as:
> - Selecting the kind of pattern uses
[Python-Dev] Re: PEP 642 v2: Explicit constraint patterns *without* question marks in the syntax
Paul Sokolovsky wrote:
> Use punctuation ("sigils") to mark as-binding terms. This choice still seems
> to be under-considered. (As in: it doesn't seem like many people, including
> the PEP authors, tried to say "indeed, what if?" and feel thru it. I mean,
> try really hard. I trust the "gang of 4" spent maybe whole few hours on that
> and delivered "no" to all us. It's still not the same as dozens of people
> trying it over a few months).
To anyone actually wondering how much time and mental energy we’ve spent on a
particular issue: please take a look at our tracker before guessing “maybe
whole few hours”:
- Issue #1(!), April, 29 comments: https://github.com/gvanrossum/patma/issues/1
- Issue #90, June, 84 comments: https://github.com/gvanrossum/patma/issues/90
- Issue #92, June, 33 comments: https://github.com/gvanrossum/patma/issues/92
- Issue #105, June, 17 comments: https://github.com/gvanrossum/patma/issues/105
- Issue #143, August, 7 comments: https://github.com/gvanrossum/patma/issues/143
(I won't judge anyone for skimming a bit; it's a *lot* of discussion. Do note,
though, that for months I was one of the proponents of store sigils like `?`
until I was eventually convinced otherwise.)
That's also not counting side-discussions in other issues, countless mailing
list threads, two competing PEPs that make many of the same choices, a video
call with the SC, etc.
I'll also add, for anyone considering choosing yet another ASCII symbol off
their keyboard and proposing it as a “novel”, “intuitive” marker: one problem
with most of the hastily suggested adornments are that they do not nest very
well, for even simple cases. Further, the fact that constructions like `p =
Point(x, y)` look exactly the same as deconstructions like `case Point(x, y):`
is absolutely intentional (which touches back on Guido’s “rhyme” comment last
night).
Very briefly, compare the current syntax:
```
case first, *middle, last:
rebuilt = first, *middle, last
case {"key": value, **rest}:
rebuilt = {"key": value, **rest}
case Point(x=a, y=b):
rebuilt = Point(x=a, y=b)
```
with (using your own syntactic flavor):
```
case >first, *>middle, >last:
rebuilt = first, *middle, last
case {"key": >value, **>rest}:
rebuilt = {"key": value, **rest}
case Point(x=>a, y=>b):
rebuilt = Point(x=a, y=b)
```
(I had to stop and think *hard* where exactly the `>` should go in `*middle`
and `**rest`. I'm not confident I made the correct guess, either.)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/7SJ2SOPRD6XOAISOYXONKXUKB3JMLYNU/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 642: Constraint Pattern Syntax for Structural Pattern Matching
Glenn Linderman wrote: > So what _is_ the syntax for "a tuple of two equal values" ? If you’re asking about PEP 634: ``` case x, y if x == y: ``` Which is much clearer, in my opinion. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UM6ZACEZLF2B6YLMMO4TASWCVVKD2KPM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Thoughts on PEP 634 (Structural Pattern Matching)
> Can we discuss whether we want pattern matching in Python and the broader > semantics first, before dealing with low level details? This is a huge step backward. These discussions have already taken place, over the last 10 years. Here's just a sampling: - https://mail.python.org/archives/list/python-id...@python.org/thread/IEJFUSFC5GBDKFIPCAGS7JYXV5WGVAXP/ - https://mail.python.org/archives/list/python-id...@python.org/thread/GTRRJHUG4W2LXGDH4AU46SI3DLWXJF6A/ - https://mail.python.org/archives/list/python-id...@python.org/thread/EURSG3MYEFHXDDL2474PQNQZFJ3CUIOX/ - https://mail.python.org/archives/list/python-id...@python.org/thread/NTQEL3HRUJMULQYI6RDBTXQ2H3KHBBRO/ - https://mail.python.org/archives/list/python-id...@python.org/thread/NEC54II2RB3JRGHDP6PX3NOEALRAK6BV/ - https://mail.python.org/archives/list/python-id...@python.org/thread/T3VBUFECTLZMB424MBBGUHCI24YA4FPT/ We read all of these and more back way in March, before we even started brainstorming syntax and semantics. > Do we want a fancy switch statement, or a powerful expression? It's right here that you lose me. Anyone who reduces pattern matching to "a fancy switch statement" probably isn't the right person to be discussing its semantics and usefulness with. It seems that some people just can't separate the two ideas in their mind. It's like calling a class a "fancy module". It's okay that you feel that way, but hopefully you'll understand if people start to tune out messages that contain these sorts of comments. > What special method(s) should be added? None. PEP 622 originally added one, but even that is more than we need right now. Some people may need to register their mappings or sequences as Mappings or Sequences, but otherwise that's it. > I would ask anyone who wants pattern matching adding to Python, to not > support PEP 634. Seriously? I would ask anyone who wants pattern matching added to Python to carefully consider the PEPs for themselves (particularly PEP 636, which is much less dry and contains more examples and commentary). We've written four of the largest, most detailed PEPs of any new feature I've seen, complete with a working implementation that we've made available from any browser. Of course it's not the *only* way of getting pattern matching... but if you want it, this is probably your *best* shot at getting it. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DEUZMFMTDBYUD3OHB5HNN7MWWNP237VV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: docs: I'd like new features to references their PEPs
Agreed. To prevent the docs from going stale, the "Originally proposed in :pep:`XXX`." wording should probably be used for *all* of the new links, not just the ones that are currently out-of-date. Depending on the scope of these changes, we could also just consider adding a new ".. pepadded:: XXX" directive for reuse and consistency. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2R4JJSJTCLVLWWQ4FKMMTVJ3UE3DVC2T/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deferred, coalescing, and other very recent reference counting optimization
Tim Peters wrote: > `zip` then creates `n` 2-tuple objects, each of which lives only long enough > to be unpacked into `x` and `y`... With "immediate" reclamation of garbage > via refcounting, memory use is trival regardless of how large `n` is, as > CPython reuses the same heap space over & over & over, ASAP. The space for > each 2-tuple is reclaimed before `x-y` is computed... It's also worth noting that the current refcounting scheme allows for some pretty sneaky optimizations under-the-hood. In your example, `zip` only ever creates one 2-tuple, and keeps reusing it over and over: https://github.com/python/cpython/blob/c96d00e88ead8f99bb6aa1357928ac4545d9287c/Python/bltinmodule.c#L2623 This is thanks to the fact that most `zip` usage looks exactly like yours, where the tuple is only around long enough to be unpacked. If `zip.__next__` knows it's not referenced anywhere else anymore, it is free to mutate(!) it in place. I believe PyUnicode_Append does something similar for string concatenation, as well. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MJ4BL42YSMP5BUGWT7EEK3EKNVGBDH35/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)
Federico Salerno wrote: > Is there anything (bar tradition or other subjective arguments) that speaks > in favour of this, especially in light of the fact that having the same > indentation level would also solve other problems? ...if anyone made an > argument against it I must have missed it: We spend a fair bit of time discussing this exact proposal in the PEP (I just searched for "indent"): https://www.python.org/dev/peps/pep-0622/#use-a-flat-indentation-scheme ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KLP6CTEVLYXVGBGBWCTUARV2KQAPYJVS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 version 2 (Structural Pattern Matching)
Ethan Furman wrote: > Why is this no longer an issue? My apologies if I missed it in the PEP. This problem was an artifact of the default `object.__match__` implementation, which allowed one positional argument by default when `__match_args__` was missing or `None`. Since we've removed `__match__` from the proposal (and therefore the default `__match__` implementation from `object`), this issue no longer exists. (Note that most common built-in types like `int` and `tuple` will still work this way, but this behavior is not inherited by *all* objects anymore). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NFP22VPMLZ4EERYU6KEB2KO7KQQ7ETA5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622: Structural Pattern Matching -- followup
Inada Naoki wrote: > Since this is very new system, can we have some restriction to allow > aggressive optimization than regular Python code? The authors were just discussing a related question yesterday (more specifically, can the compiler fold `C() | C()` -> `C( | )`). The answer we arrived at is "yes"; in general patterns may take reasonable shortcuts, and should not be expected to follow all the same rules as expressions. This means that users should never count on `__contains__`/`__getitem__`/`__instancecheck__`/`__len__`/`__match_args__` or other attributes being looked up or called more than once with the same arguments, and that name lookups *may* be "frozen", in a sense. We don't feel a need to cater to code that relies on these side-effecty behaviors (or doing even nastier things like changing local/global name bindings); in the eyes of the authors, code like that is buggy. However, these rules only apply as long as we are still in "pattern-land", meaning all of our knowledge about the world is invalidated as soon as we hit a guard or stop matching. In practice, I am currently experimenting with building decision-trees at compile-time. Given a match block of the following form: ``` match : case | : ... case | if : ... case | : ... ``` It's safe to use the same decision tree for through , but it must be rebuilt for and , since could have done literally *anything*. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XY3FXVB7HDYJIVKSOOBHW5BV2UB522FL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Why does _ need to be special ?
Henk-Jaap Wagenaar wrote: > The not binding is there only to allow the main way in which "_" is special > in match/case: ... The non-binding behavior is useful in other ways: match range(HUGE_INT): case [*_, last]: print(last) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2BYZQUCYIJKJSS6DRDVLRWI5B767S6ZO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 railroaded through?
Since I took it upon myself to implement PEP 622, I just have a few thoughts to
add to the other excellent responses here. Hopefully these will help clarify
that the intent is not to "railroad" anything.
Rob Cliffe wrote:
> PEP 622 only seems to have been presented to the Python community only after
> a well-developed (if not finalised) implementation was built.
Well, thanks for the "well-developed" bit, but I promise that the
implementation is far from final. Since we first published the PEP, I've made
dozens of commits touching pretty much every new line of code several times.
Just last week we gutted the __match__ method in response to well-reasoned
calls to defer it.
> So there will inevitably be resistance from the developers to accept changes
> suggested on python-dev.
As the one who has been doing almost all of the development, I assure you this
isn't true. 80% of the PEP's authors have been almost entirely detached from
development of the reference implementation, so I would be very surprised if
they gave my existing work more weight than the opinions of the community... I
wrote the thing, and I certainly don't!
I volunteered to implement it because I thought the PEP and implementation
would be an interesting project while trapped at home this spring. I *like*
doing this stuff, so I'm not really worried about getting to do more of it. ;)
> And since the PEP has Guido's authority behind it, I think it is likely that
> it will eventually be accepted pretty much as it was originally written.
It has already changed quite substantially from how it was originally written.
Here's everything that's changed since we posted the first draft (we've been
pretty much dominating PEP repo traffic over the past week):
https://github.com/python/peps/commits/master/pep-0622.rst
Again, you'll notice that the entire __match__ protocol was deferred based on
feedback we received, and we've made an effort to describe the reasoning behind
many decisions that seemed obvious to us but weren't to others. The opening
sections are also getting a rewrite (again, based on Python-Dev feedback).
> Guido's 2nd email ("PEP 622: Structural Pattern Matching -- followup")
> already to me (YMMV) reads rather like "OK, you've had your fun, now try not
> to joggle our elbows too much while we get on with the work".
That's an extremely odd way to interpret his thread, which exists solely to
collect of all of the "unheard" critiques in one central location.
> I do think it's a pity that the Python community did not have the chance to
> supply feedback earlier down the road (when IMO it would have been taken more
> seriously).
It did. The very first thing we did was perform detailed surveys of 8 different
Python-Ideas threads (spanning 9 years), 13 other languages, and a handful of
the most popular Python packages for pattern matching. This is not at all the
first time this has been brought up by the community; I personally worked my
way through hundreds of emails and dozens of documents before writing a single
line of code (or PEP).
> While Guido and the other developers have obviously already put a huge amount
> of work into this PEP (and by now probably have a significant emotional
> investment in it), I do hope that they will take the time to consider
> seriously and on their merits most/all suggested changes, rather than being
> tempted to rush through the acceptance and implementation of the PEP.
Don't worry; I have a much stronger emotional connection to Python's continued
success and community than to a quarantine project that Guido nerd-sniped me
with a few months ago. ;)
Brandt
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/JB7WSMUUVBNDYSNLECJEYK75WFSJMM6X/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622 (match statement) playground
Walter Dörwald wrote: > This looks strange to me. In all other cases of variable lookup the global > variable z would be found. The next case assigns to z, making z local to whereis. This is consistent with python's existing scoping rules (for example, try rewriting this as the equivalent if-elif chain and you'll get the same error). It sounds like you want to add "global z" to the top of the function definition. > whereis(23) however works. This branch is hit before the unbound local lookup is attempted. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CZAZYFSWWEYTXDEXEZ75TNTVKSIZZ27K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622: Structural Pattern Matching -- followup
Ethan Furman wrote: > Ouch. That seems like a pretty serious drawback. Will this issue be > resolved? It's currently being revisited. Realistically, I'd imagine that we either find some straightforward way of opting-in to the current default behavior (allowing one arg to be positionally matched against the proxy), or lose the nice behavior altogether. Obviously the former is preferable, since it's not trivial to reimplement yourself. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UZ77WPWH4DK3IK3ANKQYOCXKVW4AERIE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622: Structural Pattern Matching -- followup
Pablo Galindo Salgado wrote:
> ...users can do a positional match against the proxy with a name pattern:
>
> match input:
> case datetime.date(dt):
> print(f"The date {dt.isoformat()}"
>
> ...if 'datetime.date' were updated to implement a non-default __match_args__,
> allowing individual fields to be pulled out of it like this, then the first
> block would be valid, correct code before the change, but would raise an
> ImpossibleMatch after the change because 'dt' is not a field in
> __match_args__. Is this argument misinterpreting something about the PEP or
> is missing some important detail?
Well yeah, it's actually a fair bit worse than you describe. Since dt is
matched positionally, it wouldn't raise during matching - it would just succeed
as before, but instead binding the year attribute (not the whole object) to the
name "dt". So it wouldn't fail until later, when your method call raises a
TypeError.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/Y7DJJA2ONIRJSMMA6PYKAYCZSYIECY4D/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622: Structural Pattern Matching
Guido van Rossum wrote: > So why not .get(key, )? You can reuse the sentinel, and this way > it's a single call instead of two -- e.g. the code in Mapping implements both > __contains__() and get() by calling __getitem__() and catching KeyError. Good point. Another option I've considered is using the `keys` method, since most non-dict mappings would get this called anyways for patterns with `**rest`. All the more reason why we should allow the implementation some flexibility. ;) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RLBJKQNWFOTCBSAJT47J7RO2K26ZFQAL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 622: Structural Pattern Matching
Guido van Rossum wrote: > IIUC the pattern matching uses either .get(key, ) or > .__contains__(key) followed by .__getitem__(key). Neither of those will > auto-add the item to a defaultdict (and the Mapping protocol supports both). > @Brandt: what does your implementation currently do? Do you think we need to > specify this in the PEP? I still prefer under-specifying rather than over-specifying, in order to allow the implementation some flexibility. Basically, "if you've got a funny mapping, it may behave funny". For example, we currently perform a length check before even getting to this point, which is allowed because the spec isn't very strict. This is a good thing; we already have a section in the PEP that explicitly allows compiler to perform transformations such as C(x) | C(y) -> C(x | y). If we have to specify it, __contains__ followed by __getitem__ is the way to do it. The current behavior is subtly different (and probably wrong), but I was going to change it today anyways... I had an outstanding TODO for it. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FGHL3HV3Z5W5DBURTE76WU4YI23IRZ3Z/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 618: zip(strict=True)
Woo! Many thanks to Ram for the idea, Antoine for sponsoring, Guido for PEP-Delegating, and everyone on -Ideas and -Dev for the spirited discussion and review. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/F6M3F2HZ5DMBPLERGACGKMUGVNRUMNP6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584 underspecified
Maybe I'm missing something here. The PEP specifically says: > Similarly, the iteration order of the key-value pairs in the dictionary will > follow the same semantics as the examples above, with each newly added key > (and its value) being appended to the current sequence. That seems sufficient, no? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NM75M5N7BJOHSG3ZELIVBUNOG7ZQ3VQ4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
In the last 24 hours, this thread has grown a bit beyond my capacity to continue several different lines of discussion with each individual. I count 22 messages from 14 different people since my last reply, and I assure you that I've carefully read each response and am considering them as I work on the next draft. I'd like to thank everyone who took the time to read the PEP and provide thoughtful, actionable feedback here! Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7V7A2C4EXCEJD2Y2Q3OEQBZYKGJI5QVU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
Ethan Furman wrote: > Can you do those with _pydecimal? If performance were an issue anywhere I > would expect to see it with number crunching. No difference, probably because those methods look like they spend most of their time doing string manipulation: ``` $ export PYPERFSETUP='from _pydecimal import Decimal; from random import getrandbits; l = Decimal(bin(getrandbits(28))[2:]); r = Decimal(bin(getrandbits(28))[2:])' $ export PYPERFRUN='l.logical_and(r); l.logical_or(r); l.logical_xor(r)' $ ./python-master -m pyperf timeit -s "$PYPERFSETUP" "$PYPERFRUN" Mean +- std dev: 53.4 us +- 2.8 us $ ./python-zip-strict -m pyperf timeit -s "$PYPERFSETUP" "$PYPERFRUN" Mean +- std dev: 53.8 us +- 2.5 us $ ./python-zip-strict -m pyperf timeit -s "$PYPERFSETUP" "$PYPERFRUN" # This time, with strict=True in each method. Mean +- std dev: 53.6 us +- 3.0 us ``` I would encourage those who are still curious to pull the branch and experiment for themselves. Let's try to keep this a design discussion, since we've established that performance isn't a problem (and there is plenty of time for code review later). > Paul Moore and Chris Angelico have made good arguments in favor of an > itertools addition which haven't been answered yet. I don't consider their arguments particularly strong, but yeah, I was getting to those. I wanted to address your points first since you weren't part of the Ideas discussion! Paul Moore wrote: > ... so it's another beast because (among other reasons) it lives in a > separate namespace, and it should live in a separate namespace because it's > another beast? That's circular logic. Sorry, that's on me for trying to respond to two questions with one answer right before bed. Strike the namespace argument, then. The rest stands. > So importing zip_strict from itertools is an entirely reasonable way for > users to enable the check, then. Still agreed. But I think they would be *better* served by the proposed keyword argument. This whole sub-thread of discussion has left me very confused. Was anything unclear in the PEP's phrasing here? If so, I'd like to improve it. The original quote is: "The goal here is not just to provide a way to catch bugs, but to also make it easy (even tempting) for a user to enable the check whenever using `zip` at a call site with this property." > It's very easy to suggest bad ways of using a feature. That doesn't make the > feature bad. You seem to be arguing that zip_strict is bad because people can > misuse it. Well, I addressed this "irrelevant" point because right out of the gate people started suggesting that they want a separate function *because* it makes shadowing easy. Which brings me to my next quote: Chris Angelico wrote: > I am most in favour of the separate-functions option *because* it makes > shadowing easy. Not an anti-pattern at all. I *really* hope this isn't how people use this (and I don't *think* it would be predominantly used this way), but at least it's clear to me now why you want it to be a separate function. It would still be quite simple to follow this pattern, though, with `functools.partial` or a custom wrapper. > Python is *deliberately* designed so that you can shadow things. I wouldn't confuse "can" and "should" here. Python is deliberately designed to make *many* design patterns possible, good and bad. > And considering that "from __future__ import print_function" is an > officially-sanctioned way to cause a semantic change to print, I don't think > it's really that strong an argument. Well that's a parser directive that is just there for 2/3 compatibility (I'm pretty sure - I've never used Python 2). I see it as very, very different from my `from pprint import pprint as print` headache that was quoted two levels up. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/67O5FJA5MB26WWAC7VV5IUQMEKMLCNYK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
Ethan Furman wrote: > So half of your examples are actually counter-examples. I claimed to have found "dozens of other call sites in Python's standard library and tooling where it would be appropriate to enable this new feature". You asked for references, and I provided two dozen cases of zipping what must be equal length iterables. I said they were "appropriate", not "needed" or even "recommended". These are call sites where unequal-length iterables, if encountered, would be an error that I would hope wouldn't pass silently. Besides, I don't think it's beyond the realm of imagination for a future refactoring of several of the "Mismatch cannot happen." cases to introduce a bug of this kind. > Did you vet them, or just pick matches against `zip(`? Of course. I spent hours vetting them, to the point of researching the GNU tar extended sparse header and Apple property list formats (and trying to figure out what the hell was happening in `os._fwalk`) just to make sure my understanding was correct. Ethan Furman wrote: > Not the call itself, but the running of zip. Absent some clever programming > it seems to me that there are two choices if we have a flag: I wouldn't call my implementation "clever", but it differs from both of these options. We only need to check if we're strict when an error occurs in one of our iterators, which is a situation the C code for `zip` already needs to explicitly handle with a branch. So this condition is only hit on the "last" `__next__` call, not on every single iteration. As a reminder, the actual C implementation is linked in the PEP (there's no PR yet but branch reviews are welcome), though I'd prefer if the PEP discussion didn't get bogged down in those specifics. The pure-Python implementation in the PEP is *very* close to it, but it uses different abstractions for some of the details regarding error handling and argument parsing.[0] However, for those who are interested, there is no measurable performance regression (and no additional parsing overhead for no-keyword-argument calls). Parsing the keyword argument (if present) adds <0.2us of overhead at creation time on my machine. I went ahead and ran some rough PGO/LTO benchmarks: Creation time: ``` $ ./python-master -m pyperf timeit 'zip()' Mean +- std dev: 79.4 ns +- 4.3 ns $ ./python-zip-strict -m pyperf timeit 'zip()' Mean +- std dev: 79.0 ns +- 1.9 ns $ ./python-zip-strict -m pyperf timeit 'zip(strict=True)' Mean +- std dev: 240 ns +- 8 ns ``` Creation time + iteration time: ``` $ ./python-master -m pyperf timeit -s 'r = range(10)' '[*zip(r, r)]' Mean +- std dev: 577 ns +- 35 ns $ ./python-zip-strict -m pyperf timeit -s 'r = range(10)' '[*zip(r, r)]' Mean +- std dev: 565 ns +- 16 ns $ ./python-zip-strict -m pyperf timeit -s 'r = range(10)' '[*zip(r, r, strict=True)]' Mean +- std dev: 756 ns +- 27 ns $ ./python-master -m pyperf timeit -s 'r = range(100)' '[*zip(r, r)]' Mean +- std dev: 3.54 us +- 0.14 us $ ./python-zip-strict -m pyperf timeit -s 'r = range(100)' '[*zip(r, r)]' Mean +- std dev: 3.49 us +- 0.07 us $ ./python-zip-strict -m pyperf timeit -s 'r = range(100)' '[*zip(r, r, strict=True)]' Mean +- std dev: 3.73 us +- 0.13 us $ ./python-master -m pyperf timeit -s 'r = range(1000)' '[*zip(r, r)]' Mean +- std dev: 44.1 us +- 2.0 us $ ./python-zip-strict -m pyperf timeit -s 'r = range(1000)' '[*zip(r, r)]' Mean +- std dev: 45.2 us +- 2.0 us $ ./python-zip-strict -m pyperf timeit -s 'r = range(1000)' '[*zip(r, r, strict=True)]' Mean +- std dev: 45.2 us +- 1.4 us ``` Additionally, the size of a `zip` instance has not changed. Pickles for non-strict `zip` instances are unchanged as well. Brandt [0] And zip's current tuple caching, which is *very* clever. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZVCBS2S7IRIEU346AAJUEVH45VNMVOKI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
Thanks for all of your feedback. Antoine Pitrou wrote: > I'm not sure what the iters bring here. The snippet would be more readable > without, IMHO. Good point. I was trying to demonstrate that it works with iterators, but I agree it's clearer to just use the lists here. Ethan Furman wrote: > > Many Python users find that most of their zip usage > I don't think you have enough data to make that claim, unless by "many" you > mean five or more. It's based on a combination of my own experience, the experiences of several others, and a survey of the CPython repo. I can dial back the wording, though, since this isn't necessarily representative of the larger userbase... > > but silently start producing shortened, mismatched results if items is > > refactored by the caller to be a consumable iterator > This seems like a weak argument; static type checking could catch it. Well, that's why I go on to make stronger, non-toy ones immediately after. :) This is mainly just to introduce the problem in an easy-to-understand way. > > the author has counted dozens of other call sites in Python's standard > > library > References, please. Here are two dozens: - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/_pydecimal.py#L3394 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/_pydecimal.py#L3418 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/_pydecimal.py#L3435 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L94-L95 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1184 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1275 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1363 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/ast.py#L1391 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/copy.py#L217 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/csv.py#L142 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/dis.py#L462 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/filecmp.py#L142 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/filecmp.py#L143 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/inspect.py#L1440 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/inspect.py#L2095 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/os.py#L510 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/plistlib.py#L577 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/tarfile.py#L1317 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/tarfile.py#L1323 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/tarfile.py#L1339 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/turtle.py#L3015 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/turtle.py#L3071 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/Lib/turtle.py#L3901 - https://github.com/python/cpython/blob/27c0d9b54abaa4112d5a317b8aa78b39ad60a808/setup.py#L455 I'll go ahead and link these in the PEP. > > A good rule of thumb is that "mode-switches" which change return types or > > significantly alter functionality are indeed an anti-pattern, > Source? This was based on a chat with someone who has chosen not to become involved in the larger discussion, and it was lifted almost verbatim from my notes into the draft. Looking at it again, though, I don't think this sentence belongs in the PEP... it really shouldn't be prescribing design philosophies like this. > > while ones which enable or disable complementary checks or behavior are not. > None of the listed examples change behavior between "working" and "raising > exceptions", and none of the listed examples are for "complementary checks". Thanks for pointing this out. If I keep this bit, I'll include some other examples from the stdlib that specifically behave this way. > > At most one additional item may be consumed from one of the iterators when > > compared to normal zip usage. > How, exactly? I'm actually considering just leaving this line out, too. We don't currently make any promises about how "extra" items are drawn (just that they are drawn left-to-right, which is still true here), so I don't think this needs to be in the spec. > > However, zip_longest is really another beast entirely > No, it isn't. It has a completely
[Python-Dev] PEP 618: Add Optional Length-Checking To zip
I have pushed a second draft of PEP 618: https://www.python.org/dev/peps/pep-0618 Please let me know what you think – I'd love to hear any new feedback that hasn't yet been addressed in the PEP! Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6HTDN4L5PFMOYMUXLTZCGGJYDYMHJGHC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Question about bytecode stability
> The best way to look at this is to consider how long a .pyc file is valid. > They're currently named something like __pycache__/modulename.cpython-38.pyc > which is a fairly clear indication that the cached compiled module should be > valid for any CPython 3.8.x release. Perhaps an even better indicator is the “magic number” that invalidates the cached bytecode. This can change several times during the alpha and beta phases, and a (mostly complete) record of changes can be found here: https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py#L147-L284 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NKNIMQCET2OKNUY6MCYMBFYVGJY7ZN7L/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 614 accepted
Welcome Bhavani! This is only my second PEP, but I'm sure somebody will correct me if I'm wrong! Basically, the next steps are: - Mark the PEP as "Accepted": https://github.com/python/peps/commit/841188a45f17f0252267f016d7b2f11ac27c5aa1 - Review and merge the implementation/tests: https://github.com/python/cpython/pull/18570 - Add documentation: (in progress, sometime today) - Mark the PEP as "Final": (last step) PEP 1 (the meta-PEP), outlines the PEP lifecycle quite well: https://www.python.org/dev/peps/pep-0001/#pep-review-resolution Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Y3BVAS6MU22GTJHSETKLJFJYJJQFGHVT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 614 accepted
Thanks to Guido for sponsoring, and to the SC for their acceptance! Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/F3LVBGPDLB26QFJEKOLSPOHJ6DG7LWKJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 584: Add Union Operators To dict
Thanks for your response Serhiy.
> Yes, but this is a different thing. You must to implement new __or__ in a
> subclass to make the behavior of d1 | d2 be different from {**d1, **d2}.
Really? I'm sorry, but this doesn't feel like a real argument to me.
There was no clear compatible path before. Now there is. It works by default,
returning a dict for dict subclasses. If you want to change the return type (or
do something else), you do have to do a little work. Same for every other
overridden thing in any other subclass.
Besides, our argument was just the opposite: "you can't implement anything in a
subclass to make the type of {**d1, **d2} be different".
Now that we've provided a way, it's a problem because you have to define custom
behavior (if you want it)?
> you want an operation for merging OrderedDict, defaultdict, etc and add
> dict.__or__ just for symmetry, even if it is not actually necessary for dicts.
Except that now every dict subclass gets this for free. And now everything
quacks alike.
Are you really suggesting that it would make more sense to implement this for
all subclasses, but not dict itself? I don't think you are, but it sure sounds
like it.
> Let's take a closer look at these examples.
I appreciate that you've taken the time to go through each one, but please note
that the PEP finishes with:
> The above examples show that sometimes the | operator leads to a clear
> increase in readability... However other examples using the | operator lead
> to long, complex single expressions... As with any other language feature,
> the programmer should use their own judgement about whether | improves their
> code.
We're not suggesting that all of the examples would be better, or even
equivalent with the new code. During the discussions last year, people kept
asking for before-and-after examples. We decided to extract real examples
rather than fabricate toys, so for the purposes of this PEP you can assume that
they stand on their own, void of context.
For example, I considered removing the `globs` example because, like you, I
thought it was uglier. However, I figured that it would be wrong to omit it,
since the goal of the section was to show as many candidates for the operator
that we could trivially find, not "all of the ones that Brandt likes better
with |". :)
I'm not sure that continuing this thread of discussion is very productive. As I
have mentioned before, the huge PEP is roughly:
- 50% responses to arguments.
- 35% examples.
- 10% specification.
- 5% links to prior discussions.
We've addressed anything that we felt needed addressing, and they have been
enough to persuade the necessary decision makers. Many of the points you make
have been responded to in some way or another in this monster of a document. :)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/RHFTVX7IV7VG3GLEVE4ZWOIHTCB2JQBK/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 584: Add Union Operators To dict
> Are these three cookie classes sufficiently popular that we need to support > `|` on them? I don't have much experience with `http`, so I figured I'd open a BPO issue and let the relevant folks make the call. The main reason I'm considering these is that `Morsel` was updated back in 3.5 with `copy` and `update` overrides: https://bugs.python.org/issue2211 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NWJIVQ35JYM2VAG6JHTNWPHZKAAM2GJQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 614: Relaxing Grammar Restrictions On Decorators
Happy to help. Since it doesn't seem like anybody has issues with the PEP, and the PR looks solid, I think I'll contact the Steering Council about moving this forward. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4WVJPAG5PCWFF3JWC54LA2QSR2NANZBF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 584: Add Union Operators To dict
> we'll also want to look at collections.UserDict, collections.ChainMap, and > types.MappingProxyType. UserDict has already been updated in the PR (it has tests that fail if it doesn't have all of dict's methods). I'll look into the others... thanks for reminding me! > collections.Mapping and collections.MutableMapping could provide concrete > method implementations that make subclasses behave in a way that's similar to > built-in dicts Hm, haven't thought too much about this (I don't have much experience with the ABCs). Would they return dicts, or call the self.copy and self.update methods? Those are just hypothetical questions for now; I don't necessarily want to dig too far into that discussion again. But I agree that it's definitely worth considering. ;) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QRX3NQPWTHFQFSQR6NJ54N2IKRGL3VET/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 584: Add Union Operators To dict
Just to follow up on this, here are the subclasses I've found. Should be updated: - collections.OrderedDict - collections.defaultdict - http.cookies.BaseCookie - http.cookies.Morsel - http.cookies.SimpleCookie Don’t think so: - typing.TypedDict Already defines the operator: - collections.Counter Not Public (or at least not documented): - _strptime.TimeRE - builtins.StgDict (this one's only available internally in _ctypes). - email._encoded_words._QByteMap - enum._EnumDict - logging.config.ConvertingDict - multiprocessing.pool._PoolCache - urllib.parse.Quoter ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BRNLVR4G7TDG7Z2BQMURCSYSTFHDXYW5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Three trivial PRs from first-timers in need of merging!
All of these have been landed. Thanks, everybody! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GU6AWXQP4KIYMRSHPLGSNOATLR4RBTAK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Three trivial PRs from first-timers in need of merging!
Hi everybody! I've found a handful of trivial PRs from new contributors that have gone un-core-reviewed since November. CLAs are signed, tests are passing, and I've verified the quality and correctness of each one. If anybody has a few spare minutes to merge these, I know that it would make the authors very happy! :) Clarify numeric padding behaviour in string formatting: - A helpful doc addition (just a few words). - https://github.com/python/cpython/pull/17036 argparse unittest tracebacks are confusing if an error is raised when not expected: - This just changes a "raise" to a "raise from None" in the argparse unit test machinery. - https://github.com/python/cpython/pull/17120 Reuse identifier of PREDICT macros as PREDICT_ID: - Unifies some shared naming logic in the ceval prediction macros (don't worry, it's simple). - https://github.com/python/cpython/pull/17155/files Thanks. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/46U2H5U2BAACZZETIWJTAJAGXJSD6MKO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 614: Relaxing Grammar Restrictions On Decorators
PEP 614 has recently completed a round of review on Python-Ideas: https://www.python.org/dev/peps/pep-0614/ It proposes that the current decorator syntax restrictions be relaxed to allow any valid expression. Nobody has raised any objections, but I wanted to gather more feedback here prior to bringing it before the Steering Council. Please let me know what you think... thanks! :) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SLKFAR56RA6A533O5ZOZ7XTJ764EMB7I/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 584: Add Union Operators To dict
> In particular, you may want to update the implementations of defaultdict and > ordereddict. (are there others?) I have a checklist for PEP 584 follow-up PRs I'm planning to make after (and if) it is accepted. typeshed stubs, docs, and `defaultdict` were on it. I'll add `OrderedDict` as well (for some reason I didn't think it was a `dict` subclass). `Counter` already defines this operator for other purposes, so no need to do anything there. I'm not aware of other public subclasses, but I may just try importing the whole stdlib and recursing through `dict.__subclasses__()` just to be sure. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3SJTNSTTZ4G5CXQQWNWRBY7WZHEWBYEY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 584: Add Union Operators To dict
> I am accepting this PEP. Congratulations Steven and Brandt!
Thank you for your guidance, especially the suggestions late last year. And
thanks Steven for taking me on as a co-author and shaping the bulk of the
proposal.
> Hm, the PEP should probably also link to that PR rather than to Brandt's
> private branch.
Agreed. This can probably be changed when the status is updated.
Now, addressing Serhiy's points :)...
> ...it was decided that `d1 | d2` also should ignore the types of the operands
> and always return a dict. And it accepts only dicts, not general mappings, in
> difference to `{**d1, **d2}`. So the only disadvantage of `{**d1, **d2}` is
> that it is not well known and "looks ugly".
Not quite. While this point *has* been weakened a bit with the recent semantic
change, you don't mention that `dict` subclasses can (and likely would)
override the `__or__` trio with wrapped `super()` calls. So while `{**d1,
**d2}` will *never* be anything but a `dict`, I can trust that well-written
`dict` subclasses my code encounters will still be able to preserve themselves
with `d1 | d2`, if desired.
> The pure-Python implementation of the non-inplace operator can be simpler if
> use dict unpacking.
The purpose of the pure-Python implementation in the PEP is to be a clear,
understandable example that serves as a codification of the semantics, not to
be the shortest one-liner. The suggested changes make the code less readable,
making it harder to see exactly what will happen if I do `self | other`. I'm
against changing it, for that reason.
> I suggest to include to objections that non-inplace merging of two dicts is
> much less common operation than concatenating of two strings or other
> sequences, and that the existing syntax {**d1, **d2} completely satisfies the
> need.
This is really two objections.
Regarding the commonality of this operation, we've provided eighteen detailed
examples of third-party library code that are candidates for these new
operators. To the best of my knowledge, a large survey like this is
unprecedented in a PEP. Whether or not it is more common than a different
operation on other types isn't relevant here. We have gone above and beyond in
demonstrating the use cases in detail.
A rewording of your second objection has already been addressed:
https://www.python.org/dev/peps/pep-0584/#more-than-one-way-to-do-it
I feel that I've adequately responded to the other points you've raised here in
this email (please let me know if I missed anything, and perhaps Steven may
want to chime in as well). I don't know what the policy for changing the PEP is
once it's gotten to this stage, but it's already grown to a huge size in its
attempts to address every possible concern that has arisen over the past year
(even some that, in my opinion, weren't serious proposals that needed to be
acknowledged). I'd prefer not to bloat it even more this late in the game, if
at all possible.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/TEC4G6LVTHFL566ELDC4J6S527LI5C37/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> So can we just finish PEP 584 without the .copy() call? Sounds good. I’ll update the implementation and open a PR to the PEPs repo later tonight. Thanks everybody! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IPSPMBI6YKWWLVJJT7V2O5X7HHMPD7FJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
After a few days of thinking and experimenting, I’ve been convinced that `copy` (and also `__copy__`) is not the right protocol for what we want to do here. I believe that 584 can likely continue without subclass-preserving behavior, but that better behavior could perhaps could be added to *all* built-in types later, since it’s outside the scope of this PEP. > My opinion is that Python built-in types make subclassing unnecessarily(?) > awkward to use, and I would like to see that change. Yes! But, on further reflection, I don’t think this is the correct way of approaching it. > For example, consider subclassing float. If you want your subclass to be > actually usable, you have to write a whole bunch of boilerplate, otherwise > the first time you perform arithmetic on your subclass, it will be converted > to a regular old float… This is painful and adds a great amount of friction > to subclassing. `float` is a *perfect* example of the problems with the way things are currently, so let’s focus on this. Currently, subclassing `float` requires ~30 overridden methods of repetitive (but non-trivial) boilerplate to get everything working right. However, calling the `float` equivalent of `dict.copy()` on the LHS before proceeding with the default implementation wouldn’t help us, because floats (like many built-in types) are immutable. So a new, plain, built-in `float` would still be returned by the default machinery. It doesn’t know how to construct a new, different instance of our subclass, and it can’t change one it’s already built. This leads me to believe that we’re approaching the problem wrong. Rather than making a copy and working on it, I think the problem would be better served by a protocol that runs the default implementation, *then* calls some under hook on the subclass to build a new instance. Let’s call this method `__build__`. I’m not sure what its arguments would look like, but it would probably need at least `self`, and an instance of the built-in base class (in this case a `float`), and return a new instance of the subclass based on the two. It would likely also need to work with `cls` instead of `self` for `classmethod` constructors like `dict.fromkeys`, or have a second hook for that case. By subclassing `float` and defining `__build__` to something like this: ``` class MyFloat(float): … def __build__(self, result): Return MyFloat(result, some_state=self.some_state) … ``` I could now trust the built-in `float` machinery to try calling `lhs.__build__(result)` on the result that *would* have been returned *before* returning it. This is a simple example, but a protocol like this would work for mutables as well. > A more pertinent example, from dict itself: If `dict` *were* to grow more operators, they would likely be `^`, `&`, and `-`. You can consider the case of subclassing `set` or `frozenset`, since they currently has those. Calling `lhs.copy()` first before updating is fine for additive operations like `|`, but for subtractive operations like the others, this can be very bad for performance, especially if we’re now *required* to call them. Again, doing things the default way, and *then* constructing the appropriate subclass in an agreed-upon way seems like the path to take here. > Changing all builtins is a big, backwards-incompatible change. If implemented right, a system like the one described above (`__build__`) wouldn’t be backward-incompatible, as long as nobody was already using the name. Just food for thought. I think this is a much bigger issue than PEP 584, but I'm convinced that the consistent status quo should prevail until a suitable solution for all types can be worked out (if ever). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7HNJ6RVVEX2VD37HYR5F5P5W2YAOMPDH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> > We already have an API for copying a dict (dict.copy). I still fail to see > > problem with using a method that doesn't start and end with underscores, > > other than that we "haven't done it". > > Before Python 3.0, iterators had a next() method, and that was explicitly and > consciously changed to __next__(). The arguments there seem relevant here too. Thanks for bringing this up. PEP 3114 does a great job of explaining how and why Python uses dunders for *language-level* constructs. I would probably be opposed to language features or built-in functions doing magical things with the `copy` method of dicts; however, in this case, it's being used by the dict itself! As the PEP 3114 says: > In Python, double underscores before and after a name are used to distinguish > names that belong to the language itself... Not all things that are called > "protocols" are made of methods with double-underscore names... even though > the read method is part of the file protocol, it does not have double > underscores because there is no language construct that implicitly invokes > x.read(). The language itself doesn't call `copy`, but `dict` should feel free to. Especially if it makes life easier for subclasses (which PEP 584 actually encourages users to make in the "Rejected Semantics" section: https://www.python.org/dev/peps/pep-0584/#rejected-semantics)! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FIPVZTUM4CJ5WJWAAYGS4E55Y7AP7J6E/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
Sorry Paul, I sent my reply too soon. I see what you're saying, and I'm pretty firmly -1 on reinventing (or importing) copy.copy. We already have an API for copying a dict (dict.copy). I still fail to see problem with using a method that doesn't start and end with underscores, other than that we "haven't done it". Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7TQI54BEUN6GERZA3Y2GCXPBD3CHXKM6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> but I think the desired semantics can all be achieved using only magic > methods on the objects themselves Hm. So, just to clarify, you're suggesting we use `__copy__`, if it exists? Interesting... ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZEB2VAVPGWHL6B55UAMYYQRALLILGQLE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> It would create an exception of two rules: I don't think these are "rules", I think they're just "the way things are". If I'm subclassing `dict`, and I see in the docs something to the effect of: > By default, `dict` subclasses will return `dict` objects for `|` operations. > To force the creation of a new instance of the subclass, users can override > the `copy` method. In that case, the return value from this method will be > used instead. Then my life suddenly becomes a lot better, because chances are I've already thought to override `copy`. And if I want the "legacy" behavior, it's as simple as not bothering with "copy" (or, if I need to, overriding the `__or__` trio)... but I'm sure this is the less common case. If we're quoting the Zen, then let's not elevate past design patterns to "rules". Besides, practicality beats purity. ;) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L5ODZ2TEEY2T53C5KB25WJPFGJAU7IIP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> I see this asymmetry between the | and |= mentioned a few times in the PEP, > but I don't see any rationale other than "the authors have decided". I agree that this probably deserves to be addressed. I can't speak for Steven, but I know that my motivation here is to restrict `|` in order to avoid confusing casting. It suddenly becomes a very complicated problem with strange edge-cases as soon as reflected operations get involved. In contrast, `|=` appears in dramatically simpler contexts, so we don't need to worry about resolving all of the combinations of possible LHS and RHS types (or whether the user can keep up with what we're doing under-the-hood). I have no idea if this is what motivated the decision in `list`, but I think it's a good one. > The specification mentions "Dict union will return a new dict containing the > left operand merged with the right operand, which must be a dict (or an > instance of a dict subclass)." Can you clarify if it is part of the spec that > it will always return a dict even if one or both of the operands is a dict > subclass? See my recent post on this here. I believe that we should call an overridden `lhs.copy()`, but others disagreed during code review. For what it's worth, it's probably not good for us to use "dict" and "`dict`" (with monospace/backticks) to mean different things in the PEP... ;) > Unless there is compelling reason to do otherwise, I am in favor of trying to > retain subclass identity after operation. I'll count you as a vote for `new = lhs.copy(); dict.update(new, rhs)`, then. ;) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MRYKAFWIJFODH7PBVH5CKPWAS2M26VFO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
> We should fix that (by reverting to 3.8.1 behaviour) before 3.8.2 gets > released. The commits which changed the behavior were bytecode/compiler changes that only went to master. I don't think they are present on any other branches. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GC3PGIGXFY47RQVOA4HLE3VX5IQ2VGTB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
One issue that's come up during PR review with Guido and Serhiy is, when
evaluating `a | b`, whether or not the default implementation should call an
overridden `copy()` method on `a` in order to create an instance of the correct
subclass (rather than a plain-ol' `dict`). For example, `defaultdict` works
correctly without any additional modification:
```
>>> from collections import defaultdict
>>> i = defaultdict(int, {1: 1, 2: 2})
>>> s = defaultdict(str, {2: "2", 3: "3"})
>>> i | s
defaultdict(, {1: 1, 2: '2', 3: '3'})
>>> s | i
defaultdict(, {2: 2, 3: '3', 1: 1})
```
So this has immediate benefits for both usability and maintenance: subclasses
only need to override `copy()` to get working `__or__`/`__ror__` behavior.
While this isn't what `list` and `set` do, for example, I argue that `dict` is
more often subclassed, and we shouldn't blindly follow the precedent of their
behavior when designing a new API for `dict`.
We decided to bring the discussion here to get input from a larger audience.
I'm currently +0 on calling `copy()`, but I know Steven feels a bit more
strongly about this than I do:
> I think the standard handling of subclasses in Python builtins is wrong, and
> I don't wish to emulate that wrong behaviour without a really good reason. Or
> at least a better reason than "other methods break subclassing unless
> explicitly overloaded, so this should do so too". Or at least not without a
> fight :-)
The more detailed thread of discussion starts at
https://github.com/python/cpython/pull/12088#issuecomment-582609024 (note that
we are no longer considering calling an overridden `update` method).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/Z5SSGCH736UWHKFGJEZROH6VKPKXIT7W/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Clarification of unpacking semantics.
Commits 13bc139 and 8a4cd70 introduced subtle changes in the evaluation logic of unpacking operations. Previously, all elements were evaluated prior to being collected in a container. Now, these operations are interleaved. For example, the code `[*a, *b]` used to evaluate in the order `a` -> `b` -> `a.__iter__()` -> `b.__iter__()`. Now, it evaluates as `a` -> `a.__iter__()` -> `b` -> `b.__iter__()`. I believe this breaking semantic change is a bug, and I've opened a PR to fix it (https://github.com/python/cpython/pull/18264). My reasoning is that "unary *" isn't an operator; it doesn't appear on the operator precedence table in the docs, and you can't evaluate `*x`. Like the brackets and the comma, it's part of the syntax of the outer display expression, not the inner one. It specifies how the list should be built, so it should be evaluated last, as part of the list construction. And it has always been this way since PEP 448 (as far as I can tell). The docs themselves seem to support this line of reasoning (https://docs.python.org/3/reference/expressions.html#evaluation-order): > In the following lines, expressions will be evaluated in the arithmetic order > of their suffixes: > ... > expr1(expr2, expr3, *expr4, **expr5) Note that the stars are not part of expressions 1-5, but are a part of the top-level call expression that operates on them all. Mark Shannon disagrees with me (I'll let him reply rather than attempt to summarize his argument for him), but we figured it might be better to get more input here on exactly whether you all think the behavior should change or not. You can see the discussion on the PR itself for some additional points and context. Thanks! Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4HS2ZEQWWFMQ7IO5ZHBNWEYLN4PIYDCD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> I have one suggestion: Wouldn't it be useful for these operators to also > accept sets (functionally acting like a dict with None for all values)? > Why None? Why not 0, or False, or 42? This sort of thing belongs more in a > function or method, IMHO. Well, in their defense, None is the null object in Python, so it would be a natural choice. With that said, I am strongly against this for several reasons: - The proposal in its current form is very easy to wrap your head around: "|" takes dicts, "|=" takes anything dict.update does. - Python doesn't allow sets to be dictionaries anywhere else. It's much more natural to use a dict like a set than to use a set like a dict. Hence, this PEP! - A consistent argument that we've gotten since the very beginning is something to the effect of "I don't know what types are being used in the expression 'a | b'." While we argue that this isn't the issue in practice that people make it out to be, accepting sets here would definitely muddy the waters. > This would make it very elegant to 'normalize' dicts by pruning (dict & set) > or padding (set | dict) dictionaries. Well, if the PEP lands, the padding part is easy: padded = dict.fromkeys(expected) | given Or even better, just do the dict.fromkeys construction at module level once, and do: padded = EXPECTED | given Or, implement the behavior in a subclass if you truly need it to be a set! We're intentionally very subclass-friendly (unlike list, for example). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/G3NWWUFR3SJECCXIAUGHNEJLSCBQBEVG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
It already has a PR open against master, with all tests passing: https://github.com/python/cpython/pull/12088 Sorry about the messy history - this proposal has changed significantly several times over the past year (at least as far as the implementation is concerned). At one point, both operators were implemented for comparison with each other! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6DVUQ45EFTX3G5GF6SOZCYZK62GYFQ7V/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 584: Add Union Operators To dict
Steven D'Aprano and I have pushed a third draft of PEP 584: https://www.python.org/dev/peps/pep-0584/ The accompanying reference implementation is on GitHub: https://github.com/brandtbucher/cpython/tree/addiction For those who have been following the discussions over the past year on python-ideas, this new draft does not contain much additional content; the most notable difference is that the choice of operator has been changed from + to |. The rest of the revisions are mostly reformatting and reorganizing the information to bring the document in line with PEP standards. Please let us know what you think – we'd love to hear any *new* feedback that hasn't yet been addressed in the PEP or the related discussions it links to! Thanks! Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TTIKCDIPC2CDHX23Y57CPHDSVYOWCCER/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should set objects maintain insertion order too?
> On Dec 27, 2019, at 19:48, Tim Peters wrote: > > So, ya, I've seen and implemented lots of work queues along these > lines - but an OrderedSet would be an "attractive nuisance" > (offhandedly appearing to solve a problem it doesn't actually > address): > > jobs = some_kind_of_queue() > finished_jobs = set() > ... > while jobs: > job = jobs.get() > if job in finished_jobs: > continue > try: > work on the job > possibly adding (sub)jobs to the `jobs` queue > except TransientExceptions: > jobs.put(job) # try again later > else: > finished_jobs.add(job) Well, if an OrderedSet were designed to gracefully handle resizes during iteration, something like this may make sense: jobs = OrderedSet(initial_jobs) for job in jobs: new_jobs = process(job) jobs |= new_jobs ... # jobs is now a set of every job processed A dictionary with None values comes close if you replace the union line with a jobs.update(new_jobs) call (and ignore resizing issues), but it breaks because repeated jobs are shuffled to the end of the sequence and would be processed again. Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SXGD5G5EY4YMXDG42AU7OSNVCUUU25DI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Helpers for dynamic bytecode generation
Interesting that you bring this up. Just earlier this week I published the first version of a new package that lets you write compiled bytecode instructions inline with pure-Python syntax. The code's still a bit messy, being only a week old and all, but it works as advertised for CPython 3.6.2 through 3.9.0a0, and even includes neat features like labeled jumps, unused name/constant removal, stack size adjustments, etc... Perhaps it'll be useful to you (or at least you'll find it interesting): https://github.com/brandtbucher/hax Victor's Stinner's Bytecode package (already mentioned) is surely better for *dynamic* generation... I've never used it personally, but it looks great. Definitely not for the faint of heart, though! ;) Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PAQUZZ4MHZ6TJFIBNCPZGZBW6DOZFUJG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: 4 first-time contributions that need core review.
Thanks everybody for the quick response! These all now have core review (from Steering Council members, no less)! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NTOCTGY2XR6AMGAPTUGA5CFM5LCFBZUS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] 4 first-time contributions that need core review.
Hi all. There are a few *very simple* PRs from first-time contributors that have been sitting un-core-reviewed for several weeks. https://github.com/python/cpython/pull/16680: bpo-38419: fix "check-c-globals" path https://github.com/python/cpython/pull/16678: bpo-38421: Update email.utils documentation https://github.com/python/cpython/pull/16557: bpo-38361: syslog: fixed making default "ident" from sys.argv[0] https://github.com/python/cpython/pull/16438: bpo-38293: Allow shallow and deep copying of property objects I've reviewed them and they look good. If somebody has a small bit of time to look at these, I'm sure the authors would appreciate it! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BT5QELS56DHDXUPQVU6NFBFXB3EAXHJ2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: EnHackathon 2019 - seeking core dev support
Hi Lewis. I'm in the Pacific time zone. Barring any surprises (I'm getting married late next month), I can commit to being online in the morning (the last few hours of your business day), as well as during my normal business hours (late evening/night for you). Just tag "@brandtbucher" on any PRs you'd like reviewed. You can also tag or email me directly with any questions you have. It would also help if you could nosy "brandtbucher" on any bpo issues you plan on working on. Looking forward to getting some new contributors! Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Z7SGJRZA5TAMN2O6VNFVPVA4XLNMVMFR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: EnHackathon 2019 - seeking core dev support
Hi Lewis. I'm a trigger (not a core developer), but I'd be happy to make myself available during that time for expedited PR reviews. Ideally, the core review / merge stage would be your only bottleneck then. I'm a bit busy next month though, so do you know the actual dates / time / time zones you plan to be working in? Also, are there any specific issues you have identified that you want to work on within your stated areas of interest? If so, I can do some homework before in order to speed up the reviews. I'd recommend that all PR authors review the contributing section of the developers guide beforehand: https://devguide.python.org/pullrequest Specifically, make sure that every contributor has signed the CLA at least a couple of business days before (it's often the longest blocker for first-time contributions): https://devguide.python.org/pullrequest/#licensing Looking forward to your help! Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5GMVGDLVCTP67VFMUBQDLWNNP22EADAU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Migrate typing in builtin
I’m a bit confused. For my own understanding: what’s stopping
PyObject_IsInstance/PyObject_IsSubclass from just trying
PyImport_GetModule("typing") here?
If NULL, carry on. Otherwise, check the Union case.
Brandt
> On Oct 8, 2019, at 05:44, Philippe Prados wrote:
>
>
> Glups.
>
> I am not an expert of Pyhton source code. May be after this patch ;-)
>
> I think I should discuss with the authors of the module typing, to identify
> the best strategy.
> Who is he ?
>
> Philippe
>
>
>> Le mar. 8 oct. 2019 à 10:19, Ivan Levkivskyi a écrit :
>> You will need to rewrite most of things in C.
>>
>> --
>> Ivan
>>
>>
>>> On Tue 8 Oct 2019, 08:53 Philippe Prados, wrote:
>>> Ok,
>>>
>>> But _GenericAlias and dependencies are written with Python (Lib/typing.py),
>>> not with C.
>>> So, I must rewrite the _GenericAlias in C or it's possible to merge the C
>>> and Python in builtin and add a direct reference to _GenericAlias with C,
>>> and add the reference in builtin module ?
>>>
>>> Philippe
>>>
>>>
Le lun. 7 oct. 2019 à 22:58, Random832 a écrit :
On Mon, Oct 7, 2019, at 12:02, Philippe Prados wrote:
> Because this PEP propose to accept, for all classes
> assert isinstance("", int | str)
> assert issubclass(int, int | str)
> and add an operator __or__() for type type.
> def f(list: List[int | str], param: int | None) -> float | str:
> pass
Oh, sorry, I didn't realize that this also included the | operator, I
thought this was just for isinstance("", Union[int, str]).
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/GD7WXPD26VUPMZT6WAATCJJBB42DDYYQ/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4K7WZ3RBGG7K6E6XK65MS44VQYZIKQS2/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/RYGHVWJZT76GNDWNRSOLA74SHZFNDW42/
Code of Conduct: http://python.org/psf/codeofconduct/
