[Python-Dev] Re: PEP 584: Add Union Operators To dict
> > We already have an API for copying a dict (dict.copy). I still fail to see > > problem with using a method that doesn't start and end with underscores, > > other than that we "haven't done it". > > Before Python 3.0, iterators had a next() method, and that was explicitly and > consciously changed to __next__(). The arguments there seem relevant here too. Thanks for bringing this up. PEP 3114 does a great job of explaining how and why Python uses dunders for *language-level* constructs. I would probably be opposed to language features or built-in functions doing magical things with the `copy` method of dicts; however, in this case, it's being used by the dict itself! As the PEP 3114 says: > In Python, double underscores before and after a name are used to distinguish > names that belong to the language itself... Not all things that are called > "protocols" are made of methods with double-underscore names... even though > the read method is part of the file protocol, it does not have double > underscores because there is no language construct that implicitly invokes > x.read(). The language itself doesn't call `copy`, but `dict` should feel free to. Especially if it makes life easier for subclasses (which PEP 584 actually encourages users to make in the "Rejected Semantics" section: https://www.python.org/dev/peps/pep-0584/#rejected-semantics)! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FIPVZTUM4CJ5WJWAAYGS4E55Y7AP7J6E/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
How did we move from [*a,...] to print(*a,...)? They are quite different. On Thu, Feb 6, 2020 at 14:07 Serhiy Storchaka wrote: > 06.02.20 08:28, Brandt Bucher пише: > > Commits 13bc139 and 8a4cd70 introduced subtle changes in the evaluation > logic of unpacking operations. Previously, all elements were evaluated > prior to being collected in a container. Now, these operations are > interleaved. For example, the code `[*a, *b]` used to evaluate in the order > `a` -> `b` -> `a.__iter__()` -> `b.__iter__()`. Now, it evaluates as `a` -> > `a.__iter__()` -> `b` -> `b.__iter__()`. > > > > I believe this breaking semantic change is a bug, and I've opened a PR > to fix it (https://github.com/python/cpython/pull/18264). My reasoning is > that "unary *" isn't an operator; it doesn't appear on the operator > precedence table in the docs, and you can't evaluate `*x`. Like the > brackets and the comma, it's part of the syntax of the outer display > expression, not the inner one. It specifies how the list should be built, > so it should be evaluated last, as part of the list construction. And it > has always been this way since PEP 448 (as far as I can tell). > > > > The docs themselves seem to support this line of reasoning ( > https://docs.python.org/3/reference/expressions.html#evaluation-order): > > > >> In the following lines, expressions will be evaluated in the arithmetic > order of their suffixes: > >> ... > >> expr1(expr2, expr3, *expr4, **expr5) > > > > Note that the stars are not part of expressions 1-5, but are a part of > the top-level call expression that operates on them all. > > > > Mark Shannon disagrees with me (I'll let him reply rather than attempt > to summarize his argument for him), but we figured it might be better to > get more input here on exactly whether you all think the behavior should > change or not. You can see the discussion on the PR itself for some > additional points and context. > > I have two problems with this change. > > 1. It changes error messages. > > >>> print(*1) > Traceback (most recent call last): >File "", line 1, in > TypeError: print() argument after * must be an iterable, not int > >>> print(*1, *2) > Traceback (most recent call last): >File "", line 1, in > TypeError: Value after * must be an iterable, not int > > In 3.8 you got the same error message. > > >>> print(*1) > Traceback (most recent call last): >File "", line 1, in > TypeError: print() argument after * must be an iterable, not int > >>> print(*1, *2) > Traceback (most recent call last): >File "", line 1, in > TypeError: print() argument after * must be an iterable, not int > > I am not sure whether the function name is a useful information, but > some effort was spend to preserve it. In any case, error messages should > be consistent. > > > 2. It introduces performance regression. > > In 3.8 the bytecode for `(*a, *b, *c)` was: > >1 0 LOAD_NAME0 (a) >2 LOAD_NAME1 (b) >4 LOAD_NAME2 (c) >6 BUILD_TUPLE_UNPACK 3 > > In master it is: > >1 0 BUILD_LIST 0 >2 LOAD_NAME0 (a) >4 LIST_EXTEND 1 >6 LOAD_NAME1 (b) >8 LIST_EXTEND 1 > 10 LOAD_NAME2 (c) > 12 LIST_EXTEND 1 > 14 LIST_TO_TUPLE > > The bytecode is larger, therefore slower. It also prevents possible > optimization of BUILD_TUPLE_UNPACK and similar opcodes for common case > of tuples and lists which would allow to minimize the number of memory > allocations. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/CZZKWFW22TBJ5VLO7GUIF7A7QBFTBAC2/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PVCX2G6FJFCTW2P7D27UM76AAGSDJAT3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
I don't have a terribly strong opinion about whether or not it is acceptable to use dict.copy, my point was that the desired semantics can be achieved using only dunder methods if desired, and I think at this point getting the semantics right probably matters more than the implementation details. If we all agree on the semantics and we're just trying to decide how to get there, then I suppose I don't have a dog in the fight. I will note that it doesn't seem to be true that operators never call standard methods. Looks like date.__add__ calls date.toordinal and date.fromordinal (at least in the pure Python implementation), and datetime calls those plus tzinfo.utcoffset. Not sure if the rule Serhiy is citing is only intended to apply to builtins, though. On February 6, 2020 10:25:52 PM UTC, Brandt Bucher wrote: >Sorry Paul, I sent my reply too soon. > >I see what you're saying, and I'm pretty firmly -1 on reinventing (or >importing) copy.copy. We already have an API for copying a dict >(dict.copy). > >I still fail to see problem with using a method that doesn't start and >end with underscores, other than that we "haven't done it". > >Brandt >___ >Python-Dev mailing list -- python-dev@python.org >To unsubscribe send an email to python-dev-le...@python.org >https://mail.python.org/mailman3/lists/python-dev.python.org/ >Message archived at >https://mail.python.org/archives/list/python-dev@python.org/message/7TQI54BEUN6GERZA3Y2GCXPBD3CHXKM6/ >Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SCDWUIQX4ZLLU72QI4PYEG25W6JVIBFT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
On Fri, Feb 7, 2020 at 9:30 AM Brandt Bucher wrote: > > Sorry Paul, I sent my reply too soon. > > I see what you're saying, and I'm pretty firmly -1 on reinventing (or > importing) copy.copy. We already have an API for copying a dict (dict.copy). > > I still fail to see problem with using a method that doesn't start and end > with underscores, other than that we "haven't done it". > Before Python 3.0, iterators had a next() method, and that was explicitly and consciously changed to __next__(). The arguments there seem relevant here too. https://www.python.org/dev/peps/pep-3114/ ChrisA ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SGWZHDDSMZEH53N2MK2MSY7UQPG543FU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
Sorry Paul, I sent my reply too soon. I see what you're saying, and I'm pretty firmly -1 on reinventing (or importing) copy.copy. We already have an API for copying a dict (dict.copy). I still fail to see problem with using a method that doesn't start and end with underscores, other than that we "haven't done it". Brandt ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7TQI54BEUN6GERZA3Y2GCXPBD3CHXKM6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> but I think the desired semantics can all be achieved using only magic > methods on the objects themselves Hm. So, just to clarify, you're suggesting we use `__copy__`, if it exists? Interesting... ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZEB2VAVPGWHL6B55UAMYYQRALLILGQLE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
On 2/6/20 4:23 PM, Serhiy Storchaka wrote: > It would create an exception of two rules: > > 1. Operators on subclasses of builtin classes do not depend on > overridden methods of arguments (except the corresponding dunder > method). `list.__add__` and `set.__or__` do not call copy() and > extend()/update(). You should override the corresponding dunder method > to change the behavior of the operator. > > 2. Operators do not depend on non-dunder methods. > > This looks to me as a direct violation of the principle "Special cases > aren't special enough to break the rules." > I may not fully understand the implications of #1, but I would think you could implement the semantics Brandt wants using only dunder methods and copy.copy (which itself dispatches to one of a number of dunder methods - __copy__, __reduce__, __setstate__, depending on which ones are defined - we could presumably avoid the `copy` import by partially porting that logic into `__or__`): def __or__(self, other): new_value = copy.copy(self) for key in other.__keys__(): new_value.__setitem__(key, other.__getitem__(key)) return new_value Obviously the actual implementation would be in C and handle more edge cases and whatnot, but I think the desired semantics can all be achieved using only magic methods on the objects themselves (though we'd probably want to bypass all that stuff in favor of a "fast path" in the case of `dict` | `dict`). Best, Paul signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QXLOQ6QZLF35FGCTTARFSUBNGK4U22MB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
Then there’s nothing to do here right? Or just add it to whatsnew? On Thu, Feb 6, 2020 at 13:20 Brandt Bucher wrote: > > We should fix that (by reverting to 3.8.1 behaviour) before 3.8.2 gets > released. > > The commits which changed the behavior were bytecode/compiler changes that > only went to master. I don't think they are present on any other branches. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/GC3PGIGXFY47RQVOA4HLE3VX5IQ2VGTB/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido (mobile) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R6X4FROFYY5VG6VRSRHQ25PLMBUGXNA3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
06.02.20 08:28, Brandt Bucher пише: Commits 13bc139 and 8a4cd70 introduced subtle changes in the evaluation logic of unpacking operations. Previously, all elements were evaluated prior to being collected in a container. Now, these operations are interleaved. For example, the code `[*a, *b]` used to evaluate in the order `a` -> `b` -> `a.__iter__()` -> `b.__iter__()`. Now, it evaluates as `a` -> `a.__iter__()` -> `b` -> `b.__iter__()`. I believe this breaking semantic change is a bug, and I've opened a PR to fix it (https://github.com/python/cpython/pull/18264). My reasoning is that "unary *" isn't an operator; it doesn't appear on the operator precedence table in the docs, and you can't evaluate `*x`. Like the brackets and the comma, it's part of the syntax of the outer display expression, not the inner one. It specifies how the list should be built, so it should be evaluated last, as part of the list construction. And it has always been this way since PEP 448 (as far as I can tell). The docs themselves seem to support this line of reasoning (https://docs.python.org/3/reference/expressions.html#evaluation-order): In the following lines, expressions will be evaluated in the arithmetic order of their suffixes: ... expr1(expr2, expr3, *expr4, **expr5) Note that the stars are not part of expressions 1-5, but are a part of the top-level call expression that operates on them all. Mark Shannon disagrees with me (I'll let him reply rather than attempt to summarize his argument for him), but we figured it might be better to get more input here on exactly whether you all think the behavior should change or not. You can see the discussion on the PR itself for some additional points and context. I have two problems with this change. 1. It changes error messages. >>> print(*1) Traceback (most recent call last): File "", line 1, in TypeError: print() argument after * must be an iterable, not int >>> print(*1, *2) Traceback (most recent call last): File "", line 1, in TypeError: Value after * must be an iterable, not int In 3.8 you got the same error message. >>> print(*1) Traceback (most recent call last): File "", line 1, in TypeError: print() argument after * must be an iterable, not int >>> print(*1, *2) Traceback (most recent call last): File "", line 1, in TypeError: print() argument after * must be an iterable, not int I am not sure whether the function name is a useful information, but some effort was spend to preserve it. In any case, error messages should be consistent. 2. It introduces performance regression. In 3.8 the bytecode for `(*a, *b, *c)` was: 1 0 LOAD_NAME0 (a) 2 LOAD_NAME1 (b) 4 LOAD_NAME2 (c) 6 BUILD_TUPLE_UNPACK 3 In master it is: 1 0 BUILD_LIST 0 2 LOAD_NAME0 (a) 4 LIST_EXTEND 1 6 LOAD_NAME1 (b) 8 LIST_EXTEND 1 10 LOAD_NAME2 (c) 12 LIST_EXTEND 1 14 LIST_TO_TUPLE The bytecode is larger, therefore slower. It also prevents possible optimization of BUILD_TUPLE_UNPACK and similar opcodes for common case of tuples and lists which would allow to minimize the number of memory allocations. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CZZKWFW22TBJ5VLO7GUIF7A7QBFTBAC2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> It would create an exception of two rules: I don't think these are "rules", I think they're just "the way things are". If I'm subclassing `dict`, and I see in the docs something to the effect of: > By default, `dict` subclasses will return `dict` objects for `|` operations. > To force the creation of a new instance of the subclass, users can override > the `copy` method. In that case, the return value from this method will be > used instead. Then my life suddenly becomes a lot better, because chances are I've already thought to override `copy`. And if I want the "legacy" behavior, it's as simple as not bothering with "copy" (or, if I need to, overriding the `__or__` trio)... but I'm sure this is the less common case. If we're quoting the Zen, then let's not elevate past design patterns to "rules". Besides, practicality beats purity. ;) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L5ODZ2TEEY2T53C5KB25WJPFGJAU7IIP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
06.02.20 21:38, Brandt Bucher пише: One issue that's come up during PR review with Guido and Serhiy is, when evaluating `a | b`, whether or not the default implementation should call an overridden `copy()` method on `a` in order to create an instance of the correct subclass (rather than a plain-ol' `dict`). It would create an exception of two rules: 1. Operators on subclasses of builtin classes do not depend on overridden methods of arguments (except the corresponding dunder method). `list.__add__` and `set.__or__` do not call copy() and extend()/update(). You should override the corresponding dunder method to change the behavior of the operator. 2. Operators do not depend on non-dunder methods. This looks to me as a direct violation of the principle "Special cases aren't special enough to break the rules." I think the standard handling of subclasses in Python builtins is wrong, and I don't wish to emulate that wrong behaviour without a really good reason. Or at least a better reason than "other methods break subclassing unless explicitly overloaded, so this should do so too". Or at least not without a fight :-) We can discuss and change the standard handling of subclasses in Python builtins. But if it be changed, it should be changed for all builtin classes, without exceptions and special cases. This could resolve a problem with rule 1. But there is still a problem with rule 2. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4NDSEYFAHZBZOPIYKC4URMFAUGAIE42G/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
I like Mark’s new semantics better, but agree with the point about this
being a “feature”.
On Thu, Feb 6, 2020 at 13:06 Paul Moore wrote:
> On Thu, 6 Feb 2020 at 20:17, Mark Shannon wrote:
> >
> > I recently unintentionally changed the semantics of this expression
> > `[print("a"), *None, print("b")]`.
> > PEP 448 states that this should raise an exception, but does not specify
> > evaluation order.
> >
> > My implementation was based on the general principle that evaluation in
> > Python is left to right unless specified otherwise.
> >
> > The question is, what should
> > >>> [print("a"), *None, print("b")]
> >
> > print before raising an exception?
> > I think just "a", Brandt thinks "a" and "b".
> >
> > Brandt argues that I have introduced a bug. I think I have fixed one,
> > admittedly one that I didn't previously realize existed.
>
> I think that if this were a new feature, either order would be
> arguable. But given that this code previously printed "a b", I think
> that changing the order is a change in user-visible behaviour and
> therefore the existing behaviour should be preserved (certainly in
> bugfix releases).
>
> > There is a precedent for fixing evaluation order to be left to right:
> > https://bugs.python.org/issue29652
>
> Changing the order as part of a feature release, listing it as a
> user-visible behaviour change in "What's New" seems acceptable to me.
> But characterising this change as a bug fix doesn't.
>
> > On 06/02/2020 6:28 am, Brandt Bucher wrote:
> > > Commits 13bc139 and 8a4cd70 introduced subtle changes in the
> evaluation logic of unpacking operations. Previously, all elements were
> evaluated prior to being collected in a container. Now, these operations
> are interleaved. For example, the code `[*a, *b]` used to evaluate in the
> order `a` -> `b` -> `a.__iter__()` -> `b.__iter__()`. Now, it evaluates as
> `a` -> `a.__iter__()` -> `b` -> `b.__iter__()`.
> > >
> > > I believe this breaking semantic change is a bug, and I've opened a PR
> to fix it (https://github.com/python/cpython/pull/18264). My reasoning is
> that "unary *" isn't an operator; it doesn't appear on the operator
> precedence table in the docs, and you can't evaluate `*x`. Like the
> brackets and the comma, it's part of the syntax of the outer display
> expression, not the inner one. It specifies how the list should be built,
> so it should be evaluated last, as part of the list construction. And it
> has always been this way since PEP 448 (as far as I can tell).
>
> I agree that it's the *change in behaviour* that's the issue here. We
> should fix that (by reverting to 3.8.1 behaviour) before 3.8.2 gets
> released.
>
> > The lack of explicitly listed precedence for an operator does not mean
> > that it isn't an operator, merely that it doesn't need precedence due to
> > the grammar. For example the slice creation operator `x:y` in `a[x:y]`
> > needs no precedence as it is constrained to only occur in indexing
> > operations. Likewise the unpacking operation `*a` can only occur in
> > certain expressions. That doesn't mean that is not an operation.
>
> I don't think this is a particularly helpful way of looking at it. I
> don't think there's any particular need here to try to argue that
> either behaviour is "right" or "wrong". The previous behaviour has
> been round for some time, and the change in behaviour was (by your own
> admission) inadvertent. Therefore it seems obvious to me that the
> reasonable thing to do is to apply Brandt's PR, that restores the old
> evaluation order (with the *intended* fix from your patch intact, as I
> understand it). If, once this has been done, you still care strongly
> enough to argue for a behaviour change, targeted at 3.9 (assuming
> no-one insists on a deprecation period for the change!), then that's
> fine. Personally I think the arguments either way are weak, and I'd be
> inclined not to care, or to mildly prefer not bothering, in such a
> debate - but let's have the debate once the pressure of "is it OK to
> do this in a bugfix release?" has been removed.
>
> Paul
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/SUIATL7A7M2P3YWYIVIWP5EAM7NRZ5D3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/A46XTV6ZHVKSEZYB32SB4M2PNCHIKYSF/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
> I see this asymmetry between the | and |= mentioned a few times in the PEP, > but I don't see any rationale other than "the authors have decided". I agree that this probably deserves to be addressed. I can't speak for Steven, but I know that my motivation here is to restrict `|` in order to avoid confusing casting. It suddenly becomes a very complicated problem with strange edge-cases as soon as reflected operations get involved. In contrast, `|=` appears in dramatically simpler contexts, so we don't need to worry about resolving all of the combinations of possible LHS and RHS types (or whether the user can keep up with what we're doing under-the-hood). I have no idea if this is what motivated the decision in `list`, but I think it's a good one. > The specification mentions "Dict union will return a new dict containing the > left operand merged with the right operand, which must be a dict (or an > instance of a dict subclass)." Can you clarify if it is part of the spec that > it will always return a dict even if one or both of the operands is a dict > subclass? See my recent post on this here. I believe that we should call an overridden `lhs.copy()`, but others disagreed during code review. For what it's worth, it's probably not good for us to use "dict" and "`dict`" (with monospace/backticks) to mean different things in the PEP... ;) > Unless there is compelling reason to do otherwise, I am in favor of trying to > retain subclass identity after operation. I'll count you as a vote for `new = lhs.copy(); dict.update(new, rhs)`, then. ;) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MRYKAFWIJFODH7PBVH5CKPWAS2M26VFO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
> We should fix that (by reverting to 3.8.1 behaviour) before 3.8.2 gets > released. The commits which changed the behavior were bytecode/compiler changes that only went to master. I don't think they are present on any other branches. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GC3PGIGXFY47RQVOA4HLE3VX5IQ2VGTB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
On Thu, 6 Feb 2020 at 20:17, Mark Shannon wrote:
>
> I recently unintentionally changed the semantics of this expression
> `[print("a"), *None, print("b")]`.
> PEP 448 states that this should raise an exception, but does not specify
> evaluation order.
>
> My implementation was based on the general principle that evaluation in
> Python is left to right unless specified otherwise.
>
> The question is, what should
> >>> [print("a"), *None, print("b")]
>
> print before raising an exception?
> I think just "a", Brandt thinks "a" and "b".
>
> Brandt argues that I have introduced a bug. I think I have fixed one,
> admittedly one that I didn't previously realize existed.
I think that if this were a new feature, either order would be
arguable. But given that this code previously printed "a b", I think
that changing the order is a change in user-visible behaviour and
therefore the existing behaviour should be preserved (certainly in
bugfix releases).
> There is a precedent for fixing evaluation order to be left to right:
> https://bugs.python.org/issue29652
Changing the order as part of a feature release, listing it as a
user-visible behaviour change in "What's New" seems acceptable to me.
But characterising this change as a bug fix doesn't.
> On 06/02/2020 6:28 am, Brandt Bucher wrote:
> > Commits 13bc139 and 8a4cd70 introduced subtle changes in the evaluation
> > logic of unpacking operations. Previously, all elements were evaluated
> > prior to being collected in a container. Now, these operations are
> > interleaved. For example, the code `[*a, *b]` used to evaluate in the order
> > `a` -> `b` -> `a.__iter__()` -> `b.__iter__()`. Now, it evaluates as `a` ->
> > `a.__iter__()` -> `b` -> `b.__iter__()`.
> >
> > I believe this breaking semantic change is a bug, and I've opened a PR to
> > fix it (https://github.com/python/cpython/pull/18264). My reasoning is that
> > "unary *" isn't an operator; it doesn't appear on the operator precedence
> > table in the docs, and you can't evaluate `*x`. Like the brackets and the
> > comma, it's part of the syntax of the outer display expression, not the
> > inner one. It specifies how the list should be built, so it should be
> > evaluated last, as part of the list construction. And it has always been
> > this way since PEP 448 (as far as I can tell).
I agree that it's the *change in behaviour* that's the issue here. We
should fix that (by reverting to 3.8.1 behaviour) before 3.8.2 gets
released.
> The lack of explicitly listed precedence for an operator does not mean
> that it isn't an operator, merely that it doesn't need precedence due to
> the grammar. For example the slice creation operator `x:y` in `a[x:y]`
> needs no precedence as it is constrained to only occur in indexing
> operations. Likewise the unpacking operation `*a` can only occur in
> certain expressions. That doesn't mean that is not an operation.
I don't think this is a particularly helpful way of looking at it. I
don't think there's any particular need here to try to argue that
either behaviour is "right" or "wrong". The previous behaviour has
been round for some time, and the change in behaviour was (by your own
admission) inadvertent. Therefore it seems obvious to me that the
reasonable thing to do is to apply Brandt's PR, that restores the old
evaluation order (with the *intended* fix from your patch intact, as I
understand it). If, once this has been done, you still care strongly
enough to argue for a behaviour change, targeted at 3.9 (assuming
no-one insists on a deprecation period for the change!), then that's
fine. Personally I think the arguments either way are weak, and I'd be
inclined not to care, or to mildly prefer not bothering, in such a
debate - but let's have the debate once the pressure of "is it OK to
do this in a bugfix release?" has been removed.
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/SUIATL7A7M2P3YWYIVIWP5EAM7NRZ5D3/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
Hi Brandt, very nice PEP. I have two questions here.
First:
> - The proposal in its current form is very easy to wrap your head around: "|"
> takes dicts, "|=" takes anything dict.update does.
I see this asymmetry between the | and |= mentioned a few times in the
PEP, but I don't see any rationale other than "the authors have
decided". I am not saying that this is the wrong decision, but the
reasoning behind this choice is not obvious, and I think it might be a
good idea to include the rationale in the PEP. I'd say the asymmetry
between list's `__add__` and `__iadd__` semantics is actually fairly
confusing for anyone who hasn't encountered it before:
>>> a = []
>>> a = a + "one"
---
TypeError Traceback (most recent
call last)
in
> 1 a = a + "one"
TypeError: can only concatenate list (not "str") to list
>>> a += "one"
>>> a
['o', 'n', 'e']
I think most people would be surprised at the difference in semantics
here and this also an example of a situation where it's not obvious that
the "call `list.extend`" behavior is the right thing to do. It would be
nice to see why you rejected:
1. Giving |= `.update` semantics rather than the semantics you chose for |.
2. Giving `|` the same semantics as `|=`.
Second question:
The specification mentions "Dict union will return a new dict containing
the left operand merged with the right operand, which must be a dict (or
an instance of a dict subclass)." Can you clarify if it is part of the
spec that it will always return a `dict` even if one or both of the
operands is a dict subclass? You mentioned in another post that this is
deliberately intended to be subclass-friendly, but if it always returns
`dict`, then I would expect this:
>>> class MyDict(dict):
... pass
...
>>> MyDict({1: 2}) | MyDict({3: 4})
{1: 2, 3: 4}
I realize that there's a lot of precedent for this with other builtin
types (int subclasses reverting to int with +, etc), though generally
the justifications for this are two-fold:
1. For symmetrical operations like addition it's not obvious which
operand's type should prevail, particularly if the two are both
different dict subclasses. I think in this case | is already
asymmetrical and it would be relatively natural to think of the right
thing to do as "make a copy of the LHS then update it with the RHS"
(thus retaining the subtype of the LHS).
2. Subclasses may override the constructor in unpredictable ways, so we
don't know how to construct arbitrary subtypes. I /think/ this objection
could be satisfied by using logic equivalent to `copy.copy` for the LHS
when it is a dict subclass, and then using the mapping protocol on the RHS.
Unless there is compelling reason to do otherwise, I am in favor of
trying to retain subclass identity after operations, but either way it
would be good to be explicit about it in the specification and maybe
include a bit of the rationale one way or the other.
Best,
Paul
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/L44V6WTFI4OWM7TPYIMCACJHKBRJQMIU/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
Hi everyone,
I recently unintentionally changed the semantics of this expression
`[print("a"), *None, print("b")]`.
PEP 448 states that this should raise an exception, but does not specify
evaluation order.
My implementation was based on the general principle that evaluation in
Python is left to right unless specified otherwise.
The question is, what should
>>> [print("a"), *None, print("b")]
print before raising an exception?
I think just "a", Brandt thinks "a" and "b".
Brandt argues that I have introduced a bug. I think I have fixed one,
admittedly one that I didn't previously realize existed.
There is a precedent for fixing evaluation order to be left to right:
https://bugs.python.org/issue29652
On 06/02/2020 6:28 am, Brandt Bucher wrote:
Commits 13bc139 and 8a4cd70 introduced subtle changes in the evaluation logic of unpacking
operations. Previously, all elements were evaluated prior to being collected in a container. Now,
these operations are interleaved. For example, the code `[*a, *b]` used to evaluate in the order
`a` -> `b` -> `a.__iter__()` -> `b.__iter__()`. Now, it evaluates as `a` ->
`a.__iter__()` -> `b` -> `b.__iter__()`.
I believe this breaking semantic change is a bug, and I've opened a PR to fix it
(https://github.com/python/cpython/pull/18264). My reasoning is that "unary *"
isn't an operator; it doesn't appear on the operator precedence table in the docs, and
you can't evaluate `*x`. Like the brackets and the comma, it's part of the syntax of the
outer display expression, not the inner one. It specifies how the list should be built,
so it should be evaluated last, as part of the list construction. And it has always been
this way since PEP 448 (as far as I can tell).
The lack of explicitly listed precedence for an operator does not mean
that it isn't an operator, merely that it doesn't need precedence due to
the grammar. For example the slice creation operator `x:y` in `a[x:y]`
needs no precedence as it is constrained to only occur in indexing
operations. Likewise the unpacking operation `*a` can only occur in
certain expressions. That doesn't mean that is not an operation.
The docs themselves seem to support this line of reasoning
(https://docs.python.org/3/reference/expressions.html#evaluation-order):
In the following lines, expressions will be evaluated in the arithmetic order
of their suffixes:
...
expr1(expr2, expr3, *expr4, **expr5)
Note that the stars are not part of expressions 1-5, but are a part of the
top-level call expression that operates on them all.
There are many layers of grammar that make up a call. It is entirely
arbitrary what you call an expression or some other grammatical entity.
`*expr4` is parsed as an argument, the same as `expr2`.
Cheers,
Mark.
Mark Shannon disagrees with me (I'll let him reply rather than attempt to
summarize his argument for him), but we figured it might be better to get more
input here on exactly whether you all think the behavior should change or not.
You can see the discussion on the PR itself for some additional points and
context.
Thanks!
Brandt
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/4HS2ZEQWWFMQ7IO5ZHBNWEYLN4PIYDCD/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/VUSNP43ALB6YS2TINDTON6F5Y56EG7HV/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
On 2/6/2020 2:26 PM, Mark Shannon wrote: In the python grammar, an 'expression' is a 'starred_item' but a 'starred_item' need not be an expression. starred_item ::= expression | "*" or_expr expression ::= conditional_expression | lambda_expr conditional_expression ::= or_test ["if" or_test "else" expression] '*a' is a 'starred_item' but not an 'expression'. I don't know where you got that grammar from, but not GitHub https://github.com/python/cpython/blob/master/Grammar/Grammar#L142 From the human readable docs https://docs.python.org/3/reference/expressions.html#expression-lists https://docs.python.org/3/reference/expressions.html#conditional-expressions ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YYVDKK2FODTHFXN26LEUE2DJTOXGYWXB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
One issue that's come up during PR review with Guido and Serhiy is, when
evaluating `a | b`, whether or not the default implementation should call an
overridden `copy()` method on `a` in order to create an instance of the correct
subclass (rather than a plain-ol' `dict`). For example, `defaultdict` works
correctly without any additional modification:
```
>>> from collections import defaultdict
>>> i = defaultdict(int, {1: 1, 2: 2})
>>> s = defaultdict(str, {2: "2", 3: "3"})
>>> i | s
defaultdict(, {1: 1, 2: '2', 3: '3'})
>>> s | i
defaultdict(, {2: 2, 3: '3', 1: 1})
```
So this has immediate benefits for both usability and maintenance: subclasses
only need to override `copy()` to get working `__or__`/`__ror__` behavior.
While this isn't what `list` and `set` do, for example, I argue that `dict` is
more often subclassed, and we shouldn't blindly follow the precedent of their
behavior when designing a new API for `dict`.
We decided to bring the discussion here to get input from a larger audience.
I'm currently +0 on calling `copy()`, but I know Steven feels a bit more
strongly about this than I do:
> I think the standard handling of subclasses in Python builtins is wrong, and
> I don't wish to emulate that wrong behaviour without a really good reason. Or
> at least a better reason than "other methods break subclassing unless
> explicitly overloaded, so this should do so too". Or at least not without a
> fight :-)
The more detailed thread of discussion starts at
https://github.com/python/cpython/pull/12088#issuecomment-582609024 (note that
we are no longer considering calling an overridden `update` method).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/Z5SSGCH736UWHKFGJEZROH6VKPKXIT7W/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
On 06/02/2020 6:30 pm, Terry Reedy wrote:
On 2/6/2020 1:28 AM, Brandt Bucher wrote:
Commits 13bc139 and 8a4cd70 introduced subtle changes in the
evaluation logic of unpacking operations. Previously, all elements
were evaluated prior to being collected in a container. Now, these
operations are interleaved. For example, the code `[*a, *b]` used to
evaluate in the order `a` -> `b` -> `a.__iter__()` -> `b.__iter__()`.
Now, it evaluates as `a` -> `a.__iter__()` -> `b` -> `b.__iter__()`.
A simpler example, which sharpens the contrast, is [*a, b]. The
unpacking of *b is last either way. The change is from eval(a), eval(b),
extend(a.__iter__()), append b
to eval(a), extend(a.__iter__()), eval(b), append b
I believe this breaking semantic change is a bug, and I've opened a PR
to fix it (https://github.com/python/cpython/pull/18264).
I carefully read and considered the original issue and the discussion on
the PR and agree with the intent of the PR.
The semantic change can have visible effects due to interaction of
side-effects. Examples on the PR are
1. a.__iter__ raising while b prints
([*None, print('executed)]), and
2. a and b both involving the same iterator
(*it, next(it))
These previously unannounced semantic changes are apparently gratuitous
side-effects of an internal refactoring. They should only be made, if
at all, after discussion and agreement, then announcement and a
deprecation period. But I seen no reason to change the status quo
semantics.
These changes were unannounced because I didn't realize the current
implementation was broken. There were no tests for raising an exception
in the middle of unpacking a list, and it didn't occur to me to add them.
My reasoning is that "unary *" isn't an operator; it doesn't appear on
the operator precedence table in the docs, and you can't evaluate
`*x`. Like the brackets and the comma, it's part of the syntax of the
outer display expression, not the inner one. It specifies how the list
should be built, so it should be evaluated last, as part of the list
construction. And it has always been this way since PEP 448 (as far as
I can tell).
I agree that '*a' is not an expression in the meaning relevant here.
https://docs.python.org/3/glossary.html
says "A piece of syntax which can be evaluated to some value." This is
the common math/logic/CS meaning. '*a' cannot be evaluated to a Python
object. It is not an 'expression statement and cannot be passed to eval().
In the python grammar, an 'expression' is a 'starred_item' but a
'starred_item' need not be an expression.
starred_item ::= expression | "*" or_expr
expression ::= conditional_expression | lambda_expr
conditional_expression ::= or_test ["if" or_test "else" expression]
'*a' is a 'starred_item' but not an 'expression'.
I don't know where you got that grammar from, but not GitHub
https://github.com/python/cpython/blob/master/Grammar/Grammar#L142
The docs themselves seem to support this line of reasoning
(https://docs.python.org/3/reference/expressions.html#evaluation-order):
In the following lines, expressions will be evaluated in the
arithmetic order of their suffixes:
...
expr1(expr2, expr3, *expr4, **expr5)
Note that the stars are not part of expressions 1-5, but are a part of
the top-level call expression that operates on them all.
Mark Shannon disagrees with me (I'll let him reply rather than attempt
to summarize his argument for him), but we figured it might be better
to get more input here on exactly whether you all think the behavior
should change or not. You can see the discussion on the PR itself for
some additional points and context.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/THKSHB2PSZBJPHMM5AACFLDJRCV6I24P/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Clarification of unpacking semantics.
On 2/6/2020 1:28 AM, Brandt Bucher wrote:
Commits 13bc139 and 8a4cd70 introduced subtle changes in the evaluation logic of unpacking
operations. Previously, all elements were evaluated prior to being collected in a container. Now,
these operations are interleaved. For example, the code `[*a, *b]` used to evaluate in the order
`a` -> `b` -> `a.__iter__()` -> `b.__iter__()`. Now, it evaluates as `a` ->
`a.__iter__()` -> `b` -> `b.__iter__()`.
A simpler example, which sharpens the contrast, is [*a, b]. The
unpacking of *b is last either way. The change is from eval(a), eval(b),
extend(a.__iter__()), append b
to eval(a), extend(a.__iter__()), eval(b), append b
I believe this breaking semantic change is a bug, and I've opened a PR to fix
it (https://github.com/python/cpython/pull/18264).
I carefully read and considered the original issue and the discussion on
the PR and agree with the intent of the PR.
The semantic change can have visible effects due to interaction of
side-effects. Examples on the PR are
1. a.__iter__ raising while b prints
([*None, print('executed)]), and
2. a and b both involving the same iterator
(*it, next(it))
These previously unannounced semantic changes are apparently gratuitous
side-effects of an internal refactoring. They should only be made, if
at all, after discussion and agreement, then announcement and a
deprecation period. But I seen no reason to change the status quo
semantics.
My reasoning is that "unary *" isn't an operator; it doesn't appear on the
operator precedence table in the docs, and you can't evaluate `*x`. Like the brackets and
the comma, it's part of the syntax of the outer display expression, not the inner one. It
specifies how the list should be built, so it should be evaluated last, as part of the
list construction. And it has always been this way since PEP 448 (as far as I can tell).
I agree that '*a' is not an expression in the meaning relevant here.
https://docs.python.org/3/glossary.html
says "A piece of syntax which can be evaluated to some value." This is
the common math/logic/CS meaning. '*a' cannot be evaluated to a Python
object. It is not an 'expression statement and cannot be passed to eval().
In the python grammar, an 'expression' is a 'starred_item' but a
'starred_item' need not be an expression.
starred_item ::= expression | "*" or_expr
expression ::= conditional_expression | lambda_expr
conditional_expression ::= or_test ["if" or_test "else" expression]
'*a' is a 'starred_item' but not an 'expression'.
The docs themselves seem to support this line of reasoning
(https://docs.python.org/3/reference/expressions.html#evaluation-order):
In the following lines, expressions will be evaluated in the arithmetic order
of their suffixes:
...
expr1(expr2, expr3, *expr4, **expr5)
Note that the stars are not part of expressions 1-5, but are a part of the
top-level call expression that operates on them all.
Mark Shannon disagrees with me (I'll let him reply rather than attempt to
summarize his argument for him), but we figured it might be better to get more
input here on exactly whether you all think the behavior should change or not.
You can see the discussion on the PR itself for some additional points and
context.
--
Terry Jan Reedy
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/R74GKY6HL5SB5O4GBWPYYMINQAVLXXQH/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
On Thu, Feb 6, 2020 at 7:42 PM Musbur wrote:
>
> Depends on what d is.
>
> >>> type({})
>
>
> So the result is {(1, 2): None}, but the ambiguity comes from the
> definition of {}, not from my proposal.
>
Actually, what Serhiy hinted at was a consequence (and, I would say, a
rather weird corner case) of the current definition of |= as being
"equivalent to update()". Since update() will accept a number of
things, including an iterable of pairs, it's actually possible to use
a set of two-element tuples as a representation of keys and values. It
seems pretty unlikely that anyone would use a *set* for this (as
opposed to, say, a list or generator), but it does mean that your
proposal would conflict with that. Which in turn means you're actually
asking for |= to special-case sets; not impossible, by any means, but
it's a definite change and not simply a logical extension of current
behaviour.
I don't think supporting sets in |= is worth this confusion,
especially since there's no easy way to override the choice of value.
It'd be just as easy to do thing|=dict.fromkeys(s) and avoid the whole
issue. However, IMO the &= operator would be better able to accept a
set, and this is something that wouldn't conflict (and thus can be
safely added later).
ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/BTYIZSBWR3O5JRIGFHOZWVIPUISUGDHB/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 584: Add Union Operators To dict
Depends on what d is.
type({})
So the result is {(1, 2): None}, but the ambiguity comes from the
definition of {}, not from my proposal.
Am 05.02.2020 18:19 schrieb Serhiy Storchaka:
05.02.20 14:27, Musbur пише:
I have one suggestion: Wouldn't it be useful for these operators to
also accept sets (functionally acting like a dict with None for all
values)? This would make it very elegant to 'normalize' dicts by
pruning (dict & set) or padding (set | dict) dictionaries. I would
find this useful for efficient data sanitation purposes, as when
processing input from web forms.
d = {}
d |= {(1, 2)}
What is d now? {1: 2} or {(1, 2): None}
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ZUXJBYUHXGAKLH3TOVB4UUXQUBFOQAIK/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/2ENA4EV2IVHTE64YVXT4MK52MYU5PLGR/
Code of Conduct: http://python.org/psf/codeofconduct/
