Re: [Python-ideas] PEP: Dict addition and subtraction
Oops, meant for the list: > >> > >> > Chris seems to accept that sometimes you can use a dict subclass, and > >> > that my proposal will give some representation of "practical > >> > experience". > >> > >> I said "definitely won't", not "will give some". So, no. > > > > > > Jonathan: Chris A. is right. But if you think an example implementation > is useful, go ahead and make one — it’s actually quite easy, and I’m pretty > sure code has already been posted here as well. > > -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 22, 2019 at 07:59:03AM +, Jonathan Fine wrote: > Steven D'Aprano wrote: > > > (For the record, the PEP isn't set in stone in regards to the choice of > > operator. > > Steven: Please say what parts of the PEP you consider to be set in > stone. This will allow discussion to focus on essentials rather than > details. The PEP is primarily about making a merge operator, so that stays, regardless of whether it's spelled + | or something else. Otherwise there's no point to the PEP. If there is demand for a merged() method/function, that go into a competing PEP, but it won't be part of this PEP. If anyone wants to propose syntax for chained method calls (fluent programming) so we can write d.copy().update(), that won't be in this PEP either. Likewise for new syntax to turn method calls into operators. Feel free to propose a competing PEP (and I might even support yours, if it makes a good enough case). A grey area is the "last wins" merge behaviour matching update(). In theory, if somebody made an absolutely brilliant case for some other behaviour, I could change my mind, but it would have to be pretty amazing. In the absence of such, I'm going to use my perogative as PEP author to choose the behaviour I prefer to see, and leave alternatives to subclasses. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Python fits well in the mind, because (1) by design it reduces
cognitive load, and (2) it encourages its users to reduce cognitive
load, and (3) we have a culture of reading code, taking pride in our
code. Readability counts.
https://en.wikipedia.org/wiki/Cognitive_load
Steven D'Aprano says that examples such as below don't help us discuss
the cognitive load associated with dict + dict.
> 1. Toy examples using generic names don't count.
> eggs += spam
I assume he's referring to my example
>>> items.update(points)
>>> items += points
In this example, items.update gives useful additional information.
We expect, from duck typing and sensible naming, that points can be
iterated to give key value pairs. In Python, when
>>> a + b
gives one of
TypeError: unsupported operand type(s) for +: 'int' and 'str'
TypeError: Can't convert 'int' object to str implicitly
we get a very strong hint to write instead something like
a + int(b)
str(a) + b
so that the nature of the addition is made clear to the next person
who reads the code (who might be ourselve, in a crisis, in ten years
time.)
(JavaScript does implicit conversion. This makes the code easier to
write, harder to read, and harder to maintain.)
For certain values of dct and lst we get
>>> lst += dct
>>> lst
[('a', 1), ('b', 2), 'c', 'd']
For the same values of dct and lst (if proposal allowed)
>>> dct += lst
>>> dct
{'a': 1, 'b': 2, 'c': 3, 'd': 4}
In these examples, dct is a dict, and lst is a list. This behaviour is
something Python users will have to learn, and have in their mind,
whenever they see '+=' in unfamiliar code.
I find this as much an unwelcome cognitive load as that produced by Javascript's
> 2 * "8"
16
> 2 + "8"
"28"
To be fair, this may in part be a problem with our expectations about +=.
--
Jonathan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Steven D'Aprano wrote: > (For the record, the PEP isn't set in stone in regards to the choice of > operator. Steven: Please say what parts of the PEP you consider to be set in stone. This will allow discussion to focus on essentials rather than details. -- Jonathan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 22, 2019 at 6:47 PM Jonathan Fine wrote: > > Chris Angelico wrote: > > > The trouble with that is that you can't always use a dict subclass (or > > a non-subclass MutableMapping implementation, etc, etc, etc). There > > are MANY situations in which Python will give you an actual real dict, > > and it defeats the purpose if you then have to construct an > > AddableDict out of it just so you can add something to it. Not every > > proposed change makes sense on PyPI, and it definitely won't get a > > fair representation in "practical experience". > > Chris seems to accept that sometimes you can use a dict subclass, and > that my proposal will give some representation of "practical > experience". I said "definitely won't", not "will give some". So, no. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 22, 2019 at 1:58 PM David Mertz wrote: > > On Thu, Mar 21, 2019, 10:15 PM Steven D'Aprano wrote: >> > Of course I could learn it and teach it, but it will always feel >> > like a wart in the language. >> >> Would that wartness be lessoned if it were spelled | or << instead? > > Yes, definitely. Both those spellings feel pretty natural to me. They don't > have the misleading associations '+' carries. I'm kinda fond of '<<' because > it visitation resembles an arrow that I can think of as "put the stuff here > into there". > Please no. The "cuteness" value of abusing the operator to indicate information flow got old shortly after C++ did it, and it doesn't help. With normal operator overloading, you can say "the + operator means addition", and then define "addition" for different types. Perhaps that ship has sailed, since we already have division between path objects, but at least in that example it is VERY closely related. There's no use of "<<" inside string literals with dictionaries the way there's "/foo/bar/spam" in paths. Dictionary merging is a form of addition. It's also related to set union, which is well known as part of the pipe operator. Either of those is far better than abusing left shift. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Chris Angelico wrote: > The trouble with that is that you can't always use a dict subclass (or > a non-subclass MutableMapping implementation, etc, etc, etc). There > are MANY situations in which Python will give you an actual real dict, > and it defeats the purpose if you then have to construct an > AddableDict out of it just so you can add something to it. Not every > proposed change makes sense on PyPI, and it definitely won't get a > fair representation in "practical experience". Chris seems to accept that sometimes you can use a dict subclass, and that my proposal will give some representation of "practical experience". Even if not perfect, such benefits are I think worth having. And Chris gives no evidence (or examples) beyond his own assertions, that my proposal would not produce a fair representation of practical experience. Why don't we just try it and see. This would engage us with the users. And it would, as I suggested, clarify and document the syntax and semantics. And provide backporting to current versions of Python. By the way, in "Masterminds of Programming" [page 20], Guido gives four lines of defence against the unwise addition of a "favorite feature" to the language. They are [1] Explain to people that they can already do what they want. [2] Tell them to write their own module or class to encapsulate the feature. [3] Accept the feature, as pure Python, in the standard library. [4] Accept the feature as a C-Python extension standard. And [4] is, in Guido's words > the last line of defense before we have to admit [...] > this is so useful [...] so we'll have to change the language I think the pure Python implementation is important. If the supporters of this proposal are not willing to provide this, then I will (along with anyone else who volunteers). http://shop.oreilly.com/product/9780596515171.do # Masterminds of Programming -- Jonathan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 2019-03-22 02:40, Dan Sommers wrote: On 3/21/19 9:19 PM, Christopher Barker wrote: https://docs.python.org/3.8/library/collections.html has some examples using collections.Counter, which is clearly described as being a subclass of dict. Amongst the examples: c + d # add two counters together: c[x] + d[x] That's the + operator operating on two dicts (don't make me quote the Liskov Substitution Principle), but doing something really different than the base operator. So if I know that c and d (or worse, that one of them) is a dict, then interpreting c + d becomes much more interesting, Killing a use of a common operator with a very common built in data type because the operator is used in a different way by a specialized object in the stdlib seems a bit backwards to me. Perhaps. Note that Counter also uses | and & for other operations that probably wouldn't make much sense on base dicts. Frankly, I think considering Counter as a dict subclass is the mistake here, even if it is true. I had the same thought that Counter is misdesigned in one way or another, but (a) that ship has long sailed, and (b) I didn't want to run off on that tangent. [snip] Counter is trying to provide the functionality of 2 kinds of container: 1. A counting container. 2. A multi-set. + makes sense for counting (sum); | makes sense for multi-sets (union). ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019, 10:15 PM Steven D'Aprano wrote:
> What would be the most useful behaviour for dict "addition" in your
> opinion?
>
Probably what I would use most often was a "lossless" merging in which
duplicate keys resulted in the corresponding value becoming a set
containing all the merged values. E.g.
>>> d1 = {1: 55, 2: 77, 3: 88}
>>> d2 = {3: 99, 4: 22}
>>> add(d1, d2)
{1: 55, 2: 77, 3: {88, 99}, 4:22}
I'm sure most users would hate this too. It changes the type of values
between a thing and a set of things, and that has to be sorted out
downstream. But it is lossless in a similar way to Counter or sequence
addition.
I can write what I want perfectly well. Perhaps useing defaultdict as a
shortcut to get there. And I know there are some behaviors I have not
specified here, but my function can do whatever I want in the edge cases.
If we're to see 'd1 + d2' for the first time without having followed this
discussion, my guess would be behavior similar to what I show.
> Of course I could learn it and teach it, but it will always feel
> > like a wart in the language.
>
> Would that wartness be lessoned if it were spelled | or << instead?
>
Yes, definitely. Both those spellings feel pretty natural to me. They don't
have the misleading associations '+' carries. I'm kinda fond of '<<'
because it visitation resembles an arrow that I can think of as "put the
stuff here into there".
> In contrast, once you tell me about the special object "vectorised
> arrays",
> > `arr1 + arr2` does exactly what is expect in NumPy.
>
> I don't know Numpy well enough to know whether that is elementwise
> addition or concatenation or something else, so that example doesn't
> resonate with me. I can't guess what you expect, and I have no confidence
> that my guess (matrix addition of equal-sized arrays, an exception if
> unequal) will be what Numpy does
>
Fair enough. I've worked with NumPy long enough that perhaps I forget what
my first intuition was. I accept that it's non-obvious to many users.
FWIW, I really love NumPy behavior, but it's a shift in thinking vs lists.
E.g.
>>> a = array([1, 2, 3])
>>> b = array([[10, 11, 12], [100, 200, 300]])
>>> a + b
[[ 11 13 15 ]
[ 101 202 303]]
This is "broadcasting" of compatible shapes.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 3/21/19 9:19 PM, Christopher Barker wrote: https://docs.python.org/3.8/library/collections.html has some examples using collections.Counter, which is clearly described as being a subclass of dict. Amongst the examples: c + d # add two counters together: c[x] + d[x] That's the + operator operating on two dicts (don't make me quote the Liskov Substitution Principle), but doing something really different than the base operator. So if I know that c and d (or worse, that one of them) is a dict, then interpreting c + d becomes much more interesting, Killing a use of a common operator with a very common built in data type because the operator is used in a different way by a specialized object in the stdlib seems a bit backwards to me. Perhaps. Note that Counter also uses | and & for other operations that probably wouldn't make much sense on base dicts. Frankly, I think considering Counter as a dict subclass is the mistake here, even if it is true. I had the same thought that Counter is misdesigned in one way or another, but (a) that ship has long sailed, and (b) I didn't want to run off on that tangent. My point remains: because Counter is a subclass of dict, and Counter uses the + operator for something that doesn't apply to base dicts, adding + to dicts *may* cause confusion that wasn't there before. Presently, +, -, |, and & all raise an exception when given a Counter and a dict. This (raising an exception) is probably still the Right Thing to do in that case, even with a + operator on dicts, but that violates the LSP and IMO the PLS. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
> > https://docs.python.org/3.8/library/collections.html has some > examples using collections.Counter, which is clearly described > as being a subclass of dict. Amongst the examples: > > c + d # add two counters together: c[x] + d[x] > > That's the + operator operating on two dicts (don't make me > quote the Liskov Substitution Principle), but doing something > really different than the base operator. > > So if I know that c and d (or worse, that one of them) is a > dict, then interpreting c + d becomes much more interesting, Killing a use of a common operator with a very common built in data type because the operator is used in a different way by a specialized object in the stdlib seems a bit backwards to me. Frankly, I think considering Counter as a dict subclass is the mistake here, even if it is true. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019 at 08:13:01PM -0400, David Mertz wrote: > On Thu, Mar 21, 2019, 7:48 PM Steven D'Aprano wrote: [...] > Nonetheless, if I see `dict1 + dict2` the meaning you intend in the PEP > does not jump out as the obvious behavior. Nor even as the most useful > behavior. What would be the most useful behaviour for dict "addition" in your opinion? > Of course I could learn it and teach it, but it will always feel > like a wart in the language. Would that wartness be lessoned if it were spelled | or << instead? > In contrast, once you tell me about the special object "vectorised arrays", > `arr1 + arr2` does exactly what is expect in NumPy. I don't know Numpy well enough to know whether that is elementwise addition or concatenation or something else, so that example doesn't resonate with me. I can't guess what you expect, and I have no confidence that my guess (matrix addition of equal-sized arrays, an exception if unequal) will be what Numpy does. > > And your subjective feeling is well-noted :-) > > This is more than "merely subjective." If it is more than subjective, then there must be an objective test that anyone, or a computer program, could do to tell whether or not the + operator on dicts will be ... um, what? A wart? Ugly? Both of those are subjective value judgements, so I'm not sure what objective claim you believe you are making which is "more than" subjective. The point is, I'm not *discounting* the subjective claims that + on dicts is ugly. I've acknowledged them, and the next draft of the PEP will do so too. But repetition doesn't make a subjective value judgement objective. It might boil down to a subjective preference for + over | or visa versa, or another operator, or no operator at all. That's fine: language design is partly subjective. But I'd like to see more comments based on objective reasons we can agree on, and fewer arguments that boil down to "I just don't like it". -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 3/21/19 6:46 PM, Steven D'Aprano wrote: Antoine and Serhiy seem to worry that there are existing uses of + which are currently easy to understand but will become less so if dict.__add__ is added. I respect that worry, even if I doubt that they are correct. If someone can demonstrate that their fear is well-founded, that would be an excellent counter-argument to the PEP's proposal to use +. https://docs.python.org/3.8/library/collections.html has some examples using collections.Counter, which is clearly described as being a subclass of dict. Amongst the examples: c + d # add two counters together: c[x] + d[x] That's the + operator operating on two dicts (don't make me quote the Liskov Substitution Principle), but doing something really different than the base operator. So if I know that c and d (or worse, that one of them) is a dict, then interpreting c + d becomes much more interesting, but arguably no worse than c.update(d). Yes, it's "just" polymorphism, but IMO it violates the Principle of Least Surprise. My apologies if this is covered elsewhere in this thread, or it doesn't meet the bar Steven set. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019, 7:48 PM Steven D'Aprano wrote: > A number of people including Antoine and Serhiy seem to have taken the > position that merely adding dict.__add__ will make existing code using + > harder to understand, as you will need to consider not just numeric > addition and concatenation, but also merging, when reading code. > > Would you agree that this example of + is perfectly clear today? > > for digit in digits: > num = num*10 + digit > > By both the naming (digit, num) and presence of multiplication by the > literal 10, it should be pretty obvious that this is probably doing > integer addition. > Yep. This is clear and will not become less clear if some more objects grow an .__add__() methods. Already, it is POSSIBLE that `num` and `digit` mean something other than numbers. Bad naming of variables if so, but not prohibited. For example, NumPy uses '+' and '*' for elementwise operations, often with broadcasting to different areas shapes. Maybe that's code dealing with vectorised arrays... But probably not. Holoviews users '+' and '*' to combine elements of graphs. E.g labelled = low_freq * high_freq * linpoints overlay + labelled + labelled.Sinusoid.Low_Frequency ggplot in R has similar behavior. Maybe your loop is composing a complex graph... But probably not. Nonetheless, if I see `dict1 + dict2` the meaning you intend in the PEP does not jump out as the obvious behavior. Nor even as the most useful behavior. Of course I could learn it and teach it, but it will always feel like a wart in the language. In contrast, once you tell me about the special object "vectorised arrays", `arr1 + arr2` does exactly what is expect in NumPy. And your subjective feeling is well-noted :-) > This is more than "merely subjective." I teach Python. I write books about Python. I've had tens of millions of readers of articles I've written about Python. I'm not the only person in this discussion with knowledge of learners and programmers and scientists... But the opinions I'm expressing ARE on their behalf too (as I perceive likely surprise and likely bugs). I like most of the design of Python. Almost all, even. But there are a few warts in it. This would be a wart. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019 at 03:10:48PM -0700, Brandt Bucher wrote: > For anyone interested in "trying it out": if you're not against cloning and > compiling CPython yourself, here is a PEP 584 C implementation I have PR'd > against master right now. I'm keeping it in sync with the draft PEP as it > changes, so subtraction performance is not overly optimized yet, but it > will show you the *exact* behavior outlined in the PEP on the dict builtin > and its subclasses. The relevant branch is called "addiction". You can > clone it from: That's great, thank you! For the sake of comparisons, could you support | as an alias? That will allow people to get a feel for whether a+b or a|b looks nicer. (For the record, the PEP isn't set in stone in regards to the choice of operator. > https://github.com/brandtbucher/cpython.git -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019 at 06:02:05PM -0400, David Mertz wrote:
> I dislike the symbol '+' to mean "dictionary merging with value updates." I
> have no objection to, and mildly support, adding '|' with this meaning.
>
> It's not really possible to give "that one example" where + for meeting
> makes code less clear... In my eyes it would be EVERY such use.
I suspect that I may not have explained myself properly. Sorry. Let me
try to explain again.
A number of people including Antoine and Serhiy seem to have taken the
position that merely adding dict.__add__ will make existing code using +
harder to understand, as you will need to consider not just numeric
addition and concatenation, but also merging, when reading code.
*If this were true* it would be an excellent argument against using +
for dict merges. But is it true?
Would you agree that this example of + is perfectly clear today?
for digit in digits:
num = num*10 + digit
By both the naming (digit, num) and presence of multiplication by the
literal 10, it should be pretty obvious that this is probably doing
integer addition.
(I suppose it is conceivable that this is doing sequence repetition and
concatenation, but given the names that interpretation would be rather
unexpected.)
We shouldn't find it hard to understand that code, using nothing more
than *local* context. There's no need to search the global context to
find out what num and digits are.
(Although in the specific example I copied that snippet from, that
information is only two or three lines away. But in principle, we might
have needed to search an arbitrarily large code base to determine what
they were.)
Adding dict.__add__ isn't going to make that example harder to
understand. If it did, that would be a big blow to the + proposal.
Antoine and Serhiy seem to worry that there are existing uses of + which
are currently easy to understand but will become less so if dict.__add__
is added. I respect that worry, even if I doubt that they are correct.
If someone can demonstrate that their fear is well-founded, that would
be an excellent counter-argument to the PEP's proposal to use +.
What *doesn't* count as a demonstration:
1. Toy examples using generic names don't count. With generic,
meaningless names, they're not meaningful now and so adding dict.__add__
won't make them *less* meaningful:
# is this concatenation or numeric addition? who can tell?
for spam in spammy_macspamface:
eggs += spam
Regardless of whether dicts support + or not, we would still have to
search the global context to work out what eggs and spam are.
Adding dict.__add__ doesn't make this harder.
2. Purely opinion-based subjective statements, since they basically boil
down to "I don't like the use of + for dict merging." That point has
been made, no need to keep beating that drum.
3. Arguments based on unfamiliarity to the new operator:
preferences += {'EDITOR': 'ed', 'PAGESIZE': 'A4'}
might give you a bit of a double-take the first time you see it, but it
surely won't still be surprising you in five years time.
I realise that this is a high bar to reach, but if somebody does reach
it, and demonstrates that Antoine and Serhiy's fears are well-founded,
that would be a very effective and convincing argument.
> Every
> example presented in this thread or in the PEP feels wrong to me. I know
> about operator overloading and dunder methods and custom classes. My
> intuition about '+' from math, other programming languages, and Python,
> simply does not lead me to expect the proposed meaning.
And your subjective feeling is well-noted :-)
--
Steven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 22, 2019 at 3:42 AM Steven D'Aprano wrote:
> And to those who support this PEP, code examples where a dict merge
> operator will help are most welcome!
Since Python examples don't really exist yet, I'm reaching for another
language that DOES have this feature. Pike's mappings (broadly
equivalent to Python's dicts) can be added (actually, both + and | are
supported), with semantics equivalent to PEP 584's. Translated into
Python syntax, here's a section from the implementation of
Process.run():
def run(cmd, modifiers={}):
...
...
p = Process(cmd, modifiers + {
"stdout": mystdout->pipe(),
"stderr": mystderr->pipe(),
"stdin": mystdin->pipe(),
})
In Val.TimeTZ, a subclass that adds a timezone attribute overrides a
mapping-returning method to incorporate the timezone in the result
mapping. Again, translated into Python syntax:
def tm(self):
return super().tm() + {"timezone": self.timezone}
To spawn a subprocess with a changed environment variable:
//from the Process.create_process example
Process.create_process(({ "/usr/bin/env" }),
(["env" : getenv() + (["TERM":"vt100"]) ]));
# equivalent Python code
subprocess.Popen("/usr/bin/env",
env=os.environ + {"TERM": "vt100"})
All of these examples could be done with the double-star syntax, as
they all use simple literals. But addition looks a lot cleaner IMO,
and even more so if you're combining multiple variables rather than
using literals.
ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
I dislike the symbol '+' to mean "dictionary merging with value updates." I have no objection to, and mildly support, adding '|' with this meaning. It's not really possible to give "that one example" where + for meeting makes code less clear... In my eyes it would be EVERY such use. Every example presented in this thread or in the PEP feels wrong to me. I know about operator overloading and dunder methods and custom classes. My intuition about '+' from math, other programming languages, and Python, simply does not lead me to expect the proposed meaning. On Thu, Mar 21, 2019, 12:43 PM Steven D'Aprano wrote: > I'd like to make a plea to people: > > I get it, there is now significant opposition to using the + symbol for > this proposed operator. At the time I wrote the first draft of the PEP, > there was virtually no opposition to it, and the | operator had very > little support. This has clearly changed. > > At this point I don't think it is productive to keep making subjective > claims that + will be more confusing or surprising. You've made your > point that you don't like it, and the next draft^1 of the PEP will make > that clear. > > But if you have *concrete examples* of code that currently is easy to > understand, but will be harder to understand if we add dict.__add__, > then please do show me! > > For those who oppose the + operator, it will help me if you made it > clear whether it is *just* the + symbol you dislike, and would accept > the | operator instead, or whether you hate the whole operator concept > regardless of how it is spelled. > > And to those who support this PEP, code examples where a dict merge > operator will help are most welcome! > > > > > ^1 Coming Real Soon Now™. > > > -- > Steven > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 22, 2019 at 8:44 AM Jonathan Fine wrote: > > Antoine Pitrou wrote: > > Note that, if you're able to live with a third-party dependency, the > > `toolz` package has what you need (and lots of other things too): > > https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.merge > > I suggest that the supporters of dict + dict make (and put up on PyPi) > a pure-Python subclass of dict that has the desired properties. This > would > > 1. Clarify and document the syntax and semantics. > 2. Help with exploration and testing. > 3. Provide a 'back-port' mechanism to current Python. > 4. Give the proposal the benefit of practical experience. > The trouble with that is that you can't always use a dict subclass (or a non-subclass MutableMapping implementation, etc, etc, etc). There are MANY situations in which Python will give you an actual real dict, and it defeats the purpose if you then have to construct an AddableDict out of it just so you can add something to it. Not every proposed change makes sense on PyPI, and it definitely won't get a fair representation in "practical experience". If someone's proposing adding a new module to the standard library, then by all means, propose PyPI. But changes to core types can't be imported from other modules. Python is not Ruby. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Antoine Pitrou wrote: > Note that, if you're able to live with a third-party dependency, the > `toolz` package has what you need (and lots of other things too): > https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.merge I suggest that the supporters of dict + dict make (and put up on PyPi) a pure-Python subclass of dict that has the desired properties. This would 1. Clarify and document the syntax and semantics. 2. Help with exploration and testing. 3. Provide a 'back-port' mechanism to current Python. 4. Give the proposal the benefit of practical experience. I find this last very important, when we can do it. And we can, in this case. Language changes are 'cast in stone' and hard to reverse. And afterwards, on this list, we're sometime told that we've 'missed the boat' for a particular change. Let's take the benefit of a reference pure Python implementation, when we can. Steven D'A. Please would you include or respond to this suggestion, in the next revision of the PEP. -- Jonathan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, 21 Mar 2019 17:59:41 + Rhodri James wrote: > > >> And to those who support this PEP, code examples where a dict merge > >> operator will help are most welcome! > > I don't use Python often enough to have much to offer, I'm afraid. The > sort of occasion I would use dict merging is passing modified > environments to subcommands. Something like: > > def process(): > if time_to_do_thing1(): > thing1(base_env + thing1_env_stuff + env_tweaks) > if time_to_do_thing2(): > thing2(base_env + thing2_env_stuff + env_tweaks) > > ...and so on. The current syntax for doing this is a tad verbose: > > def process(): > if time_to_do_thing1(): > env = base_env.copy() > env.update(thing1_env_stuff) > env.update(env_tweaks) > thing1(env) > del env > if time_to_do_thing2(): > env = base_env.copy() > env.update(thing2_env_stuff) > env.update(env_tweaks) > thing2(env) > del env Ah, you convinced me there is a use case indeed (though `del env` isn't necessary above). I would still prefer something that's not an operator, but I agree there is potential to improve the current state of affairs. Note that, if you're able to live with a third-party dependency, the `toolz` package has what you need (and lots of other things too): https://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoolz.merge Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 21/03/2019 17:59, Rhodri James wrote:
def process():
if time_to_do_thing1():
thing1(base_env + thing1_env_stuff + env_tweaks)
if time_to_do_thing2():
thing2(base_env + thing2_env_stuff + env_tweaks)
...and so on. The current syntax for doing this is a tad verbose:
def process():
if time_to_do_thing1():
env = base_env.copy()
env.update(thing1_env_stuff)
env.update(env_tweaks)
thing1(env)
del env
if time_to_do_thing2():
env = base_env.copy()
env.update(thing2_env_stuff)
env.update(env_tweaks)
thing2(env)
del env
Of course I forgot:
def process():
if time_to_do_thing1():
thing1({**base_env, **thing1_env_stuff, **env_tweaks})
if time_to_do_thing2():
thing2({**base_env, **thing2_env_stuff, **env_tweaks})
...which says something about how memorable that syntax is.
--
Rhodri James *-* Kynesim Ltd
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
I'd also like to add what I consider to be another point in favor of an
operator:
Throughout all of these related threads, I have seen many typos and
misspellings of current dict merging idioms, from messing up the number of
asterisks in "{**a, **b}", to even Guido(!) accidentally writing the common
copy/update idiom as
d = d1.copy()
d = d1.update(d2)
in a thoughtful email... and it was then copied-and-pasted (unquoted and
verbatim) by others!
I still have yet to see somebody (even those who claim to be confused by
it) mess up the PEP's current definition of "+" or "+=" in this context.
Brandt
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 21/03/2019 17:06, Antoine Pitrou wrote: On Fri, 22 Mar 2019 03:42:00 +1100 Steven D'Aprano wrote: For those who oppose the + operator, it will help me if you made it clear whether it is *just* the + symbol you dislike, and would accept the | operator instead, or whether you hate the whole operator concept regardless of how it is spelled. I'd rather see a method. Dict merging just doesn't occur often enough that an operator is desirable for it. Analogous to the relationship between list.sort() and sorted(), I can't help but think that a dict.merge() method would be a terrible idea. A merged() function is more defensible. And to those who support this PEP, code examples where a dict merge operator will help are most welcome! I don't use Python often enough to have much to offer, I'm afraid. The sort of occasion I would use dict merging is passing modified environments to subcommands. Something like: def process(): if time_to_do_thing1(): thing1(base_env + thing1_env_stuff + env_tweaks) if time_to_do_thing2(): thing2(base_env + thing2_env_stuff + env_tweaks) ...and so on. The current syntax for doing this is a tad verbose: def process(): if time_to_do_thing1(): env = base_env.copy() env.update(thing1_env_stuff) env.update(env_tweaks) thing1(env) del env if time_to_do_thing2(): env = base_env.copy() env.update(thing2_env_stuff) env.update(env_tweaks) thing2(env) del env -- Rhodri James *-* Kynesim Ltd ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 3/21/19 1:01 PM, Jonathan Fine wrote: Rémi Lapeyre wrote: Not matter the notation you end up choosing, I think this code: https://github.com/jpadilla/pyjwt/blob/master/jwt/utils.py#L71-L81 [...] would greatly benefit from a new merge to merge dicts. I've looked at the merge_dict defined in this code. It's similar to def gapfill(self, other): # See also: https://cobrapy.readthedocs.io/en/latest/gapfilling.html # Cobra's gapfill adds items to a model, to meet a requirement. for key in other.keys(): if key not in self: self[key] = other[key] (This is code I've written, that's not yet on PyPi.) The usage is different. Instead of writing one of aaa = merge_dict(aaa, bbb) ccc = merge_dict(aaa, bbb) you write one of gapfill(aaa, bbb) aaa.gapfill(bbb) # If gapfill added to dict methods. With merge_dict, you never really know if ccc is the same object as aaa, or a different one. Sometimes this is important. With gapfill, you get the same behaviour as the already familiar and loved dict.update. But of course with a different merge rule. With gapfill, I can never remeber whether it's gapfill(aaa, bbb) or gapfill(bbb, aaa). This is always important. :-) At least with aaa.gapfill(bbb), I have some sense of the "direction" of the asymmetry, or I would if I had some frame of reference into which to put the "gapfill" operation. (With the proposed + or | operator syntax, that gets lost.) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Chris Angelico writes:
> ... then, in the interests of productive discussion, could you please
> explain? What is it about dict addition that makes it harder to
> understand than other addition?
Antoine didn't say what dict addition does is harder to understand
than other addition. He said he wants to understand it without
knowing what it does.
I can't say for sure what he means precisely, but I take it that he
wants dict "+" to obey certain regularities that other instances of
"+" do, possibly including outside of Python. As you'll see I find it
hard to make this precise, but it's a pretty strong feeling for me, as
well.
To me, those regularities include associativity (satisfied in Python
except for floats) and commutativity where possible (integers and I
believe floats do satisfy it, while strings cannot and other sequences
in Python in general do not, although they very often do in
mathematics).
For mappings, the mathematical meaning of "+" is usually pointwise.
This wouldn't make sense for strings (interpreted as mappings from a
prefix of the natural numbers) at all except that by accident in
Python s1[n] + s2[n] does make sense, but not pointwise (because the
length of the result is 2, not 1, for each n).
For sequences in general pointwise doesn't make sense (there's no
restriction to homogeneous sequences, and if there were, like strings
it's not clear that the elements would be summable in an appropriate
sense). But concatenation always makes sense, especially by analogy
to the somehow (IMO) canonical case of strings. For sets, the only
plausible interpretation of "addition" is union, but in fact Python
used .add asymmetrically as "add to", not "add together" (self is a
set, argument is a generic object), and the union operator is "|", not
"+".
For dictionaries, neither pointwise addition nor concatenation makes
sense in general, and update is "too asymmetric" for my taste, and has
no analog in the usual algebras of mappings.
In some sense string concatenation, though noncommutative, doesn't
lose information, and it does obey a sort of antisymmetry in that
a + b == reversed(reversed(b) + reversed(a)). Dictionary update does
lose the original settings.
If people really think it's so important to spell
d = d0.copy()
d.update(d1)
as "d0 + d1" despite the noncommutativity (and the availability of
"{**d0, **d1}" for "true" dicts), and by extension the redundant "d |=
d1" for "d.update(d1)", I won't get terribly upset, but I will be sad
because it offends my sense of "beautiful code" (including TOOWTDI,
where "+" for dicts would violate both the "obvious" and the
parenthetical "only one" conditions IMO). I would consider it a wart
in the same way that many people consider str.join a wart, as it
breaks even more of the regularities I associate with "+" than string
concatenation does.
Again, I don't know what Antoine meant, but I might say the same kind
of thing in the same words, and the above is what I would mean.
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Brandt Bucher wrote:
> Before:
>
> tmp = keep.copy()
> tmp.update(separate)
> result = function(param=tmp)
> del tmp
> After:
>
> result = f(param=keep+separate)
I'd rewrite this example as:
Before:
fn(param={**keep, **separate})
After:
fn(param=keep + separate)
--
Jonathan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Rémi Lapeyre wrote: > Not matter the notation you end up choosing, I think this code: > https://github.com/jpadilla/pyjwt/blob/master/jwt/utils.py#L71-L81 > [...] would greatly benefit from a new merge to merge dicts. I've looked at the merge_dict defined in this code. It's similar to def gapfill(self, other): # See also: https://cobrapy.readthedocs.io/en/latest/gapfilling.html # Cobra's gapfill adds items to a model, to meet a requirement. for key in other.keys(): if key not in self: self[key] = other[key] (This is code I've written, that's not yet on PyPi.) The usage is different. Instead of writing one of aaa = merge_dict(aaa, bbb) ccc = merge_dict(aaa, bbb) you write one of gapfill(aaa, bbb) aaa.gapfill(bbb) # If gapfill added to dict methods. With merge_dict, you never really know if ccc is the same object as aaa, or a different one. Sometimes this is important. With gapfill, you get the same behaviour as the already familiar and loved dict.update. But of course with a different merge rule. -- Jonathan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
> > And to those who support this PEP, code examples where a dict merge > operator will help are most welcome! I would definitely include the example you alluded to in the operators thread: Before: tmp = keep.copy() tmp.update(separate) result = function(param=tmp) del tmp After: result = f(param=keep+separate) Thanks for drafting the PEP for this. There seems to be a bit of an echo in these 5+ threads, and your commentary has definitely been more constructive/original than most. Looking forward to the next revision! Brandt On Thu, Mar 21, 2019 at 9:42 AM Steven D'Aprano wrote: > I'd like to make a plea to people: > > I get it, there is now significant opposition to using the + symbol for > this proposed operator. At the time I wrote the first draft of the PEP, > there was virtually no opposition to it, and the | operator had very > little support. This has clearly changed. > > At this point I don't think it is productive to keep making subjective > claims that + will be more confusing or surprising. You've made your > point that you don't like it, and the next draft^1 of the PEP will make > that clear. > > But if you have *concrete examples* of code that currently is easy to > understand, but will be harder to understand if we add dict.__add__, > then please do show me! > > For those who oppose the + operator, it will help me if you made it > clear whether it is *just* the + symbol you dislike, and would accept > the | operator instead, or whether you hate the whole operator concept > regardless of how it is spelled. > > And to those who support this PEP, code examples where a dict merge > operator will help are most welcome! > > > > > ^1 Coming Real Soon Now™. > > > -- > Steven > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, 21 Mar 2019 at 17:27, Rémi Lapeyre wrote: > > Le 21 mars 2019 à 17:43:31, Steven D'Aprano > (st...@pearwood.info(mailto:st...@pearwood.info)) a écrit: > > > I'd like to make a plea to people: > > > > I get it, there is now significant opposition to using the + symbol for > > this proposed operator. At the time I wrote the first draft of the PEP, > > there was virtually no opposition to it, and the | operator had very > > little support. This has clearly changed. > > > > At this point I don't think it is productive to keep making subjective > > claims that + will be more confusing or surprising. You've made your > > point that you don't like it, and the next draft^1 of the PEP will make > > that clear. > > > > But if you have *concrete examples* of code that currently is easy to > > understand, but will be harder to understand if we add dict.__add__, > > then please do show me! > > > > For those who oppose the + operator, it will help me if you made it > > clear whether it is *just* the + symbol you dislike, and would accept > > the | operator instead, or whether you hate the whole operator concept > > regardless of how it is spelled. > > Thanks for the work you are doing on this PEP and for debunking my > misconceptions regarding types, I’m currently learning a lot about them. > > I don’t know if it matters but I’m in favor of the method > > > And to those who support this PEP, code examples where a dict merge > > operator will help are most welcome! > > Not matter the notation you end up choosing, I think this code: > https://github.com/jpadilla/pyjwt/blob/master/jwt/utils.py#L71-L81 > > which is part of a widely used library to validate JWTs would greatly > benefit from a new merge to merge dicts. (This package is 78 on > https://hugovk.github.io/top-pypi-packages/) It's already got a function that does the job. How much benefit is there *really* from being able to replace it with d1 + d2 once you drop support for Python < 3.8? But point taken that new code would have been able to avoid the function in the first place. ... or would it? def merge_dict(original, updates): if not updates: return original With the + operator. d1 + None will fail with an error. With your code, updates=None means "return the original unchanged". Does that matter with your current code? The point is that in many real world cases, you'd write a function *anyway*, to handle corner cases, and a new operator doesn't make much difference at that point. Having said all of that, I'm mostly indifferent to the idea of having a built in "dictionary merge" capability - I doubt I'd use it *much*, but if it were there I'm sure I'd find useful it on the odd occasion. I'm somewhat against an operator, I really don't see why this couldn't be a method (although the asymmetry in d1.merge(d2) makes me have a mild preference for a class method or standalone function). I can't form an opinion between + and |, I find | significantly uglier (I tend to avoid using it for sets, in favour of the union method) but I am mildly uncomfortable with more overloading of +. Serious suggestion - why not follow the lead of sets, and have *both* an operator and a method? And if you think that's a bad idea, it would be worth considering *why* it's a bad idea for dictionaries, when it's OK for sets (and "well, I didn't like it when sets did it" isn't sufficient ;-)) And having said that, I'll go back to lurking and not really caring one way or the other. Paul ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Steven D'Aprano wrote:
> But if you have *concrete examples* of code that currently is easy to
> understand, but will be harder to understand if we add dict.__add__,
> then please do show me!
# What does this do?
>>> items. update(points)
# And what does this do?
>>> items += points
What did you get? Here's one possible context.
>>> Point = namedtuple('Point', ['x', 'y'])
>>> p, q, r = Point(1,2), Point(3, 4), Point(5, 6)
>>> points = set([p, q, r])
>>> points
{Point(x=1, y=2), Point(x=5, y=6), Point(x=3, y=4)}
>>> items = dict(a=4, b=8)
--
Jonathan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Le 21 mars 2019 à 17:43:31, Steven D'Aprano (st...@pearwood.info(mailto:st...@pearwood.info)) a écrit: > I'd like to make a plea to people: > > I get it, there is now significant opposition to using the + symbol for > this proposed operator. At the time I wrote the first draft of the PEP, > there was virtually no opposition to it, and the | operator had very > little support. This has clearly changed. > > At this point I don't think it is productive to keep making subjective > claims that + will be more confusing or surprising. You've made your > point that you don't like it, and the next draft^1 of the PEP will make > that clear. > > But if you have *concrete examples* of code that currently is easy to > understand, but will be harder to understand if we add dict.__add__, > then please do show me! > > For those who oppose the + operator, it will help me if you made it > clear whether it is *just* the + symbol you dislike, and would accept > the | operator instead, or whether you hate the whole operator concept > regardless of how it is spelled. Thanks for the work you are doing on this PEP and for debunking my misconceptions regarding types, I’m currently learning a lot about them. I don’t know if it matters but I’m in favor of the method > And to those who support this PEP, code examples where a dict merge > operator will help are most welcome! Not matter the notation you end up choosing, I think this code: https://github.com/jpadilla/pyjwt/blob/master/jwt/utils.py#L71-L81 which is part of a widely used library to validate JWTs would greatly benefit from a new merge to merge dicts. (This package is 78 on https://hugovk.github.io/top-pypi-packages/) Rémi > ^1 Coming Real Soon Now™. > > > -- > Steven > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, 22 Mar 2019 03:42:00 +1100 Steven D'Aprano wrote: > > For those who oppose the + operator, it will help me if you made it > clear whether it is *just* the + symbol you dislike, and would accept > the | operator instead, or whether you hate the whole operator concept > regardless of how it is spelled. I'd rather see a method. Dict merging just doesn't occur often enough that an operator is desirable for it. > And to those who support this PEP, code examples where a dict merge > operator will help are most welcome! Yes, I still have no idea why this operator would supposedly be useful. How many dict merges do you write per month? Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
I'd like to make a plea to people: I get it, there is now significant opposition to using the + symbol for this proposed operator. At the time I wrote the first draft of the PEP, there was virtually no opposition to it, and the | operator had very little support. This has clearly changed. At this point I don't think it is productive to keep making subjective claims that + will be more confusing or surprising. You've made your point that you don't like it, and the next draft^1 of the PEP will make that clear. But if you have *concrete examples* of code that currently is easy to understand, but will be harder to understand if we add dict.__add__, then please do show me! For those who oppose the + operator, it will help me if you made it clear whether it is *just* the + symbol you dislike, and would accept the | operator instead, or whether you hate the whole operator concept regardless of how it is spelled. And to those who support this PEP, code examples where a dict merge operator will help are most welcome! ^1 Coming Real Soon Now™. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019 at 03:16:44PM +0200, Serhiy Storchaka wrote:
> 21.03.19 14:51, Chris Angelico пише:
> >... then, in the interests of productive discussion, could you please
> >explain? What is it about dict addition that makes it harder to
> >understand than other addition?
>
> Currently the + operator has 2 meanings for builtin types (both are
> widely used), after adding it for dicts it will have 3 meanings.
Just two meanings? I get at least eight among the builtins:
- int addition;
- float addition;
- complex addition;
- string concatenation;
- list concatenation;
- tuple concatenation;
- bytes concatenation;
- bytearray concatenation.
I suppose if you cover one eye and focus on the "big picture", ignoring
vital factors like "you can't add a list to a string" and "float
addition and int addition aren't precisely the same", we might pretend
that this is just two operations:
- numeric addition;
- sequence concatenation.
But in practice, when reading code, it's usually not enough to know that
some use of the + operator means "concatenation", you need to know
*what* is being concatenated. There's no point trying to add a tuple if
a bytearray is required.
> 3 > 2, is not?
Okay, but how does this make it harder to determine what a piece of code
using + does? Antoine insists that *if we allow dict addition*, then we
won't be able to tell what
spam + eggs # for example
does unless we know what spam and eggs are. This is very true. But it is
*equally true today*, right now, and its been equally true going back to
Python 1.0 or before.
This proposed change doesn't add any uncertainty that doesn't already
exist, nor will it make code that is clear today less clear tomorrow.^1
And don't forget that Python allows us to create non-builtin types that
overload operators. If you don't know what spam and eggs are, you can't
assume they are builtins. With operator overloading, any operator can
mean literally anything at all. In practice though, this rarely becomes
a serious problem.
Is there a significant increase in difficulty between the current
situation:
# Is this addition or concatenation or something else?
spam + eggs
versus the proposed:
# Is this addition or concatenation or merge or something else?
spam + eggs
Obviously there's *one more builtin* to consider, but I don't think that
changes the process of understanding the meaning of the operation.
I think that the problem you and Antoine fear ("dict.__add__ will make
it harder to read code") requires a process that goes something like
this:
1. Here's a mysterious "spam + eggs" operation we need to understand.
2. For each operation in ("numeric addition", "concatenation"):
3. assume + represents that operation;
4. if we understand the spam+eggs expression now, break
If that's how we read code, then adding one more operation would make it
harder to understand. We'd have to loop three times, not twice:
2. For each operation in ("numeric addition", "concatenation", "dict merging"):
Three is greater than two, so we may have to do more work to understand
the code. But I don't think that's how people actually read code. I
think they do this:
1. Here's a mysterious "spam + eggs" operation we need to understand.
2. Read the code to find out what spam and eggs are.
3. Knowing what they are (tuples, lists, floats, etc) immediately tells
you what the plus operator does; at worst, a programmer unfamiliar
with the type may need to read the docs.
Adding dict.__add__ doesn't make it any harder to work out what the
operands spam and eggs are. The process we go through to determine what
the operands are remains the same:
- if one of operands is a literal, that gives you a strong hint that
the other is the same type;
- the names or context may make it clear ("header + text" probably
isn't doing numeric addition);
- read back through the code looking for where the variables are
defined; etc.
That last bit isn't always easy. People can write obfuscated, complex
code using poor or misleading names. But allowing dict.__add__ doesn't
make it more obfuscated or more complex. Usually the naming and context
will make it clear. Most code is not terrible.
At worst, there will be a transition period where people have a
momentary surprise:
"Wait, what, these are dicts??? How can you add dicts???"
but then they will read the docs (or ask StackOverflow) and the second
time they see it, it shouldn't be a surprise.
^1 That's my assertion, but if anyone has a concrete example of actual
code which is self-evident today but will become ambiguous if this
proposal goes ahead, please show me!
--
Steven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
21.03.19 15:24, Chris Angelico пише: On Fri, Mar 22, 2019 at 12:17 AM Serhiy Storchaka wrote: 21.03.19 14:51, Chris Angelico пише: ... then, in the interests of productive discussion, could you please explain? What is it about dict addition that makes it harder to understand than other addition? Currently the + operator has 2 meanings for builtin types (both are widely used), after adding it for dicts it will have 3 meanings. 3 > 2, is not? I suppose you could call it two (numeric addition and sequence concatenation), but there are subtleties to the way that lists concatenate that don't apply to strings (esp since lists are mutable), so I'd call it at least three already. I do not understand what are these subtleties that you treat list concatenation different from string concatenation. Could you please explain? In any case, it does not matter how you count meanings, n + 1 > n. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019 at 9:17 AM Serhiy Storchaka wrote: > 21.03.19 14:51, Chris Angelico пише: > > ... then, in the interests of productive discussion, could you please > > explain? What is it about dict addition that makes it harder to > > understand than other addition? > > Currently the + operator has 2 meanings for builtin types (both are > widely used), after adding it for dicts it will have 3 meanings. > > 3 > 2, is not? > It depends how abstractly you define the "meanings". If you define + as "arithmetic addition" and "sequence concatenation", then yes, there are 2. But novices have to learn that the same concatenation operator applies to strings as well as lists/tuples. And when reading x + y, it is probably relevant whether x and y are numbers, strings, or sequence containers like lists. The proposal would generalize "sequence concatenation" to something like "asymmetric sequence/collection combination". (Asymmetric because d1 + d2 may not equal d2 + d1.) It seems a natural extension to me, though the | alternative is also reasonable (interpreted as taking the OR of keys in the two dicts; but unlike unioning two sets, the dict-merge operator would be asymmetric). The third proposed alternative, <<, has no "baggage" from an existing use as a combination operator, but at the same time it is a more obscure choice. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 22, 2019 at 12:17 AM Serhiy Storchaka wrote: > > 21.03.19 14:51, Chris Angelico пише: > > ... then, in the interests of productive discussion, could you please > > explain? What is it about dict addition that makes it harder to > > understand than other addition? > > Currently the + operator has 2 meanings for builtin types (both are > widely used), after adding it for dicts it will have 3 meanings. > > 3 > 2, is not? I suppose you could call it two (numeric addition and sequence concatenation), but there are subtleties to the way that lists concatenate that don't apply to strings (esp since lists are mutable), so I'd call it at least three already. And what about non-builtin types? You can add two numpy arrays and it does pairwise addition, quite different from how lists add together. But in every case, the + operator means "add these things together". It will be the same with dicts: you add the dicts together. Antoine has stated that the problem is NOT understanding what dict.__add__ does, so I am at a loss as to what the problem IS. We *already* have many different definitions of "add", according to the data types involved. That is exactly what polymorphism is for. Why is it such a bad thing for a dict? Now, my own opinion is that | would be a better operator than +, but it's only a weak preference, and I'd be happy with either. Also, to my understanding, the concerns about "what does addition mean" apply identically to "what does Or mean", but as we've already seen, my understanding doesn't currently extend as far as comprehending this issue. Hence asking. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 21/03/2019 11:34, Antoine Pitrou wrote: On Wed, 20 Mar 2019 15:46:24 -1000 Christopher Barker wrote: This is precisely why I worded my question this way: what has changed in the last 20 years that make a "+" dict operator more compelling today than it was? Do we merge dicts much more frequently than we did? The analogy doesn't hold because @ was a new operator -- a MUCH bigger change than dimply defining the use of + (or | ) for dicts. But it's less disruptive when reading code, because "x @ y" is unambiguous: it's a matrix multiplication. "x + y" can be many different things, and now it can be one more thing. "x @ y" is unambiguous once you know what it means. Until then, it's just mysterious. I wouldn't mind the new operator if its meaning was clear-cut. But here we have potential for confusion, both for writers and readers of code. but it's NOT a new operator, it is making use of an existing one, and sure you could guess at a couple meanings, but the merge one is probably one of the most obvious to guess, and one quick test and you know -- I really can't see it being a ongoing source of confusion. Did you actually read what I said? The problem is not to understand what dict.__add__ does. It's to understand what code using the + operator does, without knowing upfront whether the inputs are dicts. Welcome to polymorphism. -- Rhodri James *-* Kynesim Ltd ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
21.03.19 14:51, Chris Angelico пише: ... then, in the interests of productive discussion, could you please explain? What is it about dict addition that makes it harder to understand than other addition? Currently the + operator has 2 meanings for builtin types (both are widely used), after adding it for dicts it will have 3 meanings. 3 > 2, is not? ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, 21 Mar 2019 23:51:12 +1100 Chris Angelico wrote: > On Thu, Mar 21, 2019 at 11:45 PM Antoine Pitrou wrote: > > > > On Thu, 21 Mar 2019 23:35:36 +1100 > > Chris Angelico wrote: > > > On Thu, Mar 21, 2019 at 10:35 PM Antoine Pitrou > > > wrote: > > > > > but it's NOT a new operator, it is making use of an existing one, and > > > > > sure > > > > > you could guess at a couple meanings, but the merge one is probably > > > > > one of > > > > > the most obvious to guess, and one quick test and you know -- I really > > > > > can't see it being a ongoing source of confusion. > > > > > > > > Did you actually read what I said? The problem is not to understand > > > > what dict.__add__ does. It's to understand what code using the + > > > > operator does, without knowing upfront whether the inputs are dicts. > > > > > > The + operator adds two things together. I don't understand the issue > > > here. > > > > I'm not expecting you to understand, either. > > > > ... then, in the interests of productive discussion, could you please > explain? What is it about dict addition that makes it harder to > understand than other addition? "Productive discussion" is something that requires mutual implication. Asking me to repeat exactly what I spelled out above (and that you even quoted) is not productive. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019 at 11:45 PM Antoine Pitrou wrote: > > On Thu, 21 Mar 2019 23:35:36 +1100 > Chris Angelico wrote: > > On Thu, Mar 21, 2019 at 10:35 PM Antoine Pitrou wrote: > > > > but it's NOT a new operator, it is making use of an existing one, and > > > > sure > > > > you could guess at a couple meanings, but the merge one is probably one > > > > of > > > > the most obvious to guess, and one quick test and you know -- I really > > > > can't see it being a ongoing source of confusion. > > > > > > Did you actually read what I said? The problem is not to understand > > > what dict.__add__ does. It's to understand what code using the + > > > operator does, without knowing upfront whether the inputs are dicts. > > > > The + operator adds two things together. I don't understand the issue here. > > I'm not expecting you to understand, either. > ... then, in the interests of productive discussion, could you please explain? What is it about dict addition that makes it harder to understand than other addition? ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, 21 Mar 2019 23:35:36 +1100 Chris Angelico wrote: > On Thu, Mar 21, 2019 at 10:35 PM Antoine Pitrou wrote: > > > but it's NOT a new operator, it is making use of an existing one, and sure > > > you could guess at a couple meanings, but the merge one is probably one of > > > the most obvious to guess, and one quick test and you know -- I really > > > can't see it being a ongoing source of confusion. > > > > Did you actually read what I said? The problem is not to understand > > what dict.__add__ does. It's to understand what code using the + > > operator does, without knowing upfront whether the inputs are dicts. > > The + operator adds two things together. I don't understand the issue here. I'm not expecting you to understand, either. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 21, 2019 at 10:35 PM Antoine Pitrou wrote: > > but it's NOT a new operator, it is making use of an existing one, and sure > > you could guess at a couple meanings, but the merge one is probably one of > > the most obvious to guess, and one quick test and you know -- I really > > can't see it being a ongoing source of confusion. > > Did you actually read what I said? The problem is not to understand > what dict.__add__ does. It's to understand what code using the + > operator does, without knowing upfront whether the inputs are dicts. The + operator adds two things together. I don't understand the issue here. You can add integers: 1 + 2 == 3 You can add floats: 0.5 + 1.25 == 1.75 You can add lists: [1,2] + [3,4] == [1,2,3,4] You can add strings: "a" + "b" == "ab" And soon you'll be able to add dictionaries. The exact semantics need to be defined, but it's not fundamentally changing how you interpret the + operator. I don't understand the panic here - or rather, I don't understand why it's happening NOW, not back when lists got the ability to be added (if that wasn't in the very first release). Conversely, if it's the | operator, it's a matter of merging, and the same is true. You can merge integers, treating them as bit sets. You can merge sets. And now you'll be able to merge dictionaries. Same same. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, 20 Mar 2019 15:46:24 -1000 Christopher Barker wrote: > > > This is precisely why I worded my question this way: what has changed > > in the last 20 years that make a "+" dict operator more compelling > > today than it was? Do we merge dicts much more frequently than we > > did? > > The analogy doesn't hold because @ was a new operator -- a MUCH bigger > change than dimply defining the use of + (or | ) for dicts. But it's less disruptive when reading code, because "x @ y" is unambiguous: it's a matrix multiplication. "x + y" can be many different things, and now it can be one more thing. > I wouldn't mind the new operator if its meaning was clear-cut. But > > here we have potential for confusion, both for writers and readers of > > code. > > > > but it's NOT a new operator, it is making use of an existing one, and sure > you could guess at a couple meanings, but the merge one is probably one of > the most obvious to guess, and one quick test and you know -- I really > can't see it being a ongoing source of confusion. Did you actually read what I said? The problem is not to understand what dict.__add__ does. It's to understand what code using the + operator does, without knowing upfront whether the inputs are dicts. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Sat, Mar 16, 2019 at 12:39 AM Antoine Pitrou wrote: > On Sat, 16 Mar 2019 01:41:59 +1100 > Steven D'Aprano wrote: > > > Matrix multiplication is a perfect example: adding the @ operator could > > have been done in Python 0.1 if anyone had thought of it, but it took 15 > > years of numerical folk "whinging" about the lack until it happened: > > Not so perfect, as the growing use of Python for scientific computing > has made it much more useful to promote a dedicated matrix > multiplication operator than, say, 15 or 20 years ago. > Theres more to it than that, really, but not really relevant here... > This is precisely why I worded my question this way: what has changed > in the last 20 years that make a "+" dict operator more compelling > today than it was? Do we merge dicts much more frequently than we > did? The analogy doesn't hold because @ was a new operator -- a MUCH bigger change than dimply defining the use of + (or | ) for dicts. I wouldn't mind the new operator if its meaning was clear-cut. But > here we have potential for confusion, both for writers and readers of > code. > but it's NOT a new operator, it is making use of an existing one, and sure you could guess at a couple meanings, but the merge one is probably one of the most obvious to guess, and one quick test and you know -- I really can't see it being a ongoing source of confusion. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Sat, 16 Mar 2019 03:44:02 +1100 Steven D'Aprano wrote: > On Fri, Mar 15, 2019 at 12:34:45PM +0100, Antoine Pitrou wrote: > > On Thu, 7 Mar 2019 10:58:02 +1100 > > Chris Angelico wrote: > > > > > > Lots of words that basically say: Stuff wouldn't be perfectly pure. > > > > Chris, please learn to think twice before contributing what is > > essentially a trivialization of someone else's arguments. You're not > > doing anything useful here, and are just sounding like an asshole who > > wants to shut people up. > > I don't think you are being fair here, and I'd rather avoid getting into > unhelpful arguments about tone and whether Chris is "trivializing" (a > perjorative term) or "simplifying" (a more neutral term) Josh's > position. But if you feel that Chris (and I) have missed parts of Josh's > argument, then by all means point out what we missed. When someone posts an elaborate argument (regardless of whether they are right or not) and someone else responds a one-liner that reduces it to "lots of words" and claims to rephrase it as a short caricatural statement, then it seems fair to me to characterize it as "trivializing". But if you feel that I missed a subtlety in Chris' position, and if you feel he was more respectful of the OP than I felt he was, then by all means point out what I missed. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Sat, 16 Mar 2019 01:59:07 +1100 Steven D'Aprano wrote: > On Fri, Mar 15, 2019 at 12:25:22PM +0100, Antoine Pitrou wrote: > > > Yeah, well I do think "+=" for lists was a mistake. I *still* have > > trouble remembering the exact difference between "list +=" and > > "list.extend" (yes, there is one: one accepts more types than the > > other... which one it is, and why, I never remember; > > Both accept arbitrary iterables, and the documentation suggests that > they are the same: > > https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types > > Perhaps you are thinking of the difference between list + list versus > list += iterable? Hmm, it looks like I misremembered indeed. Thanks for correcting this. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Sat, 16 Mar 2019 01:41:59 +1100 Steven D'Aprano wrote: > > Matrix multiplication is a perfect example: adding the @ operator could > have been done in Python 0.1 if anyone had thought of it, but it took 15 > years of numerical folk "whinging" about the lack until it happened: Not so perfect, as the growing use of Python for scientific computing has made it much more useful to promote a dedicated matrix multiplication operator than, say, 15 or 20 years ago. This is precisely why I worded my question this way: what has changed in the last 20 years that make a "+" dict operator more compelling today than it was? Do we merge dicts much more frequently than we did? I don't think so. > Or the infamous := operator, which ultimately is a useful but minor > syntactic and semantic change but generated a huge amount of debate, > argument and negativity. ... and is likely to be a mistake as well. Justifying future mistakes with past mistakes doesn't sound very reasonable ;-) > I still remember being told in no uncertain terms by the core devs that > adding a clear() method to lists was a waste of time because there was > already a perfectly good way to spell it with slicing. And then ABCs > came along and now lists have a clear method. So opinions change too. Not really the same problem. The "+" dict operator is not intuitively obvious in its meaning, while a "clear()" method on lists is. I wouldn't mind the new operator if its meaning was clear-cut. But here we have potential for confusion, both for writers and readers of code. > > Besides, if I have two dicts with e.g. lists as values, I *really* > > dislike the fact that the + operator will clobber the values rather than > > concatenate them. It's a recipe for confusion. > > Are you confused that the update method clobbers list values rather than > concatenate them? I doubt that you are. > > So why would it be confusing to say that + does a copy-and-update? Because it's named "+" precisely. You know, names are important. ;-) Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Sat, Mar 16, 2019 at 09:04:22PM +1300, Greg Ewing wrote: > Another random thought about this: Mathematicians use addition as a > metaphor for quite a range of different things, but they tend to only > use the symbols ∪ and ∩ for actual sets, or things that are very > set-like. So maybe that's an argument for using '+' rather than '|' > for dict merging. If one views an ordered dict as an assoc list, '+' would mean prepending the new values to the existing ones. If one views an unordered dict as a set of ordered pairs, '|' would make sense. Stefan Krah ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Another random thought about this: Mathematicians use addition as a metaphor for quite a range of different things, but they tend to only use the symbols ∪ and ∩ for actual sets, or things that are very set-like. So maybe that's an argument for using '+' rather than '|' for dict merging. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 11:15 AM Steven D'Aprano wrote: > On Fri, Mar 15, 2019 at 10:34:45AM -0700, Brett Cannon wrote: > > > Watch the tone please. > > Brett, you might have missed my comment about wanting to avoid unhelpful > arguments about tone, but if you are going to complain about people's > tone, the considerate thing to do is to say what it is that you're > objecting to. > The phrasing of "just sounding like an asshole who wants to shut people up" is unnecessary. > > Otherwise we're left guessing as to what it is and whether or not you > are making an implied threat to apply the CoC. > No implied "threat". If it was an official warning then I would have said so. > > I responded to Antoine's post earlier, but thought that it was a > respectful disagreement. Do you think that's not the case? > I think it skirts the edge of being disrespectful, hence the request to please be aware of how one comes across. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 10:34:45AM -0700, Brett Cannon wrote: > Watch the tone please. Brett, you might have missed my comment about wanting to avoid unhelpful arguments about tone, but if you are going to complain about people's tone, the considerate thing to do is to say what it is that you're objecting to. Otherwise we're left guessing as to what it is and whether or not you are making an implied threat to apply the CoC. I responded to Antoine's post earlier, but thought that it was a respectful disagreement. Do you think that's not the case? -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 4:36 AM Antoine Pitrou wrote: > On Thu, 7 Mar 2019 10:58:02 +1100 > Chris Angelico wrote: > > > > Lots of words that basically say: Stuff wouldn't be perfectly pure. > > Chris, please learn to think twice before contributing what is > essentially a trivialization of someone else's arguments. You're not > doing anything useful here, and are just sounding like an asshole who > wants to shut people up. > Watch the tone please. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 3/15/2019 11:21 AM, Steven D'Aprano wrote: On Fri, Mar 15, 2019 at 11:54:51AM -0300, Andre Roberge wrote: On Fri, Mar 15, 2019 at 11:42 AM Steven D'Aprano wrote: [snip] I still remember being told in no uncertain terms by the core devs that adding a clear() method to lists was a waste of time because there was already a perfectly good way to spell it with slicing. And then ABCs came along and now lists have a clear method. So opinions change too. I agree with the opinions expressed in the (partially) quoted message but I don't think that this is how this particular change happened. https://mail.python.org/pipermail/python-ideas/2009-April/003897.html You proposed that in April 2009, but there was nothing added to the bug tracker for 18 months until it was finally added by Terry Reedy in Actually, I opened the tracker issue with a succinct message, after the discussion and Guido's approval changed my mind. https://bugs.python.org/issue10516 However, Eli Bendersky wrote the patch with help from others and then merged it. November 2010, based on discussion in a completely different thread (one about sets!): https://mail.python.org/pipermail/python-ideas/2010-November/008722.html -- Terry Jan Reedy ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 12:34:45PM +0100, Antoine Pitrou wrote: > On Thu, 7 Mar 2019 10:58:02 +1100 > Chris Angelico wrote: > > > > Lots of words that basically say: Stuff wouldn't be perfectly pure. > > Chris, please learn to think twice before contributing what is > essentially a trivialization of someone else's arguments. You're not > doing anything useful here, and are just sounding like an asshole who > wants to shut people up. I don't think you are being fair here, and I'd rather avoid getting into unhelpful arguments about tone and whether Chris is "trivializing" (a perjorative term) or "simplifying" (a more neutral term) Josh's position. But if you feel that Chris (and I) have missed parts of Josh's argument, then by all means point out what we missed. Josh, the same applies to you: I do want to give your objections a fair hearing in the updated PEP, so if you think I've missed something, please point it out. In context, I think Chris' response was valid: he was responding to a post by Josh whose entire argument was that using + for dict merging is an abuse of the + symbol because it isn't like numeric addition. If there is more to Josh's argument, can you point out to me what I have missed please? That's a genuine request, not a rhetorical question. Here's Josh's argument: https://mail.python.org/pipermail/python-ideas/2019-March/055733.html and for context, here is Chris' dismissal of Josh's argument: https://mail.python.org/pipermail/python-ideas/2019-March/055734.html and his explanation of why he is dismissing it. Chris is well within his right to dismiss an argument that doesn't impress him, which he did by summarizing it as "Stuff wouldn't be perfectly pure". (Pure in the sense of being like numeric addition.) I think that's pretty much an accurate summary: Josh apparently doesn't like using + for anything that isn't purely like + for real numbers. He calls using + for concatentation a "minor abuse" of the operator and argues that it would be bad for dict meging to use + because merging has different properties to numeric addition. (He has also criticised the use of + for concatenation in at least one other post.) He even gives qualified support for a dict merge operator: "there's nothing wrong with making dict merges easier" but just doesn't like the choice of + as the operator. He's entitled to his opinion, and Chris is entitled to dismiss it. (Aside: your email appears to have broken threading. I'm not sure why, your other emails seem to be threaded okay.) -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 11:54:51AM -0300, Andre Roberge wrote: > On Fri, Mar 15, 2019 at 11:42 AM Steven D'Aprano > wrote: > > > [snip] > > > > I still remember being told in no uncertain terms by the core devs that > > adding a clear() method to lists was a waste of time because there was > > already a perfectly good way to spell it with slicing. And then ABCs > > came along and now lists have a clear method. So opinions change too. > > > > I agree with the opinions expressed in the (partially) quoted message > but I don't think that this is how this particular change happened. > > https://mail.python.org/pipermail/python-ideas/2009-April/003897.html You proposed that in April 2009, but there was nothing added to the bug tracker for 18 months until it was finally added by Terry Reedy in November 2010, based on discussion in a completely different thread (one about sets!): https://mail.python.org/pipermail/python-ideas/2010-November/008722.html Contrary-wise, a few years earlier the same request had been roundly dismissed by core-devs and Python luminaries as "inane", "redundant" and "trivial". https://mail.python.org/pipermail/python-list/2006-April/356236.html People can change their mind -- something that is dismissed one year may be accepted some years later on. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 12:25:22PM +0100, Antoine Pitrou wrote: > Yeah, well I do think "+=" for lists was a mistake. I *still* have > trouble remembering the exact difference between "list +=" and > "list.extend" (yes, there is one: one accepts more types than the > other... which one it is, and why, I never remember; Both accept arbitrary iterables, and the documentation suggests that they are the same: https://docs.python.org/3/library/stdtypes.html#mutable-sequence-types Perhaps you are thinking of the difference between list + list versus list += iterable? [...] > There is a virtue to > > """There should be one-- and preferably only one --obvious way to do > it""" "It" here refers to two different things: "I want to update a dict in place": The Obvious Way is to use the update method; the fact that += works as well is just a side-effect of the way augmented assignments are defined. "I want a new dict that merges two existing dicts": The Obvious Way is to use the merge operator (possibly spelled + but that's not written in stone yet). -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 11:42 AM Steven D'Aprano wrote: > [snip] > > I still remember being told in no uncertain terms by the core devs that > adding a clear() method to lists was a waste of time because there was > already a perfectly good way to spell it with slicing. And then ABCs > came along and now lists have a clear method. So opinions change too. > > I agree with the opinions expressed in the (partially) quoted message but I don't think that this is how this particular change happened. https://mail.python.org/pipermail/python-ideas/2009-April/003897.html ;-) ;-) André Roberge ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 15, 2019 at 12:20:21PM +0100, Antoine Pitrou wrote:
> Agreed with this. What is so useful exactly in this new dict operator
> that it hasn't been implemented, say, 20 years ago?
One could say the same thing about every new feature. Since Python 1.5
was so perfect, why add Unicode, decorators, matrix multiplication,
async, descriptors, Decimal, iterators, ...
Matrix multiplication is a perfect example: adding the @ operator could
have been done in Python 0.1 if anyone had thought of it, but it took 15
years of numerical folk "whinging" about the lack until it happened:
https://mail.python.org/pipermail/python-ideas/2014-March/027053.html
In some ways, it is often easier to get community buy-in for *big*
changes, provided they are backwards compatible. With a big change,
people often either want it, or don't care one way or another.
(Sometimes because the big change is too big or complicated or difficult
for them to understand -- I feel that way about async. Some day I'll
deal with it, but right now it's so far down my list of priorities that
I have no opinion on anything to do with async.)
But *little* changes are easy enough for everyone to understand, and so
they trigger the impulse to bike-shed. Everyone has an opinion on
whether or not dicts should support an update operator, and whether to
spell it + or | or <- or << or something else.
Or the infamous := operator, which ultimately is a useful but minor
syntactic and semantic change but generated a huge amount of debate,
argument and negativity. A far smaller change to the language than
adding type hinting, but it generated far more argument.
I still remember being told in no uncertain terms by the core devs that
adding a clear() method to lists was a waste of time because there was
already a perfectly good way to spell it with slicing. And then ABCs
came along and now lists have a clear method. So opinions change too.
Things happen when they happen, because if they had happened earlier we
wouldn't still be arguing about them.
> I rarely find
> myself merging dicts and, when I do, calling dict.update() is entirely
> acceptable
The code we write is shaped by the operators and methods that
exist. You use dict.update() because *it exists* so when you want a new
dict merged with another, you write the code that is possible today:
new = spam.copy()
new.update(eggs)
process(new)
and you are content because you "rarely find myself merging dicts".
But perhaps those who *frequently* merge dicts have a different option,
and would prefer to write one line rather than three and avoid naming
something that doesn't need a name:
process(spam + eggs) # or spam|eggs if you prefer
> (I think the "{**d}" notation was already a mistake, making
> a perfectly readable operation more cryptic simply for the sake of
> saving a few keystrokes).
I don't know if it was a mistake, but disatisfaction with its lack of
readability and discoverability is one of the motivations of this PEP.
[...]
> Besides, if I have two dicts with e.g. lists as values, I *really*
> dislike the fact that the + operator will clobber the values rather than
> concatenate them. It's a recipe for confusion.
Are you confused that the update method clobbers list values rather than
concatenate them? I doubt that you are.
So why would it be confusing to say that + does a copy-and-update?
(In any case, popular opinion may be shifting towards preferring the |
operator over + so perhaps confusion over concatenation may not be an
issue in the future.)
--
Steven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, 7 Mar 2019 10:58:02 +1100 Chris Angelico wrote: > > Lots of words that basically say: Stuff wouldn't be perfectly pure. Chris, please learn to think twice before contributing what is essentially a trivialization of someone else's arguments. You're not doing anything useful here, and are just sounding like an asshole who wants to shut people up. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Mon, 4 Mar 2019 15:57:38 -0800 Guido van Rossum wrote: > > > Those two points make me uncomfortable with "+=" strictly behaving > > like ".update()". > > And yet that's how it works for lists. (Note that dict.update() still has > capabilities beyond +=, since you can also invoke it with keyword args.) Yeah, well I do think "+=" for lists was a mistake. I *still* have trouble remembering the exact difference between "list +=" and "list.extend" (yes, there is one: one accepts more types than the other... which one it is, and why, I never remember; and, of course, there might be the obscure performance difference because of CPython's execution details). I should not have to remember whether I want to use "list +=" or "list.extend" every time I need to extend a list. There is a virtue to """There should be one-- and preferably only one --obvious way to do it""" and we shouldn't break it more than we already did. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Mon, 4 Mar 2019 16:02:06 +0100 Stefan Behnel wrote: > INADA Naoki schrieb am 04.03.19 um 11:15: > > Why statement is not enough? > > I'm not sure I understand why you're asking this, but a statement is "not > enough" because it's a statement and not an expression. This is an argument for Perl 6, not for Python. Regards Antoine. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, 6 Mar 2019 00:46:57 +
Josh Rosenberg
wrote:
>
> Overloading + lacks the clear descriptive aspect of update that describes
> the goal of the operation, and contradicts conventions (in Python and
> elsewhere) about how + works (addition or concatenation, and a lot of
> people don't even like it doing the latter, though I'm not that pedantic).
>
> A couple "rules" from C++ on overloading are "*Whenever the meaning of an
> operator is not obviously clear and undisputed, it should not be
> overloaded.* *Instead, provide a function with a well-chosen name.*"
> and "*Always
> stick to the operator’s well-known semantics".* (Source:
> https://stackoverflow.com/a/4421708/364696 , though the principle is
> restated in many other places).
Agreed with this. What is so useful exactly in this new dict operator
that it hasn't been implemented, say, 20 years ago? I rarely find
myself merging dicts and, when I do, calling dict.update() is entirely
acceptable (I think the "{**d}" notation was already a mistake, making
a perfectly readable operation more cryptic simply for the sake of
saving a few keystrokes).
Built-in operations should be added with regard to actual user needs
(such as: a first-class notation for matrix multiplication, making
formulas easier to read and understand), not a mere "hmm this might
sometimes be useful".
Besides, if I have two dicts with e.g. lists as values, I *really*
dislike the fact that the + operator will clobber the values rather than
concatenate them. It's a recipe for confusion.
Regards
Antoine.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Just in case I'm not the only one that had a hard time finding the latest version of this PEP, here it is in the PEPS Repo: https://github.com/python/peps/blob/master/pep-0584.rst -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Sat, Mar 09, 2019 at 11:39:39AM -0800, Stephan Hoyer wrote: > Would __iadd__ and __isub__ be added to collections.abc.MutableMapping? No, that will not be part of the PEP. The proposal is only to change dict itself. If people want to add this to MutableMapping, that could be considered seperately. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Would __iadd__ and __isub__ be added to collections.abc.MutableMapping? This would be consistent with other infix operations on mutable ABCs, but could potentially break backwards compatibility for anyone who has defined a MutableMapping subclass that implements __add__ but not __iadd__. On Sat, Mar 9, 2019 at 8:55 AM Steven D'Aprano wrote: > Thanks to everyone who has contributed to the discussion, I have been > reading all the comments even if I haven't responded. > > I'm currently working on an update to the PEP which will, I hope, > improve some of the failings of the current draft. > > > -- > Steven > ___ > Python-ideas mailing list > Python-ideas@python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Thanks to everyone who has contributed to the discussion, I have been reading all the comments even if I haven't responded. I'm currently working on an update to the PEP which will, I hope, improve some of the failings of the current draft. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 8, 2019 at 11:25 AM João Matos wrote:
> I've just read your PEP 585 draft and have some questions.
> When you say
> "
>
> Like the merge operator and list concatenation, the difference operator
> requires both operands to be dicts, while the augmented version allows any
> iterable of keys.
>
> >>> d - {'spam', 'parrot'}
> Traceback (most recent call last):
> ...
> TypeError: cannot take the difference of dict and set
>
> >>> d -= {'spam', 'parrot'}
> >>> print(d)
> {'eggs': 2, 'cheese': 'cheddar'}
>
> >>> d -= [('spam', 999)]
> >>> print(d)
> {'spam': 999, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
>
>
> "
>
> The option d -= {'spam', 'parrot'} where parrot does not exist in the d
> dict, will raise an exception (eg. KeyNotFound) or be silent?
>
> The option d -= [('spam', 999)] should remove the pair from the dict,
> correct? But the print that follows still shows it there. It's a mistake or
> am I missing something?
>
My understanding is that:
- (Q1) Attempting to discard a key not in the target of the augmented
assignment would *not *raise a KeyError (or any Exception for that
matter). This is analogous to how the - operator works on sets and is
consistent with the pure python implementation towards the bottom of the
PEP.
- (Q2) This one got me as well while implementing the proposal in cpython,
but there is a difference in what "part" of the RHS the operators "care
about" if the RHS isn't a dict. The += operator expects 2-tuples and will
treat them as (key, value) pairs. The -= operator doesn't attempt to
unpack the RHS's elements as += does and expects keys. So d -= [('spam',
999)] treated the tuple as a *key *and attempted to discard it.
IOW,
d = {
'spam': 999,
('spam', 999): True
}
d -= [('spam', 999)]
Would discard the *key* ('spam', 999) and corresponding value True.
Which highlights a possibly surprising incongruence between the operators:
d = {}
update = [(1,1), (2,2), (3,3)]
d += update
d -= update
assert d == {} # will raise, as d still has 3 items
Similarly,
d = {}
update = {1:1, 2:2, 3:3}
d += update.items()
d -= update.items()
assert d == {} # will raise, for the same reason
d -= update.keys()
assert d == {} # would pass without issue
That being said I (personally) wouldn't consider it a deal-breaker and
still would very much appreciate of the added functionality (regardless of
the choice of operator).
- Jim
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Guido van Rossum wrote: I guess everybody's high school math(s) class was different. I don't ever recall seeing + and * for boolean OR/AND; we used ∧ and ∨. Boolean algebra was only touched on briefly in my high school years. I can't remember exactly what notation was used, but it definitely wasn't ∧ and ∨ -- I didn't encounter those until much later. However, I've definitely seen texts on boolean alegbra in relation to logic circuits that write 'A and B' as 'AB', and 'A or B' as 'A + B'. (And also use an overbar for negation instead of the mathematical ¬). Maybe it depends on whether you're a mathematician or an engineer? The multiplication-addition notation seems a lot more readable when you have a complicated boolean expression, so I can imagine it being favoured by pragmatic engineering type people. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Fri, Mar 8, 2019 at 3:33 PM Greg Ewing wrote: > Guido van Rossum wrote: > > I guess this explains the behavior of removing results <= 0; it makes > > sense as multiset subtraction, since in a multiset a negative count > > makes little sense. (Though the name Counter certainly doesn't seem to > > imply multiset.) > > It doesn't even behave consistently as a multiset, since c[k] -= n > is happy to let the value go negative. > > > For sets, > > union and intersection are distributive over each other. > > > Note that this is *not* the case for + and * when used with > > (mathematical) numbers... So in a sense, SETL (which uses + and * > > for union and intersection got the operators wrong. > > But in another sense, it didn't. In Boolean algebra, "and" and "or" > (which also distribute over each other) are often written using the > same notations as multiplication and addition. There's no rule in > mathematics saying that these notations must be distributive in one > direction but not the other. > I guess everybody's high school math(s) class was different. I don't ever recall seeing + and * for boolean OR/AND; we used ∧ and ∨. I learned | and & for set operations only after I learned programming; I think it was in PL/1. But of course it stuck because of C bitwise operators (which are also boolean OR/AND and set operations). This table suggests there's a lot of variety in how these operators are spelled: https://en.wikipedia.org/wiki/List_of_logic_symbols -- --Guido van Rossum (python.org/~guido) ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Guido van Rossum wrote: I guess this explains the behavior of removing results <= 0; it makes sense as multiset subtraction, since in a multiset a negative count makes little sense. (Though the name Counter certainly doesn't seem to imply multiset.) It doesn't even behave consistently as a multiset, since c[k] -= n is happy to let the value go negative. For sets, union and intersection are distributive over each other. Note that this is *not* the case for + and * when used with (mathematical) numbers... So in a sense, SETL (which uses + and * > for union and intersection got the operators wrong. But in another sense, it didn't. In Boolean algebra, "and" and "or" (which also distribute over each other) are often written using the same notations as multiplication and addition. There's no rule in mathematics saying that these notations must be distributive in one direction but not the other. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 2019-03-08 16:55, Guido van Rossum wrote: [snip] If we were to use "|" and "&" for dict "union" and "intersection", the mutual distributive properties will hold. Since "|" (especially "|=") *is* suitable for "update", I think we should reserve "+" for some future commutative extension. One argument is that sets have an update() method aliased to "|=", so this makes it more reasonable to do the same for dicts, which also have a. update() method, with similar behavior (not surprising, since sets were modeled after dicts). [snip] One way to think of it is that a dict is like a set, except that each of its members has an additional associated value. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 7, 2019 at 9:12 PM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:
> Ka-Ping Yee writes:
> > On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico wrote:
>
> > > But adding dictionaries is fundamentally *useful*. It is expressive.
> >
> > It is useful. It's just that + is the wrong name.
>
> First, let me say that I prefer ?!'s position here, so my bias is made
> apparent. I'm also aware that I have biases so I'm sympathetic to
> those who take a different position.
>
TBH, I am warming up to "|" as well.
> Rather than say it's "wrong", let me instead point out that I think
> it's pragmatically troublesome to use "+". I can think of at least
> four interpretations of "d1 + d2"
>
> 1. update
> 2. multiset (~= Collections.Counter addition)
>
I guess this explains the behavior of removing results <= 0; it makes sense
as multiset subtraction, since in a multiset a negative count makes little
sense. (Though the name Counter certainly doesn't seem to imply multiset.)
> 3. addition of functions into the same vector space (actually, a
> semigroup will do ;-), and this is the implementation of
> Collections.Counter
> 4. "fiberwise" set addition (ie, of functions into relations)
>
> and I'm very jet-lagged so I may be missing some.
>
> There's also the fact that the operations denoted by "|" and "||" are
> often implemented as "short-circuiting", and therefore not
> commutative, while "+" usually is (and that's reinforced for
> mathematicians who are trained to think of "+" as the operator for
> Abelian groups, while "*" is a (possibly) non-commutative operator. I
> know commutativity of "+" has been mentioned before, but the
> non-commutativity of "|" -- and so unsuitability for many kinds of
> dict combination -- hasn't been emphasized before IIRC.
>
I've never heard of single "|" being short-circuiting. ("||" of course is
infamous for being that in C and most languages derived from it.)
And "+" is of course used for many non-commutative operations in Python
(e.g. adding two lists/strings/tuples together). It is only *associative*,
a weaker requirement that just says (A + B) + C == A + (B + C). (This is
why we write A + B + C, since the grouping doesn't matter for the result.)
Anyway, while we're discussing mathematical properties, and since SETL was
briefly mentioned, I found an interesting thing in math. For sets, union
and intersection are distributive over each other. I can't type the
operators we learned in high school, so I'll use Python's set operations.
We find that A | (B & C) == (A | B) & (A | C). We also find that A & (B |
C) == (A & B) | (A & C).
Note that this is *not* the case for + and * when used with (mathematical)
numbers: * distributes over +: a * (b + c) == (a * b) + (a * c), but + does
not distribute over *: a + (b * c) != (a + b) * (a + c). So in a sense,
SETL (which uses + and * for union and intersection) got the operators
wrong.
Note that in Python, + and * for sequences are not distributive this way,
since (A + B) * n is not the same as (A * n) + (B * n). OTOH A * (n + m) ==
A * n + A * m. (Assuming A and B are sequences of the same type, and n and
m are positive integers.)
If we were to use "|" and "&" for dict "union" and "intersection", the
mutual distributive properties will hold.
> Since "|" (especially "|=") *is* suitable for "update", I think we
> should reserve "+" for some future commutative extension.
>
One argument is that sets have an update() method aliased to "|=", so this
makes it more reasonable to do the same for dicts, which also have a.
update() method, with similar behavior (not surprising, since sets were
modeled after dicts).
> In the spirit of full disclosure:
> Of these, 2 is already implemented and widely used, so we don't need
> to use dict.__add__ for that. I've never seen 4 in the mathematical
> literature (union of relations is not the same thing). 3, however, is
> very common both for mappings with small domain and sparse
> representation of mappings with a default value (possibly computed
> then cached), and "|" is not suitable for expressing that sort of
> addition (I'm willing to say it's "wrong" :-).
>
--
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Hello,
I've just read your PEP 585 draft and have some questions.
When you say
"
Like the merge operator and list concatenation, the difference
operator requires both operands to be dicts, while the augmented
version allows any iterable of keys.
>>> d - {'spam', 'parrot'}
Traceback (most recent call last):
...
TypeError: cannot take the difference of dict and set
>>> d -= {'spam', 'parrot'}
>>> print(d)
{'eggs': 2, 'cheese': 'cheddar'}
>>> d -= [('spam', 999)]
>>> print(d)
{'spam': 999, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
"
The option d -= {'spam', 'parrot'} where parrot does not exist in
the d dict, will raise an exception (eg. KeyNotFound) or be silent?
The option d -= [('spam', 999)] should remove the pair from the
dict, correct? But the print that follows still shows it there. It's
a mistake or am I missing something?
Best regards,
João Matos
smime.p7s
Description: S/MIME Cryptographic Signature
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Ka-Ping Yee writes: > On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico wrote: > > But adding dictionaries is fundamentally *useful*. It is expressive. > > It is useful. It's just that + is the wrong name. First, let me say that I prefer ?!'s position here, so my bias is made apparent. I'm also aware that I have biases so I'm sympathetic to those who take a different position. Rather than say it's "wrong", let me instead point out that I think it's pragmatically troublesome to use "+". I can think of at least four interpretations of "d1 + d2" 1. update 2. multiset (~= Collections.Counter addition) 3. addition of functions into the same vector space (actually, a semigroup will do ;-), and this is the implementation of Collections.Counter 4. "fiberwise" set addition (ie, of functions into relations) and I'm very jet-lagged so I may be missing some. There's also the fact that the operations denoted by "|" and "||" are often implemented as "short-circuiting", and therefore not commutative, while "+" usually is (and that's reinforced for mathematicians who are trained to think of "+" as the operator for Abelian groups, while "*" is a (possibly) non-commutative operator. I know commutativity of "+" has been mentioned before, but the non-commutativity of "|" -- and so unsuitability for many kinds of dict combination -- hasn't been emphasized before IIRC. Since "|" (especially "|=") *is* suitable for "update", I think we should reserve "+" for some future commutative extension. In the spirit of full disclosure: Of these, 2 is already implemented and widely used, so we don't need to use dict.__add__ for that. I've never seen 4 in the mathematical literature (union of relations is not the same thing). 3, however, is very common both for mappings with small domain and sparse representation of mappings with a default value (possibly computed then cached), and "|" is not suitable for expressing that sort of addition (I'm willing to say it's "wrong" :-). Steve ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Ka-Ping Yee writes: > On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico wrote: > > But adding dictionaries is fundamentally *useful*. It is expressive. > > It is useful. It's just that + is the wrong name. First, let me say that I prefer ?!'s position here, so my bias is made apparent. I'm also aware that I have biases so I'm sympathetic to those who take a different position. Rather than say it's "wrong", let me instead point out that I think it's pragmatically troublesome to use "+". I can think of at least four interpretations of "d1 + d2" 1. update 2. multiset (~= Collections.Counter addition) 3. addition of functions into the same vector space (actually, a semigroup will do ;-), and this is the implementation of Collections.Counter 4. "fiberwise" addition (ie, assembling functions into relations) and I'm very jet-lagged so I may be missing some. Since "|" (especially "|=") *is* suitable for "update", I think we should reserve "+" for some alternative future commutative extension, of which there are several possible (all of 2, 3, 4 are commutative). Again in the spirit of full disclosure, of those above, 2 is already implemented and widely used, so we don't need to use "+" for that. I've never seen 4 except in the mathematical literature (union of relations is not the same thing). 3, however, is very common both for mappings with small domain and sparse representation of mappings with a default value (possibly computed then cached), and "|" is not suitable for expressing that sort of addition (I'm willing to say it's "wrong" :-). There's also the fact that the operations denoted by "|" and "||" are often implemented as "short-circuiting", and therefore not commutative, while "+" usually is (and that's reinforced for mathematicians who are trained to think of "+" as the operator for Abelian groups, while "*" is a (possibly) non-commutative operator. I know commutativity of "+" has been mentioned before, but the non-commutativity of "|" -- and so unsuitability for many kinds of dict combination -- hasn't been emphasized before IIRC. Steve ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Now, this belongs as a separate PEP, and I probably will write one, but I propose: d1 << d2 makes a copy of d1 and merges d2 into it, and when the keys conflict, d2 takes priority. (Works like copy/update.) d1 + d2 makes a new dictionary, taking keys from d1 and d2. If d1 and d2 have a different value for same key, a KeyError is thrown. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, Mar 6, 2019 at 4:01 PM Chris Angelico wrote: > On Thu, Mar 7, 2019 at 10:52 AM Josh Rosenberg > wrote: > > > > Allowing dicts to get involved in + means: > > Lots of words that basically say: Stuff wouldn't be perfectly pure. > > But adding dictionaries is fundamentally *useful*. It is expressive. > It is useful. It's just that + is the wrong name. Filtering and subtracting from dictionaries are also useful! Those are operations we do all the time. It would be useful if & and - did these things too—and if we have & and -, it's going to be even more obvious that the merge operator should have been |. Josh Rosenberg wrote: > If we were inventing programming languages in a vacuum, you could say + > can mean "arbitrary combination operator" and it would be fine. But we're > not in a vacuum; every major language that uses + with general purpose > containers uses it to mean element-wise addition or concatenation, not just > "merge". If we were inventing Python from scratch, we could have decided that we always use "+" to combine collections. Sets would combine with + and then it would make sense that dictionaries also combine with + . But that is not Python. Lists combine with + and sets combine with |. Why? Because lists add (put both collections together and keep everything), but sets merge (put both collections together and keep some). So, Python already has a merge operator. The merge operator is "|". For lists, += is shorthand for list.extend(). For sets, |= is shorthand for set.update(). Is dictionary merge more like extend() or more like update()? Python already took a position on that when it was decided to name the dictionary method update(). That ship sailed a long time ago. —Ping ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 7, 2019 at 10:52 AM Josh Rosenberg wrote: > The closest I can come to a thorough definition of what + does in Python (and > most languages) right now is that: > > 1. Returns a new thing of the same type (or a shared coerced type for number > weirdness) > 2. That combines the information of the input operands > 3. Is associative ((a + b) + c produces the same thing as a + (b + c)) > (modulo floating point weirdness) > 4. Is "reversible": Knowing the end result and *one* of the inputs is > sufficient to determine the value of the other input; that is, for c = a + b, > knowing any two of a, b and c allows you to determine a single unambiguous > value for the remaining value (numeric coercion and floating point weirdness > make this not 100%, but you can at least know a value equal to other value; > e.g. for c = a + b, knowing c is 5.0 and a is 1.0 is sufficient to say that b > is equal to 4, even if it's not necessarily an int or float). For numbers, > reversal is done with -; for sequences, it's done by slicing c using the > length of a or b to "subtract" the elements that came from a/b. > 5. (Actual addition only) Is commutative (modulo floating point weirdness); a > + b == b + a > 6. (Concatenation only) Is order preserving (really a natural consequence of > #4, but a property that people expect) > > Allowing dicts to get involved in + means: > > 1. Fewer consistent rules apply to +; > 2. The particular idiosyncrasies of Python dict ordering and "which value > wins" rules are now tied to +. for concatenation, there is only one set of > possible rules AFAICT so every language naturally agrees on behavior, but > dict merging obviously has many possible rules that would be unlikely to > match the exact rules of any other language except by coincidence). a winning > on order and b winning on value is a historical artifact of how Python's dict > developed; I doubt any other language would intentionally choose to split > responsibility like that if they weren't handcuffed by history. > > Again, there's nothing wrong with making dict merges easier. But it shouldn't > be done by (further) abusing +. Lots of words that basically say: Stuff wouldn't be perfectly pure. But adding dictionaries is fundamentally *useful*. It is expressive. It will, in pretty much all situations, do exactly what someone would expect, based on knowledge of how Python works in other areas. The semantics for edge cases have to be clearly defined, but they'll only come into play on rare occasions; most of the time, for instance, we don't have to worry about identity vs equality in dictionary keys. If you tell people "adding two dictionaries combines them, with the right operand winning collisions", it won't matter that this isn't how lists or floats work; it'll be incredibly useful as it is. Practicality. Let's have some. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, Mar 6, 2019 at 10:31 PM Greg Ewing wrote: > > You might as well say that using the + operator on vectors is > nonsense, because len(v1 + v2) is not in general equal to > len(v1) + len(v2). > > Yet mathematicians are quite happy to talk about "addition" > of vectors. > > Vectors addition is *actual* addition, not concatenation. You're so busy loosening the definition of + as relates to , to make it make sense for dicts that you've forgotten that + is, first and foremost, about addition in the mathematical sense, where vector addition is just one type of addition. Concatenation is already a minor abuse of +, but one commonly accepted by programmers, thanks to it having some similarities to addition and a single, unambiguous set of semantics to avoid confusion. You're defending + on dicts because vector addition isn't concatenation already, which only shows how muddled things get when you try to use + to mean multiple concepts that are at best loosely related. The closest I can come to a thorough definition of what + does in Python (and most languages) right now is that: 1. Returns a new thing of the same type (or a shared coerced type for number weirdness) 2. That combines the information of the input operands 3. Is associative ((a + b) + c produces the same thing as a + (b + c)) (modulo floating point weirdness) 4. Is "reversible": Knowing the end result and *one* of the inputs is sufficient to determine the value of the other input; that is, for c = a + b, knowing any two of a, b and c allows you to determine a single unambiguous value for the remaining value (numeric coercion and floating point weirdness make this not 100%, but you can at least know a value equal to other value; e.g. for c = a + b, knowing c is 5.0 and a is 1.0 is sufficient to say that b is equal to 4, even if it's not necessarily an int or float). For numbers, reversal is done with -; for sequences, it's done by slicing c using the length of a or b to "subtract" the elements that came from a/b. 5. (Actual addition only) Is commutative (modulo floating point weirdness); a + b == b + a 6. (Concatenation only) Is order preserving (really a natural consequence of #4, but a property that people expect) Note that these rules are consistent across most major languages that allow + to mean combine collections (the few that disagree, like Pascal, don't support | as a union operator). Concatenation is missing element #5, but otherwise aligns with actual addition. dict merges (and set unions for that matter) violate #4 and #6; for c = a + b, knowing c and either a or b still leaves a literally infinite set of possible inputs for the other input (it's not infinite for sets, where the options would be a subset of the result, but for dicts, there would be no such limitation; keys from b could exist with any possible value in a). dicts order preserving aspect *almost* satisfies #6, but not quite (if 'x' comes after 'y' in b, there is no guarantee that it will do so in c, because a gets first say on ordering, and b gets the final word on value). Allowing dicts to get involved in + means: 1. Fewer consistent rules apply to +; 2. The particular idiosyncrasies of Python dict ordering and "which value wins" rules are now tied to +. for concatenation, there is only one set of possible rules AFAICT so every language naturally agrees on behavior, but dict merging obviously has many possible rules that would be unlikely to match the exact rules of any other language except by coincidence). a winning on order and b winning on value is a historical artifact of how Python's dict developed; I doubt any other language would intentionally choose to split responsibility like that if they weren't handcuffed by history. Again, there's nothing wrong with making dict merges easier. But it shouldn't be done by (further) abusing +. -Josh Rosenberg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Ka-Ping Yee wrote: len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + operator is nonsense. You might as well say that using the + operator on vectors is nonsense, because len(v1 + v2) is not in general equal to len(v1) + len(v2). Yet mathematicians are quite happy to talk about "addition" of vectors. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
>
> If we use this "literally concat" metaphor, I still think set should have
> `+` as alias to `|` for consistency.
>
I agree.
I think "|" keeps commutativity only because it's minor than `+`.
>
I suppose that's true, fair point.
I guess I would be ok with | no longer always implying commutativity if we
were repurposing it for some radically different purpose. But dicts and
sets are similar enough that I think having them both use similar but
ultimately different definitions of "|" is going to have non-zero cost,
especially when reading or modifying future code that makes heavy use of
both data structures.
Maybe that cost is worth it. I'm personally not convinced, but I do think
it should be taken into account..
Hmm. The PEP proposed dict - dict, which is similar to set - set
> (difference).
>
Now that you point it out, I think I also dislike `d1 - d2` for the same
reasons I listed earlier: it's not consistent with set semantics. One other
objection I overlooked is that the PEP currently requires both operands to
be dicts when doing "d1 - d2" . So doing {"a": 1, "b": 2, "c": 3} - ["a",
"b"] is currently disallowed (though doing d1 -= ["a", "b"] is apparently
ok).
I can sympathize: allowing "d1 - some_iter" feels a little too magical to
me. But it's unfortunately restrictive -- I suspect removing keys stored
within a list or something would be just as common of a use-case if not
more so then removing keys stored in another dict.
I propose that we instead add methods like "d1.without_keys(...)" and
"d1.remove_keys(...)" that can accept any iterable of keys. These two
methods would replace "d1.__sub__(...)" and "d1.__isub__(...)"
respectively. The exact method names and semantics could probably do with a
little more bikeshedding, but I think this idea would remove a false
symmetry between "d1 + d2" and "d1 - d2" that doesn't actually really exist
while being more broadly useful.
Or I guess we could just remove that restriction: "it feels too magical"
isn't a great objection on my part. Either way, that part of the PEP could
use some more refinement, I think.
-- Michael
On Wed, Mar 6, 2019 at 8:29 AM Inada Naoki wrote:
> On Wed, Mar 6, 2019 at 10:59 PM Michael Lee
> wrote:
>
> >
> > I think the behavior proposed in the PEP makes sense whether you think
> of "+" as meaning "concatenation" or "merging".
> >
> > If your instinct is to assume "+" means "concatenation", then it would
> be natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be
> identical to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the
> key-value pairs into a new dict.
> >
>
> Nice explanation. You reduced my opposite to `+` by "literally concat".
> Better example, {"a": 1, "b": 2} + {"c": 4, "b": 3} == {"a": 1, "b":
> 2, "c": 4, "b": 3} == {"a": 1, "b": 3, "c": 4}
>
> On the other hand, union of set is also "literally concat". If we use
> this "literally concat" metaphor,
> I still think set should have `+` as alias to `|` for consistency.
>
> >
> > Using "|" would also violate an important existing property of unions:
> the invariant "d1 | d2 == d2 | d1" is no longer true. As far as I'm aware,
> the union operation is always taken to be commutative in math, and so I
> think it's important that we preserve that property in Python. At the very
> least, I think it's far more important to preserve commutativity of unions
> then it is to preserve some of the invariants I've seen proposed above,
> like "len(d1 + d2) == len(d1) + len(d2)".
> >
>
> I think both rule are "rather a coincidence than a conscious decision".
>
> I think "|" keeps commutativity only because it's minor than `+`. Easy
> operator
> is abused easily more than minor operator.
>
> And I think every "coincidence" rules are important. They makes
> understanding Python easy.
> Every people "discover" rules and consistency while learning language.
>
> This is a matter of balance. There are no right answer. Someone
> *feel* rule A is important than B.
> Someone feel opposite.
>
>
> > But I do know that I'm a strong -1 on adding set operations to dicts:
> it's not possible to preserve the existing semantics of union (and
> intersection) with dict and think expressions like "d1 | d2" and "d1 & d2"
> would just be confusing and misleading to encounter in the wild.
>
> Hmm. The PEP proposed dict - dict, which is similar to set - set
> (difference).
> To me, {"a": 1, "b": 2} - {"b": 3} = {"a": 1} is confusing than {"a":
> 1, "b": 2} - {"b"} = {"a": 1}.
>
> So I think borrow some semantics from set is good idea.
> Both of `dict - set` and `dict & set` makes sense to me.
>
> * `dict - set` can be used to remove private keys by "blacklist".
> * `dict & set` can be used to choose public keys by "whiltelist".
>
> --
> Inada Naoki
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
I wrote:
> I wonder, is this behaviour of {'a': 0, 'a': 1} documented (or tested)
> anywhere?
I've answered my own question here:
[Python-ideas] dict literal allows duplicate keys
https://mail.python.org/pipermail/python-ideas/2019-March/055717.html
Finally, Christopher Barker wrote:
> Yes, and had already been brought up in this thread ( I think by Guido).
> (Maybe not well documented, but certainly well understood and deliberate)
Thank you for this, Christopher.
--
Jonathan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, Mar 6, 2019 at 10:59 PM Michael Lee wrote:
>
> I think the behavior proposed in the PEP makes sense whether you think of "+"
> as meaning "concatenation" or "merging".
>
> If your instinct is to assume "+" means "concatenation", then it would be
> natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical
> to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs
> into a new dict.
>
Nice explanation. You reduced my opposite to `+` by "literally concat".
Better example, {"a": 1, "b": 2} + {"c": 4, "b": 3} == {"a": 1, "b":
2, "c": 4, "b": 3} == {"a": 1, "b": 3, "c": 4}
On the other hand, union of set is also "literally concat". If we use
this "literally concat" metaphor,
I still think set should have `+` as alias to `|` for consistency.
>
> Using "|" would also violate an important existing property of unions: the
> invariant "d1 | d2 == d2 | d1" is no longer true. As far as I'm aware, the
> union operation is always taken to be commutative in math, and so I think
> it's important that we preserve that property in Python. At the very least, I
> think it's far more important to preserve commutativity of unions then it is
> to preserve some of the invariants I've seen proposed above, like "len(d1 +
> d2) == len(d1) + len(d2)".
>
I think both rule are "rather a coincidence than a conscious decision".
I think "|" keeps commutativity only because it's minor than `+`. Easy operator
is abused easily more than minor operator.
And I think every "coincidence" rules are important. They makes
understanding Python easy.
Every people "discover" rules and consistency while learning language.
This is a matter of balance. There are no right answer. Someone
*feel* rule A is important than B.
Someone feel opposite.
> But I do know that I'm a strong -1 on adding set operations to dicts: it's
> not possible to preserve the existing semantics of union (and intersection)
> with dict and think expressions like "d1 | d2" and "d1 & d2" would just be
> confusing and misleading to encounter in the wild.
Hmm. The PEP proposed dict - dict, which is similar to set - set (difference).
To me, {"a": 1, "b": 2} - {"b": 3} = {"a": 1} is confusing than {"a":
1, "b": 2} - {"b"} = {"a": 1}.
So I think borrow some semantics from set is good idea.
Both of `dict - set` and `dict & set` makes sense to me.
* `dict - set` can be used to remove private keys by "blacklist".
* `dict & set` can be used to choose public keys by "whiltelist".
--
Inada Naoki
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Michael Lee wrote:
> If your instinct is to assume "+" means "concatenation", then it would be
> natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical
> to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs
> into a new dict.
> But of course, you can't have duplicate keys in Python. So, you would either
> recall or look up how duplicate keys are handled when constructing a dict and
> learn that the rule is that the right-most key wins. So the natural
> conclusion is that "+" would follow this existing rule -- and you end up with
> exactly the behavior described in the PEP.
This is a nice argument. And well presented. And it gave me surprise,
that taught me something. Here goes:
>>> {'a': 0}
{'a': 0}
>>> {'a': 0, 'a': 0}
{'a': 0}
>>> {'a': 0, 'a': 1}
{'a': 1}
>>> {'a': 1, 'a': 0}
{'a': 0}
This surprised me quite a bit. I was expecting to get an exception. However
>>> dict(a=0)
{'a': 0}
>>> dict(a=0, a=0)
SyntaxError: keyword argument repeated
does give an exception.
I wonder, is this behaviour of {'a': 0, 'a': 1} documented (or tested)
anywhere? I didn't find it in these URLs:
https://docs.python.org/3/library/stdtypes.html#mapping-types-dict
https://docs.python.org/3/tutorial/datastructures.html#dictionaries
I think this behaviour might give rise to gotchas. For example, if we
define inverse_f by
>>> inverse_f = { f(a): a, f(b): b }
then is the next statement always true (assuming a <> b)?
>>> inverse_f[ f(a) ] == a
Well, it's not true with these values
>>> a, b = 1, 2
>>> def f(n): pass # There's a bug here, f(n) should be a bijection.
A quick check that len(inverse) == 2 would provide a sanity check. Or
perhaps better, len(inverse_f) == len(set(a, b)). (I don't have an
example of this bug appearing 'in the wild'.)
Once again, I thank Michael for his nice, instructive and
well-presented example.
--
Jonathan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Thu, Mar 7, 2019 at 12:59 AM Michael Lee wrote:
> If your instinct is to assume "+" means "concatenation", then it would be
> natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be identical
> to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the key-value pairs
> into a new dict.
>
> But of course, you can't have duplicate keys in Python. So, you would either
> recall or look up how duplicate keys are handled when constructing a dict and
> learn that the rule is that the right-most key wins. So the natural
> conclusion is that "+" would follow this existing rule -- and you end up with
> exactly the behavior described in the PEP.
>
Which, by the way, is also consistent with assignment:
d = {}; d["a"] = 1; d["b"] = 2; d["c"] = 3; d["b"] = 4
Rightmost one wins. It's the most logical behaviour.
ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
>
> I strongly agree with Ka-Ping. '+' is intuitively concatenation not
> merging. The behavior is overwhelmingly more similar to the '|' operator in
> sets (whether or not a user happens to know the historical implementation
> overlap).
I think the behavior proposed in the PEP makes sense whether you think of
"+" as meaning "concatenation" or "merging".
If your instinct is to assume "+" means "concatenation", then it would be
natural to assume that {"a": 1, "b": 2} + {"c": 3, "b": 4} would be
identical to {"a": 1, "b": 2, "c": 3, "b": 4} -- literally concat the
key-value pairs into a new dict.
But of course, you can't have duplicate keys in Python. So, you would
either recall or look up how duplicate keys are handled when constructing a
dict and learn that the rule is that the right-most key wins. So the
natural conclusion is that "+" would follow this existing rule -- and you
end up with exactly the behavior described in the PEP.
This also makes explaining the behavior of "d1 + d2" slightly easier than
explaining "d1 | d2". For the former, you can just say "d1 + d2 means we
concat the two dicts together" and stop there. You almost don't need to
explain the merging/right-most key wins behavior at all, since that
behavior is the only one consistent with the existing language rules.
In contrast, you *would* need to explain this with "d1 | d2": I would
mentally translate this expression to mean "take the union of these two
dicts" and there's no real way to deduce which key-value pair ends up in
the final dict given that framing. Why is it that key-value pairs in d2 win
over pairs in d1 here? That choice seems pretty arbitrary when you think of
this operation in terms of unions, rather than either concat or merge.
Using "|" would also violate an important existing property of unions: the
invariant "d1 | d2 == d2 | d1" is no longer true. As far as I'm aware, the
union operation is always taken to be commutative in math, and so I think
it's important that we preserve that property in Python. At the very least,
I think it's far more important to preserve commutativity of unions then it
is to preserve some of the invariants I've seen proposed above, like
"len(d1 + d2) == len(d1) + len(d2)".
Personally, I don't really have a strong opinion on this PEP, or the other
one I've seen proposed where we add a "d1.merge(d2, d3, ...)". But I do
know that I'm a strong -1 on adding set operations to dicts: it's not
possible to preserve the existing semantics of union (and intersection)
with dict and think expressions like "d1 | d2" and "d1 & d2" would just be
confusing and misleading to encounter in the wild.
-- Michael
On Wed, Mar 6, 2019 at 4:53 AM David Mertz wrote:
> I strongly agree with Ka-Ping. '+' is intuitively concatenation not
> merging. The behavior is overwhelmingly more similar to the '|' operator in
> sets (whether or not a user happens to know the historical implementation
> overlap).
>
> I think growing the full collection of set operations world be a pleasant
> addition to dicts. I think shoe-horning in plus would always be jarring to
> me.
>
> On Wed, Mar 6, 2019, 5:30 AM Ka-Ping Yee wrote:
>
>> len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the +
>> operator is nonsense.
>>
>> len(dict1 + dict2) cannot even be computed by any expression
>> involving +. Using len() to test the semantics of the operation is not
>> arbitrary; the fact that the sizes do not add is a defining quality of a
>> merge. This is a merge, not an addition. The proper analogy is to sets,
>> not lists.
>>
>> The operators should be |, &, and -, exactly as for sets, and the
>> behaviour defined with just three rules:
>>
>> 1. The keys of dict1 [op] dict2 are the elements of dict1.keys() [op]
>> dict2.keys().
>>
>> 2. The values of dict2 take priority over the values of dict1.
>>
>> 3. When either operand is a set, it is treated as a dict whose values are
>> None.
>>
>> This yields many useful operations and, most importantly, is simple to
>> explain. "sets and dicts can |, &, -" takes up less space in your brain
>> than "sets can |, &, - but dicts can only + and -, where dict + is like set
>> |".
>>
>> merge and update some items:
>>
>> {'a': 1, 'b': 2} | {'b': 3, 'c': 4} => {'a': 1, 'b': 3, 'c': 4}
>>
>> pick some items:
>>
>> {'a': 1, 'b': 2} & {'b': 3, 'c': 4} => {'b': 3}
>>
>> remove some items:
>>
>> {'a': 1, 'b': 2} - {'b': 3, 'c': 4} => {'a': 1}
>>
>> reset values of some keys:
>>
>> {'a': 1, 'b': 2} | {'b', 'c'} => {'a': 1, 'b': None, 'c': None}
>>
>> ensure certain keys are present:
>>
>> {'b', 'c'} | {'a': 1, 'b': 2} => {'a': 1, 'b': 2, 'c': None}
>>
>> pick some items:
>>
>> {'b', 'c'} | {'a': 1, 'b': 2} => {'b': 2}
>>
>> remove some items:
>>
>> {'a': 1, 'b': 2} - {'b', 'c'} => {'a': 1}
>>
>> On Wed, Mar 6, 2019 at 1:51 AM Rémi Lapeyre
>> wrote:
>>
>>> Le 6 mars 2019 à 10:26:15, Brice Parent
>>> (cont...@brice.xyz(mailto:cont...@brice.xyz)) a écrit:
>>>
Re: [Python-ideas] PEP: Dict addition and subtraction
Le 06/03/2019 à 13:53, Chris Angelico a écrit : On Wed, Mar 6, 2019 at 11:18 PM Brice Parent wrote: The major implication to such a modification of the Dict.update method, is that when you're using it with keyword arguments (by opposition to passing another dict/iterable as positional), you're making a small non-backward compatible change in that if in some code, someone was already using the keyword that would be chosing (here "on_collision"), their code would be broken by the new feature. Anyway, if the keyword is slected wisely, the collision case will almost never happen, and be quite easy to correct if it ever happened. You can make it unlikely, yes, but I'd dispute "easy to correct". Let's suppose that someone had indeed used the chosen keyword (and remember, the more descriptive the argument name, the more likely that it'll be useful elsewhere and therefore have a collision). How would they discover this? If they're really lucky, there MIGHT be an exception (if on_collision accepts only a handful of keywords, and the collision isn't one of them), but if your new feature is sufficiently flexible, that might not happen. There'll just be incorrect behaviour. As APIs go, using specific keyword args at the same time as **kw is a bit odd. Consider: button_options.update(button_info, on_click=frobnicate, style="KDE", on_collision="replace") It's definitely not obvious which of those will end up in the dictionary and which won't. Big -1 from me on that change. That's indeed a good point. Even if the correction is quite easy to make in most cases. With keyword only changes: button_options.update(dict(on_click=frobnicate, style="KDE", on_collision="replace")) # or button_options.update(dict(on_collision="replace"), on_click=frobnicate, style="KDE") In the exact case you proposed, it could become a 2-liners: button_options.update(button_info) button_options.update(dict(on_click=frobnicate, style="KDE", on_collision="replace")) In my code, I would probably make it into 2 lines, to make clear that we have 2 levels of data merging, one that is general (the first), and one that is specific to this use-case (as it's hard written in the code), but not everyone doesn't care about the number of lines. But for the other part of your message, I 100% agree with you. The main problem with such a change is not (to me) that it can break some edge cases, but that it would potentially break them silently. And that, I agree, is worth a big -1 I guess. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, Mar 6, 2019 at 11:18 PM Brice Parent wrote: > The major implication to such a > modification of the Dict.update method, is that when you're using it > with keyword arguments (by opposition to passing another dict/iterable > as positional), you're making a small non-backward compatible change in > that if in some code, someone was already using the keyword that would > be chosing (here "on_collision"), their code would be broken by the new > feature. > Anyway, if > the keyword is slected wisely, the collision case will almost never > happen, and be quite easy to correct if it ever happened. You can make it unlikely, yes, but I'd dispute "easy to correct". Let's suppose that someone had indeed used the chosen keyword (and remember, the more descriptive the argument name, the more likely that it'll be useful elsewhere and therefore have a collision). How would they discover this? If they're really lucky, there MIGHT be an exception (if on_collision accepts only a handful of keywords, and the collision isn't one of them), but if your new feature is sufficiently flexible, that might not happen. There'll just be incorrect behaviour. As APIs go, using specific keyword args at the same time as **kw is a bit odd. Consider: button_options.update(button_info, on_click=frobnicate, style="KDE", on_collision="replace") It's definitely not obvious which of those will end up in the dictionary and which won't. Big -1 from me on that change. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Rhodri James wrote: > Making assumptions about length where any dictionary > manipulations are concerned seems unwise to me I think you're a bit hasty here. Some assumptions are sensible. Suppose a = len(d1) b = len(d2) c = len(d1 + d2) # Using the suggested syntax. Then we know max(a, b) <= c <= a + b And this is, in broad terms, characteristic of merge operations. -- Jonathan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Ka-Ping Yee wrote: > > len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + > operator is nonsense. > > len(dict1 + dict2) cannot even be computed by any expression involving +. > Using len() to test the semantics of the operation is not arbitrary; the fact > that the sizes do not add is a defining quality of a merge. This is a merge, > not an addition. The proper analogy is to sets, not lists. For me, this comment is excellent. It neatly expresses the central concern about this proposal. I think most us will agree that the proposal is to use '+' to express a merge operation, namely update. (There are other merge operations, when there are two values to combine, such as taking the min or max of the two values.) Certainly, many of the posts quite naturally use the word merge. Indeed PEP 584 writes "This PEP suggests adding merge '+' and difference '-' operators to the built-in dict class." We would all agree that it would be obviously wrong to suggest adding merge '-' and difference '+' operators. (Note: I've swapped '+' and '-'.) And why? Because it is obviously wrong to use '-' to denote merge, etc. Some of us are also upset by the use of '+' to denote merge. By the way, there is already a widespread symbol for merge. It appears on many road signs. It looks like an upside down 'Y'. It even has merge left and merge right versions. Python already has operator symbols '+', '-', '*', '/' and so on. See https://docs.python.org/3/reference/lexical_analysis.html#operators Perhaps we should add a merge or update symbol to this list, so that we don't overload to breaking point the humble '+' operator. Although that would make Python a bit more like APL. By the way, Pandas already has a merge operation, called merge, that takes many parameters. I've only glanced at it. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html -- Jonathan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, Mar 6, 2019 at 11:52 AM Rhodri James wrote: > On 06/03/2019 10:29, Ka-Ping Yee wrote: > > len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + > > operator is nonsense. > > I'm sorry, but you're going to have to justify why this identity is > important. Making assumptions about length where any dictionary > manipulations are concerned seems unwise to me, which makes a nonsense > of your claim that this is nonsense :-) > It's not "nonsense" per se. If we were inventing programming languages in a vacuum, you could say + can mean "arbitrary combination operator" and it would be fine. But we're not in a vacuum; every major language that uses + with general purpose containers uses it to mean element-wise addition or concatenation, not just "merge". Concatenation is what imposes that identity (and all the others people are defending, like no loss of input values); you're taking a sequence of things, and shoving another sequence of things on the end of it, preserving order and all values. The argument here isn't that you *can't* make + do arbitrary merges that don't adhere to these semantics. It's that adding yet a third meaning to + (and it is a third meaning; it has no precedent in any existing type in Python, nor in any other major language; even in the minor languages that allow it, they use + for sets as well, so Python using + is making Python itself internally inconsistent with the operators used for set), for limited benefit. - Josh Rosenberg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Le 06/03/2019 à 10:50, Rémi Lapeyre a écrit :
Le 05/03/2019 à 23:40, Greg Ewing a écrit :
Steven D'Aprano wrote:
The question is, is [recursive merge] behaviour useful enough and
common enough to be built into dict itself?
I think not. It seems like just one possible way of merging
values out of many. I think it would be better to provide
a merge function or method that lets you specify a function
for merging values.
That's what this conversation led me to. I'm not against the addition
for the most general usage (and current PEP's describes the behaviour I
would expect before reading the doc), but for all other more specific
usages, where we intend any special or not-so-common behaviour, I'd go
with modifying Dict.update like this:
foo.update(bar, on_collision=updator) # Although I'm not a fan of the
keyword I used
This won’t be possible update() already takes keyword arguments:
foo = {}
bar = {'a': 1}
foo.update(bar, on_collision=lambda e: e)
foo
{'a': 1, 'on_collision': at 0x10b8df598>}
I don't see that as a problem at all.
Having a function's signature containing a **kwargs doesn't disable to
have explicit keyword arguments at the same time:
`def foo(bar="baz", **kwargs):` is perfectly valid, as well as `def
spam(ham: Dict, eggs="blah", **kwargs):`, so `update(other,
on_collision=None, **added) is too, no? The major implication to such a
modification of the Dict.update method, is that when you're using it
with keyword arguments (by opposition to passing another dict/iterable
as positional), you're making a small non-backward compatible change in
that if in some code, someone was already using the keyword that would
be chosing (here "on_collision"), their code would be broken by the new
feature.
I had never tried to pass a dict and kw arguments together, as it seemed
to me that it wasn't supported (I would even have expected an exception
to be raised), but it's probably my level of English that isn't high
enough to get it right, or this part of the doc that doesn't describe
well the full possible usage of the method (see here:
https://docs.python.org/3/library/stdtypes.html#dict.update). Anyway, if
the keyword is slected wisely, the collision case will almost never
happen, and be quite easy to correct if it ever happened.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On 06/03/2019 10:29, Ka-Ping Yee wrote: len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the + operator is nonsense. I'm sorry, but you're going to have to justify why this identity is important. Making assumptions about length where any dictionary manipulations are concerned seems unwise to me, which makes a nonsense of your claim that this is nonsense :-) -- Rhodri James *-* Kynesim Ltd ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
len(dict1 + dict2) does not equal len(dict1) + len(dict2), so using the +
operator is nonsense.
len(dict1 + dict2) cannot even be computed by any expression involving +.
Using len() to test the semantics of the operation is not arbitrary; the
fact that the sizes do not add is a defining quality of a merge. This is a
merge, not an addition. The proper analogy is to sets, not lists.
The operators should be |, &, and -, exactly as for sets, and the behaviour
defined with just three rules:
1. The keys of dict1 [op] dict2 are the elements of dict1.keys() [op]
dict2.keys().
2. The values of dict2 take priority over the values of dict1.
3. When either operand is a set, it is treated as a dict whose values are
None.
This yields many useful operations and, most importantly, is simple to
explain. "sets and dicts can |, &, -" takes up less space in your brain
than "sets can |, &, - but dicts can only + and -, where dict + is like set
|".
merge and update some items:
{'a': 1, 'b': 2} | {'b': 3, 'c': 4} => {'a': 1, 'b': 3, 'c': 4}
pick some items:
{'a': 1, 'b': 2} & {'b': 3, 'c': 4} => {'b': 3}
remove some items:
{'a': 1, 'b': 2} - {'b': 3, 'c': 4} => {'a': 1}
reset values of some keys:
{'a': 1, 'b': 2} | {'b', 'c'} => {'a': 1, 'b': None, 'c': None}
ensure certain keys are present:
{'b', 'c'} | {'a': 1, 'b': 2} => {'a': 1, 'b': 2, 'c': None}
pick some items:
{'b', 'c'} | {'a': 1, 'b': 2} => {'b': 2}
remove some items:
{'a': 1, 'b': 2} - {'b', 'c'} => {'a': 1}
On Wed, Mar 6, 2019 at 1:51 AM Rémi Lapeyre wrote:
> Le 6 mars 2019 à 10:26:15, Brice Parent
> (cont...@brice.xyz(mailto:cont...@brice.xyz)) a écrit:
>
> >
> > Le 05/03/2019 à 23:40, Greg Ewing a écrit :
> > > Steven D'Aprano wrote:
> > >> The question is, is [recursive merge] behaviour useful enough and
> > > > common enough to be built into dict itself?
> > >
> > > I think not. It seems like just one possible way of merging
> > > values out of many. I think it would be better to provide
> > > a merge function or method that lets you specify a function
> > > for merging values.
> > >
> > That's what this conversation led me to. I'm not against the addition
> > for the most general usage (and current PEP's describes the behaviour I
> > would expect before reading the doc), but for all other more specific
> > usages, where we intend any special or not-so-common behaviour, I'd go
> > with modifying Dict.update like this:
> >
> > foo.update(bar, on_collision=updator) # Although I'm not a fan of the
> > keyword I used
>
> Le 6 mars 2019 à 10:26:15, Brice Parent
> (cont...@brice.xyz(mailto:cont...@brice.xyz)) a écrit:
>
> >
> > Le 05/03/2019 à 23:40, Greg Ewing a écrit :
> > > Steven D'Aprano wrote:
> > >> The question is, is [recursive merge] behaviour useful enough and
> > > > common enough to be built into dict itself?
> > >
> > > I think not. It seems like just one possible way of merging
> > > values out of many. I think it would be better to provide
> > > a merge function or method that lets you specify a function
> > > for merging values.
> > >
> > That's what this conversation led me to. I'm not against the addition
> > for the most general usage (and current PEP's describes the behaviour I
> > would expect before reading the doc), but for all other more specific
> > usages, where we intend any special or not-so-common behaviour, I'd go
> > with modifying Dict.update like this:
> >
> > foo.update(bar, on_collision=updator) # Although I'm not a fan of the
> > keyword I used
>
> This won’t be possible update() already takes keyword arguments:
>
> >>> foo = {}
> >>> bar = {'a': 1}
> >>> foo.update(bar, on_collision=lambda e: e)
> >>> foo
> {'a': 1, 'on_collision': at 0x10b8df598>}
>
> > `updator` being a simple function like this one:
> >
> > def updator(updated, updator, key) -> Any:
> > if key == "related":
> > return updated[key].update(updator[key])
> >
> > if key == "tags":
> > return updated[key] + updator[key]
> >
> > if key in ["a", "b", "c"]: # Those
> > return updated[key]
> >
> > return updator[key]
> >
> > There's nothing here that couldn't be made today by using a custom
> > update function, but leaving the burden of checking for values that are
> > in both and actually inserting the new values to Python's language, and
> > keeping on our side only the parts that are specific to our use case,
> > makes in my opinion the code more readable, with fewer possible bugs and
> > possibly better optimization.
> >
> >
> > ___
> > Python-ideas mailing list
> > Python-ideas@python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Re: [Python-ideas] PEP: Dict addition and subtraction
Le 6 mars 2019 à 10:26:15, Brice Parent
(cont...@brice.xyz(mailto:cont...@brice.xyz)) a écrit:
>
> Le 05/03/2019 à 23:40, Greg Ewing a écrit :
> > Steven D'Aprano wrote:
> >> The question is, is [recursive merge] behaviour useful enough and
> > > common enough to be built into dict itself?
> >
> > I think not. It seems like just one possible way of merging
> > values out of many. I think it would be better to provide
> > a merge function or method that lets you specify a function
> > for merging values.
> >
> That's what this conversation led me to. I'm not against the addition
> for the most general usage (and current PEP's describes the behaviour I
> would expect before reading the doc), but for all other more specific
> usages, where we intend any special or not-so-common behaviour, I'd go
> with modifying Dict.update like this:
>
> foo.update(bar, on_collision=updator) # Although I'm not a fan of the
> keyword I used
Le 6 mars 2019 à 10:26:15, Brice Parent
(cont...@brice.xyz(mailto:cont...@brice.xyz)) a écrit:
>
> Le 05/03/2019 à 23:40, Greg Ewing a écrit :
> > Steven D'Aprano wrote:
> >> The question is, is [recursive merge] behaviour useful enough and
> > > common enough to be built into dict itself?
> >
> > I think not. It seems like just one possible way of merging
> > values out of many. I think it would be better to provide
> > a merge function or method that lets you specify a function
> > for merging values.
> >
> That's what this conversation led me to. I'm not against the addition
> for the most general usage (and current PEP's describes the behaviour I
> would expect before reading the doc), but for all other more specific
> usages, where we intend any special or not-so-common behaviour, I'd go
> with modifying Dict.update like this:
>
> foo.update(bar, on_collision=updator) # Although I'm not a fan of the
> keyword I used
This won’t be possible update() already takes keyword arguments:
>>> foo = {}
>>> bar = {'a': 1}
>>> foo.update(bar, on_collision=lambda e: e)
>>> foo
{'a': 1, 'on_collision': at 0x10b8df598>}
> `updator` being a simple function like this one:
>
> def updator(updated, updator, key) -> Any:
> if key == "related":
> return updated[key].update(updator[key])
>
> if key == "tags":
> return updated[key] + updator[key]
>
> if key in ["a", "b", "c"]: # Those
> return updated[key]
>
> return updator[key]
>
> There's nothing here that couldn't be made today by using a custom
> update function, but leaving the burden of checking for values that are
> in both and actually inserting the new values to Python's language, and
> keeping on our side only the parts that are specific to our use case,
> makes in my opinion the code more readable, with fewer possible bugs and
> possibly better optimization.
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
Le 05/03/2019 à 23:40, Greg Ewing a écrit : Steven D'Aprano wrote: The question is, is [recursive merge] behaviour useful enough and > common enough to be built into dict itself? I think not. It seems like just one possible way of merging values out of many. I think it would be better to provide a merge function or method that lets you specify a function for merging values. That's what this conversation led me to. I'm not against the addition for the most general usage (and current PEP's describes the behaviour I would expect before reading the doc), but for all other more specific usages, where we intend any special or not-so-common behaviour, I'd go with modifying Dict.update like this: foo.update(bar, on_collision=updator) # Although I'm not a fan of the keyword I used `updator` being a simple function like this one: def updator(updated, updator, key) -> Any: if key == "related": return updated[key].update(updator[key]) if key == "tags": return updated[key] + updator[key] if key in ["a", "b", "c"]: # Those return updated[key] return updator[key] There's nothing here that couldn't be made today by using a custom update function, but leaving the burden of checking for values that are in both and actually inserting the new values to Python's language, and keeping on our side only the parts that are specific to our use case, makes in my opinion the code more readable, with fewer possible bugs and possibly better optimization. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
INADA Naoki schrieb am 05.03.19 um 08:03:> On Tue, Mar 5, 2019 at 12:02 AM
Stefan Behnel wrote:
>> INADA Naoki schrieb am 04.03.19 um 11:15:
>>> Why statement is not enough?
>>
>> I'm not sure I understand why you're asking this, but a statement is
>> "not enough" because it's a statement and not an expression. It does
>> not replace the convenience of an expression.
>
> It seems tautology and say nothing.
That's close to what I thought when I read your question. :)
> What is "convenience of an expression"?
It's the convenience of being able to write an expression that generates
the thing you need, rather than having to split code into statements that
create it step by step before you can use it.
Think of comprehensions versus for-loops. Comprehensions are expressions
that don't add anything to the language that a for-loop cannot achieve.
Still, everyone uses them because they are extremely convenient.
> Is it needed to make Python more readable language?
No, just like comprehensions, it's not "needed". It's just convenient.
> Anyway, If "there is expression" is the main reason for this proposal,
> symbolic operator is not necessary.
As said, "needed" is not the right word. Being able to use a decorator
closes a gap in the language. Just like list comprehensions fit generator
expressions and vice versa. There is no "need" for being able to write
[x**2 for x in seq]
{x**2 for x in seq}
when you can equally well write
list(x**2 for x in seq)
set(x**2 for x in seq)
But I certainly wouldn't complain about that redundancy in the language.
> `new = d1.updated(d2)` or `new = dict.merge(d1, d2)` are enough. Python
> preferred name over symbol in general. Symbols are readable and
> understandable only when it has good math metaphor.
>
> Sets has symbol operator because it is well known in set in math, not
> because set is frequently used.
>
> In case of dict, there is no simple metaphor in math.
So then, if "list+list" and "tuple+tuple" wasn't available through an
operator, would you also reject the idea of adding it, argueing that we
could use this:
L = L1.extended(L2)
I honestly do not see the math relation in concatenation via "+".
But, given that "+" and "|" already have the meaning of "merging two
containers into one" in Python, I think it makes sense to allow that also
for dicts.
> It just cryptic and hard to Google.
I honestly doubt that it's something people would have to search for any
more than they have to search for the list "+" operation. My guess is that
it's pretty much what most people would try first when they have the need
to merge two dicts, and only failing that, they would start a web search.
In comparison, very few users would be able to come up with "{**d1, **d2}"
on their own, or even "d1.updated(d2)".
My point is, given the current language, "dict+dict" is a gap that is worth
closing.
Stefan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] PEP: Dict addition and subtraction
On Wed, Mar 6, 2019 at 12:08 AM Guido van Rossum wrote:
> On Tue, Mar 5, 2019 at 3:50 PM Josh Rosenberg <
> shadowranger+pythonid...@gmail.com> wrote:
>
>>
>> On Tue, Mar 5, 2019 at 11:16 PM Steven D'Aprano
>> wrote:
>>
>>> On Sun, Mar 03, 2019 at 09:28:30PM -0500, James Lu wrote:
>>>
>>> > I propose that the + sign merge two python dictionaries such that if
>>> > there are conflicting keys, a KeyError is thrown.
>>>
>>> This proposal is for a simple, operator-based equivalent to
>>> dict.update() which returns a new dict. dict.update has existed since
>>> Python 1.5 (something like a quarter of a century!) and never grown a
>>> "unique keys" version.
>>>
>>> I don't recall even seeing a request for such a feature. If such a
>>> unique keys version is useful, I don't expect it will be useful often.
>>>
>>
>> I have one argument in favor of such a feature: It preserves
>> concatenation semantics. + means one of two things in all code I've ever
>> seen (Python or otherwise):
>>
>> 1. Numeric addition (including element-wise numeric addition as in
>> Counter and numpy arrays)
>> 2. Concatenation (where the result preserves all elements, in order,
>> including, among other guarantees, that len(seq1) + len(seq2) == len(seq1 +
>> seq2))
>>
>> dict addition that didn't reject non-unique keys wouldn't fit *either*
>> pattern; the main proposal (making it equivalent to left.copy(), followed
>> by .update(right)) would have the left hand side would win on ordering, the
>> right hand side on values, and wouldn't preserve the length invariant of
>> concatenation. At least when repeated keys are rejected, most concatenation
>> invariants are preserved; order is all of the left elements followed by all
>> of the right, and no elements are lost.
>>
>
> I must by now have seen dozens of post complaining about this aspect of
> the proposal. I think this is just making up rules (e.g. "+ never loses
> information") to deal with an aspect of the design where a *choice* must be
> made. This may reflect the Zen of Python's "In the face of ambiguity,
> refuse the temptation to guess." But really, that's a pretty silly rule
> (truly, they aren't all winners). Good interface design constantly makes
> choices in ambiguous situations, because the alternative is constantly
> asking, and that's just annoying.
>
> We have a plethora of examples (in fact, almost all alternatives
> considered) of situations related to dict merging where a choice is made
> between conflicting values for a key, and it's always the value further to
> the right that wins: from d[k] = v (which overrides the value when k is
> already in the dict) to d1.update(d2) (which lets the values in d2 win),
> including the much lauded {**d1, **d2} and even plain {'a': 1, 'a': 2} has
> a well-defined meaning where the latter value wins.
>
> Yeah. And I'm fine with the behavior for update because the name itself is
descriptive; we're spelling out, in English, that we're update-ing the
thing it's called on, so it makes sense to have the thing we're sourcing
for updates take precedence.
Similarly, for dict literals (and by extension, unpacking), it's following
an existing Python convention which doesn't contradict anything else.
Overloading + lacks the clear descriptive aspect of update that describes
the goal of the operation, and contradicts conventions (in Python and
elsewhere) about how + works (addition or concatenation, and a lot of
people don't even like it doing the latter, though I'm not that pedantic).
A couple "rules" from C++ on overloading are "*Whenever the meaning of an
operator is not obviously clear and undisputed, it should not be
overloaded.* *Instead, provide a function with a well-chosen name.*"
and "*Always
stick to the operator’s well-known semantics".* (Source:
https://stackoverflow.com/a/4421708/364696 , though the principle is
restated in many other places). Obviously the C++ community isn't perfect
on this (see iostream and <> operators), but they're otherwise pretty
consistent. + means addition, and in many languages including C++ strings,
concatenation, but I don't know of any languages outside the "esoteric"
category that use it for things that are neither addition nor
concatenation. You've said you don't want the whole plethora of set-like
behaviors on dicts, but dicts are syntactically and semantically much more
like sets than sequences, and if you add + (with semantics differing from
both sets and sequences), the language becomes less consistent.
I'm not against making it easier to merge dictionaries. But people seem to
be arguing that {**d1, **d2} is bad because of magic punctuation that
obscures meaning, when IMO:
d3 = d1 + d2
is obscuring meaning by adding yet a third rule for what + means,
inconsistent with both existing rules (from both Python and the majority of
languages I've had cause to use). A named method (class or instance) or
top-level function (a la sorted) is more explicit, easier to look up (after
all, the
