Re: [Python-Dev] sum(...) limitation

2014-08-13 Thread Stephen J. Turnbull
Redirecting to python-ideas, so trimming less than I might. Chris Barker writes: On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull step...@xemacs.org wrote: I'm referring to removing the unnecessary information that there's a better way to do it, and simply raising an error (as

Re: [Python-Dev] sum(...) limitation

2014-08-13 Thread Ronald Oussoren
On 12 Aug 2014, at 10:02, Armin Rigo ar...@tunes.org wrote: Hi all, The core of the matter is that if we repeatedly __add__ strings from a long list, we get O(n**2) behavior. For one point of view, the reason is that the additions proceed in left-to-right order. Indeed, sum() could

Re: [Python-Dev] sum(...) limitation

2014-08-13 Thread Chris Barker
On Tue, Aug 12, 2014 at 11:21 PM, Stephen J. Turnbull step...@xemacs.org wrote: Redirecting to python-ideas, so trimming less than I might. reasonable enough -- you are introducing some more significant ideas for changes. I've said all I have to say about this -- I don't seem to see anything

Re: [Python-Dev] sum(...) limitation

2014-08-12 Thread Stephen J. Turnbull
Ethan Furman writes: On 08/11/2014 08:50 PM, Stephen J. Turnbull wrote: Chris Barker - NOAA Federal writes: It seems pretty pedantic to say: we could make this work well, but we'd rather chide you for not knowing the proper way to do it. Nobody disagrees. But backward

Re: [Python-Dev] sum(...) limitation

2014-08-12 Thread Nick Coghlan
On 12 Aug 2014 11:21, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: Sorry for the bike shedding here, but: The quadratic behaviour of repeated str summation is a subtle, silent error. OK, fair enough. I suppose it would be hard and ugly to catch those instances and raise an

Re: [Python-Dev] sum(...) limitation

2014-08-12 Thread Armin Rigo
Hi all, The core of the matter is that if we repeatedly __add__ strings from a long list, we get O(n**2) behavior. For one point of view, the reason is that the additions proceed in left-to-right order. Indeed, sum() could proceed in a more balanced tree-like order: from [x0, x1, x2, x3, ...],

Re: [Python-Dev] sum(...) limitation

2014-08-12 Thread Chris Barker
On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull step...@xemacs.org wrote: I'm referring to removing the unnecessary information that there's a better way to do it, and simply raising an error (as in Python 3.2, say) which is all a RealProgrammer[tm] should ever need! I can't imagine

Re: [Python-Dev] sum(...) limitation

2014-08-12 Thread Stefan Richthofer
chris.bar...@noaa.gov An:Kein Empfnger Cc:Python Dev python-dev@python.org Betreff:Re: [Python-Dev] sum(...) limitation On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull step...@xemacs.org wrote: Im referring to removing the unnecessary information that theres a better way to do it, and simply

Re: [Python-Dev] sum(...) limitation

2014-08-12 Thread Nikolaus Rath
Chris Barker chris.bar...@noaa.gov writes: What I fail to see is why it's better to raise an exception and point users to a better way, than to simply provide an optimization so that it's a mute issue. The only justification offered here is that will teach people that summing strings (and

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Ben Hoyt
It seems to me this is something of a pointless discussion -- I highly doubt the current situation is going to change, and it works very well. Even if not perfect, sum() is for numbers, sep.join() for strings. However, I will add one comment: I'm overall -1 on trying to change the current

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Chris Barker - NOAA Federal
I'm very sympathetic to Steven's explanation that we wouldn't be having this discussion if we used a different operator for string concatenation. Sure -- but just imagine the conversations we could be having instead : what does bit wise and of a string mean? A bytes object? I cod see it as a

Re: [Python-Dev] sum(...) limitation - temporary elision take 2

2014-08-11 Thread Julian Taylor
On 04.08.2014 22:22, Jim J. Jewett wrote: Sat Aug 2 12:11:54 CEST 2014, Julian Taylor wrote (in https://mail.python.org/pipermail/python-dev/2014-August/135623.html ) wrote: Andrea Griffini agriff at tin.it wrote: However sum([[1,2,3],[4],[],[5,6]], []) concatenates the lists.

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Terry Reedy
On 8/11/2014 8:26 AM, Ben Hoyt wrote: It seems to me this is something of a pointless discussion -- I highly doubt the current situation is going to change, and it works very well. Even if not perfect, sum() is for numbers, sep.join() for strings. However, I will add one comment: I'm

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Nick Coghlan
On 12 Aug 2014 03:03, Chris Barker - NOAA Federal chris.bar...@noaa.gov wrote: My confusion is still this: Repeated summation of strings has been optimized in cpython even though it's not the recommended way to solve that problem. The quadratic behaviour of repeated str summation is a

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Alexander Belopolsky
On Mon, Aug 11, 2014 at 8:19 PM, Nick Coghlan ncogh...@gmail.com wrote: Teaching users the difference between linear time operations and quadratic ones isn't about purity, it's about passing along a fundamental principle of algorithm scalability. I would understand if this was done in

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Chris Barker - NOAA Federal
Sorry for the bike shedding here, but: The quadratic behaviour of repeated str summation is a subtle, silent error. OK, fair enough. I suppose it would be hard and ugly to catch those instances and raise an exception pointing users to .join. *is* controversial that CPython silently optimises

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Stephen J. Turnbull
Chris Barker - NOAA Federal writes: Is there anything in the language spec that says string concatenation is O(n^2)? Or for that matter any of the performs characteristics of build in types? Those striker as implementation details that SHOULD be particular to the implementation.

Re: [Python-Dev] sum(...) limitation

2014-08-11 Thread Ethan Furman
On 08/11/2014 08:50 PM, Stephen J. Turnbull wrote: Chris Barker - NOAA Federal writes: It seems pretty pedantic to say: we could make this work well, but we'd rather chide you for not knowing the proper way to do it. Nobody disagrees. But backward compatibility gets in the way. Something

Re: [Python-Dev] sum(...) limitation

2014-08-10 Thread Stephen J. Turnbull
Alexander Belopolsky writes: On Sat, Aug 9, 2014 at 3:08 AM, Stephen J. Turnbull step...@xemacs.org wrote: All the suggestions I've seen so far are (IMHO, YMMV) just as ugly as the present situation. What is ugly about allowing strings? CPython certainly has a way to to

Re: [Python-Dev] sum(...) limitation

2014-08-10 Thread Barry Warsaw
On Aug 10, 2014, at 05:24 PM, Stephen J. Turnbull wrote: Actually ... if I were a fan of the .join() idiom, I'd seriously propose 0.sum(numeric_iterable) as the RightThang{tm]. Then we could deprecate .join(string_iterable) in favor of .sum(string_iterable) (with the same efficient semantics).

Re: [Python-Dev] sum(...) limitation

2014-08-10 Thread Glenn Linderman
On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote: Actually ... if I were a fan of the .join() idiom, I'd seriously propose 0.sum(numeric_iterable) as the RightThang{tm]. Then we could deprecate .join(string_iterable) in favor of .sum(string_iterable) (with the same efficient semantics).

Re: [Python-Dev] sum(...) limitation

2014-08-10 Thread R. David Murray
On Sun, 10 Aug 2014 13:12:26 -0700, Glenn Linderman v+pyt...@g.nevcal.com wrote: On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote: Actually ... if I were a fan of the .join() idiom, I'd seriously propose 0.sum(numeric_iterable) as the RightThang{tm]. Then we could deprecate

Re: [Python-Dev] sum(...) limitation

2014-08-10 Thread R. David Murray
On Sun, 10 Aug 2014 13:12:26 -0700, Glenn Linderman v+pyt...@g.nevcal.com wrote: On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote: Actually ... if I were a fan of the .join() idiom, I'd seriously propose 0.sum(numeric_iterable) as the RightThang{tm]. Then we could deprecate

Re: [Python-Dev] sum(...) limitation

2014-08-10 Thread Stephen J. Turnbull
Glenn Linderman writes: On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote: Actually ... if I were a fan of the .join() idiom, I'd seriously propose 0.sum(numeric_iterable) as the RightThang{tm]. Then we could deprecate .join(string_iterable) in favor of .sum(string_iterable) (with

Re: [Python-Dev] sum(...) limitation

2014-08-09 Thread Stephen J. Turnbull
Alexander Belopolsky writes: Why have builtin sum at all if its use comes with so many caveats? Because we already have it. If the caveats had been known when it was introduced, maybe it wouldn't have been. The question is whether you can convince python-dev that it's worth changing the

Re: [Python-Dev] sum(...) limitation

2014-08-09 Thread Paul Moore
On 9 August 2014 06:08, Steven D'Aprano st...@pearwood.info wrote: py with Stopwatch(): ... sum(carray) # carray is a numpy array of 7500 floats. ... 11250.0 time taken: 52.659770 seconds py with Stopwatch(): ... numpy.sum(carray) ... 11250.0 time taken: 0.161263

Re: [Python-Dev] sum(...) limitation

2014-08-09 Thread Alexander Belopolsky
On Sat, Aug 9, 2014 at 3:08 AM, Stephen J. Turnbull step...@xemacs.org wrote: All the suggestions I've seen so far are (IMHO, YMMV) just as ugly as the present situation. What is ugly about allowing strings? CPython certainly has a way to to make sum(x, '') at least as efficient as y='';for

Re: [Python-Dev] sum(...) limitation

2014-08-09 Thread Alexander Belopolsky
On Sat, Aug 9, 2014 at 2:02 PM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: y='';for in in x; y+= x Should have been y='' for i in x; y += i ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] sum(...) limitation

2014-08-09 Thread Alexander Belopolsky
On Sat, Aug 9, 2014 at 1:08 AM, Steven D'Aprano st...@pearwood.info wrote: We wouldn't be having these interminable arguments about using sum() to concatenate strings (and lists, and tuples) if the operator was used for concatenation and + was only used for numeric addition. But we would

Re: [Python-Dev] sum(...) limitation

2014-08-09 Thread Devin Jeanpierre
On Sat, Aug 9, 2014 at 12:20 PM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: On Sat, Aug 9, 2014 at 1:08 AM, Steven D'Aprano st...@pearwood.info wrote: We wouldn't be having these interminable arguments about using sum() to concatenate strings (and lists, and tuples) if the

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Chris Barker
On Thu, Aug 7, 2014 at 4:01 PM, Ethan Furman et...@stoneleaf.us wrote: I don't remember where, but I believe that cPython has an optimization built in for repeated string concatenation, which is probably why you aren't seeing big differences between the + and the sum(). Indeed -- clearly so.

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Ethan Furman
On 08/08/2014 08:23 AM, Chris Barker wrote: So my final question is this: repeated string concatenation is not the recommended way to do this -- but nevertheless, cPython has an optimization that makes it fast and efficient, to the point that there is no practical performance reason to

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Raymond Hettinger
On Aug 8, 2014, at 11:09 AM, Ethan Furman et...@stoneleaf.us wrote: So why not apply a similar optimization to sum() for strings? That I cannot answer -- I find the current situation with sum highly irritating. It is only irritating if you are misusing sum(). The str.__add__

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Ethan Furman
On 08/08/2014 05:34 PM, Raymond Hettinger wrote: On Aug 8, 2014, at 11:09 AM, Ethan Furman et...@stoneleaf.us mailto:et...@stoneleaf.us wrote: So why not apply a similar optimization to sum() for strings? That I cannot answer -- I find the current situation with sum highly irritating.

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Alexander Belopolsky
On Fri, Aug 8, 2014 at 8:56 PM, Ethan Furman et...@stoneleaf.us wrote: I don't use sum at all, or at least very rarely, and it still irritates me. You are not alone. When I see sum([a, b, c]), I think it is a + b + c, but in Python it is 0 + a + b + c. If we had a join operator for strings

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Steven D'Aprano
On Fri, Aug 08, 2014 at 10:20:37PM -0400, Alexander Belopolsky wrote: On Fri, Aug 8, 2014 at 8:56 PM, Ethan Furman et...@stoneleaf.us wrote: I don't use sum at all, or at least very rarely, and it still irritates me. You are not alone. When I see sum([a, b, c]), I think it is a + b + c,

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Greg Ewing
Steven D'Aprano wrote: I've long believed that + is the wrong operator for concatenating strings, and that makes a much better operator. Do you have a reason for preferring '' in particular, or do you just want something different from '+'? Personally I can't see why bitwise and on strings

Re: [Python-Dev] sum(...) limitation

2014-08-08 Thread Antoine Pitrou
Le 09/08/2014 01:08, Steven D'Aprano a écrit : On Fri, Aug 08, 2014 at 10:20:37PM -0400, Alexander Belopolsky wrote: On Fri, Aug 8, 2014 at 8:56 PM, Ethan Furman et...@stoneleaf.us wrote: I don't use sum at all, or at least very rarely, and it still irritates me. You are not alone. When I

Re: [Python-Dev] sum(...) limitation

2014-08-07 Thread Chris Barker
On Mon, Aug 4, 2014 at 11:10 AM, Steven D'Aprano st...@pearwood.info wrote: On Mon, Aug 04, 2014 at 09:25:12AM -0700, Chris Barker wrote: Good point -- I was trying to make the point about .join() vs + for strings in an intro python class last year, and made the mistake of having the

Re: [Python-Dev] sum(...) limitation

2014-08-07 Thread Ethan Furman
On 08/07/2014 03:06 PM, Chris Barker wrote: [snip timings, etc.] I don't remember where, but I believe that cPython has an optimization built in for repeated string concatenation, which is probably why you aren't seeing big differences between the + and the sum(). A little testing shows how

Re: [Python-Dev] sum(...) limitation

2014-08-07 Thread Ethan Furman
On 08/07/2014 04:01 PM, Ethan Furman wrote: On 08/07/2014 03:06 PM, Chris Barker wrote: the + and the sum(). Yeah, that 'sum' should be 'join' :/ -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] sum(...) limitation

2014-08-07 Thread Ethan Furman
On 08/07/2014 04:01 PM, Ethan Furman wrote: On 08/07/2014 03:06 PM, Chris Barker wrote: -- timeit.Timer(for string in ['booya'] * 10: blah = blah + string, blah = '').repeat(3, 1) [0.021117210388183594, 0.013692855834960938, 0.00768280029296875] -- timeit.Timer(for string in ['booya'] *

Re: [Python-Dev] sum(...) limitation

2014-08-04 Thread Chris Barker
On Sat, Aug 2, 2014 at 1:35 PM, David Wilson dw+python-...@hmmz.org wrote: Repeated list and str concatenation both have quadratic O(N**2) performance, but people frequently build up strings with + join() isn't preferable in cases where it damages readability while simultaneously

Re: [Python-Dev] sum(...) limitation

2014-08-04 Thread Steven D'Aprano
On Mon, Aug 04, 2014 at 09:25:12AM -0700, Chris Barker wrote: Good point -- I was trying to make the point about .join() vs + for strings in an intro python class last year, and made the mistake of having the students test the performance. You need to concatenate a LOT of strings to see any

Re: [Python-Dev] sum(...) limitation

2014-08-04 Thread Stefan Behnel
Steven D'Aprano schrieb am 04.08.2014 um 20:10: On Mon, Aug 04, 2014 at 09:25:12AM -0700, Chris Barker wrote: Good point -- I was trying to make the point about .join() vs + for strings in an intro python class last year, and made the mistake of having the students test the performance.

Re: [Python-Dev] sum(...) limitation

2014-08-04 Thread Jim J. Jewett
Sat Aug 2 12:11:54 CEST 2014, Julian Taylor wrote (in https://mail.python.org/pipermail/python-dev/2014-August/135623.html ) wrote: Andrea Griffini agriff at tin.it wrote: However sum([[1,2,3],[4],[],[5,6]], []) concatenates the lists. hm could this be a pure python case that

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread Terry Reedy
On 8/2/2014 1:57 AM, Allen Li wrote: On Fri, Aug 01, 2014 at 02:51:54PM -0700, Guido van Rossum wrote: No. We just can't put all possible use cases in the docstring. :-) On Fri, Aug 1, 2014 at 2:48 PM, Andrea Griffini agr...@tin.it wrote: help(sum) tells clearly that it should be used

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread Julian Taylor
On 02.08.2014 08:35, Terry Reedy wrote: On 8/2/2014 1:57 AM, Allen Li wrote: On Fri, Aug 01, 2014 at 02:51:54PM -0700, Guido van Rossum wrote: No. We just can't put all possible use cases in the docstring. :-) On Fri, Aug 1, 2014 at 2:48 PM, Andrea Griffini agr...@tin.it wrote:

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread Stefan Behnel
Julian Taylor schrieb am 02.08.2014 um 12:11: On 02.08.2014 08:35, Terry Reedy wrote: On 8/2/2014 1:57 AM, Allen Li wrote: On Fri, Aug 01, 2014 at 02:51:54PM -0700, Guido van Rossum wrote: No. We just can't put all possible use cases in the docstring. :-) On Fri, Aug 1, 2014 at 2:48 PM,

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread Alexander Belopolsky
On Sat, Aug 2, 2014 at 3:39 AM, Steven D'Aprano st...@pearwood.info wrote: String concatenation with + is an attractive nuisance for many people, including some who actually know better but nevertheless do it. Also, for reasons I don't understand, many people dislike or cannot remember to use

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread Stefan Behnel
Alexander Belopolsky schrieb am 02.08.2014 um 16:52: On Sat, Aug 2, 2014 at 3:39 AM, Steven D'Aprano wrote: String concatenation with + is an attractive nuisance for many people, including some who actually know better but nevertheless do it. Also, for reasons I don't understand, many people

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread Steven D'Aprano
On Sat, Aug 02, 2014 at 10:52:07AM -0400, Alexander Belopolsky wrote: On Sat, Aug 2, 2014 at 3:39 AM, Steven D'Aprano st...@pearwood.info wrote: String concatenation with + is an attractive nuisance for many people, including some who actually know better but nevertheless do it. Also, for

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread MRAB
On 2014-08-02 16:27, Steven D'Aprano wrote: On Sat, Aug 02, 2014 at 10:52:07AM -0400, Alexander Belopolsky wrote: On Sat, Aug 2, 2014 at 3:39 AM, Steven D'Aprano st...@pearwood.info wrote: String concatenation with + is an attractive nuisance for many people, including some who actually know

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread Alexander Belopolsky
On Sat, Aug 2, 2014 at 11:06 AM, Stefan Behnel stefan...@behnel.de wrote: I don't think sum(strings) is beautiful enough sum(strings) is more beautiful than ''.join(strings) in my view, but unfortunately it does not work even for lists because the initial value defaults to 0. sum(strings, '')

Re: [Python-Dev] sum(...) limitation

2014-08-02 Thread David Wilson
On Sat, Aug 02, 2014 at 05:39:12PM +1000, Steven D'Aprano wrote: Repeated list and str concatenation both have quadratic O(N**2) performance, but people frequently build up strings with + and rarely do the same for lists. String concatenation with + is an attractive nuisance for many people,

[Python-Dev] sum(...) limitation

2014-08-01 Thread Andrea Griffini
help(sum) tells clearly that it should be used to sum numbers and not strings, and with strings actually fails. However sum([[1,2,3],[4],[],[5,6]], []) concatenates the lists. Is this to be considered a bug? Andrea ___ Python-Dev mailing list

Re: [Python-Dev] sum(...) limitation

2014-08-01 Thread Guido van Rossum
No. We just can't put all possible use cases in the docstring. :-) On Fri, Aug 1, 2014 at 2:48 PM, Andrea Griffini agr...@tin.it wrote: help(sum) tells clearly that it should be used to sum numbers and not strings, and with strings actually fails. However sum([[1,2,3],[4],[],[5,6]], [])

Re: [Python-Dev] sum(...) limitation

2014-08-01 Thread Allen Li
On Fri, Aug 01, 2014 at 02:51:54PM -0700, Guido van Rossum wrote: No. We just can't put all possible use cases in the docstring. :-) On Fri, Aug 1, 2014 at 2:48 PM, Andrea Griffini agr...@tin.it wrote: help(sum) tells clearly that it should be used to sum numbers and not strings,