Re: [Python-Dev] sum(...) limitation
Alexander Belopolsky writes: > Why have builtin sum at all if its use comes with so many caveats? Because we already have it. If the caveats had been known when it was introduced, maybe it wouldn't have been. The question is whether you can convince python-dev that it's worth changing the definition of sum(). IMO that's going to be very hard to do. All the suggestions I've seen so far are (IMHO, YMMV) just as ugly as the present situation. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On 9 August 2014 06:08, Steven D'Aprano wrote: > py> with Stopwatch(): > ... sum(carray) # carray is a numpy array of 7500 floats. > ... > 11250.0 > time taken: 52.659770 seconds > py> with Stopwatch(): > ... numpy.sum(carray) > ... > 11250.0 > time taken: 0.161263 seconds > > >> Why have builtin sum at all if its use comes with so many caveats? > > Because sum() is a perfectly reasonable general purpose tool for adding > up small amounts of numbers where high floating point precision is not > required. It has been included as a built-in because Python comes with > "batteries included", and a basic function for adding up a few numbers > is an obvious, simple battery. But serious programmers should be > comfortable with the idea that you use the right tool for the right job. Changing the subject a little, but the Stopwatch function you used up there is "an obvious, simple battery" for timing a chunk of code at the interactive prompt. I'm amazed there's nothing like it in the timeit module... Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] os.walk() is going to be *fast* with scandir
Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be. Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especially faster than listdir / walk. Note that benchmark results can vary a lot, depending on operating system, file system, hard drive type, and the OS's caching state. Anyway, os.walk() can be FIFTY times as fast using os.scandir(). # Old ctypes implementation of scandir in scandir.py: C:\work\scandir>\work\python\cpython\python benchmark.py -r Using slower ctypes version of scandir os.walk took 1.144s, scandir.walk took 0.060s -- 19.2x as fast # Existing "half C" implementation of scandir in _scandir.c: C:\work\scandir>\Python34-x86\python.exe benchmark.py -r Using fast C version of scandir os.walk took 1.160s, scandir.walk took 0.042s -- 27.6x as fast # New "all C" os.scandir implementation in posixmodule.c: C:\work\scandir>\work\python\cpython\python benchmark.py -r Using Python 3.5's builtin os.scandir() os.walk took 1.141s, scandir.walk took 0.022s -- 53.0x as fast [1] Work in progress implementation as part of Python 3.5's posixmodule.c available here: https://github.com/benhoyt/scandir/blob/master/posixmodule.c -Ben ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Sat, Aug 9, 2014 at 3:08 AM, Stephen J. Turnbull wrote: > All the suggestions > I've seen so far are (IMHO, YMMV) just as ugly as the present > situation. > What is ugly about allowing strings? CPython certainly has a way to to make sum(x, '') at least as efficient as y='';for in in x; y+= x is now. What is ugly about making sum([a, b, ..]) be equivalent to a + b + .. so that non-empty lists of arbitrary types can be "summed"? What is ugly about harmonizing sum(x) and reduce(operator.add, x) behaviors? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Sat, Aug 9, 2014 at 2:02 PM, Alexander Belopolsky < [email protected]> wrote: > y='';for in in x; y+= x Should have been y='' for i in x; y += i ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] class Foo(object) vs class Foo: should be clearly explained in python 2 and 3 doc
Hi. Referring to my discussion on [1] and then on #python this afternoon. A little background would help people to understand where this was coming from. 1. I write Python 2 code and have done zero Python-3 specific code. 2. I have always been using class Foo(object) so I do not know the new style is no longer required in Python 3. I feel "stupid" and "wrong" by thinking (object) is still a convention in Python 3. 3. Many Python 2 tutorials do not use object as the base class whether for historical reason, or lack of information/education, and can cause confusing to newcomers searching for answers when they consult the official documentation. While Python 3 code no longer requires object be the base class for the new-style class definition, I believe (object) is still required if one has to write a 2-3 compatible code. But this was not explained or warned anywhere in Python 2 and Python 3 code, AFAIK. (if I am wrong, please correct me) I propose the followings: * It is desirable to state boldly to users that (object) is no longer needed in Python-3 **only** code and warn users to revert to (object) style if the code needs to be 2 and 3 compatible. * In addition, Python 2 doc [2] should be fixed by introducing the new-style classes. This problem was noted a long long time ago according to [4]. * I would like to see warnings from suggested action item 1 on [2] and [3], for python 2 and 3 documentations. Possible objections(s): * We are pushing toward Python 3, some years later we don't need to maintain both Python 2 and 3 code. And many people, especially the newcomers will probably have no need to maintain Python 2 and 3 compatible codes. My answer to that is we need to be careful with marketing. First, it is a little embarrassing to assume and to find out the assumption is not entirely accurate. Secondly, Python 2 will not go away any time soon and most tutorials available on the Internet today are still written for Python 2. Furthermore, this CAN be a "gotcha" for new developers knowing only Python 3 writing Python 2 & 3 compatible code. * Books can do a better job I haven't actually reviewed/read any Python 3 books knowing most of my code should work without bothering Python 3-2 incompatibility yet. So I don't have an accurate answer, but a very very quick glance over a popular Python 3 book (I am not sure if naming it out is ethical or not so I am going to grey it out here) the book just writes class Foo: and doesn't note the different between 2 and 3 with classes. It is not wrong since the book is about programming in Python 3, NOT writing 2 and 3, but this is where the communication breaks. Docs and books don't give all the answers needed. P.S. Sorry if I should've have asked on #python-dev first or made a ticket but I've decided to send to mailing list before making a bug ticket. First time! Thanks. Best, Yeuk Hon [1]: https://news.ycombinator.com/item?id=8154471 [2]: https://docs.python.org/2/tutorial/classes.html https://docs.python.org/3/tutorial/classes.html [3]: https://docs.python.org/3/tutorial/classes.html [4]: https://www.python.org/doc/newstyle/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Sat, Aug 9, 2014 at 1:08 AM, Steven D'Aprano wrote: > We wouldn't be having > these interminable arguments about using sum() to concatenate strings > (and lists, and tuples) if the & operator was used for concatenation and > + was only used for numeric addition. > But we would probably have a similar discussion about all(). :-) Use of + is consistent with the use of * for repetition. What would you use use for repetition if you use & instead? Compare, for example s + ' ' * (n - len(s)) and s & ' ' * (n - len(s)) Which one is clearer? It is sum() that need to be fixed, not +. Not having sum([a, b]) equivalent to a + b for any a, b pair is hard to justify. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] class Foo(object) vs class Foo: should be clearly explained in python 2 and 3 doc
On 8/9/2014 2:44 PM, John Yeuk Hon Wong wrote: Hi. Referring to my discussion on [1] and then on #python this afternoon. A little background would help people to understand where this was coming from. 1. I write Python 2 code and have done zero Python-3 specific code. 2. I have always been using class Foo(object) so I do not know the new style is no longer required in Python 3. I feel "stupid" and "wrong" by thinking (object) is still a convention in Python 3. If someone else tried to make you feel that way, they are Code of Conduct violators who should be ignored. If you are beating yourself on the head, stop. 3. Many Python 2 tutorials do not use object as the base class whether for historical reason, or lack of information/education, Probably both. Either way, the result is a disservice to readers. and can cause confusing to newcomers searching for answers > when they consult the official documentation. I and some other people STRONGLY recommend that newcomers start with Python 3 and Python 3 docs and completely ignore Python 2 unless they cannot. While Python 3 code no longer requires object be the base class for the new-style class definition, I believe (object) is still required if one has to write a 2-3 compatible code. But this was not explained or warned anywhere in Python 2 and Python 3 code, AFAIK. (if I am wrong, please correct me) I propose the followings: * It is desirable to state boldly to users that (object) is no longer needed in Python-3 **only** code and warn users to revert to (object) style if the code needs to be 2 and 3 compatible. I think 'boldly' and 'warn' are a bit overstated. * In addition, Python 2 doc [2] should be fixed by introducing the new-style classes. Definitely. The 2.x tutorial start with class x: and continues that way half way through the chapter. I think it should start with class x(object): and at the end of the first half, briefly mention that class x in 2.x gets something slightly different that beginners can mostly ignore, while class x: in 3.x == class x(object): and that the latter works the same for both. The 3.x tutorial, in the same place could *briefly* mention that class x: == class x(object): and the the latter is usually only used in code that also runs on 2.x or has been converted without removing the extra code. The 3.x tutorial should *not* mention old style classes. > This problem was noted a long long time ago according to [4]. The opening statement "Unfortunately, new-style classes have not yet been integrated into Python's standard documention." is perhaps a decade out of date. That page should not have been included in the new site design without being modified. [1]: https://news.ycombinator.com/item?id=8154471 [2]: https://docs.python.org/2/tutorial/classes.html https://docs.python.org/3/tutorial/classes.html [3]: https://docs.python.org/3/tutorial/classes.html [4]: https://www.python.org/doc/newstyle/ -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sum(...) limitation
On Sat, Aug 9, 2014 at 12:20 PM, Alexander Belopolsky wrote: > > On Sat, Aug 9, 2014 at 1:08 AM, Steven D'Aprano wrote: >> >> We wouldn't be having >> these interminable arguments about using sum() to concatenate strings >> (and lists, and tuples) if the & operator was used for concatenation and >> + was only used for numeric addition. > > > But we would probably have a similar discussion about all(). :-) > > Use of + is consistent with the use of * for repetition. What would you use > use for repetition if you use & instead? If the only goal is to not be tempted to use sum() for string concatenation, how about using *? This is more consistent with mathematics terminology, where a * b is not necessarily the same as b * a (unlike +, which is commutative). As an example, consider matrix multiplication. Then, to answer your question, repetition would have been s ** n. (In fact, this is the notation for concatenation and repetition used in formal language theory.) (If we really super wanted to add this to Python, obviously we'd use the @ and @@ operators. But it's a bit late for that.) -- Devin ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] class Foo(object) vs class Foo: should be clearly explained in python 2 and 3 doc
On Sat, Aug 09, 2014 at 02:44:10PM -0400, John Yeuk Hon Wong wrote: > Hi. > > Referring to my discussion on [1] and then on #python this afternoon. > > A little background would help people to understand where this was > coming from. > > 1. I write Python 2 code and have done zero Python-3 specific code. > 2. I have always been using class Foo(object) so I do not know the new > style is no longer required in Python 3. I feel "stupid" and "wrong" by > thinking (object) is still a convention in Python 3. But object is still a convention in Python 3. It is certainly required when writing code that will behave the same in version 2 and 3, and it's optional in 3-only code, but certainly not frowned upon or discouraged. There's nothing wrong with explicitly inheriting from object in Python 3, and with the Zen of Python "Explicit is better than implicit" I would argue that *leaving it out* should be very slightly discouraged. class Spam: # okay, but a bit lazy class Spam(object): # better Perhaps PEP 8 should make a recommendation, but if so, I think it should be a very weak one. In Python 3, it really doesn't matter which you write. My own personal practice is to explicitly inherit from object when the class is "important" or more than half a dozen lines, and leave it out if the class is a stub or tiny. > 3. Many Python 2 tutorials do not use object as the base class whether > for historical reason, or lack of information/education, and can cause > confusing to newcomers searching for answers when they consult the > official documentation. We can't do anything about third party tutorials :-( > While Python 3 code no longer requires object be the base class for the > new-style class definition, I believe (object) is still required if one > has to write a 2-3 compatible code. But this was not explained or warned > anywhere in Python 2 and Python 3 code, AFAIK. (if I am wrong, please > correct me) It's not *always* required, only if you use features which require new-style classes, e.g. super, or properties. > I propose the followings: > > * It is desirable to state boldly to users that (object) is no longer > needed in Python-3 **only** code I'm against that. Stating this boldly will be understood by some readers that object should not be used, and I'm strongly against that. I believe explicitly inheriting from object should be mildly preferred, not strongly discouraged. > and warn users to revert to (object) > style if the code needs to be 2 and 3 compatible. I don't think that should be necesary, but have no objections to it being mentioned. I think it should be obvious: if you need new-style behaviour in Python 2, then obviously you have to inherit from object otherwise you have a classic class. That requirement doesn't go away just because your code will sometimes run under Python 3. Looking at your comment here: > [1]: https://news.ycombinator.com/item?id=8154471 there is a reply from zeckalpha, who says: "Actually, leaving out `object` is the preferred convention for Python 3, as they are semantically equivalent." How does (s)he justify this claim? "Explicit is better than implicit." which is not logical. If you leave out `object`, that's implicit, not explicit. -- Steven ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] class Foo(object) vs class Foo: should be clearly explained in python 2 and 3 doc
On Sun, Aug 10, 2014 at 10:44 AM, Steven D'Aprano wrote: > Looking at your comment here: > >> [1]: https://news.ycombinator.com/item?id=8154471 > > there is a reply from zeckalpha, who says: > >"Actually, leaving out `object` is the preferred convention for > Python 3, as they are semantically equivalent." > > How does (s)he justify this claim? > >"Explicit is better than implicit." > > which is not logical. If you leave out `object`, that's implicit, not > explicit. The justification is illogical. However, I personally believe boilerplate should be omitted where possible; that's why we have a whole lot of things that "just work". Why does Python not have explicit boolification for if/while checks? REXX does (if you try to use anything else, you get a run-time error "Logical value not 0 or 1"), and that's more explicit - Python could require you to write "if bool(x)" for the case where you actually want the truthiness magic, to distinguish from "if x is not None" etc. But that's unnecessary boilerplate. Python could have required explicit nonlocal declarations for all names used in closures, but that's unhelpful too. Python strives to eliminate that kind of thing. So, my view would be: Py3-only tutorials can and probably should omit it, for the same reason that we don't advise piles of __future__ directives. You can always add stuff later for coping with Py2+Py3 execution; chances are any non-trivial code will have much bigger issues than accidentally making an old-style class. ChrisA ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.walk() is going to be *fast* with scandir
Le 09/08/2014 12:43, Ben Hoyt a écrit : Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be. Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especially faster than listdir / walk. Note that benchmark results can vary a lot, depending on operating system, file system, hard drive type, and the OS's caching state. Anyway, os.walk() can be FIFTY times as fast using os.scandir(). Very nice results, thank you :-) Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.walk() is going to be *fast* with scandir
On 10 August 2014 13:20, Antoine Pitrou wrote: > Le 09/08/2014 12:43, Ben Hoyt a écrit : > >> Just thought I'd share some of my excitement about how fast the all-C >> version [1] of os.scandir() is turning out to be. >> >> Below are the results of my scandir / walk benchmark run with three >> different versions. I'm using an SSD, which seems to make it >> especially faster than listdir / walk. Note that benchmark results can >> vary a lot, depending on operating system, file system, hard drive >> type, and the OS's caching state. >> >> Anyway, os.walk() can be FIFTY times as fast using os.scandir(). > > > Very nice results, thank you :-) Indeed! This may actually motivate me to start working on a redesign of walkdir at some point, with scandir and DirEntry objects as the basis. My original approach was just too slow to be useful in practice (at least when working with trees on the scale of a full Fedora or RHEL build hosted on an NFS share). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.walk() is going to be *fast* with scandir
A small tip from my bzr days - cd into the directory before scanning it - especially if you'll end up statting more than a fraction of the files, or are recursing - otherwise the VFS does a traversal for each path you directly stat / recurse into. This can become a dominating factor in some workloads (I shaved several hundred milliseconds off of bzr stat on kernel trees doing this). -Rob On 10 August 2014 15:57, Nick Coghlan wrote: > On 10 August 2014 13:20, Antoine Pitrou wrote: >> Le 09/08/2014 12:43, Ben Hoyt a écrit : >> >>> Just thought I'd share some of my excitement about how fast the all-C >>> version [1] of os.scandir() is turning out to be. >>> >>> Below are the results of my scandir / walk benchmark run with three >>> different versions. I'm using an SSD, which seems to make it >>> especially faster than listdir / walk. Note that benchmark results can >>> vary a lot, depending on operating system, file system, hard drive >>> type, and the OS's caching state. >>> >>> Anyway, os.walk() can be FIFTY times as fast using os.scandir(). >> >> >> Very nice results, thank you :-) > > Indeed! > > This may actually motivate me to start working on a redesign of > walkdir at some point, with scandir and DirEntry objects as the basis. > My original approach was just too slow to be useful in practice (at > least when working with trees on the scale of a full Fedora or RHEL > build hosted on an NFS share). > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net -- Robert Collins Distinguished Technologist HP Converged Cloud ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.walk() is going to be *fast* with scandir
On 08/09/2014 10:40 PM, Robert Collins wrote: A small tip from my bzr days - cd into the directory before scanning it I doubt that's permissible for a library function like os.scandir(). //arry/ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
