Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Gregory P. Smith
On Jun 28, 2014 12:49 PM, "Ben Hoyt" wrote: > > >> But the underlying system calls -- ``FindFirstFile`` / > >> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- > > > > What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide readdir? > > I guess it'd be better to say "

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Nick Coghlan
On 29 June 2014 05:55, Ben Hoyt wrote: > Re is_dir etc being properties rather than methods: > >>> I find this behaviour a bit misleading: using methods and have them >>> return cached results. How much (implementation and/or performance >>> and/or memory) overhead would incur by using property-li

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Nick Coghlan
On 29 June 2014 05:48, Ben Hoyt wrote: >>> But the underlying system calls -- ``FindFirstFile`` / >>> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- >> >> What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide >> readdir? > > I guess it'd be better to say "Windows

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Ben Hoyt
Re is_dir etc being properties rather than methods: >> I find this behaviour a bit misleading: using methods and have them >> return cached results. How much (implementation and/or performance >> and/or memory) overhead would incur by using property-like access here? >> I think this would underlin

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Ben Hoyt
>> But the underlying system calls -- ``FindFirstFile`` / >> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- > > What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide readdir? I guess it'd be better to say "Windows" and "Unix-based OSs" throughout the PEP? Because

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Chris Angelico
On Sat, Jun 28, 2014 at 11:05 PM, Akira Li <4kir4...@gmail.com> wrote: > Have you considered adding support for paths relative to directory > descriptors [1] via keyword only dir_fd=None parameter if it may lead to > more efficient implementations on some platforms? > > [1]: https://docs.python.org

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Akira Li
Ben Hoyt writes: > Hi Python dev folks, > > I've written a PEP proposing a specific os.scandir() API for a > directory iterator that returns the stat-like info from the OS, *the > main advantage of which is to speed up os.walk() and similar > operations between 4-20x, depending on your OS and fil

Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7

2014-06-28 Thread Paul Sokolovsky
Hello, On Thu, 26 Jun 2014 22:49:40 +1000 Chris Angelico wrote: > On Thu, Jun 26, 2014 at 9:04 PM, Antoine Pitrou > wrote: > > For the same reason, I agree with Victor that we should ditch the > > threading-disabled builds. It's too much of a hassle for no actual, > > practical benefit. People

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Nick Coghlan
On 28 June 2014 16:17, Gregory P. Smith wrote: > On Fri, Jun 27, 2014 at 2:58 PM, Nick Coghlan wrote: >> * it would be nice to see some relative performance numbers for NFS and >> CIFS network shares - the additional network round trips can make excessive >> stat calls absolutely brutal from a sp

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-06-28 Thread Nick Coghlan
On 28 June 2014 19:17, Nick Coghlan wrote: > Agreed, but walking even a moderately large tree over the network can > really hammer home the point that this offers a significant > performance enhancement as the latency of access increases. I've found > that kind of comparison can be eye-opening for