Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 6/26/2014 6:59 PM, Ben Hoyt wrote: Rationale = Python's built-in ``os.walk()`` is significantly slower than it needs to be, because -- in addition to calling ``os.listdir()`` on each directory -- it executes the system call ``os.stat()`` or ``GetFileAttributes()`` on each file to dete

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

2014-07-01 4:04 GMT+02:00 Glenn Linderman : >> +0 for stat fields to be None on all platforms unless ensure_lstat=True. > > This won't work well if lstat info is only needed for some entries. Is > that a common use-case? It was mentioned earlier in the thread. > > If it is, use ensure_lstat=False,

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 6/30/2014 9:44 PM, Ethan Furman wrote: On 06/30/2014 06:28 PM, Ben Hoyt wrote: I suppose the exact behavior is still under discussion, as there are only two or three fields one gets "for free" on Windows (I think...), where as an os.stat call would get everything available for the platform.

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 6/30/2014 10:17 PM, Nick Coghlan wrote: > > On 30 Jun 2014 19:13, "Glenn Linderman" > wrote: >> >> >> If it is, use ensure_lstat=False, and use the proposed (by me) > .refresh() API to update the data for those that need it. > > I'm -1 on a refresh API for DirE

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 06/30/2014 06:28 PM, Ben Hoyt wrote: I suppose the exact behavior is still under discussion, as there are only two or three fields one gets "for free" on Windows (I think...), where as an os.stat call would get everything available for the platform. No, Windows is nice enough to give you all

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 30 Jun 2014 19:13, "Glenn Linderman" wrote: > > > If it is, use ensure_lstat=False, and use the proposed (by me) .refresh() API to update the data for those that need it. I'm -1 on a refresh API for DirEntry - just use pathlib in that case. Cheers, Nick. > > _

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

The proposal I was replying to was that: - There is no .refresh() - ensure_lstat=False means no OS has populated attributes - ensure_lstat=True means ever OS has populated attributes Even if we add a .refresh(), the latter two items mean that you can't avoid doing extra work (either too much on w

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 6/30/2014 4:25 PM, Devin Jeanpierre wrote: On Mon, Jun 30, 2014 at 3:07 PM, Tim Delaney wrote: On 1 July 2014 03:05, Ben Hoyt wrote: So, here's my alternative proposal: add an "ensure_lstat" flag to scandir() itself, and don't have *any* methods on DirEntry, only attributes. ... Most im

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> I suppose the exact behavior is still under discussion, as there are only > two or three fields one gets "for free" on Windows (I think...), where as an > os.stat call would get everything available for the platform. No, Windows is nice enough to give you all the same stat_result fields during s

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 06/30/2014 04:15 PM, Tim Delaney wrote: On 1 July 2014 08:38, Ethan Furman wrote: On 06/30/2014 03:07 PM, Tim Delaney wrote: I'm torn between whether I'd prefer the stat fields to be populated on Windows if ensure_lstat=False or not. There are good arguments each way, but overall I'm inclin

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Mon, Jun 30, 2014 at 3:07 PM, Tim Delaney wrote: > On 1 July 2014 03:05, Ben Hoyt wrote: >> >> > So, here's my alternative proposal: add an "ensure_lstat" flag to >> > scandir() itself, and don't have *any* methods on DirEntry, only >> > attributes. >> ... >> >> > Most importantly, *regardless

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 1 July 2014 08:38, Ethan Furman wrote: > On 06/30/2014 03:07 PM, Tim Delaney wrote: > >> I'm torn between whether I'd prefer the stat fields to be populated >> on Windows if ensure_lstat=False or not. There are good arguments each >> way, but overall I'm inclining towards having it consistent

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 06/30/2014 03:07 PM, Tim Delaney wrote: On 1 July 2014 03:05, Ben Hoyt wrote: So, here's my alternative proposal: add an "ensure_lstat" flag to scandir() itself, and don't have *any* methods on DirEntry, only attributes. ... Most importantly, *regardless of platform*, the cached stat result

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 1 July 2014 03:05, Ben Hoyt wrote: > > So, here's my alternative proposal: add an "ensure_lstat" flag to > > scandir() itself, and don't have *any* methods on DirEntry, only > > attributes. > ... > > Most importantly, *regardless of platform*, the cached stat result (if > > not None) would ref

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> So, here's my alternative proposal: add an "ensure_lstat" flag to > scandir() itself, and don't have *any* methods on DirEntry, only > attributes. > > That would make the DirEntry attributes: > > is_dir: boolean, always populated > is_file: boolean, always populated > is_symlink boole

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 6/29/2014 5:28 AM, Nick Coghlan wrote: There'd still be a slight window of discrepancy (since the filesystem state may change between reading the directory entry and making the lstat() call), but this could be effectively eliminated from the perspective of the Python code by making the result

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 29.06.2014 19:04, Ethan Furman wrote: > On 06/29/2014 04:12 AM, Jonas Wielicki wrote: >> >> If the flag is set to False, all the fields in the DirEntry will be >> None, for consistency, even on Windows. > > -1 > > This consistency is unnecessary. I’m not sure -- similar to the windows_wildcard

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Chris Angelico writes: > On Sat, Jun 28, 2014 at 11:05 PM, Akira Li <4kir4...@gmail.com> wrote: >> Have you considered adding support for paths relative to directory >> descriptors [1] via keyword only dir_fd=None parameter if it may lead to >> more efficient implementations on some platforms? >>

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 06/29/2014 04:12 AM, Jonas Wielicki wrote: If the flag is set to False, all the fields in the DirEntry will be None, for consistency, even on Windows. -1 This consistency is unnecessary. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.o

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 06/29/2014 05:28 AM, Nick Coghlan wrote: So, here's my alternative proposal: add an "ensure_lstat" flag to scandir() itself, and don't have *any* methods on DirEntry, only attributes. That would make the DirEntry attributes: is_dir: boolean, always populated is_file: boolean, alwa

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 29.06.2014 13:08, Nick Coghlan wrote: > On 29 June 2014 20:52, Steven D'Aprano wrote: >> Speaking of caching, is there a way to freshen the cached values? > > Switch to a full Path object instead of relying on the cached DirEntry data. > > This is what makes me wary of including lstat, even t

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 29 June 2014 21:45, Paul Moore wrote: > On 29 June 2014 12:08, Nick Coghlan wrote: >> This is what makes me wary of including lstat, even though Windows >> offers it without the extra stat call. Caching behaviour is *really* >> hard to make intuitive, especially when it *sometimes* returns dat

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 29 June 2014 12:08, Nick Coghlan wrote: > This is what makes me wary of including lstat, even though Windows > offers it without the extra stat call. Caching behaviour is *really* > hard to make intuitive, especially when it *sometimes* returns data > that looks fresh (as it on first call on PO

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 29 June 2014 20:52, Steven D'Aprano wrote: > Speaking of caching, is there a way to freshen the cached values? Switch to a full Path object instead of relying on the cached DirEntry data. This is what makes me wary of including lstat, even though Windows offers it without the extra stat call.

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Sat, Jun 28, 2014 at 03:55:00PM -0400, Ben Hoyt wrote: > Re is_dir etc being properties rather than methods: [...] > The problem with this is that properties "look free", they look just > like attribute access, so you wouldn't normally handle exceptions when > accessing them. But .lstat() and .i

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 28 Jun 2014, at 21:48, Ben Hoyt wrote: [...] Crazy idea: would it be possible to "convert" a DirEntry object to a pathlib.Path object without losing the cache? I guess that pathlib.Path expects a full stat_result object. The main problem is that pathlib.Path objects explicitly don't cache

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Jun 28, 2014 12:49 PM, "Ben Hoyt" wrote: > > >> But the underlying system calls -- ``FindFirstFile`` / > >> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- > > > > What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide readdir? > > I guess it'd be better to say "

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 29 June 2014 05:55, Ben Hoyt wrote: > Re is_dir etc being properties rather than methods: > >>> I find this behaviour a bit misleading: using methods and have them >>> return cached results. How much (implementation and/or performance >>> and/or memory) overhead would incur by using property-li

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 29 June 2014 05:48, Ben Hoyt wrote: >>> But the underlying system calls -- ``FindFirstFile`` / >>> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- >> >> What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide >> readdir? > > I guess it'd be better to say "Windows

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Re is_dir etc being properties rather than methods: >> I find this behaviour a bit misleading: using methods and have them >> return cached results. How much (implementation and/or performance >> and/or memory) overhead would incur by using property-like access here? >> I think this would underlin

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

>> But the underlying system calls -- ``FindFirstFile`` / >> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- > > What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide readdir? I guess it'd be better to say "Windows" and "Unix-based OSs" throughout the PEP? Because

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Sat, Jun 28, 2014 at 11:05 PM, Akira Li <4kir4...@gmail.com> wrote: > Have you considered adding support for paths relative to directory > descriptors [1] via keyword only dir_fd=None parameter if it may lead to > more efficient implementations on some platforms? > > [1]: https://docs.python.org

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Ben Hoyt writes: > Hi Python dev folks, > > I've written a PEP proposing a specific os.scandir() API for a > directory iterator that returns the stat-like info from the OS, *the > main advantage of which is to speed up os.walk() and similar > operations between 4-20x, depending on your OS and fil

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 28 June 2014 16:17, Gregory P. Smith wrote: > On Fri, Jun 27, 2014 at 2:58 PM, Nick Coghlan wrote: >> * it would be nice to see some relative performance numbers for NFS and >> CIFS network shares - the additional network round trips can make excessive >> stat calls absolutely brutal from a sp

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 28 June 2014 19:17, Nick Coghlan wrote: > Agreed, but walking even a moderately large tree over the network can > really hammer home the point that this offers a significant > performance enhancement as the latency of access increases. I've found > that kind of comparison can be eye-opening for

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Fri, Jun 27, 2014 at 2:58 PM, Nick Coghlan wrote: > > * -1 on including Windows specific globbing support in the API > * -0 on including cross platform globbing support in the initial iteration > of the API (that could be done later as a separate RFE instead) > Agreed. Globbing or filtering s

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 28 Jun 2014 01:27, "Jonas Wielicki" wrote: > > On 27.06.2014 00:59, Ben Hoyt wrote: > > Specifics of proposal > > = > > [snip] Each ``DirEntry`` object has the following > > attributes and methods: > > [snip] > > Notes on caching > > > > > > The ``DirEntry``

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 27.06.2014 00:59, Ben Hoyt wrote: > Specifics of proposal > = > [snip] Each ``DirEntry`` object has the following > attributes and methods: > [snip] > Notes on caching > > > The ``DirEntry`` objects are relatively dumb -- the ``name`` attribute > is obviousl

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 27.06.2014 03:50, MRAB wrote: > On 2014-06-27 02:37, Ben Hoyt wrote: >> I don't mind iterdir() and would take it :-), but I'll just say why I >> chose the name scandir() -- though it wasn't my suggestion originally: >> >> iterdir() sounds like just an iterator version of listdir(), kinda >> like

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Hello, On Fri, 27 Jun 2014 12:08:41 +1000 Steven D'Aprano wrote: > On Fri, Jun 27, 2014 at 03:07:46AM +0300, Paul Sokolovsky wrote: > > > With my MicroPython hat on, os.scandir() would make things only > > worse. With current interface, one can either have inefficient > > implementation (like C

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Hello, On Thu, 26 Jun 2014 21:52:43 -0400 Ben Hoyt wrote: [] > It's a fair point that os.walk() can be implemented efficiently > without adding a new function and API. However, often you'll want more > info, like the file size, which scandir() can give you via > DirEntry.lstat(), which is free

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Jun 26, 2014, at 4:38 PM, Tim Delaney wrote: On 27 June 2014 09:28, MRAB wrote: > > -1 for windows_wildcard (it would be an attractive nuisance to write windows-only code) Could you emulate it on other platforms? +1 on the rest of it. -Chris __

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Hi, You wrote a great PEP Ben, thanks :-) But it's now time for comments! > But the underlying system calls -- ``FindFirstFile`` / > ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide readdir? You should add a

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 26 June 2014 23:59, Ben Hoyt wrote: > Would love feedback on the PEP, but also of course on the proposal itself. A solid +1 from me. Some specific points: - I'm in favour of it being in the os module. It's more discoverable there, as well as the other reasons mentioned. - I prefer scandir as

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

I'm generally +1, with opinions noted below on these two topics. On 6/26/2014 3:59 PM, Ben Hoyt wrote: Should there be a way to access the full path? -- Should ``DirEntry``'s have a way to get the full path without using ``os.path.join(path, entry.nam

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Thu, Jun 26, 2014 at 09:37:50PM -0400, Ben Hoyt wrote: > I don't mind iterdir() and would take it :-), but I'll just say why I > chose the name scandir() -- though it wasn't my suggestion originally: > > iterdir() sounds like just an iterator version of listdir(), kinda > like keys() and iterke

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Fri, Jun 27, 2014 at 03:07:46AM +0300, Paul Sokolovsky wrote: > With my MicroPython hat on, os.scandir() would make things only worse. > With current interface, one can either have inefficient implementation > (like CPython chose) or efficient implementation (like MicroPython > chose) - all tra

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

+1 on getting this in for 3.5. If the only objection people are having is the stupid paint color of the name I don't care what it's called! scandir matches the libc API of the same name. iterdir also makes sense to anyone reading it. Whoever checks this in can pick one and be done with it. We

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> os.listdir() when I worked on "os" module for MicroPython. I essentially > did what your PEP suggests - introduced internal generator function > (ilistdir_ex() in > https://github.com/micropython/micropython-lib/blob/master/os/os/__init__.py#L85 > ), in terms of which both os.listdir() and os.wal

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 2014-06-27 02:37, Ben Hoyt wrote: I don't mind iterdir() and would take it :-), but I'll just say why I chose the name scandir() -- though it wasn't my suggestion originally: iterdir() sounds like just an iterator version of listdir(), kinda like keys() and iterkeys() in Python 2. Whereas in

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

I don't mind iterdir() and would take it :-), but I'll just say why I chose the name scandir() -- though it wasn't my suggestion originally: iterdir() sounds like just an iterator version of listdir(), kinda like keys() and iterkeys() in Python 2. Whereas in actual fact the return values are quite

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

+1 for scandir. -1 for iterdir(scandir sounds fancier). - for windows_wildcard. Tim Delaney wrote: >On 27 June 2014 09:28, MRAB wrote: > >> Personally, I'd prefer the name 'iterdir' because it emphasises that >> it's an iterator. > > >Exactly what I was going to post (with the added note

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Hello, On Thu, 26 Jun 2014 17:35:21 -0700 Benjamin Peterson wrote: > On Thu, Jun 26, 2014, at 17:07, Paul Sokolovsky wrote: > > > > With my MicroPython hat on, os.scandir() would make things only > > worse. With current interface, one can either have inefficient > > implementation (like CPython

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On Thu, Jun 26, 2014, at 17:07, Paul Sokolovsky wrote: > > With my MicroPython hat on, os.scandir() would make things only worse. > With current interface, one can either have inefficient implementation > (like CPython chose) or efficient implementation (like MicroPython > chose) - all transparent

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 06/26/2014 04:36 PM, Tim Delaney wrote: On 27 June 2014 09:28, MRAB wrote: Personally, I'd prefer the name 'iterdir' because it emphasises that it's an iterator. Exactly what I was going to post (with the added note that thee's an obvious symmetry with listdir). +1 for iterdir rather tha

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Hello, On Thu, 26 Jun 2014 18:59:45 -0400 Ben Hoyt wrote: > Hi Python dev folks, > > I've written a PEP proposing a specific os.scandir() API for a > directory iterator that returns the stat-like info from the OS, the > main advantage of which is to speed up os.walk() and similar > operations b

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 27 June 2014 09:28, MRAB wrote: > Personally, I'd prefer the name 'iterdir' because it emphasises that > it's an iterator. Exactly what I was going to post (with the added note that thee's an obvious symmetry with listdir). +1 for iterdir rather than scandir Other than that: +1 for adding

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

On 2014-06-26 23:59, Ben Hoyt wrote: Hi Python dev folks, I've written a PEP proposing a specific os.scandir() API for a directory iterator that returns the stat-like info from the OS, the main advantage of which is to speed up os.walk() and similar operations between 4-20x, depending on your OS

[Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Hi Python dev folks, I've written a PEP proposing a specific os.scandir() API for a directory iterator that returns the stat-like info from the OS, the main advantage of which is to speed up os.walk() and similar operations between 4-20x, depending on your OS and file system. Full details, backgro