A small tip from my bzr days - cd into the directory before scanning it - especially if you'll end up statting more than a fraction of the files, or are recursing - otherwise the VFS does a traversal for each path you directly stat / recurse into. This can become a dominating factor in some workloads (I shaved several hundred milliseconds off of bzr stat on kernel trees doing this).
-Rob On 10 August 2014 15:57, Nick Coghlan <ncogh...@gmail.com> wrote: > On 10 August 2014 13:20, Antoine Pitrou <anto...@python.org> wrote: >> Le 09/08/2014 12:43, Ben Hoyt a écrit : >> >>> Just thought I'd share some of my excitement about how fast the all-C >>> version [1] of os.scandir() is turning out to be. >>> >>> Below are the results of my scandir / walk benchmark run with three >>> different versions. I'm using an SSD, which seems to make it >>> especially faster than listdir / walk. Note that benchmark results can >>> vary a lot, depending on operating system, file system, hard drive >>> type, and the OS's caching state. >>> >>> Anyway, os.walk() can be FIFTY times as fast using os.scandir(). >> >> >> Very nice results, thank you :-) > > Indeed! > > This may actually motivate me to start working on a redesign of > walkdir at some point, with scandir and DirEntry objects as the basis. > My original approach was just too slow to be useful in practice (at > least when working with trees on the scale of a full Fedora or RHEL > build hosted on an NFS share). > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Converged Cloud _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
