Seems like if I wanted to do what is proposed in that Pep, I would have to drop down to the platform specific syscalls. Although at least it is already working like a "generator" and streaming the listing on each subsequent call.
On Wed, 17 Dec 2014 11:13 AM Anthony Tan <[email protected]> wrote: > Hm, interesting, I'll definitely have to have a poke when I get some > time - been looking for an excuse to do more Go but I can't justify any of > it here at work. Homework! :D > > > On Wed, Dec 17, 2014, at 05:52 AM, Justin Israel wrote: > > Oh woops. You were talking about the listdir impl in python. The > equivalent is readdir, and here: > https://golang.org/src/os/file_unix.go#L154 > > They are reading the names of the location and doing an lstat on each. > > On Wed, 17 Dec 2014 7:28 AM Justin Israel <[email protected]> wrote: > > If you are curious, the source for Go's walk is here: > https://golang.org/src/path/filepath/path.go#L389 > > It isn't quite what the python Pep describes, as it uses an lstat in > combination with readdirnames. It also isn't really the fastest of walk > implementations as some have pointed out. I'm actually using someone's > modified implementation which does the walk concurrently (although it > doesn't change the way it stats) : > https://github.com/MichaelTJones/walk/blob/master/walk.go#L140 > > I was actually thinking about this last night, now that you mention it. > Because my walk process in seqls doesn't need the stats for all the files. > I could probably get improved results if I just did a recursive > Readdirnames, as that is all I am interested in on each path. > > Also, in my benchmarks, just to mention, there wasn't much walking going > on except for the "all" test. The tests were all doing a readdir for the > listing of a given dir, to process the files into sequences. > > On Wed, 17 Dec 2014 3:13 AM Marcus Ottosson <[email protected]> > wrote: > > Specifically, i’m thinking about PEP-471. https://www.python.org/dev/ > peps/pep-0471/ > > That’s some really interesting stuff! Looks like it’s available for 2.7 > too. > > > On 16 December 2014 at 12:26, Anthony Tan <[email protected]> > wrote: > > > Wildish speculation on my part, I wouldn't be surprised if the go version > of had a better equivalent implementation of the os.listdir() call giving > you a lot of that speed up - esp if you're using NFS/CIFS - which would > make the go version better-er even without implementing any swanky > concurrency stuff. Specifically, i'm thinking about PEP-471. > https://www.python.org/dev/peps/pep-0471/ > > Some notes that show a rewritten os.walk is going to be a tad faster ( > http://www.gossamer-threads.com/lists/python/dev/1153484) so 25x doesn't > sound immediately improbable either. (Would be curious to see what the > speed up is from python 2.7 vs python 3.5.. maybe I'll run a benchmark post > Christmas for my own curiosity..) > > > On Tue, Dec 16, 2014, at 09:32 PM, Justin Israel wrote: > > Here are some benchmarks: > https://gist.github.com/justinfx/9c07ce03693f03c4bc46 > > I don't have a case on hand to reproduce where my co-worker was getting a > 25x speedup. I will have to see if I can find that. It was more a case of > "what were you getting before? Ok what are you getting now?" :-) > These were done on an SSD local drive, where as his tests were across an > NFS mounted filesystem. > > Also, the gofileseq library itself isn't yet using any concurrency for its > logic. Only the seqls tool uses concurrency for walking directories, and > processing lists of frames into sequences. I will probably do another pass > on gofileseq to get the FindSequencesOnDisk function processing files in > parallel. Haven't done much of any profiling yet. > > > On Tue Dec 16 2014 at 9:18:04 PM Justin Israel <[email protected]> > wrote: > > I can work up some benchmarks against the python fileseq. > > On Tue, 16 Dec 2014 8:57 PM Marcus Ottosson <[email protected]> > wrote: > > Sounds good, no benchmarks? > > > On 15 December 2014 at 18:35, Justin Israel <[email protected]> > wrote: > > The Go version was getting about a 25x speedup when using > FindSequencesOnDisk (which makes use of all of the other facilities) . The > parsing and string building of the paths, frame ranges, and results are > faster. I reuse buffers instead of constantly allocating new strings. Also > the find operation is concurrent. > An example was that my co-worker was using the python version to drive a > graphical file sequence browser. He saw that massively apparent speedup in > the interface when he switched to the Go version. > > On Tue, 16 Dec 2014 1:54 AM Marcus Ottosson <[email protected]> > wrote: > > I haven't used the Python library, but what would be the advantage(s) of a > Go version compared with the Python version? > > > On 13 December 2014 at 22:31, Justin Israel <[email protected]> > wrote: > > Hey all, > > This isn't directly python related, but I thought I would share anyways. > If anyone has been using the python library fileseq > <https://pypi.python.org/pypi/Fileseq/0.5.1>, which I help maintain, to > work with file sequence and frame range patterns, I've built a tool around > this functionality (kind of similar to rvls). > > gofileseq <https://github.com/justinfx/gofileseq> is a port of fileseq, > to the Go language. Included in this repo is a utility called seqls > <https://github.com/justinfx/gofileseq/tree/master/cmd/seqls>, which uses > gofileseq to provide a way to list files on the filesystem that are rolled > up into sequence formats. So far it can do short and long listings, rolling > up the file sizes and showing the latest mod time. I've cross compiled it > for win/linux/osx > <https://bintray.com/justinfx/utils/seqls/_latestVersion> > > If you find it useful or have suggestions, please let me know! Also feel > free to register issues for it on github if needed. I develop on osx and > linux, so I have only briefly tested it on windows.. > > Enjoy! > > > > Justin > > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/python_inside_maya/CAPGFgA2KpONX8cs9%3Df%2B4zzQx11yo1e7q > V09bNTrT9zwudEfTRA%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2KpONX8cs9%3Df%2B4zzQx11yo1e7qV09bNTrT9zwudEfTRA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > > > -- > *Marcus Ottosson* > [email protected] > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/python_inside_maya/CAFRtmODiTWjOJ3jm2pH5RN-CKSqoAvpQ6kgh > L4ja752PMiXoQQ%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODiTWjOJ3jm2pH5RN-CKSqoAvpQ6kghL4ja752PMiXoQQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To view this discussion on the web visit https://groups.google.com/d/ms > gid/python_inside_maya/CAPGFgA1XKJy%2BWS4V5nPr53t8x%3D% > 3D2eTdULbvuzF_XGihM2Q1-GQ%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1XKJy%2BWS4V5nPr53t8x%3D%3D2eTdULbvuzF_XGihM2Q1-GQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > *Marcus Ottosson* > [email protected] > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/python_inside_maya/CAFRtmOCsf7uiqwUKWaO6q9QBrZiCxSP- > rZLu4DbtxKyuY4qzHA%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCsf7uiqwUKWaO6q9QBrZiCxSP-rZLu4DbtxKyuY4qzHA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/python_inside_maya/CAPGFgA1zQ%2BNpmmqR2gWDe0vKEAd%2B% > 3DB47x2fkFL_y1JEErQOieQ%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1zQ%2BNpmmqR2gWDe0vKEAd%2B%3DB47x2fkFL_y1JEErQOieQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/python_inside_maya/1418732783.2884423.203482413.7E971BE9%40webmail. > messagingengine.com > <https://groups.google.com/d/msgid/python_inside_maya/1418732783.2884423.203482413.7E971BE9%40webmail.messagingengine.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > *Marcus Ottosson* > [email protected] > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/python_inside_maya/CAFRtmOA784b-Qspr1-VCCAVDESTuB1ZxTep95AfdBAv5V4Ph > aw%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOA784b-Qspr1-VCCAVDESTuB1ZxTep95AfdBAv5V4Phaw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0maPnOO9OQ%2BDNQrQ3sX8cjUW-91i1sikQTs6D7PRWkEA%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0maPnOO9OQ%2BDNQrQ3sX8cjUW-91i1sikQTs6D7PRWkEA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/1418767983.738136.203723453.3A60AF73%40webmail.messagingengine.com > <https://groups.google.com/d/msgid/python_inside_maya/1418767983.738136.203723453.3A60AF73%40webmail.messagingengine.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2E4%2BKorJMi6eeRbJcHY5-D61f%2BWTF1PCAKkXBUitGR%2BA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
