Hm, interesting, I'll definitely have to have a poke when I get some time - been looking for an excuse to do more Go but I can't justify any of it here at work. Homework! :D
On Wed, Dec 17, 2014, at 05:52 AM, Justin Israel wrote: > Oh woops. You were talking about the listdir impl in python. The > equivalent is readdir, and here: > https://golang.org/src/os/file_unix.go#L154 > They are reading the names of the location and doing an lstat on each. > > On Wed, 17 Dec 2014 7:28 AM Justin Israel > <[email protected]> wrote: >> If you are curious, the source for Go's walk is here: >> https://golang.org/src/path/filepath/path.go#L389 >> It isn't quite what the python Pep describes, as it uses an lstat in >> combination with readdirnames. It also isn't really the fastest of >> walk implementations as some have pointed out. I'm actually using >> someone's modified implementation which does the walk concurrently >> (although it doesn't change the way it stats) : >> https://github.com/MichaelTJones/walk/blob/master/walk.go#L140 >> I was actually thinking about this last night, now that you >> mention it. Because my walk process in seqls doesn't need the >> stats for all the files. I could probably get improved results if >> I just did a recursive Readdirnames, as that is all I am >> interested in on each path. >> Also, in my benchmarks, just to mention, there wasn't much walking >> going on except for the "all" test. The tests were all doing a >> readdir for the listing of a given dir, to process the files into >> sequences. >> >> On Wed, 17 Dec 2014 3:13 AM Marcus Ottosson >> <[email protected]> wrote: >>>> Specifically, i’m thinking about PEP-471. >>>> https://www.python.org/dev/__peps/pep-0471/ >>> That’s some really interesting stuff! Looks like it’s available for >>> 2.7 too. >>> >>> >>> On 16 December 2014 at 12:26, Anthony Tan >>> <[email protected]> wrote: >>>> __ >>>> Wildish speculation on my part, I wouldn't be surprised if the go >>>> version of had a better equivalent implementation of the >>>> os.listdir() call giving you a lot of that speed up - esp if you're >>>> using NFS/CIFS - which would make the go version better-er even >>>> without implementing any swanky concurrency stuff. Specifically, >>>> i'm thinking about PEP-471. >>>> https://www.python.org/dev/__peps/pep-0471/ >>>> >>>> Some notes that show a rewritten os.walk is going to be a tad >>>> faster (http://www.gossamer-threads.__com/lists/python/dev/1153484) >>>> so 25x doesn't sound immediately improbable either. (Would be >>>> curious to see what the speed up is from python 2.7 vs python 3.5.. >>>> maybe I'll run a benchmark post Christmas for my own curiosity..) >>>> >>>> >>>> On Tue, Dec 16, 2014, at 09:32 PM, Justin Israel wrote: >>>>> Here are some benchmarks: >>>>> https://gist.github.com/__justinfx/9c07ce03693f03c4bc46 >>>>> >>>>> I don't have a case on hand to reproduce where my co-worker was >>>>> getting a 25x speedup. I will have to see if I can find that. It >>>>> was more a case of "what were you getting before? Ok what are you >>>>> getting now?" :-) These were done on an SSD local drive, where as >>>>> his tests were across an NFS mounted filesystem. >>>>> >>>>> Also, the gofileseq library itself isn't yet using any concurrency >>>>> for its logic. Only the seqls tool uses concurrency for walking >>>>> directories, and processing lists of frames into sequences. I will >>>>> probably do another pass on gofileseq to get the >>>>> FindSequencesOnDisk function processing files in parallel. Haven't >>>>> done much of any profiling yet. >>>>> >>>>> >>>>> On Tue Dec 16 2014 at 9:18:04 PM Justin Israel >>>>> <[email protected]> wrote: >>>>>> I can work up some benchmarks against the python fileseq. >>>>>> >>>>>> On Tue, 16 Dec 2014 8:57 PM Marcus Ottosson >>>>>> <[email protected]> wrote: >>>>>>> Sounds good, no benchmarks? >>>>>>> >>>>>>> >>>>>>> On 15 December 2014 at 18:35, Justin Israel >>>>>>> <[email protected]> wrote: >>>>>>>> The Go version was getting about a 25x speedup when using >>>>>>>> FindSequencesOnDisk (which makes use of all of the other >>>>>>>> facilities) . The parsing and string building of the paths, >>>>>>>> frame ranges, and results are faster. I reuse buffers instead >>>>>>>> of constantly allocating new strings. Also the find operation >>>>>>>> is concurrent. >>>>>>>> An example was that my co-worker was using the python version to drive a graphical file sequence browser. He saw that massively apparent speedup in the interface when he switched to the Go version. >>>>>>>> >>>>>>>> On Tue, 16 Dec 2014 1:54 AM Marcus Ottosson >>>>>>>> <[email protected]> wrote: >>>>>>>>> I haven't used the Python library, but what would be the >>>>>>>>> advantage(s) of a Go version compared with the Python version? >>>>>>>>> >>>>>>>>> >>>>>>>>> On 13 December 2014 at 22:31, Justin Israel >>>>>>>>> <[email protected]> wrote: >>>>>>>>>> Hey all, >>>>>>>>>> >>>>>>>>>> This isn't directly python related, but I thought I would >>>>>>>>>> share anyways. If anyone has been using the python library >>>>>>>>>> fileseq[1], which I help maintain, to work with file sequence >>>>>>>>>> and frame range patterns, I've built a tool around this >>>>>>>>>> functionality (kind of similar to rvls). >>>>>>>>>> >>>>>>>>>> gofileseq[2] is a port of fileseq, to the Go language. >>>>>>>>>> Included in this repo is a utility called seqls[3], which >>>>>>>>>> uses gofileseq to provide a way to list files on the >>>>>>>>>> filesystem that are rolled up into sequence formats. So far >>>>>>>>>> it can do short and long listings, rolling up the file sizes >>>>>>>>>> and showing the latest mod time. I've cross compiled it for >>>>>>>>>> win/linux/osx[4] >>>>>>>>>> >>>>>>>>>> If you find it useful or have suggestions, please let me >>>>>>>>>> know! Also feel free to register issues for it on github if >>>>>>>>>> needed. I develop on osx and linux, so I have only briefly >>>>>>>>>> tested it on windows.. >>>>>>>>>> >>>>>>>>>> Enjoy! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Justin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit https://groups.google.com/d/__ms__gid/python_inside_maya/__CAPGFgA__2KpONX8cs9%3Df%__2B4zzQx11yo1e7q__V09bNTrT9zwudEf__TRA%40mail.__gmail.com[5]. >>>>>>>>>> For more options, visit https://groups.google.com/d/__op__tout. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Marcus Ottosson* [email protected] >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the >>>>>>>>> Google Groups "Python Programming for Autodesk Maya" group. To >>>>>>>>> unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to >>>>>>>>> [email protected]. To view >>>>>>>>> this discussion on the web visit >>>>>>>>> https://groups.google.com/d/__ms__gid/python_inside_maya/__CAFRtmO__DiTWjOJ3jm2pH5RN-__CKSqoAvpQ6kgh__L4ja752PMiXoQQ%__40mail.gmail.__com[6]. >>>>>>>>> For more options, visit https://groups.google.com/d/__op__tout. >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/__ms__gid/python_inside_maya/__CAPGFgA__1XKJy%2BWS4V5nPr53t8x%__3D%__3D2eTdULbvuzF_XGihM2Q1-GQ%__40ma__il.gmail.com[7]. >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/__op__tout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Marcus Ottosson* [email protected] >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the >>>>>>> Google Groups "Python Programming for Autodesk Maya" group. To >>>>>>> unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to >>>>>>> [email protected]. To view >>>>>>> this discussion on the web visit >>>>>>> https://groups.google.com/d/__ms__gid/python_inside_maya/__CAFRtmO__Csf7uiqwUKWaO6q9QBrZiCx__SP-__rZLu4DbtxKyuY4qzHA%40mail.__gmai__l.com[8]. >>>>>>> For more options, visit https://groups.google.com/d/__op__tout. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Python Programming for Autodesk Maya" group. To >>>>> unsubscribe from this group and stop receiving emails from it, >>>>> send an email to >>>>> [email protected]. To view this >>>>> discussion on the web visit >>>>> https://groups.google.com/d/__msgid/python_inside_maya/__CAPGFgA1zQ%__2BNpmmqR2gWDe0vKEAd%2B%__3DB47x2fkFL_y1JEErQOieQ%__40mail.gmail.com[9]. >>>>> For more options, visit https://groups.google.com/d/__optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Python Programming for Autodesk Maya" group. To unsubscribe >>>> from this group and stop receiving emails from it, send an email to >>>> [email protected]. To view this >>>> discussion on the web visit >>>> https://groups.google.com/d/__msgid/python_inside_maya/__1418732783.2884423.203482413.__7E971BE9%40webmail.__messagingengine.com[10]. >>>> >>>> For more options, visit https://groups.google.com/d/__optout. >>> >>> >>> >>> -- >>> *Marcus Ottosson* [email protected] >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Python Programming for Autodesk Maya" group. To unsubscribe >>> from this group and stop receiving emails from it, send an email to >>> [email protected]. To view this >>> discussion on the web visit >>> https://groups.google.com/d/__msgid/python_inside_maya/__CAFRtmOA784b-Qspr1-__VCCAVDESTuB1ZxTep95AfdBAv5V4Ph__aw%40mail.gmail.com[11]. >>> For more options, visit https://groups.google.com/d/__optout. > > -- > You received this message because you are subscribed to the Google > Groups "Python Programming for Autodesk Maya" group. To unsubscribe > from this group and stop receiving emails from it, send an email to > [email protected]. To view this > discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0maPnOO9OQ%2BDNQrQ3sX8cjUW-91i1sikQTs6D7PRWkEA%40mail.gmail.com[12]. > For more options, visit https://groups.google.com/d/optout. Links: 1. https://pypi.python.org/pypi/Fileseq/0.5.1 2. https://github.com/justinfx/gofileseq 3. https://github.com/justinfx/gofileseq/tree/master/cmd/seqls 4. https://bintray.com/justinfx/utils/seqls/_latestVersion 5. https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2KpONX8cs9%3Df%2B4zzQx11yo1e7qV09bNTrT9zwudEfTRA%40mail.gmail.com?utm_medium=email&utm_source=footer 6. https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODiTWjOJ3jm2pH5RN-CKSqoAvpQ6kghL4ja752PMiXoQQ%40mail.gmail.com?utm_medium=email&utm_source=footer 7. https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1XKJy%2BWS4V5nPr53t8x%3D%3D2eTdULbvuzF_XGihM2Q1-GQ%40mail.gmail.com?utm_medium=email&utm_source=footer 8. https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCsf7uiqwUKWaO6q9QBrZiCxSP-rZLu4DbtxKyuY4qzHA%40mail.gmail.com?utm_medium=email&utm_source=footer 9. https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1zQ%2BNpmmqR2gWDe0vKEAd%2B%3DB47x2fkFL_y1JEErQOieQ%40mail.gmail.com?utm_medium=email&utm_source=footer 10. https://groups.google.com/d/msgid/python_inside_maya/1418732783.2884423.203482413.7E971BE9%40webmail.messagingengine.com?utm_medium=email&utm_source=footer 11. https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOA784b-Qspr1-VCCAVDESTuB1ZxTep95AfdBAv5V4Phaw%40mail.gmail.com?utm_medium=email&utm_source=footer 12. https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0maPnOO9OQ%2BDNQrQ3sX8cjUW-91i1sikQTs6D7PRWkEA%40mail.gmail.com?utm_medium=email&utm_source=footer -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/1418767983.738136.203723453.3A60AF73%40webmail.messagingengine.com. For more options, visit https://groups.google.com/d/optout.
