Wildish speculation on my part, I wouldn't be surprised if the go version of had a better equivalent implementation of the os.listdir() call giving you a lot of that speed up - esp if you're using NFS/CIFS - which would make the go version better-er even without implementing any swanky concurrency stuff. Specifically, i'm thinking about PEP-471. https://www.python.org/dev/peps/pep-0471/
Some notes that show a rewritten os.walk is going to be a tad faster (http://www.gossamer-threads.com/lists/python/dev/1153484) so 25x doesn't sound immediately improbable either. (Would be curious to see what the speed up is from python 2.7 vs python 3.5.. maybe I'll run a benchmark post Christmas for my own curiosity..) On Tue, Dec 16, 2014, at 09:32 PM, Justin Israel wrote: > Here are some benchmarks: > https://gist.github.com/justinfx/9c07ce03693f03c4bc46 > > I don't have a case on hand to reproduce where my co-worker was > getting a 25x speedup. I will have to see if I can find that. It was > more a case of "what were you getting before? Ok what are you getting > now?" :-) These were done on an SSD local drive, where as his tests > were across an NFS mounted filesystem. > > Also, the gofileseq library itself isn't yet using any concurrency for > its logic. Only the seqls tool uses concurrency for walking > directories, and processing lists of frames into sequences. I will > probably do another pass on gofileseq to get the FindSequencesOnDisk > function processing files in parallel. Haven't done much of any > profiling yet. > > > On Tue Dec 16 2014 at 9:18:04 PM Justin Israel > <[email protected]> wrote: >> I can work up some benchmarks against the python fileseq. >> >> On Tue, 16 Dec 2014 8:57 PM Marcus Ottosson >> <[email protected]> wrote: >>> Sounds good, no benchmarks? >>> >>> >>> On 15 December 2014 at 18:35, Justin Israel <[email protected]> >>> wrote: >>>> The Go version was getting about a 25x speedup when using >>>> FindSequencesOnDisk (which makes use of all of the other >>>> facilities) . The parsing and string building of the paths, frame >>>> ranges, and results are faster. I reuse buffers instead of >>>> constantly allocating new strings. Also the find operation is >>>> concurrent. >>>> An example was that my co-worker was using the python version to drive a graphical file sequence browser. He saw that massively apparent speedup in the interface when he switched to the Go version. >>>> >>>> On Tue, 16 Dec 2014 1:54 AM Marcus Ottosson >>>> <[email protected]> wrote: >>>>> I haven't used the Python library, but what would be the >>>>> advantage(s) of a Go version compared with the Python version? >>>>> >>>>> >>>>> On 13 December 2014 at 22:31, Justin Israel >>>>> <[email protected]> wrote: >>>>>> Hey all, >>>>>> >>>>>> This isn't directly python related, but I thought I would share >>>>>> anyways. If anyone has been using the python library fileseq[1], >>>>>> which I help maintain, to work with file sequence and frame range >>>>>> patterns, I've built a tool around this functionality (kind of >>>>>> similar to rvls). >>>>>> >>>>>> gofileseq[2] is a port of fileseq, to the Go language. Included >>>>>> in this repo is a utility called seqls[3], which uses gofileseq >>>>>> to provide a way to list files on the filesystem that are rolled >>>>>> up into sequence formats. So far it can do short and long >>>>>> listings, rolling up the file sizes and showing the latest mod >>>>>> time. I've cross compiled it for win/linux/osx[4] >>>>>> >>>>>> If you find it useful or have suggestions, please let me know! >>>>>> Also feel free to register issues for it on github if needed. I >>>>>> develop on osx and linux, so I have only briefly tested it on >>>>>> windows.. >>>>>> >>>>>> Enjoy! >>>>>> >>>>>> >>>>>> Justin >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. >>>>>> To view this discussion on the web visit https://groups.google.com/d/__msgid/python_inside_maya/__CAPGFgA2KpONX8cs9%3Df%__2B4zzQx11yo1e7qV09bNTrT9zwudEf__TRA%40mail.gmail.com[5]. >>>>>> For more options, visit https://groups.google.com/d/__optout. >>>>> >>>>> >>>>> >>>>> -- >>>>> *Marcus Ottosson* [email protected] >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/__msgid/python_inside_maya/__CAFRtmODiTWjOJ3jm2pH5RN-__CKSqoAvpQ6kghL4ja752PMiXoQQ%__40mail.gmail.com[6]. >>>>> For more options, visit https://groups.google.com/d/__optout. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/__msgid/python_inside_maya/__CAPGFgA1XKJy%2BWS4V5nPr53t8x%__3D%3D2eTdULbvuzF_XGihM2Q1-GQ%__40mail.gmail.com[7]. >>>> >>>> For more options, visit https://groups.google.com/d/__optout. >>>> >>> >>> >>> >>> -- >>> *Marcus Ottosson* [email protected] >>> >>> -- >>> You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. >>> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/__msgid/python_inside_maya/__CAFRtmOCsf7uiqwUKWaO6q9QBrZiCx__SP-rZLu4DbtxKyuY4qzHA%40mail.__gmail.com[8]. >>> For more options, visit https://groups.google.com/d/__optout. > > -- > You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1zQ%2BNpmmqR2gWDe0vKEAd%2B%3DB47x2fkFL_y1JEErQOieQ%40mail.gmail.com[9]. > For more options, visit https://groups.google.com/d/optout. Links: 1. https://pypi.python.org/pypi/Fileseq/0.5.1 2. https://github.com/justinfx/gofileseq 3. https://github.com/justinfx/gofileseq/tree/master/cmd/seqls 4. https://bintray.com/justinfx/utils/seqls/_latestVersion 5. https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2KpONX8cs9%3Df%2B4zzQx11yo1e7qV09bNTrT9zwudEfTRA%40mail.gmail.com?utm_medium=email&utm_source=footer 6. https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODiTWjOJ3jm2pH5RN-CKSqoAvpQ6kghL4ja752PMiXoQQ%40mail.gmail.com?utm_medium=email&utm_source=footer 7. https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1XKJy%2BWS4V5nPr53t8x%3D%3D2eTdULbvuzF_XGihM2Q1-GQ%40mail.gmail.com?utm_medium=email&utm_source=footer 8. https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCsf7uiqwUKWaO6q9QBrZiCxSP-rZLu4DbtxKyuY4qzHA%40mail.gmail.com?utm_medium=email&utm_source=footer 9. https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1zQ%2BNpmmqR2gWDe0vKEAd%2B%3DB47x2fkFL_y1JEErQOieQ%40mail.gmail.com?utm_medium=email&utm_source=footer -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/1418732783.2884423.203482413.7E971BE9%40webmail.messagingengine.com. For more options, visit https://groups.google.com/d/optout.
