OK, I'm going to guess, then, that it's highly likely to either be (a) something that was fixed between 1.5.14 and 1.6.11, or (b) something that is specific to Windows.
I would recommend trying to upgrade to 1.6 on your Windows boxes. Fixing this would not be the only advantage of doing so. If you continue to have trouble on a modern 1.6, I'm happy to dig into it further. Also, if you haven't done so, those cache statistics and the invalidation trick still may help (though, I'm less sure of the Python support for either of those in 1.5, IIRC 1.6 included some big Python overhauls). > On Apr 11, 2016, at 10:00 PM, Jonathan Tilden (2K) <[email protected]> > wrote: > > Looks like it is not the file type for me. TGA had the same issue. > > I just fired up the test on my macbook at home – everything seems to behave > like I would expect (v1.6.11 via homebrew). > > Looks like it’s an issue with my (possibly) janky Windows build (built from > v1.5.14). If you (or anyone else) has a windows build handy, it would be good > to know if you’re getting similar results. > > It’s not a blocking issue for us though – when things are slow, we just > distribute it on the build machines :) > > Thanks for the pointers! > -J > > From: Oiio-dev [mailto:[email protected]] On Behalf Of > Larry Gritz > Sent: Monday, April 11, 2016 8:02 PM > To: OpenImageIO developers <[email protected]> > Subject: Re: [Oiio-dev] decrease performance when opening 1000s of files with > python > > Have you gotten a chance to try this on a different file type? It's possible > (I haven't audited the code yet) that the DDS reader has some kind of > persistent state that is growing for each file it opens? > > This is all just a guess. I'll try to recreate on my end with a script, and > I'll be curious to hear if you find anything looking at the cache statistics. > > > > On Apr 11, 2016, at 7:57 PM, Jonathan Tilden (2K) <[email protected] > <mailto:[email protected]>> wrote: > > Yeah I thought so too - I wrote the test in a huge hurry on the way out the > door. It's possible I did something very embarrassingly dumb in the test :) > > I intend to revisit in a bit here just to make sure... > > > > -----Original Message----- > From: Larry Gritz [[email protected] <mailto:[email protected]>] > Sent: Monday, April 11, 2016 10:48 PM Eastern Standard Time > To: OpenImageIO developers > Subject: Re: [Oiio-dev] decrease performance when opening 1000s of files with > python > > So the cost of each additional one is proportional to the number or previous > images opened? That's weird. > > > > On Apr 11, 2016, at 6:51 PM, Jonathan Tilden (2K) <[email protected] > > <mailto:[email protected]>> wrote: > > > >> My suspicion is that this is related to a growing number of files held > >> open by the underlying ImageCache, which will not be immediately closed or > >> free just because you clear() or even destroy the ImageBuf. Or possibly by > >> the overhead of what happens when the maximum number of open files is > >> reached? > > > > I was coming to a similar conclusion when I was trying to modify files in > > my test bed, and the OS was complaining that I still had open filehandles. > > > >> The default is 100, I'm curious what happens if you just raise that. On > >> Linux/OSX it should be safe to go into thousands, but Windows usually has > >> some lower limit, to let's try 500 just to see what happens to your > >> timings. In particular, does the slowdown seem to come later, i.e. after > >> more files have been touched? > > > > I ran some more tests locally and, unfortunately, I actually didn't see any > > improvement using the cache. Nor do I see much improvement using the > > invalidate method you described. > > > > I've attached a script and some graphs if you wanted to take a look. The > > only real requirement is that you populate a local directory with the same > > image 2500x (if you can't get the attachments through here, let me know and > > I'll email them direct). > > > > The interesting bit, is that I always get a linear-ish increase in time > > with the imagebuf, but if I graph the time it takes to open an ImageInput > > and query the spec, its pretty constant. > > > > Curious to know if you have any other thoughts! Thanks again! > > -J > > > > > > <oiiotest_imgbuf_perf.py><oiiotest_2500_cache.png><oiiotest_2500_cache_invalidate.png><oiiotest_2500_nocache.png>_______________________________________________ > > Oiio-dev mailing list > > [email protected] <mailto:[email protected]> > > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > > <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> > > -- > Larry Gritz > [email protected] <mailto:[email protected]> > > > _______________________________________________ > Oiio-dev mailing list > [email protected] <mailto:[email protected]> > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> > _______________________________________________ > Oiio-dev mailing list > [email protected] <mailto:[email protected]> > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org > <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> > > -- > Larry Gritz > [email protected] <mailto:[email protected]> > > _______________________________________________ > Oiio-dev mailing list > [email protected] > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org -- Larry Gritz [email protected]
_______________________________________________ Oiio-dev mailing list [email protected] http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
