As yet another followup - rewrote the test in C++. I'm seeing the same results as the Python example. Which is good. Easier to debug this than Python glue code :)
From: Jonathan Tilden (2K) Sent: Monday, April 11, 2016 10:01 PM To: OpenImageIO developers <[email protected]> Subject: RE: [Oiio-dev] decrease performance when opening 1000s of files with python Looks like it is not the file type for me. TGA had the same issue. I just fired up the test on my macbook at home - everything seems to behave like I would expect (v1.6.11 via homebrew). Looks like it's an issue with my (possibly) janky Windows build (built from v1.5.14). If you (or anyone else) has a windows build handy, it would be good to know if you're getting similar results. It's not a blocking issue for us though - when things are slow, we just distribute it on the build machines :) Thanks for the pointers! -J From: Oiio-dev [mailto:[email protected]] On Behalf Of Larry Gritz Sent: Monday, April 11, 2016 8:02 PM To: OpenImageIO developers <[email protected]<mailto:[email protected]>> Subject: Re: [Oiio-dev] decrease performance when opening 1000s of files with python Have you gotten a chance to try this on a different file type? It's possible (I haven't audited the code yet) that the DDS reader has some kind of persistent state that is growing for each file it opens? This is all just a guess. I'll try to recreate on my end with a script, and I'll be curious to hear if you find anything looking at the cache statistics. On Apr 11, 2016, at 7:57 PM, Jonathan Tilden (2K) <[email protected]<mailto:[email protected]>> wrote: Yeah I thought so too - I wrote the test in a huge hurry on the way out the door. It's possible I did something very embarrassingly dumb in the test :) I intend to revisit in a bit here just to make sure... -----Original Message----- From: Larry Gritz [[email protected]<mailto:[email protected]>] Sent: Monday, April 11, 2016 10:48 PM Eastern Standard Time To: OpenImageIO developers Subject: Re: [Oiio-dev] decrease performance when opening 1000s of files with python So the cost of each additional one is proportional to the number or previous images opened? That's weird. > On Apr 11, 2016, at 6:51 PM, Jonathan Tilden (2K) > <[email protected]<mailto:[email protected]>> wrote: > >> My suspicion is that this is related to a growing number of files held open >> by the underlying ImageCache, which will not be immediately closed or free >> just because you clear() or even destroy the ImageBuf. Or possibly by the >> overhead of what happens when the maximum number of open files is reached? > > I was coming to a similar conclusion when I was trying to modify files in my > test bed, and the OS was complaining that I still had open filehandles. > >> The default is 100, I'm curious what happens if you just raise that. On >> Linux/OSX it should be safe to go into thousands, but Windows usually has >> some lower limit, to let's try 500 just to see what happens to your timings. >> In particular, does the slowdown seem to come later, i.e. after more files >> have been touched? > > I ran some more tests locally and, unfortunately, I actually didn't see any > improvement using the cache. Nor do I see much improvement using the > invalidate method you described. > > I've attached a script and some graphs if you wanted to take a look. The only > real requirement is that you populate a local directory with the same image > 2500x (if you can't get the attachments through here, let me know and I'll > email them direct). > > The interesting bit, is that I always get a linear-ish increase in time with > the imagebuf, but if I graph the time it takes to open an ImageInput and > query the spec, its pretty constant. > > Curious to know if you have any other thoughts! Thanks again! > -J > > > <oiiotest_imgbuf_perf.py><oiiotest_2500_cache.png><oiiotest_2500_cache_invalidate.png><oiiotest_2500_nocache.png>_______________________________________________ > Oiio-dev mailing list > [email protected]<mailto:[email protected]> > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org<http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> -- Larry Gritz [email protected]<mailto:[email protected]> _______________________________________________ Oiio-dev mailing list [email protected]<mailto:[email protected]> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org<http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> _______________________________________________ Oiio-dev mailing list [email protected]<mailto:[email protected]> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org<http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org> -- Larry Gritz [email protected]<mailto:[email protected]>
_______________________________________________ Oiio-dev mailing list [email protected] http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
