IIRC from last time I tested this (please don't trust my memory 100%), it was a 
substantial per-thread overhead issue. 

I have not combed through the IlmImf code to verify this, but what it smelled 
like to me is if each thread in the pool has a retained buffer that it uses as 
scratch space for the decoding/uncompresion, data format conversions, and so 
on. (You know, you have to read the compressed data into some buffer, and then 
perform the decompression into another buffer... that sort of thing.) I get the 
feeling that each thread in the pool has its own area for this, and it's 
probably quite large so that it can do these decode/conversions on a big batch 
of pixels at once.

Soren, I think it would be worth repeating your experiment with the renderer 
and OpenEXR set to use just one thread (or maybe 1, 2, 4, ...). Obviously, it's 
not quite apples-to-apples because the access pattern will be totally different 
as you change thread counts. But if you see this "unaccounted overhead" in the 
process size growing steadily with the number of threads (while the number of 
textures you access and the maximum open at once stays the same), then that 
tells you something.

Do you know how many threads you were using?

Remember, also, that Soren is not comparing ImageCache to *nothing*, he's 
comparing ImageCache with exr files to the same ImageCache with tiff files, 
rendering the same frame with the same texture access patterns. This really is 
a direct test of speed and overhead of the two file format libraries.


> On Sep 16, 2016, at 10:12 AM, Søren Ragsdale <sor...@gmail.com> wrote:
> 
> Thanks, Larry! My EXR textures are already mipmapped and tiled so I don't 
> think we're using any extra memory on auto-tile, but it certainly was true 
> that we were using OIIO with the default 100 file handle cache size. We'll 
> give it a go with a much higher value.
> 
> I'm still surprised that libIlmImf requires that much more memory. If our 
> handle cache size was limited to 100 files at a time, that's an extra 38.75 
> MB per texture. Even if it's caching all 1957 texture handles at once that's 
> still an extra 1.98 MB per texture.
> 
> On Fri, Sep 16, 2016 at 5:56 PM, Larry Gritz <l...@larrygritz.com 
> <mailto:l...@larrygritz.com>> wrote:
> See how there were 1957 unique textures, but you created 144168 ImageInputs? 
> That means your file (handle) cache was too small, you were accessing the 
> files in an incoherent order and the "working set" was much larger than the 
> file cache size (100), so you ended up closing and reopening files a massive 
> number of times (73 times each, on average).
> 
> The solution to this is to raise the TextureSystem's (that is, the underlying 
> ImageCache's) "max_open_files" significantly. The default of 100 is extremely 
> conservative -- trying to keep you out of trouble on operating systems with a 
> low handle count per process, or with ImageCache's file operations being 
> potentially only a small portion of your process's file needs.
> 
> On a modern Linux system, you should be able to have several thousand files 
> open simultaneously. I would do that and see what it does to all your 
> figures. At the very least, it should substantially lower the "file open 
> time" and maybe the file I/O time overall.
> 
> But, it may make the memory issues worse -- if libIlmImf is internally 
> holding more memory per open file than libtiff does (my tests indicate this 
> as well), then having more files open at a time may exacerbate the problem. I 
> haven't tested this for a while, though, but I think that Karl is correct 
> that part of this is that each of the threads in OpenEXR's pool is holding a 
> bit of working memory, and that adds up, so the higher your thread count, the 
> more memory use overhead there will be for libIlmImf, and it's very hard for 
> the app to account for that or control it.
> 
> It is simply a fact that on a per-thread, per-file basis, libIlmImf uses more 
> overhead memory per open file than libtiff does. I have not looked into it 
> deeply enough to know if that's a good design tradeoff or not (for example, 
> maybe that memory is put to good use, and helps speed up some operations).
> 
> This is the kind of thing you would never notice reading OpenEXR files 
> sequentially, but in something like a texture system where you may have 
> thousands of files open simultaneously, an extra 256KB of overhead per file 
> adds up fast.
> 
> 
> > On Sep 16, 2016, at 3:34 AM, Søren Ragsdale <sor...@gmail.com 
> > <mailto:sor...@gmail.com>> wrote:
> >
> > Hello, OpenEXR devs. I've been doing some comparative rendering tests I've 
> > found something a bit surprising.
> >
> > TIFF and EXR texture access *times* seems more or less the same, which is 
> > fine because the underlying data is equivalent. (Same data type, 
> > compression, tile size, etc.) But the RAM overhead seems much higher for 
> > EXRs. We've got a 9GB render using TIFFs and a 13GB render using EXRs.
> >
> > Does anyone have some theories why EXR texture access is requiring 4GB more 
> > memory?
> >
> >
> > Prman-20.11, OSL shaders, OIIO/TIFF textures:
> > real 00:21:46
> > VmRSS 9,063.45 MB
> > OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
> >   Options:  max_memory_MB=4000.0 max_open_files=100 autotile=64
> >             autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
> >             accept_unmipped=1 read_before_insert=0 deduplicate=1
> >             unassociatedalpha=0 failure_retries=0
> >   Images : 1957 unique
> >     ImageInputs : 136432 created, 100 current, 796 peak
> >     Total size of all images referenced : 166.0 GB
> >     Read from disk : 55.5 GB
> >     File I/O time : 7h 2m 33.9s (16m 54.2s average per thread)
> >     File open time only : 27m 44.0s
> >
> >
> > Prman-20.11, OSL shaders, OIIO/EXR textures:
> > real 00:21:14
> > VmRSS 12,938.83 MB
> > OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
> >   Options:  max_memory_MB=4000.0 max_open_files=100 autotile=64
> >             autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
> >             accept_unmipped=1 read_before_insert=0 deduplicate=1
> >             unassociatedalpha=0 failure_retries=0
> >   Images : 1957 unique
> >     ImageInputs : 133168 created, 100 current, 771 peak
> >     Total size of all images referenced : 166.0 GB
> >     Read from disk : 55.5 GB
> >     File I/O time : 6h 15m 42.1s (15m 1.7s average per thread)
> >     File open time only : 1m 22.5s
> >
> > _______________________________________________
> > Openexr-devel mailing list
> > Openexr-devel@nongnu.org <mailto:Openexr-devel@nongnu.org>
> > https://lists.nongnu.org/mailman/listinfo/openexr-devel 
> > <https://lists.nongnu.org/mailman/listinfo/openexr-devel>
> 
> --
> Larry Gritz
> l...@larrygritz.com <mailto:l...@larrygritz.com>
> 
> 
> 
> _______________________________________________
> Openexr-devel mailing list
> Openexr-devel@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/openexr-devel

--
Larry Gritz
l...@larrygritz.com


_______________________________________________
Openexr-devel mailing list
Openexr-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/openexr-devel

Reply via email to