Thanks Mr.Gritz
I think the problem is solved.
about the memory usage. where the memory performance i checked is in the memory 
monitor from Visual studio. it show me that the memory is truly increasing.
but when i checked memory in Task Manager from windows. it is OK!
i think it is a Visual studio's bug.
sorry for that. it is my mistake.T_T


and here is another thing. in my application i want to load EXR sequence files 
as soon as possible.
i trying to do this with multi-threading like the code i wrote u. when all the 
ImageBuf::read() function was done. the memory usage is very decent.
but the memory peak when loading is still a little bit high. especially i 
trying to load more files.


i trying to change the OIIO::attribute("exr_threads") to 1 or 3 then use 
multi-threading.  the memory peak is down but more slower as it will be.


so. how can i balance from the "speed" and "memory peak"? it will be great if 
you can give your suggestion!


Thanks again!
Kong.






??????????
15952002...@qq.com



 




------------------ Original ------------------
From:                                                                           
                                             "oiio-dev"                         
                                                           
<oiio-dev-requ...@lists.openimageio.org&gt;;
Date:&nbsp;Mon, May 16, 2022 03:44 AM
To:&nbsp;"oiio-dev"<oiio-dev@lists.openimageio.org&gt;;

Subject:&nbsp;Oiio-dev Digest, Vol 164, Issue 8



Send Oiio-dev mailing list submissions to
        oiio-dev@lists.openimageio.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
or, via email, send a message with subject or body 'help' to
        oiio-dev-requ...@lists.openimageio.org

You can reach the person managing the list at
        oiio-dev-ow...@lists.openimageio.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Oiio-dev digest..."


Today's Topics:

&nbsp;&nbsp; 1. Re: How to Load channel from EXR file with a little memory
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; usage. (Larry Gritz)


----------------------------------------------------------------------

Message: 1
Date: Sun, 15 May 2022 12:44:32 -0700
From: Larry Gritz <l...@larrygritz.com&gt;
To: OpenImageIO dev list <oiio-dev@lists.openimageio.org&gt;
Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
        little memory usage.
Message-ID: <9d247bfa-4988-486c-8978-b66ceae0f...@larrygritz.com&gt;
Content-Type: text/plain; charset="utf-8"

Here's what I've found so far.

First of all, the way you wrote the loop, when I tried it, the threads were 
spawned but the app terminated before they did much useful. I made more 
progress rewriting the loop slightly with a true join():

&nbsp;&nbsp;&nbsp; std::string base = "tmp/S25_";
&nbsp;&nbsp;&nbsp; std::vector<std::unique_ptr<std::thread&gt;&gt; threads;
&nbsp;&nbsp;&nbsp; for (int i = 10; i < 80; i++) {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; std::string file_allpath = 
Strutil::fmt::format("{}{:04}.exr", base, i);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; threads.emplace_back(new 
std::thread([=]() {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Strutil::print("creating {}: {}\n", i, file_allpath);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //just 
construction.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ImageBuf 
buf(file_allpath);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Strutil::print("reading {}\n", i);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // Do the 
read!!
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; buf.read(0, 
0, 15, 18, true, TypeDesc::UINT8);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }));
&nbsp;&nbsp;&nbsp; }
&nbsp;&nbsp;&nbsp; for (auto&amp; t : threads)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; t-&gt;join();

And I created a bunch of test files like this:

&nbsp;&nbsp;&nbsp; oiiotool --frames 10-79 --create 1920x1080 20 -d half -o 
tmp/S25_#.exr

I'm just guessing here, making an HD res frame with 20 'half' channels, just to 
simulate what happens.

Then I ran with some special debug environment variables set (you can try 
these, too, though my example is using bash, you may need to do it somewhat 
differently on Windows):

&nbsp;&nbsp;&nbsp; export OPENIMAGEIO_OPTIONS="debug=2"
&nbsp;&nbsp;&nbsp; export OPENIMAGEIO_IMAGECACHE_OPTIONS="statistics:level=1"
&nbsp;&nbsp;&nbsp; test_program

These variables will cause debug messages to print to the console whenever 
ImageBuf allocates or deallocates pixels. You will see messages like this:

&nbsp;&nbsp;&nbsp; OIIO DEBUG: IB allocated 5 MB, global IB memory now 403 MB
&nbsp;&nbsp;&nbsp; OIIO DEBUG: IB allocated 5 MB, global IB memory now 409 MB
&nbsp;&nbsp;&nbsp; ...

and also at the end, a statistics report from the ImageCache, where it might 
say something like:

&nbsp;&nbsp;&nbsp; Total pixel data size of all images referenced : 5.4 GB
&nbsp;&nbsp;&nbsp; Total actual file size of all images referenced : 5.6 MB
&nbsp;&nbsp;&nbsp; Pixel data read : 0 B
&nbsp;&nbsp;&nbsp; File I/O time : 0.9s (0.0s average per thread, for 70 
threads)
&nbsp;&nbsp;&nbsp; File open time only : 0.9s
&nbsp;&nbsp;&nbsp; Peak cache memory : 0 B

The files are opened (the ImageBuf uses ImageCache to read the file headers), 
but no pixels are read and no ImageCache tile cache space is used (because of 
that 'true' parameter we passed to force an immediate read rather than using 
the cache).

Now, if you comment out the read() call from the loop, so that it's just the 
ImageBuf construction, what happens is that NO messages will print about 
ImageBuf allocations.

This seems to confirm two things (at least, for me running top-of-master OIIO 
on a Mac):

1. ImageBuf construction alone opens the file and reads the header, but doesn't 
read pixels at all and doesn't allocate any pixel memory.
2. With the read, it definitely allocates memory, but the peak for all images 
is pretty manageable, with a peak of&nbsp; around 415MB. (But note: the 
resolution and channels of my test images may not be as much as yours.)

So I'm curious to hear your results if you replicate this experiment, and also 
the following information from you that might help diagnose the problem:

* Version of OIIO
* Full "oiiotool -info -v" of one of your files so I can see the resolution, 
channels, data type, if it's tiled, etc.
* Your peak memory as reported by those debug statistics.
* Peak memory if you run the loop SERIALIZED (that is, one at a time, not 
spawning threads at all).
* I'm also curious to know what happens with the thread spawn, but if you also 
comment out the ImageBuf construction -- that is, how much of the memory use do 
you think you are seeing is inherent to running that number of threads, and 
doesn't have anything to do with ImageBuf at all?




&gt; On May 15, 2022, at 9:18 AM, Larry Gritz <l...@larrygritz.com&gt; wrote:
&gt; 
&gt; Oh, interesting. So you are making all those ImageBuf's *simultaneously*. 
But on the other hand, it sure looks like it shouldn't consume much memory 
since you are never accessing the pixels. 
&gt; 
&gt; I will try it on my end and see what happens.
&gt; 
&gt; Can you answer a few questions for me?
&gt; 
&gt; 1. What version of OIIO are you using?
&gt; 
&gt; 2. What is the size of the exr files you're loading? For example, do
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; oiiotool -info -v S25_00.10.exr
&gt;&nbsp;&nbsp;&nbsp;&nbsp; and show us what it says. (I'm looking especially 
at resolution, channels, data type.)
&gt; 
&gt; 3. If you switch to sequential access, what happens to the memory 
consumption? In other words, commend out the thread launch, so you only have
&gt; 
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for (int i = 10; i < 80; 
i++) {
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 std::string file_allpath = base + std::to_string(i) + std::string(".exr");
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 //just construction.
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 ImageBuf buf(file_allpath);
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
&gt; 
&gt; Meanwhile, I will investigate on my end and see if I can get any insight.
&gt; 
&gt; 
&gt;&gt; On May 15, 2022, at 8:25 AM, ????? <15952002...@qq.com 
<mailto:15952002...@qq.com&gt;&gt; wrote:
&gt;&gt; 
&gt;&gt; 
&gt;&gt; it looks like this Email can't hold image.
&gt;&gt; 
&gt;&gt; here is my simple test code:
&gt;&gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //exr sequence from 
S25_0010.exr to S25_0079.exr
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; std::string base = 
"C:/Users/Lo_Fnatic/Desktop/nima/S25_00";
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for (int i = 10; i < 
80; i++) {
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 std::string file_allpath = base + std::to_string(i) + std::string(".exr");
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 std::thread t([=]() {
&gt;&gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 //just construction.
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 ImageBuf buf(file_allpath);
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 //buf.read(0, 0, 15, 18, true, TypeDesc::UINT8);
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 });
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 t.detach();
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
&gt;&gt; 
&gt;&gt; it will cost almost 32G memory even though i am not ready to read 
these pixel data.
&gt;&gt; thanks
&gt;&gt; 
&gt;&gt; 
&gt;&gt; ------------------ Original ------------------
&gt;&gt; From: "oiio-dev" <oiio-dev-requ...@lists.openimageio.org 
<mailto:oiio-dev-requ...@lists.openimageio.org&gt;&gt;;
&gt;&gt; Date: Sun, May 15, 2022 02:15 AM
&gt;&gt; To: "oiio-dev"<oiio-dev@lists.openimageio.org 
<mailto:oiio-dev@lists.openimageio.org&gt;&gt;;
&gt;&gt; Subject: Oiio-dev Digest, Vol 164, Issue 4
&gt;&gt; 
&gt;&gt; Send Oiio-dev mailing list submissions to
&gt;&gt; oiio-dev@lists.openimageio.org 
<mailto:oiio-dev@lists.openimageio.org&gt;
&gt;&gt; 
&gt;&gt; To subscribe or unsubscribe via the World Wide Web, visit
&gt;&gt; http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org 
<http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org&gt;
&gt;&gt; or, via email, send a message with subject or body 'help' to
&gt;&gt; oiio-dev-requ...@lists.openimageio.org 
<mailto:oiio-dev-requ...@lists.openimageio.org&gt;
&gt;&gt; 
&gt;&gt; You can reach the person managing the list at
&gt;&gt; oiio-dev-ow...@lists.openimageio.org 
<mailto:oiio-dev-ow...@lists.openimageio.org&gt;
&gt;&gt; 
&gt;&gt; When replying, please edit your Subject line so it is more specific
&gt;&gt; than "Re: Contents of Oiio-dev digest..."
&gt;&gt; 
&gt;&gt; 
&gt;&gt; Today's Topics:
&gt;&gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp; 1. How to Load channel from EXR file with a little 
memory usage.
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (=?gb18030?B?qICogKiAqIC5zg==?=)
&gt;&gt;&nbsp;&nbsp;&nbsp; 2. Re: How to Load channel from EXR file with a 
little memory
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; usage. (Larry Gritz)
&gt;&gt;&nbsp;&nbsp;&nbsp; 3. Re: How to Load channel from EXR file with a 
little memory
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; usage. (Larry Gritz)
&gt;&gt; 
&gt;&gt; 
&gt;&gt; ----------------------------------------------------------------------
&gt;&gt; 
&gt;&gt; Message: 1
&gt;&gt; Date: Sat, 14 May 2022 23:28:29 +0800
&gt;&gt; From: "=?gb18030?B?qICogKiAqIC5zg==?=" <15952002...@qq.com 
<mailto:15952002...@qq.com&gt;&gt;
&gt;&gt; To: "=?gb18030?B?b2lpby1kZXY=?=" <oiio-dev@lists.openimageio.org 
<mailto:oiio-dev@lists.openimageio.org&gt;&gt;
&gt;&gt; Subject: [Oiio-dev] How to Load channel from EXR file with a little
&gt;&gt; memory usage.
&gt;&gt; Message-ID: <tencent_77ef28dd6d2d2359af2ccc5ca8ce66b74...@qq.com 
<mailto:tencent_77ef28dd6d2d2359af2ccc5ca8ce66b74...@qq.com&gt;&gt;
&gt;&gt; Content-Type: text/plain; charset="gb18030"
&gt;&gt; 
&gt;&gt; Hello.
&gt;&gt; I'm trying to load OpenEXR sequence to memory then draw them with 
Opengl.
&gt;&gt; but these EXR files are so big with many sub-channels in(one file with 
300M and more than 30 channels).
&gt;&gt; &amp;nbsp;i just want to load the "Beauty" pass.
&gt;&gt; 
&gt;&gt; 
&gt;&gt; here is what i was tried:
&gt;&gt; 
&gt;&gt; 
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; ImageBuf buf = 
ImageBuf::ImageBuf(exr_filepath);
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; const ImageSpec&amp;amp; 
spec = buf.spec();
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; int xres = spec.width;
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; int yres = spec.height;
&gt;&gt; 
&gt;&gt; 
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; //&amp;nbsp; &amp;nbsp;i 
just want to load the RGB data from the "Beauty" pass
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; std::vector<UINT8&amp;gt; 
pixels(xres * yres * 3);
&gt;&gt; 
&gt;&gt; 
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; //&amp;nbsp; &amp;nbsp;for 
example the Beauty pass start with index of 15
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; ROI roi(0, xres, 0, yres, 
0, 1, /*chans:*/ 15, 18);
&gt;&gt; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; buf.get_pixels(roi, 
TypeDesc::UINT8, &amp;amp;pixels[0]);
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; it works. but cost a huge amount of memory use with 
multi-threading(sometimes more than 80G) during loading all files.
&gt;&gt; here is my quesion:
&gt;&gt; like what i did. is this way to load all data together then save the 
"Beauty data" from them to the std::vector?
&gt;&gt; or just load the "Beauty" pass without cache all exr data first?
&gt;&gt; is this memory usage normal?
&gt;&gt; 
&gt;&gt; 
&gt;&gt; and i have checked the document. it looks like the ImageCache class is 
what i want. can i use this class in my situation?
&gt;&gt; is there anywhere i can find more example about this class?
&gt;&gt; 
&gt;&gt; 
&gt;&gt; thanks!
&gt;&gt; kong
&gt;&gt; -------------- next part --------------
&gt;&gt; An HTML attachment was scrubbed...
&gt;&gt; URL: 
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/a7118f41/attachment-0001.html
 
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/a7118f41/attachment-0001.html&gt;&gt;
&gt;&gt; 
&gt;&gt; ------------------------------
&gt;&gt; 
&gt;&gt; Message: 2
&gt;&gt; Date: Sat, 14 May 2022 10:59:36 -0700
&gt;&gt; From: Larry Gritz <l...@larrygritz.com 
<mailto:l...@larrygritz.com&gt;&gt;
&gt;&gt; To: OpenImageIO dev list <oiio-dev@lists.openimageio.org 
<mailto:oiio-dev@lists.openimageio.org&gt;&gt;
&gt;&gt; Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
&gt;&gt; little memory usage.
&gt;&gt; Message-ID: <68429dd3-80ae-444c-b06b-70dfed031...@larrygritz.com 
<mailto:68429dd3-80ae-444c-b06b-70dfed031...@larrygritz.com&gt;&gt;
&gt;&gt; Content-Type: text/plain; charset="utf-8"
&gt;&gt; 
&gt;&gt; I think that the problem here is that the ImageBuf will read in all 
the channels, even though the get_pixels only copies out the 3 channels you 
need. The I/O itself happens "lazily", triggered by the get_pixels call itself 
needing pixel values, but at that point, it still doesn't know that you won't 
subsequently try to access other channels in the ImageBuf, so the channel 
restriction of get_pixels doesn't turn into a channel restriction for how the 
file is read and stored.
&gt;&gt; 
&gt;&gt; Additionally, the ImageBuf will by default store the pixels in 
whatever the "widest" data type of the file itself has (in the case of an 
OpenEXR, that would mean half at best, and maybe float depending on what's in 
your file), even though ultimately you only want UINT8 data in your program. 
(And if the file is large, there's also probably yet another copy, or partial 
copy, in the underlying ImageCache.)
&gt;&gt; 
&gt;&gt; But I think you can fix all of these issues by doing an explicit 
ImageBuf::read() before the first time you access the pixels, using the version 
of read() that lets you select the channel range and data type:
&gt;&gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; buf.read(/*subimage*/ 0, /*miplevel*/ 0,
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 /*chbegin*/ 15, /*chend*/ 18,
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 /*forceread*/ true, /*convert*/ TypeDesc::UINT8);
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; // ... then ...
&gt;&gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; // BEWARE! now channels 15... of the file are 
stored in 0... of the buf!
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 0, 
4);
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; buf.get_pixels(roi, TypeDesc::UINT8, 
&amp;pixels[0]);
&gt;&gt; 
&gt;&gt; Now the buf will internally store only 3 channels of UINT8.
&gt;&gt; 
&gt;&gt; See where I passed true for the "forceread" parameter? That forces it 
to read the image right then, skipping use of the ImageCache and storing the 
entire image in memory (well, the channels you asked for, I mean). So the 
ImageBuf itself holds the channels you asked for, without an extra copy in the 
ImageCache. The ImageCache is usually very helpful, especially if you only need 
a small portion of a large image at any one time by one thread. The one case 
where it's wasteful is if you need an entire image all at once, and it can fit 
into memory so there is no benefit to reading it piece by piece only as needed.
&gt;&gt; 
&gt;&gt; And given that we have the whole image loaded into the ImageBuf 
without any ImageCache backing, it's possible that you can do one more better 
thing, depending on how you use the data. With the forced read, the ImageBuf 
itself holds channels 15,16,17 as UINT8, contiguously in memory starting at 
address localpixels(). So do you really need the pixels to be copied (and 
stored again) in your std::vector? Or can you directly use the ImageBuf's 
internal storage of the pixels?
&gt;&gt; 
&gt;&gt; You can test if an ImageBuf's pixels are all in memory:
&gt;&gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; if (buf.storage() == ImageBuf::LOCALBUFFER)
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; address = (const 
uint8*) buf.localpixels();
&gt;&gt; 
&gt;&gt; Actually, I think localpixels() will return nullptr if it doesn't use 
local storage, so you don't need the separate check of storage(), but I still 
wanted to point that out.
&gt;&gt; 
&gt;&gt; (I think everything I have written is true. Please double check it.)
&gt;&gt; 
&gt;&gt; 
&gt;&gt; &gt; On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com 
<mailto:15952002...@qq.com&gt;&gt; wrote:
&gt;&gt; &gt; 
&gt;&gt; &gt; Hello.
&gt;&gt; &gt; I'm trying to load OpenEXR sequence to memory then draw them with 
Opengl.
&gt;&gt; &gt; but these EXR files are so big with many sub-channels in(one file 
with 300M and more than 30 channels).
&gt;&gt; &gt;&nbsp; i just want to load the "Beauty" pass.
&gt;&gt; &gt; 
&gt;&gt; &gt; here is what i was tried:
&gt;&gt; &gt; 
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ImageBuf buf = 
ImageBuf::ImageBuf(exr_filepath);
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; const 
ImageSpec&amp; spec = buf.spec();
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int xres = 
spec.width;
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int yres = 
spec.height;
&gt;&gt; &gt; 
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp; i 
just want to load the RGB data from the "Beauty" pass
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
std::vector<UINT8&gt; pixels(xres * yres * 3);
&gt;&gt; &gt; 
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; //&nbsp;&nbsp; 
for example the Beauty pass start with index of 15
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ROI roi(0, xres, 
0, yres, 0, 1, /*chans:*/ 15, 18);
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
buf.get_pixels(roi, TypeDesc::UINT8, &amp;pixels[0]);
&gt;&gt; &gt; 
&gt;&gt; &gt; it works. but cost a huge amount of memory use with 
multi-threading(sometimes more than 80G) during loading all files.
&gt;&gt; &gt; here is my quesion:
&gt;&gt; &gt; like what i did. is this way to load all data together then save 
the "Beauty data" from them to the std::vector?
&gt;&gt; &gt; or just load the "Beauty" pass without cache all exr data first?
&gt;&gt; &gt; is this memory usage normal?
&gt;&gt; &gt; 
&gt;&gt; &gt; and i have checked the document. it looks like the ImageCache 
class is what i want. can i use this class in my situation?
&gt;&gt; &gt; is there anywhere i can find more example about this class?
&gt;&gt; &gt; 
&gt;&gt; &gt; thanks!
&gt;&gt; &gt; kong
&gt;&gt; &gt; 
&gt;&gt; &gt; _______________________________________________
&gt;&gt; &gt; Oiio-dev mailing list
&gt;&gt; &gt; Oiio-dev@lists.openimageio.org 
<mailto:Oiio-dev@lists.openimageio.org&gt;
&gt;&gt; &gt; 
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org 
<http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org&gt;
&gt;&gt; 
&gt;&gt; --
&gt;&gt; Larry Gritz
&gt;&gt; l...@larrygritz.com <mailto:l...@larrygritz.com&gt;
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; -------------- next part --------------
&gt;&gt; An HTML attachment was scrubbed...
&gt;&gt; URL: 
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/80e79745/attachment-0001.html&gt;
&gt;&gt; 
&gt;&gt; ------------------------------
&gt;&gt; 
&gt;&gt; Message: 3
&gt;&gt; Date: Sat, 14 May 2022 11:15:51 -0700
&gt;&gt; From: Larry Gritz <l...@larrygritz.com&gt;
&gt;&gt; To: OpenImageIO dev list <oiio-dev@lists.openimageio.org&gt;
&gt;&gt; Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
&gt;&gt; little memory usage.
&gt;&gt; Message-ID: <b12c7b0e-e77c-497e-a7d2-669b73d1c...@larrygritz.com&gt;
&gt;&gt; Content-Type: text/plain; charset="utf-8"
&gt;&gt; 
&gt;&gt; By the way, for those following along, if you're wondering how to make 
oiiotool take these very same shortcuts:
&gt;&gt; 
&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp; oiiotool -i:ch=R,G,B:type=uint8:now=1 foo.exr 
...
&gt;&gt; 
&gt;&gt; 
&gt;&gt; &gt; On May 14, 2022, at 10:59 AM, Larry Gritz 
<l...@larrygritz.com&gt; wrote:
&gt;&gt; &gt; 
&gt;&gt; &gt; I think that the problem here is that the ImageBuf will read in 
all the channels, even though the get_pixels only copies out the 3 channels you 
need. The I/O itself happens "lazily", triggered by the get_pixels call itself 
needing pixel values, but at that point, it still doesn't know that you won't 
subsequently try to access other channels in the ImageBuf, so the channel 
restriction of get_pixels doesn't turn into a channel restriction for how the 
file is read and stored.
&gt;&gt; &gt; 
&gt;&gt; &gt; Additionally, the ImageBuf will by default store the pixels in 
whatever the "widest" data type of the file itself has (in the case of an 
OpenEXR, that would mean half at best, and maybe float depending on what's in 
your file), even though ultimately you only want UINT8 data in your program. 
(And if the file is large, there's also probably yet another copy, or partial 
copy, in the underlying ImageCache.)
&gt;&gt; &gt; 
&gt;&gt; &gt; But I think you can fix all of these issues by doing an explicit 
ImageBuf::read() before the first time you access the pixels, using the version 
of read() that lets you select the channel range and data type:
&gt;&gt; &gt; 
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; buf.read(/*subimage*/ 0, /*miplevel*/ 0,
&gt;&gt; 
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 /*chbegin*/ 15, /*chend*/ 18,
&gt;&gt; 
&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
 /*forceread*/ true, /*convert*/ TypeDesc::UINT8);
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; // ... then ...
&gt;&gt; &gt; 
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; // BEWARE! now channels 15... of the file 
are stored in 0... of the buf!
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; ROI roi(0, xres, 0, yres, 0, 1, 
/*chans:*/ 0, 4);
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; buf.get_pixels(roi, TypeDesc::UINT8, 
&amp;pixels[0]);
&gt;&gt; &gt; 
&gt;&gt; &gt; Now the buf will internally store only 3 channels of UINT8.
&gt;&gt; &gt; 
&gt;&gt; &gt; See where I passed true for the "forceread" parameter? That 
forces it to read the image right then, skipping use of the ImageCache and 
storing the entire image in memory (well, the channels you asked for, I mean). 
So the ImageBuf itself holds the channels you asked for, without an extra copy 
in the ImageCache. The ImageCache is usually very helpful, especially if you 
only need a small portion of a large image at any one time by one thread. The 
one case where it's wasteful is if you need an entire image all at once, and it 
can fit into memory so there is no benefit to reading it piece by piece only as 
needed.
&gt;&gt; &gt; 
&gt;&gt; &gt; And given that we have the whole image loaded into the ImageBuf 
without any ImageCache backing, it's possible that you can do one more better 
thing, depending on how you use the data. With the forced read, the ImageBuf 
itself holds channels 15,16,17 as UINT8, contiguously in memory starting at 
address localpixels(). So do you really need the pixels to be copied (and 
stored again) in your std::vector? Or can you directly use the ImageBuf's 
internal storage of the pixels?
&gt;&gt; &gt; 
&gt;&gt; &gt; You can test if an ImageBuf's pixels are all in memory:
&gt;&gt; &gt; 
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp; if (buf.storage() == 
ImageBuf::LOCALBUFFER)
&gt;&gt; &gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; address = (const 
uint8*) buf.localpixels();
&gt;&gt; &gt; 
&gt;&gt; &gt; Actually, I think localpixels() will return nullptr if it doesn't 
use local storage, so you don't need the separate check of storage(), but I 
still wanted to point that out.
&gt;&gt; &gt; 
&gt;&gt; &gt; (I think everything I have written is true. Please double check 
it.)
&gt;&gt; &gt; 
&gt;&gt; &gt; 
&gt;&gt; &gt;&gt; On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com 
<mailto:15952002...@qq.com&gt;&gt; wrote:
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt; Hello.
&gt;&gt; &gt;&gt; I'm trying to load OpenEXR sequence to memory then draw them 
with Opengl.
&gt;&gt; &gt;&gt; but these EXR files are so big with many sub-channels in(one 
file with 300M and more than 30 channels).
&gt;&gt; &gt;&gt;&nbsp; i just want to load the "Beauty" pass.
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt; here is what i was tried:
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ImageBuf buf 
= ImageBuf::ImageBuf(exr_filepath);
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; const 
ImageSpec&amp; spec = buf.spec();
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int xres = 
spec.width;
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int yres = 
spec.height;
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
//&nbsp;&nbsp; i just want to load the RGB data from the "Beauty" pass
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
std::vector<UINT8&gt; pixels(xres * yres * 3);
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
//&nbsp;&nbsp; for example the Beauty pass start with index of 15
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ROI roi(0, 
xres, 0, yres, 0, 1, /*chans:*/ 15, 18);
&gt;&gt; &gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
buf.get_pixels(roi, TypeDesc::UINT8, &amp;pixels[0]);
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt; it works. but cost a huge amount of memory use with 
multi-threading(sometimes more than 80G) during loading all files.
&gt;&gt; &gt;&gt; here is my quesion:
&gt;&gt; &gt;&gt; like what i did. is this way to load all data together then 
save the "Beauty data" from them to the std::vector?
&gt;&gt; &gt;&gt; or just load the "Beauty" pass without cache all exr data 
first?
&gt;&gt; &gt;&gt; is this memory usage normal?
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt; and i have checked the document. it looks like the ImageCache 
class is what i want. can i use this class in my situation?
&gt;&gt; &gt;&gt; is there anywhere i can find more example about this class?
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt; thanks!
&gt;&gt; &gt;&gt; kong
&gt;&gt; &gt;&gt; 
&gt;&gt; &gt;&gt; _______________________________________________
&gt;&gt; &gt;&gt; Oiio-dev mailing list
&gt;&gt; &gt;&gt; Oiio-dev@lists.openimageio.org 
<mailto:Oiio-dev@lists.openimageio.org&gt;
&gt;&gt; &gt;&gt; 
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
&gt;&gt; &gt; 
&gt;&gt; &gt; --
&gt;&gt; &gt; Larry Gritz
&gt;&gt; &gt; l...@larrygritz.com <mailto:l...@larrygritz.com&gt;
&gt;&gt; &gt; 
&gt;&gt; &gt; 
&gt;&gt; &gt; 
&gt;&gt; &gt; 
&gt;&gt; &gt; 
&gt;&gt; &gt; _______________________________________________
&gt;&gt; &gt; Oiio-dev mailing list
&gt;&gt; &gt; Oiio-dev@lists.openimageio.org
&gt;&gt; &gt; http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
&gt;&gt; 
&gt;&gt; --
&gt;&gt; Larry Gritz
&gt;&gt; l...@larrygritz.com
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; 
&gt;&gt; -------------- next part --------------
&gt;&gt; An HTML attachment was scrubbed...
&gt;&gt; URL: 
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/dce8c2a6/attachment.html&gt;
&gt;&gt; 
&gt;&gt; ------------------------------
&gt;&gt; 
&gt;&gt; Subject: Digest Footer
&gt;&gt; 
&gt;&gt; _______________________________________________
&gt;&gt; Oiio-dev mailing list
&gt;&gt; Oiio-dev@lists.openimageio.org
&gt;&gt; http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
&gt;&gt; 
&gt;&gt; 
&gt;&gt; ------------------------------
&gt;&gt; 
&gt;&gt; End of Oiio-dev Digest, Vol 164, Issue 4
&gt;&gt; ****************************************
&gt;&gt; _______________________________________________
&gt;&gt; Oiio-dev mailing list
&gt;&gt; Oiio-dev@lists.openimageio.org 
<mailto:Oiio-dev@lists.openimageio.org&gt;
&gt;&gt; http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
&gt; 
&gt; --
&gt; Larry Gritz
&gt; l...@larrygritz.com <mailto:l...@larrygritz.com&gt;
&gt; 
&gt; 
&gt; 
&gt; 
&gt; 
&gt; _______________________________________________
&gt; Oiio-dev mailing list
&gt; Oiio-dev@lists.openimageio.org
&gt; http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org

--
Larry Gritz
l...@larrygritz.com





-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220515/70a65d85/attachment.html&gt;

------------------------------

Subject: Digest Footer

_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org


------------------------------

End of Oiio-dev Digest, Vol 164, Issue 8
****************************************
_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org

Reply via email to