Here's what I've found so far.
First of all, the way you wrote the loop, when I tried it, the threads were
spawned but the app terminated before they did much useful. I made more
progress rewriting the loop slightly with a true join():
std::string base = "tmp/S25_";
std::vector<std::unique_ptr<std::thread>> threads;
for (int i = 10; i < 80; i++) {
std::string file_allpath = Strutil::fmt::format("{}{:04}.exr", base, i);
threads.emplace_back(new std::thread([=]() {
Strutil::print("creating {}: {}\n", i, file_allpath);
//just construction.
ImageBuf buf(file_allpath);
Strutil::print("reading {}\n", i);
// Do the read!!
buf.read(0, 0, 15, 18, true, TypeDesc::UINT8);
}));
}
for (auto& t : threads)
t->join();
And I created a bunch of test files like this:
oiiotool --frames 10-79 --create 1920x1080 20 -d half -o tmp/S25_#.exr
I'm just guessing here, making an HD res frame with 20 'half' channels, just to
simulate what happens.
Then I ran with some special debug environment variables set (you can try
these, too, though my example is using bash, you may need to do it somewhat
differently on Windows):
export OPENIMAGEIO_OPTIONS="debug=2"
export OPENIMAGEIO_IMAGECACHE_OPTIONS="statistics:level=1"
test_program
These variables will cause debug messages to print to the console whenever
ImageBuf allocates or deallocates pixels. You will see messages like this:
OIIO DEBUG: IB allocated 5 MB, global IB memory now 403 MB
OIIO DEBUG: IB allocated 5 MB, global IB memory now 409 MB
...
and also at the end, a statistics report from the ImageCache, where it might
say something like:
Total pixel data size of all images referenced : 5.4 GB
Total actual file size of all images referenced : 5.6 MB
Pixel data read : 0 B
File I/O time : 0.9s (0.0s average per thread, for 70 threads)
File open time only : 0.9s
Peak cache memory : 0 B
The files are opened (the ImageBuf uses ImageCache to read the file headers),
but no pixels are read and no ImageCache tile cache space is used (because of
that 'true' parameter we passed to force an immediate read rather than using
the cache).
Now, if you comment out the read() call from the loop, so that it's just the
ImageBuf construction, what happens is that NO messages will print about
ImageBuf allocations.
This seems to confirm two things (at least, for me running top-of-master OIIO
on a Mac):
1. ImageBuf construction alone opens the file and reads the header, but doesn't
read pixels at all and doesn't allocate any pixel memory.
2. With the read, it definitely allocates memory, but the peak for all images
is pretty manageable, with a peak of around 415MB. (But note: the resolution
and channels of my test images may not be as much as yours.)
So I'm curious to hear your results if you replicate this experiment, and also
the following information from you that might help diagnose the problem:
* Version of OIIO
* Full "oiiotool -info -v" of one of your files so I can see the resolution,
channels, data type, if it's tiled, etc.
* Your peak memory as reported by those debug statistics.
* Peak memory if you run the loop SERIALIZED (that is, one at a time, not
spawning threads at all).
* I'm also curious to know what happens with the thread spawn, but if you also
comment out the ImageBuf construction -- that is, how much of the memory use do
you think you are seeing is inherent to running that number of threads, and
doesn't have anything to do with ImageBuf at all?
> On May 15, 2022, at 9:18 AM, Larry Gritz <l...@larrygritz.com> wrote:
>
> Oh, interesting. So you are making all those ImageBuf's *simultaneously*. But
> on the other hand, it sure looks like it shouldn't consume much memory since
> you are never accessing the pixels.
>
> I will try it on my end and see what happens.
>
> Can you answer a few questions for me?
>
> 1. What version of OIIO are you using?
>
> 2. What is the size of the exr files you're loading? For example, do
> oiiotool -info -v S25_00.10.exr
> and show us what it says. (I'm looking especially at resolution,
> channels, data type.)
>
> 3. If you switch to sequential access, what happens to the memory
> consumption? In other words, commend out the thread launch, so you only have
>
> for (int i = 10; i < 80; i++) {
> std::string file_allpath = base + std::to_string(i) +
> std::string(".exr");
> //just construction.
> ImageBuf buf(file_allpath);
> }
>
> Meanwhile, I will investigate on my end and see if I can get any insight.
>
>
>> On May 15, 2022, at 8:25 AM, ████刮 <15952002...@qq.com
>> <mailto:15952002...@qq.com>> wrote:
>>
>>
>> it looks like this Email can't hold image.
>>
>> here is my simple test code:
>>
>> //exr sequence from S25_0010.exr to S25_0079.exr
>> std::string base = "C:/Users/Lo_Fnatic/Desktop/nima/S25_00";
>> for (int i = 10; i < 80; i++) {
>> std::string file_allpath = base + std::to_string(i) +
>> std::string(".exr");
>> std::thread t([=]() {
>>
>> //just construction.
>> ImageBuf buf(file_allpath);
>> //buf.read(0, 0, 15, 18, true, TypeDesc::UINT8);
>>
>> });
>> t.detach();
>> }
>>
>> it will cost almost 32G memory even though i am not ready to read these
>> pixel data.
>> thanks
>>
>>
>> ------------------ Original ------------------
>> From: "oiio-dev" <oiio-dev-requ...@lists.openimageio.org
>> <mailto:oiio-dev-requ...@lists.openimageio.org>>;
>> Date: Sun, May 15, 2022 02:15 AM
>> To: "oiio-dev"<oiio-dev@lists.openimageio.org
>> <mailto:oiio-dev@lists.openimageio.org>>;
>> Subject: Oiio-dev Digest, Vol 164, Issue 4
>>
>> Send Oiio-dev mailing list submissions to
>> oiio-dev@lists.openimageio.org <mailto:oiio-dev@lists.openimageio.org>
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>> or, via email, send a message with subject or body 'help' to
>> oiio-dev-requ...@lists.openimageio.org
>> <mailto:oiio-dev-requ...@lists.openimageio.org>
>>
>> You can reach the person managing the list at
>> oiio-dev-ow...@lists.openimageio.org
>> <mailto:oiio-dev-ow...@lists.openimageio.org>
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Oiio-dev digest..."
>>
>>
>> Today's Topics:
>>
>> 1. How to Load channel from EXR file with a little memory usage.
>> (=?gb18030?B?qICogKiAqIC5zg==?=)
>> 2. Re: How to Load channel from EXR file with a little memory
>> usage. (Larry Gritz)
>> 3. Re: How to Load channel from EXR file with a little memory
>> usage. (Larry Gritz)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sat, 14 May 2022 23:28:29 +0800
>> From: "=?gb18030?B?qICogKiAqIC5zg==?=" <15952002...@qq.com
>> <mailto:15952002...@qq.com>>
>> To: "=?gb18030?B?b2lpby1kZXY=?=" <oiio-dev@lists.openimageio.org
>> <mailto:oiio-dev@lists.openimageio.org>>
>> Subject: [Oiio-dev] How to Load channel from EXR file with a little
>> memory usage.
>> Message-ID: <tencent_77ef28dd6d2d2359af2ccc5ca8ce66b74...@qq.com
>> <mailto:tencent_77ef28dd6d2d2359af2ccc5ca8ce66b74...@qq.com>>
>> Content-Type: text/plain; charset="gb18030"
>>
>> Hello.
>> I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
>> but these EXR files are so big with many sub-channels in(one file with 300M
>> and more than 30 channels).
>> i just want to load the "Beauty" pass.
>>
>>
>> here is what i was tried:
>>
>>
>> ImageBuf buf = ImageBuf::ImageBuf(exr_filepath);
>> const ImageSpec& spec = buf.spec();
>> int xres = spec.width;
>> int yres = spec.height;
>>
>>
>> // i just want to load the RGB data
>> from the "Beauty" pass
>> std::vector<UINT8> pixels(xres * yres * 3);
>>
>>
>> // for example the Beauty pass start
>> with index of 15
>> ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 15,
>> 18);
>> buf.get_pixels(roi, TypeDesc::UINT8,
>> &pixels[0]);
>>
>>
>>
>> it works. but cost a huge amount of memory use with
>> multi-threading(sometimes more than 80G) during loading all files.
>> here is my quesion:
>> like what i did. is this way to load all data together then save the "Beauty
>> data" from them to the std::vector?
>> or just load the "Beauty" pass without cache all exr data first?
>> is this memory usage normal?
>>
>>
>> and i have checked the document. it looks like the ImageCache class is what
>> i want. can i use this class in my situation?
>> is there anywhere i can find more example about this class?
>>
>>
>> thanks!
>> kong
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/a7118f41/attachment-0001.html
>>
>> <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/a7118f41/attachment-0001.html>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sat, 14 May 2022 10:59:36 -0700
>> From: Larry Gritz <l...@larrygritz.com <mailto:l...@larrygritz.com>>
>> To: OpenImageIO dev list <oiio-dev@lists.openimageio.org
>> <mailto:oiio-dev@lists.openimageio.org>>
>> Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
>> little memory usage.
>> Message-ID: <68429dd3-80ae-444c-b06b-70dfed031...@larrygritz.com
>> <mailto:68429dd3-80ae-444c-b06b-70dfed031...@larrygritz.com>>
>> Content-Type: text/plain; charset="utf-8"
>>
>> I think that the problem here is that the ImageBuf will read in all the
>> channels, even though the get_pixels only copies out the 3 channels you
>> need. The I/O itself happens "lazily", triggered by the get_pixels call
>> itself needing pixel values, but at that point, it still doesn't know that
>> you won't subsequently try to access other channels in the ImageBuf, so the
>> channel restriction of get_pixels doesn't turn into a channel restriction
>> for how the file is read and stored.
>>
>> Additionally, the ImageBuf will by default store the pixels in whatever the
>> "widest" data type of the file itself has (in the case of an OpenEXR, that
>> would mean half at best, and maybe float depending on what's in your file),
>> even though ultimately you only want UINT8 data in your program. (And if the
>> file is large, there's also probably yet another copy, or partial copy, in
>> the underlying ImageCache.)
>>
>> But I think you can fix all of these issues by doing an explicit
>> ImageBuf::read() before the first time you access the pixels, using the
>> version of read() that lets you select the channel range and data type:
>>
>> buf.read(/*subimage*/ 0, /*miplevel*/ 0,
>> /*chbegin*/ 15, /*chend*/ 18,
>> /*forceread*/ true, /*convert*/ TypeDesc::UINT8);
>> // ... then ...
>>
>> // BEWARE! now channels 15... of the file are stored in 0... of the buf!
>> ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 0, 4);
>> buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
>>
>> Now the buf will internally store only 3 channels of UINT8.
>>
>> See where I passed true for the "forceread" parameter? That forces it to
>> read the image right then, skipping use of the ImageCache and storing the
>> entire image in memory (well, the channels you asked for, I mean). So the
>> ImageBuf itself holds the channels you asked for, without an extra copy in
>> the ImageCache. The ImageCache is usually very helpful, especially if you
>> only need a small portion of a large image at any one time by one thread.
>> The one case where it's wasteful is if you need an entire image all at once,
>> and it can fit into memory so there is no benefit to reading it piece by
>> piece only as needed.
>>
>> And given that we have the whole image loaded into the ImageBuf without any
>> ImageCache backing, it's possible that you can do one more better thing,
>> depending on how you use the data. With the forced read, the ImageBuf itself
>> holds channels 15,16,17 as UINT8, contiguously in memory starting at address
>> localpixels(). So do you really need the pixels to be copied (and stored
>> again) in your std::vector? Or can you directly use the ImageBuf's internal
>> storage of the pixels?
>>
>> You can test if an ImageBuf's pixels are all in memory:
>>
>> if (buf.storage() == ImageBuf::LOCALBUFFER)
>> address = (const uint8*) buf.localpixels();
>>
>> Actually, I think localpixels() will return nullptr if it doesn't use local
>> storage, so you don't need the separate check of storage(), but I still
>> wanted to point that out.
>>
>> (I think everything I have written is true. Please double check it.)
>>
>>
>> > On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com
>> > <mailto:15952002...@qq.com>> wrote:
>> >
>> > Hello.
>> > I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
>> > but these EXR files are so big with many sub-channels in(one file with
>> > 300M and more than 30 channels).
>> > i just want to load the "Beauty" pass.
>> >
>> > here is what i was tried:
>> >
>> > ImageBuf buf = ImageBuf::ImageBuf(exr_filepath);
>> > const ImageSpec& spec = buf.spec();
>> > int xres = spec.width;
>> > int yres = spec.height;
>> >
>> > // i just want to load the RGB data from the "Beauty" pass
>> > std::vector<UINT8> pixels(xres * yres * 3);
>> >
>> > // for example the Beauty pass start with index of 15
>> > ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 15, 18);
>> > buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
>> >
>> > it works. but cost a huge amount of memory use with
>> > multi-threading(sometimes more than 80G) during loading all files.
>> > here is my quesion:
>> > like what i did. is this way to load all data together then save the
>> > "Beauty data" from them to the std::vector?
>> > or just load the "Beauty" pass without cache all exr data first?
>> > is this memory usage normal?
>> >
>> > and i have checked the document. it looks like the ImageCache class is
>> > what i want. can i use this class in my situation?
>> > is there anywhere i can find more example about this class?
>> >
>> > thanks!
>> > kong
>> >
>> > _______________________________________________
>> > Oiio-dev mailing list
>> > Oiio-dev@lists.openimageio.org <mailto:Oiio-dev@lists.openimageio.org>
>> > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>> > <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>>
>> --
>> Larry Gritz
>> l...@larrygritz.com <mailto:l...@larrygritz.com>
>>
>>
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/80e79745/attachment-0001.html>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Sat, 14 May 2022 11:15:51 -0700
>> From: Larry Gritz <l...@larrygritz.com>
>> To: OpenImageIO dev list <oiio-dev@lists.openimageio.org>
>> Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
>> little memory usage.
>> Message-ID: <b12c7b0e-e77c-497e-a7d2-669b73d1c...@larrygritz.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> By the way, for those following along, if you're wondering how to make
>> oiiotool take these very same shortcuts:
>>
>> oiiotool -i:ch=R,G,B:type=uint8:now=1 foo.exr ...
>>
>>
>> > On May 14, 2022, at 10:59 AM, Larry Gritz <l...@larrygritz.com> wrote:
>> >
>> > I think that the problem here is that the ImageBuf will read in all the
>> > channels, even though the get_pixels only copies out the 3 channels you
>> > need. The I/O itself happens "lazily", triggered by the get_pixels call
>> > itself needing pixel values, but at that point, it still doesn't know that
>> > you won't subsequently try to access other channels in the ImageBuf, so
>> > the channel restriction of get_pixels doesn't turn into a channel
>> > restriction for how the file is read and stored.
>> >
>> > Additionally, the ImageBuf will by default store the pixels in whatever
>> > the "widest" data type of the file itself has (in the case of an OpenEXR,
>> > that would mean half at best, and maybe float depending on what's in your
>> > file), even though ultimately you only want UINT8 data in your program.
>> > (And if the file is large, there's also probably yet another copy, or
>> > partial copy, in the underlying ImageCache.)
>> >
>> > But I think you can fix all of these issues by doing an explicit
>> > ImageBuf::read() before the first time you access the pixels, using the
>> > version of read() that lets you select the channel range and data type:
>> >
>> > buf.read(/*subimage*/ 0, /*miplevel*/ 0,
>> > /*chbegin*/ 15, /*chend*/ 18,
>> > /*forceread*/ true, /*convert*/ TypeDesc::UINT8);
>> > // ... then ...
>> >
>> > // BEWARE! now channels 15... of the file are stored in 0... of the
>> > buf!
>> > ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 0, 4);
>> > buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
>> >
>> > Now the buf will internally store only 3 channels of UINT8.
>> >
>> > See where I passed true for the "forceread" parameter? That forces it to
>> > read the image right then, skipping use of the ImageCache and storing the
>> > entire image in memory (well, the channels you asked for, I mean). So the
>> > ImageBuf itself holds the channels you asked for, without an extra copy in
>> > the ImageCache. The ImageCache is usually very helpful, especially if you
>> > only need a small portion of a large image at any one time by one thread.
>> > The one case where it's wasteful is if you need an entire image all at
>> > once, and it can fit into memory so there is no benefit to reading it
>> > piece by piece only as needed.
>> >
>> > And given that we have the whole image loaded into the ImageBuf without
>> > any ImageCache backing, it's possible that you can do one more better
>> > thing, depending on how you use the data. With the forced read, the
>> > ImageBuf itself holds channels 15,16,17 as UINT8, contiguously in memory
>> > starting at address localpixels(). So do you really need the pixels to be
>> > copied (and stored again) in your std::vector? Or can you directly use the
>> > ImageBuf's internal storage of the pixels?
>> >
>> > You can test if an ImageBuf's pixels are all in memory:
>> >
>> > if (buf.storage() == ImageBuf::LOCALBUFFER)
>> > address = (const uint8*) buf.localpixels();
>> >
>> > Actually, I think localpixels() will return nullptr if it doesn't use
>> > local storage, so you don't need the separate check of storage(), but I
>> > still wanted to point that out.
>> >
>> > (I think everything I have written is true. Please double check it.)
>> >
>> >
>> >> On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com
>> >> <mailto:15952002...@qq.com>> wrote:
>> >>
>> >> Hello.
>> >> I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
>> >> but these EXR files are so big with many sub-channels in(one file with
>> >> 300M and more than 30 channels).
>> >> i just want to load the "Beauty" pass.
>> >>
>> >> here is what i was tried:
>> >>
>> >> ImageBuf buf = ImageBuf::ImageBuf(exr_filepath);
>> >> const ImageSpec& spec = buf.spec();
>> >> int xres = spec.width;
>> >> int yres = spec.height;
>> >>
>> >> // i just want to load the RGB data from the "Beauty" pass
>> >> std::vector<UINT8> pixels(xres * yres * 3);
>> >>
>> >> // for example the Beauty pass start with index of 15
>> >> ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 15, 18);
>> >> buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
>> >>
>> >> it works. but cost a huge amount of memory use with
>> >> multi-threading(sometimes more than 80G) during loading all files.
>> >> here is my quesion:
>> >> like what i did. is this way to load all data together then save the
>> >> "Beauty data" from them to the std::vector?
>> >> or just load the "Beauty" pass without cache all exr data first?
>> >> is this memory usage normal?
>> >>
>> >> and i have checked the document. it looks like the ImageCache class is
>> >> what i want. can i use this class in my situation?
>> >> is there anywhere i can find more example about this class?
>> >>
>> >> thanks!
>> >> kong
>> >>
>> >> _______________________________________________
>> >> Oiio-dev mailing list
>> >> Oiio-dev@lists.openimageio.org <mailto:Oiio-dev@lists.openimageio.org>
>> >> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>> >
>> > --
>> > Larry Gritz
>> > l...@larrygritz.com <mailto:l...@larrygritz.com>
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Oiio-dev mailing list
>> > Oiio-dev@lists.openimageio.org
>> > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>
>> --
>> Larry Gritz
>> l...@larrygritz.com
>>
>>
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/dce8c2a6/attachment.html>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Oiio-dev mailing list
>> Oiio-dev@lists.openimageio.org
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>
>>
>> ------------------------------
>>
>> End of Oiio-dev Digest, Vol 164, Issue 4
>> ****************************************
>> _______________________________________________
>> Oiio-dev mailing list
>> Oiio-dev@lists.openimageio.org <mailto:Oiio-dev@lists.openimageio.org>
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
> --
> Larry Gritz
> l...@larrygritz.com <mailto:l...@larrygritz.com>
>
>
>
>
>
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
l...@larrygritz.com
_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
