Oh, interesting. So you are making all those ImageBuf's *simultaneously*. But
on the other hand, it sure looks like it shouldn't consume much memory since
you are never accessing the pixels.
I will try it on my end and see what happens.
Can you answer a few questions for me?
1. What version of OIIO are you using?
2. What is the size of the exr files you're loading? For example, do
oiiotool -info -v S25_00.10.exr
and show us what it says. (I'm looking especially at resolution, channels,
data type.)
3. If you switch to sequential access, what happens to the memory consumption?
In other words, commend out the thread launch, so you only have
for (int i = 10; i < 80; i++) {
std::string file_allpath = base + std::to_string(i) +
std::string(".exr");
//just construction.
ImageBuf buf(file_allpath);
}
Meanwhile, I will investigate on my end and see if I can get any insight.
> On May 15, 2022, at 8:25 AM, ████刮 <15952002...@qq.com> wrote:
>
>
> it looks like this Email can't hold image.
>
> here is my simple test code:
>
> //exr sequence from S25_0010.exr to S25_0079.exr
> std::string base = "C:/Users/Lo_Fnatic/Desktop/nima/S25_00";
> for (int i = 10; i < 80; i++) {
> std::string file_allpath = base + std::to_string(i) +
> std::string(".exr");
> std::thread t([=]() {
>
> //just construction.
> ImageBuf buf(file_allpath);
> //buf.read(0, 0, 15, 18, true, TypeDesc::UINT8);
>
> });
> t.detach();
> }
>
> it will cost almost 32G memory even though i am not ready to read these pixel
> data.
> thanks
>
>
> ------------------ Original ------------------
> From: "oiio-dev" <oiio-dev-requ...@lists.openimageio.org>;
> Date: Sun, May 15, 2022 02:15 AM
> To: "oiio-dev"<oiio-dev@lists.openimageio.org>;
> Subject: Oiio-dev Digest, Vol 164, Issue 4
>
> Send Oiio-dev mailing list submissions to
> oiio-dev@lists.openimageio.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
> or, via email, send a message with subject or body 'help' to
> oiio-dev-requ...@lists.openimageio.org
>
> You can reach the person managing the list at
> oiio-dev-ow...@lists.openimageio.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Oiio-dev digest..."
>
>
> Today's Topics:
>
> 1. How to Load channel from EXR file with a little memory usage.
> (=?gb18030?B?qICogKiAqIC5zg==?=)
> 2. Re: How to Load channel from EXR file with a little memory
> usage. (Larry Gritz)
> 3. Re: How to Load channel from EXR file with a little memory
> usage. (Larry Gritz)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 14 May 2022 23:28:29 +0800
> From: "=?gb18030?B?qICogKiAqIC5zg==?=" <15952002...@qq.com>
> To: "=?gb18030?B?b2lpby1kZXY=?=" <oiio-dev@lists.openimageio.org>
> Subject: [Oiio-dev] How to Load channel from EXR file with a little
> memory usage.
> Message-ID: <tencent_77ef28dd6d2d2359af2ccc5ca8ce66b74...@qq.com>
> Content-Type: text/plain; charset="gb18030"
>
> Hello.
> I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
> but these EXR files are so big with many sub-channels in(one file with 300M
> and more than 30 channels).
> i just want to load the "Beauty" pass.
>
>
> here is what i was tried:
>
>
> ImageBuf buf = ImageBuf::ImageBuf(exr_filepath);
> const ImageSpec& spec = buf.spec();
> int xres = spec.width;
> int yres = spec.height;
>
>
> // i just want to load the RGB data
> from the "Beauty" pass
> std::vector<UINT8> pixels(xres * yres * 3);
>
>
> // for example the Beauty pass start
> with index of 15
> ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 15,
> 18);
> buf.get_pixels(roi, TypeDesc::UINT8,
> &pixels[0]);
>
>
>
> it works. but cost a huge amount of memory use with multi-threading(sometimes
> more than 80G) during loading all files.
> here is my quesion:
> like what i did. is this way to load all data together then save the "Beauty
> data" from them to the std::vector?
> or just load the "Beauty" pass without cache all exr data first?
> is this memory usage normal?
>
>
> and i have checked the document. it looks like the ImageCache class is what i
> want. can i use this class in my situation?
> is there anywhere i can find more example about this class?
>
>
> thanks!
> kong
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/a7118f41/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Sat, 14 May 2022 10:59:36 -0700
> From: Larry Gritz <l...@larrygritz.com>
> To: OpenImageIO dev list <oiio-dev@lists.openimageio.org>
> Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
> little memory usage.
> Message-ID: <68429dd3-80ae-444c-b06b-70dfed031...@larrygritz.com>
> Content-Type: text/plain; charset="utf-8"
>
> I think that the problem here is that the ImageBuf will read in all the
> channels, even though the get_pixels only copies out the 3 channels you need.
> The I/O itself happens "lazily", triggered by the get_pixels call itself
> needing pixel values, but at that point, it still doesn't know that you won't
> subsequently try to access other channels in the ImageBuf, so the channel
> restriction of get_pixels doesn't turn into a channel restriction for how the
> file is read and stored.
>
> Additionally, the ImageBuf will by default store the pixels in whatever the
> "widest" data type of the file itself has (in the case of an OpenEXR, that
> would mean half at best, and maybe float depending on what's in your file),
> even though ultimately you only want UINT8 data in your program. (And if the
> file is large, there's also probably yet another copy, or partial copy, in
> the underlying ImageCache.)
>
> But I think you can fix all of these issues by doing an explicit
> ImageBuf::read() before the first time you access the pixels, using the
> version of read() that lets you select the channel range and data type:
>
> buf.read(/*subimage*/ 0, /*miplevel*/ 0,
> /*chbegin*/ 15, /*chend*/ 18,
> /*forceread*/ true, /*convert*/ TypeDesc::UINT8);
> // ... then ...
>
> // BEWARE! now channels 15... of the file are stored in 0... of the buf!
> ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 0, 4);
> buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
>
> Now the buf will internally store only 3 channels of UINT8.
>
> See where I passed true for the "forceread" parameter? That forces it to read
> the image right then, skipping use of the ImageCache and storing the entire
> image in memory (well, the channels you asked for, I mean). So the ImageBuf
> itself holds the channels you asked for, without an extra copy in the
> ImageCache. The ImageCache is usually very helpful, especially if you only
> need a small portion of a large image at any one time by one thread. The one
> case where it's wasteful is if you need an entire image all at once, and it
> can fit into memory so there is no benefit to reading it piece by piece only
> as needed.
>
> And given that we have the whole image loaded into the ImageBuf without any
> ImageCache backing, it's possible that you can do one more better thing,
> depending on how you use the data. With the forced read, the ImageBuf itself
> holds channels 15,16,17 as UINT8, contiguously in memory starting at address
> localpixels(). So do you really need the pixels to be copied (and stored
> again) in your std::vector? Or can you directly use the ImageBuf's internal
> storage of the pixels?
>
> You can test if an ImageBuf's pixels are all in memory:
>
> if (buf.storage() == ImageBuf::LOCALBUFFER)
> address = (const uint8*) buf.localpixels();
>
> Actually, I think localpixels() will return nullptr if it doesn't use local
> storage, so you don't need the separate check of storage(), but I still
> wanted to point that out.
>
> (I think everything I have written is true. Please double check it.)
>
>
> > On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com> wrote:
> >
> > Hello.
> > I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
> > but these EXR files are so big with many sub-channels in(one file with 300M
> > and more than 30 channels).
> > i just want to load the "Beauty" pass.
> >
> > here is what i was tried:
> >
> > ImageBuf buf = ImageBuf::ImageBuf(exr_filepath);
> > const ImageSpec& spec = buf.spec();
> > int xres = spec.width;
> > int yres = spec.height;
> >
> > // i just want to load the RGB data from the "Beauty" pass
> > std::vector<UINT8> pixels(xres * yres * 3);
> >
> > // for example the Beauty pass start with index of 15
> > ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 15, 18);
> > buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
> >
> > it works. but cost a huge amount of memory use with
> > multi-threading(sometimes more than 80G) during loading all files.
> > here is my quesion:
> > like what i did. is this way to load all data together then save the
> > "Beauty data" from them to the std::vector?
> > or just load the "Beauty" pass without cache all exr data first?
> > is this memory usage normal?
> >
> > and i have checked the document. it looks like the ImageCache class is what
> > i want. can i use this class in my situation?
> > is there anywhere i can find more example about this class?
> >
> > thanks!
> > kong
> >
> > _______________________________________________
> > Oiio-dev mailing list
> > Oiio-dev@lists.openimageio.org
> > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
> --
> Larry Gritz
> l...@larrygritz.com
>
>
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/80e79745/attachment-0001.html>
>
> ------------------------------
>
> Message: 3
> Date: Sat, 14 May 2022 11:15:51 -0700
> From: Larry Gritz <l...@larrygritz.com>
> To: OpenImageIO dev list <oiio-dev@lists.openimageio.org>
> Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
> little memory usage.
> Message-ID: <b12c7b0e-e77c-497e-a7d2-669b73d1c...@larrygritz.com>
> Content-Type: text/plain; charset="utf-8"
>
> By the way, for those following along, if you're wondering how to make
> oiiotool take these very same shortcuts:
>
> oiiotool -i:ch=R,G,B:type=uint8:now=1 foo.exr ...
>
>
> > On May 14, 2022, at 10:59 AM, Larry Gritz <l...@larrygritz.com> wrote:
> >
> > I think that the problem here is that the ImageBuf will read in all the
> > channels, even though the get_pixels only copies out the 3 channels you
> > need. The I/O itself happens "lazily", triggered by the get_pixels call
> > itself needing pixel values, but at that point, it still doesn't know that
> > you won't subsequently try to access other channels in the ImageBuf, so the
> > channel restriction of get_pixels doesn't turn into a channel restriction
> > for how the file is read and stored.
> >
> > Additionally, the ImageBuf will by default store the pixels in whatever the
> > "widest" data type of the file itself has (in the case of an OpenEXR, that
> > would mean half at best, and maybe float depending on what's in your file),
> > even though ultimately you only want UINT8 data in your program. (And if
> > the file is large, there's also probably yet another copy, or partial copy,
> > in the underlying ImageCache.)
> >
> > But I think you can fix all of these issues by doing an explicit
> > ImageBuf::read() before the first time you access the pixels, using the
> > version of read() that lets you select the channel range and data type:
> >
> > buf.read(/*subimage*/ 0, /*miplevel*/ 0,
> > /*chbegin*/ 15, /*chend*/ 18,
> > /*forceread*/ true, /*convert*/ TypeDesc::UINT8);
> > // ... then ...
> >
> > // BEWARE! now channels 15... of the file are stored in 0... of the buf!
> > ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 0, 4);
> > buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
> >
> > Now the buf will internally store only 3 channels of UINT8.
> >
> > See where I passed true for the "forceread" parameter? That forces it to
> > read the image right then, skipping use of the ImageCache and storing the
> > entire image in memory (well, the channels you asked for, I mean). So the
> > ImageBuf itself holds the channels you asked for, without an extra copy in
> > the ImageCache. The ImageCache is usually very helpful, especially if you
> > only need a small portion of a large image at any one time by one thread.
> > The one case where it's wasteful is if you need an entire image all at
> > once, and it can fit into memory so there is no benefit to reading it piece
> > by piece only as needed.
> >
> > And given that we have the whole image loaded into the ImageBuf without any
> > ImageCache backing, it's possible that you can do one more better thing,
> > depending on how you use the data. With the forced read, the ImageBuf
> > itself holds channels 15,16,17 as UINT8, contiguously in memory starting at
> > address localpixels(). So do you really need the pixels to be copied (and
> > stored again) in your std::vector? Or can you directly use the ImageBuf's
> > internal storage of the pixels?
> >
> > You can test if an ImageBuf's pixels are all in memory:
> >
> > if (buf.storage() == ImageBuf::LOCALBUFFER)
> > address = (const uint8*) buf.localpixels();
> >
> > Actually, I think localpixels() will return nullptr if it doesn't use local
> > storage, so you don't need the separate check of storage(), but I still
> > wanted to point that out.
> >
> > (I think everything I have written is true. Please double check it.)
> >
> >
> >> On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com
> >> <mailto:15952002...@qq.com>> wrote:
> >>
> >> Hello.
> >> I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
> >> but these EXR files are so big with many sub-channels in(one file with
> >> 300M and more than 30 channels).
> >> i just want to load the "Beauty" pass.
> >>
> >> here is what i was tried:
> >>
> >> ImageBuf buf = ImageBuf::ImageBuf(exr_filepath);
> >> const ImageSpec& spec = buf.spec();
> >> int xres = spec.width;
> >> int yres = spec.height;
> >>
> >> // i just want to load the RGB data from the "Beauty" pass
> >> std::vector<UINT8> pixels(xres * yres * 3);
> >>
> >> // for example the Beauty pass start with index of 15
> >> ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 15, 18);
> >> buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
> >>
> >> it works. but cost a huge amount of memory use with
> >> multi-threading(sometimes more than 80G) during loading all files.
> >> here is my quesion:
> >> like what i did. is this way to load all data together then save the
> >> "Beauty data" from them to the std::vector?
> >> or just load the "Beauty" pass without cache all exr data first?
> >> is this memory usage normal?
> >>
> >> and i have checked the document. it looks like the ImageCache class is
> >> what i want. can i use this class in my situation?
> >> is there anywhere i can find more example about this class?
> >>
> >> thanks!
> >> kong
> >>
> >> _______________________________________________
> >> Oiio-dev mailing list
> >> Oiio-dev@lists.openimageio.org <mailto:Oiio-dev@lists.openimageio.org>
> >> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
> >
> > --
> > Larry Gritz
> > l...@larrygritz.com <mailto:l...@larrygritz.com>
> >
> >
> >
> >
> >
> > _______________________________________________
> > Oiio-dev mailing list
> > Oiio-dev@lists.openimageio.org
> > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
> --
> Larry Gritz
> l...@larrygritz.com
>
>
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/dce8c2a6/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
>
> ------------------------------
>
> End of Oiio-dev Digest, Vol 164, Issue 4
> ****************************************
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
l...@larrygritz.com
_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
