it looks like this Email can't hold image.
here is my simple test code:
//exr sequence from S25_0010.exr to S25_0079.exr
std::string base =
"C:/Users/Lo_Fnatic/Desktop/nima/S25_00";
for (int i = 10; i < 80; i++) {
std::string file_allpath = base +
std::to_string(i) + std::string(".exr");
std::thread t([=]() {
//just construction.
ImageBuf
buf(file_allpath);
//buf.read(0, 0, 15,
18, true, TypeDesc::UINT8);
});
t.detach();
}
it will cost almost 32G memory even though i am not ready to read these pixel
data.
thanks
------------------ Original ------------------
From:
"oiio-dev"
<oiio-dev-requ...@lists.openimageio.org>;
Date: Sun, May 15, 2022 02:15 AM
To: "oiio-dev"<oiio-dev@lists.openimageio.org>;
Subject: Oiio-dev Digest, Vol 164, Issue 4
Send Oiio-dev mailing list submissions to
oiio-dev@lists.openimageio.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
or, via email, send a message with subject or body 'help' to
oiio-dev-requ...@lists.openimageio.org
You can reach the person managing the list at
oiio-dev-ow...@lists.openimageio.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Oiio-dev digest..."
Today's Topics:
1. How to Load channel from EXR file with a little memory usage.
(=?gb18030?B?qICogKiAqIC5zg==?=)
2. Re: How to Load channel from EXR file with a little memory
usage. (Larry Gritz)
3. Re: How to Load channel from EXR file with a little memory
usage. (Larry Gritz)
----------------------------------------------------------------------
Message: 1
Date: Sat, 14 May 2022 23:28:29 +0800
From: "=?gb18030?B?qICogKiAqIC5zg==?=" <15952002...@qq.com>
To: "=?gb18030?B?b2lpby1kZXY=?=" <oiio-dev@lists.openimageio.org>
Subject: [Oiio-dev] How to Load channel from EXR file with a little
memory usage.
Message-ID: <tencent_77ef28dd6d2d2359af2ccc5ca8ce66b74...@qq.com>
Content-Type: text/plain; charset="gb18030"
Hello.
I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
but these EXR files are so big with many sub-channels in(one file with 300M and
more than 30 channels).
&nbsp;i just want to load the "Beauty" pass.
here is what i was tried:
&nbsp; &nbsp; &nbsp; &nbsp; ImageBuf buf =
ImageBuf::ImageBuf(exr_filepath);
&nbsp; &nbsp; &nbsp; &nbsp; const ImageSpec&amp; spec =
buf.spec();
&nbsp; &nbsp; &nbsp; &nbsp; int xres = spec.width;
&nbsp; &nbsp; &nbsp; &nbsp; int yres = spec.height;
&nbsp; &nbsp; &nbsp; &nbsp; //&nbsp; &nbsp;i just want
to load the RGB data from the "Beauty" pass
&nbsp; &nbsp; &nbsp; &nbsp; std::vector<UINT8&gt;
pixels(xres * yres * 3);
&nbsp; &nbsp; &nbsp; &nbsp; //&nbsp; &nbsp;for example
the Beauty pass start with index of 15
&nbsp; &nbsp; &nbsp; &nbsp; ROI roi(0, xres, 0, yres, 0, 1,
/*chans:*/ 15, 18);
&nbsp; &nbsp; &nbsp; &nbsp; buf.get_pixels(roi,
TypeDesc::UINT8, &amp;pixels[0]);
it works. but cost a huge amount of memory use with multi-threading(sometimes
more than 80G) during loading all files.
here is my quesion:
like what i did. is this way to load all data together then save the "Beauty
data" from them to the std::vector?
or just load the "Beauty" pass without cache all exr data first?
is this memory usage normal?
and i have checked the document. it looks like the ImageCache class is what i
want. can i use this class in my situation?
is there anywhere i can find more example about this class?
thanks!
kong
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/a7118f41/attachment-0001.html>
------------------------------
Message: 2
Date: Sat, 14 May 2022 10:59:36 -0700
From: Larry Gritz <l...@larrygritz.com>
To: OpenImageIO dev list <oiio-dev@lists.openimageio.org>
Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
little memory usage.
Message-ID: <68429dd3-80ae-444c-b06b-70dfed031...@larrygritz.com>
Content-Type: text/plain; charset="utf-8"
I think that the problem here is that the ImageBuf will read in all the
channels, even though the get_pixels only copies out the 3 channels you need.
The I/O itself happens "lazily", triggered by the get_pixels call itself
needing pixel values, but at that point, it still doesn't know that you won't
subsequently try to access other channels in the ImageBuf, so the channel
restriction of get_pixels doesn't turn into a channel restriction for how the
file is read and stored.
Additionally, the ImageBuf will by default store the pixels in whatever the
"widest" data type of the file itself has (in the case of an OpenEXR, that
would mean half at best, and maybe float depending on what's in your file),
even though ultimately you only want UINT8 data in your program. (And if the
file is large, there's also probably yet another copy, or partial copy, in the
underlying ImageCache.)
But I think you can fix all of these issues by doing an explicit
ImageBuf::read() before the first time you access the pixels, using the version
of read() that lets you select the channel range and data type:
buf.read(/*subimage*/ 0, /*miplevel*/ 0,
/*chbegin*/ 15, /*chend*/ 18,
/*forceread*/ true, /*convert*/ TypeDesc::UINT8);
// ... then ...
// BEWARE! now channels 15... of the file are stored in 0...
of the buf!
ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 0, 4);
buf.get_pixels(roi, TypeDesc::UINT8, &pixels[0]);
Now the buf will internally store only 3 channels of UINT8.
See where I passed true for the "forceread" parameter? That forces it to read
the image right then, skipping use of the ImageCache and storing the entire
image in memory (well, the channels you asked for, I mean). So the ImageBuf
itself holds the channels you asked for, without an extra copy in the
ImageCache. The ImageCache is usually very helpful, especially if you only need
a small portion of a large image at any one time by one thread. The one case
where it's wasteful is if you need an entire image all at once, and it can fit
into memory so there is no benefit to reading it piece by piece only as needed.
And given that we have the whole image loaded into the ImageBuf without any
ImageCache backing, it's possible that you can do one more better thing,
depending on how you use the data. With the forced read, the ImageBuf itself
holds channels 15,16,17 as UINT8, contiguously in memory starting at address
localpixels(). So do you really need the pixels to be copied (and stored again)
in your std::vector? Or can you directly use the ImageBuf's internal storage of
the pixels?
You can test if an ImageBuf's pixels are all in memory:
if (buf.storage() == ImageBuf::LOCALBUFFER)
address = (const uint8*)
buf.localpixels();
Actually, I think localpixels() will return nullptr if it doesn't use local
storage, so you don't need the separate check of storage(), but I still wanted
to point that out.
(I think everything I have written is true. Please double check it.)
> On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com> wrote:
>
> Hello.
> I'm trying to load OpenEXR sequence to memory then draw them with Opengl.
> but these EXR files are so big with many sub-channels in(one file with
300M and more than 30 channels).
> i just want to load the "Beauty" pass.
>
> here is what i was tried:
>
> ImageBuf buf =
ImageBuf::ImageBuf(exr_filepath);
> const ImageSpec& spec
= buf.spec();
> int xres = spec.width;
> int yres = spec.height;
>
> // i just want
to load the RGB data from the "Beauty" pass
> std::vector<UINT8>
pixels(xres * yres * 3);
>
> // for example
the Beauty pass start with index of 15
> ROI roi(0, xres, 0, yres,
0, 1, /*chans:*/ 15, 18);
> buf.get_pixels(roi,
TypeDesc::UINT8, &pixels[0]);
>
> it works. but cost a huge amount of memory use with
multi-threading(sometimes more than 80G) during loading all files.
> here is my quesion:
> like what i did. is this way to load all data together then save the
"Beauty data" from them to the std::vector?
> or just load the "Beauty" pass without cache all exr data first?
> is this memory usage normal?
>
> and i have checked the document. it looks like the ImageCache class is
what i want. can i use this class in my situation?
> is there anywhere i can find more example about this class?
>
> thanks!
> kong
>
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
l...@larrygritz.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/80e79745/attachment-0001.html>
------------------------------
Message: 3
Date: Sat, 14 May 2022 11:15:51 -0700
From: Larry Gritz <l...@larrygritz.com>
To: OpenImageIO dev list <oiio-dev@lists.openimageio.org>
Subject: Re: [Oiio-dev] How to Load channel from EXR file with a
little memory usage.
Message-ID: <b12c7b0e-e77c-497e-a7d2-669b73d1c...@larrygritz.com>
Content-Type: text/plain; charset="utf-8"
By the way, for those following along, if you're wondering how to make oiiotool
take these very same shortcuts:
oiiotool -i:ch=R,G,B:type=uint8:now=1 foo.exr ...
> On May 14, 2022, at 10:59 AM, Larry Gritz <l...@larrygritz.com> wrote:
>
> I think that the problem here is that the ImageBuf will read in all the
channels, even though the get_pixels only copies out the 3 channels you need.
The I/O itself happens "lazily", triggered by the get_pixels call itself
needing pixel values, but at that point, it still doesn't know that you won't
subsequently try to access other channels in the ImageBuf, so the channel
restriction of get_pixels doesn't turn into a channel restriction for how the
file is read and stored.
>
> Additionally, the ImageBuf will by default store the pixels in whatever
the "widest" data type of the file itself has (in the case of an OpenEXR, that
would mean half at best, and maybe float depending on what's in your file),
even though ultimately you only want UINT8 data in your program. (And if the
file is large, there's also probably yet another copy, or partial copy, in the
underlying ImageCache.)
>
> But I think you can fix all of these issues by doing an explicit
ImageBuf::read() before the first time you access the pixels, using the version
of read() that lets you select the channel range and data type:
>
> buf.read(/*subimage*/ 0, /*miplevel*/ 0,
>
/*chbegin*/ 15, /*chend*/ 18,
>
/*forceread*/ true, /*convert*/ TypeDesc::UINT8);
> // ... then ...
>
> // BEWARE! now channels 15... of the file are
stored in 0... of the buf!
> ROI roi(0, xres, 0, yres, 0, 1, /*chans:*/ 0, 4);
> buf.get_pixels(roi, TypeDesc::UINT8,
&pixels[0]);
>
> Now the buf will internally store only 3 channels of UINT8.
>
> See where I passed true for the "forceread" parameter? That forces it to
read the image right then, skipping use of the ImageCache and storing the
entire image in memory (well, the channels you asked for, I mean). So the
ImageBuf itself holds the channels you asked for, without an extra copy in the
ImageCache. The ImageCache is usually very helpful, especially if you only need
a small portion of a large image at any one time by one thread. The one case
where it's wasteful is if you need an entire image all at once, and it can fit
into memory so there is no benefit to reading it piece by piece only as needed.
>
> And given that we have the whole image loaded into the ImageBuf without
any ImageCache backing, it's possible that you can do one more better thing,
depending on how you use the data. With the forced read, the ImageBuf itself
holds channels 15,16,17 as UINT8, contiguously in memory starting at address
localpixels(). So do you really need the pixels to be copied (and stored again)
in your std::vector? Or can you directly use the ImageBuf's internal storage of
the pixels?
>
> You can test if an ImageBuf's pixels are all in memory:
>
> if (buf.storage() == ImageBuf::LOCALBUFFER)
> address = (const uint8*)
buf.localpixels();
>
> Actually, I think localpixels() will return nullptr if it doesn't use
local storage, so you don't need the separate check of storage(), but I still
wanted to point that out.
>
> (I think everything I have written is true. Please double check it.)
>
>
>> On May 14, 2022, at 8:28 AM, ????? <15952002...@qq.com
<mailto:15952002...@qq.com>> wrote:
>>
>> Hello.
>> I'm trying to load OpenEXR sequence to memory then draw them with
Opengl.
>> but these EXR files are so big with many sub-channels in(one file with
300M and more than 30 channels).
>> i just want to load the "Beauty" pass.
>>
>> here is what i was tried:
>>
>> ImageBuf buf =
ImageBuf::ImageBuf(exr_filepath);
>> const ImageSpec&
spec = buf.spec();
>> int xres = spec.width;
>> int yres = spec.height;
>>
>> // i just
want to load the RGB data from the "Beauty" pass
>> std::vector<UINT8>
pixels(xres * yres * 3);
>>
>> // for
example the Beauty pass start with index of 15
>> ROI roi(0, xres, 0,
yres, 0, 1, /*chans:*/ 15, 18);
>> buf.get_pixels(roi,
TypeDesc::UINT8, &pixels[0]);
>>
>> it works. but cost a huge amount of memory use with
multi-threading(sometimes more than 80G) during loading all files.
>> here is my quesion:
>> like what i did. is this way to load all data together then save the
"Beauty data" from them to the std::vector?
>> or just load the "Beauty" pass without cache all exr data first?
>> is this memory usage normal?
>>
>> and i have checked the document. it looks like the ImageCache class is
what i want. can i use this class in my situation?
>> is there anywhere i can find more example about this class?
>>
>> thanks!
>> kong
>>
>> _______________________________________________
>> Oiio-dev mailing list
>> Oiio-dev@lists.openimageio.org
<mailto:Oiio-dev@lists.openimageio.org>
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
> --
> Larry Gritz
> l...@larrygritz.com <mailto:l...@larrygritz.com>
>
>
>
>
>
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
l...@larrygritz.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.openimageio.org/pipermail/oiio-dev-openimageio.org/attachments/20220514/dce8c2a6/attachment.html>
------------------------------
Subject: Digest Footer
_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
------------------------------
End of Oiio-dev Digest, Vol 164, Issue 4
****************************************_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
