Replying just to put the quoted email back into circulation -- this was the
message that bounced last night for all gmail users due to gmail-side glitch (I
presume). But I think it has some really interesting information that people
may want to know if they find themselves trying to turn gigapixel images into
textures.
-- lg
> On May 4, 2023, at 11:21 PM, Larry Gritz <l...@larrygritz.com> wrote:
>
> Part 2 of this effort was contained in this patch:
> https://github.com/OpenImageIO/oiio/pull/3807
> <https://github.com/OpenImageIO/oiio/pull/3807>
>
> In that one, I did some refactoring of the PSD reading which I found was
> basically holding an entire extra copy of the image in the PSDInput itself,
> so... essentially doubling the memory footprint of reading a PSD.
>
> And part 3 in https://github.com/OpenImageIO/oiio/pull/3829
> <https://github.com/OpenImageIO/oiio/pull/3829>, currently under review,
> addresses efficiency (and performance statistics) of the make_texture process
> for large images.
>
> After those patches above, I was able to read your whole enormous psd file if
> I give oiiotool some extra hints:
>
> oiiotool -v -i:type=uint8:now=1 0001-F_3a_Left.psb -otex:forcefloat=0
> out.tx
>
> I'm using -i explicitly instead of just naming the file, so that I can take
> advantage of a couple of optional modifiers that control the read process:
>
> now=1
> causes the read to happen fully and immediately, forcing it to avoid using
> the ImageCache, which it would normally do for large files to limit memory
> use and read lazily. Since making a texture requires the whole thing to be in
> memory at once, there is no point to using IC in this case and really hurts
> performance (since the image is so much larger than cache size) and the last
> thing you want is for an extra copy to be stored in the IC.
> type=uint8
> Forces oiiotool to read into a uint8 buffer, which is the minimal size it can
> be since we know that as a PSD file, it starts off as uint8 data in the file.
> This is to combat oiiotool's default behavior of reading everything into
> internal float buffers for maximum precision and performance of math
> operations. But in this case, it's worth paying the penalty of uint8 -> float
> math -> uint8 conversions on the fly to keep the in-memory representation
> from exploding by 4x (32 bit float instead of 8 bit int per channel datum).
>
> And there's a special modifier on -otex as well:
>
> forcefloat=0
> Disables the usual behavior of converting to a float buffer and using float
> intermediate buffers when down-resing to generate MIP levels, instead keeping
> it in the native format (in this case, int8, thus avoiding the 4x memory
> expansion, at the expense of some time and probably worse math precision for
> the downsize).
>
> With those patches applied, the above command successfully creates a texture
> from your psb file on my MacBookPro with 32GB RAM, though it takes about 1/2
> hour! I think that's because it's swapping between memory and disk
> constantly. I also tried on my work machine with oodles of memory, and it
> takes 12 minutes, most of which appears to be actually reading the input file
> and writing the resulting texture to disk. There's still lots of room for
> improvement, but it basically works now.
>
>
>
>> On Apr 27, 2023, at 12:39 PM, Jens Olsson
>> <jens.gustaf.johan.ols...@gmail.com
>> <mailto:jens.gustaf.johan.ols...@gmail.com>> wrote:
>>
>> Hey, that's great news! Thanks for looking into it, looking forward to
>> testing the fix!
>> (Still not getting anything from the mailing list to my gmail, so manual
>> copy-paste reply again)
>>
>> Cheers,
>> Jens
>> > Sorry it took me a while to get to this, but I have the first part of the
>> > solution here: https://github.com/OpenImageIO/oiio/pull/3806
>> > <https://github.com/OpenImageIO/oiio/pull/3806>
>> > <https://github.com/OpenImageIO/oiio/pull/3806
>> > <https://github.com/OpenImageIO/oiio/pull/3806>>
>> > This solves an issue where PSD files wider than 64k pixels per scanline
>> > would simply not correctly decode rle-compressed data. It was simply a
>> > case of a couple parameters declared as 16 bit data types that should have
>> > been 32 bits. I can't fathom why it was done that way, I think it was just
>> > a mistake that was never caught.
>>
>> > This allows iconvert, etc, to successfully copy the file, though it still
>> > uses a lot of memory.
>>
>> > I see now that the psd reader ALSO uses way too much memory, hanging onto
>> > memory it doesn't need to. I will try to address that separately. But at
>> > least now it works, if you have enough memory in the machine.
>> _______________________________________________
>> Oiio-dev mailing list
>> Oiio-dev@lists.openimageio.org <mailto:Oiio-dev@lists.openimageio.org>
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
> --
> Larry Gritz
> l...@larrygritz.com <mailto:l...@larrygritz.com>
>
>
>
>
>
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
l...@larrygritz.com
_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
