Part 2 of this effort was contained in this patch: https://github.com/OpenImageIO/oiio/pull/3807 <https://github.com/OpenImageIO/oiio/pull/3807>
In that one, I did some refactoring of the PSD reading which I found was basically holding an entire extra copy of the image in the PSDInput itself, so... essentially doubling the memory footprint of reading a PSD. And part 3 in https://github.com/OpenImageIO/oiio/pull/3829, currently under review, addresses efficiency (and performance statistics) of the make_texture process for large images. After those patches above, I was able to read your whole enormous psd file if I give oiiotool some extra hints: oiiotool -v -i:type=uint8:now=1 0001-F_3a_Left.psb -otex:forcefloat=0 out.tx I'm using -i explicitly instead of just naming the file, so that I can take advantage of a couple of optional modifiers that control the read process: now=1 causes the read to happen fully and immediately, forcing it to avoid using the ImageCache, which it would normally do for large files to limit memory use and read lazily. Since making a texture requires the whole thing to be in memory at once, there is no point to using IC in this case and really hurts performance (since the image is so much larger than cache size) and the last thing you want is for an extra copy to be stored in the IC. type=uint8 Forces oiiotool to read into a uint8 buffer, which is the minimal size it can be since we know that as a PSD file, it starts off as uint8 data in the file. This is to combat oiiotool's default behavior of reading everything into internal float buffers for maximum precision and performance of math operations. But in this case, it's worth paying the penalty of uint8 -> float math -> uint8 conversions on the fly to keep the in-memory representation from exploding by 4x (32 bit float instead of 8 bit int per channel datum). And there's a special modifier on -otex as well: forcefloat=0 Disables the usual behavior of converting to a float buffer and using float intermediate buffers when down-resing to generate MIP levels, instead keeping it in the native format (in this case, int8, thus avoiding the 4x memory expansion, at the expense of some time and probably worse math precision for the downsize). With those patches applied, the above command successfully creates a texture from your psb file on my MacBookPro with 32GB RAM, though it takes about 1/2 hour! I think that's because it's swapping between memory and disk constantly. I also tried on my work machine with oodles of memory, and it takes 12 minutes, most of which appears to be actually reading the input file and writing the resulting texture to disk. There's still lots of room for improvement, but it basically works now. > On Apr 27, 2023, at 12:39 PM, Jens Olsson > <jens.gustaf.johan.ols...@gmail.com> wrote: > > Hey, that's great news! Thanks for looking into it, looking forward to > testing the fix! > (Still not getting anything from the mailing list to my gmail, so manual > copy-paste reply again) > > Cheers, > Jens > > Sorry it took me a while to get to this, but I have the first part of the > > solution here: https://github.com/OpenImageIO/oiio/pull/3806 > > <https://github.com/OpenImageIO/oiio/pull/3806> > > <https://github.com/OpenImageIO/oiio/pull/3806 > > <https://github.com/OpenImageIO/oiio/pull/3806>> > > This solves an issue where PSD files wider than 64k pixels per scanline > > would simply not correctly decode rle-compressed data. It was simply a case > > of a couple parameters declared as 16 bit data types that should have been > > 32 bits. I can't fathom why it was done that way, I think it was just a > > mistake that was never caught. > > > This allows iconvert, etc, to successfully copy the file, though it still > > uses a lot of memory. > > > I see now that the psd reader ALSO uses way too much memory, hanging onto > > memory it doesn't need to. I will try to address that separately. But at > > least now it works, if you have enough memory in the machine. > _______________________________________________ > Oiio-dev mailing list > Oiio-dev@lists.openimageio.org > http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org -- Larry Gritz l...@larrygritz.com
_______________________________________________ Oiio-dev mailing list Oiio-dev@lists.openimageio.org http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
