For us, the performance win is indeed generally that a single part can
be read individually, whereas its generally harder to 'cherrypick'
channels out of a 'flat' multichannel format For this reason (and the
ability to easily interrogate and deal with metadata in a single EXR
'container') we heavily favour multipart EXRs for frames that contain
'optional' or 'layered' elements over other formats.
Since a lot of stuff is more-or-less played directly off disk (its fast,
SAN disk, but still disk) being able to just read specific parts into
buffers for display just works better than having to read the data for a
huge number of channels and throw most of it away each frame.
-Pete
I'd be interested in any performance bottlenecks that people run into
with the multi-part path. One of the bigger wins we saw was with
stereo image pipelines where only one view was being asked for from
the file, a simple seek to the requested part vs a not an
insignificant amount of IO.
-Piotr
On 7 March 2016 at 12:25, Larry Gritz <[email protected]
<mailto:[email protected]>> wrote:
I want to add that while I'm happy to add this flexibility to the
OIIO API, I make NO GUARANTEES about libIlmImf (or any other
format libraries) performance with random part access. I don't
know how it buffers internally or whether keeping 200 parts alive
in an open file will result in massive memory allocation internal
to the library that I can't control.
-- lg
> On Mar 7, 2016, at 12:18 PM, Larry Gritz <[email protected]
<mailto:[email protected]>> wrote:
>
> Thanks for the additional input, Pete.
>
> So is the current proposal is:
>
> * ImageOutput::supports("random_subimage_access") will return
true for formats that support writing to subimages other than the
currently designated one (set by the last open() call).
>
> * Add a variant of ImageOutput::write_tiles that accepts an "int
subimage" parameter to specify which subimage the tile belongs to.
>
> Is that sufficient to meet your needs?
>
>
>
>> On Mar 7, 2016, at 12:10 PM, Pete Black <[email protected]
<mailto:[email protected]>> wrote:
>>
>> Hi Larry,
>>
>> I personally can see a need to write entire image parts in a
more flexible fashion, rather than tiles - I cant talk about the
details on-list, but recently I had to deal with stereo pairs of
EXRS each with 400+ channels, and selectively output multipart
stereo EXRs with 200+ parts per frame - I could do the job with
OIIO, but having more flexible APIs for multipart would be nice.
>>
>> Thanks
>>
>> -Pete
>>
>>
>>> On 8/03/2016, at 8:48 AM, Larry Gritz <[email protected]
<mailto:[email protected]>> wrote:
>>>
>>> Wow, I guess I never fully realized that OpenEXR's API allowed
for this! OpenEXR's addition of "multi-part" was a latecomer,
compared to TIFF which has allowed multiple images in a file, but
not random read or write access to them, and I guess I never
reexamined those assumptions when OpenEXR added the ability.
>>>
>>> Yes, in light of that, I think it does make sense to extend
the API to have a variant of write_tiles that lets you specify the
subimage. Of course, since most image formats (such as TIFF) do
not allow this flexibility, apps will be cautioned to only use it
for output formats for which supports("random_subimage_access") is
true, otherwise it would be an error to write tiles to anything
but the currently-opened subimage.
>>>
>>> Do you think we need write_scanlines similarly modified?
write_image? Or is the practical use case for this always going to
be tiles?
>>>
>>> What about input? Do you think it's important to allow reading
of a tile from one part in a random access way, but without
needing a seek_subimage to set the current subimage?
>>>
>>> Let me think a bit about how to structure this. It seems to me
that this is going to have to be a master-only feature, since
adding it is going to break link compatibility with the existing
ImageOutput class API and vtable.
>>>
>>> -- lg
>>>
>>>
>>>> On Mar 7, 2016, at 3:38 AM, Ramon Montoya Vozmediano
<[email protected] <mailto:[email protected]>> wrote:
>>>>
>>>> Hi Larry,
>>>>
>>>> OIIO supports output to multipart EXR images, and they have
to be written out like this:
>>>>
>>>> open(multipart file with n parts)
>>>> write_tiles(...)
>>>>
>>>> open(...AppendSubimage)
>>>> write_tiles(...)
>>>>
>>>> open(...AppendSubimage)
>>>> write_tiles(...)
>>>>
>>>> close()
>>>>
>>>> So you write out the first subimage, then the next, etc...
>>>>
>>>> The motivation for this email is that with this setup, when
you are rendering, you need to keep all the tiles in memory before
they can be written out.
>>>>
>>>> OpenEXR itself lets you write out tiles for any part in any
order.
>>>>
>>>> So, when using the EXR API directly, a renderer could write
out tiles as soon as they are done, and the physical layout of the
multipart image would end up looking like this:
>>>>
>>>> [chunk offset index]
>>>> [part 0, data for tile (0,0)]
>>>> [part 1, data for tile (0,0)]
>>>> [part 2, data for tile (0,0)]
>>>> ...
>>>> [part 0, data for tile (0,1)]
>>>> [part 1, data for tile (0,1)]
>>>> [part 2, data for tile (0,1)]
>>>> ...
>>>>
>>>> It looks like supporting this means extending OIIO's API, so
that you can set the part you want to write to, without having to
close/open a "subimage".
>>>>
>>>> Another way to look at this conceptually, is to consider an
EXR part as a channel grouping rather than a tiff style subimage.
>>>>
>>>> Have you considered extending OIIO's API in this direction?
>>>>
>>>> Best,
>>>>
>>>> r
>>>>
>>>
>>> --
>>> Larry Gritz
>>> [email protected] <mailto:[email protected]>
>>>
>>>
>>> _______________________________________________
>>> Oiio-dev mailing list
>>> [email protected]
<mailto:[email protected]>
>>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>>
>> _______________________________________________
>> Oiio-dev mailing list
>> [email protected]
<mailto:[email protected]>
>> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
>
> --
> Larry Gritz
> [email protected] <mailto:[email protected]>
>
>
> _______________________________________________
> Oiio-dev mailing list
> [email protected]
<mailto:[email protected]>
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
[email protected] <mailto:[email protected]>
_______________________________________________
Oiio-dev mailing list
[email protected] <mailto:[email protected]>
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
_______________________________________________
Oiio-dev mailing list
[email protected]
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
_______________________________________________
Oiio-dev mailing list
[email protected]
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
