Ahh got it. I thought exrheader would explicitly say "zip, individual scanlines" next to those parts and not "dwa, small scanline". blocks". Is there a way to introspect the compression for each part that dwaa is doing auto-magically?
Richard: sorry for hijacking your thread 😈 It appears a lot of this is explained in the header for ImfDwaCompressor.cpp. Most of the time the top of the EXR source is just copy-write data and basic comments so I didn't realize there as a whole book chapter in there on the topic. 😀 On Mon, Mar 28, 2016 at 4:43 PM, Karl Rasche <karlras...@gmail.com> wrote: > > Ahh; I see what you are saying. > > The filtering we're talking about happens internally when you set the > compressor = DWAA or DWAB. You won't see the affect in exrheader. > > So in your example there are multiple parts, all set to DWAB. But only > part 3 will be lossy compressed, and of that, only the A channel. > > Parts 0-2 will be lossless. > > Karl > > On Mon, Mar 28, 2016 at 4:35 PM, Deke Kincaid <dekekinc...@gmail.com> > wrote: > >> The channel filtering you and Piotr mentioned as being automatic does not >> occur. Every part whether it has XYZ, UV, Z, etc... is written with dwa >> compression. >> >> On Mon, Mar 28, 2016 at 4:29 PM, Karl Rasche <karlras...@gmail.com> >> wrote: >> >>> >>> So.. what's the issue at hand then? >>> >>> On Mon, Mar 28, 2016 at 4:17 PM, Deke Kincaid <dekekinc...@gmail.com> >>> wrote: >>> >>>> They must be implementing something incorrectly then. Below is a >>>> truncated exrheader on a file out of Nuke with the "interleave" knob set to >>>> "channels" which means each layer is put into a separate part.. >>>> >>>> file testMultiPartChannelsdwaa.exr: >>>> >>>> file format version: 2, flags 0x1000 >>>> >>>> part 0: >>>> channels (type chlist): >>>> x, 16-bit floating-point, sampling 1 1 >>>> y, 16-bit floating-point, sampling 1 1 >>>> z, 16-bit floating-point, sampling 1 1 >>>> chunkCount (type int): 34 >>>> compression (type compression): dwa, small scanline blocks >>>> dataWindow (type box2i): (0 240) - (2047 1315) >>>> displayWindow (type box2i): (0 0) - (2047 1555) >>>> dwaCompressionLevel (type float): 45 >>>> lineOrder (type lineOrder): increasing y >>>> name (type string): "N.left" >>>> nuke/full_layer_names (type int): 0 >>>> nuke/node_hash (type string): "368b9d1b87a4b787" >>>> nuke/version (type string): "9.0v8" >>>> pixelAspectRatio (type float): 1 >>>> screenWindowCenter (type v2f): (0 0) >>>> screenWindowWidth (type float): 1 >>>> type (type string): "scanlineimage" >>>> version (type int): 1 >>>> view (type string): "left" >>>> >>>> part 1: >>>> channels (type chlist): >>>> x, 16-bit floating-point, sampling 1 1 >>>> y, 16-bit floating-point, sampling 1 1 >>>> z, 16-bit floating-point, sampling 1 1 >>>> chunkCount (type int): 34 >>>> compression (type compression): dwa, small scanline blocks >>>> dataWindow (type box2i): (0 240) - (2047 1315) >>>> displayWindow (type box2i): (0 0) - (2047 1555) >>>> dwaCompressionLevel (type float): 45 >>>> lineOrder (type lineOrder): increasing y >>>> name (type string): "P.left" >>>> pixelAspectRatio (type float): 1 >>>> screenWindowCenter (type v2f): (0 0) >>>> screenWindowWidth (type float): 1 >>>> type (type string): "scanlineimage" >>>> version (type int): 1 >>>> view (type string): "left" >>>> >>>> part 2: >>>> channels (type chlist): >>>> Z, 16-bit floating-point, sampling 1 1 >>>> chunkCount (type int): 34 >>>> compression (type compression): dwa, small scanline blocks >>>> dataWindow (type box2i): (0 240) - (2047 1315) >>>> displayWindow (type box2i): (0 0) - (2047 1555) >>>> dwaCompressionLevel (type float): 45 >>>> lineOrder (type lineOrder): increasing y >>>> name (type string): "depth.left" >>>> pixelAspectRatio (type float): 1 >>>> screenWindowCenter (type v2f): (0 0) >>>> screenWindowWidth (type float): 1 >>>> type (type string): "scanlineimage" >>>> version (type int): 1 >>>> view (type string): "left" >>>> >>>> part 3: >>>> channels (type chlist): >>>> u, 16-bit floating-point, sampling 1 1 >>>> v, 16-bit floating-point, sampling 1 1 >>>> chunkCount (type int): 34 >>>> compression (type compression): dwa, small scanline blocks >>>> dataWindow (type box2i): (0 240) - (2047 1315) >>>> displayWindow (type box2i): (0 0) - (2047 1555) >>>> dwaCompressionLevel (type float): 45 >>>> lineOrder (type lineOrder): increasing y >>>> name (type string): "motion.left" >>>> pixelAspectRatio (type float): 1 >>>> screenWindowCenter (type v2f): (0 0) >>>> screenWindowWidth (type float): 1 >>>> type (type string): "scanlineimage" >>>> version (type int): 1 >>>> view (type string): "left" >>>> >>>> part 4: >>>> channels (type chlist): >>>> A, 16-bit floating-point, sampling 1 1 >>>> B, 16-bit floating-point, sampling 1 1 >>>> G, 16-bit floating-point, sampling 1 1 >>>> R, 16-bit floating-point, sampling 1 1 >>>> chunkCount (type int): 34 >>>> compression (type compression): dwa, small scanline blocks >>>> dataWindow (type box2i): (0 240) - (2047 1315) >>>> displayWindow (type box2i): (0 0) - (2047 1555) >>>> dwaCompressionLevel (type float): 45 >>>> lineOrder (type lineOrder): increasing y >>>> name (type string): "rgba.left" >>>> pixelAspectRatio (type float): 1 >>>> screenWindowCenter (type v2f): (0 0) >>>> screenWindowWidth (type float): 1 >>>> type (type string): "scanlineimage" >>>> version (type int): 1 >>>> view (type string): "left" >>>> >>>> >>>> >>>> On Mon, Mar 28, 2016 at 3:16 PM, Piotr Stanczyk < >>>> piotr.stanc...@gmail.com> wrote: >>>> >>>>> +1 .. there is nothing that the client code really has to do. It's >>>>> purely based off the channel names. >>>>> >>>>> -Piotr >>>>> >>>>> >>>>> On 28 March 2016 at 15:05, Karl Rasche <karlras...@gmail.com> wrote: >>>>> >>>>>> Deke - >>>>>> >>>>>> All the channel rule setup should happen automagically; there isn't >>>>>> anything for Nuke to setup. >>>>>> >>>>>> That said, if your channels are all named RGB going into the write >>>>>> node, they'll all be handled the same by the compressor. >>>>>> >>>>>> Karl >>>>>> >>>>>> On Monday, March 28, 2016, Deke Kincaid <dekekinc...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Thanks for the info Karl. Currently Nuke doesn't follow these rules >>>>>>> and when you pick DWAA and just compresses all channels the same >>>>>>> instead of >>>>>>> selectively based on channel names. So we are having to write some >>>>>>> hackery >>>>>>> to work around it at the moment. >>>>>>> >>>>>>> I'll put this into my feature request/bug to the Foundry. >>>>>>> >>>>>>> On Mon, Mar 28, 2016 at 2:15 PM, Karl Rasche <karlras...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> This won't help on the documentation front, but for your question >>>>>>>> about how channels are handled... >>>>>>>> >>>>>>>> This is setup in initializeChannelRules() >>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3120>. >>>>>>>> That logic is basically saying: >>>>>>>> >>>>>>>> >>>>>>>> - If the channel name is A, use a lossless RLE compression on it >>>>>>>> - If the channel name is {Y, BY, RY}, and the data is {FLOAT, >>>>>>>> HALF}, lossy compress the channel >>>>>>>> - If the channel name is {R, G, B}, and the data is {FLOAT, >>>>>>>> HALF}, lossy compress the channels. >>>>>>>> - If we have R, G, and B channels, convert to a color >>>>>>>> difference format prior to quantization. Otherwise, deal with the >>>>>>>> channels >>>>>>>> independently as-is. >>>>>>>> - Otherwise, use a lossless compression on the data >>>>>>>> >>>>>>>> So channels named {y, u, U, v, V} should not be lossy compressed. >>>>>>>> >>>>>>>> The rule set that Jonathan is referring to is in >>>>>>>> initializeLegacyChannelRules() >>>>>>>> <https://github.com/openexr/openexr/blob/master/OpenEXR/IlmImf/ImfDwaCompressor.cpp#L3149>, >>>>>>>> and are slightly different from what 2.2 does by default. >>>>>>>> >>>>>>>> Most of this is hold-over from the time before multi-part files >>>>>>>> allowed more control over compression on different channel sets. >>>>>>>> >>>>>>>> Karl >>>>>>>> >>>>>>>> On Mon, Mar 28, 2016 at 1:56 PM, Richard Hadsell < >>>>>>>> hads...@blueskystudios.com> wrote: >>>>>>>> >>>>>>>>> There has been discussion on the [nuke-prerelease] e-mail list >>>>>>>>> about DWAA and DWAB compression with OpenExr 2.2. I checked the >>>>>>>>> OpenExr >>>>>>>>> Technical Introduction, but was disappointed to see that it has not >>>>>>>>> been >>>>>>>>> updated since November 2013. Is there any chance that someone is >>>>>>>>> working >>>>>>>>> on an update? >>>>>>>>> >>>>>>>>> It is not difficult to use DWAA and DWAB compressions, which are >>>>>>>>> available in the compression header. That is not the problem. The >>>>>>>>> questions that I would like to see answered pertain more to lossy vs. >>>>>>>>> lossless compression. The [nuke-prerelease] discussion included >>>>>>>>> interesting information like this: >>>>>>>>> >>>>>>>>> On Thu, Mar 17, 2016 at 1:03 PM, Jonathan Egstad < >>>>>>>>> jonathan.egs...@dreamworks.com> wrote: >>>>>>>>> ... >>>>>>>>> > 3. Does the lossyness cause issues with data aov's, motion >>>>>>>>> vectors, uv, position, normals, etc...? >>>>>>>>> >>>>>>>>> Yup - it will destroy data AOVs like normals, positions, etc. I >>>>>>>>> believe the codec recognizes typical color-channel names like >>>>>>>>> 'r/R/red/RED', etc and only lossy compresses those. Channels it >>>>>>>>> doesn't >>>>>>>>> recognize will get lossless compressed with I believe ZIP - alpha for >>>>>>>>> example is lossless compressed. >>>>>>>>> One gotcha is I think it traps 'Y/y, u/U, v/V' as valid color >>>>>>>>> channels and lossy-compresses those, which is a problem for xyz and uv >>>>>>>>> channels... >>>>>>>>> This is how our internal codec works so you might want to check >>>>>>>>> that the one in the official OpenEXR2.2+ release has these behaviors. >>>>>>>>> ... >>>>>>>>> (end quote) >>>>>>>>> >>>>>>>>> It would be great to see a list of those special channel names. >>>>>>>>> Are they the only channels that are compressed with DWAA/DWAB? What >>>>>>>>> compression would be applied to the other channels? Does this >>>>>>>>> behavior >>>>>>>>> apply to all lossy compressions? >>>>>>>>> >>>>>>>>> I hope to see answers to these questions in an updated Technical >>>>>>>>> Intro (eventually). >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Dick Hadsell 203-992-6320 Fax: 203-992-6001 >>>>>>>>> Reply-to: hads...@blueskystudios.com >>>>>>>>> Blue Sky Studios http://www.blueskystudios.com >>>>>>>>> 1 American Lane, Greenwich, CT 06831-2560 >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Openexr-devel mailing list >>>>>>>>> Openexr-devel@nongnu.org >>>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Openexr-devel mailing list >>>>>>>> Openexr-devel@nongnu.org >>>>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel >>>>>>>> >>>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Openexr-devel mailing list >>>>>> Openexr-devel@nongnu.org >>>>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel >>>>>> >>>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Openexr-devel mailing list >>>> Openexr-devel@nongnu.org >>>> https://lists.nongnu.org/mailman/listinfo/openexr-devel >>>> >>>> >>> >> >> _______________________________________________ >> Openexr-devel mailing list >> Openexr-devel@nongnu.org >> https://lists.nongnu.org/mailman/listinfo/openexr-devel >> >> >
_______________________________________________ Openexr-devel mailing list Openexr-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/openexr-devel
