Heya,
as far as i understood the original question it was specifically about multipart images. There is no doubt that non-multipart monolithic files are WAY WAY slower (if you do not use all passes that is). Multipart is supposed to get rid of that limitation though. It DOES re-add the limitations that scanline interleaving was supposed to solve in the first place (allowing per scanline access to parts of a scanline to allow for highly efficient ROI access). That being said the tests i have been doing so far were done using the multipart converter included in the openexr samples. Nuke can write them too, but Nuke has its own set of limitations for the EXR spec (e.g. not being able to write different bitdepth per channel). These files are a LOT faster in our use cases. I have however not done proper benchmarking yet let alone for your default VFX scenario (we are relying on multipart in some very specific use cases). Cheers, Thorsten --- Thorsten Kaufmann Production Pipeline Architect Mackevision Medien Design GmbH Forststraße 7 70174 Stuttgart T +49 711 93 30 48 606 F +49 711 93 30 48 90 M +49 151 19 55 55 02 [email protected] www.mackevision.com Geschäftsführer: Armin Pohl, Joachim Lincke HRB 243735 Amtsgericht Stuttgart --- VFX: Game of Thrones, Season 5 - VFX making of reel<https://vimeo.com/133433110>. TWITTER | ADOBE BEHANCE: Follow us on Twitter<https://twitter.com/Mackevision> and Adobe Behance<https://www.behance.net/mackevision>. ________________________________ Von: [email protected] <[email protected]> im Auftrag von Howard Jones <[email protected]> Gesendet: Samstag, 12. September 2015 11:07 An: Nuke user discussion Betreff: Re: [Nuke-users] Multipart multichannel exrs vs. separate exrs That is a good point about which layers are left unused, however the time lag in not including these passes, then requesting a pass that could have been included, waiting for it would have a very negative impact on a fast turn around project. Far more than having them available anyway. On the other point If i needed an extra pass I would expect it to be as a separate file regardless of the original delivery. Howard On 12 Sep 2015, at 9:55 am, Elias Ericsson Rydberg <[email protected]<mailto:[email protected]>> wrote: As a 3D lighter I prefer having it split into multiple files. Sure it's nice to deliver one big exr to comp because then you know all the available channels will show up in nuke. Easy and quick for compositor to bring into the app. Neat. But I see a few drawbacks, at least with the way the compositors used the resulting passes. Some would just use the beauty pass and then maybe add/subtract some lighting passes or use a mask or two. Making the majority of the passes redundant But since they were still included, because they might be nice to have, they mostly just caused a strain on the network. Albeit this was a few years ago, and exr might have improved since. But even so, when the compositor requested a new special pass. Like a second more broad ambient occlusion pass. It became a bit of a hassle to copy that one pass into the previously delivered main file in nuke and then write out a new main file. I know this I deviating from the original topic, which is faster, multipart or monolithic? And the answer is of course "it depends". Does the compositor use all the passes and does the monolith only include the very essentials then that may be the faster route. Do you include a lot of masks and special utility passes, then size of the file grows and the load on the network unnecessary. Excuse the long post, and for possibly just stating the obvious. Den 12 sep 2015 2:06 fm skrev "Randy Little" <[email protected]<mailto:[email protected]>>: was Just saying if someone would like I think there has been in depth discussion about it I believe. Wasn't meaning to contradict you if it came off like that. 3 weeks no days off shortest day 13 hours brain left last week. Randy S. Little http://www.rslittle.com/ http://www.imdb.com/name/nm2325729/ On Fri, Sep 11, 2015 at 4:37 PM, Nathan Rusch <[email protected]<mailto:[email protected]>> wrote: Yup, and that's part of what I said. I just wanted to point out that multiple file sequences *does* introduce its own form of overhead, but it's still almost always the better option. From: Randy Little<mailto:[email protected]> Sent: Friday, September 11, 2015 4:29 PM To: Nuke user discussion<mailto:[email protected]> Subject: Re: [Nuke-users] Multipart multichannel exrs vs. separate exrs Can't speak to multipart at all but there has always been discussion on the list about monolithic multichannel files not being faster in traditional networked workflows. I"m sure I can find the archives and repost those results. (pretty sure actually data) Randy S. Little http://www.rslittle.com/ http://www.imdb.com/name/nm2325729/ On Fri, Sep 11, 2015 at 3:38 PM, Nathan Rusch <[email protected]<mailto:[email protected]>> wrote: It has to *open* the file, yes, but opening does not imply reading. One of the main points of a multi-part file is to allow discrete access to each part. As far as I know, hardly any renderers currently support writing multi-part files though, and your 3D guys may not know how to set them up even if they could, so chances are you're looking at 1.x files. To partially echo Thorsten, I think in theory multi-part files should be faster than separate file sequences (I say "in theory" because I haven't done any benchmarking of my own). Multiple file sequences generally beat monolithic single-part (i.e. multi-layer EXR 1.x) files because of Nuke's selective channel access behavior. In other words, you are rarely reading all of your AOVs at the same time and same place in the node tree, but you are still paying the decompression/read cost for all of them whenever one is read. Incremental improvements to Nuke's caching system may have improved the situation with mutli-layer 1.x files somewhat though. On the other side, using multiple sequences trades the "read-and-decompress-everything" requirement for the overhead of requiring Nuke to at least open and read each file's header in order to determine what channels are available (regardless of what channels are actually being read). If you use network storage like most people, this overhead can increase quite a bit due to latency, cluster cache eviction, etc. However, it's still generally better than the alternative. On paper, multi-part files should ideally give you the best of both worlds: encapsulation alongside piecemeal data access. To inspect the available channels, Nuke only needs to open a single file, though it will still need to read the header for each part (I'm assuming there's a jump table at the top of the file, but I'm not sure). You can then also seek to and read the data for any one layer without touching data for any others (assuming each is stored in its own part). This relies on smart reader plugin design on the Nuke side, as well as support for parallel access to different parts in the same file. -Nathan From: Randy Little<mailto:[email protected]> Sent: Friday, September 11, 2015 2:57 PM To: Nuke user discussion<mailto:[email protected]> Subject: Re: [Nuke-users] Multipart multichannel exrs vs. separate exrs Has to load the file to read it always. Can't just load part of a little endian file pretty sure. I know here I can load a beauty pass and scrub it and play around with it and do all kinds of fun things. When some 3d guy decides to put 50 passes (10-15) it take longer to load and they are don't scrub. Randy S. Little http://www.rslittle.com/ http://www.imdb.com/name/nm2325729/ On Fri, Sep 11, 2015 at 1:53 PM, Thorsten Kaufmann <[email protected]<mailto:[email protected]>> wrote: This is not really true with multipart EXR files anymore. If files are interleaved by layer rathern than by scanline you loose the benefits but gain performance compared to multiple sequences. I have yet to do a full performance check but the boost seems to be rather big if you do not use all channels (if you use all channels it should not matter anyways, should it). I expect multichannel to still have some overhead maybe, but given the downsides of splitting exrs (too many open file handles anyone?) This is a use case decicsion i would say. Cheers, Thorsten --- Thorsten Kaufmann Production Pipeline Architect Mackevision Medien Design GmbH Forststraße 7 70174 Stuttgart T +49 711 93 30 48 606<tel:%2B49%20711%2093%2030%2048%20606> F +49 711 93 30 48 90<tel:%2B49%20711%2093%2030%2048%2090> M +49 151 19 55 55 02<tel:%2B49%20151%2019%2055%2055%2002> [email protected]<mailto:[email protected]> www.mackevision.com<http://www.mackevision.com> Geschäftsführer: Armin Pohl, Joachim Lincke HRB 243735 Amtsgericht Stuttgart ________________________________ Von:[email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> im Auftrag von Randy Little <[email protected]<mailto:[email protected]>> Gesendet: Freitag, 11. September 2015 22:06 An: Nuke user discussion Betreff: Re: [Nuke-users] Multipart multichannel exrs vs. separate exrs big huge all passes in a exr is slow. Has to read entire file (75MB+) to find the channel you want that might be 200KB every single time you want to deal with that little mask channel. They also seem to take longer to render out of 3d. We usually break them up by math type and try not to stuff 20 passes into a file. Randy S. Little http://www.rslittle.com/ http://www.imdb.com/name/nm2325729/ On Fri, Sep 11, 2015 at 2:31 AM, Johannes Hezer <[email protected]<mailto:[email protected]>> wrote: Hey everyone, A bit of a survey question... Is anyone using multipart exrs ? We have been using them for 2 projects and we have not done any performance profiling, but I am not 100% sure if it is a speed bost or not, compared to multichannel 1.xxx exrs. I was hoping might come close to splitted exrs (each layer in a single file) ... Looking forward to some input Cheers Johannes Von meinem iPhone gesendet ____ ESET 12236 (20150911) ____ The message was checked by ESET Mail Security. _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users ________________________________ _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users ________________________________ _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users _______________________________________________ Nuke-users mailing list [email protected]<mailto:[email protected]>,http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
_______________________________________________ Nuke-users mailing list [email protected], http://forums.thefoundry.co.uk/ http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
