As a 3D lighter I prefer having it split into multiple files.
Sure it's nice to deliver one big exr to comp because then you
know all the available channels will show up in nuke. Easy and
quick for compositor to bring into the app. Neat.
But I see a few drawbacks, at least with the way the compositors
used the resulting passes. Some would just use the beauty pass
and then maybe add/subtract some lighting passes or use a mask or
two. Making the majority of the passes redundant But since they
were still included, because they might be nice to have, they
mostly just caused a strain on the network.
Albeit this was a few years ago, and exr might have improved
since. But even so, when the compositor requested a new special
pass. Like a second more broad ambient occlusion pass. It became
a bit of a hassle to copy that one pass into the previously
delivered main file in nuke and then write out a new main file.
I know this I deviating from the original topic, which is faster,
multipart or monolithic? And the answer is of course "it
depends". Does the compositor use all the passes and does the
monolith only include the very essentials then that may be the
faster route. Do you include a lot of masks and special utility
passes, then size of the file grows and the load on the network
unnecessary.
Excuse the long post, and for possibly just stating the obvious.
Den 12 sep 2015 2:06 fm skrev "Randy Little"
<randyslit...@gmail.com <mailto:randyslit...@gmail.com>>:
was Just saying if someone would like I think there has been
in depth discussion about it I believe. Wasn't meaning to
contradict you if it came off like that. 3 weeks no days off
shortest day 13 hours brain left last week.
Randy S. Little
http://www.rslittle.com/
http://www.imdb.com/name/nm2325729/
On Fri, Sep 11, 2015 at 4:37 PM, Nathan Rusch
<nathan_ru...@hotmail.com <mailto:nathan_ru...@hotmail.com>>
wrote:
Yup, and that's part of what I said. I just wanted to
point out that multiple file sequences *does* introduce
its own form of overhead, but it's still almost always
the better option.
*From:*Randy Little <mailto:randyslit...@gmail.com>
*Sent:* Friday, September 11, 2015 4:29 PM
*To:*Nuke user discussion
<mailto:nuke-users@support.thefoundry.co.uk>
*Subject:* Re: [Nuke-users] Multipart multichannel exrs
vs. separate exrs
Can't speak to multipart at all but there has always been
discussion on the list about monolithic multichannel
files not being faster in traditional networked
workflows. I"m sure I can find the archives and repost
those results. (pretty sure actually data)
Randy S. Little
http://www.rslittle.com/
http://www.imdb.com/name/nm2325729/
On Fri, Sep 11, 2015 at 3:38 PM, Nathan Rusch
<nathan_ru...@hotmail.com
<mailto:nathan_ru...@hotmail.com>> wrote:
It has to *open* the file, yes, but opening does not
imply reading.
One of the main points of a multi-part file is to
allow discrete access to each part. As far as I know,
hardly any renderers currently support writing
multi-part files though, and your 3D guys may not
know how to set them up even if they could, so
chances are you're looking at 1.x files.
To partially echo Thorsten, I think in theory
multi-part files should be faster than separate file
sequences (I say "in theory" because I haven't done
any benchmarking of my own).
Multiple file sequences generally beat monolithic
single-part (i.e. multi-layer EXR 1.x) files because
of Nuke's selective channel access behavior. In other
words, you are rarely reading all of your AOVs at the
same time and same place in the node tree, but you
are still paying the decompression/read cost for all
of them whenever one is read. Incremental
improvements to Nuke's caching system may have
improved the situation with mutli-layer 1.x files
somewhat though.
On the other side, using multiple sequences trades
the "read-and-decompress-everything" requirement for
the overhead of requiring Nuke to at least open and
read each file's header in order to determine what
channels are available (regardless of what channels
are actually being read). If you use network storage
like most people, this overhead can increase quite a
bit due to latency, cluster cache eviction, etc.
However, it's still generally better than the
alternative.
On paper, multi-part files should ideally give you
the best of both worlds: encapsulation alongside
piecemeal data access. To inspect the available
channels, Nuke only needs to open a single file,
though it will still need to read the header for each
part (I'm assuming there's a jump table at the top of
the file, but I'm not sure). You can then also seek
to and read the data for any one layer without
touching data for any others (assuming each is stored
in its own part). This relies on smart reader plugin
design on the Nuke side, as well as support for
parallel access to different parts in the same file.
-Nathan
*From:*Randy Little <mailto:randyslit...@gmail.com>
*Sent:* Friday, September 11, 2015 2:57 PM
*To:*Nuke user discussion
<mailto:nuke-users@support.thefoundry.co.uk>
*Subject:* Re: [Nuke-users] Multipart multichannel
exrs vs. separate exrs
Has to load the file to read it always. Can't just
load part of a little endian file pretty sure. I
know here I can load a beauty pass and scrub it and
play around with it and do all kinds of fun things.
When some 3d guy decides to put 50 passes (10-15) it
take longer to load and they are don't scrub.
Randy S. Little
http://www.rslittle.com/
http://www.imdb.com/name/nm2325729/
On Fri, Sep 11, 2015 at 1:53 PM, Thorsten Kaufmann
<thorsten.kaufm...@mackevision.com
<mailto:thorsten.kaufm...@mackevision.com>> wrote:
This is not really true with multipart EXR files
anymore. If files are interleaved by layer
rathern than by scanline you loose the benefits
but gain performance compared to multiple
sequences. I have yet to do a full performance
check but the boost seems to be rather big if you
do not use all channels (if you use all channels
it should not matter anyways, should it).
I expect multichannel to still have some overhead
maybe, but given the downsides of splitting exrs
(too many open file handles anyone?) This is a
use case decicsion i would say.
Cheers,
Thorsten
---
Thorsten Kaufmann
Production Pipeline Architect
Mackevision Medien Design GmbH
Forststraße 7
70174 Stuttgart
T +49 711 93 30 48 606
<tel:%2B49%20711%2093%2030%2048%20606>
F +49 711 93 30 48 90
<tel:%2B49%20711%2093%2030%2048%2090>
M +49 151 19 55 55 02
<tel:%2B49%20151%2019%2055%2055%2002>
thorsten.kaufm...@mackevision.com
<mailto:thorsten.kaufm...@mackevision.com>
www.mackevision.com <http://www.mackevision.com>
Geschäftsführer: Armin Pohl, Joachim Lincke
HRB 243735 Amtsgericht Stuttgart
------------------------------------------------------------------------
*Von:*nuke-users-boun...@support.thefoundry.co.uk
<mailto:nuke-users-boun...@support.thefoundry.co.uk>
<nuke-users-boun...@support.thefoundry.co.uk
<mailto:nuke-users-boun...@support.thefoundry.co.uk>>
im Auftrag von Randy Little
<randyslit...@gmail.com
<mailto:randyslit...@gmail.com>>
*Gesendet:* Freitag, 11. September 2015 22:06
*An:* Nuke user discussion
*Betreff:* Re: [Nuke-users] Multipart
multichannel exrs vs. separate exrs
big huge all passes in a exr is slow. Has to read
entire file (75MB+) to find the channel you want
that might be 200KB every single time you want to
deal with that little mask channel. They also
seem to take longer to render out of 3d. We
usually break them up by math type and try not to
stuff 20 passes into a file.
Randy S. Little
http://www.rslittle.com/
http://www.imdb.com/name/nm2325729/
On Fri, Sep 11, 2015 at 2:31 AM, Johannes Hezer
<j.he...@studiorakete.de
<mailto:j.he...@studiorakete.de>> wrote:
Hey everyone,
A bit of a survey question... Is anyone using
multipart exrs ?
We have been using them for 2 projects and we
have not done any performance profiling, but
I am not 100% sure if it is a speed bost or
not, compared to multichannel 1.xxx exrs.
I was hoping might come close to splitted
exrs (each layer in a single file) ...
Looking forward to some input
Cheers
Johannes
Von meinem iPhone gesendet
____ ESET 12236 (20150911) ____
The message was checked by ESET Mail Security.
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
------------------------------------------------------------------------
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
------------------------------------------------------------------------
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
_______________________________________________
Nuke-users mailing list
Nuke-users@support.thefoundry.co.uk
<mailto:Nuke-users@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-users
