Good answer Asi, got a rant here below, hope it helps ( got too much time on my hands over the holidays apparently)
Asi wrote: After some rethinking I changed my approach a little, reason is,EXR have a good support for long/custom metadata but others don't have limited support, DPX for example, which we use as well. Cool, I’m not sure about how awesome exr is when it comes to long/custom metadata, it’s better than dpx true, don’t get me wrong, the first time .exr was opened to the public I was so excited I learnt to write them out command line pixel by pixel, exr rocks beyond believe but the character limit can get annoying, have you gotten around the 31 character limit yet ( By your description I assume so, but sending the note your way so it doesn’t come as a surprise if it hits ) http://www.openexr.com/openexrfilelayout.pdf http://www.openexr.com/MultiViewOpenEXR.pdf http://lists.nongnu.org/archive/html/openexr-devel/2006-05/msg00009.html It gets better though: http://download.autodesk.com/global/docs/maya2014/en_us/index.html?url=files/Vari_Subfolders_and_names_of_rendered_images.htm,topicNumber=d30e685154 And with exr2 and all that limits are closer to the sky than ever before. Asi wrote: So instead of figuring out each format metadata support, I rather dump the metadata in a file or a Database, Good call, it’s what most studios do, less risk regarding a frame that's been sitting on the farm for 48 hours and such, however my attitude towards that has always been; a) Why don't those guys hire a render pipeline developer? a 48 hour render has not been needed since at least 5 years ago, it’s horribly risky and regeneration is near impossible in a tight deadline, trust me I’ve seen the biggest frames being shown in cinema these days go through massive farms and there is nothing in the entire world of games/film/commercials which would justify a pipeline where one falls to the grace of the processors and won’t know a thing until days later. b) If done right, this information belongs inside the actual frame as the base location, any database or file system info file should derive it’s information from the image header. It’s only sensible to include information about things inside things, of course! It’s only because most people who attempt this completely mess it up that I feel most decision makers rightfully choose the path which has burnt the least and additional databases, info files, excel sheets, and so forth will forever exist. Asi wrote: located where the images/movie are, Careful; [pro] Easy access to the data for everyone. [con] -You increase the number of files in the project directory, making for more convulated browsing, raising more questions to inherently inquisitive groups of people who really should rather be focusing on their modelling task at hand than be wondering about what the difference between this_shot.mb / this_shot.info / this_shot.dep files are. -One can actually foolhardily enough script enough files to get generated into there so it goes beyond 2000 files, loading times from a database environment turns into massive debug sessions to enhance the loading process when all that is happening is unix struggling due to sloppy coding, check out this link, it’s a really good read, suggests only 1020 files per directory which is an absolute most i.m.h.o http://www.frank4dd.com/howto/various/maxfiles-per-dir.htm Additional things to back this up below, it’s so crazy I still ran into people in 2013 hitting this limit 😊 http://www.linuxquestions.org/questions/general-10/maximum-number-of-files-in-a-directory-183643/ http://stackoverflow.com/questions/466521/how-many-files-in-a-directory-is-too-many http://serverfault.com/questions/59454/whats-the-maxium-number-of-files-a-unix-folder-can-hold It’s always good to have that stuff accessible but if it’s not inside the image file I’d say keep it away from the general staff ( tell them where it is if they are interested of course but don't complicate their workspace with metadata information they don’t need ) Asi wrote: I could have infinite amount of metadata without relying on each format ability to store metadata. this is doable in our environment since the rendered files won't get moved from the metadata file. It’s a good idea, the reason I’m getting into this in further detail is it’s getting to be a bit of an old sport. Works of course but history has proven requires massive amounts of human resources to support it. Alternates do exist out there, for example since you bring up the environment have you considered a context sensitive directory scheme? Roughly it goes { setenv this_x[1-nearinfinity] for levels of directory depth; that_[x-nearinfinity] for token in naming convention + non identifiable elements} , It’s a fun one, means you only ever have 1 configuration file that you drop in the root of projects and all metadata is actually generated into the environment based on where you are looking at it from ( hooks into cd on Unix and folder callbacks on Windows, change directory; your environment is different, this can include things all the way down to artist notes even, all still driven from a single algorithm dependant on who is reading it from where, some studios have a rudimentary scheme like this going on but I’ve never seen it fully utilized ) Then of course, you can always try out the Shotgun Pipeline Toolkit (Formerly Tank) which handles all this at a phenomenal level and since it’s so widespread these days you will be working with a scheme that is consistent across studios, I do think the top of the line with regards to technology we have at hand today is generating metadata into the rendered files ( internally by the renderer as you describe, not a post process, vray and Arnold have the ease of access nailed down ) then push that information towards a hosted Shotgun server as a centralized container. Must note I have no affiliations with Shotgun development beyond customer level but can verify after having used it since it’s first iteration then develop towards it as Tank became the pipeline toolkit it just makes everything so straight forward. Keep it in mind, Shotgun rocks for what you are after but mostly so if you manage to embed your metadata. http://www.shotgunsoftware.com/ Asi wrote: I still wish MR would have a built in hook to inject metadata :) but this workaround would help improve on other wider issues as well hehe, ok, I fold, just promise to be careful, mental ray picks up custom attributes based on a naming convention, as easy as it gets Experimental: if an attribute with name staring with "meta:" prefix is used (like attribute string "meta:my_key" "my_value"), the value is passed to the image file headers (i.e."my_key" "my_value" pair). This is currently only supported for OpenEXR images. Boolean parameters are converted to integer ones. http://docs.autodesk.com/MENTALRAY/2013/ENU/mental-ray-help/files/relnotes/relnotes.html Care to move on to compression? I can show you how to turn single render into something which halts a 50T server in a few seconds vs. renders without anyone noticing by only toggling a compression setting :) exr is awesome to play with, the only format which lets you actually just set that other setting and not lose over 30 hours a day of accumulated time only spent due to number of files in directories and compression schemes, without it the visual effects industry would not have expanded as it has done in the last few years, it’s a weird thing to claim I know but the efficiency of the format has allowed teams to increase their attentiveness to detail and volume in such a manner that the limitless environment allows for amazing things to be created, the learning curve regarding it though can be a bit tricky as it’s not like other formats. But hey! it doesn’t have to, it does it’s thing really well :) http://forum.nvidia-arc.com/showthread.php?9434-OpenExr-and-zip-scanline-compressions Have a nice holiday ahead hope your metadata adventures go well -aeVar -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/52b9014a.aacbc20a.5f76.0242%40mx.google.com. For more options, visit https://groups.google.com/groups/opt_out.
