and here is the file actually attached
On 18/12/13 17:36, Frank Rueter wrote:
Here is an updated version that implements a bunch of suggestions from
this thread.
This supports stereo and offers to use 1-4 concurrent threads for the
file copying.
This is ignoring the setting for disk cache size in the preferences,
so be clean up after yourself ;).
It'd be great to get some feedback on this so I can throw it up on
Nukepedia if it works for everybody.
Cheers,
frank
On 16/12/13 21:29, Frank Rueter wrote:
Hola everybody,
I had a quick look at this this morning and realised I would have to
re-write everything from scratch - but then couldn't resists :-D.
Could some of you test the attached file and tell me how you get on?
Just put it into your NUKE_PATH and put this into your menu.py:
import LocaliseThreaded
LocaliseThreaded.register()
This will replace the default localising behaviour with a threaded one.
The maximum threads are half of your Nuke threads (nuke.THREADS) at
the moment (I will make this a preference though). There will be info
about concurrent threads in the progress bar as it does it's thing.
It's work in progress at this stage but since my day is coming to an
end, I thought it would be good to get it out there for a test run,
so I know more in the morning.
Things I still need/want to do:
* support split file knobs for stereo projects
* support proxy knobs (should I, not sure if the default does?)
* refactor the code so that multiple threads can tackle the same
read node (at the moment one Read node is allocated one task)
* benchmark the copy function (currently shutil.copy2). Pretty sure
it's not the fastest one for large files and I might have to roll
my own to speed things up.
Let me know how it works, especially if you are on windows as I can't
test that here.
Cheers,
frank
P.S.: If somebody is an expert with python threading with a little
bit of time on their hands, get in touch, I'm pretty sure the way I'm
doing this can be optimised, especially for trying to dynamically
allocate threads to efficiently deal with outstanding tasks.
On 14/12/13 23:23, Howard Jones wrote:
I was wondering that. It was going to be my next question (honest)
Howard
On 14 Dec 2013, at 08:19, Thorsten Kaufmann<thorsten.kaufm...@mackevision.de>
wrote:
This also sounds like a job for "import nuke" no? ;)
Thorsten Kaufmann
Production Pipeline Architect
____________________________________
Mackevision Medien Design GmbH
Forststra?e 7
D-70174 Stuttgart
T T +49 711 93 30 48 78
F +49 711 93 30 48 90
M +49 151 19 55 55 02
thorsten.kaufm...@mackevision.de
http://www.mackevision.de
Gesch?ftsf?hrer: Armin Pohl, Joachim Lincke, Karin Suttheimer
HRB 243735 Amtsgericht Stuttgart
________________________________________
Von:nuke-python-boun...@support.thefoundry.co.uk
[nuke-python-boun...@support.thefoundry.co.uk] im Auftrag von Frank Rueter
[fr...@beingfrank.info]
Gesendet: Samstag, 14. Dezember 2013 01:11
An: Nuke Python discussion; Justin Fpc
Betreff: Re: [Nuke-python] nuke localise
I have wrote my own localising script from scratch just before this feature was
implemented. I will have a peek next week if I can quickly adapt it to use the
localising settings in the preferences and nodes. If so it will be threaded and
we should get the best of both worlds until the built in feature is more
flexible to allow background processing.
Sent with AquaMail for Android
http://www.aqua-mail.com
On 13 December 2013 9:58:43 PM Justin Fpc wrote:
Hi all,
I would be very interested if there is anyway to manage this localising in
background.
I've also tested to use the threading method and found the same problem/cause
as Frank.
Justin
2013/12/13 Howard Jones
<mrhowardjo...@yahoo.com<mailto:mrhowardjo...@yahoo.com>>
Thanks for testing. That would have stumped me.
I contacted support.
Howard
On 12 Dec 2013, at 23:48, Frank Rueter
<fr...@beingfrank.info<mailto:fr...@beingfrank.info>> wrote:
I remember now:
I tried this a while ago myself and failed because doLocalise() is a wrapper
function using nuke.localiseFiles which seems to be compiled.
Since nuke.localiseFiles takes care of the progress bar (presumably juggling
it's own threads) it's not just a matter of using
thread = threading.Thread(target=doLocalise, args=(True,))
thread.start()
or
thread = threading.Thread(target=nuke.localiseFiles, args=(readKnobList,))
thread.start()
Both the above do the job, but you won't get the progress bar and the main
thread is still blocked.
There might be a way but I don't know how, other than basically writing the
localisation logic yourself.
So best to push that feature request to make nuke.localiseFiles thread-able.
Cheers,
frank
On 13/12/13 12:15, Frank Rueter wrote:
Yes, you should be able to. I have a quick peek...
On 13/12/13 11:29, Howard Jones wrote:
Ok done. Out of interest can this be run in a separate thread? My python brain
hasn't got round threading, but i can run doLocalise(0) so could I thread it
instead?
Howard
On 12 Dec 2013, at 22:02, Frank
Rueter<fr...@beingfrank.info><mailto:fr...@beingfrank.info> wrote:
I have asked for this in the pas as well, so please bug support to up the
priority ;)
On 11/12/13 05:23, Howard Jones wrote:
Hi
Is it possible to run localise from a shell or in the background?
H
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
import errno
import filecmp
import nuke
import os
import threading
import time
from subprocess import call
if nuke.env['WIN32']:
# IMPORT WINDLL FOR FAST COPY UNDER WINDOWS
from ctypes import windll
class LocaliseThreaded(object):
'''
Mimic nuke.localiseFile but make it threaded so it can run in the background
WARNING:
This does not implement the preferences' disk cache size knob yet!!!
To install put this into your menu.py:
import LocaliseThreaded
LocaliseThreaded.register()
'''
def __init__(self, fileDict, maxThreads=1):
'''
Threaded interface for copying files
fileDict - dictionary where key is the name of the sequence (used for progress bar) and value is a list of files to be copied
'''
self.fileDict = fileDict
self.cachePath = nuke.value('preferences.localCachePath')
self.taskCount = len(self.fileDict)
self.totalFileCount = sum([len(v) for v in self.fileDict.values()])
self.progress = 0.0
self.cachePath = nuke.value('preferences.localCachePath')
self.finishedThreads = 0
self.threadLimit = maxThreads
self.threadLimiter = threading.BoundedSemaphore(self.threadLimit)
def start(self):
'''start copying files'''
self.start = time.time()
self.mainTask = nuke.ProgressTask('LOCALISING %s files' % self.totalFileCount)
self.__updateMainTaskMessage()
for seqName, fileList in self.fileDict.iteritems():
thread = threading.Thread(name=seqName, target=self.copyFiles, args=(seqName, fileList))
thread.start()
def copyFiles(self, taskName, fileList):
'''Copy all files'''
self.threadLimiter.acquire()
task = nuke.ProgressTask('%s (%s files)' % (taskName, len(fileList)))
for i, filePath in enumerate(fileList):
if task.isCancelled() or self.mainTask.isCancelled():
break
# COPY FILE
self.copyFile(filePath, self.getTargetDir(filePath))
# UPDATE LOCAL TASK
task.setMessage('localising %s' % filePath)
task.setProgress(int(float(i) / len(fileList) * 100))
# UPDATE GLOBAL TASK
self.progress += 1
self.mainTask.setProgress(int(self.progress / self.totalFileCount * 100))
self.reportFinishedThread()
self.threadLimiter.release()
def reportFinishedThread(self):
'''Used to update the main task message and invoke the indicator on all nodes after localisation finsishes'''
self.finishedThreads += 1
self.__updateMainTaskMessage()
if self.finishedThreads == self.taskCount:
self.__forceUpdate()
self.end = time.time()
#print 'localising took %s seconds' % (self.end - self.start)
def __updateMainTaskMessage(self):
self.mainTask.setMessage('%s/%s tasks' % (self.finishedThreads, self.taskCount))
def __forceUpdate(self):
'''Silly workaround to update the node indicators. node.update() doesn't do the trick'''
n = nuke.nodes.NoOp()
nuke.delete(n)
def copyFile(self, filePath, destPath):
'''
Copy filePath to destPath. destPath will be created if it doesn't exist.
filePath will not be copied if the file already exists in destPath unless the local copy has an older time stamp
'''
# CREATE TARGET DIR IF NEED BE
if not os.path.isdir(destPath):
os.makedirs(destPath)
maxTries = 5 # NUMBER OF COPY ATTEMPTS IF STALE NFS HANDLE IS ENCOUNTERED
localFile = os.path.join(destPath, os.path.basename(filePath))
if os.path.isfile(localFile) and filecmp.cmp(filePath, localFile):
# LOCAL FILE IS UP-TO-DATE - NOTHING TO DO
pass
else:
# FILE DOES NOT EXISTS LOCALLY OR
# LOCAL COPY SEEMS OUT OF SYNC - COPY IT AGAIN
tryCount = 0
while True:
try:
# TRY TO COPY FILE
self.fastCopy(filePath, destPath)
break
except (OSError, IOError) as e:
if e.errno == errno.ESTALE:
# IF STALE NFS HANDLE IS ENCOUNTERED WAIT AND TRY AGAIN
if tryCount >= maxTries:
# TOO MANY UNSUCCESSFUL TRIES - GIVING UP
raise
time.sleep(.5)
tryCount += 1
else:
# SOME UNKNOWN ERROR OCCURRED
raise
def fastCopy(self, filePath, destPath):
'''use a fast copy functuion based on OS'''
if nuke.env['WIN32']:
# COPY UNDER WINDOWS SYSTEM
windll.kernel32.CopyFileA(filePath, destPath, False)
else:
# COPY UNDER*NIX SYSTEM
call(['cp', '-p', filePath, destPath])
def getTargetDir(self, filePath):
'''Get the target directory for filePath based on Nuke's cache preferences and localisation rules'''
parts = filePath.split('/') # NUKE ALREADY CONVERTS BACK SLASHES TO FORWARD SLASHES ON WINDOWS
if not filePath.startswith('/'):
# DRIVE LETTER
driveLetter = parts[0]
parts = parts [1:] # REMOVE DRIVE LETTER FROM PARTS BECAUSE WE ARE STORING IT IN PREFIX
prefix = driveLetter.replace(':', '_')
else:
# REPLACE EACH LEADING SLASH WITH UNDERSCORE
slashCount = len([i for i in parts if not i])
root = [p for p in parts if p][0]
parts = parts[slashCount + 1:] # REMOVE SLASHES AND ROOT FROM PARTS BECAUSE WE ARE STORING THOSE IN PREFIX
prefix = '_' * slashCount + root
# RE-ASSEMBLE TO LOCALISED PATH
parts.insert(0, prefix)
parts = self.cachePath.split('/') + parts
return '/'.join(parts[:-1]) # RETURN LOCAL DIRECTORY USING FORWARD SLASHES TO BE CONSISTENT WITH NUKE
def getFrameList(fileKnob, existingFilePaths):
'''
Return a list of frames that are part of the sequence that fileKnob is pointing to.
If the file path is already in existingFilePaths it will not be included.
'''
node = fileKnob.node()
originalCacheMode = node['cacheLocal'].value()
node['cacheLocal'].setValue('never')
frameRange = nuke.FrameRange(node.firstFrame(), node.lastFrame(), 1)
outputContext = nuke.OutputContext()
frameList = []
# Cycle over views
for viewNumber in xrange(outputContext.viewcount()):
viewName = outputContext.viewname(viewNumber)
# Skip "default" view
if viewName not in nuke.views():
continue
# Set context to viewNumber
outputContext.setView(viewNumber)
# Cycle over frame range
for frameNumber in frameRange:
outputContext.setFrame(frameNumber)
filePath = fileKnob.getEvaluatedValue(outputContext)
if filePath not in existingFilePaths:
frameList.append(filePath)
node['cacheLocal'].setValue(originalCacheMode)
return frameList
def localiseFileThreaded(readKnobList):
'''Wrapper to duck punch default method'''
p = nuke.Panel('Localiser (threaded)')
knobName = 'concurrent copy tasks'
p.addEnumerationPulldown(knobName, ' '.join([str(i+1) for i in xrange(min(nuke.THREADS, 4))]))
if p.show():
maxThreads = int(p.value(knobName))
else:
return
fileDict = {}
allFilesPaths = []
for knob in readKnobList:
first = knob.node().firstFrame()
last = knob.node().lastFrame()
filePathList = getFrameList(knob, allFilesPaths)
fileDict[knob.node().name()] = filePathList
allFilesPaths.extend(filePathList)
localiseThread = LocaliseThreaded(fileDict, maxThreads)
localiseThread.start()
def register():
nuke.localiseFilesHOLD = nuke.localiseFiles #BACKUP ORIGINAL
nuke.localiseFiles = localiseFileThreaded
_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python
