Re: [Nuke-python] nuke localise

Mon, 16 Dec 2013 15:27:20 -0800 

Hi Holger,
>>if i remember correctly from back then you can basically already set the 'cacheLocal' knob to 'always' I am not touching that part but leave the settings up to the user so the default workflow remains the same. 



>>e.g. for files from a drive mapped to the letter 'Y' these are the 
two different results

>>original Nuke function:
>><localCachePath>/Y_/folder1/folder2/etc/...
>>
>>your LocaliseThreaded
>><localCachePath>/Y__folder1/folder2/etc/...


Great, thanks for that, looks like some Nuke docs are wrong then. I will 
adjust my code accordingly.


Cheers,
frank


thanks for the info, looks like some of the Nuke documentaion is wrong then
On 17/12/13 11:43, Holger Hummel|Celluloid VFX wrote:

hey Frank,


i can defnitely second Sebastian's statement regarding Windows. a 
while ago i quickly wrote our own localise function which is using 
MS's robocopy tool.
this already sped things up considerably compared to Nuke's built-in 
localise function so i didn't really dig into it any further to see if 
there's even more to be gained.
if i remember correctly from back then you can basically already set 
the 'cacheLocal' knob to 'always' even before copying and Nuke will just
read the localised files as they become available. Nuke will then show 
the localised frames with the orange bar above the Read node and 
update that indicator

while stepping through the timeline.


i also quickly tried your tool and found that it won't copy the 
localised files into the exact same location on my drive.
e.g. for files from a drive mapped to the letter 'Y' these are the two 
different results


original Nuke function:
<localCachePath>/Y_/folder1/folder2/etc/...

your LocaliseThreaded
<localCachePath>/Y__folder1/folder2/etc/...


so the Nuke session didn't find the local files and thus did not use 
them.


cheers,
Holger



Sebastian Elsner wrote:

Am 16.12.2013 20:43, schrieb Frank Rueter:

Thanks Sebastian. On linux/osx running a native "cp" command via 
os.system or Popen is heaps faster as well than copy2 last time I 
checked.



This is because Python has its own implementation of a copy written 
in Python, which is of course slower (have a look at the code, its 
basically a indefinite loop of a file.read() and newFile.write()) So 
maybe on unix, it would be nice to use ctypes with the kernel native 
"sendfile" function.



Interesting to hear about threaded copies not being faster for you 
though. I will double check my setup but am pretty sure it makes a 
massive difference. If it's so unpredictable though I might just pop 
up a little dialog when the user wishes to localise to ask for the 
desired amount of concurrent threads.



This is probably different on unix especially if you are using nfs, 
but with samba and window, having more than one or two threads is 
slowing things down.





Thanks for the heads up no QThreads, that is exactly my plan (as you 
may have guessed based on my post on the PySide forum :)). Though I 
might just leave it as is and not overcomplicate it. Will decided 
today...




On 16/12/13 22:24, Sebastian Elsner wrote:

On 12/16/2013 09:29 AM, Frank Rueter wrote:

Hola everybody,


I had a quick look at this this morning and realised I would have 
to re-write everything from scratch - but then couldn't resists :-D.

Could some of you test the attached file and tell me how you get on?
Just put it into your NUKE_PATH and put this into your menu.py:

    import LocaliseThreaded
    LocaliseThreaded.register()



This will replace the default localising behaviour with a threaded 
one.
The maximum threads are half of your Nuke threads (nuke.THREADS) 
at the moment (I will make this a preference though). There will 
be info about concurrent threads in the progress bar as it does 
it's thing.



It's work in progress at this stage but since my day is coming to 
an end, I thought it would be good to get it out there for a test 
run, so I know more in the morning.


Things I still need/want to do:

  * support split file knobs for stereo projects
  * support proxy knobs (should I, not sure if the default does?)
  * refactor the code so that multiple threads can tackle the same
    read node (at the moment one Read node is allocated one task)
  * benchmark the copy function (currently shutil.copy2). Pretty sure
    it's not the fastest one for large files and I might have to roll
    my own to speed things up.




On windows copy2 is really slow compared to native copying. You can 
use this snippet to speed things up:


import platform


if platform.system() == "Windows":
    from ctypes import windll

    def copy2(src, dst):
        windll.kernel32.CopyFileA(src, dst, False)
else:
    from shutil import copy2


Also copying in multiple threads on windows does not help at all to 
speed things up. Worse, it will make things slower. But this is 
mostly depending on your fileserver. I never do more than one 
thread on our machines, its not worth the effort of handling the 
threads dynamically. If you still want that and need an easy 
solution try QThreadPool and QRunnable. Its not as flexible as 
handling the threads yourself, but it is difficult to get it wrong, 
which is something you cannot say about QThread.


Cheers

Sebastian



 *





Let me know how it works, especially if you are on windows as I 
can't test that here.


Cheers,
frank


P.S.: If somebody is an expert with python threading with a little 
bit of time on their hands, get in touch, I'm pretty sure the way 
I'm doing this can be optimised, especially for trying to 
dynamically allocate threads to efficiently deal with outstanding 
tasks.




On 14/12/13 23:23, Howard Jones wrote:

I was wondering that. It was going to be my next question (honest)

Howard


On 14 Dec 2013, at 08:19, Thorsten 
Kaufmann<thorsten.kaufm...@mackevision.de> wrote:


This also sounds like a job for "import nuke" no? ;)
Thorsten Kaufmann
Production Pipeline Architect
____________________________________

Mackevision Medien Design GmbH
Forststra?e 7
D-70174 Stuttgart

T  T  +49 711 93 30 48 78
F  +49 711 93 30 48 90
M +49 151 19 55 55 02

thorsten.kaufm...@mackevision.de
http://www.mackevision.de

Gesch?ftsf?hrer: Armin Pohl, Joachim Lincke, Karin Suttheimer
HRB 243735 Amtsgericht Stuttgart
________________________________________

Von:nuke-python-boun...@support.thefoundry.co.uk 
[nuke-python-boun...@support.thefoundry.co.uk] im Auftrag von 
Frank Rueter [fr...@beingfrank.info]

Gesendet: Samstag, 14. Dezember 2013 01:11
An: Nuke Python discussion; Justin Fpc
Betreff: Re: [Nuke-python] nuke localise


I have wrote my own localising script from scratch just before 
this feature was implemented. I will have a peek next week if I 
can quickly adapt it to use the localising settings in the 
preferences and nodes. If so it will be threaded and we should 
get the best of both worlds until the built in feature is more 
flexible to allow background processing.


Sent with AquaMail for Android
http://www.aqua-mail.com

On 13 December 2013 9:58:43 PM Justin Fpc wrote:

Hi all,


I would be very interested if there is anyway to manage this 
localising in background.
I've also tested to use the threading method and found the same 
problem/cause as Frank.



Justin



2013/12/13 Howard Jones 
<mrhowardjo...@yahoo.com<mailto:mrhowardjo...@yahoo.com>>

Thanks for testing. That would have stumped me.

I contacted support.

Howard


On 12 Dec 2013, at 23:48, Frank Rueter 
<fr...@beingfrank.info<mailto:fr...@beingfrank.info>> wrote:


I remember now:

I tried this a while ago myself and failed because doLocalise() 
is a wrapper function using nuke.localiseFiles which seems to be 
compiled.
Since nuke.localiseFiles takes care of the progress bar 
(presumably juggling it's own threads) it's not just a matter of 
using


thread = threading.Thread(target=doLocalise, args=(True,))

thread.start()

or



thread = threading.Thread(target=nuke.localiseFiles, 
args=(readKnobList,))


thread.start()



Both the above do the job, but you won't get the progress bar 
and the main thread is still blocked.



There might be a way but I don't know how, other than basically 
writing the localisation logic yourself.
So best to push that feature request to make nuke.localiseFiles 
thread-able.



Cheers,
frank



On 13/12/13 12:15, Frank Rueter wrote:
Yes, you should be able to. I have a quick peek...

On 13/12/13 11:29, Howard Jones wrote:

Ok done. Out of interest can this be run in a separate thread? 
My python brain hasn't got round threading, but i can run 
doLocalise(0) so could I thread it instead?


Howard


On 12 Dec 2013, at 22:02, Frank 
Rueter<fr...@beingfrank.info><mailto:fr...@beingfrank.info> wrote:



I have asked for this in the pas as well, so please bug support 
to up the priority ;)



On 11/12/13 05:23, Howard Jones wrote:
Hi
Is it possible to run localise from a shell or in the background?
H
_______________________________________________
Nuke-python mailing list

Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/ 

http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python 


_______________________________________________
Nuke-python mailing list

Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/ 

http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python 


_______________________________________________
Nuke-python mailing list

Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/ 

http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python 



_______________________________________________
Nuke-python mailing list

Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/ 

http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python 



_______________________________________________
Nuke-python mailing list

Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/ 

http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python 



_______________________________________________
Nuke-python mailing list

Nuke-python@support.thefoundry.co.uk<mailto:Nuke-python@support.thefoundry.co.uk>,http://forums.thefoundry.co.uk/ 

http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python 




_______________________________________________
Nuke-python mailing list

Nuke-python@support.thefoundry.co.uk,http://forums.thefoundry.co.uk/ 

http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python 


_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python




_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk,http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python



--
check outwww.pointcloud9.com

Sebastian Elsner - Pipeline Technical Director - RISE

t: +49 30 20180300flor...@risefx.com
f: +49 30 61651074www.risefx.com

RISE FX GmbH
Schlesische Strasse 28, Aufgang B, 10997 Berlin
c/o action concept, An der Hasenkaule 1-7, 50354 Hürth
Geschaeftsfuehrer: Sven Pannicke, Robert Pinnow
Handelsregister Berlin HRB 106667 B


_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python





_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python


------------------------------------------------------------------------

_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python




_______________________________________________
Nuke-python mailing list
Nuke-python@support.thefoundry.co.uk, http://forums.thefoundry.co.uk/
http://support.thefoundry.co.uk/cgi-bin/mailman/listinfo/nuke-python





























					Reply via email to