On Fri, Jan 20, 2017, 5:15 PM Christopher Evans <[email protected]> wrote:
The idea wasn't really to render anything, but to send multiple headless mayas 'tasks', do many people use these render management solutions to do something like batch export 4000 fbx files from 3000 maya files? I thought they were always very specific to rendering. Ya, "render" is used loosely here as a term they usual call these VFX targeted distributed task queues. Obviously rendering is a key task, but really they are just systems designed to schedule and dispatch work to available nodes on a farm. We use them to do every imaginable type of pipeline operation that we don't want taking up resources on our local machines. Disk migrations, image conversions, publish processes, review system processing. So yea, running match operations on a Maya file would be a reasonable task to submit to a "render" farm. Keeping a long running Maya instance on a booked node is a special optimisation to avoid the overhead of starting it for each batch. We don't do it in our own render farm since its more complicated and tends to not always net a significant benefit over just standard task batching. I did try using pipe before, but it wasn't really working. I believe the issue is that I am running this popen from here: https://github.com/chrisevans3d/mayaTaskServer/blob/master/server.py#L275 What I think is happening that is giving you trouble is that your maya work happens in a forked process. Within that process you have no direct access back to what is happening in your main process. You can't consider any state really useful outside the arguments you have been passed to your target function. So when you run you maya process you need some way to ship the stdout/stderr back to the main process. You can look at multiprocessing.Pipe <https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Pipe> to accomplish this. You create it on the main process side and pass the write side of the pipe into the child process of the pool. Then in that child process function you can probably again pass that write pipe as the stdout for your subprocess mayapy. It should then write the stdout over the pipe and you can collect it somehow in your main process. Hope this gives you some direction? On Thu, Jan 19, 2017 at 7:17 PM, Justin Israel <[email protected]> wrote: On Fri, Jan 20, 2017 at 10:34 AM Christopher Evans <[email protected]> wrote: https://github.com/chrisevans3d/mayaTaskServer It's on github here, I am no master of python, I have written one of these at a previous company, but I used RPYC for distributed computing and a 'task finished' signal coupled with a timeout to know when something crashed. I would now like to parse the stdout in realtime and also print it to each worker tab in the UI in realtime to give someone an idea why it crashed. Are you talking about this bit of code? https://github.com/chrisevans3d/mayaTaskServer/blob/master/serverUtils.py#L59 Currently you are calling Popen without piping the stdout/stderr anywhere so it would go to the console or default output location. I see you have commented out an attempt to pipe stdout, but you don't have any code that actually uses it. If you are looking to capture stdout as the application is running (that is, without waiting for mayapy to exit before getting the output) then something needs to be constantly reading from that pipe. Usually you can accomplish this in a reader thread. You can read from the stdout pipe in a thread until the process ends and the pipe closes. This is something every company needs, so I thought that this would be useful for people. >From a glance at the readme, it looks like a very specific implementation of a render queue solution (Rush, Dr Queue, Qube, Smedge, ...). Is the difference here that the Maya process stays running instead of loading for each batch of tasks? I know Qube uses this approach for their custom support for Maya Batch tasks. When they schedule a task to a node they start a maya session and feed it work over the commandPort until the job has no more tasks. If anyone wants access, let me know how to give people edit access on github, I am a github newb. On Thu, Jan 19, 2017 at 1:22 PM, Justin Israel <[email protected]> wrote: On Fri, Jan 20, 2017, 6:33 AM Tony Barbieri <[email protected]> wrote: Previously I've used the communicate method on a Popen object to get the output and currently I am using QtCore.QProcess with the readyReadStandardOutput, readyReadStandardError signals to get the output to display in Qt widgets. I'm using QtCore.QProcess in a situation similar to your's where I have workers executing mayapy processes. I've done probably one or both of those in the past as well, without an issue in collecting stdout for display. @Christopher, we probably need to see a small example of what you are doing, that isn't working as expected. On Thu, Jan 19, 2017 at 5:34 AM, Marcus Ottosson <[email protected]> wrote: Anyone have experience with getting feedback out of mayapy? Something like this? $ mayapy -c "import sys;sys.stdout.write('Hello world!')" Hello world! -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBAp5CqQ-YTgw9nN97DmyWyyVBuY-D6HaPpJakssW8Fig%40mail.gmail.com <https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBAp5CqQ-YTgw9nN97DmyWyyVBuY-D6HaPpJakssW8Fig%40mail.gmail.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- Tony -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsSGvt2gMT-4sh4-RUo1Uo6iiJ02mUrAqQzjpD-f-zt5mQ%40mail.gmail.com <https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsSGvt2gMT-4sh4-RUo1Uo6iiJ02mUrAqQzjpD-f-zt5mQ%40mail.gmail.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2idOnzBMfb6mF6V_7mDQdDexD7jD_f%2BRa_pLk8qO9oDw%40mail.gmail.com <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2idOnzBMfb6mF6V_7mDQdDexD7jD_f%2BRa_pLk8qO9oDw%40mail.gmail.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- CE -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAEdZ1gyawRvEpVMzxzScJ6fXE%3DFZw9sfyFVkGG6BNrjppkEnjw%40mail.gmail.com <https://groups.google.com/d/msgid/python_inside_maya/CAEdZ1gyawRvEpVMzxzScJ6fXE%3DFZw9sfyFVkGG6BNrjppkEnjw%40mail.gmail.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2_eX3SzcjVJpbYO6N%3D0MhkP4j-aSvggvWCzAhZcetamA%40mail.gmail.com <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2_eX3SzcjVJpbYO6N%3D0MhkP4j-aSvggvWCzAhZcetamA%40mail.gmail.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- CE -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAEdZ1gwon2wX1j6p2TBhoTeZvggnCRfHY9_uwa-rjRzvG0NQ1Q%40mail.gmail.com <https://groups.google.com/d/msgid/python_inside_maya/CAEdZ1gwon2wX1j6p2TBhoTeZvggnCRfHY9_uwa-rjRzvG0NQ1Q%40mail.gmail.com?utm_medium=email&utm_source=footer> . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA29sE-x5z%2BT4hvh60j1UL0ejr43WhMaGxttNwtB2TC8cA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
