Re: [Maya-Python] Weird issue with Widget deletion

Justin Israel Mon, 10 Nov 2014 10:42:19 -0800

 On Tue, 11 Nov 2014 6:14 AM Joe Weidenbach <[email protected]> wrote:

Honestly, Justin, that might solve more than one piece of this for me, and
what you're saying makes sense. I'm going to have to apologize for the
wall of text in advance, but I need it to explain what I'm trying to
accomplish.

My code is a lot more intensive than the sample is, but I'm imagining it's
the same general issue--this is my first dive into mesh processing outside
of simple deformers (I'm normally a rigger, so this is the most complex
piece of code I've written as a single function), so I'm not always
entirely sure what I'm doing as I work through it. The system works great
on small meshes, but once I get up to full character scale, it slows down
exponentially, and on a full-res mesh it's actually crashing my system
(after eating up 32 gigs of memory). I've not been able to find any
unnecessary loops, so I don't think I'm running a flawed algorithm, but
breaking it up to give the system a chance to catch up is probably the
solution. After some research, it looks like the garbage collector might
not be getting time to do its job, so I'm getting a high-water effect from
processing each vert, edge, and face. I'm still not entirely sure how I'm
getting a 6 meg .ma file to take up 32+ gigs in memory, but so it goes
(and, as I said, the reason I was putting the progress bars in in the first
place, to give myself feedback on what the system is doing when the memory
starts to get eaten). The reason I think it's the garbage collector is
this article--
http://python.dzone.com/articles/diagnosing-memory-leaks-python . I
haven't installed objgraph yet, or heapy, but I'm imagining I'm going to
find lots of small objects that haven't been deallocated.

Ya memory profiling may be a lot easier than trying to write a UI around
the problem to help you solve it.

As I understand it, if I need to proceed sequentially from one task to
another, I don't think things will benefit from breaking them into
individual functions, but then I'm used to lower-level systems (C and
Assembly at the lowest end--I spent my first year in the field developing
new hardware controllers for the games we were developing), so I'm used to
manual memory management, and Garbage Collection might as well be magic.
So, does the event loop work work off of a function boundary?

We shouldn't confuse the issues you have with garbage collection with the
qt event loop, or try to equate the benefits of breaking down tasks between
Qt and non Qt applications. I can't say I have intimate knowledge of the
internal of the Qt event loop, but the basic idea is that objects can
register more tasks into the event loop and eventually it wants to unwind a
stack and get back to the event loop to process. Things like the
processEvents call can force the event loop to flush, but if you check out
the docs it will note that deferred deletions do not happen as part of a
call to processEvents. I presume this is also related to avoiding deferred
deletions during a deeply nested stack. I have had issues with bad pointers
and clean-ups from a call to processEvents in a deeply nested stack, or
when an event handler method is part of the stack. So maybe this is really
just your issue, that you need to allow the event loop to run, simply for
the UI reasons and not your task reasons. Also, I have no idea how Maya
manages it's memory in relation to Qt.

I also noticed that you're using QTimer. Quick info, just so I understand,
is that similar to .NET's Timer in that it sets up a second thread under
the hood? And, if so, is that the key to making this work in a way that
can help with refreshing? I ask because that's the only thing I could
think of that would benefit from smaller functions other than code
readability. As I understand it, chaining a series of functions together
(or calling one function after the other) doesn't really change the
resulting bytecode.

QTimer doesn't create threads. It schedules a callable to run after a
given amount of time, in the same thread. Using a value of 0 is just a way
to tell it to run right when the event loop gets back to idle again. I'm
not necessarily suggesting that a QTimer is the real solution here. Only
that it was a simple way to illustrate fixing the UI problem in your
example, by breaking up just the UI deletions and additions. Your example
couples the task loops with the creation and deletion of the widgets over
and over in that same stack. Maybe using the QueuedConnection for the
progress signals might have helped too. I didn't check. But I think a
different architecture could make it so that your progress UI stuff happens
deferred, separate from the actual work.

Here's my use case: I'm writing a tool to serialize a mesh, and then
optionally mirror it intelligently. I'm starting with just working across
the X axis, but from there I plan to add in other modes of
mirroring--arbitrary plane for sure, possibly even radial.

The steps I'm using are like this:
1) Capture the verts.
2) Capture the polygon index arrays
3) Capture the edge smoothness for each edge.
4) Wrap these into a JSON data structure (a dict) for later use.
5) Iterate through the vertices and mirror those that should be mirrored.
Add the mirrored verts to the data structure.
6) Iterate through the polys. Mirror. Add.
7) Iterate through the edges. Mirror smoothness. Add.

Steps 1-4 are in one function currently, and 5-7 are in another. I have a
third function that rebuilds a mesh from the resulting data structure. As
I said, for small meshes it has no issue. The mesh I'm using for my
testing is actually the beast from Hyper-Real creature Creation (the 2005
masterclasses), modeled by a WETA guy, and sitting at about 70,000 polys.
I figure that if I can make my tool work on it, it's about as bulletproof
as it can get.

That's the first stage. The second stage, once I have the first part done,
is to use the serialized mesh data to build a rig. So, I can store a
database of defined meshes, hand off a base mesh to a modeler, bring it
back in after sculpting, and have my tool build the rig to it as long as
the modeler didn't mess with the topology. I've seen these systems in the
wild, but what I haven't seen (although I'm sure it exists) is the part I'm
really focused on, which is tying a graphical workflow setup into it, so as
a rigger I can add an arbitrary mesh and then define how it should build
the rig, all graphically. All of that, I'm confident in my ability on, but
it all depends on getting this initial processing to work--the whole point
is that I can define the rig for the portion being mirrored, and then the
tool can build the rig for the entire system, with all necessary
mirroring. So, in a nutshell, as a one-man show, I'm trying to replicate
the rigging systems big houses with a dedicated development team have :).
Yep, I'm just crazy that way.

So, you mentioned QThreads. Threads are still an area of voodoo to me. I
work in games, and Unity (the engine my current company uses) actively
works against you if you even try to bring in threading. I get that
they're useful for parallel processing, or not blocking the UI, but I'm not
sure I need them for a sequential series of mesh analyses. If using them
gives the garbage collector/UI the opportunity to recover though, I'm glad
to give it a shot. I'd read that Maya doesn't like threads either,
especially when you're calling the API. I've done basic threading before
(mostly using BackgroundWorker in .NET to handle keeping the UI
responsive), but most of the information I've found on threading assumes a
much higher level of knowledge about it than I have (I only took a year of
actual CS classes before changing majors to animation, the rest is 10 years
of experience teaching myself)--so, it says "Here's how you do X with Y
Language's threading," when I have no idea what X means or why I'd want to
do it. I've always assumed that when I needed it and had a specific use
case, it would make sense. Any good resources on QThreads (or how the
event loop runs) you could point me towards? Information at that level
seems to be sparse from my searches.

QThreads do make a program more complicated for sure, because you have to
then consider memory barriers and logic running in completely different
branches. But the general rule of thumb in UI frameworks with an event loop
is that you aim to not block the main thread and do long running tasks in
another thread, while communicating over the signals system. Using Qt in
Maya is a bit different, because like you said there are restrictions on
being able to make Maya calls in another thread. Those usually have to be
called through executeInMainThreadWithResult() So that you call into the
main thread from your other thread.

In the meantime, I'll try integrating the example you sent back and see if
it makes a difference in the results.

Thanks again!

Joe

On 11/10/2014 1:59 AM, Justin Israel wrote:

Hey Joe,

The thing is, since your example is really trivial, I can really only
comment on the trivial example itself. Normally one wouldn't want to
process a whole bunch of long running/blocking tasks one after the next, in
a single method, while adding and removing widgets from a layout. The
layout is obviously having a drawing issue related to timing, from
everything happening within that same stack.

I can offer a tweak for the example that shows how it could be corrected,
by splitting up your tasks into pieces that allow the stack to return to
the event loop again.

http://pastebin.com/5s5mx6KE

In a more real world example, you might abstract these tasks into objects
emit their own progress (QThread already does this), and then be able to
chain them up so one starts after the next. Or when one is done, a queue is
checked and the next pending task runs. Are these tasks that can be run in
a thread? If so, you can use a thread pool or QThreads and coordinate with
their signals.

Anyways, basically the issue is that the layout is unhappy having itself
modified so much from a single stack.

- Justin

On Mon Nov 10 2014 at 7:03:33 PM Joe Weidenbach <[email protected]> wrote:

Hey All,

I'm having an issue with a tool I'm working on regarding slot-based
widget deletion.

I'm trying to create a UI where my object which handles data processing
reports it's progress on various (possibly nested) tasks back to the
UI. It's a mesh processor, so it can be pretty intensive. I'm actually
using these progress bars to try and identify where a memory leak is
right now, so I can track it down, but it would be dead useful to have
for general purposes as well.

I've made up a simpler script to demo the situation, here:
http://pythonfiddle.com/progress-bar-test

The relevant code is in the final class, in the three slots that connect
to PBTester (lines 84-113). I don't think I need to call self.update()
on every loop, but it does slow things down for the visual of the
progress bars.

The issue I'm having is that when the processCompleted Signal fires, it
doesn't seem to actually delete the given progress bar. It's removed
from the layout, but the visual stays in place and gets overlaid by the
new progress bar. My guess from the research I've done is that this has
something to do with deleteLater not being called while in the process,
but I don't understand the event loop well enough yet to make much sense
of it. The closest page I've been able to find recommended calling
emit() with Qt.QueuedConnection, and I did try that, but I don't think I
was doing it right, as it didn't seem to do anything. I'm not using
threads either, so I don't think it's a sync issue, but I'm not sure if
Qt is using them under the hood or not.

Hopefully that makes sense. Any thoughts from the pros?

Thanks,

Joe

---
This email is free from viruses and malware because avast! Antivirus
protection is active.
http://www.avast.com

--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/5460552B.3000500%40gmail.com
.
For more options, visit https://groups.google.com/d/optout.

To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3AZ8gG9zLc%3Dsa7aH1ME1P%3D9yp9nSnOR7A0ENKmjVDoiQ%40mail.gmail.com
<https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3AZ8gG9zLc%3Dsa7aH1ME1P%3D9yp9nSnOR7A0ENKmjVDoiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

For more options, visit https://groups.google.com/d/optout.

This email is free from viruses and malware because avast! Antivirus
<http://www.avast.com/> protection is active.

--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/5460F261.2050708%40gmail.com
<https://groups.google.com/d/msgid/python_inside_maya/5460F261.2050708%40gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0V2VRXUW%2B%2BAmMqG7x98ZyrFK9fGz6x%3DpNV1244Dw%2B8UQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Maya-Python] Weird issue with Widget deletion

Reply via email to