On Tue Nov 11 2014 at 7:42:01 AM Justin Israel <[email protected]> wrote:
> > On Tue, 11 Nov 2014 6:14 AM Joe Weidenbach <[email protected]> wrote: > > Honestly, Justin, that might solve more than one piece of this for me, and > what you're saying makes sense. I'm going to have to apologize for the > wall of text in advance, but I need it to explain what I'm trying to > accomplish. > > My code is a lot more intensive than the sample is, but I'm imagining it's > the same general issue--this is my first dive into mesh processing outside > of simple deformers (I'm normally a rigger, so this is the most complex > piece of code I've written as a single function), so I'm not always > entirely sure what I'm doing as I work through it. The system works great > on small meshes, but once I get up to full character scale, it slows down > exponentially, and on a full-res mesh it's actually crashing my system > (after eating up 32 gigs of memory). I've not been able to find any > unnecessary loops, so I don't think I'm running a flawed algorithm, but > breaking it up to give the system a chance to catch up is probably the > solution. After some research, it looks like the garbage collector might > not be getting time to do its job, so I'm getting a high-water effect from > processing each vert, edge, and face. I'm still not entirely sure how I'm > getting a 6 meg .ma file to take up 32+ gigs in memory, but so it goes > (and, as I said, the reason I was putting the progress bars in in the first > place, to give myself feedback on what the system is doing when the memory > starts to get eaten). The reason I think it's the garbage collector is > this article-- > http://python.dzone.com/articles/diagnosing-memory-leaks-python . I > haven't installed objgraph yet, or heapy, but I'm imagining I'm going to > find lots of small objects that haven't been deallocated. > > Ya memory profiling may be a lot easier than trying to write a UI around > the problem to help you solve it. > > As I understand it, if I need to proceed sequentially from one task to > another, I don't think things will benefit from breaking them into > individual functions, but then I'm used to lower-level systems (C and > Assembly at the lowest end--I spent my first year in the field developing > new hardware controllers for the games we were developing), so I'm used to > manual memory management, and Garbage Collection might as well be magic. > So, does the event loop work work off of a function boundary? > > We shouldn't confuse the issues you have with garbage collection with > the qt event loop, or try to equate the benefits of breaking down tasks > between Qt and non Qt applications. I can't say I have intimate knowledge > of the internal of the Qt event loop, but the basic idea is that objects > can register more tasks into the event loop and eventually it wants to > unwind a stack and get back to the event loop to process. Things like the > processEvents call can force the event loop to flush, but if you check out > the docs it will note that deferred deletions do not happen as part of a > call to processEvents. I presume this is also related to avoiding deferred > deletions during a deeply nested stack. I have had issues with bad pointers > and clean-ups from a call to processEvents in a deeply nested stack, or > when an event handler method is part of the stack. So maybe this is really > just your issue, that you need to allow the event loop to run, simply for > the UI reasons and not your task reasons. Also, I have no idea how Maya > manages it's memory in relation to Qt. > Actually, what I meant to say here was that I believe you are dealing with more than one issue: Python garbage collection, Maya's memory management, or Qt deferred deletion, or the Qt event loop and updates/redraws. I think the python garbage collector runs at both intervals and object thresholds. If your garbage is coming from python objects and you aren't explicitly calling delete on them, and instead relying on them to collect when they fall off the stack, when you do a deeply nested stack that keeps calling more functions, those objects in the previous stack frames are still alive until the stack unwinds past their frames. If the garbage is coming from Qt, then maybe it is related to the deferred deletions piling up. And if the garbage is Maya, I don't really know very well what controls its memory management. But it may also be related to the event loop. I also noticed that you're using QTimer. Quick info, just so I > understand, is that similar to .NET's Timer in that it sets up a second > thread under the hood? And, if so, is that the key to making this work in > a way that can help with refreshing? I ask because that's the only thing I > could think of that would benefit from smaller functions other than code > readability. As I understand it, chaining a series of functions together > (or calling one function after the other) doesn't really change the > resulting bytecode. > > QTimer doesn't create threads. It schedules a callable to run after a > given amount of time, in the same thread. Using a value of 0 is just a way > to tell it to run right when the event loop gets back to idle again. I'm > not necessarily suggesting that a QTimer is the real solution here. Only > that it was a simple way to illustrate fixing the UI problem in your > example, by breaking up just the UI deletions and additions. Your example > couples the task loops with the creation and deletion of the widgets over > and over in that same stack. Maybe using the QueuedConnection for the > progress signals might have helped too. I didn't check. But I think a > different architecture could make it so that your progress UI stuff happens > deferred, separate from the actual work. > > Here's my use case: I'm writing a tool to serialize a mesh, and then > optionally mirror it intelligently. I'm starting with just working across > the X axis, but from there I plan to add in other modes of > mirroring--arbitrary plane for sure, possibly even radial. > > The steps I'm using are like this: > 1) Capture the verts. > 2) Capture the polygon index arrays > 3) Capture the edge smoothness for each edge. > 4) Wrap these into a JSON data structure (a dict) for later use. > 5) Iterate through the vertices and mirror those that should be mirrored. > Add the mirrored verts to the data structure. > 6) Iterate through the polys. Mirror. Add. > 7) Iterate through the edges. Mirror smoothness. Add. > > Steps 1-4 are in one function currently, and 5-7 are in another. I have a > third function that rebuilds a mesh from the resulting data structure. As > I said, for small meshes it has no issue. The mesh I'm using for my > testing is actually the beast from Hyper-Real creature Creation (the 2005 > masterclasses), modeled by a WETA guy, and sitting at about 70,000 polys. > I figure that if I can make my tool work on it, it's about as bulletproof > as it can get. > > That's the first stage. The second stage, once I have the first part > done, is to use the serialized mesh data to build a rig. So, I can store a > database of defined meshes, hand off a base mesh to a modeler, bring it > back in after sculpting, and have my tool build the rig to it as long as > the modeler didn't mess with the topology. I've seen these systems in the > wild, but what I haven't seen (although I'm sure it exists) is the part I'm > really focused on, which is tying a graphical workflow setup into it, so as > a rigger I can add an arbitrary mesh and then define how it should build > the rig, all graphically. All of that, I'm confident in my ability on, but > it all depends on getting this initial processing to work--the whole point > is that I can define the rig for the portion being mirrored, and then the > tool can build the rig for the entire system, with all necessary > mirroring. So, in a nutshell, as a one-man show, I'm trying to replicate > the rigging systems big houses with a dedicated development team have :). > Yep, I'm just crazy that way. > > So, you mentioned QThreads. Threads are still an area of voodoo to me. I > work in games, and Unity (the engine my current company uses) actively > works against you if you even try to bring in threading. I get that > they're useful for parallel processing, or not blocking the UI, but I'm not > sure I need them for a sequential series of mesh analyses. If using them > gives the garbage collector/UI the opportunity to recover though, I'm glad > to give it a shot. I'd read that Maya doesn't like threads either, > especially when you're calling the API. I've done basic threading before > (mostly using BackgroundWorker in .NET to handle keeping the UI > responsive), but most of the information I've found on threading assumes a > much higher level of knowledge about it than I have (I only took a year of > actual CS classes before changing majors to animation, the rest is 10 years > of experience teaching myself)--so, it says "Here's how you do X with Y > Language's threading," when I have no idea what X means or why I'd want to > do it. I've always assumed that when I needed it and had a specific use > case, it would make sense. Any good resources on QThreads (or how the > event loop runs) you could point me towards? Information at that level > seems to be sparse from my searches. > > > QThreads do make a program more complicated for sure, because you have to > then consider memory barriers and logic running in completely different > branches. But the general rule of thumb in UI frameworks with an event loop > is that you aim to not block the main thread and do long running tasks in > another thread, while communicating over the signals system. Using Qt in > Maya is a bit different, because like you said there are restrictions on > being able to make Maya calls in another thread. Those usually have to be > called through executeInMainThreadWithResult() So that you call into the > main thread from your other thread. > > > In the meantime, I'll try integrating the example you sent back and see if > it makes a difference in the results. > > Thanks again! > > Joe > > On 11/10/2014 1:59 AM, Justin Israel wrote: > > Hey Joe, > > The thing is, since your example is really trivial, I can really only > comment on the trivial example itself. Normally one wouldn't want to > process a whole bunch of long running/blocking tasks one after the next, in > a single method, while adding and removing widgets from a layout. The > layout is obviously having a drawing issue related to timing, from > everything happening within that same stack. > > I can offer a tweak for the example that shows how it could be corrected, > by splitting up your tasks into pieces that allow the stack to return to > the event loop again. > > http://pastebin.com/5s5mx6KE > > In a more real world example, you might abstract these tasks into objects > emit their own progress (QThread already does this), and then be able to > chain them up so one starts after the next. Or when one is done, a queue is > checked and the next pending task runs. Are these tasks that can be run in > a thread? If so, you can use a thread pool or QThreads and coordinate with > their signals. > > Anyways, basically the issue is that the layout is unhappy having itself > modified so much from a single stack. > > - Justin > > > > On Mon Nov 10 2014 at 7:03:33 PM Joe Weidenbach <[email protected]> wrote: > > Hey All, > > I'm having an issue with a tool I'm working on regarding slot-based > widget deletion. > > I'm trying to create a UI where my object which handles data processing > reports it's progress on various (possibly nested) tasks back to the > UI. It's a mesh processor, so it can be pretty intensive. I'm actually > using these progress bars to try and identify where a memory leak is > right now, so I can track it down, but it would be dead useful to have > for general purposes as well. > > I've made up a simpler script to demo the situation, here: > http://pythonfiddle.com/progress-bar-test > > The relevant code is in the final class, in the three slots that connect > to PBTester (lines 84-113). I don't think I need to call self.update() > on every loop, but it does slow things down for the visual of the > progress bars. > > The issue I'm having is that when the processCompleted Signal fires, it > doesn't seem to actually delete the given progress bar. It's removed > from the layout, but the visual stays in place and gets overlaid by the > new progress bar. My guess from the research I've done is that this has > something to do with deleteLater not being called while in the process, > but I don't understand the event loop well enough yet to make much sense > of it. The closest page I've been able to find recommended calling > emit() with Qt.QueuedConnection, and I did try that, but I don't think I > was doing it right, as it didn't seem to do anything. I'm not using > threads either, so I don't think it's a sync issue, but I'm not sure if > Qt is using them under the hood or not. > > Hopefully that makes sense. Any thoughts from the pros? > > Thanks, > > Joe > > --- > This email is free from viruses and malware because avast! Antivirus > protection is active. > http://www.avast.com > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/5460552B.3000500%40gmail.com > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3AZ8gG9zLc%3Dsa7aH1ME1P%3D9yp9nSnOR7A0ENKmjVDoiQ%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3AZ8gG9zLc%3Dsa7aH1ME1P%3D9yp9nSnOR7A0ENKmjVDoiQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > > For more options, visit https://groups.google.com/d/optout. > > > > <img > src="https://ci3.googleusercontent.com/proxy/ugHIP5cOMxDtnT_N1iOHaSA-eaW54QTlGlm6cWeLsTztLmjyZiaQGO5iCzO-MGezy0MDrIjPeV-miBnwZphFX93F4wjDUcXA3w=s0-d-e1-ft#http://static.avast.com/emails/avast-mail-stamp.png";> > <http://www.avast.com/> > > This email is free from viruses and malware because avast! Antivirus > <http://www.avast.com/> protection is active. > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/5460F261.2050708%40gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/5460F261.2050708%40gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3-SGo7Cud4BmzSjcN3g-qX3fE5QzE2C%2BUk%2B1gLkFxj6g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
![https://ci3.googleusercontent.com/proxy/ugHIP5cOMxDtnT_N1iOHaSA-eaW54QTlGlm6cWeLsTztLmjyZiaQGO5iCzO-MGezy0MDrIjPeV-miBnwZphFX93F4wjDUcXA3w=s0-d-e1-ft#http://static.avast.com/emails/avast-mail-stamp.png"](https://ci3.googleusercontent.com/proxy/ugHIP5cOMxDtnT_N1iOHaSA-eaW54QTlGlm6cWeLsTztLmjyZiaQGO5iCzO-MGezy0MDrIjPeV-miBnwZphFX93F4wjDUcXA3w=s0-d-e1-ft#http://static.avast.com/emails/avast-mail-stamp.png"){kind=link}