Nice. Glad you figured out a solution! I tend to figure out some expected time for each iteration of my task and come up with a reasonable interval to call processEvents. Like:
if i %10 = 0: processEvents() On Wed, 12 Nov 2014 1:27 PM Joe Weidenbach <[email protected]> wrote: > Huzzah! > > I think I've got a workflow down that will be functional for what I'm > trying to do. > > I found an article at http://doc.qt.digia.com/qq/qq27-responsive-guis.html > that went through some ways to keep things responsive, both with threads > and without. The key I was missing was using a second event loop to pass > control back to the application while waiting for my functions to finish, > while ensuring that I'm calling my processing tasks with a single-shot > timer to ensure everything's in the loop. This solved my issue of not > wanting to chain my functions together, while still enabling me to break > out functionality where needed, while giving Qt some breathing room. So > thank you! And, I'll keep you posted on my findings with the other issues. > > Here's the working code that I settled on: > > http://pythonfiddle.com/progress-bar-test/ > > I'm probably calling QtGui.QApplication.processEvents() a few times too > often, and will worry about that as I go. > > If you see any other potential issues in this part of things, don't > hesitate to mention them! > > Thanks, > > Joe > > On Mon, Nov 10, 2014 at 1:02 PM, Joe Weidenbach <[email protected]> wrote: > >> >> But I think a different architecture could make it so that your >> progress UI stuff happens deferred, separate from the actual work. >> >> I completely agree with this. That decoupling is what I was trying to do >> by simply sending signals as I did the work, and letting the UI sort it all >> out. I've spent too much time with C# events it seems :). Might just be >> me, but they seem more predictable. >> >> On Mon, Nov 10, 2014 at 12:08 PM, Joe Weidenbach <[email protected]> >> wrote: >> >>> Yeah, I'm pretty sure you're right about that :). I'm running fairly >>> regular gc.collect()'s, but it seems to have limited effectiveness, which >>> was where that article I included came in--it seems there's no way to force >>> a collection mid-loop, although that article also pointed out that that was >>> Linux-specific behavior and that they didn't believe Windows was affected. >>> Either way, that QTimer could help, as could setting the UI in a seperate >>> thread and calling the maya API stuff with >>> executeInMainThreadWithResult(), just by having a true delay for all >>> of the systems to regroup as it were. >>> >>> I do want to profile it (that's usually my first step, although I'm used >>> to having tools like Visual Studio or Unity to handle the profiling. Is >>> there a way to setup a profiling environment (like a VirtualEnv) that I can >>> run this in while keeping Maya functionality, without affecting the global >>> mayapy? I like to keep my mayapy relatively pristine so that I don't >>> introduce outside dependencies without deliberation, so I'd prefer to avoid >>> using pip outside of a VirtualEnv, but not sure if this is possible for the >>> Maya Version. >>> >>> On Mon, Nov 10, 2014 at 11:18 AM, Justin Israel <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On Tue Nov 11 2014 at 7:42:01 AM Justin Israel <[email protected]> >>>> wrote: >>>> >>>>> >>>>> On Tue, 11 Nov 2014 6:14 AM Joe Weidenbach <[email protected]> wrote: >>>>> >>>>> Honestly, Justin, that might solve more than one piece of this for me, >>>>> and what you're saying makes sense. I'm going to have to apologize for >>>>> the >>>>> wall of text in advance, but I need it to explain what I'm trying to >>>>> accomplish. >>>>> >>>>> My code is a lot more intensive than the sample is, but I'm imagining >>>>> it's the same general issue--this is my first dive into mesh processing >>>>> outside of simple deformers (I'm normally a rigger, so this is the most >>>>> complex piece of code I've written as a single function), so I'm not >>>>> always >>>>> entirely sure what I'm doing as I work through it. The system works great >>>>> on small meshes, but once I get up to full character scale, it slows down >>>>> exponentially, and on a full-res mesh it's actually crashing my system >>>>> (after eating up 32 gigs of memory). I've not been able to find any >>>>> unnecessary loops, so I don't think I'm running a flawed algorithm, but >>>>> breaking it up to give the system a chance to catch up is probably the >>>>> solution. After some research, it looks like the garbage collector might >>>>> not be getting time to do its job, so I'm getting a high-water effect from >>>>> processing each vert, edge, and face. I'm still not entirely sure how I'm >>>>> getting a 6 meg .ma file to take up 32+ gigs in memory, but so it goes >>>>> (and, as I said, the reason I was putting the progress bars in in the >>>>> first >>>>> place, to give myself feedback on what the system is doing when the memory >>>>> starts to get eaten). The reason I think it's the garbage collector is >>>>> this article-- >>>>> http://python.dzone.com/articles/diagnosing-memory-leaks-python . I >>>>> haven't installed objgraph yet, or heapy, but I'm imagining I'm going to >>>>> find lots of small objects that haven't been deallocated. >>>>> >>>>> Ya memory profiling may be a lot easier than trying to write a UI >>>>> around the problem to help you solve it. >>>>> >>>>> As I understand it, if I need to proceed sequentially from one task to >>>>> another, I don't think things will benefit from breaking them into >>>>> individual functions, but then I'm used to lower-level systems (C and >>>>> Assembly at the lowest end--I spent my first year in the field developing >>>>> new hardware controllers for the games we were developing), so I'm used to >>>>> manual memory management, and Garbage Collection might as well be magic. >>>>> So, does the event loop work work off of a function boundary? >>>>> >>>>> We shouldn't confuse the issues you have with garbage collection >>>>> with the qt event loop, or try to equate the benefits of breaking down >>>>> tasks between Qt and non Qt applications. I can't say I have intimate >>>>> knowledge of the internal of the Qt event loop, but the basic idea is that >>>>> objects can register more tasks into the event loop and eventually it >>>>> wants >>>>> to unwind a stack and get back to the event loop to process. Things like >>>>> the processEvents call can force the event loop to flush, but if you check >>>>> out the docs it will note that deferred deletions do not happen as part of >>>>> a call to processEvents. I presume this is also related to avoiding >>>>> deferred deletions during a deeply nested stack. I have had issues with >>>>> bad >>>>> pointers and clean-ups from a call to processEvents in a deeply nested >>>>> stack, or when an event handler method is part of the stack. So maybe this >>>>> is really just your issue, that you need to allow the event loop to run, >>>>> simply for the UI reasons and not your task reasons. Also, I have no idea >>>>> how Maya manages it's memory in relation to Qt. >>>>> >>>> Actually, what I meant to say here was that I believe you are dealing >>>> with more than one issue: Python garbage collection, Maya's memory >>>> management, or Qt deferred deletion, or the Qt event loop and >>>> updates/redraws. >>>> >>>> I think the python garbage collector runs at both intervals and object >>>> thresholds. If your garbage is coming from python objects and you aren't >>>> explicitly calling delete on them, and instead relying on them to collect >>>> when they fall off the stack, when you do a deeply nested stack that keeps >>>> calling more functions, those objects in the previous stack frames are >>>> still alive until the stack unwinds past their frames. If the garbage is >>>> coming from Qt, then maybe it is related to the deferred deletions piling >>>> up. And if the garbage is Maya, I don't really know very well what controls >>>> its memory management. But it may also be related to the event loop. >>>> >>>> I also noticed that you're using QTimer. Quick info, just so I >>>>> understand, is that similar to .NET's Timer in that it sets up a second >>>>> thread under the hood? And, if so, is that the key to making this work in >>>>> a way that can help with refreshing? I ask because that's the only thing >>>>> I >>>>> could think of that would benefit from smaller functions other than code >>>>> readability. As I understand it, chaining a series of functions together >>>>> (or calling one function after the other) doesn't really change the >>>>> resulting bytecode. >>>>> >>>>> QTimer doesn't create threads. It schedules a callable to run after >>>>> a given amount of time, in the same thread. Using a value of 0 is just a >>>>> way to tell it to run right when the event loop gets back to idle again. >>>>> I'm not necessarily suggesting that a QTimer is the real solution here. >>>>> Only that it was a simple way to illustrate fixing the UI problem in your >>>>> example, by breaking up just the UI deletions and additions. Your example >>>>> couples the task loops with the creation and deletion of the widgets over >>>>> and over in that same stack. Maybe using the QueuedConnection for the >>>>> progress signals might have helped too. I didn't check. But I think a >>>>> different architecture could make it so that your progress UI stuff >>>>> happens >>>>> deferred, separate from the actual work. >>>>> >>>>> Here's my use case: I'm writing a tool to serialize a mesh, and then >>>>> optionally mirror it intelligently. I'm starting with just working across >>>>> the X axis, but from there I plan to add in other modes of >>>>> mirroring--arbitrary plane for sure, possibly even radial. >>>>> >>>>> The steps I'm using are like this: >>>>> 1) Capture the verts. >>>>> 2) Capture the polygon index arrays >>>>> 3) Capture the edge smoothness for each edge. >>>>> 4) Wrap these into a JSON data structure (a dict) for later use. >>>>> 5) Iterate through the vertices and mirror those that should be >>>>> mirrored. Add the mirrored verts to the data structure. >>>>> 6) Iterate through the polys. Mirror. Add. >>>>> 7) Iterate through the edges. Mirror smoothness. Add. >>>>> >>>>> Steps 1-4 are in one function currently, and 5-7 are in another. I >>>>> have a third function that rebuilds a mesh from the resulting data >>>>> structure. As I said, for small meshes it has no issue. The mesh I'm >>>>> using for my testing is actually the beast from Hyper-Real creature >>>>> Creation (the 2005 masterclasses), modeled by a WETA guy, and sitting at >>>>> about 70,000 polys. I figure that if I can make my tool work on it, it's >>>>> about as bulletproof as it can get. >>>>> >>>>> That's the first stage. The second stage, once I have the first part >>>>> done, is to use the serialized mesh data to build a rig. So, I can store >>>>> a >>>>> database of defined meshes, hand off a base mesh to a modeler, bring it >>>>> back in after sculpting, and have my tool build the rig to it as long as >>>>> the modeler didn't mess with the topology. I've seen these systems in the >>>>> wild, but what I haven't seen (although I'm sure it exists) is the part >>>>> I'm >>>>> really focused on, which is tying a graphical workflow setup into it, so >>>>> as >>>>> a rigger I can add an arbitrary mesh and then define how it should build >>>>> the rig, all graphically. All of that, I'm confident in my ability on, >>>>> but >>>>> it all depends on getting this initial processing to work--the whole point >>>>> is that I can define the rig for the portion being mirrored, and then the >>>>> tool can build the rig for the entire system, with all necessary >>>>> mirroring. So, in a nutshell, as a one-man show, I'm trying to replicate >>>>> the rigging systems big houses with a dedicated development team have :). >>>>> Yep, I'm just crazy that way. >>>>> >>>>> So, you mentioned QThreads. Threads are still an area of voodoo to >>>>> me. I work in games, and Unity (the engine my current company uses) >>>>> actively works against you if you even try to bring in threading. I get >>>>> that they're useful for parallel processing, or not blocking the UI, but >>>>> I'm not sure I need them for a sequential series of mesh analyses. If >>>>> using them gives the garbage collector/UI the opportunity to recover >>>>> though, I'm glad to give it a shot. I'd read that Maya doesn't like >>>>> threads either, especially when you're calling the API. I've done basic >>>>> threading before (mostly using BackgroundWorker in .NET to handle keeping >>>>> the UI responsive), but most of the information I've found on threading >>>>> assumes a much higher level of knowledge about it than I have (I only took >>>>> a year of actual CS classes before changing majors to animation, the rest >>>>> is 10 years of experience teaching myself)--so, it says "Here's how you do >>>>> X with Y Language's threading," when I have no idea what X means or why >>>>> I'd >>>>> want to do it. I've always assumed that when I needed it and had a >>>>> specific use case, it would make sense. Any good resources on QThreads >>>>> (or >>>>> how the event loop runs) you could point me towards? Information at that >>>>> level seems to be sparse from my searches. >>>>> >>>>> >>>>> QThreads do make a program more complicated for sure, because you have >>>>> to then consider memory barriers and logic running in completely different >>>>> branches. But the general rule of thumb in UI frameworks with an event >>>>> loop >>>>> is that you aim to not block the main thread and do long running tasks in >>>>> another thread, while communicating over the signals system. Using Qt in >>>>> Maya is a bit different, because like you said there are restrictions on >>>>> being able to make Maya calls in another thread. Those usually have to be >>>>> called through executeInMainThreadWithResult() So that you call into the >>>>> main thread from your other thread. >>>>> >>>>> >>>>> In the meantime, I'll try integrating the example you sent back and >>>>> see if it makes a difference in the results. >>>>> >>>>> Thanks again! >>>>> >>>>> Joe >>>>> >>>>> On 11/10/2014 1:59 AM, Justin Israel wrote: >>>>> >>>>> Hey Joe, >>>>> >>>>> The thing is, since your example is really trivial, I can really only >>>>> comment on the trivial example itself. Normally one wouldn't want to >>>>> process a whole bunch of long running/blocking tasks one after the next, >>>>> in >>>>> a single method, while adding and removing widgets from a layout. The >>>>> layout is obviously having a drawing issue related to timing, from >>>>> everything happening within that same stack. >>>>> >>>>> I can offer a tweak for the example that shows how it could be >>>>> corrected, by splitting up your tasks into pieces that allow the stack to >>>>> return to the event loop again. >>>>> >>>>> http://pastebin.com/5s5mx6KE >>>>> >>>>> In a more real world example, you might abstract these tasks into >>>>> objects emit their own progress (QThread already does this), and then be >>>>> able to chain them up so one starts after the next. Or when one is done, a >>>>> queue is checked and the next pending task runs. Are these tasks that can >>>>> be run in a thread? If so, you can use a thread pool or QThreads and >>>>> coordinate with their signals. >>>>> >>>>> Anyways, basically the issue is that the layout is unhappy having >>>>> itself modified so much from a single stack. >>>>> >>>>> - Justin >>>>> >>>>> >>>>> >>>>> On Mon Nov 10 2014 at 7:03:33 PM Joe Weidenbach <[email protected]> >>>>> wrote: >>>>> >>>>> Hey All, >>>>> >>>>> I'm having an issue with a tool I'm working on regarding slot-based >>>>> widget deletion. >>>>> >>>>> I'm trying to create a UI where my object which handles data processing >>>>> reports it's progress on various (possibly nested) tasks back to the >>>>> UI. It's a mesh processor, so it can be pretty intensive. I'm >>>>> actually >>>>> using these progress bars to try and identify where a memory leak is >>>>> right now, so I can track it down, but it would be dead useful to have >>>>> for general purposes as well. >>>>> >>>>> I've made up a simpler script to demo the situation, here: >>>>> http://pythonfiddle.com/progress-bar-test >>>>> >>>>> The relevant code is in the final class, in the three slots that >>>>> connect >>>>> to PBTester (lines 84-113). I don't think I need to call self.update() >>>>> on every loop, but it does slow things down for the visual of the >>>>> progress bars. >>>>> >>>>> The issue I'm having is that when the processCompleted Signal fires, it >>>>> doesn't seem to actually delete the given progress bar. It's removed >>>>> from the layout, but the visual stays in place and gets overlaid by the >>>>> new progress bar. My guess from the research I've done is that this >>>>> has >>>>> something to do with deleteLater not being called while in the process, >>>>> but I don't understand the event loop well enough yet to make much >>>>> sense >>>>> of it. The closest page I've been able to find recommended calling >>>>> emit() with Qt.QueuedConnection, and I did try that, but I don't think >>>>> I >>>>> was doing it right, as it didn't seem to do anything. I'm not using >>>>> threads either, so I don't think it's a sync issue, but I'm not sure if >>>>> Qt is using them under the hood or not. >>>>> >>>>> Hopefully that makes sense. Any thoughts from the pros? >>>>> >>>>> Thanks, >>>>> >>>>> Joe >>>>> >>>>> --- >>>>> This email is free from viruses and malware because avast! Antivirus >>>>> protection is active. >>>>> http://www.avast.com >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Python Programming for Autodesk Maya" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/python_inside_maya/5460552B.3000500%40gmail.com >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Python Programming for Autodesk Maya" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3AZ8gG9zLc%3Dsa7aH1ME1P%3D9yp9nSnOR7A0ENKmjVDoiQ%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3AZ8gG9zLc%3Dsa7aH1ME1P%3D9yp9nSnOR7A0ENKmjVDoiQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>>> <img >>>>> src="https://ci3.googleusercontent.com/proxy/ugHIP5cOMxDtnT_N1iOHaSA-eaW54QTlGlm6cWeLsTztLmjyZiaQGO5iCzO-MGezy0MDrIjPeV-miBnwZphFX93F4wjDUcXA3w=s0-d-e1-ft#http://static.avast.com/emails/avast-mail-stamp.png";> >>>>> <http://www.avast.com/> >>>>> >>>>> This email is free from viruses and malware because avast! Antivirus >>>>> <http://www.avast.com/> protection is active. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Python Programming for Autodesk Maya" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/python_inside_maya/5460F261.2050708%40gmail.com >>>>> <https://groups.google.com/d/msgid/python_inside_maya/5460F261.2050708%40gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Python Programming for Autodesk Maya" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3-SGo7Cud4BmzSjcN3g-qX3fE5QzE2C%2BUk%2B1gLkFxj6g%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3-SGo7Cud4BmzSjcN3g-qX3fE5QzE2C%2BUk%2B1gLkFxj6g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >> > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAM33%3Da5moNGiUHD%3D1EhqcFbd-vis7xUUwT0aRAUu5oUQoRp0YA%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAM33%3Da5moNGiUHD%3D1EhqcFbd-vis7xUUwT0aRAUu5oUQoRp0YA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA2B4nOn1YMALqzT_aKi4k2%3DvW%3Dku2NZ5xKA4Xhfvhm7vw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
![https://ci3.googleusercontent.com/proxy/ugHIP5cOMxDtnT_N1iOHaSA-eaW54QTlGlm6cWeLsTztLmjyZiaQGO5iCzO-MGezy0MDrIjPeV-miBnwZphFX93F4wjDUcXA3w=s0-d-e1-ft#http://static.avast.com/emails/avast-mail-stamp.png"](https://ci3.googleusercontent.com/proxy/ugHIP5cOMxDtnT_N1iOHaSA-eaW54QTlGlm6cWeLsTztLmjyZiaQGO5iCzO-MGezy0MDrIjPeV-miBnwZphFX93F4wjDUcXA3w=s0-d-e1-ft#http://static.avast.com/emails/avast-mail-stamp.png"){kind=link}