While I cant comment in terms of direct experience with this, it seems that according to the docs, passing the modifier SHOULD populate it with the undoable command. This is kind of a weird shot in the dark but its work a try. I noticed for some reason the docs omit referencing the modifier input on setVertexColors, whereas on setVertexColor it is included. It makes me wonder if the setVertexColors is possibly ignoring the modified parameter. Have you tried iterating over your vertex list and calling:
MStatus<http://download.autodesk.com/us/maya/2010help/API/class_m_status.html> MFnMesh::setVertexColor(MColor<http://download.autodesk.com/us/maya/2010help/API/class_m_color.html> & *color*,int *vertexIndex*,MDGModifier<http://download.autodesk.com/us/maya/2010help/API/class_m_d_g_modifier.html> * *modifier* = NULL ) ... just for poops and giggles? But then again, the approach of maintaining the previous values manually is pretty much the purpose of the undoIt() in the command. So if this really doesn't work with the modifier.undoIt(), I think its totally appropriate to remember the state and manually restore it, which is what you would have to do in any command if you werent using the MDGModifier. On Fri, Nov 11, 2011 at 5:19 PM, AK Eric <[email protected]> wrote: > I wrote a tool (Python function, not scripted plugin) that assigns > vert colors via the API, through MFnMesh.setVertexColor(). Works > great, really fast. Problem is, it won't undo. > Unless I'm missing something, I've deduced that when API calls modify > the DG (add\delete nodes\connections, color verts, etc) unless they're > authored via a scripted plugin supporting undo (via MDGModifier), you > can't undo them. If anyone has a workaround for /that/, I'd love to > hear it. But I digress... > > So I went about re-authoring this function as a scripted plugin so I > can support undo. However, it's still not undoing... properly. It > "sort of undo's", more on that below. > > Other scripted plugins I've written that use the functionality of the > methods on a MDGModifier (create, rename, delete, etc), I can get them > to undo just fine: > http://download.autodesk.com/us/maya/2010help/API/class_m_d_g_modifier.html > > But the MDGModifier has nothing for setting vert colors, that's where > MFnMesh comes in obviously. MFnMesh has a setVertexColors() method: > > http://download.autodesk.com/us/maya/2010help/API/class_m_fn_mesh.html#7226f44a9d5d14689a8f1f7c822d6d0f > It can optionally take a MDGModifier reference. But if I pass one in > or not seems to have no effect on the undo operation: No undo > whatsoever. > > So as a workaround, I've gone about authoring code that will remember > the previous vert colors & store them (during redoIt()), so that when > the command is undone (and the undoIt() method is called), that state > is restored. And it works... sort of. The first time yes, but after > that it gets pretty hairy. > > Any thoughts on this? Anyone been successful at passing a MDGModifier > reference to a MFnMesh.serVertexColors() method and getting undo to > properly work? > > thanks > > -- > view archives: http://groups.google.com/group/python_inside_maya > change your subscription settings: > http://groups.google.com/group/python_inside_maya/subscribe > -- view archives: http://groups.google.com/group/python_inside_maya change your subscription settings: http://groups.google.com/group/python_inside_maya/subscribe
