Rob Miller <[EMAIL PROTECTED]> writes: > Ross Patterson wrote: >> Rob Miller <[EMAIL PROTECTED]> writes: >> >>> hi, >>> >>> i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with >>> over 29,000 indexed objects. it churns for a good long time, and >>> eventually fails with a long set of tracebacks, of which i've included >>> a sample at the end of this message. >>> >>> i think i understand the gist of the issue... it's trying to write an >>> object (probably a CatalogBrain) to the database, but this object's >>> __dict__ contains a value that is of type instancemethod, which isn't >>> allowed for persistent objects. >>> >>> the problem is that i can't figure out which specific objects are >>> causing the problem. i've used pdb.post_mortem to get a debug prompt >>> way down in the traceback, but the code goes in and out of C modules, >>> so i'm missing a lot of what's happening. and when i interactively >>> peek at the objects that are being indexed when the error happens, >>> there doesn't seem to be anything wrong, and i can index the objects >>> w/ no problem. i've even tried dropping the subtransaction threshold >>> down to 1, so it will try to commit a savepoint after every object, >>> but none of the objects being indexed seemed to have any problems. >>> >>> i CAN verify that the instancemethod that is causing the problem >>> renders like this: >>> >>> <bound method SessionDataManager.getSessionData of <SessionDataManager >>> at /session_data_manager>> >>> >>> even that hasn't proven enough for me to concretely identify the >>> source of the problem, though. >>> >>> i've been working on this for a full day already, and am not sure how >>> to proceed. does anyone have any debugging tips that might help me >>> figure out what, exactly, is causing the reindex attempts to blow up? >>> >>> thanks! >> >> Can you get a pdb.post_mortem prompt at the actual ZODB/serialize.py:416 >> error or only at transaction/_transaction.py:267 where some sort of >> previous error is handled? > > yes, i put the post_mortem around the "transaction.savepoint" call at > line 559 of Products/ZCatalog/ZCatalog.py. this is the place where > the error is happening, but it's where the subtransaction threshold > has been reached, so IIUC the error could be caused by any of the > objects in the subtransaction.
Then it seems like you're not getting at the actual pickle error and thus aren't looking at the object whose __getstate__ returns the offending instance method. If you have enough memory, try running the refreshCatalog with subtransactions disabled to get the *actual* pickle error. >> If the former, I often find it informative to step up to >> ZODB/serialize.py:407 where obj.__getstate__() is called and find out >> what obj is. Is it the same object every time? If not, is it of the >> same type every time? > > right. as long as the subtxn threshold number is the same, then the > object is the same. but if i change the subtxn threshold, then the > object will be different. as i said, i even tried reducing the subtxn > threshold to '1', so, assuming my understanding is correct, this would > be triggered after every object. it took many hours to get to my > debug prompt; when i did, the obj was FileAttachment object that > didn't seem to have any problems. i was able to interactively index > the object from the pdb prompt, and even commit the transaction, w/ no > problems. The actual error of your TB is the error that savepoint raises when a previous error has occurred. So changing the subtransaction threshold just changes which object is being indexed when the savepoint *happens* to be run and notices that an error has been raised for a previous object in the savepoint. At least, that's my read of it. >> A next step can also be to put a pdb.set_trace() at >> transaction/_transaction.py:340 in the register() method such that >> it's is only called when the offending object is added to the >> transaction. > > i'll give this a shot, thx. > >> Hope some of that helps, > > even just having someone to discuss it with is helpful, for my sanity > at least... ;-) Well, then I'll have to enjoy your sanity vicariously. :) Ross _______________________________________________ Zope maillist - [email protected] http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
