Re: [Zope] help debugging a can't pickle error deep within a catalog reindex

2008-06-19 Thread Dieter Maurer
Rob Miller wrote at 2008-6-17 15:47 -0700:
 ...
Traceback (most recent call last):
 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py,
 
line 407, in serialize
 return self._dump(meta, obj.__getstate__())
   File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py,
 
line 416, in _dump
 self._p.dump(state)
   File copy_reg.py, line 69, in _reduce_ex
 raise TypeError, can't pickle %s objects % base.__name__
TypeError: can't pickle instancemethod objects

Reproduce the problem in an interactive Python interpreter
(bin/zopectl debug on *nix; alternatives for Win*).
Call pdb.pm() (or probably better dm.pdb.zpdb.pm() (dm.pdb
can be found on PyPI)) and go up two call frames.
Look at obj. With a bit of looks, you see which objects
this is and where it comes from.

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


[Zope] help debugging a can't pickle error deep within a catalog reindex

2008-06-17 Thread Rob Miller

hi,

i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with over 
29,000 indexed objects.  it churns for a good long time, and eventually fails 
with a long set of tracebacks, of which i've included a sample at the end of 
this message.


i think i understand the gist of the issue... it's trying to write an object 
(probably a CatalogBrain) to the database, but this object's __dict__ contains 
a value that is of type instancemethod, which isn't allowed for persistent 
objects.


the problem is that i can't figure out which specific objects are causing the 
problem.  i've used pdb.post_mortem to get a debug prompt way down in the 
traceback, but the code goes in and out of C modules, so i'm missing a lot of 
what's happening.  and when i interactively peek at the objects that are being 
indexed when the error happens, there doesn't seem to be anything wrong, and i 
can index the objects w/ no problem.  i've even tried dropping the 
subtransaction threshold down to 1, so it will try to commit a savepoint after 
every object, but none of the objects being indexed seemed to have any problems.


i CAN verify that the instancemethod that is causing the problem renders like 
this:


bound method SessionDataManager.getSessionData of SessionDataManager at 
/session_data_manager


even that hasn't proven enough for me to concretely identify the source of the 
problem, though.


i've been working on this for a full day already, and am not sure how to 
proceed.  does anyone have any debugging tips that might help me figure out 
what, exactly, is causing the reindex attempts to blow up?


thanks!

-r




Traceback (most recent call last):
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py, 
line 296, in refreshCatalog

self.catalog_object(obj, p, pghandler=pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py, 
line 367, in catalog_object

self._increment_counter()
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py, 
line 395, in _increment_counter

self._counter.change(1)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/BTrees/Length.py, 
line 55, in change

self.value += delta
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py, 
line 890, in register

self._register(obj)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py, 
line 900, in _register

self.transaction_manager.get().join(self)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py, 
line 273, in join

self._prior_operation_failed() # doesn't return
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py, 
line 267, in _prior_operation_failed

raise TransactionFailedError(An operation previously failed, 
TransactionFailedError: An operation previously failed, with traceback:

  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZServer/PubCore/ZServerPublisher.py, 
line 25, in __init__

response=b)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 401, in publish_module

environ, debug, request, response)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 202, in publish_module_standard

response = publish(request, module_name, after_list, debug=debug)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 119, in publish

request, bind=1)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/mapply.py, 
line 88, in mapply

if debug is not None: return debug(object,args,context)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py, 
line 42, in call_object

result=apply(object,args) # Type scr to step into published object.
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py, 
line 260, in manage_catalogReindex

self.refreshCatalog(clear=1, pghandler=handler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py, 
line 296, in refreshCatalog

self.catalog_object(obj, p, pghandler=pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py, 
line 385, in catalog_object

update_metadata, pghandler=pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch.py, 
line 96, in catalog_object

uid, idxs, update_metadata, pghandler)
  File 
/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch_utils.py, 
line 6, in call

return getattr(self, PATTERN % __name__)(*args, **kw)
  File