On 12/11/2012 05:33 PM, Jeroen Michiel wrote:
Hi,
I'm having serious trouble getting my DB evolved to a new version. I'm
runnong a Grok 1.4 site using ZODB 3.10.2
The problem happens when I add a new index to a new catalog.
As soon as the index is added, a subscriber from zop.catalog (I believe)
will automatically loop over all objects in the DB trying to index them. For
some reason, it apparently tries to keep all these objects into memory,
while only a very small part of them are effectively need indexing, and even
then, indexing shouldn't touch them.
After some time of running I see the process taking 1.9GB of mem on windows,
(or 3G on linux), and then I first get these errors:
2012-12-11 16:52:56,617 ERROR [ZODB.Connection] Couldn't load state for
0x0a45a2
Traceback (most recent call last):
File
"c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\Connection.py",
line 856, in setstate
self._setstate(obj)
File
"c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\Connection.py",
line 910, in _setstate
self._reader.setGhostState(obj, p)
File
"c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\serialize.py",
line 612, in setGhostState
state = self.getState(pickle)
File
"c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\serialize.py",
line 605, in getState
return unpickler.load()
MemoryError
I have not a single clue why it would need that much memory.
I tried using savepoints, but that doesn't help.
How can I see what exactly is eating all that memory, where do I start
debugging this?
ANY help appreciated!
Well it loads too many objects in a single transaction.
Doing this after some iterations (10k?, depends on your object sizes)
helps usually:
def forceSavepoint(anyPersistentObject=None):
transaction.savepoint(optimistic=True)
if anyPersistentObject is not None:
#and clear picklecache
conn = anyPersistentObject._p_jar
conn.cacheGC()
--
Best regards,
Adam GROSZER
--
Quote of the day:
A liberal is someone too poor to be a capitalist and too rich to be a
communist.
_______________________________________________
Zope3-users mailing list
Zope3-users@zope.org
https://mail.zope.org/mailman/listinfo/zope3-users
