Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
OK. Thanks a lot, once again. Regards, Pedro Jim Fulton wrote: On May 26, 2009, at 9:35 AM, Jim Fulton wrote: ... FileStorage indexes can't be saved after they reach a certain size, where size roughly based on the number of objects. I need to find a way to fix this. The fix was pretty easy. I just checked it in and the fix will be in the next 3.9 release. Jim -- Jim Fulton Zope Corporation -- José Pedro Ferreira (Software Developer, Indico Project) IT-UDS-AVC CERN Geneva, Switzerland Office 513-R-042 Tel. +41 22 76 77159 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On May 26, 2009, at 9:35 AM, Jim Fulton wrote: ... FileStorage indexes can't be saved after they reach a certain size, where size roughly based on the number of objects. I need to find a way to fix this. The fix was pretty easy. I just checked it in and the fix will be in the next 3.9 release. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Hanno Schlichting wrote: Hanno Schlichting recently posted a nice graph showing the persistent structure of a Plone Page object and it's 9 (!) sub-objects. http://blog.hannosch.eu/2009/05/visualizing-persistent-structure-of.html That graph isn't quite correct ;-) workflow_history has DateTime objects in it, and I think they get their own pickle. Nope. DateTime objects are plain old-style classes and don't inherit from persistent.*. Hmm, oh well, my bad... In that case it must just be that their pickled form is huge compared to an int ;-) They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call __init__ and I don't see why the DWIM magic would be needed anyway, since everything has already been parsed. It's actually the list of PersistentMappings that is the problem. Once that gets large (ie: a content object that's been around for a long time) then everything that adds workflow history commits a new copy of that list. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
2009/5/27 Chris Withers ch...@simplistix.co.uk: Laurence Rowe wrote: Jim Fulton wrote: Well said. A feature I'd like to add is the ability to have persistent objects that don't get their own database records, so that you can get the benefit of having them track their changes without incuring the expense of a separate database object. +lots Hanno Schlichting recently posted a nice graph showing the persistent structure of a Plone Page object and it's 9 (!) sub-objects. http://blog.hannosch.eu/2009/05/visualizing-persistent-structure-of.html That graph isn't quite correct ;-) workflow_history has DateTime objects in it, and I think they get their own pickle. I had a major win on one CMFWorkflow project by changing the workflow implementation to use a better data structure *and* store ints instead of DateTime object. CMF should change this... Good point, though it is 'correct' for an object that has not undergone any workflow transitions yet, as is the case here ;) Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Chris Withers wrote: Hanno Schlichting wrote: Nope. DateTime objects are plain old-style classes and don't inherit from persistent.*. Hmm, oh well, my bad... In that case it must just be that their pickled form is huge compared to an int ;-) Sure: len(cPickle.dumps(DateTime.DateTime(), 1)) == 392 Where as their float representation is: len(cPickle.dumps(DateTime.DateTime().timeTime(), 1)) == 10 They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call __init__ and I don't see why the DWIM magic would be needed anyway, since everything has already been parsed. How would a new instance of a class be constructed without calling the init or new? Look at the _instantiate method in pickle.py, when it does: value = klass(*args) What happens on unpickling is that a new DateTime instance representing just now is generated and then that instance is updated with the values from the pickle. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Wed, May 27, 2009 at 12:17:36PM +0200, Hanno Schlichting wrote: Chris Withers wrote: Hanno Schlichting wrote: They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call __init__ and I don't see why the DWIM magic would be needed anyway, since everything has already been parsed. How would a new instance of a class be constructed without calling the init or new? You're cleverly changing the question ;-) *Most* objects are unpickled by calling __new__ followed by __setstate__, but without calling __init__. Chris is therefore understandably surprised. Old-style classes that define __getinitargs__ will get their __init__ called during unpickling, though, and DateTime is such a class. You could also cause the __init__ to be called by defining __reduce__ or __reduce_ex__ appropriately. This is all described in great detail at http://python.org/doc/current/library/pickle.html#pickle-protocol Look at the _instantiate method in pickle.py, when it does: value = klass(*args) What happens on unpickling is that a new DateTime instance representing just now is generated and then that instance is updated with the values from the pickle. Marius Gedminas -- Every nonempty totally-disconnected perfect compact metric space is homeomorphic to the Cantor set. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Marius Gedminas wrote: On Wed, May 27, 2009 at 12:17:36PM +0200, Hanno Schlichting wrote: Chris Withers wrote: Hanno Schlichting wrote: They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call __init__ and I don't see why the DWIM magic would be needed anyway, since everything has already been parsed. How would a new instance of a class be constructed without calling the init or new? You're cleverly changing the question ;-) *Most* objects are unpickled by calling __new__ followed by __setstate__, but without calling __init__. Chris is therefore understandably surprised. Hhm, no. From what I read most objects are created by the following: class _EmptyClass: pass value = _EmptyClass() value.__class__ = klass The __new__ is only called when your new-style class has a __getnewargs__ method, which none of the standard types have. And even then it would only be used for pickle protocol 2, but the ZODB uses protocol 1. Old-style classes that define __getinitargs__ will get their __init__ called during unpickling, though, and DateTime is such a class. Ah, ok. I missed that this only happens in conjunction with __getinitargs__. Thanks, Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Wed, May 27, 2009 at 03:35:26PM +0200, Hanno Schlichting wrote: Marius Gedminas wrote: On Wed, May 27, 2009 at 12:17:36PM +0200, Hanno Schlichting wrote: Chris Withers wrote: Hanno Schlichting wrote: They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call __init__ and I don't see why the DWIM magic would be needed anyway, since everything has already been parsed. How would a new instance of a class be constructed without calling the init or new? You're cleverly changing the question ;-) *Most* objects are unpickled by calling __new__ followed by __setstate__, but without calling __init__. Chris is therefore understandably surprised. Hhm, no. From what I read most objects are created by the following: class _EmptyClass: pass value = _EmptyClass() value.__class__ = klass The __new__ is only called when your new-style class has a __getnewargs__ method, which none of the standard types have. And even then it would only be used for pickle protocol 2, but the ZODB uses protocol 1. It's an Internet Law that every correction must have at least one mistake. Mine was assuming that __new__ is always invoked. Old-style classes that define __getinitargs__ will get their __init__ called during unpickling, though, and DateTime is such a class. Ah, ok. I missed that this only happens in conjunction with __getinitargs__. Marius Gedminas -- Bumper sticker: No radio - Already stolen. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Tres Seaver wrote:
Pedro Ferreira wrote:
Thanks a lot for your help. In fact, it was a matter of increasing the
maximum recursion limit.
There's still an unsolved issue, though. Each time we try to recover a
backup using repozo, we get a CRC error. Is this normal? Has it happened
to anyone?
I don't recall anything like that. Can you provide a traceback?
pferr...@pcituds01 ~/projects/indico/db $ repozo -Rv -r
/afs/cern.ch/project/indico/db_backups/indico/ -D 2009-05-23-21-31-33 -o
data.fs
looking for files between last full backup and 2009-05-23-21-31-33...
files needed to recover state as of 2009-05-23-21-31-33:
/afs/cern.ch/project/indico/db_backups/indico/2009-05-23-21-31-33.fsz
Recovering file to data.fs
Traceback (most recent call last):
File /usr/bin/repozo, line 8, in module
load_entry_point('ZODB3==3.8.1', 'console_scripts', 'repozo')()
File
//usr/lib/python2.5/site-packages/ZODB3-3.8.1-py2.5-linux-i686.egg/ZODB/scripts/repozo.py,
line 513, in main
do_recover(options)
File
//usr/lib/python2.5/site-packages/ZODB3-3.8.1-py2.5-linux-i686.egg/ZODB/scripts/repozo.py,
line 501, in do_recover
reposz, reposum = concat(repofiles, outfp)
File
//usr/lib/python2.5/site-packages/ZODB3-3.8.1-py2.5-linux-i686.egg/ZODB/scripts/repozo.py,
line 263, in concat
bytesread += dofile(func, ifp)
File
//usr/lib/python2.5/site-packages/ZODB3-3.8.1-py2.5-linux-i686.egg/ZODB/scripts/repozo.py,
line 200, in dofile
data = fp.read(todo)
File /usr/lib/python2.5/gzip.py, line 227, in read
self._read(readsize)
File /usr/lib/python2.5/gzip.py, line 292, in _read
self._read_eof()
File /usr/lib/python2.5/gzip.py, line 311, in _read_eof
raise IOError, CRC check failed
IOError: CRC check failed
I guess we have a very large database, for what is normal in ZODB
applications.
Not really: I know of clients whose database routinely grow much larger
than yours (15 Gb, packing down to 6 Gb, right?)
We were wondering if there's any way to optimize the size
(and performance) of such a large database, through the removal of
unused objects and useless data. We perform packs in a weekly basis, but
we're not sure if this is enough, or if there are other ways of
lightening up the DB. Any recommendations regarding this point?
Without knowing anything about the application:
- Check that it is not holding onto old data inappropriately
(e.g., maintaining lots of archival versions of content).
Yes, I think we can do some improvements there. We actually store some
deleted content as a a safeguard, we're considering a major cleanup
operation.
- Check into the catalog / index usage: you may be able to slow
the growth by batching updates, especially to text indexes.
I'm not sure I understand this one... we're not using ZCatalog, nor
full-text indexes, though...
Thanks!
Pedro
--
José Pedro Ferreira
(Software Developer, Indico Project)
IT-UDS-AVC
CERN
Geneva, Switzerland
Office 513-R-042
Tel. +41 22 76 77159
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Jim Fulton wrote:
On May 25, 2009, at 9:23 AM, Pedro Ferreira wrote:
Dear Andreas, Marius,
This means that you're using ZEO, right? Have you tried to use strace
to see what it's doing? Is it using any CPU time?
Yes, we're using ZEO.
It's doing a lot of lseek() and read() calls, i.e.:
read(6, eq\7cdatetime\ndatetime\nq\10(U\n\7\326\t\r\f..., 4096) = 4096
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
read(6, \n_authorGenq9(U\10\0\0\0\0\3\v\375\367q:h\5tQU\t..., 4096)
= 4096
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
And the allocated memory grows up to ~200 MB, data.fs.index.index_tmp is
created, and then the process seems to die and restart (different PID).
What is in the ZEO server log when this happens?
2009-05-25T09:14:47 (30942) opening storage '1' using FileStorage
2009-05-25T09:15:05 Ignoring index for /opt2/indico/db/data.fs
2009-05-25T09:31:15 (31017) opening storage '1' using FileStorage
2009-05-25T09:31:35 Ignoring index for /opt2/indico/db/data.fs
2009-05-25T09:47:38 (31056) opening storage '1' using FileStorage
2009-05-25T09:47:57 Ignoring index for /opt2/indico/db/data.fs
...
We noticed there was a problem when a pack failed (yesterday, around
12:00 CET):
Traceback (most recent call last):
File
/opt/python24/lib/python2.4/site-packages/MaKaC/tools/packDB.py,
line 24, in ?
DBMgr.getInstance().pack(days=1)
File /opt/python24/lib/python2.4/site-packages/MaKaC/common/db.py,
line 135, in pack
self._storage.pack(days=days)
File
/opt/python24/lib/python2.4/site-packages/ZEO/ClientStorage.py,
line 865, in pack
return self._server.pack(t, wait)
File /opt/python24/lib/python2.4/site-packages/ZEO/ServerStub.py,
line 161, in pack
self.rpc.call('pack', t, wait)
File
/opt/python24/lib/python2.4/site-packages/ZEO/zrpc/connection.py,
line 536, in call
raise inst # error raised by server
RuntimeError: maximum recursion depth exceeded
What was in the ZEO server log when this happened?
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45138):
ManagedServerConnection ('137.138.128.213', 45138)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45139):
ManagedServerConnection ('137.138.128.213', 45139)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59721):
ManagedServerConnection ('137.138.128.212', 59721)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59722):
ManagedServerConnection ('137.138.128.212', 59722)
2009-05-24T12:22:54 (137.138.128.213:45138) received handshake 'Z303'
2009-05-24T12:22:54 (28965) Error raised in delayed method
Traceback (most recent call last):
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
1009, in run
result = self._method(*self._args)
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
345, in _pack_impl
self.storage.pack(time, referencesf)
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1372, in pack
self._save_index()
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 252, in _save_index
p.dump(info)
RuntimeError: maximum recursion depth exceeded
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59723):
ManagedServerConnection ('137.138.128.212', 59723)
2009-05-24T12:22:54 (137.138.128.213:45139) received handshake 'Z303'
Also, how many objects are in your database?
Hmmm... I have no idea... is there an easy way of calculating that?
We were packing a 15GB (which normally results in a 6-7 GB) database.
So, we'll try increasing the recursion depth limit (maybe the process is
dying because it reaches the limit?).
Silly question: is there any way to access (read-only) the data without
generating the index?
Yes, if you open the file-storage read-only (in your ZEO conf, use
read-only true in the filestorage section), an index file won't be
written. An in memory index will still be created. I suspect the
problem is occurring when writing the index to disk.
Good, this may be useful, if a problem like this ever happens again
(fingers crossed).
Thanks, once again,
Pedro
--
José Pedro Ferreira
(Software Developer, Indico Project)
IT-UDS-AVC
CERN
Geneva, Switzerland
Office 513-R-042
Tel. +41 22 76 77159
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Tue, May 26, 2009 at 12:00:51PM +0200, Pedro Ferreira wrote:
Jim Fulton wrote:
What was in the ZEO server log when this happened?
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45138):
ManagedServerConnection ('137.138.128.213', 45138)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45139):
ManagedServerConnection ('137.138.128.213', 45139)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59721):
ManagedServerConnection ('137.138.128.212', 59721)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59722):
ManagedServerConnection ('137.138.128.212', 59722)
2009-05-24T12:22:54 (137.138.128.213:45138) received handshake 'Z303'
2009-05-24T12:22:54 (28965) Error raised in delayed method
Traceback (most recent call last):
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
1009, in run
result = self._method(*self._args)
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
345, in _pack_impl
self.storage.pack(time, referencesf)
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1372, in pack
self._save_index()
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 252, in _save_index
p.dump(info)
RuntimeError: maximum recursion depth exceeded
That's shorter than I expected.
OTOH the `p` here probably refers to a cPickle.Pickler object, and I've
never seen recursion depth errors caused by C code before. It makes
sense.
Also, how many objects are in your database?
Hmmm... I have no idea... is there an easy way of calculating that?
There's a script called analyze.py that comes with ZODB. It will give
you summaries grouped by object class of the counts and sizes of all
your persistent objects.
Marius Gedminas
--
The only way to learn a new programming language is by writing programs in it.
-- Brian Kernighan
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On May 26, 2009, at 6:00 AM, Pedro Ferreira wrote:
Jim Fulton wrote:
What was in the ZEO server log when this happened?
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45138):
ManagedServerConnection ('137.138.128.213', 45138)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45139):
ManagedServerConnection ('137.138.128.213', 45139)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59721):
ManagedServerConnection ('137.138.128.212', 59721)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59722):
ManagedServerConnection ('137.138.128.212', 59722)
2009-05-24T12:22:54 (137.138.128.213:45138) received handshake 'Z303'
2009-05-24T12:22:54 (28965) Error raised in delayed method
Traceback (most recent call last):
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
1009, in run
result = self._method(*self._args)
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
345, in _pack_impl
self.storage.pack(time, referencesf)
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1372, in pack
self._save_index()
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 252, in _save_index
p.dump(info)
RuntimeError: maximum recursion depth exceeded
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59723):
ManagedServerConnection ('137.138.128.212', 59723)
2009-05-24T12:22:54 (137.138.128.213:45139) received handshake 'Z303'
That's what I was afraid of.
FileStorage indexes can't be saved after they reach a certain size,
where size roughly based on the number of objects.
I need to find a way to fix this.
Also, how many objects are in your database?
Hmmm... I have no idea... is there an easy way of calculating that?
import ZEO.ClientStorage
len(ZEO.ClientStorage.ClientStorage(addr)
where addr if the address of your storage server as a host, port tuple.
Jim
--
Jim Fulton
Zope Corporation
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Hello Jim, Where's that certain size on the scale? Tuesday, May 26, 2009, 3:35:56 PM, you wrote: JF FileStorage indexes can't be saved after they reach a certain size, JF where size roughly based on the number of objects. -- Best regards, Adam GROSZERmailto:agros...@gmail.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On May 26, 2009, at 10:16 AM, Pedro Ferreira wrote: That's what I was afraid of. FileStorage indexes can't be saved after they reach a certain size, where size roughly based on the number of objects. I need to find a way to fix this. So, from this I infer that our database has grown in such a proportion that we're reaching some of the current limits (?) of ZODB. It's a limit of FileStorage. Is there any other possible failure related to this that can be foreseen? I mean, are there any other limits that can cause problems, because of the large number of objects? It would be very important for us to know so. The only other issue is memory size, since FileStorage keeps its index in memory. Also, how many objects are in your database? Hmmm... I have no idea... is there an easy way of calculating that? import ZEO.ClientStorage len(ZEO.ClientStorage.ClientStorage(addr) where addr if the address of your storage server as a host, port tuple. So, this returns 19283681. Does this include object revisions? That's interesting. We have much larger databases. In any case, it's not such a surprising number, since we have ~73141 event objects and ~344484 contribution objects, plus ~492016 resource objects, and then each one of these may contain authors, and fore sure some associated objects that store different bits of info... So, even if it doesn't include revisions, 19M is not such a surprising number. I've also tried to run the analyze.py script, but it returns me a stream of '''type' object is unsubscriptable errors, due to: classinfo = pickle.loads(record.data)[0] any suggestion? No. Unfortunately, most of the scripts in ZODB aren't tested or documented well and tend to bitrot. Also, is there any documentation about the basic structures of the database available? We found some information spread through different sites, but we couldn't find exhaustive documentation for the API (information about the different kinds of persistent classes, etc...). Is there any documentation on this? No. Comprehensive ZODB documentation is needed. This is an upcoming project for me. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Hi,
Pedro Ferreira wrote:
I've also tried to run the analyze.py script, but it returns me a
stream of '''type' object is unsubscriptable errors, due to:
classinfo = pickle.loads(record.data)[0]
any suggestion?
I personally apply the attached patch to analyze.py, that does not load
pickled data in get_type() to prevent 'unsubscriptable' errors. It is
also much faster.
Regards,
--
Kazuhiko Shiozaki, Nexedi SA Senior Consultant
Nexedi: Consulting and Development of Free / Open Source Software
http://www.nexedi.com
ERP5: Full Featured High End Open Source ERP
http://www.erp5.com
ERP5 Wiki: Developer Zone for ERP5 Community
http://www.erp5.org
--- analyze.py 2009-05-02 21:57:30.0 +0200
+++ analyze.py 2009-05-20 14:01:11.0 +0200
@@ -7,6 +7,7 @@
import sys
import types
from ZODB.FileStorage import FileStorage
+from ZODB.utils import get_pickle_metadata
class Report:
def __init__(self):
@@ -86,21 +87,8 @@
analyze_rec(report, rec)
def get_type(record):
-try:
-classinfo = pickle.loads(record.data)[0]
-except SystemError, err:
-s = str(err)
-mo = re.match('Failed to import class (\S+) from module (\S+)', s)
-if mo is None:
-raise
-else:
-klass, mod = mo.group(1, 2)
-return %s.%s % (mod, klass)
-if isinstance(classinfo, types.TupleType):
-mod, klass = classinfo
-return %s.%s % (mod, klass)
-else:
-return str(classinfo)
+mod, klass = get_pickle_metadata(record.data)
+return %s.%s % (mod, klass)
def analyze_rec(report, record):
oid = record.oid
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Jim Fulton wrote: On May 26, 2009, at 10:16 AM, Pedro Ferreira wrote: In any case, it's not such a surprising number, since we have ~73141 event objects and ~344484 contribution objects, plus ~492016 resource objects, and then each one of these may contain authors, and fore sure some associated objects that store different bits of info... So, even if it doesn't include revisions, 19M is not such a surprising number. I've also tried to run the analyze.py script, but it returns me a stream of '''type' object is unsubscriptable errors, due to: classinfo = pickle.loads(record.data)[0] any suggestion? No. Unfortunately, most of the scripts in ZODB aren't tested or documented well and tend to bitrot. Also, is there any documentation about the basic structures of the database available? We found some information spread through different sites, but we couldn't find exhaustive documentation for the API (information about the different kinds of persistent classes, etc...). Is there any documentation on this? No. Comprehensive ZODB documentation is needed. This is an upcoming project for me. I have a patch at https://bugs.launchpad.net/zodb/+bug/223331 which fixes this. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On May 26, 2009, at 11:17 AM, Laurence Rowe wrote: Jim Fulton wrote: On May 26, 2009, at 10:16 AM, Pedro Ferreira wrote: In any case, it's not such a surprising number, since we have ~73141 event objects and ~344484 contribution objects, plus ~492016 resource objects, and then each one of these may contain authors, and fore sure some associated objects that store different bits of info... So, even if it doesn't include revisions, 19M is not such a surprising number. I've also tried to run the analyze.py script, but it returns me a stream of '''type' object is unsubscriptable errors, due to: classinfo = pickle.loads(record.data)[0] any suggestion? No. Unfortunately, most of the scripts in ZODB aren't tested or documented well and tend to bitrot. Also, is there any documentation about the basic structures of the database available? We found some information spread through different sites, but we couldn't find exhaustive documentation for the API (information about the different kinds of persistent classes, etc...). Is there any documentation on this? No. Comprehensive ZODB documentation is needed. This is an upcoming project for me. I have a patch at https://bugs.launchpad.net/zodb/+bug/223331 which fixes this. Wow! Comprehensive ZODB documentation in a patch. ;) Feel free to check this in. Feel even freer to write a test. :) Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
2009/5/26 Pedro Ferreira jose.pedro.ferre...@cern.ch: In any case, it's not such a surprising number, since we have ~73141 event objects and ~344484 contribution objects, plus ~492016 resource objects, and then each one of these may contain authors, and fore sure some associated objects that store different bits of info... I had a similar problem packing a large 15GB database. This may sound obvious, and is totally application specific, but you might like to look into whether it really is necessary to have so many individual ZODB objects. I found that often I would have a class that inherited from one of the Persistent classes that was part of a larger Persistent class, resulting in an explosion of Persistent objects. It was possible to make many of them non-Persistent, so that they only get serialised and stored in the database as part of whatever larger Persistent object that they're part of. Another thing to watch out for is dangling pointers - given the dynamic nature of Python, it's very easy to leave a pointer hanging around to an old Persistent object that you don't really need to store in the database, which in turn points to more objects, etc. Any object that can be reached in a traversal of the object graph will be kept in the database forever. The lack of a formal schema and rigid type checking make these kind of errors much easier to make, and with today's hardware they don't become a problem until the database grows big. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On May 26, 2009, at 10:21 AM, Adam GROSZER wrote: Hello Jim, Where's that certain size on the scale? Tuesday, May 26, 2009, 3:35:56 PM, you wrote: JF FileStorage indexes can't be saved after they reach a certain size, JF where size roughly based on the number of objects. I shouldn't have used the word certain. :) It's really a limit on the number of buckets in the file-storage index BTree. The number of buckets is related to the number of objects, but as buckets become sparse, there can be more buckets per objects. The bottom line is that I need to fix save_index so this isn't a problem, at any size. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Well said. A feature I'd like to add is the ability to have persistent objects that don't get their own database records, so that you can get the benefit of having them track their changes without incuring the expense of a separate database object. Jim On May 26, 2009, at 11:48 AM, Chris Bainbridge wrote: 2009/5/26 Pedro Ferreira jose.pedro.ferre...@cern.ch: In any case, it's not such a surprising number, since we have ~73141 event objects and ~344484 contribution objects, plus ~492016 resource objects, and then each one of these may contain authors, and fore sure some associated objects that store different bits of info... I had a similar problem packing a large 15GB database. This may sound obvious, and is totally application specific, but you might like to look into whether it really is necessary to have so many individual ZODB objects. I found that often I would have a class that inherited from one of the Persistent classes that was part of a larger Persistent class, resulting in an explosion of Persistent objects. It was possible to make many of them non-Persistent, so that they only get serialised and stored in the database as part of whatever larger Persistent object that they're part of. Another thing to watch out for is dangling pointers - given the dynamic nature of Python, it's very easy to leave a pointer hanging around to an old Persistent object that you don't really need to store in the database, which in turn points to more objects, etc. Any object that can be reached in a traversal of the object graph will be kept in the database forever. The lack of a formal schema and rigid type checking make these kind of errors much easier to make, and with today's hardware they don't become a problem until the database grows big. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Jim Fulton wrote: Well said. A feature I'd like to add is the ability to have persistent objects that don't get their own database records, so that you can get the benefit of having them track their changes without incuring the expense of a separate database object. +lots Hanno Schlichting recently posted a nice graph showing the persistent structure of a Plone Page object and it's 9 (!) sub-objects. http://blog.hannosch.eu/2009/05/visualizing-persistent-structure-of.html A sub-persitent type would allow us to fix the latency problems we experience without needing to re-engineer Archetypes at the same time. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Jim Fulton wrote: FileStorage indexes can't be saved after they reach a certain size, where size roughly based on the number of objects. I need to find a way to fix this. It might be interesting to use SQLite for FileStorage indexes. With SQLite, we wouldn't have to store the whole index as a unit. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Tue, May 26, 2009 at 04:16:29PM +0200, Pedro Ferreira wrote: Also, is there any documentation about the basic structures of the database available? We found some information spread through different sites, but we couldn't find exhaustive documentation for the API (information about the different kinds of persistent classes, etc...). Is there any documentation on this? No, but you have something even better than documentation -- the source! More seriously, when I needed it, I've been able to get enough information about the FileStorage internal structure from code comments and docstrings, and some jumping around following the flow of functions. Marius Gedminas -- I read I forget; I see I remember; I teach I understand. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Tue, May 26, 2009 at 1:58 PM, Shane Hathaway sh...@hathawaymix.org wrote: Jim Fulton wrote: FileStorage indexes can't be saved after they reach a certain size, where size roughly based on the number of objects. I need to find a way to fix this. It might be interesting to use SQLite for FileStorage indexes. With SQLite, we wouldn't have to store the whole index as a unit. Or possibly BDB? The indices appear to be fairly straight forward to store. I know people have had nightmares with BDB. We use it in our Enfold Proxy cache and it works fine. We have not had corruption or other issues that tend to come along with BDB usage. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Laurence Rowe wrote: Jim Fulton wrote: Well said. A feature I'd like to add is the ability to have persistent objects that don't get their own database records, so that you can get the benefit of having them track their changes without incuring the expense of a separate database object. +lots Hanno Schlichting recently posted a nice graph showing the persistent structure of a Plone Page object and it's 9 (!) sub-objects. http://blog.hannosch.eu/2009/05/visualizing-persistent-structure-of.html That graph isn't quite correct ;-) workflow_history has DateTime objects in it, and I think they get their own pickle. I had a major win on one CMFWorkflow project by changing the workflow implementation to use a better data structure *and* store ints instead of DateTime object. CMF should change this... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Chris Withers wrote: Laurence Rowe wrote: Jim Fulton wrote: Well said. A feature I'd like to add is the ability to have persistent objects that don't get their own database records, so that you can get the benefit of having them track their changes without incuring the expense of a separate database object. +lots Hanno Schlichting recently posted a nice graph showing the persistent structure of a Plone Page object and it's 9 (!) sub-objects. http://blog.hannosch.eu/2009/05/visualizing-persistent-structure-of.html That graph isn't quite correct ;-) workflow_history has DateTime objects in it, and I think they get their own pickle. Nope. DateTime objects are plain old-style classes and don't inherit from persistent.*. They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Hello, We're using ZODB for the Indico Project (at CERN), since 2004, without any kind of problem. However, today, our database went down and we can't find a way to recover it. This is a major issue, since we have ~4000 users depending on this application, and we're simply not able to access the data in any way. Around 00:30 tonight the database went down, and since, all the connections are refused. We tried to restart the database, but the script seems to hang, while trying to create the index: -rw-r--r-- 1 root root 6396734704 May 25 13:21 dataout.fs -rw-r--r-- 1 root root 173 May 25 12:21 dataout.fs.index -rw-r--r-- 1 root root 229755165 May 25 13:22 dataout.fs.index.index_tmp -rw-r--r-- 1 root root 7 May 25 12:21 dataout.fs.lock -rw-r--r-- 1 root root70956364 May 25 13:21 dataout.fs.tmp We tried to do fsrecovery, but it says 0 bytes removed during recovery, and the result ends up being the same. We tried it in different machines, with no success. In one of them, after a while trying to create the index, a Python exception was thrown, saying maximum recursion depth exceeded. We're using 3.4 in production, but we've tried with 3.9 and 3.9b0 as well, with no success. We're getting kind of desperate, since the same seems to happen with yesterday's backup, and trying to restore previous backups with repozo raises a CRC Error. Has anyone ever experienced this? Any clues on how to solve this problem? We'd really appreciate you could help us out, since this is becoming a big issue here at CERN (a lot of people's work depends on this). Thank you very much in advance, Best regards, Pedro Ferreira -- José Pedro Ferreira (Software Developer, Indico Project) IT-UDS-AVC CERN Geneva, Switzerland Office 513-R-042 Tel. +41 22 76 77159 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Mon, May 25, 2009 at 01:44:46PM +0200, Pedro Ferreira wrote: We're using ZODB for the Indico Project (at CERN), since 2004, without any kind of problem. However, today, our database went down and we can't find a way to recover it. This is a major issue, since we have ~4000 users depending on this application, and we're simply not able to access the data in any way. Ouch. Around 00:30 tonight the database went down, and since, all the connections are refused. This means that you're using ZEO, right? Have you tried to use strace to see what it's doing? Is it using any CPU time? We tried to restart the database, but the script seems to hang, while trying to create the index: -rw-r--r-- 1 root root 6396734704 May 25 13:21 dataout.fs -rw-r--r-- 1 root root 173 May 25 12:21 dataout.fs.index -rw-r--r-- 1 root root 229755165 May 25 13:22 dataout.fs.index.index_tmp -rw-r--r-- 1 root root 7 May 25 12:21 dataout.fs.lock -rw-r--r-- 1 root root70956364 May 25 13:21 dataout.fs.tmp We tried to do fsrecovery, but it says 0 bytes removed during recovery, and the result ends up being the same. We tried it in different machines, with no success. In one of them, after a while trying to create the index, a Python exception was thrown, saying maximum recursion depth exceeded. I'm not intimately familiar with the internals of ZODB. If it's doing object graph traversals recursively, and if your object graph is very deep, you may mitigate this by calling, e.g. sys.setrecursionlimit(2 * sys.getrecursionlimit()) We're using 3.4 in production, but we've tried with 3.9 and 3.9b0 as well, with no success. We're getting kind of desperate, since the same seems to happen with yesterday's backup, and trying to restore previous backups with repozo raises a CRC Error. Has anyone ever experienced this? Any clues on how to solve this problem? We'd really appreciate you could help us out, since this is becoming a big issue here at CERN (a lot of people's work depends on this). Good luck! Marius Gedminas -- Linux: The Ultimate NT Service Pack signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Dear Andreas, Marius,
This means that you're using ZEO, right? Have you tried to use strace
to see what it's doing? Is it using any CPU time?
Yes, we're using ZEO.
It's doing a lot of lseek() and read() calls, i.e.:
read(6, eq\7cdatetime\ndatetime\nq\10(U\n\7\326\t\r\f..., 4096) = 4096
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
read(6, \n_authorGenq9(U\10\0\0\0\0\3\v\375\367q:h\5tQU\t..., 4096) = 4096
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
And the allocated memory grows up to ~200 MB, data.fs.index.index_tmp is
created, and then the process seems to die and restart (different PID).
It seems to go up to 100% for a significant time (~20 min), then slowly
goes down (moment in which some data seems to be written to index_tmp),
and then comes back to 100% again, repeating it maybe a couple of times
before dying and starting all over again.
We tried to restart the database, but the
script seems to hang, while trying to create the index:
-rw-r--r-- 1 root root 6396734704 May 25 13:21 dataout.fs
-rw-r--r-- 1 root root 173 May 25 12:21 dataout.fs.index
-rw-r--r-- 1 root root 229755165 May 25 13:22
dataout.fs.index.index_tmp
-rw-r--r-- 1 root root 7 May 25 12:21 dataout.fs.lock
-rw-r--r-- 1 root root70956364 May 25 13:21 dataout.fs.tmp
We tried to do fsrecovery, but it says 0 bytes removed during
recovery, and the result ends up being the same. We tried it in
different machines, with no success. In one of them, after a while
trying to create the index, a Python exception was thrown, saying
maximum recursion depth exceeded.
I'm not intimately familiar with the internals of ZODB. If it's doing
object graph traversals recursively, and if your object graph is very
deep, you may mitigate this by calling, e.g.
sys.setrecursionlimit(2 * sys.getrecursionlimit())
OK, we'll give it a try. Thanks a lot!
We noticed there was a problem when a pack failed (yesterday, around
12:00 CET):
Traceback (most recent call last):
File /opt/python24/lib/python2.4/site-packages/MaKaC/tools/packDB.py, line
24, in ?
DBMgr.getInstance().pack(days=1)
File /opt/python24/lib/python2.4/site-packages/MaKaC/common/db.py, line
135, in pack
self._storage.pack(days=days)
File /opt/python24/lib/python2.4/site-packages/ZEO/ClientStorage.py, line
865, in pack
return self._server.pack(t, wait)
File /opt/python24/lib/python2.4/site-packages/ZEO/ServerStub.py, line 161,
in pack
self.rpc.call('pack', t, wait)
File /opt/python24/lib/python2.4/site-packages/ZEO/zrpc/connection.py, line
536, in call
raise inst # error raised by server
RuntimeError: maximum recursion depth exceeded
We were packing a 15GB (which normally results in a 6-7 GB) database.
So, we'll try increasing the recursion depth limit (maybe the process is
dying because it reaches the limit?).
Silly question: is there any way to access (read-only) the data without
generating the index?
Thanks, once again,
We really appreciate your help.
Regards,
Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
--
José Pedro Ferreira
(Software Developer, Indico Project)
IT-UDS-AVC
CERN
Geneva, Switzerland
Office 513-R-042
Tel. +41 22 76 77159
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Dear all,
Thanks a lot for your help. In fact, it was a matter of increasing the
maximum recursion limit.
There's still an unsolved issue, though. Each time we try to recover a
backup using repozo, we get a CRC error. Is this normal? Has it happened
to anyone?
I guess we have a very large database, for what is normal in ZODB
applications. We were wondering if there's any way to optimize the size
(and performance) of such a large database, through the removal of
unused objects and useless data. We perform packs in a weekly basis, but
we're not sure if this is enough, or if there are other ways of
lightening up the DB. Any recommendations regarding this point?
Thanks in advance, and, once again, thanks a lot for the precious help,
Best regards,
Pedro
Pedro Ferreira wrote:
Dear Andreas, Marius,
This means that you're using ZEO, right? Have you tried to use strace
to see what it's doing? Is it using any CPU time?
Yes, we're using ZEO.
It's doing a lot of lseek() and read() calls, i.e.:
read(6, eq\7cdatetime\ndatetime\nq\10(U\n\7\326\t\r\f..., 4096) = 4096
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
read(6, \n_authorGenq9(U\10\0\0\0\0\3\v\375\367q:h\5tQU\t..., 4096) = 4096
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
And the allocated memory grows up to ~200 MB, data.fs.index.index_tmp is
created, and then the process seems to die and restart (different PID).
It seems to go up to 100% for a significant time (~20 min), then slowly
goes down (moment in which some data seems to be written to index_tmp),
and then comes back to 100% again, repeating it maybe a couple of times
before dying and starting all over again.
We tried to restart the database, but the
script seems to hang, while trying to create the index:
-rw-r--r-- 1 root root 6396734704 May 25 13:21 dataout.fs
-rw-r--r-- 1 root root 173 May 25 12:21 dataout.fs.index
-rw-r--r-- 1 root root 229755165 May 25 13:22
dataout.fs.index.index_tmp
-rw-r--r-- 1 root root 7 May 25 12:21 dataout.fs.lock
-rw-r--r-- 1 root root70956364 May 25 13:21 dataout.fs.tmp
We tried to do fsrecovery, but it says 0 bytes removed during
recovery, and the result ends up being the same. We tried it in
different machines, with no success. In one of them, after a while
trying to create the index, a Python exception was thrown, saying
maximum recursion depth exceeded.
I'm not intimately familiar with the internals of ZODB. If it's doing
object graph traversals recursively, and if your object graph is very
deep, you may mitigate this by calling, e.g.
sys.setrecursionlimit(2 * sys.getrecursionlimit())
OK, we'll give it a try. Thanks a lot!
We noticed there was a problem when a pack failed (yesterday, around
12:00 CET):
Traceback (most recent call last):
File /opt/python24/lib/python2.4/site-packages/MaKaC/tools/packDB.py,
line 24, in ?
DBMgr.getInstance().pack(days=1)
File /opt/python24/lib/python2.4/site-packages/MaKaC/common/db.py, line
135, in pack
self._storage.pack(days=days)
File /opt/python24/lib/python2.4/site-packages/ZEO/ClientStorage.py, line
865, in pack
return self._server.pack(t, wait)
File /opt/python24/lib/python2.4/site-packages/ZEO/ServerStub.py, line
161, in pack
self.rpc.call('pack', t, wait)
File /opt/python24/lib/python2.4/site-packages/ZEO/zrpc/connection.py,
line 536, in call
raise inst # error raised by server
RuntimeError: maximum recursion depth exceeded
We were packing a 15GB (which normally results in a 6-7 GB) database.
So, we'll try increasing the recursion depth limit (maybe the process is
dying because it reaches the limit?).
Silly question: is there any way to access (read-only) the data without
generating the index?
Thanks, once again,
We really appreciate your help.
Regards,
Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
--
José Pedro Ferreira
(Software Developer, Indico Project)
IT-UDS-AVC
CERN
Geneva, Switzerland
Office 513-R-042
Tel. +41 22 76 77159
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Pedro Ferreira wrote: Thanks a lot for your help. In fact, it was a matter of increasing the maximum recursion limit. There's still an unsolved issue, though. Each time we try to recover a backup using repozo, we get a CRC error. Is this normal? Has it happened to anyone? I don't recall anything like that. Can you provide a traceback? I guess we have a very large database, for what is normal in ZODB applications. Not really: I know of clients whose database routinely grow much larger than yours (15 Gb, packing down to 6 Gb, right?) We were wondering if there's any way to optimize the size (and performance) of such a large database, through the removal of unused objects and useless data. We perform packs in a weekly basis, but we're not sure if this is enough, or if there are other ways of lightening up the DB. Any recommendations regarding this point? Without knowing anything about the application: - - Check that it is not holding onto old data inappropriately (e.g., maintaining lots of archival versions of content). - - Check into the catalog / index usage: you may be able to slow the growth by batching updates, especially to text indexes. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKGsvu+gerLs4ltQ4RApw+AKCOHgdPvKFsEXYnGRngrKLkwz5JrwCeNk+9 6RG4zZcUSw881nEWw5qKAEE= =biZk -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
Pedro Ferreira wrote: Dear all, Thanks a lot for your help. In fact, it was a matter of increasing the maximum recursion limit. There's still an unsolved issue, though. Each time we try to recover a backup using repozo, we get a CRC error. Is this normal? Has it happened to anyone? I guess we have a very large database, for what is normal in ZODB applications. We were wondering if there's any way to optimize the size (and performance) of such a large database, through the removal of unused objects and useless data. We perform packs in a weekly basis, but we're not sure if this is enough, or if there are other ways of lightening up the DB. Any recommendations regarding this point? You might want to try packing without garbage collection, which is a much cheaper operation. See http://mail.zope.org/pipermail/zodb-dev/2009-January/012365.html Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On May 25, 2009, at 9:23 AM, Pedro Ferreira wrote:
Dear Andreas, Marius,
This means that you're using ZEO, right? Have you tried to use
strace
to see what it's doing? Is it using any CPU time?
Yes, we're using ZEO.
It's doing a lot of lseek() and read() calls, i.e.:
read(6, eq\7cdatetime\ndatetime\nq\10(U\n\7\326\t\r\f..., 4096) =
4096
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
lseek(6, 3156590592, SEEK_SET) = 3156590592
read(6, \n_authorGenq9(U\10\0\0\0\0\3\v\375\367q:h\5tQU\t...,
4096) = 4096
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
lseek(6, 3156594688, SEEK_SET) = 3156594688
And the allocated memory grows up to ~200 MB,
data.fs.index.index_tmp is
created, and then the process seems to die and restart (different
PID).
What is in the ZEO server log when this happens?
...
We noticed there was a problem when a pack failed (yesterday, around
12:00 CET):
Traceback (most recent call last):
File /opt/python24/lib/python2.4/site-packages/MaKaC/tools/
packDB.py, line 24, in ?
DBMgr.getInstance().pack(days=1)
File /opt/python24/lib/python2.4/site-packages/MaKaC/common/
db.py, line 135, in pack
self._storage.pack(days=days)
File /opt/python24/lib/python2.4/site-packages/ZEO/
ClientStorage.py, line 865, in pack
return self._server.pack(t, wait)
File /opt/python24/lib/python2.4/site-packages/ZEO/ServerStub.py,
line 161, in pack
self.rpc.call('pack', t, wait)
File /opt/python24/lib/python2.4/site-packages/ZEO/zrpc/
connection.py, line 536, in call
raise inst # error raised by server
RuntimeError: maximum recursion depth exceeded
What was in the ZEO server log when this happened?
Also, how many objects are in your database?
We were packing a 15GB (which normally results in a 6-7 GB) database.
So, we'll try increasing the recursion depth limit (maybe the
process is
dying because it reaches the limit?).
Silly question: is there any way to access (read-only) the data
without
generating the index?
Yes, if you open the file-storage read-only (in your ZEO conf, use
read-only true in the filestorage section), an index file won't be
written. An in memory index will still be created. I suspect the
problem is occurring when writing the index to disk.
Jim
--
Jim Fulton
Zope Corporation
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
