On Mon, May 25, 2009 at 01:44:46PM +0200, Pedro Ferreira wrote: > We're using ZODB for the Indico Project (at CERN), since 2004, without > any kind of problem. However, today, our database went down and we can't > find a way to recover it. This is a major issue, since we have ~4000 > users depending on this application, and we're simply not able to access > the data in any way.
Ouch. > Around 00:30 tonight the database went down, and since, all the > connections are refused. This means that you're using ZEO, right? Have you tried to use strace to see what it's doing? Is it using any CPU time? > We tried to restart the database, but the > script seems to hang, while trying to create the index: > > -rw-r--r-- 1 root root 6396734704 May 25 13:21 dataout.fs > -rw-r--r-- 1 root root 173 May 25 12:21 dataout.fs.index > -rw-r--r-- 1 root root 229755165 May 25 13:22 > dataout.fs.index.index_tmp > -rw-r--r-- 1 root root 7 May 25 12:21 dataout.fs.lock > -rw-r--r-- 1 root root 70956364 May 25 13:21 dataout.fs.tmp > > We tried to do fsrecovery, but it says "0 bytes removed during > recovery", and the result ends up being the same. We tried it in > different machines, with no success. In one of them, after a while > trying to create the index, a Python exception was thrown, saying > "maximum recursion depth exceeded". I'm not intimately familiar with the internals of ZODB. If it's doing object graph traversals recursively, and if your object graph is very deep, you may mitigate this by calling, e.g. sys.setrecursionlimit(2 * sys.getrecursionlimit()) > We're using 3.4 in production, but we've tried with 3.9 and 3.9b0 as > well, with no success. > We're getting kind of desperate, since the same seems to happen with > yesterday's backup, and trying to restore previous backups with repozo > raises a CRC Error. > Has anyone ever experienced this? Any clues on how to solve this problem? > We'd really appreciate you could help us out, since this is becoming a > big issue here at CERN (a lot of people's work depends on this). Good luck! Marius Gedminas -- Linux: The Ultimate NT Service Pack
Description: Digital signature
_______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev