Re: [Zope-dev] Packing (still) hosed in 2.6.2?
Woohoo! Fixed. I created an empty file, then used copyTransactionsFrom to copy all transactions across. This patched the backpointers. fsrecover.py should probably get some smarts to do something like this as well, but I've spent far too long on this problem already... :-/ from ZODB import FileStorage infilename='/app/zope/dev_csr_server/recover/Data.fs' outfilename='/app/zope/dev_csr_server/recover/Out.fs' outfs = FileStorage.FileStorage(outfilename) infs = FileStorage.FileStorage(infilename) outfs.copyTransactionsFrom(infs) I have some small patches to fsdump that outputs expected prev records alongside the actual ones, is this useful enough to check in? Anthony ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Packing (still) hosed in 2.6.2?
Jeremy Hylton wrote I don't recall that any of the pack bugs that we fixed caused this sort of error. My first guess would be that the file is a bit damaged, perhaps in a way that the old pack did not check. Can you run fsdump.py and see if that output sheds any light. It will give you a detailed text summary of all the transactions and data records. You could use the dump to figure out if, for example, the two numbers in the error message are indeed transaction header locations. Ok. The original error, reformatted for clarity: ZODB.fspack.CorruptedError: /app/zope/dev_csr_server/recover/Data.fs:9200838: data record does not point to transaction header: 17391715 != 9200760 Here's the 3 transactions (one before, dead one, one after). There's definately a screwed transaction there. offset: 9198385 end pos: 9200752 transaction id: 033bb0f41f05f3aa trec len: 2367 status: ' ' user: ' jasonb' description: '/admin_tools/accounts/update_account/manage_edit' len(extra): 0 offset: 9198463 oid: 000206bb revid: 033bb0f41f05f3aa previous record offset: 9193826 transaction offset: 9198385 len(data): 2247 redundant trec len: 2367 offset: 9200760 end pos: 9202513 transaction id: 033bb0f4b80ad42a trec len: 1753 status: 'u' user: ' jasonb' description: '/admin_tools/accounts/update_account/manage_edit' len(extra): 0 offset: 9200838 oid: 000206bb revid: 033bb0f4b80ad42a previous record offset: 17389418 # woopsie transaction offset: 17391715 # woopsie len(data): 1633 redundant trec len: 1753 offset: 9202521 end pos: 9204883 transaction id: 033bb0f7be87736e trec len: 2362 status: ' ' user: ' jasonb' description: '/admin_tools/accounts/update_account/manage_edit' len(extra): 0 offset: 9202599 oid: 000206bb revid: 033bb0f7be87736e previous record offset: 9198463 transaction offset: 9202521 len(data): 2242 redundant trec len: 2362 So, fixing this. fsrecover.py doesn't do the right thing, unfortunately. It just leaves the broken record alone. I'm not sure if making fsrecover handle this class of error is useful... It should be simply a matter of walking through the transactions, and patching the one that's busted. Or alternately, simply skipping it. I'll look into fixing this now. More in a bit. Anthony ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Packing (still) hosed in 2.6.2?
I don't recall that any of the pack bugs that we fixed caused this sort of error. My first guess would be that the file is a bit damaged, perhaps in a way that the old pack did not check. Can you run fsdump.py and see if that output sheds any light. It will give you a detailed text summary of all the transactions and data records. You could use the dump to figure out if, for example, the two numbers in the error message are indeed transaction header locations. Jeremy -Original Message- From: Anthony Baxter [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 12, 2003 4:18 AM To: [EMAIL PROTECTED] Subject: [Zope-dev] Packing (still) hosed in 2.6.2? I remember that 2.6 had issues with packing, but I was under the impression that they'd been fixed... However, attempting to pack a Data.fs in 2.6-current-cvs fails for me: File /export/01/zope/dev_csr_code/lib/python/ZODB/DB.py, line 526, in pack try: self._storage.pack(t,referencesf) File /export/01/zope/dev_csr_code/lib/python/ZODB/FileStorage.py, line 1503, in pack opos = p.pack() File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 680, in pack self.gc.findReachable() File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 464, in findReachable self.buildPackIndex() File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 486, in buildPackIndex self.checkData(th, tpos, dh, pos) File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 189, in checkData : %d != %d, dh.tloc, tpos) File /export/01/zope/dev_csr_code/lib/python/ZODB/fspack.py, line 170, in fail raise CorruptedError(s) ZODB.fspack.CorruptedError: /app/zope/dev_csr_server/var/Data.fs.packtest:9200838:data record does not point to transaction header: 17391715 != 9200760 This is reproducible. The particular Zope that uses this file was shut down cleanly before attempting to pack the DB. The code that fails to pack is: import ZODB from ZODB import FileStorage filename='/app/zope/dev_csr_server/var/Data.fs.packtest' store = FileStorage.FileStorage(filename) db = ZODB.DB(store) db.pack() # boom. Help? I'm using the ZODB that's in the current 2.6 branch of CVS - according to ZODB/__init__.py it's version 3.1.2. Is a newer version of ZODB going to help here? Anthony ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Packing (still) hosed in 2.6.2?
Anthony Baxter wrote at 2003-8-12 18:18 +1000: I remember that 2.6 had issues with packing, but I was under the impression that they'd been fixed... However, attempting to pack a Data.fs in 2.6-current-cvs fails for me: ... raise CorruptedError(s) ZODB.fspack.CorruptedError: /app/zope/dev_csr_server/var/Data.fs.packtest:9200838:data record does not point to transaction header: 17391715 != 9200760 This is reproducible. The particular Zope that uses this file was shut down cleanly before attempting to pack the DB. The code that fails to pack is: Maybe, the Data.fs was already corrupted before you switched to 2.6.2. Does the problem remain after a fsrecover? Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
