Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 11:24, Shane Hathaway wrote: On 01/27/2011 03:57 AM, Chris Withers wrote: It would also be *really* handy if zodbpack could run off a normal zope.conf for both logging and storage config (although, in my case, I'd then need to be able to specify which storage to pack, I want to avoid packing one of them!) Please note that the zodbpack utility is simple, short, and has no extra dependencies. I prefer to keep it that way. If you want to do something more interesting, please fork zodbpack. Fair enough, simple is good for me :-) If errors or logging occur inside zodbpack, where do they go? However, more than happy to poke, just tell me where and for what... Is the problem repeatable? Yes. If you start with the same database twice and pack, do you end up with POSKeyErrors on the same OIDs? Not sure. I know that's probably a long test to run, but I'm not yet sure what else to suggest. Okay, some more info... I re-converted the .fs file for this storage. This time, I've taken a mysqldump backup. I then packed, and *then* started the batch process: no problems. While the batch was running, I tried packing again but the pack finished without any problems. Okay, so next up I restored the backup. This time I set the batch running and *then* started the pack. This eventually caused the POSKeyErrors to begin again. I'm going to have a go at knocking up a small batch script that you can run to reproduce the issue, but there's definitely an issue here when packing while changing lots of data in a ZODB. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/28/2011 02:21 AM, Chris Withers wrote: Okay, so next up I restored the backup. This time I set the batch running and *then* started the pack. This eventually caused the POSKeyErrors to begin again. I'm going to have a go at knocking up a small batch script that you can run to reproduce the issue, but there's definitely an issue here when packing while changing lots of data in a ZODB. I look forward to your results! Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 03:15, Shane Hathaway wrote: Okay, so I'll do: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? The MySQL filesystem could be unmounted (after I shut down MySQL!), so I ran e2fsck on it: e2fsck -f /dev/sdb1 e2fsck 1.41.3 (12-Oct-2008) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 166/16777216 files (10.2% non-contiguous), 8244979/67107513 blocks - mysqlcheck -c mysqlcheck -c dev_packed -u root -p Enter password: dev_packed.new_oid OK dev_packed.object_ref OK dev_packed.object_refs_added OK dev_packed.object_stateOK dev_packed.pack_object OK What logs should I hunt through and what kind of things am I looking for? /var/log/messages and the like. Look for kernel-level errors such as block I/O errors, oops, and system panics. Any such errors have a chance of corrupting files. We need to rule out errors at the kernel or below. None of the above, /var/log/messages is actually pretty empty for the last few days. If it helps, both the app server and database server are VMWare virtual machines... Next, follow the directions in relstorage/tests/README.txt to create the 4 test databases. Then run bin/test -p -m relstorage. All tests should pass. Okay, first problem, the tests only connect to localhost, which means I can't exactly test as the app server is one machine and the database server is another. However, the two machines are identical, so I setup the buildout on the database server with the new test section added. First up, I get the following failures: IOError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/blob/blob_connection.txt' OSError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/replicas.conf' My guess is that these files aren't included by setuptools? So, I checked out the 1.4.0 tag and added it as a develop egg in the buildout. Now I get: Running .HFMySQLBlobTests tests: Set up .HFMySQLBlobTests in 0.000 seconds. Running: Ran 70 tests with 0 failures and 0 errors in 9.903 seconds. Running .HPMySQLBlobTests tests: Tear down .HFMySQLBlobTests in 0.000 seconds. Set up .HPMySQLBlobTests in 0.000 seconds. Running: Ran 81 tests with 0 failures and 0 errors in 10.511 seconds. Running zope.testing.testrunner.layer.UnitTests tests: Tear down .HPMySQLBlobTests in 0.005 seconds. Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Running: 78/269 (29.0%) Error in test check16MObject (relstorage.tests.testmysql.HPMySQLTests) Traceback (most recent call last): File /usr/local/lib/python2.6/unittest.py, line 279, in run testMethod() File /home/zope/relstorage_co/relstorage/tests/reltestbase.py, line 214, in check16MObject self._dostoreNP(oid, data=data) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 202, in _dostoreNP return self._dostore(oid, revid, data, 1, user, description) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 190, in _dostore r1 = self._storage.store(oid, revid, data, '', t) File /home/zope/relstorage_co/relstorage/storage.py, line 565, in store cursor, self._batcher, oid_int, prev_tid_int, data) File /home/zope/relstorage_co/relstorage/adapters/mover.py, line 453, in mysql_store_temp command='REPLACE', File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 67, in insert_into self.flush() File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 74, in flush self._do_inserts() File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 110, in _do_inserts self.cursor.execute(stmt, tuple(params)) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py, line 174, in execute self.errorhandler(self, exc, value) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1153, Got a packet bigger than 'max_allowed_packet' bytes) 196/269 (72.9%) Error in test check16MObject (relstorage.tests.testmysql.HFMySQLTests) Traceback (most recent call last): File /usr/local/lib/python2.6/unittest.py, line 279, in run testMethod() File /home/zope/relstorage_co/relstorage/tests/reltestbase.py, line 214, in check16MObject self._dostoreNP(oid, data=data) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 202, in _dostoreNP return self._dostore(oid, revid, data, 1, user,
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On Thu, 27 Jan 2011 10:32:09 +, Chris Withers ch...@simplistix.co.uk wrote: On 27/01/2011 03:15, Shane Hathaway wrote: Okay, so I'll do: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? The MySQL filesystem could be unmounted (after I shut down MySQL!), so I ran e2fsck on it: e2fsck -f /dev/sdb1 e2fsck 1.41.3 (12-Oct-2008) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 166/16777216 files (10.2% non-contiguous), 8244979/67107513 blocks - mysqlcheck -c mysqlcheck -c dev_packed -u root -p Enter password: dev_packed.new_oid OK dev_packed.object_ref OK dev_packed.object_refs_added OK dev_packed.object_stateOK dev_packed.pack_object OK What logs should I hunt through and what kind of things am I looking for? /var/log/messages and the like. Look for kernel-level errors such as block I/O errors, oops, and system panics. Any such errors have a chance of corrupting files. We need to rule out errors at the kernel or below. None of the above, /var/log/messages is actually pretty empty for the last few days. If it helps, both the app server and database server are VMWare virtual machines... Next, follow the directions in relstorage/tests/README.txt to create the 4 test databases. Then run bin/test -p -m relstorage. All tests should pass. Okay, first problem, the tests only connect to localhost, which means I can't exactly test as the app server is one machine and the database server is another. However, the two machines are identical, so I setup the buildout on the database server with the new test section added. First up, I get the following failures: IOError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/blob/blob_connection.txt' OSError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/replicas.conf' My guess is that these files aren't included by setuptools? So, I checked out the 1.4.0 tag and added it as a develop egg in the buildout. Now I get: Running .HFMySQLBlobTests tests: Set up .HFMySQLBlobTests in 0.000 seconds. Running: Ran 70 tests with 0 failures and 0 errors in 9.903 seconds. Running .HPMySQLBlobTests tests: Tear down .HFMySQLBlobTests in 0.000 seconds. Set up .HPMySQLBlobTests in 0.000 seconds. Running: Ran 81 tests with 0 failures and 0 errors in 10.511 seconds. Running zope.testing.testrunner.layer.UnitTests tests: Tear down .HPMySQLBlobTests in 0.005 seconds. Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Running: 78/269 (29.0%) Error in test check16MObject (relstorage.tests.testmysql.HPMySQLTests) Traceback (most recent call last): File /usr/local/lib/python2.6/unittest.py, line 279, in run testMethod() File /home/zope/relstorage_co/relstorage/tests/reltestbase.py, line 214, in check16MObject self._dostoreNP(oid, data=data) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 202, in _dostoreNP return self._dostore(oid, revid, data, 1, user, description) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 190, in _dostore r1 = self._storage.store(oid, revid, data, '', t) File /home/zope/relstorage_co/relstorage/storage.py, line 565, in store cursor, self._batcher, oid_int, prev_tid_int, data) File /home/zope/relstorage_co/relstorage/adapters/mover.py, line 453, in mysql_store_temp command='REPLACE', File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 67, in insert_into self.flush() File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 74, in flush self._do_inserts() File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 110, in _do_inserts self.cursor.execute(stmt, tuple(params)) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py, line 174, in execute self.errorhandler(self, exc, value) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1153, Got a packet bigger than 'max_allowed_packet' bytes) 196/269 (72.9%) Error in test check16MObject (relstorage.tests.testmysql.HFMySQLTests) Traceback (most recent call last): File /usr/local/lib/python2.6/unittest.py, line 279, in run testMethod() File /home/zope/relstorage_co/relstorage/tests/reltestbase.py, line 214, in check16MObject
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 10:40, Jürgen Herrmann wrote: i also had to up max_allowed_packet to 32M to make the tests work. Indeed, I upped to 32M and now I get no failures: Total: 420 tests, 0 failures, 0 errors in 59.173 seconds. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/27/2011 03:32 AM, Chris Withers wrote: On 27/01/2011 03:15, Shane Hathaway wrote: Okay, so I'll do: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? You don't. Don't even try. :-) The MySQL filesystem could be unmounted (after I shut down MySQL!), so I ran e2fsck on it: e2fsck -f /dev/sdb1 e2fsck 1.41.3 (12-Oct-2008) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 166/16777216 files (10.2% non-contiguous), 8244979/67107513 blocks - mysqlcheck -c mysqlcheck -c dev_packed -u root -p Enter password: dev_packed.new_oid OK dev_packed.object_ref OK dev_packed.object_refs_added OK dev_packed.object_stateOK dev_packed.pack_object OK Ok, thanks for checking. First up, I get the following failures: IOError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/blob/blob_connection.txt' OSError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/replicas.conf' My guess is that these files aren't included by setuptools? No, they are included AFAICT. Let's ignore this for now. This is a little weird, as I have max_allowed_packet set to 16M. Increase max_allowed_packet to at least 32M. 16M is just a bit too low. Should these tests fail? No. I think they will all pass once you increase max_allowed_packet. That said, I don't think this has anything to do with the packing bug as The point was to eliminate some of the most likely causes. Now we'll move on. I didn't see any exceptions or, in fact, any logging or output at all from zodbpack, and the only other exceptions seen were the POSKeyErrors... Hmm, you do bring up a good point: zodbpack doesn't configure the logging package. It should. Can you send me your database? Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 10:47, Shane Hathaway wrote: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? You don't. Don't even try. :-) Yeah, I got that from the warning message it squealed when I tried ;-) I was more curious about how you'd do this if you needed to (how do you unmount / so you can check it?) but I don't think it's the problem here... This is a little weird, as I have max_allowed_packet set to 16M. Increase max_allowed_packet to at least 32M. 16M is just a bit too low. Done, and tests all pass now. I didn't see any exceptions or, in fact, any logging or output at all from zodbpack, and the only other exceptions seen were the POSKeyErrors... Hmm, you do bring up a good point: zodbpack doesn't configure the logging package. It should. Yep ;-) ZConfig would be great so I can plug in a mailinglogger... It would also be *really* handy if zodbpack could run off a normal zope.conf for both logging and storage config (although, in my case, I'd then need to be able to specify which storage to pack, I want to avoid packing one of them!) Can you send me your database? In a word, no :-( It's 4GB+ in size (down from 26GB-ish in FileStorage!) and contains loads of confidential customer data. However, more than happy to poke, just tell me where and for what... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/27/2011 03:57 AM, Chris Withers wrote: It would also be *really* handy if zodbpack could run off a normal zope.conf for both logging and storage config (although, in my case, I'd then need to be able to specify which storage to pack, I want to avoid packing one of them!) Please note that the zodbpack utility is simple, short, and has no extra dependencies. I prefer to keep it that way. If you want to do something more interesting, please fork zodbpack. However, more than happy to poke, just tell me where and for what... Is the problem repeatable? If you start with the same database twice and pack, do you end up with POSKeyErrors on the same OIDs? I know that's probably a long test to run, but I'm not yet sure what else to suggest. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the discussion here with Anton Stonor. We are still only hypothesizing that there's a bug. Also, does RelStorage have a bug tracker anywhere? Not yet. The need for one has not been clear until very recently. RelStorage is turning into a community project and every community project needs a bug tracker. I suggest we use Launchpad. I have a main ZODB storage in a normal RelStorage with several BTreeFolder2's mounted into it that come from a history free storage. I'm using Zope 2.12.7 with ZODB 3.9.6 I attempted to pack the history-free storage with the following config: relstorage destination pack-gc true mysql ... /mysql keep-history false /relstorage The pack was going on while a batch process was pouring data into the ZODB by way for a loop with one transaction per iteration. The pack succeeded fine with no errors but about 10 minutes after the pack started, the batch process started throwing POSKeyErrors of the following sort: File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 897, in _setstate p, serial = self._storage.load(obj._p_oid, '') File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py, line 462, in load raise POSKeyError(oid) POSKeyError: 0xf30dbd -- 2011-01-26T10:26:00 WARNING POSKeyError on oid 15925081: no tid found; history-free adapter -- 2011-01-26T10:26:00 ERROR Couldn't load state for 0xf2ff59 Traceback (most recent call last): File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 847, in setstate self._setstate(obj) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 897, in _setstate p, serial = self._storage.load(obj._p_oid, '') File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py, line 462, in load raise POSKeyError(oid) POSKeyError: 0xf2ff59 It looks like these errors are occurring through the web client to this ZODB as well now. Any ideas what's going on here? What more useful information can I provide? That may be related, but first, are you mounting databases? You have to be careful with mounted databases and packing. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the discussion here with Anton Stonor. We are still only hypothesizing that there's a bug. Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? Also, does RelStorage have a bug tracker anywhere? Not yet. The need for one has not been clear until very recently. RelStorage is turning into a community project and every community project needs a bug tracker. I suggest we use Launchpad. Meh, I'm no fan of Launchpad, I vastly prefer Trac, but it's not my project and I'll use whatever your choose more than happily ;-) That may be related, but first, are you mounting databases? You have to be careful with mounted databases and packing. This exact setup has been used with FileStorages served over ZEO for getting on for a decade now... Does RelStorage do anything different in this area? (in the affected project, I'm fairly certain there are no cross-database references...) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On Wed, 26 Jan 2011 21:54:52 +, Chris Withers ch...@simplistix.co.uk wrote: On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the discussion here with Anton Stonor. We are still only hypothesizing that there's a bug. Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? Also, does RelStorage have a bug tracker anywhere? Not yet. The need for one has not been clear until very recently. RelStorage is turning into a community project and every community project needs a bug tracker. I suggest we use Launchpad. Meh, I'm no fan of Launchpad, I vastly prefer Trac, but it's not my project and I'll use whatever your choose more than happily ;-) That may be related, but first, are you mounting databases? You have to be careful with mounted databases and packing. This exact setup has been used with FileStorages served over ZEO for getting on for a decade now... Does RelStorage do anything different in this area? (in the affected project, I'm fairly certain there are no cross-database references...) cheers, Chris is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. jürgen -- XLhost.de ® - Webspace von supersmall bis eXtra Large XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 Web: http://www.XLhost.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 21:57, Jürgen Herrmann wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Googling for cross database references zodb should give you all the necessary background. I don't know of any scripts or examples to explicitly search for these, though... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 21:54, Chris Withers wrote: On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the discussion here with Anton Stonor. We are still only hypothesizing that there's a bug. Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? My guess is not. Shane, I went to try your suggestion: If you're sure you have found all of the corruption, you can do this (with all app servers shut down) to re-create the current_object table: delete from current_object; insert into current_object (zoid, tid) select zoid, max(tid) from object_state; ...but of course, history-free storages have no current_object table. Any ideas? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/26/2011 03:21 PM, Chris Withers wrote: Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? My guess is not. Shane, I went to try your suggestion: If you're sure you have found all of the corruption, you can do this (with all app servers shut down) to re-create the current_object table: delete from current_object; insert into current_object (zoid, tid) select zoid, max(tid) from object_state; ...but of course, history-free storages have no current_object table. Indeed, which is why I think your issue could be quite different from Anton's. The odd thing is I have seen no reports of pack issues for a long time. Why two at once? I still wonder if there might be no bug at all. If you and Anton could check all your logs and database integrity, that would save me a lot of time. FWIW, RelStorage has extensive tests of packing, including stress tests that write to the database while packing. I run those tests automatically using Buildbot on several platforms and with several combinations of ZODB and SQL database versions. None of the pack tests are failing on any platform, with any version of ZODB or any SQL database. If there is a bug, then the next step is to add a test that exercises the bug. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26 January 2011 21:57, Jürgen Herrmann juergen.herrm...@xlhost.de wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Packing with garbage collection disabled (pack-gc = false) should definitely be safe. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 22:49, Laurence Rowe wrote: On 26 January 2011 21:57, Jürgen Herrmannjuergen.herrm...@xlhost.de wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Packing with garbage collection disabled (pack-gc = false) should definitely be safe. Am I right in thinking this is pointless if you're using a history-free storage? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26 January 2011 23:11, Chris Withers ch...@simplistix.co.uk wrote: On 26/01/2011 22:49, Laurence Rowe wrote: On 26 January 2011 21:57, Jürgen Herrmannjuergen.herrm...@xlhost.de wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Packing with garbage collection disabled (pack-gc = false) should definitely be safe. Am I right in thinking this is pointless if you're using a history-free storage? Yes. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 01/26/2011 04:07 PM, Chris Withers wrote: On 26/01/2011 22:38, Shane Hathaway wrote: The odd thing is I have seen no reports of pack issues for a long time. Why two at once? I still wonder if there might be no bug at all. I'm afraid my wondering is on the other side of that argument ;-) If you and Anton could check all your logs and database integrity, that would save me a lot of time. Okay, so I'll do: - e2fsck -f - mysqlcheck -c Anything else? What logs should I hunt through and what kind of things am I looking for? /var/log/messages and the like. Look for kernel-level errors such as block I/O errors, oops, and system panics. Any such errors have a chance of corrupting files. We need to rule out errors at the kernel or below. FWIW, RelStorage has extensive tests of packing, including stress tests that write to the database while packing. How can I run these? If you don't already have bin/test in your buildout, add a section that creates it. Here is an example based on the buildout.cfg in RelStorage: [buildout] parts = test [test] recipe = zc.recipe.testrunner eggs = relstorage MySQL-python ZODB3 [test] zope.testrunner Next, follow the directions in relstorage/tests/README.txt to create the 4 test databases. Then run bin/test -p -m relstorage. All tests should pass. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
