Re: [ZODB-Dev] [Zope-dev] when did transaction lose the ability to be usable as a context manager?
On 06/02/2013 11:47, Wichert Akkerman wrote: You can also use the transaction manager directly instead of the bound methods transaction lifts out of it: import transaction with transaction.manager: pass yeah, this is what I'll do. Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] API question
On 16/01/2013 18:01, Tres Seaver wrote: from ZDOB.DB import DB # This one can even be ambiguous now FTR, I don't like this style. Somewhat a matter of taste. Sure. I don't like using APIs via long, multi-dotted paths. What Tres said ;-) - - Nose gives us easy access to Ned Batchelder's coverage package, which allows me to get seriously good test coverage in place before trying to port code to the straddle dialect. Yeah, easy, good coverage reporting and intergration with Jenkins' test reporting, coverage reporting and various other plugins to nose are the reasons I've moved all my stuff over.. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage breaks history tab
Hi Shane, On 22/05/2012 06:37, Chris Withers wrote: So, the problem is that, with a Zope 2 app server, any revisions made to an object (say, a Page Template) don't show in the history tab of that object *until* the app server is restarted. Okay, the issue appears to be that, in some circumstances, RelStorage is leaving the read connection with an open transaction that isn't rolled back. I couldn't find the source of this, but in the meantime, I less heavyweight hack is: diff --git a/relstorage/storage.py b/relstorage/storage.py index 2d592d3..459469b 100644 --- a/relstorage/storage.py +++ b/relstorage/storage.py @@ -1025,6 +1025,7 @@ class RelStorage( self._lock_acquire() try: self._before_load() +self._load_conn.rollback() cursor = self._load_cursor oid_int = u64(oid) try: However, that open transaction is likely to cause other problems (I'm fairly certain I've seen other effects of this) so do you have any idea where to look for it? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage breaks history tab - solution?
Hi Shane, On 23/05/2012 16:27, Chris Withers wrote: Okay, the issue appears to be that, in some circumstances, RelStorage is leaving the read connection with an open transaction that isn't rolled back. Upon investigation, this turns out to be the history method itself! :-) Switching to the following patch also solves the problem: diff --git a/relstorage/storage.py b/relstorage/storage.py index 2d592d3..3fa792b 100644 --- a/relstorage/storage.py +++ b/relstorage/storage.py @@ -1053,6 +1053,7 @@ class RelStorage( break return res finally: +self._load_conn.rollback() self._lock_release() def undo(self, transaction_id, transaction): However, that raised more questions for me: - How does RelStorage usually rollback the _load_conn in a non-write transaction? - Why doesn't this kick in when viewing the history method? My guess as to why ZODB.tests.HistoryStorage:HistoryStorage.checkSimpleHistory passes is that everything is done with the same thread/connection, and so a consistent view of the data is seen. More confusing: having dropped the zserver-threads down to one and from looking at the results of: select * from pg_catalog.pg_stat_activity where datname='myrelstorage'; ...it appears that there's: - one connection for reads - one connection for writes, which is opened when first used. Okay, as expected. - one connection for viewing history. I can't see where/why this final connection is opened... Shane, little help? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage breaks history tab
Hi All, I've been chatting with Shane about this off-list but wanted to see if anyone else has experience of this and has found a solution. So, the problem is that, with a Zope 2 app server, any revisions made to an object (say, a Page Template) don't show in the history tab of that object *until* the app server is restarted. I've verified this on both MySQL and Postgres with trunk of RelStorage as well as 1.5.2 and 1.4.2. It appears to be some kind of cursor re-use / relational database transaction isolation bug given that the following hack fixes the problem: --- a/relstorage/storage.py +++ b/relstorage/storage.py @@ -1025,7 +1025,7 @@ class RelStorage( self._lock_acquire() try: self._before_load() -cursor = self._load_cursor +conn, cursor = self._adapter.connmanager.open() oid_int = u64(oid) try: rows = self._adapter.dbiter.iter_object_history( @@ -1053,6 +1053,7 @@ class RelStorage( break return res finally: +self._adapter.connmanager.close(conn, cursor) self._lock_release() It seems a little heavyweight to have to open a whole new connection each time someone views a history tab, though :-/ A couple of surprising things: - ZODB.tests.HistoryStorage:HistoryStorage.checkSimpleHistory passes. I'd expect this to fail, so I wonder what it's doing differently? - The read connection is obviously getting the transaction's data, otherwise the wrong version of the object would be rendered, right? Any ideas very gratefully received... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] git clone of relstorage
Hi All, In case anyone else is interested, I've git svn clone'd svn://svn.zope.org/repos/main/relstorage and pushed the result up here: https://github.com/cjw296/relstorage It's only a master branch, sine git svn clone does svn branches and separate remotes by default and I didn't want to do my manual hack-fix steps since Shane likely wants to stay with svn: cp -Rf .git/refs/remotes/tags/* .git/refs/tags/ rm -rf .git/refs/remotes/tags cp -Rf .git/refs/remotes/* .git/refs/heads/ rm -rf .git/refs/remotes cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Unable to acquire commit lock from RelStorage
Hi Shane, What does this exception mean: Traceback (innermost last): Module ZPublisher.Publish, line 135, in publish Module Zope2.App.startup, line 291, in commit Module transaction._manager, line 89, in commit Module transaction._transaction, line 329, in commit Module transaction._transaction, line 426, in _commitResources Module ZODB.Connection, line 776, in tpc_vote Module relstorage.storage, line 783, in tpc_vote Module relstorage.storage, line 817, in _vote Module relstorage.storage, line 689, in _prepare_tid Module relstorage.adapters.locker, line 104, in hold_commit_lock StorageError: Unable to acquire commit lock What can cause it? What are the recommended solutions? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] where does 32it vs 64bit matter?
Hi Alan, On 21/12/2011 20:12, Alan Runyan wrote: Are there any migration steps or minimum ZODB/RelStorage versions needed to make this step? We run 32bit windows clients against 64bit linux zeo server; works fine. Good to know, but my concern is taking a system which is currently running 32-bit clients against RelStorage running on a MySQL on a 64-bit server and replacing the 32-bit clients with 64-bit clients. Is that going to cause any issues? Has anyone done any similar migrations? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] where does 32it vs 64bit matter?
Hi All, I have a set of ZODBs, they're currently served from a 64-bit mysql cluster running relstorage. At the moment, all the clients are 32-bit. At some point, those may be changing to be 64-bit. What are the implications of this? Do all relstorage clients need to be the same bit width? What about FileStorage and ZEO connections? Are there any migration steps or minimum ZODB/RelStorage versions needed to make this step? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage breaks History tab
On 02/09/2011 08:26, Martijn Pieters wrote: On Thu, Sep 1, 2011 at 22:56, Chris Withersch...@simplistix.co.uk wrote: I see the resulting transactions in both the root Undo tab and the Undo tab of the page template, but not in the History tab of the page template. Without looking, I'd say the history tab relies on the internal structure of the stock FileStorage implementation, not on public ZODB APIs. I haven't touched the archetypes History code in 5 years at least though, so I am not sure about this. Nope, RelStorage has specific history hooks in dbiter.py, just doesn't look like they work for me ;-) Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and MySQL 5.5
On 22/06/2011 22:25, Shane Hathaway wrote: I think RelStorage is ready to become more of a community project, and part of that is changing the meaning of supported. Until now, it has meant Shane is confident that this configuration will not lose your data, because he has tested it repeatedly in a few different ways. In the future, supported should mean several people have tested it, and not necessarily me. I think we need some kind of support matrix that gathers input from the community. I wonder if a wiki would be sufficient. Tox and Jenkins would solve the lot ;-) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] weird MySQL gone away error when viewing history tab on Script(Python) with RelStorage 1.4.2
On 27/04/2011 18:11, Shane Hathaway wrote: OperationalError: (2006, 'MySQL server has gone away') This is happening across at least two separate instances with separate storages. Any ideas? Maybe the query is killing MySQL, for whatever reason. I don't think so, since I'd then expect all the other storages served by that cluster to complain similarly. However, it's literally just the thread handling this particular request that shows the error... Are the MySQL process IDs the same before and after this happens? Yes. Nothing in the logs either... Ideas gratefully received! Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] weird MySQL gone away error when viewing history tab on Script(Python) with RelStorage 1.4.2
On 03/05/2011 09:02, Shane Hathaway wrote: Maybe the query is killing MySQL, for whatever reason. I don't think so, since I'd then expect all the other storages served by that cluster to complain similarly. Not necessarily. RelStorage reconnects automatically. ...but logs, and I haven't seen any log entries for this. Even for the thread which showed this error, which is weird... The reconnection is invisible unless the disconnect happens in the middle of a request, in which case RelStorage raises the error then automatically reconnects on the next request. ...or maybe not so weird. However, it's literally just the thread handling this particular request that shows the error... Two ideas: - MySQL is dropping the connection after a long period of inactivity, then RelStorage doesn't notice it's dropped until it's too late. Don't think so, this is 100% reproducible, even after the Zope instance has just been restarted... Connections that poll take defensive steps to prevent that, but maybe you're using a connection that does not poll. I'm using whatever the default that you've specified is ;-) - A single MySQL thread is dying. (Is MySQL multithreaded?) MySQL does indeed appear to be multithreaded, but I've seen no hard evidence of thread death, although it's hard to spot... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] weird MySQL gone away error when viewing history tab on Script(Python) with RelStorage 1.4.2
Hi Shane,
Attempting to view the /manage_change_history_page of a history-keeping
relstorage is giving me:
script statement failed: '\nSELECT 1 FROM current_object WHERE
zoid = %(oid)s\n'; parameters: {'oid': 1163686}
...and then:
Traceback (innermost last):
Module ZPublisher.Publish, line 127, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 47, in call_object
Module Shared.DC.Scripts.Bindings, line 324, in __call__
Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
Module App.special_dtml, line 185, in _exec
Module OFS.History, line 120, in manage_change_history
Module relstorage.storage, line 1011, in history
Module relstorage.adapters.dbiter, line 120, in iter_object_history
Module relstorage.adapters.scriptrunner, line 53, in run_script_stmt
Module MySQLdb.cursors, line 174, in execute
Module MySQLdb.connections, line 36, in defaulterrorhandler
OperationalError: (2006, 'MySQL server has gone away')
This is happening across at least two separate instances with separate
storages.
Any ideas?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] transaction as context manager, exception during commit
Hi Jim, The current __exit__ for transaction managers looks like this: def __exit__(self, t, v, tb): if v is None: self.commit() else: self.abort() ..which means that if you're using the transaction package as a context manager and, say, a relational database integrity constraint is violated, then you're left with a hosed transaction that still needs aborting. How would you feel about the above changing to: def __exit__(self, t, v, tb): if v is None: try: self.commit() except: self.abort() raise else: self.abort() If this is okay, I'll be happy to write the tests and make the changes provided someone does a release when I have... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction as context manager, exception during commit
On 24/02/2011 11:38, Laurence Rowe wrote: Hi Jim, The current __exit__ for transaction managers looks like this: def __exit__(self, t, v, tb): if v is None: self.commit() else: self.abort() ..which means that if you're using the transaction package as a context manager and, say, a relational database integrity constraint is violated, then you're left with a hosed transaction that still needs aborting. How would you feel about the above changing to: def __exit__(self, t, v, tb): if v is None: try: self.commit() except: self.abort() raise else: self.abort() If this is okay, I'll be happy to write the tests and make the changes provided someone does a release when I have... Looking at the way ZPublisher handles this, I think you're right. Cool, I'll wait for Jim's verdict (and some time to implement it ;-)) before diving in... I think you might also need to modify the __exit__ in Attempt, which additionally handles retrying transactions that fail. Yeah, I see a bug relating to this was reported yesterday: https://bugs.launchpad.net/bugs/724332 I know it's a different problem, but sounds like the code for attempt needs some love... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and MySQL wait-timeout
On 01/02/2011 18:11, Shane Hathaway wrote: On 02/01/2011 07:51 PM, Chris Withers wrote: I can understand the problem being fairly terminal if there was a disconnect *during* a timeout, and I'd expect an exception, but not a segfault ;-) I haven't seen segfaults except when the dynamic linker used an incorrect library. Use ldd to check the .so files generated during buildout. Surely these would occur all the time? I've only seen them occur on two occasions, and the same buildouts have worked fine the rest of the time... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
On 01/02/2011 23:10, Ruda Porto Filgueiras wrote: I had similar issues on past and now I follow some rules: - pack only with zodbpack - pack-gc = false ...which means your database will still slowly (or quickly in my case!) grow over time with unreferenced objects. - pack when the database is not been active update or not update at all. I'm glad you have an app with quiet spells! ;-) This isn't realistic for any active, public web app. But it does not imply RelStorage bug, since MySQL is know to not be so safe. Er, since when? If that were the case, I'm sure Shane would place explicit instructions that it should not be used... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
On 02/02/2011 09:14, Shane Hathaway wrote: On 02/02/2011 10:57 AM, Chris Withers wrote: Er, since when? If that were the case, I'm sure Shane would place explicit instructions that it should not be used... Safe is relative. MySQL is a good choice for Facebook, but if I knew my bank was storing my account balance in MySQL, I would close my accounts immediately. I'll have to respectfully disagree... MySQL has come a long way over the last 10 years, and I'm as confident in it as I am in any relational database, particularly when used with the InnoDB engine. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage and MySQL wait-timeout
On 01/02/2011 04:11, Shane Hathaway wrote: My guess is that the zap_all took so long that the server had gone away by the time the sql statement had be executed. My guess is MySQL is configured to drop connections when they are idle. Indeed, Rackspace had configured a wait-timeout of 60 second! (why on earth they would do that is beyond me, answers on a post card...) That is a bad idea IMHO, so I think raising that exception is the right thing to do, not a bug. Okay, but, with it at 60s, I was getting the following behaviour when rendering pages: Couldn't load state for 0x11ae03 Traceback (most recent call last): File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 847, in setstate self._setstate(obj) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 897, in _setstate p, serial = self._storage.load(obj._p_oid, '') File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py, line 448, in load state, tid_int = cache.load(cursor, oid_int) File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/cache.py, line 279, in load state, tid_int = self.adapter.mover.load_current(cursor, oid_int) File /var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/adapters/mover.py, line 125, in mysql_load_current cursor.execute(stmt, (oid,)) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py, line 174, in execute self.errorhandler(self, exc, value) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (2006, 'MySQL server has gone away') ...which feels a little on the serious side for (what is for MySQL) quite a normal situation to be in. (Mike Bayer could probably shed some light given the connection pool stuff that SQLAlchemy does to deal with MySQL's behaviour...) I also had a segfault trying to do the same conversion which I'm attributing to the MySQL server being restarted by an overeager DBA mid-converstion but still, that shouldn't cause a segfault, right? I don't know why it would. I'll let you know if I see it again... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage - what tables and indexes should be present?
Hi Shane, I notice that one of my history-free storages only has 'new_oid' and 'object_state' tables while all the others have 'new_oid', 'object_ref', 'object_refs_added', 'object_state' and 'pack_object' tables. What's special about this storage? I also note that while the ones with lots of tables have this schema for object_state: CREATE TABLE `object_state` ( `zoid` bigint(20) NOT NULL, `tid` bigint(20) NOT NULL, `state` longblob, PRIMARY KEY (`zoid`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1$$ ...the one with only two tables has this schema: CREATE TABLE `object_state` ( `zoid` bigint(20) NOT NULL, `tid` bigint(20) NOT NULL, `state` longblob, PRIMARY KEY (`zoid`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1$$ ...which causes the following statement, which gets executed a lot, to be very slow: SELECT tid FROM object_state ORDER BY tid DESC LIMIT 1 cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] (anecdotal) Restarting MySQL under RelStorage gives segfault
Hi All, I've had a couple of instances now where RelStorage-backed ZODB client processes have crashed/segfaulted. As best I can tell, both times have been the result of the MySQL server being restarted while the client is connected. However, other server restarts haven't caused this problem, so I'm wondering if it's a restart at a particular point in the transaction cycle that causes the segfault? Anyway, just putting this out there in case anyone else has had similar experiences... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Error from MySQL when attempting to run zodbconvert
On 01/02/2011 04:03, Shane Hathaway wrote: On 01/31/2011 06:30 PM, Chris Withers wrote: Hi Shane, I got the following error when trying to run zodbconvert against a clustered MySQL environment running RHCS: File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.OperationalError: (1598, Binary logging not possible. Message: Transaction level 'READ-COMMITTED' in InnoDB is not safe for binlog mode 'STATEMENT') That's why you have to use row-based replication instead of statement-based replication. Is this in the RelStorage docs anywhere? If so, I missed it, but if not, I guess it probably should be... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage - what tables and indexes should be present?
On 01/02/2011 14:42, Stephan Richter wrote: On Tuesday, February 01, 2011, Chris Withers wrote: I also note that while the ones with lots of tables have this schema for object_state: CREATE TABLE `object_state` ( `zoid` bigint(20) NOT NULL, `tid` bigint(20) NOT NULL, `state` longblob, PRIMARY KEY (`zoid`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1$$ ...the one with only two tables has this schema: CREATE TABLE `object_state` ( `zoid` bigint(20) NOT NULL, `tid` bigint(20) NOT NULL, `state` longblob, PRIMARY KEY (`zoid`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1$$ Did you paste correctly? They seem identical to me. Nope, I phail... but the above is the incorrect version. Try creating a RelStorage and you'll see. The above ends up being really slow as it has no index on tid... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage - what tables and indexes should be present?
On 01/02/2011 17:33, Shane Hathaway wrote: What's special about this storage? It sounds like RelStorage didn't get a chance to finish creating the schema. In MySQL, DDL statements are not transactional, so errors during schema creation (such as a timeout) leave a partial schema. Sounds like something to warn about in the docs for zodbconvert and storage creation... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and MySQL wait-timeout
Hi Shane, On 01/02/2011 17:45, Shane Hathaway wrote: On 02/01/2011 10:01 AM, Chris Withers wrote: OperationalError: (2006, 'MySQL server has gone away') ...which feels a little on the serious side for (what is for MySQL) quite a normal situation to be in. Random disconnects are unacceptable for RelStorage. If MySQL goes away outside transaction boundaries, we *have* to propagate the exception to the application; Are you sure you mean outside? Surely outside is fine, you just reconnect as part of starting the transaction? otherwise consistency is lost. We can only reconnect at the next request or transaction boundary. Indeed, and am I right in thinking RelStorage does this just fine, along with emitting a warning message to say that it has occurred? I can understand the problem being fairly terminal if there was a disconnect *during* a timeout, and I'd expect an exception, but not a segfault ;-) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 28/01/2011 21:58, Jürgen Herrmann wrote: Afaics you use zodbpack's default of days=0. This is known to produce POSKeyErrors if the database is written to while packing. Where is this documented? Does this apply to history-keeping or history-free storages? Try with something like days=0.1 . I'll not pack a history-free storage until Shane gives it the okay... However, I find it a bit odd that RelStorage ships with a default that could cause these kind of problems and no warning in the readme that this is the case... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Error from MySQL when attempting to run zodbconvert
On 31/01/2011 16:30, Chris Withers wrote: I got the following error when trying to run zodbconvert against a clustered MySQL environment running RHCS: File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.OperationalError: (1598, Binary logging not possible. Message: Transaction level 'READ-COMMITTED' in InnoDB is not safe for binlog mode 'STATEMENT') I fixed this by setting binlog-format to MIXED rather than STATEMENT in my.cnf. There don't appear to be any downsides to this, so maybe this should go in the docs somewhere? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 11:24, Shane Hathaway wrote: On 01/27/2011 03:57 AM, Chris Withers wrote: It would also be *really* handy if zodbpack could run off a normal zope.conf for both logging and storage config (although, in my case, I'd then need to be able to specify which storage to pack, I want to avoid packing one of them!) Please note that the zodbpack utility is simple, short, and has no extra dependencies. I prefer to keep it that way. If you want to do something more interesting, please fork zodbpack. Fair enough, simple is good for me :-) If errors or logging occur inside zodbpack, where do they go? However, more than happy to poke, just tell me where and for what... Is the problem repeatable? Yes. If you start with the same database twice and pack, do you end up with POSKeyErrors on the same OIDs? Not sure. I know that's probably a long test to run, but I'm not yet sure what else to suggest. Okay, some more info... I re-converted the .fs file for this storage. This time, I've taken a mysqldump backup. I then packed, and *then* started the batch process: no problems. While the batch was running, I tried packing again but the pack finished without any problems. Okay, so next up I restored the backup. This time I set the batch running and *then* started the pack. This eventually caused the POSKeyErrors to begin again. I'm going to have a go at knocking up a small batch script that you can run to reproduce the issue, but there's definitely an issue here when packing while changing lots of data in a ZODB. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, MySQL - errors
Hi Shane,
Okay, so here's the test harness:
svn co https://secure.simplistix.co.uk/svn/relstorage_pack -r 4594
It uses the normal RelStorage test users and databases.
After running the buildout, I do each of the following in their own
terminal:
bin/generate --zap
bin/zodbpack pack.conf
On 28/01/2011 10:26, Shane Hathaway wrote:
I'm going to have a go at knocking up a small batch script that you can
run to reproduce the issue, but there's definitely an issue here when
packing while changing lots of data in a ZODB.
I look forward to your results!
I initially had the file size set to 200Kb and everything seemed to be
behaving as it should.
I dropped the file size to 1kb and tried again, getting the following
after a short time:
Traceback (most recent call last):
File bin/generate, line 100, in module
generate.generate('/home/zope/relstorage_pack/zodb.conf')
File /home/zope/relstorage_pack/generate.py, line 62, in generate
app.manage_delObjects([id])
File
/var/buildout-eggs/Zope2-2.12.7-py2.6-linux-i686.egg/OFS/ObjectManager.py,
line 536, in manage_delObjects
if v.wl_isLocked():
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line
847, in setstate
self._setstate(obj)
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line
897, in _setstate
p, serial = self._storage.load(obj._p_oid, '')
File
/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py,
line 448, in load
state, tid_int = cache.load(cursor, oid_int)
File
/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/cache.py,
line 234, in load
self._check_tid_after_load(oid_int, actual_tid_int, tid_int)
File
/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/cache.py,
line 187, in _check_tid_after_load
'thread_ident': thread.get_ident(),
AssertionError: Detected an inconsistency between the RelStorage cache
and the database while loading an object using the delta_after0 dict.
Please verify the database is configured for ACID compliance and that
all clients are using the same commit lock. (oid_int=2509,
expect_tid_int=20891637739451L, actual_tid_int=None,
current_tid=20892026220390L, cp0=20891176006912L,
cp1=20891176006912L, len(delta_after0)=2376, len(delta_after1)=0,
pid=6087, thread_ident=-1219049280)
Not the POSKeyError I'm having trouble with, but looks like you might
want to know ;-)
Could this have anything to do with the POSKeyErrors?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
Bingo:
svn co https://secure.simplistix.co.uk/svn/relstorage_pack -r 4595
cd relstorage_pack
python2.6 bootstrap.py
bin/buildout
Then in one terminal:
bin/generate --zap --folder_depth 3
Leave it running for a minute or two, and then in another do:
bin/zodbpack pack.conf
After a bit I get:
Traceback (most recent call last):
File bin/generate, line 100, in module
generate.generate('/home/zope/relstorage_pack/zodb.conf')
File /home/zope/relstorage_pack/generate.py, line 64, in generate
container.manage_delObjects([path[-1]])
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line
847, in setstate
self._setstate(obj)
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line
897, in _setstate
p, serial = self._storage.load(obj._p_oid, '')
File
/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py,
line 462, in load
raise POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x1675
cheers,
Chris
--
Simplistix - Content Management, Batch Processing Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage, history-free, pack causes POSKeyError with BTreeFolder2
On 28 Jan 2011, at 21:58, Jürgen Herrmann juergen.herrm...@xlhost.de wrote:
On Fri, 28 Jan 2011 12:34:35 +, Chris Withers wrote:
Bingo:
svn co https://secure.simplistix.co.uk/svn/relstorage_pack -r 4595
cd relstorage_pack
python2.6 bootstrap.py
bin/buildout
Then in one terminal:
bin/generate --zap --folder_depth 3
Leave it running for a minute or two, and then in another do:
bin/zodbpack pack.conf
After a bit I get:
Traceback (most recent call last):
File bin/generate, line 100, in module
generate.generate('/home/zope/relstorage_pack/zodb.conf')
File /home/zope/relstorage_pack/generate.py, line 64, in
generate
container.manage_delObjects([path[-1]])
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py,
line
847, in setstate
self._setstate(obj)
File
/var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py,
line
897, in _setstate
p, serial = self._storage.load(obj._p_oid, '')
File
/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/storage.py,
line 462, in load
raise POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x1675
cheers,
Chris
Afaics you use zodbpack's default of days=0. This is known to produce
POSKeyErrors if the database is written to while packing. Try with
something
like days=0.1 .
Regards,
Jürgen
--
XLhost.de ® - Webhosting von supersmall bis eXtra Large
XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany
Geschäftsführer: Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218
Fon: +49 (0)800 XLHOSTDE [0800 95467833]
Fax: +49 (0)800 95467830
Web: http://www.XLhost.de
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
__
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
__
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 03:15, Shane Hathaway wrote: Okay, so I'll do: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? The MySQL filesystem could be unmounted (after I shut down MySQL!), so I ran e2fsck on it: e2fsck -f /dev/sdb1 e2fsck 1.41.3 (12-Oct-2008) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 166/16777216 files (10.2% non-contiguous), 8244979/67107513 blocks - mysqlcheck -c mysqlcheck -c dev_packed -u root -p Enter password: dev_packed.new_oid OK dev_packed.object_ref OK dev_packed.object_refs_added OK dev_packed.object_stateOK dev_packed.pack_object OK What logs should I hunt through and what kind of things am I looking for? /var/log/messages and the like. Look for kernel-level errors such as block I/O errors, oops, and system panics. Any such errors have a chance of corrupting files. We need to rule out errors at the kernel or below. None of the above, /var/log/messages is actually pretty empty for the last few days. If it helps, both the app server and database server are VMWare virtual machines... Next, follow the directions in relstorage/tests/README.txt to create the 4 test databases. Then run bin/test -p -m relstorage. All tests should pass. Okay, first problem, the tests only connect to localhost, which means I can't exactly test as the app server is one machine and the database server is another. However, the two machines are identical, so I setup the buildout on the database server with the new test section added. First up, I get the following failures: IOError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/blob/blob_connection.txt' OSError: [Errno 2] No such file or directory: '/var/buildout-eggs/RelStorage-1.4.0-py2.6.egg/relstorage/tests/replicas.conf' My guess is that these files aren't included by setuptools? So, I checked out the 1.4.0 tag and added it as a develop egg in the buildout. Now I get: Running .HFMySQLBlobTests tests: Set up .HFMySQLBlobTests in 0.000 seconds. Running: Ran 70 tests with 0 failures and 0 errors in 9.903 seconds. Running .HPMySQLBlobTests tests: Tear down .HFMySQLBlobTests in 0.000 seconds. Set up .HPMySQLBlobTests in 0.000 seconds. Running: Ran 81 tests with 0 failures and 0 errors in 10.511 seconds. Running zope.testing.testrunner.layer.UnitTests tests: Tear down .HPMySQLBlobTests in 0.005 seconds. Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Running: 78/269 (29.0%) Error in test check16MObject (relstorage.tests.testmysql.HPMySQLTests) Traceback (most recent call last): File /usr/local/lib/python2.6/unittest.py, line 279, in run testMethod() File /home/zope/relstorage_co/relstorage/tests/reltestbase.py, line 214, in check16MObject self._dostoreNP(oid, data=data) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 202, in _dostoreNP return self._dostore(oid, revid, data, 1, user, description) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 190, in _dostore r1 = self._storage.store(oid, revid, data, '', t) File /home/zope/relstorage_co/relstorage/storage.py, line 565, in store cursor, self._batcher, oid_int, prev_tid_int, data) File /home/zope/relstorage_co/relstorage/adapters/mover.py, line 453, in mysql_store_temp command='REPLACE', File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 67, in insert_into self.flush() File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 74, in flush self._do_inserts() File /home/zope/relstorage_co/relstorage/adapters/batch.py, line 110, in _do_inserts self.cursor.execute(stmt, tuple(params)) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py, line 174, in execute self.errorhandler(self, exc, value) File /var/buildout-eggs/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/connections.py, line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1153, Got a packet bigger than 'max_allowed_packet' bytes) 196/269 (72.9%) Error in test check16MObject (relstorage.tests.testmysql.HFMySQLTests) Traceback (most recent call last): File /usr/local/lib/python2.6/unittest.py, line 279, in run testMethod() File /home/zope/relstorage_co/relstorage/tests/reltestbase.py, line 214, in check16MObject self._dostoreNP(oid, data=data) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/tests/StorageTestBase.py, line 202, in _dostoreNP return self._dostore(oid, revid, data, 1, user,
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 10:40, Jürgen Herrmann wrote: i also had to up max_allowed_packet to 32M to make the tests work. Indeed, I upped to 32M and now I get no failures: Total: 420 tests, 0 failures, 0 errors in 59.173 seconds. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 27/01/2011 10:47, Shane Hathaway wrote: - e2fsck -f Hmm, how do I e2fsck a mounted filesystem? You don't. Don't even try. :-) Yeah, I got that from the warning message it squealed when I tried ;-) I was more curious about how you'd do this if you needed to (how do you unmount / so you can check it?) but I don't think it's the problem here... This is a little weird, as I have max_allowed_packet set to 16M. Increase max_allowed_packet to at least 32M. 16M is just a bit too low. Done, and tests all pass now. I didn't see any exceptions or, in fact, any logging or output at all from zodbpack, and the only other exceptions seen were the POSKeyErrors... Hmm, you do bring up a good point: zodbpack doesn't configure the logging package. It should. Yep ;-) ZConfig would be great so I can plug in a mailinglogger... It would also be *really* handy if zodbpack could run off a normal zope.conf for both logging and storage config (although, in my case, I'd then need to be able to specify which storage to pack, I want to avoid packing one of them!) Can you send me your database? In a word, no :-( It's 4GB+ in size (down from 26GB-ish in FileStorage!) and contains loads of confidential customer data. However, more than happy to poke, just tell me where and for what... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the discussion here with Anton Stonor. We are still only hypothesizing that there's a bug. Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? Also, does RelStorage have a bug tracker anywhere? Not yet. The need for one has not been clear until very recently. RelStorage is turning into a community project and every community project needs a bug tracker. I suggest we use Launchpad. Meh, I'm no fan of Launchpad, I vastly prefer Trac, but it's not my project and I'll use whatever your choose more than happily ;-) That may be related, but first, are you mounting databases? You have to be careful with mounted databases and packing. This exact setup has been used with FileStorages served over ZEO for getting on for a decade now... Does RelStorage do anything different in this area? (in the affected project, I'm fairly certain there are no cross-database references...) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 21:57, Jürgen Herrmann wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Googling for cross database references zodb should give you all the necessary background. I don't know of any scripts or examples to explicitly search for these, though... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 21:54, Chris Withers wrote: On 26/01/2011 21:05, Shane Hathaway wrote: On 01/26/2011 11:52 AM, Chris Withers wrote: On 26/01/2011 14:08, Shane Hathaway wrote: I've checked in a fix in Subversion. Please try it out. I need to look at the possible pack issue recently reported before we make a release. Where is this pack issue documented/discussed? See the discussion here with Anton Stonor. We are still only hypothesizing that there's a bug. Well, my case matches that case pretty exactly... MySQL 5.1, RelStorage 1.4.0... Anton, were you using a history-free storage? My guess is not. Shane, I went to try your suggestion: If you're sure you have found all of the corruption, you can do this (with all app servers shut down) to re-create the current_object table: delete from current_object; insert into current_object (zoid, tid) select zoid, max(tid) from object_state; ...but of course, history-free storages have no current_object table. Any ideas? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage pack with history-free storage results in POSKeyErrors
On 26/01/2011 22:49, Laurence Rowe wrote: On 26 January 2011 21:57, Jürgen Herrmannjuergen.herrm...@xlhost.de wrote: is there a script or some example code to search for cross db references? i'm also eager to find out... for now i disabled my packing cronjobs. Packing with garbage collection disabled (pack-gc = false) should definitely be safe. Am I right in thinking this is pointless if you're using a history-free storage? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage recommended maintenance
Hi Shane, I'm wondering what the recommended maintenance for these two types of storage are that I use: - keep-history=true, never want to lose any revisions My guess is zodbpack with pack-gc as true, but what do I specify for the number of days in order to keep all history? - keep-history=false, never want any revisions or unreferenced objects My guess is zodbpack with pack-gc as true and days as 0? All storages will be in MySQL, are there any duty-cycle or other parameters I should tweak from their default values? How often do you recommend running zodbpack? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage recommended maintenance
On 21/01/2011 22:09, Laurence Rowe wrote: On 21 January 2011 20:57, Shane Hathawaysh...@hathawaymix.org wrote: On 01/21/2011 10:46 AM, Chris Withers wrote: I'm wondering what the recommended maintenance for these two types of storage are that I use: - keep-history=true, never want to lose any revisions My guess is zodbpack with pack-gc as true, but what do I specify for the number of days in order to keep all history? Is 100 years enough? 365.24 * 100 = 36524 ;-) Why would you pack a database from which you don't want to lose any revisions. It's a fair point. I can't think of anything, but given that Shane didn't make your point, I wonder if he can? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Write on master, read from slave(s)?
On 26/10/2010 11:27, Shane Hathaway wrote: On 10/26/2010 04:04 AM, Anton Stonor wrote: In order to scale an application using RelStorage I was thinking about seperating reads and writes accross databases. Writes would go to a Mysql master and the app would read from one or more Mysql slaves. If you mean that you intend to set up some clients to write to a master database, while other clients only read from asynchronous replicas, then yes, you'll be in good shape. You should be able to get amazing scalability that way. Wouldn't this require zero replication lag? Even if the read replicas had only a second or two delay (or spike delays caused by large writes?), would there not potentially be quite serious problems? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] 32-bit vs 64-bit
Hi All, Are there any intricacies with migrating a ZODB-backed Zope 2 app from 32-bit to 64-bit machines? The specific scenario I have in mind is moving a 32-bit set of ZODBs into MySQL-backed RelStorages where the MySQL server runs 64-bit but the clients are still 32-bit, although in one instance the clients will also be 64-bit. Any pointers/pitfalls/etc gratefully recieved! cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 32-bit vs 64-bit - RelStorage on MySQL
On 18/11/2010 15:39, Marius Gedminas wrote: About the only noticeable difference -- other than the obvious memory growth What obvious memory growth? I guess I'm more concerned with a mix of 32 and 64 bit in a RelStorage setup... Does that mean my question is essentially do 32-bit MySQL clients play nicely with a 64-bit MySQL server or is it more complicated than that? Do pickles go nicely across 32-bit/64-bit boundaries? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 32-bit vs 64-bit - RelStorage on MySQL
On 18/11/2010 16:48, Hanno Schlichting wrote: On Thu, Nov 18, 2010 at 5:19 PM, Leonardo Santagadasantag...@gmail.com wrote: On Thu, Nov 18, 2010 at 1:47 PM, Chris Withersch...@simplistix.co.uk wrote: On 18/11/2010 15:39, Marius Gedminas wrote: About the only noticeable difference -- other than the obvious memory growth What obvious memory growth? The one from pointers and anything related to memory going from 32bits to 64bits in size. Py_objects get fatter because of that. For Zope based applications I've generally seen 50% to 100% memory growth when moving to 64bit. That's why we stick to 32bit for a number of memory hungry applications (*wink* Plone). Right, but, even if the MySQL server is 64-bit, if the clients are 32-bit, they won't have this problem, will they? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] transaction timeouts raise no exceptions and cause strange errors on clients
Hi All,
I have a ZODB 3.9.6 ZEO storage server configured with a transaction
time of 20 seconds.
On one zeo client I saw the following:
--
2010-11-04T10:40:50 INFO ZEO.ClientStorage unpackedstorage Disconnected
from storage: ('xx', 6000)
--
2010-11-04T10:40:50 INFO ZEO.ClientStorage unpackedstorage Testing
connection ManagedClientConnection ('xx', 6000)
--
2010-11-04T10:40:50 INFO ZEO.zrpc.Connection(C) (xx:6000) received
handshake 'Z309'
--
2010-11-04T10:40:50 ERROR ZODB.Connection Couldn't load state for 0x15ef67
Traceback (most recent call last):
File ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 847,
in setstate
self._setstate(obj)
File ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line 897,
in _setstate
p, serial = self._storage.load(obj._p_oid, '')
File ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/Connection.py, line
1282, in load
return self._storage.load(oid, '')
File ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/ClientStorage.py, line
813, in load
data, tid = self._server.loadEx(oid)
File ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/ClientStorage.py, line
88, in __getattr__
raise ClientDisconnected()
ClientDisconnected
--
2010-11-04T10:40:51 INFO ZEO.ClientStorage unpackedstorage Server
authentication protocol None
--
2010-11-04T10:40:51 INFO ZEO.ClientStorage unpackedstorage Connected to
storage: ('xx', 6000)
--
2010-11-04T10:40:51 INFO ZEO.ClientStorage unpackedstorage No
verification necessary (last_inval_tid up-to-date)
--
2010-11-04T10:40:51 ERROR Zope.SiteErrorLog 1288867251.890.173783627248
https://xx
Traceback (innermost last):
...
Module ZODB.Connection, line 847, in setstate
Module ZODB.Connection, line 897, in _setstate
Module ZODB.Connection, line 1282, in load
Module ZEO.ClientStorage, line 813, in load
Module ZEO.ClientStorage, line 88, in __getattr__
ClientDisconnected
--
2010-11-04T10:40:54 CRITICAL txn.-1271567456 A storage error occurred
during the second phase of the two-phase commit. Resources may be in an
inconsistent state.
--
2010-11-04T10:40:55 ERROR Zope.SiteErrorLog 1288867255.090.422928174987
https://xx
Traceback (innermost last):
Module ZPublisher.Publish, line 135, in publish
Module Zope2.App.startup, line 291, in commit
Module transaction._manager, line 89, in commit
Module transaction._transaction, line 329, in commit
Module transaction._transaction, line 431, in _commitResources
Module ZODB.Connection, line 794, in tpc_finish
Module ZEO.ClientStorage, line 1148, in tpc_finish
ClientDisconnected: Calling tpc_finish() on a disconnected transaction
This was all pretty alarming, especially as I could see no errors logged
anywhere else.
Eevntually, in the storage server logs, I found:
2010-11-04T10:40:50 INFO ZEO.StorageServer (7008/xx) Transaction timeout
after 20 seconds
--
2010-11-04T10:40:50 INFO ZEO.StorageServer (7008/xx) disconnected during
transaction transaction._transaction.Transaction object at 0x9b4106c
--
2010-11-04T10:40:50 INFO ZEO.StorageServer (7008) new connection ('xx',
34996): ManagedServerConnection ('xx', 34996)
--
2010-11-04T10:40:50 INFO ZEO.zrpc.Connection(S) (xx) received handshake
'Z309'
--
2010-11-04T10:40:55 WARNING ZEO.StorageServer (7008/xx) no current
transaction: tpc_abort()
--
I'm curious:
- Why is the timeout not logged at ERROR or even CRITICAL?
- Is the inconsistent state warning genuine here? I restarted the
client to be on the safe side, but it'd be good to know for sure.
- If that warning is genuine, is there any way the timeout could not
leave the client in an inconsistent state?
cheers,
Chris
--
Simplistix - Content Management, Batch Processing Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage: getting keep-history wrong results in silent death
Hi Shane, I was having trouble starting a relstorage 1.4.0-backed Zope instance this morning. The symptoms were that it would get going fine, until the point it tried to open the storages, at which point it would simply restart with no logging as to why. Turns out I had keep-history set to true whereas the database have been created with it set to false. Should this not result in an exception being logged rather than just a silent death that causes zdaemon to lamely attempt to restart the instance? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage versus temp_folder
Hi Shane, I have the following config for the temp_folders of my ZODB instances: zodb_db temporary relstorage destination mysql host dr2.private user zope .. /mysql keep-history false /relstorage mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer /zodb_db However, even after exercising the site that uses this temp_folder and verifying that there are objects containing within it using the ZMI, I still see the database as totally empty when using MySQL's gui administrator. Any idea what's going on here? Where is the data actually being stored? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction timeouts raise no exceptions and cause strange errors on clients
On 04/11/2010 13:43, Jim Fulton wrote: I'm curious: - Why is the timeout not logged at ERROR or even CRITICAL? Because it was considered to be neither. It's also not a server error. It is an erroneous condition; a transaction has taken long enough that a timeout has occurred. Thinking about it, it feels more like a TimeoutException should be raised, but then would the storage server clean things up properly? - Is the inconsistent state warning genuine here? Yes. You don't really know whether the transaction committed or not. It's much worse if multiple storages are involved. ...which of course, they are ;-) - If that warning is genuine, is there any way the timeout could not leave the client in an inconsistent state? Absolutely. My wording was poor: If that warning is genuine, is there any way the timeout could be made to not leave the client in an inconsistent state? This is why I tend to view transaction timeouts as a last resort and set the limit much higher, 5 minutes. Fair enough. Seems odd to have a feature which you seem to be suggestion should not be used... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: getting keep-history wrong results in silent death
On 04/11/2010 14:13, Shane Hathaway wrote: Turns out I had keep-history set to true whereas the database have been created with it set to false. Should this not result in an exception being logged rather than just a silent death that causes zdaemon to lamely attempt to restart the instance? It outputs an error using the logging module and raises an exception. Have you verified this in a normal Zope 2 setup? In my case, I have 2 normal storages, both of which now have keep-history as false and a third for the temp folder. I'm using MySQL as the back end and saw no logging either from RelStorage or anything that might have caught an exception from RelStorage... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction timeouts raise no exceptions and cause strange errors on clients
On 04/11/2010 14:50, Jim Fulton wrote: Actually, I think it should be used, at least for applications with high availability requirements. For applications like our's (ZC's) it amounts to a don't be stupid setting. For some applications, it's better to risk inconsistency than to have your application grind to a halt. I consider it an emergency device though, which is why I'd set it pretty high. ...except that it logs at INFO ;-) Can you raise that, or would you mind if I raised that to CRITICAL? (it does feel like a CRITICAL kind of thing, based on the information you've provided today) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction timeouts raise no exceptions and cause strange errors on clients
On 04/11/2010 15:09, Jim Fulton wrote: Can you raise that, or would you mind if I raised that to CRITICAL? (it does feel like a CRITICAL kind of thing, based on the information you've provided today) +1 Feel free to do it yourself, or file a bug. https://bugs.launchpad.net/zodb/+bug/670986 I don't have a ZODB development handy, but will tackle it next time I do. Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Default comparison considered harmful in BTrees.
On 25/10/2010 23:34, Marius Gedminas wrote: Or perhaps make it emit DeprecationWarnings, but continue working. Then make it a fatal error in the next minor/major release. Well, not a DeprecationWarning... Is there a DataLossWarning? Still, +1 on the warning followed by exception in 2 releases... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] MySQL server has gone away with RelStorage
On 25/10/2010 15:57, Paul Winkler wrote: On Sun, Oct 24, 2010 at 05:55:05PM +0100, Chris Withers wrote: Restarting Zope appeared to fixed the problem, but I'm slightly confused as I can't find any entries in the logs of the Apache instance that sits in front of the test instances for the failed requests... Maybe turn on Zope's trace log? Okay, but regardless of what happens with Zope, surely I'd expect a row in the Apache log saying what happened? I'm just wondering whether I should be suspicious of MySQL + RelStorage + Zope or if I should be looking for some weird Apache issue... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] MySQL server has gone away with RelStorage
On 23/10/2010 18:48, Shane Hathaway wrote: On 10/23/2010 01:29 AM, Chris Withers wrote: Hi Shane, With the server I'm testing on, when coming back to test after a couple of days working on something else, I get the following logged at Warning: Reconnecting load_conn: (2006, 'MySQL server has gone away') Does this mean the reconnection is being handled by RelStorage or that I need to do some work? That just means MySQL is timing out connections, which is fine. That message was probably followed immediately by an info message saying that it reconnected. Yep, that matches. Also, after this, it seems life the ZODB object cache is emptied. Is that happening or is it just my imagination? That's possible, but reconnection alone should not cause it. What could cause a cache to empty like this? More worryingly, another of the test sites (these sites only get hit in patches every few days when people come to do test runs) just timed out this morning. I can't see any evidence of the timed out requests in either the access log nor anything in the event log. Restarting Zope appeared to fixed the problem, but I'm slightly confused as I can't find any entries in the logs of the Apache instance that sits in front of the test instances for the failed requests... Has anyone seen anything like this? cheers. Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] MySQL server has gone away with RelStorage
Hi Shane, With the server I'm testing on, when coming back to test after a couple of days working on something else, I get the following logged at Warning: Reconnecting load_conn: (2006, 'MySQL server has gone away') Does this mean the reconnection is being handled by RelStorage or that I need to do some work? Also, after this, it seems life the ZODB object cache is emptied. Is that happening or is it just my imagination? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Status of gocept.zeoraid
On 20/10/2010 17:03, Sylvain Viollon wrote: One of my customers would like to use gocept.zeoraid in production. So him, and I have some questions: - Is it stable enough to be used in production ? No. - Does it bring speed improvements over regular ZEO setup (I don't think so) ? No. It is currently 5 times slower when tested with zodbshootout. - Does it work with blobs ? I will have a blob directory shared between different servers. Can I use it to access and edit directly the blob in that directory ? Don't know. Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 14/10/2010 02:40, Darryl Dixon - Winterhouse Consulting wrote: ...and fsdump appears to have no way of saying show me the last few transactions in this 35GB file :-( Yeah it's pretty verbose. I usually just pump the output to a text file ...yeah, but a 35GB zodb is going to result in, what, a 120? 200? GB text file. I don't think I actually have that much disk space lying around on these servers, let alone how long fsdump would take to produce it ;-) I wonder how hard it'd be to add some kind of start-at-the-end feature to fstail? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage Questions
Hi Shane, For all of these, if there are docs, please point me in their direction. I've been working from: http://pypi.python.org/pypi/RelStorage In my case, I'm looking at ZODB 3.9.6, Zope 2.12.7 onto MySQL 5.1 provided as a two machine cluster maintained by a hosting company. This sound good? Any different versions I should look at? From the features section: multiple ZODB instances can share the same database; does this mean I can put more than one storage in the same mysql database? Is this a good idea or should I stick to one storage per mysql database? From the RelStorage options section: - I'll have one storage with keep-history true and one with it false. Both will require packing and garbage collection, right? What's the recommended method and frequency of doing that? I'm guessing the zodbpack script but wanted to doubt check that does both pack and garbage collection. - How do you pick the right pack-duty-cycle? - poll-interval - when is it recommended to use memcached caching? How do you decide to pick what values to use for cache-prefix, cache-local-mb and cache-delta-size-limit? - Under what situations would you recommend changing the commit-lock-timeout from its default of 30 seconds? From the MySQL Adapter options section: - When is/isn't gzip compression a good idea? What is actually being compressed by this options? - Should I worry about init_command, read_default_file, read_default_group and client_flag or just ignore them? - What does load_infile actually do? Should it be used or avoided? Sorry for all the questions, hope these aren't too much of a pain to answer! cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] One more RelStorage question
Hi Shane, I forgot to ask... Is RelStorage suitable for a temporary storage? If so, what would be the recommended options for the following storage: zodb_db temporary # Temporary storage database (for sessions) temporarystorage name temporary storage for sessioning /temporarystorage mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer /zodb_db ...and could one database containing such a temporary storage be used by more than one client, and would that be a good idea? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
On 14/10/2010 17:09, Jim Fulton wrote: What does this error message imply? That two processes were trying to access the cache. The above did not, however, occur if I stopped and then started the app server... Race condition? Hopefully, you screwed up somehow and started the process while it was already running. :) I don't think so. This is using zc.recipe.rhrc to drive zope2instance Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
On 14/10/2010 17:13, Alan Runyan wrote: I went for the following in the end: ... cache-size 200Mb client app wait on /zeoclient mount-point / cache-size 5 cache-size-bytes 500Mb Any obvious issues with that? No. each zeoclient needs a name in your conf. That's what the 'client' parameter provides, and there's only one client per machine. These clients do open two storages, but the storage name forms part of the cache file name. so the .zec files dont collide. this is what is happening to you. look at component.xml again for details on assigning zeo client a name or identifier. See above ;-) please open issue at launchpad; if this fails zope should NOT start. Sure, which tracker do you recommend? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] blocked transactions?
Hi All, What do the following error message mean: -- 2010-10-12T17:05:58 INFO ZEO.StorageServer (24471/192.168.100.100:54407) Transaction blocked waiting for storage. Clients waiting: 3. -- 2010-10-12T17:07:16 INFO ZEO.StorageServer (24471/192.168.100.105:50112) Blocked transaction restarted. Clients waiting: 2 ...and would ZODB 3.10 help fix them? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage and ZODB 3.10?
Hi Shane, Does RelStorage 1.4 work with ZODB 3.10? (In particular, the Undo changes and the tightened storage API..) I see the mention of 3.10.0b7, but just wanted to check that means you're okay with 3.10 final ;-) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blocked transactions?
On 13/10/2010 14:06, Jim Fulton wrote: On Wed, Oct 13, 2010 at 4:02 AM, Chris Withersch...@simplistix.co.uk wrote: Hi All, What do the following error message mean: -- 2010-10-12T17:05:58 INFO ZEO.StorageServer (24471/192.168.100.100:54407) Transaction blocked waiting for storage. Clients waiting: 3. -- 2010-10-12T17:07:16 INFO ZEO.StorageServer (24471/192.168.100.105:50112) Blocked transaction restarted. Clients waiting: 2 ...and would ZODB 3.10 help fix them? These aren't error messages. You can tell because they say INFO. If you say so ;-) Their occurences coincided with an app server cluster of 4 clients locking up completely... I would dearly love to know why that happened. Any way to interrogate a running zeo server to find out what it thinks it's up to? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blocked transactions?
On 13/10/2010 15:40, Jim Fulton wrote: Wouldn't I see storage errors in that case? No. Unless you set a transaction timeout, the storage will wait for the finish indefinitely. ...I meant on the client side. There were no errors other than the protocol timeout ones below: File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/ServerStub.py, line 377, in stub raise ValueError(Timeout waiting for protocol handshake) ValueError: Timeout waiting for protocol handshake These were on the client, not the server. That's odd. This suggests that there's something else going on with your storage server and that the transactions waiting messages are a red herring. Indeed. I'd dearly love to know what, sadly, the storage server is zeoraid, and I can't expect you to help with that ;-) (zeoraid then talks to back end zeo servers, and the only out-of-the-ordinary stuff logged was on the zeoraid servers, and it was those blocked messages) When that's configured, what information does it provide and how do I get it? (if there are docs, lemme know and I'll go read them instead) Go read the code or try it. Will do. Related: how can I find out how long transactions are taking? Note that we're really talking about how long commits are taking, specifically the time between vote and finish. You can determine that from the waiting messages. So there's no logging I can turn on to see how long commits are taking, unless I can get more of those 'transaction blocked' messages to show up? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blocked transactions?
On 13/10/2010 15:59, Jim Fulton wrote: So there's no logging I can turn on to see how long commits are taking, unless I can get more of those 'transaction blocked' messages to show up? You can set the log level to debug. This will log the entire ZEO protocol. Analysis will take require a bit of work. (Note that ZEO 3.10 no longer does this level of logging even at the debug level, as it slowed the server down.) yeah, slowing the server down is what I'm worried about so I'm not gonna do this unless someone tells me I really need to ;-) You could temporarily hack the ZEO storage server code to collect statisics. That would probably be easier than debug logging. It worries me a little that this kind of thing, which feels important for ZODB tuning, isn't available without hackery. Are there any plans to get some storage server stats accumulation? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 13/10/2010 15:23, Jim Fulton wrote: You can connect to the monitor port in 3.9 and earlier, if the monitor port is configured. In 3.10, the monitor server is replaced by a ZEO client method, server_status. This tells you much the same information that's in the log messages. Okay, monitor port up and running now. I see commits listed when I'm not expecting any. Do we have any kind of tail -f /some/filestorage.fs yet? (or have we always had such a tool) to see what the last few transactions in the underlying file storage look like in a human-readable form? cheers, Chrus -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
On 12/10/2010 17:56, Jim Fulton wrote: Currently the only configuration docs are in the .xml Alan provided a link to. The best size will depend on the app. Is there a downside to too big, provided there's plenty of disk space free on the app servers? There are some docs in http://svn.zope.org/ZODB/trunk/doc/, although I don't know if they're accurate. ...yeah, those look a lot like the original docs that Guido(?) wrote. In ZODB 3.10, the cache tracing analysis tools actually work and produce meaningful numbers if you start tracing with an empty cache. Okay, does that mean the stuff that's documented should be ignored for ZODB 3.9? I went for the following in the end: ... cache-size 200Mb client app wait on /zeoclient mount-point / cache-size 5 cache-size-bytes 500Mb Any obvious issues with that? One problem I did hit, though, was if I restarted an app server, I got: File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZODB/config.py, line 210, in open **options) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/ClientStorage.py, line 395, in __init__ self._cache = self.ClientCacheClass(cache_path, size=cache_size) File /var/buildout-eggs/ZODB3-3.9.6-py2.6-linux-i686.egg/ZEO/cache.py, line 194, in __init__ self._lock_file = zc.lockfile.LockFile(path + '.lock') File /var/buildout-eggs/zc.lockfile-1.0.0-py2.6.egg/zc/lockfile/__init__.py, line 76, in __init__ _lock_file(fp) File /var/buildout-eggs/zc.lockfile-1.0.0-py2.6.egg/zc/lockfile/__init__.py, line 59, in _lock_file raise LockError(Couldn't lock %r % file.name) LockError: Couldn't lock 'zec.lock' ...and yet the app server came up fine. What does this error message imply? The above did not, however, occur if I stopped and then started the app server... Race condition? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 14/10/2010 01:28, Darryl Dixon - Winterhouse Consulting wrote: Do we have any kind of tail -f /some/filestorage.fs yet? (or have we always had such a tool) to see what the last few transactions in the underlying file storage look like in a human-readable form? fsdump.py gets you pretty close (ZODB/scripts/fsdump.py). Between that and the Undo log for the DB inside Zope, you might be able to figure it out... ..well, the Undo log shows no entries ;-) ...and fsdump appears to have no way of saying show me the last few transactions in this 35GB file :-( Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodb monitor port / tailing a .fs
On 14/10/2010 01:30, Laurence Rowe wrote: There's also fstail: $ bin/zopepy -m ZODB.scripts.fstail var/filestorage/Data.fs That looks more like it.. Any ideas what options there are to it? From reading the source, I see a -n, but no idea what it should do. I'm looking for a more detail knob, and also a 'tail -f' equivalent... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction commits don't have to write data?
On 14/10/2010 01:21, Chris Withers wrote: On 13/10/2010 15:23, Jim Fulton wrote: You can connect to the monitor port in 3.9 and earlier, if the monitor port is configured. In 3.10, the monitor server is replaced by a ZEO client method, server_status. This tells you much the same information that's in the log messages. Okay, monitor port up and running now. I see commits listed when I'm not expecting any. ...but using fstail, I see no new transactions in the last 6 hrs. Is it normal/ok/etc to commits in the monitor server output but without and actual transaction showing up in fstail? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] zodb 3.9.6 performance tuning?
Hi, After recently moving a medium-sized customer project from Zope 2.9.8 to Zope 2.12.7 sitting on ZODB 3.9.6 by way of zeoraid b7, I'm now seeing some fairly weird and bad behaviour that we didn't have before. So, everything was fine after the migration. However, today I turned on a batch job which pumps content into the storage. This appears to be *much* slower than on Zope 2.9.8 (with its included ZODB release) and also appears to cause the web clients to be slow enough when its running that they start dropping out of the load balancer pool. How can I debug what's going on? Is there any way to get diagnostics out of a 3.9.6 zeo server to find out what it's up to? cheers, Chris PS: If it helps, it looks like the zeoraid process is using an order of magnitude more cpu than the zeo process, according to top, but even that's only 30% of the CPU load... -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
Hi All, I'm wondering if some of my issues might be alleviated by persistent client caches. Now, I have the impression that these were unreliable and buggy and had been so for some time. Is this still the case? If not, where are the docs on how to configure them nowadays? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] persistent client caches okay to use again in ZODB 3.9.6?
On 12/10/2010 17:11, Jim Fulton wrote: Now, I have the impression that these were unreliable and buggy and had been so for some time. They became much better in later 3.8 releases and still much better lately. We've been using them in production for quite a while. Any docs on recommended settings around? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] semantics of _v_ attributes?
On 06/10/2010 15:19, Jim Fulton wrote: _v_ attributes should be dropped whenever an object's state is cleared. It would be great if there was some clear documentation covering all this in the ZODB package... So, you want to update an object using old state. I would use: base._p_invalidate() # remove state, including volatiles, no matter what base._p_activate() # make sure we're not a ghost base.__setstate__(state) # change the state base._p_changed = True # marke object as dirty This isn't foolproof, depending on how an object uses _v_. For example if __setstate__ sets _v_s if they aren't already set, then the above code would likely fail, as the _v_s would reflect the old state, not the new. Okay, it works for the Zope 2.12 problem at hand: https://bugs.launchpad.net/zope2/+bug/649605 If this use case is important, it would be better for ZODB tp provide an API to handle this directly, so you didn't have to hack together existing APIs. Well, I have customers who think it's important, but it depends on whether you still think Zope 2 with TTW editing is important ;-) I assume this code will ultimately be followed by a commit. Correct. Well, for your use case, you're going to be committing a change to the object. That will cause instances of the object in all other connections to be invalidated, dropping their volatile attributes. Cool :-) I'll take this on trust, I don't want to contemplate trying to write a test that proves it ;-) Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] semantics of _v_ attributes?
Hi All, I'm trying to fix this bug: https://bugs.launchpad.net/zope2/+bug/649605 ...and I'm struggling to find documentation on the semantics of _v_ attributes. In particular, surely this code: base._p_activate() # make sure we're not a ghost base.__setstate__(state) # change the state base._p_changed = True # marke object as dirty ...should be enough to indicate that all _v_ attributes should be dropped? If it isn't, what is the correct way to indicate to Persistent/ZODB that all _v_ attribute should be dropped not only for the current object but for all occurrences of that object across all connections to the current ZODB? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB history problem with Zope 2.12.7
On 28/09/2010 12:55, Tres Seaver wrote:
On 09/28/2010 03:36 AM, Chris Withers wrote:
On 28/09/2010 00:12, Marius Gedminas wrote:
--- ./OFS/History.py.orig 2010-09-28 02:11:56.535745440 +0300
+++ ./OFS/History.py2010-09-28 02:12:00.043764683 +0300
@@ -151,6 +151,9 @@
base = aq_base(self)
base._p_activate() # make sure we're not a ghost
base.__setstate__(state) # change the state
+for attr in dir(base):
+if attr.startswith('_v_'):
+delattr(base, attr)
base._p_changed = True # marke object as dirty
self.manage_afterHistoryCopy()
Thanks, I guess I'll monkey patch for now, here's the bug:
https://bugs.launchpad.net/zope2/+bug/649605
However, I'm curious, so the above will fix the object in the current
thread, but what about objects in other threads?
(or do _v_ attributes get killed off at the start of each transaction?)
Only when objects are ghostified (due to an invalidation from another
thread or process) or evicted from the cache. I'm not quite sure how
the case you are triggering occurs, but if that code saves the new-old
version of the template to the ZODB, then any instances in other
threads' connection caches should be invalidated.
Feel free to repeat the steps I originally posted and are in the bug
description ;-)
Looks like the History.py code isn't doing something it should, but I
don't know when or what changed to make this so. I'll CC zodb-dev in
case Jim knows of any changes in ZODB (I'm using 3.9.6) that might be
relevant...
cheers,
Chris
--
Simplistix - Content Management, Batch Processing Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] attempting to undo an un-undoable transaction gives NotImplementedError with ZODB 3.9.5
On 09/09/2010 21:47, Jim Fulton wrote: 2010-08-26 22:49:01 ERROR txn.-1338986496 Failed to abort resource manager:MultiObjectResourceAdapter forZODB.DB.TransactionalUndo object at 0x2d509f0 at 47516176 Traceback (most recent call last): File transaction-1.1.0-py2.6.egg/transaction/_transaction.py, line 475, in abort rm.abort(self) File transaction-1.1.0-py2.6.egg/transaction/_transaction.py, line 548, in abort self.manager.abort(o, txn) File ZODB3-3.9.5-py2.6-macosx-10.3-fat.egg/ZODB/DB.py, line 985, in abort raise NotImplementedError NotImplementedError This used to give a meaningful error message. In 3.7 and earlier. Anyone know why it no longer does? The resource manager abort method used to be a pass. In 3.9 in raises an exception, which hides the undo exception. That's bad, right? An app should capture the undo exception information before calling abort. the app here is Zope 2's publisher process, right? My guess is that this works much better in ZODB 3.10. Any particular reason? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] attempting to undo an un-undoable transaction gives NotImplementedError with ZODB 3.9.5
Here's an example: 2010-08-26 22:49:01 ERROR txn.-1338986496 Failed to abort resource manager: MultiObjectResourceAdapter for ZODB.DB.TransactionalUndo object at 0x2d509f0 at 47516176 Traceback (most recent call last): File transaction-1.1.0-py2.6.egg/transaction/_transaction.py, line 475, in abort rm.abort(self) File transaction-1.1.0-py2.6.egg/transaction/_transaction.py, line 548, in abort self.manager.abort(o, txn) File ZODB3-3.9.5-py2.6-macosx-10.3-fat.egg/ZODB/DB.py, line 985, in abort raise NotImplementedError NotImplementedError This used to give a meaningful error message. Anyone know why it no longer does? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] tracking back to what code caused a conflicting object change
Hi All, Forwarding here as there may be more people with the relevant knowledge around. So, the key thing I'm trying to find is what code is changing the conflicting objects? (primarily since I'm not aware of any doing so, so it must be inadvertantly getting executed) Any ideas gratefully received! Chris PS: apologies for the inline forwarding, zope.org's MTAs are still incorrectly configured to reject messages forwarded as attachments... Original Message Subject: [Zope] ConflictError from Products.Transience.TransientObject.TransientObject Date: Tue, 27 Jul 2010 11:45:26 +0100 From: Chris Withers ch...@simplistix.co.uk To: zope list user z...@zope.org Hi All, I have an old Zope 2.9.8 instance which has recently started throwing the following unresolved ConflictErrors: Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 479, in commit Module ZODB.Connection, line 1050, in _commit_savepoint Module tempstorage.TemporaryStorage, line 202, in store ConflictError: database conflict error (oid 0x046a, class Products.Transience.TransientObject.TransientObject, serial this txn started with 0x0387c9f23edea155 2010-07-27 10:26:14.735074, serial currently committed 0x0387c9f28a1ff766 2010-07-27 10:26:32.373016) To my knowledge, the request that throws this isn't touching the session or temp_folder, but I guess something is. How can I find out what code is actually touch this and so causing this conflict? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - z...@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev ) Original Message Subject: [Zope] (also) ConflictError from Products.Transience.Transience.Increaser Date: Tue, 27 Jul 2010 11:47:09 +0100 From: Chris Withers ch...@simplistix.co.uk To: zope list user z...@zope.org References: 4c4eb8c6.8050...@simplistix.co.uk Chris Withers wrote: To my knowledge, the request that throws this isn't touching the session or temp_folder, but I guess something is. How can I find out what code is actually touch this and so causing this conflict? In similar circumstances, I also see: Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 479, in commit Module ZODB.Connection, line 1050, in _commit_savepoint Module tempstorage.TemporaryStorage, line 202, in store ConflictError: database conflict error (oid 0x29, class Products.Transience.Transience.Increaser, serial this txn started with 0x0387c9ffb271bc44 2010-07-27 10:39:41.822878, serial currently committed 0x0387ca0108963711 2010-07-27 10:41:02.012526) How can I find out what's going on to cause this conflict? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - z...@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev ) -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
Christian Theune wrote: and a mysterious bug WRT oid generation This one's new to me, where can I find out more? The one you reported quite a while ago that ended in mixed-up OID results. Have you got a url to the tracker issue? I think it was this one: https://bugs.edge.launchpad.net/gocept.zeoraid/+bug/485976 Okay, but again, this one seems to be mitigated by only running one zeoraid server at a time, right? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
Christian Theune wrote: On 06/11/2010 05:50 PM, Alan Runyan wrote: I suppose you could look at zeoraid. Hadn't thought of that, will look at it, thanks! Anyone using ZEORaid in production? I hope not. There's still open issues with split brain situations IIUC, that can be mitigated by only having one zeoraid server live at a time? and a mysterious bug WRT oid generation This one's new to me, where can I find out more? Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEORaid was Re: Restoring from repozo and reusing an index file?
Christian Theune wrote: On 06/12/2010 09:47 AM, Chris Withers wrote: Christian Theune wrote: On 06/11/2010 05:50 PM, Alan Runyan wrote: I suppose you could look at zeoraid. Hadn't thought of that, will look at it, thanks! Anyone using ZEORaid in production? I hope not. There's still open issues with split brain situations IIUC, that can be mitigated by only having one zeoraid server live at a time? Yes. Our goal is to mitigate even splits by making the system read-only and manually allowing to specify which parts are active in that case. An automatic voting procedure would then require more than 2 participants, but with even splits you at least won't fail horribly. *nods sagely while looking a little blank* ;-) and a mysterious bug WRT oid generation This one's new to me, where can I find out more? The one you reported quite a while ago that ended in mixed-up OID results. Have you got a url to the tracker issue? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Restoring from repozo and reusing an index file?
Paul Winkler wrote: 3 should be pretty fast as its an incremental backup. 4 may take a little while but iirc it's not too bad for this db - maybe 10 minutes. Most of the downtime is caused by rebuilding the index which takes way longer (30 min?) So I had two questions: 1) would it be safe to copy the index file from the old host and just use that with the filestorage generated by repozo? Almost certainly not. 2) should I maybe ditch repozo and just rsync everything? That'd replace step 3 with a second call to rsync, and eliminate steps 4 and 6. But I'm not sure how fast it is at updating a big binary file. I think I'll do an experiment today and report back on this option. I suppose you could look at zeoraid. Have you tried just starting with an empty index file? Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] The database root: It's a trap!
Jim Fulton wrote: A. Change ZODB so that new databases have BTrees as root objects. This has the advantage that BTrees are scalable. It has a number of down sides: - BTrees should only be used when keys are known to have a stable ordering. If we had a scalable hash data structure, that would be better. - This encourages mistake 2 above. B. Change ZODB so that the root object is a variant of persistent mapping that either refuses to store more than a small number of objects, or at least issues a warning when more than a small number of objects is stored. C. I'd really like to be able to configure what the root object is as database creation time. That way it can be application specific. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Problem with handling of data managers that join transactions after savepoints
Jim Fulton wrote: I plan to implement A soon if there are no objections. Unless someone somehow convinced me to do D, I'll also add an assertion in the Transaction.join method to raise an error if a data manager joins more than once. Option A sounds sensible. It also means I won't have to change anything in the zope.sqlalchemy data manager. Very cool. I was hoping non-ZODB-data-manager authors were paying attention. :) If anyone knows of any other, I would appreciate someone forwarding this thread to them. zope.sendmail and MaildropHost have data managers. I've seen some file-based things that was a data manager and a few others in people's BFG stacks, maybe they can pipe up and/or let the others of those wsgi components know. I'm also likely about to write one, but I'm dumb, so can't comment meaningfully on the options ;-) Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: compatibility of ZEO clients and servers running different Python versions
Jim Fulton wrote: Because of the complexity of the solution, I'm, sadly, inclined to leave things as they are: - Can't use Python 2.4 clients with Python 2.5 or 2.6 servers - Can use Python 2.6 clients with 2.4 servers. - Can't use Python 2.5 or 2.6 servers with clients running ZODB 3.7 or earlier. I wonder what other people think. Personally, I'm certainly okay with this. It'd be great to have it advertised somewhere prominent though... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
Marius Gedminas wrote: * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 it's gone. Solved by peeking into the method signature with inspect.getargspec() and supplying a version only if it's needed. I believe this may be fixed with 3.9.4 is released. ...but it sounds like your code isn't ending up using the Z303 adapter, which is why you're running into the problem. How are you mixing versions? 3.8 on client and 3.9 on server or? (sorry, I lost the previous thread and saw this as a new one when Jim changed the subject line...) cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Making ZODB / ZEO faster
Jim Fulton wrote: On Fri, Dec 4, 2009 at 11:31 AM, Erik Dahl ed...@zenoss.com wrote: ... I haven't dove into the relstorage yet but have heard it will perform better. It doesn't. See: https://mail.zope.org/pipermail/zodb-dev/2009-October/012758.html Jim, That's a little sweeping and some might see you as a little biased in this ;-) Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jim Fulton wrote: BTW, offline back ups if file-storage even with Blobs is straightforward without repozo. I couldn't parse this, could you restate particularly wrt to what ou mean by offline? If there's a way other than repozo, then it would be great to know what it was... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Jeff Shell wrote: I'm not sure whose expectations you're satisfying as the items in ZODB.scripts are a wild and inconsistent mess. It's been trial and error just to find ones that seem to work, outside of repozo. Oh the irony of the one script that Jim is looking to remove, because someone dared to do some maintenance on it ;-) Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
Martin Aspeli wrote: - there were no tests for the existing code There are tests, they're just manual and spew a lot of output even when passing, but they do (as best I could tell) exercise the backup and restore cycle quite heavily. I ran these before and after to verify they didn't fail but did exhibit the behaviour I was trying to fix before I started and didn't fail *and* didn't exhibit the behaviour I was trying to fix after. - Tres wrote a trivial test for the trivial fix Indeed. His test exhibited the problem prior to the fix and showed it to be fixed after. (since the problem was a warning generated on import) - You now found some different problem in the same code, ...but didn't say what it is. Jim seemed to hint it was a Python 3 problem. I'd warrant there are much more serious problems to be worrying about for Python 3 first. If Python 3 compatibility is now a requirement for any patches submitted, then we're well and truly fucked. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] client cache bugs when a client switches from one zeo server to another
Hi All, So, I got the following errors when using a setup I'll describe shortly: o Module Products.ZCatalog.CatalogBrains, line 86, in getObject o Module OFS.Traversable, line 250, in restrictedTraverse o Module OFS.Traversable, line 226, in unrestrictedTraverse __traceback_info__: ([], 'tvi-101108-rb3') o Module OFS.ObjectManager, line 728, in __getitem__ KeyError: 'An object id' This catalog is maintained in sync with the content as changes to one are always committed in the same transaction as changes to the other. o Module Products.BTreeFolder2.BTreeFolder2, line 293, in getBatchObjectListing IndexError: 124 This looks like an impossible error that occurs when a BTreeFolder's object count doesn't match the length of its objectIds(). These errors would be bizarre and disturbing were it not that they went away as soon as I restarted the Zope instance. That suggests client cache corruption to me as the client in question has no persistent cache. So anyway, the setup is one Zope 2.9.8 client connected to two zeoraid 1.0b7 zeo servers. The client has been switching between the two zeoraid servers while I've been testing zeoraid by bringing it up and down. So, I'm left wondering: - have there been any client cache bugs fixed with Zope 2.9.8 that would cause this kind of behaviour when a client has multiple zeo servers and switches between the two due to failure of one of them? - (mainly at Christian) can you think of anything in zeoraid that might cause this behaviour? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9.3 history call causing problems for storages that still accept version parameters
Christian Theune wrote: Traceback (most recent call last): File ZODB3-3.9.3-py2.6-linux-i686.egg/ZEO/zrpc/connection.py, line 581, in handle_request ret = meth(*args) File gocept.zeoraid-1.0b6-py2.6.egg/gocept/zeoraid/storage.py, line 219, in history assert version is '' There is no ZEOStorage instance involved in this as far as I can tell. What am I missing? Check the setup_delegation method: the history method is patched through directly so your traceback doesn't see the ZEOStorage anymore. OK, I've committed tests and patches to deal with this issue and the similar issue in ZEOStorage308Adapter on the 3.9 branch. Should I merge these changes to the trunk too? cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
Jim Fulton wrote: On Thu, Nov 19, 2009 at 7:01 PM, Chris Withers ch...@simplistix.co.uk wrote: Jim Fulton wrote: There's nothing official or supported about a backup solution without automated tests. So I guess there isn't one. Right, so what does Zope Corp use? We use ZRS, of course. Well, ZRS solves the HA challenge the same way as zeoraid, if I understand correctly, but what about offsite backups and the like? The project I'm currently working on uses repozo to create backups that: - get hoovered by the hosting provider's backup mechanisms and rotated offsite daily - get sprayed by rsync over ssh to a DR site on another continent How would ZRS solve these problems? I'd prefer that there be a file-storage backup solution out of the box. repozo is the logical choice. It sounds like it needs some love though. This isn't something I'd likely get to soon. I'm not sure how much love repozo needs. It works, and it won't need changing until FileStorage's format changes, which I don't see happening any time soon. cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
Jim Fulton wrote: I need to review the changes before the release. I'll probably reject the repozo change without an automated test. Are you serious? You'd rather have a broken tool than one that isn't broken on the basis that the existing tests aren't part of the test suite that gets run by default?! In the future, please don't check changes directly into trunk or a release branch. Check them into a development branch. I'll review and merge. Sure, although these are both miniscule changes... I've learned my lesson, I won't try and contribute to ZODB development in future... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
Tim Peters wrote: That's what he said -- and you made him repeat it several times by now. Yes, I find it hard to believe that someone would deliberately break something that someone else has taken the trouble to fix (and run the tests for!)... I've learned my lesson, I won't try and contribute to ZODB development in future... I think that's the wrong lesson to take. repozo had several fatal bugs when I inherited it, and part of the reason is that there were no tests of any kind. As the checkin comment said when I added testrepozo.py: Better Than Nothing -- which is what we had before. There wasn't time then to finish the job (i.e., to automate the testing and dope out some way to make failure output more /helpful/ than just guts don't match). What I checked in then was essential, though, to verify the slew of bugfixes that went in around the same time. Alas, 5 1/2 years later, repozo testing apparently remains just barely Better Than Nothing, but that's really not good enough for a supported approach. I'm not arguing with the above, and I'm not asking for anything more supported than already exists. However, requiring someone to completely rewrite a test suite for software that they're never needed to understand on the basis that they corrected some imports to make them compatible with a newer version of python seems unreasonable. So the right lessons are: (1) to do development on a development branch; Where is the development process that requires this documented? and, (2) to finish the job I started if repozo is to be treated as more than just another random piece of flotsam in the ZODB ocean. I'm fine with the status quo... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
Jim Fulton wrote: On Fri, Nov 20, 2009 at 9:32 AM, Chris Withers ch...@simplistix.co.uk wrote: ... I'm not sure how much love repozo needs. It works, and it won't need changing until FileStorage's format changes, which I don't see happening any time soon. It just occurred to me that repozo doesn't support blobs. Indeed, I've never been comfortable with blobs due to their spotty support in older tools and the need to make a filesystem HA for them to be used in a HA environment. As a result, I don't use them, so I'm okay with repozo not suppporting them. Yes, I'd love it if repozo *did* support them, but I wouldn't have the faintest idea of how to do that, even if I did have time to work on it... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo, neither official nor supported, apparently...
Jeff Shell wrote: For what it's worth, I've dealt with broken / unloved / mildly-out-of-date ZODB scripts by making a new package that my little company uses internally. I don't agree with this because it's fragmenting a tiny development group even further, no to mention creating private forks of tools that mean everyone loses in terms of eyeballing... It allowed me to get some useful tools available again, Such as? And what stopped you working on them back in the ZODB source tree? and was independent from any ZODB release cycle. I have no problems with the ZODB release cycle... You could always explore this option, and even do it better than me by sharing with the community. This just feels wrong to me. If I *was* changing a script's behaviour such that it needed new tests, I'd make sure those were automatically run. The changes to repozo didn't require any new tests, the bug was provoked just by running the existing tests with python 2.6. So if 'repozo' needs love, even if it's simple love, I would recommend that someone make a 'z3c.zodbtools' or 'z3c.repozo' or 'z3c.fsbackup' project. I couldn't agree less. This kind of fragmentation helps no-one. It's something I'd only consider as a last resort, and then the most likely action is just to roll my own sdist with a funky version number and stick it on the customer's index server. Hell, I'll look at the package I've made and see if I can do something along those lines as a starting/example point. I'd much prefer to see your fixes worked back into the 3.9 branch and trunk of ZODB... Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
