[ZODB-Dev] transaction as context manager, exception during commit
Hi Jim, The current __exit__ for transaction managers looks like this: def __exit__(self, t, v, tb): if v is None: self.commit() else: self.abort() ..which means that if you're using the transaction package as a context manager and, say, a relational database integrity constraint is violated, then you're left with a hosed transaction that still needs aborting. How would you feel about the above changing to: def __exit__(self, t, v, tb): if v is None: try: self.commit() except: self.abort() raise else: self.abort() If this is okay, I'll be happy to write the tests and make the changes provided someone does a release when I have... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction as context manager, exception during commit
On 24 February 2011 10:17, Chris Withers ch...@simplistix.co.uk wrote: Hi Jim, The current __exit__ for transaction managers looks like this: def __exit__(self, t, v, tb): if v is None: self.commit() else: self.abort() ..which means that if you're using the transaction package as a context manager and, say, a relational database integrity constraint is violated, then you're left with a hosed transaction that still needs aborting. How would you feel about the above changing to: def __exit__(self, t, v, tb): if v is None: try: self.commit() except: self.abort() raise else: self.abort() If this is okay, I'll be happy to write the tests and make the changes provided someone does a release when I have... Looking at the way ZPublisher handles this, I think you're right. I think you might also need to modify the __exit__ in Attempt, which additionally handles retrying transactions that fail. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour
On Wed, Feb 23, 2011 at 15:08, Martijn Pieters m...@zopatista.com wrote: I've started a optimistic locking strategy patch in my patch queue, contains this locking strategy change only for now: https://bitbucket.org/mjpieters/relstorage-mq/src/tip/optimistic_commitlock_pack.patch I've made progress on the patch this afternoon. Next up are tests for both patches. The above patch now uses the nowait locking strategy to run pack batches. It has been renamed though, and now lives at: https://bitbucket.org/mjpieters/relstorage-mq/src/tip/nowait_commitlock_pack.patch -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Advice on whether to run relstorage database and Zope on different servers
Hi, We've been successfully running Plone 3.3 + Zope 2.10 + Relstorage + MySql for a couple of years now. We're running low on memory on our existing server (which has numerous other services on it) and will soon need to deploy an additional Plone 4.1 + Zope 2.13 + Relstorage + Mysql environment. We've been using virtual environments since September. For the purposes of this discussion please assume we don't have the option of increasing the memory on our existing server and so need to add a new one.Some questions I'd really appreciate your help with please: * I'm familiar with enterprise environments where you would have an app server and a database server but are there any advantages to putting Zope and MySQL on different servers? * I'd expect a performance hit if we run Zope + MySQL on separate servers but is this hit manageable? * Can we put anything in place to mitigate this? * Would you simply recommend having the existing mysql + zope on one server and the new mysql + zope on the other? Thanks in advance Anthony ___ Anthony Gerrard Email: anthonygerr...@gmail.com http://www.sponsorachild.org.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour
On Wed, Feb 23, 2011 at 14:41, Martijn Pieters m...@zopatista.com wrote: I've moved this patch to bitbucket at https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch and updated the README a little more to document the options to zodbpack. The two-phase pack patch has been updated again: https://bitbucket.org/mjpieters/relstorage-mq/src/tip/twophasepack.patch I've now updated the tests, and ironed out a flaw the tests revealed. I now consider this patch mergable into RelStorage. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage 1.5.0b1 dry-run two phase pack, better pack lock behaviour
On Thu, Feb 24, 2011 at 14:26, Martijn Pieters m...@zopatista.com wrote: I've made progress on the patch this afternoon. Next up are tests for both patches. The above patch now uses the nowait locking strategy to run pack batches. It has been renamed though, and now lives at: https://bitbucket.org/mjpieters/relstorage-mq/src/tip/nowait_commitlock_pack.patch I see a lot of transaction aborted errors on the ZODB multi-thread tests with this patch in place, so I'll have to investigate more. Thread debugging joy! -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Advice on whether to run relstorage database and Zope on different servers
We've been successfully running Plone 3.3 + Zope 2.10 + Relstorage + MySql for a couple of years now. We're running low on memory on our existing server (which has numerous other services on it) and will soon need to deploy an additional Plone 4.1 + Zope 2.13 + Relstorage + Mysql environment. We've been using virtual environments since September. For the purposes of this discussion please assume we don't have the option of increasing the memory on our existing server and so need to add a new one.Some questions I'd really appreciate your help with please: * I'm familiar with enterprise environments where you would have an app server and a database server but are there any advantages to putting Zope and MySQL on different servers? * I'd expect a performance hit if we run Zope + MySQL on separate servers but is this hit manageable? * Can we put anything in place to mitigate this? * Would you simply recommend having the existing mysql + zope on one server and the new mysql + zope on the other? Anthony this isnt a ZODB question; its a plone related application question. Ask plone-users. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Advice on whether to run relstorage database and Zope on different servers
Hi. On Thu, Feb 24, 2011 at 4:20 PM, Anthony Gerrard anthonygerr...@gmail.com wrote: * I'm familiar with enterprise environments where you would have an app server and a database server but are there any advantages to putting Zope and MySQL on different servers? Database servers (be it MySQL or ZODB) have different requirements than application servers (Zope). For small installations this generally doesn't matter, but if you have larger deployments you can adjust your hardware to fit the different roles. Database servers are generally disk I/O bound. Using the fastest possible and very reliable storage for these can help a lot. Be that either a RAID 1 or 10 of SAS drives with 10k or 15k rpm drives or a similar RAID of SSD's. Be prepared for some major performance degradation if you put your database on non-local NAS or SAN setups. Unless optimized correctly these will slow things down considerably. The ZODB has almost no CPU load, the SQL queries done by RelStorage also incur almost no CPU load. Memory isn't all too important for databases as long as the indexes fit into memory and you can handle savepoints / rollback data. Still some memory for the OS disk cache does help. Application servers are generally CPU bound and in case of Zope benefit a lot from memory for the connection cache. With RelStorage you can also use other cache setups like the shared memcached one. The disk types of application servers don't matter much and can be simple local disks, as there's no actual business data on the servers but just reproducible configuration setup - if you use proper version control for your code stored elsewhere that is. * I'd expect a performance hit if we run Zope + MySQL on separate servers but is this hit manageable? As long as the servers are on the same physical network with a low latency (~1ms) there's almost no overhead compared to running things locally. If you get into higher latencies of 10ms or more, you will see a noticeable drop in performance with Zope/Plone though. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage and PostgreSQL's VACUUM
Op 23-02-11 12:41, Maurits van Rees schreef: CLUSTER does not do anything unless you first tell it which index to use for clustering a table. Using postgres 8.4 I have done this for the four tables for which I think this is useful (the other tables had no content): cluster blob_chunk using blob_chunk_pkey; cluster current_object using current_object_pkey; cluster object_state using object_state_pkey ; cluster transaction using transaction_pkey; This did not have an effect on the above mentioned test database though. But that may be because it had already been fully vacuumed. Would these be the correct tables and indexes? Some of these tables have other indexes as well, but these are the primary ones. I just did this and the database went down from 38 GB to 21 GB, so I'd say this works. Would it make sense to add these lines to the postgres code in relstorage that creates the tables? Then users would only need to perform the 'CLUSTER;' command without any extra parameters and it would work. -- Maurits van Rees Web App Programmer at Zest Software: http://zestsoftware.nl Personal website: http://maurits.vanrees.org/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction as context manager, exception during commit
On 24/02/2011 11:38, Laurence Rowe wrote: Hi Jim, The current __exit__ for transaction managers looks like this: def __exit__(self, t, v, tb): if v is None: self.commit() else: self.abort() ..which means that if you're using the transaction package as a context manager and, say, a relational database integrity constraint is violated, then you're left with a hosed transaction that still needs aborting. How would you feel about the above changing to: def __exit__(self, t, v, tb): if v is None: try: self.commit() except: self.abort() raise else: self.abort() If this is okay, I'll be happy to write the tests and make the changes provided someone does a release when I have... Looking at the way ZPublisher handles this, I think you're right. Cool, I'll wait for Jim's verdict (and some time to implement it ;-)) before diving in... I think you might also need to modify the __exit__ in Attempt, which additionally handles retrying transactions that fail. Yeah, I see a bug relating to this was reported yesterday: https://bugs.launchpad.net/bugs/724332 I know it's a different problem, but sounds like the code for attempt needs some love... cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
