[ZODB-Dev] propose to change undo support in ZEO
I'm working on improving the zeo architecture to make it perform better. Undo is causing some problems. For storages that support undo *and* work with ZEO, I'd like to add a new method, transaction_oids, that returns the oids modified by a transaction. Currently AFAIK, there are only 2 storages that work with ZEO and support undo, FileStorage and DirectoryStorage. I'm not 100% that DirectoryStorage still works with ZEO. I propose to add a new interface that includes this new method and require this interface to support undo with ZEO. This will break undo support for DirectoryStorage in ZEO, but I doubt that anyone will care. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] propose to change undo support in ZEO
On Wed, Dec 23, 2009 at 10:24 AM, Jim Fulton j...@zope.com wrote: I'm working on improving the zeo architecture to make it perform better. Undo is causing some problems. For storages that support undo *and* work with ZEO, I'd like to add a new method, transaction_oids, that returns the Actually, undo_transaction_oids. oids modified by a transaction. Currently AFAIK, there are only 2 storages that work with ZEO and support undo, FileStorage and DirectoryStorage. I'm not 100% that DirectoryStorage still works with ZEO. I propose to add a new interface that includes this new method and require this interface to support undo with ZEO. This will break undo support for DirectoryStorage in ZEO, but I doubt that anyone will care. Jim -- Jim Fulton -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] propose to change undo support in ZEO
Sorry, I retract this proposal. :( Jim On Wed, Dec 23, 2009 at 11:25 AM, Jim Fulton j...@zope.com wrote: On Wed, Dec 23, 2009 at 10:24 AM, Jim Fulton j...@zope.com wrote: I'm working on improving the zeo architecture to make it perform better. Undo is causing some problems. For storages that support undo *and* work with ZEO, I'd like to add a new method, transaction_oids, that returns the Actually, undo_transaction_oids. oids modified by a transaction. Currently AFAIK, there are only 2 storages that work with ZEO and support undo, FileStorage and DirectoryStorage. I'm not 100% that DirectoryStorage still works with ZEO. I propose to add a new interface that includes this new method and require this interface to support undo with ZEO. This will break undo support for DirectoryStorage in ZEO, but I doubt that anyone will care. Jim -- Jim Fulton -- Jim Fulton -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] undo (and storage interface) brokenness
I'm looking at undo today because it is complicating ZEO transaction management. Undo is broken in a number of ways. Does anyone care? Does anyone use undo? Undo is broken in the following ways: - The TransactionalUndo resource manager invalidates objects too early. - If undo is mixed with stores, store invalidations may not be handled properly. This is because, with undo, the two-phase commit calls are made on a storage multiple times. This was OK a long long time ago, but not since tpc_vote started returning important information. Of course, mixing undo and stores is problematic in a number of ways, but there's nothing to prevent it now. - Because undo returns affected oids immediately, it's impossible to delay getting the storage lock until tpc_vote is called. This is a performance issue and is less serious than the problems mentioned above. Undo can be fixed, but not in a way that preserves the existing API. I'm tempted to remove undo support, at least from ZEO. So, I repeat: is anyone actually using undo these days? BTW, this analysis reveals a flaw in the storage APIs that I'd like to fix. You're allowed to call tpc_begin multiple times with the same transaction. Extra calls are ignored. Also, calling tpc_finish with a wrong transaction (a transaction other than the one the storage is dealing with or when a storage isn't committing) is ignored. This feels like a bug magnet to me, but it's there, afaict, to support the current broken undo implementation. Changing existing standard storages to be more restrictive (and fixing tests that specifically tested the permissive behavior) doesn't break anything but the current undo implementation. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
We use undo from time to time in a zeo setup. When it works it can be a life saver! :) If it goes away I guess it wouldn't be the end of the world though. -EAD Erik A. Dahl Co-Founder and CTO, Zenoss Inc. phone: 443-837-2597 http://www.zenoss.com On Dec 23, 2009, at 3:26 PM, Jim Fulton wrote: I'm looking at undo today because it is complicating ZEO transaction management. Undo is broken in a number of ways. Does anyone care? Does anyone use undo? Undo is broken in the following ways: - The TransactionalUndo resource manager invalidates objects too early. - If undo is mixed with stores, store invalidations may not be handled properly. This is because, with undo, the two-phase commit calls are made on a storage multiple times. This was OK a long long time ago, but not since tpc_vote started returning important information. Of course, mixing undo and stores is problematic in a number of ways, but there's nothing to prevent it now. - Because undo returns affected oids immediately, it's impossible to delay getting the storage lock until tpc_vote is called. This is a performance issue and is less serious than the problems mentioned above. Undo can be fixed, but not in a way that preserves the existing API. I'm tempted to remove undo support, at least from ZEO. So, I repeat: is anyone actually using undo these days? BTW, this analysis reveals a flaw in the storage APIs that I'd like to fix. You're allowed to call tpc_begin multiple times with the same transaction. Extra calls are ignored. Also, calling tpc_finish with a wrong transaction (a transaction other than the one the storage is dealing with or when a storage isn't committing) is ignored. This feels like a bug magnet to me, but it's there, afaict, to support the current broken undo implementation. Changing existing standard storages to be more restrictive (and fixing tests that specifically tested the permissive behavior) doesn't break anything but the current undo implementation. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
On Wed, Dec 23, 2009 at 3:38 PM, Erik Dahl ed...@zenoss.com wrote: We use undo from time to time in a zeo setup. When it works it can be a life saver! :) If it goes away I guess it wouldn't be the end of the world though. Do you use it via Zope 2 or do you call ZODB undo apis directly? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
Zope 2. -EAD Erik A. Dahl Co-Founder and CTO, Zenoss Inc. phone: 443-837-2597 http://www.zenoss.com On Dec 23, 2009, at 3:40 PM, Jim Fulton wrote: On Wed, Dec 23, 2009 at 3:38 PM, Erik Dahl ed...@zenoss.com wrote: We use undo from time to time in a zeo setup. When it works it can be a life saver! :) If it goes away I guess it wouldn't be the end of the world though. Do you use it via Zope 2 or do you call ZODB undo apis directly? Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
On Wed, Dec 23, 2009 at 9:26 PM, Jim Fulton j...@zope.com wrote: Undo is broken in a number of ways. Does anyone care? Does anyone use undo? Speaking from the Zope2/Plone crowd: I'm using it during development with a local file storage at times. In a real ZEO production setup, undo in Plone most of the time only works for a quick: revert the last transaction. For any transaction that happened a while ago our catalog-overuse will cause some persistent object inside the catalog to have changed in a later transaction. Throwing away all changes done in the meantime is usually not practical. For recovery purposes of older data, we have found a bit of manual work and zc.beforestorage to be the better approach. So if undo is gone from ZEO it wouldn't be tragic. If the remove last transaction could be retained, that would be nice. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
On Dec 23, 2009, at 2:26 PM, Jim Fulton wrote: On Wed, Dec 23, 2009 at 4:20 PM, Jeff Shell j...@bottlerocket.net wrote: Are you asking at the ZODB level, or at a higher level, such as Zope? I'm asking if applications are using undo these days. We use it in our Zope 3 based CMS. We usually only expose a single-level undo and have not exposed any interface to undo more. What do you mean by single undo? Undoing just one transaction? Undoing just one transaction. When doing like an action like 'delete' inside of a folder, we add a link to the end of the message that is displayed on the next page to undo: Deleted 3 items (undo). If 'undo' is clicked, then the next page has a message to redo (undo the undo). It's always meant to undo a single, recent (but maybe not the most recent) transaction, there as a quick oops! recovery mechanism. Once they move on to the next page, the link to 'undo' is gone. Thanks, Jeff Shell j...@bottlerocket.net ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
Jim Fulton wrote: So, I repeat: is anyone actually using undo these days? I haven't used undo (not the APIs nor via any Zope UI) in years. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
Jim Fulton wrote: I'm looking at undo today because it is complicating ZEO transaction management. Undo is broken in a number of ways. Does anyone care? Does anyone use undo? Plone users tend to use Undo quite a lot. If you accidentally delete something, say, it's pretty much your only way out save for a full restore from backup (which would likely be at least a few hours out of date). Undo can be fixed, but not in a way that preserves the existing API. I'm tempted to remove undo support, at least from ZEO. Please, please don't. It'll upset a lot of people by removing an important (if imperfect) safety mechanism. It'll also make stronger the ZODB is too opaque, I can't trust it FUD. I think we can work to make Zope (2?) use a new API for undo is necessary. But *please* don't remove the feature. So, I repeat: is anyone actually using undo these days? I'd say thousands of Plone users, yes. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
Hanno Schlichting wrote: On Wed, Dec 23, 2009 at 9:26 PM, Jim Fultonj...@zope.com wrote: Undo is broken in a number of ways. Does anyone care? Does anyone use undo? Speaking from the Zope2/Plone crowd: I'm using it during development with a local file storage at times. In a real ZEO production setup, undo in Plone most of the time only works for a quick: revert the last transaction. For any transaction that happened a while ago our catalog-overuse will cause some persistent object inside the catalog to have changed in a later transaction. Throwing away all changes done in the meantime is usually not practical. ... although sometimes it is preferable to a full backup restore or living with whatever it is you want to restore (usually a delete). For recovery purposes of older data, we have found a bit of manual work and zc.beforestorage to be the better approach. What is zc.beforestorage? So if undo is gone from ZEO it wouldn't be tragic. If the remove last transaction could be retained, that would be nice. I actually think it would be tragic. Or at least pretty miserable. People who manage high availability setups can probably find other ways (like very frequent backups and good restore routines). For a lot of average installations, it can be a lifesaver. It was a lifesaver for one of my clients about a week ago, for example. The late night phonecall kind of lifesaver. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
On Thu, Dec 24, 2009 at 1:34 AM, Martin Aspeli optilude+li...@gmail.com wrote: Hanno Schlichting wrote: Throwing away all changes done in the meantime is usually not practical. ... although sometimes it is preferable to a full backup restore or living with whatever it is you want to restore (usually a delete). Right. That's why we use beforestorage. For recovery purposes of older data, we have found a bit of manual work and zc.beforestorage to be the better approach. What is zc.beforestorage? Look at its PyPi page http://pypi.python.org/pypi/zc.beforestorage Basically it's a convenient way to open a database as it was at a certain point in time. So for Plone in practice you get a separate copy of the live database or from a backup, open it via beforestorage at the time you want and export the content you are looking for via a zexp. Put that zexp into the import of your live server and import the content. You usually have to reindex the content, as you cannot take the catalog entries with you. This works nicely for restoring deleted content and we have done so a number of times. If there's some unwanted changes made to individual content items, you can revert those via CMFEditions / application level versioning. So if undo is gone from ZEO it wouldn't be tragic. If the remove last transaction could be retained, that would be nice. I actually think it would be tragic. Or at least pretty miserable. In practice it hardly ever works. Relying on it as a substitution for backup or tested restore procedures is dangerous. People who manage high availability setups can probably find other ways (like very frequent backups and good restore routines). For a lot of average installations, it can be a lifesaver. It was a lifesaver for one of my clients about a week ago, for example. The late night phonecall kind of lifesaver. Frequent backups and good restore routines have nothing to do with high availability setups. They are a basic requirement for any type of setup. But even if you don't do frequent backups, as long as you have not packed your database, your data is still there and can be retrieved without too much hassle. But doing partial database recovery is something that requires a good deal of knowledge of the underlying application and database. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
Hanno Schlichting wrote: What is zc.beforestorage? Look at its PyPi page http://pypi.python.org/pypi/zc.beforestorage Thanks for the pointer! In practice it hardly ever works. Relying on it as a substitution for backup or tested restore procedures is dangerous. I know that a procedure based on backups and something like zc.beforestorage is going to be better. It's also an order of magnitude more difficult for people to set up and recover from. I think we'd see a pretty big backlash if we just removed the Undo feature, imperfect though it is. Unfortunately, I think the people most likely to have used it in a pinch are less likely to be reading this list. If the current API is broken, we should fix it. If that means fixing some of the UI in Zope to use a new API, that's OK too. Few people will have customised this or relied on the old API. If we had to say you only get step-by-step undo from the most recent transaction, then that'd probably be OK too. In this case, we should have a UI that lets you specify a number (undo N transactions) or a limit (undo up to and including transaction X). I think people understand that undoing a single transaction that's not the most recent one is error-prone anyway. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martin Aspeli wrote: Hanno Schlichting wrote: What is zc.beforestorage? Look at its PyPi page http://pypi.python.org/pypi/zc.beforestorage Thanks for the pointer! There is also dm.historical when you need to deal with earlier versions of objects: http://pypi.python.org/pypi/dm.historical/1.1 Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksy4iYACgkQCJIWIbr9KYxbFgCfQYanCFqC7dNmlU4P3M9esurK Y9IAn39geL7s4MltJZd5pmtyir9pbeEf =ib3E -END PGP SIGNATURE- attachment: lists.vcf___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
I'd like to add my vote to retaining Undo, I think it's a crucial feature. Changing the API is not a problem, and changing the UI could probably be an improvement. I use it all the time, especially when fiddling with customer datatabases for upgrades or bugfixes. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
