Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
On Wed, 14 Apr 2004, Kapil Thangavelu wrote: although i wonder if there is some hand waving in progress here that i can't see. i guess my semantic notion of versions has been that of long lived transactions, and is there a better means of thinking of them? how do they play across with multiple mounted zodbs? what would something like merge mean in the context of a catalog? By my understanding, outside the storage, different versions are completely independent. You could even simulate multiple databases using versions. It is the storage that knows how to create new versions, merge versions, and prevent objects being used in a version from being changed. Mounted databases try to use the same version as the root connection. To merge a catalog, you could bring one of the stored catalogs to life within the storage and replay the sequence of transactions that occurred in the other catalog. This could be tricky, and opening a catalog within a ZEO server would make the ZEO server less stable. I think there might be other ways to look at it. One way to simplify would be to put catalogs in their own storage, separated from the content they index. That way, you could (ab)use the transaction machinery to abort or commit indexes as you see fit within the catalog conflict resolver. I've experimented with this before, and it seemed promising. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
On Tue, 2004-04-13 at 12:01, Shane Hathaway wrote: On Tue, 13 Apr 2004, Kapil Thangavelu wrote: since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site. This is only true of FileStorage. Some other storage could implement ZODB versions with merging capability rather than locking. good point, just because hasn't been doesn't mean it can't ;-) although i wonder if there is some hand waving in progress here that i can't see. i guess my semantic notion of versions has been that of long lived transactions, and is there a better means of thinking of them? how do they play across with multiple mounted zodbs? what would something like merge mean in the context of a catalog? i'm also curious how you dealt with svn transactions as part of the ape integration work to date. The same way it tries to impose transactions on the filesystem: in the vote phase, Ape looks for possible problems and aborts early if it detects anything that will cause the transaction to fail. Obviously, this provides no guarantee, but covers many cases. i was more curious how jean-francois was doing the svn ops in fileops, as svn is fundamentally a transactional store (as opposed to the fs), ie is there some record boundary notion of ape signalling the end of serialization for an object set, or was each operation being conducted in a separate svn transaction. -kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
granted the svn integration has lots of coolness factors including richer client integration, but things like zope version control (zope.org cvs) are already production quality and not so hard to integrate in for basic version control. -kapil On Tue, 2004-04-13 at 14:13, Arthur Chan Chi Chuen wrote: Hi all, I've read your discussion about version control, it seems a cool thing you guys making good progress. Btw, can I ask is the Ape using Subversion in Zope stable? how able CMF stuff? I wanna make/find a document management system which can provide some kinda version control in Plone. Thanks Arthur On Tue, 13 Apr 2004 04:06:26 -0400, Kapil Thangavelu wrote On Mon, 2004-04-12 at 18:03, [EMAIL PROTECTED] wrote: G'Day, Well, step one is done ... I now have Zope + Ape using Subversion as it's filesystem !! cool! This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem. Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support: - Use zope's username for SVN logging. using AccessControl.getSecurityManager().getUser().getId() and setting it up as a revision prop ( or directly when creating the repo transaction) should do it. - History/Undo support: View past revisions of an object, and revert to such a past revision. perhaps the code for this would be helpful, (remove image for code link) http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevisions. png - Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?) zope version support isn't really all that worthwhile, esp in a cmf context. in general zope's version support (or zodb more particularly) is a database level feature masquerading as an application one. since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site. otoh, integration with zope's transaction manager would be a good thing, although its problematic to integrate between svn and zope txn models, more on that in a moment. In the longer term, there's great opportunity for: - Built-in conflict management and resolution: No more need for a SafetyBelt type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error. i don't know that conflict management is really useful in this context. svn like zope relies on optimistic concurrency control, and currently doesn't support dav locking (which zope does). ie, it will just throw an error if the content has been changed since the transaction began. the 'normal' concurrency control of svn is better but dependent on using the working copy (wc) layer, which is additional programming and storage overhead. so at the layer of the svn client this is already done and working and good, but integrating this functionality into zope is a bit harder without wc overhard. this also makes the transaction integration becomes harder because both zope and svn are using what amounts to optimistic concurrency control which makes it impossible afaics, to get real txn integration, ie in zope's two phase commit strategy, the last chance for a participant to safely abort is tpc_vote, but there is no real way of knowing if the svn txn will suceed or not until its tried. if it is tried at this stage and succeeds then there is the possibility of a latter txn participant failing the tpc_vote and the txn being aborted, and if waits till tpc_finish (last part of two phase commit) and the svn txn fails it can hose the composite txn integrity. once svn supports dav locks, doing txn integration via resource locking as part of tpc_vote (or earlier) would be possible, till then.. i dunno, i can't see a way around this for real txn integration. i'm also curious how you dealt with svn transactions as part of the ape integration work to date. - Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird
Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
On Mon, 2004-04-12 at 18:03, [EMAIL PROTECTED] wrote: G'Day, Well, step one is done ... I now have Zope + Ape using Subversion as it's filesystem !! cool! This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem. Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support: - Use zope's username for SVN logging. using AccessControl.getSecurityManager().getUser().getId() and setting it up as a revision prop ( or directly when creating the repo transaction) should do it. - History/Undo support: View past revisions of an object, and revert to such a past revision. perhaps the code for this would be helpful, (remove image for code link) http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevisions.png - Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?) zope version support isn't really all that worthwhile, esp in a cmf context. in general zope's version support (or zodb more particularly) is a database level feature masquerading as an application one. since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site. otoh, integration with zope's transaction manager would be a good thing, although its problematic to integrate between svn and zope txn models, more on that in a moment. In the longer term, there's great opportunity for: - Built-in conflict management and resolution: No more need for a SafetyBelt type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error. i don't know that conflict management is really useful in this context. svn like zope relies on optimistic concurrency control, and currently doesn't support dav locking (which zope does). ie, it will just throw an error if the content has been changed since the transaction began. the 'normal' concurrency control of svn is better but dependent on using the working copy (wc) layer, which is additional programming and storage overhead. so at the layer of the svn client this is already done and working and good, but integrating this functionality into zope is a bit harder without wc overhard. this also makes the transaction integration becomes harder because both zope and svn are using what amounts to optimistic concurrency control which makes it impossible afaics, to get real txn integration, ie in zope's two phase commit strategy, the last chance for a participant to safely abort is tpc_vote, but there is no real way of knowing if the svn txn will suceed or not until its tried. if it is tried at this stage and succeeds then there is the possibility of a latter txn participant failing the tpc_vote and the txn being aborted, and if waits till tpc_finish (last part of two phase commit) and the svn txn fails it can hose the composite txn integrity. once svn supports dav locks, doing txn integration via resource locking as part of tpc_vote (or earlier) would be possible, till then.. i dunno, i can't see a way around this for real txn integration. i'm also curious how you dealt with svn transactions as part of the ape integration work to date. - Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort. to me this is one of the fundamental benefits of using svn, giving users the ability to use interfaces like tortoisesvn (win shell extensions) or mac finder extensions to interact directly with content. - Leveraging SVN's property management. Content in SVN has properties, much like Zope does. I haven't looked at it yet, but I've noticed .properties file appearing ... I'm guessing those are the Zope properties, which would be better handled by subversion's property mechanism. And properties are versioned too ! definitely! In the realm of the wishful thinking, there's even more: Right now, HEAD (Latest/youngest revision) is always used and worked with. The really powerful feature I want to eventually
Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
Hi all, I've read your discussion about version control, it seems a cool thing you guys making good progress. Btw, can I ask is the Ape using Subversion in Zope stable? how able CMF stuff? I wanna make/find a document management system which can provide some kinda version control in Plone. Thanks Arthur On Tue, 13 Apr 2004 04:06:26 -0400, Kapil Thangavelu wrote On Mon, 2004-04-12 at 18:03, [EMAIL PROTECTED] wrote: G'Day, Well, step one is done ... I now have Zope + Ape using Subversion as it's filesystem !! cool! This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem. Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support: - Use zope's username for SVN logging. using AccessControl.getSecurityManager().getUser().getId() and setting it up as a revision prop ( or directly when creating the repo transaction) should do it. - History/Undo support: View past revisions of an object, and revert to such a past revision. perhaps the code for this would be helpful, (remove image for code link) http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevisions. png - Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?) zope version support isn't really all that worthwhile, esp in a cmf context. in general zope's version support (or zodb more particularly) is a database level feature masquerading as an application one. since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site. otoh, integration with zope's transaction manager would be a good thing, although its problematic to integrate between svn and zope txn models, more on that in a moment. In the longer term, there's great opportunity for: - Built-in conflict management and resolution: No more need for a SafetyBelt type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error. i don't know that conflict management is really useful in this context. svn like zope relies on optimistic concurrency control, and currently doesn't support dav locking (which zope does). ie, it will just throw an error if the content has been changed since the transaction began. the 'normal' concurrency control of svn is better but dependent on using the working copy (wc) layer, which is additional programming and storage overhead. so at the layer of the svn client this is already done and working and good, but integrating this functionality into zope is a bit harder without wc overhard. this also makes the transaction integration becomes harder because both zope and svn are using what amounts to optimistic concurrency control which makes it impossible afaics, to get real txn integration, ie in zope's two phase commit strategy, the last chance for a participant to safely abort is tpc_vote, but there is no real way of knowing if the svn txn will suceed or not until its tried. if it is tried at this stage and succeeds then there is the possibility of a latter txn participant failing the tpc_vote and the txn being aborted, and if waits till tpc_finish (last part of two phase commit) and the svn txn fails it can hose the composite txn integrity. once svn supports dav locks, doing txn integration via resource locking as part of tpc_vote (or earlier) would be possible, till then.. i dunno, i can't see a way around this for real txn integration. i'm also curious how you dealt with svn transactions as part of the ape integration work to date. - Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort. to me this is one of the fundamental benefits of using svn, giving users the ability to use interfaces like tortoisesvn (win shell extensions) or mac finder extensions to interact directly with content. - Leveraging SVN's property
Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
On 12 Apr 2004 at 18:03, [EMAIL PROTECTED] wrote: Anyways, I'm just rambling by now ... Comments, thoughts and constructive criticism welcome ! This sounds wonderful! Regarding versions. Perhap you can have a control panel setting that selects the branch to publish by default. The ZMI could offer another mechanism that uses cookies to select the desired branch or version That is, a branch name or PREV, HEAD etc. But, I use jEdit and FTP for most of my Zope work, so would really like to have a way to access version/branches via FTP. Maybe that can be done by having pseudo subdirectories named after revisions or branches. So /myFolder/.Prev/index_html would do what you expect (note you can't save to old versions anyway ..) -- Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
G'Day, Well, step one is done ... I now have Zope + Ape using Subversion as it's filesystem !! This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem. Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support: - Use zope's username for SVN logging. - History/Undo support: View past revisions of an object, and revert to such a past revision. - Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?) In the longer term, there's great opportunity for: - Built-in conflict management and resolution: No more need for a SafetyBelt type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error. - Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort. - Leveraging SVN's property management. Content in SVN has properties, much like Zope does. I haven't looked at it yet, but I've noticed .properties file appearing ... I'm guessing those are the Zope properties, which would be better handled by subversion's property mechanism. And properties are versioned too ! In the realm of the wishful thinking, there's even more: Right now, HEAD (Latest/youngest revision) is always used and worked with. The really powerful feature I want to eventually get to is publsihing something of a given revision, while editing another. One potential paradigm for distiguishing between the two modes of operation could be to use anonymous vs. authenticated. This is not useful to everyone, but can be in certain circumstances, most notably where authenticated = authors/developpers and anonymous = normal users. This however requires ZMI interfaces, and in my case CMF ones as well ... This would be global though ... Eventually it'd be nice to have per object control of this stuff. Andy McKay says it can't be done, anybody care to contradict him ? :P I image I'd have to monkey pathc something DEEP in the Zope code base, but I find the mix-in class that's the commonn denominator ... why not ? Right now I've been working within the fs module of apelib. I'm going to split it off into a seperate one so that it's a clean merge with Ape's code base, in case someone wants to try it, or it eventually makes into Ape's default distribution ?? ;) Anyways, I'm just rambling by now ... Comments, thoughts and constructive criticism welcome ! Thanks, J.F. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Shane Hathaway Sent: April 8, 2004 11:20 AM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: [Zope-dev] Using a truely revision based storage for Zope ? [EMAIL PROTECTED] wrote: I've started looking at the ZODB and APE packages to try and get some understanding of how the whole storage interaction works, but it'll take me some time to figure it all out ... So I thought I'd get feedback on the idea first ... Sounds great! If I were you, I would start by replacing Ape/lib/apelib/fs/fileops.py with something that uses the Subversion module. It's a very simple module and you should be able to get pretty far that way. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
On Mon, 12 Apr 2004 [EMAIL PROTECTED] wrote: Well, step one is done ... I now have Zope + Ape using Subversion as it's filesystem !! That's fantastic! I'll write a more detailed reply soon. :-) Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
