[Zope-dev] RFC: A proposal for ZODB schema migration in Zope 3
I've posted a proposal: http://dev.zope.org/Zope3/DatabaseGenerations for a framework for managing the migration of application/database schemas in ZODB databases in an orderly manner. The proposal is to apply this new framework in Zope 3, however, if it works well, it would be applicable to Zope 2 and other ZODB applications. Comments are welcome. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope3-dev] RFC: A proposal for ZODB schema migration in Zope 3
I am keen on such functionality. I will be working on something related to this in the near future to support a customer. I would be interested in implementing something like this for Zope 2 as a result. I had planned on implementing it as a completely external kind of thing, but maybe some support from Zope itself would be useful. There is another practical problem that is logically related to this one which has the potential to be solved by the same machinery. I'm not sure we want to conflate the problems, but I mention it here just because it's possible that we do. The "upgrade problem" isn't always limited to the updating the schema of database objects. Sometimes an upgrade requires a mass update of "already-schema-current" objects. For example, during an upgrade, you might want to reset local role values across a number of objects in Zope 2. Or you might want to change a data value across a number of objects. Or whatever. This is a common problem in production, and usually it's solved by writing one-off scripts that connect to the database and do recursion and a commit. There's nothing in your proposal that implies that the proposed machinery couldn't be used to perform these kinds of mass-update duties except the names "schema" (and maybe "generation"). So immediately the only thing I suggest (if we want to conflate the two problems) is to change the name "schema" to "state" and the name "generation" to "version". If we don't want to conflate the two problems, do you have a suggestion as to how to more uniformly deal with the data-value-update problem? (BTW, it's very hard to follow threads following a message that is cross-posted to a number of lists. I've made this mistake a number of times in the past, and it's almost never worth doing as invariably I miss something or need to reread every followup on every list). On Sat, 2004-04-17 at 08:14, Jim Fulton wrote: > I've posted a proposal: > >http://dev.zope.org/Zope3/DatabaseGenerations > > for a framework for managing the migration of application/database schemas in ZODB > databases in an orderly manner. The proposal is to apply this new framework in Zope > 3, > however, if it works well, it would be applicable to Zope 2 and other ZODB > applications. > > Comments are welcome. > > Jim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)
On Wed, 2004-04-14 at 19:28, Shane Hathaway wrote: > In Subversion, is it possible to read/write a versioned object by unique > identifier rather than by path? If so, it's a great blessing and we don't > need a transaction script. not really, yes every svn node has an a node id, but there is no way to retrieve a particular node from the svn fs via node id. it is possible to compare to compare and see if their related, (part of the node id is a copy id which gets incremented when a node is copied). in short not really. -kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)
On Wed, 2004-04-14 at 10:04, [EMAIL PROTECTED] wrote: > Kapil, > > Right now, the svn transactions are entirely contained within a single > fileops operation: for example a "mkdir" connects to a transaction root, > performs the necessary operations, and commits, all in one shot. > ok, thats what about what i expected. not ideal but it works. > Last night I took some more time to try and learn more about Ape's > functionning (Where events come from, which interfaces are used for what, > and TPC), so I'm starting to understand more ... > > The more I learn, the more I think closer integration between SVN txn's and > Ape's TPC would be a good place to start before looking at adding features > like history support and so on: defining a model for what happens in svn for > each TPC related call (connect, vote, finish), and then as Shane had said, > look at IFSReader/IFSWriter (Which I now call > ISubversionReader/ISubversionWriter :P) to match. part of the reason i never used ape as a means for svn integration was was that this imo, mix of high level application operations with zodb storage level operations never seemed a proper fit as it required bypassing the storage interface for richer semantic operations. ie i see reverting a revision, tagging a content tree, or diffing mulitiple revisions as application operations. still, there have been the some compelling ideas here about using zodb/ape as an interface. > > Right now the fs implementation stores "script commands" that are cummulated > upon connect() (I think?), validated as best as possible upon vote() and run > upon finish(). I don't see why this couldn't be adapted to SVN txn's ... > connect() = start a txn, vote() = validation (what this entails needs to be > defined, could involve delta operations, revision number matching, etc ... > ?), finish() = commit the svn txn. > not exactly, you just start a txn, do whatever work, and commit in vote. see my previous message outlining txn integration difficulties, and why this is not ideal but currently the best option. > Because we're within an svn transaction, there would be no need for fs style > script command accumulation however, which is nice. > sure. should we move this discussion to an ape specific mailing list? -kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)
its (cmfsvnbrowser) not what your looking for, if you want a versioning system integrated with plone (as i recall your original request) i still think working with zope version control (as i suggested earlier) is more applicable in the short term.. as an integrated example of doing object version control entirely in zope see, http://occams.objectrealms.net -kapil On Thu, 2004-04-15 at 06:36, Arthur Chan Chi Chuen wrote: > cool, let me try the cmfsvnbrowser first > > thanks. =) > > Arthur > On Wed, 14 Apr 2004 10:35:14 -0400, Jean-Francois.Doyon wrote > > Well there you go, perfect :) > > > > -Original Message- > > From: Kapil Thangavelu [mailto:[EMAIL PROTECTED] > > Sent: April 14, 2004 6:49 AM > > To: [EMAIL PROTECTED] > > Cc: [EMAIL PROTECTED] > > Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely > > r evis ion based storage for Zope ?) > > > > On Tue, 2004-04-13 at 20:53, [EMAIL PROTECTED] wrote: > > > Hello, > > > > > > Hmmm, well it's as stable as Ape and Subversion are respectively :) > > > > > > I wouldn't call it stable no, it's something I did over the long week-end > > we > > > just had, and that's about it :) > > > > > > Ape is at 0.8 and therefore becoming quite mature, I'd have to let others > > > speak as to it stability however ... > > > > > > Subversion is also probably quite stable (It just reached 1.0), though I > > > don't know how heavily tested it's been in a long running process (Might > > it > > > have some memeory leaks ?) ... > > > > the svn apache server model is a long running process and is fairly > > stable on memory usage. the python bindings require doing some manual > > memory management, but the pool api makes it easy to deallocate > > arbitrary allocations safely. the cmfsvnbrowser code i ref'd early > > has been in production by myself and others for almost a year and > > runs with a stable memory footprint. > > > > -kapil > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)
On Wed, 2004-04-14 at 19:14, Shane Hathaway wrote: > On Wed, 14 Apr 2004 [EMAIL PROTECTED] wrote: > > > My initial, uneducated thoughts on the topic were simplistic, but then I'm a > > big K.I.S.S. fan: simply pickle the entire object back and forth as one > > entity. This means for each object, there is one file on the fs. The > > benefit is greater simplicity ... the downside is that you couldn't check > > those files out of subversion and interact with them. I also have to > > imagine the former is faster ? > > If you store pickles, you can't merge, you can't store properties, in fact > you can't even store folders as directories or keep track of references > between objects. At that point, you'd do much better to just use > FileStorage--KISS, after all. Ape's real utility is in breaking objects > apart for storage. If you don't need that, you don't need Ape. But I > like to think it makes the job of breaking objects apart for storage > relatively simple. > > > But if usnig the latter, then I'd think that in SVN there would be only 1 > > file, written by a specific mapper for a specific content type, say image. > > the data written to the file is such that if checked out, the file "works", > > so I can open it in photoshop or something. > > > > I however, would simply put everything else in properties, if that makes > > sense at all. Zope properties, security information, and so on. 1 porperty, > > 1 piece of data. No need for reading/writing multi-line text. > > That's fine, although you'll need to distinguish Zope properties from > other kinds of properties. For example, if some object has a class_name > Zope property, you'd be in trouble if the system mixed that up with the > class_name key of the object classification. i don't think needs to much worrying about, just prefix zope properties with 'zope:' much the same way svn does with its properties 'svn:externals' > > > The only thing I'm not so sure about is this "remainder" ... Though if it > > really is a base64 encoded string, there's no reason not to put that in a > > property as well. > > That's correct, it's always base64 encoded (with some comments.) > > > The benefits I see: > > > > - The filesystem reflects the object structure, no extra files lying around. > > Definitely. > > > - a more consistent way to setup mappers/gateways: there's "data", and > > there's everything else. Have a common/defined/smart way of handling > > "everything else". > > I would recommend you defer this for a while. Lots objects don't > easily fit this model. For example, what is the "data" of a CMF tool? > We can make plenty of progress on Subversion without thinking about > changing the way mappers are defined. > > > - with the use of the properties related svn funtions, we better leverage > > SVN's features. > > Yup. > another thing to keep in mind here, svn is basically a versioned fs, and in addition to node/content history tracking and it has facilities/functions for apply deltas to content, and recieving delta diffs back etc, but these apply to node/content bodies and not to properties. ie. there are no facilties for diffing properties, and this would need to be implemented in python. -kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)
On Wed, 2004-04-14 at 10:44, [EMAIL PROTECTED] wrote: > The property schema thing is a good point, though I'm not sure we could ever > do anything about it, not with the purpose to help "naive gui" clients work > better with the repository. agreed, its not something to worry about, and such naive clients are imo broken. > > By nature the object structure (Class instance) is not fixed, so the > amount/name/data of properties could vary arbitrarily. This is after all > the major benefit of an OODBMS. > > Fact is if one wanted a client to interact with a svn repository that stores > zope objects, it would need to be fairly specifically designed for it ... > How would a client (Say dreamweavnr with a subversion plug-in) know that > when editing an image, the x and y size properties need to be updated ? What > if someone monkey patches the image type to add some other > property/attribute ? > well i think its also useful to think of svn as a sychronization point, like an rdbms. for other processes in other languages to interact with the content, like doing rich transforms/classification based on the content and storing as properties. using proper names for properties i think avoids the issue of dealing with client overlap, i would treat properties as though they were in xml namespaces, ie serialize dublincore author to dc:author, zope localroles -> zsecurity:localroles, etc. also its possible for zope to have total awareness of content changes done by external clients ( i depend on this functionality actually to reindex, get notification of deleted, apply wf to added content, etc.) but all of this happens at an application level not the zodb layer. > All we could hope to do is find a consistent and predictable way to handle > properties, so that clients would know at the very least how to read/write > them, but not necessarily know which one does what. I suppose there's > possible ways around that actually, by providing some way to describe > properties and what they do and so on ... but that's beyond the scope of > what I'm trying to do I think ... > it is beyond scope, at min it requires a schema set for commonly defined properties which beyond the zope core properties are application specific. i wouldn't worry about to well into the future. -kapil > J.F. > > -Original Message- > From: Kapil Thangavelu [mailto:[EMAIL PROTECTED] > Sent: April 14, 2004 6:51 AM > To: Shane Hathaway > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely > r evis ion based storage for Zope ?) > > > On Tue, 2004-04-13 at 22:46, Shane Hathaway wrote: > > On Tue, 13 Apr 2004 [EMAIL PROTECTED] wrote: > > > > > Thanks for the extra tips, I'll check out those interfaces! I'm also > getting > > > up to speed on the whole mapper concept, where the work regarding > properties > > > handling seems to be ? > > > > Ape supports both annotations and Zope properties. Annotations are blocks > > of multi-line text, while Zope properties are less constrained. On the > > filesystem, Ape stores many Zope properties together in a single > > annotation called "properties". Other annotations include object > > classification, the remainder pickle (encoded in base 64), and security > > information. > > > > What are the expectations and limits of Subversion properties? If they > > are blocks of multi-line text, you can store the annotations as Subversion > > properties. If not, we'll still need a .properties file. (Note that > > the name ".properties" is thus confusing. Oops.) > > > > svn supports aribtrary property values. ie you can store binary > (remainder pickles, object classification, etc.) of arbitrary size on > them. there is some question of how well some naive gui client will do > when trying to list such properties, as there is currently no notion of > metaproperties/property schema beyond the property namespace for > introspection of a property. > > -kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)
On Thu, 2004-04-15 at 00:16, [EMAIL PROTECTED] wrote: > Oh, a very good idea indeed! We'll have to look at that eventually. > > The mechanism you describe is preferable, but it should be noted that > subversion properties are easily accessible using the clients. > > So long as said properties are human readable/writable, that's also an > option. > > The main issue for me (And many others I suppose), is the fact that every > single object type I use is either completely custom, or a monkey patched > version of a built-in type (Notably OFS.Image for instance). > > So there would have to be a reliable serializer for "other" objects, which I > beleive you already have, so that's OK :) i dunno if you use or have tried archetypes, but it has experimental support for generating the needed ape structures from its schema.. see archgenxml and archetypes/apesupport.. its pretty cool actually you cane go from a uml model straight to a content model with support. http://cvs.sourceforge.net/viewcvs.py/archetypes/Archetypes/ApeSupport.py?rev=1.6&view=auto cheers, -kapil ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
