RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-19 Thread Shane Hathaway
On Sun, 18 Apr 2004, Kapil Thangavelu wrote:

 part of the reason i never used ape as a means for svn integration was
 was that this imo, mix of high level application operations with zodb
 storage level operations never seemed a proper fit as it required
 bypassing the storage interface for richer semantic operations. ie i see
 reverting a revision, tagging a content tree, or diffing mulitiple
 revisions as application operations. still, there have been the some
 compelling ideas here about using zodb/ape as an interface.

FWIW, Ape is also designed to be useful as an import/export mechanism.  
In theory, if you set up a mapper configuration for a set of classes,
you've gained both a database bridge and a clean format for importing and
exporting objects.  I imagine that would benefit many kinds of 
applications.  In practice, I haven't tried it yet. :-)

 should we move this discussion to an ape specific mailing list?

I'm happy with zope-dev and zodb-dev, but if there's a good reason to make 
a new list, that's fine.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-19 Thread Kapil Thangavelu
On Mon, 2004-04-19 at 13:33, Shane Hathaway wrote:
 On Sun, 18 Apr 2004, Kapil Thangavelu wrote:

  should we move this discussion to an ape specific mailing list?
 
 I'm happy with zope-dev and zodb-dev, but if there's a good reason to make 
 a new list, that's fine.
 

its a knowledge discovery issue, there's a lot of useful info ape info
floating around on diseparate mailing lists which makes it hard to get a
hold of, having a single mailing focused on ape would make it much
easier for folks to get up to speed and see all the cool stuff that can
be done and that people are doing with ape ;-)

cheers,

-kapil



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-18 Thread Kapil Thangavelu
On Wed, 2004-04-14 at 10:04, [EMAIL PROTECTED] wrote:
 Kapil,
 
 Right now, the svn transactions are entirely contained within a single
 fileops operation: for example a mkdir connects to a transaction root,
 performs the necessary operations, and commits, all in one shot.
 

ok, thats what about what i expected. not ideal but it works.

 Last night I took some more time to try and learn more about Ape's
 functionning (Where events come from, which interfaces are used for what,
 and TPC), so I'm starting to understand more ...
 
 The more I learn, the more I think closer integration between SVN txn's and
 Ape's TPC would be a good place to start before looking at adding features
 like history support and so on: defining a model for what happens in svn for
 each TPC related call (connect, vote, finish), and then as Shane had said,
 look at IFSReader/IFSWriter (Which I now call
 ISubversionReader/ISubversionWriter :P) to match.

part of the reason i never used ape as a means for svn integration was
was that this imo, mix of high level application operations with zodb
storage level operations never seemed a proper fit as it required
bypassing the storage interface for richer semantic operations. ie i see
reverting a revision, tagging a content tree, or diffing mulitiple
revisions as application operations. still, there have been the some
compelling ideas here about using zodb/ape as an interface.

 
 Right now the fs implementation stores script commands that are cummulated
 upon connect() (I think?), validated as best as possible upon vote() and run
 upon finish().  I don't see why this couldn't be adapted to SVN txn's ...
 connect() = start a txn, vote() = validation (what this entails needs to be
 defined, could involve delta operations, revision number matching, etc ...
 ?), finish() = commit the svn txn.
 

not exactly, you just start a txn, do whatever work, and commit in vote.
see my previous message outlining txn integration difficulties, and why
this is not ideal but currently the best option.

 Because we're within an svn transaction, there would be no need for fs style
 script command accumulation however, which is nice.
 

sure.

should we move this discussion to an ape specific mailing list?

-kapil



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-18 Thread Kapil Thangavelu
its (cmfsvnbrowser) not what your looking for, if you want a versioning
system integrated with plone (as i recall your original request) i still
think working with zope version control (as i suggested earlier) is more
applicable in the short term.. as an integrated example of doing object
version control entirely in zope see,

http://occams.objectrealms.net

-kapil

On Thu, 2004-04-15 at 06:36, Arthur Chan Chi Chuen wrote:
 cool, let me try the cmfsvnbrowser first
 
 thanks. =)
 
 Arthur
 On Wed, 14 Apr 2004 10:35:14 -0400, Jean-Francois.Doyon wrote
  Well there you go, perfect :)
  
  -Original Message-
  From: Kapil Thangavelu [mailto:[EMAIL PROTECTED]
  Sent: April 14, 2004 6:49 AM
  To: [EMAIL PROTECTED]
  Cc: [EMAIL PROTECTED]
  Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
  r evis ion based storage for Zope ?)
  
  On Tue, 2004-04-13 at 20:53, [EMAIL PROTECTED] wrote:
   Hello,
   
   Hmmm, well it's as stable as Ape and Subversion are respectively :)
   
   I wouldn't call it stable no, it's something I did over the long week-end
  we
   just had, and that's about it :)
   
   Ape is at 0.8 and therefore becoming quite mature, I'd have to let others
   speak as to it stability however ...
   
   Subversion is also probably quite stable (It just reached 1.0), though I
   don't know how heavily tested it's been in a long running process (Might
  it
   have some memeory leaks ?) ...
  
  the svn apache server model is a long running process and is fairly
  stable on memory usage. the python bindings require doing some manual
  memory management, but the pool api makes it easy to deallocate
  arbitrary allocations safely. the cmfsvnbrowser code i ref'd early 
  has been in production by myself and others for almost a year and 
  runs with a stable memory footprint.
  
  -kapil
  


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-18 Thread Kapil Thangavelu
On Wed, 2004-04-14 at 19:14, Shane Hathaway wrote:
 On Wed, 14 Apr 2004 [EMAIL PROTECTED] wrote:
 
  My initial, uneducated thoughts on the topic were simplistic, but then I'm a
  big K.I.S.S. fan: simply pickle the entire object back and forth as one
  entity.  This means for each object, there is one file on the fs.  The
  benefit is greater simplicity ... the downside is that you couldn't check
  those files out of subversion and interact with them.  I also have to
  imagine the former is faster ?
 
 If you store pickles, you can't merge, you can't store properties, in fact
 you can't even store folders as directories or keep track of references
 between objects.  At that point, you'd do much better to just use
 FileStorage--KISS, after all.  Ape's real utility is in breaking objects
 apart for storage.  If you don't need that, you don't need Ape.  But I
 like to think it makes the job of breaking objects apart for storage
 relatively simple.
 
  But if usnig the latter, then I'd think that in SVN there would be only 1
  file, written by a specific mapper for a specific content type, say image.
  the data written to the file is such that if checked out, the file works,
  so I can open it in photoshop or something.
  
  I however, would simply put everything else in properties, if that makes
  sense at all. Zope properties, security information, and so on.  1 porperty,
  1 piece of data. No need for reading/writing multi-line text.
 
 That's fine, although you'll need to distinguish Zope properties from 
 other kinds of properties.  For example, if some object has a class_name 
 Zope property, you'd be in trouble if the system mixed that up with the 
 class_name key of the object classification.

i don't think needs to much worrying about, just prefix zope properties
with 'zope:' much the same way svn does with its properties
'svn:externals'

 
  The only thing I'm not so sure about is this remainder ... Though if it
  really is a base64 encoded string, there's no reason not to put that in a
  property as well.
 
 That's correct, it's always base64 encoded (with some comments.)
 
  The benefits I see:
  
  - The filesystem reflects the object structure, no extra files lying around.
 
 Definitely.
 
  - a more consistent way to setup mappers/gateways: there's data, and
  there's everything else. Have a common/defined/smart way of handling
  everything else.
 
 I would recommend you defer this for a while.  Lots objects don't
 easily fit this model.  For example, what is the data of a CMF tool?  
 We can make plenty of progress on Subversion without thinking about 
 changing the way mappers are defined.
 
  - with the use of the properties related svn funtions, we better leverage
  SVN's features.
 
 Yup.
 

another thing to keep in mind here, svn is basically a versioned fs, and
in addition to node/content history tracking and it has
facilities/functions for apply deltas to content, and recieving delta
diffs back etc, but these apply to node/content bodies and not to
properties. ie. there are no facilties for diffing properties, and this
would need to be implemented in python.

-kapil


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-18 Thread Kapil Thangavelu
On Wed, 2004-04-14 at 10:44, [EMAIL PROTECTED] wrote:
 The property schema thing is a good point, though I'm not sure we could ever
 do anything about it, not with the purpose to help naive gui clients work
 better with the repository.

agreed, its not something to worry about, and such naive clients are imo
broken.

 
 By nature the object structure (Class instance) is not fixed, so the
 amount/name/data of properties could vary arbitrarily.  This is after all
 the major benefit of an OODBMS.
 
 Fact is if one wanted a client to interact with a svn repository that stores
 zope objects, it would need to be fairly specifically designed for it ...
 How would a client (Say dreamweavnr with a subversion plug-in) know that
 when editing an image, the x and y size properties need to be updated ? What
 if someone monkey patches the image type to add some other
 property/attribute ?
 

well i think its also useful to think of svn as a sychronization point,
like an rdbms. for other processes in other languages to interact with
the content, like doing rich transforms/classification based on the
content and storing as properties. using proper names for properties i
think avoids the issue of dealing with client overlap, i would treat
properties as though they were in xml namespaces, ie serialize
dublincore author to dc:author, zope localroles - zsecurity:localroles,
etc. 

also its possible for zope to have total awareness of content changes
done by external clients ( i depend on this functionality actually to
reindex, get notification of deleted, apply wf to added content, etc.)
but all of this happens at an application level not the zodb layer.

 All we could hope to do is find a consistent and predictable way to handle
 properties, so that clients would know at the very least how to read/write
 them, but not necessarily know which one does what.  I suppose there's
 possible ways around that actually, by providing some way to describe
 properties and what they do and so on ... but that's beyond the scope of
 what I'm trying to do I think ...
 

it is beyond scope, at min it requires a schema set for commonly defined
properties which beyond the zope core properties are application
specific. i wouldn't worry about to well into the future.

-kapil

 J.F.
 
 -Original Message-
 From: Kapil Thangavelu [mailto:[EMAIL PROTECTED]
 Sent: April 14, 2004 6:51 AM
 To: Shane Hathaway
 Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
 Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
 r evis ion based storage for Zope ?)
 
 
 On Tue, 2004-04-13 at 22:46, Shane Hathaway wrote:
  On Tue, 13 Apr 2004 [EMAIL PROTECTED] wrote:
  
   Thanks for the extra tips, I'll check out those interfaces! I'm also
 getting
   up to speed on the whole mapper concept, where the work regarding
 properties
   handling seems to be ?
  
  Ape supports both annotations and Zope properties.  Annotations are blocks
  of multi-line text, while Zope properties are less constrained.  On the
  filesystem, Ape stores many Zope properties together in a single
  annotation called properties.  Other annotations include object
  classification, the remainder pickle (encoded in base 64), and security
  information.
  
  What are the expectations and limits of Subversion properties?  If they
  are blocks of multi-line text, you can store the annotations as Subversion
  properties.  If not, we'll still need a .properties file.  (Note that 
  the name .properties is thus confusing.  Oops.)
  
 
 svn supports aribtrary property values. ie you can store binary
 (remainder pickles, object classification, etc.) of arbitrary size on
 them. there is some question of how well some naive gui client will do
 when trying to list such properties, as there is currently no notion of
 metaproperties/property schema beyond the property namespace for
 introspection of a property.
 
 -kapil



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-18 Thread Kapil Thangavelu
On Thu, 2004-04-15 at 00:16, [EMAIL PROTECTED] wrote:
 Oh, a very good idea indeed! We'll have to look at that eventually.
 
 The mechanism you describe is preferable, but it should be noted that
 subversion properties are easily accessible using the clients.
 
 So long as said properties are human readable/writable, that's also an
 option.
 
 The main issue for me (And many others I suppose), is the fact that every
 single object type I use is either completely custom, or a monkey patched
 version of a built-in type (Notably OFS.Image for instance).
 
 So there would have to be a reliable serializer for other objects, which I
 beleive you already have, so that's OK :)


i dunno if you use or have tried archetypes, but it has experimental
support for generating the needed ape structures from its schema.. see 
archgenxml and archetypes/apesupport.. its pretty cool actually you cane
go from a uml model straight to a content model with support.

http://cvs.sourceforge.net/viewcvs.py/archetypes/Archetypes/ApeSupport.py?rev=1.6view=auto

cheers,

-kapil


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-17 Thread Kapil Thangavelu
On Wed, 2004-04-14 at 19:28, Shane Hathaway wrote:

 In Subversion, is it possible to read/write a versioned object by unique
 identifier rather than by path?  If so, it's a great blessing and we don't
 need a transaction script.

not really, yes every svn node has an a node id, but there is no way to
retrieve a particular node from the svn fs via node id. it is possible
to compare to compare and see if their related, (part of the node id is
a copy id which gets incremented when a node is copied).

in short not really.

-kapil


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-15 Thread Shane Hathaway
On Thu, 15 Apr 2004 [EMAIL PROTECTED] wrote:

 What happens is you write a new serializer, notably write a custom
 serializer for an object type you've already been using for a while, using
 the default one ? (Presumably because you WANT to be able to access the
 contents from other tools)
 
 I think I noticed that somewhere in the .properties file there was a note of
 the class used for serializing ... Is there a way for a serialized instance
 to remember which serializer was used ? Or does everything break when I
 change the definition of a serializer (Old instances can no longer be read)
 ?

Ape has a solution.

Every object is stored with a classification.  The classification (a
dictionary of strings, stored in whatever format is chosen by apeconf.xml)  
has at least two keys: class_name and mapper_name.  The mapper_name tells
Ape which mapper was used to store the object, enabling Ape to reliably
use the same mapper to load the object later.  The class_name is used by
mappers that work with multiple classes.

When Ape stores a new state for an object previously loaded, it disregards
the earlier choice of mapper.  So, for example, lets say you've made a
class called UltraPageTemplate, which subclasses ZopePageTemplate.  If you
store that using the default configuration, Ape won't try to guess how you
meant to store it and will mostly just pickle it.  The mapper_name will be
anyfile.

Later, you add to apeconf.xml a mapper that specifically serializes
UltraPageTemplate objects, using the mapper_name
Products.UltraPT.UltraPT.UltraPageTemplate.  Ape will continue to load
UltraPageTemplates stored using the anyfile mapper because the
mapper_name supercedes the class_name when loading.  However, when you
store new UltraPageTemplates or update existing ones, Ape will use your
new mapper, since the old choice of mapper is not consulted at save time.  
Therefore, to update all existing instances of UltraPageTemplate to use
your new mapper, you only need to get ZODB to load all the existing
instances and store them again unchanged.  If your new mapper is 
complete, the old pickles will go away and people will be able to edit 
UltraPageTemplates outside Zope.

Using strategies like this, Ape is trying to achieve a mode of development
that doesn't require you to decide how to store objects at the same time
you write classes.  You start with ZODB, a radically simple, unintrusive
storage layer, then you customize gradually with Ape.  I think Ape is 
making good progress toward that goal.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Kapil Thangavelu
On Tue, 2004-04-13 at 20:53, [EMAIL PROTECTED] wrote:
 Hello,
 
 Hmmm, well it's as stable as Ape and Subversion are respectively :)
 
 I wouldn't call it stable no, it's something I did over the long week-end we
 just had, and that's about it :)
 
 Ape is at 0.8 and therefore becoming quite mature, I'd have to let others
 speak as to it stability however ...
 
 Subversion is also probably quite stable (It just reached 1.0), though I
 don't know how heavily tested it's been in a long running process (Might it
 have some memeory leaks ?) ...

the svn apache server model is a long running process and is fairly
stable on memory usage. the python bindings require doing some manual
memory management, but the pool api makes it easy to deallocate
arbitrary allocations safely. the cmfsvnbrowser code i ref'd early has
been in production by myself and others for almost a year and runs with
a stable memory footprint.

-kapil


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Kapil Thangavelu
On Tue, 2004-04-13 at 22:46, Shane Hathaway wrote:
 On Tue, 13 Apr 2004 [EMAIL PROTECTED] wrote:
 
  Thanks for the extra tips, I'll check out those interfaces! I'm also getting
  up to speed on the whole mapper concept, where the work regarding properties
  handling seems to be ?
 
 Ape supports both annotations and Zope properties.  Annotations are blocks
 of multi-line text, while Zope properties are less constrained.  On the
 filesystem, Ape stores many Zope properties together in a single
 annotation called properties.  Other annotations include object
 classification, the remainder pickle (encoded in base 64), and security
 information.
 
 What are the expectations and limits of Subversion properties?  If they
 are blocks of multi-line text, you can store the annotations as Subversion
 properties.  If not, we'll still need a .properties file.  (Note that 
 the name .properties is thus confusing.  Oops.)
 

svn supports aribtrary property values. ie you can store binary
(remainder pickles, object classification, etc.) of arbitrary size on
them. there is some question of how well some naive gui client will do
when trying to list such properties, as there is currently no notion of
metaproperties/property schema beyond the property namespace for
introspection of a property.

-kapil


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Jean-Francois . Doyon
Kapil,

Right now, the svn transactions are entirely contained within a single
fileops operation: for example a mkdir connects to a transaction root,
performs the necessary operations, and commits, all in one shot.

Last night I took some more time to try and learn more about Ape's
functionning (Where events come from, which interfaces are used for what,
and TPC), so I'm starting to understand more ...

The more I learn, the more I think closer integration between SVN txn's and
Ape's TPC would be a good place to start before looking at adding features
like history support and so on: defining a model for what happens in svn for
each TPC related call (connect, vote, finish), and then as Shane had said,
look at IFSReader/IFSWriter (Which I now call
ISubversionReader/ISubversionWriter :P) to match.

Right now the fs implementation stores script commands that are cummulated
upon connect() (I think?), validated as best as possible upon vote() and run
upon finish().  I don't see why this couldn't be adapted to SVN txn's ...
connect() = start a txn, vote() = validation (what this entails needs to be
defined, could involve delta operations, revision number matching, etc ...
?), finish() = commit the svn txn.

Because we're within an svn transaction, there would be no need for fs style
script command accumulation however, which is nice.

J.F.



-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of Kapil Thangavelu
Sent: April 14, 2004 6:59 AM
To: Shane Hathaway
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
revis ion based storage for Zope ?)


On Tue, 2004-04-13 at 12:01, Shane Hathaway wrote:
 On Tue, 13 Apr 2004, Kapil Thangavelu wrote:
 
  since
  objects modified in a version are in essence locked from participating
  in other transactions, actions like modifying content in a version in a
  cmf site amounts to locking the catalog from changes outside of the
  version, which amounts to shutting down write activities to a cmf site.
 
 This is only true of FileStorage.  Some other storage could implement ZODB
 versions with merging capability rather than locking.
 

good point, just because hasn't been doesn't mean it can't ;-)

although i wonder if there is some hand waving in progress here that i
can't see. i guess my semantic notion of versions has been that of long
lived transactions, and is there a better means of thinking of them? how
do they play across with multiple mounted zodbs? what would something
like merge mean in the context of a catalog?

  i'm also curious how you dealt with svn transactions as part of the ape
  integration work to date.
 
 The same way it tries to impose transactions on the filesystem: in the 
 vote phase, Ape looks for possible problems and aborts early if it detects

 anything that will cause the transaction to fail.  Obviously, this 
 provides no guarantee, but covers many cases.
 

i was more curious how jean-francois was doing the svn ops in fileops,
as svn is fundamentally a transactional store (as opposed to the fs), ie
is there some record boundary notion of ape signalling the end of
serialization for an object set, or was each operation being conducted
in a separate svn transaction.

-kapil


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Jean-Francois . Doyon
Shane,

As Kapil already mentionned, svn's support for properties is quite flexible,
so that doesn't worry me too much.

I would imagine and administrator having the flexibility of choosing how
objects get translateds to files might be handy.

My initial, uneducated thoughts on the topic were simplistic, but then I'm a
big K.I.S.S. fan: simply pickle the entire object back and forth as one
entity.  This means for each object, there is one file on the fs.  The
benefit is greater simplicity ... the downside is that you couldn't check
those files out of subversion and interact with them.  I also have to
imagine the former is faster ?

But if usnig the latter, then I'd think that in SVN there would be only 1
file, written by a specific mapper for a specific content type, say image.
the data written to the file is such that if checked out, the file works,
so I can open it in photoshop or something.

I however, would simply put everything else in properties, if that makes
sense at all. Zope properties, security information, and so on.  1 porperty,
1 piece of data. No need for reading/writing multi-line text.

The only thing I'm not so sure about is this remainder ... Though if it
really is a base64 encoded string, there's no reason not to put that in a
property as well.

The benefits I see:

- The filesystem reflects the object structure, no extra files lying around.
- a more consistent way to setup mappers/gateways: there's data, and
there's everything else. Have a common/defined/smart way of handling
everything else.
- with the use of the properties related svn funtions, we better leverage
SVN's features.
- possibly better performance/simpler code by not needing the whole multiple
file management and read/write of text files.  Heck the whole
AnnotatedFileSystem, thing could be dropped/replaced with a
PropertyFileSystem implementation of some kind maybe ??

What do you think ?

J.F.

-Original Message-
From: Shane Hathaway [mailto:[EMAIL PROTECTED]
Sent: April 13, 2004 10:46 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
r evis ion based storage for Zope ?)


On Tue, 13 Apr 2004 [EMAIL PROTECTED] wrote:

 Thanks for the extra tips, I'll check out those interfaces! I'm also
getting
 up to speed on the whole mapper concept, where the work regarding
properties
 handling seems to be ?

Ape supports both annotations and Zope properties.  Annotations are blocks
of multi-line text, while Zope properties are less constrained.  On the
filesystem, Ape stores many Zope properties together in a single
annotation called properties.  Other annotations include object
classification, the remainder pickle (encoded in base 64), and security
information.

What are the expectations and limits of Subversion properties?  If they
are blocks of multi-line text, you can store the annotations as Subversion
properties.  If not, we'll still need a .properties file.  (Note that 
the name .properties is thus confusing.  Oops.)

 As for the seperation of code ... I now have a subversion directory as
a
 sibbling to fs ... I had to edit a couple of files outside that
directory,
 but still the seperation is nicer.
 
 eventually I'd have to make it into a product, is that doable ?

Why not add your code to Ape?  I think it would be easier for both of us
to maintain that way.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Jean-Francois . Doyon
Well there you go, perfect :)

-Original Message-
From: Kapil Thangavelu [mailto:[EMAIL PROTECTED]
Sent: April 14, 2004 6:49 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
r evis ion based storage for Zope ?)


On Tue, 2004-04-13 at 20:53, [EMAIL PROTECTED] wrote:
 Hello,
 
 Hmmm, well it's as stable as Ape and Subversion are respectively :)
 
 I wouldn't call it stable no, it's something I did over the long week-end
we
 just had, and that's about it :)
 
 Ape is at 0.8 and therefore becoming quite mature, I'd have to let others
 speak as to it stability however ...
 
 Subversion is also probably quite stable (It just reached 1.0), though I
 don't know how heavily tested it's been in a long running process (Might
it
 have some memeory leaks ?) ...

the svn apache server model is a long running process and is fairly
stable on memory usage. the python bindings require doing some manual
memory management, but the pool api makes it easy to deallocate
arbitrary allocations safely. the cmfsvnbrowser code i ref'd early has
been in production by myself and others for almost a year and runs with
a stable memory footprint.

-kapil

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Jean-Francois . Doyon
The property schema thing is a good point, though I'm not sure we could ever
do anything about it, not with the purpose to help naive gui clients work
better with the repository.

By nature the object structure (Class instance) is not fixed, so the
amount/name/data of properties could vary arbitrarily.  This is after all
the major benefit of an OODBMS.

Fact is if one wanted a client to interact with a svn repository that stores
zope objects, it would need to be fairly specifically designed for it ...
How would a client (Say dreamweaver with a subversion plug-in) know that
when editing an image, the x and y size properties need to be updated ? What
if someone monkey patches the image type to add some other
property/attribute ?

All we could hope to do is find a consistent and predictable way to handle
properties, so that clients would know at the very least how to read/write
them, but not necessarily know which one does what.  I suppose there's
possible ways around that actually, by providing some way to describe
properties and what they do and so on ... but that's beyond the scope of
what I'm trying to do I think ...

J.F.

-Original Message-
From: Kapil Thangavelu [mailto:[EMAIL PROTECTED]
Sent: April 14, 2004 6:51 AM
To: Shane Hathaway
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
r evis ion based storage for Zope ?)


On Tue, 2004-04-13 at 22:46, Shane Hathaway wrote:
 On Tue, 13 Apr 2004 [EMAIL PROTECTED] wrote:
 
  Thanks for the extra tips, I'll check out those interfaces! I'm also
getting
  up to speed on the whole mapper concept, where the work regarding
properties
  handling seems to be ?
 
 Ape supports both annotations and Zope properties.  Annotations are blocks
 of multi-line text, while Zope properties are less constrained.  On the
 filesystem, Ape stores many Zope properties together in a single
 annotation called properties.  Other annotations include object
 classification, the remainder pickle (encoded in base 64), and security
 information.
 
 What are the expectations and limits of Subversion properties?  If they
 are blocks of multi-line text, you can store the annotations as Subversion
 properties.  If not, we'll still need a .properties file.  (Note that 
 the name .properties is thus confusing.  Oops.)
 

svn supports aribtrary property values. ie you can store binary
(remainder pickles, object classification, etc.) of arbitrary size on
them. there is some question of how well some naive gui client will do
when trying to list such properties, as there is currently no notion of
metaproperties/property schema beyond the property namespace for
introspection of a property.

-kapil

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Shane Hathaway
On Wed, 14 Apr 2004 [EMAIL PROTECTED] wrote:

 My initial, uneducated thoughts on the topic were simplistic, but then I'm a
 big K.I.S.S. fan: simply pickle the entire object back and forth as one
 entity.  This means for each object, there is one file on the fs.  The
 benefit is greater simplicity ... the downside is that you couldn't check
 those files out of subversion and interact with them.  I also have to
 imagine the former is faster ?

If you store pickles, you can't merge, you can't store properties, in fact
you can't even store folders as directories or keep track of references
between objects.  At that point, you'd do much better to just use
FileStorage--KISS, after all.  Ape's real utility is in breaking objects
apart for storage.  If you don't need that, you don't need Ape.  But I
like to think it makes the job of breaking objects apart for storage
relatively simple.

 But if usnig the latter, then I'd think that in SVN there would be only 1
 file, written by a specific mapper for a specific content type, say image.
 the data written to the file is such that if checked out, the file works,
 so I can open it in photoshop or something.
 
 I however, would simply put everything else in properties, if that makes
 sense at all. Zope properties, security information, and so on.  1 porperty,
 1 piece of data. No need for reading/writing multi-line text.

That's fine, although you'll need to distinguish Zope properties from 
other kinds of properties.  For example, if some object has a class_name 
Zope property, you'd be in trouble if the system mixed that up with the 
class_name key of the object classification.

 The only thing I'm not so sure about is this remainder ... Though if it
 really is a base64 encoded string, there's no reason not to put that in a
 property as well.

That's correct, it's always base64 encoded (with some comments.)

 The benefits I see:
 
 - The filesystem reflects the object structure, no extra files lying around.

Definitely.

 - a more consistent way to setup mappers/gateways: there's data, and
 there's everything else. Have a common/defined/smart way of handling
 everything else.

I would recommend you defer this for a while.  Lots of objects don't
easily fit this model.  For example, what is the data of a CMF tool?  
We can make plenty of progress on Subversion without thinking about 
changing the way mappers are defined.

 - with the use of the properties related svn funtions, we better leverage
 SVN's features.

Yup.

 - possibly better performance/simpler code by not needing the whole multiple
 file management and read/write of text files.  Heck the whole
 AnnotatedFileSystem, thing could be dropped/replaced with a
 PropertyFileSystem implementation of some kind maybe ??

I'd guess there's no need for any such thing in Subversion.

I think your code might eventually share a lot of code with the fs
package, but if you'd like to start by making a clean break from the fs
package, by all means go ahead.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Shane Hathaway
On Wed, 14 Apr 2004 [EMAIL PROTECTED] wrote:

 Right now the fs implementation stores script commands that are cummulated
 upon connect() (I think?), validated as best as possible upon vote() and run
 upon finish().  I don't see why this couldn't be adapted to SVN txn's ...
 connect() = start a txn, vote() = validation (what this entails needs to be
 defined, could involve delta operations, revision number matching, etc ...
 ?), finish() = commit the svn txn.

The transaction script is necessary for exactly one thing: translation
between OIDs and paths.  I agonized over this for months, but translating
OIDs to paths simply requires complex machinery like this.  Also, there is
only one reason we have to translate OIDs to paths: opening files by inode
number is disallowed (and probably unreliable) in most operating systems.

In Subversion, is it possible to read/write a versioned object by unique
identifier rather than by path?  If so, it's a great blessing and we don't
need a transaction script.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Shane Hathaway
On Wed, 14 Apr 2004 [EMAIL PROTECTED] wrote:

 Fact is if one wanted a client to interact with a svn repository that stores
 zope objects, it would need to be fairly specifically designed for it ...
 How would a client (Say dreamweaver with a subversion plug-in) know that
 when editing an image, the x and y size properties need to be updated ? What
 if someone monkey patches the image type to add some other
 property/attribute ?

I would say Ape is quite good at accepting data from non-Zope-aware tools,
and I'd like to make it better.  The fact that Ape stores the image
dimensions separately from the image content is an implementation decision
in the serializer.  You could instead tell the serializer to detect image
dimensions when loading the object.  The detection is a relatively
inexpensive operation.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Arthur Chan Chi Chuen
cool, let me try the cmfsvnbrowser first

thanks. =)

Arthur
On Wed, 14 Apr 2004 10:35:14 -0400, Jean-Francois.Doyon wrote
 Well there you go, perfect :)
 
 -Original Message-
 From: Kapil Thangavelu [mailto:[EMAIL PROTECTED]
 Sent: April 14, 2004 6:49 AM
 To: [EMAIL PROTECTED]
 Cc: [EMAIL PROTECTED]
 Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
 r evis ion based storage for Zope ?)
 
 On Tue, 2004-04-13 at 20:53, [EMAIL PROTECTED] wrote:
  Hello,
  
  Hmmm, well it's as stable as Ape and Subversion are respectively :)
  
  I wouldn't call it stable no, it's something I did over the long week-end
 we
  just had, and that's about it :)
  
  Ape is at 0.8 and therefore becoming quite mature, I'd have to let others
  speak as to it stability however ...
  
  Subversion is also probably quite stable (It just reached 1.0), though I
  don't know how heavily tested it's been in a long running process (Might
 it
  have some memeory leaks ?) ...
 
 the svn apache server model is a long running process and is fairly
 stable on memory usage. the python bindings require doing some manual
 memory management, but the pool api makes it easy to deallocate
 arbitrary allocations safely. the cmfsvnbrowser code i ref'd early 
 has been in production by myself and others for almost a year and 
 runs with a stable memory footprint.
 
 -kapil
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - 
  http://mail.zope.org/mailman/listinfo/zope-announce
  http://mail.zope.org/mailman/listinfo/zope )


--
Open WebMail Project (http://openwebmail.org)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-14 Thread Jean-Francois . Doyon
Oh, a very good idea indeed! We'll have to look at that eventually.

The mechanism you describe is preferable, but it should be noted that
subversion properties are easily accessible using the clients.

So long as said properties are human readable/writable, that's also an
option.

The main issue for me (And many others I suppose), is the fact that every
single object type I use is either completely custom, or a monkey patched
version of a built-in type (Notably OFS.Image for instance).

So there would have to be a reliable serializer for other objects, which I
beleive you already have, so that's OK :)

What happens is you write a new serializer, notably write a custom
serializer for an object type you've already been using for a while, using
the default one ? (Presumably because you WANT to be able to access the
contents from other tools)

I think I noticed that somewhere in the .properties file there was a note of
the class used for serializing ... Is there a way for a serialized instance
to remember which serializer was used ? Or does everything break when I
change the definition of a serializer (Old instances can no longer be read)
?

J.F.

-Original Message-
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: 14/04/2004 7:39 PM
Subject: RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r
evis ion based storage for Zope ?)

On Wed, 14 Apr 2004 [EMAIL PROTECTED] wrote:

 Fact is if one wanted a client to interact with a svn repository that
stores
 zope objects, it would need to be fairly specifically designed for it
...
 How would a client (Say dreamweaver with a subversion plug-in) know
that
 when editing an image, the x and y size properties need to be updated
? What
 if someone monkey patches the image type to add some other
 property/attribute ?

I would say Ape is quite good at accepting data from non-Zope-aware
tools,
and I'd like to make it better.  The fact that Ape stores the image
dimensions separately from the image content is an implementation
decision
in the serializer.  You could instead tell the serializer to detect
image
dimensions when loading the object.  The detection is a relatively
inexpensive operation.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-13 Thread Jean-Francois . Doyon
Hello,

Hmmm, well it's as stable as Ape and Subversion are respectively :)

I wouldn't call it stable no, it's something I did over the long week-end we
just had, and that's about it :)

Ape is at 0.8 and therefore becoming quite mature, I'd have to let others
speak as to it stability however ...

Subversion is also probably quite stable (It just reached 1.0), though I
don't know how heavily tested it's been in a long running process (Might it
have some memeory leaks ?) ...

Basically, I wouldn't recommend for usage yet.  I'm not even distributing it
just yet, mostly simply because right now it's code is still intermingled
with Ape's ... As soon as I pull it out into a seperate product I'll be sure
ot post it on zope.org and notify the list, as I'm sure I'll be looking for
feedback/testers.

I would estimate that moment to be weeks away though ...

Thanks,
J.F.

-Original Message-
From: Arthur Chan Chi Chuen
To: Kapil Thangavelu; [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: 13/04/2004 2:13 PM
Subject: Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
revis ion based storage for Zope ?)

Hi all,

I've read your discussion about version control, it seems a cool thing
you 
guys making good progress. Btw, can I ask is the Ape using Subversion in
Zope 
stable? how able CMF stuff? I wanna make/find a document management
system 
which can provide some kinda version control in Plone.

Thanks
Arthur
On Tue, 13 Apr 2004 04:06:26 -0400, Kapil Thangavelu wrote
 On Mon, 2004-04-12 at 18:03, [EMAIL PROTECTED]
wrote:
  G'Day,
  
  Well, step one is done ... I now have Zope + Ape using Subversion as
it's
  filesystem !!
 
 
 cool!
 
  This is step one because, as Shawn suggested (Thanks for the
pointer, 
that's
  what I needed!), this simply means that Zope uses SVN purely as a
  filesystem.
 
 
  Because of subversion's nature, I want to look at 2 things beyond
this 
that
  traditional filesystems don't support:
  
  - Use zope's username for SVN logging.
 
 using AccessControl.getSecurityManager().getUser().getId() and 
 setting it up as a revision prop ( or directly when creating the 
 repo transaction) should do it.
 
  - History/Undo support: View past revisions of an object, and revert
to 
such
  a past revision.
 
 perhaps the code for this would be helpful, (remove image for code 
 link) 
http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevi
sions.
png
 
  - Zope Version support: SVN is fully transactional and atomic, this
should
  allow for support of Zope versions (I think ?)
 
 
 zope version support isn't really all that worthwhile, esp in a cmf
 context. in general zope's version support (or zodb more 
 particularly) is a database level feature masquerading as an 
 application one. since objects modified in a version are in essence 
 locked from participating in other transactions, actions like 
 modifying content in a version in a cmf site amounts to locking the 
 catalog from changes outside of the version, which amounts to 
 shutting down write activities to a cmf site. otoh, integration with 
 zope's transaction manager would be a good thing, although its 
 problematic to integrate between svn and zope txn models, more on 
 that in a moment.
 
  In the longer term, there's great opportunity for:
  
  - Built-in conflict management and resolution: No more need for a
  SafetyBelt type approach.  Right now I haven't looked at this at
all.  I
  plan to implement smart merging where possible (It might work
already
  actually, I just need to test it).  True conflicts (Where a merge
can't be
  accomplished withouth user interaction) would raise some sort of
conflict
  error.
 
 
 i don't know that conflict management is really useful in this
context.
 svn like zope relies on optimistic concurrency control, and currently
 doesn't support dav locking (which zope does). ie, it will just 
 throw an error if the content has been changed since the transaction 
 began. the 'normal' concurrency control of svn is better but 
 dependent on using the working copy (wc) layer, which is additional 
 programming and storage overhead. so at the layer of the svn client 
 this is already done and working and good, but integrating this 
 functionality into zope is a bit harder without wc overhard.
 
 this also makes the transaction integration becomes harder because
both
 zope and svn are using what amounts to optimistic concurrency control
 which makes it impossible afaics, to get real txn integration, ie in
 zope's two phase commit strategy, the last chance for a participant 
 to safely abort is tpc_vote, but there is no real way of knowing if 
 the svn txn will suceed or not until its tried. if it is tried at 
 this stage and succeeds then there is the possibility of a latter 
 txn participant failing the tpc_vote and the txn being aborted, and 
 if waits till tpc_finish (last part of two phase commit) and the svn 
 txn fails it can hose the 

RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-13 Thread Jean-Francois . Doyon
Brad,

About the branch thing ... That's basically the idea!

The bigger problem here is how to manage this both internally and from a
user/administrator perspective.

This paradigm only really makes sense in the CMF world anyways, and I want
to focus on basic Zope before moving up to the extra layer CMF represents.
I'm also worried getting something like this to work with CMF woudl require
an ungodly amount of monkey patching :P

The main things with CMF would be:

1) published type states vs. other states: how does one tell/define the
difference ? How does this information get passed to/from the publishing
system or the storage layer ?

Possible solution: Have the subversion/zope interface interact with
DCWorkflow.  When an object is changed, consult DCWorkflow settings and
determine what state the object is in.  If the state is deemed published,
copy the object into the publishing branch.

2) If we got this far, then I need to know when to use the branch vs. the
trunk.  From a user prespective this isn't too big a deal, a ZMI interface,
cookies, etc ... That's all ok ... The problem is that by the time the
request comes to the storage side of things, that information is most likely
gone (Don't know though, I'm still learning).  Then there's the whole
caching thing that would probably complicate things.

2) Message logging: CMF comments when doing a workflow action could go
straight into SVN's logging, again how this might be accomplished isn't
clear as of yet.

This is all for the future though, I'm nowhere near this yet!!

Also, I'm thinking the cooler features may just have to wait for Zope 3 ...
Implementing them in Zope might be too convoluted.

I should probably start a Wiki on this topic !

J.F.


-Original Message-
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: 13/04/2004 8:40 AM
Subject: Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
revis   ion based storage for Zope ?)

On 12 Apr 2004 at 18:03, [EMAIL PROTECTED] wrote:

 
 Anyways, I'm just rambling by now ... Comments, thoughts and
constructive
 criticism welcome !

This sounds wonderful!

Regarding versions. Perhap you can have a control panel setting that
selects the 
branch to publish by default. The ZMI could offer another mechanism
that uses 
cookies to select the desired branch or version  That is, a branch
name or PREV, 
HEAD etc.

But, I use jEdit and FTP for most of my Zope work, so would really like
to have a way to 
access version/branches via FTP. Maybe that can be done by having pseudo

subdirectories named after revisions or branches. So
/myFolder/.Prev/index_html 
would do what you expect (note you can't save to old versions anyway ..)




-- 
Brad Clements,[EMAIL PROTECTED]   (315)268-1000
http://www.murkworks.com  (315)268-9812 Fax
http://www.wecanstopspam.org/   AOL-IM: BKClements


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-13 Thread Jean-Francois . Doyon
Shane,

Thanks for the extra tips, I'll check out those interfaces! I'm also getting
up to speed on the whole mapper concept, where the work regarding properties
handling seems to be ?

I've done some reading, and I need to do some more, but I'll get there :)

As for the seperation of code ... I now have a subversion directory as  a
sibbling to fs ... I had to edit a couple of files outside that directory,
but still the seperation is nicer.

eventually I'd have to make it into a product, is that doable ?

J.F.

-Original Message-
From: Shane Hathaway
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Sent: 13/04/2004 11:43 AM
Subject: Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
revis ion based storage for Zope ?)

On Mon, 12 Apr 2004 [EMAIL PROTECTED] wrote:

 This is step one because, as Shawn suggested (Thanks for the pointer,
that's
 what I needed!), this simply means that Zope uses SVN purely as a
 filesystem.
 
 Because of subversion's nature, I want to look at 2 things beyond this
that
 traditional filesystems don't support:

To take this to the next level, you need to implement IFSReader and
IFSWriter instead of the more basic file operations interface.  See
Ape/lib/apelib/fs/interfaces.py.  In IFSReader/IFSWriter, you're a
little
closer to objects, and you work with OIDs instead of paths.  I expect
that
history and version support will require us to augment IFSReader or add
another interface.

 - Use zope's username for SVN logging.
 - History/Undo support: View past revisions of an object, and revert
to such
 a past revision.
 - Zope Version support: SVN is fully transactional and atomic, this
should
 allow for support of Zope versions (I think ?)

These seem doable.

 In the longer term, there's great opportunity for:
 
 - Built-in conflict management and resolution: No more need for a
 SafetyBelt type approach.  Right now I haven't looked at this at
all.  I
 plan to implement smart merging where possible (It might work already
 actually, I just need to test it).  True conflicts (Where a merge
can't be
 accomplished withouth user interaction) would raise some sort of
conflict
 error.

This could be complicated, because after the merge you have to be 
sure Zope caches the merged state rather than either of the intermediate

states.  The idea has potential, though!

 - Editing Zope content objects through interaction with the svn
repository.
 I can checkout the repository, edit some objects, and chek them back
in,
 never interacting with Zope directly ... I've already tried this !
Works
 great for text based content types such as PageTemplates or DTML
Documents
 and so on ... I even did it with a JPG, though because the properties
hold
 width and height, you get some weird looking pictures :) The concept
is
 valid though.  There may someday be a way to leverage this
functionality
 better with a purpose built client of some sort.

Sounds neat.  Fortunately, Ape doesn't have to be aware of this. :-)

 - Leveraging SVN's property management.  Content in SVN has
properties, much
 like Zope does.  I haven't looked at it yet, but I've noticed
.properties
 file appearing ... I'm guessing those are the Zope properties, which
would
 be better handled by subversion's property mechanism.  And properties
are
 versioned too !

I think you'll find this easy to do once you implement 
IFSReader/IFSWriter.

 In the realm of the wishful thinking, there's even more:
 
 Right now, HEAD (Latest/youngest revision) is always used and worked
with.
 The really powerful feature I want to eventually get to is publsihing
 something of a given revision, while editing another.  One potential
 paradigm for distiguishing between the two modes of operation could be
to
 use anonymous vs. authenticated.  This is not useful to everyone, but
can be
 in certain circumstances, most notably where authenticated =
 authors/developpers and anonymous = normal users.  This however
requires ZMI
 interfaces, and in my case CMF ones as well ... This would be global
though
 ... Eventually it'd be nice to have per object control of this stuff.
Andy
 McKay says it can't be done, anybody care to contradict him ? :P I
image I'd
 have to monkey pathc something DEEP in the Zope code base, but I find
the
 mix-in class that's the commonn denominator ... why not ?

Well, CMFStaging is designed to address this need.  To do this at the 
database level, you could use versions.

 Right now I've been working within the fs module of apelib.  I'm
going to
 split it off into a seperate one so that it's a clean merge with Ape's
code
 base, in case someone wants to try it, or it eventually makes into
Ape's
 default distribution ?? ;)

You can add a new module, perhaps 'svnops.py', to the 'fs' package.
Then 
we should make the choice between fileops and svnops configurable.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML 

RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-13 Thread Arthur Chan Chi Chuen
Hi J.F,

Thanks for your comments first. I understand more about the process of SVN in 
zope now. I am very eager to have/make a document management system with good 
version ctrl in Plone. Some products like Silva has versions but it's just 
not exactly what we need.

Wish you have a good progress, and let me know if I can help you in the dev. 
or testing pharse.

Arthur

On Tue, 13 Apr 2004 20:53:30 -0400, Jean-Francois.Doyon wrote
 Hello,
 
 Hmmm, well it's as stable as Ape and Subversion are respectively :)
 
 I wouldn't call it stable no, it's something I did over the long 
 week-end we just had, and that's about it :)
 
 Ape is at 0.8 and therefore becoming quite mature, I'd have to let others
 speak as to it stability however ...
 
 Subversion is also probably quite stable (It just reached 1.0),
  though I don't know how heavily tested it's been in a long running 
 process (Might it have some memeory leaks ?) ...
 
 Basically, I wouldn't recommend for usage yet.  I'm not even 
 distributing it just yet, mostly simply because right now it's code 
 is still intermingled with Ape's ... As soon as I pull it out into a 
 seperate product I'll be sure ot post it on zope.org and notify the 
 list, as I'm sure I'll be looking for feedback/testers.
 
 I would estimate that moment to be weeks away though ...
 
 Thanks,
 J.F.
 
 -Original Message-
 From: Arthur Chan Chi Chuen
 To: Kapil Thangavelu; [EMAIL PROTECTED]
 Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
 Sent: 13/04/2004 2:13 PM
 Subject: Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
 revis ion based storage for Zope ?)
 
 Hi all,
 
 I've read your discussion about version control, it seems a cool 
 thing you guys making good progress. Btw, can I ask is the Ape using 
 Subversion in Zope stable? how able CMF stuff? I wanna make/find a 
 document management system which can provide some kinda version 
 control in Plone.
 
 Thanks
 Arthur
 On Tue, 13 Apr 2004 04:06:26 -0400, Kapil Thangavelu wrote
  On Mon, 2004-04-12 at 18:03, [EMAIL PROTECTED]
 wrote:
   G'Day,
   
   Well, step one is done ... I now have Zope + Ape using Subversion as
 it's
   filesystem !!
  
  
  cool!
  
   This is step one because, as Shawn suggested (Thanks for the
 pointer, 
 that's
   what I needed!), this simply means that Zope uses SVN purely as a
   filesystem.
  
  
   Because of subversion's nature, I want to look at 2 things beyond
 this 
 that
   traditional filesystems don't support:
   
   - Use zope's username for SVN logging.
  
  using AccessControl.getSecurityManager().getUser().getId() and 
  setting it up as a revision prop ( or directly when creating the 
  repo transaction) should do it.
  
   - History/Undo support: View past revisions of an object, and revert
 to 
 such
   a past revision.
  
  perhaps the code for this would be helpful, (remove image for code 
  link) 
 http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevi
 sions.
 png
  
   - Zope Version support: SVN is fully transactional and atomic, this
 should
   allow for support of Zope versions (I think ?)
  
  
  zope version support isn't really all that worthwhile, esp in a cmf
  context. in general zope's version support (or zodb more 
  particularly) is a database level feature masquerading as an 
  application one. since objects modified in a version are in essence 
  locked from participating in other transactions, actions like 
  modifying content in a version in a cmf site amounts to locking the 
  catalog from changes outside of the version, which amounts to 
  shutting down write activities to a cmf site. otoh, integration with 
  zope's transaction manager would be a good thing, although its 
  problematic to integrate between svn and zope txn models, more on 
  that in a moment.
  
   In the longer term, there's great opportunity for:
   
   - Built-in conflict management and resolution: No more need for a
   SafetyBelt type approach.  Right now I haven't looked at this at
 all.  I
   plan to implement smart merging where possible (It might work
 already
   actually, I just need to test it).  True conflicts (Where a merge
 can't be
   accomplished withouth user interaction) would raise some sort of
 conflict
   error.
  
  
  i don't know that conflict management is really useful in this
 context.
  svn like zope relies on optimistic concurrency control, and currently
  doesn't support dav locking (which zope does). ie, it will just 
  throw an error if the content has been changed since the transaction 
  began. the 'normal' concurrency control of svn is better but 
  dependent on using the working copy (wc) layer, which is additional 
  programming and storage overhead. so at the layer of the svn client 
  this is already done and working and good, but integrating this 
  functionality into zope is a bit harder without wc overhard.
  
  this also makes the transaction integration becomes harder because
 both
  zope and svn 

Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-13 Thread robert rottermann
[EMAIL PROTECTED] wrote:

Brad,

About the branch thing ... That's basically the idea!

The bigger problem here is how to manage this both internally and from a
user/administrator perspective.
This paradigm only really makes sense in the CMF world anyways, and I want
to focus on basic Zope before moving up to the extra layer CMF represents.
I'm also worried getting something like this to work with CMF woudl require
an ungodly amount of monkey patching :P
The main things with CMF would be:

1) published type states vs. other states: how does one tell/define the
difference ? How does this information get passed to/from the publishing
system or the storage layer ?
Possible solution: Have the subversion/zope interface interact with
DCWorkflow.  When an object is changed, consult DCWorkflow settings and
determine what state the object is in.  If the state is deemed published,
copy the object into the publishing branch.
2) If we got this far, then I need to know when to use the branch vs. the
trunk.  From a user prespective this isn't too big a deal, a ZMI interface,
cookies, etc ... That's all ok ... The problem is that by the time the
request comes to the storage side of things, that information is most likely
gone (Don't know though, I'm still learning).  Then there's the whole
caching thing that would probably complicate things.
2) Message logging: CMF comments when doing a workflow action could go
straight into SVN's logging, again how this might be accomplished isn't
clear as of yet.
This is all for the future though, I'm nowhere near this yet!!

Also, I'm thinking the cooler features may just have to wait for Zope 3 ...
Implementing them in Zope might be too convoluted.
I should probably start a Wiki on this topic !

J.F.

-Original Message-
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Sent: 13/04/2004 8:40 AM
Subject: Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely
revis   ion based storage for Zope ?)
On 12 Apr 2004 at 18:03, [EMAIL PROTECTED] wrote:

 

Anyways, I'm just rambling by now ... Comments, thoughts and
   

constructive
 

criticism welcome !
   

This sounds wonderful!

Regarding versions. Perhap you can have a control panel setting that
selects the 
branch to publish by default. The ZMI could offer another mechanism
that uses 
cookies to select the desired branch or version  That is, a branch
name or PREV, 
HEAD etc.

But, I use jEdit and FTP for most of my Zope work, so would really like
to have a way to 
access version/branches via FTP. Maybe that can be done by having pseudo

subdirectories named after revisions or branches. So
/myFolder/.Prev/index_html 
would do what you expect (note you can't save to old versions anyway ..)



 

Fran├žois,

we did a versioning Product here at redCOR which is about to go into 
production. It is served trough DCWorkflow state changes.
It stores a series of objects per versioned object. One for each 
version. No diff's or such.
I would be very much interested to adapt it such that it uses an APE/svn 
augmented storage for it.

Robert

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)

2004-04-13 Thread Shane Hathaway
On Tue, 13 Apr 2004 [EMAIL PROTECTED] wrote:

 Thanks for the extra tips, I'll check out those interfaces! I'm also getting
 up to speed on the whole mapper concept, where the work regarding properties
 handling seems to be ?

Ape supports both annotations and Zope properties.  Annotations are blocks
of multi-line text, while Zope properties are less constrained.  On the
filesystem, Ape stores many Zope properties together in a single
annotation called properties.  Other annotations include object
classification, the remainder pickle (encoded in base 64), and security
information.

What are the expectations and limits of Subversion properties?  If they
are blocks of multi-line text, you can store the annotations as Subversion
properties.  If not, we'll still need a .properties file.  (Note that 
the name .properties is thus confusing.  Oops.)

 As for the seperation of code ... I now have a subversion directory as  a
 sibbling to fs ... I had to edit a couple of files outside that directory,
 but still the seperation is nicer.
 
 eventually I'd have to make it into a product, is that doable ?

Why not add your code to Ape?  I think it would be easier for both of us
to maintain that way.

Shane

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )