Re: [Zope-dev] Zope hangs, with processes being idle ... (rather urgent problem)
I have a Zope 2.5.1 using ZEO on Linux, with Apache as proxy server. The two ZEO clients run fine for a couple of hours, and then suddenly one of them stops responding. It's the type of Zope hangs when all processes are fine, but ZServer just doesn't respond (load is very low, with the CPU 90% idle, so it is not a performance issue or recursion problem). There is no error before this situation happens. The processes just don't respond any more. The second ZEO keeps running, but that machine also hangs from time to time, so I guess ZEO is not the problem, and the hardware or any Linux-specific problems aren't either. When the ZServer hangs, I can not access it directly, so it is no problem with the Apache. You see, I don't have any clue on what is happening here. The servers worked fine for months, and I don't know what specific change could have caused the problem. How can such a situation be debugged? Dieter Maurer wrote: Maybe, you use the -M logging and analyse the log with Chris' requestprofiler. You may see unfinished requests. Thanks! That's what I did. I think I found the problem: I have some Objects that inherit from Image, and their data attribute (that's where the image data is held) wasn't initialized correctly in some cases. So whenever somebody tried to display one of those objects, Zope started streaming back the image data, which made the request hang when the data attribute was broken. Cheers Joachim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Removing the acquisition wrapper from an object (Python script)
Hi, aq_base(someObject) doesn't work in a python script. I need to get an object without its acquisition wrapper. Any hint ? Many thanks in advance --Gilles ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Refer to zope attribute or object in source
A lot of people would like to see these things cleaned up, but unfortunately unless it becomes a pressing need for someone it doesn't get done (for lack of time). If you have some experience with that, and have ideas on how to fix things (and check that they work ok), please do write some patches. Many of us will be grateful to you. Florent In article [EMAIL PROTECTED] you write: In PropertySheets.py of zope/lib/python/OFS def xml_escape(v): . return unicode(v, latin-1).encode(utf-8) Are there anyway to substitute the latin-1 with an attribute of the zope site's attribute say: zope-charset which a site maintainer can modify through the web? I raise this question because the current-cvs version of zope is turned into unicode, but all the source everywhere is assumed that the data input is latin-1 but I am very sure that it will not always the case. Some sites simply don't work in that assumption, turning many sites using zope stop upgrade or stop using zope. This will seriously hurt the zope community. The initial act to make zope understand unicode is to help zope internationalize, but the actual effect was not. I urge the zope developer, the zope coporation as well as product developers (some products like ieDocument and worldPilot have 8859-1 specified in their source) alert on this issue, and take action before it is too late. The best solution I could come up with is to allow individual site manager to choose its default charset and allow other site-branch manager to overwrite it with acquisition, thus the attribute zope-charset will be used to determine both the Content-type: text/plain; charset=dtml-var zope-charset and the code convertion of Zope like PropertySheets.py -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Removing the acquisition wrapper from an object (Python script)
- Original Message - From: Toby Dickenson [EMAIL PROTECTED] To: Gilles Lenfant [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Thursday, August 01, 2002 1:46 PM Subject: Re: [Zope-dev] Removing the acquisition wrapper from an object (Python script) On Thursday 01 Aug 2002 12:49 pm, Gilles Lenfant wrote: Hi, aq_base(someObject) doesn't work in a python script. I need to get an object without its acquisition wrapper. It cant be done in untrusted code, because it would let you strip away its security settings. You could write an external method, or similar trusted code. Thanks Toby I can't understand that reason because it's also easy to strip away an object's security settings in an untrusted python script that has a Manager proxy. Well, I'm gonna make my 2 or 3 lines External method :( Cheers --Gilles ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Removing the acquisition wrapper from an object (Python script)
On Thursday 01 Aug 2002 2:44 pm, Gilles Lenfant wrote: I can't understand that reason because it's also easy to strip away an object's security settings in an untrusted python script that has a Manager proxy. Well, I'm gonna make my 2 or 3 lines External method :( If thats true, its a bug. a serious one too. Please file an example in the collector ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] DTML and REQUEST data changes about to be checked in
Hi folks,
I am about to land some big changes in the way DTML deals with data taken
from the REQUEST object when accessed implicitly, in both the Zope Trunk
and the Zope 2.5 branch. These changes could potentially break existing
Zope sites.
Without these changes, Zope is somewhat vulnerable to cross-scripting
attacks, where a well-crafted URL can cause a Zope server to serve out
arbitrary HTML. Because DTML does not automatically html quote any data,
and can implicitly get information out of the REQUEST even when it was not
the intention of the template author, it is easy to cause REQUEST data to
be rendered as HTML on a page.
My changes cause the REQUEST to keep track of suspected strings, where
suspect is defined as any string with a ''. These are marked as tainted.
Any normal, explicit access to the REQUEST will still give you normal
values. However, as soon as a DTML template requests a variable from the
general namespace, and this variable is then satisfied from the REQUEST,
the value of this variable could potentially be a TaintedString object
instead of the original string. When rendering such a value, DTML will
automatically HTML quote it if not already done so explicitly. All DTML
string operations dealing with TaintedString objects are careful to retain
the TaintedString status.
I also fixed all exceptions raised in Zope that I could find, where
untrusted REQUEST data was used in the exception message; these exceptions
now html quote the data. I also made sure that the REQUEST calculated
variables URLx and BASEx and such were not shadowed by untrusted form
variables of the same name.
These changes can break existing sites in the following ways:
- If you relied on getting HTML-like data from the REQUEST in DTML and
want to render this as HTML, and you got this data implicitly, this data
will now be HTML quoted. Note that you were vulnerable to a
cross-scripting attack here already.
You can retrieve your information from the REQUEST directly (with
dtml-with REQUEST for example), at your own risk. ;)
- HTML quoting will also take place in templates that do not otherwise
generate HTML to be sent back to the browser, such as email forms and Z
SQL Methods. For Z SQL Methods, dtml-sqlvar does not quote
TaintedStrings and is otherwise ignorant of them. For emails, use
explicit access to REQUEST instead.
- If you relied on being able to override URLx or BASEx variables through
a form variable, this no longer works. Use explicit access to
REQUEST.form instead.
- Using the string method .join (''join(items)) cannot handle
TaintedString objects. You can use _.string.join instead.
- Passing a TaintedString value from a DTML template to other objects such
as Python code, External Methods, Python Scripts, etc, may cause them to
break because they did not anticipate a TaintedString object.
What doesn't break (among others):
- Accessing REQUEST data from Python code, Python scripts, or ZPT. Only
DocumentTemplate.DT_String derivatives (DTML Document, DTML Method, etc)
and DTMLFile objects are affected.
- If you already HTML quoted, nothing gets double quoted.
- Using the _.string module in DTML retains taints.
- Zope 2.6 unicode marshalling (var:ustring:utf8) works with
TaintedStrings as well.
TaintedString objects try to mimic strings as best as they can, but until
we move to python 2.2 definitely and we can inherit from str directly,
certain python code will not accept TaintedString objects as substitutes.
I found that the normal string module, and the string ''.join module don't
accept TaintedString objects for example.
Also, using the string interpolation operator % will cause TaintedString
objects to be unwrapped. When TaintedString becomes a subclass of str,
more operations will unwrap them, such as unicode() and ''.join; or just
about any operation that manipulates strings through other ways than
string methods.
Because of size of the change and the impact on existing DTML code, well
release betas of Zope 2.5 and 2.6 soon to facilitate wider testing. For
those following CVS, please test the changes rigorously and let me know what
you find.
--
Martijn Pieters
| Software Engineer mailto:[EMAIL PROTECTED]
| Zope Corporation http://www.zope.com/
| Creators of Zope http://www.zope.org/
-
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Removing the acquisition wrapper from an object (Python script)
- Original Message - From: Toby Dickenson [EMAIL PROTECTED] To: Gilles Lenfant [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Thursday, August 01, 2002 3:51 PM Subject: Re: [Zope-dev] Removing the acquisition wrapper from an object (Python script) On Thursday 01 Aug 2002 2:44 pm, Gilles Lenfant wrote: I can't understand that reason because it's also easy to strip away an object's security settings in an untrusted python script that has a Manager proxy. Well, I'm gonna make my 2 or 3 lines External method :( If thats true, its a bug. a serious one too. Please file an example in the collector Toby, It's definitively *NOT* a bug but a feature that's completely documented. Most Zope objects inherit of RoleManager class. This class has (among others) this method : manage_permission(self, permission_to_manage, roles=[], acquire=0, REQUEST=None) Just use this method in an untrusted python script on any Zope object, add to it Manager proxy, and you're done. --Gilles ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: DTML and REQUEST data changes about to be checked in
On Thu, Aug 01, 2002 at 10:46:44AM -0400, Martijn Pieters wrote: I am about to land some big changes in the way DTML deals with data taken from the REQUEST object when accessed implicitly, in both the Zope Trunk and the Zope 2.5 branch. These changes could potentially break existing Zope sites. It's in. Let the testing begin! -- Martijn Pieters | Software Engineer mailto:[EMAIL PROTECTED] | Zope Corporation http://www.zope.com/ | Creators of Zope http://www.zope.org/ - ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: DTML and REQUEST data changes about to bechecked in
On 8/1/02 10:08 AM, Martijn Pieters [EMAIL PROTECTED] wrote: On Thu, Aug 01, 2002 at 10:46:44AM -0400, Martijn Pieters wrote: I am about to land some big changes in the way DTML deals with data taken from the REQUEST object when accessed implicitly, in both the Zope Trunk and the Zope 2.5 branch. These changes could potentially break existing Zope sites. It's in. Let the testing begin! Hopefully I'll get a chance to test it with some of our 2.5 sites - I have a small worry that old code on small sites that we don't have much worry about will break if this is put into a 2.5.2 or later release. Could there be a way to disable this feature in 2.5 via a z2/environment variable or some other configuration setting, but have it be automatic in 2.6? Potential code breakage and point point release leave me a little worried about maintaining 2.5 sites. It may not be an issue - I have to digest the changes in more depth that I've had (or currently have) time for, but that's the thought that crossed my mind earlier. -- Jeffrey P Shell www.cuemedia.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: DTML and REQUEST data changes about to be checked in
On Thu, Aug 01, 2002 at 10:29:36AM -0600, Jeffrey P Shell wrote: Hopefully I'll get a chance to test it with some of our 2.5 sites - I have a small worry that old code on small sites that we don't have much worry about will break if this is put into a 2.5.2 or later release. Could there be a way to disable this feature in 2.5 via a z2/environment variable or some other configuration setting, but have it be automatic in 2.6? Potential code breakage and point point release leave me a little worried about maintaining 2.5 sites. It may not be an issue - I have to digest the changes in more depth that I've had (or currently have) time for, but that's the thought that crossed my mind earlier. From a technical standpoint I can indeed add a switch that would disable the occurence of tainted strings, yes. I'll discuss this with Brian, it shouldn't be hard to add. But note that breakage only occurs when REQUEST data actually contains possibly dangerous markup, and your site was vulnerable in those areas that now break. Disabeling the tainting will leave you vulnerable. -- Martijn Pieters | Software Engineer mailto:[EMAIL PROTECTED] | Zope Corporation http://www.zope.com/ | Creators of Zope http://www.zope.org/ - ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Removing the acquisition wrapper from an object (Python script)
- Original Message - From: Chris Withers [EMAIL PROTECTED] To: Gilles Lenfant [EMAIL PROTECTED] Cc: Toby Dickenson [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Thursday, August 01, 2002 6:18 PM Subject: Re: [Zope-dev] Removing the acquisition wrapper from an object (Python script) Gilles Lenfant wrote: manage_permission(self, permission_to_manage, roles=[], acquire=0, REQUEST=None) Just use this method in an untrusted python script on any Zope object, add to it Manager proxy, and you're done. That method just tweaks the permissions settings. It certainly doesn't let you strip off the acquisition wrappers (and hence security settings). Please give examples :-) cheers, Chris Chris, There's some misunderstanding: I was formerly looking for a way to get rid of the acquisition wraper of an object from an untrusted python script. Toby Dickenson replied me that aq_base (as opposite to aq_parent) is not part of the acquisition utilities available from an untrusted python script because it enables to tweak the object's security settings. I replied there are ways to tweak permission settings of an object from a python script with a Manager proxy. I didn't say in any way that it's possible to remove the acquisition wrapper of an object with obj.manage_permission(...) of course :)) That's all ! Now I'm always looking for an alternate way to get rid of the acquisition wrapper and access only the objects own props. TIA --Gilles ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Removing the acquisition wrapper from an object (Pythonscript)
Gilles Lenfant wrote: manage_permission(self, permission_to_manage, roles=[], acquire=0, REQUEST=None) Just use this method in an untrusted python script on any Zope object, add to it Manager proxy, and you're done. That method just tweaks the permissions settings. It certainly doesn't let you strip off the acquisition wrappers (and hence security settings). Please give examples :-) cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope components and revision control with cvs
Mehran erfani wrote: I am a newbie to the zope world and recently started using zope in our corporate development environment. One thing that I couldn't find so far is an easy way to use cvs for revision control for individual components developed in zope. I understand that zope has its own version control system and mechanism. But still in a corporate frame work it might be desirable to find a way to be able to use systems such as cvs to do revision control outside the frame of zope. I am pretty much familiar with export and import facilities in zope, but I am sure we all agree that importing and exporting individual components of a big project is going to be quite cumbersome. On the other hand import and export of a zope project as whole is not going to be helpful in using cvs either so I am puzzled! Is there anyone out there who has a suggestion on this issue? Is there anyway around this? Does zope provide some API to be able to export and import components in group...Is there any product out there that would automate one such a procedure? I would appreciate it if anyone can help. Thanks. There are a few products on zope.org for integrating with CVS. Look through the downloadable products. I think there are CVSFolder and ZCVSFile (or something like that), maybe more. However, most version control tools are difficult to integrate into applications in a platform-neutral way. To integrate with CVS, for example, Zope spawns a CVS client, but the CVS client runs with the privileges of the Zope process rather than the user's privileges. It works but it's not very clean. Subversion, however, gives us a new opportunity. The first alpha has been released. Communicating with a Subversion repository involves HTTP, WebDAV, and DeltaV--all documented, open standards. So Zope could talk to a Subversion repository in a very transparent, well-integrated way. Anyone out there looking to write a nice, useful product for Zope? A Zope-Subversion adapter would be a great choice. :-) http://subversion.tigris.org Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Removing the acquisition wrapper from an object(Python script)
On Thu, 2002-08-01 at 14:38, Gilles Lenfant wrote: [...] Now I'm always looking for an alternate way to get rid of the acquisition wrapper and access only the objects own props. To access an object's own props (and get an error if it doesn't find them instead of looking up the aq-chain) you don't need to remove it's acquisition wrappers, you just need to tell him not to use acquisition implicitly. The way you do that is to ask the object for a non-implicit-acquisition version of itself, that is, an explicit acquisition object: non_implicit_obj = obj.aq_explicit obj.some_attr or just: obj.aq_explicit.some_attr for short. This way, 'some_attr' won't be looked up in the object's acquisition chain Cheers, Leo -- Ideas don't stay in some minds very long because they don't like solitary confinement. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Removing the acquisition wrapper from an object(Python script)
- Original Message - From: Leonardo Rochael Almeida [EMAIL PROTECTED] To: Zope Developers list [EMAIL PROTECTED] Sent: Thursday, August 01, 2002 11:37 PM Subject: Re: [Zope-dev] Removing the acquisition wrapper from an object(Python script) On Thu, 2002-08-01 at 14:38, Gilles Lenfant wrote: [...] Now I'm always looking for an alternate way to get rid of the acquisition wrapper and access only the objects own props. To access an object's own props (and get an error if it doesn't find them instead of looking up the aq-chain) you don't need to remove it's acquisition wrappers, you just need to tell him not to use acquisition implicitly. The way you do that is to ask the object for a non-implicit-acquisition version of itself, that is, an explicit acquisition object: non_implicit_obj = obj.aq_explicit obj.some_attr or just: obj.aq_explicit.some_attr Leo, Many thanks, exactly what I needed ! I didn't find much doc about these aq_xxx methods/functions. Where can I find this ? Cheers --Gilles ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope components and revision control with cvs
On Thu, 2002-08-01 at 15:04, Shane Hathaway wrote: There are a few products on zope.org for integrating with CVS. Look through the downloadable products. I think there are CVSFolder and ZCVSFile (or something like that), maybe more. However, most version control tools are difficult to integrate into applications in a platform-neutral way. To integrate with CVS, for example, Zope spawns a CVS client, but the CVS client runs with the privileges of the Zope process rather than the user's privileges. It works but it's not very clean. Subversion, however, gives us a new opportunity. The first alpha has been released. Communicating with a Subversion repository involves HTTP, WebDAV, and DeltaV--all documented, open standards. So Zope could talk to a Subversion repository in a very transparent, well-integrated way. Anyone out there looking to write a nice, useful product for Zope? A Zope-Subversion adapter would be a great choice. :-) http://subversion.tigris.org Shane Hi Shane. I've been thinking about Zope versioning, and I also did a bit of list searches for past discussions on this general topic. One problem that seems pertinent to really any external-to-zope versioning system, including Subversion, is dealing with non-folder object managers. If you want to manage the contents of these special object managers individually in your versioning system, you're running into some trouble, some parts of which seem insurmountable to me at the moment (from my admittedly limited knowledge ;-). I find myself agreeing with earlier posters (earlier as in 2001--Paul, for one, I think) who said that Zope itself might need to support full versioning, a la DeltaV or somesuch, itself. This makes sense to me, but I didn't find any historical documents on zope.org as to progress or as to reasons for abandonment of this approach. What's the deal? Would this be a huge effort, or is it definitively problematic for a reason I don't see, or is it stalled for some other reason? Thanks Gary ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope components and revision control with cvs
Gary Poster wrote: On Thu, 2002-08-01 at 15:04, Shane Hathaway wrote: There are a few products on zope.org for integrating with CVS. Look through the downloadable products. I think there are CVSFolder and ZCVSFile (or something like that), maybe more. However, most version control tools are difficult to integrate into applications in a platform-neutral way. To integrate with CVS, for example, Zope spawns a CVS client, but the CVS client runs with the privileges of the Zope process rather than the user's privileges. It works but it's not very clean. Subversion, however, gives us a new opportunity. The first alpha has been released. Communicating with a Subversion repository involves HTTP, WebDAV, and DeltaV--all documented, open standards. So Zope could talk to a Subversion repository in a very transparent, well-integrated way. Anyone out there looking to write a nice, useful product for Zope? A Zope-Subversion adapter would be a great choice. :-) http://subversion.tigris.org Shane Hi Shane. I've been thinking about Zope versioning, and I also did a bit of list searches for past discussions on this general topic. One problem that seems pertinent to really any external-to-zope versioning system, including Subversion, is dealing with non-folder object managers. If you want to manage the contents of these special object managers individually in your versioning system, you're running into some trouble, some parts of which seem insurmountable to me at the moment (from my admittedly limited knowledge ;-). This is true. the problem is that simple versioning tools don't know the meta type of a file that's on the file system. For example: you have a file MyFile.dtml on the file system. is the meta-type DTMLMethod or DTMLDocument ? Not even talking about the more complex container types here ( The same goes with the FTP access to zope objects. uploading new objects is a problem... appearantly, you first have to create an empty object of the correct type in zope and after that you can upload the content of that object. We at ZZICT too suffer severly from the restriction that versioning a product that consists of code ass well as ZODB objects is a real pain, yet from an architetural point of view splitting the product into core stuff (functionality) that resides in a filesystem based product and presentation layer stuff that resides in the ZODB is a natural way to go about. If you take a poll, I bet this problem bothers a lot of us. Maybe, you could do it the other way around: add CVS like (hopefully better or at least less sucky) functionality to zope. Sloot. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope components and revision control with cvs
On 1 Aug 2002, Gary Poster wrote: Given a hypothetical folder-like instance called myFLI, we would presumably want, in CVS (or Subversion, or whatever) a folder named myFLI containing the children and a file named, to borrow your example, myFLI.properties.zexp that *only* contains the non-ObjectManager-children properties, whatever they are. But, as I understand it, when you pickle an object for storage as a zexp--in the way the ZCVSFolder does it, for instance--you are pickling the object *and* its (ObjectManager) children: not what we want. This is the bigger stumbling block for me. Is this fixable? Overriding __getstate__ (I assume?) just for this seems fragile (can we guarantee the source of the ObjectManager children in the object, for instance? I don't think so). So that was my concern. AdaptableStorage, which I just presented at OSCON 2002, faced the same problem, but I wrestled until I found a solution. A lot of it is based on ideas from ZODB. Here's my train of thought. I'm trying to refine my explanation each time I give it. 1) You don't have to store or version pickles. Instead, you can ask a series of serializers to convert your object to a simpler representation, then you can store or version the simpler format. 2) The simpler representation can encode inter-object references just like ZODB does. 3) As the serializers convert the object to a simpler representation, they can report to a serialization tracker which parts of the object have been recorded. 4) To finish idea 2, when the serializers record an inter-object reference, they are required to report the reference to the serialization tracker. This tells the controlling software to look at the referenced object, and if it is a new object not previously recorded, it is added to the stack of objects to record. This is similar to what ZODB.Connection.commit() does. 5) One kind of serializer is the leftover pickle serializer, which records the state of attributes and subobjects not recorded by other serializers. This serializer makes it safe to version any kind of object, with a caveat that the leftover data is stored in a binary format. 6) Serializers should also be deserializers, helping ensure that anything that can be serialized can also be deserialized. 7) The thing that I spent months (maybe even years) pondering was the simple format. I knew that object serializers could be very useful for storage anywhere, as well as versioning and merging, if we could just come up with a common format that would let us connect serializers and storage adapters together. I tried XML and DOM, but it was too cumbersome, and I tried several kinds of custom classes. Nothing really fit the bill. I finally gave up, but by giving up I found the solution. Sometimes having a deadline really helps! I started using sequences of records or, in most cases, tuples of tuples. I rediscovered what RDBMS experts have known all along--that data can be easily represented as sequences of schema-bound records. So now all the serializers in AdaptableStorage serialize to a sequence of records. I admit that the requirement is a stretch in some places, since I ended up writing code like return ((value,),), but there are great benefits to having a common, simple serialization format, and Python provides just the right ingredients. 8) Serializers are configurable components. Through the configuration mechanism, we will be able to add serializers to the component system in Zope 3. We will configure which serializers should apply to which kinds of objects. Many applications will use them, including object persistence mechanisms, version control adapters, merging tools, etc. It will open up a lot of possibilities, I think. I'm sure I left a few things out, so ask questions about the unclear parts. It's probably more info than you were expecting. ;-) Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
