Re: [Zope] Re: Ape
Now, if using svn for source control is the only reason he was using APE, surely you'd agree that DirecotyViews are a more appropriate method? cheers, Chris Yes, you are more or less correct. The primary reason we went to APE was after having tried the nightmares of the various zope cvs products which just seemed to be able to tie themselves into knots after awhile. APElib allowed us to persist all objects as plain text files on the file system so that normal file system tools can be used for development. After we went to APE we moved from CVS to subversion. Now most of us develop using a text editor directly modifying our zope objects. We have the refresh interval dialed down to 1s on our dev boxes and 5-10 minutes on the live servers. This allows us to easily shoehorn minor patches (fix - commit - update server) without any service interruption on the live server. So, our reasons for using Apelib were... 1. Allow use of source control (branching, diffs, revision history, etc) on all content types as plain text files, not python pickles. 2. Allow editing of zope objects (DTML, ZPT, ZSQL, py script) externally in appropriate editor. 3. Allow us to quickly push out minor patches to running servers. 4. A distant final item was using TTW editing to do minor tweaks. The only downsides we've found for apelib are that 1, it's slow and it's been end of lifed by it's creator. Two of us have taken stabs at trying to make it so it can run in newer versions of zope, but our deep zope fu isn't up to the challenge (nor do we have a lot of time). And it seems that low level stuff for zodb storage's change with every release (as shane noted in his eol announcement). -Chris -- Chris Kratz ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Ape
On Thursday 16 March 2006 12:18 pm, Paul Winkler wrote: On Thu, Mar 16, 2006 at 11:00:11AM -0500, Chris Kratz wrote: Now, if using svn for source control is the only reason he was using APE, surely you'd agree that DirecotyViews are a more appropriate method? cheers, Chris Yes, you are more or less correct. The primary reason we went to APE was after having tried the nightmares of the various zope cvs products which just seemed to be able to tie themselves into knots after awhile. APElib allowed us to persist all objects as plain text files on the file system so that normal file system tools can be used for development. After we went to APE we moved from CVS to subversion. Now most of us develop using a text editor directly modifying our zope objects. We have the refresh interval dialed down to 1s on our dev boxes and 5-10 minutes on the live servers. This allows us to easily shoehorn minor patches (fix - commit - update server) without any service interruption on the live server. So, our reasons for using Apelib were... 1. Allow use of source control (branching, diffs, revision history, etc) on all content types as plain text files, not python pickles. 2. Allow editing of zope objects (DTML, ZPT, ZSQL, py script) externally in appropriate editor. 3. Allow us to quickly push out minor patches to running servers. 4. A distant final item was using TTW editing to do minor tweaks. DirectoryViews would mean you'd lose #4. Otherwise, I think they'd work for you and it's a much simpler solution than ape. Also, if you have any persistent objects for which there isn't an existing FSObject-derived version, you'd have to write one. Another issue is that AFAIK there's no way to control the refresh interval of FSObject subclasses. I guess you could monkeypatch FSObject._updateFromFS. You might want to do that anyway so you don't have to run your production servers in debug mode. Hello Paul, How does the refresh interval work for DirectoryView? Obviously it is not adjustable, so does it only read when it needs to the first time in and after that it hits the cache (until evicted)? I did extensive profiling of Apelib at one point and all the time was getting soaked up with ape checking mod times on thousands of files on the check interval. You don't notice this with a small site, but as the site grows it becomes noticeable. At one point I seriously thought about writing an external process/thread that would periodically sweep the files and check for modified items and then attempt to evict those items from the cache within each zope thread. Or even better, load the object into the cache out of band so it's there the next time the thread runs. I think that would have made apelib much faster and still responsive to changes on the file system. We never got it production ready though. -Chris -- Chris Kratz ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Ape
I wasn't aware of that project. We are doing the same sort thing in a manual way, ie the developers get to decide when the commits happen. But it's interesting. Unfortunately, as you noted, it's based on Ape as well and so suffers the same problems with newer versions of zope. Thanks for the pointer though. It's interesting to see where these things get taken. -Chris On Tuesday 14 March 2006 09:28 am, Doyon, Jean-Francois wrote: This might be of interest to you: http://www.zope.org/Members/jdoyon/svnapezope Doesn't fix the issue of Ape only working with 2.7 however, obviously :( J.F. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Chris Kratz Sent: March 13, 2006 3:07 PM To: zope@zope.org Subject: Re: [Zope] Re: Ape On Saturday 11 March 2006 11:16 am, Shane Hathaway wrote: Dennis Allison wrote: I am looking to map a portion of the local file system into Zope. One way to do this is to use LocalFS, another is to use Ape. I am sure there are onter approaches as well. We are using Zop 2.9.X. Will Ape work in this context? What would you suggest? Ape doesn't work with Zope 2.9 yet. Shane We are very interested in this as well. We have a large application using Apelib right now to persist our zope application onto the filesystem so we can use subversion. It's quite elegant for multiple developers, branching, etc. I'd be interested to know if anyone else has had success with other file system persistence and zope. Most seem to save as blobs (at last our last survey of the options did). Apelib is wonderfull because in most cases it saves the contents in text files which can be accessed by external editors which is how we do all our development now. Diffs and merges are easy plus we get history of changes over time. Because APE doesn't work in 2.9, we are kind of stuck as far as upradeing zope and will need to go to localFS or something like it I guess at some point in the future. I would be curious to know where you come out with this Dennis. -Chris ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Chris Kratz ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Ape
On Saturday 11 March 2006 11:16 am, Shane Hathaway wrote: Dennis Allison wrote: I am looking to map a portion of the local file system into Zope. One way to do this is to use LocalFS, another is to use Ape. I am sure there are onter approaches as well. We are using Zop 2.9.X. Will Ape work in this context? What would you suggest? Ape doesn't work with Zope 2.9 yet. Shane We are very interested in this as well. We have a large application using Apelib right now to persist our zope application onto the filesystem so we can use subversion. It's quite elegant for multiple developers, branching, etc. I'd be interested to know if anyone else has had success with other file system persistence and zope. Most seem to save as blobs (at last our last survey of the options did). Apelib is wonderfull because in most cases it saves the contents in text files which can be accessed by external editors which is how we do all our development now. Diffs and merges are easy plus we get history of changes over time. Because APE doesn't work in 2.9, we are kind of stuck as far as upradeing zope and will need to go to localFS or something like it I guess at some point in the future. I would be curious to know where you come out with this Dennis. -Chris ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope-dev] apelib performance
Hello All, Our application contains about 38k objects running via apelib into the filesystem using the filesystem mapper. We have been extremely happy with the ability to use version control on everything in zope which apelib affords us as well as allowing our developers to use editors/ide of choice. The problem we keep running into is performance. On even fast machines, right after zope starts up, it is extremely slow. Once the cache get's populated, the site runs as normal (we have the cache set to 10k objects). On a server, we set the scan-interval to 5 minutes or higher. But on our development boxes, we have the scan-interval set to 1s. This allows changes as they are made on the file system to show up in zope for testing immediately. This works quite well, but is very, very slow for developing under unless you have a pretty heavy duty processor. In profiling this, it doesn't appear to be I/O bound (as I would have guessed), but instead processor bound. Profiling points to Apelib as being the bottleneck (getmtime, getannotations, compile, etc). To verify this, I copied our application back into a standard zope storage and the performance after startup was night and day different. Very snappy. Has anyone found a way to make apelib faster, or have hints on performance tuning? Is there a better way then apelib to get zope objects (dtml, python scripts, zsql statements) into text files? Thanks for your time, -Chris ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
