[Zope-dev] proposal: serving static content faster
I have put a new proposal up at http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing which deals with serving large static content objects faster from Zope 2. This is based on some work that Paul Winkler and I did at the PyCon Zope 2 sprint. Comments appreciated. I think this would also be very useful for Zope 3; the concept would just need to be adapted to the new server architecture in use there instead of ZServer. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] proposal: serving static content faster
Chris McDonough wrote: http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing Sounds good. WRT your comments on the need for a cache multiplexer so one can handle the case of HTTP cache control headers opaque server-side caches working together--I'm really wondering if a better solution isn't to just remove the cache header manipulations from where they are now (in a seperate product) and integrate it more into an API any where object [that wants to] can use. HTTP cache control really is a protocol level thing, and the way its bundled as an add-on service right now feels pretty awkward. But anyway, thats a digression from the main thrust of your let them eat producer proposal, which I think is a good idea in general. -- Jamie Heilman http://audible.transient.net/~jamie/ I was in love once -- a Sinclair ZX-81. People said, No, Holly, she's not for you. She was cheap, she was stupid and she wouldn't load -- well, not for me, anyway. -Holly ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] proposal: serving static content faster
On Wed, 2004-03-24 at 08:00, Chris McDonough wrote: I have put a new proposal up at http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing which deals with serving large static content objects faster from Zope 2. This is based on some work that Paul Winkler and I did at the PyCon Zope 2 sprint. Comments appreciated. I think this would also be very useful for Zope 3; the concept would just need to be adapted to the new server architecture in use there instead of ZServer. Cool and very appreciated. Christian -- Christian Theune, gocept gmbh co. kg http://www.gocept.com - [EMAIL PROTECTED] fon: 03496 3099112 fax: 03496 3099118 mobile: 0179 7808366 signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] proposal: serving static content faster
On Wed, 2004-03-24 at 08:00, Chris McDonough wrote: I have put a new proposal up at http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing which deals with serving large static content objects faster from Zope 2. This is based on some work that Paul Winkler and I did at the PyCon Zope 2 sprint. Comments appreciated. I think this would also be very useful for Zope 3; the concept would just need to be adapted to the new server architecture in use there instead of ZServer. Btw: does this also - decrease memory usage during delivery time? - provide a solution of having too many threads consumed up by long-taking downloads? Christian -- Christian Theune, gocept gmbh co. kg http://www.gocept.com - [EMAIL PROTECTED] fon: 03496 3099112 fax: 03496 3099118 mobile: 0179 7808366 signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope3-dev] Re: [Zope-dev] proposal: serving static content faster
+1 from me I think its a great idea, I also agree with Jamie about HTTP cache control feeling awkward at the moment not sure how it could be better though Jamie Heilman wrote: Chris McDonough wrote: http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing Sounds good. WRT your comments on the need for a cache multiplexer so one can handle the case of HTTP cache control headers opaque server-side caches working together--I'm really wondering if a better solution isn't to just remove the cache header manipulations from where they are now (in a seperate product) and integrate it more into an API any where object [that wants to] can use. HTTP cache control really is a protocol level thing, and the way its bundled as an add-on service right now feels pretty awkward. But anyway, thats a digression from the main thrust of your let them eat producer proposal, which I think is a good idea in general. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] proposal: serving static content faster
On Wed, 2004-03-24 at 03:25, Jamie Heilman wrote: Chris McDonough wrote: http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing Darn, I always forget that cross-posting is bad due to replies to both lists. Sorry. I've limited this reply to the zope-dev list. Sounds good. WRT your comments on the need for a cache multiplexer so one can handle the case of HTTP cache control headers opaque server-side caches working together--I'm really wondering if a better solution isn't to just remove the cache header manipulations from where they are now (in a seperate product) and integrate it more into an API any where object [that wants to] can use. HTTP cache control really is a protocol level thing, and the way its bundled as an add-on service right now feels pretty awkward. I agree... it kinda feels like the services which actually get Zope involved in caching data (RAMCacheManager) should be distinct from those that let other systems know they should cache data (HTTPCacheManager). This is mostly because the need for caching something in an upstream HTTP cache isn't mutually exclusive with potentially wanting to cache it locally too. Well, I guess literally, it probably is (because you probably wouldn't bother caching something in Zope if you could just let Squid do it, at least in the common case), but the adminstrative headache of selectively associating content with the right cache manager just seems like too much of a practical burden to make people bear. It would just be way easier to let people be sloppy about it by: a) letting people associate content with multiple cache managers. b) creating a cache manager multiplexer (which might expose a bit of policy for dynamic cache selection). c) creating a different cache management architecture that provided caching services based more on policy than on a direct association (ala CMF's CachingPolicyManager). But I really don't have the time to do the work implied by any of the enumerated options. I'll hopefully be able to wait until somebody else gets suffiently indignant about the situation and takes it upon themselves to implement one of the alternatives. ;-) But anyway, thats a digression from the main thrust of your let them eat producer proposal, which I think is a good idea in general. Thanks, - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] proposal: serving static content faster
moved just to zope-dev instead of both zope3-dev and zope-dev Btw: does this also - decrease memory usage during delivery time? Probably. I haven't done any work to prove it, though. It can theoretically prevent ZODB/ZEO cache thrashing, where large static objects might consume a significant portion of the ZODB cache at any given time if you have a large enough working set of static content and lots of consumers of different parts of that working set. I believe also that the ZODB object cache stores copies of the same objects for each active connection, so presumably if you've got 4 connections and 4 people download the same content at exactly the same time, you presumably wind up with 4 copies of the same objects representing that content in memory at once. Optimally, if you were able to serve the content mostly from disk, you'd have only one copy of the content in one thread's cache at any given time and the other threads would never access the ZODB for the content at all. And at some point, once the disk-cached content had expired from the ZODB cache, you'd have no memory consumed by cache at all for that content, freeing cache memory to deal with truly dynamic things. - provide a solution of having too many threads consumed up by long-taking downloads? This isn't currently an issue in Zope 2 (and presumably not in Zope 3 either, although I haven't looked). I thought this was the case too up until a couple of days ago, but Jim let me in on the fact that Zope worker threads aren't really tied up at all during slow downloads (at least for large File/Image objects) for longer than it takes to retrieve the data out of the ZODB, because the machinery which the RESPONSE.write protocol consumes spools large chunks of data out to a tempfile. This operation doesn't block based on consumer speed, so the worker thread finishes as fast as it can write to the tempfile. Medusa alone is responsible for delivering the data out of the tempfile to the consumer afterwards, but that happens in the main thread and is transparent to Zope itself. Medusa/asyncore is very fast and consumes very few resources, so serving many copies of this content isn't much of a hit to Zope itself. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] APE and SQL Storages in Zope 2.7
Hi all, I'm trying to use APE (yesterday CVS version) on Zope 2.7.0 and Postgresql, but I'm not able to make it work. With some minor changes in datatypes.py (change loadConf for load_conf), it works fine using FSConnection, but DBAPIConnection seems not to be ready to work together with ZConfig machinery. Is there any way to have a working config for postgresql and zope 2.7 ? Or everybody is working with FileSystem storages and SQL storages are just working with zope 2.6 ? Thanks in advance Santi Camps http://zetadb.sourceforge.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PageTemplateFile vs. Bindings vs. Security
Shane Hathaway wrote: To do this, I would change Products/PageTemplates/Expressions.py. It creates an expression evaluation engine and adds expression types to it. It chooses the unrestricted or the restricted expression types based on whether the Zope module exists. This is a wart. Instead, I think it should create two engines, one restricted and one unrestricted. Then you should be able to tell the PageTemplateFile constructor which engine to use. That sounds mighty handy. What needs to happen for that to happen? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] manage_access patch
Hi, I've made a patch to manage_access (and AccessControl.Role.permission_settings) that displays the current effective security setting (something that have been bugging me for ages). It also uses a small javascript and disabled property of the checkboxes (with potential problems with old browsers that don't support the disabled property). So where should I send the patch, I'm not currently active as a Zope contributor. Regards, Johan Carlsson -- Johan Carlsson Tel: + 46 8 31 24 94 Colliberty Mob: + 46 70 558 25 24 Torsgatan 72Email: [EMAIL PROTECTED] SE-113 37 STOCKHOLM Skype: colliberty ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PageTemplateFile vs. Bindings vs. Security
On Wed, 24 Mar 2004, Chris Withers wrote: Shane Hathaway wrote: To do this, I would change Products/PageTemplates/Expressions.py. It creates an expression evaluation engine and adds expression types to it. It chooses the unrestricted or the restricted expression types based on whether the Zope module exists. This is a wart. Instead, I think it should create two engines, one restricted and one unrestricted. Then you should be able to tell the PageTemplateFile constructor which engine to use. That sounds mighty handy. What needs to happen for that to happen? A voluntary volunteer needs to volunteer voluntarily. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] APE and SQL Storages in Zope 2.7
[EMAIL PROTECTED] wrote: I'm trying to use APE (yesterday CVS version) on Zope 2.7.0 and Postgresql, but I'm not able to make it work. You could do with being a bit more explicit about the problems you encountered. Exception type, value and traceback please... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
Chris McDonough wrote: I have put a new proposal up at http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing which deals with serving large static content objects faster from Zope 2. This is based on some work that Paul Winkler and I did at the PyCon Zope 2 sprint. Comments appreciated. I think this would also be very useful for Zope 3; the concept would just need to be adapted to the new server architecture in use there instead of ZServer. This sounds useful for serving content from the filesystem. However, I'm a little concerned about this because producers must not read from the object database. After the application finishes interpreting a request and returns a producer to be processed asynchronously, it closes the ZODB connection. If the producer then tried to load some ZODB object, ZODB would do one of two things. If the connection is still closed, ZODB would raise an error. If the connection happens to be re-opened by another thread, ZODB might allow it, but it has a chance of going insane if it happens to be loading or storing something in the other thread at the same time. To work around this, you can give to the producer an object that contains the chunks already loaded from the database. If you do that, though, I think you've lost all the benefit of using a producer. Another workaround might be to open a special ZODB connection just for the main thread. The producer should load objects from that connection instead of the application thread's connection. Hey, I think that would solve another problem at the same time: multiple requests for the same large object would use the same ZODB cache. Wow, do you see any holes in that plan? I'm thinking that would win both speed and memory. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Pointless exception re-raising in DA.py
Hi All, In Shared/DC/ZRDB/DA.py, Line 399 of Zope 2.6.1 and line 419 in 2.7.0, there's this rather pointless lump of code: try: DB__=dbc() except: raise DatabaseError, ( '%s is not connected to a database' % self.id) ...which only serves to mask the real cause of the problem when trying to obtain a connection. Would anyone have any objections if I changed this to simply: DB__ = dbc() ? (if no responses, I'll assume that means you agree with my proposed change ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Pointless exception re-raising in DA.py
Hi Chris, In Shared/DC/ZRDB/DA.py, Line 399 of Zope 2.6.1 and line 419 in 2.7.0, there's this rather pointless lump of code: try: DB__=dbc() except: raise DatabaseError, ( '%s is not connected to a database' % self.id) ...which only serves to mask the real cause of the problem when trying to obtain a connection. I remeber I made a collector issue about that liens of code: http://www.zope.org/Collectors/Zope/927 Previously this has been a string valued exception. Actually I like it to be an exception of a certain type now, because I can now selectively catching this DatabaseError and distinguish it from other errors. No need for a evil bare except. The reason is that I would like to treat errors when accessing an external data base different from other errors; often the data obtained there is only optional to the page, so I don't want to show usered the error page in this case, only to fill up doem slot with sorry, that certain piece of information is not availabe, because our sql data base sucks. I see Your point. However is there any chance that dbc() does raise something more specific? I am afraid it does not. If it does not, I would have to go back to an evil bare except ... Cheers, Clemens ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Pointless exception re-raising in DA.py
Hi there, Clemens Robbenhaar wrote: I remeber I made a collector issue about that liens of code: http://www.zope.org/Collectors/Zope/927 Indeed. Catching everything and raising a string exception is evil, and I was glad to see that go. Previously this has been a string valued exception. Actually I like it to be an exception of a certain type now, because I can now selectively catching this DatabaseError and distinguish it from other errors. No need for a evil bare except. Well, I'm afraid I don't agree here... The reason is that I would like to treat errors when accessing an external data base different from other errors; often the data obtained there is only optional to the page, so I don't want to show usered the error page in this case, only to fill up doem slot with sorry, that certain piece of information is not availabe, because our sql data base sucks. ...this is an application level decision. The code currently makes it very easy, but at the expense of debugging any unexpected exceptions that code throws. While relational data may be optional for you, for many people it is ESSENTIAL, and is used for things like their auth details, content storage, etc. For them, this piece of code causes MAJOR suffering, especially for intermittent failures where you can't just insert a print_traceback and try again ;-) I see Your point. However is there any chance that dbc() does raise something more specific? I am afraid it does not. No, that's the whole point. If that code raises an exception, it's much more useful if it can be logged and dealt with in its original form rather than have it morphed into something uniform and meaningless. If it does not, I would have to go back to an evil bare except ... I wouldn't if I were you: try: *your zsql method except ConflictError: raise except Exception: return sorry, that certain piece of information is not availabe, because our sql data base sucks If your database adapter raises a string exception, THEN you have to add a bare except on the end of that, but if that's the case you should beat the DA author with a stick until he fixes it. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] APE and SQL Storages in Zope 2.7
[EMAIL PROTECTED] wrote: Is there any way to have a working config for postgresql and zope 2.7 ? Or everybody is working with FileSystem storages and SQL storages are just working with zope 2.6 ? Thanks for reporting this. I just now brought the ZConfig schema up to date. See the updated component.xml for an example of setting up a connection to Postgres. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Pointless exception re-raising in DA.py
Hi Chris, [...] ...this is an application level decision. The code currently makes it very easy, but at the expense of debugging any unexpected exceptions that code throws. While relational data may be optional for you, for many people it is ESSENTIAL, and is used for things like their auth details, content storage, etc. For them, this piece of code causes MAJOR suffering, especially for intermittent failures where you can't just insert a print_traceback and try again ;-) Ok, understand (I guess ;-) [...] If it does not, I would have to go back to an evil bare except ... I wouldn't if I were you: try: *your zsql method except ConflictError: raise except Exception: return sorry, that certain piece of information is not availabe, because our sql data base sucks Ok, if this is the proper dance to catch such exceptions with Zope without risking ZODB corruption I will go with it. So far I have not been sure, that ConflictError is the only type needing special treatement in these cases. Clemens ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: Zope2.7.0rc2 AttributeError: RESPONSE object has noattribute 'setHeader'
After investigating (but not solving the problem...) The objects are there, but some security stuff returns None instead of the object when we aren't supposed to have the rights... (could someone tell me what function is first called to get the value of an object such as AUTHENTICATED_USER ???). In my case, this has everything to do with the class inheriting from ObjectManager and declaring security.declareObjectPrivate(). Inherited methods such as manage_workspace can be called when you're Manager, but some objects (AUTHENTICATED_USER and RESPONSE among others) evaluate to None (if security is really at stake here, raising an appropriate exception may be a better answer). And it's not very intuitive for Manager to have some closed doors. For now, I've removed the declareObjectPrivate to work around the problem, but I'm not sure it's an intended behavior, in particular after having read the CHANGES.txt. Regards, Ch. Tronche On Tuesday 23 March 2004 20:36, Christophe Tronche wrote: Hello all. Has this problem been solved ? I'm stuck in the very same problem after having switched from python 2.6.2 to 2.6.4, both using the same python2.2.3. Pure Zope, no Product other than my own (where not a single line of code has been modified) and no funny name like Tony's SYS or whatever. The problem occurs when trying to interpret App/dtml/manage_page_header.dtml and /OFS/dtml/main.dtml in a management tab. Any idea someone ? Regards, Ch. Tronche Thanks again for the reply Dieter, On 6 Feb 2004, at 21:14, Dieter Maurer wrote: REQUEST.RESPONSE is set up in ZServer.HTTPServer.zhttp_handler.continue_request. It should be impossible that is is None. Agreed. Maybe, we have a memory corruption. I have seen objects magically becoming None in Python versions with memory corruption problems (Python 2.1.1). -- Dieter Could be memory perhaps, but this is python2.3.3. Christian Heimes wrote: [...] And the next one: Python2.3.3/Zope2.70rc2/Plone2rc5. Exception Type AttributeError Exception Value 'str' object has no attribute 'RESPONSE' [...] Module Products.CMFPhoto.Photo, line 510, in clearCache AttributeError: 'str' object has no attribute 'RESPONSE' The relevant code line is self.REQUEST.RESPONSE.setHeader(). The error occured at object creation. Something really bad is going on! Christian The problem I'm having is that these are all stock Zope products from a Zope2.7.0rc2 release. The traceback is coming from the Zope core. Traceback (innermost last): ... Module Shared.DC.Scripts.Bindings, line 320, in __render_with_namespace__ Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec Module App.special_dtml, line 175, in _exec Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: RESPONSE Module string, line 0, in ? AttributeError: 'NoneType' object has no attribute 'setHeader' This also seems different from what Christian is reporting; Christian Heimes wrote at 2004-2-6 23:01 +0100: And the next one: Python2.3.3/Zope2.70rc2/Plone2rc5. Exception Type AttributeError Exception Value 'str' object has no attribute 'RESPONSE' [...] Module Products.CMFPhoto.Photo, line 510, in clearCache AttributeError: 'str' object has no attribute 'RESPONSE' The relevant code line is self.REQUEST.RESPONSE.setHeader(). The error occured at object creation. Something really bad is going on! So, I'm still baffled by this. Many thanks to all those providing more data points. I *think* this is a 'Zope' thing and not a 'Tone' thing, but I've been wrong in that area before ;) cheers tone. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] APE and SQL Storages in Zope 2.7
Shane Hathaway escribió: [EMAIL PROTECTED] wrote: Is there any way to have a working config for postgresql and zope 2.7 ? Or everybody is working with FileSystem storages and SQL storages are just working with zope 2.6 ? Thanks for reporting this. I just now brought the ZConfig schema up to date. See the updated component.xml for an example of setting up a connection to Postgres. Shane Thanks a lot for your answer !! It works fine :-) Let me make a suggestion: In next releases, leave SQLMultiTableProperties option as default. That's what all Zope newbies are looking for, I think. I meet a lot of developers that like Zope, but they are disturbed if data aren't in a relational database, but also if data aren't in a relational way (just what SQLMultiTableProperties does). It could seems a silliness, but it's real, and stops a lot of people to use Zope. I think Ape will help a lot to expand Zope. I will begin to intesively use Ape with SQLMultiTableProperties option over Postgresql.Of course, I will report eventual bugs in the collector. Thanks again Santi Camps ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
moved to zope-dev only On Wed, 2004-03-24 at 09:28, Shane Hathaway wrote: This sounds useful for serving content from the filesystem. However, I'm a little concerned about this because producers must not read from the object database. I think the easiest thing to do is to turn this into a rule. ;-) I'd be comfortable saying you can't access ZODB from a producer for now and then thinking about allowing people to do it later. To work around this, you can give to the producer an object that contains the chunks already loaded from the database. If you do that, though, I think you've lost all the benefit of using a producer. IMO code that needs to read from the database shouldn't return a producer. Instead, it should probably continue using the RESPONSE.write streaming protocol in the worker thread when it needs to do producer-like things. Returning a producer to ZPublisher seems to only really useful when the producer's more generator is guaranteed to be exceedingly cheap because as you noted it is meant to operate in the context of the main thread. Another workaround might be to open a special ZODB connection just for the main thread. The producer should load objects from that connection instead of the application thread's connection. Hey, I think that would solve another problem at the same time: multiple requests for the same large object would use the same ZODB cache. Wow, do you see any holes in that plan? I'm thinking that would win both speed and memory. The time spent waiting for the code that accessed the database would block all other asyncore operations, though, right? We'd need to test it, but I suspect it might be a net lose for the multiple requests for the same object case because the overhead of reading from the database cache would be realized serially for each request. And if the object isn't in cache, it could potentially block for quite a long time. That said, I dunno. Do you think it might be a win? I guess my worry is that the the operation of producer should be more or less guaranteed to be cheap and it seems hard to make that promise about ZODB access, especially as the data might be coming over the wire from ZEO. FWIW, Jim intimated a while back that he might be interested in providing blob support directly within ZODB. I can imagine an implementation of this where maybe you can mark an object as blobifiable and when you do so, the ZODB caching code writes a copy of that object into a named file on disk during normal operations hand-waving goes here ;- Then we could use a producer to spool the file data out without ever actually reading data out of a database from a ZODB connection; we'd just ask the connection for the filename. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
Chris McDonough wrote: IMO code that needs to read from the database shouldn't return a producer. Instead, it should probably continue using the RESPONSE.write streaming protocol in the worker thread when it needs to do producer-like things. Returning a producer to ZPublisher seems to only really useful when the producer's more generator is guaranteed to be exceedingly cheap because as you noted it is meant to operate in the context of the main thread. I'll note that iterators probably ought to replace producers. Just spell more as next and they look pretty much the same. The time spent waiting for the code that accessed the database would block all other asyncore operations, though, right? We'd need to test it, but I suspect it might be a net lose for the multiple requests for the same object case because the overhead of reading from the database cache would be realized serially for each request. Look at it this way: - Don't ghostify anything manually. Let ZODB handle that. - Use a larger ZODB cache for the main thread's connection than you do for the other connections, to increase the chance that objects will be served directly from RAM. - As long as other threads aren't reading/writing the large objects, there will be at most one copy of a large object in memory at any given time. - Periodically ask the connection to collect garbage. It uses a LRU strategy, which seems much more optimal than immediate deactivation. And if the object isn't in cache, it could potentially block for quite a long time. That said, I dunno. Do you think it might be a win? I guess my worry is that the the operation of producer should be more or less guaranteed to be cheap and it seems hard to make that promise about ZODB access, especially as the data might be coming over the wire from ZEO. If the object is not loaded and not in the ZEO cache, the producer could say it's not ready yet and ask ZEO to fetch it in the background. Jeremy has suggested that object pre-fetching could be added to ZODB. FWIW, Jim intimated a while back that he might be interested in providing blob support directly within ZODB. I can imagine an implementation of this where maybe you can mark an object as blobifiable and when you do so, the ZODB caching code writes a copy of that object into a named file on disk during normal operations hand-waving goes here ;- Then we could use a producer to spool the file data out without ever actually reading data out of a database from a ZODB connection; we'd just ask the connection for the filename. That's a possibility, although it would complicate the storage, and making it work with ZEO would require a distributed filesystem. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster
On Wed, 2004-03-24 at 13:32, Shane Hathaway wrote: Chris McDonough wrote: IMO code that needs to read from the database shouldn't return a producer. Instead, it should probably continue using the RESPONSE.write streaming protocol in the worker thread when it needs to do producer-like things. Returning a producer to ZPublisher seems to only really useful when the producer's more generator is guaranteed to be exceedingly cheap because as you noted it is meant to operate in the context of the main thread. I'll note that iterators probably ought to replace producers. Just spell more as next and they look pretty much the same. I did consider that, but since the idea was to make it as fast as possible, I figured we'd just return something that medusa could deal with directly. But since medusa doesn't know beans about StopIteration coming out of an iterator, we can't just alias more to next and expect it to work, at least without changing medusa. But maybe that's the right thing to do anyway (medusa is pretty overdue for some spring cleaning) , or maybe we just wrap the iterator up in something medusa currently understands. It doesn't matter to me either way, really. The time spent waiting for the code that accessed the database would block all other asyncore operations, though, right? We'd need to test it, but I suspect it might be a net lose for the multiple requests for the same object case because the overhead of reading from the database cache would be realized serially for each request. Look at it this way: - Don't ghostify anything manually. Let ZODB handle that. - Use a larger ZODB cache for the main thread's connection than you do for the other connections, to increase the chance that objects will be served directly from RAM. - As long as other threads aren't reading/writing the large objects, there will be at most one copy of a large object in memory at any given time. - Periodically ask the connection to collect garbage. It uses a LRU strategy, which seems much more optimal than immediate deactivation. OK. I'll let you handle that. ;-) And if the object isn't in cache, it could potentially block for quite a long time. That said, I dunno. Do you think it might be a win? I guess my worry is that the the operation of producer should be more or less guaranteed to be cheap and it seems hard to make that promise about ZODB access, especially as the data might be coming over the wire from ZEO. If the object is not loaded and not in the ZEO cache, the producer could say it's not ready yet and ask ZEO to fetch it in the background. Right. We'd need to come up with a protocol that lets the producer return not ready yet. I suppose this could just be implemented as an exception. Jeremy has suggested that object pre-fetching could be added to ZODB. I'll let you handle that too. ;-) FWIW, Jim intimated a while back that he might be interested in providing blob support directly within ZODB. I can imagine an implementation of this where maybe you can mark an object as blobifiable and when you do so, the ZODB caching code writes a copy of that object into a named file on disk during normal operations hand-waving goes here ;- Then we could use a producer to spool the file data out without ever actually reading data out of a database from a ZODB connection; we'd just ask the connection for the filename. That's a possibility, although it would complicate the storage, and making it work with ZEO would require a distributed filesystem. It would actually complicate the ZODB connection caching code but the storage would have really nothing to do with it. It also wouldn't require a distributed filesystem, because all we'd be doing is storing cached copies of the data on the local disk of each ZEO client. An implementation could go something like this: Objects that want to participate in the blob caching scheme can implement a _p_makeBlob method (or whatever), which returns an iterator representing the serialized data stream. When a request for an object is provided to the connection: - if it is not in the ZODB cache, return a ghost like normal. - if it is in the cache and it has a _p_makeBlob method, check if a file on disk exists with its oid. if a file doesn't exist on disk, call _p_makeBlob and create the file using the iterator it returns. set _p_blob_filename on the object to the filename of the file created. - App code can now use check for _p_blob_filename to see if a cached copy representing the serialized data exists on disk. If it does, it can make use of it how it sees fit. - when a cached object is invalidated out of the ZODB caches, delete the cached file too. This happens on every ZEO client. Solving race conditions and locking is an exercise left to the reader. ;-) - C ___ Zope-Dev maillist - [EMAIL PROTECTED]
Re: [Zope-dev] PageTemplateFile vs. Bindings vs. Security
Shane Hathaway wrote: On Wed, 24 Mar 2004, Chris Withers wrote: That sounds mighty handy. What needs to happen for that to happen? A voluntary volunteer needs to volunteer voluntarily. I'll probably tackle it, but not before next month due to more immediate fires. -- Jamie Heilman http://audible.transient.net/~jamie/ I was in love once -- a Sinclair ZX-81. People said, No, Holly, she's not for you. She was cheap, she was stupid and she wouldn't load -- well, not for me, anyway. -Holly ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Pointless exception re-raising in DA.py
Chris Withers wrote at 2004-3-24 15:36 +:
...
The reason is that I would like to treat errors when accessing
an external data base different from other errors; often the data
obtained there is only optional to the page, so I don't want to show
usered the error page in this case, only to fill up doem slot with
sorry, that certain piece of information is not availabe, because our
sql data base sucks.
...this is an application level decision. The code currently makes it very easy,
but at the expense of debugging any unexpected exceptions that code throws.
You can have both!
It is just a little more work for you (I know you are lazy...):
try: DB__=dbc()
except:
exc_type, exc_value, trc = sys.exc_info()
raise DatabaseError('%s is not connected to a database' % self.id,
exc_type,
exc_value), trc
trc = None
Define DatabaseError in such a way that its __str__ includes
information about the original exception.
--
Dieter
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
