Re: [Zope-dev] Zope.pipeline proposal
Hi there, One issue I have with using paste deploy's pipeline configuration for endware is that such configuration sometimes really wants to be part of a library. I.e. I don't want to configure a tower of endwares each time I write an application, I want to reuse some premade configuration that comes along with some released library. This as opposed to middleware. The middleware is generic across WSGI servers and the person who decides to use it is frequently the developer. That said, there's likely a grey area between the two. Perhaps there's a way you can use paste's mechanism and ship it in a library. If not, you'll end up stacking together WSGI's in Python code. I can then see the benefit in a declarative version of this that can ship along with a library. With the right application of grokkers that could probably be made to look very compact too. :) Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Chris McDonough wrote: [snip] While I think that would be a good thing, I do want to mention that it's not really the point of the whatsitdoing benchmark. Right, agreed. I think it's more important to make the Zope Framework more comprehensible than it is to improve its performance. Its performance is already fine for many purposes and it'll be much easier to improve once the structure is more comprehensible anyway. We have currently two efforts going on to increase comprehensibility of the Zope Framework: * cutting away code that isn't needed or is unnecessarily pulled in. This is the dependency refactoring work. * Shane's zope.pipeline work. I'm pretty pleased we have all of this going, as long as we keep it up. :) That we need to clean up the framework became clear to me much before Chris started pointing out things in the whatsitdoing benchmark though, but earlier discussions about Repoze (in particular with Tres at DZUG last september) did help shape my ideas. Repoze stuff typically tries to simplify each component in the set of components used to service a goal, sometimes at the expense of at least some backwards compatibility or excessive configurability. On the other hand, Zope3 and Grok tend to keep backwards compatibility and configurability sometimes at the expense of verbosity and extra runtime expense. They are just somewhat different goals. Repoze stuff therefore has the major advantage of not needing to carry around 6-10 years of backwards compatibility baggage; it owes any 20-20 hindsight in these cases of course directly to Zope. I do believe we can move the existing Zope Framework forward quite a bit in the right direction too. Perhaps we'll have to give up some backwards compatibility, but we can probably keep most of the code working. We haven't been afraid to break things occasionally with Grok but we've found that with good upgrade notes we can keep the pain to a minimum. This work should also result in more reusable components for Repoze and more use of Repoze components within code that uses the Zope Framework (such as Grok), so that should make us all happy. :) Luckily we're not starting with a very bad situation here. We have the benefit that the Zope Framework (and Grok) are made out of pieces that are at least *somewhat* tractable and replaceable. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope.pipeline proposal
Hi all, I've put up a draft of a zope.pipeline proposal: http://wiki.zope.org/zope3/ZopePipeline The proposal is intended to explain my thoughts on the subject more thoroughly. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Dan Korostelev wrote: Also, how easy is to integrate existing non-zopeish WSGI middlewares into the zope.pipeline? Like some resource injectors or XHTML slimmers and so on. It would be really great to be able to do that with single ZCML directive. You can do that with two ZCML directives. I probably need to elaborate on the directives in the proposal. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
2009/2/25 Martijn Faassen faas...@startifact.com: One area that I'd like to see support for is some easy way to turn off security proxies. It's rumored there is such a way but with Grok, we ended up ripping them off repeatedly anyway. Am I right in that it should be possible to put a WSGI endware on top of this whole stack that does an explicit security check? I think so. Currently, the main entry point for security proxies is the get root method of the publication, so if you'll use modified publication object that don't wrap root object in the ProxyFactory, you'll rip most of them. However, some things, like trusted adapters rewrap objects using ProxyFactory, so, maybe we could add some modifier to the ProxyFactory function that just makes it return object as is w/o wrapping. This way we could turn off proxies globally without need to modify code that uses ProxyFactory. -- WBR, Dan Korostelev ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roger Ineichen wrote: Hi Shane Betreff: [Zope-dev] Zope.pipeline proposal Hi all, I've put up a draft of a zope.pipeline proposal: http://wiki.zope.org/zope3/ZopePipeline The proposal is intended to explain my thoughts on the subject more thoroughly. Do you know something about the performance of WSGI? I whould be happy to see some perfomance tests comparing WSGI with other server concepts. Chris McDonough wrote up some complexity benchmarks here: - http://plope.com/whats_your_web_framework_doing - http://plope.com/whatsitdoing2 We did some work early on in the repoze effort comparing the various WSGI servers: http://lists.repoze.org/pipermail/repoze-dev/2007-September/000487.html Note that the relative differences between the various servers drop off steeply once the application begins doing any real work (even the Z2 quickstart page). This is because requests-per-second is actually a deceptive measure: we should probably switch to talking microseconds per request instead. In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer). Real speed wins only come from more radical simplifications: for instance, repoze.bfg drops the adapter-based traversal semantics in favor of using only __getitem__. The fact that the BFG hello world app is ~20 times faster[1] than the Grok hello world is due to such simplifications. [1] 163 us/req for the BFG vs.3012 us/req for Grok in Chris' profiling, the difference is more marked with profiling turned off. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpXd3+gerLs4ltQ4RAiQXAJ9/RP7UWcLZ/lCW0jDdlTCqko8cgACdHhoC yx1PS0mMm7eCNrtvAr3QMYY= =hfNz -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Roger Ineichen wrote: Do you know something about the performance of WSGI? I whould be happy to see some perfomance tests comparing WSGI with other server concepts. WSGI is extremely lightweight, so WSGI itself isn't going to affect performance. The WSGI servers I know about are reasonably fast. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Shane Hathaway wrote: Dan Korostelev wrote: Also, how easy is to integrate existing non-zopeish WSGI middlewares into the zope.pipeline? Like some resource injectors or XHTML slimmers and so on. It would be really great to be able to do that with single ZCML directive. You can do that with two ZCML directives. I probably need to elaborate on the directives in the proposal. Agreed, I'd like to hear more about these. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tres Seaver wrote: Roger Ineichen wrote: Hi Shane Betreff: [Zope-dev] Zope.pipeline proposal Hi all, I've put up a draft of a zope.pipeline proposal: http://wiki.zope.org/zope3/ZopePipeline The proposal is intended to explain my thoughts on the subject more thoroughly. Do you know something about the performance of WSGI? I whould be happy to see some perfomance tests comparing WSGI with other server concepts. Sorry for following up to myself: I had meant to include links to the recent songlist thread which examined raw WSGI vs. various frameworks: - - The original, raw WSGI writeup: http://www.eflorenzano.com/blog/post/writing-blazing-fast-infinitely-scalable-pure-wsgi/ - - Martijn Faassen's Grok take: http://faassen.n--tree.net/blog/view/weblog/2009/01/10/0 - - Carlos de la Guardia's repoze.bfg writeup: http://blog.delaguardia.com.mx/index.php?op=ViewArticlearticleId=119blogId=1 Please note the comparison Carlos makes between raw WSGI on his box, using the default Paste http server (1780 requests/second) vs. the repoze.bfg version with the same server (1060 requests/second). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpYPX+gerLs4ltQ4RAiihAJwNxxxiU3L8c2aBCyE1D6qu3CfHYQCgySlm LpYSFDGcULBueL/9+Oz2OjE= =4a6d -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Martin Aspeli wrote: I'm used to using Paste Deploy ini files to configure a WSGI pipeline. Is this simply an alternative to that? If so, do we really need our own alternative, or could we try to use the Paste Deploy stuff directly? Yes, you can just use Paste Deploy instead of the ZCML-based pipeline builder. However, the pipeline you get with Paste Deploy will not vary according to the request type, so you won't get the publisher's special handling for non-browser methods like PUT and DELETE, nor XML-RPC support, etc. OTOH, I'm sure people can find creative ways to make Paste Deploy vary the pipeline according to the request type. I am a little worried about the conceptual overhead of having both the ZCA and the WSGI pipeline provide swappability services to application builders. It feels like those two things overlap somewhat in scope. They do overlap, but the ZCA-configured pipeline can easily vary according to the request type. I need to add more info about this to the proposal. Also, looking at your endware, there are some that seem to overlap with Repoze stuff. Is this on purpose? I think the relationship with Repoze should be made more clear in the proposal. I don't know what the relationship is yet because I've chosen to only get code from Zope, for now, since I'm trying to maintain compatibility. virtual_host -- is this the same as repoze.vhm? Could be. Not sure. retry -- is this the same as repoze.retry? I think it might be different, because this version of retry resets the environment rather than the request. It has to be in the pipeline before the create_request application. This design makes it possible to eliminate the complex retry code from Zope requests. create_request -- should this maybe have some compatibility with WebOb requests? I've looked at WebOb, and my impression is that Zope requests and WebOb requests serve completely different purposes. A Zope request is essentially an input decoder and information holder. A WebOb request is a whole framework, AFAICT. That said, I expect zope.pipeline to give people a greater opportunity to prove I'm wrong about things like that. switch_pipeline -- could this be made non-Zope specific? It sounds useful. I need to understand better what you mean by non-Zope specific. How would you use it? log -- both repoze and paste have logging middleware, IIRC Sure. I haven't bothered to write this app at all since I know lots of people have written such things. open_root -- I thought repoze had something similar, but I may be wrong I think this one is significantly different from Repoze. It would be good to communicate about stuff like this at a sprint. clean_transaction -- is this not the same as repoze.tm2? No. To mimic the current Zope publisher, we need to commit the transaction shortly after the call application is finished, but then a lot of things can still happen before the response leaves the server, so we need to make sure any open transaction is aborted before letting the open_root application close the database. This application is very small. It might make sense to have the open_root application do its job instead. handle_error -- again, I thought Repoze had something like this This error handling is very different from Repoze, I think. It's going to handle errors the Zope way, by looking up error handlers in the database. I imagine a lot of people will want to remove or customize it. end_transaction -- sounds like the other end of repoze.tm2 This is like repoze.tm2, but it will also annotate the transaction. Note that the code in my sandbox does not yet reflect this thinking. authenticate -- sounds like repoze.who? I doubt it. This does the IAuthentication dance, allowing authentication to be added during traversal. fix_relative_links -- sounds generally useful outside Zope as well I wish, but this application depends on Zope's traversal logic to determine the canonical URL of an object. I don't think the same traversal information will be present in most frameworks. OTOH, I imagine fix_relative_links could be rewritten as a more broadly useful thing, if someone can figure out how without breaking performance. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Hey, Tres Seaver wrote: [snip] In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer). As for Grok I hope we can break some backwards compatibility and get some larger performance speedups. We definitely need to aggressively keep moving forward in this area. Not even primarily for speed gains but also for comprehensibility; I find Chris's what's it doing report far more worrying than differences in speed at this point: http://plope.com/whatsitdoing2 This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area. Real speed wins only come from more radical simplifications: for instance, repoze.bfg drops the adapter-based traversal semantics in favor of using only __getitem__. The fact that the BFG hello world app is ~20 times faster[1] than the Grok hello world is due to such simplifications. In a hello world the overhead of the publisher dominates in a way that in normal applications it probably doesn't, so what are real speed wins, as usual, will depend. Improvements in template language implementations might gain us more. That said, recent profiling of Grok with chameleon make that a small component too, so perhaps it *is* the publisher. Oh the joys of profiling! :) Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Hanno Schlichting wrote: What's the overhead of a WSGI middleware? Is the overhead cost in the same order of magnitude as a simple function call with a return value or is there something inherently more complex going on? A WSGI middleware app is simply a callable thing that calls the next app, so the overhead is literally the same as a simple function call. There won't be security proxies or any other such thing getting in the way. From my current experience I usually classify things into the groups of: 1. attribute access 2. function calls 3. adapter lookups Going from one of these groups into the next one has a very noticeable performance cost involved, if used on the critical path of the application. The current publisher is chock full of function calls and adapter lookups along the critical path. I expect the zope.pipeline project to open everyone's eyes about just how much we're already doing in the critical path. I think much of it is frequently not needed at all. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Hey, Shane Hathaway wrote: Martin Aspeli wrote: [snip] create_request -- should this maybe have some compatibility with WebOb requests? I've looked at WebOb, and my impression is that Zope requests and WebOb requests serve completely different purposes. A Zope request is essentially an input decoder and information holder. A WebOb request is a whole framework, AFAICT. As far as I know WebOb provides a request and response implementation on top of WSGI. I guess that makes it similar to what zope.publisher does. Using WebOb straight away won't be very easy as the API isn't compatible with that of Zope's request object. You could adapt one to the other, but that won't gain you much. It should be fairly straightforward, I hope, to obtain a WebOb-style request instead of a Zope-style request in application code. That said, I expect zope.pipeline to give people a greater opportunity to prove I'm wrong about things like that. On the longer term I'd like to experiment with a WebOb-based Grok and this should allow a way in. But that's definitely long-term thinking. Regards, Martijn ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Hi all Betreff: Re: [Zope-dev] Zope.pipeline proposal Hey, Tres Seaver wrote: [snip] In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer). As for Grok I hope we can break some backwards compatibility and get some larger performance speedups. We definitely need to aggressively keep moving forward in this area. Not even primarily for speed gains but also for comprehensibility; I find Chris's what's it doing report far more worrying than differences in speed at this point: http://plope.com/whatsitdoing2 This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area. I absolutly agree! As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable. Regards Roger Ineichen ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roger Ineichen wrote: Hi all Betreff: Re: [Zope-dev] Zope.pipeline proposal Hey, Tres Seaver wrote: [snip] In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer). As for Grok I hope we can break some backwards compatibility and get some larger performance speedups. We definitely need to aggressively keep moving forward in this area. Not even primarily for speed gains but also for comprehensibility; I find Chris's what's it doing report far more worrying than differences in speed at this point: http://plope.com/whatsitdoing2 This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area. I absolutly agree! As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable. I think you've made Chris' point for him: nothing about the application being tested *requires* that there be a ZODB connection open; Grok's design forces opening one unconditionally, which is a layer of complexity which *can't be turned off.* The conceptual overhead of each of the frameworks is at least as important as the performance overhead. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpdHk+gerLs4ltQ4RAutNAJ91GlaX91Kl4hzKlv9NKUezrdK4zQCgyLBL We7uIeSpZ+KPepKU3/eCey8= =U1s/ -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Hi Tres http://plope.com/whatsitdoing2 This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area. I absolutly agree! As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable. I think you've made Chris' point for him: nothing about the application being tested *requires* that there be a ZODB connection open; Grok's design forces opening one unconditionally, which is a layer of complexity which *can't be turned off.* The conceptual overhead of each of the frameworks is at least as important as the performance overhead. Yes, that's true. But I like to see the real performance win within a *real* application. My applications normaly do not only show a hello world page ;-) I also like to see how much the repoze method calls will grow if the authentication, transaction, ZODB etc is involved. I'm pretty sure that it will much less then 20 times like in the given test. I'll also be happy if we will gain a 2 time perfomance factor in a real comparable test app. I've developed a prototyping package in my own personal repos which tries to setup a site within some objects which I use with the old zope server and a paste WSGI server setup. This let us run performance tests against the ZODB. Probably I should add that to the zope repos and we could start adding all our different publishing implementations. This whould let us compare the same things in a real world use case. Regards Roger Ineichen ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Shane Hathaway wrote: Martin Aspeli wrote: I'm used to using Paste Deploy ini files to configure a WSGI pipeline. Is this simply an alternative to that? If so, do we really need our own alternative, or could we try to use the Paste Deploy stuff directly? Yes, you can just use Paste Deploy instead of the ZCML-based pipeline builder. However, the pipeline you get with Paste Deploy will not vary according to the request type, so you won't get the publisher's special handling for non-browser methods like PUT and DELETE, nor XML-RPC support, etc. You can configure multiple pipelines in a Paste Deploy file. It's up to the fronting web server to decide which 'pipeline app' to invoke. OTOH, I'm sure people can find creative ways to make Paste Deploy vary the pipeline according to the request type. The Paste#composite web app basically does this based on inbound subpath. It's not exactly hard to do in WSGI. I am a little worried about the conceptual overhead of having both the ZCA and the WSGI pipeline provide swappability services to application builders. It feels like those two things overlap somewhat in scope. They do overlap, but the ZCA-configured pipeline can easily vary according to the request type. I need to add more info about this to the proposal. If this is the only reason to use the ZCA to register the pipeline, then maybe there's no need: only a fronting piece of middleware that picks the right pipeline. Of course, there may be other things that make using the ZCA attractive. Also, looking at your endware, there are some that seem to overlap with Repoze stuff. Is this on purpose? I think the relationship with Repoze should be made more clear in the proposal. I don't know what the relationship is yet because I've chosen to only get code from Zope, for now, since I'm trying to maintain compatibility. I think it'd be a shame to produce a lot of new middleware that overlaps with the Repoze middleware, when both are born out of Zope. At least, we should try to re-use Repoze stuff wherever we can, rather than roll our own. virtual_host -- is this the same as repoze.vhm? Could be. Not sure. I suspect they are. retry -- is this the same as repoze.retry? I think it might be different, because this version of retry resets the environment rather than the request. It has to be in the pipeline before the create_request application. This design makes it possible to eliminate the complex retry code from Zope requests. Ok. create_request -- should this maybe have some compatibility with WebOb requests? I've looked at WebOb, and my impression is that Zope requests and WebOb requests serve completely different purposes. A Zope request is essentially an input decoder and information holder. A WebOb request is a whole framework, AFAICT. Yes, kind of. I'm not sure using it would make sense for Zope. However, lots of other middleware does use WebOb requests, so it'd be useful to think about and document how they relate, if nothing else. That said, I expect zope.pipeline to give people a greater opportunity to prove I'm wrong about things like that. ;-) switch_pipeline -- could this be made non-Zope specific? It sounds useful. I need to understand better what you mean by non-Zope specific. How would you use it? I'd imagine, to have multiple pipelines set up and have this middleware know which one to invoke. At the end of the day, to one piece of middleware, the 'pipeline' is merely one or possibly many callables that invoke the next piece of middleware/application. open_root -- I thought repoze had something similar, but I may be wrong I think this one is significantly different from Repoze. It would be good to communicate about stuff like this at a sprint. Yes I think they'd be different, reading Tres' reply. clean_transaction -- is this not the same as repoze.tm2? No. To mimic the current Zope publisher, we need to commit the transaction shortly after the call application is finished, but then a lot of things can still happen before the response leaves the server, so we need to make sure any open transaction is aborted before letting the open_root application close the database. Why is it desirable to do things in this way? I do find it kind of confusing/error prone that we have two pieces of middleware: one that opens the transaction and another that closes it, with ordering dependencies between the two. This application is very small. It might make sense to have the open_root application do its job instead. Sure. authenticate -- sounds like repoze.who? I doubt it. This does the IAuthentication dance, allowing authentication to be added during traversal. repoze.who seems to be turning into something of a widely used standard. I think it'd be worth looking into whether it can be used 'upstream' and something else could do the IAuthentication stuff based on what
Re: [Zope-dev] Zope.pipeline proposal
Hanno Schlichting wrote: Shane Hathaway wrote: Roger Ineichen wrote: Do you know something about the performance of WSGI? I whould be happy to see some perfomance tests comparing WSGI with other server concepts. WSGI is extremely lightweight, so WSGI itself isn't going to affect performance. The WSGI servers I know about are reasonably fast. What's the overhead of a WSGI middleware? Is the overhead cost in the same order of magnitude as a simple function call with a return value or is there something inherently more complex going on? No. A WSGI app is basically just a callable. Of course, the app can do lots of crazy things. There may also be some setup cost if you use things like Paste Deploy, but that's a one-time thing. zope.pipeline may of course do lots of crazy things in its apps. ;-) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Martin Aspeli wrote: Shane Hathaway wrote: Martin Aspeli wrote: clean_transaction -- is this not the same as repoze.tm2? No. To mimic the current Zope publisher, we need to commit the transaction shortly after the call application is finished, but then a lot of things can still happen before the response leaves the server, so we need to make sure any open transaction is aborted before letting the open_root application close the database. Why is it desirable to do things in this way? I do find it kind of confusing/error prone that we have two pieces of middleware: one that opens the transaction and another that closes it, with ordering dependencies between the two. I'm trying to express something different, actually. The first app, clean_transaction, only exists to cancel accidental writes. The Zope publisher does this today. The actual transaction management is in the second application, which I now realize is misnamed as end_transaction because it both begins and ends transactions. repoze.who seems to be turning into something of a widely used standard. I think it'd be worth looking into whether it can be used 'upstream' and something else could do the IAuthentication stuff based on what repoze.who does. Ok, that changes my perspective a bit. Shane ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope.pipeline proposal
Roger Ineichen wrote: Hi Tres http://plope.com/whatsitdoing2 This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area. I absolutly agree! As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable. I think you've made Chris' point for him: nothing about the application being tested *requires* that there be a ZODB connection open; Grok's design forces opening one unconditionally, which is a layer of complexity which *can't be turned off.* The conceptual overhead of each of the frameworks is at least as important as the performance overhead. Yes, that's true. But I like to see the real performance win within a *real* application. My applications normaly do not only show a hello world page ;-) I also like to see how much the repoze method calls will grow if the authentication, transaction, ZODB etc is involved. I'm pretty sure that it will much less then 20 times like in the given test. I'll also be happy if we will gain a 2 time perfomance factor in a real comparable test app. I've developed a prototyping package in my own personal repos which tries to setup a site within some objects which I use with the old zope server and a paste WSGI server setup. This let us run performance tests against the ZODB. Probably I should add that to the zope repos and we could start adding all our different publishing implementations. This whould let us compare the same things in a real world use case. While I think that would be a good thing, I do want to mention that it's not really the point of the whatsitdoing benchmark. Repoze stuff typically tries to simplify each component in the set of components used to service a goal, sometimes at the expense of at least some backwards compatibility or excessive configurability. On the other hand, Zope3 and Grok tend to keep backwards compatibility and configurability sometimes at the expense of verbosity and extra runtime expense. They are just somewhat different goals. Repoze stuff therefore has the major advantage of not needing to carry around 6-10 years of backwards compatibility baggage; it owes any 20-20 hindsight in these cases of course directly to Zope. In the meantime, we are proud of the work we've been doing to simplify Zopelike publishing, and the whatsitdoing benchmark was really just a mechanism to toot that particular horn. I'm not sure how to toot that horn without comparing it to Zope, so while I see that it has ruffled some feathers, I'll stand by the benchmark as a useful comparison. I'm pretty sure the stuff would fare pretty well on some other less objective benchmark (like developer productvity as you seem to be advocating). FTR, the publishing implementation you speak of above in the case of repoze is is really a different framework altogether: http://static.repoze.org/bfgdocs/ . - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
