Re: [ZODB-Dev] Re: ZEO and Twisted
David Pratt wrote: Jim, what do you think? In principal, I'd libe to be able to leverage Twised in ZEO's networking layer. I would not want to make it a requirement. It's not something I'd want to see until there was a more basic refactoring. Do you think you could give us a place to experiment in zope's svn? Sure I could. Submit a contributor's ageement and make a branch. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
Rodrigo Dias Arruda Senra wrote: You probably already know all of these, but just in case ... [ Jim Fulton ]: --- | | I am strongly against linking ZEO to an application's main loop. Twisted performs better if the protocols build upon it rely on its reactor (asynchronous loop). That means breaking long computations to use promises (deferreds in Twisted vocabulary), therefore increasing the interleaving of data handlers. > Implementing ZEO in Twisted would probably isolate the zeo protocol from the underlying transport (tcp, udp, whatever). OTOH, the implementation would be dependent on the typical deferred patterns (only common in Twisted nowadays). Twisted also support threads, but that defeats most of the other benefits of using Twisted in the first place (like not bothering about critical sections and other synch stuff), so they are just used when blocking calls must be made. I'm familiar with how Twisted works, ZEO currently uses asyncore, which is similar in philosophy. They are both based on the reactor pattern, which doesn't really fit ZODB very well. There are operations that can take a long time. In Zope, we got around this by interfacing a separate thread pool with the asyncore main loop. ZODB will never use Twisted's computational model internally. Forget it. It's not on the table. It's not even in the same room. That's not to say that I'd be opposed to leveraging Twisted for ZEO's networking. This would be in a dedicated thread just for this I/O. This would be separate from any main loop the application might run. | If twisted supports using multiple independent main loops, then it | would be an option. I know it supports multiple implementations of reactors, each specialized for a particular purpose, for instance to conciliate two different loops into a single reactor, which is the case of twisted_networking_loop + GUI_event_loop. Therefore, it would be possible (my educated guess) to create a zeoreactor, would that suffice ? That was done to marry Twisted to: gtk, gtk2, glib2, wxPython, win32event loops. I would want to use a dedicated loop just for ZEO's networking. In fact, if an application used multiple ZEO storages, it would have a separate loop for each storage. This would be in addition to any main loop the application might use for it's own purposes. ... [ David Pratt ]: - | > Is there is any strong opposition to using the twistd daemon for a | > twisted zeo service? I volunteer to help in that effort to the best of my modest skills and limited (just like everybody else) spare time. Subject to what I've said above, that would be great some time in the future. I'd be happy to have Twisted as an option. But before I'd consider that, there really needs to be a more basic refactoring of the ZEO networking architecture. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
Hi Rodrigo! Great post on this. You provide an insightful and compelling summary of what can be accomplished. Add on top of this, the things that twisted "just does" like SSL or SSH out of the box and we can also have the security issue beat. Jim, what do you think? Do you think you could give us a place to experiment in zope's svn? Regards, David Rodrigo Dias Arruda Senra wrote: You probably already know all of these, but just in case ... [ Jim Fulton ]: --- | | I am strongly against linking ZEO to an application's main loop. Twisted performs better if the protocols build upon it rely on its reactor (asynchronous loop). That means breaking long computations to use promises (deferreds in Twisted vocabulary), therefore increasing the interleaving of data handlers. Implementing ZEO in Twisted would probably isolate the zeo protocol from the underlying transport (tcp, udp, whatever). OTOH, the implementation would be dependent on the typical deferred patterns (only common in Twisted nowadays). Twisted also support threads, but that defeats most of the other benefits of using Twisted in the first place (like not bothering about critical sections and other synch stuff), so they are just used when blocking calls must be made. | If twisted supports using multiple independent main loops, then it | would be an option. I know it supports multiple implementations of reactors, each specialized for a particular purpose, for instance to conciliate two different loops into a single reactor, which is the case of twisted_networking_loop + GUI_event_loop. Therefore, it would be possible (my educated guess) to create a zeoreactor, would that suffice ? That was done to marry Twisted to: gtk, gtk2, glib2, wxPython, win32event loops. | Then the possible issues are: | | - Whether we want Twisted to be a dependency of ZEO If it comes to that, probably just TwistedCore would be required, and perhaps even a subset. | - Performance There is a lot of this-and-that [1] about what communication pattern performs better. Twisted is definetely better for long sessions, and not necessarily much worse for short-termed sessions (where "people" say thread-pools are much sexier). IMVHO, the strong argument behind Twisted adoption would be flexibility and not huge speedups. [ David Pratt ]: - | > Is there is any strong opposition to using the twistd daemon for a | > twisted zeo service? I volunteer to help in that effort to the best of my modest skills and limited (just like everybody else) spare time. [1] http://www.cs.wustl.edu/~schmidt/patterns-ace.html cheers, Senra - Rodrigo Senra GPr Sistemas ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
You probably already know all of these, but just in case ... [ Jim Fulton ]: --- | | I am strongly against linking ZEO to an application's main loop. Twisted performs better if the protocols build upon it rely on its reactor (asynchronous loop). That means breaking long computations to use promises (deferreds in Twisted vocabulary), therefore increasing the interleaving of data handlers. Implementing ZEO in Twisted would probably isolate the zeo protocol from the underlying transport (tcp, udp, whatever). OTOH, the implementation would be dependent on the typical deferred patterns (only common in Twisted nowadays). Twisted also support threads, but that defeats most of the other benefits of using Twisted in the first place (like not bothering about critical sections and other synch stuff), so they are just used when blocking calls must be made. | If twisted supports using multiple independent main loops, then it | would be an option. I know it supports multiple implementations of reactors, each specialized for a particular purpose, for instance to conciliate two different loops into a single reactor, which is the case of twisted_networking_loop + GUI_event_loop. Therefore, it would be possible (my educated guess) to create a zeoreactor, would that suffice ? That was done to marry Twisted to: gtk, gtk2, glib2, wxPython, win32event loops. | Then the possible issues are: | | - Whether we want Twisted to be a dependency of ZEO If it comes to that, probably just TwistedCore would be required, and perhaps even a subset. | - Performance There is a lot of this-and-that [1] about what communication pattern performs better. Twisted is definetely better for long sessions, and not necessarily much worse for short-termed sessions (where "people" say thread-pools are much sexier). IMVHO, the strong argument behind Twisted adoption would be flexibility and not huge speedups. [ David Pratt ]: - | > Is there is any strong opposition to using the twistd daemon for a | > twisted zeo service? I volunteer to help in that effort to the best of my modest skills and limited (just like everybody else) spare time. [1] http://www.cs.wustl.edu/~schmidt/patterns-ace.html cheers, Senra - Rodrigo Senra GPr Sistemas ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
Hi Jim. My experience has been in running a single selectreactor in an app together with its own main loop. I found a patch on google to allow multiple selectreactors but have not seen the patch applied to svn or attempted the change in twisted as yet. I have emailed the maintainer to ask about it. I was looking at the zeo server service first which should be happy to run on its own. There is no risk at this stage to experiment with my ideas other than learning more than I did before about twisted and ZEO. Regards, David Jim Fulton wrote: David Pratt wrote: ... Is there is any strong opposition to using the twistd daemon for a twisted zeo service? I am strongly against linking ZEO to an application's main loop. If twisted supports using multiple independent main loops, then it would be an option. Then the possible issues are: - Whether we want Twisted to be a dependency of ZEO - Performance Switching to twisted would be a big change. If I was to make such a change, which I anticipate, it would be great if twised could be an option. Jim ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
David Pratt wrote: ... Is there is any strong opposition to using the twistd daemon for a twisted zeo service? I am strongly against linking ZEO to an application's main loop. If twisted supports using multiple independent main loops, then it would be an option. Then the possible issues are: - Whether we want Twisted to be a dependency of ZEO - Performance Switching to twisted would be a big change. If I was to make such a change, which I anticipate, it would be great if twised could be an option. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
Florent Guillaume wrote: On 25 Apr 2006, at 15:23, Jim Fulton wrote: Florent Guillaume wrote: [Ccing zodb-dev] On 25 Apr 2006, at 15:09, David Pratt wrote: The protocol is simple yes, but the iteractions w.r.t threading are sometimes subtle. Hi Florent. This could be set up using a twisted's application object so that you have a twisted app using a twisted .tac. I guess this could still use the .conf as in this but twisted already has a means of daemonizing its services with twisted. This would give you something like this to start up zeo. twistd -y zeo.tac What are your thoughts? Huh, I thought you were talking about the ZEO client, ClientStorage, not the ZEO server. For the ZEO server I don't see the point of changing it, it works well. OTOH a ClientStorage has to integrate with the other servers in Zope, and that's the one that would benefit from being moved to the twisted event loop if twisted is used. (I'm mostly not paying attention to this thread but I have to speak up.) I disagree. ZEO should not be dependent on an application main loop at all. It was a mistake to couple ZEO's networking at an application's asyncore main loop. This *greatly* complicated ZEO's implementation. I'd much rather that ZEO's networking be independent of an application. Fortunately, asyncore allows multiple independent main loops. Does Twisted? When I propose to move it to the twisted event loop, I mean refactor it so that you can plug it into whatever main loop you want, with nice entry points. Today the code that has to switch depending on whether TheadedAsync is available or not, for example, is horrible. Well, first, making ZEO's networking that pluggable would be a very major refactoring. The twisted and syncore apis are quite different. Second, you don't want to plug into the application's main loop because you generally want to do database operations before the main loop has started. That is the reason for the horrible code you mention. IMO, it would be far better to give the database it's own main loop. Then you don't need to plug into anything. This is my long-term plan for ZEO. I also do contemplate a major change to ZEO's networking code. The main reason is to make it more testable. This could also make it more pluggable. Sadly, it will be a long time before I have time to work on this. :( Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
On 25 Apr 2006, at 15:23, Jim Fulton wrote: Florent Guillaume wrote: [Ccing zodb-dev] On 25 Apr 2006, at 15:09, David Pratt wrote: The protocol is simple yes, but the iteractions w.r.t threading are sometimes subtle. Hi Florent. This could be set up using a twisted's application object so that you have a twisted app using a twisted .tac. I guess this could still use the .conf as in this but twisted already has a means of daemonizing its services with twisted. This would give you something like this to start up zeo. twistd -y zeo.tac What are your thoughts? Huh, I thought you were talking about the ZEO client, ClientStorage, not the ZEO server. For the ZEO server I don't see the point of changing it, it works well. OTOH a ClientStorage has to integrate with the other servers in Zope, and that's the one that would benefit from being moved to the twisted event loop if twisted is used. (I'm mostly not paying attention to this thread but I have to speak up.) I disagree. ZEO should not be dependent on an application main loop at all. It was a mistake to couple ZEO's networking at an application's asyncore main loop. This *greatly* complicated ZEO's implementation. I'd much rather that ZEO's networking be independent of an application. Fortunately, asyncore allows multiple independent main loops. Does Twisted? When I propose to move it to the twisted event loop, I mean refactor it so that you can plug it into whatever main loop you want, with nice entry points. Today the code that has to switch depending on whether TheadedAsync is available or not, for example, is horrible. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZEO and Twisted
Florent Guillaume wrote: [Ccing zodb-dev] On 25 Apr 2006, at 15:09, David Pratt wrote: The protocol is simple yes, but the iteractions w.r.t threading are sometimes subtle. Hi Florent. This could be set up using a twisted's application object so that you have a twisted app using a twisted .tac. I guess this could still use the .conf as in this but twisted already has a means of daemonizing its services with twisted. This would give you something like this to start up zeo. twistd -y zeo.tac What are your thoughts? Huh, I thought you were talking about the ZEO client, ClientStorage, not the ZEO server. For the ZEO server I don't see the point of changing it, it works well. OTOH a ClientStorage has to integrate with the other servers in Zope, and that's the one that would benefit from being moved to the twisted event loop if twisted is used. (I'm mostly not paying attention to this thread but I have to speak up.) I disagree. ZEO should not be dependent on an application mail loop at all. It was a mistake to couple ZEO's networking at an application's asyncore main loop. This *greatly* complicated ZEO's implementation. I'd much rather that ZEO's networking be independent of an application. Fortunately, asyncore allows multiple independent main loops. Does Twisted? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
