Re: [ZODB-Dev] Re: ZEO and Twisted

2006-05-01 Thread Jim Fulton

Rodrigo Dias Arruda Senra wrote:

You probably already know all of these, but just in case ...

[ Jim Fulton ]:
---
|
|  I am strongly against linking ZEO to an application's main loop.

Twisted performs better if the protocols build upon it rely on its
reactor (asynchronous loop). That means breaking long computations
to use promises (deferreds in Twisted vocabulary), therefore increasing
the interleaving of data handlers.



Implementing ZEO in Twisted would probably isolate the zeo protocol
from the underlying transport (tcp, udp, whatever). OTOH, the
implementation would be dependent on the typical deferred patterns
(only common in Twisted nowadays). Twisted also support threads, but
that defeats most of the other benefits of using Twisted in the first
place (like not bothering about critical sections and other synch
stuff), so they are just used when blocking calls must be made.


I'm familiar with how Twisted works,  ZEO currently uses asyncore,
which is similar in philosophy.  They are both based on the reactor
pattern, which doesn't really fit ZODB very well.  There are operations
that can take a long time.  In Zope, we got around this by interfacing
a separate thread pool with the asyncore main loop.

ZODB will never use Twisted's computational model internally.  Forget
it.  It's not on the table.  It's not even in the same room.

That's not to say that I'd be opposed to leveraging Twisted for
ZEO's networking.  This would be in a dedicated thread just for
this I/O.  This would be separate from any main loop the application
might run.



|  If twisted supports using multiple independent main loops, then it
| would be an option.

I know it supports multiple implementations of reactors, each
specialized for a particular purpose, for instance to conciliate
two different loops into a single reactor, which is the case of
twisted_networking_loop + GUI_event_loop.
Therefore, it would be possible (my educated guess) to create a
zeoreactor, would that suffice ?  


That was done to marry Twisted to: gtk, gtk2, glib2, wxPython,
win32event loops.


I would want to use a dedicated loop just for ZEO's networking.
In fact, if an application used multiple ZEO storages, it would have
a separate loop for each storage.  This would be in addition to
any main loop the application might use for it's own purposes.

...



[ David Pratt ]:
-
|   Is there is any strong opposition to using the twistd daemon for a 
|   twisted zeo service?


I volunteer to help in that effort to the best of my modest skills
and limited (just like everybody else) spare time.


Subject to what I've said above, that would be great some time
in the future.  I'd be happy to have Twisted as an option.
But before I'd consider that, there really needs to be a more
basic refactoring of the ZEO networking architecture.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-05-01 Thread Jim Fulton

David Pratt wrote:

Jim, what do you think?


In principal, I'd libe to be able to leverage Twised in ZEO's
networking layer.  I would not want to make it a requirement.
It's not something I'd want to see until there was a more basic
refactoring.

Do you think you could 
give us a place to experiment in zope's svn?


Sure I could. Submit a contributor's ageement and make a branch.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread Jim Fulton

Florent Guillaume wrote:

[Ccing zodb-dev]

On 25 Apr 2006, at 15:09, David Pratt wrote:

The protocol is simple yes, but the iteractions w.r.t threading  are 
sometimes subtle.



Hi Florent.  This could be set up using a twisted's application  
object so that you have a twisted app using a twisted .tac. I guess  
this could still use the .conf as in this but twisted already has a  
means of daemonizing its services with twisted. This would give you  
something like this to start up zeo.


twistd -y zeo.tac

What are your thoughts?



Huh, I thought you were talking about the ZEO client, ClientStorage,  
not the ZEO server.

For the ZEO server I don't see the point of changing it, it works well.
OTOH a ClientStorage has to integrate with the other servers in Zope,  
and that's the one that would benefit from being moved to the twisted  
event loop if twisted is used.


(I'm mostly not paying attention to this thread but I have to speak up.)

I disagree.  ZEO should not be dependent on an application mail loop
at all.  It was a mistake to couple ZEO's networking at an application's
asyncore main loop.  This *greatly* complicated ZEO's implementation.
I'd much rather that ZEO's networking be independent of
an application.  Fortunately, asyncore allows multiple independent main loops.
Does Twisted?

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread Florent Guillaume

On 25 Apr 2006, at 15:23, Jim Fulton wrote:

Florent Guillaume wrote:

[Ccing zodb-dev]
On 25 Apr 2006, at 15:09, David Pratt wrote:
The protocol is simple yes, but the iteractions w.r.t threading   
are sometimes subtle.



Hi Florent.  This could be set up using a twisted's application   
object so that you have a twisted app using a twisted .tac. I  
guess  this could still use the .conf as in this but twisted  
already has a  means of daemonizing its services with twisted.  
This would give you  something like this to start up zeo.


twistd -y zeo.tac

What are your thoughts?
Huh, I thought you were talking about the ZEO client,  
ClientStorage,  not the ZEO server.
For the ZEO server I don't see the point of changing it, it works  
well.
OTOH a ClientStorage has to integrate with the other servers in  
Zope,  and that's the one that would benefit from being moved to  
the twisted  event loop if twisted is used.


(I'm mostly not paying attention to this thread but I have to speak  
up.)


I disagree.  ZEO should not be dependent on an application main loop
at all.  It was a mistake to couple ZEO's networking at an  
application's

asyncore main loop.  This *greatly* complicated ZEO's implementation.
I'd much rather that ZEO's networking be independent of
an application.  Fortunately, asyncore allows multiple independent  
main loops.

Does Twisted?


When I propose to move it to the twisted event loop, I mean refactor  
it so that you can plug it into whatever main loop you want, with  
nice entry points.
Today the code that has to switch depending on whether TheadedAsync  
is available or not, for example, is horrible.


Florent

--
Florent Guillaume, Nuxeo (Paris, France)   Director of RD
+33 1 40 33 71 59   http://nuxeo.com   [EMAIL PROTECTED]


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread Jim Fulton

Florent Guillaume wrote:

On 25 Apr 2006, at 15:23, Jim Fulton wrote:


Florent Guillaume wrote:


[Ccing zodb-dev]
On 25 Apr 2006, at 15:09, David Pratt wrote:

The protocol is simple yes, but the iteractions w.r.t threading   
are sometimes subtle.




Hi Florent.  This could be set up using a twisted's application   
object so that you have a twisted app using a twisted .tac. I  
guess  this could still use the .conf as in this but twisted  
already has a  means of daemonizing its services with twisted.  This 
would give you  something like this to start up zeo.


twistd -y zeo.tac

What are your thoughts?


Huh, I thought you were talking about the ZEO client,  
ClientStorage,  not the ZEO server.

For the ZEO server I don't see the point of changing it, it works  well.
OTOH a ClientStorage has to integrate with the other servers in  
Zope,  and that's the one that would benefit from being moved to  the 
twisted  event loop if twisted is used.



(I'm mostly not paying attention to this thread but I have to speak  up.)

I disagree.  ZEO should not be dependent on an application main loop
at all.  It was a mistake to couple ZEO's networking at an  application's
asyncore main loop.  This *greatly* complicated ZEO's implementation.
I'd much rather that ZEO's networking be independent of
an application.  Fortunately, asyncore allows multiple independent  
main loops.

Does Twisted?



When I propose to move it to the twisted event loop, I mean refactor  it 
so that you can plug it into whatever main loop you want, with  nice 
entry points.
Today the code that has to switch depending on whether TheadedAsync  is 
available or not, for example, is horrible.


Well, first, making ZEO's networking that pluggable would be a very
major refactoring.  The twisted and syncore apis are quite
different.

Second, you don't want to plug into the application's main loop
because you generally want to do database operations before the
main loop has started.  That is the reason for the horrible code
you mention.  IMO, it would be far better to give the database it's
own main loop.  Then you don't need to plug into anything. This is
my long-term plan for ZEO.

I also do contemplate a major change to ZEO's networking code. The
main reason is to make it more testable.  This could also make it
more pluggable.  Sadly, it will be a long time before I have time to
work on this. :(

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread Jim Fulton

David Pratt wrote:
...
Is there is any strong opposition to using the twistd daemon for a 
twisted zeo service?


I am strongly against linking ZEO to an application's main loop.

If twisted supports using multiple independent main loops, then it would
be an option.

Then the possible issues are:

- Whether we want Twisted to be a dependency of ZEO

- Performance

Switching to twisted would be a big change.  If I was to make such a
change, which I anticipate, it would be great if twised could be an
option.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread Rodrigo Dias Arruda Senra

You probably already know all of these, but just in case ...

[ Jim Fulton ]:
---
|
|  I am strongly against linking ZEO to an application's main loop.

Twisted performs better if the protocols build upon it rely on its
reactor (asynchronous loop). That means breaking long computations
to use promises (deferreds in Twisted vocabulary), therefore increasing
the interleaving of data handlers.

Implementing ZEO in Twisted would probably isolate the zeo protocol
from the underlying transport (tcp, udp, whatever). OTOH, the
implementation would be dependent on the typical deferred patterns
(only common in Twisted nowadays). Twisted also support threads, but
that defeats most of the other benefits of using Twisted in the first
place (like not bothering about critical sections and other synch
stuff), so they are just used when blocking calls must be made.
  

|  If twisted supports using multiple independent main loops, then it
| would be an option.

I know it supports multiple implementations of reactors, each
specialized for a particular purpose, for instance to conciliate
two different loops into a single reactor, which is the case of
twisted_networking_loop + GUI_event_loop.
Therefore, it would be possible (my educated guess) to create a
zeoreactor, would that suffice ?  

That was done to marry Twisted to: gtk, gtk2, glib2, wxPython,
win32event loops.

|  Then the possible issues are:
|  
|  - Whether we want Twisted to be a dependency of ZEO

If it comes to that, probably just TwistedCore would be required,
and perhaps even a subset.
 
|  - Performance

There is a lot of this-and-that [1] about what communication pattern
performs better. Twisted is definetely better for long sessions, and
not necessarily much worse for short-termed sessions (where people
say thread-pools are much sexier). IMVHO, the strong argument behind
Twisted adoption would be flexibility and not huge speedups. 


[ David Pratt ]:
-
|   Is there is any strong opposition to using the twistd daemon for a 
|   twisted zeo service?

I volunteer to help in that effort to the best of my modest skills
and limited (just like everybody else) spare time.

[1] http://www.cs.wustl.edu/~schmidt/patterns-ace.html

cheers,
Senra

-
Rodrigo Senra
GPr Sistemas
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev