Re: [ZODB-Dev] Storage Interfaces
Jim Fulton wrote: On Apr 27, 2007, at 9:30 PM, David Pratt wrote: Hi Jim. I have been quite interested in this also so have experimented and continue on this also. The way I am doing the networking it similar to ngi I have put this together before you defined these interfaces. I'm not sure which this you're referring to. I assume you mean using Twisted for ZEO. I don't see that that has much to do with the new storage interfaces. Yes, Twisted for ZEO. No, I was referring to the interfaces you were setting up - the interfaces to ngi. In what way is your networking similar to ngi? The interfaces are similar. The system I am putting together is pretty symmetrical and I like the language which is more consistent with twisted's use of factories. You were using this language before in your initial proposal to ngi but many of the interfaces are now name differently named. See how nice and symmetrical this looks :-) clientconnfactoryhandler clientconnhandler clientconnfactory clientreconnfactory clientmethods clientstorage connection serverconnfactory serverconnhandler serverconnfactoryhandler servermethods serverstorage Note that I wrote ngi based on a misunderstanding of Twisted. :( I didn't realize that Twisted application code didn't touch sockets and therefore could be tested without using sockets, threads or subprocesses. IMO, this is Twisted's most important feature. I recently finished my first Twisted application and I was very pleased with my ability to create sane tests for it, although it's test support infrastructure could use improvement, which I plan to do. I have got a basic transport to use prospective broker and banana protocol. So a client and server but I have not yet put together the methods for interacting with the storage. I plan to stick with the existing ZEO protocol, both for compatibility and for performance reasons. Banana protocol is efficient and extensible. It has its own producer/consumer so no need to roll you own system as you have for ZEO. I think you will find banana an efficient serialization of objects. As far as compatibility, I am not sure the past always needs to equal the future. It is probably the methods for zrpc that are important, not the transport - at least that is my thinking. This is the obvious downside of ZEO at the moment is that is too tied to asyncore. In fact, I started by trying to replicate much of what was in ZEO - then I threw it away because something simpler I believe is possible. If a new system can transport the objects and execute the methods, isn't that all it is supposed to do? On top of it, what if something better comes around - the methods that you are executing on both sides are the important part and the transport should be transparent. I think this could produce a much cleaner and understandable system. On top of it, it doesn't upset anything with current zeo for users that currently use this. It could in fact be developed and run in parallel, not that you would want to run it that way :-). Another obvious advantage is that twisted project has tests for most of this already so that is also a big plus. Out of curiosity, does perspective broker support one-way calls, messages sent without replies? Scanning the docs, I can't tell. Of course, not waiting for replies is inherent in Twisted's defered model I suppose. Yes To set up something to work with I create both a TCP and SSL clientfactory so I bring them into the multiservice when zope is started. This way you do not need a separate loop to run the service which is the current way zeo runs. At the same time this does not interfere with the current zope setup. I have no proof one way or the other that this will not work as efficiently on a single twisted loop with the second asyncore loop. I still have to hook up database methods and I had put this on a backburner for a while since I have had other things to do. In any case, the client setup in zope requires only minor modification of the zope.app.twisted main.py since the script only deals with servers and not clients at the moment. The clients are reconnecting clients so will continue to try reconnecting if connection is lost in gradually increasing time intervals. On the server side I have the server working using zconfig and zdaemon in a similar way to way it is setup for zope itself. I have been looking at the zope3recipes since I am working on getting this into a buildout to continue since I am quite happy with the way buildouts work. Regards, David ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage Interfaces
Bottom line is that storage methods need to be executed on both sides. I don't think the storages need to care what or how objects are transported nor what version of zope you are using so long as in the end, the object is one that can be used by a storage method - and that the storage methods are compatible with the older zopes (which seems to be what you are concerned about). At least this is how I am looking at it - whether it is zope 2 or 3, it should not matter if they are methods that can act on ZODB and are suited to the zodb version. In any case, thats the direction I have been going. On top of it, who wants a transport that does only one thing, that can get tired pretty fast. There are many more possibilities than exchanging zodb data using a more generic transport. That's the other key reason I tossed out my attempt to replicate ZEO directly using twisted - if you want to think a bit beyond ZEO. Regards, David Jim Fulton wrote: On Apr 28, 2007, at 9:13 AM, David Pratt wrote: ... As far as compatibility, I am not sure the past always needs to equal the future. It is probably the methods for zrpc that are important, not the transport - at least that is my thinking. I can use current ZEO servers with ZODB 3.2 clients running in very old versions of Zope (2.6 and 2.7). This is extremely useful to me at this point. I couldn't do this if I changed the wire protocol. Jim -- Jim Fultonmailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporationhttp://www.zope.comhttp://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage Interfaces
On Apr 27, 2007, at 10:21 AM, Stephan Richter wrote: On Thursday 26 April 2007 19:33, Jim Fulton wrote: Comments on the APIs are very welcome. Hopefully this will clarify some things. I have read the checkin messages with interest. In the next month or so I will probably need to implement a storage that encrypts its data, so I will get a taste of the new API then. I'll make sure to use the trunk. :-) I assume/home this will be a proxy, although I'll have more to say about proxies. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storage Interfaces
On 4/27/07, Jim Fulton [EMAIL PROTECTED] wrote: On Apr 27, 2007, at 10:21 AM, Stephan Richter wrote: ... In the next month or so I will probably need to implement a storage that encrypts its data, so I will get a taste of the new API then. I'll make sure to use the trunk. :-) I suspect that Stephan is talking about encrypted storage, and Jim is talking about encrypted transport? Of course you probably need both. :) -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Storage Interfaces
I've just checked in a fair bit of re-factoring to try to define reasonable storage interfaces. The results can be found at: http://svn.zope.org/ZODB/trunk/src/ZODB/interfaces.py?view=auto and http://svn.zope.org/ZODB/trunk/src/ZEO/interfaces.py?view=auto Some notes: - I added a new tpc_transaction method to allow ZEO to get some information it needs without groping for _transaction. Storage servers will still work with storages that don't have this method, but will log problems. Current storage implementations should add this method. - I'm not sure whether history, loadSerial, pack, and registerDB, should be in IStorage. I have a feeling that packing and other database-management tasks should be exposed some other way. history and loadSerial are fairly special, although they can be wildly useful. (IMO, there should be a history view for all Zope objects that lets one at least see the transaction history for an object even if looking at old data isn't supported.) Only very specialized storages need registerDB AFAICT. - See my earlier message about getExtensionMethods. - I was able to retire a poorly documented and understood method that was only needed by ZEO: loadEx. - Intentionally included version-support methods from the APIs. - I arranged that storages that don't support undo or versions don't need to provide any of the methods. I cleaned up various other sundry things along the way and identified some bugs, fixing at least one. Comments on the APIs are very welcome. Hopefully this will clarify some things. One of my text tasks will be to add some documentation about concurrency requirements. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
