[Zope-dev] Re: [medusa] Twisted, medusa, ZServer, and VFS's
I'm trying to get selective with the cross-posting based on what the thread is focusing on... On Wed, Oct 10, 2001 at 08:48:37AM -0500, Skip Montanaro wrote: > > > Donovan> ZServer grew out of Medusa I'm not sure how tightly tied to > Donovan> Zope it is > > Not tightly at all. I use it completely independent of Zope. Hmm, what for? How? What are you serving out of? > Donovan> My problem with Medusa is its http and ftp servers assume that > Donovan> the VFS can deliver files wrapped in producers without > Donovan> blocking. > > I believe Sam Rushing has said all along that Medusa is intended for > I/O-bound situations. If you have producers that are computationally bound > or that do I/O outside of the Medusa framework, you'll suffer. The problem is the existing http and ftp servers use file producers to serve content, and the framework assumes producers don't block. There is no way to make a producer that does I/O _inside_ the Medusa async framework, at least not without using my ready() patch. The proper way to fix this is to change the ftp and http server so they don't use file producers, instead they use file dispatchers to push async data onto the http/ftp dispatcher. -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key -- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Twisted, medusa, ZServer, and VFS's
G'day, I've just been looking at Twisted Python for the past hour or two, read the mailing list archives, and have some comments and questions. I've been working on using Medusa for serving a virtual mirror via http and ftp. I'm at the point where I'm close, but I'm starting to re-think some stuff, in particular my choice of Medusa. First my impressions of the three contenders; Twisted, Medusa, ZServer. Please correct me if I'm wrong in the following summaries; Medusa seems to be the daddy of them all. It's the oldest, which has benefits and problems. It is a little messy from its evolution, but seems pretty mature. It uses an async select loop to drive everything. It uses "asynchat" derived class objects to communicate on sockets. Data can be sent by pushing "producers" onto asynchat objects. Producers can be complex objects that produce data, execute callbacks, whatever. It's ftp and http server classes use a primative VFS to serve from. The asyncore and asynchat modules it is built on are now part of Python. Twisted seems to be a from-the-ground up re-invention of Medusa. It's newer, but surprisingly it's bigger, dispite it's apparently less mature feature set. It is similar in structure to Medusa, but simplifys it by dispensing with producers. It can use a variety of event-loops, including Tk and GTK, or it's own. It doesen't have a VFS (yet) so its ftp and http servers serve from the underlying os filesystem. ZServer grew out of Medusa. It uses the same basic underlying architecture, but throws in threads to get around the problem of delayed producers blocking the event loop. I'm not sure how tightly tied to Zope it is, but its http and ftp servers generally serve from a ZODB database, presumably wrapped in a Medusa VFS, though I have a feeling they might have changed that. ZServer also supports webDAV serving. It is possible that some of the enhancements could be merged back into Medusa, but probably it has changed so much it would be difficult. My problem with Medusa is its http and ftp servers assume that the VFS can deliver files wrapped in producers without blocking. I've fixed this by creating a patch for Medusa's asynchat that adds support for a ready() method to producers, so they can block without blocking the event loop. I'm currently in the process of writing Medusa VFS's for ftp and eventualy http backends. In the process I've also found that the Medusa ftp server is not as full featured as I want. In my search for Python VFS's I found PyVFS (http://www.pycage.de/). This is modelled on the Gnome/MC VFS, so it supports '/dir/somefile.tar.gz#tgz:/somepath' style paths to look inside tar, tgz, ftp, whatever. The various different VFS backends are loaded dynamicly as pluggins. These pluggins execute as a seperate process that are communicated with over a channel. The API is too simplistic for me, with files being "projected" out of the VFS to a local file to be manipulated/used. I don't like the pluggin-process-channel architecture either. In the reading of the Twisted mailing list, I saw a comment to the affect that the Medusa VFS was an example of how _not_ to do it, which lead to using webDAV as the API for a VFS. My gut feeling is DAV is a cool protocol for a VFS backend, but I dunno about using it as the primary API. Sure, it supports meta- data etc, but the reality is the API that is most widely used and understood is the POSIX filesytem API, as exposed in Python by the os and os.path modules. My solution for a VFS has been, upto now, based on Medusa's, but extending it to be more like os and os.path. So far it's a filesystem class with most of the os and os.path methods. One of the derived classes is a mountable_filesystem that allows you to mount other VFS filesystems off it. At this point I'm tempted to make a vfs module that emulates os and os.path so that you can mount whatever vfs's you want first, and then just replace all your os.* calls with vfs.* calls. Note that the one catch would be open() would need to be replaced with vfs.open(). I'm sort of fishing for general suggestions, comments, and interest. I'm at the point where I've just convinced myself my vfs is worth finishing, and my ready () patch to asynchat is worth updating, but I'm not sure what to use as the http and ftp server front-end, though I'm still leaning towards medusa. It looks like Twister is not ready, and ZServer would be too hard to seperate from Zope. PS... I'm not on the zope-dev list but I am on the twister and medusa lists. The zope-dev list is too much non-ZServer stuff and that's all I'm interested in. So zope-dev'ers, please reply to me or one/both of the other lists directly. -- ABO: finger [EMAIL PROTECTED] for more information. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lis
Re: [Zope-dev] asynchat producer ready() method patch.
On Sun, Jan 28, 2001 at 09:31:07PM +0100, Dieter Maurer wrote: > Donovan Baarda writes: > > I remember reading ages ago on a wishlist for ZServer that someone was > > thinking of adding blocking support to producers. This patch adds support > > for just that in a non-intrusive way. I haven't tried applying this patch to > > ZServer, but it _should_ work. > What does "blocking support dor producers" mean? Disclaimer: you must be moderately familiar with ZServer/medusa/asynchat for the following to make sense; Asynchat used by ZServer and medusa use producers as a way to "push" data through an async interface. The current version of asynchat requires that producers be able to produce data immediately on calling their more() method, untill they are finished, when they return ''. My patch adds support for producers being able to "block" by letting asynchat know if they are ready via a ready() method. This allows you to use the simple producer model to push data that may not be immediately available. An example would be a http-fetch producer that fetches data from upstream for a proxy. This patch is only useful for people creating or modifying things like medusa or ZServer. -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key -- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: asynchat producer ready() method patch.
On Sat, Jan 27, 2001 at 09:30:16PM +1100, abo wrote: > G'day zope-dev, > > I'm not subscribed to the zope-dev list (got enough email thanks :-), but I > thought some of you might be interested in the following patch to medusa > that I sent to the medusa list (ZServer is based on medusa). [...] whups... didn't attach the patch. Rather than bombard everyone with a 10k patch, it can be found at; ftp://minkirri.apana.org.au/pub/python/medusa-ready_0.3.patch.bz2 -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key -- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] asynchat producer ready() method patch.
G'day zope-dev, I'm not subscribed to the zope-dev list (got enough email thanks :-), but I thought some of you might be interested in the following patch to medusa that I sent to the medusa list (ZServer is based on medusa). I remember reading ages ago on a wishlist for ZServer that someone was thinking of adding blocking support to producers. This patch adds support for just that in a non-intrusive way. I haven't tried applying this patch to ZServer, but it _should_ work. The two things to look out for when applying these changes to older programs that use asynchat.py and producers.py are; old producers with existing and conflicting ready() methods, anything that uses the compound_producer because the __init__ method has been simplified. I hope someone finds this useful, and would appreciate any feedback. Thanks, ABO - Forwarded message from Donovan Baarda <[EMAIL PROTECTED]> - G'day, I've recieved one note of interest in my patch for medusa to allow producers to block using a ready() method. The gist was "this is a good idea, but it doesn't work". At the time I was focusing on the ftp server part of medusa, so I hadn't checked the http side much at all. I also noticed that asyncore.py and asynchat.py were distributed without producers.py in python itself. My original changes made asynchat.py dependant on producers.py, which was probably a bad idea. So I have gone right over my changes and produced a new, more thoroughly tested patch that tries to minimize the side affects. I have tested the changes as they are incrementaly applied to ensure that my changes don't break old code that doesn't use the ready() feature. This patch contains all changes made to all files, but you only need to accept the changes to some files to use ready() The main change is to asynchat.py, where support for the ready() method has been added. Also added are a variety of producer base classes that allow new producers with ready() support to be easily derived (and removed the asynchat.py dependancy on producers.py that my early patch introduced). The main async_chat class is actualy significantly simpler for the change, mostly due to replacing the fifo class with a fifo_producer class. The find_prefix_at_end procedure for finding terminators has also been simplified and speeded up. Overall, asynchat.py is only 23 lines bigger dispite all the additional functionality. The good news is, the changes to asynchat.py are fully backwards compatible and medusa runs fine with all other files unchanged, _except_ that filesys.py and status_handler.py have conflicting ready() methods in producers. These ready() methods are unused in the original medusa, but cause the new asynchat.py to think the producers never complete. The patch to these two files simply removes these ready() methods so medusa can run with the new asynchat. The next lot of serious changes are to producers.py and http_server.py. These changes are not needed to run medusa as it stands, but are needed if you start using blocking producers. The changes primarily make the producers inherited from the base producer classes defined in asynchat.py. This ensures that producers that wrap other producers will block correctly. The inheritance of features makes producers.py 62 lines smaller than it was. All the changes to producers.py are backwards compatible _except_ the compound_producer.__init__() method now takes a python list of producers not a fifo. The changes to http_server.py take this into account. The final trivial change is a commented out alternative to use the new async_producer instead of file_producer in ftp_server.py. This allows medusa filesystems to return file types that could block without blocking the whole server. I hope someone gets a chance to try these out, and I would appreciate any form of feedback. I'm tempted to submit my patch to asynchat.py direct to the Python people for inclusion in Python 2.1, but would like to hear more feedback before I do. BTW, ignore the patch to "medusa.prj"... that is a side affect of using prcs to revision control my changes. This patch is to the 2601 version of medusa. -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key -- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
