Baruch Even wrote: >I'm interested in packaging mod_webkit for Debian, I know >of the effort of Federico Di Gregorio to package the whole >of webware for Debian, but for now I understand this >attempt is delayed until webware will switch to some >directory layout and installation method that is easier to >integrate with the Debian system. I've been quietly working away an experimental redesign and repackaging of Webware. It is not a 'rewrite' of Webware. Chuck, Geoff, Jay, and the other Webware developers have done some excellent work with the existing Webware. Most of the code in this redesign is theirs. You can download it from http://calrudd.com/Webware-experimental_redesign.tar.gz It's well documented and there is much more information in the README file I've attached. Unlike earlier versions of this redesign, which I released under the name 'WebwareXP', it aims for 99% compatibility with existing servlet code. Furthermore, the earlier releases only attempted to reimplement the 'WebKit' application server. It now attempts to provide the functionality of Webware as whole. The early 'WebwareXP' releases were a joke in comparison to this release. It is more mature, more complete, and leverages the existing code base to a greater extent. However, it is still incomplete. It's probably 90% compatible with existing servlet code, but there are still some packaging issues that need addressing and a few sections of the API that need fleshing out. See the TODO section in the README for information. Also, it hasn't benefitted from extensive testing like the existing Webware so there will be bugs and implementation errors. I've now taken it as far as I can by myself. I hope you will see merit in the ideas behind this redesign and that we can work together to complete the implementation. I have no intention of creating a fork. I'm going to be offline for most of the next month, but Federico's expressed interest in taking this to completion when he gets back from holiday next week. If anyone is interested in working with him, we could setup a temporary project on sourceforge to get it into CVS and then fill in the unfinished bits and test it. Of course the final decision of whether it is suitable for integration into the main Webware CVS is up to Chuck, Jay and Geoff. Cheers, Tavis p.s. Jay and Geoff, that tarball that Chuck forwarded to you a week ago contained a buggy and rather incomplete version of this code. Ignore it.
This is an experimental redesign and repackaging of Webware for Python. Contents: - Description - Similarities - Differences - TODO - Architectural Overview - Installation Description ================================================================================ This is an experimental redesign and repackaging of 'Webware for Python'. It is not a 'rewrite' of Webware. Chuck, Geoff, Jay, and the other Webware developers have done some excellent work with the existing Webware. Most of the code in this redesign is theirs. Unlike earlier versions of this redesign, which I released under the name 'WebwareXP', it aims for 99% compatibility with existing servlet code. Furthermore, the earlier releases only attempted to reimplement the 'WebKit' application server. It now attempts to provide the full functionality of Webware as whole. The early 'WebwareXP' releases were a joke in comparison to this release. It is more mature, more complete, and leverages the existing code base to a greater extent. However, it is still incomplete. It's probably 90% compatible with existing servlet code, but there are still some packaging issues that need addressing and a few sections of the API that need fleshing out. See the TODO section below for information. Also, it hasn't benefitted from extensive testing like the existing Webware so there will be bugs and implementation errors. I've now taken it as far as I can by myself. I hope you will see merit in the ideas behind this redesign and that we can work together to complete the implementation. I have no intention of creating a fork. Similarities with the existing Webware ================================================================================ - It incorporates the latest fixes from the Webware CVS. - It uses streamed input and output, just like Webware. - It uses exactly the same approach for importing servlet modules. - The APIs for Responses, Sessions, and Transactions are identical. - All import statements in existing servlet code will work, although the package structure is radically different. - Exception handling and error logging are virtually identical. - The PID and run-settings are recorded to file at startup. - It comes with an HTTPServer. (it needs work though!) - It allows you to drop the extension on filenames in the URI. - It works with MS' COM. - It includes WebUtils, MiscUtils and all the core user classes of WebKit. - The code is modularized and well documented. Differences from the existing Webware ================================================================================ - It is packaged with distutils and thus can install all the core modules to a system directory. There are numerous advantages to using distutils: * users don't need to maintain a local copy of all the modules. * the core modules can be used easily by other frameworks * it can be packaged for Debian and other OS distros. * installs on Windows now make use of the GUI installer. * distutils offers many advanced packaging facilities! * distutils is the Pythonic way to do installs. - It allows multiple applications to run concurrently in the same process, each with its own settings, data caches, servlet caches, and sessions. Think of Applications as beefed up 'contexts'. The mapping of requests to the appropriate application is very flexible and simple to modify. Applications can be associated with virtual host names, specific directories, PATH_INFO's, cookies, GET vars, etc. - It allows request-to-servlet mapping using SCRIPT_NAME rather than PATH_INFO as WebKit currently requires. See the file '.webware_config_annotated' for more information. This makes it far simpler to serve static files from the web-server rather than from Webware, and removes the need for mod_rewrite manipulations of the URI when you want to serve webware servlets directly from the root dir of your web-server. It also frees up PATH_INFO for other uses. - The Launcher script is now object-orientated and thus is more flexble and can be extended easily. - It is significantly faster. Chuck pushed me to do some more realistic benchmarks than hello_world.py so I created a sample servlet that used sessions, cookies and fields then ran it with both the existing Webware and the framework. When testing with a concurrency of 50 and 600 hits the results were 214 req/sec for the new code and 133 req/sec for the status quo. This was on an AMD 650 with 256 MB running Linux 2.2.18. - It allows multiple services to be run on multiple ports from one process and allows these services to communicate. This permits us to build command-line monitoring and admin tools that can communicate with running AppServers, and allows us to to run a simple python-based webserver (like AsyncThreadeHTTPServer) at the same time as accepting requests from apache/IIS via the adaptors. It also allows us to add a encrypted-communications service on another port that can run concurrently with the standard non-encrypted webkit port. PIE-IN-THE-SKY, but possible. - It is possible to run the AppServer without requiring write access to the file-system. All use of external files (logs, etc.) can easily be disabled via the config settings. This allows the AppServer to run as the user 'nobody'. - When dropping the extensions from filenames in requests the existing Webware will raise an exception if there are multiple files with different extensions for the requested basename. This version allows you to specify a list of extensions to cascade through and use the first one found. If you list .html before .py in your list of extensions and a request can map to 'file.html' or 'file.py', 'file.html' will be returned rather than raising an exception. - More of the internal settings are exposed via the config-file. - The Servlet class now has an extra hook method '.shutdown()' for breaking any reference cycles when a Servlet is disposed of. - There is no Async version of the AppServer. - COMKit's functionality has been rolled directly into the AppServer. - It is even more modularized than the existing code, thus allowing more flexibility and more options for code reuse. - 'Cans' have been removed. - plus more ... see the code TODO ================================================================================ - Figure out where to put MiddleKit, UserKit, and PSP in the new package structure. I think the 'contrib_packages' dir is a good spot. They can't go under the 'Webware' dir as everything under it will appear as part of the 'Webware' package in the Python import system. They should appear as top-level packages in Python rather than as sub-packages of Webware. Ditto for the Apaptors: where should they go? And where should packages like FunFormKit and Cheetah, that are under separate CVS control, be placed? - Flesh out the remaining bits of the Application and Request APIs to make sure they are compatible: - forwardRequest, etc. - all the methods of Request that I've marked as needing work. - Integrate Clark's Path-based session patches. - Port PSP to the new framework. - Get 'oneshot' working with the new framework ... I haven't even looked at this yet. - Finish off the AdminServer in AdminServer.py and implement the behaviour of Monitor.py in the Launcher class (read Launcher.py) - Make the HTTPServer interface more robust. - Create some more example Servlets and rebuild the existing 'Example' application. - Create the regression testing framework and cases. - Update the Users' Guides. I propose that LaTeX be used for the docs, like all the Python library modules. Furthermore, I feel that the 'install guides' and 'users guides' in the current Webware documentation should be rolled into one and that the core Users Guide should include all the documentation for MiscUtils, WebUtils, the Adapters, and the AppServer (aka 'Webkit'). - Examine the possibility of using the ZODB and ZEO for storing session data. Also for 'Application-wide' data. This could provide session safe fail-over support. - Experiment with encrypted webserver <--> AppServer communication (maybe with M2Crypto) +++ secure communication with the AdminServer. Architectural Overview ================================================================================ Webware contains a Python-Powered Application Server for serving dynamic websites programmed in Python. As it highly modularized, it can also be used as Software Development Kit for building custom Web Application Servers, chat servers, HTTP servers, etc.. The diagrams below outline how Webware is used as an Application Server: | .---------------------. requests | | Client Web Browser | http://www.mysite.com/examples/helloworld.py | '---------------------' | . | / \ | | | | HTTP request/response | | | | | \ / | ' | .-----------------------. | | Web Server (Apache) | Apache sends all .py requests to mod_webkit. | | | mod_webkit marshals the CGI env data to a | | with mod_webkit.so | string, sends it to a WebwareXP AppServer and | | module installed | waits for a reply to send back to the Client | | | Web Browser. | | | | | | | '-----------------------' | . | / \ | | | | | | | | | \ / | ' | .-------------------------. | | Webware | | | Application Server | The AppServer unmarshals the env dict and puts | | | it in a python dictionary. It then examines | | listening on a port | the request dictionary to identify which | | for requests from | 'servlet' file was requested. It gets a | | mod_webkit | response from the servlet and sends the | | | response back to mod_webkit. | | Details in INSET A | | | | | '-------------------------' Notes: - This diagram only shows one of several possiblp configurations using Apache. IIS and other web servers can be used as well. - mod_webkit can be configured to communicate with multiple AppServers. These AppServers can be run on any port that the user running the AppServer has permissions for. AppServers can also be run on remote machines (behind a firewall for example). Thus you can spread the processing load between several machines and implement replication and fail-over support. INSET A ------- Anatomy of Webware's Application Server .-----------------------------------------------------------------------. | | | | | | | | M | A | H | O | | .---------------------------. .---------------------. | O | D | T | T | | | servlet files | | ditto | | N | M | T | H | | | servlet factories | | | | I | I | P | E | | | live servlet caches | | | | T | N | | R | | | (servlets can talk to | | | | O | | S | | | | each other) | | | | R | S | E | S | | | | | see INSET B | | | E | R | E | | | application settings | | for more info | | S | R | V | R | | | (can be used by servlets) | | on Applications | | E | V | I | V | | | | | | | R | I | C | I | | | shared data | | | | V | C | E | C | | | shared DB connections | | | | I | E | | E | | | | | | | C | | | S | | | user session data | | | | E | | | | | | | | | | | | | | | | plus current transactions | | | | | | | | | | | | | | | | | | | '---------------------------' '---------------------' | | | | | | Application One Application Two | 8 | 8 | 8 | 8 | | | 0 | 0 | 0 | 0 | | | 8 | 8 | 8 | ? | | | 4 | 5 | 7 | ? | | AppServer Service (PORT 8086) | | | | | |-----------------------------------------------------------------------| | | | | | MultiPortServer | | (listening to ports 8084-87) | '-----------------------------------------------------------------------' The 'MultiPortServer' is the outer layer of Webware. It listens to a set of network ports for incoming connections and dispatches them to the service bound to the port the connection came in on. Webware can be run with any number of services on any number of ports. One service can be bound to multiple ports, but not vice-versa. Services can be any callable object. They can be simple functions that accept the connection and communicate with the client at the other end, or they can be a method of a full-blown Application Server that dispatches the request to an appropriate Application/Servlet and returns the Servlet's output. This modularized approach makes it very easy to create custom services or to debug this MultiPortServer independently of service. The main service used in WebKit is the AppServer, which handles requests from mod_webkit. The AppServer unmarshals request data sent from mod_webkit and stores it in 'reqDict', a python dictionary. It then examines reqDict to determine which Application is responsible for handling the request. See the notes in the file '.webware_config_annotated' for more information on the mapping of requests-to-applications/servlets. The mapping of requests to Applications is completely customizable so you can change this if needed. The Application also examines reqDict, this time to determine which of its servlet files was requested. If the servlet file has been requested before the Application will retrieve cached instance of the servlet class defined in the file. If not, it'll import the servlet file as a Python module and create an instance of the class. The Application then creates an Transaction object which handles communication between the Servlet and the Application and also provides a convienient wrapper around Request and Response objects. It also provides methods for dealing with sessions. The Transaction object is responsible for executing the Servlet's awake(), respond(), and sleep() methods. See Inset B for more details. INSET B ------- | .---------------------. | | Webware adapter | sends marshalled request data to port 8086 | '---------------------' | . | / \ | | | | 2-way socket communication | | | | | \ / | ' | .-----------------------. | | MultiPortServer | Receives socket connection on port 8086 and | | | dispatches the connection to the AppServer | | | object. | | | | '-----------------------' | . | / \ | | | | Python calls and return statements | | + communication with the Adapter via the socket connection. | | | \ / | ' | .-------------------------. | | AppServer | | | | The AppServer receives the marshalled data, | | | unmarshals it, and puts it in a Python dict | | | called 'reqDict'. | | | It creates two file objects out of the | | | socket connection: write-mode and read-mode. | | | The read-mode file is stored as reqDict['input']. | | | | | | Then it examines the request dict. to determine | | | which Application should process the request | | | and calls that application's | | | .handleRequest(reqDict, connWriteFile) method. | | | | '-------------------------' | | . | / \ | | | | Python calls and return statements | | + communication with the Adapter via the socket file objects. | | | \ / | ' | .-------------------------. | | An Application | | | | Then it examines the request dict. to determine | | | which servlet should process the request. | | | | | | If an instance of the servlet class isn't already | | | in a cache it creates a new instance. | | | | | | It creates a Transaction object for the request | | | and feeds the servlet to the transaction. The | | | trans. creates Request and Response objects | | | and then calls the servlet's awake(trans), | | | respond(trans), and sleep(trans) methods. | | | | | | Response.write() is mapped directly to | | | connWriteFile.write(). Ditto for flush(). | | | | '-------------------------' | Installation ================================================================================ IMPORTANT NOTE --------------- This is not ready for use in production systems. It is an experiment and needs testing and code review. This is NOT a drop-in replacement for the existing WebKit. It is standalone and requires nothing from the existing Webware. Do not place it in the same directory! If you hack it, don't attempt to use classes from the original WebKit. If you want to test it: ----------------------- 1- compile and install apache with DSO support. use configure command like: ./configure --enable-module=so 2- compile and install mod_webkit from the original webware package. The source code available through 3- read the notes in '.webware_config_annotated' and configure Apache and your .webware_config file accordingly. 4- use the 'webware' launcher script in the 'Webware/Bin/' dir to launch the AppServer using you config script. Type 'webware -h' for more details on the launcher script.
