AOLserver 4.5.0 Released June 27, 2006 ========================================
On behalf of the AOLserver Team, I have the honor of announcing the latest major release of AOLserver: 4.5.0. AOLserver 4.5.0 is a major upgrade including several new Tcl commands, C API's, and resource management improvements. A summary of changes is provided at the end of this document. WHAT IS AOLSERVER? AOLserver is America Online's Open-Source web server. AOLserver is the backbone of the largest and busiest production environments in the world. AOLserver is a multithreaded, Tcl-enabled web server used for large scale, dynamic web sites. Visit the project's website: http://aolserver.com/ HOW CAN I GET AOLSERVER? Download the source code from SourceForge: http://aolserver.com/downloads/aolserver-4.5.0-src.tar.gz I FOUND A BUG! WHAT DO I DO? File it in the Bug Tracker at SourceForge: http://aolserver.com/sf/bugs IS THERE A MAILING LIST? WHERE'S THE USER COMMUNITY? Yes! There's an announcements-only and a general discussion mailing list. Instructions on how to subscribe are here: http://aolserver.com/lists.php There is also a wiki-web set up for AOLserver: http://aolserver.com/wiki/ CHANGES SINCE LAST RELEASE? AOLserver 4.5 - June 2006 ------------------------- AOLserver 4.5 is a major upgrade including several new Tcl commands, C API's, and resource management improvements. Contents: -------- New and Updated Binaries New Modules New and Updated Tcl Commands New and Updated C Routines Additional Updates Changes to the Build Process Bug Fixes Known Issues New and Updated Binaries: ------------------------ libnsd: The AOLserver library now includes an entry point suitable for loading into an ordinary, thread-enabled, tclsh, e.g.,: # tclsh % load /usr/local/aolserver/lib/libnsd.so % ns_time This provides the full set of Tcl commands within the AOLserver core although several commands will generate errors for lack of an HTTP request context (e.g., ns_conn). nstclsh: A simple Tcl-shell linked against libnsd. It's equivalent to loading libnsd into an ordinary tclsh as described above. New Modules: ----------- nszlib: The popular nszlib module written by Vlad Seryakov has been integrated into the core release with modifications to enable the new Ns_Gzip routine. The module also provides the "ns_zlib" command for compressing and uncompressing strings and files. nsproxy: The nsproxy modules provides the "ns_proxy" command which enables sending Tcl scripts to a proxy process connected via a pipe. The proxy includes core Tcl commands as well as AOLserver commands from the libnsd library. Evaluating scripts in a proxy process can be used to isolate and/or timeout thread-unsafe or otherwise unsafe 3rd party code. New and Updated Tcl Commands: ---------------------------- The following Tcl commands are new or updated from the last release. As few of the man pages have yet to be written, see the source code for usage details and options. ns_pools: The "ns_pools" command enables configuration of one or more pools of connection processing threads. The pools allow certain requests to be handled by specific threads. This could, for example, ensure multiple long running requests don't block other short running requests. Pools are selected based on method/url pairs similar to the mappings managed by the "ns_register_proc" command. By default, all requests are handled by a single, unlimited, "default" pool. There is also an "error" pool as described below. Coupled with the new "ns_limits" command, pools can provide for sophisticated resource management. ns_limit: The "ns_limit" command enables setting various resource limits for specified method/url combinations. These limits include such items as max concurrent connections, max file upload size, and timeouts waiting for connection processing. When limits are exceeded, connections are immediately dispatched to a dedicated "error" connection processing pool to generate a quick error response. By default all requests share the same default limits. Coupled with the new "ns_pools" command, URL-based limits can provide for sophisticated resource management. ns_register_fastpath: The new "ns_register_fastpath" command enables mapping of specific directories to the internal static file response code. See also the new Ns_FastPathOp API. ns_adp_flush: The "ns_adp_flush" command allows buffered ADP output to be immediately flushed to the client. If no output has been sent, the server will first generate the appropriate HTTP response headers for a streaming response. An implicit flush will also be performed when the ADP buffer overflows. ns_adp_close: The "ns_adp_close" command flushes content as with ns_adp_flush and then closes the connection so no additional content can be sent. This can be useful to keep the client from waiting while post-processing steps continue. ns_adp_registerscript The "ns_adp_registerscript" command is similar to the ns_adp_regsiteradp command except it registers an ordinary script instead of text/script ADP code to execute for a given tag. ns_adp_ictl: The new "ns_adp_ictl" command provides various options to manage the ADP execution environment for current Tcl interpreter, e.g.: ns_adp_ctl bufsize ?size?; # Max output before forced flush ns_adp_ctl autoabort ?bool?; # Abort execution on client drop? ns_adp_ctl singlescript ?bool?; # Compile pages as a single script? ns_adp_ctl gzip ?bool?; # Gzip output if enabled in server? ns_adp_ctl trimspace ?bool?; # Trim leading whitespace in response? ns_adp_ctl channel ?chan?; # Set channel for ADP output ns_adp_ctl trace ?bool?; # Trace ADP execution ns_adp_ctl detailerror ?bool?; # Generate detailed error messages ns_adp_ctl displayerror ?bool?; # Format error message in page output ns_adp_ctl stricterror ?bool?; # Abort ADP execution on flush error ns_adp_include: The "ns_adp_include" command has been enhanced with the new "-cache ttl" and "-nocache" options to support execution results caching. Applications which isolate fully dynamic page components (e.g., personalized content) from partially dynamic page components (e.g., slowly changing news headlines) can benefit from using these options to cache and reuse the results of slowly changing components. This approach has the potential to dramatically improve the performance of such applications. ns_adp_ident: The new "ns_adp_ident" command can be used to tag and query ADP file versions. ns_cache: The "ns_cache" command enables creation and management of memory-based caches utilizing the underlying Ns_Cache C API's. ns_driver The new "ns_driver" command provides a "list" option to enumerate all driver instances and a "query" option to return a Tcl list of various statistics for a given driver. ns_ictl: The "ns_ictl" command has been enhanced with several new options including: trace: Enables Tcl-script based callbacks at points defined by the Ns_TclRegisterTrace routine. threads: Returns a list of all threads with Tcl interps for current virtual server. cancel: Sends an async-cancel message to a given thread, interrupting any Tcl execution in any AOLserver-based interpreter executing on the thread. package: Registers a Tcl package to be automatically required in each interp for the current virtual server. once: Registers a script to be executed exactly once for a virtual server. ns_internalredirect: The "ns_internalredirect" command enables re-authorization and restart of a connection as if it where originally for a different URL. This command provides access to the previously private routines used internally for such purposes, e.g., by the fastpath to handle directory URL's mapped to an ADP index files such as "index.adp". ns_for: ns_foreach: ns_while: These commands are replacements for the standard "for", "foreach", and "while" command which have the same semantics but also allow for monitoring and control by the "ns_loop_ctl" command described below. ns_loop_ctl: The "ns_loop_ctl" command enables listing, pause, resume, cancel, and evaluation of a script within "ns_for", "ns_foreach", and "ns_while" loops executing in other threads. In addition, the "install" option can be used at to swap the "ns_for", "ns_foreach", and "ns_while" commands for the "for", "foreach", and "while" commands, enabling monitoring of all loops in unmodified code. ns_register_encoding The new "ns_register_encoding" command enables mapping of method/URL combinations to specific charset encodings used to decode the request. If no mapping exists for a given method/URL, the server will use that defined for the virtual server or process wide by the "urlcharset" config setting. This setting defaults to NULL, or no encoding, thus assuming UTF-8 characters in the request data. ns_returnmoved: The "ns_returnmoved" command is similar to the "ns_returnredirect" command but returns a 301 "permanently moved" HTTP status code along with the redirected location. New and Updated C Routines: -------------------------- The following routines are new or updated from the last release. As few of the man pages have yet to be written, see the source code for parameter and results details. Ns_BinPath: New routine to construct file pathnames relative to the bin/ subdirectory similar to the Ns_LibPath and Ns_HomePath routines. Ns_ConnContent: Return a pointer to request content. If the content is not in memory and has been spooled to a temp file, a new file mapping will be created and returned. Note that the content is likely, but not guaranteed, to be followed by a terminating null byte. See also the "content" option of the "ns_conn" command. Ns_ConnContentFd: Return a file descriptor open on a temp file containing request content. If the content is in memory only, it will first be copied to a newly allocated temp file. You must not close the returned fd as it is owned by the server core which will reuse the open file for subsequent requests. For an example, see nscgi/nscgi.c which has been updated to use Ns_ConnContentFd instead of spooling to a temp file directly. See also the "contentchannel" option to the "ns_conn" command which returns a duplicate of the fd returned via Ns_ConnContentFd registered as a Tcl channel. Ns_ConnContentOnDisk: Returns 1 if the request content is already located in a temp file or 0 if the content is located in memory. A call to Ns_ConnContent after Ns_ConnContentOnDisk has returned 1 will result in a new file memory mapping being created. Ns_ConnGetFile: Ns_ConnFirstFile: Ns_ConnNextFile: Routines to find a specific file or enumerate all file objects embedded within a multipart form. See also the "files" option to the "ns_conn" command. Ns_ConnSendFdEx: Send contents of an open file descriptor from a specific file offset. This routine uses pread on Unix to enable multiple simultaneous calls with the same open fd and is currently unsupported on Win32. Ns_ConnReturnOpenFdEx: Generate a complete response with content from open file descriptor at given offset. This routine uses Ns_ConnSendFdEx and as such is also unsupported on Win32. Ns_ConnReturnServiceUnavailable: New routine to return an HTTP 503 response. Ns_CheckStack: Checks if the stack appears to have overrun. Note that the ADP evaluation engine now calls Ns_CheckStack on entry to each call frame and will generate an exception in the case of an endlessly recursive ADP condition. Ns_ClsAlloc: Ns_ClsSet: Ns_ClsGet: New interface, similar to thread-local storage, which enables user data to be saved and retrieved within the connection structure. This interface is intended to be used with the Ns_QueueWait interface so that data can be saved in the driver thread and retrieved later in a connection processing thread. Ns_ConnSetStatus: Ns_ConnGetStatus: New routines to set and get the HTTP status code to be used in responses generated via Ns_ConnFlush. See also the "status" option to the "ns_conn" command. Ns_ConnSetType: Enhanced to more intelligently set the input and output character encodings based on intended output charset. See also the "ns_adp_mimetype" command and the "encoding" and "urlencoding" options to the "ns_conn" command. Ns_ConnGetGzipFlag: Ns_ConnSetGzipFlag: Ns_ConnGetKeepAliveFlag: Ns_ConnSetKeepAliveFlag: New utility routines to query or set the gzip and connection keep-alive advisory flags. Ns_ConnFlush: New general purpose routine to construct and return dynamically generated, text-based responses, e.g., as generated from ADP. The routine incorporates much of the logic previously embedded in the ADP interface including the ability to automatically covert from UTF-8 to the output encoding and/or gzip-compress the output. In addition, the routine supports streaming responses, generating headers on the first call followed by HTTP/1.1-style chunked-encoding blocks on subsequent calls. See also the "ns_adp_flush" and "ns_adp_close" commands. Ns_ConnFlushDirect: Similar to Ns_ConnFlush but without any modification of the given buffer, thus useful for non-text content. Ns_DevNull: Returns a shared file descriptor open on /dev/null. This fd is owned by the AOLserver process and must not be closed. Ns_FastPathOp: New request callback to utilize the internal "fastpath" caching-file response code. See also the new "ns_register_fastpath" command. Ns_AdpRequestEx: Extended version of Ns_AdpRequest with an additional Ns_Time argument, specifying the timeout of optional execution results caching. Ns_FindCharset Locates the charset string within a full mimetype string, e.g., locating "utf-8" in "text/html; charset=utf-8". Ns_Gzip: Ns_SetGzipProc: Routine to gzip-compress a buffer if a compression function has been registered with a call to Ns_SetGzipProc. The new "nszlib" module registers such a callback. Ns_GetAllAddrByHost: New routine to return all addresses for a given host. See also the "ns_addrbyhost -all" command. Ns_ParseRequestEx: New extended version of Ns_ParseRequest which takes an optional Tcl_Encoding argument specifying the charset to decode the URL. The default is based on the context of the request. Ns_TaskCreate: Ns_TaskEnqueue: Ns_TaskRun: Ns_TaskCancel: Ns_TaskWait: Ns_TaskCallback: Ns_TaskDone: Ns_TaskFree: Ns_CreateTaskQueue: Ns_DestroyTaskQueue: New facility for event-based callback queues designed to replace the legacy Ns_SockCallback interface. As an example, see the "ns_http" command which has been updated to use the task interface. Ns_TclRegisterTrace: New routine to registere callbacks at specific points in the lifetime of a Tcl interpreter. This routine is a replacement for the legacy Ns_TclRegisterAt* routines. See also the "trace", "runtraces", and "gettraces", options to the "ns_ictl" command. Ns_QueueWait: New interface to enable event-driven callbacks in the driver thread before dispatching to pools for processing. This allows drivers to augment data received from the client (headers, request, content) with additional data fetched over the network (likely stored in connection-local storage via Ns_Cls). An example would be to add certain personalization data received via a Web service. The benefit of this approach is to accumulate additional data efficiently with driver-thread based event callbacks instead of through potentially blocking calls within connection threads. Ns_SockWaitEx: New extended version of Ns_SockWait using higher-resolution millisecond timeouts instead of 1-second resolution. ns_poll: New wrapper around system-provided poll or an emulation based on select when poll isn't available (Win32) or broken (OS/X). Ns_LibInit: NsThreads_LibInit: New routines to initialize the libnsd and libnsthread libraries. It is normally not necessary to call these routines as the dynamic library entry points as well as the Ns_Main and Ns_TclInit routines call these init routines at startup. They are available if necessary to initialize specialized, statically linked applications. Additional Updates: ------------------ * Enabled a global, pseudo-server, to support all Tcl commands outside the context of a virtual server, e.g., when the libnsd.so library is loaded into a Tcl shell. * Updated Tcl trace execution order to be FIFO and LIFO for init and cleanup callbacks respectively. * Moved more of the legacy (and generally confusing) Tcl interp duplication code from C into the the bootstrap init.tcl script utilizing the ns_ictl command. * ADP output buffer is now configurable with a maximum size (see the "ns_adp_ctl" command above). Upon overflow, the current output will be flushed to the client immediately in streaming mode to avoid consuming unbounded memory. * Restored AOLserver 3.x ADP parsing order, based on a patch from Alastair Young, where registered tags, <% ... %> scripts, and <script> ... </script> tags all have the same precedence. The ADP parser now also properly handles nested sequences, e.g.: <% ns_adp_puts [ns_adp_eval {<% ... %>} %> * Enhanced ADP error messaging capabilities which can be managed on a per-interp basis with ns_adp_ctl or configured by default, e.g.: ns_section "ns/server/server1/adp" ns_param stricterror false ;# Stop execution on error ns_param detailerror true ;# Include connection detail ns_param displayerror false ;# Return error message to client When "detailerror" and/or "displayerror" are set, an error message similar to the following would be appended to the server log and/or included in the output sent to the client: This is an error. while executing "error "This is an error."" at line 4 of adp file "/aolserver/servers/server1/pages/error.adp" while processing connection #2: GET /error.adp HTTP/1.1 Accept: */* Accept-Language: en Accept-Encoding: gzip, deflate User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) ... Connection: keep-alive Host: 192.168.2.1:8000 * New ADP execution tracing which can be enabled for a request with the ns_adp_ctl command or configured by default, e.g.: ns_section "ns/server/server1/adp" ns_param trace false ;# Enable text/script debug info. ns_param tracesize 40 ;# Max chars of each chunk to log. The output of ADP tracing is somewhat limited and poorly formatted. * Communication driver threads are now dedicated to each instance of loaded drivers instead of a single thread multiplexing all I/O. This both simplified the code and enabled the new Ns_QueueWait API. * Communication driver threads now truncate instead of freeing I/O dstring buffers, avoiding repeated realloc calls between connections. * Re-ordered inits to ensure startup fds and log file are initialized first. * Communication drivers and connection data access routines have been updated to support large request content spooled to temp files instead of always copying to potentially large in-memory regions. See the Ns_ConnContentFd API for more details. * Updated to HTTP response version (1.0 or 1.1) to match client version. * Updated "ns_db" Tcl API to be a native Tcl_Obj command which should improve performance over the previous technique of multiple comparisons of command option strings. * Added debug driver configuration option which will enable logging of unusual I/O or protocol errors. * Simplified charset/encoding config for process-wide url decoding and per-virtual server defaults for dynamic Tcl requests. * Several new default file extension to mimetype mappings have been added in nsd/mimetype.c, e.g., .mp3 = audio/mpeg. * Created new default index.adp page which displays a number of AOLserver configuration settings. * The OS/X dynamic library loading code has been removed in favor of the more standard dlopen/dlsym API now provided by OS/X 10.4. Changes to the Build Process: ---------------------------- * The Unix and Win32 build environments have been tightly integrated. Basically, the makefiles support both gmake on Unix and nmake on Win32 by including a platform-specific include/ns.mak created by the nsconfig.tcl Tcl script. As a side effect, several makefile variable names had to be changed, e.g., "LIB" on Unix is now "DLL" on Unix and Win32. In addition, platform-independent Tcl scripts are used for file removal and copying, a further dependency on a working Tcl installation while building. * On Unix, backwards compatible Makefile.module and Makefile.global are provided which simply include the new ns.mak and map old variables to new names if present, e.g., "LIB" to "DLL". * On Win32, the legacy Visual Studio build directories have been removed entirely has their maintenance proved irritating and complicated. * A new util/nsinstall-man.sh script installs man pages and also optionally converts the pages to HTML using groff on Unix. * Several new makefile install directives have been added, including install-tcl, install-skel, install-util, install-bins, and install-docs. * Added new stubs and makefile targets to support building a library or module in a consistent way which separates the library code installed as both a static archive and dynamic lib from the library and/or module init calls. * Added mynsd example directory showing how to build a custom static linked nsd binary. The previous conditional startup code and makefile support has been removed. * Compiling for 64-bit is known to work in general on Unix. It is unclear if AOLserver has ever been successfully compiled on 64-bit Windows and there is a known issue with the Win32 thread interface which would appear to make this not possible (the thread exit status pointer is cast to an smaller-sized integer). In addition, while AOLserver runs on 64-bit Unix systems, the interfaces have not been scrubbed and updated to support all 64-bit features, e.g., large file offsets. A more extensive 64-bit certification effort is planned for the next release. * Applied patches from Stephen Deasey for better compile time error checking, e.g., now checking printf-style arguments in Ns_DStringPrintf. Bug Fixes: --------- * Fixed crash bug caused by certain command sequences generated when using the "ns_chan" Tcl API (SF #1094480). * Fixed crash bug in channel tables cleanup and closing code. * Fixed crash bug loading nsdb outside a virtual server. * Fixed invalid return code in Ns_ConnReadLine(). * Updated simple HTML response routines (e.g., Ns_ConnReturnNotice, ns_returnnotice) to properly encode output from UTF-8. * Fixed bug where log level "dev" wasn't properly supported. * Switched to strtol() from atoi() for improved thread safety. * Added check for Darwin (OS/X) to disable detection of poll() which appears broken, using the poll() emulation code based on select() instead. * Fixed infinite recursion crash bug in GetAddr(). * Removed minor memory leak when no CGI mappings are present. * Removed most panics when compiling on 64 bit machines. * A thread race condition which could lead to a crash when using "ns_server threads" has been fixed. Known Issues: ------------ * Documentation remains incomplete. We are still missing useful overview and getting started materials and, while significant progress has been made, most of the man pages are still empty stubs. * Loading libnsd into a tclsh and then creating new threads with the ns_thread command will result in a crash when those threads exit. The issues has to do with finalization of the async-cancel context used to support the new "ns_ictl cancel" feature. This bug is not present when using the "nstclsh" binary. * The Win32 port does not support all of the new features of temp-file based request content. In particular, while large requests will spool to a temp file, a subsequent call to Ns_ConnContent will fail as the corresponding file mapping code has not yet been written. * The build environment in general is annoying. As described, there was an effort in this release to support Unix and Win32 from the same makefiles which, while a bit clever, likely irritated more Unix developers than excited Win32 developers. In addition, the Unix configure script is weak, overly reliant on the context from a properly built Tcl installation, and the makefiles rely too much on GNU make features. Creating a proper Unix configure script which creates proper Makefiles is planned for the next release along with a possible shift in the Win32 strategy (see next note). * AOLserver does not compile with the latest Visual C++ 2005 from Microsoft. While some of the new safety and security features in Visual C++ are interesting and the debugger has always been excellent, the complexity of Visual Studio and incompatibility of the Microsoft command line tools with Unix equivalents makes supporting Win32 increasingly irritating. As such, we will be exploring a shift to Msys-based Win32 build in the next release, enabling us to utilize our Unix-based development approach while still allowing AOLserver to run against native Win32 libraries. Feedback is appreciated. For now, AOLserver can be compiled on Win32 as described in the README file using the previous release of Visual C++. -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70)