Hi everyone,
I’m glad to report that NaviServer 5.0 RC 1 is now available on SourceForge:
https://sourceforge.net/projects/naviserver/files/NaviServer5.0/
The updated online documentation can be found here:
https://naviserver.org/docs/5.0/
This marks our feature freeze - over the coming weeks we’ll focus strictly on
bug fixes and stability improvements. Our goal is to deliver the final 5.0
release before the OpenACS/EuroTcl/Conference in Bologna (July 10–11, 2025):
https://openacs.km.at/
Below is the preliminary changelog for the core package. We continue work on a
summary of the main changes in the NaviServer modules repositories.
Since raw changelog since the 4.99 release is quite large (>22K lines),
something may have slipped through. If you spot any omissions that should be on
our radar, please let me know.
The changes are quite extensive:
522 files changed, 78016 insertions(+), 32993 deletions(-)
According to our documentation, this is the largest number of changes between
releases.
Please, if possible, download RC 1, give it a spin, and let us know of any
regressions or show-stopper bugs by opening an issue on our tracker.
https://github.com/naviserver-project/naviserver
Thank you for your testing and feedback - let’s make the 5.0 release rock solid
for Bologna!
Best regards,
The NaviServer Team
Changelog
New Features
Security Enhancements
Secure by Default when NaviServer is used as an HTTP client (ns_http and
ns_connchan)
All HTTP client requests now validate the peer server certificate by default.
Introduced the -insecure flag to bypass validation in trusted scenarios.
Bundled a ca-bundle.crt with well-known root certificates for out-of-the-box
verification support.
Fine-Grained Validation Management:
Configure validation exceptions in the httpclient section to allow specific
certificate issues (e.g., self-signed certs) for defined IP ranges.
Define a directory for storing rejected certificates, enabling later review,
manual trust decisions, or audit trails.
For details, consult the admin-config manual.
Certificate Management Added new commands:
ns_certctl list – List loaded certificates.
ns_certctl reload – Reload certificates at runtime.
Pluggable Authorization Framework
Introduced flexible, scriptable request- and user-level authorization:
Supports multiple authorization handlers ("authorities") with ordered
evaluation.
Handlers can return continuation codes (TCL_OK, TCL_BREAK, TCL_RETURN) to
control the authorization chain.
New Tcl commands:
ns_auth request – Evaluate configured request-level auth handlers.
ns_auth user – Evaluate user-level authorization logic.
Improved Reverse Proxy Trust Management
Define trusted upstream proxies via CIDR ranges.
Configurable handling of X-Forwarded-For chains, including:
Skipping non-routable addresses.
Alternative resolution strategies.
Improves accuracy and security of client IP logging (e.g., in access.log).
For implementation details, see commit ab23158ece.
Argon2 Password Hashing New command: ns_crypto::argon2 – Provides support for
modern, secure password hashing.
User Interface and HTML Improvements
Modernized Default Start Page
Redesigned layout with responsive and accessible design.
Added modal dialog for password updates.
Improved security guidance for publicly exposed installations.
Installer now fetches resources from GitHub (previously Bitbucket).
Dark Mode Support
Automatically respects system preferences via the prefers-color-scheme media
query.
Supported across modern versions of Windows, macOS, iOS, and Android.
Consistent Visual Styling
Unified look and feel across:
Default start page
NaviServer documentation
nsstats module interface
Configuration and File Management
Configuration via environment variables
Added ability to pull in environment variables with a certain prefix as
configuration variables for configuration files (new command
ns_configure_variables). This helps configuration significantly for
container-based deployments, where ports etc. have to be adjusted on
standardized configuration files.
New File Location Options
Unified log path resolution via logdir or serverdir.
Moved serverdir definition from fastpath to the top-level server section; it
now acts as the root for both logdirand page directories.
Added bindir parameter for custom binary module and nsproxy worker locations.
Improved Log Configuration
Relative log file paths are now resolved consistently based on:
serverrootproc (for mass virtual hosting), or
serverdir.
Supports dynamic log file naming (e.g., using Host header) for per-vhost
logging.
Log rotation now applies across multiple log targets.
Improved Section Parameter Updates
New -update flag for ns_section, allowing ns_param to overwrite previously
defined values rather than appending.
Proxy Features
Reverse Proxy Integration
The reverse proxy module is now built into the NaviServer core.
Leverages improvements in ns_http, including:
Persistent connections
Streaming request handling
New features:
-use_target_host_header flag for ::revproxy::upstream to control Host header
forwarding.
Configurable transport mechanism using either ns_http or ns_connchan.
Support for backend connections via Unix domain sockets.
Forward Proxy Implementation
Base implementation included, built on the same scalable architecture as the
reverse proxy.
Full support for the HTTP CONNECT method:
Enables tunneling HTTPS connections through the proxy.
Eliminates the need for insecure plaintext fallbacks.
Additional Functional Improvements
Unix Domain Socket Support
Both ns_http and ns_connchan now support connections via Unix domain sockets.
Multibyte HTML Entity Support
ns_striphtml and ns_unquotehtml now correctly decode multibyte numeric HTML
entities.
New Log Severities
Security – For security-related events.
Deprecated – For usage of deprecated features (e.g., API calls, parameters).
TclPro Debugger Support
Significantly improved integration with the TclPro debugger.
Resolved missing file references and corrected documentation.
See the ns_adp_debug manual for usage.
Tcl 9 Compatibility
Internal updates to support Tcl 9 features and constraints.
Support for Tcl command argument counts >2^31.
License Upgrade
Changed from MPL 1.1 + GPL to MPL 1.2.
Announced previously on the mailing list and Tcl forums.
Tcl API Changes
ns_cache
New option cachingmode: Accepts full or none to toggle caching behavior.
ns_configsection
New option -filter: Filters variables by unread, defaulted, or defaults.
ns_configure_variables
New command to pull in configuration variables from environment variables (to
be used in configuration files)
ns_conn
General enhancements:
ns_conn host, ns_conn port, ns_conn protocol: Now return appropriate values for
all request types (not limited to forward proxy requests).
ns_conn target: Returns the complete request target, including query
parameters, from the HTTP start line.
ns_conn fragment: Returns the fragment identifier (after #) if present.
ns_conn urldict: Parses the request URL into a Tcl dictionary for structured
access.
ns_conn host: Supports an optional default value if the host is not determined.
ns_conn details: Returns a dictionary with driver-specific connection metadata.
ns_connchan
Security and diagnostics improvements:
ns_connchan debug: Set or query the debug level.
ns_connchan connect / ns_connchan open: Now accept additional security-related
options: -cafile, -capath, -cert, -hostname, -insecure, -driver, and
-unixsocket.
ns_connchan status: Added -server option for multi-server introspection.
ns_connchan write: Now performs buffered writes.
ns_driver
info, names, stats, and threads support the -server option to allow
multi-server diagnostics.
ns_http
Connection handling and introspection:
Persistent HTTP connections supported via the keepalive option in httpclient
configuration.
Introduced support for multiple task threads.
Streaming response support for incremental data handling (useful for large file
transfers and LLM interaction).
Default timeout configuration for requests.
New callbacks: response_header_callback and response_data_callback.
Renamed -donecallback to -done_callback (old name deprecated).
Added support for informational HTTP status codes (e.g., 100 Continue).
New subcommands:
ns_http keepalives: Displays active persistent connections.
ns_http taskthreads: Provides insight into HTTP client threads.
Major internal refactoring of tclhttp.c for modularity and maintainability.
ns_ictl
getmodules: Now supports the -server option for per-server introspection.
ns_info
Added subcommands:
argv: Returns the original argument vector.
bindir: Returns the binary directory path.
buildinfo: Displays build-time config and version info.
logdir: Path to the log directory.
meminfo: Displays memory statistics (when using tcmalloc).
ns_ip
Added subcommands:
inany: Checks if an IP matches any configured address.
properties: Returns metadata about configured IPs.
public: Determines if an IP is publicly routable.
trusted: Checks if an IP is trusted.
valid: Validates IP syntax.
These enhancements supersede the need for ns_subnetmatch.
ns_issmallint
New command to check if a value is a valid small integer (replaces the old, now
deprecated issmallint).
ns_logctl
New subcommand grep: Searches log files with pattern matching, stripping color
codes and handling line continuations.
ns_register* Enhancements
All ns_register_* commands now support the -constraints option, enabling
context-sensitive behavior.
New command: ns_register_auth allows registration of user- and request-level
authorization handlers.
ns_server
New subcommands:
authprocs: Lists registered request/user auth procedures.
hosts: Lists registered hostnames for the server.
logdir: Returns the server log directory.
realm: returns or sets the realm of the server
serverdir: Returns the base directory for the virtual server (-effective
returns the resolved runtime path).
vhostenabled: Returns a boolean indicating virtual hosting status.
ns_server mapped: New option -all returns a dict with handler and pool info.
ns_set
Case-insensitive support:
New -nocase flag for ns_set create and related operations.
Deprecated the i* subcommands (e.g., iget, ifind) in favor of unified interface.
Multi-valued key support:
-all option retrieves all values for a given key in ns_set and ns_config.
New subcommands:
format: Pretty-prints the set contents.
stats: Returns memory usage statistics.
delkey: Now returns a boolean success flag.
Internal improvements:
Refactored ns_set internals using dense storage for lower memory use and better
cache locality.
Replaced deeply nested switch logic with modular, maintainable code.
ns_thread
Unified thread creation:
Use ns_thread create for all thread types.
Deprecated begin and begindetached to align with ns_cond, ns_mutex, and ns_sema
usage.
ns_urlspace
Wildcard matching now supports path-segment-level matches (previously limited
to leaf nodes).
Applied improved matching in nscgi to support directory-wide CGI mapping.
Renamed option -contextfilter to -constraints in ns_urlspace set/unset to match
the new registration API.
Database Enhancements
ns_db info: Returns a dictionary of metadata for a given DB handle.
ns_dbpooldescription: Renamed from the previous ns_pooldescription for
consistent naming.
ns_db rowcount: Fixed in this release to return actual row counts.
New Utilities
ns_fseekchars: Efficiently scans a stream for a string (e.g., for
multipart/form-data parsing).
ns_joinurl: Constructs well-formed URLs from path components.
ns_mkdtemp: Creates a unique temporary directory (POSIX-style mkdtemp()).
ns_parsehtml: Parses HTML fragments into a structured Tcl dict.
ns_parsemessage: Parses MIME-style messages (e.g., emails or HTTP headers).
ns_percentencode / ns_percentdecode: Replace deprecated ns_urlencode and
ns_urldecode for robust URL-safe encoding/decoding.
Changes in Core Modules
nscgi
New command: ns_register_cgi for dynamic CGI handler registration (supports
-noinherit, -path, etc.).
Supports unregistration via ns_unregister_op.
New environment variables: SCRIPT_FILENAME, REQUEST_URI.
Refactored to support major web apps (e.g., WordPress, Joomla).
See commit 36027b70215 for implementation details.
nscp
Integrated with new authorization system (e.g., via nsperm users).
New command nscp users: Lists registered users for nscp authentication.
nsperm
Acts as a pluggable authorization provider for request/user scopes.
New config parameter: allowLoopbackEmptyUser—permits unauthenticated loopback
access when enabled.
Added support for setting default server start pages and nsstats.
nsproxy
New subcommand: ns_proxy workers—provides detailed runtime info on proxy worker
processes.
Bug Fixes
Stability and Crash Resolutions
Addressed multiple potential crash scenarios:
Robust handling of HTTP CONNECT requests.
Fixed crashes caused by missing or empty argument lists in commands such as
ns_filestat, ns_sockcallback, and ns_ictl oncleanup.
Prevented crash in ns_log when invoked with an empty message.
Resolved a 24-year-old bug in Ns_AdjTime() that could lead to fatal errors due
to microsecond overflow in multithreaded environments.
Fixed crash in ns_conn copy when operating on empty content.
Prevented crash in ns_sema release with invalid semaphore counts.
Corrected off-by-one error in ns_adp_bind_args that could access uninitialized
Tcl_Obj values.
Fixed crash in ns_inet_ntop due to unsafe memory operations with overlapping
regions (notably on aarch64 with musl).
Resolved crash in Ns_SetIUpdateSz() caused by case mismatch in header keys with
the legacy C API.
Fixed crash during nscp startup when the users section was not configured.
Prevented crash in debug mode when the Host: header could not be mapped to a
virtual server and the driver was installed locally.
Avoided crash when launching nsd with -c and -t options and no home parameter
defined.
Fixed crash during computation of ns_conn location when the network driver was
not globally installed (global installation now recommended).
Fixed crash due to self-destructive header replacement when ns_conn
outputheaders are passed via ns_respond ... -headers ...
Functional Correctness and Logic Fixes:
ns_conn status: Fixed issue where updated status codes were silently ignored.
ns_conn doneCallback: Ensured this callback is always invoked in ns_http.
nsv_dict get: Fixed a memory leak in value retrieval.
ns_conn peeraddr: Resolved race condition in pipelined requests that could
yield incorrect peer addresses.
ns_cache_eval -force: Fixed race condition that could produce obsolete results.
ns_sema create: Corrected handling of initial count values (e.g., 1000).
ns_trim: Fixed spacing logic to trim only leading and trailing whitespace,
preserving internal spacing.
ns_config -int: Corrected fallback behavior when invalid values are supplied,
now correctly using documented defaults.
ns_sockcallback: Now gracefully handles missing arguments.
ns_socknread: Fixed inaccurate results for buffered connections.
ns_crypto::aead::encrypt/decrypt: Restored compatibility with OpenSSL 1.1.1.
Fixed incorrect parsing of encoded backslashes in URLs.
Ns_StrTrimRight(): Corrected UTF-8 handling in right-side string trimming.
Fixed ns_conn location when running behind a reverse proxy to always returns a
value, even in broken configurations.
OpenSSL fixes
Added support for detecting and validating OCSP Must-Staple and AIA presence in
certificates. Without that, NaviServer might crash, when OCSP is turned on, and
NULL values are passed for AIA URLs (letsencrypt)
Introduced stable output buffers for send operations to support retries after
SSL_ERROR_WANT_WRITE, preventing connection failures under high load.
Improved error handling in OpenSSL integration by draining the error stack via
DrainErrorStack().
HTTP Client Fixes
ns_http:
Reordered initialization in NsInitServer() to ensure submodules can access a
fully configured server state.
HTML and ADP Parsing Fixes
ADP Parser:
Enhanced support for quoted > characters inside attribute values, aligning with
modern HTML parsing rules.
return-notice handling:
Suppressed spurious error messages when fallback ADP templates are processed
outside of a full ADP context.
ns_striphtml:
Fixed long-standing bug where adjacent HTML entities were incorrectly
decoded—only the first entity was processed.
Logging and Diagnostics Fixes
Fixed misleading log output when dynamically changing extended headers via
ns_accesslog extendedheaders ....
Improved error messages when port binding fails due to conflicts with driver
assignment.
Fixed off-by-one error in the virtual server port configuration logic.
Clarified or corrected multiple logging messages across modules for better
diagnostics.
nscgi Module Fixes
Fixed file upload failures when uploads were internally spooled by NaviServer —
previously returned 500 errors.
Corrected processing of CGI script exit codes not 0.
Fixed hostname and port reporting for SERVER_NAME.
Database Fixes
ns_db rowcount: Fixed regression where the row count logic was not invoked,
rendering the command a no-op.
Build and Compatibility Fixes
Suppressed obsolete --enable-threads warning for Tcl versions where thread
support is now enabled by default.
Fixed compilation failure with glibc 2.38+ due to PTHREAD_STACK_MIN becoming
dynamic via sysconf().
General Cleanups
Fixed small memory leak triggered by serverrootproc reset.
Numerous minor typo corrections, comment clarifications, and small logic
cleanups across modules.
C-Level Infrastructure, C API Enhancements, and Build System
Core Infrastructure Improvements
Socket Layer Enhancements:
Added sendErrno field to the Sock structure to improve diagnostics and tracking
of write errors.
Modernized Initialization:
Introduced NS_INIT_ONCE() macro for thread-safe one-time initialization,
replacing legacy double-lock patterns.
Data Structure Utilities:
Added utility functions Ns_DListSaveString() and Ns_DListFreeElements() to
simplify dynamic list management.
Replaced use of legacy Ns_DString functions with standard Tcl_DString,
modernizing internal data handling.
Debugging Support:
Added NsHexPrint() for hex-dumping of byte sequences, aiding in low-level
debugging and analysis.
Introspection Enhancements:
Introduced Ns_TclReturnCodeString() and Ns_ReturnCodeString() to convert
internal return codes into readable strings, improving log clarity and
diagnostics.
Code Quality and Performance:
Applied extensive internal refactorings to improve performance, cache locality,
and maintainability across multiple subsystems.
C API Enhancements
General:
Added typedefs for Ns_AuthorizeRequestProc,
Ns_AuthorizeUserProc,Ns_UrlSpaceMatchInfo, Ns_DriverConnInfoProc
Added enum for Ns_RequestType, Ns_UrlSpaceOp, Ns_DriverClientInitArg
Added API calls in ns.h: Ns_ConfigFilename, Ns_ConnTarget,
Ns_UrlSpaceMatchInfo, Ns_ConnServPtr, Ns_DStringAppendSockState,
Ns_RegisterFilter2, Ns_TaskQueueLength, Ns_TaskQueueName, Ns_TaskQueueRequests,
Ns_ObjvTablePrint, Ns_InfoLogPath, Ns_LogPath, Ns_ServerLogDir,
Ns_ServerRootProcEnabled, Ns_ServerLogGetFd, Ns_ServerLogCloseAll,
Ns_ServerLogRollAll, Ns_GetServer, Ns_ServerName, Ns_SockSetSendErrno,
Ns_SockGetSendErrno, Ns_SockGetSendRejected, Ns_SockGetSendCount,
Ns_SockFlagAdd, Ns_SockFlagClear, Ns_SockSendBufsEx, Ns_SockConnectUnix,
Ns_SockGetClientSockAddr, Ns_SockGetConfiguredSockAddr,
Ns_SockaddrPublicIpAddress, Ns_SockaddrTrustedReverseProxy, Ns_SockaddrInAny,
Ns_SockaddrAddToDictIpProperties, Ns_TclReturnCodeString, Ns_ReturnCodeString,
Ns_TclInterpServPtr, Ns_LogDeprecatedParameter, Ns_RegisterFastUrl2File
Extended Ns_Request, Ns_TclCallback
Refactored Ns_ConnReturnMoved() and Ns_ConnReturnRedirect() to unify
redirection logic.
Secure Communication and Validation:
Added fine-grained certificate validation APIs
Improved error stack draining for OpenSSL operations.
Extended C API for ns_connchan:
Introduced a minimal C-level API for connchan connections, enabling finer
control over lower-level network operations.
Build Platform Compatibility:
Updated Windows build system to use NS_IMPORT in place of deprecated NS_EXTERN.
Avoided use of deprecated C functions such as mktemp() to improve portability
and security.
Build System and Tooling
Configuration and Sample Support:
Replaced hard-coded OS commands (ls, mv, ...) consistently with build variables
Replaced hard-coded OpenSSL binary references with the $(OPENSSL) build
variable.
Enhanced log path and configuration variable handling to support more dynamic
setups.
Test Infrastructure:
Extended regression and compatibility test coverage to validate edge cases and
cross-platform behavior.
Build Metadata Introspection:
Added support for build-time introspection of environment settings, including
allocator type (e.g., malloc), compiler version, and Tcl build info.
Optional Deprecation-Free Builds:
Introduced build flag NS_NO_DEPRECATED to exclude deprecated functions and
APIs, mirroring Tcl’s TCL_NO_DEPRECATED mechanism.
CI/CD Modernization:
Upgraded GitHub Actions workflows from version 3 to version 4 for improved
performance and compatibility.
Sample Configuration Files:
Improved sample configurations (nsd-config and openacs-config) to use
environment-specific settings via ns_configure_variables
prefer names http and https instead of nsock and nsssl inside sample
configuration files to ease configuration for new users.
Documentation Updates
Comprehensive Overhaul
Conducted a thorough review and restructuring of all documentation:
Tcl Command Documentation:
Ensured all implemented Tcl commands and their options are fully documented.
Removed documentation for obsolete or unimplemented commands.
Deprecated commands are no longer advertised in manuals or used in examples.
Introduced a dedicated section listing deprecated commands.
Enabled automated generation of a complete command reference.
Test Alignment:
Verified that every implemented Tcl command is covered by regression tests.
Ensured that all documented commands are also implemented and tested.
Syntax and Formatting Consistency
Standardized placeholder syntax across the documentation and syntax error
messages (see commit ffbd32774db for details).
Manual Page Improvements
nscgi: Significantly revised documentation and usage examples for clarity and
completeness.
admin-config.man: Added a new section on “Customizing File Locations”.
Added and updated sections covering:
Basic templating features.
Error handling mechanisms.
Recently introduced Tcl commands and options.
Visual and Structural Enhancements
Updated diagrams and usage examples related to:
Request processing flow.
Reverse proxy (revproxy) behavior.
Additional Improvements
Numerous corrections to spelling, grammar, formatting, and internal linking
throughout the documentation.
Improved WebSocket documentation with expanded details on:
Timeout handling.
Use of connchan for bidirectional communication.
Deprecations
Tcl-Level Command Deprecations
Deprecated the following Tcl commands:
ns_set print → replaced by ns_set format
ns_checkurl and ns_requestauthorize → replaced by ns_auth request
ns_thread begin → replaced by ns_thread create
ns_thread begindetached → replaced by ns_thread create -detached
ns_event → replaced by ns_cond
ns_pooldescription → replaced by ns_dbpooldescription
keyldel, keylget, keylkeys, keylset (from TclX) → replaced by native dict
functionality in Tcl
Legacy experimental functions (marked TBD for over 20 years) are now deprecated:
ns_browsermatch, ns_choosecharset, ns_cookiecharset, ns_formfieldcharset,
ns_formvalueput, ns_paren, ns_tagelement, ns_tagelementset
Deprecated non-namespaced functions:
getformdata, issmallint
ns_parsetime: now officially deprecated (was internally marked "To be removed"
for ~15 years)
ns_set_precision: deprecated in favor of standard Tcl idioms
Tcl-Level Option Deprecations
Deprecated options:
-buffered in ns_connchan write
-donecallback in ns_http → replaced by -done_callback (naming consistency)
-binary (previously used to indicate Tcl objects with binary data) → replaced
by -data, aligning with conventions where -binary is a boolean flag
C-Level API Deprecations
Deprecated or removed C functions:
Ns_ObjvFlags() → replaced by Ns_ObjvIndex() for option parsing (removed
redundancy)
Ns_SockSendBufs2() → replaced by Ns_SockSendBufsEx() which returns an
additional errorCode
Ns_TclInitInterps() → removed (marked as deprecated since 2005)
Deprecated internal usage of OpenSSL and Tcl functions that have been marked as
deprecated upstream
All C functions previously marked as deprecated in source comments are now
officially deprecated
Configuration Parameter Deprecations
Global config parameters:
logroll: → replaced by logrollonsignal (standardized log rotation behavior on
SIGHUP)
serverlog: → replaced by systemlog to reduce ambiguity between system and
per-server logs
Section parameter changes:
serverdir in the fastpath section is now deprecated → use serverdir in the main
per-server section instead (reflecting its broader usage scope)
Deprecation Management Infrastructure
Introduced compiler warnings for usage of deprecated Ns_DString* functions
Added log severity level Deprecated to better surface deprecated usage at
runtime
Documented all deprecated commands in a dedicated section of the command
reference
Deprecated the use of manual double-checked locking for one-time initialization
replaced by the NS_INIT_ONCE() macro for safer and cleaner initialization
semantics_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
