Dear all,
on sourceforge is a release candidate for NaviServer 4.99.19 [1].
Below is a - preliminary - summary of changes; especially the
summary of changes in the modules is missing. In case you have more
changes for this release, please commit or get in touch with me.
If everything goes well, the release should follow in about one week.
Please test if possible.
best regards
-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/
=======================================
NaviServer 4.99.19, released 2019-XX-XX
=======================================
244 files changed, 18221 insertions(+), 7899 deletions(-)
New Features:
-------------
- Bandwidth management: Limit download rate to a maximum (specified
as kilobytes per second)
On servers with bad internet connections, it is possible that a few
eager downloaders can use up all available bandwidth. NaviServer
allows now admins and developers to specify the maximum outgoing
bandwidth per connection or per connection pool. The rate limit can
be specified per connection (highest weight) or per connection
thread pool (second highest weight) or per network driver. The rate
limitation is only active, when writer threads are used. Per
default, the rate limitation is turned off (value 0).
New option for ns_conn to query set the per-connection limit
ns_conn ratelimit ?limit?
Introspecton "ns_writer list" returns the current
transferrates per writer job.
Configuration option for connection pools:
ns_param connectionratelimit ...
ns_param poolratelimit ...
Configuration option for network drivers:
ns_param writerratelimit ...
Commands for changing the rate limits dynamically:
ns_server ?-server s? ?-pool p? connectionratelimit ?value?
ns_server ?-server s? ?-pool p? poolratelimit ?value?
- Context filter for urlspace:
NaviServer's urlspace trie data structure used for various purposes
such as registering methods/URL pairs for procs or for mapping of
requests to connection thread pools. The classical trie mapping for
e.g. "GET /foo/bar/*.html" is based on method + path + string match
pattern, where the last part "*.html" is used as a "filter".
The new version of NaviServer allows in addition to the string
based glob pattern so-called "context filter", which includes the
IP-address or header fields in the matching process. The context
filters are applied after the classical urlspace processing, such
that the behavior is fully backwards compatible. The context
filters are kept in a stable order (IP before header-based filters,
specific before less specific) such that the behavior is not order
dependent.
The new feature can be used map e.g. bots or certain IP ranges to
the specified connection pools. When connection pools named "bots"
and "local" are defined, one can use the following mapping commands
to map incoming requests to the specified pools. Examples for
adding urlspace mappings at runtime:
ns_server -pool bots map "GET /* {user-agent *bot*}"
ns_server -pool bots map "GET /* {user-agent *crawl*}"
ns_server -pool bots map "GET /* {user-agent *baidu*}"
ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}"
ns_server -pool local map "GET /* {X-NS-ip 127.0.0.1}"
ns_server -pool local map "GET /* {X-NS-ip 137.208.1.0/16}"
Note that arbitrary header fields can be used for the mapping,
containing potentially some glob style match characters. The
special header field "X-NS-ip" denotes IP based filters, which can
be provided fully qualified or in CIDR notation (denoting
significant bits) for IPv4 and IPv6.
- Allow a single driver to listen on multiple IP addresses:
This feature greatly simplifies setups, where a single server is
listening on multiple IP addresses (e.g. IPv4 and
IPv6). Previously, it was necessary to define separate drivers for
these, which need different names but which are often
configured identically. Furthermore, when the address is omitted,
the server performs a lookup from the hostname to determine the IP
address. Previously, it used the first one, now it can work with
all returned addresses.
To use this feature, simply specify the configure values of
"address" as a Tcl list containing multiple IP addresses. The
change is fully backwards compatible, old configuration files will
continue to work.
- Added sending of multiple file chunks with a single command:
ns_writer sendfiles /list of filespecs/
Every filespec is a dict which must contain a "filename" element
and can contain additionally an "-offset" and/or a "-size" element.
ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}}
This function is e.g. useful for video streaming applications,
where multiple video file chunks should be transferred (sometimes)
concatenated in a single chunk. Therefore, this function avoids the
necessity to concatenate these files in advance (causing increased
latency and disk usage).
- Added per-server "extraheaders". These extra headers fields are
merged with the following precedence (the earlier ones have more
weight)
1) application specific headers
2) per-server extra headers
3) per-driver extra headers
This change makes it possible to provide in the config files
defaults, which can be overwritten by certain pages in the
applications. Furthermore, this change makes it easy to add
header fields to all requests of a virtual server (e.g.
Strict-Transport-Security, ...)
- Logging improvements:
* Support separate access logs for requests sent via
different drivers.
The nslog module accepts now an additional parameter named
"driver". This option can be used to produce different access
logs for requests submitted via different drivers to sort out
e.g. local server talk. Per default, every request is
logged. When a value is provided for this parameter, only
requests are logged in this log file when these come from a
driver matching the provided glob pattern.
* log-file sanitizer:
When data entered into a log file contains special characters
(e.g. end line characters), these could alter the appearance of
content within the log file. Single entries may appear as
multiple entries. Attackers may leverage log forging to insert
fake entries to that obfuscate malicious acts, or these can
confuse log-file analyzers to avoid analysis.
The new global parameter "sanitizelogfiles" can control this
behavior (values 0: none, 1: full, 2: human-friendly; default:
2);
* Additional debugging flag "Debug(access)": When activated, entries
of the access log are mirrored in the system log
(error.log). This option eases tracking the end of requests in
the system log.
* Additional debugging flag "Debug(writer)" for debugging of writer.
* Provided access to request AND reply header fields via
"extendedheaders" specification (in "ns_accesslog" and "nslog"
configuration section):
It is now optionally possible to specify the header fields in the
provided "extendedheaders" list with a prefix "request:" or
"response:" to denote request and reply header fields. In case,
no tag is provided, field names are interpreted as request header
fields (like before). The change is fully backwards compatible,
- ns_set:
* Added sucommand "ns_set imerge", a is the case insignificant
version of "ns_set merge". This option is useful especially for
working with header fields.
* Added subcommand "ns_set iupdate": this command is the case
insenstive counterpart of "ns_set update".
* Fixed "nsv_set a b" when no key nor array exist
- ns_http:
* Revamp of ns_http and related supportive code.
* Added chunked-encoding parser
* Tcl channel support for body (PUT, POST) and for result.
This makes it possible to use e.g. reflected Tcl channels
in "ns_http".
New Options for "ns_http run"
- ?-body_size size?
- ?-body_chan chan?
- ?-outputfile fn?
- ?-outputchan chan?
* Make clear that "ns_http run" is the preferred interface
(rather than "queue" + "wait")
* Deprecated output variables of "ns_http wait" (since everything
is included in the resulting dict)
- New commands:
* ns_asynclogfile:
Facility for thread-safe writing async
log files of various kinds using the AsyncWriterThread.
Usage example:
# ... at startup
set fd [ns_asynclogfile open /tmp/test.log]
# ... during run
ns_asynclogfile write $fd hello\n
# ... at shutdown
ns_asynclogfile close $fd
* ns_parsefieldvalue:
This function parses the provided field value (from an HTTP
request or reply header field) into its parts and returns these
in the form of a list of Tcl dicts or a single Tcl dict depending
on parameters. The syntax of the contents of these header fields
is specified in RFC 7230 section 3.2.6.
- New features for existing commands/subcommands
* Binary reform: Some NaviServer commands accepted previously
implicitly binary or non-binary input, some commands used
the flag "-binary" to denote the differences (e.g. ns_return).
For a more uniform and expectable behavior binary input the
explicit option "-binary" flag to the following commands:
ns_base64decode
ns_base64encode
ns_base64urldecode
ns_base64urlencode
ns_md5
ns_sha1
ns_uudecode
ns_uuencode
ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input")
ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input")
ns_crypto::eckey import ("-string")
ns_crypto::eckey sharedsecret ("pubkey")
ns_crypto::hmac add (for "message")
ns_crypto::hmac new (for "key")
ns_crypto::hmac string (for "key", "message")
ns_crypto::md add (for "message")
ns_crypto::md hkdf (for "-salt" "-secret" "-info")
ns_crypto::md string (for "message")
ns_crypto::md vapidsign (for "message")
ns_crypto::scrypt (for "-salt" "-secret")
Note that this option is not fully backward compatible. Early
versions of "ns_sha1" were expecting always non-binary input, some
new version assumed binary input. The new version is now more
compatible with AOLserver and older NaviServer versions.
* "ns_server threads" reports now how often a thread of this pool
was started. This statistic can help to identify
installations, where threads are started and stopped in a too
eager fashion.
* "ns_server.... stats": added "sendbodysize" and "replybodysize"
to the dict of the per-pool statistics.
* "ns_server ... connectionratelimit ?value?": Query or set
the default per-connection rate limit.
* "ns_server ... poolratelimit ?value?": Query or set
the pool connection rate limit.
* "ns_writer list" returns now the current transferrates per
writer job.
* "ns_writer submitfile -offset X -size Y" and "ns_writer size X"
accept now memory units, the option ?-driver driver? was added
to the "ns_writer" subcommands "size", and "streaming"
(defaulting to the current driver).
One can now write
ns_writer size 150KB
instead of
ns_writer size nssock [expr 150*1024]
Performance Improvements:
-------------------------
Bug Fixes:
----------
- Blueprint serializer: moved serialized objects in the blueprint
after the namespace imports and ensemble recreators since
constructors can call functions depending on it.
- Include *xml* and *json* in non-binary MIME types. Note that
there is no exact definition, what MIME types are exactly.
- nslog:
* Abort in error situation with an error message instead of crashing
* Added server name in log entries to make it easier to distinguish
messages per-server during bootup
- Range requests:
* Fixed potential problem with too many non-contiguous byte ranges
* Don't silently ignore invalid syntax of range requests
- Improved handling of already closed connections (data delivery was
already delegated to writer threads, but driver still wants to
return data directly; this could happen in error situations)
- connchan:
* Fixed potential invalid reads in (error) cases, where
LogConnchanDebug is enabled and the callback is deleted during a
Tcl_Eval().
* Fixed potential problem, where call to Tcl_Eval() might clean
structures used in the callback handler
- Make sure, memory allocated by Tcl is returned by Tcl (important,
when NaviServer is compiled with -DSYSTEM_MALLOC)
- Fixed Tcl argument parsing of "ns_roll", "ns_fmttime",
"nsv_bucket" and "ns_critsec eval".
- Fixed bug in "nsv_set -default" overwriting pre-existing values
- Fixed potential crashes in (desperate) error situations, where the
connection was already closed.
- Fixed leaking tmp file, when writerstraming is activated and
streaming HTML output is used. This bug could appear, when
"writerstreaming" is turned on in the config file (default off).
See also: https://sourceforge.net/p/naviserver/mailman/naviserver-
devel/thread/dd508519-1262-bd91-288a-fa30a7634224%40digital-
concepts.com/#msg36794472
- Made table used for enumeration values for Ns_ObjvIndex static.
Background: This is a fix for a tricky Tcl_Obj sharing bug
triggered by the usage of C-level IndexObjs based on volatile
tables. Consider the following example:
proc foo {x} {
return [bar -value x ...]
}
The Tcl_Obj "x" is shared as name of argument and as a value of
the non-positional parameter "-value". When bar
(e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in
a table of options, which is volatile, the involved Tcl_Obj will
be converted to an indexObj.
On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl
will try to give a nice error message, saying that "foo x" can be
called only with one argument. When printing argument "x", it sees
that "x" is an indexObj, and for these kinds of objects, "x" might
be an abbreviated version of a full name. Since the table behind
the indexObj is in the case above volatile, a crash might happen.
- Added API call Ns_SockInErrorState() since SSL_shutdown() must not
be called if a previous fatal error has occurred on a connection
i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or
SSL_ERROR_SSL.
- Provided local buffers for OpenSSL ERR_error_string handling to
avoid potential race conditions (OpenSSL keeps otherwise error
messages in static memory).
- Crypto support:
* New feature: scrypt Password-Based Key Derivation Function (RFC 7914)
The scrypt function is a modern replacement for crypt and bcrypt
and derives secret keys from a secret string. It is based on
memory- hard functions, which offer added protection against
attacks using custom hardware and GPU arrays.
The function requires the compilation of NaviServer against
OpenSSL 3.0 or newer (not yet released)
Example from RFC 7914:
% ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16
fdbabe1c9d3472007856e7190d01e9fe7c6ad7cbc8237830e77376634b373162
2eaf30d92e22a3886ff109279d9830dac727afb94a83ee6d8360cbdfa2cc0640
* "ns_md" and "ns_hmac": added flag "-encoding"
* Added options "-passphrase" to all "ns_crypto" commands
accepting a PEM file. In principle, every PEM file can be
protected by a pass phrase.
* Signing and verifying signatures: "::ns_crypto::md string" can
be used for signing and verifying of cryptgraphic signatures:
% set sig [::ns_crypto::md string \
-digest sha1 \
-encoding binary \
-sign /usr/local/src/naviserver/myprivate.pem \
"abcdefghijklmnopqrstuxvwxyz\n"]
% set vfy [::ns_crypto::md string \
-digest sha1 \
-verify /usr/local/src/naviserver/myprivate.pem \
-signature $sig \
"abcdefghijklmnopqrstuxvwxyz\n"]
- nscgi:
* ...
Documentation improvements:
---------------------------
- Improved cross references between man pages
- Added documentation to undocumented flags
- Added more example
- Improved spelling
- The following 48 man pages were updated since the last release:
admin-db.man, admin-install.man, admin-maintenance.man,
admin-tuning.man, commandlist.man, ns_adp.man, ns_adp_ctl.man,
ns_adp_info.man, ns_adp_parse.man, ns_asynclogfile.man,
ns_base64.man, ns_conn.man, ns_connchan.man, ns_crypt.man,
ns_crypto.man, ns_driver.man, ns_filestat.man, ns_hmac.man,
ns_hotp.man, ns_http.man, ns_job.man, ns_locationproc.man,
ns_log.man, ns_md.man, ns_md5.man, ns_parsefieldvalue.man,
ns_parseheader.man, ns_parseurl.man, ns_rand.man,
ns_register.man, ns_return.man, ns_server.man, ns_set.man,
ns_sha1.man, ns_sls.man, ns_sockcallback.man,
ns_sockselect.man, ns_tmpnam.man, ns_totp.man,
ns_urlspace.man, ns_uudecode.man, ns_uuencode.man,
ns_writer.man, nsv.man, returnstatus-cmds.man,
tcl-lib-file.man, tcl-libraries.man, tcl-overview.man
Configuration Changes:
----------------------
- Output warning to system log, when NaviServer was compiled without
zlib support, but the config file request compression. Before, the
compress enable request was silently ignored, giving no hint, why
compression attempts of a developer were ignored.
- Sample configurations:
* Documented usage of connection pools in sample config files.
* Added sample entries for new features (e.g. rate limits, log
file sanitizer)
* openacs-config.tcl:
. Changed driver installation to "global" in sample configuration
file. This change makes it easier for site admins to add
further servers (e.g. for virtual hosting).
. Simplified sample config file by using multiple IP addresses
per driver
* nsd-config.tcl:
. Changed driver installation to "global" in sample configuration
file. This change makes it easier for site admins to add
Code Changes:
-------------
- Improved scalability: provide different locks per urlspace IDs.
- Improved range checking for Ns_ObjvInt, Ns_ObjvLong,
Ns_ObjvWideInt and MemUnits:
* built-in support for objv parser This eases specification of the
C API, makes accepted ranges clear and provides consistent error
messages. These change fixes several potential crashes, when e.g.
passed-in values are used as array-indices (crash on negative
numbers) or where the values were casted to unsigned types (very
high unsigned values, potential memory alloc problems).
* Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj,
and Tcl_GetWideintFromObj was used without a range check.
- Argument parser: skip processing of optional arguments when the
number of provided arguments is equal to the number of required
parameters.
Background: This change allows a fully backward compatible version
of e.g. "ns_md5 $foo", no matter what the value of "$foo" is,
although the new version of "ns_md5" has now a "-binary" option.
- Refactored low-level socket I/O:
* All driver operations are now strictly non-blocking
* Use same infrastructure for all network I/O operations
(driver, connchan, ns_http, ...)
* Rewritten socketfile/sendfile handling (including *BSD support)
* Improved handling of closed and detached channels, e.g.delegated
to writers (new macros NS_CONN_REQUIRE_CONNECTED,
NS_CONN_REQUIRE_OPEN, NS_CONN_REQUIRE_CONFIGURED,
NS_CONN_REQUIRE_ALL)
* Improved protection for SIGPIPE signals.
- New API infrastructure: Ns_DList
The Ns_DList is similar to Tcl_DString, but operates on pointers
instead of characters. Like Tcl_DString, Ns_DList pre-allocates
some data and extends it when necessary. Ns_DList structures are
especially useful alternative to linked lists, when usually data
is added to the end. Since the data is kept as a dense
(potentially growing) array, the memory locality is much better
than with linked lists, resulting in better CPU cache hit rates.
- Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(),
where the first is for the file system and the latter for URLs.
Previously, both cases were handled identically.
- Function pointer handling:
* Created a Tcl hash type for function pointers, since ANSI/ISO C
forbids casting/comparison of data and function pointers. The
new type is used whenever a hash lookup is performed for a
function pointer.
* Fixed all other occurrences of such comparisons.
- Regressions testing:
* Removed false positive
* Switch to testing via ns_http in regression tests
(this makes it as well possible for testing https connections)
* Extended tests:
+ Added regression tests for "ns_rand"
+ Added nsssl tests
+ Added automatic certificate generation for test-server
* The following 35 tests were added or extended since the last release
adp.test, encoding.test, http.test, http_byteranges.test,
http_chunked.test, http_keep.test, https.test, misc.test,
ns_adp_compress.test, ns_base64.test, ns_cache.test,
ns_conn.test, ns_conn_host.test, ns_crypto.test,
ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test,
ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test,
ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test,
ns_reflow_text.test, ns_schedule.test, ns_server.test,
ns_serverpath.test, ns_set.test, ns_sha1.test,
ns_urlencode.test, ns_urlspace.test, ns_uuencode.test,
ns_writer.test, tclconnio.test, tclresp.test
- Improved portability:
* improved compatibility with LibreSSL (2.7 and 2.9)
* Windows changes (Many thanks to Andrew Piskorski):
+ Fixed Ns_LogRoll() to work on Windows.
+ WSASend receives as 5th argument flags, but not a pointer to flags.
+ Improved makefiles
+ Updated _MSC_VER version numbers to include Visual Studio 2019
+ Added macro NS_INLINE to achieve higher portability with (older?)
Microsoft compilers.
- Added warnings about potential misconfiguration of NaviServer "tcllib"
setting (Tcl written NaviServer modules)
- Marked compatibility wrappers Ns_SetThreadServer() and
Ns_SetThreadServer() explicitly as deprecated. Use
Ns_ThreadSetName() and Ns_ThreadGetName() instead.
- Build-system
* Added -DSYSTEM_MALLOC to default compile flags.
* Added nsssl to the default test target.
* Require RSA keys of size 2048: newer versions of OpenSSL refuse to work
with RSA keys of size 1024.
* Improved alignment with Tcl's current .m4 file to get rid of
"-prebind" deprecated message under macOS.
* Improved handling of posix thread library for FreeBSD and OpenBSD
- Use native thread_local storage for log handling when available
(experimental).
- Improved type cleanness for function pointers
- Improved code locality
- Improved error messages
- Improved structure packing
- Reduced variable scopes
- Added missing "extern" declarations
- Aligned function prototypes
- Dropped potentially dangerous call to alloca()
- Added typedefs for commonly used functions
- Reduce potential dangling pointer dereferences
- Aligned names of arguments in prototype with function definition
- Introduced use attribute-based approach for denoting
fall through in case statements
- Adding more declarations for PURE and CONST functions
- Improved configurability for clang-tidy
- Fixed macro name-clash with PostgreSQL
- Improved spelling
Modules:
--------
....
_______________________________________________
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel
