On 25.08.2006, at 19:09, Rick Cobb wrote:
Here's what we ran into when we did something similar; just cautionary
information, not trying to shoot anything down here.
Actually this gave me another idea...
(BTW...
The reason why I'm still not letting loose on this point:
althogh I like the idea of tool-based config, it has one
strong drawback. How am I going to make this persistent?
Per module? This is just plain unpracticable as it would
result in a "config-file" per module adding to complexity
if I would like to pull to config data from some other
source(s)).
Our approach was to create a separate repository / heirarchy of
configuration variables that is *not* module oriented,
Which is also what I'm after...
but
administrative task oriented. So, for example, we have a section
called
"ServiceNetwork" with a variable called "HTTPInterface". That
variable
gets one or more IP addresses. Only these variables are dynamic. To
get a dynamic update for one of these variables, any piece of C/C++
code
registers a callback based on the variable name.
.. but what I'd do is make this "callback" entirely transparent to the
caller. Instead, I'd split the config repository in 2 contexts:
shared and thread-private. The shared gets loaded, updated and the
thread-private is a partial copy of the shared, build-up during the
config lookup, dynamically. Each "reader" of the config will first go
ask its private context and then shared context for a value.
If found in private: ok. If found in shared: ok. Otherwise it updtes
its both private and shared context with the default value.
The locking would be minimized so that during the normal operation
only two threads (the reader and writer) would content for a mutex,
which is, given the majority of reads and tiny fractions of writes,
neglectable. Now you'd ask: where is the callback?
Simple: the reader therad, not founding the value in its private context
will set it (under lock) in the shared and register itself automatically
for having "interest" in this section. Any writer thereafter, which
updates
any of the keys of that section will walk the list of interested threads
and update their own *private* context after updating the shared one.
This way, the locking is minimized to absolute minimum. You will see
that
when I'm ready with the code. The task of the writer will not be cheap
as it will neeed to update not one but N places (depeding on howmany
threads expressed "interest" in particular section) but this way the
readers are cheap and fast as they will mostly find what they're
looking for in their own private cache.
This repository is read at startup, before the normal
ns_section/ns_param file.
OK. But I'd opt to give the startup option to read it before
or after the ns_section/ns_param file. This way there is
more flexibility what overrides what.
The declarations for these variables state whether changes to the
variable require a server restart to take effect, or not.
This is a good idea. Although I will not enforce a declaration
of the variable (it will be as dynamic as the current config)
I will add one qualifier: scope. This will be either startup
or runtime depicting wether the thing is read/set only at
startup only or not. The scope can be selected during the
lookup-falure followed by the shared config update automatically.
Our management console works entirely in terms of this repository;
YES. This is the other benefit of having it centralized as you
need NOT know in advance what modules you have out there. All is
in one place and can be saved, restored, uniformly controlled etc.
The nsd.tcl-equivalent, which is still written in terms of
ns_section/ns_param, then has access to those variables as TCL global
variables (just a convenience, not necessarily my best design
idea). If
more than one module needs access to a configuration variable, they
just
use it in this section. That lets us use normal AOLServer modules
out-of-the-box, yet coordinate their tuning by writing complicated
TCL.
This is where I'm still not sure what to do. I can easily attach
a (x)dbm hash underneath which would be perfectly OK. But with a
drawback of not being human-readable. I will have to think more
here...
The positive implications have been:
* People configuring our system rarely have to copy the same parameter
to different sections.
* People configuring our system never have to write TCL.
* No changes to existing modules are required
* Our modules, or our own edits to AOLServer modules, get dynamic
capabilities
* All configuration changes can be made over HTTP
* All of our modules have dynamic debug capability; in fact, for
almost
all of them, it's possible to adjust their debug settings over
HTTP, so
you can almost get to the point where you can debug a specific HTTP
request (of course, with lots of traffic, that doesn't work).
The negative implications have been:
* Our nsd.tcl-equivalent is *very* long and complicated (~3000 lines),
and is gradually becoming dominated by TCL control statements
(if/for/etc), not sections & params.
* If a variable requires calculation to become an internal
configuration
state, the calculation is often repeated in both TCL (nsd.tcl) and C.
* Startup time is slowed by all this processing, but it's not a big
deal
because we also recover a database during our startups, which
dominates
the startup time.
* It is very hard to see all the coupling for a specific variable.
* Any module that uses a dynamic variable has to code for it three
times, in three different areas: once for its use for
ns_section/ns_param, once for the global configuration area, and
once in
the callback.
We could mitigate some of these with better generators or code
conventions, but haven't.
Reading the above, I see no mention about how you do the locking?
If a writer chages a variable, he'd have to propagate this change
to readers, or? Now, readers, accessing the same variable must
access it under lock as some writer can call their registered callback
which updates the value at the same time, right?
In my (still) "vapourware", this is/must-be so, but the locking is
actually very cheap while most of the time reader will just lock
its private context, lookup and unlock. There will be no lock contenton
on that mutex. Only when one of the writers (which is a rare event) goes
and updates reader's private context (this being a very short operation
anyway) a short contention may happen. I believe this can be entirely
neglected. The cost of locking a mutex is low, if there is nobody
else I have to fight with. Actually, I'm sure this will work excellent.
Anyway, I will have to write this one in the next few days as
we already have plans for expading our product. We cannot
tolerate shutdowns of the server just to flip a bit any more.
This is now pressing us for some time and we've come to a point
where this must be done.
Thanks for the valuable information. I will make (our) nsconf or
nsregistry module public and we can then all see if this is usable
for others or not.
Cheers,
Zoran
