In terms of distributed key/value pair data structures, there are two
additional methods we've used here at AOL:
* AV (auto variables)
* NV (network variables)
AV's are a read-only data structure that under the covers is built on a
sorted list for quick access, and small size. The files get built and
then distributed to the front-end servers where the updated versions are
automatically swapped in by code that looks at the last modification
time for the AV file, and also some additional code which maintains
reference counts for AV usage across interps. We tend to use RDIST to
distribute these files from the server that creates them to the front-ends.
NV's are best suited for relatively small data sets which have frequent
updates. The classic example is a stock ticker application. Updates are
"streamed" to connected front-ends via persistent DCI_RPC connections
between the NV writer and those front-end servers.
Both of the mechanisms are similar to how you would use NSV (NaviServer
variables) locally, but can be used with a farm of connected front-end
hosts. NV's also provide some additional functionality that you don't
get with NSV's out the box, for example, a configuration option that
will periodically dump the contents from memory to disk, as well as
reading the version stored on disk at startup for persistent data
storage across restarts.
A third approach we use is to use our SOB (small object blaster) servers
and APIs to store data. The data can be anything, but most often is a
Tcl list. The SOB APIs provide various caching levels, which depending
on your application, might help with memory size since you'd only be
caching data that was requested, as opposed to loading an entire data
set into memory as you would with NSVs, AVs, and NVs.
Anyhow, these APIs are part of the nsdci AOLserver module which we will
be releasing soon.
- n
[EMAIL PROTECTED] wrote:
On Monday 03 July 2006 08:33, John Buckman wrote:
One of the reasons I want to use Berkeley DB is that I'd like every
web page string to be a BDB database lookup, allowing wiki-style
correcting of strings on a web page (ie, anyone can correct, on the
spot, any translated text on any page). SQL just wouldn't work with
that design, with a dozen or so db lookups per page.
So the design requires a number of queries, too many for responsive
performance from an SQL database. However, if you use the BDB database, you
will still have to go through some sort of database API, either the one which
exists, or a custom designed API. The purpose of the built in API is to share
connections across threads and even virtual servers, but it isn't really
suited for simple key/value lookup.
If the number of key/value pairs is small, everything could be loaded into
memory, into an nsv_array.
If the number of key/value pairs is large, then the solution is to load pairs
into memory only on first use. Maybe a high-use cache could be developed and
pre-loaded on server startup. If the cache will eventually grow too large,
then ns_cache could be used to expire stale data. Maybe BDB doesn't have the
option to unload memory mapped regions. If this is true, then ns_cache would
be the better choice to control memory usage.
But none of these solutions deals with how the key/value pairs are added to
the database and distributed to one or more AOLservers. At some point there
has to be a single master list of these pairs. Loading all these into memory,
or into a BDB or DJB cdb requires only one SQL query. You can even do one
query to write the database on the master, then distribute this with rsync or
whatever to replace the old database.
Overall, it seems like BDB or cdb would have the same memory requirements of
an nsv array, but they require some kind of API to get to this memory. Since
it is just key/value pairs, this API overhead is a total waste: you have to
use it even when the data is already in memory.
tom jackson
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]>
with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject:
field of your email blank.
