On Tue, 20 Aug 2002 [EMAIL PROTECTED] wrote:
> We are investigating using IPC rather then a file based
> structure but its purely investigation at this point.
>
> What are the speed diffs between an IPC cache and a
> Berkely DB cache. My gut instinct always screams 'Stay Off
> The Disk' but my gut is not always right.. Ok, rarely
> right.. ;)
IPC (for many definitions of that) has all sorts of odd limitations
and isn't that fast. Don't go there.
The disk is usually much faster than you think. Often overlooked
for caching is a simple file based cache.
Here's a story about that:
A while ago Graham Barr and I spend some time going through a number
of iterations for a "self cleaning" cache system. It would take
lots of writes and fewer reads. In each cache entry a number of
integers would be stored. Just storing the last thousand entries
would be enough.
We tried quite a few different approaches; the most noteworthy was a
system of semaphores to control access to a number of slots in a
BerkeleyDB. That should be pretty fast, right?
It got a bit complicated as our systems didn't support that many
semaphores, so we had to come up with a system for sharing the
semaphores across multiple "slots" in the database.
Designing and writing this implementation took a few days. It was
really cool.
Anyway, after fixing that and a few deadlocks we were benchmarking
away. The system was so clever. We thought it was simple and neat.
Okay, neat at least. And it was really slow. Slow. (~200 writes a
second on a 400MHz Pentium II if I recall correctly).
First we suspected we did something wrong with the semaphores, but
further benchmarking showed that the BerkeleyDB just wasn't that
fast for writing.
30 minutes thinking and 30 minutes typing code later we had a
prototype for a simple filebased system.
Now using good old Fcntl to control access to simple "flat files".
(Data serialized with pack("N*", ...); I don't think anything beats
"pack" and "unpack" for serializing data).
The expiration went into the data and purging the cache was a simple
cronjob to find files older than a few minutes and deleting them.
The performance? I don't remember the exact figure, but it was at
least several times faster than the BerkeleyDB system. And *much*
simpler.
The morale of the story: Flat files rock! ;-)
- ask
--
ask bjoern hansen, http://www.askbjoernhansen.com/ !try; do();
