On Mon, 18 Sep 2006, Brian Akins wrote:
Niklas Edmundsson wrote:
Extra tracking sounds unnecessary if you can do it in a way that
doesn't need it.
It's not "extra" it just adding some tracking. When an objects gets cached
log (sql, db, whatever) that /blah/foo/bar.html is cached as
/cache/x/y/something.meta. Then it's very easy to ask the "store" what is
/blah/foo/bar.html cached as? There may be multiples because of vary.
"Extra" because you already have the needed info to puzzle the things
together...
* Clients read from the cache as files are being cached.
That's the hard one, IMO
But the implementation was rather easy once the "cache to separate
file and mv to correct location"-stuff was ripped out. Or, as easy as
building your own bucket-type is.
* Only one session caches the same file.
Easy to do if we use deterministic tmp files and not the way we currently do
it. Then all you have to do is when creating temp files use O_EXCL.
Or, if we skip the tmp files altogether.
* Header/Body updates.
Eaiser with seperate files like mod_disk_cache does now.
True.
* No index/files out-of-sync issues. Ever.
Hard to guarantee, but not impossible. Always to index when storing file and
remove when deleting. This should use something like providers so it's not
in core cache code and can be easily modified.
With locks, yes it's possible but also a hassle to get right with
performance intact.
Not really that hard. Trust me it has been done...
I'll take your word for that.
We, as a ftp mirror operated by a non-profit computer club, have a
slightly different usecase with single files larger than machine RAM
and a working set of approx 40 times larger than RAM. Some bad design
decisions in mod_disk_cache becomes really visible in this
environment.
Seems to me you should approach problem differently, like rsyncing the
mirrored content. I don't know your environment, but was just what I cam up
with off the top of my head.
Try rsyncing a few TB of content onto a few hundred GB of cache disk
and see how that works out for you :)
Our setup is briefly described here by the way:
http://ftp.acc.umu.se/mirror/ftp-about.html
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---------------------------------------------------------------------------
A closed mouth gathers no feet.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
