We implemented something quick yesterday using the fuse examples and
libmemcache (replicating that python example in C). We configured it to
just wrap an existing GlusterFS volume. Our initial test very crude test
was just to accelerate stating. Stating 18500 files over GlusterFS took
consistently ~22.5 seconds without the memcache layer. Stating the same
files (after hydrating the cache) took ~4.5 seconds consistently with
the memcache caching layer. We didn't really do anything for caching
content, no more time to play with it. . . didn't even try to do any
optimizations. I was just happy it helped validate idea.
If implemented truely as a caching solution, you shouldn't have to worry
about losing files either. Cache misses would fall back to the
filesystem and just rehydrate the cache. Or like you said, you could
have it fall back to a database instead of the filesystem. What I like
about using the filesystem as opposed to a database, is that it can be
used more generally to accelerate any exported filesystem.
Erik Osterman
Peter van Dijk wrote:
On 9-mei-2007, at 20:21, Erik Osterman wrote:
Python proof of concept code.
http://blog.koshigoe.jp/archives/2007/04/fuse.html
It doesn't appear to cache contents, only stat.
Also, on a philisophical side note. I am floored by how often almost
identical ideas are conceived of by different people at the same
time. Look at the date on that blog entry!
You have no idea how much this is scaring me right now ;)
Over the last two weeks i spent a lot of time discussing a memcachefs
(fuse-based) with two fellow geeks - applications that came to mind
were (a) the smarty cache (b) php sessions; for both cases, losing
files (as a whole, not random parts inside) is ok and readdir is
irrelevant, which allows cutting a lot of corners.
We're also pondering using mysql as the write-through bit, instead of
an actual filesystem, which wouldmake the storage stable and allow
readdir. No code so far, though.
Cheers, Peter
