Hi David,
We've been experimenting quite a bit with memory caches (on our Solaris
2.5.1 sparc machines), for much the same reasons: memory is cheap, and
for certain services (e.g., our web servers, which are serving most
documents out of AFS), a disk-based AFS cache simply hammers the disk
that contains the cache into the ground (consistently >%20 busy; >200ms
service times).
Unfortunately, despite the seeming advantages of memory caches, I'd have
to classify our experiences with memory caches as "mixed".
First of all, Transarc doesn't really consider the AFS memory cache (for
unix platforms) to be a "real" product. On March 5 1998, I posed the
following (exact quote) question to Transarc:
> In the past, Transarc has told us (in so many words) that the original
> reason the memory cache was provided was to allow diskless clients to be
> AFS clients. It was strongly implied that Transarc was not interested
> in focusing any developer resources on memory cache issues. Is this
> still an accurate summary of Transarc's attitude towards AFS memory
> caches?
On March 18 1998, we received this (exact quote) response from Transarc
development:
> This continues to be true. We test the memory cache with each release
> to make sure it is not broken but we do not expend too much of effort to
> stress test it like the disk cache. We do not have the resources to
> work on performance problems and optimizations for the mem cache.
On March 20 1998, we opened TR-42504, and requested the following enhancement:
> We have multiple machines that perform intensive/extensive AFS
> operations, such as our interactive timesharing service machines, and
> our web server machines (that serve most of their data out of AFS
> space). For these machines, the disk-based AFS cache crushes the disk
> that contains the cache into the ground--even when the disk cache is
> located on a disk dedicated to the AFS cache.
> We need to find some way to reduce the I/O bottleneck that a disk-based
> AFS cache becomes when AFS operations become intensive. However,
> Transarc is shooting down our options. Using a TMPFS for the AFS cache
> might reduce the disk I/O bottleneck, but it's not feasible for Transarc
> to support placing the AFS cache on a TMPFS (see TR42019). Using a
> memory cache instead of a disk cache would obviously eliminate the disk
> I/O bottleneck, but we feel extremely uneasy about doing this in a
> production environment when Transarc's attitude towards the memory cache
> is:
>> "We test the memory cache with each release to make sure it is not
>> broken but we do not expend too much of effort to stress test it like
>> the disk cache. We do not have the resources to work on performance
>> problems and optimizations for the mem cache."
> ...
> About the only option we have left is to try placing the AFS cache
> across multiple disks (by software striping, software RAID, et. al.).
> But we don't even know yet if this will work: the AFS cache manager
> might make low-level assumptions about the AFS cache's filesystem that
> are valid for a simple UFS, but are not valid when the UFS is in fact on
> top of a software stripe or software RAID. If the software
> striping/RAID wouldn't work, then we'd be forced to go out and buy
> dedicated hardware RAID systems! Either endeavor (software or hardware)
> represents a great deal of expense (both in terms of money, and in terms
> of the support levels required to assemble and maintain the systems).
> In short, the AFS client cache does not scale well with respect to AFS
> load. We need to find some way around this problem.
> From our point of view, currently the best way to alleviate this problem
> is for Transarc to treat the memory cache as a "real product". In other
> words, if Transarc were to dedicate the resources to stress-test the
> memory cache, work on performance problems, add optimizations, and
> generally give the memory cache as much attention as the disk cache, it
> would be feasible for us to use the memory cache to bypass the disk
> cache I/O bottlenecks that result in a machine that performs intensive
> AFS operations.
> Therefore, we request that Transarc treat the memory cache as no less
> important than the disk cache, as described in the previous paragraph.
We have yet to receive a response. Every time we prod Transarc
concerning TR-42504, we're told something that reduces to "product
development is still considering your request".
Also, we've discovered that AFS memory caches don't work very well on
the sun4m architecture, due to kernel memory restrictions. On sun4m, no
matter how much physical memory is in the machine, the kernel memory can
never occupy more than ~100MB. Therefore, with a decent-sized memory
cache (e.g., 64MB), it's very easy for AFS to request a bigger chunk of
contiguous kernel memory than is actually available, and completely hang
the machine as a result. As Transarc explained in TR-45737, on July 31
1998:
> [T]he hangs you've been experiencing on your web servers are caused by
> the AFS memcache implementation. What is happening is that the machines
> are running out of memory. In your case, AFS data in a memcache is
> stored in 8 KB chunks (as governed by "afsd -chunksize 13", but each
> directory structure is stored in a single cache chunk. Unlike AFS data
> which is stored in 8 KB chunks, each directory structure is stored in a
> single cache file which can be as large as neccessary to hold the entire
> directory listing. At the time of the hang, a user had tried to access
> a directory whose directory structure would occupy 468 pages (1.8 MB)
> and there is not enough memory to satisfy that request ... AFS
> development is working on a change to the src code...Delta
> afs3.4-9250-hang.when.memcache.runs.out.of.memory would ensure that the
> machine does not hang when it runs out of memory. Instead, users would
> see an error message; the request for the large directory would fail
> thereby stopping the machine from running out of physical memory.
We've been waiting for delta
afs3.4-9250-hang.when.memcache.runs.out.of.memory ever since. In the
meantime, we've had to revert to AFS disk caches on our web server
machines. (We had hoped to be able to use memory caches on those
machines despite Transarc's lack of commitment to support, but we really
had no choice but to go back to disk caches.)
Transarc's seeming unwillingness to devote any appreciable resources to
AFS memory caches is extremely frustrating. As a result, despite the
fact that our experiences with memory caches subjectively indicate that
memory caches can offer markedly better performance than disk caches,
going with memory caches isn't something we'd recommend to others. The
only thing we can really recommend is that if you agree with our
enhancement request (TR-42504), it would probably be a good idea to let
Transarc know.
Regards,
James
--
James Ralston / Systems & Networks / Computing and Information Services
University of Pittsburgh / 600 Epsilon Drive / Pittsburgh PA 15238-2887
"Computer, you and I need to have a little talk." - O'Brien, ST:DS9
