> Random thoughts: > - Did the content have short expiration times (or recent change dates which > would result in the cache making agressive expiration estimates). That could > churn the cache.
No. files literally never change, when updates appear they are always new files, web pages just point to new ones each update. In this application these are all executable downloadable files, think FTP repository over HTTP. > - Was CacheMaxStreamingBuffer set appopropriately? (it may not be needed at all > if the content length header is included on all replies). C-L is included (SSIs and all similar dynamic stuff is disabled), not sure about CacheMaxStreamingBuffer, I'd need to go check. > - Did you try caching open file descriptors? I am rather curious if caching open > fds will be useful/practicle on Unix systems. Oh..., but this probably will not > help your disk throughput... nevermind. :-D > - It's probably worth noting in the doc that -each- child process will cache up > to MCacheSize KBytes. If you have 10 child processes, then you need > 10xMCacheSize Kbytes RAM available just for the cache (the same files could be > cached in each process). I wonder if we should, at startup, allocate MCacheSize > KB of shared storage and have mod_mem_cache allocate out of the shared pool. > Each child process would have it's own unique reference to the object, but the > object itself would only be cached once for all processes to access. I suspected that's where our memory running out was coming from, but it would have been helpful to have confirmation of my suspicion in the docs, yes. The problem is our cache needed to be quite large to cache very many of those large files, and we needed to run a lot of processes due to the mutex contentions with too many threads in one process (see the "[PATCH] remove some mutex locks in the worker MPM" thread)... so we kind of gave up on mod_mem_cache. This is kind of how this discussion branched off of that thread, sorry I didn't state that clearly earlier. It would be nice if there were some kind of shared cache, shared between processes. With large files like this it only needs to be read once it's primed with the most popular files... and which files are the most popular files doesn't change that often since we only make new releases every couple months... mod_file_cache works ok for this, but we need to develop something that "guesses" what will be most popular and generates the httpd.conf list and restarts apache before each new release is publicly linked on web pages but after the files are put live, to avoid our servers falling over with each new release... it's quite a pain and quite scary what will happen if those steps aren't followed correctly, that's why I was hoping Apache could manage it automatically with mod_mem_cache. Dave > > Bill > > > Apache 2.0.43, Solaris 8, Sun E220R, 4 gig memory, gig ethernet. We tried > > both Sun forte and gcc compilers. The problem was mod_mem_cache was just > > way too resource intensive when pounding on a machine that hard, trying to > > see if everything would fit into the cache... cpu/mutexes were very high, > > especially memory was out of control (we had many very large files, ranging > > from half dozen to two dozen megs, the most popular of those were what we > > really wanted cached), and we were running several hundred concurrent > > connections at once. Maybe a new cache loading/hit/removal algorithm that > > works better for many hits to very large files would solve it I dunno. > > > > We finally settled on listing out some of the most popular files out in the > > httpd.conf file for mod_file_cache, but that presents a management problem > > as what's most popular changes. It would have been nicer if apache could > > auto-sense the most popular files. Also it seems mod_file_cache has a file > > size limit but at least we got enough in there the disk wasn't bottlenecking > > anymore... > > > > Dave > > > > ----- Original Message ----- > > From: "Bill Stoddard" <[EMAIL PROTECTED]> > > To: <[EMAIL PROTECTED]> > > Sent: Wednesday, January 01, 2003 6:38 AM > > Subject: RE: [PATCH] remove some mutex locks in the worker MPM > > > > > > > > it may also have to do with caching we were doing (mod_mem_cache crashed > > and > > > burned, > > > What version were you running? What was the failure? If you can give me > > enough > > > info to debug the problem, I'll work on it. > > > > > > Bill > > > >
