Re: [PATCH] mod_cache fixes: #8

2004-08-03 Thread Brian Akins
Bill Stoddard wrote: Please, no more specialized knobs which 99.9% of the world cares nothing about. How do you define that percentage? By domains? In that case 99.999% probably care nothing about what we are doing. If you look at total traffic, however, would not options that help the to

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Graham Leggett
Bill Stoddard wrote: To get mod_cache/mod_mem_cache (I know little or nothing about mod_disk_cache) really performing competatively against best-of-breed caches will require bypassing output filters (and prebuilding headers) and possibly bypassing or at least reworking input filters. The initial

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Bill Stoddard
Mads Toftum wrote: On Mon, Aug 02, 2004 at 04:54:07PM -0400, Brian Akins wrote: Any reason md4 was not used in mod_cache? In my ad hoc tests, it seems much faster. I do not know the in-and-outs of encryption, but is there any compelling reason to use md5 over md4 in this case? We don't care i

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Graham Leggett
Brian Akins wrote: On one hand, I think doing MD5 is sort of silly - just use the URL itself. *shrug* -- justin One of the goals of the new mod_cache was to support the caching of URL variants. I actually have somewhat of a solution: URL encode the uri and any vary elements: www.cnn.com/index.

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Mads Toftum
On Mon, Aug 02, 2004 at 04:54:07PM -0400, Brian Akins wrote: > Any reason md4 was not used in mod_cache? In my ad hoc tests, it seems > much faster. I do not know the in-and-outs of encryption, but is there > any compelling reason to use md5 over md4 in this case? We don't care > if someone "

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Brian Akins
Justin Erenkrantz wrote: MD5 has the possibility for collisions, too. Any reason md4 was not used in mod_cache? In my ad hoc tests, it seems much faster. I do not know the in-and-outs of encryption, but is there any compelling reason to use md5 over md4 in this case? We don't care if someo

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Brian Akins
Justin Erenkrantz wrote: MD5 has the possibility for collisions, too. What do squid or other proxies do? True. I'll see what others do. On one hand, I think doing MD5 is sort of silly - just use the URL itself. *shrug* -- justin With mod_disk_cache, how about urls such as: http://www.cnn.com

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Justin Erenkrantz
--On Monday, August 2, 2004 2:49 PM -0400 Bill Stoddard <[EMAIL PROTECTED]> wrote: To get mod_cache/mod_mem_cache (I know little or nothing about mod_disk_cache) really performing competatively against best-of-breed caches will require bypassing output filters (and prebuilding headers) and possib

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Justin Erenkrantz
--On Monday, August 2, 2004 3:01 PM -0400 Brian Akins <[EMAIL PROTECTED]> wrote: (and prebuilding headers) Is there a way to send pre built headers? mod_asis uses ap_scan_script_header_er which is fairly slow. That was what I fixed with apr_file_gets() last night. The code there was really nai

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Justin Erenkrantz
--On Monday, August 2, 2004 2:54 PM -0400 Brian Akins <[EMAIL PROTECTED]> wrote: The other bottleneck I looked at was MD5 as the on-disk naming scheme. I think MD5 is a poor choice here because it's not very fast. Ideally, switching to a variant of the times-33 hash might work out better. *shr

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Brian Akins
Bill Stoddard wrote: To get mod_cache/mod_mem_cache (I know little or nothing about mod_disk_cache) really performing competatively against best-of-breed caches will require bypassing output filters Yes. (and prebuilding headers) Is there a way to send pre built headers? mod_asis uses ap_

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Brian Akins
Justin Erenkrantz wrote: The other bottleneck I looked at was MD5 as the on-disk naming scheme. I think MD5 is a poor choice here because it's not very fast. Ideally, switching to a variant of the times-33 hash might work out better. *shrug* How to handle collisions? -- Brian Akins Senior Sy

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Bill Stoddard
Justin Erenkrantz wrote: mod_cache isn't doing anything stupid or damaging performance-wise, I'd like to start being more aggressive about what we can cache. From my perspective, these patches I've posted (and started to commit) are just the beginning of trying to get mod_cache on more solid gr

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Brian Akins
Justin Erenkrantz wrote: post_config's not a bad place for that. But, I've yet to get a good handle on my thoughts for the storage mechanism that mod_disk_cache is using. My hunch so far is that it's really inefficient. (The reading of the headers one-byte at a time with the brain-dead apr_fi

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Justin Erenkrantz
--On Monday, August 2, 2004 9:21 AM -0400 Brian Akins <[EMAIL PROTECTED]> wrote: In a low traffic site, yes. In a very high traffic site, with lots of objects, the mkdir's kill you. After a "while," most of the directories will be created. However, bringing up a "fresh" server behind a very b

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Justin Erenkrantz
--On Monday, August 2, 2004 8:46 AM -0400 Brian Akins <[EMAIL PROTECTED]> wrote: Another big speed-up may be to pre-make all of the directories. A simple script could use CacheRoot, |CacheDirLength|, and |CacheDirLevels to create them all. Just require that this script be ran before starting a

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Brian Akins
Jeff Trawick wrote: Wouldn't assuming that the directory is already there be sufficient (and then create the directory structure on the error path)? It looks to me that we only assume the directory structure exists if we had this very same file cached previously. In a low traffic site, yes. In

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Jeff Trawick
On Mon, 02 Aug 2004 08:46:07 -0400, Brian Akins <[EMAIL PROTECTED]> wrote: > Justin Erenkrantz wrote: > > > - > > static void mkdir_structure(disk_cache_conf *conf, char *file, > > apr_pool_t *pool) > > Another big speed-up may be to pre-make all of the directories. A simple script > could use

Re: [PATCH] mod_cache fixes: #8

2004-08-02 Thread Brian Akins
Justin Erenkrantz wrote: - static void mkdir_structure(disk_cache_conf *conf, char *file, apr_pool_t *pool) Another big speed-up may be to pre-make all of the directories. A simple script could use CacheRoot, |CacheDirLength|, and |CacheDirLevels to create them all. Just require that this scr

[PATCH] mod_cache fixes: #8

2004-08-02 Thread Justin Erenkrantz
--On Sunday, August 1, 2004 11:25 AM -0400 Bill Stoddard <[EMAIL PROTECTED]> wrote: Too many changes in one patch. Break this up into multiple consumable in 15 minute patches and I'll review them. Some more work, analysis, and tests yielded apr_file_gets() and MD5 as two more bottlenecks. I've