Re: Problem with file descriptor handling in httpd 2.3.1
On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Configuration is very close to original, except for: 40c40 Listen myhost:8000 --- Listen 80 455,456c455,456 EnableMMAP off EnableSendfile off --- #EnableMMAP off #EnableSendfile off (because installation is on NFS, but the problem also occurs with those switches on) The following Modules are loaded: LoadModule authn_file_module modules/mod_authn_file.so LoadModule authn_anon_module modules/mod_authn_anon.so LoadModule authn_core_module modules/mod_authn_core.so LoadModule authz_host_module modules/mod_authz_host.so LoadModule authz_groupfile_module modules/mod_authz_groupfile.so LoadModule authz_user_module modules/mod_authz_user.so LoadModule authz_owner_module modules/mod_authz_owner.so LoadModule authz_core_module modules/mod_authz_core.so LoadModule access_compat_module modules/mod_access_compat.so LoadModule auth_basic_module modules/mod_auth_basic.so LoadModule auth_digest_module modules/mod_auth_digest.so LoadModule log_config_module modules/mod_log_config.so LoadModule env_module modules/mod_env.so LoadModule mime_magic_module modules/mod_mime_magic.so LoadModule cern_meta_module modules/mod_cern_meta.so LoadModule expires_module modules/mod_expires.so LoadModule headers_module modules/mod_headers.so LoadModule ident_module modules/mod_ident.so LoadModule usertrack_module modules/mod_usertrack.so LoadModule unique_id_module modules/mod_unique_id.so LoadModule setenvif_module modules/mod_setenvif.so LoadModule version_module modules/mod_version.so LoadModule mime_module modules/mod_mime.so LoadModule unixd_module modules/mod_unixd.so LoadModule status_module modules/mod_status.so LoadModule autoindex_module modules/mod_autoindex.so LoadModule asis_module modules/mod_asis.so LoadModule info_module modules/mod_info.so LoadModule suexec_module modules/mod_suexec.so LoadModule vhost_alias_module modules/mod_vhost_alias.so LoadModule negotiation_module modules/mod_negotiation.so LoadModule dir_module modules/mod_dir.so LoadModule imagemap_module modules/mod_imagemap.so LoadModule actions_module modules/mod_actions.so LoadModule speling_module modules/mod_speling.so LoadModule userdir_module modules/mod_userdir.so LoadModule alias_module modules/mod_alias.so LoadModule rewrite_module modules/mod_rewrite.so To reproduce you must use KeepAlive and your MaxKeepAliveRequests (Default:100) times concurrency must exceed the maximum number of FDs. Even without exceeding, you can use httpd -X and look at /proc/PID/fd during the test run. You should be able to notice a huge number of fds, all pointing to the index.html. Regards, Rainer
Re: Problem with file descriptor handling in httpd 2.3.1
On 04.01.2009 14:14, Ruediger Pluem wrote: On 01/04/2009 11:24 AM, Rainer Jung wrote: On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Can you try if you run in the same problem on 2.2.x with a file of size 257 bytes? I tried on the same type of system with event MPM and 2.2.11. Can't reproduce even with content file of size 257 bytes. The same file with trunk immediately reproduces the problem. Will try your patch/hack next. Thanks Rainer
Re: Problem with file descriptor handling in httpd 2.3.1
On 01/04/2009 03:26 PM, Rainer Jung wrote: On 04.01.2009 14:14, Ruediger Pluem wrote: On 01/04/2009 11:24 AM, Rainer Jung wrote: On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Can you try if you run in the same problem on 2.2.x with a file of size 257 bytes? I tried on the same type of system with event MPM and 2.2.11. Can't reproduce even with content file of size 257 bytes. Possibly you need to increase the number of threads per process with event MPM and the number of concurrent requests from ab. Regards Rüdiger
Re: Problem with file descriptor handling in httpd 2.3.1
On 01/04/2009 03:48 PM, Rainer Jung wrote: On 04.01.2009 15:40, Ruediger Pluem wrote: On 01/04/2009 03:26 PM, Rainer Jung wrote: On 04.01.2009 14:14, Ruediger Pluem wrote: On 01/04/2009 11:24 AM, Rainer Jung wrote: On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Can you try if you run in the same problem on 2.2.x with a file of size 257 bytes? I tried on the same type of system with event MPM and 2.2.11. Can't reproduce even with content file of size 257 bytes. Possibly you need to increase the number of threads per process with event MPM and the number of concurrent requests from ab. I increased the maximum KeepAlive Requests and the KeepAlive timeout a lot and during a longer running test I see always exactly as many open FDs for the content file in /proc/PID/fd as I had concurrency in ab. So it seems the FDs always get closed before handling the next request in the connection. After testing the patch, I'll try it again with 257 bytes on 2.2.11 with prefork or worker. IMHO this cannot happen with prefork on 2.2.x. So I guess it is not worth testing. It still confuses me that this happens on trunk as it looks like that ab does not do pipelining. Regards Rüdiger
Re: Problem with file descriptor handling in httpd 2.3.1
On 04.01.2009 16:22, Rainer Jung wrote: On 04.01.2009 15:56, Ruediger Pluem wrote: On 01/04/2009 03:48 PM, Rainer Jung wrote: On 04.01.2009 15:40, Ruediger Pluem wrote: On 01/04/2009 03:26 PM, Rainer Jung wrote: On 04.01.2009 14:14, Ruediger Pluem wrote: On 01/04/2009 11:24 AM, Rainer Jung wrote: On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Can you try if you run in the same problem on 2.2.x with a file of size 257 bytes? I tried on the same type of system with event MPM and 2.2.11. Can't reproduce even with content file of size 257 bytes. Possibly you need to increase the number of threads per process with event MPM and the number of concurrent requests from ab. I increased the maximum KeepAlive Requests and the KeepAlive timeout a lot and during a longer running test I see always exactly as many open FDs for the content file in /proc/PID/fd as I had concurrency in ab. So it seems the FDs always get closed before handling the next request in the connection. After testing the patch, I'll try it again with 257 bytes on 2.2.11 with prefork or worker. IMHO this cannot happen with prefork on 2.2.x. So I guess it is not worth testing. It still confuses me that this happens on trunk as it looks like that ab does not do pipelining. ^The strace log shows, that the sequence really is - new connection - read request - open file - send response - log request repeat this triplet a lot of times (maybe as long as KeepAlive is active) and then there are a lot of close() for the content files. Not sure, about the exact thing that triggers the close. So I don't necessarily see pipelining (in the sense of sending more requests before responses return) being necessary. I tested your patch (worker, trunk): It does not help. I then added an error log statement directly after the requests++ and it shows this number is always 1. I can now even reproduce without load. Simply open a connection and send hand crafted KeepAlive requests via telnet. The file descriptors are kept open as long as the connection is alive. I'll run under the debugger to see, how the stack looks like, when the file gets closed. Since the logging is done much earlier (directly after eahc request) the problem does not seem to be directly related to EOR. It looks like somehow the close file cleanup does not run when the request pool is destroyed or maybe it is registered with the connection pool. gdb should help. More later. Rainer
Re: Problem with file descriptor handling in httpd 2.3.1
On 04.01.2009 15:40, Ruediger Pluem wrote: On 01/04/2009 03:26 PM, Rainer Jung wrote: On 04.01.2009 14:14, Ruediger Pluem wrote: On 01/04/2009 11:24 AM, Rainer Jung wrote: On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Can you try if you run in the same problem on 2.2.x with a file of size 257 bytes? I tried on the same type of system with event MPM and 2.2.11. Can't reproduce even with content file of size 257 bytes. Possibly you need to increase the number of threads per process with event MPM and the number of concurrent requests from ab. I increased the maximum KeepAlive Requests and the KeepAlive timeout a lot and during a longer running test I see always exactly as many open FDs for the content file in /proc/PID/fd as I had concurrency in ab. So it seems the FDs always get closed before handling the next request in the connection. After testing the patch, I'll try it again with 257 bytes on 2.2.11 with prefork or worker. Regards, Rainer
Re: Problem with file descriptor handling in httpd 2.3.1
On 01/04/2009 12:49 AM, Rainer Jung wrote:
On 04.01.2009 00:36, Paul Querna wrote:
Rainer Jung wrote:
During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too
many open files. I used strace and the problem looks like this:
- The test case is using ab with HTTP keep alive, concurrency 20 and a
small file, so doing about 2000 requests per second.
MaxKeepAliveRequests=100 (Default)
- the file leading to EMFILE is the static content file, which can be
observed to be open more than 1000 times in parallel although ab
concurrency is only 20
- From looking at the code it seems the file is closed during a
cleanup function associated to the request pool, which is triggered by
an EOR bucket
Now what happens under KeepAlive is that the content files are kept
open longer than the handling of the request, more precisely until the
closing of the connection. So when MaxKeepAliveRequests*Concurrency
MaxNumberOfFDs we run out of file descriptors.
I observed the behaviour with 2.3.1 on Linux (SLES10 64Bit) with
Event, Worker and Prefork. I didn't yet have the time to retest with
2.2.
It should only happen in 2.3.x/trunk because the EOR bucket is a new
feature to let MPMs do async writes once the handler has finished
running.
And yes, this sounds like a nasty bug.
I verified I can't reproduce with the same platform and 2.2.11.
Not sure I understand the EOR asynchronicity good enough to analyze the
root cause.
Can you try the following patch please?
Index: server/core_filters.c
===
--- server/core_filters.c (Revision 731238)
+++ server/core_filters.c (Arbeitskopie)
@@ -367,6 +367,7 @@
#define THRESHOLD_MIN_WRITE 4096
#define THRESHOLD_MAX_BUFFER 65536
+#define MAX_REQUESTS_QUEUED 10
/* Optional function coming from mod_logio, used for logging of output
* traffic
@@ -381,6 +382,7 @@
apr_bucket_brigade *bb;
apr_bucket *bucket, *next;
apr_size_t bytes_in_brigade, non_file_bytes_in_brigade;
+int requests;
/* Fail quickly if the connection has already been aborted. */
if (c-aborted) {
@@ -466,6 +468,7 @@
bytes_in_brigade = 0;
non_file_bytes_in_brigade = 0;
+requests = 0;
for (bucket = APR_BRIGADE_FIRST(bb); bucket != APR_BRIGADE_SENTINEL(bb);
bucket = next) {
next = APR_BUCKET_NEXT(bucket);
@@ -501,11 +504,22 @@
non_file_bytes_in_brigade += bucket-length;
}
}
+else if (bucket-type == ap_bucket_type_eor) {
+/*
+ * Count the number of requests still queued in the brigade.
+ * Pipelining of a high number of small files can cause
+ * a high number of open file descriptors, which if it happens
+ * on many threads in parallel can cause us to hit the OS limits.
+ */
+requests++;
+}
}
-if (non_file_bytes_in_brigade = THRESHOLD_MAX_BUFFER) {
+if ((non_file_bytes_in_brigade = THRESHOLD_MAX_BUFFER)
+|| (requests MAX_REQUESTS_QUEUED)) {
/* ### Writing the entire brigade may be excessive; we really just
- * ### need to send enough data to be under THRESHOLD_MAX_BUFFER.
+ * ### need to send enough data to be under THRESHOLD_MAX_BUFFER or
+ * ### under MAX_REQUESTS_QUEUED
*/
apr_status_t rv = send_brigade_blocking(net-client_socket, bb,
(ctx-bytes_written), c);
This is still some sort of a hack, but maybe helpful to understand if this is
the problem.
Regards
Rüdiger
Index: server/core_filters.c
===
--- server/core_filters.c (Revision 731238)
+++ server/core_filters.c (Arbeitskopie)
@@ -367,6 +367,7 @@
#define THRESHOLD_MIN_WRITE 4096
#define THRESHOLD_MAX_BUFFER 65536
+#define MAX_REQUESTS_QUEUED 10
/* Optional function coming from mod_logio, used for logging of output
* traffic
@@ -381,6 +382,7 @@
apr_bucket_brigade *bb;
apr_bucket *bucket, *next;
apr_size_t bytes_in_brigade, non_file_bytes_in_brigade;
+int requests;
/* Fail quickly if the connection has already been aborted. */
if (c-aborted) {
@@ -466,6 +468,7 @@
bytes_in_brigade = 0;
non_file_bytes_in_brigade = 0;
+requests = 0;
for (bucket = APR_BRIGADE_FIRST(bb); bucket != APR_BRIGADE_SENTINEL(bb);
bucket = next) {
next = APR_BUCKET_NEXT(bucket);
@@ -501,11 +504,22 @@
non_file_bytes_in_brigade += bucket-length;
}
}
+else if (bucket-type == ap_bucket_type_eor) {
+/*
+ * Count the number of requests still queued in the brigade.
+ * Pipelining of a high number of small files can cause
+ * a high number of open file descriptors, which if it happens
+ * on many threads in parallel can
Re: Problem with file descriptor handling in httpd 2.3.1
On 04.01.2009 15:04, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. MaxKeepAliveRequests=100 (Default) - the file leading to EMFILE is the static content file, which can be observed to be open more than 1000 times in parallel although ab concurrency is only 20 - From looking at the code it seems the file is closed during a cleanup function associated to the request pool, which is triggered by an EOR bucket Now what happens under KeepAlive is that the content files are kept open longer than the handling of the request, more precisely until the closing of the connection. So when MaxKeepAliveRequests*Concurrency MaxNumberOfFDs we run out of file descriptors. I observed the behaviour with 2.3.1 on Linux (SLES10 64Bit) with Event, Worker and Prefork. I didn't yet have the time to retest with 2.2. It should only happen in 2.3.x/trunk because the EOR bucket is a new feature to let MPMs do async writes once the handler has finished running. And yes, this sounds like a nasty bug. I verified I can't reproduce with the same platform and 2.2.11. Not sure I understand the EOR asynchronicity good enough to analyze the root cause. Can you try the following patch please? Here's the gdb story: When the content file gets opened, its cleanup is correctly registered with the request pool. Later in core_filters.c at the end of function ap_core_output_filter() line 528 we call setaside_remaining_output(). This goes down the stack via ap_save_brigade(), file_bucket_setaside() to apr_file_setaside(). This kills the cleanup for the request pool and adds it instead to the transaction (=connection) pool. There we are. 2.2.x has a different structure, although I can also see two calls to ap_save_brigade() in ap_core_output_filter(), but they use different pools as new targets, namely a deferred_write_pool resp. input_pool. So now we know, how it happens, but I don't have an immediate idea how to solve it. Regards, Rainer
Re: Problem with file descriptor handling in httpd 2.3.1
On 04.01.2009 17:57, Rainer Jung wrote:
When the content file gets opened, its cleanup is correctly registered
with the request pool. Later in core_filters.c at the end of function
ap_core_output_filter() line 528 we call setaside_remaining_output().
...
2.2.x has a different structure, although I can also see two calls to
ap_save_brigade() in ap_core_output_filter(), but they use different
pools as new targets, namely a deferred_write_pool resp. input_pool.
And the code already contains the appropriate hint:
static void setaside_remaining_output(...)
{
...
if (make_a_copy) {
/* XXX should this use a separate deferred write pool, like
* the original ap_core_output_filter?
*/
ap_save_brigade(f, (ctx-buffered_bb), bb, c-pool);
...
}
Re: Problem with file descriptor handling in httpd 2.3.1
On 04.01.2009 15:56, Ruediger Pluem wrote: On 01/04/2009 03:48 PM, Rainer Jung wrote: On 04.01.2009 15:40, Ruediger Pluem wrote: On 01/04/2009 03:26 PM, Rainer Jung wrote: On 04.01.2009 14:14, Ruediger Pluem wrote: On 01/04/2009 11:24 AM, Rainer Jung wrote: On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Can you try if you run in the same problem on 2.2.x with a file of size 257 bytes? I tried on the same type of system with event MPM and 2.2.11. Can't reproduce even with content file of size 257 bytes. Possibly you need to increase the number of threads per process with event MPM and the number of concurrent requests from ab. I increased the maximum KeepAlive Requests and the KeepAlive timeout a lot and during a longer running test I see always exactly as many open FDs for the content file in /proc/PID/fd as I had concurrency in ab. So it seems the FDs always get closed before handling the next request in the connection. After testing the patch, I'll try it again with 257 bytes on 2.2.11 with prefork or worker. IMHO this cannot happen with prefork on 2.2.x. So I guess it is not worth testing. It still confuses me that this happens on trunk as it looks like that ab does not do pipelining. ^The strace log shows, that the sequence really is - new connection - read request - open file - send response - log request repeat this triplet a lot of times (maybe as long as KeepAlive is active) and then there are a lot of close() for the content files. Not sure, about the exact thing that triggers the close. So I don't necessarily see pipelining (in the sense of sending more requests before responses return) being necessary. I tested your patch (worker, trunk): It does not help. I then added an error log statement directly after the requests++ and it shows this number is always 1. Regards, Rainer
Re: Where to initialize a global pool/hash - server create or child_init?
On Sun, Jan 4, 2009 at 05:03, Jacques Amar jal...@amar.com wrote: Where to begin I am creating a global pool/hash in which I save cached, hard to calculate data (pre-compiled regex expressions etc.). I also store pages I've created using this data in memcached (using the APR interface). I can't save the calculated regex data in memcached. I initially followed the advice of creating a private pool, hash and mutex inside a child_init hook in my server config structure, protecting all access with the mutex. The module works well enough in regular httpd. However, when I tried this in worker MPM I got constant: [notice] child pid x exit signal Segmentation fault (11) after a few page loads. On a whim, I moved the whole creation of these structures into a server config hook. All these issues seem to have vanished. I have theorized that the pool I am creating at every child creation does not properly work. I am looking for a discussion of the pros and cons of creating this in a per child hook, versus the one time server create hook and any other pointers to help decide (and debug) where/when I should use one or the other. pre/post_config are run as the user who starts the server (root typically). child_init is run as the httpd user (configured with the User config directive). If the hash is read-only (it is never changed after initialisation) I guess you don't even need the mutex. In this case, I guess there's no difference between child_init and post_config (except the effective UID that runs the code, as said above). If the data is changed by one worker and you want the change be visible to all other workers, you have to put your initialisation in post_config. Regarding debugging, run your apache in debug (single-process) mode (apache2 -X) and check if you still get the segfaults. If yes, load apache2 in a debugger (I use gdb): gdb $HOME/usr/sbin/apache2 set args -f $HOME/etc/apache2/apache2.conf -X break my_handler run Issue a request and check where it segfaults. Sorin -- A: Because it reverses the logical flow of conversation. Q: Why is top-posting frowned upon? A: Top-posting. Q: What is the most annoying thing in e-mail?
Re: Why is r-handler a garbled string?
On Jan 2, 2009, at 12:14 PM, John David Duncan wrote: Yes, in fact the package does contain config.nice, which begins: CC=/opt/SUNWspro/bin/cc; export CC CFLAGS=-xO3 -xarch=386 -xchip=pentium -xspace -Xa -xildoff - xc99=all -DSSL_EXPERIMENTAL -DSSL_ENGINE -xO4; export CFLAGS That's what was passed in by the builders: the configure script will make some stuff up itself. Those flags alone don't do the trick for me, but the ones from apr- config make it all work. Good to hear. Yes, we've been struggling with documentation and/or a reusable build environment for out-of-tree modules. I think there was some initiative in that direction, but don't know what came of it. S. -- Sander Temme scte...@apache.org PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF smime.p7s Description: S/MIME cryptographic signature
Re: Problem with file descriptor handling in httpd 2.3.1
On 01/04/2009 11:24 AM, Rainer Jung wrote: On 04.01.2009 01:51, Ruediger Pluem wrote: On 01/04/2009 12:49 AM, Rainer Jung wrote: On 04.01.2009 00:36, Paul Querna wrote: Rainer Jung wrote: During testing 2.3.1 I noticed a lot of errors of type EMFILE: Too many open files. I used strace and the problem looks like this: - The test case is using ab with HTTP keep alive, concurrency 20 and a small file, so doing about 2000 requests per second. What is the exact size of the file? It is the index.html, via URL /, so size is 45 Bytes. Can you try if you run in the same problem on 2.2.x with a file of size 257 bytes? Regards Rüdiger
Re: Problem with file descriptor handling in httpd 2.3.1
On 01/04/2009 06:28 PM, Rainer Jung wrote:
On 04.01.2009 17:57, Rainer Jung wrote:
When the content file gets opened, its cleanup is correctly registered
with the request pool. Later in core_filters.c at the end of function
ap_core_output_filter() line 528 we call setaside_remaining_output().
...
2.2.x has a different structure, although I can also see two calls to
ap_save_brigade() in ap_core_output_filter(), but they use different
pools as new targets, namely a deferred_write_pool resp. input_pool.
And the code already contains the appropriate hint:
static void setaside_remaining_output(...)
{
...
if (make_a_copy) {
/* XXX should this use a separate deferred write pool, like
* the original ap_core_output_filter?
*/
ap_save_brigade(f, (ctx-buffered_bb), bb, c-pool);
...
}
Thanks for the analysis and good catch. Maybe I have a look into this by
tomorrow.
Regards
Rüdiger
Issues with mod_disk_cache and htcacheclean
I posted this on the users list, but was advised to post it to dev as well, since it seemed relevant to developers. Hope that's ok... I am using Apache 2.2.9 on Linux AMD64, built from source. There is one server running two builds of Apache - a lightweight front-end caching reverse proxy configuration using mod_disk_cache, and a heavyweight mod_perl back end. I use caching to relieve load on the server when many people request the same page at once. The website is dynamic and contains millions of page permutations. Thus the cache has a tendency to get fairly large, unless it is pruned. So I have been trying to use htcacheclean to achieve this. There have been some issues, which I will outline below. First, I found that htcacheclean was not able to keep up with pruning the cache. It just kept growing. I initially ran htcacheclean in daemon mode, thus: htcacheclean -i -t -n -d60 -p/var/cache/www -l1000M CacheDirLevels was 3 and CacheDirLength 1. The cache would just keep getting bigger, to multiple GB. Additionally, even doing a du on the cache could take hours to complete. I also noticed that iowait would spike when I tried running htcacheclean in non-daemon mode. It would not keep up at all using the -n (nice) option; when I took that off, the iowait would go through the roof and the process would take hours to complete. This was on a quad core AMD64 server with 4 x 10k SCSI drives in hardware RAID0. Upon investigation, I discovered that the cache was a lot deeper than I expected. In addition to the three levels specified in CacheDirLevels, there were then additional levels of subdirectories beneath .vary subdirs. For each .header file, there was a .vary subdir with three levels of directory below that. Simply traversing this tree with du could take a long time - hours sometimes, depending on how long the server had been running without a cache clear. I discovered that the .vary subdirs were caused by my configuration, which was introducing a Vary http header. This came from two sources: First, mod_deflate. I found this out from this helpful page: http://www.digitalsanctuary.com/tech-blog/general/apache-mod_deflate-and-mod_cache-issues.html So I disabled mod_deflate, since it seemed to be producing a huge number of cache entries for each file - a different one for every browser. But after disabling mod_deflate, the .vary subdirs were still there. I also had this line in my config: Header add Vary Cookie This is necessary because users on my site set options for how the site is displayed. When I tried disabling this cookie Vary header, the number of directories went down substantially, to the expected three levels. The cache structure was much simpler, and it seemed that htcacheclean could keep up with this. However, the site was broken - since the same page for different users with different options would be cached only once. So someone who had no ads or no pics would request a page that someone else had recently requested (with different options), and they would get that other person's options. Not good. So I had to switch the vary header for cookies back on, so that pages would get differentiated in the cache based on cookie. But now I was back to square one - six effective levels of subdirectory, which htcacheclean could not keep up with. After some thought, I ended up changing CacheDirLevels to 2, to try to reduce the depth of the tree. Now I had fewer subdirs, but more files in each one. Also, the size of the cache, via du, always seems to be much higher than specified for htcacheclean. I lowered the limit to 100M, but still the cache is regularly up at 180MB or 200MB. This seems counter-intuitive, since htcacheclean doesn't appear to be taking the true size of the cache into account (i.e. including all the subdirs, which also take up space and presumably are what cause the discrepancy). I also noticed something else: htcacheclean was leaving behind .header files. When it cleaned the .vary subdirs, it seemed to leave behind the corresponding .header files. These would accumulate, causing the iowait to gradually increase, presumably due to the size of the directories. I would rotate (clear) the cache manually at midnight. The behavior I would see (via munin monitoring tool) was that iowait would then remain at zero for about 12 hours, but then would gradually become visible as the number of .header files would accumulate. So I wrote a perl script which could go through the cache, and look for .header files, and for each one found, see if a corresponding .vary subdir exists for it. If not, then the .header file is deleted. I then run another script to prune empty subdirectories. Currently I run this combination every 10 minutes - first a non-daemon invocation of htcacheclean, followed by the header prune script, followed by the empty subdirs pruning script. This seems to keep the cache small, and iowait is not noticeable any more, since the junk .header files are now disposed of
Bug report for Apache httpd-1.3 [2009/01/04]
+---+ | Bugzilla Bug ID | | +-+ | | Status: UNC=Unconfirmed NEW=New ASS=Assigned| | | OPN=ReopenedVER=Verified(Skipped Closed/Resolved) | | | +-+ | | | Severity: BLK=Blocker CRI=Critical REG=Regression MAJ=Major | | | | MIN=Minor NOR=NormalENH=Enhancement TRV=Trivial | | | | +-+ | | | | Date Posted | | | | | +--+ | | | | | Description | | | | | | | |10744|New|Nor|2002-07-12|suexec might fail to open log file| |10747|New|Maj|2002-07-12|ftp SIZE command and 'smart' ftp servers results i| |10760|New|Maj|2002-07-12|empty ftp directory listings from cached ftp direc| |14518|Opn|Reg|2002-11-13|QUERY_STRING parts not incorporated by mod_rewrite| |16013|Opn|Nor|2003-01-13|Fooling mod_autoindex + IndexIgnore | |16631|Inf|Min|2003-01-31|.htaccess errors logged outside the virtual host l| |17318|Inf|Cri|2003-02-23|Abend on deleting a temporary cache file if proxy | |19279|Inf|Min|2003-04-24|Invalid chmod options in solaris build| |21637|Inf|Nor|2003-07-16|Timeout causes a status code of 200 to be logged | |21777|Inf|Min|2003-07-21|mod_mime_magic doesn't handle little gif files| |22618|New|Maj|2003-08-21|MultiViews invalidates PATH_TRANSLATED if cgi-wrap| |25057|Inf|Maj|2003-11-27|Empty PUT access control in .htaccess overrides co| |26126|New|Nor|2004-01-14|mod_include hangs with request body | |26152|Ass|Nor|2004-01-15|Apache 1.3.29 and below directory traversal vulner| |26790|New|Maj|2004-02-09|error deleting old cache file | |29257|Opn|Nor|2004-05-27|Problem with apache-1.3.31 and mod_frontpage (dso,| |29498|New|Maj|2004-06-10|non-anonymous ftp broken in mod_proxy | |29538|Ass|Enh|2004-06-12|No facility used in ErrorLog to syslog| |30207|New|Nor|2004-07-20|Piped logs don't close read end of pipe | |30877|New|Nor|2004-08-26|htpasswd clears passwd file on Sun when /var/tmp i| |30909|New|Cri|2004-08-28|sporadic segfault resulting in broken connections | |31975|New|Nor|2004-10-29|httpd-1.3.33: buffer overflow in htpasswd if calle| |32078|New|Enh|2004-11-05|clean up some compiler warnings | |32539|New|Trv|2004-12-06|[PATCH] configure --enable-shared= brocken on SuSE| |32974|Inf|Maj|2005-01-06|Client IP not set | |33086|New|Nor|2005-01-13|unconsistency betwen 404 displayed path and server| |33495|Inf|Cri|2005-02-10|Apache crashes with WSADuplicateSocket failed for| |33772|New|Nor|2005-02-28|inconsistency in manual and error reporting by sue| |33875|New|Enh|2005-03-07|Apache processes consuming CPU| |34108|New|Nor|2005-03-21|mod_negotiation changes mtime to mtime of Document| |34114|New|Nor|2005-03-21|Apache could interleave log entries when writing t| |34404|Inf|Blk|2005-04-11|RewriteMap prg can not handle fpout | |34571|Inf|Maj|2005-04-22|Apache 1.3.33 stops logging vhost| |34573|Inf|Maj|2005-04-22|.htaccess not working / mod_auth_mysql| |35424|New|Nor|2005-06-20|httpd disconnect in Timeout on CGI| |35439|New|Nor|2005-06-21|Problem with remove /../ in util.c and mod_rewri| |35547|Inf|Maj|2005-06-29|Problems with libapreq 1.2 and Apache::Cookie | |3|New|Nor|2005-06-30|Can't find DBM on Debian Sarge| |36375|Opn|Nor|2005-08-26|Cannot include http_config.h from C++ file| |37166|New|Nor|2005-10-19|Under certain conditions, mod_cgi delivers an empt| |37185|New|Enh|2005-10-20|AddIcon, AddIconByType for OpenDocument format| |37252|New|Reg|2005-10-26|gen_test_char reject NLS string | |38989|New|Nor|2006-03-15|restart + piped logs stalls httpd for 24 minutes (| |39104|New|Enh|2006-03-25|[FR] fix build with -Wl,--as-needed | |39287|New|Nor|2006-04-12|Incorrect If-Modified-Since validation (due to syn| |39937|New|Nor|2006-06-30|Garbage output if README.html is gzipped or compre| |40176|New|Nor|2006-08-03|magic and mime| |40224|Ver|Nor|2006-08-10|System time crashes Apache @year 2038 (win32 only?| |41279|New|Nor|2007-01-02|Apache 1.3.37 htpasswd is vulnerable to buffer ove| |42355|New|Maj|2007-05-08|Apache 1.3 permits non-rfc HTTP error code = 600 | |43626|New|Maj|2007-10-15|r-path_info returning invalid value |
