[STATUS] (flood) Wed Jul 21 23:45:32 EDT 2004
flood STATUS: -*-text-*- Last modified at [$Date: 2003/07/01 20:55:12 $] Release: 1.0: Released July 23, 2002 milestone-03: Tagged January 16, 2002 ASF-transfer: Released July 17, 2001 milestone-02: Tagged August 13, 2001 milestone-01: Tagged July 11, 2001 (tag lost during transfer) RELEASE SHOWSTOPPERS: * Everything needs to work perfectly Other bugs that need fixing: * I get a SIGBUS on Darwin with our examples/round-robin-ssl.xml config, on the second URL. I'm using OpenSSL 0.9.6c 21 dec 2001. * iPlanet sends Content-length - there is a hack in there now to recognize it. However, all HTTP headers need to be normalized before checking their values. This isn't easy to do. Grr. * OpenSSL 0.9.6 Segfaults under high load. Upgrade to OpenSSL 0.9.6b. Aaron says: I just found a big bug that might have been causing this all along (we weren't closing ssl sockets). How can I reproduce the problem you were seeing to verify if this was the fix? * SEGVs when /tmp/.rnd doesn't exist are bad. Make it configurable and at least bomb with a good error message. (See Doug's patch.) Status: This is fixed, no? * If APR has disabled threads, flood should as well. We might want to have an enable/disable parameter that does this also, providing an error if threads are desired but not available. * flood needs to clear pools more often. With a long running test it can chew up memory very quickly. We should just bite the bullet and create/destroy/clear pools for each level of our model: farm, farmer, profile, url/request-cycle, etc. * APR needs to have a unified interface for ephemeral port exhaustion, but aparently Solaris and Linux return different errors at the moment. Fix this in APR then take advantage of it in flood. * The examples/analyze-relative scripts fail when there are less than 5 unique URLs. Other features that need writing: * More analysis and graphing scripts are needed * Write robust tool (using tethereal perhaps) to take network dumps and convert them to flood's XML format. Status: Justin volunteers. Aaron had a script somewhere that is a start. Jacek is working on a Mozilla application, codename Flood URL bag (much like Live HTTP Headers) and small HTTP proxy. * Get chunked encoding support working. Status: Justin volunteers. He got sidetracked by the httpd implementation of input filtering and never finished this. This is a stopgap until apr-serf is completed. * Maybe we should make randfile and capath runtime directives that come out of the XML, instead of autoconf parameters. * We are using apr_os_thread_current() and getpid() in some places when what we really want is a GUID. The GUID will be used to correlate raw output data with each farmer. We may wish to print a unique ID for each of farm, farmer, profile, and url to help in postprocessing. * We are using strtol() in some places and strtoll() in others. Pick one (Aaron says strtol(), but he's not sure). * Validation of responses (known C-L, specific strings in response) Status: Justin volunteers * HTTP error codes (ie. teach it about 302s) Justin says: Yeah, this won't be with round_robin as implemented. Need a linked list-based profile where we can insert new URLs into the sequence. * Farmer (Single-thread, multiple profiles) Status: Aaron says: If you have threads, then any Farmer can be run as part of any Farm. If you don't have threads, you can currently only run one Farmer named Joe right now (this will be changed so that if you don't have threads, flood will attempt to run all Farmers in serial under one process). * Collective (Single-host, multiple farms) This is a number of Farms that have been fork()ed into child processes. * Megaconglomerate (Multiple hosts each running a collective) This is a number of Collectives running on a number of hosts, invoked via RSH/SSH or maybe even some proprietary mechanism. * Other types of urllists a) Random / Random-weighted b) Sequenced (useful with cookie propogation) c) Round-robin d) Chaining of the above strategies Status: Round-robin is complete. * Other types of reports Status: Aaron says: simple reports are functional. Justin added a new type that simply prints the approx. timestamp when the test was run, and the result as OK/FAIL; it is called easy reports (see flood_easy_reports.h). Furthermore,
Some benchs results : WAS: Invitation to HTTPD commiters in tomcat-dev
I made some benchs on my Linux Fedora Core 2 on a P4 2.8ghz / 1Gb RAM : Apache 2.0.50 in - Apache 2.0.50 alone (simple html file) - TC 3.3.2/Coyote 1.1 - Apache 2.0.50 + jk 1.2.6 + TC 3.3.2/jk2 JkMount /examples/* local worker.local.port=8009 worker.local.host=localhost worker.local.type=ajp13 worker.local.cachesize=16 worker.local.cache_timeout=600 worker.local.socket_keepalive=1 worker.local.socket_timeout=300 - Apache 2.0.50 + mod_proxy + TC 3.3.2 (Coyote 1.1). ProxyPass /tc3/ http://localhost:11011/ ProxyPassReverse /tc3/ http://localhost:11011/ Apache Bench is running on another machine, Windows 2000 P3 1Ghz, and both systems are on a switched 100Mbps network : Apache 2 alone 1202 req/s TC/Coyote 883 req/s Apache 2 + jk + TC 906 req/s Apache 2 + proxy + TC 497.req/s (but with 8000 errors ;( Constatation : - Remy make a tremendous works since Coyote HTTP 1.1 is only 15% slower than the Apache 2 native HTTP. - mod_proxy is 50% slower than mod_jk and that's a really bad news. Also many errors appears, about 4% errors. - Tomcat via jk or mod_proxy, when on the same machine make a cpu load of 60% system and 30% user. Tomcat alone is 33% system and 10% user. How could we optimize mod_proxy settings since I'm using the standard httpd.conf ? -- Complete bench results : Apache 2.0.50 alone (simple html file) C:\Program Files\Apache Group\Apache2\binab -k -n 20 -c 16 http://machone/HelloWorldExample.html This is ApacheBench, Version 2.0.40-dev $Revision: 1.121.2.8 $ apache-2.0 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/ Benchmarking machone (be patient) Completed 2 requests Completed 4 requests Completed 6 requests Completed 8 requests Completed 10 requests Completed 12 requests Completed 14 requests Completed 16 requests Completed 18 requests Finished 20 requests Server Software:Apache/2.0.50 Server Hostname:machone Server Port:80 Document Path: /HelloWorldExample.html Document Length:459 bytes Concurrency Level: 16 Time taken for tests: 166.299127 seconds Complete requests: 20 Failed requests:0 Write errors: 0 Keep-Alive requests:0 Total transferred: 14580 bytes HTML transferred: 9180 bytes Requests per second:1202.65 [#/sec] (mean) Time per request: 13.304 [ms] (mean) Time per request: 0.831 [ms] (mean, across all concurrent requests) Transfer rate: 856.18 [Kbytes/sec] received Tomcat 3.3.2 using the Coyote 1.1 HTTP connector C:\Program Files\Apache Group\Apache2\binab -k -n 20 -c 16 http://machone:11011/examples/servlet/HelloWorldExample This is ApacheBench, Version 2.0.40-dev $Revision: 1.121.2.8 $ apache-2.0 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/ Benchmarking machone (be patient) Completed 2 requests Completed 4 requests Completed 6 requests Completed 8 requests Completed 10 requests Completed 12 requests Completed 14 requests Completed 16 requests Completed 18 requests Finished 20 requests Server Software:Apache-Coyote/1.1 Server Hostname:machone Server Port:11011 Document Path: /examples/servlet/HelloWorldExample Document Length:400 bytes Concurrency Level: 16 Time taken for tests: 226.485671 seconds Complete requests: 20 Failed requests:0 Write errors: 0 Keep-Alive requests:0 Total transferred: 10540 bytes HTML transferred: 8000 bytes Requests per second:883.06 [#/sec] (mean) Time per request: 18.119 [ms] (mean) Time per request: 1.132 [ms] (mean, across all concurrent requests) Transfer rate: 454.46 [Kbytes/sec] received Apache 2.0.50 + jk 1.2.6 + Tomcat 3.3.2 (via Coyote JK2 connector) C:\Program Files\Apache Group\Apache2\binab -k -n 20 -c 16 http://machone/examples/servlet/HelloWorldExample This is ApacheBench, Version 2.0.40-dev $Revision: 1.121.2.8 $ apache-2.0 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/ Benchmarking machone (be patient) Completed 2 requests Completed 4 requests Completed 6 requests Completed 8 requests Completed 10 requests Completed 12 requests Completed 14 requests Completed 16 requests Completed 18 requests Finished 20 requests Server Software:Apache/2.0.50 Server Hostname:machone Server Port:80 Document Path: /examples/servlet/HelloWorldExample Document Length:400 bytes Concurrency Level: 16 Time taken for tests: 220.637261 seconds Complete
Re: cvs commit: httpd-2.0/modules/filters mod_deflate.c
On Mon, 19 Jul 2004, Joe Orton wrote:
Nothing like that was posted to the list, at least. Patch below is
still sufficient to fix the proxy+304 case; does it work for you too?
Yes, mostly (it fixes the important bug that was previously a
showstopper). And it's an improvement on my hack by virtue of
simplicity.
But it should still set the Content-Encoding header on a HEAD request
that would normally be deflated (and unset content-length if present).
So your:
+/* Deflating a zero-length response would make it longer; the
+ * proxy may pass through an empty response for a 304 too. */
+if (APR_BUCKET_IS_EOS(APR_BRIGADE_FIRST(bb))) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f-next, bb);
+}
+
should move after the the
if ( ! force-gzip )
block, and if then if we reach the EOS-only test we should fix up
the headers.
That test also seems to lose the pathological case of a brigade
with no data but one or more FLUSH buckets followed by EOS.
Could that ever happen in a HEAD or a 204/304?
Investigating this has revealed a similar bug with HEAD requests
in inflate_out_filter, which I shall now have to fix:-(
--
Nick Kew
Re: cvs commit: httpd-2.0/modules/filters mod_deflate.c
On Thu, 22 Jul 2004, Nick Kew wrote:
On Mon, 19 Jul 2004, Joe Orton wrote:
Nothing like that was posted to the list, at least. Patch below is
still sufficient to fix the proxy+304 case; does it work for you too?
Yes, mostly (it fixes the important bug that was previously a
showstopper).
I attach a new patch based on yours. it fixes my testcases including
headers for HEAD requests. Look OK to you?
--
Nick Kew--- mod_deflate.c.bak 2004-07-22 11:12:53.0 +0100
+++ mod_deflate.c 2004-07-22 12:17:13.0 +0100
@@ -247,7 +247,6 @@
apr_bucket_brigade *bb, *proc_bb;
} deflate_ctx;
-static void* const deflate_yes = (void*)YES;
static apr_status_t deflate_out_filter(ap_filter_t *f,
apr_bucket_brigade *bb)
{
@@ -255,14 +254,14 @@
request_rec *r = f-r;
deflate_ctx *ctx = f-ctx;
int zRC;
-char* buf;
-int eos_only = 1;
-apr_bucket *bkt;
-char *token;
-const char *encoding = NULL;
deflate_filter_config *c = ap_get_module_config(r-server-module_config,
deflate_module);
+/* Do nothing if asked to filter nothing. */
+if (APR_BRIGADE_EMPTY(bb)) {
+return APR_SUCCESS;
+}
+
/* If we don't have a context, we need to ensure that it is okay to send
* the deflated content. If we have a context, that means we've done
* this before and we liked it.
@@ -270,6 +269,8 @@
* we're in better shape.
*/
if (!ctx) {
+char *buf, *token;
+const char *encoding;
/* only work on main request/no subrequests */
if (r-main) {
@@ -349,7 +350,6 @@
*/
apr_table_setn(r-headers_out, Vary, Accept-Encoding);
-
/* force-gzip will just force it out regardless if the browser
* can actually do anything with it.
*/
@@ -384,39 +384,22 @@
}
}
-/* don't deflate responses with zero length e.g. proxied 304's but
- * we do set the header on eos_only at this point for headers_filter
- *
- * if we get eos_only and come round again, we want to avoid redoing
- * what we've already done, so set f-ctx to a flag here
+/* Deflating a zero-length response would make it longer; the
+ * proxy may pass through an empty response for a 304 too.
+ * So we just need to fix up the headers as if we had a body.
*/
-f-ctx = ctx = deflate_yes;
-}
-if (ctx == deflate_yes) {
-/* deal with the pathological case of lots of empty brigades and
- * no knowledge of whether content will follow
- */
-for (bkt = APR_BRIGADE_FIRST(bb);
- bkt != APR_BRIGADE_SENTINEL(bb);
- bkt = APR_BUCKET_NEXT(bkt))
-{
-if (!APR_BUCKET_IS_EOS(bkt)) {
- eos_only = 0;
- break;
-}
-}
-if (eos_only) {
-if (!encoding || !strcasecmp(encoding, identity)) {
+if (APR_BUCKET_IS_EOS(APR_BRIGADE_FIRST(bb))) {
+ if (!encoding || !strcasecmp(encoding, identity)) {
apr_table_set(r-headers_out, Content-Encoding, gzip);
}
else {
apr_table_merge(r-headers_out, Content-Encoding, gzip);
}
apr_table_unset(r-headers_out, Content-Length);
+
+ap_remove_output_filter(f);
return ap_pass_brigade(f-next, bb);
}
-}
-if (!ctx || (ctx==deflate_yes)) {
/* We're cool with filtering this. */
ctx = f-ctx = apr_pcalloc(r-pool, sizeof(*ctx));
@@ -912,6 +895,11 @@
apr_status_t rv;
deflate_filter_config *c;
+/* Do nothing if asked to filter nothing. */
+if (APR_BRIGADE_EMPTY(bb)) {
+return APR_SUCCESS;
+}
+
c = ap_get_module_config(r-server-module_config, deflate_module);
if (!ctx) {
@@ -950,6 +938,13 @@
}
apr_table_unset(r-headers_out, Content-Encoding);
+/* No need to inflate HEAD or 204/304 */
+if (APR_BUCKET_IS_EOS(APR_BRIGADE_FIRST(bb))) {
+ap_remove_output_filter(f);
+return ap_pass_brigade(f-next, bb);
+}
+
+
f-ctx = ctx = apr_pcalloc(f-r-pool, sizeof(*ctx));
ctx-proc_bb = apr_brigade_create(r-pool, f-c-bucket_alloc);
ctx-buffer = apr_palloc(r-pool, c-bufferSize);
@@ -983,9 +978,10 @@
apr_size_t len;
/* If we actually see the EOS, that means we screwed up! */
+/* no it doesn't - not in a HEAD or 204/304 */
if (APR_BUCKET_IS_EOS(bkt)) {
inflateEnd(ctx-stream);
-return APR_EGENERAL;
+return ap_pass_brigade(f-next, bb);
}
if (APR_BUCKET_IS_FLUSH(bkt)) {
Re: Invitation to HTTPD commiters in tomcat-dev
Guernsey, Byron (GE Consumer Industrial) wrote: We are using mod_proxy and a patched mod_rewrite to do sticky load balancing. Mod_rewrite supports cookies, but not session based cookies. I added this functionality and posted the patch here (see mod_rewrite cookie patch (PR#28391))- still trying to figure out how to get it included in the httpd. Post it to bugzilla so the patch doesn't fall through the cracks, then follow up with a message to [EMAIL PROTECTED] so as to generate some discussion - you might have to do this more than once. I would find it very useful if keepalive connections were supported in mod_proxy. They are, but only if the client was using keepalives. This might change to a full connection pool at a later stage. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: cvs commit: httpd-2.0/modules/filters mod_deflate.c
On Thu, Jul 22, 2004 at 12:20:33PM +0100, Nick Kew wrote:
On Thu, 22 Jul 2004, Nick Kew wrote:
On Mon, 19 Jul 2004, Joe Orton wrote:
Nothing like that was posted to the list, at least. Patch below is
still sufficient to fix the proxy+304 case; does it work for you too?
Yes, mostly (it fixes the important bug that was previously a
showstopper).
I attach a new patch based on yours. it fixes my testcases including
headers for HEAD requests. Look OK to you?
Yup, other than the tab character.
+if (APR_BUCKET_IS_EOS(APR_BRIGADE_FIRST(bb))) {
+ if (!encoding || !strcasecmp(encoding, identity)) {
^^^
