Re: strange usage pattern for child processes
Ruediger Pluem wrote: The following patch should fix the above thing. Comments? Index: server/core_filters.c +1. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: strange usage pattern for child processes
On Oct 25, 2008, at 11:17 AM, Ruediger Pluem wrote:
On 10/19/2008 03:06 PM, Ruediger Pluem wrote:
Another maybe funny sidenote: Because of the way the read method on
socket buckets
work and the way the core input filter works, the ap_get_brigade
call when processing
the http body of the backend response in mod_proxy_http never
returns a brigade that
contains more then 8k of data no matter what you set for
ProxyIOBufferSize.
And this is the case since 2.0.x days. So the optimization was
always limited to
sending at most 8k and in this case the TCP buffer (the send
buffer) should have
fixed this in many cases.
The following patch should fix the above thing. Comments?
Index: server/core_filters.c
===
--- server/core_filters.c (Revision 707744)
+++ server/core_filters.c (Arbeitskopie)
@@ -267,6 +267,35 @@
return APR_SUCCESS;
}
+/* Have we read as much data as we wanted (be greedy)? */
+if (len readbytes) {
+apr_size_t bucket_len;
+
+rv = APR_SUCCESS;
+/* We already registered the data in e in len */
+e = APR_BUCKET_NEXT(e);
+while ((len readbytes) (rv == APR_SUCCESS)
+(e != APR_BRIGADE_SENTINEL(ctx-b))) {
+/* Check for the availability of buckets with known
length */
+if (e-length != -1) {
+len += e-length;
+e = APR_BUCKET_NEXT(e);
+continue;
+}
+/*
+ * Read from bucket, but non blocking. If there
isn't any
+ * more data, well than this is fine as well, we will
+ * not wait for more since we already got some and
we are
+ * only checking if there isn't more.
+ */
+rv = apr_bucket_read(e, str, bucket_len,
APR_NONBLOCK_READ);
+if (rv == APR_SUCCESS) {
+len += bucket_len;
+e = APR_BUCKET_NEXT(e);
+}
+}
+1 for the concept, but I'm not sure I like the logic flow
of the continue. This is really an if/else so how about:
while ((len readbytes) (rv == APR_SUCCESS)
(e != APR_BRIGADE_SENTINEL(ctx-b))) {
/* Check for the availability of buckets with known
length */
if (e-length != -1) {
len += e-length;
e = APR_BUCKET_NEXT(e);
} else {
/*
* Read from bucket, but non blocking. If there
isn't any
* more data, well than this is fine as well, we
will
* not wait for more since we already got some
and we are
* only checking if there isn't more.
*/
rv = apr_bucket_read(e, str, bucket_len,
APR_NONBLOCK_READ);
if (rv == APR_SUCCESS) {
len += bucket_len;
e = APR_BUCKET_NEXT(e);
}
}
}
Re: strange usage pattern for child processes
On 10/26/2008 04:58 PM, Jim Jagielski wrote:
On Oct 25, 2008, at 11:17 AM, Ruediger Pluem wrote:
On 10/19/2008 03:06 PM, Ruediger Pluem wrote:
Another maybe funny sidenote: Because of the way the read method on
socket buckets
work and the way the core input filter works, the ap_get_brigade call
when processing
the http body of the backend response in mod_proxy_http never returns
a brigade that
contains more then 8k of data no matter what you set for
ProxyIOBufferSize.
And this is the case since 2.0.x days. So the optimization was always
limited to
sending at most 8k and in this case the TCP buffer (the send buffer)
should have
fixed this in many cases.
The following patch should fix the above thing. Comments?
Index: server/core_filters.c
===
--- server/core_filters.c (Revision 707744)
+++ server/core_filters.c (Arbeitskopie)
@@ -267,6 +267,35 @@
return APR_SUCCESS;
}
+/* Have we read as much data as we wanted (be greedy)? */
+if (len readbytes) {
+apr_size_t bucket_len;
+
+rv = APR_SUCCESS;
+/* We already registered the data in e in len */
+e = APR_BUCKET_NEXT(e);
+while ((len readbytes) (rv == APR_SUCCESS)
+(e != APR_BRIGADE_SENTINEL(ctx-b))) {
+/* Check for the availability of buckets with known
length */
+if (e-length != -1) {
+len += e-length;
+e = APR_BUCKET_NEXT(e);
+continue;
+}
+/*
+ * Read from bucket, but non blocking. If there isn't
any
+ * more data, well than this is fine as well, we will
+ * not wait for more since we already got some and we
are
+ * only checking if there isn't more.
+ */
+rv = apr_bucket_read(e, str, bucket_len,
APR_NONBLOCK_READ);
+if (rv == APR_SUCCESS) {
+len += bucket_len;
+e = APR_BUCKET_NEXT(e);
+}
+}
+1 for the concept, but I'm not sure I like the logic flow
of the continue. This is really an if/else so how about:
Sure it is an if/else. I don't care very much whether it is continue
of if/else. It just saved some indenting. So It can be in the
way you proposed below.
while ((len readbytes) (rv == APR_SUCCESS)
(e != APR_BRIGADE_SENTINEL(ctx-b))) {
/* Check for the availability of buckets with known
length */
if (e-length != -1) {
len += e-length;
e = APR_BUCKET_NEXT(e);
} else {
/*
* Read from bucket, but non blocking. If there
isn't any
* more data, well than this is fine as well, we will
* not wait for more since we already got some and
we are
* only checking if there isn't more.
*/
rv = apr_bucket_read(e, str, bucket_len,
APR_NONBLOCK_READ);
if (rv == APR_SUCCESS) {
len += bucket_len;
e = APR_BUCKET_NEXT(e);
}
}
}
Regards
Rüdiger
Re: strange usage pattern for child processes
On Oct 26, 2008, at 12:15 PM, Ruediger Pluem wrote: Sure it is an if/else. I don't care very much whether it is continue of if/else. It just saved some indenting. So It can be in the way you proposed below. :) As you mean, functionally it's the same. I'm just a stickler for certain programming designs. :)
Re: strange usage pattern for child processes
On 10/19/2008 03:06 PM, Ruediger Pluem wrote:
Another maybe funny sidenote: Because of the way the read method on socket
buckets
work and the way the core input filter works, the ap_get_brigade call when
processing
the http body of the backend response in mod_proxy_http never returns a
brigade that
contains more then 8k of data no matter what you set for ProxyIOBufferSize.
And this is the case since 2.0.x days. So the optimization was always limited
to
sending at most 8k and in this case the TCP buffer (the send buffer) should
have
fixed this in many cases.
The following patch should fix the above thing. Comments?
Index: server/core_filters.c
===
--- server/core_filters.c (Revision 707744)
+++ server/core_filters.c (Arbeitskopie)
@@ -267,6 +267,35 @@
return APR_SUCCESS;
}
+/* Have we read as much data as we wanted (be greedy)? */
+if (len readbytes) {
+apr_size_t bucket_len;
+
+rv = APR_SUCCESS;
+/* We already registered the data in e in len */
+e = APR_BUCKET_NEXT(e);
+while ((len readbytes) (rv == APR_SUCCESS)
+(e != APR_BRIGADE_SENTINEL(ctx-b))) {
+/* Check for the availability of buckets with known length */
+if (e-length != -1) {
+len += e-length;
+e = APR_BUCKET_NEXT(e);
+continue;
+}
+/*
+ * Read from bucket, but non blocking. If there isn't any
+ * more data, well than this is fine as well, we will
+ * not wait for more since we already got some and we are
+ * only checking if there isn't more.
+ */
+rv = apr_bucket_read(e, str, bucket_len, APR_NONBLOCK_READ);
+if (rv == APR_SUCCESS) {
+len += bucket_len;
+e = APR_BUCKET_NEXT(e);
+}
+}
+}
+
/* We can only return at most what we read. */
if (len readbytes) {
readbytes = len;
Regards
Rüdiger
Re: strange usage pattern for child processes
On 10/18/08 4:28 PM, William A. Rowe, Jr. [EMAIL PROTECTED] wrote: Also consider today that a server on broadband can easily spew 1gb/sec bandwidth at the client. If this is composed content (or proxied, etc, but not sendfiled) it would make sense to allow multiple buffer pages and/or resizing buffer pages, in a Location or Files or Proxy specific context. Would not the generic store-and-forward approach I sent last week help all of these situations? It effective turns any request into a sendfiled response. Let me do some checking and I may be able to just donate the code, since it's basically a very hacked up mod_deflate crossed with mod_xsendfile. It works for us, but we don't use the stock proxy, so not 100% it will help if the frontend and backend pools are somehow linked. -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: strange usage pattern for child processes
Akins, Brian wrote: Would not the generic store-and-forward approach I sent last week help all of these situations? It effective turns any request into a sendfiled response. Let me do some checking and I may be able to just donate the code, since it's basically a very hacked up mod_deflate crossed with mod_xsendfile. It works for us, but we don't use the stock proxy, so not 100% it will help if the frontend and backend pools are somehow linked. This technique was used in the large disk cache work from 2006, and worked very well. Whether it will work in the current flow though I am not so sure. The current flow looks like this: read-backend - write-and-flush-frontend - disconnect-backend As I understand it, the flush part is the killer. The disconnect-backend should happen before the flush, not after. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: strange usage pattern for child processes
On Oct 19, 2008, at 4:20 PM, Ruediger Pluem wrote: On 10/19/2008 07:35 PM, Jim Jagielski wrote: On Oct 18, 2008, at 4:22 PM, Graham Leggett wrote: Ruediger Pluem wrote: As a result, the connection pool has made the server slower, not faster, and very much needs to be fixed. I agree in theory. But I don't think so in practice. Unfortunately I know so in practice. In this example we are seeing single connections being held open for 30 second or more. :( 1. 2.0.x behaviour: If you did use keepalive connections to the backend the connection to the backenend was kept alive and as it was bound to the frontend connection in 2.0.x it couldn't be used by other connections. Depending on the backend server it wasted the same number of resources as without the optimization (backend like httpd worker, httpd prefork) or a small amount of resources (backend like httpd event with HTTP or a recent Tomcat web connector). So you didn't benefit very well from this optimization in 2.0.x as long as you did not turn off the keepalives to the backend. Those who did need the optimisation, would have turned off keepalives to the backend. Trying to grok things better, but doesn't this imply that for those who needed the optimization, disabling the connection pool would be the required work-around for 2.2? No. Without a connection pool (e.g. the default reverse worker) the backend connection does not get freed any faster than without a connection pool. Ok strictly spoken you cannot turn off the connection pools at all (reverse is also one), you can only turn off a reuse of the connections. I thought that was the concern; that the pool wasn't released immediately. If you disable reuse, then you don't need to worry about when it is released... or I must be missing something obvious here :/
Re: strange usage pattern for child processes
Jim Jagielski wrote: I thought that was the concern; that the pool wasn't released immediately. If you disable reuse, then you don't need to worry about when it is released... or I must be missing something obvious here :/ Whether the connection is released and returned to the pool (when pools are used), or when the connection is closed (when pools are not used), as I understand it now neither of these make any difference - in both cases, the frontend connection to the client is flushed - causing a long delay - before the backend is released. The backend connection is held open during this delay, and isn't returned to the pool or closed, and this has the double effect of keeping expensive backend resources tied up for a long time, and reducing the effectiveness of the connection poool. If the backend connection is returned to the pool or disconnected the moment the backend request is finished, the delay is avoided, and when the pools are used, the backend connection can be reused immediately by another thread, instead of only after the flush to frontend is complete. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: strange usage pattern for child processes
On Oct 20, 2008, at 10:13 AM, Graham Leggett wrote: Jim Jagielski wrote: I thought that was the concern; that the pool wasn't released immediately. If you disable reuse, then you don't need to worry about when it is released... or I must be missing something obvious here :/ Whether the connection is released and returned to the pool (when pools are used), or when the connection is closed (when pools are not used), as I understand it now neither of these make any difference - in both cases, the frontend connection to the client is flushed - causing a long delay - before the backend is released. The backend connection is held open during this delay, and isn't returned to the pool or closed, and this has the double effect of keeping expensive backend resources tied up for a long time, and reducing the effectiveness of the connection poool. Ahhh... I got it now. Basically, as soon as we know we have all the data from the backend, we should release it.
Re: strange usage pattern for child processes
On 10/18/2008 10:22 PM, Graham Leggett wrote: Ruediger Pluem wrote: Plus the default socket and TCP buffers on most OS should be already larger then this. So in order to profit from the optimization the time the client needs to consume the ProxyIOBufferSize needs to be remarkable. It makes no difference how large the TCP buffers are, the backend will only be released for reuse when the frontend has completely flushed and Sorry I maybe wrong here, but I don't think this is the case. If there is space left in the TCP buffer you write to the socket non-blocking and the data seems to be processed for the sending process then. It does not block until the data is sent by the OS. And even flush buckets do not seem to cause any special processing like a blocking flush. So returning to your CNN example I am pretty much sure that if the TCP buffer for the socket of the client connection holds 92k plus the header overhead your connection to the backend will be released almost instantly. I don't even think so that a close or shutdown on the socket will block until all data in the TCP buffer is sent. But this wouldn't matter on keepalive connection to the client anyway since the connection isn't closed. Regards Rüdiger
Re: strange usage pattern for child processes
On 10/19/2008 01:21 PM, Ruediger Pluem wrote:
On 10/18/2008 10:22 PM, Graham Leggett wrote:
Ruediger Pluem wrote:
Plus the default socket and TCP buffers on most OS should be already
larger then this. So in order to profit from the optimization the time
the client needs to consume the ProxyIOBufferSize needs to be
remarkable.
It makes no difference how large the TCP buffers are, the backend will
only be released for reuse when the frontend has completely flushed and
Sorry I maybe wrong here, but I don't think this is the case. If there is
space left in the TCP buffer you write to the socket non-blocking and the
data seems to be processed for the sending process then. It does not block
until the data is sent by the OS. And even flush buckets do not seem to cause
any special processing like a blocking flush. So returning to your CNN
example I am pretty much sure that if the TCP buffer for the socket of the
client connection holds 92k plus the header overhead your connection to the
backend will be released almost instantly.
I don't even think so that a close or shutdown on the socket will block until
all data in the TCP buffer is sent. But this wouldn't matter on keepalive
connection to the client anyway since the connection isn't closed.
I did some further investigations in the meantime. Using the following
configuration
ProxyPass /cnn/ http://www.cnn.com/
Sendbuffersize 285168
ProxyIOBufferSize 131072
and the following slow testclient
!/usr/bin/perl -w
use strict;
use Socket;
my $proto;
my $port;
my $sin;
my $addr;
my $url;
my $oldfh;
$proto = getprotobyname('tcp');
$port = getservbyname('http','tcp');
$addr = 192.168.2.4;
$url = /cnn/;
socket(SOCKET_H,PF_INET,SOCK_STREAM,$proto);
$sin = sockaddr_in($port,inet_aton($addr));
setsockopt(SOCKET_H, SOL_SOCKET, SO_RCVBUF, 1);
connect(SOCKET_H,$sin);
$oldfh = select SOCKET_H;
$| = 1;
select $oldfh;
print SOCKET_H GET $url HTTP/1.0\r\n\r\n;
sleep(500);
close SOCKET_H;
I was able to have the connection to www.cnn.com returned back to the pool
immediately.
The strange thing that remains is that I needed to set the Sendbuffersize about
3 times higher than the actual size of the page to get this done. I currently
do not know why this is the case.
Another maybe funny sidenote: Because of the way the read method on socket
buckets
work and the way the core input filter works, the ap_get_brigade call when
processing
the http body of the backend response in mod_proxy_http never returns a brigade
that
contains more then 8k of data no matter what you set for ProxyIOBufferSize.
And this is the case since 2.0.x days. So the optimization was always limited to
sending at most 8k and in this case the TCP buffer (the send buffer) should have
fixed this in many cases.
Regards
Rüdiger
Re: strange usage pattern for child processes
On Oct 18, 2008, at 4:22 PM, Graham Leggett wrote: Ruediger Pluem wrote: As a result, the connection pool has made the server slower, not faster, and very much needs to be fixed. I agree in theory. But I don't think so in practice. Unfortunately I know so in practice. In this example we are seeing single connections being held open for 30 second or more. :( 1. 2.0.x behaviour: If you did use keepalive connections to the backend the connection to the backenend was kept alive and as it was bound to the frontend connection in 2.0.x it couldn't be used by other connections. Depending on the backend server it wasted the same number of resources as without the optimization (backend like httpd worker, httpd prefork) or a small amount of resources (backend like httpd event with HTTP or a recent Tomcat web connector). So you didn't benefit very well from this optimization in 2.0.x as long as you did not turn off the keepalives to the backend. Those who did need the optimisation, would have turned off keepalives to the backend. Trying to grok things better, but doesn't this imply that for those who needed the optimization, disabling the connection pool would be the required work-around for 2.2?
Re: strange usage pattern for child processes
On 10/19/2008 07:35 PM, Jim Jagielski wrote: On Oct 18, 2008, at 4:22 PM, Graham Leggett wrote: Ruediger Pluem wrote: As a result, the connection pool has made the server slower, not faster, and very much needs to be fixed. I agree in theory. But I don't think so in practice. Unfortunately I know so in practice. In this example we are seeing single connections being held open for 30 second or more. :( 1. 2.0.x behaviour: If you did use keepalive connections to the backend the connection to the backenend was kept alive and as it was bound to the frontend connection in 2.0.x it couldn't be used by other connections. Depending on the backend server it wasted the same number of resources as without the optimization (backend like httpd worker, httpd prefork) or a small amount of resources (backend like httpd event with HTTP or a recent Tomcat web connector). So you didn't benefit very well from this optimization in 2.0.x as long as you did not turn off the keepalives to the backend. Those who did need the optimisation, would have turned off keepalives to the backend. Trying to grok things better, but doesn't this imply that for those who needed the optimization, disabling the connection pool would be the required work-around for 2.2? No. Without a connection pool (e.g. the default reverse worker) the backend connection does not get freed any faster than without a connection pool. Ok strictly spoken you cannot turn off the connection pools at all (reverse is also one), you can only turn off a reuse of the connections. Regards Rüdiger
Re: strange usage pattern for child processes
Ruediger Pluem wrote: The code Graham is talking about was introduced by him in r93811 and was removed in r104602 about 4 years ago. So I am not astonished any longer that I cannot remember this optimization. It was before my time :-). This optimization was never in 2.2.x (2.0.x still ships with it). BTW: This logic flow cannot be restored easily, because due to the current pool and bucket allocator usage it *must* be ensured that all buckets are flushed down the chain before we can return the backend connection to the connection pool. By the time this was removed and in 2.0.x we do *not* use a connection pool for backend connections. Right now, the problem we are seeing is that the expensive backends are tied up until slow frontends eventually get round to completely consuming responses, which is a huge waste of backend resources. As a result, the connection pool has made the server slower, not faster, and very much needs to be fixed. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: strange usage pattern for child processes
Ruediger Pluem wrote: As a result, the connection pool has made the server slower, not faster, and very much needs to be fixed. I agree in theory. But I don't think so in practice. Unfortunately I know so in practice. In this example we are seeing single connections being held open for 30 second or more. :( 1. 2.0.x behaviour: If you did use keepalive connections to the backend the connection to the backenend was kept alive and as it was bound to the frontend connection in 2.0.x it couldn't be used by other connections. Depending on the backend server it wasted the same number of resources as without the optimization (backend like httpd worker, httpd prefork) or a small amount of resources (backend like httpd event with HTTP or a recent Tomcat web connector). So you didn't benefit very well from this optimization in 2.0.x as long as you did not turn off the keepalives to the backend. Those who did need the optimisation, would have turned off keepalives to the backend. 2. The optimization only helps for the last chunk being read from the backend which has the size of ProxyIOBufferSize at most. If ProxyIOBuffer size isn't set explicitly this amounts to just 8k. I guess if you are having clients or connections that take a long time to consume just 8k you are troubled anyway. Which is why you would increase the size from 8k to something big enough to hold your complete pages. The CNN home page for example is 92k. As recent as 3 years ago, I had one client running an entire company on a 64kbps legacy telco connection that cost over USD1000 per month. These clients tie up your backend for many seconds, sometimes minutes, and protecting you from this is one of the key reasons you would use a reverse proxy. Plus the default socket and TCP buffers on most OS should be already larger then this. So in order to profit from the optimization the time the client needs to consume the ProxyIOBufferSize needs to be remarkable. It makes no difference how large the TCP buffers are, the backend will only be released for reuse when the frontend has completely flushed and acknowledged the request, so all your buffers don't help at all. As soon as the backend has provided the very last bit of the request, the backend should be released immediately and placed back in the pool. The backend might only take tens or hundreds of milliseconds to complete its work, but is then tied up frozen for many orders of magnitude more than that waiting for the client to say that is is done. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: strange usage pattern for child processes
Graham Leggett wrote: 2. The optimization only helps for the last chunk being read from the backend which has the size of ProxyIOBufferSize at most. If ProxyIOBuffer size isn't set explicitly this amounts to just 8k. I guess if you are having clients or connections that take a long time to consume just 8k you are troubled anyway. Which is why you would increase the size from 8k to something big enough to hold your complete pages. The CNN home page for example is 92k. Also consider today that a server on broadband can easily spew 1gb/sec bandwidth at the client. If this is composed content (or proxied, etc, but not sendfiled) it would make sense to allow multiple buffer pages and/or resizing buffer pages, in a Location or Files or Proxy specific context. Since mismatched buffer pages are a loser from the recycling pov, it seems sensible to set that based on traffic and bandwidth, but then take advantage of multiple pages to complete the handoff from the backend connection pool to the client facing connection pool.
Re: strange usage pattern for child processes
On Oct 15, 2008, at 6:56 PM, Graham Leggett wrote: Ruediger Pluem wrote: Something else to try is to look at the ProxyIOBufferSize parameter. The proxy reads from the backend in blocks, and as soon as a block is not full (ie it's the last block), the proxy will complete and terminate the backend request before sending the last block on to the client. No. See below. Oh dear - I know this optimisation is supposed to be there - I wrote it. Picking through the proxy code on v2.2, it looks like the optimisation has been removed for some reason. The logic should flow like this: H I'm not seeing it in trunk.
Re: strange usage pattern for child processes
On 10/17/2008 05:38 PM, Jim Jagielski wrote: On Oct 15, 2008, at 6:56 PM, Graham Leggett wrote: Ruediger Pluem wrote: Something else to try is to look at the ProxyIOBufferSize parameter. The proxy reads from the backend in blocks, and as soon as a block is not full (ie it's the last block), the proxy will complete and terminate the backend request before sending the last block on to the client. No. See below. Oh dear - I know this optimisation is supposed to be there - I wrote it. Picking through the proxy code on v2.2, it looks like the optimisation has been removed for some reason. The logic should flow like this: H I'm not seeing it in trunk. The code Graham is talking about was introduced by him in r93811 and was removed in r104602 about 4 years ago. So I am not astonished any longer that I cannot remember this optimization. It was before my time :-). This optimization was never in 2.2.x (2.0.x still ships with it). BTW: This logic flow cannot be restored easily, because due to the current pool and bucket allocator usage it *must* be ensured that all buckets are flushed down the chain before we can return the backend connection to the connection pool. By the time this was removed and in 2.0.x we do *not* use a connection pool for backend connections. Regards Rüdiger
Re: strange usage pattern for child processes
On 10/15/08 6:56 PM, Graham Leggett [EMAIL PROTECTED] wrote:
Obviously, if the loop comes round more than once, then the client comes
into play. This definitely needs to be fixed, it is a big performance issue.
Could a more general purpose optimization be done? I was thinking of a
generic store and forward filter. Basically, just dump the entire brigade
into some storage (memory, file) in a loop (like deflate sorta does) without
sending to client at all until the response is finished. This would work
for proxy, php, some strange handler, etc.
A small test I did was a module had a filter that just dumped all the
brigade contents into a temp file. When it saw EOS, it would then send the
complete file to the client (via sendfile, mmap, etc..) and remove the
tempfile. Almost like an in process X-Sendfile. This very rudimentary
module increased overall throughput of some large SSI files by about 20% and
completely shielded origin servers from clients in some proxy tests. It
also did not consume very much memory (although I was writing the temp files
into /dev/shm on Linux).
Basic logic - I ignored flush and meta buckets in my tests:
while (!APR_BRIGADE_EMPTY(bb)) {
e = APR_BRIGADE_FIRST(bb);
if (APR_BUCKET_IS_EOS(e)) {
create a file bucket with the tempfile and send it along
tempfile was opened with APR_DELONCLOSE
} else {
apr_bucket_read(e,
write content to temp file
}
apr_bucket_delete(e);
}
apr_brigade_cleanup(bb);
return APR_SUCCESS;
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: strange usage pattern for child processes
Ruediger Pluem wrote: This is a pity, because then it will become much harder to debug this issue. Any chance you get shell access or that you can instruct the administrators in the service company to get the needed information for you? Getting shell access is very unlikely ... However, initial tests using mod_disk_cache have been very good. The performance of mod_mem_cache compared to mod_disk_cache is just very bad ... It seems the main issue is/was that under high load the child process(es) of Apache just starve while trying to read something from the mem cache or to write something to it. But well, without access to the box I couldn't really dig into what exactly is happening in such a case. Ok. MaxSpareThreads is set to 75 with ThreadsPerChild 256. This means that StartServers 3 is pointless because after starting *one* process we already have way too much spare threads *overall*. So the other 2 processes will get killed immediately :-). Yes you are right, I missed that MaxSpareThreads applies to all child processes. But I didn't wrote the config. ;-) However, I still find it a bit odd that we actually use only a single process and only start using threads of another child process once all threads of the other process are busy. Wouldn't it make more sense to keep spare threads in all child processes instead of just one? Especially if the config has low MaxRequestsPerChild limit. Or am I missing something? ciao... -- Lars Eilebrecht [EMAIL PROTECTED]
Re: strange usage pattern for child processes
On 10/16/2008 02:35 PM, Lars Eilebrecht wrote: Ruediger Pluem wrote: This is a pity, because then it will become much harder to debug this issue. Any chance you get shell access or that you can instruct the administrators in the service company to get the needed information for you? Getting shell access is very unlikely ... However, initial tests using mod_disk_cache have been very good. The performance of mod_mem_cache compared to mod_disk_cache is just very bad ... It seems the main issue is/was that under high load the child process(es) of Apache just starve while trying to read something from the mem cache or to write something to it. But well, without access to the box I couldn't really dig into what exactly is happening in such a case. Maybe while writing as mod_mem_cache needs to do locking over all threads during several processing phases. Ok. MaxSpareThreads is set to 75 with ThreadsPerChild 256. This means that StartServers 3 is pointless because after starting *one* process we already have way too much spare threads *overall*. So the other 2 processes will get killed immediately :-). Yes you are right, I missed that MaxSpareThreads applies to all child processes. But I didn't wrote the config. ;-) I have not assumed that you wrote the config :-). However, I still find it a bit odd that we actually use only a single process and only start using threads of another child process once all threads of the other process are busy. Wouldn't it make more sense to keep spare threads in all child processes instead of just one? Especially if the config has low MaxRequestsPerChild limit. Or am I missing something? Two points that can happen: 1. Because of the configuration you only have one process started. Once it reaches MinSpareThreads it starts a second one. Depending on the request types the first process still has a lot of keepalive connections and thus is more busy than the second process. 2. Unfair lock handling on OS side. Each process tries to get the accept mutex. If getting the lock is not fairly distributed over the two processes then one process has to handle more connections than the other one. Given the MaxRequestsPerChild limit (why is there one at all), the configuration is especially unfortunate as if there is only one process there are times where httpd does not accept new connections at all until it has started an additional process. So MPM settings should be in a way that they allow an idle system to run more than one process. But, who I am a telling, you didn't wrote the config :-). Regards Rüdiger
Re: strange usage pattern for child processes
On Wed, Oct 15, 2008 at 7:55 AM, Lars Eilebrecht [EMAIL PROTECTED] wrote: Hi, The first odd thing is that I would have expected that Apache uses all child processes about equally. Especially I would have expected that there are at least 25 threads for the second process in state _ (waiting for connection), because the MinSpareThread directive is set to 25. requests check in, but they don't check out -- but only for this process. That would explain the backlog here, the other two processes are not ignores they're just doing their work in a timely fashion. The second odd thing is that the connections/threads in W state seem to be hanging, i.e., no data is being transferred over the connection and these threads/connection time out after about 256 seconds. However, the general Timeout setting is 30s so why isn't the connection killed after 30s? This timeout only applies to an individual read/write, and the meter isn't running if we're not talking to the client (e.g. cgi chugging away) What happens during peek times is that all threads of one child process end up in state W, then Apache starts using another child process, etc. and at some point basically all threads are in state W and we are hitting the MaxClients limit which makes the server unusable. take a few pstack snapshots of one of those spinning processes? -- Eric Covener [EMAIL PROTECTED]
Re: strange usage pattern for child processes
On 10/15/2008 01:55 PM, Lars Eilebrecht wrote: Hi, I'm trying to debug a performance issue on an Apache infrastructure using 2.2.9 as reverse proxies. The Apache servers don't do much except for ProxyPass'ing data from others backend servers, and caching the content using mod_mem_cache. Is it really a good idea to use mod_mem_cache? Keep in mind that mod_mem_cache uses local caches per process and cannot use sendfile to send cached data. It seems that mod_disk_cache with a cache root on a ram disk could be more efficient here. This is Apache with the worker mpm on Solaris. Currently each box is configured with a ServerLimit of 3. The attached screenshot of the server-status page shows that almost all threads of one child process (at the bottom) are in W state. The thread status of the process shown at the top look OK, but the second child process doesn't seem to be used at all. The first odd thing is that I would have expected that Apache uses all child processes about equally. Especially I would have expected that there are at least 25 threads for the second process in state _ (waiting for connection), because the MinSpareThread directive is set to 25. This is indeed strange. Mind to 1. Attach an ASCII-output of the whole status page to see the exact process / thread slot usage. 2. Your MPM configuration and your reverse proxy configuration. The second odd thing is that the connections/threads in W state seem to be hanging, i.e., no data is being transferred over the connection and these threads/connection time out after about 256 seconds. However, the general Timeout setting is 30s so why isn't the connection killed after 30s? It would be very helpful to have a backtrace of this process. If they aren't killed after the timeout it is highly likely that they don't wait for IO but do or wait for something else. What happens during peek times is that all threads of one child process end up in state W, then Apache starts using another child process, etc. and at some point basically all threads are in state W and we are hitting the MaxClients limit which makes the server unusable. Regards Rüdiger
Re: strange usage pattern for child processes
Ruediger Pluem wrote: Is it really a good idea to use mod_mem_cache? Keep in mind that mod_mem_cache uses local caches per process and cannot use sendfile to send cached data. It seems that mod_disk_cache with a cache root on a ram disk could be more efficient here. No, it really isn't a good idea, and it wasn't my idea. ;-) I just started working at that company, and the frontend servers are even managed by another service company so I don't even have shell access to the servers. But I'm pushing for a switch to mod_disk_cache. The first odd thing is that I would have expected that Apache uses all child processes about equally. Especially I would have expected that there are at least 25 threads for the second process in state _ (waiting for connection), because the MinSpareThread directive is set to 25. This is indeed strange. Mind to 1. Attach an ASCII-output of the whole status page to see the exact process / thread slot usage. 2. Your MPM configuration and your reverse proxy configuration. Here's the mpm config: MaxMemFree 1024 ThreadLimit 256 ServerLimit 3 StartServers3 MaxClients 768 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 256 MaxRequestsPerChild 200 ciao... -- Lars Eilebrecht [EMAIL PROTECTED]
Re: strange usage pattern for child processes
On 15 Oct 2008, at 14:41, Ruediger Pluem wrote: On 10/15/2008 01:55 PM, Lars Eilebrecht wrote: I'm trying to debug a performance issue on an Apache infrastructure using 2.2.9 as reverse proxies. The Apache servers don't do much except for ProxyPass'ing data from others backend servers, and caching the content using mod_mem_cache. Is it really a good idea to use mod_mem_cache? Keep in mind that mod_mem_cache uses local caches per process and cannot use sendfile to send cached data. It seems that mod_disk_cache with a cache root on a ram disk could be more efficient here. They may have gotten driven into mod_mem_cache as it was (too) hard or would take to long to set up a ram disk with ease on that type of ops environment,. And if you assume that: This is Apache with the worker mpm on Solaris. Currently each box is configured with a ServerLimit of 3. The attached screenshot ServerLimit goes up - you really want to go towards disk/sendfile(). Dw.
Re: strange usage pattern for child processes
On 10/15/2008 08:25 PM, Lars Eilebrecht wrote: Ruediger Pluem wrote: Is it really a good idea to use mod_mem_cache? Keep in mind that mod_mem_cache uses local caches per process and cannot use sendfile to send cached data. It seems that mod_disk_cache with a cache root on a ram disk could be more efficient here. No, it really isn't a good idea, and it wasn't my idea. ;-) I just started working at that company, and the frontend servers are even managed by another service company so I don't even have shell access to the servers. This is a pity, because then it will become much harder to debug this issue. Any chance you get shell access or that you can instruct the administrators in the service company to get the needed information for you? But I'm pushing for a switch to mod_disk_cache. The first odd thing is that I would have expected that Apache uses all child processes about equally. Especially I would have expected that there are at least 25 threads for the second process in state _ (waiting for connection), because the MinSpareThread directive is set to 25. This is indeed strange. Mind to 1. Attach an ASCII-output of the whole status page to see the exact process / thread slot usage. 2. Your MPM configuration and your reverse proxy configuration. Here's the mpm config: MaxMemFree 1024 ThreadLimit 256 ServerLimit 3 StartServers3 MaxClients 768 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 256 MaxRequestsPerChild 200 Ok. MaxSpareThreads is set to 75 with ThreadsPerChild 256. This means that StartServers 3 is pointless because after starting *one* process we already have way too much spare threads *overall*. So the other 2 processes will get killed immediately :-). So the slot with all threads in '.' is one of these currently unneeded processes. The picture shows that there are still more than MaxSpareThreads (=25) threads idle *overall*. Works as designed :-). Regards Rüdiger
Re: strange usage pattern for child processes
Lars Eilebrecht wrote: The second odd thing is that the connections/threads in W state seem to be hanging, i.e., no data is being transferred over the connection and these threads/connection time out after about 256 seconds. However, the general Timeout setting is 30s so why isn't the connection killed after 30s? Something else to try is to look at the ProxyIOBufferSize parameter. The proxy reads from the backend in blocks, and as soon as a block is not full (ie it's the last block), the proxy will complete and terminate the backend request before sending the last block on to the client. If the request is small enough to fit into a single block (or if the buffer size is big enough for a bigger request to fit), what this means is that a slow client on the front end will not hold up the backend, which will be released immediately. In theory, setting the buffer size larger on the caches will release the origin backends earlier, and setting the buffer size larger on proxies before the cache will release the caches earlier. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: strange usage pattern for child processes
Ruediger Pluem wrote:
Something else to try is to look at the ProxyIOBufferSize parameter.
The proxy reads from the backend in blocks, and as soon as a block is
not full (ie it's the last block), the proxy will complete and terminate
the backend request before sending the last block on to the client.
No. See below.
Oh dear - I know this optimisation is supposed to be there - I wrote it.
Picking through the proxy code on v2.2, it looks like the optimisation
has been removed for some reason. The logic should flow like this:
do {
ap_get_brigade()
if (backend_is_done) {
backend_close
mark_we_are_done
}
ap_pass_brigade()
} while(not_yet_done)
This makes sure that for reasonable sized requests, the
probably-expensive backend is released immediately, before any time is
invested shipping the response to the potentially-slow client.
Obviously, if the loop comes round more than once, then the client comes
into play. This definitely needs to be fixed, it is a big performance issue.
Regards,
Graham
--
smime.p7s
Description: S/MIME Cryptographic Signature
