AW: mod_proxy, oooled backend connections and the keep-alive race condition
-Ursprüngliche Nachricht- Von: Joe Orton Gesendet: Dienstag, 1. Oktober 2013 14:23 An: Plüm, Rüdiger, Vodafone Group Cc: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition On Fri, Aug 02, 2013 at 12:33:58PM +, Plüm, Rüdiger, Vodafone Group wrote: The typical way to solve this today is to know the keepalive timeout of the backend and set ttl for this worker to a value a few seconds below. I just looked at a case where the user is hitting this problem and does already have the ttl keepalive-timeout configured like that; and a few seconds difference is not necessarily enough. The problem is that the prefetch of the 16K request body is making it a lot worse - the worst case is (16K/packet size) * Timeout seconds to complete the prefetch. True. That's time when the proxy *thinks* the connection is valid but the backend thinks the connection is idle. And in most reverse-proxy cases that prefetch is adding basically no value AFAICT - the backend is a known vintage and probably HTTP/1.1. So... could we make the prefetch stuff configurable away? IMHO no issue with this. Let's hear what others say. I guess the main point of prefetch was to make better decisions whether to use chunked encoding when sending to the backend. Or provide a CL to the backend when the real client does not. Regards Rüdiger
AW: mod_proxy, oooled backend connections and the keep-alive race condition
The typical way to solve this today is to know the keepalive timeout of the backend and set ttl for this worker to a value a few seconds below. Regards Rüdiger -Ursprüngliche Nachricht- Von: Jim Jagielski Gesendet: Freitag, 2. August 2013 14:29 An: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition +1 for the theory, but I'm not sure if it's feasible or not. On Aug 2, 2013, at 5:28 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: So I've been seeing lots of proxy: error reading status line from remote server by mod_proxy lately. Usually this is caused by the race condition between checking the connection state and the backend closing the connection due to the keep-alive timeout. As Covener pointed out to me in IRC, using mod_proxy_http's env variable proxy-initial-not- pooled does offer a solution to the problem albeit at the cost of performance. The call to ap_proxy_http_process_response() in mod_proxy_http.c eventually boils down to ap_rgetline_core() which calls ap_get_brigade() on r-input_filters. This looks to me like a simple input filter might do the trick if it only checked for a possibility to read on the socket and reopens the connection upon failure type reset by peer. I took a short look at core_create_proxy_req() in server/core.c to see how connections are set up and I wonder if it's possible to recreate/reuse that logic in an input filter. If so, this input filter would offer a nice alternative if hard coding this behavior into mod_proxy/core is frowned upon. Simply make the filter dependant on an env variable, just like proxy-initial-not-pooled.
AW: mod_proxy, oooled backend connections and the keep-alive race condition
But effective and simple :-) -Ursprüngliche Nachricht- Von: Jim Jagielski Gesendet: Freitag, 2. August 2013 15:20 An: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition True enough, but that's inelegant ;) On Aug 2, 2013, at 8:33 AM, Plüm, Rüdiger, Vodafone Group ruediger.pl...@vodafone.com wrote: The typical way to solve this today is to know the keepalive timeout of the backend and set ttl for this worker to a value a few seconds below. Regards Rüdiger -Ursprüngliche Nachricht- Von: Jim Jagielski Gesendet: Freitag, 2. August 2013 14:29 An: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition +1 for the theory, but I'm not sure if it's feasible or not. On Aug 2, 2013, at 5:28 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: So I've been seeing lots of proxy: error reading status line from remote server by mod_proxy lately. Usually this is caused by the race condition between checking the connection state and the backend closing the connection due to the keep-alive timeout. As Covener pointed out to me in IRC, using mod_proxy_http's env variable proxy-initial-not- pooled does offer a solution to the problem albeit at the cost of performance. The call to ap_proxy_http_process_response() in mod_proxy_http.c eventually boils down to ap_rgetline_core() which calls ap_get_brigade() on r-input_filters. This looks to me like a simple input filter might do the trick if it only checked for a possibility to read on the socket and reopens the connection upon failure type reset by peer. I took a short look at core_create_proxy_req() in server/core.c to see how connections are set up and I wonder if it's possible to recreate/reuse that logic in an input filter. If so, this input filter would offer a nice alternative if hard coding this behavior into mod_proxy/core is frowned upon. Simply make the filter dependant on an env variable, just like proxy-initial-not-pooled.
AW: mod_proxy, oooled backend connections and the keep-alive race condition
-Ursprüngliche Nachricht- Von: Eric Covener [mailto:cove...@gmail.com] Gesendet: Freitag, 2. August 2013 15:47 An: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition +1 for the theory, but I'm not sure if it's feasible or not. Some other ideas: * Does something like a select/poll/getpeername() ever tell us of an error when a write() doesn't? I don't think so. See the code where we detect if the connection is still there (is_connected ???) * Could we give people an Expect: 100-continue option on the initial client request? I think this does not work for GET requests or request without a request body. Regards Rüdiger
