Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-16 Thread Micha Lenk
Hi Yann, Am 09.12.2013 00:30, schrieb Yann Ylavic: Now, if trying to send the first bytes of the request immediately fails because the socket isn't connected anymore (e.g. EPIPE), you *know* that exactly *none* bits of your current request reached the server. For this reason

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-08 Thread Micha Lenk
Hi all, Am 05.12.2013 22:03, schrieb Yann Ylavic: I'm not talking about retrying requests but retrying writing on the socket after trying to re-open a connection. When mod_proxy tries to forward the client request to the backends and encounters a closed connection due to the

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-08 Thread Yann Ylavic
On Sun, Dec 8, 2013 at 11:51 AM, Micha Lenk mi...@lenk.info wrote: Am 05.12.2013 22:03, schrieb Yann Ylavic: When mod_proxy encounters a closed (reused) connection while forwarding the client's request, it has just sent the request once, right? That depends on the actual state. What

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-06 Thread Yann Ylavic
I don't want to overdo this thread (apologies if it is already), last comment yet... On Thu, Dec 5, 2013 at 10:03 PM, Yann Ylavic ylavic@gmail.com wrote: Is sending a partial request first what you'd like to (be) implement(ed), and only if that succeeds send the remaining data? The issue

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Jan Kaluža
Hi, should this be fixed in trunk already? I see some commits in proxy code based on your ideas Yann, but I'm not sure if they address this particular problem too. Jan Kaluza On 10/17/2013 04:52 PM, Yann Ylavic wrote: On Thu, Oct 17, 2013 at 11:36 AM, Thomas Eckert

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Yann Ylavic
Hi Jan, I don't think it is fixed in trunk, but I may have missed the commits. Which ones are you talking about? Regards, Yann. On Thu, Dec 5, 2013 at 1:51 PM, Jan Kaluža jkal...@redhat.com wrote: Hi, should this be fixed in trunk already? I see some commits in proxy code based on your

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Jim Jagielski
There hardly seemed any consensus on the patch... It also seems that it adds more cycles to Apache on the front to reduce a race condition that can't really be removed. IMO, a reverse proxy should get out of the way as quickly as possible. Plus, if we do this here, shouldn't we do it for all

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Thomas Eckert
It also seems that it adds more cycles to Apache on the front to reduce a race condition that can't really be removed. While it's true that the race condition itself cannot be avoided we can definitely work around the resulting problem situation, e.g. by trying to open the connection again once

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Yann Ylavic
On Thu, Dec 5, 2013 at 4:05 PM, Jim Jagielski j...@jagunet.com wrote: There hardly seemed any consensus on the patch... It also seems that it adds more cycles to Apache on the front to reduce a race condition that can't really be removed. I don't think more cycles are added by this patch.

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Yann Ylavic
On Thu, Dec 5, 2013 at 5:04 PM, Thomas Eckert thomas.r.w.eck...@gmail.comwrote: It also seems that it adds more cycles to Apache on the front to reduce a race condition that can't really be removed. While it's true that the race condition itself cannot be avoided we can definitely work

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Yann Ylavic
On Thu, Dec 5, 2013 at 5:45 PM, Yann Ylavic ylavic@gmail.com wrote: On Thu, Dec 5, 2013 at 5:07 PM, Yann Ylavic ylavic@gmail.com wrote: On Thu, Dec 5, 2013 at 4:05 PM, Jim Jagielski j...@jagunet.com wrote: There hardly seemed any consensus on the patch... It also seems that it adds

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Thomas Eckert
You can't retry all the requests (particularly non idempotent ones), not even once. Suppose it is a charged $100 order, you wouldn't like any proxy to double that because of network problems... I'm not talking about retrying requests but retrying writing on the socket after trying to re-open a

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-12-05 Thread Yann Ylavic
On Thu, Dec 5, 2013 at 7:03 PM, Thomas Eckert thomas.r.w.eck...@gmail.comwrote: I'm not talking about retrying requests but retrying writing on the socket after trying to re-open a connection. When mod_proxy tries to forward the client request to the backends and encounters a closed connection

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-17 Thread Thomas Eckert
Sorry for the delayed reply. At the moment I don't have time to look at the patch proposal in detail, sorry about that too. I'll get back to it soon, I hope. Pre-fetching 16K (or waiting for input filters to provide these) is not always a fast operation, and the case where the backend closes its

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-17 Thread Yann Ylavic
On Thu, Oct 17, 2013 at 11:36 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: Hence, why cannot mod_proxy_http prefetch the client's body *before* trying anything with the backend connection, and only if it's all good, connect (or reuse a connection to) the backend and then start

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-08 Thread Micha Lenk
Hi Yann, Am 03.10.2013 15:33, schrieb Yann Ylavic: On Thu, Oct 3, 2013 at 2:07 PM, Micha Lenk mi...@lenk.info wrote: Independent from how the HRS issue (CVE-2005-2088) was fixed at that time, I still believe that it is a bad idea in terms of security to flush the buffer and forward

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-07 Thread Yann Ylavic
Sorry to insist about this issue but I don't see how the current mod_proxy_http's behaviour of connecting the backend (or checking established connection reusability) before prefetching the client's body is not a problem. Pre-fetching 16K (or waiting for input filters to provide these) is not

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-03 Thread Yann Ylavic
-Ursprüngliche Nachricht- Von: Jim Jagielski Gesendet: Freitag, 2. August 2013 14:29 An: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition +1 for the theory, but I'm not sure if it's feasible or not. On Aug 2, 2013

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-03 Thread Micha Lenk
Hi Yann, Am 01.10.2013 17:08, schrieb Yann Ylavic: As far as I understand the issue, the main point of prefetch was to fix CVE-2005-2088, a HTTP Request Smuggling attack (see also http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-2088). This is discussed in PR40029 and is

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-03 Thread Yann Ylavic
Hi Micha, On Thu, Oct 3, 2013 at 2:07 PM, Micha Lenk mi...@lenk.info wrote: Independent from how the HRS issue (CVE-2005-2088) was fixed at that time, I still believe that it is a bad idea in terms of security to flush the buffer and forward its content to the backend before the *whole*

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-02 Thread Thomas Eckert
: Re: mod_proxy, oooled backend connections and the keep-alive race condition On Fri, Aug 02, 2013 at 12:33:58PM +, Plüm, Rüdiger, Vodafone Group wrote: The typical way to solve this today is to know the keepalive timeout of the backend and set ttl for this worker to a value a few

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-02 Thread Yann Ylavic
On Wed, Oct 2, 2013 at 9:40 AM, Thomas Eckert thomas.r.w.eck...@gmail.comwrote: Yann, although I do expect it to solve the issue discussed here, I don't think simply flushing everything instantly is the right way to go. For example, how do the proposed changes work with modules which scan

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-02 Thread Yann Ylavic
A late (little) fix below... On Thu, Oct 3, 2013 at 12:14 AM, Yann Ylavic ylavic@gmail.com wrote: Index: modules/proxy/proxy_util.c === --- modules/proxy/proxy_util.c(revision 1528615) +++ modules/proxy/proxy_util.c

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-02 Thread Yann Ylavic
Is the Step Three-and-a-Half really required since ap_proxy_connect_backend() (that does the same is_socket_connected() check) is run almost before? May the ap_proxy_connection_create() function in between take some time or is it a last chance to catch the race? Regards On Thu, Oct 3, 2013 at

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-01 Thread Joe Orton
On Fri, Aug 02, 2013 at 12:33:58PM +, Plüm, Rüdiger, Vodafone Group wrote: The typical way to solve this today is to know the keepalive timeout of the backend and set ttl for this worker to a value a few seconds below. I just looked at a case where the user is hitting this problem and

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-01 Thread Yann Ylavic
- Von: Joe Orton Gesendet: Dienstag, 1. Oktober 2013 14:23 An: Plüm, Rüdiger, Vodafone Group Cc: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition On Fri, Aug 02, 2013 at 12:33:58PM +, Plüm, Rüdiger, Vodafone Group

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-01 Thread Yann Ylavic
On Tue, Oct 1, 2013 at 3:39 PM, Yann Ylavic ylavic@gmail.com wrote: It should address this issue, and in the same time allow protocols like MS RPC-over-HTTP to work with : SetEnvIf Request_Method ^RPC_IN_DATA proxy-flushall See PR40029 for RPC-over-HTTP debate, it was finally

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-01 Thread Jim Jagielski
connections and the keep-alive race condition That's time when the proxy *thinks* the connection is valid but the backend thinks the connection is idle. And in most reverse-proxy cases that prefetch is adding basically no value AFAICT - the backend is a known vintage and probably HTTP/1.1. So

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-01 Thread Micha Lenk
Hi all, Am 01.10.2013 14:36, schrieb Plüm, Rüdiger, Vodafone Group: That's time when the proxy *thinks* the connection is valid but the backend thinks the connection is idle. And in most reverse-proxy cases that prefetch is adding basically no value AFAICT - the backend is a known vintage

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-10-01 Thread Yann Ylavic
On Tue, Oct 1, 2013 at 4:53 PM, Micha Lenk mi...@lenk.info wrote: As far as I understand the issue, the main point of prefetch was to fix CVE-2005-2088, a HTTP Request Smuggling attack (see also http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-2088). This is discussed in PR40029 and is

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-05 Thread Thomas Eckert
One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. I don't see why an OPTIONS request should not encounter problems where a GET request will. After all, the problem is on the transport layer, not on the application

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-05 Thread Eric Covener
On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. I don't see why an OPTIONS request should not encounter problems where a GET

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-05 Thread Rainer Jung
On 05.08.2013 13:18, Eric Covener wrote: On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: One could do an 'OPTIONS *' request. But I am not sure if that is any better than proxy-initial-not-pooled in terms of performance. I don't see why an OPTIONS request

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-05 Thread Jim Jagielski
On Aug 5, 2013, at 10:13 AM, Rainer Jung rainer.j...@kippdata.de wrote: On 05.08.2013 13:18, Eric Covener wrote: On Mon, Aug 5, 2013 at 2:49 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: One could do an 'OPTIONS *' request. But I am not sure if that is any better than

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-04 Thread Jim Jagielski
I've tried looking into that, and I found it more trouble than it was worth... (I'm sure that the list archives have posts about the 'http ping' tests). The problem is that the OPTIONS request could be that request that kicks the backend from being a keptalive connection to closing it. :/ On Aug

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-03 Thread Stefan Fritsch
Am Freitag, 2. August 2013, 11:21:56 schrieb Eric Covener: I think this does not work for GET requests or request without a request body. Just re-read spec, you are right -- we are abusing this in a module as a sort of extended handshake even w/ no body, but not against heterogenous

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-02 Thread Jim Jagielski
+1 for the theory, but I'm not sure if it's feasible or not. On Aug 2, 2013, at 5:28 AM, Thomas Eckert thomas.r.w.eck...@gmail.com wrote: So I've been seeing lots of proxy: error reading status line from remote server by mod_proxy lately. Usually this is caused by the race condition between

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-02 Thread Thomas Eckert
Yes, the theory thing ... I wish I could have added an experimental patch for such an input filter but I'm afraid that might take a long time for me to finish. I'll try though I hope someone more knowledgeable will pick this up. On Fri, Aug 2, 2013 at 2:28 PM, Jim Jagielski j...@jagunet.com

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-02 Thread Jim Jagielski
Rüdiger -Ursprüngliche Nachricht- Von: Jim Jagielski Gesendet: Freitag, 2. August 2013 14:29 An: dev@httpd.apache.org Betreff: Re: mod_proxy, oooled backend connections and the keep-alive race condition +1 for the theory, but I'm not sure if it's feasible or not. On Aug 2, 2013

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-02 Thread Eric Covener
+1 for the theory, but I'm not sure if it's feasible or not. Some other ideas: * Does something like a select/poll/getpeername() ever tell us of an error when a write() doesn't? * Could we give people an Expect: 100-continue option on the initial client request?

Re: mod_proxy, oooled backend connections and the keep-alive race condition

2013-08-02 Thread Eric Covener
I think this does not work for GET requests or request without a request body. Just re-read spec, you are right -- we are abusing this in a module as a sort of extended handshake even w/ no body, but not against heterogenous backends.