Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 11:25:19AM +0530, Sachin Shetty wrote: Hey Willy, We have 1.4.11 - so how do we use the workaround - This is already in prod and I need to work around it before we can get to the root cause. The workaround is part of the code, so you're hitting a different issue I think. Please do not hesitate to send me a network capture of what passes between apache and haproxy. Maybe it's not hard to improve the workaround to cover your case if that's the issue. Regards, Willy
RE: Apache translates 500 to 502 from haproxy
Hey Willy, tcpdump would be fine? Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, June 14, 2011 12:14 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 11:25:19AM +0530, Sachin Shetty wrote: Hey Willy, We have 1.4.11 - so how do we use the workaround - This is already in prod and I need to work around it before we can get to the root cause. The workaround is part of the code, so you're hitting a different issue I think. Please do not hesitate to send me a network capture of what passes between apache and haproxy. Maybe it's not hard to improve the workaround to cover your case if that's the issue. Regards, Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 12:22:03PM +0530, Sachin Shetty wrote: Hey Willy, tcpdump would be fine? Yes, for instance : tcpdump -s0 -npi eth0 port 80 -w trace1.cap Please note that -s0 is very important otherwise packets will be truncated. Regards, Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 01:09:12PM +0530, Sachin Shetty wrote: here is the tcp dump containing that precise request: tcpdump -s0 -npi lo port 9910 -w trace1.cap There is nothing abnormal there, the response was complete and not truncated. The connection was eventually closed before all of the request was read, but I see nothing wrong with that and it is quite common with redirects or 401. It's very strange that your apache does return a 502 here, because it received the response long before the connection was closed, so it should already have forwarded it. Also many people are using apache in front of haproxy without such a problem. Are you sure there is not anything in your apache config/modules which buffers the whole response before responding ? It might explain this different behaviour. Willy
can't figure out why this is causing a CQ
I can't see what I am missing here.
Any help is appreciated
Jun 14 02:00:00 localhost haproxy[3052]: 10.101.1.2:2892
[14/Jun/2011:02:00:00.088] w wi/wi-9 35/111/-1/-1/146 503 212 W=9 - CQ--
202/202/27/18/0 1/0 {w.x.y.z|Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)
AppleWebKit/534.7 (KHTML, like Gecko) Chrome/1.0.11.4
Safari/534.7|1549|http://w.x.y.z/w/web/w.jsp?ID=1sid=42||lang=en;
auth=5/z/Z; W=9||} {} POST /w/web/w.jsp?ID=1sid=42 HTTP/1.1
config:
defaults
#option splice-auto
option tcp-smart-connect
option http-server-close
timeout queue 21s
timeout http-request 5s
timeout client 38s
timeout connect 8s
timeout server 38s
timeout http-keep-alive 8s
timeout tarpit 120s
global
node w1
loglocalhost local0
# loglocalhost local0 err
maxconn32768
uid99
gid99
chroot /var/empty
pidfile/var/run/haproxy.pid
daemon
quiet
spread-checks 6
frontend w
bind 10.1.1.1:80
mode http
log global
option httplog
option http-server-close
option log-separate-errors
maxconn 32768
capture request header Host len 32
capture request header User-Agent len 256
capture request header Content-Length len 10
capture request header Refererlen 384
capture request header Vialen 64
capture request header Cookie len 128
capture request header X-xylen 64
capture response header Content-Length len 10
capture request header X-xxlen 32
# block any unwanted source IP addresses or networks
acl forbidden_src src 0.0.0.0/7 224.0.0.0/3
# acl forbidden_src src_port 0:1023
block if forbidden_src
default_backend wi
capture cookie W= len 13
backend wi
modehttp
balance roundrobin
cookie W
server wi-7 10.1.1.7:80 cookie 7 check observe layer7 inter 90s
fastinter 12s downinter 18s rise 2 fall 4 slowstart 360s weight 100 maxconn
18 maxqueue 64
server wi-9 10.1.1.9:80 cookie 9 check observe layer7 inter 90s
fastinter 12s downinter 18s rise 2 fall 4 slowstart 360s weight 100 maxconn
18 maxqueue 64
server wi-0 10.1.1.0:80 cookie 0 check observe layer7 inter 90s fastinter 12s
downinter 18s rise 2 fall 4 slowstart 360s weight 100 maxconn 18 maxqueue 64
server wi-1 10.1.1.1:80 cookie 1 check observe layer7 inter 90s fastinter 12s
downinter 18s rise 2 fall 4 slowstart 360s weight 100 maxconn 18 maxqueue 64
server wi-2 10.1.1.2:80 cookie 2 check observe layer7 inter 90s fastinter 12s
downinter 18s rise 2 fall 4 slowstart 360s weight 100 maxconn 18 maxqueue 64
server wi-3 10.1.1.3:80 cookie 3 check observe layer7 inter 90s fastinter 12s
downinter 18s rise 2 fall 4 slowstart 360s weight 100 maxconn 18 maxqueue 64
server wi-8 10.1.1.8:80 cookie 8 check observe layer7 inter 90s fastinter 12s
downinter 18s rise fall 4 slowstart 360s weight 100 maxconn 18 maxqueue 64
#server wi-2 10.1.1.2:80 cookie 2 check observe layer7 inter 90s
fastinter 12s downinter 18s rise 2 fall 4 slowstart 360s backup weight 100
maxconn 24
option redispatch
retries 3
option forwardfor
option http-server-close
#option forceclose
option abortonclose
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 02:14:15PM +0530, Sachin Shetty wrote: I looked at all the apache modules we have and haven't seen anything glaring there. Attached tcpdump from cherrypy server when Apache is bypassing haproxy and forwarding to cherrypy directly. Obviously cherrypy does it in order, loads the whole file and then sends the response back which works ok for Apache. I understand, but I see nothing in HTTP which makes it mandatory to read all the content once you have responded. If you respond, you don't need those data anymore. The common practice is to read at least some of them in order to avoid the issue with the close causing a reset. Haproxy reads as much as it can until the close. Once the connection is closed, it cannot read anymore. Apache accepts to read a lot more of data,but with a limit too. Willy
RE: Apache translates 500 to 502 from haproxy
Yeah, I understand. So what could we do? I am really stuck with this and not able to figure out any workaround either. -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, June 14, 2011 4:17 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 02:14:15PM +0530, Sachin Shetty wrote: I looked at all the apache modules we have and haven't seen anything glaring there. Attached tcpdump from cherrypy server when Apache is bypassing haproxy and forwarding to cherrypy directly. Obviously cherrypy does it in order, loads the whole file and then sends the response back which works ok for Apache. I understand, but I see nothing in HTTP which makes it mandatory to read all the content once you have responded. If you respond, you don't need those data anymore. The common practice is to read at least some of them in order to avoid the issue with the close causing a reset. Haproxy reads as much as it can until the close. Once the connection is closed, it cannot read anymore. Apache accepts to read a lot more of data,but with a limit too. Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 04:27:00PM +0530, Sachin Shetty wrote: Yeah, I understand. So what could we do? I am really stuck with this and not able to figure out any workaround either. I really have no idea, because the server already replies before the data are sent. Maybe you could disable the http-server-close option so that haproxy works in tunnel mode. It will then not analyse any request nor response beyond headers and will let client and server agree on when to close the connection. You may even resort to the plain old option httpclose so that each other are aware that the connection is used only for one request. Willy
RE: Apache translates 500 to 502 from haproxy
man We already had option httpclose in our config and removing it fixed it. We haven't tweaked any configs for a few months, I dont even know why we had this in the first place :) I read through the documentation, I dont think we need it. Do you have any reservations about taking it out? Thanks Sachin -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Tuesday, June 14, 2011 5:15 PM To: Sachin Shetty Cc: 'Cassidy, Bryan'; haproxy@formilux.org Subject: Re: Apache translates 500 to 502 from haproxy On Tue, Jun 14, 2011 at 04:27:00PM +0530, Sachin Shetty wrote: Yeah, I understand. So what could we do? I am really stuck with this and not able to figure out any workaround either. I really have no idea, because the server already replies before the data are sent. Maybe you could disable the http-server-close option so that haproxy works in tunnel mode. It will then not analyse any request nor response beyond headers and will let client and server agree on when to close the connection. You may even resort to the plain old option httpclose so that each other are aware that the connection is used only for one request. Willy
Re: Apache translates 500 to 502 from haproxy
On Tue, Jun 14, 2011 at 06:16:23PM +0530, Sachin Shetty wrote: man We already had option httpclose in our config and removing it fixed it. We haven't tweaked any configs for a few months, I dont even know why we had this in the first place :) I read through the documentation, I dont think we need it. Do you have any reservations about taking it out? If you don't have the option, then only the first request and first response of each connection is analysed. So if Apache does keep-alive with the server over haproxy, then haproxy won't parse the second and subsequent requests. If you were already having httpclose, then haproxy did not close by itself, so that means that the server was closing the connection after it had nothing else to send. In other words, we have two servers (haproxy and the server behind it) who agree on acting the same way and the Apache definitely is the issue here. Could you just make a try with option http-server-close then ? I think it won't work due to the server closing, but if it does it would be better. I'll have to think about implementing a drain mode over keep-alive connections for this precise case : if the connection to the server is dead while the connection to the client is active with data still coming, we should silently drain those data. Regards, Willy
Re: acl and multiple header values
On Tue, Jun 14, 2011 at 04:43:47PM -0700, John Fieber wrote: I want to create an ACL based on X-Forwarded-For: acl whitelist hdr_ip(X-Forwarded-For) -f whitelist.txt block unless whitelist Which is just grand, EXCEPT I'm only interested in (and trust) the last address in the X-Forwarded-For header. The above acl matches any address in the header. I've been digging for a good chunk of the day how to do that and come up empty handed. Help? Since we have not yet reworked the ACLs to rely on the pattern subsystem, it's still not possible to make use of hdr_ip(X-f-f,-1) as we do on the balance or source keywords. One thing you could do, despite not being very good, is to remove all occurrences of values in the header. Basically, remove everything from the first char to the last comma : reqirep ^(X-Forwarded-For:\ ).*,([^,]*) \1\2 Then your ACL could match based on what is left in this header. Regards, Willy
Re: can't figure out why this is causing a CQ
Hi Hank,
On Tue, Jun 14, 2011 at 03:12:27AM -0700, Hank A. Paulson wrote:
I can't see what I am missing here.
Any help is appreciated
Jun 14 02:00:00 localhost haproxy[3052]: 10.101.1.2:2892
[14/Jun/2011:02:00:00.088] w wi/wi-9 35/111/-1/-1/146 503 212 W=9 - CQ--
202/202/27/18/0 1/0 {w.x.y.z|Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/1.0.11.4
Safari/534.7|1549|http://w.x.y.z/w/web/w.jsp?ID=1sid=42||lang=en;
auth=5/z/Z; W=9||} {} POST /w/web/w.jsp?ID=1sid=42 HTTP/1.1
You're not missing anything, CQ means the client disconnected while
the request was waiting in the queue. The client aborted 111 ms after
his request went into the queue, and since you have option abortonclose
the request was never forwarded to the server.
Regards,
Willy
