Re: Backend Server UP/Down Debugging?
On Thu, 27 Aug 2009, Willy Tarreau wrote: Hi, Hi, On Wed, Aug 26, 2009 at 02:00:42PM -0700, Jonah Horowitz wrote: I???m watching my servers on the back end and occasionally they flap. I???m wondering if there is a way to see why they are taken out of service. I???d like to see the actual response, or at least a HTTP status code. right now it's not archived. I would like to keep a local copy of the last request sent and response received which caused a state change, but that's not implemented yet. I wanted to clean up the stats socket first, but now I realize that we could keep at least some info (eg: HTTP status, timeout, ...) in the server struct itself and report it in the log. Nothing of that is performed right now, so you may have to tcpdump at best :-( As always, I have a patch for that, solving it nearly exactly like you described it. ;) However for the last half year I have been rather silent, mostly because it is very important time in my private life, so I think I'm partially excused. ;) I know that there are some unfinished tasks (acl for exapmple) so I'll try to push ASAP, maybe starting from the easier patches, likt this ones. The rest will have to wait when I get back from honeymoon. One trick you can do to make this experience better is to set a port or addr option on your server line to run the checks on a different ip/port combination. That way, you can filter that on tcpdump and only get the checks. Or to use something like: echo -e GET query HTTP/1.0\r\nhost: host\r\n|nc host port|less -S Best regards, Krzysztof Olędzki
Re: Backend Server UP/Down Debugging?
Hi Krzysztof, glad to get news from you ! On Thu, Aug 27, 2009 at 08:45:23AM +0200, Krzysztof Oledzki wrote: (...) right now it's not archived. I would like to keep a local copy of the last request sent and response received which caused a state change, but that's not implemented yet. I wanted to clean up the stats socket first, but now I realize that we could keep at least some info (eg: HTTP status, timeout, ...) in the server struct itself and report it in the log. Nothing of that is performed right now, so you may have to tcpdump at best :-( As always, I have a patch for that, solving it nearly exactly like you described it. ;) excellent. I was at least twice tempted to do it but lacked the time for it! When the stats socket processing looks better, I hope to quickly add a show health entry reporting all health check status. However for the last half year I have been rather silent, mostly because it is very important time in my private life, so I think I'm partially excused. ;) Hey, you're completely excused, private life is more important than code :-) I know that there are some unfinished tasks (acl for exapmple) so I'll try to push ASAP, maybe starting from the easier patches, likt this ones. The rest will have to wait when I get back from honeymoon. No problem, your patches will be welcome as usual ! One trick you can do to make this experience better is to set a port or addr option on your server line to run the checks on a different ip/port combination. That way, you can filter that on tcpdump and only get the checks. Or to use something like: echo -e GET query HTTP/1.0\r\nhost: host\r\n|nc host port|less -S that's an option too, indeed. Best regards, Willy
Re: round robin
Ok thanks Willy for your help. - Original Message - From: Willy Tarreau w...@1wt.eu To: Johan Duflost mail...@decizium.com Cc: Angelo Höngens a.hong...@netmatch.nl; haproxy@formilux.org Sent: Thursday, August 27, 2009 6:32 AM Subject: Re: round robin On Wed, Aug 26, 2009 at 03:39:44PM +0200, Johan Duflost wrote: Hello Willy, Ok I can understand this issue but when session stickyness is enabled, all the objects should be retreived from the same server, isn't it? yes you're right, they should. So it's probably not our problem. Then check your server's downtime on the stats page. It's possible that one server sometimes goes down and does not receive requests. Also check the Redis column, which will indicate if a server sometimes does not respond to connection requests, causing them to be retries on the other server. Willy No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.409 / Virus Database: 270.13.69/2328 - Release Date: 08/26/09 12:16:00
Re: TCP log format question
On Thu, Aug 27, 2009 at 06:39:51AM +0200, Willy Tarreau wrote: I'm seeing that you have both tcplog and httplog. Since they both add a set of flags, the union of both is enabled which means httplog to me. I should add a check for this so that tcplog disables httplog. In my log file I see the following lines: Aug 26 18:19:50 balancer0-00 haproxy[66301]: A.B.C.D:28689 [26/Aug/2009:18:19:50.034] M-front M-native/ms1 -1/1/0/-1/3 -1 339 - - CD-- 0/0/0/0/0 0/0 BADREQ 1) What does BADREQ mean? I see no description of that field in documentation of TCP log format. this is because of option httplog. Aha, I see, i had an impression 'option httplog' will be ignored in TCP mode. I removed it and BADREQ disappeared from the log. 2) Why *all* requests are being logged? (note option dontlog-normal in default section). How should I change configuration to log only important events (errors) and do not log the fact connection was made and served? Hmmm dontlog-normal only works in HTTP mode. Ok, I see, though it is completely unclean after reading the manual. This should probably be explicitly mentioned. Could you please explain what type of normal connections you would want to log and what type you would not want to log ? It could help making a choice of implementation of dontlog-normal for tcplog. I want to log exactly what manual states: ### Setting this option ensures that normal connections, those which experience no error, no timeout, no retry nor redispatch, will not be logged. ### ... but for TCP mode proxy. I mean I want to see in logs only those connection that were redispatched, timeouted, etc. Thanks!
