Re: Haproxy and UTF8-encoded chars
On 25.7.2012 11:21, Sander Klein wrote: We are experiencing the same issue, but it only happens with Internet Explorer. So I figured it must be a bug on the internet explorer side and not on the HAProxy side since internet explorer doesn't seem to encode the URL correctly. I'm afraid I don't have any control over what browsers the users are using, and I'm sure a fair amount of those are IE . And the fact that I'm seeing \x escaped characters in both GET and Referrer headers isn't helping any either. How do you deal with IE users then ? br, Stojan
Re: IPv6 for source directive?
Hi Stephen, On Tue, Jul 24, 2012 at 11:41:27AM -0700, Stephen Balukoff wrote: Hi Willy, Awesome! Ok, I was able to figure out how to make this work on each 'server' line. Note that it would probably be a good idea to update the documentation for the 'source' directive such it doesn't specify IPv4 explicitly (ie. giving an implication that IPv6 addresses are not allowed). Plus it would be a good idea to have an example of an IPv6 server line so that people see the right way to specify an IPv6 address without specifying the port (ie. IPv6 address with trailing colon-- somewhat different than I've seen with other software where the IPv6 address is surrounded by square brackets). These are good ideas. Could you please propose a patch with such changes, it is always better when the people who experience issues with the doc propose fixes, as they don't rely on things that are implied for me (and it scales better too). In any case, the functionality that's in haproxy 1.5 meets our needs and I can verify that having both IPv4 and IPv6 servers in the same back-end pool appears to be working as expected! It would be awesome if one could specify a pool of possible source addresses and have haproxy pick an appropriate source address for the address family of the back-end server, depending on whether the back-end uses an IPv4 or IPv6 address. But in any case, again, what's in 1.5 will definitely work for what we're trying to do, eh! I have this feature in my traffic generator, from which I took the port range allocation. I think that I will implement it after the connection handling is reworked, because health checks and traffic will then be able to use the same code to establish a connection. The syntax I'm expecting is the same as I already use in the other tool: ip1-ip2:port1-port2. Anyway, there is little need for multiple source IPs right now, it only happens for people who want to deal with more than 64k connections per server behind the LB, and you probably know like me that such setups are extremely rare with todays technologies ! That's why this task remains low on the priority list. Regards, Willy
Re: Linux kernel crash with haproxy 1.5-dev11 and ipv6 listener
On Tue, Jul 24, 2012 at 11:55:17AM -0700, Stephen Balukoff wrote: Hi Willy, I'm compiling a main-line 2.6.32.27 kernel and will let you know if I'm able to crash this as easily as the RHEL-6 kernel. (I couldn't find a 2.6.32.59 kernel-- it doesn't appear to be on ftp.kernel.org.) You have it here : http://www.kernel.org/pub/linux/kernel/v2.6/longterm/v2.6.32/ I know that the move to the longterm subdir causes a real mess to users, but I didn't choose it :-/ It's really important to be on the updated version, because if the issue was fixed between .27 and .59, there's no point trying to find whether you can reproduce it or not. In short, if .59 is OK and RHEL is KO, then you can simply report it to them and they'll be able to figure out what patch in their kernel causes this. If it's also in mainline, we need to fix it and the fix will naturally find its way to RHEL's kernel. And in any case, it's extremely reproducible: All I have to do is attempt to make an IPv6 TCP connection to an IPv6 haproxy listener and the kernel crashes instantly (ie. reproducible with essentially one packet). Impressed ! Also note: I've verified that I get no crashing with a main-line 3.4.5 kernel. OK that's great! Thanks, Willy
Re: Haproxy and UTF8-encoded chars
On 26.07.2012 09:44, Stojan Rancic (Iprom) wrote: On 25.7.2012 11:21, Sander Klein wrote: We are experiencing the same issue, but it only happens with Internet Explorer. So I figured it must be a bug on the internet explorer side and not on the HAProxy side since internet explorer doesn't seem to encode the URL correctly. I'm afraid I don't have any control over what browsers the users are using, and I'm sure a fair amount of those are IE . And the fact that I'm seeing \x escaped characters in both GET and Referrer headers isn't helping any either. How do you deal with IE users then ? This is always a bit problematic. If the URL is being generated from our software then we fix our software to create pre-encoded URLs. If it's 3rd party software, we tell the 3rd party to fix their stuff. Currently we have one case where the 3rd party doesn't understand the issue, and then we just tell the users to start using a browser which does proper encoding. Because of your question I wiresharked a bit yesterday to make sure I was giving you the right info. My tests showed that Safari, Firefox and Chrome do proper encoding of the URL before sending it and Internet Explorer only encodes some parts of the URL. I also check RFC3986 and it says in section 2.5 paragraph 6: --- When a new URI scheme defines a component that represents textual data consisting of characters from the Universal Character Set [UCS], the data should first be encoded as octets according to the UTF-8 character encoding [STD63]; then only those octets that do not correspond to characters in the unreserved set should be percent- encoded. For example, the character A would be represented as A, the character LATIN CAPITAL LETTER A WITH GRAVE would be represented as %C3%80, and the character KATAKANA LETTER A would be represented as %E3%82%A2. --- So I definitely think Internet Explorer is doing it wrong. It relies on the fact that most web servers will encode the URL for them, which most actually do If you really want to accept the 'bad' URLs then you might enable 'option accept-invalid-http-request' but I strongly recommend to not enable this in a production environment. Greets, Sander Klein
Backend redirection issue - log interpretation help please
I'm having some trouble interpreting an HAProxy log file line - it doesn't seem to make much sense to me, so I'm fairly certain that I'm reading it incorrectly. My full configuration file is at the end of this email. Symptomatically, I've been seeing situations in which people are briefly redirected unexpectedly onto a different 'gui' backend server. Its rare, but when it happens it often appears at the same point in the GUI use flow. This may be a red herring however. This morning I was able to reproduce it even after restarting our appserver proceses and our haproxy process (which had been up for an extended period of time - thanks!). When it reproduced, I noticed that the backend appserver was marked as DOWN in the statistics report even though the logfiles gave no indication that there were any problems. From the user's experience, the system hung waiting for a response for some 40 seconds and then showed a login screen, which was served when the incorrect backend received a page request. Simply re-requesting the previous page was successful, as the previously authenticated backend was shown as UP by that point. In the logfile we see the following entry at the appropriate time: Jul 26 06:21:09 localhost haproxy[3853]: 172.25.200.177:3124 [26/Jul/2012:06:21:01.826] secure gui/gui2 4993/0/0/-1/7992 -1 0 - - CH-- 40/40/1/1/0 0/0 HEAD /panel/healthcheck.html HTTP/1.1 This file - while similar to the HAProxy healthcheck URL (and in fact produced the same way on the backend server) - is actually the one being requested by our hardware LBs (we use existing hardware LBs to distribute between machines, and a single haproxy daemon on each box to distribute to appservers within each machine). It appears that it saw some sort of timeout, but I'm not sure why that would have caused haproxy to mark the backend as DOWN. We add backend server markers to our generated pages, so I'm confident that the hardware LB kept us on the same physical system the entire time. Needless to say I'm a little confused as to what I'm seeing. If anyone can shed a little more light on the subject it would be very much appreciated! Other than this recurrent issue, everything's been working incredibly well. Configuration file below in full in case there's something odd: global maxconn 2 tune.bufsize 128000 tune.maxrewrite 8192 ulimit-n 6 stats socket /tmp/haproxy mode 0777 level admin log 127.0.0.1 local0 defaults mode http timeout connect 5s timeout client 1m timeout server 10m option srvtcpka option http-server-close option http-pretend-keepalive option redispatch option abortonclose option httpchk HEAD /panel/healthcheck.txt HTTP/1.0 backend gui balance source appsession JSESSIONID len 64 timeout 3h server gui1 172.25.200.53:8080 check maxconn 2000 server gui2 172.25.200.54:8080 check maxconn 2000 backend platform balance roundrobin server platform1 172.25.200.50:8080 check maxconn 2000 server platform2 172.25.200.51:8080 check maxconn 2000 server platform3 172.25.200.52:8080 check maxconn 2000 frontend secure bind 172.25.200.70:8080 maxconn 2 option forwardfor option clitcpka acl javascript url_beg /js acl widgetsurl_beg /widgets acl reportsurl_beg /datafeed acl deny url_beg /server-status /balancer-manager /jmx-console /web-console /invoker block if deny default_backend gui use_backend platform if javascript or widgets or reports log global option httplog option dontlognull option dontlog-normal listen stats :7583 mode http stats uri /
Re: Haproxy and UTF8-encoded chars
On 07/25/2012 08:22 AM, Stojan Rancic (Iprom) wrote: Hello, we're experiencing issues with HAproxy 1.5-dev11 rejecting GET requests with UTF8-encoded characters. The encoding happens with Javascript's Encode function for east european characters (š, č, ž, etc) . The requests (as seen from 'echo show errors | socat stdio unix-connect:/var/run/haproxy.sock') are: Total events captured on [25/Jul/2012:08:10:16.070] : 1347 [25/Jul/2012:08:10:16.033] frontend http-in (#2): invalid request backend NONE (#-1), server NONE (#-1), event #1346 src 91.216.172.145:40752, session #527199, session flags 0x HTTP msg state 27, msg flags 0x, tx flags 0x HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x00809002, out 0 bytes, total 873 bytes pending 873 bytes, wrapping at 16384, error at position 51: 0 GET /XXX/YYY?z=39;t=js;sid=index;ssid=\xC2\xA7=index;m=ZZ 00064+ ZZ;ref=http://www.ZZZ.com/lala;num=9;kw=;flash=0;res=lala;r 00134+ e=http%3A%2F%2Fwww.QQQ.com%2Fsi%2FZZZ.html;rmc=1343196615443;cpre 00204+ mium=false;url=http%3A//www.ZZZ.com/lala HTTP/1.1\r\n I think that you should urlencode utf8 strings you want to put into URI or query string. Check out encodeURIComponent() -- Brane F. Gračnar skrbnik aplikacij/applications manager e: brane.grac...@tsmedia.si TSmedia, d.o.o. a: Cigaletova 15, 1000 Ljubljana; Slovenia t: +386 1 473 00 10 f: +386 1 473 00 16
Re: Haproxy and UTF8-encoded chars
On 26.7.2012 14:51, Brane F. Gračnar wrote: 0 GET /XXX/YYY?z=39;t=js;sid=index;ssid=\xC2\xA7=index;m=ZZ 00064+ ZZ;ref=http://www.ZZZ.com/lala;num=9;kw=;flash=0;res=lala;r 00134+ e=http%3A%2F%2Fwww.QQQ.com%2Fsi%2FZZZ.html;rmc=1343196615443;cpre 00204+ mium=false;url=http%3A//www.ZZZ.com/lala HTTP/1.1\r\n I think that you should urlencode utf8 strings you want to put into URI or query string. Check out encodeURIComponent() After talking to our team, this is exactly what we're supposed to be using.. encodeURIComponent(header.referrer)
HC admin语您共享了照片
Have a nice day! Thank you for reading my email!
Re: Backend redirection issue - log interpretation help please
Hi Richard, Le 26/07/2012 13:46, Richard Stanford a écrit : This morning I was able to reproduce it even after restarting our appserver proceses and our haproxy process (which had been up for an extended period of time - thanks!). When it reproduced, I noticed that the backend appserver was marked as DOWN in the statistics report even though the logfiles gave no indication that there were any problems. Please add log global in your backend sections (or in your defaults), this explains why your log files didn't give you any indication. After adding this line everywhere (not only in the frontend), you'll see when servers go UP and DOWN, and why. This will also probably help us to know what happens. From the user's experience, the system hung waiting for a response for some 40 seconds and then showed a login screen, which was served when the incorrect backend received a page request. Simply re-requesting the previous page was successful, as the previously authenticated backend was shown as UP by that point. In the logfile we see the following entry at the appropriate time: Jul 26 06:21:09 localhost haproxy[3853]: 172.25.200.177:3124 [26/Jul/2012:06:21:01.826] secure gui/gui2 4993/0/0/-1/7992 -1 0 - - CH-- 40/40/1/1/0 0/0 HEAD /panel/healthcheck.html HTTP/1.1 From this previous log line, it looks like something becomes slow (your haproxy server or your backend servers). Configuration file below in full in case there's something odd: global maxconn 2 tune.bufsize 128000 Wow, are you sure you really want to use such a big buffer size ? Also, ensure you're running the last stable version of HAProxy (currently 1.4.21), which fixes a major bug when using a larger buffer size (it doesn't explain what you observe but it's an advise for more stability). For more details : http://haproxy.1wt.eu/git?p=haproxy-1.4.git;a=commit;h=30297cb17147a8d339eb160226bcc08c91d9530b defaults mode http timeout connect 5s timeout client 1m timeout server 10m option srvtcpka option http-server-close option http-pretend-keepalive option redispatch option abortonclose option httpchk HEAD /panel/healthcheck.txt HTTP/1.0 As said at the beginning, please add : log global backend gui balance source appsession JSESSIONID len 64 timeout 3h If using cookies is not an issue for your clients, I'd recommend you not to use appsession but cookie insert or cookie prefix instead. http://cbonte.github.com/haproxy-dconv/configuration-1.4.html#4-cookie server gui1 172.25.200.53:8080 check maxconn 2000 server gui2 172.25.200.54:8080 check maxconn 2000 You didn't provide any timeout check nor inter value. The default will be 2 seconds, which is maybe too low for your case. http://cbonte.github.com/haproxy-dconv/configuration-1.4.html#inter http://cbonte.github.com/haproxy-dconv/configuration-1.4.html#timeout%20check Hope this helps. -- Cyril Bonté
