Re: Haproxy and UTF8-encoded chars
On Fri, Jul 27, 2012 at 8:49 AM, Stojan Rancic (Iprom) sto...@iprom.si wrote: On 26.7.2012 10:15, Sander Klein wrote: So I definitely think Internet Explorer is doing it wrong. It relies on the fact that most web servers will encode the URL for them, which most actually do If you really want to accept the 'bad' URLs then you might enable 'option accept-invalid-http-request' but I strongly recommend to not enable this in a production environment. I did a bit of ngrep-ing yesterday as well, and we are indeed seeing non-ASCII encoded characters flowing in URLs and Referrers, from MSIE clients. We'll see how we handle those on our actual backends (without HAproxy forwarding the traffic), and then decide on what the next step would be. Thanks for the input so far. br, Stojan Hi, You could use as well the option option accept-invalid-http-request to tell HAProxy not to run character compliance testing. This could be used temporary, the time for the applications to be fixed. Baptiste
Re: Haproxy and UTF8-encoded chars
On 26.7.2012 10:15, Sander Klein wrote: So I definitely think Internet Explorer is doing it wrong. It relies on the fact that most web servers will encode the URL for them, which most actually do If you really want to accept the 'bad' URLs then you might enable 'option accept-invalid-http-request' but I strongly recommend to not enable this in a production environment. I did a bit of ngrep-ing yesterday as well, and we are indeed seeing non-ASCII encoded characters flowing in URLs and Referrers, from MSIE clients. We'll see how we handle those on our actual backends (without HAproxy forwarding the traffic), and then decide on what the next step would be. Thanks for the input so far. br, Stojan
Re: Haproxy and UTF8-encoded chars
On 25.7.2012 11:21, Sander Klein wrote: We are experiencing the same issue, but it only happens with Internet Explorer. So I figured it must be a bug on the internet explorer side and not on the HAProxy side since internet explorer doesn't seem to encode the URL correctly. I'm afraid I don't have any control over what browsers the users are using, and I'm sure a fair amount of those are IE . And the fact that I'm seeing \x escaped characters in both GET and Referrer headers isn't helping any either. How do you deal with IE users then ? br, Stojan
Re: Haproxy and UTF8-encoded chars
On 26.07.2012 09:44, Stojan Rancic (Iprom) wrote: On 25.7.2012 11:21, Sander Klein wrote: We are experiencing the same issue, but it only happens with Internet Explorer. So I figured it must be a bug on the internet explorer side and not on the HAProxy side since internet explorer doesn't seem to encode the URL correctly. I'm afraid I don't have any control over what browsers the users are using, and I'm sure a fair amount of those are IE . And the fact that I'm seeing \x escaped characters in both GET and Referrer headers isn't helping any either. How do you deal with IE users then ? This is always a bit problematic. If the URL is being generated from our software then we fix our software to create pre-encoded URLs. If it's 3rd party software, we tell the 3rd party to fix their stuff. Currently we have one case where the 3rd party doesn't understand the issue, and then we just tell the users to start using a browser which does proper encoding. Because of your question I wiresharked a bit yesterday to make sure I was giving you the right info. My tests showed that Safari, Firefox and Chrome do proper encoding of the URL before sending it and Internet Explorer only encodes some parts of the URL. I also check RFC3986 and it says in section 2.5 paragraph 6: --- When a new URI scheme defines a component that represents textual data consisting of characters from the Universal Character Set [UCS], the data should first be encoded as octets according to the UTF-8 character encoding [STD63]; then only those octets that do not correspond to characters in the unreserved set should be percent- encoded. For example, the character A would be represented as A, the character LATIN CAPITAL LETTER A WITH GRAVE would be represented as %C3%80, and the character KATAKANA LETTER A would be represented as %E3%82%A2. --- So I definitely think Internet Explorer is doing it wrong. It relies on the fact that most web servers will encode the URL for them, which most actually do If you really want to accept the 'bad' URLs then you might enable 'option accept-invalid-http-request' but I strongly recommend to not enable this in a production environment. Greets, Sander Klein
Re: Haproxy and UTF8-encoded chars
On 07/25/2012 08:22 AM, Stojan Rancic (Iprom) wrote: Hello, we're experiencing issues with HAproxy 1.5-dev11 rejecting GET requests with UTF8-encoded characters. The encoding happens with Javascript's Encode function for east european characters (š, č, ž, etc) . The requests (as seen from 'echo show errors | socat stdio unix-connect:/var/run/haproxy.sock') are: Total events captured on [25/Jul/2012:08:10:16.070] : 1347 [25/Jul/2012:08:10:16.033] frontend http-in (#2): invalid request backend NONE (#-1), server NONE (#-1), event #1346 src 91.216.172.145:40752, session #527199, session flags 0x HTTP msg state 27, msg flags 0x, tx flags 0x HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x00809002, out 0 bytes, total 873 bytes pending 873 bytes, wrapping at 16384, error at position 51: 0 GET /XXX/YYY?z=39;t=js;sid=index;ssid=\xC2\xA7=index;m=ZZ 00064+ ZZ;ref=http://www.ZZZ.com/lala;num=9;kw=;flash=0;res=lala;r 00134+ e=http%3A%2F%2Fwww.QQQ.com%2Fsi%2FZZZ.html;rmc=1343196615443;cpre 00204+ mium=false;url=http%3A//www.ZZZ.com/lala HTTP/1.1\r\n I think that you should urlencode utf8 strings you want to put into URI or query string. Check out encodeURIComponent() -- Brane F. Gračnar skrbnik aplikacij/applications manager e: brane.grac...@tsmedia.si TSmedia, d.o.o. a: Cigaletova 15, 1000 Ljubljana; Slovenia t: +386 1 473 00 10 f: +386 1 473 00 16
Re: Haproxy and UTF8-encoded chars
On 26.7.2012 14:51, Brane F. Gračnar wrote: 0 GET /XXX/YYY?z=39;t=js;sid=index;ssid=\xC2\xA7=index;m=ZZ 00064+ ZZ;ref=http://www.ZZZ.com/lala;num=9;kw=;flash=0;res=lala;r 00134+ e=http%3A%2F%2Fwww.QQQ.com%2Fsi%2FZZZ.html;rmc=1343196615443;cpre 00204+ mium=false;url=http%3A//www.ZZZ.com/lala HTTP/1.1\r\n I think that you should urlencode utf8 strings you want to put into URI or query string. Check out encodeURIComponent() After talking to our team, this is exactly what we're supposed to be using.. encodeURIComponent(header.referrer)
Haproxy and UTF8-encoded chars
Hello, we're experiencing issues with HAproxy 1.5-dev11 rejecting GET requests with UTF8-encoded characters. The encoding happens with Javascript's Encode function for east european characters (š, č, ž, etc) . The requests (as seen from 'echo show errors | socat stdio unix-connect:/var/run/haproxy.sock') are: Total events captured on [25/Jul/2012:08:10:16.070] : 1347 [25/Jul/2012:08:10:16.033] frontend http-in (#2): invalid request backend NONE (#-1), server NONE (#-1), event #1346 src 91.216.172.145:40752, session #527199, session flags 0x HTTP msg state 27, msg flags 0x, tx flags 0x HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x00809002, out 0 bytes, total 873 bytes pending 873 bytes, wrapping at 16384, error at position 51: 0 GET /XXX/YYY?z=39;t=js;sid=index;ssid=\xC2\xA7=index;m=ZZ 00064+ ZZ;ref=http://www.ZZZ.com/lala;num=9;kw=;flash=0;res=lala;r 00134+ e=http%3A%2F%2Fwww.QQQ.com%2Fsi%2FZZZ.html;rmc=1343196615443;cpre 00204+ mium=false;url=http%3A//www.ZZZ.com/lala HTTP/1.1\r\n 00260 Via: 1.0 SREMISA2\r\n 00279 Cookie: CT=true; UID=121617214516589645; CA1590=0:0:0:4:4:3:0:13395679 00349+ 37; AD8413=0:0:0:1:1:1:0:1339563959; CA1645=0:0:0:4:4:3:0:1339568497; 00419+ CA1617=0:0:0:4:1:1:0:1341379400; AD8579=0:0:0:5:2:2:0:1340173696; AD86 00489+ 86=0:0:0:3:3:1:0:1343195149; CA1686=0:0:0:3:3:1:0:1343195149; CA1688=0 00559+ :0:0:1:1:1:0:1343195915\r\n 00584 Referer: http://www.ZZZ.com/lala\r\n 00623 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .N 00693+ ET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 00763+ 3.5.30729; InfoPath.3)\r\n 00787 Host: XXX.com\r\n 00813 Accept: */*\r\n 00826 Accept-Language: sl\r\n 00847 Connection: Keep-Alive\r\n 00871 \r\n The configuration of haproxy attached. global log 127.0.0.1 local1 notice maxconn 409600 user haproxy group haproxy daemon stats socket /var/run/haproxy.sock gid 117 mode 0660 level operator defaults log global modehttp option httplog option dontlognull retries 3 option redispatch maxconn 4 timeout connect 3000 timeout client 5 timeout server 5 option httpclose option forwardfor frontend http-in option httplog log global option log-separate-errors bind XXX:80 acl is_XXX hdr_end(host) -i XXX.net use_backend XXX if is_XXX default_backend YYY backend XXX mode http timeout connect 1000 timeout server 1000 timeout http-request 1000 timeout queue 1000 balance leastconn server A XXX:80 check inter 2000 rise 2 fall 5 server B YYY:80 check inter 2000 rise 2 fall 5 backend YYY mode http log global option httplog timeout connect 3000 timeout server 3000 timeout http-request 3000 timeout queue 3000 balance leastconn server C ZZZ:80 check inter 2000 rise 2 fall 5 server D QQQ:80 check inter 2000 rise 2 fall 5
Re: Haproxy and UTF8-encoded chars
Hi, On 25.07.2012 08:22, Stojan Rancic (Iprom) wrote: Hello, we're experiencing issues with HAproxy 1.5-dev11 rejecting GET requests with UTF8-encoded characters. The encoding happens with Javascript's Encode function for east european characters (š, č, ž, etc) . We are experiencing the same issue, but it only happens with Internet Explorer. So I figured it must be a bug on the internet explorer side and not on the HAProxy side since internet explorer doesn't seem to encode the URL correctly. Greets, Sander