Re: cookie prefix strange behavior
Hi willy. Am 27-07-2015 07:26, schrieb Willy Tarreau: Hi Aleks, On Sun, Jul 26, 2015 at 11:42:42PM +0200, Aleksandar Lazic wrote: As far as I know there is not yet a flow chart like http://redmine.lighttpd.net/projects/lighttpd/repository/entry/trunk/doc/state.dot You can see the picture when you paste the text into this website. http://sandbox.kidstrythisathome.com/erdos/ Excellent, thanks for the links! I guess this could be an option for better documentation about all the processing. I started to describe the chaining of rulesets last year and it was a nightmare, I stopped after just 4 levels. I think I can retry with this (especially for 1.6 which became a bit simpler in this regard). Well graphviz ( http://www.graphviz.org/ ) is very powerful even with that 'simple' syntax ;-) http://www.graphviz.org/Gallery.php Cheers Aleks
RE: cookie prefix strange behavior
Ok. For now we'll use cookie insert method. Thank you Roberto -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: domenica 26 luglio 2015 21.54 To: mlist Cc: 'Baptiste'; haproxy@formilux.org Subject: Re: cookie prefix strange behavior On Sat, Jul 25, 2015 at 07:05:42AM +, mlist wrote: Hi Willy, any new on the strange cookie behavior ? Not yet, and for now I'm away for the whole week so it will have to wait even more. Regards, Willy -- Il messaggio e' stato analizzato alla ricerca di virus o contenuti pericolosi da MailScanner, ed e' risultato non infetto.
RE: cookie prefix strange behavior
Hi Willy, any new on the strange cookie behavior ? Also I ask you for haproxy configuration problem cannot I found a solution searching hard on Internet... To be honest I can't believe this. Have you tried to read the documentation carefully and more then one time? http://lmgtfy.com/?q=haproxy+use+backend+acl+based http://www.haproxy.org/#docs https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#7 https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#7.2 I do not need link for acl... We just use working acl, we ask if you know, there is, a best practice, i.e. to avoid cycle and be more performance and flexibility on request management if we need redirection to ssl. If is best to have separate http frontend for http and https and different backend. But ok. We'll test different configurations. 1. We want to redirect all non HTTPS request to HTTPS except some request (ex: path_beg based) but we want to have exception like this: If Request = http://dom1/(A) - backend1 (http) If Request = https://dom1/(A) - SSL Termination - backend1 (http) If Request = http://dom1/(NOT A) - Redirect SSL - SSL Termination - backend1 (http) If Request = https://dom1/(NOT A) - SSL Termination - backend1 (http) http://serverfault.com/questions/589494/haproxy-use-acl-in-backend-shrink-my-config Maybe you get the idea for a possible solution way. We do not need to redirect to a single server, as stated we just uses acl to redirect to different backend. There are best practice for configuration to avoid redundancy in configuration file but having best performance, 2 front-end + 2 back-end, 2 front-end + 1 back-end, 1 front-end + 1 backend-end ? 2. In a configuration like that in point 1. (with SSL termination with exception) is a good solution to mix backend :80 with :443 servers without a risk for redirection loop ? There is as flow chart of haproxy request-response flow to can evaluate these configurations ? Maybe you could find some answer in the design-thoughts directory like. http://git.haproxy.org/?p=haproxy.git;a=blob;f=doc/design-thoughts/how-it-works.txt;h=2d1cb89a059e477469b2f980e970c22f4af6da66;hb=HEAD http://git.haproxy.org/?p=haproxy.git;a=tree;f=doc/design-thoughts;h=42c124145020b7c538d6011a7508c45e4e0780e1;hb=HEAD No useful info in this documents… As far as I know there is not yet a flow chart like http://redmine.lighttpd.net/projects/lighttpd/repository/entry/trunk/doc/state.dot You can see the picture when you paste the text into this website. http://sandbox.kidstrythisathome.com/erdos/ I mean a general flow of request-response management by haproxy including precedence i.e. when acl are evaluated and in which precedence) and sequence of step, so one can evaluate in advance if a possible configuration can be good. An architectural view of haproxy behavior. 3. In haproxy.log I see only client request, is there a configuration to see also backend server response or the only solution is to use debugging to see all traffic ? Also, I use on command line haproxy -d to debug, there is a method so we do not need to stop haproxy daemon to use command line to do debugging ? so we can debug changes on production haproxy ? https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#3.3 https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#log No info there. I just used –d option in command line. I cannot found info if such option can be used so such info can be written on the haproxy.log instead of stopping daemon and running haproxy on command line with –d switch. Also in the –d output not all the header information are present, so for some problem we have to divide analysis in different tool output (haproxy –d, tcpdump, ecc.), spreading info on different tools make analysis harder… But ok ! 4. what standard syntax haproxy uses for Regular Expression (perl, POSIX) ? http://git.haproxy.org/?p=haproxy.git;a=blob;f=Makefile;h=a1383a38c539fe4f0101a986746d8290ad3eebfa;hb=HEAD#l16 # USE_PCRE : enable use of libpcre for regex. Recommended. Ok .Thank you. 5. What about client certificates ? I think haproxy can SSL Terminate also client certificates verification, is so ? What about client certificates if backend server have to authenticate client using their certificate, can haproxy manage this situation passing client certificates to backend server ? http://lmgtfy.com/?q=haproxy+client+certificate https://raymii.org/s/tutorials/haproxy_client_side_ssl_certificates.html Ok. On this probably I could look better :D Thank you. Roberto -Original Message- From: Aleksandar Lazic [mailto:al-hapr...@none.at] Sent: domenica 26 luglio 2015 23.43 To: mlist Cc: 'Willy Tarreau'; 'Baptiste'; haproxy@formilux.org Subject: RE: cookie prefix strange behavior Hi Roberto. Am 25-07
Re: cookie prefix strange behavior
On Sat, Jul 25, 2015 at 07:05:42AM +, mlist wrote: Hi Willy, any new on the strange cookie behavior ? Not yet, and for now I'm away for the whole week so it will have to wait even more. Regards, Willy
RE: cookie prefix strange behavior
Hi Roberto. Am 25-07-2015 09:05, schrieb mlist: Hi Willy, any new on the strange cookie behavior ? Also I ask you for haproxy configuration problem cannot I found a solution searching hard on Internet... To be honest I can't believe this. Have you tried to read the documentation carefully and more then one time? http://lmgtfy.com/?q=haproxy+use+backend+acl+based http://www.haproxy.org/#docs https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#7 https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#7.2 1. We want to redirect all non HTTPS request to HTTPS except some request (ex: path_beg based) but we want to have exception like this: If Request = http://dom1/(A) - backend1 (http) If Request = https://dom1/(A) - SSL Termination - backend1 (http) If Request = http://dom1/(NOT A) - Redirect SSL - SSL Termination - backend1 (http) If Request = https://dom1/(NOT A) - SSL Termination - backend1 (http) http://serverfault.com/questions/589494/haproxy-use-acl-in-backend-shrink-my-config Maybe you get the idea for a possible solution way. There are best practice for configuration to avoid redundancy in configuration file but having best performance, 2 front-end + 2 back-end, 2 front-end + 1 back-end, 1 front-end + 1 backend-end ? 2. In a configuration like that in point 1. (with SSL termination with exception) is a good solution to mix backend :80 with :443 servers without a risk for redirection loop ? There is as flow chart of haproxy request-response flow to can evaluate these configurations ? Maybe you could find some answer in the design-thoughts directory like. http://git.haproxy.org/?p=haproxy.git;a=blob;f=doc/design-thoughts/how-it-works.txt;h=2d1cb89a059e477469b2f980e970c22f4af6da66;hb=HEAD http://git.haproxy.org/?p=haproxy.git;a=tree;f=doc/design-thoughts;h=42c124145020b7c538d6011a7508c45e4e0780e1;hb=HEAD As far as I know there is not yet a flow chart like http://redmine.lighttpd.net/projects/lighttpd/repository/entry/trunk/doc/state.dot You can see the picture when you paste the text into this website. http://sandbox.kidstrythisathome.com/erdos/ 3. In haproxy.log I see only client request, is there a configuration to see also backend server response or the only solution is to use debugging to see all traffic ? Also, I use on command line haproxy -d to debug, there is a method so we do not need to stop haproxy daemon to use command line to do debugging ? so we can debug changes on production haproxy ? https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#3.3 https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#log 4. what standard syntax haproxy uses for Regular Expression (perl, POSIX) ? http://git.haproxy.org/?p=haproxy.git;a=blob;f=Makefile;h=a1383a38c539fe4f0101a986746d8290ad3eebfa;hb=HEAD#l16 # USE_PCRE : enable use of libpcre for regex. Recommended. 5. What about client certificates ? I think haproxy can SSL Terminate also client certificates verification, is so ? What about client certificates if backend server have to authenticate client using their certificate, can haproxy manage this situation passing client certificates to backend server ? http://lmgtfy.com/?q=haproxy+client+certificate https://raymii.org/s/tutorials/haproxy_client_side_ssl_certificates.html Thank you in advance. Hth Best Regards Cheers Roberto Aleks -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: lunedì 20 luglio 2015 12.15 To: mlist Cc: 'Baptiste'; haproxy@formilux.org Subject: Re: cookie prefix strange behavior On Mon, Jul 20, 2015 at 09:51:28AM +, mlist wrote: Hi Willy Hi Baptiste, as you can see using prefix or sticky table we found this invalid cookie problem. - Why without haproxy in the middle we do not have this problem ? why a browser send an INVALID cookie ? Because it learned it another way, maybe before you installed haproxy, maybe on a direct connection or anything. I'm sure enough this is not the case. After we get this behavior, we cleaned all cookies. After some testing passing only by haproxy for these domain, we get the problem again, with Chrome and with IE. OK, interesting. - How we can match absence of prefix ? can be done directly by haproxy ? I think haproxy should fix it, yes, otherwise it can continue this way forever. That said, it *will* break existing sessions, but if haproxy applies load balancing, such session will be broken as well. What version is this, 1.6-dev or 1.5 ? What do you mean with: it *will* break existing sessions ? if we load balance web application with haproxy, session coming in must have a cookie inserted/prefixed by haproxy, I'm wrong ? if so any request with an Invalid cookie is INVALID :D so no session will be broken... What I mean is that if the cookie is invalid, haproxy cannot use the cookie to decide what server to send the request to, so it will pick one server in the farm which
Re: cookie prefix strange behavior
Hi Aleks, On Sun, Jul 26, 2015 at 11:42:42PM +0200, Aleksandar Lazic wrote: As far as I know there is not yet a flow chart like http://redmine.lighttpd.net/projects/lighttpd/repository/entry/trunk/doc/state.dot You can see the picture when you paste the text into this website. http://sandbox.kidstrythisathome.com/erdos/ Excellent, thanks for the links! I guess this could be an option for better documentation about all the processing. I started to describe the chaining of rulesets last year and it was a nightmare, I stopped after just 4 levels. I think I can retry with this (especially for 1.6 which became a bit simpler in this regard). Cheers, Willy
RE: cookie prefix strange behavior
Hi Willy, any new on the strange cookie behavior ? Also I ask you for haproxy configuration problem cannot I found a solution searching hard on Internet... 1. We want to redirect all non HTTPS request to HTTPS except some request (ex: path_beg based) but we want to have exception like this: If Request = http://dom1/(A) - backend1 (http) If Request = https://dom1/(A) - SSL Termination - backend1 (http) If Request = http://dom1/(NOT A) - Redirect SSL - SSL Termination - backend1 (http) If Request = https://dom1/(NOT A) - SSL Termination - backend1 (http) There are best practice for configuration to avoid redundancy in configuration file but having best performance, 2 front-end + 2 back-end, 2 front-end + 1 back-end, 1 front-end + 1 backend-end ? 2. In a configuration like that in point 1. (with SSL termination with exception) is a good solution to mix backend :80 with :443 servers without a risk for redirection loop ? There is as flow chart of haproxy request-response flow to can evaluate these configurations ? 3. In haproxy.log I see only client request, is there a configuration to see also backend server response or the only solution is to use debugging to see all traffic ? Also, I use on command line haproxy -d to debug, there is a method so we do not need to stop haproxy daemon to use command line to do debugging ? so we can debug changes on production haproxy ? 4. what standard syntax haproxy uses for Regular Expression (perl, POSIX) ? 5. What about client certificates ? I think haproxy can SSL Terminate also client certificates verification, is so ? What about client certificates if backend server have to authenticate client using their certificate, can haproxy manage this situation passing client certificates to backend server ? Thank you in advance. Best Regards Roberto -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: lunedì 20 luglio 2015 12.15 To: mlist Cc: 'Baptiste'; haproxy@formilux.org Subject: Re: cookie prefix strange behavior On Mon, Jul 20, 2015 at 09:51:28AM +, mlist wrote: Hi Willy Hi Baptiste, as you can see using prefix or sticky table we found this invalid cookie problem. - Why without haproxy in the middle we do not have this problem ? why a browser send an INVALID cookie ? Because it learned it another way, maybe before you installed haproxy, maybe on a direct connection or anything. I'm sure enough this is not the case. After we get this behavior, we cleaned all cookies. After some testing passing only by haproxy for these domain, we get the problem again, with Chrome and with IE. OK, interesting. - How we can match absence of prefix ? can be done directly by haproxy ? I think haproxy should fix it, yes, otherwise it can continue this way forever. That said, it *will* break existing sessions, but if haproxy applies load balancing, such session will be broken as well. What version is this, 1.6-dev or 1.5 ? What do you mean with: it *will* break existing sessions ? if we load balance web application with haproxy, session coming in must have a cookie inserted/prefixed by haproxy, I'm wrong ? if so any request with an Invalid cookie is INVALID :D so no session will be broken... What I mean is that if the cookie is invalid, haproxy cannot use the cookie to decide what server to send the request to, so it will pick one server in the farm which is not necessarily the right one (in fact it has a (N-1)/N chance of picking a wrong one in a farm of N servers). That's why I think that haproxy should fix this when this happens. Most likely the problem is that once a wrong cookie flows from the client to the server, the server will not emit this cookie anymore so no prefixing will occur. For this reason I think that we should remove the invalid cookies from the requests when running in prefix mode. In order to know exactly how the situation happened, you'll need to look through all the logs affecting the client which exhibited the problem. The cookie flags will indicate when the cookie was inserted/prefixed, present or valid/invalid etc... And maybe we'll find what produces this situation. It is also possible that the cookie is built by the application using javascript. We compiled from source HA-Proxy version 1.6-dev2-25f4e3e 2015/07/10 OK thanks. Do you know if 1.5 also produces the same problem ? It could be a regression, though I don't remember that we ever touched that area recently. Willy -- Il messaggio e' stato analizzato alla ricerca di virus o contenuti pericolosi da MailScanner, ed e' risultato non infetto.
RE: cookie prefix strange behavior
It can make repeatable the test case per this issue. I do not know what is the problem, so I tell you situation in which this issue arise: 1. I Set persistence method cookie prefix on haproxy for an application cookie (as ASP.NET_SessionId) 2. I opened two or three Chrome session with some web pages different from the target application site for which we get this issue 3. On one of these Chrome session, with ctrl-shift-N we opened an incognito Chrome entering the application URL 4. Login on the application page and navigate on some pages / funtions to see if persistence works well (no problem for this session) 5. I opened a new normal Chrome session entering the application URL . this session break out the issue Sometimes the session at (5.) reuses the cookie for session at (4.), so no issue, as all Crome session share the some session with the application server. Often the new session send the strange cookie without server request to Set-Cookie, generating the issue. I hope this can help you. Best regards Roberto From: mlist Sent: lunedì 20 luglio 2015 15.09 To: 'Willy Tarreau' Cc: 'Baptiste'; haproxy@formilux.org Subject: RE: cookie prefix strange behavior I tested with 1.5.14. I think this is not a regression. The error is the same. But using insert instead of prefix, also with 1.5.6 the strange cookie is managed in different manner strange cookie has a prefix, probably taken by old haproxy connection with prefix, this old cookie is marked ---NI instead of ---IN and haproxy now configured with insert instead of prefix, insert a cookie so all works well. I think this is a bug or something you can manage in haproxy. If not only cookie insert can be used to have a robust cookie / session management with haproxy. And more... Can be used sticky table as insert mode ? without specify an application cookie ? This is the BLOCK1 Jul 20 14:39:49 haproxy_machine haproxy[2846]: 172.16.3.122:7144 [20/Jul/2015:14:39:49.411] front_end~ back_end/SERVER1 132/0/1/89/222 302 541 ASP.NET_SessionId=SERVER1_https~3k5uin550flaxa55dl1twd45 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 --NI 1/1/0/1/0 0/0 GET /app1/ HTTP/1.1 Jul 20 14:39:49 haproxy_machine haproxy[2846]: 172.16.3.122:7144 [20/Jul/2015:14:39:49.633] front_end~ back_end/SERVER1 3/0/1/11/15 200 6444 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 1/1/0/1/0 0/0 GET /app1/login.aspx HTTP/1.1 Jul 20 14:39:49 haproxy_machine haproxy[2846]: 172.16.3.122:7144 [20/Jul/2015:14:39:49.649] front_end~ back_end/SERVER1 7/0/1/6/14 304 184 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 1/1/0/1/0 0/0 GET /app1/templatelibrary/menutop/templatenoheader.css HTTP/1.1 Jul 20 14:39:49 haproxy_machine haproxy[2846]: 172.16.3.122:7144 [20/Jul/2015:14:39:49.664] front_end~ back_end/SERVER1 1/0/1/9/11 304 184 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 3/3/1/2/0 0/0 GET /app1/templatelibrary/menutop/img/logoheader/logo.gif HTTP/1.1 Jul 20 14:39:49 haproxy_machine haproxy[2846]: 172.16.3.122:7146 [20/Jul/2015:14:39:49.664] front_end~ back_end/SERVER1 2/0/1/11/14 304 184 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 3/3/0/1/0 0/0 GET /app1/templatelibrary/menutop/common.css HTTP/1.1 Jul 20 14:39:49 haproxy_machine haproxy[2846]: 172.16.3.122:7144 [20/Jul/2015:14:39:49.675] front_end~ back_end/SERVER1 8/0/1/8/17 304 184 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 3/3/1/2/0 0/0 GET /app1/templatelibrary/menutop/aps/jquery-ui-1.10.3.custom.min.css HTTP/1.1 Jul 20 14:39:49 haproxy_machine haproxy[2846]: 172.16.3.122:7146 [20/Jul/2015:14:39:49.679] front_end~ back_end/SERVER1 3/0/1/14/18 304 184 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 3/3/0/1/0 0/0 GET /app1/templatelibrary/menutop/jquery.multiselect.css HTTP/1.1 Jul 20 14:39:56 haproxy_machine haproxy[2846]: 172.16.3.122:7146 [20/Jul/2015:14:39:49.697] front_end~ back_end/SERVER1 6495/0/3/132/6630 302 6481 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 3/3/0/1/0 0/0 POST /app1/login.aspx HTTP/1.1 Jul 20 14:39:56 haproxy_machine haproxy[2846]: 172.16.3.122:7146 [20/Jul/2015:14:39:56.327] front_end~ back_end/SERVER1 3/0/1/138/143 200 21668 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 3/3/0/0/0 0/0 GET /app1/ HTTP/1.1 Jul 20 14:39:56 haproxy_machine haproxy[2846]: 172.16.3.122:7159 [20/Jul/2015:14:39:56.486] front_end~ back_end/SERVER1 4/0/1/1/6 304 184 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 6/6/3/4/0 0/0 GET /app1/templatelibrary/menutop/img/module5.gif HTTP/1.1 Jul 20 14:39:56 haproxy_machine haproxy[2846]: 172.16.3.122:7144 [20/Jul/2015:14:39:49.692] front_end~ back_end/SERVER1 6796/0/1/9/6806 304 184 ASP.NET_SessionId=qbasnn45mfhxi155lyncbw45 - --VN 6/6/3/4/0 0/0 GET /app1/templatelibrary/menutop/img/homebackground/proclist.jpg HTTP/1.1 Jul 20 14:39:56 haproxy_machine haproxy[2846]: 172.16.3.122:7159 [20/Jul/2015:14:39:56.493] front_end~ back_end/SERVER1 1/0/1/2/4 304 184
Re: cookie prefix strange behavior
[please avoid top-posting, it makes it really painful to reply to e-mails with the relevant context] On Sat, Jul 18, 2015 at 09:56:23AM +, mlist wrote: Hi Baptiste, as you can see using prefix or sticky table we found this invalid cookie problem. - Why without haproxy in the middle we do not have this problem ? why a browser send an INVALID cookie ? Because it learned it another way, maybe before you installed haproxy, maybe on a direct connection or anything. - How we can match absence of prefix ? can be done directly by haproxy ? I think haproxy should fix it, yes, otherwise it can continue this way forever. That said, it *will* break existing sessions, but if haproxy applies load balancing, such session will be broken as well. What version is this, 1.6-dev or 1.5 ? Willy
RE: cookie prefix strange behavior
Hi Willy Hi Baptiste, as you can see using prefix or sticky table we found this invalid cookie problem. - Why without haproxy in the middle we do not have this problem ? why a browser send an INVALID cookie ? Because it learned it another way, maybe before you installed haproxy, maybe on a direct connection or anything. I'm sure enough this is not the case. After we get this behavior, we cleaned all cookies. After some testing passing only by haproxy for these domain, we get the problem again, with Chrome and with IE. - How we can match absence of prefix ? can be done directly by haproxy ? I think haproxy should fix it, yes, otherwise it can continue this way forever. That said, it *will* break existing sessions, but if haproxy applies load balancing, such session will be broken as well. What version is this, 1.6-dev or 1.5 ? What do you mean with: it *will* break existing sessions ? if we load balance web application with haproxy, session coming in must have a cookie inserted/prefixed by haproxy, I'm wrong ? if so any request with an Invalid cookie is INVALID :D so no session will be broken... We compiled from source HA-Proxy version 1.6-dev2-25f4e3e 2015/07/10 Roberto -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: lunedì 20 luglio 2015 10.19 To: mlist Cc: 'Baptiste'; haproxy@formilux.org Subject: Re: cookie prefix strange behavior [please avoid top-posting, it makes it really painful to reply to e-mails with the relevant context] On Sat, Jul 18, 2015 at 09:56:23AM +, mlist wrote: Hi Baptiste, as you can see using prefix or sticky table we found this invalid cookie problem. - Why without haproxy in the middle we do not have this problem ? why a browser send an INVALID cookie ? Because it learned it another way, maybe before you installed haproxy, maybe on a direct connection or anything. - How we can match absence of prefix ? can be done directly by haproxy ? I think haproxy should fix it, yes, otherwise it can continue this way forever. That said, it *will* break existing sessions, but if haproxy applies load balancing, such session will be broken as well. What version is this, 1.6-dev or 1.5 ? Willy -- Il messaggio e' stato analizzato alla ricerca di virus o contenuti pericolosi da MailScanner, ed e' risultato non infetto.
Re: cookie prefix strange behavior
On Mon, Jul 20, 2015 at 09:51:28AM +, mlist wrote: Hi Willy Hi Baptiste, as you can see using prefix or sticky table we found this invalid cookie problem. - Why without haproxy in the middle we do not have this problem ? why a browser send an INVALID cookie ? Because it learned it another way, maybe before you installed haproxy, maybe on a direct connection or anything. I'm sure enough this is not the case. After we get this behavior, we cleaned all cookies. After some testing passing only by haproxy for these domain, we get the problem again, with Chrome and with IE. OK, interesting. - How we can match absence of prefix ? can be done directly by haproxy ? I think haproxy should fix it, yes, otherwise it can continue this way forever. That said, it *will* break existing sessions, but if haproxy applies load balancing, such session will be broken as well. What version is this, 1.6-dev or 1.5 ? What do you mean with: it *will* break existing sessions ? if we load balance web application with haproxy, session coming in must have a cookie inserted/prefixed by haproxy, I'm wrong ? if so any request with an Invalid cookie is INVALID :D so no session will be broken... What I mean is that if the cookie is invalid, haproxy cannot use the cookie to decide what server to send the request to, so it will pick one server in the farm which is not necessarily the right one (in fact it has a (N-1)/N chance of picking a wrong one in a farm of N servers). That's why I think that haproxy should fix this when this happens. Most likely the problem is that once a wrong cookie flows from the client to the server, the server will not emit this cookie anymore so no prefixing will occur. For this reason I think that we should remove the invalid cookies from the requests when running in prefix mode. In order to know exactly how the situation happened, you'll need to look through all the logs affecting the client which exhibited the problem. The cookie flags will indicate when the cookie was inserted/prefixed, present or valid/invalid etc... And maybe we'll find what produces this situation. It is also possible that the cookie is built by the application using javascript. We compiled from source HA-Proxy version 1.6-dev2-25f4e3e 2015/07/10 OK thanks. Do you know if 1.5 also produces the same problem ? It could be a regression, though I don't remember that we ever touched that area recently. Willy
RE: cookie prefix strange behavior
14:40:39 haproxy_machine haproxy[2846]: 172.16.3.122:7241 [20/Jul/2015:14:40:39.327] front_end~ back_end/SERVER2 8/0/2/2/12 304 184 ASP.NET_SessionId=tahvauvuahzipfuq2fisio55 - --VN 6/6/0/1/0 0/0 GET /app1/templatelibrary/menutop/img/module3.gif HTTP/1.1 Roberto -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: lunedì 20 luglio 2015 12.15 To: mlist Cc: 'Baptiste'; haproxy@formilux.org Subject: Re: cookie prefix strange behavior On Mon, Jul 20, 2015 at 09:51:28AM +, mlist wrote: Hi Willy Hi Baptiste, as you can see using prefix or sticky table we found this invalid cookie problem. - Why without haproxy in the middle we do not have this problem ? why a browser send an INVALID cookie ? Because it learned it another way, maybe before you installed haproxy, maybe on a direct connection or anything. I'm sure enough this is not the case. After we get this behavior, we cleaned all cookies. After some testing passing only by haproxy for these domain, we get the problem again, with Chrome and with IE. OK, interesting. - How we can match absence of prefix ? can be done directly by haproxy ? I think haproxy should fix it, yes, otherwise it can continue this way forever. That said, it *will* break existing sessions, but if haproxy applies load balancing, such session will be broken as well. What version is this, 1.6-dev or 1.5 ? What do you mean with: it *will* break existing sessions ? if we load balance web application with haproxy, session coming in must have a cookie inserted/prefixed by haproxy, I'm wrong ? if so any request with an Invalid cookie is INVALID :D so no session will be broken... What I mean is that if the cookie is invalid, haproxy cannot use the cookie to decide what server to send the request to, so it will pick one server in the farm which is not necessarily the right one (in fact it has a (N-1)/N chance of picking a wrong one in a farm of N servers). That's why I think that haproxy should fix this when this happens. Most likely the problem is that once a wrong cookie flows from the client to the server, the server will not emit this cookie anymore so no prefixing will occur. For this reason I think that we should remove the invalid cookies from the requests when running in prefix mode. In order to know exactly how the situation happened, you'll need to look through all the logs affecting the client which exhibited the problem. The cookie flags will indicate when the cookie was inserted/prefixed, present or valid/invalid etc... And maybe we'll find what produces this situation. It is also possible that the cookie is built by the application using javascript. We compiled from source HA-Proxy version 1.6-dev2-25f4e3e 2015/07/10 OK thanks. Do you know if 1.5 also produces the same problem ? It could be a regression, though I don't remember that we ever touched that area recently. Willy -- Il messaggio e' stato analizzato alla ricerca di virus o contenuti pericolosi da MailScanner, ed e' risultato non infetto.
RE: cookie prefix strange behavior
Hi Baptiste, as you can see using prefix or sticky table we found this invalid
cookie problem.
- Why without haproxy in the middle we do not have this problem ? why a browser
send an INVALID cookie ?
- How we can match absence of prefix ? can be done directly by haproxy ?
Roberto
-Original Message-
From: Baptiste [mailto:bed...@gmail.com]
Sent: venerdì 17 luglio 2015 22.09
To: mlist
Cc: haproxy@formilux.org
Subject: Re: cookie prefix strange behavior
Hi Roberto,
Look in your log lines, block 2, HAProxy says --IN. 'IN' is for
cookie persistence.
'I' means the cookie sent by the client is invalid and 'N' means
HAProxy did not perform any action on persistence.
In such case, you could try to match that there are no prefix in your
cookie and do a redirect to a page which cleans up the cookie then
redirect the user to the login page.
Baptiste
On Fri, Jul 17, 2015 at 5:49 PM, mlist ml...@apsystems.it wrote:
We found this behavior does not appears if we manually clean cookie in the
browser. There is a configuration option to invalidate old cookie so client
does not reuse this strange cookie not recognized by server ?
Roberto
From: mlist
Sent: venerdì 17 luglio 2015 16.19
To: 'haproxy@formilux.org'
Subject: cookie prefix strange behavior
We have compiled and installed haproxy version 1.6 dev2.
If we use cookie insert all works, but if we use cookie prefix
(ASP.NET_SessionId) or sticky table in which one have to specify cookie to
be sticked, so using cookie name = ASP.NET_SessionId) we have a strange
behavior.
BLOCK1
As you can see below, we open a browser and make a request, cookie prefix
mechanism works well, we can login and use the application (all subsequent
requests go on the some server).
BLOCK2
But if we open a new browser instance (chrome in this case, but this happen
also if we open IE) the client uses a strange “ASP.NET_SessionId” cookie
without haproxy prefix with back-end server, server does not ask client to
set a new cookie.
As of login, clearly, backend server does not recognize the cookie sent by
client (haproxy do a plain roundrobin distribution, no cookie management was
done) and so backend server return an error.
Can this be a bug of haproxy or a bad configuration by us ?
BLOCK1
Jul 17 14:55:49 ha_server1 haproxy[5604]: client_ip:37322
[17/Jul/2015:14:55:49.414] front_end_https~ back_end_https/SERVER1
2/0/44/18/64 302 467 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/ HTTP/1.1
Jul 17 14:55:49 ha_server1 haproxy[5604]: client_ip:37324
[17/Jul/2015:14:55:49.484] front_end_https~ back_end_https/SERVER1
2/0/5/22/29 200 6449 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/login.aspx HTTP/1.1
Jul 17 14:55:52 ha_server1 haproxy[5604]: client_ip:37328
[17/Jul/2015:14:55:52.284] front_end_https~ back_end_https/SERVER1 2/0/1/3/7
404 1424 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l}
GET /app1/segreteria HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37344
[17/Jul/2015:14:55:59.452] front_end_https~ back_end_https/SERVER1 2/0/1/2/5
301 448 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l}
GET /app1/anagrafenet HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37345
[17/Jul/2015:14:55:59.461] front_end_https~ back_end_https/SERVER1
2/0/1/32/36 302 435 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/anagrafenet/ HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37346
[17/Jul/2015:14:55:59.501] front_end_https~ back_end_https/SERVER1
2/0/1/27/31 200 6625 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/login.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37359
[17/Jul/2015:14:56:06.515] front_end_https~ back_end_https/SERVER1
1/0/4/64/69 302 6712 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {|946|Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, l} POST /app1/login.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37361
[17/Jul/2015:14:56:06.588] front_end_https~ back_end_https/SERVER1
2/0/1/175/179 200 28897 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/anagrafenet/default.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37364
[17/Jul/2015:14:56:06.777
RE: cookie prefix strange behavior
We found this behavior does not appears if we manually clean cookie in the
browser. There is a configuration option to invalidate old cookie so client
does not reuse this strange cookie not recognized by server ?
Roberto
From: mlist
Sent: venerdì 17 luglio 2015 16.19
To: 'haproxy@formilux.org'
Subject: cookie prefix strange behavior
We have compiled and installed haproxy version 1.6 dev2.
If we use cookie insert all works, but if we use cookie prefix
(ASP.NET_SessionId) or sticky table in which one have to specify cookie to be
sticked, so using cookie name = ASP.NET_SessionId) we have a strange behavior.
BLOCK1
As you can see below, we open a browser and make a request, cookie prefix
mechanism works well, we can login and use the application (all subsequent
requests go on the some server).
BLOCK2
But if we open a new browser instance (chrome in this case, but this happen
also if we open IE) the client uses a strange ASP.NET_SessionId cookie
without haproxy prefix with back-end server, server does not ask client to set
a new cookie.
As of login, clearly, backend server does not recognize the cookie sent by
client (haproxy do a plain roundrobin distribution, no cookie management was
done) and so backend server return an error.
Can this be a bug of haproxy or a bad configuration by us ?
BLOCK1
Jul 17 14:55:49 ha_server1 haproxy[5604]: client_ip:37322
[17/Jul/2015:14:55:49.414] front_end_https~ back_end_https/SERVER1 2/0/44/18/64
302 467 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0 0/0
{||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/ HTTP/1.1
Jul 17 14:55:49 ha_server1 haproxy[5604]: client_ip:37324
[17/Jul/2015:14:55:49.484] front_end_https~ back_end_https/SERVER1 2/0/5/22/29
200 6449 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/login.aspx HTTP/1.1
Jul 17 14:55:52 ha_server1 haproxy[5604]: client_ip:37328
[17/Jul/2015:14:55:52.284] front_end_https~ back_end_https/SERVER1 2/0/1/3/7
404 1424 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/segreteria HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37344
[17/Jul/2015:14:55:59.452] front_end_https~ back_end_https/SERVER1 2/0/1/2/5
301 448 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0 0/0
{||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/anagrafenet HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37345
[17/Jul/2015:14:55:59.461] front_end_https~ back_end_https/SERVER1 2/0/1/32/36
302 435 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0 0/0
{||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/anagrafenet/ HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37346
[17/Jul/2015:14:55:59.501] front_end_https~ back_end_https/SERVER1 2/0/1/27/31
200 6625 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/login.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37359
[17/Jul/2015:14:56:06.515] front_end_https~ back_end_https/SERVER1 1/0/4/64/69
302 6712 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {|946|Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l}
POST /app1/login.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37361
[17/Jul/2015:14:56:06.588] front_end_https~ back_end_https/SERVER1
2/0/1/175/179 200 28897 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/anagrafenet/default.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37364
[17/Jul/2015:14:56:06.777] front_end_https~ back_end_https/SERVER1 5/0/2/11/18
200 2049 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 5/5/5/5/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/templatelibrary/menutop/sf-menu.css HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37363
[17/Jul/2015:14:56:06.776] front_end_https~ back_end_https/SERVER1 6/0/3/13/22
200 3580 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 4/4/4/4/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/menu/supersubs.js HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37367
[17/Jul/2015:14:56:06.779] front_end_https~ back_end_https/SERVER1 3/0/3/15/22
200 17829 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 4/4/3/3/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l} GET
/app1/script/dwfunctions.js HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37369
Re: cookie prefix strange behavior
Hi Roberto,
Look in your log lines, block 2, HAProxy says --IN. 'IN' is for
cookie persistence.
'I' means the cookie sent by the client is invalid and 'N' means
HAProxy did not perform any action on persistence.
In such case, you could try to match that there are no prefix in your
cookie and do a redirect to a page which cleans up the cookie then
redirect the user to the login page.
Baptiste
On Fri, Jul 17, 2015 at 5:49 PM, mlist ml...@apsystems.it wrote:
We found this behavior does not appears if we manually clean cookie in the
browser. There is a configuration option to invalidate old cookie so client
does not reuse this strange cookie not recognized by server ?
Roberto
From: mlist
Sent: venerdì 17 luglio 2015 16.19
To: 'haproxy@formilux.org'
Subject: cookie prefix strange behavior
We have compiled and installed haproxy version 1.6 dev2.
If we use cookie insert all works, but if we use cookie prefix
(ASP.NET_SessionId) or sticky table in which one have to specify cookie to
be sticked, so using cookie name = ASP.NET_SessionId) we have a strange
behavior.
BLOCK1
As you can see below, we open a browser and make a request, cookie prefix
mechanism works well, we can login and use the application (all subsequent
requests go on the some server).
BLOCK2
But if we open a new browser instance (chrome in this case, but this happen
also if we open IE) the client uses a strange “ASP.NET_SessionId” cookie
without haproxy prefix with back-end server, server does not ask client to
set a new cookie.
As of login, clearly, backend server does not recognize the cookie sent by
client (haproxy do a plain roundrobin distribution, no cookie management was
done) and so backend server return an error.
Can this be a bug of haproxy or a bad configuration by us ?
BLOCK1
Jul 17 14:55:49 ha_server1 haproxy[5604]: client_ip:37322
[17/Jul/2015:14:55:49.414] front_end_https~ back_end_https/SERVER1
2/0/44/18/64 302 467 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/ HTTP/1.1
Jul 17 14:55:49 ha_server1 haproxy[5604]: client_ip:37324
[17/Jul/2015:14:55:49.484] front_end_https~ back_end_https/SERVER1
2/0/5/22/29 200 6449 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/login.aspx HTTP/1.1
Jul 17 14:55:52 ha_server1 haproxy[5604]: client_ip:37328
[17/Jul/2015:14:55:52.284] front_end_https~ back_end_https/SERVER1 2/0/1/3/7
404 1424 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l}
GET /app1/segreteria HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37344
[17/Jul/2015:14:55:59.452] front_end_https~ back_end_https/SERVER1 2/0/1/2/5
301 448 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 - --VN 0/0/0/0/0
0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, l}
GET /app1/anagrafenet HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37345
[17/Jul/2015:14:55:59.461] front_end_https~ back_end_https/SERVER1
2/0/1/32/36 302 435 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/anagrafenet/ HTTP/1.1
Jul 17 14:55:59 ha_server1 haproxy[5604]: client_ip:37346
[17/Jul/2015:14:55:59.501] front_end_https~ back_end_https/SERVER1
2/0/1/27/31 200 6625 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/login.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37359
[17/Jul/2015:14:56:06.515] front_end_https~ back_end_https/SERVER1
1/0/4/64/69 302 6712 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {|946|Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, l} POST /app1/login.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37361
[17/Jul/2015:14:56:06.588] front_end_https~ back_end_https/SERVER1
2/0/1/175/179 200 28897 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 0/0/0/0/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/anagrafenet/default.aspx HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37364
[17/Jul/2015:14:56:06.777] front_end_https~ back_end_https/SERVER1
5/0/2/11/18 200 2049 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN 5/5/5/5/0 0/0 {||Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36
(KHTML, l} GET /app1/templatelibrary/menutop/sf-menu.css HTTP/1.1
Jul 17 14:56:06 ha_server1 haproxy[5604]: client_ip:37363
[17/Jul/2015:14:56:06.776] front_end_https~ back_end_https/SERVER1
6/0/3/13/22 200 3580 ASP.NET_SessionId=SERVER1~fi2b5smpq33tgwfy0qqmog45 -
--VN
