Re: Consistent hashing based on cookie - across multiple HAProxy boxes
For those interested, this turned out to be some wierdness in haproxy - one of the servers was only storing the first 24 (despite being configured for 32, and the other identical binary/config working fine). The way to debug was the following on both servers for a given cookie value: echo show table webservers | socat /var/lib/haproxy/stats - | fgrep 4start_of_session From my config, it was also important to store text not binary data in the table. -Alex On Sat, Mar 2, 2013 at 5:32 PM, Alex Davies a...@davz.net wrote: I was trying to troubleshoot this with a packet dump on the peer traffic. The raw tcpdump does not mean anything to me, and Wireshark is decoding it as Java RMI traffic which isnt much use, and it looks like a binary protocol. So I can report that 'peer' traffic is certainly working because when I restart one of the haproxy processes there are a bunch of different packets sent (Wireshark doesent even attempt to decode these). I was hoping to be able to grep for one of the offending session IDs to see if that gave me a hint. Does anybody have a working configuration for multi peer persistence based on a cookie that I could test, by chance? Thanks, -Alex -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently.
Re: Consistent hashing based on cookie - across multiple HAProxy boxes
I was trying to troubleshoot this with a packet dump on the peer traffic. The raw tcpdump does not mean anything to me, and Wireshark is decoding it as Java RMI traffic which isnt much use, and it looks like a binary protocol. So I can report that 'peer' traffic is certainly working because when I restart one of the haproxy processes there are a bunch of different packets sent (Wireshark doesent even attempt to decode these). I was hoping to be able to grep for one of the offending session IDs to see if that gave me a hint. Does anybody have a working configuration for multi peer persistence based on a cookie that I could test, by chance? Thanks, -Alex
Re: Consistent hashing based on cookie - across multiple HAProxy boxes
H Baptiste and others, I've not been able to get this to work. I have two different haproxy servers with identical config (relevant sections below). When I hit each one with the same PHPSESSID (domain set to .domain.tld) I get put on a (different) consistent server. I've setup cookie capturing in the log to show that PHPSESSID is good; I can telnet across from one peer to the other. I've started both process with -L peer_name_and_also_hostname, but it still wont work. I'm running 1.5-dev17. Relevant log - server 1 [root@frontend1 ~]# tail -f /var/log/haproxy.log | grep /info.php Feb 28 22:55:24 localhost haproxy[18069]: 213.146.182.195:47073[28/Feb/2013:22:55:11.237] main webservers/tc_ww6 13063/0/1/0/13064 200 318 PHPSESSID=d40c28be77d7c9e364ae9fd35f87f62b 11670/11670/17/1/0 0/0 GET /info.php HTTP/1.1 Relevant log - server 2 [root@frontend2 ~]# tail -f /var/log/haproxy.log | grep info.php Feb 28 22:55:37 localhost haproxy[30244]: 213.146.182.195:47120[28/Feb/2013:22:55:36.991] main webservers/tc_ww1 7/0/0/3/10 200 318 PHPSESSID=d40c28be77d7c9e364ae9fd35f87f62b 5/5/0/0/0 0/0 GET /info.php HTTP/1.1 Connection from 1 to 2: [root@frontend1 ~]# telnet 10.99.99.162 1099 Trying 10.99.99.162... Connected to 10.99.99.162. Escape character is '^]'. SOMETHING 501 Connection closed by foreign host. Relevant config: peers frontends peer frontend1 10.99.99.161:1099 peer frontend2 10.99.99.162:1099 frontend main bind :80 capture cookie PHPSESSID len 50 .. backend webservers balance roundrobin stick on cookie(PHPSESSID) stick-table type string size 32k peers frontends expire 24h stick store-response set-cookie(PHPSESSID) server ww1 10.99.99.201:80 weight 5 check observe layer4 inter 1 server ww2 10.99.99.202:80 weight 6 check observe layer4 inter 1 server ww3 10.99.99.203:80 weight 10 check observe layer4 inter 1 ... Thanks, Alex On Sun, Feb 10, 2013 at 9:47 PM, Baptiste bed...@gmail.com wrote: Hi, It's weird you don't need the store-response. Test it as best as you can to avoid any issues later. There is no way to monitor the peering, as far as I know. If a peer is unavailable, it means both haproxy can't reach each others. so you may have a bigger issue to fix before :) This is a multi-master way of working: each haproxy notifies its peers about the changes it made into the table. cheers On Sun, Feb 10, 2013 at 9:36 PM, Alex Davies a...@davz.net wrote: Hi Baptiste (apologies!) I did some testing with this, it actually seemed that stick store-response was not needed. I think this is working, at least it works in my test environment. Will test it for real shortly. One question I had - what is the best way to monitor this peering? I didnt see anything in the logs about the peers, nor anything in the haproxy status URL. Obviously if the peering were to silently break down, this would be bad for me! Thanks, -Alex On Fri, Feb 8, 2013 at 4:22 AM, Baptiste bed...@gmail.com wrote: ahah, you can call me Baptiste :) You miss a stick on cookie(PHPSESSID). Also consider using the same expire delay you have on your application server. And last but not least, add a peers section (and a peer directive on the stick-table definition) where you provide all your HAProxy server IPs in order to get the table of each HAProxy synchronized. then you're done. Baptiste On 2/8/13, Alex Davies a...@davz.net wrote: Hi Willy, Thanks for your suggestion. I'm guessing you mean something like this backend: backend x balance roundrobin stick-table type string size 32k peers other_haproxy_server expire 24h stick store-response set-cookie(PHPSESSID) If I understand you correctly, you are saying that this will only mean that sessions become persistant once PHPSESSID is set. So, to translate into practicality, as long as the login page creates the relevant cooke (and it does not subsequently change once logged in), this should work nicely. Thanks, -Alex On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote: Hi, the only way you could do what you want for now is using stick tables (and haproxy 1.5-dev17). You can learn the Set-Cookie from the server and match the Cookie in the table from the client. That way, all the request from a user will be sent to the same server, from the first to the last one. Today, haproxy is able to hash a HTTP header for load-balancing, so a configuration like: balance hdr(Cookie) could do the trick, but it means that ALL clients cookie to load-balance. And worste, since there is no phpsessionid cookie on the first request, there are chances that the first and the second requests won't be routed to the same server. I guess it would be possible soon to have a: balance
Re: Consistent hashing based on cookie - across multiple HAProxy boxes
Hi Baptiste (apologies!) I did some testing with this, it actually seemed that stick store-response was not needed. I think this is working, at least it works in my test environment. Will test it for real shortly. One question I had - what is the best way to monitor this peering? I didnt see anything in the logs about the peers, nor anything in the haproxy status URL. Obviously if the peering were to silently break down, this would be bad for me! Thanks, -Alex On Fri, Feb 8, 2013 at 4:22 AM, Baptiste bed...@gmail.com wrote: ahah, you can call me Baptiste :) You miss a stick on cookie(PHPSESSID). Also consider using the same expire delay you have on your application server. And last but not least, add a peers section (and a peer directive on the stick-table definition) where you provide all your HAProxy server IPs in order to get the table of each HAProxy synchronized. then you're done. Baptiste On 2/8/13, Alex Davies a...@davz.net wrote: Hi Willy, Thanks for your suggestion. I'm guessing you mean something like this backend: backend x balance roundrobin stick-table type string size 32k peers other_haproxy_server expire 24h stick store-response set-cookie(PHPSESSID) If I understand you correctly, you are saying that this will only mean that sessions become persistant once PHPSESSID is set. So, to translate into practicality, as long as the login page creates the relevant cooke (and it does not subsequently change once logged in), this should work nicely. Thanks, -Alex On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote: Hi, the only way you could do what you want for now is using stick tables (and haproxy 1.5-dev17). You can learn the Set-Cookie from the server and match the Cookie in the table from the client. That way, all the request from a user will be sent to the same server, from the first to the last one. Today, haproxy is able to hash a HTTP header for load-balancing, so a configuration like: balance hdr(Cookie) could do the trick, but it means that ALL clients cookie to load-balance. And worste, since there is no phpsessionid cookie on the first request, there are chances that the first and the second requests won't be routed to the same server. I guess it would be possible soon to have a: balance cook(PHPSessionID) but it won't fix the sticking issue between first and second request since the cookie is not present in the first request. So if you really want using the algorithm method, you must be able to share cookies between your backend servers, only lucky people will be able to get authenticated. Well, maybe there are some dirty tricks like managing a farm for cookie-less clients and configuring PHP to learn an unknown session on the fly. Baptiste On Sun, Feb 3, 2013 at 2:03 AM, Alex Davies a...@davz.net wrote: Hi All, What is the best way to configure haproxy to hash based on an application cookie (such as PHPSESSID), in a way that is consistent (meaning multiple haproxy servers will route to the same backend), ideally including the ability to weight backends (the configuration would clearly have to be the same on these different boxes). appsession obviously allows this for a single HAProxy server, but it seems from the documentation that it generates a server based on the hash at the start of each session, so if the same session hit a different but identically configured haproxy server it would end up with a Thanks, -Alex -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently.
Re: Consistent hashing based on cookie - across multiple HAProxy boxes
Hi, It's weird you don't need the store-response. Test it as best as you can to avoid any issues later. There is no way to monitor the peering, as far as I know. If a peer is unavailable, it means both haproxy can't reach each others. so you may have a bigger issue to fix before :) This is a multi-master way of working: each haproxy notifies its peers about the changes it made into the table. cheers On Sun, Feb 10, 2013 at 9:36 PM, Alex Davies a...@davz.net wrote: Hi Baptiste (apologies!) I did some testing with this, it actually seemed that stick store-response was not needed. I think this is working, at least it works in my test environment. Will test it for real shortly. One question I had - what is the best way to monitor this peering? I didnt see anything in the logs about the peers, nor anything in the haproxy status URL. Obviously if the peering were to silently break down, this would be bad for me! Thanks, -Alex On Fri, Feb 8, 2013 at 4:22 AM, Baptiste bed...@gmail.com wrote: ahah, you can call me Baptiste :) You miss a stick on cookie(PHPSESSID). Also consider using the same expire delay you have on your application server. And last but not least, add a peers section (and a peer directive on the stick-table definition) where you provide all your HAProxy server IPs in order to get the table of each HAProxy synchronized. then you're done. Baptiste On 2/8/13, Alex Davies a...@davz.net wrote: Hi Willy, Thanks for your suggestion. I'm guessing you mean something like this backend: backend x balance roundrobin stick-table type string size 32k peers other_haproxy_server expire 24h stick store-response set-cookie(PHPSESSID) If I understand you correctly, you are saying that this will only mean that sessions become persistant once PHPSESSID is set. So, to translate into practicality, as long as the login page creates the relevant cooke (and it does not subsequently change once logged in), this should work nicely. Thanks, -Alex On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote: Hi, the only way you could do what you want for now is using stick tables (and haproxy 1.5-dev17). You can learn the Set-Cookie from the server and match the Cookie in the table from the client. That way, all the request from a user will be sent to the same server, from the first to the last one. Today, haproxy is able to hash a HTTP header for load-balancing, so a configuration like: balance hdr(Cookie) could do the trick, but it means that ALL clients cookie to load-balance. And worste, since there is no phpsessionid cookie on the first request, there are chances that the first and the second requests won't be routed to the same server. I guess it would be possible soon to have a: balance cook(PHPSessionID) but it won't fix the sticking issue between first and second request since the cookie is not present in the first request. So if you really want using the algorithm method, you must be able to share cookies between your backend servers, only lucky people will be able to get authenticated. Well, maybe there are some dirty tricks like managing a farm for cookie-less clients and configuring PHP to learn an unknown session on the fly. Baptiste On Sun, Feb 3, 2013 at 2:03 AM, Alex Davies a...@davz.net wrote: Hi All, What is the best way to configure haproxy to hash based on an application cookie (such as PHPSESSID), in a way that is consistent (meaning multiple haproxy servers will route to the same backend), ideally including the ability to weight backends (the configuration would clearly have to be the same on these different boxes). appsession obviously allows this for a single HAProxy server, but it seems from the documentation that it generates a server based on the hash at the start of each session, so if the same session hit a different but identically configured haproxy server it would end up with a Thanks, -Alex -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently.
Re: Consistent hashing based on cookie - across multiple HAProxy boxes
Hi Willy, Thanks for your suggestion. I'm guessing you mean something like this backend: backend x balance roundrobin stick-table type string size 32k peers other_haproxy_server expire 24h stick store-response set-cookie(PHPSESSID) If I understand you correctly, you are saying that this will only mean that sessions become persistant once PHPSESSID is set. So, to translate into practicality, as long as the login page creates the relevant cooke (and it does not subsequently change once logged in), this should work nicely. Thanks, -Alex On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote: Hi, the only way you could do what you want for now is using stick tables (and haproxy 1.5-dev17). You can learn the Set-Cookie from the server and match the Cookie in the table from the client. That way, all the request from a user will be sent to the same server, from the first to the last one. Today, haproxy is able to hash a HTTP header for load-balancing, so a configuration like: balance hdr(Cookie) could do the trick, but it means that ALL clients cookie to load-balance. And worste, since there is no phpsessionid cookie on the first request, there are chances that the first and the second requests won't be routed to the same server. I guess it would be possible soon to have a: balance cook(PHPSessionID) but it won't fix the sticking issue between first and second request since the cookie is not present in the first request. So if you really want using the algorithm method, you must be able to share cookies between your backend servers, only lucky people will be able to get authenticated. Well, maybe there are some dirty tricks like managing a farm for cookie-less clients and configuring PHP to learn an unknown session on the fly. Baptiste On Sun, Feb 3, 2013 at 2:03 AM, Alex Davies a...@davz.net wrote: Hi All, What is the best way to configure haproxy to hash based on an application cookie (such as PHPSESSID), in a way that is consistent (meaning multiple haproxy servers will route to the same backend), ideally including the ability to weight backends (the configuration would clearly have to be the same on these different boxes). appsession obviously allows this for a single HAProxy server, but it seems from the documentation that it generates a server based on the hash at the start of each session, so if the same session hit a different but identically configured haproxy server it would end up with a Thanks, -Alex
Re: Consistent hashing based on cookie - across multiple HAProxy boxes
ahah, you can call me Baptiste :) You miss a stick on cookie(PHPSESSID). Also consider using the same expire delay you have on your application server. And last but not least, add a peers section (and a peer directive on the stick-table definition) where you provide all your HAProxy server IPs in order to get the table of each HAProxy synchronized. then you're done. Baptiste On 2/8/13, Alex Davies a...@davz.net wrote: Hi Willy, Thanks for your suggestion. I'm guessing you mean something like this backend: backend x balance roundrobin stick-table type string size 32k peers other_haproxy_server expire 24h stick store-response set-cookie(PHPSESSID) If I understand you correctly, you are saying that this will only mean that sessions become persistant once PHPSESSID is set. So, to translate into practicality, as long as the login page creates the relevant cooke (and it does not subsequently change once logged in), this should work nicely. Thanks, -Alex On Sun, Feb 3, 2013 at 7:59 AM, Baptiste bed...@gmail.com wrote: Hi, the only way you could do what you want for now is using stick tables (and haproxy 1.5-dev17). You can learn the Set-Cookie from the server and match the Cookie in the table from the client. That way, all the request from a user will be sent to the same server, from the first to the last one. Today, haproxy is able to hash a HTTP header for load-balancing, so a configuration like: balance hdr(Cookie) could do the trick, but it means that ALL clients cookie to load-balance. And worste, since there is no phpsessionid cookie on the first request, there are chances that the first and the second requests won't be routed to the same server. I guess it would be possible soon to have a: balance cook(PHPSessionID) but it won't fix the sticking issue between first and second request since the cookie is not present in the first request. So if you really want using the algorithm method, you must be able to share cookies between your backend servers, only lucky people will be able to get authenticated. Well, maybe there are some dirty tricks like managing a farm for cookie-less clients and configuring PHP to learn an unknown session on the fly. Baptiste On Sun, Feb 3, 2013 at 2:03 AM, Alex Davies a...@davz.net wrote: Hi All, What is the best way to configure haproxy to hash based on an application cookie (such as PHPSESSID), in a way that is consistent (meaning multiple haproxy servers will route to the same backend), ideally including the ability to weight backends (the configuration would clearly have to be the same on these different boxes). appsession obviously allows this for a single HAProxy server, but it seems from the documentation that it generates a server based on the hash at the start of each session, so if the same session hit a different but identically configured haproxy server it would end up with a Thanks, -Alex
Re: Consistent hashing based on cookie - across multiple HAProxy boxes
Hi, the only way you could do what you want for now is using stick tables (and haproxy 1.5-dev17). You can learn the Set-Cookie from the server and match the Cookie in the table from the client. That way, all the request from a user will be sent to the same server, from the first to the last one. Today, haproxy is able to hash a HTTP header for load-balancing, so a configuration like: balance hdr(Cookie) could do the trick, but it means that ALL clients cookie to load-balance. And worste, since there is no phpsessionid cookie on the first request, there are chances that the first and the second requests won't be routed to the same server. I guess it would be possible soon to have a: balance cook(PHPSessionID) but it won't fix the sticking issue between first and second request since the cookie is not present in the first request. So if you really want using the algorithm method, you must be able to share cookies between your backend servers, only lucky people will be able to get authenticated. Well, maybe there are some dirty tricks like managing a farm for cookie-less clients and configuring PHP to learn an unknown session on the fly. Baptiste On Sun, Feb 3, 2013 at 2:03 AM, Alex Davies a...@davz.net wrote: Hi All, What is the best way to configure haproxy to hash based on an application cookie (such as PHPSESSID), in a way that is consistent (meaning multiple haproxy servers will route to the same backend), ideally including the ability to weight backends (the configuration would clearly have to be the same on these different boxes). appsession obviously allows this for a single HAProxy server, but it seems from the documentation that it generates a server based on the hash at the start of each session, so if the same session hit a different but identically configured haproxy server it would end up with a Thanks, -Alex
Consistent hashing based on cookie - across multiple HAProxy boxes
Hi All, What is the best way to configure haproxy to hash based on an application cookie (such as PHPSESSID), in a way that is consistent (meaning multiple haproxy servers will route to the same backend), ideally including the ability to weight backends (the configuration would clearly have to be the same on these different boxes). appsession obviously allows this for a single HAProxy server, but it seems from the documentation that it generates a server based on the hash at the start of each session, so if the same session hit a different but identically configured haproxy server it would end up with a Thanks, -Alex
