I see now! One last question since you are using Pacemaker. Do you recommend it for splitbrain so that I look into that direction? I mean when you say that pacemaker restart HAProxy, does it detect network failures as well? Or only SW crashes? I assume pacemaker will be aware of both HAProxy1 and HAProxy2 in my described deployment
________________________________ From: David Coulson <da...@davidcoulson.net> To: Hermes Flying <flyingher...@yahoo.com> Cc: Baptiste <bed...@gmail.com>; "haproxy@formilux.org" <haproxy@formilux.org> Sent: Thursday, November 29, 2012 9:29 PM Subject: Re: HAproxy and detect split-brain (network failures) Both haproxy instances have the same config, with the tomcat instances with the same weight, etc. Run something like keepalived or pacemaker to manage a VIP between the two boxes. That's it. Not sure about keepalived, but pacemaker can make sure haproxy is running, then either restart it or move the VIP if it is not running. David On 11/29/12 2:27 PM, Hermes Flying wrote: Something like the following: > > HAProxy1 --------> Tomcat1 > | +++++++++/\ > | + > |--------+--->Tomcat2 > + /+\ > + + >HAProxy2+++++++ > >HAProxy1 is in the same machine as Tomcat1 >HAproxy2 is in the same machine as Tomcat2 >HAProxy1 distributes the load among Tomcat1 and Tomcat2. >I erroneously thought that HAProxy2 would take over when HAProxy1 crashed to >distribute the load among Tomcat1/Tomcat2. >So if both are independent what can I do? > > > >From: David Coulson mailto:da...@davidcoulson.net >To: Hermes Flying mailto:flyingher...@yahoo.com >Cc: Baptiste mailto:bed...@gmail.com; mailto:haproxy@formilux.org >mailto:haproxy@formilux.org >Sent: Thursday, November 29, 2012 9:12 PM >Subject: Re: HAproxy and detect split-brain (network failures) > > >Again, you are mixing everything up. > >HAProxy has it's own configuration - It defines what nodes your port 80 traffic (or whatever) is routed to. Haproxy does periodic health checks of these backend services to make sure they are available for requests. If you have multiple haproxy instances they will all independently do health checks and not share any of that information with each other. HAProxy will route traffic to all systems defined as a backend for a particular service based upon whatever criteria is in the haproxy config. > >You can run a two-node environment that is active/backup from a VIP perspective, but active/active from a haproxy service perspective - Each node would run Apache (or whatever your service is) and haproxy would distribute requests across both based on your haproxy config. But, at any point in time only one node would actually be routing requests through it's local instance of haproxy. > >I can't make it any simpler than that. Draw a diagram of what you are trying to do if it doesn't make sense. > > > >On 11/29/12 2:06 PM, Hermes Flying wrote: > >You are saying that one instance of HAProxy runs in each system and one >instance is assigned the VIP that clients hit-on (out of scope for HAProxy). >>But this HAProxy distributes the requests according to the load, either on >>system-A or system-B for which you seem to refer to as backup system. In what >>way are you now refering to it as backup system? Because I am interested in >>distributing the load to all the nodes. >> >> >> >>From: David Coulson mailto:da...@davidcoulson.net >>To: Hermes Flying mailto:flyingher...@yahoo.com >>Cc: Baptiste mailto:bed...@gmail.com; mailto:haproxy@formilux.org >>mailto:haproxy@formilux.org >>Sent: Thursday, November 29, 2012 8:57 PM >>Subject: Re: HAproxy and detect split-brain (network failures) >> >> >>You can do that, but haproxy doesn't have anything to do with the failover >>process, other than you run an instance of haproxy on one server, and another >>instance on your backup system. As I said, neither of the haproxy instances >>communicate anything, so all you need to do is move the IP clients are using >>from one server to the other in order to handle a failure. Moving the IP >>around is something keepalived, pacemaker, etc handles - Look at their >>documentation for specifics and challenges in a two-node config. >> >>HAProxy doesn't have a concent of primary and backup in terms of it's own instances. Each of them is stand alone. It's up to you, based on your network/IP config which one has traffic routed to it. >> >>David >> >> >> >>On 11/29/12 1:53 PM, Hermes Flying wrote: >> >>But if I install 2 HAProxy as load balancers, doesn't one act as the primary >>loadbalancer directing the load to the known servers while the secondary >>takes over load distribution as soon as the heartbeat fails? I remember >>reading this. Is this wrong? >>> >>> >>>From: David Coulson mailto:da...@davidcoulson.net >>>To: Hermes Flying mailto:flyingher...@yahoo.com >>>Cc: Baptiste mailto:bed...@gmail.com; mailto:haproxy@formilux.org >>>mailto:haproxy@formilux.org >>>Sent: Thursday, November 29, 2012 8:39 PM >>>Subject: Re: HAproxy and detect split-brain (network failures) >>> >>> >>>You are mixing two totally different things together. >>> >>>1) HAProxy will do periodic health checks of backend systems you are routing to. Depending if you configure something as 'backup' or 'not backup' will determine if/how traffic is routed to it. The backend systems do not 'take over'. Haproxy just routes traffic to systems based on your configuration. The backend systems don't know/care about the other backend nodes, unless your application requires it which is a different story and nothing to do with haproxy. HAproxy only cares about a single instance of itself - If you have more than one haproxy instance, they do NOT communicate anything between each other. >>> >>>2) In terms of keepalived, pacemaker, etc, it makes no difference which you use with haproxy - all they do is manage the IP address(es) which haproxy is listening on, and perhaps restart haproxy if it dies. Their configuration and how you maintain quorum in a two-node configuration is a question for one of their mailing lists, or just read their documentation. I personally use pacemaker. >>> >>> >>>On 11/29/12 1:35 PM, Hermes Flying wrote: >>> >>>Well I don't follow: >>>>"You can have a pool of primary that it routes across, then backup systems >>>>that are only used when all primary systems are unavailable." >>>>When you are saying that "the backup systems that are used when primary >>>>systems are unavailable", how do they decide to take over? How do they know >>>>that the other systems are unavailable? >>>>Are you saying that they depend on third party components like the ones you >>>>mentioned (Keepalived etc)? In this case, what is the most suitable tool to >>>>be used along with HAProxy? Is there a reference manual for this somewhere? >>>> >>>> >>>> >>>>From: David Coulson mailto:da...@davidcoulson.net >>>>To: Hermes Flying mailto:flyingher...@yahoo.com >>>>Cc: Baptiste mailto:bed...@gmail.com; mailto:haproxy@formilux.org >>>>mailto:haproxy@formilux.org >>>>Sent: Thursday, November 29, 2012 8:21 PM >>>>Subject: Re: HAproxy and detect split-brain (network failures) >>>> >>>> >>>>HAProxy only does primary and backup in terms of active backend systems - >>>>You can have a pool of primary that it routes across, then backup systems >>>>that are only used when all primary systems are unavailable. >>>> >>>>There is no concept of a cluster in terms of haproxy instances, although you can run more than one and manage them via something like pacemaker, keepalived or rgmanager. >>>> >>>> >>>>On 11/29/12 1:19 PM, Hermes Flying wrote: >>>> >>>>Hi, >>>>>From a quick look into HAProxy, I see that it is a Primary/backup >>>>>architecture. So isn't ensuring that both "nodes" don't become primary >>>>>part of HAProxy's primary/backup "protocol" ? >>>>> >>>>> >>>>>From: Baptiste mailto:bed...@gmail.com >>>>>To: Hermes Flying mailto:flyingher...@yahoo.com >>>>>Cc: mailto:haproxy@formilux.org mailto:haproxy@formilux.org >>>>>Sent: Thursday, November 29, 2012 3:02 PM >>>>>Subject: Re: HAproxy and detect split-brain (network failures) >>>>> >>>>>Hi, >>>>> >>>>>This is not HAProxy's role, this is the tool you use to ensure high >>>>>availability to do that. >>>>> >>>>>I could see a way where HAProxy can report one interface failing, >>>>>maybe this could help you to detect if you're in a split brain >>>>>situation. >>>>> >>>>>cheers >>>>> >>>>> >>>>> >>>>>On Thu, Nov 29, 2012 at 11:51 AM, Hermes Flying <flyingher...@yahoo.com> wrote: >>>>>> Hi, >>>>>> I am looking into using HAProxy as our load balancer. >>>>>> I see that you are using a primary/backup approach. I was wondering how does >>>>>> HAProxy (if it does) address split-brain situation? Do you have a mechanism >>>>>> to detect and avoid it? Do you have some standard recommendation to all >>>>>> those using your solution? >>>>>> >>>>>> Thanks >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > > >
