Re: [Openstack] Howto Nova setup with HA?
Hi everyone, apologies for the cross-post; I believe this might be interesting to people on both the openstack and the pacemaker lists. Please see below. On Tue, Feb 14, 2012 at 9:07 AM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Stackers, It seems running Openstack components in High Availability hasn't been really a focus point lately, am I right? The general docs don't really mention HA except for nova-network. So I did some resource on how to run Nova in a High Availability and have some questions about it: The docs guides you on how to setup one cloud controller (running MySQL, nova-api, RabbitMQ etc.) and 2+n nodes for nova compute/network. But it does not mention how to make the cloud controller redundant. If the cloud controller brakes we have a serious problem! So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! Best regards, Tristan van Bokkem Datacenter Operations I've taken the liberty to put together a bit of a summary of the discussion we've had here,[1] roll it into a design summit brainstorm proposal, and also post it on my blog, here: http://www.hastexo.com/blogs/florian/2012/03/21/high-availability-openstack I hope it's not a violation of list etiquette to say that instead of cross-posting all replies to both lists, everyone's welcome to make comments on that blog post, too (use your Launchpad OpenID). Please feel free to flame me to a crisp or call me an idiot; as some of you are aware I'm quite firmly an HA guy getting into OpenStack, rather than the other way around. Even if the design summit proposal doesn't make it through, perhaps a few interested people (Monty? Jay? Adam? Major?) would like to sit down over beverages to discuss this in person. All comments and feedback much appreciated. Thanks! Cheers, Florian [1] Pacemaker subscribers, for context the full thread is at http://www.mail-archive.com/openstack@lists.launchpad.net/msg07495.html ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
On 02/28/2012 04:01 AM, i3D.net - Tristan van Bokkem wrote: Hi Jay, I am using a Gelera master/master setup with Pacemaker providing an floating IP. Well in that case, please disregard my master-master comment! :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Alright, picture is getting more clear. Thanks all! Got some extra questions tho: Font-end API servers -- what does this do? * load balanced with h/w load balancer * use s/w LB for smaller deployments * run nova-scheduler on each MySQL DB * multi-master configuration with pacemaker providing an floating IP. - seems to work. RabbitMQ service * I think there are some corner cases where messages could be dropped during failover and I believe later RabbitMQ versions support full multi-master but require some client side changes - is there any plans to support this? -- I would like to know this as well! nova-volume service * I think this can be solved by 'building' a proper cloud app which does not fail if one node fails where it is living on as it should be living on multiple nodes/sites. Glance * Run on multiple servers -- how do they get synced? Compute servers * Each run their own copy of nova-api (only instances running on the node use this) * nova-network (multi-host configuration) with private network -- Needs its own discussion I guess. Keystone * How about keystone, same as glance I guess? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Dan Wendlandt [mailto:d...@nicira.com] To: Ghe Rivero [mailto:ghe.riv...@stackops.com] Cc: netst...@lists.launchpad.net, darren.birk...@gmail.com, openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 18:13:55 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? 2012/2/21 Ghe Rivero ghe.riv...@stackops.com BigSwitch and Nicira have created different blueprints in quantum to add their plugins for essex-4 about ten days ago, Cisco already have it included almost from the beginning. No news of them for the moment, but i really hope that both of them make it in time! Ghe Rivero https://blueprints.launchpad.net/quantum/+spec/quantum-nvp-plugin https://blueprints.launchpad.net/quantum/+spec/quantum-bigswitch-plugin To be clear, there are several more existing and planned plugins for Quantum, even beyond the three you mention. Each plugin will make its own trade-offs in terms of HA for control plane and data plane components. I'm planning on organizing a talk at the Folsom summit/conference where people have a chance to talk about their plugins. No need for a flame war here :) One other point of confusion that I'd like to address is that there is NO requirement that a plugin be merged into the main Quantum repo for it to be a valid plugin. I know of several plugins that are used despite not being merged in (in fact, at the summit, we're going to discuss if ANY plugins should remain in the main repo... this is likely a larger decision that should be made consistently across all openstack projects). Dan 2012/2/21 Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com This is the kind of flame war I love... ;-) if you think the HA challenges in Openstack Nova are at MySQL and Rabbit level, you are not putting the focus on the right components. The biggest HA (and scalability) challenges in Nova are in the networking side. There are a lot of companies tackling the networking challenges of Nova (BigSwitch, Nicira, Cisco, etc...), and I would like to see how they evolve in the future and how they will address this problem. Right now, only promises. Fight! -- Diego Parrilla CEO www.stackops.com | diego.parri...@stackops.com | +34 649 94 43 29 | skype:diegoparrilla On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept
Re: [Openstack] Howto Nova setup with HA?
with a simple rsync script that runs on a periodic interval that syncs the image data. Compute servers * Each run their own copy of nova-api (only instances running on the node use this) No. Compute nodes run nova-compute, not nova-api. I think the intent here was to have access to the metadata service on each compute node - hence running both n-compute and n-API (this also avoids issues in getting metadata requests between nodes) * nova-network (multi-host configuration) with private network -- Needs its own discussion I guess. Keystone * How about keystone, same as glance I guess? Similar to glance-registry. Cheers, -jay Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net E-mail Support: i...@i3d.net mailto:i...@i3d.net E-mail NOC: n...@i3d.net mailto:n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. *From:* Dan Wendlandt [mailto:d...@nicira.com] *To:* Ghe Rivero [mailto:ghe.riv...@stackops.com] *Cc:* netst...@lists.launchpad.net, darren.birk...@gmail.com, openstack@lists.launchpad.net *Sent:* Tue, 21 Feb 2012 18:13:55 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? 2012/2/21 Ghe Rivero ghe.riv...@stackops.com mailto:ghe.riv...@stackops.com BigSwitch and Nicira have created different blueprints in quantum to add their plugins for essex-4 about ten days ago, Cisco already have it included almost from the beginning. No news of them for the moment, but i really hope that both of them make it in time! Ghe Rivero https://blueprints.launchpad.net/quantum/+spec/quantum-nvp-plugin https://blueprints.launchpad.net/quantum/+spec/quantum-bigswitch-plugin To be clear, there are several more existing and planned plugins for Quantum, even beyond the three you mention. Each plugin will make its own trade-offs in terms of HA for control plane and data plane components. I'm planning on organizing a talk at the Folsom summit/conference where people have a chance to talk about their plugins. No need for a flame war here :) One other point of confusion that I'd like to address is that there is NO requirement that a plugin be merged into the main Quantum repo for it to be a valid plugin. I know of several plugins that are used despite not being merged in (in fact, at the summit, we're going to discuss if ANY plugins should remain in the main repo... this is likely a larger decision that should be made consistently across all openstack projects). Dan 2012/2/21 Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com mailto:diego.parrilla.santama...@gmail.com This is the kind of flame war I love... ;-) if you think the HA challenges in Openstack Nova are at MySQL and Rabbit level, you are not putting the focus on the right components. The biggest HA (and scalability) challenges in Nova are in the networking side. There are a lot of companies tackling the networking challenges of Nova (BigSwitch, Nicira, Cisco, etc...), and I would like to see how they evolve in the future and how they will address this problem. Right now, only promises. Fight! -- Diego Parrilla http://www.stackops.com/*CEO* **www.stackops.com* http://www.stackops.com/ | *diego.parri...@stackops.com mailto:diego.parri...@stackops.com| +34 649 94 43 29 tel:%2B34%20649%2094%2043%2029 | skype:diegoparrilla* * http://www.stackops.com/ ** On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl mailto:tristanvanbok...@i3d.nl wrote: __ Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP
Re: [Openstack] Howto Nova setup with HA?
Ho Florian, I respect your opinion of course and I am not here to start a fight. Point is, how I understand it Nova really isn't very flexible on High Availability terms. I have now a master/master mysql setup with a Virtual IP working just to work around the shortcomings Nova imo introduces. This same story goes for RabbitMQ, I need to setup RabbitMQ in HA (which is fine) but I need to setup a Virtual IP for it as well just so Nova can correctly connect to it. Without the VIP the hole point of setting it up in a cluster is useless as you still will point nova to one single host. It shouldn't be that hard to make Nova listen to multiple mysql addresses and rabbitmq hosts in a cluster? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Florian Haas [mailto:flor...@hastexo.com] To: openstack@lists.launchpad.net Sent: Fri, 17 Feb 2012 13:32:51 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? On Thu, Feb 16, 2012 at 3:28 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Tom, Thank you very much for sharing your knowledge, this will help not only me! As I am still working on the MySQL part, you mentioned multi-master which I have running indeed, but I suppose there still will need to be some kind of entering point within the cluster. How did you managed this? Virtual cluster IP addresses and highly available MySQL (either DRBD or MySQL replication based). Pacemaker is very well capable of managing this. Of course, since you also quoted http://openlife.cc/blogs/2011/july/ultimate-mysql-high-availability-solution parts of which I wholeheartedly disagree with (as with all of Henrik's Pacemaker bashing, but he's of course entitled to his own opinion), then you might run away screaming from anything that barely mentions Pacemaker in passing. But in reality[1], Pacemaker is an extremely capable HA stack that would be well suited for this. Cheers, Florian [1] To be polite, I should say in my humble opinion, but my opinion on this really isn't humble at all, so I prefer being honest over being polite. :) -- Need help with High Availability? http://www.hastexo.com/now ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept multiple locations and failover if the first location fails. Or even round-robin to distribute the load. All I am doing now is yet add another component that can possibly fail as well. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Darren Birkett [mailto:darren.birk...@gmail.com] To: i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] Cc: Florian Haas [mailto:flor...@hastexo.com], openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 12:51:11 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? Surely it depends what you want out of it? If just pure high availability (ie ability to continue functioning when primary node goes down), then doesn't a master/master with floating IP fit the bill? It sounds like what you're after is better performance/throughput rather than HA - the ability to write to multiple masters at the same time. Even here if nova *was* able to hit multiple databases, a master/master setup isn't going to suit, as each write to one master still needs to be replicated to the other so throughput is going to be the same. You'd then need to look to something like a native mysql cluster to get actual increased throughput. Darren On 21 February 2012 11:34, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Ho Florian, I respect your opinion of course and I am not here to start a fight. Point is, how I understand it Nova really isn't very flexible on High Availability terms. I have now a master/master mysql setup with a Virtual IP working just to work around the shortcomings Nova imo introduces. This same story goes for RabbitMQ, I need to setup RabbitMQ in HA (which is fine) but I need to setup a Virtual IP for it as well just so Nova can correctly connect to it. Without the VIP the hole point of setting it up in a cluster is useless as you still will point nova to one single host. It shouldn't be that hard to make Nova listen to multiple mysql addresses and rabbitmq hosts in a cluster? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. From: Florian Haas [mailto:flor...@hastexo.com] To: openstack@lists.launchpad.net Sent: Fri, 17 Feb 2012 13:32:51 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? On Thu, Feb 16, 2012 at 3:28 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Tom, Thank you very much for sharing your knowledge, this will help not only me! As I am still working on the MySQL part, you mentioned multi-master which I have running indeed, but I suppose there still will need to be some kind of entering point within the cluster. How did you managed this? Virtual cluster IP addresses and highly available MySQL (either DRBD or MySQL replication based). Pacemaker is very well capable of managing
Re: [Openstack] Howto Nova setup with HA?
On Tue, Feb 21, 2012 at 12:34 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Ho Florian, I respect your opinion of course and I am not here to start a fight. Point is, how I understand it Nova really isn't very flexible on High Availability terms. I have now a master/master mysql setup with a Virtual IP working just to work around the shortcomings Nova imo introduces. This same story goes for RabbitMQ, I need to setup RabbitMQ in HA (which is fine) but I need to setup a Virtual IP for it as well just so Nova can correctly connect to it. Without the VIP the hole point of setting it up in a cluster is useless as you still will point nova to one single host. Well yes, that's a given. Thankfully you have a set of tools at hand that make this very easy. It shouldn't be that hard to make Nova listen to multiple mysql addresses and rabbitmq hosts in a cluster? And then what? So it talks to multiple MySQLs, and you'll have to take care of database consistency by yourself. Or it talks to multiple RabbitMQs, and you'll need a way to synchronize your queues. Question is, as Darren has also said in this thread, whether you really want HA, or scaleout. Or both. But interestingly, Pacemaker can serve you just fine for all of that. Let me repeat, though, I am clearly biased -- see URL in my sig block for details. Cheers, Florian -- Need help with High Availability? http://www.hastexo.com ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
On Tue, Feb 21, 2012 at 12:51 PM, Darren Birkett darren.birk...@gmail.com wrote: Surely it depends what you want out of it? If just pure high availability (ie ability to continue functioning when primary node goes down), then doesn't a master/master with floating IP fit the bill? Master/master with a floating IP, MySQL-on-DRBD with a floating IP, MySQL-on-SAN with a floating IP, even MySQL-on-RBD (if your storage pool is RADOS based) with a floating IP might be an option. It sounds like what you're after is better performance/throughput rather than HA - the ability to write to multiple masters at the same time. Even here if nova *was* able to hit multiple databases, a master/master setup isn't going to suit, as each write to one master still needs to be replicated to the other so throughput is going to be the same. You'd then need to look to something like a native mysql cluster to get actual increased throughput. Essentially -- and there are much more acclaimed MySQL experts on this list who will correct me if I'm wrong -- most general-purpose write scalability options for MySQL ultimately boil down to sharding the database, anyhow. Which is an problem different from, and orthogonal to, replication and high availability. Cheers, Florian -- Need help with High Availability? http://www.hastexo.com/now ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
BigSwitch and Nicira have created different blueprints in quantum to add their plugins for essex-4 about ten days ago, Cisco already have it included almost from the beginning. No news of them for the moment, but i really hope that both of them make it in time! Ghe Rivero https://blueprints.launchpad.net/quantum/+spec/quantum-nvp-plugin https://blueprints.launchpad.net/quantum/+spec/quantum-bigswitch-plugin 2012/2/21 Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com This is the kind of flame war I love... ;-) if you think the HA challenges in Openstack Nova are at MySQL and Rabbit level, you are not putting the focus on the right components. The biggest HA (and scalability) challenges in Nova are in the networking side. There are a lot of companies tackling the networking challenges of Nova (BigSwitch, Nicira, Cisco, etc...), and I would like to see how they evolve in the future and how they will address this problem. Right now, only promises. Fight! -- Diego Parrilla http://www.stackops.com/*CEO* *www.stackops.com | * diego.parri...@stackops.com** | +34 649 94 43 29 | skype:diegoparrilla* * http://www.stackops.com/ ** On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: ** Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept multiple locations and failover if the first location fails. Or even round-robin to distribute the load. All I am doing now is yet add another component that can possibly fail as well. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. -- *From:* Darren Birkett [mailto:darren.birk...@gmail.com] *To:* i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] *Cc:* Florian Haas [mailto:flor...@hastexo.com], openstack@lists.launchpad.net *Sent:* Tue, 21 Feb 2012 12:51:11 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Surely it depends what you want out of it? If just pure high availability (ie ability to continue functioning when primary node goes down), then doesn't a master/master with floating IP fit the bill? It sounds like what you're after is better performance/throughput rather than HA - the ability to write to multiple masters at the same time. Even here if nova *was* able to hit multiple databases, a master/master setup isn't going to suit, as each write to one master still needs to be replicated to the other so throughput is going to be the same. You'd then need to look to something like a native mysql cluster to get actual increased throughput. Darren On 21 February 2012 11:34, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Ho Florian, I respect your opinion of course and I am not here to start a fight. Point is, how I understand it Nova really isn't very flexible on High Availability terms. I have now a master/master mysql setup with a Virtual IP working just to work around the shortcomings Nova imo introduces. This same story goes for RabbitMQ, I need to setup RabbitMQ in HA (which is fine) but I need to setup a Virtual IP for it as well just so Nova can correctly connect to it. Without the VIP the hole point of setting it up in a cluster is useless as you still will point nova to one single host. It shouldn't be that hard to make Nova listen to multiple mysql addresses and rabbitmq hosts in a cluster? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our
Re: [Openstack] Howto Nova setup with HA?
Hm, I agree on that. Yes applications should be designed for the cloud no question there. How far is Nova ready for availability zones? Can I make different zones in the US and EU and let them work together? Is there any guidance for this available? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Michael Stein [mailto:michael.st...@q9.com] To: Diego Parrilla Santamaría [mailto:diego.parrilla.santama...@gmail.com], i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] Cc: darren.birk...@gmail.com, openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 14:52:50 +0100 Subject: RE: [Openstack] Howto Nova setup with HA? I think that the idea behind Nova and “cloud” computing, in general, is that applications, middle-ware, databases and other services, will be delivered across multiple availability zones and provide HA via application-layer replication. The concept of failure is assumed and designed for by any forward thinking “cloud” designer. I think less time need to be spent on preserving nodes that maintain the “holy grail” of functionality and availability and more time needs to be spent on making apps cloud-friendly. In the case described below, I agree with Diego, networking is the real challenge. However, with a GSLB solution and application-level state/data replication you should be able to keep your services running even with an entire site failure. _ From: openstack-bounces+mstein=q9@lists.launchpad.net [mailto:openstack-bounces+mstein=q9@lists.launchpad.net] On Behalf Of Diego Parrilla Santamaría Sent: Tuesday, February 21, 2012 7:56 AM To: i3D.net - Tristan van Bokkem Cc: darren.birk...@gmail.com; openstack@lists.launchpad.net Subject: Re: [Openstack] Howto Nova setup with HA? This is the kind of flame war I love... ;-) if you think the HA challenges in Openstack Nova are at MySQL and Rabbit level, you are not putting the focus on the right components. The biggest HA (and scalability) challenges in Nova are in the networking side. There are a lot of companies tackling the networking challenges of Nova (BigSwitch, Nicira, Cisco, etc...), and I would like to see how they evolve in the future and how they will address this problem. Right now, only promises. Fight! -- Diego Parrilla CEO www.stackops.com | diego.parri...@stackops.com | +34 649 94 43 29 | skype:diegoparrilla On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept multiple locations and failover if the first location fails. Or even round-robin to distribute the load. All I am doing now is yet add another component that can possibly fail as well. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte
Re: [Openstack] Howto Nova setup with HA?
Yes, thank you. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Darren Birkett [mailto:darren.birk...@gmail.com] To: Michael Stein [mailto:michael.st...@q9.com] Cc: Diego Parrilla Santamaría [mailto:diego.parrilla.santama...@gmail.com], i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 15:36:23 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? 2012/2/21 Michael Stein michael.st...@q9.com: I think that the idea behind Nova and “cloud” computing, in general, is that applications, middle-ware, databases and other services, will be delivered across multiple availability zones and provide HA via application-layer replication. The concept of failure is assumed and designed for by any forward thinking “cloud” designer. I think less time need to be spent on preserving nodes that maintain the “holy grail” of functionality and availability and more time needs to be spent on making apps cloud-friendly. Agreed, but you're talking about HA at a different layer - within the applications that will sit on top of the cloud infrastructure. Whilst that's absolutely valid, and a message that needs to be spread more amongst those designing apps for the cloud, it's not addressing the concerns of the OP. We still need to be thinking about HA of the cloud infrastructure components themselves, so that there are less failures for the app designers to have to tolerate in the first place. Darren ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
For some reason it appears that Darren's message is late in making it into my inbox, so I'm replying to Tristan's response instead. On Tue, Feb 21, 2012 at 3:40 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Agreed, but you're talking about HA at a different layer - within the applications that will sit on top of the cloud infrastructure. Whilst that's absolutely valid, and a message that needs to be spread more amongst those designing apps for the cloud, it's not addressing the concerns of the OP. We still need to be thinking about HA of the cloud infrastructure components themselves, so that there are less failures for the app designers to have to tolerate in the first place. True. There are two different layers to be addressed here, and I believe HA of the cloud infra components is a pretty crucial one to address. That being said -- and excuse me for beating the Pacemaker drum again -- the issue of sites (multiple cloud cells, if you will, that communicate automagically, maintain a distributed consensus about site availability, and fail over automatically) is also solved in that stack. The relevant project is named booth; for those who are interested, the SUSE docs team has whipped up some useful documentation for that: http://doc.opensuse.org/products/draft/SLE-HA/SLE-ha-guide_sd_draft/cha.ha.geo.html This is an automatic site-failover mechanism based on site tickets, the state of which is maintained via a simple Paxos cluster (for those into Ceph/RADOS, this will sound familiar). Pacemaker then can simply fire up or shut down resources (OpenStack services) based on site availability. Third-party arbitration is supported. Hope this is useful. Cheers, Florian -- Need help with High Availability? http://www.hastexo.com/now ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
2012/2/21 Ghe Rivero ghe.riv...@stackops.com BigSwitch and Nicira have created different blueprints in quantum to add their plugins for essex-4 about ten days ago, Cisco already have it included almost from the beginning. No news of them for the moment, but i really hope that both of them make it in time! Ghe Rivero https://blueprints.launchpad.net/quantum/+spec/quantum-nvp-plugin https://blueprints.launchpad.net/quantum/+spec/quantum-bigswitch-plugin To be clear, there are several more existing and planned plugins for Quantum, even beyond the three you mention. Each plugin will make its own trade-offs in terms of HA for control plane and data plane components. I'm planning on organizing a talk at the Folsom summit/conference where people have a chance to talk about their plugins. No need for a flame war here :) One other point of confusion that I'd like to address is that there is NO requirement that a plugin be merged into the main Quantum repo for it to be a valid plugin. I know of several plugins that are used despite not being merged in (in fact, at the summit, we're going to discuss if ANY plugins should remain in the main repo... this is likely a larger decision that should be made consistently across all openstack projects). Dan 2012/2/21 Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com This is the kind of flame war I love... ;-) if you think the HA challenges in Openstack Nova are at MySQL and Rabbit level, you are not putting the focus on the right components. The biggest HA (and scalability) challenges in Nova are in the networking side. There are a lot of companies tackling the networking challenges of Nova (BigSwitch, Nicira, Cisco, etc...), and I would like to see how they evolve in the future and how they will address this problem. Right now, only promises. Fight! -- Diego Parrilla http://www.stackops.com/*CEO* *www.stackops.com | * diego.parri...@stackops.com** | +34 649 94 43 29 | skype:diegoparrilla* * http://www.stackops.com/ ** On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: ** Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept multiple locations and failover if the first location fails. Or even round-robin to distribute the load. All I am doing now is yet add another component that can possibly fail as well. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. -- *From:* Darren Birkett [mailto:darren.birk...@gmail.com] *To:* i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] *Cc:* Florian Haas [mailto:flor...@hastexo.com], openstack@lists.launchpad.net *Sent:* Tue, 21 Feb 2012 12:51:11 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Surely it depends what you want out of it? If just pure high availability (ie ability to continue functioning when primary node goes down), then doesn't a master/master with floating IP fit the bill? It sounds like what you're after is better performance/throughput rather than HA - the ability to write to multiple masters at the same time. Even here if nova *was* able to hit multiple databases, a master/master setup isn't going to suit, as each write to one master still needs to be replicated to the other so throughput is going to be the same. You'd then need to look to something like a native mysql cluster to get actual increased throughput. Darren On 21 February 2012 11:34, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Ho Florian, I respect your opinion of course and I am not here to start a fight. Point is, how I understand it Nova really isn't very flexible on High Availability terms. I have now a master
Re: [Openstack] Howto Nova setup with HA?
On Thu, Feb 16, 2012 at 3:09 PM, Tom Ellis tom.el...@canonical.com wrote: nova-volume service * Current weakness in the HA setup, unless you are willing to use iscsi tgtd with DRBD. I believe this would still have some problems when failing over with the initiators that are logged in. Nope, this is well understood and supported. It all amounts to getting the Pacemaker configuration right, but DRBD-backed, tgt-based iSCSI targets are perfectly capable of failover that is fully transparent to initiators. That being said, of course nova-volume with RBD (RADOS block device) would be another approach that may provide better reliability (more replicas, where DRBD is limited to 2 -- 4 with some trickery), better scaleout (seamless expansion and contraction of the storage space), and better self-healing capabilities (automatic rebalancing if individual replicas fail). Hope this is useful. Cheers, Florian -- Need help with High Availability? http://www.hastexo.com/now ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Any more thoughts about this subject? Ask? Vish? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Alexis Richardson [mailto:ale...@rabbitmq.com] To: Major Hayden [mailto:major.hay...@rackspace.com] Cc: i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net [mailto:openstack@lists.launchpad.net], Ask Solem [mailto:a...@rabbitmq.com] Sent: Tue, 14 Feb 2012 14:47:13 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? Major, The current set-up of HA in Openstack is borked in several ways. Ask Solem, cc'd, is going to write up a note helping people to do it properly. alexis On Tue, Feb 14, 2012 at 1:34 PM, Major Hayden major.hay...@rackspace.com wrote: On Feb 14, 2012, at 2:07 AM, i3D.net - Tristan van Bokkem wrote: So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! If you want to make things a little simpler, an active/passive MySQL setup with pacemaker, corosync, and DRBD will work quite well. ClusterLabs has a pretty decent howto on their site: http://www.clusterlabs.org/wiki/DRBD_MySQL_HowTo As for RabbitMQ clustering, I worked with it for several days and had quite a few problems with data consistency and availability. We eventually settled on the legacy H/A method for RabbitMQ using pacemaker and DRBD. There's a pretty good walkthrough on RabbitMQ's site: http://www.rabbitmq.com/pacemaker.html -- Major Hayden ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
FWIW, we're using DRBD + pacemaker in active/passive setups for both the MySQL and RabbitMQ setup in the trystack.org (the artist formerly known as FreeCloud) project, and it's working quite well and is pretty simple to set up. Nati Ueno could give some more details on the setup and the Chef cookbooks used... Best, -jay On 02/16/2012 03:46 AM, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net E-mail Support: i...@i3d.net mailto:i...@i3d.net E-mail NOC: n...@i3d.net mailto:n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. *From:* Alexis Richardson [mailto:ale...@rabbitmq.com] *To:* Major Hayden [mailto:major.hay...@rackspace.com] *Cc:* i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net [mailto:openstack@lists.launchpad.net], Ask Solem [mailto:a...@rabbitmq.com] *Sent:* Tue, 14 Feb 2012 14:47:13 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Major, The current set-up of HA in Openstack is borked in several ways. Ask Solem, cc'd, is going to write up a note helping people to do it properly. alexis On Tue, Feb 14, 2012 at 1:34 PM, Major Hayden major.hay...@rackspace.com mailto:major.hay...@rackspace.com wrote: On Feb 14, 2012, at 2:07 AM, i3D.net - Tristan van Bokkem wrote: So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! If you want to make things a little simpler, an active/passive MySQL setup with pacemaker, corosync, and DRBD will work quite well. ClusterLabs has a pretty decent howto on their site: http://www.clusterlabs.org/wiki/DRBD_MySQL_HowTo As for RabbitMQ clustering, I worked with it for several days and had quite a few problems with data consistency and availability. We eventually settled on the legacy H/A method for RabbitMQ using pacemaker and DRBD. There's a pretty good walkthrough on RabbitMQ's site: http://www.rabbitmq.com/pacemaker.html -- Major Hayden ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Yup, absolutely... we've been a bit slammed working on getting the project up and running, but everything is checked in to GitHub... just needs a bit of clean up, that's all. Best, -jay On 02/16/2012 07:10 AM, Alexis Richardson wrote: It would be great to see the cookbooks if they are available, since others doing that setups might want such a thing. On Thu, Feb 16, 2012 at 11:51 AM, Jay Pipesjaypi...@gmail.com wrote: FWIW, we're using DRBD + pacemaker in active/passive setups for both the MySQL and RabbitMQ setup in the trystack.org (the artist formerly known as FreeCloud) project, and it's working quite well and is pretty simple to set up. Nati Ueno could give some more details on the setup and the Chef cookbooks used... Best, -jay On 02/16/2012 03:46 AM, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net E-mail Support: i...@i3d.netmailto:i...@i3d.net E-mail NOC: n...@i3d.netmailto:n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. *From:* Alexis Richardson [mailto:ale...@rabbitmq.com] *To:* Major Hayden [mailto:major.hay...@rackspace.com] *Cc:* i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net [mailto:openstack@lists.launchpad.net], Ask Solem [mailto:a...@rabbitmq.com] *Sent:* Tue, 14 Feb 2012 14:47:13 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Major, The current set-up of HA in Openstack is borked in several ways. Ask Solem, cc'd, is going to write up a note helping people to do it properly. alexis On Tue, Feb 14, 2012 at 1:34 PM, Major Hayden major.hay...@rackspace.commailto:major.hay...@rackspace.com wrote: On Feb 14, 2012, at 2:07 AM, i3D.net - Tristan van Bokkem wrote: So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! If you want to make things a little simpler, an active/passive MySQL setup with pacemaker, corosync, and DRBD will work quite well. ClusterLabs has a pretty decent howto on their site: http://www.clusterlabs.org/wiki/DRBD_MySQL_HowTo As for RabbitMQ clustering, I worked with it for several days and had quite a few problems with data consistency and availability. We eventually settled on the legacy H/A method for RabbitMQ using pacemaker and DRBD. There's a pretty good walkthrough on RabbitMQ's site: http://www.rabbitmq.com/pacemaker.html -- Major Hayden ___ Mailing list: https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack https://launchpad.net/%7Eopenstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Interesting. Could you point us to the github location? Anyway, ive have done some further research myself and find this very interesting mysql replication setup done by wsrep (https://launchpad.net/wsrep) and galeria (https://launchpad.net/galera/) where mysql will run as a Synchronous Multi-Master replication setup. I have this running and works perfect but the problem now is. I need to add some kind of loadbalancing setup as nova can only point to one mysql location (i.e. --sql_connection=) So if I would point it to one of the mysql nodes in the cluster and that one fails everything fails. Adding a loadbalancer and point it to that will keep my mysql alive. The main issue with this is, we are adding another point where we have yet to configure and maintain several more nodes. I would like to raise the question if it would be possible for nova to accept multiple mysql connection points and therefor drop the need for extra loadbalancer component and reducing complexity. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Jay Pipes [mailto:jaypi...@gmail.com] To: Alexis Richardson [mailto:ale...@rabbitmq.com] Cc: openstack@lists.launchpad.net Sent: Thu, 16 Feb 2012 13:11:36 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? Yup, absolutely... we've been a bit slammed working on getting the project up and running, but everything is checked in to GitHub... just needs a bit of clean up, that's all. Best, -jay On 02/16/2012 07:10 AM, Alexis Richardson wrote: It would be great to see the cookbooks if they are available, since others doing that setups might want such a thing. On Thu, Feb 16, 2012 at 11:51 AM, Jay Pipesjaypi...@gmail.com wrote: FWIW, we're using DRBD + pacemaker in active/passive setups for both the MySQL and RabbitMQ setup in the trystack.org (the artist formerly known as FreeCloud) project, and it's working quite well and is pretty simple to set up. Nati Ueno could give some more details on the setup and the Chef cookbooks used... Best, -jay On 02/16/2012 03:46 AM, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net E-mail Support: i...@i3d.netmailto:i...@i3d.net E-mail NOC: n...@i3d.netmailto:n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. *From:* Alexis Richardson [mailto:ale...@rabbitmq.com] *To:* Major Hayden [mailto:major.hay...@rackspace.com] *Cc:* i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net [mailto:openstack@lists.launchpad.net], Ask Solem [mailto:a...@rabbitmq.com] *Sent:* Tue, 14 Feb 2012 14:47:13 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Major, The current set-up of HA in Openstack is borked in several ways. Ask Solem, cc'd, is going to write up a note helping people to do it properly. alexis On Tue, Feb 14, 2012 at 1:34 PM, Major Hayden major.hay...@rackspace.commailto:major.hay...@rackspace.com wrote: On Feb 14, 2012, at 2:07 AM, i3D.net - Tristan van Bokkem wrote: So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way
Re: [Openstack] Howto Nova setup with HA?
On 02/16/2012 07:32 AM, i3D.net - Tristan van Bokkem wrote: Interesting. Could you point us to the github location? I will as soon as I verify with Nati that all the most-current code has been pushed there :) Expect an email later today. Anyway, ive have done some further research myself and find this very interesting mysql replication setup done by wsrep (https://launchpad.net/wsrep) and galeria (https://launchpad.net/galera/) where mysql will run as a Synchronous Multi-Master replication setup. I have this running and works perfect but the problem now is. I need to add some kind of loadbalancing setup as nova can only point to one mysql location (i.e. --sql_connection=) So if I would point it to one of the mysql nodes in the cluster and that one fails everything fails. Adding a loadbalancer and point it to that will keep my mysql alive. Yep, this is a perfectly good solution for MySQL. The main issue with this is, we are adding another point where we have yet to configure and maintain several more nodes. I would like to raise the question if it would be possible for nova to accept multiple mysql connection points and therefor drop the need for extra loadbalancer component and reducing complexity. I would strongly resist pressure to add complexity to the Nova (or other core projects) code base when an existing solution is available that is tailored to meet the need already (install a loadbalancer...) Cheers, -kjay Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net E-mail Support: i...@i3d.net mailto:i...@i3d.net E-mail NOC: n...@i3d.net mailto:n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. *From:* Jay Pipes [mailto:jaypi...@gmail.com] *To:* Alexis Richardson [mailto:ale...@rabbitmq.com] *Cc:* openstack@lists.launchpad.net *Sent:* Thu, 16 Feb 2012 13:11:36 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Yup, absolutely... we've been a bit slammed working on getting the project up and running, but everything is checked in to GitHub... just needs a bit of clean up, that's all. Best, -jay On 02/16/2012 07:10 AM, Alexis Richardson wrote: It would be great to see the cookbooks if they are available, since others doing that setups might want such a thing. On Thu, Feb 16, 2012 at 11:51 AM, Jay Pipesjaypi...@gmail.com mailto:jaypi...@gmail.com wrote: FWIW, we're using DRBD + pacemaker in active/passive setups for both the MySQL and RabbitMQ setup in the trystack.org (the artist formerly known as FreeCloud) project, and it's working quite well and is pretty simple to set up. Nati Ueno could give some more details on the setup and the Chef cookbooks used... Best, -jay On 02/16/2012 03:46 AM, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net E-mail Support: i...@i3d.net mailto:i...@i3d.netmailto:i...@i3d.net mailto:i...@i3d.net E-mail NOC: n...@i3d.net mailto:n...@i3d.netmailto:n...@i3d.net mailto:n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. *From:* Alexis Richardson [mailto:ale...@rabbitmq.com mailto:ale...@rabbitmq.com
Re: [Openstack] Howto Nova setup with HA?
On 16/02/12 08:46, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? The team I work in has looked at a number of methods for high availability within the Diablo release and I've included some notes below, hope this helps. Font-end API servers * load balanced with h/w load balancer * use s/w LB for smaller deployments * run nova-scheduler on each MySQL DB * multi-master configuration * alternative: drbd + pacemaker in active/passive RabbitMQ service * Pacemaker with Active-passive configuration * Based on example from RabbitMQ site (already mentioned by someone else) * Virtual IP for the service - used for rabbitmq config in nova.conf * I think there are some corner cases where messages could be dropped during failover and I believe later RabbitMQ versions support full multi-master but require some client side changes - is there any plans to support this? nova-volume service * Current weakness in the HA setup, unless you are willing to use iscsi tgtd with DRBD. I believe this would still have some problems when failing over with the initiators that are logged in. * I'm hoping this will pan out for essex with some of the storage vendors committing nova-volume support via their API's. Glance * Run on multiple servers * Use another VIP in your pacemaker setup or load-balancer * Use swift as backend storage Compute servers * Each run their own copy of nova-api (only instances running on the node use this) * nova-network (multi-host configuration) with private network Swift * Run swift-proxy across all swift-storage nodes on a small setup Regards, Tom ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Hi Tom, Thank you very much for sharing your knowledge, this will help not only me! As I am still working on the MySQL part, you mentioned multi-master which I have running indeed, but I suppose there still will need to be some kind of entering point within the cluster. How did you managed this? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Tom Ellis [mailto:tom.el...@canonical.com] To: openstack@lists.launchpad.net Sent: Thu, 16 Feb 2012 15:09:15 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? On 16/02/12 08:46, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? The team I work in has looked at a number of methods for high availability within the Diablo release and I've included some notes below, hope this helps. Font-end API servers * load balanced with h/w load balancer * use s/w LB for smaller deployments * run nova-scheduler on each MySQL DB * multi-master configuration * alternative: drbd + pacemaker in active/passive RabbitMQ service * Pacemaker with Active-passive configuration * Based on example from RabbitMQ site (already mentioned by someone else) * Virtual IP for the service - used for rabbitmq config in nova.conf * I think there are some corner cases where messages could be dropped during failover and I believe later RabbitMQ versions support full multi-master but require some client side changes - is there any plans to support this? nova-volume service * Current weakness in the HA setup, unless you are willing to use iscsi tgtd with DRBD. I believe this would still have some problems when failing over with the initiators that are logged in. * I'm hoping this will pan out for essex with some of the storage vendors committing nova-volume support via their API's. Glance * Run on multiple servers * Use another VIP in your pacemaker setup or load-balancer * Use swift as backend storage Compute servers * Each run their own copy of nova-api (only instances running on the node use this) * nova-network (multi-host configuration) with private network Swift * Run swift-proxy across all swift-storage nodes on a small setup Regards, Tom ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Hi Tristan. So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! I would prefer to use MySQL Cluster: http://www.mysql.com/products/cluster/. You meant that with MySQL master-master? For RabbitMQ there are several ways to setup it: http://www.rabbitmq.com/ha.html. I think the customer site has to decide which HA setup they want use. All types have advantages and disadvantages. HTH, Christian. -- Christian Berendt Linux / Unix Consultant Developer Mail: bere...@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Hi Tristan, When I saw your post, I though what about an pacemaker ressource agent? Corosync for the messaging layer and pacemaker for the ressource management. Maybe someone as wrote a script ressource agent for nova in order to manage the failover. Also DRBD can be useful. And then after a couple of research on Google, I found this 3 links: - https://lists.launchpad.net/openstack/pdfGiNwMEtUBJ.pdf - http://wiki.openstack.org/HAforNovaDB - http://www.pixelbeat.org/docs/pacemaker-cloud/ Hope this will help you. -- Yours sincerely. Sébastien HAN. On Tue, Feb 14, 2012 at 12:05 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: ** Hi Christian, Thanks for your reply. With mysql master/master i meant the following: http://mysql-mmm.org/ but it seems they are using 2 different locations for mysql reads and writes. Something nova is (yet) not able to configure (i.e. --sql_connection= which is being used for reads and rights.) Maybe http://www.mysql.com/products/cluster/ is a better way.. RabbitMQ HA i am aware of. This is something we are looking at as well. The main question is how does other companies reach HA with openstack? I can't find any good information on the net about it. And how does Essex change things? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. -- *From:* Christian Berendt [mailto:bere...@b1-systems.de] *To:* i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] *Cc:* openstack@lists.launchpad.net *Sent:* Tue, 14 Feb 2012 11:54:38 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Hi Tristan. So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! I would prefer to use MySQL Cluster: http://www.mysql.com/products/cluster/. You meant that with MySQL master-master? For RabbitMQ there are several ways to setup it: http://www.rabbitmq.com/ha.html. I think the customer site has to decide which HA setup they want use. All types have advantages and disadvantages. HTH, Christian. -- Christian Berendt Linux / Unix Consultant Developer Mail: bere...@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
On Feb 14, 2012, at 2:07 AM, i3D.net - Tristan van Bokkem wrote: So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! If you want to make things a little simpler, an active/passive MySQL setup with pacemaker, corosync, and DRBD will work quite well. ClusterLabs has a pretty decent howto on their site: http://www.clusterlabs.org/wiki/DRBD_MySQL_HowTo As for RabbitMQ clustering, I worked with it for several days and had quite a few problems with data consistency and availability. We eventually settled on the legacy H/A method for RabbitMQ using pacemaker and DRBD. There's a pretty good walkthrough on RabbitMQ's site: http://www.rabbitmq.com/pacemaker.html -- Major Hayden ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
