Hi Logan, The biggest issue to deal with in a multi-datacenter deployment is the database, specifically latency between database nodes and between the CloudStack management servers and the database. Hosts in remote zones quite happily talk back to remote management servers over WAN links.
Currently CloudStack employs record locking which is not multi-master writing friendly. So you can only really write to one node. (this is something that is being looked at). As the latency between the management servers and the hosts is far less important than that relating to DB traffic the standard configuration is to have a master zone which contains your primary management server(s) and a MySQL master and slave pair. You then have a second zone (chosen largely by the latency on the data link) which contains secondary management server(s) and MySQL slave to the primary zone MySQL master (and probably a slave to that slave - I'll call it a secondary slave). Then you would you global server load balancing (GSLB) to switch from the primary zone to the secondary zone in the event of a primary zone failure. The slave in the secondary zone becomes the master and the secondary slave becomes 'the' slave. All other zones use the primary zone mgmt. infrastructure unless GSLB directs them to the secondary zone. Regards, Paul Angus Cloud Architect S: +44 20 3603 0540 | M: +447711418784 | T: @CloudyAngus paul.an...@shapeblue.com -----Original Message----- From: Logan Barfield [mailto:lbarfi...@tqhosting.com] Sent: 06 January 2015 19:46 To: d...@cloudstack.apache.org; users@cloudstack.apache.org Subject: Multi-Datacenter Deployment We are currently running a single location CloudStack deployment: - 1 Hardware firewall - 1 Mangement/Database Server - 1 NFS staging store (for S3 secondary storage) - Ceph RBD for primary storage - 4 Hypervisors - 1 Zone/Pod/Cluster We are looking to expand our deployment to other datacenters, and I'm trying to determine the best way to go about it. The documentation is a bit lacking for multi-site deployments. Our goal for the multi-site deployment is to have a zone for each site (E.G. US East, US West, Europe) that our customers can use to deploy instances in their preferred geographic area. Since we don't want to have different accounts for every datacenter, I don't think using Regions makes sense for us (and I'm not sure what they're actually good for without keeping accounts/users/domains in sync). Right now I'm thinking our setup will be as follows: - Firewall, Management Server, NFS staging server, primary storage, and Hypervisors in each datacenter. - All Management servers will be on the same management network. - Management servers will be connected via site-to-site VPN links over WAN. - MySQL replication (Percona?) will be set up on the management servers. Having an odd number of servers to protect against split brain, and keeping redundant database backups. - One region (default) - One zone for each datacenter - Geo-enabled DNS to direct customers to the nearest Management server - Object storage for secondary storage across cloud. My primary concerns with this setup are: - I haven't really seen multi-site deployments details anywhere. - Potential for split-brain. - How will HA be handled (e.g., if a VPN link goes down and one of the remote management servers can't contact a host, will it try to initiate HA?) - This sort of goes along with the split brain problem. Are my assumptions here sound, or is there a standard recommended way of doing multi-site deployments? Any suggestions are much appreciated. Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
