Re: [SLUG] Virtualisation and DRBD
While the setup you describe is possible there are a number of drawbacks: - DRBD disks are very slow, depending on the protocol you set them to use but still there is a performance penalty. If you DRBD the entire image of a running xen guest, your going to suffer performance hits for everything the OS does - not just the HA important data - Live migration is fantastic but you lose the ability to independently upgrade either side of the cluster. This makes rollback harder and more complex - Likewise, you kinda put all your HA eggs into one basket since this one DRBD setup might be the basis for a number of clusters. Can create some complex failure scenarios, and be hard to maintain e.g. should you wish to resize the DRBD disk maybe - Finally you now make a decision and describe the VM's themselves as cluster resources, or you have clusters on-top of clusters and have a dom0 cluster and domU cluster. For these reasons I would lean towards running DRBD inside each VM, with a minimum amount of shared state of the disk Joel On Wed, Aug 25, 2010 at 12:46 PM, Nigel Allen wrote: > > Hi All > > We're investigating both virtualisation of servers and High Availability > at the same time. > > Currently looking at Linux-HA and DRBD (amongst others). > > The idea of DRBD appeals to both me and the client as it means (or seems > to at least) that we could add a third (off-site) machine into the > equation for "real" DR. > > What happens when we then introduce Virtualisation into the equation > (currently have 4 x servers running Centos & Windoze Server - looking at > virtualising them onto one single box running Centos-5). > > I suppose the (first) question is: If we run 4 virtualised servers (B, > C, D, and E) on our working server "A" (complete with it's own storage), > can we also use DRBD to sync the entire box and dice onto server A1 > (containing servers B1, C1, D1, and E1) or do we have to sync them all > separately? Will this idea even float? Can we achieve seamless failover > with this. If not, how would you do it > > Any input (as ever) gratefully accepted. > > Confused at the Console > > Nigel. > > > > > > -- > SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ > Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html > -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Virtualisation and DRBD
On 24 August 2010 19:46, Nigel Allen wrote: > > Any input (as ever) gratefully accepted. > You should take a look at ganeti: http://code.google.com/p/ganeti/ which is a virtualisation management system that includes DRBD backed storage. Even if you don't use it, you can get some tips on DRBD setups for VMs from here. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Virtualisation and DRBD
Nigel Allen writes: > We're investigating both virtualisation of servers and High Availability at > the same time. Currently looking at Linux-HA and DRBD (amongst others). Keep in mind that both things — HA and virtualization — are actually pretty hard to get working smoothly. Just ask me how it sucks when things don't work right, after another ten hour day fighting a data corruption bug. ;) > The idea of DRBD appeals to both me and the client as it means (or seems to > at least) that we could add a third (off-site) machine into the equation for > "real" DR. DRBD is two-machine in pretty much any reasonable setup, and the performance for an off-site spare is going to be ... interesting. If you don't have a very low latency, high bandwidth link between the locations then you can expect significant pain and suffering. (The safe DRBD protocol means suffering in performance, the unsafe ones insulate you from pain until you depend on that warm-spare and find out if it actually got all corrupted or not before the failure.) > What happens when we then introduce Virtualisation into the equation > (currently have 4 x servers running Centos & Windoze Server - looking at > virtualising them onto one single box running Centos-5). Keep in mind that performance absolutely requires that you have paravirtualized drivers for those kernels. That means picking something where you have good disk and network virtual drivers ... and that probably means "not KVM", which sucks. > I suppose the (first) question is: If we run 4 virtualised servers (B, > C, D, and E) on our working server "A" (complete with it's own storage), > can we also use DRBD to sync the entire box and dice onto server A1 > (containing servers B1, C1, D1, and E1) or do we have to sync them all > separately? Yes. Specifically: you can do either, or both, depending on how you set it up, and on the capabilities of whatever management software you layer over the basic virtualization tools. > Will this idea even float? Can we achieve seamless failover with this. Maybe. You have to be very, very clear on which two of the three attributes, consistency, availability and partition tolerance you need, and make absolutely, without question certain that you deliver on that. To be clear: that means you must absolutely deliver availability (of your HA solution) and non-partitioned connectivity, because you can't live with inconsistency of data between the machines. This is much harder than it sounds: you can easily work out that you have a network cable pulled and have the entire ball of wax fall apart if you are not very, very careful. > If not, how would you do it. We just moved from this to delivering iSCSI storage in the back-end, with execute nodes that are going to start shedding local disk. This uses KVM as the virtualization technology, but anything that can talk to raw partitions should be fine on top of this. This gives us two advantages: first, we can scale as broad as we want (and have great performance) by virtue of deploying cheap storage nodes with 6TB of usable disk, tolerance of any two disk failures, and 8.5GB of disk cache between the nodes and the spinning rust. Adding another of those and moving some of the load to another system is relatively inexpensive, and can fairly easily grow with our needs; we can also scale up storage node performance by throwing a more expensive SAS array or server into the mix. (...or just more cache memory, at $100 a GB or so.) The second is that we can deliver reliability without complicating the storage nodes: a virtual machine can use Linux software RAID to mirror, stripe, or otherwise combine multiple iSCSI LUNs inside the virtual machine. This gives us similar performance and redundancy benefits to using local storage to back those devices, including the ability to lose a storage node and have the machine continue to work. This does mean that, unlike a design where replication is directly between storage nodes, we send writes out N times for N targets from the processing node — but that isn't any more bytes over the network overall, and we are not short on network capacity. (Plus, dedicated storage links on a bunch of the busier machines make this work without interference with public services.) Daniel -- ✣ Daniel Pittman✉ dan...@rimspace.net☎ +61 401 155 707 ♽ made with 100 percent post-consumer electrons -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
Re: [SLUG] Virtualisation and DRBD
It depends on your level of availability. you can achieve 98% percent with Virtualised solutions and live migration or vmotion. This means migrating a functioning virtual server from one physical server to another with no outage. You just need some shared storage that has suitable raid levels and redundant network/fibre cards and switches. We use NFS as our shared storage and find it great. This live migration has the advantage of 1 Migrate virtualised servers away from a physical host that requires maintenance. 2 Utilise both phyiscal servers or keep one in standby. 3 Straight forward recovery. Just boot the virtual machine on a different physical machine. 4 Scalable The only downside is that if there is a hard failure, it is like someone tripped over the power cord. In a lot of scenarios if you decrease your planned outages to 0, this is acceptable. The last 2% will require the biggest effort in designing, implementing, testing, testing and more testing, oh and don't forget the ongoing maintenance and documentation. It really depends on what is acceptable risk, acceptable downtime, acceptable data loss and therefore what you need to protection from. eg Disk failure, network failure, heartbeat failure, hardware failure, power failure, HTH Grant On 25/08/10 12:46, Nigel Allen wrote: Hi All We're investigating both virtualisation of servers and High Availability at the same time. Currently looking at Linux-HA and DRBD (amongst others). The idea of DRBD appeals to both me and the client as it means (or seems to at least) that we could add a third (off-site) machine into the equation for "real" DR. What happens when we then introduce Virtualisation into the equation (currently have 4 x servers running Centos& Windoze Server - looking at virtualising them onto one single box running Centos-5). I suppose the (first) question is: If we run 4 virtualised servers (B, C, D, and E) on our working server "A" (complete with it's own storage), can we also use DRBD to sync the entire box and dice onto server A1 (containing servers B1, C1, D1, and E1) or do we have to sync them all separately? Will this idea even float? Can we achieve seamless failover with this. If not, how would you do it Any input (as ever) gratefully accepted. Confused at the Console Nigel. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
[SLUG] Virtualisation and DRBD
Hi All We're investigating both virtualisation of servers and High Availability at the same time. Currently looking at Linux-HA and DRBD (amongst others). The idea of DRBD appeals to both me and the client as it means (or seems to at least) that we could add a third (off-site) machine into the equation for "real" DR. What happens when we then introduce Virtualisation into the equation (currently have 4 x servers running Centos & Windoze Server - looking at virtualising them onto one single box running Centos-5). I suppose the (first) question is: If we run 4 virtualised servers (B, C, D, and E) on our working server "A" (complete with it's own storage), can we also use DRBD to sync the entire box and dice onto server A1 (containing servers B1, C1, D1, and E1) or do we have to sync them all separately? Will this idea even float? Can we achieve seamless failover with this. If not, how would you do it Any input (as ever) gratefully accepted. Confused at the Console Nigel. -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
