Amos Shapira wrote:
We use Xen+CentOS 5+DRBD+Linux-HA to achieve similar goals.
We actually build each side of the cluster separately using automatic
deployment tools (puppet and some glue around it).
We use ext3 on the DRBD patition, the DRBD is actually managed from
inside the Xen guests, not the host (we have different DRBD partitions
for different guests).
That's an interesting idea so your giving the VM the raw partition
/dev/sdfoo and running drbd in the guest on that.
how are you getting around booting? or are you doing something with xen
for that, feeding it a running kernel it can mount / as a drbd or some such?
Linux-HA gives automatic fail-over (been tested a few times "under
fire" when hardware failed - the other side took over automatically
and all we saw from this was an SMS from Nagios about the crashed
server being down).
That is pretty much the optimal solution, nice to hear it working in the
real world.
But DRBD could come at a performance cost, depends on how much you are
pushing the setup it could hurt and we are looking at cheap SAN
replacements for the development/office stuff.
It depends on the settings for your drbd setup as well doesn't it? If
you turn its paranoia level down somewhat I was under the impression its
performance hit wasn't that large. IE set it to ok on transmission.
If you want seemless transitions your going to want something like OCFS or
We tried to setup GFS on top of DRBD (+on top of Xen) in order to move
some of the functions to primary/primary mode but the performance was
horrendous. Maybe we could get it to work if we spent more time
tweaking it but just switched back to primary/secondary and ext3 setup
for now.
What sort of load were you running, it sounds disk intensive, I've found
that even raw with paravirt drivers diskio tasks are not VM friendly.
Correct.
Another option brought by a hosting provider we talked to was to setup
a couple of CentOS servers (or FreeNAS/Openfiler as was mentioned
before) to replicate the disks between them using DRBD and serve
access to the disks through iSCSI to the "application servers".
Effectively building a highly-available SAN cluster from existing
hardware. The possible advantage there might be that you have hosts
(CPU, bus, disk controller) dedicated for disk access so even though
the applications access the disks over a network it could still free
up other resources and make the app actually run faster.
I was thinking about the possibility of running iscsi nodes and using
mdadm to perform the equivalent of DRBD but I figured if it tried to
stripe reads it would be a massive performance hit.
IE run host A and B as iscsi nodes create your VM and mount a node on
host A and on host B under mdadm.
As far as I saw on the web (a bit to my surprise), ext3 journaling is
supposed to be good enough to allow live snapshots, so you don't have
to take the client down for this. Many people on the net report doing
backups that way. Windows NTFS might be different but it might also be
good enough for such a trick.
It'd be basically like restoring the power on a machine after yanking
the cable, I wouldn't bet on that working reliably even in the hobby
scale, I've had enough corrupted tables on my mythtv install at home
resulting from that that I converted the thing to innodb rather than
myisam and stuck it on a UPS. When I said snapshot, I was referring to
the practise where you take an image of the running machines ram meaning
you can restore it to a known working state exactly, with no risk of
really screwing things up. (well no more than normal) the worst thing
the client machine would notice would be a clock warp.
In general - I try to stick to the tools which come bundled with
CentOS. It comes with Xen 3.0 so that's what we use. CentOS 6 is
expected to support KVM then we'll gladly switch to it (from what I
heard about its performance over Xen I'd love to switch to it).
libvirt should help avoid virtualisation solution dependency but so
far we haven't got around to update our home-grown scripts to use it.
Cheers,
--Amos
I like libvirt and virt-manager, pointy clicky goodness ;->
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
