On 02/08/2012 08:10 AM, Ori Mamluk wrote:
On 07/02/2012 17:47, Paolo Bonzini wrote:
On 02/07/2012 03:48 PM, Ori Mamluk wrote:
The current streaming code in QEMU only deals with the former.
Streaming to a remote server would not be supported.
I need it at the same time. The Rephub reads either the full volume or
parts of, and concurrently protects new IOs.
Why can't QEMU itself stream the full volume in the background, and
send that together with any new I/O? Is it because the rephub knows
which parts are out-of-date and need recovery? In that case, as a
first approximation the rephub can pass the sector at which streaming
should start.
Yes - it's because rephub knows. The parts that need recovery may be a
series of random IOs that were lost because of a network outage
somewhere along the replication pipe.
Easy to think of it as a bitmap holding the not-yet-replicated IOs. The
rephub occasionally reads those areas to 'sync' them, so in effect the
rephub needs read access - it's not really to trigger streaming from an
offset.
But I'm also starting to wonder whether it would be simpler to use
existing replication code. DRBD is more feature-rich, and you can use
it over loopback or NBD devices (respectively raw and non-raw), and
also store the replication metadata on a file using the loopback
device. Ceph even has a userspace library and support within QEMU.
I think there are two immediate problems that drbd poses:
1. Our replication is not a simple mirror - it maintains history. I.e.
you can recover to any point in time in the last X hours (usually 24) at
a granularity of about 5 seconds.
To be able to do that and keep the replica consistent we need to be
notified for each IO.
Can you please elaborate some more in the exact details -
In theory, you can build a setup where the drbd (or nbd) copy on the
destination side write to a intermediate image and every such write is
trapped locally on the destination and you may not immediately propagate
that to the disk image the VM sees.
2. drbd is 'below' all the Qemu block layers - if the protected volume
is qcow2 then drbd doesn't get the raw IOs, right?
That's one of the major caveats in drbd/iscsi/nbd - there is no support
for block level snapshots[1]. I wonder if the scsi protocol has
something like this so we'll get efficient replication of qcow2/lvm
snapshots that their base is already shared. If we'll gain such
functionality, we'll benefit of it for storage vm motion solution too.
Another issue w/ drbd is that a continuous backup solution requires to
do consistent snapshot and call file system freeze and sync it w/ the
current block IO transfer. DRBD doesn't do that nor the other protocols.
Of course DRBD can be enhanced but it will take allot more time.
A third requirement and similar to above is to group snapshots of
several VMs so a consistent _cross vm application view_ will be created.
It demands some control over IO tagging.
To summarize, IMHO drbd (which I used successfully 6 years ago and I
love) is not drop&replace solution to this case.
I recommend we either to fit the nbd/iscsi case and improve our vm
storage motion on the way or worse case develop proprietary logic that
can live out side of qemu using IO tapping interface, similar to the
guidelines Ori outlines.
Thanks,
Dor
[1] Check the far too basic approach for snapshots:
http://www.drbd.org/users-guide/s-lvm-snapshots.html
Ori
