于 2012-11-27 18:37, Dietmar Maurer 写道:
Just want to confirm something to understand it better:
you are backing up the block image not including VM memory state right? I
am considering a way to do live Savevm including memory and device state,
so wonder if you already had a solution for it.
Does those code for VM memory and device state lively save/restore
included in this patch serials? I quickly reviewed the patches but did not
found a hook to save VM memory state? Hope you can enlight me your way,
my thoughts is do live migration into qcow2 file, but your code seems not
Am 27.11.2012 08:15, schrieb Dietmar Maurer:
The only solution I came up with is to add before/after hooks in the
block job. I agree with the criticism, but I think it's general
enough and at the same time easy enough to implement.
IMHO, the current implementation is quite simple and easy to
于 2012-11-21 17:01, Dietmar Maurer 写道:
This series provides a way to efficiently backup VMs.
* Backup to a single archive file
* Backup contain all data to restore VM (full backup)
* Do not depend on storage type or image format
* Avoid use of temporary storage
* store sparse images
The only solution I came up with is to add before/after hooks in the
block job. I agree with the criticism, but I think it's general
enough and at the same time easy enough to implement.
Ok, here is another try - do you think that is better?
Note: This code is not tested - I just want to
Just want to confirm something to understand it better:
you are backing up the block image not including VM memory state right? I
am considering a way to do live Savevm including memory and device state,
so wonder if you already had a solution for it.
Yes, I have already code for that.
Il 26/11/2012 06:51, Dietmar Maurer ha scritto:
Which raises the question of how to distinguish whether it's a new request to
bs that must go through the filters or whether it actually comes from the
last
filter in the chain. As you can see, we don't have a well thought out plan
yet,
just
The only solution I came up with is to add before/after hooks in the block
job. I agree with the criticism, but I think it's general enough and at the
same
time easy enough to implement.
IMHO, the current implementation is quite simple and easy to maintain.
No, if (bs-backup_info)
The only solution I came up with is to add before/after hooks in the
block job. I agree with the criticism, but I think it's general
enough and at the same time easy enough to implement.
IMHO, the current implementation is quite simple and easy to maintain.
No, if (bs-backup_info)
Which raises the question of how to distinguish whether it's a new request to
bs that must go through the filters or whether it actually comes from the last
filter in the chain. As you can see, we don't have a well thought out plan
yet,
just rough ideas (otherwise it would probably be
Yup, it's already not too bad. I haven't looked into it in much
detail, but I'd like to reduce it even a bit more. In particular, the
backup_info field in the BlockDriverState feels wrong to me. In the
long term the generic block layer shouldn't know at all what a backup
is, and baking
My plan was to have something like bs-job-job_type-
{before,after}_write.
int coroutine_fn (*before_write)(BlockDriverState *bs,
int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
void **cookie);
int coroutine_fn (*after_write)(BlockDriverState *bs,
My plan was to have something like bs-job-job_type-
{before,after}_write.
int coroutine_fn (*before_write)(BlockDriverState *bs,
int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
void **cookie);
int coroutine_fn (*after_write)(BlockDriverState *bs,
Am 23.11.2012 08:38, schrieb Dietmar Maurer:
In short, the idea is that you can stick filters on top of a
BlockDriverState, so
that any read/writes (and possibly more requests, if necessary) are routed
through the filter before they are passed to the block driver of this BDS.
Filters would
Il 23/11/2012 10:05, Dietmar Maurer ha scritto:
My plan was to have something like bs-job-job_type-
{before,after}_write.
int coroutine_fn (*before_write)(BlockDriverState *bs,
int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
void **cookie);
int coroutine_fn
BTW, will such filters work with the new virtio-blk-data-plane?
No, virtio-blk-data-plane is a hack and will be slowly rewritten to support
all
fancy features.
Ah, good to know ;-) thanks.
Il 23/11/2012 08:42, Dietmar Maurer ha scritto:
My plan was to have something like bs-job-job_type-{before,after}_write.
int coroutine_fn (*before_write)(BlockDriverState *bs,
int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
void **cookie);
int
Filters would be implemented as BlockDrivers, i.e. you could
implement
.bdrv_co_write() in a filter to intercept all writes to an image.
I am quite unsure if that make things easier.
At least it would make for a much cleaner design compared to putting code
for every feature you can
Actually this was plan B, as a poor-man implementation of the filter
infrastructure. Plan A was that the block filters would materialize suddenly
in
someone's git tree.
OK, so let us summarize the options:
a.) wait untit it materialize suddenly in someone's git tree.
b.) add BlockFilter
Filters would be implemented as BlockDrivers, i.e. you could
implement
.bdrv_co_write() in a filter to intercept all writes to an image.
I am quite unsure if that make things easier.
At least it would make for a much cleaner design compared to putting
code for every feature
Am 23.11.2012 10:05, schrieb Dietmar Maurer:
My plan was to have something like bs-job-job_type-
{before,after}_write.
int coroutine_fn (*before_write)(BlockDriverState *bs,
int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
void **cookie);
int coroutine_fn
Am 23.11.2012 10:31, schrieb Dietmar Maurer:
Filters would be implemented as BlockDrivers, i.e. you could
implement
.bdrv_co_write() in a filter to intercept all writes to an image.
I am quite unsure if that make things easier.
At least it would make for a much cleaner design compared to
Kevin Wolf kw...@redhat.com writes:
Am 23.11.2012 10:05, schrieb Dietmar Maurer:
My plan was to have something like bs-job-job_type-
{before,after}_write.
int coroutine_fn (*before_write)(BlockDriverState *bs,
int64_t sector_num, int nb_sectors, QEMUIOVector *qiov,
void
On Wed, Nov 21, 2012 at 10:01:00AM +0100, Dietmar Maurer wrote:
+==Disadvantages==
+
+* we need to define a new archive format
+
+Note: Most existing archive formats are optimized to store small files
+including file attributes. We simply do not need that for VM archives.
Did you look at
Did you look at the VMDK Stream-Optimized Compressed subformat?
http://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf?
src=vmdk
It is a stream of compressed grains (data). They are out-of-order and each
grain comes with the virtual disk lba where the data should be visible to
Did you look at the VMDK Stream-Optimized Compressed subformat?
http://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf?
src=vmdk
Max file size 2TB?
It is a stream of compressed grains (data). They are out-of-order and each
grain comes with the virtual disk lba where the data should be visible to the
guest.
The stream also contains grain tables and grain directories. This
metadata makes random read access to the file possible once you
Did you look at the VMDK Stream-Optimized Compressed subformat?
http://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf?
src=vmdk
And is that covered by any patents?
On Thu, Nov 22, 2012 at 11:26:21AM +, Dietmar Maurer wrote:
Did you look at the VMDK Stream-Optimized Compressed subformat?
http://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf?
src=vmdk
It is a stream of compressed grains (data). They are out-of-order and
each
-Original Message-
From: Stefan Hajnoczi [mailto:stefa...@gmail.com]
Sent: Donnerstag, 22. November 2012 13:45
To: Dietmar Maurer
Cc: qemu-devel@nongnu.org; kw...@redhat.com
Subject: Re: [Qemu-devel] [PATCH 1/5] RFC: Efficient VM backup for qemu
(v1)
On Thu, Nov 22, 2012 at 11:26
/5] RFC: Efficient VM backup for qemu
(v1)
On Thu, Nov 22, 2012 at 11:26:21AM +, Dietmar Maurer wrote:
Did you look at the VMDK Stream-Optimized Compressed subformat?
http://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf?
src=vmdk
It is a stream of compressed
On Thu, Nov 22, 2012 at 12:40 PM, Dietmar Maurer diet...@proxmox.com wrote:
Did you look at the VMDK Stream-Optimized Compressed subformat?
http://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf?
src=vmdk
Max file size 2TB?
That is for a single .vmdk file. But vmdks can also be
On Thu, Nov 22, 2012 at 1:00 PM, Dietmar Maurer diet...@proxmox.com wrote:
It is a stream of compressed grains (data). They are out-of-order and each
grain comes with the virtual disk lba where the data should be visible to the
guest.
The stream also contains grain tables and grain
It's a naughty thing to do but we could simply pick a new constant and
support LZO as an incompatible option. The file is then no longer compatible
with existing vmdk tools but at least we then have a choice of using
compatible deflate or the LZO extension.
To be 100% incompatible to
The documents says: VMware products are covered by one or more
patents
listed at http://www.vmware.com/go/patents
I simply do not have the time to check all those things, which make that
format unusable for me.
In think proxmox ships the QEMU vmdk functionality today? In that case
On Thu, Nov 22, 2012 at 4:56 PM, Dietmar Maurer diet...@proxmox.com wrote:
It's a naughty thing to do but we could simply pick a new constant and
support LZO as an incompatible option. The file is then no longer compatible
with existing vmdk tools but at least we then have a choice of using
On Thu, Nov 22, 2012 at 4:58 PM, Dietmar Maurer diet...@proxmox.com wrote:
The documents says: VMware products are covered by one or more
patents
listed at http://www.vmware.com/go/patents
I simply do not have the time to check all those things, which make that
format unusable for me.
On Thu, Nov 22, 2012 at 12:12 PM, Stefan Hajnoczi stefa...@gmail.com wrote:
On Wed, Nov 21, 2012 at 10:01:00AM +0100, Dietmar Maurer wrote:
+==Disadvantages==
+
+* we need to define a new archive format
+
+Note: Most existing archive formats are optimized to store small files
+including
-Original Message-
From: Stefan Hajnoczi [mailto:stefa...@gmail.com]
Sent: Donnerstag, 22. November 2012 18:02
To: Dietmar Maurer
Cc: kw...@redhat.com; qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] [PATCH 1/5] RFC: Efficient VM backup for qemu
(v1)
On Thu, Nov 22, 2012 at 4
Did you look at the VMDK Stream-Optimized Compressed subformat?
We've gone down several sub-threads discussing whether VMDK is suitable.
I want to summarize why this is a good approach:
The VMDK format already allows for out-of-order data and is supported by
existing tools - this is very
The VMDK format has strong disadvantages:
- unclear License (the spec links to patents)
- they use a very slow compression algorithm (deflate), which makes it
unusable for backup
Seems they do not support multiple configuration files. You can only
a single text block, and that needs to
QEMU's implementation has partial support for Stream-Optimized
Compressed images. If you complete the code for this subformat, not only
does this benefit the VM Backup feature, but it also makes qemu-img convert
more powerful for everyone. I hope we can kill two birds with one stone
The doc
The VMDK format already allows for out-of-order data and is supported by
existing tools - this is very important for backups where people are
(rightfully) paranoid about putting their backups in an obscure format. They
want to be able to access their data years later, whether your tool is
On Thu, Nov 22, 2012 at 7:05 PM, Dietmar Maurer diet...@proxmox.com wrote:
QEMU's implementation has partial support for Stream-Optimized
Compressed images. If you complete the code for this subformat, not only
does this benefit the VM Backup feature, but it also makes qemu-img convert
more
On Thu, Nov 22, 2012 at 6:50 PM, Dietmar Maurer diet...@proxmox.com wrote:
The VMDK format has strong disadvantages:
- unclear License (the spec links to patents)
- they use a very slow compression algorithm (deflate), which makes it
unusable for backup
Seems they do not support multiple
On Thu, Nov 22, 2012 at 6:46 PM, Dietmar Maurer diet...@proxmox.com wrote:
Did you look at the VMDK Stream-Optimized Compressed subformat?
We've gone down several sub-threads discussing whether VMDK is suitable.
I want to summarize why this is a good approach:
The VMDK format already allows
On Fri, Nov 23, 2012 at 6:23 AM, Stefan Hajnoczi stefa...@gmail.com wrote:
On Thu, Nov 22, 2012 at 6:46 PM, Dietmar Maurer diet...@proxmox.com wrote:
Did you look at the VMDK Stream-Optimized Compressed subformat?
We've gone down several sub-threads discussing whether VMDK is suitable.
I
But keep in mind that any other company out there could have a patent on
out-of-order data in an image file or other aspects of what you're proposing.
Sorry, but the vmware docs explicitly include a pointer to those patents. So
this
is something completely different to me.
The VMDK format has strong disadvantages:
- unclear License (the spec links to patents)
I've already pointed out that you're taking an inconsistent position on this
point. It's FUD.
- they use a very slow compression algorithm (deflate), which makes it
unusable for backup
I've
To make progress here I'll review the RFC patches. VMDK or not isn't the
main thing, a backup feature like this looks interesting.
Yes, a 'review' would be great - thanks.
- Dietmar
In short, the idea is that you can stick filters on top of a
BlockDriverState, so
that any read/writes (and possibly more requests, if necessary) are routed
through the filter before they are passed to the block driver of this BDS.
Filters would be implemented as BlockDrivers, i.e. you could
Yup, it's already not too bad. I haven't looked into it in much
detail, but I'd like to reduce it even a bit more. In particular, the
backup_info field in the BlockDriverState feels wrong to me. In the
long term the generic block layer shouldn't know at all what a backup
is, and baking
This series provides a way to efficiently backup VMs.
* Backup to a single archive file
* Backup contain all data to restore VM (full backup)
* Do not depend on storage type or image format
* Avoid use of temporary storage
* store sparse images efficiently
The file docs/backup-rfc.txt contains
Am 21.11.2012 10:01, schrieb Dietmar Maurer:
+Some storage types/formats supports internal snapshots using some kind
+of reference counting (rados, sheepdog, dm-thin, qcow2). It would be possible
+to use that for backups, but for now we want to be storage-independent.
+
+Note: It turned out
+Note: It turned out that taking a qcow2 snapshot can take a very long
+time on larger files.
Hm, really? What are larger files? It has always been relatively quick when
I
tested it, though internal snapshots are not my focus, so that need not mean
much.
300GB or larger
If this is
Not saying that this is necessarily the best option, but I think reusing
existing
formats and implementation is always a good thing, so it's an idea to
consider.
Yes, I would really like to reuse something. Our current backup software uses
'tar' files,
but that is really inefficient. We
Am 21.11.2012 12:10, schrieb Dietmar Maurer:
+Note: It turned out that taking a qcow2 snapshot can take a very long
+time on larger files.
Hm, really? What are larger files? It has always been relatively quick
when I
tested it, though internal snapshots are not my focus, so that need not
Il 21/11/2012 13:37, Kevin Wolf ha scritto:
The active part is a bit more tricky. You're putting some code into
block.c to
achieve it, which is kind of ugly.
yes. but I tried to keep that small ;-)
Yup, it's already not too bad. I haven't looked into it in much detail,
but I'd like
AFAIK qcow2 file cannot store data out of order. In general, an backup
fd is not seekable, and we only want to do sequential writes. Image format
always requires seekable fds?
Ah, this is what you mean by out of order. Just out of curiosity, what are
these non-seekable backup fds usually?
Am 21.11.2012 14:25, schrieb Dietmar Maurer:
AFAIK qcow2 file cannot store data out of order. In general, an backup
fd is not seekable, and we only want to do sequential writes. Image format
always requires seekable fds?
Ah, this is what you mean by out of order. Just out of curiosity, what
Ah, this is what you mean by out of order. Just out of curiosity,
what are these non-seekable backup fds usually?
/dev/nst0 ;-)
Sure. :-)
But there are better examples. Usually you want to use some kind of
compression, and you do that with existing tools:
# backup to
61 matches
Mail list logo
