Re: [Qemu-devel] [PATCH v6 02/10] qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove

2014-11-07 Thread Vladimir Sementsov-Ogievskiy
+if (!name || name[0] == '\0') { Isn't is better to move name[0] == '\0' check to bdrv_create_dirty_bitmap, near existed name checking? +if (granularity 512 || is_power_of_2(granularity)) { +error_setg(errp, Granularity must be power of 2 +

Re: [Qemu-devel] [PATCH v6 05/10] block: Add bdrv_copy_dirty_bitmap and bdrv_reset_dirty_bitmap

2014-11-07 Thread Vladimir Sementsov-Ogievskiy
from [PATCH v6 02/10] +void qmp_block_dirty_bitmap_remove(const char *device, const char *name, + Error **errp) +{ +BlockDriverState *bs; +BdrvDirtyBitmap *bitmap; + +bs = bdrv_find(device); +if (!bs) { +error_set(errp,

Re: [Qemu-devel] [PATCH v6 07/10] qmp: Add support of dirty-bitmap sync mode for drive-backup

2014-11-12 Thread Vladimir Sementsov-Ogievskiy
@@ -317,7 +321,21 @@ static void coroutine_fn backup_run(void *opaque) if (alloced == 0) { continue; } +} else if (job-sync_mode == MIRROR_SYNC_MODE_DIRTY_BITMAP) { +int i, dirty = 0; +for (i =

Re: [Qemu-devel] [PATCH v6 04/10] hbitmap: Add hbitmap_copy

2014-11-17 Thread Vladimir Sementsov-Ogievskiy
+ +HBitmap *hbitmap_copy(const HBitmap *bitmap) +{ +int i; +int64_t size; +HBitmap *hb = g_memdup(bitmap, sizeof(struct HBitmap)); + +size = bitmap-size; +for (i = HBITMAP_LEVELS; i-- 0; ) { +size = MAX((size + BITS_PER_LONG - 1) BITS_PER_LEVEL, 1); +

Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series

2014-11-18 Thread Vladimir Sementsov-Ogievskiy
. Maybe, save some of bitmap levels on timer while vm is running and save the last level on shutdown? CC qemu-devel - ok. Best regards, Vladimir On 18.11.2014 02:46, John Snow wrote: On 11/13/2014 08:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi I'd just like to start working on persistent

Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series

2014-11-18 Thread Vladimir Sementsov-Ogievskiy
timer will not be co-routine based? Best regards, Vladimir On 18.11.2014 13:54, Vladimir Sementsov-Ogievskiy wrote: (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap

Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series

2014-11-18 Thread Vladimir Sementsov-Ogievskiy
, is locking required? Or sync timer will not be co-routine based? Best regards, Vladimir On 18.11.2014 13:54, Vladimir Sementsov-Ogievskiy wrote: (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data

[Qemu-devel] [PATCH v2] persistent dirty bitmap: add QDB file spec.

2014-11-20 Thread Vladimir Sementsov-Ogievskiy
use it here or not is still questionable. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- docs/specs/qdb.txt | 132 + 1 file changed, 132 insertions(+) create mode 100644 docs/specs/qdb.txt diff --git a/docs/specs

Re: [Qemu-devel] [PATCH v2] persistent dirty bitmap: add QDB file spec.

2014-11-20 Thread Vladimir Sementsov-Ogievskiy
Also, it may be better to make this as qcow2 extension. And bitmap will be saved in separate qcow2 file, which will contain only the bitmap(s) and no other data (no disk, no snapshots). Best regards, Vladimir On 20.11.2014 13:34, Vladimir Sementsov-Ogievskiy wrote: QDB file is for storing

Re: [Qemu-devel] [PATCH v2] persistent dirty bitmap: add QDB file spec.

2014-11-21 Thread Vladimir Sementsov-Ogievskiy
common procedure with cow or something like this.. Best regards, Vladimir On 20.11.2014 14:36, Stefan Hajnoczi wrote: On Thu, Nov 20, 2014 at 01:41:14PM +0300, Vladimir Sementsov-Ogievskiy wrote: Also, it may be better to make this as qcow2 extension. And bitmap will be saved in separate qcow2

Re: [Qemu-devel] [PATCH v2] persistent dirty bitmap: add QDB file spec.

2014-11-21 Thread Vladimir Sementsov-Ogievskiy
bitmap and it's snapshot used by backup. Best regards, Vladimir On 20.11.2014 14:36, Stefan Hajnoczi wrote: On Thu, Nov 20, 2014 at 01:41:14PM +0300, Vladimir Sementsov-Ogievskiy wrote: Also, it may be better to make this as qcow2 extension. And bitmap will be saved in separate qcow2 file

Re: [Qemu-devel] [PATCH v2] persistent dirty bitmap: add QDB file spec.

2014-11-24 Thread Vladimir Sementsov-Ogievskiy
not be connected with bitmap file and it's format. Best regards, Vladimir On 21.11.2014 19:55, Stefan Hajnoczi wrote: On Fri, Nov 21, 2014 at 01:27:40PM +0300, Vladimir Sementsov-Ogievskiy wrote: There is a constraint if we want to get live migration for free: The bitmap contents must

Re: [Qemu-devel] [PATCH v2] persistent dirty bitmap: add QDB file spec.

2014-11-25 Thread Vladimir Sementsov-Ogievskiy
it not touching other bitmaps.. Am I wrong? Best regards, Vladimir On 21.11.2014 19:55, Stefan Hajnoczi wrote: On Fri, Nov 21, 2014 at 01:27:40PM +0300, Vladimir Sementsov-Ogievskiy wrote: There is a constraint if we want to get live migration for free: The bitmap contents must be accessible

[Qemu-devel] [PATCH RFC] block: fix spoiling all dirty bitmaps by mirror and migration

2014-11-26 Thread Vladimir Sementsov-Ogievskiy
,reset)_dirty are made static, for internal block usage. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block-migration.c | 4 ++-- block.c | 23 --- block/mirror.c| 8 include/block/block.h | 6 -- 4 files

[Qemu-devel] [PATCH v2] block: fix spoiling all dirty bitmaps by mirror and migration

2014-11-27 Thread Vladimir Sementsov-Ogievskiy
,reset)_dirty are made static, for internal block usage. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com CC: John Snow js...@redhat.com CC: Fam Zheng f...@redhat.com CC: Denis V. Lunev d...@openvz.org CC: Stefan Hajnoczi stefa...@redhat.com CC: Kevin Wolf kw...@redhat.com

Re: [Qemu-devel] [PATCH v2] persistent dirty bitmap: add QDB file spec.

2014-11-28 Thread Vladimir Sementsov-Ogievskiy
on persistence of dirty bitmaps or persistent bitmap file format. Best regards, Vladimir On 21.11.2014 19:55, Stefan Hajnoczi wrote: On Fri, Nov 21, 2014 at 01:27:40PM +0300, Vladimir Sementsov-Ogievskiy wrote: There is a constraint if we want to get live migration for free: The bitmap contents must

Re: [Qemu-devel] [PATCH v9 05/10] block: Add bdrv_copy_dirty_bitmap and bdrv_reset_dirty_bitmap

2014-12-10 Thread Vladimir Sementsov-Ogievskiy
about naming: We will need functions for set/unset a subregion of BdrvDirtyBitmap, to fix migration and mirror (accordingly to my [PATCH v2] block: fix spoiling all dirty bitmaps by mirror and migration). Having the function 'bdrv_reset_dirty_bitmap' from this patch, we'll have to add

[Qemu-devel] [PATCH] block-migration: fix pending() return value

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
max_size, we will skip transferring the tail of the device. Currently we have set pending to BLOCK_SIZE if it is zero for bulk phase, but there no guarantee, that it will be max_size. True approach is to return, for example, max_size+1 when we are in the bulk phase. Signed-off-by: Vladimir Sementsov

[Qemu-devel] [PATCH] Fix block migration bug

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
' status. The following patch fixes this bug. Vladimir Sementsov-Ogievskiy (1): block-migration: fix pending() return value block-migration.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 1.9.1

[Qemu-devel] [PATCH 3/9] block: fix spoiling all dirty bitmaps by mirror and migration

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
,reset)_dirty are made static, for internal block usage. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com CC: John Snow js...@redhat.com CC: Fam Zheng f...@redhat.com CC: Denis V. Lunev d...@openvz.org CC: Stefan Hajnoczi stefa...@redhat.com CC: Kevin Wolf kw...@redhat.com

[Qemu-devel] [PATCH 4/9] hbitmap: store / restore

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
, HBitmap is inconsistent while restoring. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- include/qemu/hbitmap.h | 49 + util/hbitmap.c | 84 ++ 2 files changed, 133 insertions(+) diff --git

[Qemu-devel] [PATCH 1/9] block: rename bdrv_reset_dirty_bitmap

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
, uint64_t start, uint64_t count) for more transparent access to underlaying hbitmap interface. So, here we rename the considered function to 'bdrv_clear_dirty_bitmap'. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 2 +- block/backup.c| 2

[Qemu-devel] [PATCH 2/9] block-migration: fix pending() return value

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
max_size, we will skip transferring the tail of the device. Currently we have set pending to BLOCK_SIZE if it is zero for bulk phase, but there no guarantee, that it will be max_size. True approach is to return, for example, max_size+1 when we are in the bulk phase. Signed-off-by: Vladimir Sementsov

[Qemu-devel] [PATCH 5/9] block: BdrvDirtyBitmap store/restore interface

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
Several functions to provide necessary access to BdrvDirtyBitmap for block-migration.c Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 61 +-- include/block/block.h | 10 + 2 files changed

[Qemu-devel] [PATCH 6/9] block-migration: tiny refactoring

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
Add blk_create and blk_free to remove code duplicates. Otherwise, duplicates will rise in the following patches because of BlkMigBlock sturcture extendin. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block-migration.c | 56

[Qemu-devel] [PATCH 7/9] block-migration: remove not needed iothread lock

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
Instead of locking iothread, we can just swap these calls. So, if some write to our range occures before resetting the bitmap, then it will get into subsequent aio read, becouse it occures, in any case, after resetting the bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement

[Qemu-devel] [PATCH 0/8] Dirty bitmaps migration

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
is true block migration is actually skipped: no allocatoions, no bdrv_read's, no bdrv_write's, only bitmaps are migrated. The patch set includes two my previous bug fixes, which are necessary for it. The patch set is based on Incremental backup series by John Snow. Vladimir Sementsov-Ogievskiy (8

[Qemu-devel] [PATCH 9/9] block-migration: add named dirty bitmaps migration

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
Sementsov-Ogievskiy vsement...@parallels.com --- block-migration.c | 173 +++--- savevm.c | 1 + 2 files changed, 154 insertions(+), 20 deletions(-) diff --git a/block-migration.c b/block-migration.c index 908a66d..95d54a1 100644

[Qemu-devel] [PATCH 8/9] migration: add dirty parameter

2014-12-11 Thread Vladimir Sementsov-Ogievskiy
Add dirty parameter to qmp-migrate command. If this parameter is true, block-migration.c will migrate dirty bitmaps. This parameter can be used without blk parameter to migrate only dirty bitmaps, skipping block migration. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com

Re: [Qemu-devel] [PATCH 8/9] migration: add dirty parameter

2014-12-15 Thread Vladimir Sementsov-Ogievskiy
Thanks. It's interesting that checkpatch.pl doesn't warn about this line. Best regards, Vladimir On 11.12.2014 18:18, Eric Blake wrote: On 12/11/2014 07:17 AM, Vladimir Sementsov-Ogievskiy wrote: Add dirty parameter to qmp-migrate command. If this parameter is true, block-migration.c

Re: [Qemu-devel] [PATCH v2] block: fix spoiling all dirty bitmaps by mirror and migration

2014-12-15 Thread Vladimir Sementsov-Ogievskiy
ping v2 just fixes over 80 characters lines mentioned by Fam Zheng in previous version of this patch: [PATCH RFC] block: fix spoiling all dirty bitmaps by mirror and migration Best regards, Vladimir On 27.11.2014 12:40, Vladimir Sementsov-Ogievskiy wrote: Mirror and migration use dirty

Re: [Qemu-devel] [PATCH] block-migration: fix pending() return value

2014-12-19 Thread Vladimir Sementsov-Ogievskiy
Hi there.. Can someone review my bug fix? I'll surely fix typos in description after it. Best regards, Vladimir On 11.12.2014 18:07, Eric Blake wrote: On 12/11/2014 04:55 AM, Vladimir Sementsov-Ogievskiy wrote: Because of wrong return value of .save_live_pending() in block-migration

Re: [Qemu-devel] [PATCH v10 00/13] block: Incremental backup series (RFC)

2014-12-23 Thread Vladimir Sementsov-Ogievskiy
On 23.12.2014 04:12, John Snow wrote: For convenience, this patchset is available on github: https://github.com/jnsnow/qemu/commits/dbm-backup - old version here.

[Qemu-devel] [PATCH v2 1/1] migration/block: fix pending() return value

2014-12-30 Thread Vladimir Sementsov-Ogievskiy
Sementsov-Ogievskiy vsement...@parallels.com --- migration/block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/migration/block.c b/migration/block.c index 74d9eb1..2e92605 100644 --- a/migration/block.c +++ b/migration/block.c @@ -765,8 +765,8 @@ static uint64_t

[Qemu-devel] [PATCH v2 0/1] Fix block migration bug

2014-12-30 Thread Vladimir Sementsov-Ogievskiy
' it finishes with 'Pattern verification failed' status. The following patch fixes this bug. Vladimir Sementsov-Ogievskiy (1): migration/block: fix pending() return value migration/block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 1.9.1

Re: [Qemu-devel] [PATCH v10 06/13] block: Add bdrv_copy_dirty_bitmap and bdrv_clear_dirty_bitmap

2014-12-30 Thread Vladimir Sementsov-Ogievskiy
I'm sorry if it was already discussed, but I think it is inconsistent to have size in sectors and granularity in bytes in one structure. I've misused these fields because of this in my current work. At least, I think there should be comments about this. Best regards, Vladimir On 23.12.2014

Re: [Qemu-devel] [PATCH 0/8] block: persistent dirty bitmaps (RFC)

2015-02-04 Thread Vladimir Sementsov-Ogievskiy
On 13.01.2015 20:02, Vladimir Sementsov-Ogievskiy wrote: The bitmaps are saved into qcow2 file format. It provides both 'internal' and 'external' dirty bitmaps feature: - for qcow2 drives we can store bitmaps in the same file - for other formats we can store bitmaps in the separate qcow2

Re: [Qemu-devel] [PATCH RFC v2 7/8] migration: add dirty parameter

2015-02-04 Thread Vladimir Sementsov-Ogievskiy
On 27.01.2015 19:20, Eric Blake wrote: On 01/27/2015 03:56 AM, Vladimir Sementsov-Ogievskiy wrote: Add dirty parameter to qmp-migrate command. If this parameter is true, migration/block.c will migrate dirty bitmaps. This parameter can be used without blk parameter to migrate only dirty bitmaps

Re: [Qemu-devel] [PATCH 0/8] block: persistent dirty bitmaps (RFC)

2015-02-04 Thread Vladimir Sementsov-Ogievskiy
On 04.02.2015 18:20, John Snow wrote: On 02/04/2015 10:07 AM, Vladimir Sementsov-Ogievskiy wrote: On 13.01.2015 20:02, Vladimir Sementsov-Ogievskiy wrote: The bitmaps are saved into qcow2 file format. It provides both 'internal' and 'external' dirty bitmaps feature: - for qcow2 drives we

Re: [Qemu-devel] [PATCH v11 12/13] qemu-iotests: Add tests for drive-backup sync=dirty-bitmap

2015-02-06 Thread Vladimir Sementsov-Ogievskiy
On 12.01.2015 19:31, John Snow wrote: From: Fam Zheng f...@redhat.com Signed-off-by: Fam Zheng f...@redhat.com Signed-off-by: John Snow js...@redhat.com --- tests/qemu-iotests/056| 33 ++--- tests/qemu-iotests/056.out| 4 ++--

Re: [Qemu-devel] [PATCH 7/9] block-migration: remove not needed iothread lock

2015-01-16 Thread Vladimir Sementsov-Ogievskiy
On 09.01.2015 00:24, John Snow wrote: On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Instead of locking iothread, we can just swap these calls. So, if some write to our range occures before resetting the bitmap, then it will get into subsequent aio read, becouse it occures

Re: [Qemu-devel] [PATCH 7/9] block-migration: remove not needed iothread lock

2015-01-16 Thread Vladimir Sementsov-Ogievskiy
Best regards, Vladimir On 09.01.2015 01:28, Paolo Bonzini wrote: On 11/12/2014 15:17, Vladimir Sementsov-Ogievskiy wrote: -qemu_mutex_lock_iothread(); +bdrv_reset_dirty_bitmap(bs, bmds-dirty_bitmap, cur_sector, nr_sectors); + blk-aiocb = bdrv_aio_readv(bs, cur_sector, blk-qiov

Re: [Qemu-devel] [PATCH 9/9] block-migration: add named dirty bitmaps migration

2015-01-17 Thread Vladimir Sementsov-Ogievskiy
Best regards, Vladimir On 09.01.2015 01:05, John Snow wrote: CCing migration maintainers, feedback otherwise in-line. On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Just migrate parts of dirty bitmaps, corresponding to the block being migrated. Also, skip block migration

[Qemu-devel] RFC RFC

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Hi Can anybody explain what the reasons, politics and consequences of adding RFC into patch header in this mailing list? I think, it is not worth to add this information to http://wiki.qemu.org/Contribute/SubmitAPatch -- Best regards, Vladimir

Re: [Qemu-devel] [PATCH RFC v2 4/8] block: add dirty-dirty bitmaps

2015-02-12 Thread Vladimir Sementsov-Ogievskiy
On 11.02.2015 00:30, John Snow wrote: I had hoped it wouldn't come to this :) On 01/27/2015 05:56 AM, Vladimir Sementsov-Ogievskiy wrote: A dirty-dirty bitmap is a dirty bitmap for BdrvDirtyBitmap. It tracks set/unset changes of BdrvDirtyBitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy

Re: [Qemu-devel] [PATCH RFC v2 5/8] block: add bdrv_dirty_bitmap_enabled()

2015-02-12 Thread Vladimir Sementsov-Ogievskiy
On 11.02.2015 00:30, John Snow wrote: On 01/27/2015 05:56 AM, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 5 + include/block/block.h | 1 + 2 files changed, 6 insertions(+) diff --git a/block.c b

Re: [Qemu-devel] [PATCH RFC v2 8/8] migration: add migration/dirty-bitmap.c

2015-02-13 Thread Vladimir Sementsov-Ogievskiy
+ +blk_mig_reset_dirty_cursor(); +dirty_phase(f, false); + +QSIMPLEQ_FOREACH(dbms, dirty_bitmap_mig_state.dbms_list, entry) { +uint8_t flags = DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME | +DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME | +

Re: [Qemu-devel] [PATCH RFC v2 8/8] migration: add migration/dirty-bitmap.c

2015-02-13 Thread Vladimir Sementsov-Ogievskiy
/27/2015 05:56 AM, Vladimir Sementsov-Ogievskiy wrote: Live migration of dirty bitmaps. Only named dirty bitmaps are migrated. If destination qemu is already containing a dirty bitmap with the same name as a migrated bitmap, then their granularities should be the same, otherwise the error

Re: [Qemu-devel] [PATCH RFC v3 05/14] block: add meta bitmaps

2015-02-19 Thread Vladimir Sementsov-Ogievskiy
On 19.02.2015 02:45, John Snow wrote: On 02/18/2015 09:00 AM, Vladimir Sementsov-Ogievskiy wrote: Meta bitmap is a 'dirty bitmap' for the BdrvDirtyBitmap. It tracks changes (set/unset) of this BdrvDirtyBitmap. It is needed for live migration of block dirty bitmaps. Signed-off-by: Vladimir

Re: [Qemu-devel] [PATCH RFC v3 08/14] migration: add migration/block-dirty-bitmap.c

2015-02-19 Thread Vladimir Sementsov-Ogievskiy
On 19.02.2015 02:47, John Snow wrote: On 02/18/2015 09:00 AM, Vladimir Sementsov-Ogievskiy wrote: Live migration of dirty bitmaps. Only named dirty bitmaps, associated with root nodes and non-root named nodes are migrated. If destination qemu is already containing a dirty bitmap

Re: [Qemu-devel] [PATCH RFC v2 8/8] migration: add migration/dirty-bitmap.c

2015-02-17 Thread Vladimir Sementsov-Ogievskiy
On 16.02.2015 21:18, John Snow wrote: On 02/16/2015 07:06 AM, Vladimir Sementsov-Ogievskiy wrote: On 13.02.2015 23:22, John Snow wrote: On 02/13/2015 03:19 AM, Vladimir Sementsov-Ogievskiy wrote: On 11.02.2015 00:33, John Snow wrote: So in summary: using device names is probably fine

[Qemu-devel] [PATCH RFC v3 13/14] iotests: add dirty bitmap migration test

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
bitmaps. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- tests/qemu-iotests/118 | 87 ++ tests/qemu-iotests/118.out | 5 +++ tests/qemu-iotests/group | 1 + 3 files changed, 93 insertions(+) create mode 100755 tests/qemu

[Qemu-devel] [PATCH RFC v3 09/14] iotests: maintain several vms in test

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
The only problem with it is the same qmp socket name (which is vm._monitor_path) for all vms. And because of this second vm couldn't be lauched (vm.launch() fails because of socket is already in use). This patch adds a number of vm into vm._monitor_path Signed-off-by: Vladimir Sementsov-Ogievskiy

[Qemu-devel] [PATCH RFC v3 03/14] block: BdrvDirtyBitmap serialization interface

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Several functions to provide necessary access to BdrvDirtyBitmap for block-migration.c Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 36 include/block/block.h | 13 + 2 files changed, 49

[Qemu-devel] [PATCH RFC v3 04/14] block: tiny refactoring: minimize hbitmap_(set/reset) usage

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/block.c b/block.c index e95a5ae..a127fd2 100644 --- a/block.c +++ b/block.c @@ -5670,8 +5670,7 @@ void bdrv_reset_dirty_bitmap

[Qemu-devel] [PATCH RFC v3 07/14] qapi: add dirty-bitmaps migration capability

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- include/migration/migration.h | 1 + migration/migration.c | 9 + qapi-schema.json | 5 - 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/include/migration/migration.h b

[Qemu-devel] [PATCH RFC v3 12/14] qapi: add md5 checksum of last dirty bitmap level to query-block

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c| 1 + include/qemu/hbitmap.h | 8 qapi/block-core.json | 4 +++- util/hbitmap.c | 8 4 files changed, 20 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index

[Qemu-devel] [PATCH RFC v3 10/14] iotests: add add_incoming_migration to VM class

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- tests/qemu-iotests/iotests.py | 6 ++ 1 file changed, 6 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index fa756b4..75640b2 100644 --- a/tests/qemu-iotests/iotests.py +++ b

[Qemu-devel] [PATCH RFC v3 14/14] migration/qemu-file: make functions qemu_(get/put)_string public

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- include/migration/qemu-file.h | 17 + migration/block-dirty-bitmap.c | 35 --- migration/qemu-file.c | 18 ++ 3 files changed, 35 insertions(+), 35

[Qemu-devel] [PATCH RFC v3 11/14] iotests: add dirty bitmap migration test

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
The test starts two vms (vm_a, vm_b), create dirty bitmap in the first one, do several writes to corresponding device and then migrate vm_a to vm_b with dirty bitmaps. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- tests/qemu-iotests/117 | 88

[Qemu-devel] [PATCH RFC v3 05/14] block: add meta bitmaps

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Meta bitmap is a 'dirty bitmap' for the BdrvDirtyBitmap. It tracks changes (set/unset) of this BdrvDirtyBitmap. It is needed for live migration of block dirty bitmaps. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 40

[Qemu-devel] [PATCH RFC v3 06/14] block: add bdrv_next_dirty_bitmap()

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Like bdrv_next() - bdrv_next_dirty_bitmap() is a function to provide access to private dirty bitmaps list. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 10 ++ include/block/block.h | 2 ++ 2 files changed, 12 insertions(+) diff --git

Re: [Qemu-devel] RFC RFC

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
On 18.02.2015 17:01, Eric Blake wrote: On 02/18/2015 01:10 AM, Vladimir Sementsov-Ogievskiy wrote: Hi Can anybody explain what the reasons, politics and consequences of adding RFC into patch header in this mailing list? I think, it is not worth to add this information to http://wiki.qemu.org

[Qemu-devel] [PATCH RFC v3 00/14] Dirty bitmaps migration

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
fixes, which are necessary for it. The patch set is based on Incremental backup series by John Snow. Vladimir Sementsov-Ogievskiy (14): qmp: add query-block-dirty-bitmap hbitmap: serialization block: BdrvDirtyBitmap serialization interface block: tiny refactoring: minimize hbitmap_(set

[Qemu-devel] [PATCH RFC v3 01/14] qmp: add query-block-dirty-bitmap

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
Adds qmp and hmp commands to query/info dirty bitmap. This is needed only for testing. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 41 blockdev.c| 24 + hmp-commands.hx | 2

[Qemu-devel] [PATCH RFC v3 08/14] migration: add migration/block-dirty-bitmap.c

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
the migration will be done, otherwise the error will be generated. If destination qemu doesn't contain such bitmap it will be created. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- include/migration/block.h | 1 + migration/Makefile.objs| 2 +- migration/block

[Qemu-devel] [PATCH RFC v3 02/14] hbitmap: serialization

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
in several steps is available. To save performance, every step writes only the last level of the bitmap. All other levels are restored by hbitmap_deserialize_finish() as a last step of restoring. So, HBitmap is inconsistent while restoring. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement

Re: [Qemu-devel] RFC RFC

2015-02-18 Thread Vladimir Sementsov-Ogievskiy
On 18.02.2015 17:01, Eric Blake wrote: On 02/18/2015 01:10 AM, Vladimir Sementsov-Ogievskiy wrote: Hi Can anybody explain what the reasons, politics and consequences of adding RFC into patch header in this mailing list? I think, it is not worth to add this information to http://wiki.qemu.org

[Qemu-devel] [PATCH RFC v2 3/8] block: BdrvDirtyBitmap serialization interface

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
Several functions to provide necessary access to BdrvDirtyBitmap for block-migration.c Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 36 include/block/block.h | 12 2 files changed, 48 insertions

[Qemu-devel] [PATCH RFC v2 4/8] block: add dirty-dirty bitmaps

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
A dirty-dirty bitmap is a dirty bitmap for BdrvDirtyBitmap. It tracks set/unset changes of BdrvDirtyBitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 44 include/block/block.h | 5 + 2 files

[Qemu-devel] [PATCH RFC v2 6/8] block: add bdrv_next_dirty_bitmap()

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
Like bdrv_next() - bdrv_next_dirty_bitmap() is a function to provide access to private dirty bitmaps list. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 10 ++ include/block/block.h | 2 ++ 2 files changed, 12 insertions(+) diff --git

[Qemu-devel] [PATCH RFC v2 7/8] migration: add dirty parameter

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
Add dirty parameter to qmp-migrate command. If this parameter is true, migration/block.c will migrate dirty bitmaps. This parameter can be used without blk parameter to migrate only dirty bitmaps, skipping block migration. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com

[Qemu-devel] [PATCH RFC v2 0/8] Dirty bitmaps migration

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
migration is actually skipped: no allocatoions, no bdrv_read's, no bdrv_write's, only bitmaps are migrated. The patch set includes two my previous bug fixes, which are necessary for it. The patch set is based on Incremental backup series by John Snow. Vladimir Sementsov-Ogievskiy (8): qmp: print

Re: [Qemu-devel] [PATCH 0/8] block: persistent dirty bitmaps (RFC)

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
ping Best regards, Vladimir On 13.01.2015 20:02, Vladimir Sementsov-Ogievskiy wrote: The bitmaps are saved into qcow2 file format. It provides both 'internal' and 'external' dirty bitmaps feature: - for qcow2 drives we can store bitmaps in the same file - for other formats we can store

[Qemu-devel] [PATCH RFC v2 1/8] qmp: print dirty bitmap

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
Adds qmp and hmp commands to print dirty bitmap. This is needed only for testing. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 33 + blockdev.c| 13 + hmp-commands.hx | 15

[Qemu-devel] [PATCH RFC v2 8/8] migration: add migration/dirty-bitmap.c

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
Sementsov-Ogievskiy vsement...@parallels.com --- include/migration/block.h | 1 + migration/Makefile.objs | 2 +- migration/dirty-bitmap.c | 606 ++ vl.c | 1 + 4 files changed, 609 insertions(+), 1 deletion(-) create mode

[Qemu-devel] [PATCH RFC v2 2/8] hbitmap: serialization

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
in several steps is available. To save performance, every step writes only the last level of the bitmap. All other levels are restored by hbitmap_deserialize_finish() as a last step of restoring. So, HBitmap is inconsistent while restoring. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement

[Qemu-devel] [PATCH RFC v2 5/8] block: add bdrv_dirty_bitmap_enabled()

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 5 + include/block/block.h | 1 + 2 files changed, 6 insertions(+) diff --git a/block.c b/block.c index 8ab724b..15fc621 100644 --- a/block.c +++ b/block.c @@ -5551,6 +5551,11 @@ uint64_t

Re: [Qemu-devel] [PATCH] block-migration: fix pending() return value

2015-01-29 Thread Vladimir Sementsov-Ogievskiy
v2 is already pushed http://git.qemu.org/?p=qemu.git;a=commit;h=04636dc410b163c2243e66c3813dd4900a50a4ed Best regards, Vladimir On 29.01.2015 10:58, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy vsement...@parallels.com writes: Hi there.. Can someone review my bug fix? I'll surely

Re: [Qemu-devel] [PATCH v11 03/13] qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove

2015-01-29 Thread Vladimir Sementsov-Ogievskiy
s/{'command'/{ 'command'/g - to be consistent with other staff in qapi/block-core.json and the same fix for block-dirty-bitmap-enable and block-dirty-bitmap-disable Best regards, Vladimir On 12.01.2015 19:30, John Snow wrote: From: Fam Zheng f...@redhat.com The new command pair is added

Re: [Qemu-devel] [PATCH RFC v2 1/8] qmp: print dirty bitmap

2015-01-27 Thread Vladimir Sementsov-Ogievskiy
Ok, I agree. It was just a simple way to test the other staff. I'll rewrite it in the following versions of my two series. Best regards, Vladimir On 27.01.2015 19:17, Eric Blake wrote: On 01/27/2015 03:56 AM, Vladimir Sementsov-Ogievskiy wrote: Adds qmp and hmp commands to print dirty bitmap

Re: [Qemu-devel] [PATCH 9/9] block-migration: add named dirty bitmaps migration

2015-01-12 Thread Vladimir Sementsov-Ogievskiy
Best regards, Vladimir On 09.01.2015 01:36, Paolo Bonzini wrote: The bitmaps are transmitted many times in their entirety, but only the last copy actually means something. The others are lost. This means you should use the non-live interface (register_savevm). This will simplify the code a

[Qemu-devel] [PATCH 1/8] spec: add qcow2-dirty-bitmaps specification

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
Persistent dirty bitmaps will be saved into qcow2 files. It may be used as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for other drives (there may be qcow2 file with zero disk size but with several dirty bitmaps for other drives). Signed-off-by: Vladimir Sementsov-Ogievskiy

[Qemu-devel] [PATCH 6/8] qemu: command line option for dirty bitmaps

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
). granularity The granularity for the bitmap. Not necessary, the default value may be used. enabled on|off. Default is 'on'. Disabled bitmaps are not changing regardless of writes to corresponding drive. Signed-off-by: Vladimir Sementsov-Ogievskiy

[Qemu-devel] [PATCH 4/8] block: store persistent dirty bitmaps

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
in bdrv_dirty_bitmap_set_file() only once. bdrv_ref/bdrv_unref are used for BdrvDirtyBitmap.file to be sure that files will be closed and resources will be freed. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 43

[Qemu-devel] [PATCH 0/8] block: persistent dirty bitmaps (RFC)

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
. Examples: qemu -drive file=a.qcow2,id=disk -dirty-bitmap name=b,drive=disk qemu -drive file=a.raw,id=disk \ -dirty-bitmap name=b,drive=disk,file=b.qcow2,enabled=off Vladimir Sementsov-Ogievskiy (8): spec: add qcow2-dirty-bitmaps specification hbitmap: store / restore qcow2: add dirty-bitmaps

[Qemu-devel] [PATCH 5/8] block: add bdrv_load_dirty_bitmap

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
The funcion loads dirty bitmap from file, using underlying driver function. Note: the function doesn't change BdrvDirtyBitmap.file field. This field is only used by bdrv_store_dirty_bitmap() function and is ONLY written by bdrv_dirty_bitmap_set_file() function. Signed-off-by: Vladimir Sementsov

[Qemu-devel] [PATCH 8/8] iotests: test internal persistent dirty bitmap

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
The test performs several vm reloads with checking and updating dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- tests/qemu-iotests/115 | 96 ++ tests/qemu-iotests/115.out | 64

[Qemu-devel] [PATCH 2/8] hbitmap: store / restore

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
of the bitmap. All other levels are restored by hbitmap_restore_finish as a last step of restoring. So, HBitmap is inconsistent while restoring. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- include/qemu/hbitmap.h | 49 util/hbitmap.c | 87

[Qemu-devel] [PATCH 3/8] qcow2: add dirty-bitmaps feature

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
Adds dirty-bitmaps feature to qcow2 format as specified in docs/specs/qcow2.txt Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block/Makefile.objs| 2 +- block/qcow2-dirty-bitmap.c | 514 + block/qcow2.c

[Qemu-devel] [PATCH 7/8] qmp: print dirty bitmap

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
Adds qmp and hmp commands to print dirty bitmap. This is needed only for testing persistent dirty bitmap feature. Signed-off-by: Vladimir Sementsov-Ogievskiy vsement...@parallels.com --- block.c | 33 + blockdev.c| 13 + hmp

Re: [Qemu-devel] [PATCH 9/9] block-migration: add named dirty bitmaps migration

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
On 12.01.2015 17:42, Paolo Bonzini wrote: On 12/01/2015 15:20, Vladimir Sementsov-Ogievskiy wrote: On 09.01.2015 01:36, Paolo Bonzini wrote: The bitmaps are transmitted many times in their entirety, but only the last copy actually means something. The others are lost. This means you should

Re: [Qemu-devel] [PATCH v10 06/13] block: Add bdrv_copy_dirty_bitmap and bdrv_clear_dirty_bitmap

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
Hmm, can't find it in v11 Best regards, Vladimir On 10.01.2015 06:25, John Snow wrote: On 12/30/2014 08:47 AM, Vladimir Sementsov-Ogievskiy wrote: I'm sorry if it was already discussed, but I think it is inconsistent to have size in sectors and granularity in bytes in one structure. I've

Re: [Qemu-devel] [PATCH 4/9] hbitmap: store / restore

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
Best regards, Vladimir On 09.01.2015 00:21, John Snow wrote: On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Functions to store / restore HBitmap. HBitmap should be saved to linear bitmap format independently of endianess. Because of restoring in several steps, every step

Re: [Qemu-devel] [PATCH v11 01/13] block: fix spoiling all dirty bitmaps by mirror and migration

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
it is already in master and can be dropped here Best regards, Vladimir On 12.01.2015 19:30, John Snow wrote: From: Vladimir Sementsov-Ogievskiy vsement...@parallels.com Mirror and migration use dirty bitmaps for their purposes, and since commit [block: per caller dirty bitmap] they use

Re: [Qemu-devel] [PATCH v10 06/13] block: Add bdrv_copy_dirty_bitmap and bdrv_clear_dirty_bitmap

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
wrote: On 01/13/2015 04:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hmm, can't find it in v11 Best regards, Vladimir I changed approach and instead of copying bitmaps to be used with the incremental backup, I lock them in place. So copy isn't used anymore in my set, so the name of the patch

Re: [Qemu-devel] [PATCH v11 13/13] block: BdrvDirtyBitmap miscellaneous fixup

2015-01-13 Thread Vladimir Sementsov-Ogievskiy
Hmm. May be, update the size field to be uint64_t too? Negative size is not meaningful.. Best regards, Vladimir On 12.01.2015 19:31, John Snow wrote: (1) granularity type consistency: Update the granularity to be uint64_t in all places. This value is always in bytes. (2) Some

Re: [Qemu-devel] [PATCH 4/9] hbitmap: store / restore

2015-01-14 Thread Vladimir Sementsov-Ogievskiy
Best regards, Vladimir On 13.01.2015 20:08, John Snow wrote: On 01/13/2015 07:59 AM, Vladimir Sementsov-Ogievskiy wrote: On 09.01.2015 00:21, John Snow wrote: On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: +/** + * hbitmap_restore_finish + * @hb: HBitmap to operate

Re: [Qemu-devel] [PATCH 5/9] block: BdrvDirtyBitmap store/restore interface

2015-01-14 Thread Vladimir Sementsov-Ogievskiy
As in previous patch, rename store/restore to serialize/deserialize... Hmm. In this case, isn't it be better to include serialization of granularity, name and name length in these functions? Best regards, Vladimir On 11.12.2014 17:17, Vladimir Sementsov-Ogievskiy wrote: Several functions

Re: [Qemu-devel] [PATCH 6/9] block-migration: tiny refactoring

2015-01-14 Thread Vladimir Sementsov-Ogievskiy
On 09.01.2015 00:23, John Snow wrote: On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Add blk_create and blk_free to remove code duplicates. Otherwise, duplicates will rise in the following patches because of BlkMigBlock sturcture extendin. Signed-off-by: Vladimir Sementsov

  1   2   3   4   5   6   7   8   9   10   >