Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

[Vladimir Sementsov-Ogievskiy]

20.01.2018 03:37, John Snow wrote:


On 01/19/2018 01:08 PM, Vladimir Sementsov-Ogievskiy wrote:

18.01.2018 12:57, Vladimir Sementsov-Ogievskiy wrote:

17.01.2018 21:30, John Snow wrote:

On 12/28/2017 06:49 AM, Vladimir Sementsov-Ogievskiy wrote:

Thank you for reviewing my code!


Time for the re-spin? There's pretty good pressure to get this into 2.12
and say the non-nbd workflow model is feature complete.

Yes, you are right. Hope to do it today or tomorrow.


I've rebased, applied comments from review, and even one some new fixes,
but now I understand that it is too early for re-spin.

Now this series depends on
1. [PATCH v2 0/3] fix bitmaps migration through shared storage
   - we need to decide, how to fix it. May be, we just do not need
bitmaps in inactive state, and should not load them in do_qcow2_open in
this case
   [I can ignore it, just dropping one test case from new iotest, and fix
it later, but..

I'll look at this next.


2. [PATCH v2 0/6] qmp dirty bitmap API
  - here autoload is dropped, and migration should be rebased on it.


For (2) I really want to see a test case showing the utility for enable,
disable and merge to be added to the API, *or* add an x- prefix to them
for now.


tests are inevitable, I'll make them)



I want to see some use-case tests that demonstrate how they are to be
safely used, basically. You might want to expand test 124 for this.

I'm not sure we need this entire series for migration, but I don't want
to make you re-order absolutely everything. Can we merge patch one by
itself for the purposes of the migration patchset?


I think yes, will try.


so, I'll re-spin migration after these two questions are resolved.


Thanks for your patience, as always. Please send me pings every day on
whatever you have to in order to get migration in to 2.12 -- I'm going
to try to stay focused too, but I've got the attention span of a
goldfish. If I miss something, please speak up.

--js



--
Best regards,
Vladimir

Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

[John Snow]

On 01/19/2018 01:08 PM, Vladimir Sementsov-Ogievskiy wrote:
> 18.01.2018 12:57, Vladimir Sementsov-Ogievskiy wrote:
>> 17.01.2018 21:30, John Snow wrote:
>>>
>>> On 12/28/2017 06:49 AM, Vladimir Sementsov-Ogievskiy wrote:
 Thank you for reviewing my code!

>>> Time for the re-spin? There's pretty good pressure to get this into 2.12
>>> and say the non-nbd workflow model is feature complete.
>>
>> Yes, you are right. Hope to do it today or tomorrow.
>>
> 
> I've rebased, applied comments from review, and even one some new fixes,
> but now I understand that it is too early for re-spin.
> 
> Now this series depends on
> 1. [PATCH v2 0/3] fix bitmaps migration through shared storage
>   - we need to decide, how to fix it. May be, we just do not need
> bitmaps in inactive state, and should not load them in do_qcow2_open in
> this case
>   [I can ignore it, just dropping one test case from new iotest, and fix
> it later, but..

I'll look at this next.

> 2. [PATCH v2 0/6] qmp dirty bitmap API
>  - here autoload is dropped, and migration should be rebased on it.
> 

For (2) I really want to see a test case showing the utility for enable,
disable and merge to be added to the API, *or* add an x- prefix to them
for now.

I want to see some use-case tests that demonstrate how they are to be
safely used, basically. You might want to expand test 124 for this.

I'm not sure we need this entire series for migration, but I don't want
to make you re-order absolutely everything. Can we merge patch one by
itself for the purposes of the migration patchset?

> so, I'll re-spin migration after these two questions are resolved.
> 

Thanks for your patience, as always. Please send me pings every day on
whatever you have to in order to get migration in to 2.12 -- I'm going
to try to stay focused too, but I've got the attention span of a
goldfish. If I miss something, please speak up.

--js

Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

[John Snow]

On 01/19/2018 01:08 PM, Vladimir Sementsov-Ogievskiy wrote:
> 18.01.2018 12:57, Vladimir Sementsov-Ogievskiy wrote:
>> 17.01.2018 21:30, John Snow wrote:
>>>
>>> On 12/28/2017 06:49 AM, Vladimir Sementsov-Ogievskiy wrote:
 Thank you for reviewing my code!

>>> Time for the re-spin? There's pretty good pressure to get this into 2.12
>>> and say the non-nbd workflow model is feature complete.
>>
>> Yes, you are right. Hope to do it today or tomorrow.
>>
> 
> I've rebased, applied comments from review, and even one some new fixes,
> but now I understand that it is too early for re-spin.
> 
> Now this series depends on
> 1. [PATCH v2 0/3] fix bitmaps migration through shared storage
>   - we need to decide, how to fix it. May be, we just do not need
> bitmaps in inactive state, and should not load them in do_qcow2_open in
> this case
>   [I can ignore it, just dropping one test case from new iotest, and fix
> it later, but..
> 2. [PATCH v2 0/6] qmp dirty bitmap API
>  - here autoload is dropped, and migration should be rebased on it.
> 
> so, I'll re-spin migration after these two questions are resolved.
> 

You got it, thanks.

Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

[Vladimir Sementsov-Ogievskiy]

18.01.2018 12:57, Vladimir Sementsov-Ogievskiy wrote:

17.01.2018 21:30, John Snow wrote:


On 12/28/2017 06:49 AM, Vladimir Sementsov-Ogievskiy wrote:

Thank you for reviewing my code!


Time for the re-spin? There's pretty good pressure to get this into 2.12
and say the non-nbd workflow model is feature complete.


Yes, you are right. Hope to do it today or tomorrow.



I've rebased, applied comments from review, and even one some new fixes, 
but now I understand that it is too early for re-spin.


Now this series depends on
1. [PATCH v2 0/3] fix bitmaps migration through shared storage
  - we need to decide, how to fix it. May be, we just do not need 
bitmaps in inactive state, and should not load them in do_qcow2_open in 
this case
  [I can ignore it, just dropping one test case from new iotest, and 
fix it later, but..

2. [PATCH v2 0/6] qmp dirty bitmap API
 - here autoload is dropped, and migration should be rebased on it.

so, I'll re-spin migration after these two questions are resolved.

--
Best regards,
Vladimir

Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

[Vladimir Sementsov-Ogievskiy]

17.01.2018 21:30, John Snow wrote:


On 12/28/2017 06:49 AM, Vladimir Sementsov-Ogievskiy wrote:

Thank you for reviewing my code!


Time for the re-spin? There's pretty good pressure to get this into 2.12
and say the non-nbd workflow model is feature complete.


Yes, you are right. Hope to do it today or tomorrow.

--
Best regards,
Vladimir

Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

[John Snow]

On 12/28/2017 06:49 AM, Vladimir Sementsov-Ogievskiy wrote:
> Thank you for reviewing my code!
> 

Time for the re-spin? There's pretty good pressure to get this into 2.12
and say the non-nbd workflow model is feature complete.

Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

[Fam Zheng]

On Wed, 12/20 18:49, Vladimir Sementsov-Ogievskiy wrote:
> Test
> - start two vms (vm_a, vm_b)
> 
> - in a
> - do writes from set A
> - do writes from set B
> - fix bitmap sha256
> - clear bitmap
> - do writes from set A
> - start migration
> - than, in b
> - wait vm start (postcopy should start)
> - do writes from set B
> - check bitmap sha256
> 
> The test should verify postcopy migration and then merging with delta
> (changes in target, during postcopy process).
> 
> Reduce supported cache modes to only 'none', because with cache on time
> from source.STOP to target.RESUME is unpredictable and we can fail with
> timout while waiting for target.RESUME.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
>  tests/qemu-iotests/199| 105 
> ++
>  tests/qemu-iotests/199.out|   5 ++
>  tests/qemu-iotests/group  |   1 +
>  tests/qemu-iotests/iotests.py |   7 ++-
>  4 files changed, 117 insertions(+), 1 deletion(-)
>  create mode 100755 tests/qemu-iotests/199
>  create mode 100644 tests/qemu-iotests/199.out
> 
> diff --git a/tests/qemu-iotests/199 b/tests/qemu-iotests/199
> new file mode 100755
> index 00..f872040a81
> --- /dev/null
> +++ b/tests/qemu-iotests/199
> @@ -0,0 +1,105 @@
> +#!/usr/bin/env python
> +#
> +# Tests for dirty bitmaps postcopy migration.
> +#
> +# Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see .
> +#
> +
> +import os
> +import iotests
> +import time
> +from iotests import qemu_img
> +
> +disk_a = os.path.join(iotests.test_dir, 'disk_a')
> +disk_b = os.path.join(iotests.test_dir, 'disk_b')
> +size = '256G'
> +fifo = os.path.join(iotests.test_dir, 'mig_fifo')
> +
> +class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
> +
> +def tearDown(self):
> +self.vm_a.shutdown()
> +self.vm_b.shutdown()
> +os.remove(disk_a)
> +os.remove(disk_b)
> +os.remove(fifo)
> +
> +def setUp(self):
> +os.mkfifo(fifo)
> +qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
> +qemu_img('create', '-f', iotests.imgfmt, disk_b, size)
> +self.vm_a = iotests.VM(path_suffix='a').add_drive(disk_a)
> +self.vm_b = iotests.VM(path_suffix='b').add_drive(disk_b)
> +self.vm_b.add_incoming("exec: cat '" + fifo + "'")
> +self.vm_a.launch()
> +self.vm_b.launch()
> +
> +def test_postcopy(self):
> +write_size = 0x4000
> +granularity = 512
> +chunk = 4096
> +
> +result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
> +   name='bitmap', granularity=granularity)
> +self.assert_qmp(result, 'return', {});
> +
> +s = 0
> +while s < write_size:
> +self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> +s += 0x1
> +s = 0x8000
> +while s < write_size:
> +self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> +s += 0x1
> +
> +result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256',
> +   node='drive0', name='bitmap')
> +sha256 = result['return']['sha256']
> +
> +result = self.vm_a.qmp('block-dirty-bitmap-clear', node='drive0',
> +   name='bitmap')
> +self.assert_qmp(result, 'return', {});
> +s = 0
> +while s < write_size:
> +self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> +s += 0x1
> +
> +result = self.vm_a.qmp('migrate-set-capabilities',
> +   capabilities=[{'capability': 'dirty-bitmaps',
> +  'state': True}])
> +self.assert_qmp(result, 'return', {})
> +
> +result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
> +self.assertNotEqual(self.vm_a.event_wait("STOP"), None)
> +self.assertNotEqual(self.vm_b.event_wait("RESUME"), None)
> +
> +s = 0x8000
> +while s < write_size:
> +self.vm_b.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> +s += 0x1
> +
> +result = self.vm_b.qmp('query-block');
> +while