Re: [Qemu-block] [Qemu-discuss] qemu-img convert stuck

On Thu, Apr 12, 2018 at 10:03 AM, Fam Zheng wrote: > On Thu, 04/12 09:51, David Lee wrote: > > On Mon, Apr 9, 2018 at 3:35 AM, Benny Zlotnik > wrote: > > > > > $ gdb -p 13024 -batch -ex "thread apply all bt" > > > [Thread debugging using libthread_db

Re: [Qemu-block] [Qemu-discuss] qemu-img convert stuck

On Thu, 04/12 09:51, David Lee wrote: > On Mon, Apr 9, 2018 at 3:35 AM, Benny Zlotnik wrote: > > > $ gdb -p 13024 -batch -ex "thread apply all bt" > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/lib64/libthread_db.so.1". > >

Re: [Qemu-block] [Qemu-discuss] qemu-img convert stuck

On Mon, Apr 9, 2018 at 3:35 AM, Benny Zlotnik wrote: > $ gdb -p 13024 -batch -ex "thread apply all bt" > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > 0x7f98275cfaff in ppoll () from /lib64/libc.so.6 > >

Re: [Qemu-block] v2v: -o rhv-upload: Long time spent zeroing the disk

On Tue, Apr 10, 2018 at 6:53 PM Richard W.M. Jones wrote: > On Tue, Apr 10, 2018 at 03:25:47PM +, Nir Soffer wrote: > > On Tue, Apr 10, 2018 at 5:50 PM Richard W.M. Jones > > wrote: > > > > > On Tue, Apr 10, 2018 at 02:07:33PM +, Nir Soffer wrote: >

Re: [Qemu-block] [Qemu-devel] [PATCH v4 11/13] block/mirror: Add active mirroring

On 04/11/2018 01:54 PM, Max Reitz wrote: > This patch implements active synchronous mirroring. In active mode, the > passive mechanism will still be in place and is used to copy all > initially dirty clusters off the source disk; but every write request > will write data both to the source and

[Qemu-block] [PATCH v4 08/13] test-hbitmap: Add non-advancing iter_next tests

Add a function that wraps hbitmap_iter_next() and always calls it in non-advancing mode first, and in advancing mode next. The result should always be the same. By using this function everywhere we called hbitmap_iter_next() before, we should get good test coverage for non-advancing

[Qemu-block] [PATCH v4 07/13] hbitmap: Add @advance param to hbitmap_iter_next()

This new parameter allows the caller to just query the next dirty position without moving the iterator. Signed-off-by: Max Reitz Reviewed-by: Fam Zheng Reviewed-by: John Snow --- include/qemu/hbitmap.h | 5 - block/backup.c |

[Qemu-block] [PATCH v4 06/13] block: Generalize should_update_child() rule

Currently, bdrv_replace_node() refuses to create loops from one BDS to itself if the BDS to be replaced is the backing node of the BDS to replace it: Say there is a node A and a node B. Replacing B by A means making all references to B point to A. If B is a child of A (i.e. A has a reference to

[Qemu-block] [PATCH v4 for-2.13 00/13] block/mirror: Add active-sync mirroring

This series implements an active and synchronous mirroring mode. Currently, the mirror block job is passive an asynchronous: Depending on your start conditions, some part of the source disk starts as "dirty". Then, the block job will (as a background operation) continuously copy dirty parts to

[Qemu-block] [PATCH v4 11/13] block/mirror: Add active mirroring

This patch implements active synchronous mirroring. In active mode, the passive mechanism will still be in place and is used to copy all initially dirty clusters off the source disk; but every write request will write data both to the source and the target disk, so the source cannot be dirtied

[Qemu-block] [PATCH v4 03/13] block/mirror: Use CoQueue to wait on in-flight ops

Attach a CoQueue to each in-flight operation so if we need to wait for any we can use it to wait instead of just blindly yielding and hoping for some operation to wake us. A later patch will use this infrastructure to allow requests accessing the same area of the virtual disk to specifically wait

[Qemu-block] [PATCH v4 02/13] block/mirror: Convert to coroutines

In order to talk to the source BDS (and maybe in the future to the target BDS as well) directly, we need to convert our existing AIO requests into coroutine I/O requests. Signed-off-by: Max Reitz Reviewed-by: Fam Zheng --- block/mirror.c | 152

[Qemu-block] [PATCH v4 04/13] block/mirror: Wait for in-flight op conflicts

This patch makes the mirror code differentiate between simply waiting for any operation to complete (mirror_wait_for_free_in_flight_slot()) and specifically waiting for all operations touching a certain range of the virtual disk to complete (mirror_wait_on_conflicts()). Signed-off-by: Max Reitz

[Qemu-block] [PATCH v4 13/13] iotests: Add test for active mirroring

Signed-off-by: Max Reitz Reviewed-by: Fam Zheng --- tests/qemu-iotests/151 | 120 + tests/qemu-iotests/151.out | 5 ++ tests/qemu-iotests/group | 1 + 3 files changed, 126 insertions(+) create mode 100755

[Qemu-block] [PATCH v4 10/13] block/mirror: Add MirrorBDSOpaque

This will allow us to access the block job data when the mirror block driver becomes more complex. Signed-off-by: Max Reitz Reviewed-by: Fam Zheng --- block/mirror.c | 12 1 file changed, 12 insertions(+) diff --git a/block/mirror.c

[Qemu-block] [PATCH v4 12/13] block/mirror: Add copy mode QAPI interface

This patch allows the user to specify whether to use active or only background mode for mirror block jobs. Currently, this setting will remain constant for the duration of the entire block job. Signed-off-by: Max Reitz --- qapi/block-core.json | 11 +--

[Qemu-block] [PATCH v4 01/13] block/mirror: Pull out mirror_perform()

When converting mirror's I/O to coroutines, we are going to need a point where these coroutines are created. mirror_perform() is going to be that point. Signed-off-by: Max Reitz Reviewed-by: Fam Zheng Reviewed-by: Vladimir Sementsov-Ogievskiy

[Qemu-block] [PATCH v4 05/13] block/mirror: Use source as a BdrvChild

With this, the mirror_top_bs is no longer just a technically required node in the BDS graph but actually represents the block job operation. Also, drop MirrorBlockJob.source, as we can reach it through mirror_top_bs->backing. Signed-off-by: Max Reitz Reviewed-by: Fam Zheng

Re: [Qemu-block] [Qemu-devel] [PATCH 05/19] tests/test-bdrv-drain: bdrv_drain_all() works in coroutines now

On 04/11/2018 11:39 AM, Kevin Wolf wrote: > Since we use bdrv_do_drained_begin/end() for bdrv_drain_all_begin/end(), > coroutine context is automatically left with a BH, preventing the > deadlocks that made bdrv_drain_all*() unsafe in coroutine context. We > can consider it compatible now the

Re: [Qemu-block] [Qemu-devel] [PATCH 04/19] block: Don't manually poll in bdrv_drain_all()

On 04/11/2018 11:39 AM, Kevin Wolf wrote: > All involved nodes are already idle, we called bdrv_do_draine_begin() on s/draine/drain/ > them. > > The comment in the code suggested that this were not correct because the s/were/was/ > completion of a request on one node could spawn a new request

Re: [Qemu-block] [Qemu-discuss] qemu-img convert stuck

On 2018-04-09 08:04, Stefan Hajnoczi wrote: > On Sun, Apr 08, 2018 at 10:35:16PM +0300, Benny Zlotnik wrote: > > What type of storage are the source and destination images? (e.g. > source is a local qcow2 file on xfs, destination is a raw file on NFS) > >> $ gdb -p 13024 -batch -ex "thread

Re: [Qemu-block] [Qemu-devel] [PATCH 06/19] block: Avoid unnecessary aio_poll() in AIO_WAIT_WHILE()

Dear Kevin: I notice checkpatch.pl complained about code style problems, you may want to replace `while (aio_poll(bs->aio_context, false));` with ``` while (aio_poll(bs->aio_context, false)) { /* No further work */ } ``` to suppress the complaints. Best, Su Hang "Kevin Wolf"

Re: [Qemu-block] [Qemu-devel] [PATCH 00/19] Drain fixes and cleanups, part 3

Hi, This series seems to have some coding style problems. See output below for more information: Type: series Message-id: 20180411163940.2523-1-kw...@redhat.com Subject: [Qemu-devel] [PATCH 00/19] Drain fixes and cleanups, part 3 === TEST SCRIPT BEGIN === #!/bin/bash BASE=base n=1 total=$(git

Re: [Qemu-block] [PATCH v2 0/2] bitmaps persistent and migration fixes

On 2018-04-11 14:26, Vladimir Sementsov-Ogievskiy wrote: > v2: > > 01: new, proposed by Max > 02: fix leaving cat processes > > Vladimir Sementsov-Ogievskiy (2): > qcow2: try load bitmaps only once > iotests: fix 169 > > block/qcow2.h | 1 + > block/qcow2.c | 16

[Qemu-block] [PATCH 14/19] block: Defer .bdrv_drain_begin callback to polling phase

We cannot allow aio_poll() in bdrv_drain_invoke(begin=true) until we're done with propagating the drain through the graph and are doing the single final BDRV_POLL_WHILE(). Just schedule the coroutine with the callback and increase bs->in_flight to make sure that the polling phase will wait for

Re: [Qemu-block] [PATCH v2 2/2] iotests: fix 169

On 2018-04-11 14:26, Vladimir Sementsov-Ogievskiy wrote: > Improve and fix 169: > - use MIGRATION events instead of RESUME > - make a TODO: enable dirty-bitmaps capability for offline case > - recreate vm_b without -incoming near test end > > This (likely) fixes racy faults at least

[Qemu-block] [PATCH 18/19] block: Allow graph changes in bdrv_drain_all_begin/end sections

bdrv_drain_all_*() used bdrv_next() to iterate over all root nodes and did a subtree drain for each of them. This works fine as long as the graph is static, but sadly, reality looks different. If the graph changes so that root nodes are added or removed, we would have to compensate for this.

[Qemu-block] [PATCH 11/19] test-bdrv-drain: Test node deletion in subtree recursion

If bdrv_do_drained_begin() polls during its subtree recursion, the graph can change and mess up the bs->children iteration. Test that this doesn't happen. Signed-off-by: Kevin Wolf --- tests/test-bdrv-drain.c | 38 +- 1 file changed, 29

[Qemu-block] [PATCH 19/19] test-bdrv-drain: Test graph changes in drain_all section

This tests both adding and remove a node between bdrv_drain_all_begin() and bdrv_drain_all_end(), and enabled the existing detach test for drain_all. Signed-off-by: Kevin Wolf --- tests/test-bdrv-drain.c | 75 +++-- 1 file changed,

[Qemu-block] [PATCH 17/19] block: Move bdrv_drain_all_begin() out of coroutine context

Before we can introduce a single polling loop for all nodes in bdrv_drain_all_begin(), we must make sure to run it outside of coroutine context like we already do for bdrv_do_drained_begin(). Signed-off-by: Kevin Wolf --- block/io.c | 22 +- 1 file changed,

[Qemu-block] [PATCH 16/19] block: Allow AIO_WAIT_WHILE with NULL ctx

bdrv_drain_all() wants to have a single polling loop for draining the in-flight requests of all nodes. This means that the AIO_WAIT_WHILE() condition relies on activity in multiple AioContexts, which is polled from the mainloop context. We must therefore call AIO_WAIT_WHILE() from the mainloop

[Qemu-block] [PATCH 15/19] test-bdrv-drain: Test that bdrv_drain_invoke() doesn't poll

This adds a test case that goes wrong if bdrv_drain_invoke() calls aio_poll(). Signed-off-by: Kevin Wolf --- tests/test-bdrv-drain.c | 102 +--- 1 file changed, 88 insertions(+), 14 deletions(-) diff --git a/tests/test-bdrv-drain.c

[Qemu-block] [PATCH 13/19] test-bdrv-drain: Graph change through parent callback

Signed-off-by: Kevin Wolf --- tests/test-bdrv-drain.c | 130 1 file changed, 130 insertions(+) diff --git a/tests/test-bdrv-drain.c b/tests/test-bdrv-drain.c index 4af6571ca3..fdf3ce19ea 100644 --- a/tests/test-bdrv-drain.c +++

[Qemu-block] [PATCH 12/19] block: Don't poll in parent drain callbacks

bdrv_do_drained_begin() is only safe if we have a single BDRV_POLL_WHILE() after quiescing all affected nodes. We cannot allow that parent callbacks introduce a nested polling loop that could cause graph changes while we're traversing the graph. Split off bdrv_do_drained_begin_quiesce(), which

[Qemu-block] [PATCH 10/19] block: Drain recursively with a single BDRV_POLL_WHILE()

Anything can happen inside BDRV_POLL_WHILE(), including graph changes that may interfere with its callers (e.g. child list iteration in recursive callers of bdrv_do_drained_begin). Switch to a single BDRV_POLL_WHILE() call for the whole subtree at the end of bdrv_do_drained_begin() to avoid such

[Qemu-block] [PATCH 00/19] Drain fixes and cleanups, part 3

This is the third and hopefully for now last part of my work to fix drain. The main goal of this series is to make drain robust against graph changes that happen in any callbacks of in-flight requests while we drain a block node. The individual patches describe the details, but the rough plan is

[Qemu-block] [PATCH 09/19] test-bdrv-drain: Add test for node deletion

From: Max Reitz This patch adds two bdrv-drain tests for what happens if some BDS goes away during the drainage. The basic idea is that you have a parent BDS with some child nodes. Then, you drain one of the children. Because of that, the party who actually owns the parent

[Qemu-block] [PATCH 02/19] block: Use bdrv_do_drain_begin/end in bdrv_drain_all()

bdrv_do_drain_begin/end() implement already everything that bdrv_drain_all_begin/end() need and currently still do manually: Disable external events, call parent drain callbacks, call block driver callbacks. It also does two more things: The first is incrementing bs->quiesce_counter.

[Qemu-block] [PATCH 07/19] block: Really pause block jobs on drain

We already requested that block jobs be paused in .bdrv_drained_begin, but no guarantee was made that the job was actually inactive at the point where bdrv_drained_begin() returned. This introduces a new callback BdrvChildRole.bdrv_drained_poll() and uses it to make bdrv_drain_poll() consider

[Qemu-block] [PATCH 06/19] block: Avoid unnecessary aio_poll() in AIO_WAIT_WHILE()

Commit 91af091f923 added an additional aio_poll() to BDRV_POLL_WHILE() in order to make sure that all pending BHs are executed on drain. This was the wrong place to make the fix, as it is useless overhead for all other users of the macro and unnecessarily complicates the mechanism. This patch

[Qemu-block] [PATCH 05/19] tests/test-bdrv-drain: bdrv_drain_all() works in coroutines now

Since we use bdrv_do_drained_begin/end() for bdrv_drain_all_begin/end(), coroutine context is automatically left with a BH, preventing the deadlocks that made bdrv_drain_all*() unsafe in coroutine context. We can consider it compatible now the latest, after having removed the old polling code as

[Qemu-block] [PATCH 01/19] test-bdrv-drain: bdrv_drain() works with cross-AioContext events

As long as nobody keeps the other I/O thread from working, there is no reason why bdrv_drain() wouldn't work with cross-AioContext events. The key is that the root request we're waiting for is in the AioContext we're polling (which it always is for bdrv_drain()) so that aio_poll() is woken up in

[Qemu-block] [PATCH 08/19] block: Remove bdrv_drain_recurse()

For bdrv_drain(), recursively waiting for child node requests is pointless because we didn't quiesce their parents, so new requests could come in anyway. Letting the function work only on a single node makes it more consistent. For subtree drains and drain_all, we already have the recursion in

[Qemu-block] [PATCH 04/19] block: Don't manually poll in bdrv_drain_all()

All involved nodes are already idle, we called bdrv_do_draine_begin() on them. The comment in the code suggested that this were not correct because the completion of a request on one node could spawn a new request on a different node (which might have been drained before, so we wouldn't drain the

[Qemu-block] [PATCH 03/19] block: Remove 'recursive' parameter from bdrv_drain_invoke()

All callers pass false for the 'recursive' parameter now. Remove it. Signed-off-by: Kevin Wolf --- block/io.c | 13 +++-- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/block/io.c b/block/io.c index cad59db2f4..d2bd89c3bb 100644 --- a/block/io.c +++

Re: [Qemu-block] [PATCH v2 1/2] qcow2: try load bitmaps only once

On 2018-04-11 14:26, Vladimir Sementsov-Ogievskiy wrote: > Checking reopen by existence of some bitmaps is wrong, as it may be > some other bitmaps, or on the other hand, user may remove bitmaps. This > criteria is bad. To simplify things and make behavior more predictable > let's just add a flag

Re: [Qemu-block] [Qemu-devel] [PATCH] iotests: fix 169

On 2018-04-11 15:05, Vladimir Sementsov-Ogievskiy wrote: [...] > Hmm, first type? I'm now not sure about, did I really see sha256 > mismatch, or something like this (should be error, but found bitmap): > > --- /work/src/qemu/up-169/tests/qemu-iotests/169.out    2018-04-11 > 15:35:10.055027392

Re: [Qemu-block] [Qemu-devel] [PATCH v2 1/2] qcow2: try load bitmaps only once

11.04.2018 16:22, Eric Blake wrote: On 04/11/2018 07:26 AM, Vladimir Sementsov-Ogievskiy wrote: Checking reopen by existence of some bitmaps is wrong, as it may be some other bitmaps, or on the other hand, user may remove bitmaps. This criteria is bad. To simplify things and make behavior more

Re: [Qemu-block] [Qemu-devel] [PATCH] iotests: fix 169

11.04.2018 16:05, Vladimir Sementsov-Ogievskiy wrote: 11.04.2018 12:36, Vladimir Sementsov-Ogievskiy wrote: 11.04.2018 12:02, Vladimir Sementsov-Ogievskiy wrote: 03.04.2018 23:13, John Snow wrote: On 04/03/2018 12:23 PM, Max Reitz wrote: On 2018-03-30 18:10, Vladimir Sementsov-Ogievskiy

Re: [Qemu-block] [Qemu-devel] [PATCH v2 1/2] qcow2: try load bitmaps only once

On 04/11/2018 07:26 AM, Vladimir Sementsov-Ogievskiy wrote: > Checking reopen by existence of some bitmaps is wrong, as it may be > some other bitmaps, or on the other hand, user may remove bitmaps. This > criteria is bad. To simplify things and make behavior more predictable > let's just add a

Re: [Qemu-block] [Qemu-devel] [PATCH] iotests: fix 169

11.04.2018 12:36, Vladimir Sementsov-Ogievskiy wrote: 11.04.2018 12:02, Vladimir Sementsov-Ogievskiy wrote: 03.04.2018 23:13, John Snow wrote: On 04/03/2018 12:23 PM, Max Reitz wrote: On 2018-03-30 18:10, Vladimir Sementsov-Ogievskiy wrote: Use MIGRATION events instead of RESUME. Also, make

[Qemu-block] [PATCH v2 1/2] qcow2: try load bitmaps only once

Checking reopen by existence of some bitmaps is wrong, as it may be some other bitmaps, or on the other hand, user may remove bitmaps. This criteria is bad. To simplify things and make behavior more predictable let's just add a flag to remember, that we've already tried to load bitmaps on open and

[Qemu-block] [PATCH v2 2/2] iotests: fix 169

Improve and fix 169: - use MIGRATION events instead of RESUME - make a TODO: enable dirty-bitmaps capability for offline case - recreate vm_b without -incoming near test end This (likely) fixes racy faults at least of the following types: - timeout on waiting for RESUME event

[Qemu-block] [PATCH v2 0/2] bitmaps persistent and migration fixes

v2: 01: new, proposed by Max 02: fix leaving cat processes Vladimir Sementsov-Ogievskiy (2): qcow2: try load bitmaps only once iotests: fix 169 block/qcow2.h | 1 + block/qcow2.c | 16 tests/qemu-iotests/169 | 48

Re: [Qemu-block] [PATCH] iotests: Split 214 off of 122

On 2018-04-06 18:41, Max Reitz wrote: > Commit abd3622cc03cf41ed542126a540385f30a4c0175 added a case to 122 > regarding how the qcow2 driver handles an incorrect compressed data > length value. This does not really fit into 122, as that file is > supposed to contain qemu-img convert test cases,

Re: [Qemu-block] [Qemu-devel] [PATCH] iotests: fix 169

On 2018-04-11 11:02, Vladimir Sementsov-Ogievskiy wrote: > 03.04.2018 23:13, John Snow wrote: >> >> On 04/03/2018 12:23 PM, Max Reitz wrote: >>> On 2018-03-30 18:10, Vladimir Sementsov-Ogievskiy wrote: Use MIGRATION events instead of RESUME. Also, make a TODO: enable dirty-bitmaps

Re: [Qemu-block] [PATCH for-2.12 v2] qemu-iotests: update 185 output

On 2018-04-10 10:11, Stefan Hajnoczi wrote: > On Wed, Apr 04, 2018 at 06:16:12PM +0200, Max Reitz wrote: >> On 2018-04-04 17:01, Stefan Hajnoczi wrote: >> === Start mirror job and exit qemu === >> >> This seems to be independent of whether there is actually data on >> TEST_IMG (the commit

Re: [Qemu-block] [PATCH] iotests: Split 214 off of 122

On 2018-04-09 11:28, Alberto Garcia wrote: > On Fri 06 Apr 2018 06:41:08 PM CEST, Max Reitz wrote: >> Commit abd3622cc03cf41ed542126a540385f30a4c0175 added a case to 122 >> regarding how the qcow2 driver handles an incorrect compressed data >> length value. This does not

Re: [Qemu-block] [PULL 0/7] Block layer patches for 2.12.0-rc3

On 10 April 2018 at 16:37, Kevin Wolf wrote: > The following changes since commit df6378eb0e6cfd58a22a1c3ff8fa4a9039f1eaa8: > > Merge remote-tracking branch 'remotes/kraxel/tags/ui-20180410-pull-request' > into staging (2018-04-10 14:04:27 +0100) > > are available in the git

Re: [Qemu-block] [Qemu-devel] [PATCH] iotests: fix 169

11.04.2018 12:02, Vladimir Sementsov-Ogievskiy wrote: 03.04.2018 23:13, John Snow wrote: On 04/03/2018 12:23 PM, Max Reitz wrote: On 2018-03-30 18:10, Vladimir Sementsov-Ogievskiy wrote: Use MIGRATION events instead of RESUME. Also, make a TODO: enable dirty-bitmaps capability for offline

Re: [Qemu-block] [Qemu-devel] [PATCH] iotests: fix 169

03.04.2018 23:13, John Snow wrote: On 04/03/2018 12:23 PM, Max Reitz wrote: On 2018-03-30 18:10, Vladimir Sementsov-Ogievskiy wrote: Use MIGRATION events instead of RESUME. Also, make a TODO: enable dirty-bitmaps capability for offline case. This (likely) fixes racy faults at least of the