Re: [Qemu-block] [Qemu-devel] [PATCH v3 0/3] file-posix: Simplifications on image locking

2018-08-17 Thread no-reply
Hi, This series failed docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. Type: series Message-id: 20180817054758.14515-1-f...@redhat.com Subject: [Qemu-devel] [PATCH v3 0/3] file-posix:

Re: [Qemu-block] [RFC v2] new, node-graph-based fleecing and backup

2018-08-17 Thread Max Reitz
On 2018-08-14 19:01, Vladimir Sementsov-Ogievskiy wrote: > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > > [v2 is just a resend. I forget to add Den an me to cc, and I don't see the > letter in my thunderbird at all. strange. sorry for that] > > Hi all! > > Here is an idea and kind of

Re: [Qemu-block] [PATCH v2 1/3] qapi: add x-query-block-graph

2018-08-17 Thread Max Reitz
On 2018-08-17 22:32, Eric Blake wrote: > On 08/17/2018 01:04 PM, Vladimir Sementsov-Ogievskiy wrote: >> Add a new command, returning block nodes graph. >> >> Signed-off-by: Vladimir Sementsov-Ogievskiy >> --- >>   qapi/block-core.json  | 116 >> ++ >

Re: [Qemu-block] [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

2018-08-17 Thread no-reply
Hi, This series failed docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. Type: series Message-id: 20180814170126.56461-1-vsement...@virtuozzo.com Subject: [Qemu-devel] [RFC v2] new,

Re: [Qemu-block] [PATCH] qemu-img.c: increase spacing between commands in documentation

2018-08-17 Thread Eric Blake
On 08/17/2018 02:28 PM, Programmingkid wrote: -o Used with a comma separated list of format specific options in a name=value format. Use "-o ?" for an overview of the options Please spell that "-o help", not "-o ?". Otherwise, the user has to quote the ? to

Re: [Qemu-block] [Qemu-devel] [RFC v2] new, node-graph-based fleecing and backup

2018-08-17 Thread no-reply
Hi, This series failed docker-mingw@fedora build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. Type: series Message-id: 20180814170126.56461-1-vsement...@virtuozzo.com Subject: [Qemu-devel] [RFC v2] new,

Re: [Qemu-block] [PATCH v2 1/3] qapi: add x-query-block-graph

2018-08-17 Thread Eric Blake
On 08/17/2018 01:04 PM, Vladimir Sementsov-Ogievskiy wrote: Add a new command, returning block nodes graph. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 116 ++ +## +# @BlockGraphEdge: +# +# Block Graph edge

Re: [Qemu-block] [Qemu-devel] [RFC PATCH] rbd: Don't convert keypairs to JSON and back

2018-08-17 Thread Max Reitz
On 2018-08-16 08:40, Markus Armbruster wrote: > Max Reitz writes: > >> On 2018-08-15 10:12, Markus Armbruster wrote: >>> Max Reitz writes: >> >> [...] >> To me personally the issue is that if you can specify a plain filename, bdrv_refresh_filename() should give you that plain filename

Re: [Qemu-block] [Qemu-devel] [PATCH for-3.1 v10 00/31] block: Fix some filename generation issues

2018-08-17 Thread Max Reitz
On 2018-08-16 08:02, Markus Armbruster wrote: > Max Reitz writes: > >> On 2018-08-15 05:43, no-re...@patchew.org wrote: >>> Hi, >>> >>> This series seems to have some coding style problems. See output below for >>> more information: >> >> [...] >> >>> === OUTPUT BEGIN === >>> Checking PATCH

Re: [Qemu-block] [PATCH 0/7] qcow2: async handling of fragmented io

2018-08-17 Thread Denis V. Lunev
On 08/17/2018 10:34 PM, Max Reitz wrote: > On 2018-08-16 15:58, Vladimir Sementsov-Ogievskiy wrote: >> 16.08.2018 03:51, Max Reitz wrote: >>> On 2018-08-07 19:43, Vladimir Sementsov-Ogievskiy wrote: Hi all! Here is an asynchronous scheme for handling fragmented qcow2 reads and

Re: [Qemu-block] [PATCH 0/7] qcow2: async handling of fragmented io

2018-08-17 Thread Max Reitz
On 2018-08-16 15:58, Vladimir Sementsov-Ogievskiy wrote: > 16.08.2018 03:51, Max Reitz wrote: >> On 2018-08-07 19:43, Vladimir Sementsov-Ogievskiy wrote: >>> Hi all! >>> >>> Here is an asynchronous scheme for handling fragmented qcow2 >>> reads and writes. Both qcow2 read and write functions loops

Re: [Qemu-block] [PATCH] qemu-img.c: increase spacing between commands in documentation

2018-08-17 Thread Programmingkid
> On Aug 17, 2018, at 9:44 AM, Eric Blake wrote: > > On 08/16/2018 08:27 PM, Programmingkid wrote: > >> I am by no means an expert at qemu-img. But I did try my best to create what >> I think should be the new output for qemu-img --help. This is just >> the text I plan on using in a future

Re: [Qemu-block] [PATCH 2/2] qemu-img: Add dd seek= option

2018-08-17 Thread Max Reitz
On 2018-08-16 09:15, Kevin Wolf wrote: > Am 16.08.2018 um 04:49 hat Max Reitz geschrieben: >> On 2018-08-16 04:39, Eric Blake wrote: >>> If convert were more powerful, I'd be fine dropping 'qemu-img dd' after >>> a proper deprecation period. >> >> Technically it has those features already, with

[Qemu-block] [PATCH 0/7] jobs: remove job_defer_to_main_loop

2018-08-17 Thread John Snow
First, it's redundant to have each job manage this itself. Second, doing so allows us to remove a tricky case where the completion code is called under an aio_context lock, which then calls the finalization code which is itself executed under a second aio_context lock. Removing this recursive

Re: [Qemu-block] [Qemu-devel] [PATCH 4/7] block/commit: utilize job_exit shim

2018-08-17 Thread John Snow
On 08/17/2018 03:04 PM, John Snow wrote: > Change the manual deferment to commit_complete into the implicit > callback to job_exit, renaming commit_complete to commit_exit. > > This conversion does change the timing of when job_completed is > called to after the bdrv_replace_node and

Re: [Qemu-block] [PATCH v2 2/3] scripts: add render_block_graph function for QEMUMachine

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
17.08.2018 21:25, Eduardo Habkost wrote: On Fri, Aug 17, 2018 at 09:04:39PM +0300, Vladimir Sementsov-Ogievskiy wrote: Render block nodes graph with help of graphviz. This new function is for debugging, so there is no sense to put it into qemu.py as a method of QEMUMachine. Let's instead put it

[Qemu-block] [PATCH 6/7] jobs: utilize job_exit shim

2018-08-17 Thread John Snow
Utilize the job_exit shim by not calling job_defer_to_main_loop, and where applicable, converting the deferred callback into the job_exit callback. This converts backup, stream, create, and the unit tests all at once. None of these jobs undergo and order of operations changes, so it should be a

[Qemu-block] [PATCH 3/7] jobs: add exit shim

2018-08-17 Thread John Snow
All jobs do the same thing when they leave their running loop: - Store the return code in a structure - wait to receive this structure in the main thread - signal job completion via job_completed Few jobs do anything beyond exactly this. Consolidate this exit logic for a net reduction in SLOC.

[Qemu-block] [PATCH 4/7] block/commit: utilize job_exit shim

2018-08-17 Thread John Snow
Change the manual deferment to commit_complete into the implicit callback to job_exit, renaming commit_complete to commit_exit. This conversion does change the timing of when job_completed is called to after the bdrv_replace_node and bdrv_unref calls, which could have implications for bjob->blk

[Qemu-block] [PATCH 7/7] jobs: remove job_defer_to_main_loop

2018-08-17 Thread John Snow
Now that the job infrastructure is handling the job_completed call for all implemented jobs, we can remove the interface that allowed jobs to schedule their own completion. Signed-off-by: John Snow --- include/qemu/job.h | 17 - job.c | 40

Re: [Qemu-block] [PATCH v2 2/3] scripts: add render_block_graph function for QEMUMachine

2018-08-17 Thread Eduardo Habkost
On Fri, Aug 17, 2018 at 09:59:41PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 17.08.2018 21:25, Eduardo Habkost wrote: > > On Fri, Aug 17, 2018 at 09:04:39PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > Render block nodes graph with help of graphviz. This new function is > > > for

Re: [Qemu-block] [PATCH v2 2/3] scripts: add render_block_graph function for QEMUMachine

2018-08-17 Thread Eduardo Habkost
On Fri, Aug 17, 2018 at 09:04:39PM +0300, Vladimir Sementsov-Ogievskiy wrote: > Render block nodes graph with help of graphviz. This new function is > for debugging, so there is no sense to put it into qemu.py as a method > of QEMUMachine. Let's instead put it separately. > > Signed-off-by:

Re: [Qemu-block] [RFC v2] new, node-graph-based fleecing and backup

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
attached generated block graph (with help of "[PATCH v2 0/3] block nodes graph visualization") -- Best regards, Vladimir

[Qemu-block] [PATCH v2 2/3] scripts: add render_block_graph function for QEMUMachine

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Render block nodes graph with help of graphviz. This new function is for debugging, so there is no sense to put it into qemu.py as a method of QEMUMachine. Let's instead put it separately. Signed-off-by: Vladimir Sementsov-Ogievskiy --- scripts/render_block_graph.py | 78

[Qemu-block] [PATCH v2 1/3] qapi: add x-query-block-graph

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Add a new command, returning block nodes graph. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 116 ++ include/block/block.h | 1 + block.c | 80 ++ blockdev.c|

[Qemu-block] [PATCH v2 3/3] not-for-commit: example of new command usage for debugging

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/222 | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222 index 0ead56d574..c2e647fa83 100644 --- a/tests/qemu-iotests/222 +++ b/tests/qemu-iotests/222 @@ -137,6 +137,9 @@ with

Re: [Qemu-block] [PATCH 0/7] discard blockstats

2018-08-17 Thread Paolo Bonzini
On 20/11/2017 17:50, Anton Nefedov wrote: > qmp query-blockstats provides stats info for write/read/flush ops. > > Patches 1-5 implement the similar for discard (unmap) command for scsi > and ide disks. > Discard stat "unmap_ops / unmap_bytes" is supposed to account the ops that > have completed

[Qemu-block] [PATCH 2/2] qobject: modify qobject_ref() to assert on NULL

2018-08-17 Thread Marc-André Lureau
While it may be convenient to accept NULL value in qobject_unref() (for similar reasons as free() accepts NULL), it is not such a good idea for qobject_ref(): now assert() on NULL. Some code relied on that behaviour, but it's best to be explicit that NULL is accepted. We have to rely on testing,

[Qemu-block] [PATCH 1/2] qdict: add qdict_steal()

2018-08-17 Thread Marc-André Lureau
Add a new function qdict_steal(), that deletes a key from a dict, and returns the associated value, if any. Simplify related code. Signed-off-by: Marc-André Lureau --- include/qapi/qmp/qdict.h | 1 + monitor.c| 3 +-- qobject/block-qdict.c| 7 ++- qobject/qdict.c

[Qemu-block] [RFC PATCH 4/5] block: Drop AioContext lock in bdrv_drain_poll_top_level()

2018-08-17 Thread Kevin Wolf
Simimlar to AIO_WAIT_WHILE(), bdrv_drain_poll_top_level() needs to release the AioContext lock of the node to be drained before calling aio_poll(). Otherwise, callbacks called by aio_poll() would possibly take the lock a second time and run into a deadlock with a nested AIO_WAIT_WHILE() call.

[Qemu-block] [RFC PATCH 2/5] tests: Acquire AioContext around job_finish_sync()

2018-08-17 Thread Kevin Wolf
All callers in QEMU proper hold the AioContext lock when calling job_finish_sync(). The tests should do the same. Signed-off-by: Kevin Wolf --- include/qemu/job.h | 6 ++ tests/test-bdrv-drain.c | 6 ++ tests/test-blockjob.c | 6 ++ 3 files changed, 18 insertions(+) diff

[Qemu-block] [RFC PATCH 5/5] [WIP] Lock AioContext in bdrv_co_drain_bh_cb()

2018-08-17 Thread Kevin Wolf
Not sure if this is correct, but at least it makes qemu-iotests 127 pass again. Signed-off-by: Kevin Wolf --- block/io.c | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/io.c b/block/io.c index 832d2536bf..d3dde4d7fd 100644 --- a/block/io.c +++ b/block/io.c @@ -309,6 +309,10 @@

[Qemu-block] [RFC PATCH 0/5] Fix some jobs/drain/aio_poll related hangs

2018-08-17 Thread Kevin Wolf
I'm running out of time and will be offline for the next two weeks, so I'm just sending out what I have right now. This is probably not ready to be merged yet, but if need be, someone else can pick it up. Otherwise I'll do that myself when I return. This is related to the following bug reports:

[Qemu-block] [RFC PATCH 1/5] blockjob: Wake up BDS when job becomes idle

2018-08-17 Thread Kevin Wolf
In the context of draining a BDS, the .drained_poll callback of block jobs is called. If this returns true (i.e. there is still some activity pending), the drain operation may call aio_poll() with blocking=true to wait for completion. As soon as the pending activity is completed and the job

[Qemu-block] [RFC PATCH 3/5] job: Drop AioContext lock around aio_poll()

2018-08-17 Thread Kevin Wolf
Simimlar to AIO_WAIT_WHILE(), job_finish_sync() needs to release the AioContext lock of the job before calling aio_poll(). Otherwise, callbacks called by aio_poll() would possibly take the lock a second time and run into a deadlock with a nested AIO_WAIT_WHILE() call. Signed-off-by: Kevin Wolf

Re: [Qemu-block] [Qemu-devel] [PATCH 2/4] nbd/server: Advertise actual minimum block size

2018-08-17 Thread Eric Blake
On 08/17/2018 10:04 AM, Vladimir Sementsov-Ogievskiy wrote: Can you create a dirty bitmap with a granularity smaller than request_alignment?  I know you can configure dirty bitmap granularity independently from cluster size (in both directions: either smaller or larger than cluster size), but

Re: [Qemu-block] [Qemu-devel] [PATCH 4/4] nbd/client: Deal with unaligned size from server

2018-08-17 Thread Eric Blake
On 08/17/2018 08:57 AM, Vladimir Sementsov-Ogievskiy wrote: 02.08.2018 17:48, Eric Blake wrote: When a server advertises an unaligned size but no block sizes, the code was rounding up to a sector-aligned size (a known limitation of bdrv_getlength()), then assuming a request_alignment of 512

Re: [Qemu-block] [Qemu-devel] [PATCH 2/4] nbd/server: Advertise actual minimum block size

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
02.08.2018 17:48, Eric Blake wrote: Both NBD_CMD_BLOCK_STATUS and structured NBD_CMD_READ will split their reply according to bdrv_block_status() boundaries. If the block device has a request_alignment smaller than 512, but we advertise a block alignment of 512 to the client, then this can

Re: [Qemu-block] [PATCH 3/4] scripts/qemu: add render_block_graph method for QEMUMachine

2018-08-17 Thread Eduardo Habkost
On Fri, Aug 17, 2018 at 01:08:42PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 17.08.2018 04:54, Eduardo Habkost wrote: > > On Thu, Aug 16, 2018 at 08:20:26PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > Render block nodes graph with help of graphviz > > > > > > Signed-off-by: Vladimir

Re: [Qemu-block] [PATCH] qemu-img.c: increase spacing between commands in documentation

2018-08-17 Thread Eric Blake
On 08/16/2018 08:27 PM, Programmingkid wrote: I am by no means an expert at qemu-img. But I did try my best to create what I think should be the new output for qemu-img --help. This is just the text I plan on using in a future patch. It is easier to read right now than it will be in patch

Re: [Qemu-block] [Qemu-devel] [PATCH 1/4] block: Add bdrv_get_request_alignment()

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
02.08.2018 17:48, Eric Blake wrote: The next patch needs access to a device's minimum permitted alignment, since NBD wants to advertise this to clients. Add an accessor function, borrowing from blk_get_max_transfer() for accessing a backend's block limits. Signed-off-by: Eric Blake

[Qemu-block] [PATCH v2 1/7] block/qcow2-refcount: fix check_oflag_copied

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Increase corruptions_fixed only after successful fix. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 3c539f02e5..615847eb09 100644 ---

[Qemu-block] [PATCH v2 7/7] block/qcow2-refcount: fix out-of-file L2 entries to be read-as-zero

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Rewrite corrupted L2 table entry, which reference space out of underlying file. Make this L2 table entry read-as-all-zeros without any allocation. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 32 1 file changed, 32 insertions(+)

[Qemu-block] [PATCH v2 3/7] block/qcow2-refcount: check_refcounts_l2: refactor compressed case

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Separate offset and size of compressed cluster. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 15 ++- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index 566c19fbfa..0ea01e3ee2 100644 ---

[Qemu-block] [PATCH v2 2/7] block/qcow2-refcount: avoid eating RAM

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
qcow2_inc_refcounts_imrt() (through realloc_refcount_array()) can eat an unpredictable amount of memory on corrupted table entries, which are referencing regions far beyond the end of file. Prevent this, by skipping such regions from further processing. Signed-off-by: Vladimir

[Qemu-block] [PATCH v2 6/7] block/qcow2-refcount: fix out-of-file L1 entries to be zero

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Zero out corrupted L1 table entry, which reference L2 table out of underlying file. Zero L1 table entry means that "the L2 table and all clusters described by this L2 table are unallocated." Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 44

[Qemu-block] [PATCH v2 4/7] block/qcow2-refcount: check_refcounts_l2: reduce ignored overlaps

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Reduce number of structures ignored in overlap check: when checking active table ignore active tables, when checking inactive table ignore inactive ones. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 16 +--- 1 file changed, 9 insertions(+), 7 deletions(-)

[Qemu-block] [PATCH v2 5/7] block/qcow2-refcount: check_refcounts_l2: split fix_l2_entry_to_zero

2018-08-17 Thread Vladimir Sementsov-Ogievskiy
Split entry repairing to separate function, to be reused later. Note: entry in in-memory l2 table (local variable in check_refcounts_l2) is not updated after this patch. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 147