[Qemu-block] [PATCH 0/2] block/rbd: enable filename parsing on open

2018-09-10 Thread Jeff Cody
This series enables filename parsing on open, on the old key/value pair. Recent changes to new option formats for rbd broke some images. See Patch 2 for more details. Jeff Cody (2): block/rbd: pull out qemu_rbd_convert_options block/rbd: Attempt to parse legacy filenames block/rbd.c | 66

[Qemu-block] [PATCH 1/2] block/rbd: pull out qemu_rbd_convert_options

2018-09-10 Thread Jeff Cody
Code movement to pull the conversion from Qdict to BlockdevOptionsRbd into a helper function. Signed-off-by: Jeff Cody --- block/rbd.c | 36 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index

[Qemu-block] [PATCH 2/2] block/rbd: Attempt to parse legacy filenames

2018-09-10 Thread Jeff Cody
When we converted rbd to get rid of the older key/value-centric encoding format, we broke compatibility with image files with backing file strings encoded in the old format. This leaves a bit of an ugly conundrum, and a hacky solution. If the initial attempt to parse the "proper" options fails,

Re: [Qemu-block] [Qemu-devel] [PATCH v2 3/4] blockdev: rename block-dirty-bitmap-clear transaction handlers

2018-09-10 Thread John Snow
On 07/06/2018 07:36 AM, Vladimir Sementsov-Ogievskiy wrote: > Rename block-dirty-bitmap-clear transaction handlers to reuse them for > x-block-dirty-bitmap-merge transaction in the following patch. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > blockdev.c | 8 > 1 file

Re: [Qemu-block] [Qemu-devel] [PATCH v2 2/4] dirty-bitmap: restore bitmap after merge

2018-09-10 Thread John Snow
We actually don't restore anything just yet, so this commit message is a little off I think. On 07/06/2018 07:36 AM, Vladimir Sementsov-Ogievskiy wrote: > Add backup parameter to bdrv_merge_dirty_bitmap() and refactor > bdrv_undo_clear_dirty_bitmap() to use it both for undo clean and merge. > >

Re: [Qemu-block] [Qemu-devel] [PATCH v3 8/8] Revert "hbitmap: Add @advance param to hbitmap_iter_next()"

2018-09-10 Thread John Snow
On 08/14/2018 08:14 AM, Vladimir Sementsov-Ogievskiy wrote: > This reverts commit a33fbb4f8b64226becf502a123733776ce319b24. > > The functionality is unused. > > Note: in addition to automatic revert, drop second parameter in > hbitmap_iter_next() call from hbitmap_next_dirty_area() too. >

Re: [Qemu-block] [Qemu-devel] [PATCH v3 5/8] block/mirror: fix and improve do_sync_target_write

2018-09-10 Thread John Snow
On 08/14/2018 08:14 AM, Vladimir Sementsov-Ogievskiy wrote: > Use bdrv_dirty_bitmap_next_dirty_area() instead of > bdrv_dirty_iter_next_area(), because of the following problems of > bdrv_dirty_iter_next_area(): > > 1. Using HBitmap iterators we should carefully handle unaligned offset, > as

Re: [Qemu-block] [Qemu-devel] [PATCH v2 1/4] drity-bitmap: switch assert-fails to errors in bdrv_merge_dirty_bitmap

2018-09-10 Thread John Snow
s/drity-bitmap/dirty-bitmap/g On 07/06/2018 07:36 AM, Vladimir Sementsov-Ogievskiy wrote: > Move checks from qmp_x_block_dirty_bitmap_merge() to > bdrv_merge_dirty_bitmap(), to share them with dirty bitmap merge > transaction action in future commit. > > Note: for now, only

Re: [Qemu-block] [Qemu-devel] [PATCH v3 4/8] tests: add tests for hbitmap_next_dirty_area

2018-09-10 Thread John Snow
On 08/14/2018 08:14 AM, Vladimir Sementsov-Ogievskiy wrote: > Signed-off-by: Vladimir Sementsov-Ogievskiy ACK, and Tested-by: John Snow > --- > tests/test-hbitmap.c | 106 > +++ > 1 file changed, 106 insertions(+) > > diff --git

Re: [Qemu-block] [Qemu-devel] [PATCH v3 3/8] dirty-bitmap: add bdrv_dirty_bitmap_next_dirty_area

2018-09-10 Thread John Snow
On 08/14/2018 08:14 AM, Vladimir Sementsov-Ogievskiy wrote: > The function alters bdrv_dirty_iter_next_area(), which is wrong and > less efficient (see further commit > "block/mirror: fix and improve do_sync_target_write" for description). > > Signed-off-by: Vladimir Sementsov-Ogievskiy > ---

Re: [Qemu-block] [PATCH 0/4 for-3.0?] NBD fixes for unaligned images

2018-09-10 Thread Eric Blake
On 9/10/18 2:25 PM, John Snow wrote: Hi, has this series been superseded or do we need a respin for 3.1? Needs a respin. There are two issues still to be solved: 1. Asking the top-most block layer for its alignment isn't necessarily right for qemu as server. If we have: backing (512) <-

Re: [Qemu-block] [PATCH 0/4 for-3.0?] NBD fixes for unaligned images

2018-09-10 Thread John Snow
Hi, has this series been superseded or do we need a respin for 3.1? (It's not quite been a month, but it caught my eye.) --js On 08/02/2018 10:48 AM, Eric Blake wrote: > Rich reported a bug when using qemu as client to nbdkit serving > a non-sector-aligned image; in turn, I found a second bug

Re: [Qemu-block] [PATCH v2] qcow2: Release dirty entries with cache-clean-interval

2018-09-10 Thread John Snow
On 08/09/2018 09:44 AM, Alberto Garcia wrote: > The cache-clean-interval option is used to periodically release unused > entries from the L2 and refcount caches. Dirty cache entries are left > untouched, even if they are otherwise valid candidates for removal. > > This patch allows releasing

Re: [Qemu-block] [Qemu-devel] [PATCH v3 1/8] dirty-bitmap: improve bdrv_dirty_bitmap_next_zero

2018-09-10 Thread Eric Blake
On 9/10/18 11:49 AM, Vladimir Sementsov-Ogievskiy wrote: -int64_t hbitmap_next_zero(const HBitmap *hb, uint64_t start); +int64_t hbitmap_next_zero(const HBitmap *hb, uint64_t start, int64_t end); The interface looks weird because we can define a 'start' that's beyond the 'end'. I realize

Re: [Qemu-block] [Qemu-devel] [PATCH v3 1/8] dirty-bitmap: improve bdrv_dirty_bitmap_next_zero

2018-09-10 Thread Vladimir Sementsov-Ogievskiy
10.09.2018 19:55, Eric Blake wrote: On 9/10/18 11:49 AM, Vladimir Sementsov-Ogievskiy wrote: -int64_t hbitmap_next_zero(const HBitmap *hb, uint64_t start); +int64_t hbitmap_next_zero(const HBitmap *hb, uint64_t start, int64_t end); The interface looks weird because we can define a 'start'

Re: [Qemu-block] [PATCH for-3.1 v10 00/31] block: Fix some filename generation issues

2018-09-10 Thread Max Reitz
On 10.09.18 17:18, Kevin Wolf wrote: > Am 09.08.2018 um 23:34 hat Max Reitz geschrieben: >> Once more, I’ll spare both me and you another iteration of the cover >> letter, so see here: >> >> http://lists.nongnu.org/archive/html/qemu-block/2017-09/msg01030.html >> >> (Although this series no longer

Re: [Qemu-block] [Qemu-devel] [PATCH v3 1/8] dirty-bitmap: improve bdrv_dirty_bitmap_next_zero

2018-09-10 Thread Vladimir Sementsov-Ogievskiy
08.09.2018 00:49, John Snow wrote: On 08/14/2018 08:14 AM, Vladimir Sementsov-Ogievskiy wrote: Add bytes parameter to the function, to limit searched range. I'm going to assume that Eric Blake has been through here and commented on the interface itself. Signed-off-by: Vladimir

Re: [Qemu-block] [Qemu-devel] [PATCH] qemu-iotests: Test snapshot=on with nonexistent TMPDIR

2018-09-10 Thread Alberto Garcia
On Mon 10 Sep 2018 05:55:15 PM CEST, Eric Blake wrote: >> Hm, it actually doesn't crash for me even without the fix. Anyway, I >> don't have a good idea to make it more likely to crash and it's >> certainly better than nothing. > > Does running the test under valgrind reliably see the

Re: [Qemu-block] [PATCH for-3.1 v10 04/31] block: Add BDS.auto_backing_file

2018-09-10 Thread Alberto Garcia
On Fri 07 Sep 2018 02:42:53 PM CEST, Max Reitz wrote: > The whole purpose of bs->auto_backing_file is so you can compare it > with bs->backing->bs->filename, see when it differs, and then you know > that the user has changed the backing file from what it would be based > on the image header alone

Re: [Qemu-block] [Qemu-devel] [PATCH v3 1/8] dirty-bitmap: improve bdrv_dirty_bitmap_next_zero

2018-09-10 Thread John Snow
On 09/10/2018 10:59 AM, Eric Blake wrote: > On 9/7/18 4:49 PM, John Snow wrote: >> >> >> On 08/14/2018 08:14 AM, Vladimir Sementsov-Ogievskiy wrote: >>> Add bytes parameter to the function, to limit searched range. >>> >> >> I'm going to assume that Eric Blake has been through here and

Re: [Qemu-block] [Qemu-devel] [PATCH] qemu-iotests: Test snapshot=on with nonexistent TMPDIR

2018-09-10 Thread Eric Blake
On 9/10/18 4:57 AM, Kevin Wolf wrote: Am 10.09.2018 um 11:29 hat Alberto Garcia geschrieben: We just fixed a bug that was causing a use-after-free when QEMU was unable to create a temporary snapshot. This is a test case for this scenario. Signed-off-by: Alberto Garcia Hm, it actually

Re: [Qemu-block] [PATCH for-3.1 v10 00/31] block: Fix some filename generation issues

2018-09-10 Thread Kevin Wolf
Am 09.08.2018 um 23:34 hat Max Reitz geschrieben: > Once more, I’ll spare both me and you another iteration of the cover > letter, so see here: > > http://lists.nongnu.org/archive/html/qemu-block/2017-09/msg01030.html > > (Although this series no longer includes a @base-directory option.) > >

Re: [Qemu-block] [PATCH v2] job: Fix nested aio_poll() hanging in job_txn_apply

2018-09-10 Thread Kevin Wolf
Am 24.08.2018 um 04:43 hat Fam Zheng geschrieben: > All callers have acquired ctx already. Doing that again results in > aio_poll() hang. This fixes the problem that a BDRV_POLL_WHILE() in the > callback cannot make progress because ctx is recursively locked, for > example, when drive-backup

Re: [Qemu-block] backend for blk or fs with guaranteed blocking/synchronous I/O

2018-09-10 Thread Artem Pisarenko
It looks like things are even worse. Guest demonstrates strange timings even without access to anything external to machine. I've added Paolo Bonzini to CC, because issue looks related to cpu/tcg/memory stuff. I've written simple test script running parallel 'dd' utility processes operating on

Re: [Qemu-block] [Qemu-devel] [PATCH v3 1/8] dirty-bitmap: improve bdrv_dirty_bitmap_next_zero

2018-09-10 Thread Eric Blake
On 9/7/18 4:49 PM, John Snow wrote: On 08/14/2018 08:14 AM, Vladimir Sementsov-Ogievskiy wrote: Add bytes parameter to the function, to limit searched range. I'm going to assume that Eric Blake has been through here and commented on the interface itself. Actually, I haven't had time to

[Qemu-block] [PATCH 1/2] virtio: Return true from virtio_queue_empty if broken

2018-09-10 Thread Fam Zheng
Both virtio-blk and virtio-scsi use virtio_queue_empty() as the loop condition in VQ handlers (virtio_blk_handle_vq, virtio_scsi_handle_cmd_vq). When a device is marked broken in virtqueue_pop, for example if a vIOMMU address translation failed, we want to break out of the loop. This fixes a

[Qemu-block] [PATCH 2/2] virtio-scsi/virtio-blk: Disable poll handlers when stopping vq handler

2018-09-10 Thread Fam Zheng
We have this unwanted call stack: > ... > #13 0x5586602b7793 in virtio_scsi_handle_cmd_vq > #14 0x5586602b8d66 in virtio_scsi_data_plane_handle_cmd > #15 0x5586602ddab7 in virtio_queue_notify_aio_vq > #16 0x5586602dfc9f in virtio_queue_host_notifier_aio_poll > #17

[Qemu-block] [PATCH 0/2] virtio-scsi: Fix QEMU hang with vIOMMU and ATS

2018-09-10 Thread Fam Zheng
The first patch describes the bug and the reproducer. The VQ poll handler that is called by mistake within virtio_scsi_dataplane_stop enters a dead loop because it fails to detect an error state. Fix both sides of the problem: the handler should break out from the loop if no progress can be made

Re: [Qemu-block] [PATCH v8 0/8] Take the image size into account when allocating the L2 cache

2018-09-10 Thread Kevin Wolf
Hi Leonid, Am 13.08.2018 um 03:07 hat Leonid Bloch geschrieben: > This series makes the qcow2 L2 cache assignment aware of the image size, > with the intention for it to cover the entire image. The importance of > this change is in noticeable performance improvement, especially with > heavy

Re: [Qemu-block] [Qemu-devel] [PATCH] qemu-img.c: add help for each command

2018-09-10 Thread Programmingkid
> On Sep 10, 2018, at 4:16 AM, Kevin Wolf wrote: > > Am 08.09.2018 um 05:16 hat Programmingkid geschrieben: >> >>> On Sep 7, 2018, at 11:13 PM, Peter Maydell wrote: >>> >>> On 8 September 2018 at 04:01, John Arbuckle >>> wrote: >>> +/* print the help for this command */ +

Re: [Qemu-block] [PATCH v0 2/2] block: postpone the coroutine executing if the BDS's is drained

2018-09-10 Thread Kevin Wolf
Am 29.06.2018 um 14:40 hat Denis Plotnikov geschrieben: > Fixes the problem of ide request appearing when the BDS is in > the "drained section". > > Without the patch the request can come and be processed by the main > event loop, as the ide requests are processed by the main event loop > and the

Re: [Qemu-block] forward of qemu bug regarding random qemu hangs at shutdown

2018-09-10 Thread Marc Hartmayer
On Fri, Sep 07, 2018 at 06:21 PM +0200, Marc Hartmayer wrote: > On Fri, Sep 07, 2018 at 11:34 AM +0200, Kevin Wolf wrote: >> Am 06.09.2018 um 21:29 hat Christian Borntraeger geschrieben: […snip…] >> >> I think we need more information there. Can you set a breakpoint on >>

Re: [Qemu-block] forward of qemu bug regarding random qemu hangs at shutdown

2018-09-10 Thread Marc Hartmayer
On Mon, Sep 10, 2018 at 10:21 AM +0200, Kevin Wolf wrote: > Am 07.09.2018 um 18:21 hat Marc Hartmayer geschrieben: >> On Fri, Sep 07, 2018 at 11:34 AM +0200, Kevin Wolf wrote: >> > Am 06.09.2018 um 21:29 hat Christian Borntraeger geschrieben: >> >> Kevin, >> >> >> >> for reference, it seems that

Re: [Qemu-block] [PATCH] qemu-iotests: Test snapshot=on with nonexistent TMPDIR

2018-09-10 Thread Alberto Garcia
On Mon 10 Sep 2018 11:57:48 AM CEST, Kevin Wolf wrote: > Am 10.09.2018 um 11:29 hat Alberto Garcia geschrieben: >> We just fixed a bug that was causing a use-after-free when QEMU was >> unable to create a temporary snapshot. This is a test case for this >> scenario. >> >> Signed-off-by: Alberto

Re: [Qemu-block] [Qemu-devel] [PATCH v0 0/2] Postponed actions

2018-09-10 Thread Denis Plotnikov
PING PING! On 28.08.2018 13:23, Denis Plotnikov wrote: On 27.08.2018 19:05, John Snow wrote: On 08/27/2018 03:05 AM, Denis Plotnikov wrote: PING! PING! Sorry, Kevin and Stefan are both on PTO right now, I think. I can't promise I have the time to look soon, but you at least deserve an

Re: [Qemu-block] [PATCH] qemu-iotests: Test snapshot=on with nonexistent TMPDIR

2018-09-10 Thread Kevin Wolf
Am 10.09.2018 um 11:29 hat Alberto Garcia geschrieben: > We just fixed a bug that was causing a use-after-free when QEMU was > unable to create a temporary snapshot. This is a test case for this > scenario. > > Signed-off-by: Alberto Garcia Hm, it actually doesn't crash for me even without the

Re: [Qemu-block] [PATCH] block: Fix use after free error in bdrv_open_inherit()

2018-09-10 Thread Alberto Garcia
On Mon 10 Sep 2018 10:34:20 AM CEST, Kevin Wolf wrote: > Am 06.09.2018 um 16:25 hat Alberto Garcia geschrieben: >> When a block device is opened with BDRV_O_SNAPSHOT and the >> bdrv_append_temp_snapshot() call fails then the error code path tries >> to unref the already destroyed 'options' QDict.

[Qemu-block] [PATCH] qemu-iotests: Test snapshot=on with nonexistent TMPDIR

2018-09-10 Thread Alberto Garcia
We just fixed a bug that was causing a use-after-free when QEMU was unable to create a temporary snapshot. This is a test case for this scenario. Signed-off-by: Alberto Garcia --- tests/qemu-iotests/051| 3 +++ tests/qemu-iotests/051.out| 3 +++ tests/qemu-iotests/051.pc.out | 3 +++

Re: [Qemu-block] [PATCH] block: Fix use after free error in bdrv_open_inherit()

2018-09-10 Thread Kevin Wolf
Am 06.09.2018 um 16:25 hat Alberto Garcia geschrieben: > When a block device is opened with BDRV_O_SNAPSHOT and the > bdrv_append_temp_snapshot() call fails then the error code path tries > to unref the already destroyed 'options' QDict. > > This can be reproduced easily by setting TMPDIR to a

Re: [Qemu-block] forward of qemu bug regarding random qemu hangs at shutdown

2018-09-10 Thread Kevin Wolf
Am 07.09.2018 um 18:21 hat Marc Hartmayer geschrieben: > On Fri, Sep 07, 2018 at 11:34 AM +0200, Kevin Wolf wrote: > > Am 06.09.2018 um 21:29 hat Christian Borntraeger geschrieben: > >> Kevin, > >> > >> for reference, it seems that his bug report somehow got lost. > >>

Re: [Qemu-block] [Qemu-devel] [PATCH] qemu-img.c: add help for each command

2018-09-10 Thread Kevin Wolf
Am 08.09.2018 um 05:16 hat Programmingkid geschrieben: > > > On Sep 7, 2018, at 11:13 PM, Peter Maydell wrote: > > > > On 8 September 2018 at 04:01, John Arbuckle > > wrote: > > > >> +/* print the help for this command */ > >> +if (strcmp("--help", argv[optind + 1]) == 0) { > >> +