Re: [PATCH 00/18] block: Allow exporting BDSs via FUSE

2019-12-19 Thread Max Reitz
On 19.12.19 15:38, Max Reitz wrote: [...] > Final rather important notice: I didn’t really run the iotests with this > yet. I wanted to, but they appear rather broken on current master, > actually. I’m not yet sure whether that’s because something in my setup > broke in the last two weeks, or b

[PATCH v2] iotests.py: Let wait_migration wait even more

2019-12-19 Thread Max Reitz
The "migration completed" event may be sent (on the source, to be specific) before the migration is actually completed, so the VM runstate will still be "finish-migrate" instead of "postmigrate". So ask the users of VM.wait_migration() to specify the final runstate they desire and then poll the VM

[PATCH] backup-top: Begin drain earlier

2019-12-19 Thread Max Reitz
When dropping backup-top, we need to drain the node before freeing the BlockCopyState. Otherwise, requests may still be in flight and then the assertion in shres_destroy() will fail. (This becomes visible in intermittent failure of 056.) Cc: qemu-sta...@nongnu.org Signed-off-by: Max Reitz ---

Re: [PATCH v4 22/24] nvme: bump controller pci device id

2019-12-19 Thread Klaus Birkelund Jensen
On Dec 20 02:46, Keith Busch wrote: > On Thu, Dec 19, 2019 at 06:24:57PM +0100, Klaus Birkelund Jensen wrote: > > On Dec 20 01:16, Keith Busch wrote: > > > On Thu, Dec 19, 2019 at 02:09:19PM +0100, Klaus Jensen wrote: > > > > @@ -2480,7 +2480,7 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice >

Re: [PATCH v4 22/24] nvme: bump controller pci device id

2019-12-19 Thread Keith Busch
On Thu, Dec 19, 2019 at 06:24:57PM +0100, Klaus Birkelund Jensen wrote: > On Dec 20 01:16, Keith Busch wrote: > > On Thu, Dec 19, 2019 at 02:09:19PM +0100, Klaus Jensen wrote: > > > @@ -2480,7 +2480,7 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice > > > *pci_dev) > > > pci_conf[PCI_INTER

[PULL 25/30] iotests: 266: Convert to VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
Instead of having a separate blockdev_create() function, make use of the VM.blockdev_create() offered by iotests.py. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/266 | 69 +- tests/qemu-iotests/266.out | 14 2 files changed, 44 insertions(+),

[PULL 24/30] iotests: 237: Convert to VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
Instead of having a separate blockdev_create() function, make use of the VM.blockdev_create() offered by iotests.py. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/237 | 139 +++-- 1 file changed, 65 insertions(+), 74 deletions(-) diff --git a/tests/qemu-io

Re: [PATCH v4 21/24] nvme: support multiple namespaces

2019-12-19 Thread Klaus Birkelund Jensen
On Dec 19 16:11, Michal Prívozník wrote: > On 12/19/19 2:09 PM, Klaus Jensen wrote: > > This adds support for multiple namespaces by introducing a new 'nvme-ns' > > device model. The nvme device creates a bus named from the device name > > ('id'). The nvme-ns devices then connect to this and regist

[PULL 29/30] hmp: Allow using qdev ID for qemu-io command

2019-12-19 Thread Kevin Wolf
In order to issue requests on an existing BlockBackend with the 'qemu-io' HMP command, allow specifying the BlockBackend not only with a BlockBackend name, but also with a qdev ID/QOM path for a device that owns the (possibly anonymous) BlockBackend. Because qdev names could be conflicting with Bl

[PULL 30/30] iotests: Test external snapshot with VM state

2019-12-19 Thread Kevin Wolf
This tests creating an external snapshot with VM state (which results in an active overlay over an inactive backing file, which is also the root node of an inactive BlockBackend), re-activating the images and performing some operations to test that the re-activation worked as intended. Signed-off-

[PULL 26/30] iotests: 207: Remove duplication with VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
The blockdev_create() function in this test case adds another filter to the logging, but provides otherwise the same functionality as VM.blockdev_create() from iotests.py. Make it a thin wrapper around the iotests.py function. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/207 | 8 +--- 1

[PULL 28/30] block: Activate recursively even for already active nodes

2019-12-19 Thread Kevin Wolf
bdrv_invalidate_cache_all() assumes that all nodes in a given subtree are either active or inactive when it starts. Therefore, as soon as it arrives at an already active node, it stops. However, this assumption is wrong. For example, it's possible to take a snapshot of an inactive node, which resu

[PULL 27/30] iotests: 211: Remove duplication with VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
The blockdev_create() function in this test case adds an error check that skips the test in case of failure because of memory shortage, but provides otherwise the same functionality as VM.blockdev_create() from iotests.py. Make it a thin wrapper around the iotests.py function. Signed-off-by: Kevin

[PULL 21/30] iotests: 210: Convert to VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
Instead of having a separate blockdev_create() function, make use of the VM.blockdev_create() offered by iotests.py. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/210 | 81 +++--- 1 file changed, 36 insertions(+), 45 deletions(-) diff --git a/tests/qemu-io

[PULL 20/30] iotests: 206: Convert to VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
Instead of having a separate blockdev_create() function, make use of the VM.blockdev_create() offered by iotests.py. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/206 | 232 - 1 file changed, 111 insertions(+), 121 deletions(-) diff --git a/tests/qemu-

[PULL 18/30] iotests: Create VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
We have several almost identical copies of a blockdev_create() function in different test cases. Time to create one unified function in iotests.py. To keep the diff managable, this patch only creates the function and follow-up patches will convert the individual test cases. Signed-off-by: Kevin W

[PULL 17/30] qcow2: Move error check of local_err near its assignment

2019-12-19 Thread Kevin Wolf
From: Tuguoyi The local_err check outside of the if block was necessary when it was introduced in commit d1258dd0c87 because it needed to be executed even if qcow2_load_autoloading_dirty_bitmaps() returned false. After some modifications that all required the error check to remain where it is, c

[PULL 22/30] iotests: 212: Convert to VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
Instead of having a separate blockdev_create() function, make use of the VM.blockdev_create() offered by iotests.py. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/212 | 101 +++-- 1 file changed, 46 insertions(+), 55 deletions(-) diff --git a/tests/qemu-io

[PULL 11/30] iotests: Support job-complete in run_job()

2019-12-19 Thread Kevin Wolf
Automatically complete jobs that have a 'ready' state and need an explicit job-complete. Without this, run_job() would hang for such jobs. Signed-off-by: Kevin Wolf Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Stefan Hajnoczi ---

[PULL 23/30] iotests: 213: Convert to VM.blockdev_create()

2019-12-19 Thread Kevin Wolf
Instead of having a separate blockdev_create() function, make use of the VM.blockdev_create() offered by iotests.py. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/213 | 113 +++-- 1 file changed, 52 insertions(+), 61 deletions(-) diff --git a/tests/qemu-io

[PULL 15/30] iotests/273: Filter format-specific information

2019-12-19 Thread Kevin Wolf
From: Max Reitz Doing this allows running this test with e.g. -o compat=0.10 or -o compat=refcount_bits=1. Signed-off-by: Max Reitz Signed-off-by: Kevin Wolf --- tests/qemu-iotests/273 | 3 ++- tests/qemu-iotests/273.out | 27 --- 2 files changed, 2 insertions(+),

[PULL 16/30] iotests: Fix IMGOPTSSYNTAX for nbd

2019-12-19 Thread Kevin Wolf
From: Max Reitz There is no $SOCKDIR, only $SOCK_DIR. Fixes: f3923a72f199b2c63747a7032db74730546f55c6 Signed-off-by: Max Reitz Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Kevin Wolf --- tests/qemu-iotests/common.rc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/

[PULL 09/30] iotests: Add qemu_io_log()

2019-12-19 Thread Kevin Wolf
Add a function that runs qemu-io and logs the output with the appropriate filters applied. Signed-off-by: Kevin Wolf Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by: Stefan Hajnoczi --- tests/qemu-iotests/iotests.py | 5 + 1 file

[PULL 12/30] qcow2: Use offset_into_cluster()

2019-12-19 Thread Kevin Wolf
From: Alberto Garcia There's a couple of places left in the qcow2 code that still do the calculation manually, so let's replace them. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Signed-off-by: Kevin Wolf --- block/qcow2.c | 8 +++- 1 file changed, 3 insertions(+), 5 deletions(-

[PULL 10/30] iotests: Fix timeout in run_job()

2019-12-19 Thread Kevin Wolf
run_job() accepts a wait parameter for a timeout, but it doesn't actually use it. The only thing that is missing is passing it to events_wait(), so do that now. Signed-off-by: Kevin Wolf Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Reviewed-by:

[PULL 19/30] iotests: 255: Drop blockdev_create()

2019-12-19 Thread Kevin Wolf
blockdev_create() is completely unused in this test case, so we can just drop it. Signed-off-by: Kevin Wolf --- tests/qemu-iotests/255 | 10 -- 1 file changed, 10 deletions(-) diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255 index 3632d507d0..0ba03d9e61 100755 --- a/tests/qe

[PULL 07/30] blockjob: Fix error message for negative speed

2019-12-19 Thread Kevin Wolf
The error message for a negative speed uses QERR_INVALID_PARAMETER, which implies that the 'speed' option doesn't even exist: {"error": {"class": "GenericError", "desc": "Invalid parameter 'speed'"}} Make it use QERR_INVALID_PARAMETER_VALUE instead: {"error": {"class": "GenericError", "d

[PULL 14/30] iotests: Add more "_require_drivers" checks to the shell-based tests

2019-12-19 Thread Kevin Wolf
From: Thomas Huth Test 051 should be skipped if nbd is not available, and 267 should be skipped if copy-on-read is not enabled. Signed-off-by: Thomas Huth Reviewed-by: Eric Blake Signed-off-by: Kevin Wolf --- tests/qemu-iotests/051 | 1 + tests/qemu-iotests/267 | 1 + 2 files changed, 2 inse

[PULL 03/30] iotests: Skip test 079 if it is not possible to create large files

2019-12-19 Thread Kevin Wolf
From: Thomas Huth Test 079 fails in the arm64, s390x and ppc64le LXD containers on Travis (which we will hopefully enable in our CI soon). These containers apparently do not allow large files to be created. Test 079 tries to create a 4G sparse file, which is apparently already too big for these c

[PULL 13/30] MAINTAINERS: fix qcow2-bitmap.c under Dirty Bitmaps header

2019-12-19 Thread Kevin Wolf
From: Vladimir Sementsov-Ogievskiy Somehow I wrote not full path to the file. Fix that. Also, while being here, rearrange entries, so that includes go first, then block, than migration, than util. Fixes: 052db8e71444d Signed-off-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Kevin Wolf ---

Re: [PATCH v4 22/24] nvme: bump controller pci device id

2019-12-19 Thread Klaus Birkelund Jensen
On Dec 20 01:16, Keith Busch wrote: > On Thu, Dec 19, 2019 at 02:09:19PM +0100, Klaus Jensen wrote: > > @@ -2480,7 +2480,7 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice > > *pci_dev) > > pci_conf[PCI_INTERRUPT_PIN] = 1; > > pci_config_set_prog_interface(pci_conf, 0x2); > > pci

[PULL 08/30] qcow2: Declare BDRV_REQ_NO_FALLBACK supported

2019-12-19 Thread Kevin Wolf
In the common case, qcow2_co_pwrite_zeroes() already only modifies metadata case, so we're fine with or without BDRV_REQ_NO_FALLBACK set. The only exception is when using an external data file, where the request is passed down to the block driver of the external data file. We are forwarding the BD

[PULL 05/30] qemu-img: fix info --backing-chain --image-opts

2019-12-19 Thread Kevin Wolf
From: Stefan Hajnoczi Only apply --image-opts to the topmost image when listing an entire backing chain. It is incorrect to treat backing filenames as image options. Assuming we have the backing chain t.IMGFMT.base <- t.IMGFMT.mid <- t.IMGFMT, qemu-img info fails as follows: $ qemu-img info

[PULL 01/30] iotests: Provide a function for checking the creation of huge files

2019-12-19 Thread Kevin Wolf
From: Thomas Huth Some tests create huge (but sparse) files, and to be able to run those tests in certain limited environments (like CI containers), we have to check for the possibility to create such files first. Thus let's introduce a common function to check for large files, and replace the al

[PULL 02/30] iotests: Skip test 060 if it is not possible to create large files

2019-12-19 Thread Kevin Wolf
From: Thomas Huth Test 060 fails in the arm64, s390x and ppc64le LXD containers on Travis (which we will hopefully enable in our CI soon). These containers apparently do not allow large files to be created. The repair process in test 060 creates a file of 64 GiB, so test first whether such large

[PULL 04/30] qapi: better document NVMe blockdev @device parameter

2019-12-19 Thread Kevin Wolf
From: Daniel P. Berrangé Mention that this is a PCI device address & give the format it is expected in. Also mention that it must be first unbound from any host kernel driver. Signed-off-by: Daniel P. Berrangé Reviewed-by: Stefan Hajnoczi Reviewed-by: Eric Blake Signed-off-by: Kevin Wolf ---

[PULL 06/30] block: Error out on image creation with conflicting size options

2019-12-19 Thread Kevin Wolf
If both the create options (qemu-img create -o ...) and the size parameter were given, the size parameter was silently ignored. Instead, make specifying two sizes an error. Signed-off-by: Kevin Wolf Reviewed-by: Eric Blake --- block.c| 10 -- tests/qemu-iotests/049

[PULL 00/30] Block layer patches

2019-12-19 Thread Kevin Wolf
The following changes since commit aceeaa69d28e6f08a24395d0aa6915b687d0a681: Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2019-12-17' into staging (2019-12-17 15:55:20 +) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream f

Re: [PATCH] xen-bus/block: explicitly assign event channels to an AioContext

2019-12-19 Thread Anthony PERARD
On Mon, Dec 16, 2019 at 02:34:51PM +, Paul Durrant wrote: > It is not safe to close an event channel from the QEMU main thread when > that channel's poller is running in IOThread context. > > This patch adds a new xen_device_set_event_channel_context() function > to explicitly assign the chann

Re: [PATCH] iotests.py: Wait for postmigrate in wait_migration

2019-12-19 Thread Kevin Wolf
Am 19.12.2019 um 17:01 hat Kevin Wolf geschrieben: > Am 19.12.2019 um 16:51 hat Max Reitz geschrieben: > > Without this patch, I see intermittent failures in the new iotest 280 > > under high system load. I have not yet seen such failures with other > > iotests that use VM.wait_migration() and que

Re: [PATCH v5 4/4] blockdev: honor bdrv_try_set_aio_context() context requirements

2019-12-19 Thread Sergio Lopez
Sergio Lopez writes: > Kevin Wolf writes: > >> Am 13.12.2019 um 21:59 hat Eric Blake geschrieben: >>> On 12/9/19 10:06 AM, Kevin Wolf wrote: >>> > Am 28.11.2019 um 11:41 hat Sergio Lopez geschrieben: >>> > > bdrv_try_set_aio_context() requires that the old context is held, and >>> > > the new c

Re: [PATCH v4 22/24] nvme: bump controller pci device id

2019-12-19 Thread Keith Busch
On Thu, Dec 19, 2019 at 02:09:19PM +0100, Klaus Jensen wrote: > @@ -2480,7 +2480,7 @@ static void nvme_init_pci(NvmeCtrl *n, PCIDevice > *pci_dev) > pci_conf[PCI_INTERRUPT_PIN] = 1; > pci_config_set_prog_interface(pci_conf, 0x2); > pci_config_set_vendor_id(pci_conf, PCI_VENDOR_ID_IN

Re: [PATCH] qapi/block: fix nbd-server-add spec

2019-12-19 Thread Nir Soffer
On Thu, Dec 19, 2019 at 5:25 PM Vladimir Sementsov-Ogievskiy wrote: > > 19.12.2019 18:08, Nir Soffer wrote: > > On Thu, Dec 19, 2019 at 5:00 PM Vladimir Sementsov-Ogievskiy > > wrote: > >> > >> 19.12.2019 17:42, Nir Soffer wrote: > >>> On Thu, Dec 19, 2019 at 4:34 PM Vladimir Sementsov-Ogievskiy

Re: [PATCH] iotests.py: Wait for postmigrate in wait_migration

2019-12-19 Thread Kevin Wolf
Am 19.12.2019 um 16:51 hat Max Reitz geschrieben: > Without this patch, I see intermittent failures in the new iotest 280 > under high system load. I have not yet seen such failures with other > iotests that use VM.wait_migration() and query-status afterwards, but > maybe they just occur even more

Re: [PATCH] block: nbd: Fix dirty bitmap context name

2019-12-19 Thread Nir Soffer
On Thu, Dec 19, 2019 at 5:17 PM Vladimir Sementsov-Ogievskiy wrote: > > 19.12.2019 17:59, Nir Soffer wrote: > > On Thu, Dec 19, 2019 at 4:04 PM Kevin Wolf wrote: > >> > >> Am 19.12.2019 um 14:41 hat Vladimir Sementsov-Ogievskiy geschrieben: > >>> Ahh, I see, it's documented as > >>> > >>> +# @bit

[PATCH] iotests.py: Wait for postmigrate in wait_migration

2019-12-19 Thread Max Reitz
Without this patch, I see intermittent failures in the new iotest 280 under high system load. I have not yet seen such failures with other iotests that use VM.wait_migration() and query-status afterwards, but maybe they just occur even more rarely. Signed-off-by: Max Reitz --- tests/qemu-iotest

Re: [PATCH 3/3] iotests: Test external snapshot with VM state

2019-12-19 Thread Kevin Wolf
Am 19.12.2019 um 15:26 hat Max Reitz geschrieben: > On 17.12.19 15:59, Kevin Wolf wrote: > > This tests creating an external snapshot with VM state (which results in > > an active overlay over an inactive backing file, which is also the root > > node of an inactive BlockBackend), re-activating the

Re: [PATCH v2 2/2] ide: Fix incorrect handling of some PRDTs in ide_dma_cb()

2019-12-19 Thread Alexander Popov
Hello Kevin! Thanks again for your review. On 19.12.2019 18:01, Kevin Wolf wrote: > Am 16.12.2019 um 19:14 hat Alexander Popov geschrieben: >> The commit a718978ed58a from July 2015 introduced the assertion which >> implies that the size of successful DMA transfers handled in ide_dma_cb() >> shou

Re: [PATCH v2 1/2] tests/ide-test: Create a single unit-test covering more PRDT cases

2019-12-19 Thread Alexander Popov
Hello Kevin, Thanks for your review! On 19.12.2019 18:12, Kevin Wolf wrote: > Am 16.12.2019 um 19:14 hat Alexander Popov geschrieben: >> Fuzzing the Linux kernel with syzkaller allowed to find how to crash qemu >> using a special SCSI_IOCTL_SEND_COMMAND. It hits the assertion in >> ide_dma_cb() i

Re: [PATCH] qapi/block: fix nbd-server-add spec

2019-12-19 Thread Vladimir Sementsov-Ogievskiy
19.12.2019 18:08, Nir Soffer wrote: > On Thu, Dec 19, 2019 at 5:00 PM Vladimir Sementsov-Ogievskiy > wrote: >> >> 19.12.2019 17:42, Nir Soffer wrote: >>> On Thu, Dec 19, 2019 at 4:34 PM Vladimir Sementsov-Ogievskiy >>> wrote: "NAME" here may be interpreted like it should match @name, wh

Re: [PATCH] block: nbd: Fix dirty bitmap context name

2019-12-19 Thread Vladimir Sementsov-Ogievskiy
19.12.2019 17:59, Nir Soffer wrote: > On Thu, Dec 19, 2019 at 4:04 PM Kevin Wolf wrote: >> >> Am 19.12.2019 um 14:41 hat Vladimir Sementsov-Ogievskiy geschrieben: >>> Ahh, I see, it's documented as >>> >>> +# @bitmap: Also export the dirty bitmap reachable from @device, so the >>> +# NBD

Re: [PATCH v2 1/2] tests/ide-test: Create a single unit-test covering more PRDT cases

2019-12-19 Thread Kevin Wolf
Am 16.12.2019 um 19:14 hat Alexander Popov geschrieben: > Fuzzing the Linux kernel with syzkaller allowed to find how to crash qemu > using a special SCSI_IOCTL_SEND_COMMAND. It hits the assertion in > ide_dma_cb() introduced in the commit a718978ed58a in July 2015. > Currently this bug is not repr

Re: [PATCH v4 21/24] nvme: support multiple namespaces

2019-12-19 Thread Michal Prívozník
On 12/19/19 2:09 PM, Klaus Jensen wrote: > This adds support for multiple namespaces by introducing a new 'nvme-ns' > device model. The nvme device creates a bus named from the device name > ('id'). The nvme-ns devices then connect to this and registers > themselves with the nvme device. > > This

Re: [PATCH] qapi/block: fix nbd-server-add spec

2019-12-19 Thread Nir Soffer
On Thu, Dec 19, 2019 at 5:00 PM Vladimir Sementsov-Ogievskiy wrote: > > 19.12.2019 17:42, Nir Soffer wrote: > > On Thu, Dec 19, 2019 at 4:34 PM Vladimir Sementsov-Ogievskiy > > wrote: > >> > >> "NAME" here may be interpreted like it should match @name, which is > >> export name. But it was never

Re: [PATCH] block: nbd: Fix dirty bitmap context name

2019-12-19 Thread Vladimir Sementsov-Ogievskiy
19.12.2019 17:49, Nir Soffer wrote: > On Thu, Dec 19, 2019 at 3:42 PM Vladimir Sementsov-Ogievskiy > wrote: >> >> I'd not call it a "fix".. As it implies something broken. >> >> [edit: OK, now I see that something is broken, and why you called it "fix", >>see below] >> >> 19.12.2019 15:51, Nir

Re: [PATCH v2 2/2] ide: Fix incorrect handling of some PRDTs in ide_dma_cb()

2019-12-19 Thread Kevin Wolf
Am 16.12.2019 um 19:14 hat Alexander Popov geschrieben: > The commit a718978ed58a from July 2015 introduced the assertion which > implies that the size of successful DMA transfers handled in ide_dma_cb() > should be multiple of 512 (the size of a sector). But guest systems can > initiate DMA transf

Re: [PATCH] qapi/block: fix nbd-server-add spec

2019-12-19 Thread Vladimir Sementsov-Ogievskiy
19.12.2019 17:42, Nir Soffer wrote: > On Thu, Dec 19, 2019 at 4:34 PM Vladimir Sementsov-Ogievskiy > wrote: >> >> "NAME" here may be interpreted like it should match @name, which is >> export name. But it was never mentioned in such way. Make it obvious, >> that actual "" (see docs/interop/nbd.txt

Re: [PATCH] block: nbd: Fix dirty bitmap context name

2019-12-19 Thread Nir Soffer
On Thu, Dec 19, 2019 at 4:04 PM Kevin Wolf wrote: > > Am 19.12.2019 um 14:41 hat Vladimir Sementsov-Ogievskiy geschrieben: > > Ahh, I see, it's documented as > > > > +# @bitmap: Also export the dirty bitmap reachable from @device, so the > > +# NBD client can use NBD_OPT_SET_META_CONTEXT

[PATCH 16/18] iotests: Allow testing FUSE exports

2019-12-19 Thread Max Reitz
This pretends FUSE exports are a kind of protocol. As such, they are always tested under the format node. This is probably the best way to test them, actually, because this will generate more I/O load and more varied patterns. Signed-off-by: Max Reitz --- tests/qemu-iotests/check | 6

Re: [PATCH] block: nbd: Fix dirty bitmap context name

2019-12-19 Thread Nir Soffer
On Thu, Dec 19, 2019 at 3:42 PM Vladimir Sementsov-Ogievskiy wrote: > > I'd not call it a "fix".. As it implies something broken. > > [edit: OK, now I see that something is broken, and why you called it "fix", > see below] > > 19.12.2019 15:51, Nir Soffer wrote: > > When adding an export with a

[PATCH 14/18] iotests: Restrict some Python tests to file

2019-12-19 Thread Max Reitz
Most Python tests are restricted to the file protocol (without explicitly saying so), but these are the ones that would break ./check -fuse -qcow2. Signed-off-by: Max Reitz --- tests/qemu-iotests/206 | 1 + tests/qemu-iotests/242 | 1 + 2 files changed, 2 insertions(+) diff --git a/tests/qemu-i

[PATCH] iotests/279: Fix for non-qcow2 formats

2019-12-19 Thread Max Reitz
First, driver=qcow2 will not work so well with non-qcow2 formats (and this test claims to support qcow, qed, and vmdk). Second, vmdk will always report the backing file format to be vmdk. Filter that out so the output looks like for all other formats. Third, the flat vmdk subformats do not suppor

Re: [PATCH] qapi/block: fix nbd-server-add spec

2019-12-19 Thread Nir Soffer
On Thu, Dec 19, 2019 at 4:34 PM Vladimir Sementsov-Ogievskiy wrote: > > "NAME" here may be interpreted like it should match @name, which is > export name. But it was never mentioned in such way. Make it obvious, > that actual "" (see docs/interop/nbd.txt) > will match @bitmap parameter. But this

[PATCH 18/18] iotests/281: Add test for FUSE exports

2019-12-19 Thread Max Reitz
We have good coverage of the normal I/O paths now, but what remains is a test that tests some more special cases: Exporting an image on itself (thus turning a formatted image into a raw one), some error cases, and non-writable and non-growable exports. Signed-off-by: Max Reitz --- tests/qemu-iot

[PATCH 15/18] iotests: Let _make_test_img guess $TEST_IMG_FILE

2019-12-19 Thread Max Reitz
When most iotests want to create a test image that is named differently from the default $TEST_IMG, they do something like this: TEST_IMG="$TEST_IMG.base" _make_test_img $options This works fine with the "file" protocol, but not so much for anything else: _make_test_img tries to create an ima

[PATCH 10/18] iotests: Use convert -n in some cases

2019-12-19 Thread Max Reitz
qemu-img convert (without -n) can often be replaced by a combination of _make_test_img + qemu-img convert -n. Doing so allows converting to protocols that do not allow direct file creation, such as FUSE exports. So do it for some iotests, so they can run on FUSE exports. Note that doing this all

[PATCH 11/18] iotests: Avoid renaming images

2019-12-19 Thread Max Reitz
This generally does not work on non-file protocols. It is better to create the image with the final name from the start, and most tests do this already. Let 013 and 046 follow suit. Signed-off-by: Max Reitz --- tests/qemu-iotests/013 | 4 +++- tests/qemu-iotests/013.out | 2 +- tests/qemu-

[PATCH 08/18] iotests: Do not needlessly filter _make_test_img

2019-12-19 Thread Max Reitz
In most cases, _make_test_img does not need a _filter_imgfmt on top. It does that by itself. (The exception is when IMGFMT has been overwritten but TEST_IMG has not. In such cases, we do need a _filter_imgfmt on top to filter the test's original IMGFMT from TEST_IMG.) Signed-off-by: Max Reitz -

[PATCH 13/18] iotests/091: Use _cleanup_qemu instad of "wait"

2019-12-19 Thread Max Reitz
If the test environment has some other child processes running (like a storage daemon that provides a FUSE export), then "wait" will never finish. Use wait=yes _cleanup_qemu instead. (We need to discard the output so there is no change to the reference output.) Signed-off-by: Max Reitz --- tes

[PATCH 12/18] iotests: Derive image names from $TEST_IMG

2019-12-19 Thread Max Reitz
Avoid creating images with custom filenames in $TEST_DIR, because non-file protocols may want to keep $TEST_IMG (and all other test images) in some other directory. Signed-off-by: Max Reitz --- tests/qemu-iotests/200 | 3 +-- tests/qemu-iotests/200.out | 4 ++-- tests/qemu-iotests/229 |

[PATCH 17/18] iotests: Enable fuse for many tests

2019-12-19 Thread Max Reitz
Many tests (that do not support generic protocols) can run just fine with FUSE-exported images, so allow them to. Note that 221 and 250 only pass when .lseek is correctly implemented, which is only possible with a libfuse that is 3.8 or newer. Signed-off-by: Max Reitz --- tests/qemu-iotests/013

[PATCH 02/18] fuse: Allow exporting BDSs via FUSE

2019-12-19 Thread Max Reitz
fuse-export-add allows mounting block graph nodes via FUSE on some existing regular file. That file should then appears like a raw disk image, and accesses to it result in accesses to the exported BDS. Right now, we only set up the mount point and tear all mount points down in bdrv_close_all().

[PATCH 04/18] fuse: Add fuse-export-remove

2019-12-19 Thread Max Reitz
Signed-off-by: Max Reitz --- block/fuse.c| 16 qapi/block.json | 15 +++ 2 files changed, 31 insertions(+) diff --git a/block/fuse.c b/block/fuse.c index f18e8e7591..4e662e6dfb 100644 --- a/block/fuse.c +++ b/block/fuse.c @@ -123,6 +123,22 @@ fail: close_fus

[PATCH 07/18] fuse: Implement hole detection through lseek

2019-12-19 Thread Max Reitz
This is a relatively new feature in libfuse (available since 3.8.0, which was released in November 2019), so we have to let configure check whether it is available before making use of it. Signed-off-by: Max Reitz --- block/fuse.c | 77 config

[PATCH 09/18] iotests: Do not pipe _make_test_img

2019-12-19 Thread Max Reitz
Executing _make_test_img as part of a pipe will undo all variable changes it has done. As such, this could not work with FUSE (because we want to remember all of our exports and their qemu instances). Replace the pipe by a temporary file in 071 and 174 (the two tests that can run on FUSE). Signe

[PATCH 06/18] fuse: (Partially) implement fallocate()

2019-12-19 Thread Max Reitz
This allows allocating areas after the EOF, writing zeroes, and discarding. Signed-off-by: Max Reitz --- block/fuse.c | 79 1 file changed, 79 insertions(+) diff --git a/block/fuse.c b/block/fuse.c index d7c7824815..018afee6cd 100644 --- a/bl

[PATCH 03/18] fuse: Implement standard FUSE operations

2019-12-19 Thread Max Reitz
This makes the export actually useful instead of only producing errors whenever it is accessed. Signed-off-by: Max Reitz --- block/fuse.c | 222 +++ 1 file changed, 222 insertions(+) diff --git a/block/fuse.c b/block/fuse.c index 3a22579dca..f18e8

[PATCH 05/18] fuse: Allow growable exports

2019-12-19 Thread Max Reitz
These will behave more like normal files in that writes beyond the EOF will automatically grow the export size. Signed-off-by: Max Reitz --- block/fuse.c| 16 +++- qapi/block.json | 6 +- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/block/fuse.c b/block/fu

[PATCH 01/18] configure: Detect libfuse

2019-12-19 Thread Max Reitz
Signed-off-by: Max Reitz --- configure | 35 +++ 1 file changed, 35 insertions(+) diff --git a/configure b/configure index 84b413dbfc..ff7d760a0a 100755 --- a/configure +++ b/configure @@ -503,6 +503,7 @@ debug_mutex="no" libpmem="" default_devices="yes" plugin

[PATCH 00/18] block: Allow exporting BDSs via FUSE

2019-12-19 Thread Max Reitz
Preamble: This series is based on a combination of my (current) block branch and “iotests: Minor fixes”. I’ve pushed it here: https://git.xanclic.moe/XanClic/qemu fuse-exports-v1 (The base is on fuse-exports-v1-base.) Hi, Ever since I found out that you can mount FUSE filesystems on regular

[PATCH] qapi/block: fix nbd-server-add spec

2019-12-19 Thread Vladimir Sementsov-Ogievskiy
"NAME" here may be interpreted like it should match @name, which is export name. But it was never mentioned in such way. Make it obvious, that actual "" (see docs/interop/nbd.txt) will match @bitmap parameter. Fixes: 5fcbeb06812685a2 Signed-off-by: Vladimir Sementsov-Ogievskiy --- Hi all. This

Re: [PATCH 3/3] iotests: Test external snapshot with VM state

2019-12-19 Thread Max Reitz
On 17.12.19 15:59, Kevin Wolf wrote: > This tests creating an external snapshot with VM state (which results in > an active overlay over an inactive backing file, which is also the root > node of an inactive BlockBackend), re-activating the images and > performing some operations to test that the r

Re: [PATCH] block: nbd: Fix dirty bitmap context name

2019-12-19 Thread Vladimir Sementsov-Ogievskiy
19.12.2019 17:04, Kevin Wolf wrote: > Am 19.12.2019 um 14:41 hat Vladimir Sementsov-Ogievskiy geschrieben: >> Ahh, I see, it's documented as >> >> +# @bitmap: Also export the dirty bitmap reachable from @device, so the >> +# NBD client can use NBD_OPT_SET_META_CONTEXT with >> +# "

[PATCH v4 23/24] nvme: remove redundant NvmeCmd pointer parameter

2019-12-19 Thread Klaus Jensen
The command struct is available in the NvmeRequest that we generally pass around anyway. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 198 1 file changed, 99 insertions(+), 99 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index

[PATCH v4 17/24] nvme: allow multiple aios per command

2019-12-19 Thread Klaus Jensen
This refactors how the device issues asynchronous block backend requests. The NvmeRequest now holds a queue of NvmeAIOs that are associated with the command. This allows multiple aios to be issued for a command. Only when all requests have been completed will the device post a completion queue entr

[PATCH v4 13/24] nvme: additional tracing

2019-12-19 Thread Klaus Jensen
Add a trace call for nvme_enqueue_req_completion. Also, streamline nvme_identify_ns and nvme_identify_ns_list. They do not need to repeat the command, it is already in the trace name. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 8 +--- hw/block/trace-events | 5 +++-- 2 files ch

Re: [PATCH] block: nbd: Fix dirty bitmap context name

2019-12-19 Thread Kevin Wolf
Am 19.12.2019 um 14:41 hat Vladimir Sementsov-Ogievskiy geschrieben: > Ahh, I see, it's documented as > > +# @bitmap: Also export the dirty bitmap reachable from @device, so the > +# NBD client can use NBD_OPT_SET_META_CONTEXT with > +# "qemu:dirty-bitmap:NAME" to inspect the bit

[PATCH v4 21/24] nvme: support multiple namespaces

2019-12-19 Thread Klaus Jensen
This adds support for multiple namespaces by introducing a new 'nvme-ns' device model. The nvme device creates a bus named from the device name ('id'). The nvme-ns devices then connect to this and registers themselves with the nvme device. This changes how an nvme device is created. Example with t

[PATCH v4 12/24] nvme: add missing mandatory features

2019-12-19 Thread Klaus Jensen
Add support for returning a resonable response to Get/Set Features of mandatory features. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 58 --- hw/block/trace-events | 2 ++ include/block/nvme.h | 3 ++- 3 files changed, 59 insertions(+), 4 de

[PATCH v4 22/24] nvme: bump controller pci device id

2019-12-19 Thread Klaus Jensen
Since commits 9d6459d21a6e ("nvme: fix write zeroes offset and count") and c7fe50bcf1f1 ("nvme: support multiple namespaces") the controller device no longer has the quirks that the Linux kernel think it has. As the quirks are applied based on pci vendor and device id, bump the device id to get ri

[PATCH v4 05/24] nvme: populate the mandatory subnqn and ver fields

2019-12-19 Thread Klaus Jensen
Required for compliance with NVMe revision 1.2.1 or later. See NVM Express 1.2.1, Section 5.11 ("Identify command"), Figure 90 and Section 7.9 ("NVMe Qualified Names"). This also bumps the supported version to 1.2.1. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 13 ++--- 1 file cha

[PATCH v4 15/24] nvme: bump supported specification to 1.3

2019-12-19 Thread Klaus Jensen
Add new fields to the Identify Controller and Identify Namespace data structures accoding to NVM Express 1.3d. NVM Express 1.3d requires the following additional features: - addition of the Namespace Identification Descriptor List (CNS 03h) for the Identify command - support for returning

[PATCH v4 03/24] nvme: move device parameters to separate struct

2019-12-19 Thread Klaus Jensen
Move device configuration parameters to separate struct to make it explicit what is configurable and what is set internally. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 44 ++-- hw/block/nvme.h | 16 +--- 2 files changed, 35 insertions(+)

[PATCH v4 04/24] nvme: add missing fields in the identify data structures

2019-12-19 Thread Klaus Jensen
Not used by the device model but added for completeness. See NVM Express 1.2.1, Section 5.11 ("Identify command"), Figure 90 and Figure 93. Signed-off-by: Klaus Jensen --- include/block/nvme.h | 48 1 file changed, 40 insertions(+), 8 deletions(-) di

[PATCH v4 07/24] nvme: add support for the abort command

2019-12-19 Thread Klaus Jensen
Required for compliance with NVMe revision 1.2.1. See NVM Express 1.2.1, Section 5.1 ("Abort command"). The Abort command is a best effort command; for now, the device always fails to abort the given command. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 28 1 f

[PATCH v4 09/24] nvme: add temperature threshold feature

2019-12-19 Thread Klaus Jensen
It might seem wierd to implement this feature for an emulated device, but it is mandatory to support and the feature is useful for testing asynchronous event request support, which will be added in a later patch. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 47 +

[PATCH v4 06/24] nvme: refactor nvme_addr_read

2019-12-19 Thread Klaus Jensen
Pull the controller memory buffer check to its own function. The check will be used on its own in later patches. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 18 +- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index 8f7049d

[PATCH v4 10/24] nvme: add support for the get log page command

2019-12-19 Thread Klaus Jensen
Add support for the Get Log Page command and basic implementations of the mandatory Error Information, SMART / Health Information and Firmware Slot Information log pages. In violation of the specification, the SMART / Health Information log page does not persist information over the lifetime of th

[PATCH v4 02/24] nvme: remove superfluous breaks

2019-12-19 Thread Klaus Jensen
These break statements was left over when commit 3036a626e9ef ("nvme: add Get/Set Feature Timestamp support") was merged. Signed-off-by: Klaus Jensen --- hw/block/nvme.c | 4 1 file changed, 4 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index bcb1ca77e1bd..10666a3115c7 1006

[PATCH v4 20/24] nvme: add support for scatter gather lists

2019-12-19 Thread Klaus Jensen
For now, support the Data Block, Segment and Last Segment descriptor types. See NVM Express 1.3d, Section 4.4 ("Scatter Gather List (SGL)"). Signed-off-by: Klaus Jensen Acked-by: Fam Zheng --- block/nvme.c | 18 +- hw/block/nvme.c | 379 +++--

  1   2   >