Re: [PATCH 0/6] Add debug interface to kick/call on purpose

2021-03-26 Thread Dongli Zhang
Hi Jason, On 3/26/21 12:24 AM, Jason Wang wrote: > > 在 2021/3/26 下午1:44, Dongli Zhang 写道: >> The virtio device/driver (e.g., vhost-scsi or vhost-net) may hang due to >> the loss of doorbell kick, e.g., >> >>

Re: [PATCH v2 2/2] iotests/244: Test preallocation for data-file-raw

2021-03-26 Thread Max Reitz
On 26.03.21 16:17, Eric Blake wrote: On 3/26/21 9:55 AM, Max Reitz wrote: Three test cases: (1) Adding a qcow2 (metadata) file to an existing data file, see whether we can read the existing data through the qcow2 image. (2) Append data to the data file, grow the qcow2 image accordingly,

[PATCH v5 5/6] qcow2: consider in-flight-writes when freeing clusters

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
We have a bug in qcow2: assume we've started data write into host cluster A. s->lock is unlocked. During the write the refcount of cluster A may become zero, cluster may be reallocated for other needs, and our in-flight write become a use-after-free. To fix the bug let's do the following. Or

[PATCH v5 2/6] qcow2: fix cache discarding in update_refcount()

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
Here refcount of cluster at @cluster_offset reached 0, so we "free" that cluster. Not a cluster at @offset. The thing that save us from the bug is that L2 tables and refblocks are discarded one by one. Still, let's be precise. Signed-off-by: Vladimir Sementsov-Ogievskiy ---

[PATCH v5 1/6] iotests: add qcow2-discard-during-rewrite

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
Simple test: - start writing to allocated cluster A - discard this cluster - write to another unallocated cluster B (it's allocated in same place where A was allocated) - continue writing to A For now last action pollutes cluster B which is a bug fixed by the following commit. For now,

[PATCH v5 6/6] qcow2: do not discard host clusters during in-flight writes

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
Finally, after all the preparations, when we have the whole infrastructure of inflight-write-counters prepared in previous commits, let's fix the following bug: 1. Start write to qcow2. Assume guest cluster G and corresponding host cluster is H. 2. The write requests come to the point of data

[PATCH v5 3/6] qcow2: introduce is_cluster_free() helper

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
We are going to change the concept of "free host cluster", so let's clarify it now and add a helper, which we will modify later. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/qcow2-refcount.c | 31 +-- 1 file changed, 25 insertions(+), 6 deletions(-) diff

[PATCH v5 0/6] qcow2: fix parallel rewrite and discard (lockless)

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
Hi all! It's an alternative lock-less solution to [PATCH v4 0/3] qcow2: fix parallel rewrite and discard (rw-lock) It's an updated version of what was don in [PATCH v3 0/6] qcow2: compressed write cache I've split the logic into several patches, add a lot more comments and documentation, I

Re: [PATCH V4] file-posix: allow -EBUSY error during ioctl(fd, BLKZEROOUT, range) on block

2021-03-26 Thread Nir Soffer
On Fri, Mar 26, 2021 at 3:21 AM ChangLimin wrote: > >On Thu, Mar 25, 2021 at 8:07 AM ChangLimin > wrote: > >>On Wed, Mar 24, 2021 at 4:52 PM Max Reitz wrote: > >>On 22.03.21 10:25, ChangLimin wrote: > >>> For Linux 5.10/5.11, qemu write zeros to a multipath device using > >>> ioctl(fd,

[PATCH v5 4/6] qcow2: introduce inflight-write-counters

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
We have a bug in qcow2: assume we've started data write into host cluster A. s->lock is unlocked. During the write the refcount of cluster A may become zero, cluster may be reallocated for other needs, and our in-flight write become a use-after-free. More details will be in the further commit

Re: [PATCH v2 1/2] qcow2: Force preallocation with data-file-raw

2021-03-26 Thread Eric Blake
On 3/26/21 9:55 AM, Max Reitz wrote: > Setting the qcow2 data-file-raw bit means that you can ignore the > qcow2 metadata when reading from the external data file. It does not > mean that you have to ignore it, though. Therefore, the data read must > be the same regardless of whether you

Re: [PATCH v2 2/2] iotests/244: Test preallocation for data-file-raw

2021-03-26 Thread Eric Blake
On 3/26/21 9:55 AM, Max Reitz wrote: > Three test cases: > (1) Adding a qcow2 (metadata) file to an existing data file, see whether > we can read the existing data through the qcow2 image. > (2) Append data to the data file, grow the qcow2 image accordingly, see > whether we can read the

Re: [PATCH v3 4/5] qemu-iotests: let "check" spawn an arbitrary test command

2021-03-26 Thread Max Reitz
On 26.03.21 15:23, Paolo Bonzini wrote: Right now there is no easy way for "check" to print a reproducer command. Because such a reproducer command line would be huge, we can instead teach check to start a command of our choice. This can be for example a Python unit test with arguments to only

[PATCH v2 0/2] qcow2: Force preallocation with data-file-raw

2021-03-26 Thread Max Reitz
v1: https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg00992.html Hi, I think that qcow2 images with data-file-raw should always have preallocated 1:1 L1/L2 tables, so that the image always looks the same whether you respect or ignore the qcow2 metadata. The easiest way to achieve

[PATCH v2 1/2] qcow2: Force preallocation with data-file-raw

2021-03-26 Thread Max Reitz
Setting the qcow2 data-file-raw bit means that you can ignore the qcow2 metadata when reading from the external data file. It does not mean that you have to ignore it, though. Therefore, the data read must be the same regardless of whether you interpret the metadata or whether you ignore it, and

[PATCH v2 2/2] iotests/244: Test preallocation for data-file-raw

2021-03-26 Thread Max Reitz
Three test cases: (1) Adding a qcow2 (metadata) file to an existing data file, see whether we can read the existing data through the qcow2 image. (2) Append data to the data file, grow the qcow2 image accordingly, see whether we can read the new data through the qcow2 image. (3) At

Re: [PATCH 1/4] qcow2: Improve refcount structure rebuilding

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
26.03.2021 16:47, Max Reitz wrote: On 26.03.21 12:48, Vladimir Sementsov-Ogievskiy wrote: 10.03.2021 18:59, Max Reitz wrote: When rebuilding the refcount structures (when qemu-img check -r found errors with refcount = 0, but reference count > 0), the new refcount table defaults to being put at

[PATCH v3 1/5] qemu-iotests: do not buffer the test output

2021-03-26 Thread Paolo Bonzini
Instead of buffering the test output into a StringIO, patch it on the fly by wrapping sys.stdout's write method. This can be done unconditionally, even if using -d, which makes execute_unittest a bit simpler. Signed-off-by: Paolo Bonzini Reviewed-by: Vladimir Sementsov-Ogievskiy Tested-by:

[PATCH v3 2/5] qemu-iotests: allow passing unittest.main arguments to the test scripts

2021-03-26 Thread Paolo Bonzini
Python test scripts that use unittest consist of multiple tests. unittest.main allows selecting which tests to run, but currently this is not possible because the iotests wrapper ignores sys.argv. unittest.main command line options also allow the user to pick the desired options for verbosity,

[PATCH v3 3/5] qemu-iotests: move command line and environment handling from TestRunner to TestEnv

2021-03-26 Thread Paolo Bonzini
In the next patch, "check" will learn how to execute a test script without going through TestRunner. To enable this, keep only the text output and subprocess handling in the TestRunner; move into TestEnv the logic to prepare for running a subprocess. Reviewed-by: Vladimir Sementsov-Ogievskiy

[PATCH v3 5/5] qemu-iotests: fix case of SOCK_DIR already in the environment

2021-03-26 Thread Paolo Bonzini
Due to a typo, in this case the SOCK_DIR was not being created. Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Paolo Bonzini Tested-by: Emanuele Giuseppe Esposito Message-Id: <20210323181928.311862-6-pbonz...@redhat.com> --- tests/qemu-iotests/testenv.py | 2 +- 1 file changed, 1

[PATCH v3 4/5] qemu-iotests: let "check" spawn an arbitrary test command

2021-03-26 Thread Paolo Bonzini
Right now there is no easy way for "check" to print a reproducer command. Because such a reproducer command line would be huge, we can instead teach check to start a command of our choice. This can be for example a Python unit test with arguments to only run a specific subtest. Move the trailing

[PATCH v3 0/5] qemu-iotests: quality of life improvements

2021-03-26 Thread Paolo Bonzini
This series adds a few usability improvements to qemu-iotests, in particular: - arguments can be passed to Python unittests scripts, for example to run only a subset of the test cases (patches 1-2) - it is possible to do "./check -- ../../../tests/qemu-iotests/055 args..." and specify

Re: [PATCH] iotests/116: Fix reference output

2021-03-26 Thread Eric Blake
On 3/26/21 9:14 AM, Max Reitz wrote: > 15ce94a68ca ("block/qed: bdrv_qed_do_open: deal with errp") has improved > the qed driver's error reporting, though sadly did not add a test for > it. > The good news are: There already is such a test, namely 116. > The bad news are: Its reference output was

[PATCH] iotests/116: Fix reference output

2021-03-26 Thread Max Reitz
15ce94a68ca ("block/qed: bdrv_qed_do_open: deal with errp") has improved the qed driver's error reporting, though sadly did not add a test for it. The good news are: There already is such a test, namely 116. The bad news are: Its reference output was not adjusted, and so now it fails. Let's fix

Re: [PATCH 1/4] qcow2: Improve refcount structure rebuilding

2021-03-26 Thread Max Reitz
On 26.03.21 12:48, Vladimir Sementsov-Ogievskiy wrote: 10.03.2021 18:59, Max Reitz wrote: When rebuilding the refcount structures (when qemu-img check -r found errors with refcount = 0, but reference count > 0), the new refcount table defaults to being put at the image file end[1].  There is no

Re: [PATCH v2 0/5] qemu-iotests: quality of life improvements

2021-03-26 Thread Max Reitz
On 23.03.21 19:19, Paolo Bonzini wrote: This series adds a few usability improvements to qemu-iotests, in particular: - arguments can be passed to Python unittests scripts, for example to run only a subset of the test cases (patches 1-2) - it is possible to do "./check --

Re: [PATCH 1/4] qcow2: Improve refcount structure rebuilding

2021-03-26 Thread Vladimir Sementsov-Ogievskiy
10.03.2021 18:59, Max Reitz wrote: When rebuilding the refcount structures (when qemu-img check -r found errors with refcount = 0, but reference count > 0), the new refcount table defaults to being put at the image file end[1]. There is no good reason for that except that it means we will not

Re: [PULL 0/2] Block patches

2021-03-26 Thread Peter Maydell
On Wed, 24 Mar 2021 at 14:52, Stefan Hajnoczi wrote: > > The following changes since commit 67c1115edd98f388ca89dd38322ea3fadf034523: > > Merge remote-tracking branch 'remotes/kraxel/tags/ui-20210323-pull-request' > into staging (2021-03-23 23:47:30 +) > > are available in the Git

Re: [PATCH 0/7] block/nbd: decouple reconnect from drain

2021-03-26 Thread Roman Kagan
On Wed, Mar 17, 2021 at 11:35:31AM +0300, Vladimir Sementsov-Ogievskiy wrote: > 15.03.2021 09:06, Roman Kagan wrote: > > The reconnection logic doesn't need to stop while in a drained section. > > Moreover it has to be active during the drained section, as the requests > > that were caught

Re: [PATCH 7/7] block/nbd: stop manipulating in_flight counter

2021-03-26 Thread Roman Kagan
On Tue, Mar 16, 2021 at 09:37:13PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 16.03.2021 19:08, Roman Kagan wrote: > > On Mon, Mar 15, 2021 at 11:15:44PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > 15.03.2021 09:06, Roman Kagan wrote: > > > > As the reconnect logic no longer interferes

Re: [PATCH 0/6] Add debug interface to kick/call on purpose

2021-03-26 Thread Jason Wang
在 2021/3/26 下午1:44, Dongli Zhang 写道: The virtio device/driver (e.g., vhost-scsi or vhost-net) may hang due to the loss of doorbell kick, e.g., https://lists.gnu.org/archive/html/qemu-devel/2018-12/msg01711.html ... or due to the loss of IRQ, e.g., as fixed by linux kernel commit fe200ae48ef5

Re: [PATCH v2 0/5] qemu-iotests: quality of life improvements

2021-03-26 Thread Paolo Bonzini
Il gio 25 mar 2021, 19:15 Max Reitz ha scritto: > I’ve applied an s/,attr/, attr/ to patch 1 (as suggested by Vladimir) > and an s/debug: bool= False/debug: bool = False/ to patch 2. I hope > that’s OK for you. > > With that I’ve applied the series to my block branch: > >

Re: [PATCH 6/7] block/nbd: decouple reconnect from drain

2021-03-26 Thread Roman Kagan
On Tue, Mar 16, 2021 at 09:09:12PM +0300, Vladimir Sementsov-Ogievskiy wrote: > 16.03.2021 19:03, Roman Kagan wrote: > > On Mon, Mar 15, 2021 at 11:10:14PM +0300, Vladimir Sementsov-Ogievskiy > > wrote: > > > 15.03.2021 09:06, Roman Kagan wrote: > > > > The reconnection logic doesn't need to stop