Re: [Qemu-block] [RFC PATCH 4/5] block: Drop AioContext lock in bdrv_drain_poll_top_level()

2018-08-24 Thread Fam Zheng
On Fri, 08/17 19:02, Kevin Wolf wrote: > Simimlar to AIO_WAIT_WHILE(), bdrv_drain_poll_top_level() needs to > release the AioContext lock of the node to be drained before calling > aio_poll(). Otherwise, callbacks called by aio_poll() would possibly > take the lock a second time and run into a

Re: [Qemu-block] [RFC PATCH 3/5] job: Drop AioContext lock around aio_poll()

2018-08-24 Thread Fam Zheng
On Fri, 08/17 19:02, Kevin Wolf wrote: > Simimlar to AIO_WAIT_WHILE(), job_finish_sync() needs to release the > AioContext lock of the job before calling aio_poll(). Otherwise, > callbacks called by aio_poll() would possibly take the lock a second > time and run into a deadlock with a nested

Re: [Qemu-block] [RFC PATCH 2/5] tests: Acquire AioContext around job_finish_sync()

2018-08-24 Thread Fam Zheng
On Fri, 08/17 19:02, Kevin Wolf wrote: > All callers in QEMU proper hold the AioContext lock when calling > job_finish_sync(). The tests should do the same. > > Signed-off-by: Kevin Wolf A similar patch is needed for job_finalize() too, I think. Fam > --- > include/qemu/job.h | 6 ++

[Qemu-block] [PATCH v2] job: Fix nested aio_poll() hanging in job_txn_apply

2018-08-23 Thread Fam Zheng
ric Blake Signed-off-by: Fam Zheng --- v2: Add Eric's r-b and some more details in commit message. --- job.c | 18 +- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/job.c b/job.c index e36ebaafd8..a3bec7fb22 100644 --- a/job.c +++ b/job.c @@ -136,21 +136,13 @@ st

Re: [Qemu-block] [Qemu-devel] [PATCH] job: Fix nested aio_poll() hanging in job_txn_apply

2018-08-21 Thread Fam Zheng
On Tue, 08/21 11:15, Eric Blake wrote: > On 08/21/2018 01:45 AM, Fam Zheng wrote: > > All callers have acquired ctx already. Doing that again results in > > aio_poll() hang. This fixes the problem that a BDRV_POLL_WHILE() in the > > callback cannot make progress because ctx

[Qemu-block] [PATCH] job: Fix nested aio_poll() hanging in job_txn_apply

2018-08-21 Thread Fam Zheng
Signed-off-by: Fam Zheng --- job.c | 18 +- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/job.c b/job.c index fa671b431a..b07c5d0ffc 100644 --- a/job.c +++ b/job.c @@ -136,21 +136,13 @@ static void job_txn_del_job(Job *job) } } -static int job_txn_apply(JobTxn

Re: [Qemu-block] [RFC PATCH 0/5] Fix some jobs/drain/aio_poll related hangs

2018-08-21 Thread Fam Zheng
On Fri, 08/17 19:02, Kevin Wolf wrote: > I'm running out of time and will be offline for the next two weeks, so > I'm just sending out what I have right now. This is probably not ready > to be merged yet, but if need be, someone else can pick it up. Otherwise > I'll do that myself when I return.

Re: [Qemu-block] [Qemu-devel] [PATCH v3 0/3] file-posix: Simplifications on image locking

2018-08-20 Thread Fam Zheng
On Fri, 08/17 17:35, no-re...@patchew.org wrote: > ERROR:/tmp/qemu-test/src/tests/test-image-locking.c:69:check_locked_bytes: > assertion failed (perm_expected == !!qemu_lock_fd_test(fd, 100 + i, 1, > true)): (1 == 0) This image doesn't have OFD locking support in its glibc. Will add a check in

[Qemu-block] [PATCH v4 0/3] file-posix: Simplifications on image locking

2018-08-20 Thread Fam Zheng
the regression caused by v2). Fam Zheng (3): file-posix: Skip effectiveless OFD lock operations file-posix: Drop s->lock_fd tests: Add unit tests for image locking block/file-posix.c | 83 tests/Makefile.include | 2 + tests/test-image-locking.c |

[Qemu-block] [PATCH v4 1/3] file-posix: Skip effectiveless OFD lock operations

2018-08-20 Thread Fam Zheng
changes. This patch is an easy fix to this and the change is regardlessly reasonable, so do it. Signed-off-by: Fam Zheng --- v3: Don't misuse s->perm and s->shared_perm. v2: For s == NULL, unlock all bits. [Kevin] --- block/file-posix.c | 54 +-

[Qemu-block] [PATCH v4 3/3] tests: Add unit tests for image locking

2018-08-20 Thread Fam Zheng
Signed-off-by: Fam Zheng --- tests/Makefile.include | 2 + tests/test-image-locking.c | 154 + 2 files changed, 156 insertions(+) create mode 100644 tests/test-image-locking.c diff --git a/tests/Makefile.include b/tests/Makefile.include index

[Qemu-block] [PATCH v4 2/3] file-posix: Drop s->lock_fd

2018-08-20 Thread Fam Zheng
The lock_fd field is not strictly necessary because transferring locked bytes from old fd to the new one shouldn't fail anyway. This spares the user one fd per image. Signed-off-by: Fam Zheng --- block/file-posix.c | 37 + 1 file changed, 13 insertions(+), 24

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] qemu-img: Add dd seek= option

2018-08-19 Thread Fam Zheng
On Thu, 08/16 04:20, Max Reitz wrote: > No, the real issue is that dd is still not implemented just as a > frontend to convert. Which it should be. I'm not sure dd was a very > good idea from the start, and now it should ideally be a frontend to > convert. > > (My full opinion on the matter: dd

[Qemu-block] [PATCH v3 3/3] tests: Add unit tests for image locking

2018-08-16 Thread Fam Zheng
Signed-off-by: Fam Zheng --- tests/Makefile.include | 2 + tests/test-image-locking.c | 152 + 2 files changed, 154 insertions(+) create mode 100644 tests/test-image-locking.c diff --git a/tests/Makefile.include b/tests/Makefile.include index

[Qemu-block] [PATCH v3 0/3] file-posix: Simplifications on image locking

2018-08-16 Thread Fam Zheng
The first patch reduces chances of QEMU crash in unusual (but not unlikely) cases especially when used by Libvirt (see commit message). The second patch halves fd for images. The third adds some more test for patch one (would have caught the regression caused by v2). Fam Zheng (3): file-posix

[Qemu-block] [PATCH v3 1/3] file-posix: Skip effectiveless OFD lock operations

2018-08-16 Thread Fam Zheng
changes. This patch is an easy fix to this and the change is regardlessly reasonable, so do it. Signed-off-by: Fam Zheng --- v3: Don't misuse s->perm and s->shared_perm. v2: For s == NULL, unlock all bits. [Kevin] --- block/file-posix.c | 54 +-

[Qemu-block] [PATCH v3 2/3] file-posix: Drop s->lock_fd

2018-08-16 Thread Fam Zheng
The lock_fd field is not strictly necessary because transferring locked bytes from old fd to the new one shouldn't fail anyway. This spares the user one fd per image. Signed-off-by: Fam Zheng --- block/file-posix.c | 37 + 1 file changed, 13 insertions(+), 24

Re: [Qemu-block] [PATCH v2] file-posix: Skip effectiveless OFD lock operations

2018-08-14 Thread Fam Zheng
On Tue, 08/14 10:22, Kevin Wolf wrote: > Am 14.08.2018 um 10:12 hat Fam Zheng geschrieben: > > On Mon, 08/13 15:42, Kevin Wolf wrote: > > > Am 13.08.2018 um 04:39 hat Fam Zheng geschrieben: > > > > If we know we've already locked the bytes, don't do it again; similar

Re: [Qemu-block] [PATCH v2] file-posix: Skip effectiveless OFD lock operations

2018-08-14 Thread Fam Zheng
On Mon, 08/13 15:42, Kevin Wolf wrote: > Am 13.08.2018 um 04:39 hat Fam Zheng geschrieben: > > If we know we've already locked the bytes, don't do it again; similarly > > don't unlock a byte if we haven't locked it. This doesn't change the > > behavior, but fixes a corner

[Qemu-block] [PATCH] luks: Allow share-rw=on

2018-08-14 Thread Fam Zheng
on virtual devices. Suggested-by: Daniel P. Berrangé Signed-off-by: Fam Zheng --- block/crypto.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/block/crypto.c b/block/crypto.c index 146d81c90a..33ee01bebd 100644 --- a/block/crypto.c +++ b/block/crypto.c @@ -627,7 +627,9

Re: [Qemu-block] [PATCH v2] nvme: correct locking around completion

2018-08-14 Thread Fam Zheng
On Tue, 08/14 08:27, Paolo Bonzini wrote: > nvme_poll_queues is already protected by q->lock, and > AIO callbacks are invoked outside the AioContext lock. > So remove the acquire/release pair in nvme_handle_event. > > Signed-off-by: Paolo Bonzini > --- > block/nvme.c | 2 -- > 1 file changed, 2

Re: [Qemu-block] [PATCH v3 0/2] Fix aio_notify_accept()

2018-08-13 Thread Fam Zheng
On Thu, 08/09 21:22, Fam Zheng wrote: > v3: Fix commit message's bug description. [Paolo] If there's no objection, I'm queuing this for 3.1. Fam

Re: [Qemu-block] [PATCH] nvme: simplify plug/unplug

2018-08-13 Thread Fam Zheng
On Mon, 08/13 16:43, Paolo Bonzini wrote: > bdrv_io_plug/bdrv_io_unplug take care of keeping a nesting count, > so change s->plugged to just a bool. > > Signed-off-by: Paolo Bonzini Queued, thanks. Fam

Re: [Qemu-block] [PATCH] nvme: simplify code around completion

2018-08-13 Thread Fam Zheng
On Mon, 08/13 16:43, Paolo Bonzini wrote: > nvme_poll_queues is already protected by q->lock, and > AIO callbacks are invoked outside the AioContext lock. > So remove the acquire/release pair in nvme_handle_event. > > Remove unnecessary variable in in nvme_poll_cb. > > Signed-off-by: Paolo

[Qemu-block] [PATCH v2] file-posix: Skip effectiveless OFD lock operations

2018-08-12 Thread Fam Zheng
changes. This patch is an easy fix to this and the change is regardlessly reasonable, so do it. Signed-off-by: Fam Zheng --- v2: For s == NULL, unlock all bits. [Kevin] --- block/file-posix.c | 41 +++-- 1 file changed, 31 insertions(+), 10 deletions(-) diff

Re: [Qemu-block] [PATCH] file-posix: Skip effectiveless OFD lock operations

2018-08-12 Thread Fam Zheng
On Fri, 08/10 14:14, Kevin Wolf wrote: > Am 18.07.2018 um 10:43 hat Fam Zheng geschrieben: > > If we know we've already locked the bytes, don't do it again; similarly > > don't unlock a byte if we haven't locked it. This doesn't change the > > behavior, but fixes a corner

[Qemu-block] [PATCH v3 2/2] aio: Do aio_notify_accept only during blocking aio_poll

2018-08-09 Thread Fam Zheng
l()'s can steal each other's ctx->notifier event and cause hanging just like described above. Cc: qemu-sta...@nongnu.org Suggested-by: Paolo Bonzini Signed-off-by: Fam Zheng --- util/aio-posix.c | 4 ++-- util/aio-win32.c | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/

[Qemu-block] [PATCH v3 0/2] Fix aio_notify_accept()

2018-08-09 Thread Fam Zheng
o_poll(iothread->ctx, true) in the IOThread. This can cause IOThread hanging. Fam Zheng (2): aio-posix: Don't count ctx->notifier as progress when polling aio: Do aio_notify_accept only during blocking aio_poll util/aio-posix.c | 7 --- util/aio-win32.c | 3 ++- 2 files changed, 6 inse

[Qemu-block] [PATCH v3 1/2] aio-posix: Don't count ctx->notifier as progress when polling

2018-08-09 Thread Fam Zheng
The same logic exists in fd polling. This change is especially important to avoid busy loop once we limit aio_notify_accept() to blocking aio_poll(). Cc: qemu-sta...@nongnu.org Signed-off-by: Fam Zheng --- util/aio-posix.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git

[Qemu-block] [PATCH v2 2/2] aio: Do aio_notify_accept only during blocking aio_poll

2018-08-07 Thread Fam Zheng
e_thread), because otherwise two blocking aio_poll()'s can steal each other's ctx->notifier event and cause hanging just like described above. Suggested-by: Paolo Bonzini Signed-off-by: Fam Zheng --- util/aio-posix.c | 4 ++-- util/aio-win32.c | 3 ++- 2 files changed, 4 insertions(+), 3

[Qemu-block] [PATCH v2 0/2] Fix aio_notify_accept()

2018-08-07 Thread Fam Zheng
can cause IOThread hanging. Fam Zheng (2): aio-posix: Don't count ctx->notifier as progress when polling aio: Do aio_notify_accept only during blocking aio_poll util/aio-posix.c | 7 --- util/aio-win32.c | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) -- 2.17.1

Re: [Qemu-block] [RFC PATCH] async: Fix aio_notify_accept

2018-08-06 Thread Fam Zheng
On Fri, 08/03 19:08, Paolo Bonzini wrote: > On 03/08/2018 17:49, Fam Zheng wrote: > > void aio_notify_accept(AioContext *ctx) > > { > > -if (atomic_xchg(>notified, false)) { > > +/* If ctx->notify_me >= 2, another aio_poll() is waiting which may &g

[Qemu-block] [RFC PATCH] async: Fix aio_notify_accept

2018-08-03 Thread Fam Zheng
>From main loop, bdrv_set_aio_context() can call IOThread's aio_poll(). That breaks aio_notify() because the ctx->notifier event can get cleared too early by this which causes IOThread hanging. See https://bugzilla.redhat.com/show_bug.cgi?id=1562750 for details. Signed-off-by: Fam

Re: [Qemu-block] [Qemu-devel] [PATCH] qemu-img.c: Add examples section

2018-08-03 Thread Fam Zheng
On Thu, 08/02 22:34, Programmingkid wrote: > > > On Aug 2, 2018, at 10:10 PM, Fam Zheng wrote: > > > > On Thu, 08/02 20:50, John Arbuckle wrote: > >> Add an examples section to the help output. > >> > >> Signed-off-by: John Arbuckle >

[Qemu-block] [PATCH] aio-posix: Improve comment around marking node deleted

2018-08-03 Thread Fam Zheng
The counter is for qemu_lockcnt_inc/dec sections (read side), qemu_lockcnt_lock/unlock is for the write side. Suggested-by: Paolo Bonzini Signed-off-by: Fam Zheng --- util/aio-posix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/util/aio-posix.c b/util/aio-posix.c index

Re: [Qemu-block] [Qemu-devel] [PATCH] qemu-img.c: Add examples section

2018-08-02 Thread Fam Zheng
On Thu, 08/02 20:50, John Arbuckle wrote: > Add an examples section to the help output. > > Signed-off-by: John Arbuckle > --- > qemu-img.c | 11 +++ > 1 file changed, 11 insertions(+) > > diff --git a/qemu-img.c b/qemu-img.c > index 1acddf693c..f77c82695d 100644 > --- a/qemu-img.c >

Re: [Qemu-block] [PATCH RFC for-3.0-rc3 0/3] qemu-img: Disable copy offloading by default

2018-07-27 Thread Fam Zheng
On Fri, Jul 27, 2018 at 6:29 PM Kevin Wolf wrote: > > Am 27.07.2018 um 05:33 hat Fam Zheng geschrieben: > > Kevin pointed out that both glibc and kernel provides a slow fallback of > > copy_file_range which hurts thin provisioning. This is particularly true for > > thin

Re: [Qemu-block] [Qemu-devel] [PATCH] file-posix: Handle EINTR in preallocation=full write

2018-07-27 Thread Fam Zheng
On Fri, 07/27 14:53, Fam Zheng wrote: > Cc: qemu-sta...@nongnu.org > Signed-off-by: Fam Zheng Actually cc qemu-sta...@nongnu.org

[Qemu-block] [PATCH] file-posix: Handle EINTR in preallocation=full write

2018-07-27 Thread Fam Zheng
Cc: qemu-sta...@nongnu.org Signed-off-by: Fam Zheng --- block/file-posix.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/block/file-posix.c b/block/file-posix.c index ad299beb38..928b863ced 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -1646,6 +1646,9 @@ static int

[Qemu-block] [PATCH RFC for-3.0-rc3 2/3] qemu-img: Add -C option for convert with copy offloading

2018-07-26 Thread Fam Zheng
Signed-off-by: Fam Zheng --- qemu-img-cmds.hx | 2 +- qemu-img.c | 21 + qemu-img.texi| 8 +++- 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index 69758fb6e8..1526f327a5 100644 --- a/qemu-img-cmds.hx +++ b

[Qemu-block] [PATCH RFC for-3.0-rc3 1/3] Revert "qemu-img: Document copy offloading implications with -S and -c"

2018-07-26 Thread Fam Zheng
This reverts commit eb461485f4558e362fab905735b50987505bca44. Now that we introduce an explicit option, these implicit rules are not used. Signed-off-by: Fam Zheng --- qemu-img.texi | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/qemu-img.texi b/qemu-img.texi index

[Qemu-block] [PATCH 2/2] iotests: Don't lock /dev/null in 226

2018-07-24 Thread Fam Zheng
On my system (Fedora 28), this script reports a 'failed to get "consistent read" lock' error. Following docs/devel/testing.rst, it's better to add locking=off here. Signed-off-by: Fam Zheng --- tests/qemu-iotests/226 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --g

[Qemu-block] [PATCH 0/2] iotests: Fix 226 on _my_ system

2018-07-24 Thread Fam Zheng
locking, and add a doc text about using test images. Fam Zheng (2): docs: Describe using images in writing iotests iotests: Don't lock /dev/null in 226 docs/devel/testing.rst | 11 +++ tests/qemu-iotests/226 | 4 ++-- 2 files changed, 13 insertions(+), 2 deletions(-) -- 2.17.1

[Qemu-block] [PATCH 1/2] docs: Describe using images in writing iotests

2018-07-24 Thread Fam Zheng
Signed-off-by: Fam Zheng --- docs/devel/testing.rst | 11 +++ 1 file changed, 11 insertions(+) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 5e19cd50da..8e1fa3a66e 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -255,6 +255,17 @@ comparable

Re: [Qemu-block] [PATCH] block: Don't lock /dev/null and /dev/zero automatically

2018-07-23 Thread Fam Zheng
On Mon, Jul 23, 2018 at 10:37 PM Max Reitz wrote: > > On 2018-07-23 03:56, Fam Zheng wrote: > > On Sun, Jul 22, 2018 at 10:06 PM Max Reitz wrote: > >> > >> On 2018-07-22 04:37, Fam Zheng wrote: > >>> On Sun, Jul 22, 2018 at 5:08 AM Max Reitz wrote: &g

Re: [Qemu-block] [PATCH] block: Don't lock /dev/null and /dev/zero automatically

2018-07-22 Thread Fam Zheng
On Sun, Jul 22, 2018 at 10:06 PM Max Reitz wrote: > > On 2018-07-22 04:37, Fam Zheng wrote: > > On Sun, Jul 22, 2018 at 5:08 AM Max Reitz wrote: > >> > >> On 2018-07-19 05:41, Fam Zheng wrote: > >>> On my Fedora 28, /dev/null is locked by some ot

Re: [Qemu-block] [PATCH] block: Don't lock /dev/null and /dev/zero automatically

2018-07-21 Thread Fam Zheng
On Sun, Jul 22, 2018 at 5:08 AM Max Reitz wrote: > > On 2018-07-19 05:41, Fam Zheng wrote: > > On my Fedora 28, /dev/null is locked by some other process (couldn't > > inspect it due to the current lslocks limitation), so iotests 226 fails > > with some unexpected imag

Re: [Qemu-block] [Qemu-devel] [PATCH] block: Don't lock /dev/null and /dev/zero automatically

2018-07-21 Thread Fam Zheng
On Sat, Jul 21, 2018 at 12:35 AM Eric Blake wrote: > > On 07/20/2018 03:24 AM, Fam Zheng wrote: > I'm not familiar with /dev/nullb* - is that a typo? > > $ ll /dev/nullb* > ls: cannot access '/dev/nullb*': No such file or directory You probably have figured out already but

Re: [Qemu-block] [PATCH] block: Don't lock /dev/null and /dev/zero automatically

2018-07-20 Thread Fam Zheng
On Thu, 07/19 13:57, John Snow wrote: > Should we instead modify the test in this case to not attempt to take a > lock on a device we know cannot meaningfully store state, or is it your > preference to attempt to maintain such a list in the raw driver itself? > > I guess we never want QEMU to try

Re: [Qemu-block] [Qemu-devel] [PATCH] hw/block/nvme: add optional parameter num_namespaces for nvme device

2018-07-19 Thread Fam Zheng
On Thu, 07/19 12:33, Weiping Zhang wrote: > Add an optional paramter num_namespaces for device, and set it > to 1 by default. > > Signed-off-by: Weiping Zhang > --- > hw/block/nvme.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/hw/block/nvme.c b/hw/block/nvme.c

[Qemu-block] [PATCH] block: Don't lock /dev/null and /dev/zero automatically

2018-07-18 Thread Fam Zheng
-by: Fam Zheng --- block/file-posix.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/block/file-posix.c b/block/file-posix.c index 60af4b3d51..8bf034108a 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -503,7 +503,12 @@ static int raw_open_common

[Qemu-block] [PATCH] file-posix: Skip effectiveless OFD lock operations

2018-07-18 Thread Fam Zheng
changes. This patch is an easy fix to this and the change is regardlessly reasonable, so do it. Signed-off-by: Fam Zheng --- block/file-posix.c | 27 +-- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index 60af4b3d51

[Qemu-block] [PATCH] iotests: 153: Fix dead code

2018-07-11 Thread Fam Zheng
This step was left behind my mistake. As suggested by the echoed text, the intention was to test two devices with the same image, with different options. The behavior should be the same as two QEMU processes. Complete it. Signed-off-by: Fam Zheng --- tests/qemu-iotests/153 | 2 ++ tests

[Qemu-block] [PATCH v2] iotests: nbd: Stop qemu-nbd before remaking image

2018-07-11 Thread Fam Zheng
the image? Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1024 +Failed to find an available port: Address already in use read 1024/1024 bytes at offset 0 Patch _make_test_img to stop the old qemu-nbd before starting a new one, which fixes this problem, and similarly 215. Signed-off-by:

[Qemu-block] [PATCH] qemu-img: Document copy offloading implications with -S and -c

2018-07-10 Thread Fam Zheng
Explicitly enabling zero detection or compression suppresses copy offloading during convert. Document it. Suggested-by: Kevin Wolf Signed-off-by: Fam Zheng --- qemu-img.texi | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/qemu-img.texi b/qemu-img.texi index aeb1b9e66c

[Qemu-block] [PATCH] iotests: 222: Don't run with luks

2018-07-10 Thread Fam Zheng
Luks needs special parameters to operate the image. Since this test is focusing on image fleecing, skip skip that format. Signed-off-by: Fam Zheng --- tests/qemu-iotests/222 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222 index ff3bfc1470

[Qemu-block] [PATCH v3 07/10] block: Use common req handling for discard

2018-07-10 Thread Fam Zheng
Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation here is that discard requests don't affect bs->wr_highest_offset, and it cannot extend the image. Signed-off-by: Fam Zheng --- block/io.c | 32 ++-- 1 file changed, 22 insertions(+), 10 deleti

[Qemu-block] [PATCH v3 09/10] block: Fix bdrv_co_truncate overlap check

2018-07-10 Thread Fam Zheng
If we are growing the image and potentially using preallocation for the new area, we need to make sure that no write requests are made to the "preallocated" area which is [@old_size, @offset), not [@offset, offset * 2 - @old_size). Signed-off-by: Fam Zheng Reviewed-by: Eric Blake

[Qemu-block] [PATCH v3 10/10] block: Use common write req handling in truncate

2018-07-10 Thread Fam Zheng
Error parameter. Signed-off-by: Fam Zheng --- block/io.c | 57 ++ 1 file changed, 36 insertions(+), 21 deletions(-) diff --git a/block/io.c b/block/io.c index e3e2d5286d..abcb8b3de6 100644 --- a/block/io.c +++ b/block/io.c @@ -1582,14 +1

[Qemu-block] [PATCH v3 08/10] block: Use common req handling in copy offloading

2018-07-10 Thread Fam Zheng
This brings the request handling logic inline with write and discard, fixing write_gen, resize_cb, dirty bitmaps and image size refreshing. The last of these issues broke iotest case 222, which is now fixed. Signed-off-by: Fam Zheng --- block/io.c | 22 +- 1 file changed, 9

[Qemu-block] [PATCH v3 05/10] block: Extract common write req handling

2018-07-10 Thread Fam Zheng
and truncate. Suggested-by: Kevin Wolf Signed-off-by: Fam Zheng --- block/io.c | 91 ++ 1 file changed, 57 insertions(+), 34 deletions(-) diff --git a/block/io.c b/block/io.c index c56e329bb5..960e1492d0 100644 --- a/block/io.c +++ b/block/io.c

[Qemu-block] [PATCH v3 03/10] block: Use BdrvChild to discard

2018-07-10 Thread Fam Zheng
paths. Signed-off-by: Fam Zheng --- block/blkdebug.c | 2 +- block/blklogwrites.c | 2 +- block/blkreplay.c | 2 +- block/block-backend.c | 2 +- block/copy-on-read.c | 2 +- block/io.c | 18 +- block/mirror.c | 2 +- block/qcow2-refcount.c

[Qemu-block] [PATCH v3 02/10] block: Add copy offloading trace points

2018-07-10 Thread Fam Zheng
A few trace points that can help reveal what is happening in a copy offloading I/O path. Signed-off-by: Fam Zheng --- block/file-posix.c | 2 ++ block/io.c | 4 block/iscsi.c | 3 +++ block/trace-events | 6 ++ 4 files changed, 15 insertions(+) diff --git a/block/file

[Qemu-block] [PATCH v3 06/10] block: Fix handling of image enlarging write

2018-07-10 Thread Fam Zheng
-by: Fam Zheng --- block/io.c | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/block/io.c b/block/io.c index 960e1492d0..10a475302a 100644 --- a/block/io.c +++ b/block/io.c @@ -40,6 +40,7 @@ static AioWait drain_all_aio_wait; +static void bdrv_parent_cb_resize

[Qemu-block] [PATCH v3 04/10] block: Use uint64_t for BdrvTrackedRequest byte fields

2018-07-10 Thread Fam Zheng
This matches the types used for bytes in the rest parts of block layer. In the case of bdrv_co_truncate, new_bytes can be the image size which probably doesn't fit in a 32 bit int. Signed-off-by: Fam Zheng --- block/io.c| 8 +--- include/block/block_int.h | 4 ++-- 2 files

[Qemu-block] [PATCH v3 01/10] block: Prefix file driver trace points with "file_"

2018-07-10 Thread Fam Zheng
with a glob. Rename them. Suggested-by: Kevin Wolf Signed-off-by: Fam Zheng --- block/file-posix.c | 2 +- block/file-win32.c | 2 +- block/trace-events | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index 4fec8cb53c..1185c7c5cc 100644

[Qemu-block] [PATCH v3 00/10] block: Fix dst reading after tail copy offloading

2018-07-10 Thread Fam Zheng
caught by iotests 222. Follow the logic in the normal write code and update bs->total_sectors after I/O is done. While at it, add a few convenient trace points to aid future debug experiences in the topic. Fam Zheng (10): block: Prefix file driver trace points with "file_" bloc

Re: [Qemu-block] [PATCH v5 4/4] block/backup: fix fleecing scheme: use serialized writes

2018-07-09 Thread Fam Zheng
st char *job_id, > BlockDriverState *bs, > sync_bitmap : NULL; > job->compress = compress; > > +/* Detect image-fleecing (and similar) schemes */ > +job->serialize_target_writes = bdrv_chain_contains(target, bs); > + > /* If there is no backing file on the target, we cannot rely on COW if > our > * backup cluster size is smaller than the target cluster size. Even for > * targets with a backing file, try to avoid COW if possible. */ I always thought mirror is by far the most complicated job, but now backup is catching up quickly! Reviewed-by: Fam Zheng

Re: [Qemu-block] [PATCH v5 3/4] block: add BDRV_REQ_SERIALISING flag

2018-07-09 Thread Fam Zheng
BDRV_REQ_NO_SERIALISING)) { > wait_serialising_requests(); > } > @@ -2948,6 +2969,9 @@ static int coroutine_fn bdrv_co_copy_range_internal( > > /* BDRV_REQ_NO_SERIALISING is only for read operation */ > assert(!(write_flags & BDRV_REQ_NO_SERIALISING)); > +if (write_flags & BDRV_REQ_SERIALISING) { > +mark_request_serialising(, bdrv_get_cluster_size(dst->bs)); > +} > wait_serialising_requests(); > > ret = dst->bs->drv->bdrv_co_copy_range_to(dst->bs, > -- > 2.11.1 > Reviewed-by: Fam Zheng Fam

Re: [Qemu-block] [PATCH v5 2/4] block: split flags in copy_range

2018-07-09 Thread Fam Zheng
On Mon, 07/09 19:37, Vladimir Sementsov-Ogievskiy wrote: > Pass read flags and write flags separately. This is needed to handle > coming BDRV_REQ_NO_SERIALISING clearly in following patches. > > Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Fam Zheng

Re: [Qemu-block] [PATCH v5 1/4] block/io: fix copy_range

2018-07-09 Thread Fam Zheng
unsubstantiated and potentially dangerous, let's > drop it and add additional asserts and documentation. > > Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Fam Zheng

Re: [Qemu-block] [PATCH v4 1/4] block/io: fix copy_range

2018-07-09 Thread Fam Zheng
On Mon, 07/09 17:38, Vladimir Sementsov-Ogievskiy wrote: > 09.07.2018 16:17, Fam Zheng wrote: > > On Mon, 07/09 12:43, Vladimir Sementsov-Ogievskiy wrote: > > > 09.07.2018 04:15, Fam Zheng wrote: > > > > On Fri, 07/06 21:30, Vladimir Sementsov-Ogievskiy wrote: >

Re: [Qemu-block] [PATCH v4 1/4] block/io: fix copy_range

2018-07-09 Thread Fam Zheng
On Mon, 07/09 12:43, Vladimir Sementsov-Ogievskiy wrote: > 09.07.2018 04:15, Fam Zheng wrote: > > On Fri, 07/06 21:30, Vladimir Sementsov-Ogievskiy wrote: > > > Here two things are fixed: > > > > > > 1. Architecture > > > > > > On each

Re: [Qemu-block] [PATCH v2 9/9] block: Use common write req handling in truncate

2018-07-08 Thread Fam Zheng
On Fri, 07/06 17:12, Eric Blake wrote: > On 07/05/2018 02:37 AM, Fam Zheng wrote: > > Truncation is the last to convert from open coded req handling to > > reusing helpers. This time the permission check in prepare has to adapt > > to the new caller: it checks a different perm

Re: [Qemu-block] [PATCH v4 1/4] block/io: fix copy_range

2018-07-08 Thread Fam Zheng
On Fri, 07/06 21:30, Vladimir Sementsov-Ogievskiy wrote: > Here two things are fixed: > > 1. Architecture > > On each recursion step, we go to the child of src or dst, only for one > of them. So, it's wrong to create tracked requests for both on each > step. It leads to tracked requests

Re: [Qemu-block] [PATCH v3 0/4] fix image fleecing

2018-07-06 Thread Fam Zheng
On Thu, 07/05 10:46, Vladimir Sementsov-Ogievskiy wrote: > Hi all. > > This fixes image fleecing scheme for 3.0, details are in 04 patch. Looks like this breaks 'test-replication': http://patchew.org/QEMU/20180705074638.770905-1-vsement...@virtuozzo.com/ test-replication:

Re: [Qemu-block] [PATCH v2 6/9] block: Use common req handling for discard

2018-07-06 Thread Fam Zheng
On Thu, 07/05 14:44, Kevin Wolf wrote: > Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > > Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation > > here is that discard requests don't affect bs->wr_highest_offset. > > > > Signed-off-by: Fam Zheng

Re: [Qemu-block] [PATCH v2 1/9] block: Add copy offloading trace points

2018-07-06 Thread Fam Zheng
On Thu, 07/05 13:08, Kevin Wolf wrote: > Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > > A few trace points that can help reveal what is happening in a copy > > offloading I/O path. > > > > Signed-off-by: Fam Zheng > > --- > > block/file-pos

[Qemu-block] [PATCH v2 8/9] block: Fix bdrv_co_truncate overlap check

2018-07-05 Thread Fam Zheng
If we are growing the image and potentially using preallocation for the new area, we need to make sure that no write requests are made to the "preallocated" area which [@old_size, @offset), not [@offset, offset * 2 - @old_size). Signed-off-by: Fam Zheng --- block/io.c | 3 ++- 1 file

[Qemu-block] [PATCH v2 9/9] block: Use common write req handling in truncate

2018-07-05 Thread Fam Zheng
Error parameter. Signed-off-by: Fam Zheng --- block/io.c | 54 +- 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/block/io.c b/block/io.c index ed18eb0ca3..53f2bf4103 100644 --- a/block/io.c +++ b/block/io.c @@ -1556,13 +1

[Qemu-block] [PATCH v2 5/9] block: Fix handling of image enlarging write

2018-07-05 Thread Fam Zheng
-by: Fam Zheng --- block/io.c | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/block/io.c b/block/io.c index 03d9eb0a65..f06978dda0 100644 --- a/block/io.c +++ b/block/io.c @@ -40,6 +40,7 @@ static AioWait drain_all_aio_wait; +static void bdrv_parent_cb_resize

[Qemu-block] [PATCH v2 3/9] block: Use uint64_t for BdrvTrackedRequest byte fields

2018-07-05 Thread Fam Zheng
This matches the types used for bytes in the rest parts of block layer. In the case of bdrv_co_truncate, new_bytes can be the image size which probably doesn't fit in a 32 bit int. Signed-off-by: Fam Zheng --- block/io.c| 2 +- include/block/block_int.h | 4 ++-- 2 files changed

[Qemu-block] [PATCH v2 7/9] block: Use common req handling in copy offloading

2018-07-05 Thread Fam Zheng
This brings the request handling logic inline with write and discard, fixing write_gen, resize_cb, dirty bitmaps and image size refreshing. The last of these issues broke iotest case 222, which is now fixed. Signed-off-by: Fam Zheng --- block/io.c | 8 +++- 1 file changed, 7 insertions

[Qemu-block] [PATCH v2 1/9] block: Add copy offloading trace points

2018-07-05 Thread Fam Zheng
A few trace points that can help reveal what is happening in a copy offloading I/O path. Signed-off-by: Fam Zheng --- block/file-posix.c | 2 ++ block/io.c | 2 ++ block/iscsi.c | 3 +++ block/trace-events | 6 ++ 4 files changed, 13 insertions(+) diff --git a/block/file

[Qemu-block] [PATCH v2 6/9] block: Use common req handling for discard

2018-07-05 Thread Fam Zheng
Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation here is that discard requests don't affect bs->wr_highest_offset. Signed-off-by: Fam Zheng --- block/io.c | 21 ++--- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/block/io.c b/block/i

[Qemu-block] [PATCH v2 4/9] block: Extract common write req handling

2018-07-05 Thread Fam Zheng
and truncate. Suggested-by: Kevin Wolf Signed-off-by: Fam Zheng --- block/io.c | 70 ++ 1 file changed, 44 insertions(+), 26 deletions(-) diff --git a/block/io.c b/block/io.c index 443a8584c4..03d9eb0a65 100644 --- a/block/io.c +++ b/block/io.c

[Qemu-block] [PATCH v2 2/9] block: Use BdrvChild to discard

2018-07-05 Thread Fam Zheng
paths. Signed-off-by: Fam Zheng --- block/blkdebug.c | 2 +- block/blkreplay.c | 2 +- block/block-backend.c | 2 +- block/copy-on-read.c | 2 +- block/io.c | 18 +- block/mirror.c | 2 +- block/qcow2-refcount.c | 2 +- block/raw-format.c

[Qemu-block] [PATCH v2 0/9] block: Fix dst reading after tail copy offloading

2018-07-05 Thread Fam Zheng
ic in the normal write code and update bs->total_sectors after I/O is done. While at it, add a few convenient trace points to aid future debug experiences in the topic. Fam Zheng (9): block: Add copy offloading trace points block: Use BdrvChild to discard block: Use uint64_t for BdrvTracked

Re: [Qemu-block] [PATCH 1/2] block: Fix dst total_sectors after copy offloading

2018-07-04 Thread Fam Zheng
On Wed, 07/04 09:44, Kevin Wolf wrote: > Am 04.07.2018 um 08:13 hat Fam Zheng geschrieben: > > This was noticed by the new image fleecing tests case 222. The issue is > > apparent and we should just do the same right things as in > > bdrv_aligned_pwritev. > >

[Qemu-block] [PATCH 2/2] block: Add copy offloading trace points

2018-07-04 Thread Fam Zheng
A few trace points that can help reveal what is happening in a copy offloading I/O path. Signed-off-by: Fam Zheng --- block/file-posix.c | 2 ++ block/io.c | 2 ++ block/iscsi.c | 3 +++ block/trace-events | 6 ++ 4 files changed, 13 insertions(+) diff --git a/block/file

[Qemu-block] [PATCH 1/2] block: Fix dst total_sectors after copy offloading

2018-07-04 Thread Fam Zheng
This was noticed by the new image fleecing tests case 222. The issue is apparent and we should just do the same right things as in bdrv_aligned_pwritev. Reported-by: John Snow Signed-off-by: Fam Zheng --- block/io.c | 4 1 file changed, 4 insertions(+) diff --git a/block/io.c b/block

[Qemu-block] [PATCH 0/2] block: Fix dst reading after tail copy offloading

2018-07-04 Thread Fam Zheng
ic in the normal write code and update bs->total_sectors after I/O is done. While at it, add a few convenient trace points to aid future debug experiences in the topic. Fam Zheng (2): block: Fix dst total_sectors after copy offloading block: Add copy offloading trace points block/file-posix.c

[Qemu-block] [PATCH v4 2/3] block: Honour BDRV_REQ_NO_SERIALISING in copy range

2018-07-02 Thread Fam Zheng
This semantics is needed by drive-backup so implement it before using this API there. Reviewed-by: Stefan Hajnoczi Signed-off-by: Fam Zheng --- block/io.c| 6 -- include/block/block.h | 5 +++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/block/io.c b/block

[Qemu-block] [PATCH v4 1/3] block: Fix parameter checking in bdrv_co_copy_range_internal

2018-07-02 Thread Fam Zheng
off-by: Fam Zheng --- block/io.c | 29 +++-- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/block/io.c b/block/io.c index 7035b78a20..b8845708d7 100644 --- a/block/io.c +++ b/block/io.c @@ -2897,18 +2897,11 @@ static int coroutine

[Qemu-block] [PATCH v4 3/3] backup: Use copy offloading

2018-07-02 Thread Fam Zheng
request fails because of, for example, the offset is beyond EOF, but another may well be accepted by the protocol layer. This will be implemented separately. Reviewed-by: Stefan Hajnoczi Signed-off-by: Fam Zheng --- block/backup.c | 150 - block/trace

[Qemu-block] [PATCH v4 0/3] backup: Use copy offloading

2018-07-02 Thread Fam Zheng
to use the bounce buffer as well as speeding up the copy operation when the backend supports it. v3: Don't forget coroutine_fn. [Stefan] Don't reset job->use_copy_range redundantly. [Stefan] v2: Use helper functions. [Stefan] Fam Zheng (3): block: Fix parameter check

Re: [Qemu-block] [PATCH 0/4] block: Trivial fixes in offloading code

2018-07-02 Thread Fam Zheng
On Mon, 07/02 14:35, Max Reitz wrote: > On 2018-07-02 04:58, Fam Zheng wrote: > > These are the low priority ones spotted by Kevin and Max last week. > > > > Fam Zheng (4): > > qcow2: Drop unused cluster_data > > file-posix: Fix fd_open check in raw_c

Re: [Qemu-block] [PATCH 1/2] vmdk: Fix possible segfault with non-VMDK backing

2018-07-02 Thread Fam Zheng
->format_name, "vmdk")) { > +/* Backing file is not in vmdk format, so it does not have > + * a CID, which makes the overlay's parent CID invalid */ > +return 0; > +} > + Reviewed-by: Fam Zheng > if (vmdk_read_cid(p_bs, 0, _pcid) != 0) { > /* read failure: report as not valid */ > return 0; > -- > 2.17.1 >

Re: [Qemu-block] [PATCH v3 0/2] backup: Use copy offloading

2018-07-02 Thread Fam Zheng
Jeff: ping? Can we have this in 3.0? On Tue, 06/05 22:06, Fam Zheng wrote: > Based-on: <20180529055959.32002-1-f...@redhat.com> > ([PATCH v7 00/10] qemu-img convert with copy offloading) > > This enhances the backup job to make use of the copy offloading API. It > eliminates

Re: [Qemu-block] [Qemu-devel] [PATCH v2 3/3] qemu-iotests: Image fleecing test case 222

2018-07-02 Thread Fam Zheng
On Fri, 06/29 17:04, John Snow wrote: > Mine was indeed inspired by Fam's version, but I opted to rewrite it > instead of continue with the python unit tests approach. This is why I > lost the commit message, existing copyright text, etc. > > I can re-add him as author credit to my version, or at

<    1   2   3   4   5   6   7   8   9   10   >