Re: [Qemu-block] [PATCH COLO v3 01/14] docs: block replication's description

2015-04-24 Thread Paolo Bonzini
On 24/04/2015 10:58, Dr. David Alan Gilbert wrote: If we can add a filter dynamically, we can add a filter that's file is nbd dynamically after secondary qemu's nbd server is ready. In this case, I think there is no need to touch nbd client. Yes, I think maybe the harder part is

Re: [Qemu-block] [Qemu-devel] [PATCH 1/3] scsi-disk: Limit zero write request to SCSI_WRITE_SAME_MAX

2015-04-24 Thread Paolo Bonzini
On 24/04/2015 11:02, Fam Zheng wrote: However, it shouldn't be a problem for the rest of the series. It is. The request has to be splitted to aligned part and unaligned part because the latter requires a buffer, as we don't like unbounded allocation. Yes, I found that out after reading

Re: [Qemu-block] [PATCH COLO v3 01/14] docs: block replication's description

2015-04-24 Thread Paolo Bonzini
On 24/04/2015 11:53, Wen Congyang wrote: Only before starting up a new secondary. Basically you do a migration with non-shared storage, and then start the secondary in colo mode. But it's only for the failover case. Quorum (or a new block/colo.c driver or filter) is fine for normal

[Qemu-block] [PATCH 06/11] docs: Document how to stream to an intermediate layer

2015-04-24 Thread Alberto Garcia
Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- docs/live-block-ops.txt | 31 --- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/live-block-ops.txt

[Qemu-block] [PATCH 09/11] qemu-iotests: test streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
This adds test_stream_intermediate(), similar to test_stream() but streams to the intermediate image instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 18 +- tests/qemu-iotests/030.out | 4 ++-- 2

[Qemu-block] [PATCH v5 00/11] Support streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
New version of the patchset. Apart from a couple of minor documentation fixes, this one has new test cases. Since we now allow block jobs in arbitrary nodes, I wanted to test that given a sufficiently long snapshot chain, it's possible to launch several block-stream operations in parallel as long

[Qemu-block] [PATCH 03/11] block: never cancel a streaming job without running stream_complete()

2015-04-24 Thread Alberto Garcia
We need to call stream_complete() in order to do all the necessary clean-ups, even if there's an early failure. At the moment it's only useful to make sure that s-backing_file_str is not leaked, but it will become more important as we introduce support for streaming to any intermediate node.

[Qemu-block] [PATCH 1/3] scsi-disk: Limit zero write request to SCSI_WRITE_SAME_MAX

2015-04-24 Thread Fam Zheng
SBC-4 says: If the number of logical blocks specified to be unmapped or written exceeds the value indicated in the MAXIMUM WRITE SAME LENGTH field in the Block Limits VPD page (see 6.6.4), then the device server shall terminate the command with CHECK CONDITION status with the

Re: [Qemu-block] [Qemu-devel] [PATCH 1/3] scsi-disk: Limit zero write request to SCSI_WRITE_SAME_MAX

2015-04-24 Thread Fam Zheng
On Fri, 04/24 10:50, Paolo Bonzini wrote: On 24/04/2015 10:33, Fam Zheng wrote: SBC-4 says: If the number of logical blocks specified to be unmapped or written exceeds the value indicated in the MAXIMUM WRITE SAME LENGTH field in the Block Limits VPD page (see 6.6.4),

Re: [Qemu-block] [PATCH v2 2/2] block: Fix NULL deference for unaligned write if qiov is NULL

2015-04-24 Thread Paolo Bonzini
On 24/04/2015 12:33, Fam Zheng wrote: For zero write, qiov passed by callers (qemu-io write -z and scsi-disk write same) is NULL. Commit fc3959e466 fixed bdrv_co_write_zeroes which is the common case for this bug, but it still exists in bdrv_aio_write_zeroes. A simpler fix would be in

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-04-24 Thread Alberto Garcia
On Fri 24 Apr 2015 11:52:14 AM CEST, Kevin Wolf kw...@redhat.com wrote: The posix_memalign() call wastes memory. I compared: posix_memalign(memptr, 65536, 2560 * 65536); memset(memptr, 0, 2560 * 65536); with: for (i = 0; i 2560; i++) { posix_memalign(memptr, 65536,

Re: [Qemu-block] [PATCH v2 2/2] block: Fix NULL deference for unaligned write if qiov is NULL

2015-04-24 Thread Paolo Bonzini
On 24/04/2015 13:00, Paolo Bonzini wrote: -qemu_iovec_add(local_qiov, head_buf, offset (align - 1)); -qemu_iovec_concat(local_qiov, qiov, 0, qiov-size); -use_local_qiov = true; +if (qiov) { +qemu_iovec_init(local_qiov, qiov ? qiov-niov + 2 : 1);

[Qemu-block] [PATCH v2 0/2] block: Fix unaligned bdrv_aio_write_zeroes

2015-04-24 Thread Fam Zheng
An unaligned zero write causes NULL deferencing in bdrv_co_do_pwritev. That path is reachable from bdrv_co_write_zeroes and bdrv_aio_write_zeroes. You can easily trigger through the former with qemu-io, as the test case added by 61815d6e0aa. For bdrv_aio_write_zeroes, in common cases there's

[Qemu-block] [PATCH v2 1/2] Revert block: Fix unaligned zero write

2015-04-24 Thread Fam Zheng
This reverts commit fc3959e4669a1c2149b91ccb05101cfc7ae1fc05. The core write code already handles the case, so remove this duplication. Signed-off-by: Fam Zheng f...@redhat.com --- block.c | 45 ++--- 1 file changed, 6 insertions(+), 39 deletions(-) diff

[Qemu-block] [PATCH 10/11] qemu-iotests: test block-stream operations in parallel

2015-04-24 Thread Alberto Garcia
This test case checks that it's possible to launch several stream operations in parallel in the same snapshot chain, each one involving a different set of nodes. Signed-off-by: Alberto Garcia be...@igalia.com --- tests/qemu-iotests/030 | 80 ++

[Qemu-block] [PATCH 2/3] block: Fix NULL deference for unaligned write if qiov is NULL

2015-04-24 Thread Fam Zheng
For zero write, qiov passed by callers (qemu-io write -z and scsi-disk write same) is NULL. Commit fc3959e466 fixed bdrv_co_write_zeroes which is the common case for this bug, but it still exists in bdrv_aio_write_zeroes. A simpler fix would be in bdrv_co_do_pwritev which is the NULL dereference

[Qemu-block] [PATCH 0/3] block: Fix unaligned bdrv_aio_write_zeroes

2015-04-24 Thread Fam Zheng
An unaligned zero write causes NULL deferencing in bdrv_co_do_pwritev. That path is reachable from bdrv_co_write_zeroes and bdrv_aio_write_zeroes. You can easily trigger through the former with qemu-io, as the test case added by 61815d6e0aa. For bdrv_aio_write_zeroes, in common cases there's

Re: [Qemu-block] [PATCH COLO v3 01/14] docs: block replication's description

2015-04-24 Thread Dr. David Alan Gilbert
* Wen Congyang (we...@cn.fujitsu.com) wrote: On 04/24/2015 03:47 PM, Paolo Bonzini wrote: On 24/04/2015 04:16, Wen Congyang wrote: I think the primary shouldn't do any I/O after failover (and the secondary should close the NBD server) so it is probably okay to ignore the removal for

[Qemu-block] [PATCH 05/11] block: Add QMP support for streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
This patch makes the 'device' parameter of the 'block-stream' command accept a node name as well as a device name. In addition to that, operation blockers will be checked in all intermediate nodes between the top and the base node. Since qmp_block_stream() now uses the error from

[Qemu-block] [PATCH 08/11] qemu-iotests: add no-op streaming test

2015-04-24 Thread Alberto Garcia
This patch updates test_stream_partial() to test that the block-stream operation never copies data from the image specified as base. Signed-off-by: Alberto Garcia be...@igalia.com --- tests/qemu-iotests/030 | 13 + 1 file changed, 13 insertions(+) diff --git a/tests/qemu-iotests/030

[Qemu-block] [PATCH 07/11] qemu-iotests: fix test_stream_partial()

2015-04-24 Thread Alberto Garcia
This test is streaming to the top layer using the intermediate image as the base. This is a mistake since block-stream never copies data from the base image and its backing chain, so this is effectively a no-op. In addition to fixing the base parameter, this patch also writes some data to the

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-04-24 Thread Kevin Wolf
Am 24.04.2015 um 14:50 hat Alberto Garcia geschrieben: On Fri 24 Apr 2015 02:37:21 PM CEST, Stefan Hajnoczi wrote: I think it would be nice to have a way to free unused cache entries after a while. Do you think mmap plus a periodic timer would work? I'm hesitant about changes like

Re: [Qemu-block] [PATCH v6 00/21] block: transactionless incremental backup series

2015-04-24 Thread Stefan Hajnoczi
On Fri, Apr 17, 2015 at 07:49:48PM -0400, John Snow wrote: It's spring! The winter snow has thawed and so here is a new patch series to enter your life and warm your heart. This patchset enables the in-memory part of the incremental backup feature, without transactional support. Support

Re: [Qemu-block] [PATCH 08/11] qemu-iotests: add no-op streaming test

2015-04-24 Thread Max Reitz
On 24.04.2015 15:55, Alberto Garcia wrote: On Fri 24 Apr 2015 03:19:54 PM CEST, Max Reitz wrote: This patch updates test_stream_partial() to test that the block-stream operation never copies data from the image specified as base. I think it would be better to add this as an own test case. But

Re: [Qemu-block] [PATCH 08/11] qemu-iotests: add no-op streaming test

2015-04-24 Thread Max Reitz
On 24.04.2015 17:01, Alberto Garcia wrote: This patch tests that in a partial block-stream operation, no data is ever copied from the base image. Signed-off-by: Alberto Garcia be...@igalia.com --- tests/qemu-iotests/030 | 18 ++ tests/qemu-iotests/030.out | 4 ++-- 2

[Qemu-block] [PATCH 06/11] docs: Document how to stream to an intermediate layer

2015-04-24 Thread Alberto Garcia
Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- docs/live-block-ops.txt | 31 --- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/live-block-ops.txt

[Qemu-block] [PATCH 11/11] qemu-iotests: test overlapping block-stream operations

2015-04-24 Thread Alberto Garcia
This test case checks that it's not possible to perform two block-stream operations if there are nodes involved in both. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 27 +++

[Qemu-block] [PATCH 05/11] block: Add QMP support for streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
This patch makes the 'device' parameter of the 'block-stream' command accept a node name as well as a device name. In addition to that, operation blockers will be checked in all intermediate nodes between the top and the base node. Since qmp_block_stream() now uses the error from

[Qemu-block] [PATCH 09/11] qemu-iotests: test streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
This adds test_stream_intermediate(), similar to test_stream() but streams to the intermediate image instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 18 +- tests/qemu-iotests/030.out | 4 ++-- 2

[Qemu-block] [PATCH 01/11] block: keep a list of block jobs

2015-04-24 Thread Alberto Garcia
The current way to obtain the list of existing block jobs is to iterate over all root nodes and check which ones own a job. Since we want to be able to support block jobs in other nodes as well, this patch keeps a list of jobs that is updated every time one is created or destroyed. This also

[Qemu-block] [PATCH 04/11] block: Support streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
This makes sure that the image we are steaming into is open in read-write mode during the operation. Operation blockers are also set in all intermediate nodes, since they will be removed from the chain afterwards. Finally, this also unblocks the stream operation in backing files. Signed-off-by:

[Qemu-block] [PATCH 07/11] qemu-iotests: fix test_stream_partial()

2015-04-24 Thread Alberto Garcia
This test is streaming to the top layer using the intermediate image as the base. This is a mistake since block-stream never copies data from the base image and its backing chain, so this is effectively a no-op. In addition to fixing the base parameter, this patch also writes some data to the

Re: [Qemu-block] [PATCH v8 00/10] qcow2: Allow refcount_bits amendment

2015-04-24 Thread Max Reitz
On 18.03.2015 21:56, Max Reitz wrote: This series adds support to qemu for changing the refcount_bits option of an existing qcow2 file through the qemu-img amend command. Originally (up until v7), this series was called qcow2: Support refcount orders != 4, but by now that support has already

[Qemu-block] [PATCH v6 00/11] Support streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
v6: - fix the no-op test following Max's suggestions v5: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03006.html - Fix a few typos - Minor documentation updates - Update test_stream_partial() to test no-ops - New test case: test_stream_parallel() - New test case: