[Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-04-21 Thread Alberto Garcia
: the current default L2 cache size can map a whole 8GB disk, so those smaller than that are allocating cache memory that can never be used. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 35 ++- block/qcow2.c | 4 ++-- block/qcow2.h

[Qemu-block] [PATCH 06/11] docs: Document how to stream to an intermediate layer

2015-04-24 Thread Alberto Garcia
Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- docs/live-block-ops.txt | 31 --- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/live-block-ops.txt b/docs/live

[Qemu-block] [PATCH 09/11] qemu-iotests: test streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
This adds test_stream_intermediate(), similar to test_stream() but streams to the intermediate image instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 18 +- tests/qemu-iotests/030.out | 4 ++-- 2

[Qemu-block] [PATCH v5 00/11] Support streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
. - The BlockJobInfo type and all BLOCK_JOB_* events report the node name in the 'device' field if the node does not have a device name. - All intermediate nodes are blocked (and checked for blockers) during the streaming operation. Alberto Garcia (11): block: keep a list of block jobs block: allow

[Qemu-block] [PATCH 03/11] block: never cancel a streaming job without running stream_complete()

2015-04-24 Thread Alberto Garcia
. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- block/stream.c | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/block/stream.c b/block/stream.c index a628901..37bfd8b 100644 --- a/block/stream.c +++ b/block/stream.c @@ -114,21

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-04-24 Thread Alberto Garcia
On Fri 24 Apr 2015 11:52:14 AM CEST, Kevin Wolf kw...@redhat.com wrote: The posix_memalign() call wastes memory. I compared: posix_memalign(memptr, 65536, 2560 * 65536); memset(memptr, 0, 2560 * 65536); with: for (i = 0; i 2560; i++) { posix_memalign(memptr, 65536,

[Qemu-block] [PATCH 10/11] qemu-iotests: test block-stream operations in parallel

2015-04-24 Thread Alberto Garcia
This test case checks that it's possible to launch several stream operations in parallel in the same snapshot chain, each one involving a different set of nodes. Signed-off-by: Alberto Garcia be...@igalia.com --- tests/qemu-iotests/030 | 80

[Qemu-block] [PATCH 11/11] qemu-iotests: test overlapping block-stream operations

2015-04-24 Thread Alberto Garcia
This test case checks that it's not possible to perform two block-stream operations if there are nodes involved in both. Signed-off-by: Alberto Garcia be...@igalia.com --- tests/qemu-iotests/030 | 27 +++ tests/qemu-iotests/030.out | 4 ++-- 2 files changed, 29

[Qemu-block] [PATCH 05/11] block: Add QMP support for streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
() and no longer returns DeviceNotFound, iotest 030 is updated to expect GenericError instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- blockdev.c | 20 ++-- qapi/block-core.json

[Qemu-block] [PATCH 04/11] block: Support streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- block.c| 4 +++- block/stream.c | 33 + 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 25289ef..e892cb4 100644 --- a/block.c +++ b/block.c

[Qemu-block] [PATCH 08/11] qemu-iotests: add no-op streaming test

2015-04-24 Thread Alberto Garcia
This patch updates test_stream_partial() to test that the block-stream operation never copies data from the image specified as base. Signed-off-by: Alberto Garcia be...@igalia.com --- tests/qemu-iotests/030 | 13 + 1 file changed, 13 insertions(+) diff --git a/tests/qemu-iotests/030

[Qemu-block] [PATCH 07/11] qemu-iotests: fix test_stream_partial()

2015-04-24 Thread Alberto Garcia
to the intermediate image before the test, so there's something to copy and the test is meaningful. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/030

[Qemu-block] [PATCH 01/11] block: keep a list of block jobs

2015-04-24 Thread Alberto Garcia
updates qmp_query_block_jobs() to use this new list. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- blockdev.c | 19 --- blockjob.c | 13 + include/block

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-04-24 Thread Alberto Garcia
On Fri 24 Apr 2015 02:37:21 PM CEST, Stefan Hajnoczi wrote: I think it would be nice to have a way to free unused cache entries after a while. Do you think mmap plus a periodic timer would work? I'm hesitant about changes like this because they make QEMU more complex, slow down the guest,

Re: [Qemu-block] [PATCH 06/11] docs: Document how to stream to an intermediate layer

2015-04-24 Thread Alberto Garcia
On Fri 24 Apr 2015 03:11:01 PM CEST, Max Reitz wrote: Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Please remove my R-b if you change the patch in some non-insignificant way (like, obvious conflict resolution while rebasing is fine, differences

Re: [Qemu-block] [PATCH 08/11] qemu-iotests: add no-op streaming test

2015-04-24 Thread Alberto Garcia
On Fri 24 Apr 2015 03:19:54 PM CEST, Max Reitz wrote: This patch updates test_stream_partial() to test that the block-stream operation never copies data from the image specified as base. I think it would be better to add this as an own test case. But I won't oppose adding it as a special

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-04-22 Thread Alberto Garcia
On Wed 22 Apr 2015 12:26:02 PM CEST, Stefan Hajnoczi wrote: Large disk images need large L2 caches in order to maximize their I/O performance. However setting a correct size for the cache is not necessarily easy since apart from the image size, it also depends on other factors like its usage

Re: [Qemu-block] [PATCH 7/8] qemu-iotests: fix test_stream_partial()

2015-04-22 Thread Alberto Garcia
On Wed 22 Apr 2015 09:38:40 PM CEST, Eric Blake wrote: You know, we should _also_ test a no-op stream, to prove that we handle it correctly (we've had bugs in the past where 0-length active commit behaved differently in the events it generated than non-zero length), and your change is

[Qemu-block] [PATCH 10/11] qemu-iotests: test block-stream operations in parallel

2015-04-24 Thread Alberto Garcia
This test case checks that it's possible to launch several stream operations in parallel in the same snapshot chain, each one involving a different set of nodes. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 80

[Qemu-block] [PATCH 02/11] block: allow block jobs in any arbitrary node

2015-04-24 Thread Alberto Garcia
. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- block/mirror.c| 5 +++-- blockdev.c| 16 blockjob.c| 18 ++ docs/qmp/qmp-events.txt

[Qemu-block] [PATCH 06/11] docs: Document how to stream to an intermediate layer

2015-04-24 Thread Alberto Garcia
Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- docs/live-block-ops.txt | 31 --- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/live-block-ops.txt b/docs/live

[Qemu-block] [PATCH 11/11] qemu-iotests: test overlapping block-stream operations

2015-04-24 Thread Alberto Garcia
This test case checks that it's not possible to perform two block-stream operations if there are nodes involved in both. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 27 +++ tests/qemu-iotests/030

[Qemu-block] [PATCH 05/11] block: Add QMP support for streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
() and no longer returns DeviceNotFound, iotest 030 is updated to expect GenericError instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- blockdev.c | 20 ++-- qapi/block-core.json

[Qemu-block] [PATCH 09/11] qemu-iotests: test streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
This adds test_stream_intermediate(), similar to test_stream() but streams to the intermediate image instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 18 +- tests/qemu-iotests/030.out | 4 ++-- 2

[Qemu-block] [PATCH 01/11] block: keep a list of block jobs

2015-04-24 Thread Alberto Garcia
updates qmp_query_block_jobs() to use this new list. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- blockdev.c | 19 --- blockjob.c | 13 + include/block

[Qemu-block] [PATCH 04/11] block: Support streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- block.c| 4 +++- block/stream.c | 33 + 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 25289ef..e892cb4 100644 --- a/block.c +++ b/block.c

[Qemu-block] [PATCH 07/11] qemu-iotests: fix test_stream_partial()

2015-04-24 Thread Alberto Garcia
to the intermediate image before the test, so there's something to copy and the test is meaningful. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/030

[Qemu-block] [PATCH v6 00/11] Support streaming to an intermediate layer

2015-04-24 Thread Alberto Garcia
of all 'block-job-*' commands can now take a node name. - The BlockJobInfo type and all BLOCK_JOB_* events report the node name in the 'device' field if the node does not have a device name. - All intermediate nodes are blocked (and checked for blockers) during the streaming operation. Alberto

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-04-23 Thread Alberto Garcia
On Thu 23 Apr 2015 12:15:04 PM CEST, Stefan Hajnoczi wrote: For a cache size of 128MB, the PSS is actually ~10MB larger without the patch, which seems to come from posix_memalign(). Do you mean RSS or are you using a tool that reports a PSS number that I don't know about? We should

[Qemu-block] [PATCH 4/6] qcow2: remove qcow2_cache_find_entry_to_replace()

2015-04-30 Thread Alberto Garcia
A cache miss means that the whole array was traversed and the entry we were looking for was not found, so there's no need to traverse it again in order to select an entry to replace. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 45

[Qemu-block] [PATCH 0/6] qcow2 L2/refcount cache improvements

2015-04-30 Thread Alberto Garcia
find the changes worthwhile. Regards, Berto Alberto Garcia (6): qcow2: use one single memory block for the L2/refcount cache tables qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty() qcow2: use an LRU algorithm to replace entries from the L2 cache qcow2: remove

[Qemu-block] [PATCH 2/6] qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()

2015-04-30 Thread Alberto Garcia
Since all tables are now stored together, it is possible to obtain the position of a particular table directly from its address, so the operation becomes O(1). Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 22 +- 1 file changed, 5 insertions(+), 17

[Qemu-block] [PATCH 3/6] qcow2: use an LRU algorithm to replace entries from the L2 cache

2015-04-30 Thread Alberto Garcia
in positions 8 and higher are rarely (if ever) evicted. This can be seen even with the default cache size, but with larger caches the problem becomes more obvious. Using an LRU algorithm makes the chances of being removed from the cache independent from the position. Signed-off-by: Alberto Garcia

[Qemu-block] [PATCH 5/6] qcow2: use a hash to look for entries in the L2 cache

2015-04-30 Thread Alberto Garcia
. The hash is computed using the cluster number of the table, multiplied by 4 to make it perform better when there are collisions. In my tests, using a cache with 2048 entries, this reduces the average number of comparisons per lookup from 430 to 2.5. Signed-off-by: Alberto Garcia

[Qemu-block] [PATCH 6/6] qcow2: style fixes in qcow2-cache.c

2015-04-30 Thread Alberto Garcia
Fix pointer declaration to make it consistent with the rest of the code. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/block/qcow2-cache.c b/block/qcow2-cache.c index c0e0278..dd591ef 100644

[Qemu-block] [PATCH 1/6] qcow2: use one single memory block for the L2/refcount cache tables

2015-04-30 Thread Alberto Garcia
that contains all the tables using one single allocation. In my tests measuring freshly created caches with sizes 128MB (L2) and 32MB (refcount) this uses around 10MB of RAM less. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 48

Re: [Qemu-block] [Qemu-devel] [PATCH 1/6] qcow2: use one single memory block for the L2/refcount cache tables

2015-05-05 Thread Alberto Garcia
On Thu 30 Apr 2015 05:08:05 PM CEST, Eric Blake ebl...@redhat.com wrote: typedef struct Qcow2CachedTable { -void* table; int64_t offset; booldirty; int cache_hits; @@ -40,39 +39,34 @@ struct Qcow2Cache { struct Qcow2Cache* depends; int

Re: [Qemu-block] [Qemu-devel] [PATCH 2/6] qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()

2015-05-05 Thread Alberto Garcia
On Fri 01 May 2015 04:31:52 PM CEST, Stefan Hajnoczi wrote: int qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table) { -int i; +int i = (*table - c-table_array) / c-table_size; -for (i = 0; i c-size; i++) { -if (table_addr(c, i) == *table) { -

Re: [Qemu-block] [PATCH 1/6] qcow2: use one single memory block for the L2/refcount cache tables

2015-05-05 Thread Alberto Garcia
On Tue 05 May 2015 01:20:19 PM CEST, Kevin Wolf wrote: Though looking at the code again I see now that c-table_size isn't consistently used. The I/O requests still use s-cluster_size. We should either use it everywhere or not introduce it at all. c-table_size is necessary in order to

[Qemu-block] [PATCH 1/7] qcow2: use one single memory block for the L2/refcount cache tables

2015-05-06 Thread Alberto Garcia
that contains all the tables using one single allocation. In my tests measuring freshly created caches with sizes 128MB (L2) and 32MB (refcount) this uses around 10MB of RAM less. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c| 55

[Qemu-block] [PATCH 4/7] qcow2: remove qcow2_cache_find_entry_to_replace()

2015-05-06 Thread Alberto Garcia
A cache miss means that the whole array was traversed and the entry we were looking for was not found, so there's no need to traverse it again in order to select an entry to replace. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 45

[Qemu-block] [PATCH 2/7] qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()

2015-05-06 Thread Alberto Garcia
Since all tables are now stored together, it is possible to obtain the position of a particular table directly from its address, so the operation becomes O(1). Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 32 +++- 1 file changed, 15

[Qemu-block] [PATCH v2 0/7] qcow2 L2/refcount cache improvements

2015-05-06 Thread Alberto Garcia
in the cache, get it from the BDS instead v1: https://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg04355.html Regards, Berto Alberto Garcia (7): qcow2: use one single memory block for the L2/refcount cache tables qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty() qcow2

[Qemu-block] [PATCH 7/7] qcow2: style fixes in qcow2-cache.c

2015-05-06 Thread Alberto Garcia
Fix pointer declaration to make it consistent with the rest of the code. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/block/qcow2-cache.c b/block/qcow2-cache.c index 0f629c4..bea43c1 100644

[Qemu-block] [PATCH 5/7] qcow2: use a hash to look for entries in the L2 cache

2015-05-06 Thread Alberto Garcia
. The hash is computed using the cluster number of the table, multiplied by 4 to make it perform better when there are collisions. In my tests, using a cache with 2048 entries, this reduces the average number of comparisons per lookup from 430 to 2.5. Signed-off-by: Alberto Garcia

[Qemu-block] [PATCH 3/7] qcow2: use an LRU algorithm to replace entries from the L2 cache

2015-05-06 Thread Alberto Garcia
in positions 8 and higher are rarely (if ever) evicted. This can be seen even with the default cache size, but with larger caches the problem becomes more obvious. Using an LRU algorithm makes the chances of being removed from the cache independent from the position. Signed-off-by: Alberto Garcia

Re: [Qemu-block] [PATCH 5/6] qcow2: use a hash to look for entries in the L2 cache

2015-05-07 Thread Alberto Garcia
On Wed 06 May 2015 06:42:30 PM CEST, Stefan Hajnoczi wrote: By using a hash of the offset as the starting point, lookups are faster and independent from the array size. The hash is computed using the cluster number of the table, multiplied by 4 to make it perform better when there are

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-05-08 Thread Alberto Garcia
On Fri 24 Apr 2015 03:04:06 PM CEST, Kevin Wolf kw...@redhat.com wrote: I think it would be nice to have a way to free unused cache entries after a while. Do you think mmap plus a periodic timer would work? I'm hesitant about changes like this because they make QEMU more complex,

Re: [Qemu-block] [Qemu-devel] [PATCH 2/6] qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()

2015-05-06 Thread Alberto Garcia
On Wed 06 May 2015 05:00:50 PM CEST, Stefan Hajnoczi wrote: int qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table) { -int i; +int i = (*table - c-table_array) / c-table_size; -for (i = 0; i c-size; i++) { -if (table_addr(c, i) == *table) {

Re: [Qemu-block] [PATCH 5/7] qcow2: use a hash to look for entries in the L2 cache

2015-05-08 Thread Alberto Garcia
On Fri 08 May 2015 05:46:57 PM CEST, Max Reitz wrote: Let's assume the cache is full. Now this hash algorithm (direct mapped cache) basically becomes futile, because the LRU algorithm (fully associative cache) takes over That's right, although in that scenario I guess there's no good

Re: [Qemu-block] [PATCH 6/7] qcow2: make qcow2_cache_put() a void function

2015-05-08 Thread Alberto Garcia
On Fri 08 May 2015 05:51:30 PM CEST, Max Reitz wrote: -int qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table) +void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table) { int i = qcow2_cache_get_table_idx(bs, c, *table); -if (c-entries[i].offset

[Qemu-block] [PATCH 7/8] qcow2: style fixes in qcow2-cache.c

2015-05-11 Thread Alberto Garcia
Fix pointer declaration to make it consistent with the rest of the code. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com --- block/qcow2-cache.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions

[Qemu-block] [PATCH 2/8] qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()

2015-05-11 Thread Alberto Garcia
Since all tables are now stored together, it is possible to obtain the position of a particular table directly from its address, so the operation becomes O(1). Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com

[Qemu-block] [PATCH 6/8] qcow2: make qcow2_cache_put() a void function

2015-05-11 Thread Alberto Garcia
This function never receives an invalid table pointer, so we can make it void and remove all the error checking code. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com --- block/qcow2-cache.c| 7 +-- block/qcow2-cluster.c | 50

[Qemu-block] [PATCH 8/8] docs: document how to configure the qcow2 L2/refcount caches

2015-05-11 Thread Alberto Garcia
of the cluster. This document attempts to give an overview of both caches and how to configure their sizes. Signed-off-by: Alberto Garcia be...@igalia.com --- docs/qcow2-cache.txt | 117 +++ 1 file changed, 117 insertions(+) create mode 100644 docs

[Qemu-block] [PATCH 1/8] qcow2: use one single memory block for the L2/refcount cache tables

2015-05-11 Thread Alberto Garcia
that contains all the tables using one single allocation. In my tests measuring freshly created caches with sizes 128MB (L2) and 32MB (refcount) this uses around 10MB of RAM less. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com Reviewed-by: Max Reitz

[Qemu-block] [PATCH 5/8] qcow2: use a hash to look for entries in the L2 cache

2015-05-11 Thread Alberto Garcia
. The hash is computed using the cluster number of the table, multiplied by 4 to make it perform better when there are collisions. In my tests, using a cache with 2048 entries, this reduces the average number of comparisons per lookup from 430 to 2.5. Signed-off-by: Alberto Garcia

[Qemu-block] [PATCH 4/8] qcow2: remove qcow2_cache_find_entry_to_replace()

2015-05-11 Thread Alberto Garcia
A cache miss means that the whole array was traversed and the entry we were looking for was not found, so there's no need to traverse it again in order to select an entry to replace. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com Reviewed-by: Max

[Qemu-block] [PATCH 3/8] qcow2: use an LRU algorithm to replace entries from the L2 cache

2015-05-11 Thread Alberto Garcia
in positions 8 and higher are rarely (if ever) evicted. This can be seen even with the default cache size, but with larger caches the problem becomes more obvious. Using an LRU algorithm makes the chances of being removed from the cache independent from the position. Signed-off-by: Alberto Garcia

[Qemu-block] [PATCH v3 0/8] qcow2 L2/refcount cache improvements

2015-05-11 Thread Alberto Garcia
, Berto Alberto Garcia (8): qcow2: use one single memory block for the L2/refcount cache tables qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty() qcow2: use an LRU algorithm to replace entries from the L2 cache qcow2: remove qcow2_cache_find_entry_to_replace() qcow2: use

Re: [Qemu-block] [PATCH 8/8] docs: document how to configure the qcow2 L2/refcount caches

2015-05-11 Thread Alberto Garcia
On Mon 11 May 2015 03:23:25 PM CEST, Max Reitz mre...@redhat.com wrote: + disk_size = l2_cache_size * cluster_size / 8 + disk_size = refcount_cache_size * cluster_size / 2 Only with the default of refcount_bits=16. In the general case, it's refcount_cache_size * cluster_size * 8 /

Re: [Qemu-block] [PATCH 3/7] qcow2: use an LRU algorithm to replace entries from the L2 cache

2015-05-07 Thread Alberto Garcia
On Thu 07 May 2015 04:55:21 PM CEST, Eric Blake wrote: /* Give the table some hits for the start so that it won't be replaced * immediately. The number 32 is completely arbitrary. */ -c-entries[i].cache_hits = 32; c-entries[i].offset = offset; The comment is now dead.

Re: [Qemu-block] [PATCH] qcow2: do lazy allocation of the L2 cache

2015-05-08 Thread Alberto Garcia
On Fri 08 May 2015 11:47:00 AM CEST, Kevin Wolf wrote: There's also the problem that with a single chunk of memory for all cache tables it's not so easy to free individual entries. That's one of the reasons why I suggested to fix the problem by using mmap() for the individual entries rather

[Qemu-block] [PATCH 01/11] block: keep a list of block jobs

2015-05-13 Thread Alberto Garcia
updates qmp_query_block_jobs() and bdrv_drain_all() to use this new list. Signed-off-by: Alberto Garcia be...@igalia.com Cc: Max Reitz mre...@redhat.com Cc: Eric Blake ebl...@redhat.com --- block/io.c | 21 - blockdev.c | 19

[Qemu-block] [PATCH 03/11] block: never cancel a streaming job without running stream_complete()

2015-05-13 Thread Alberto Garcia
. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- block/stream.c | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/block/stream.c b/block/stream.c index a628901..37bfd8b 100644 --- a/block/stream.c +++ b/block/stream.c @@ -114,21

[Qemu-block] [PATCH 09/11] qemu-iotests: test streaming to an intermediate layer

2015-05-13 Thread Alberto Garcia
This adds test_stream_intermediate(), similar to test_stream() but streams to the intermediate image instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 18 +- tests/qemu-iotests/030.out | 4 ++-- 2

[Qemu-block] [PATCH 07/11] qemu-iotests: fix test_stream_partial()

2015-05-13 Thread Alberto Garcia
to the intermediate image before the test, so there's something to copy and the test is meaningful. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/030

[Qemu-block] [PATCH v7 00/11] Support streaming to an intermediate layer

2015-05-13 Thread Alberto Garcia
name in the 'device' field if the node does not have a device name. - All intermediate nodes are blocked (and checked for blockers) during the streaming operation. v1: https://lists.gnu.org/archive/html/qemu-devel/2015-02/msg04116.html Regards, Berto Alberto Garcia (11): block: keep a list

[Qemu-block] [PATCH 05/11] block: Add QMP support for streaming to an intermediate layer

2015-05-13 Thread Alberto Garcia
() and no longer returns DeviceNotFound, iotest 030 is updated to expect GenericError instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- blockdev.c | 20 ++-- qapi/block-core.json

[Qemu-block] [PATCH 10/11] qemu-iotests: test block-stream operations in parallel

2015-05-13 Thread Alberto Garcia
This test case checks that it's possible to launch several stream operations in parallel in the same snapshot chain, each one involving a different set of nodes. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 80

[Qemu-block] [PATCH 06/11] docs: Document how to stream to an intermediate layer

2015-05-13 Thread Alberto Garcia
Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- docs/live-block-ops.txt | 31 --- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/live-block-ops.txt b/docs/live

[Qemu-block] [PATCH 11/11] qemu-iotests: test overlapping block-stream operations

2015-05-13 Thread Alberto Garcia
This test case checks that it's not possible to perform two block-stream operations if there are nodes involved in both. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 27 +++ tests/qemu-iotests/030

[Qemu-block] [PATCH 04/11] block: Support streaming to an intermediate layer

2015-05-13 Thread Alberto Garcia
-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- block.c| 4 +++- block/stream.c | 33 + 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 7904098..8fab7e4 100644 --- a/block.c +++ b/block.c

Re: [Qemu-block] [Qemu-devel] [PATCH 11/11] qemu-iotests: test overlapping block-stream operations

2015-05-15 Thread Alberto Garcia
On Fri 15 May 2015 04:56:09 AM CEST, Fam Zheng wrote: +# If node4 is the active node, the id of the block job is drive0 +if self.num_imgs == 5: Isn't self.num_imgs a constant (9) ? Yes, but the number is arbitrary. The idea is to let users change it if they want to, so the

Re: [Qemu-block] [PATCH v2 1/2] block: Detect multiplication overflow in bdrv_getlength

2015-05-15 Thread Alberto Garcia
...@redhat.com Reviewed-by: Alberto Garcia be...@igalia.com Berto

Re: [Qemu-block] [Qemu-devel] [PATCH 04/11] block: Support streaming to an intermediate layer

2015-05-15 Thread Alberto Garcia
On Fri 15 May 2015 04:33:19 AM CEST, Fam Zheng wrote: +/* Make sure that the image is opened in read-write mode */ +orig_bs_flags = bdrv_get_flags(bs); +if (!(orig_bs_flags BDRV_O_RDWR)) { +if (bdrv_reopen(bs, orig_bs_flags | BDRV_O_RDWR, errp) != 0) { I don't think all

Re: [Qemu-block] [PATCH 1/2] block: Detect multiplication overflow in bdrv_getlength

2015-05-15 Thread Alberto Garcia
On Fri 15 May 2015 03:39:10 AM CEST, Fam Zheng f...@redhat.com wrote: int64_t ret = bdrv_nb_sectors(bs); +ret = (int64_t)(ret * BDRV_SECTOR_SIZE) 0 ? -EFBIG : ret; return ret 0 ? ret : ret * BDRV_SECTOR_SIZE; Maybe in this case you're safe, but in general there's no

[Qemu-block] [PATCH 3/3] qcow2: reorder fields in Qcow2CachedTable to reduce padding

2015-05-18 Thread Alberto Garcia
Changing the current ordering saves 8 bytes per cache entry in x86_64. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/qcow2-cache.c b/block/qcow2-cache.c index a215f5b..43590ff 100644 --- a/block

[Qemu-block] [PATCH 1/3] qcow2: mark the memory as no longer needed after qcow2_cache_empty()

2015-05-18 Thread Alberto Garcia
(and fills both caches), then it empties the top image. At this point the data in that cache is no longer needed so it's just wasting memory. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 20 1 file changed, 20 insertions(+) diff --git a/block/qcow2

[Qemu-block] [PATCH 0/3] Clean unused entries in the qcow2 L2/refcount cache

2015-05-18 Thread Alberto Garcia
These patches use MADV_DONTNEED to clean unused cache entries. Under Linux it frees the memory used by those pages. Berto Alberto Garcia (3): qcow2: mark the memory as no longer needed after qcow2_cache_empty() qcow2: add option to clean unused cache entries after some time qcow2: reorder

[Qemu-block] [PATCH 2/3] qcow2: add option to clean unused cache entries after some time

2015-05-18 Thread Alberto Garcia
of inactivity. This feature currently relies on MADV_DONTNEED to free that memory, so it is not useful in systems that don't follow that behavior. Signed-off-by: Alberto Garcia be...@igalia.com --- block/qcow2-cache.c | 35 block/qcow2.c| 56

Re: [Qemu-block] [PATCH 1/1] docs: document how to configure the qcow2 L2/refcount caches

2015-05-18 Thread Alberto Garcia
On Mon 18 May 2015 06:28:03 PM CEST, Max Reitz mre...@redhat.com wrote: +This work is licensed under the terms of the GNU GPL, version 2 or +later. See the COPYING file in the top-level directory. Well, GPL is strange for documentation but I guess it works (also, it's the implicit license

Re: [Qemu-block] [PATCH] block: Let bdrv_drain_all() to call aio_poll() for each AioContext

2015-05-13 Thread Alberto Garcia
On Wed 13 May 2015 05:18:31 PM CEST, Alexander Yarygin yary...@linux.vnet.ibm.com wrote: +if (!aio_ctxs || !g_list_find(aio_ctxs, aio_context)) { +busy |= aio_poll(aio_context, busy); +aio_ctxs = g_list_append(aio_ctxs, aio_context); +}

[Qemu-block] [PATCH 1/1] docs: document how to configure the qcow2 L2/refcount caches

2015-05-12 Thread Alberto Garcia
of the cluster and the refcount entries. This document attempts to give an overview of both caches and how to configure their sizes. Signed-off-by: Alberto Garcia be...@igalia.com --- docs/qcow2-cache.txt | 143 +++ 1 file changed, 143 insertions

[Qemu-block] [PATCH v2 0/1] Document how to configure the qcow2 L2/refcount caches

2015-05-12 Thread Alberto Garcia
Kevin applied all other patches from this series, so this is the only one left. Thanks to Max for all the feedback. v2: - Add copyright notice - Document refcount_bits and its effects v1: https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg01891.html Regards, Berto Alberto Garcia (1

Re: [Qemu-block] [PATCH] qemu-io: Use getopt() correctly

2015-05-12 Thread Alberto Garcia
loop. Signed-off-by: Eric Blake ebl...@redhat.com Reviewed-by: Alberto Garcia be...@igalia.com Berto

Re: [Qemu-block] [PATCH 4/6] block: Support streaming to an intermediate layer

2015-04-16 Thread Alberto Garcia
On Wed 15 Apr 2015 06:09:18 PM CEST, Max Reitz wrote: +orig_bs_flags = bdrv_get_flags(bs); +if (!(orig_bs_flags BDRV_O_RDWR)) { I feel like we don't want to do this if we're not streaming to an intermediate layer but to the top layer (because that means there is some reason for the

Re: [Qemu-block] [Qemu-devel] [PATCH 4/6] block: Support streaming to an intermediate layer

2015-04-16 Thread Alberto Garcia
On Thu 16 Apr 2015 02:27:39 PM CEST, Eric Blake wrote: +orig_bs_flags = bdrv_get_flags(bs); +if (!(orig_bs_flags BDRV_O_RDWR)) { I feel like we don't want to do this if we're not streaming to an intermediate layer but to the top layer (because that means there is some reason for

[Qemu-block] [PATCH 2/8] block: allow block jobs in any arbitrary node

2015-04-16 Thread Alberto Garcia
. Signed-off-by: Alberto Garcia be...@igalia.com --- block/mirror.c| 5 +++-- blockdev.c| 16 blockjob.c| 18 ++ docs/qmp/qmp-events.txt | 8 include/qapi/qmp/qerror.h | 3 --- qapi/block-core.json | 20

[Qemu-block] [PATCH 4/8] block: Support streaming to an intermediate layer

2015-04-16 Thread Alberto Garcia
-by: Alberto Garcia be...@igalia.com --- block.c| 4 +++- block/stream.c | 33 + 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 25289ef..e892cb4 100644 --- a/block.c +++ b/block.c @@ -1240,9 +1240,11 @@ void

[Qemu-block] [PATCH 5/8] block: Add QMP support for streaming to an intermediate layer

2015-04-16 Thread Alberto Garcia
() and no longer returns DeviceNotFound, iotest 030 is updated to expect GenericError instead. Signed-off-by: Alberto Garcia be...@igalia.com --- blockdev.c | 20 ++-- qapi/block-core.json | 10 +++--- tests/qemu-iotests/030 | 2 +- 3 files changed, 18 insertions

[Qemu-block] [PATCH v4 0/8] Support streaming to an intermediate layer

2015-04-16 Thread Alberto Garcia
the node name in the 'device' field if the node does not have a device name. - All intermediate nodes are blocked (and checked for blockers) during the streaming operation. Regards, Berto Alberto Garcia (8): block: keep a list of block jobs block: allow block jobs in any arbitrary node block

[Qemu-block] [PATCH 3/8] block: never cancel a streaming job without running stream_complete()

2015-04-16 Thread Alberto Garcia
. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- block/stream.c | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/block/stream.c b/block/stream.c index a628901..37bfd8b 100644 --- a/block/stream.c +++ b/block/stream.c @@ -114,21

[Qemu-block] [PATCH 6/8] docs: Document how to stream to an intermediate layer

2015-04-16 Thread Alberto Garcia
Signed-off-by: Alberto Garcia be...@igalia.com --- docs/live-block-ops.txt | 30 +++--- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/docs/live-block-ops.txt b/docs/live-block-ops.txt index a257087..3bf86be 100644 --- a/docs/live-block-ops.txt +++ b/docs

Re: [Qemu-block] [PATCH 4/6] block: Support streaming to an intermediate layer

2015-04-16 Thread Alberto Garcia
On Wed 15 Apr 2015 06:09:18 PM CEST, Max Reitz mre...@redhat.com wrote: +orig_bs_flags = bdrv_get_flags(bs); +if (!(orig_bs_flags BDRV_O_RDWR)) { I feel like we don't want to do this if we're not streaming to an intermediate layer but to the top layer (because that means there is

Re: [Qemu-block] [PATCH] qmp: fill in the image field in BlockDeviceInfo

2015-04-17 Thread Alberto Garcia
On Fri 17 Apr 2015 01:52:43 PM CEST, Alberto Garcia be...@igalia.com wrote: Anyone calling bdrv_block_device_info() directly will get a null image field. As a consequence of this, the HMP command 'info block -n -v' crashes QEMU. This patch moves the code that fills in that field from

Re: [Qemu-block] [RFC] Intermediate block mirroring

2015-04-09 Thread Alberto Garcia
On Thu, Apr 09, 2015 at 11:39:28AM +0100, Stefan Hajnoczi wrote: The goal would be to convert this: [A] - [B] - [C] - [D] into this: [A] - [B] - [X] - [D] where [D] is the active image and [X] would be a copy of [C]. The latter would be unlinked from the chain.

Re: [Qemu-block] [PATCH 3/3] block: add 'node-name' field to BLOCK_IMAGE_CORRUPTED

2015-04-09 Thread Alberto Garcia
On Wed, Apr 08, 2015 at 05:32:47PM -0600, Eric Blake wrote: +- device:Device name (json-string) +- node-name: Node name (json-string, optional) +- msg: Informative message (e.g., reason for the corruption) + (json-string) +- offset:If the corruption resulted

[Qemu-block] [PATCH RFC] mirror: allow mirroring an intermediate image

2015-04-09 Thread Alberto Garcia
This extends the drive-mirror command by allowing it to operate on intermediate images. The device parameter can now take the node name of the image to be mirrored. After the operation its overlay image will point to the new one. Signed-off-by: Alberto Garcia be...@igalia.com --- block.c

Re: [Qemu-block] [PATCH v2 2/4] block: Pause block jobs in bdrv_drain_all

2015-04-07 Thread Alberto Garcia
not the best one. Maybe we can have block_job_create/completed() maintain a list of jobs instead. https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg04800.html But anyway I don't think it's something for this patchset, therefore: Reviewed-by: Alberto Garcia be...@igalia.com Berto

  1   2   3   4   5   6   7   8   9   10   >