Re: [Qemu-devel] QCOW2 cryptography and secure key handling

2013-07-23 Thread Benoît Canet
Do you (the block maintainers) have an idea on how the code could be improved to securely pass the crypto key to the QCOW2 code ? More generally, QCow2's current encryption support is woefully inadequate from a design POV. If we wanted better encryption built-in to QEMU it is best to

Re: [Qemu-devel] QCOW2 cryptography and secure key handling

2013-07-23 Thread Benoît Canet
More generally, QCow2's current encryption support is woefully inadequate from a design POV. If we wanted better encryption built-in to QEMU it is best to just deprecate the current encryption support and define a new qcow2 extension based around something like the LUKS data format. Using the

Re: [Qemu-devel] [PATCH V2 for-1.6 2/5] block: Modify the throttling code to implement the leaky bucket algorithm.

2013-07-23 Thread Benoît Canet
1. Can we activate the timer only when requests are actually pending? Imagine a host with 1000 guests, even a 1 second timer becomes wasteful. I will try to do this. 2. You don't vary the wait time, does this mean a throttled request must wait for max 1 second? If yes, then it

Re: [Qemu-devel] [PATCH V2 for-1.6 2/5] block: Modify the throttling code to implement the leaky bucket algorithm.

2013-07-23 Thread Benoît Canet
I don't fully understand the patch yet but: The patch is a variant of the following algorithm. http://en.wikipedia.org/wiki/Leaky_bucket Best regards Benoît

[Qemu-devel] [PATCH V3 for-1.6 1/5] block: Repair the throttling code.

2013-07-23 Thread Benoît Canet
The throttling code was segfaulting since commit 02ffb504485f0920cfc75a0982a602f824a9a4f4 because some qemu_co_queue_next caller does not run in a coroutine. qemu_co_queue_do_restart assume that the caller is a coroutinne. As suggested by Stefan fix this by entering the coroutine directly. Also

[Qemu-devel] [PATCH V3 for-1.6 3/5] block: Add support for throttling burst threshold in QMP and the command line.

2013-07-23 Thread Benoît Canet
The thresholds of the leaky bucket algorithm can be used to allow some burstiness. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qapi.c | 24 + blockdev.c | 105 +++--- hmp.c| 32 +++--

[Qemu-devel] [PATCH V3 for-1.6 0/5] Leaky bucket throttling and features

2013-07-23 Thread Benoît Canet
activity [Stefan] Mark function as coroutine_fn [Stefan] Guard these function checking they are in a coroutine [Stefan] Use defines to access buckets [Stefan] Fix typo [Stefan] reset throttling metric on iddle [Benoît] rename invalid to check_io_limit [Stefan] Benoît Canet (5

[Qemu-devel] [PATCH V3 for-1.6 2/5] block: Modify the throttling code to implement the leaky bucket algorithm.

2013-07-23 Thread Benoît Canet
This patch replace the previous algorithm by the well described leaky bucket algorithm: A bucket is filled by the incoming IOs and a periodic timer decrement the counter to make the bucket leak. When a given threshold is reached the bucket is full and the IOs are hold. In this patch the threshold

[Qemu-devel] [PATCH V3 for-1.6 5/5] block: Add throttling percentage metrics.

2013-07-23 Thread Benoît Canet
This metrics show how many percent of the time I/Os are put on hold by the throttling algorithm. This metric could be used by system administrators or cloud stack developpers to decide when the throttling settings must be changed. Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c

[Qemu-devel] [PATCH V3 for-1.6 4/5] block: Add iops_sector_count to do the iops accounting for a given io size.

2013-07-23 Thread Benoît Canet
This feature can be used in case where users are avoiding the iops limit by doing jumbo I/Os hammering the storage backend. Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c |8 +++- block/qapi.c |4 blockdev.c| 22

Re: [Qemu-devel] QCOW2 cryptography and secure key handling

2013-07-24 Thread Benoît Canet
There are two ways I could see it happening. Either integrate directly into the qcow2 file format, by mapping LUKS headers key material blocks into the qcow2 header region in some manner. Alternatively do it in a completely generic block driver, that qcow2 (or any other qemu bdrv) calls

Re: [Qemu-devel] [PATCH V3 for-1.6 0/5] Leaky bucket throttling and features

2013-07-25 Thread Benoît Canet
Le Thursday 25 Jul 2013 à 20:08:22 (+0800), Fam Zheng a écrit : My ignorant question: Besides the fix, how is io throttling improved from previously, what are the advantages of leaky bucket algorithm? The advantage is simplicity and the ability to allow controlled bursts. The disavantage is it

Re: [Qemu-devel] [PATCH V3 for-1.6 2/5] block: Modify the throttling code to implement the leaky bucket algorithm.

2013-07-26 Thread Benoît Canet
I think it is easier to understand written like this: int64_t total_leak = ((bs-io_limits.iops[BLOCK_IO_LIMIT_TOTAL] * delta) / NANOSECONDS_PER_SECOND); if (ios[BLOCK_IO_LIMIT_READ] = total_leak / 2) { read_leak =

Re: [Qemu-devel] [PATCH V3 for-1.6 5/5] block: Add throttling percentage metrics.

2013-07-26 Thread Benoît Canet
If I understand it, the percentage is recalculated every leak check. So it only reflects the instant io flow, instead of historical statistics? But I think for system admin purpose, it's good to know a longer range io activity character. Or do you think management tool should sample it? Yes I

Re: [Qemu-devel] [PATCH 13/18] blockdev: Rename I/O throttling options for QMP

2013-07-26 Thread Benoît Canet
This patch will probably conflict with Benoît's work on leaky bucket throttling; can the two of you decide which one should go in first? Are we trying to target both this series and leaky bucket throttling for 1.6? I will to rebase my serie on top of this. However if anyone has suggestions

[Qemu-devel] [PATCH/FIX FOR 1.6] Repair the throttling code

2013-07-26 Thread Benoît Canet
The throttling code was segfaulting repair it for 1.6. Benoît Canet (1): block: Repair the throttling code. block.c |8 include/block/coroutine.h |9 +++-- qemu-coroutine-lock.c | 20 ++-- 3 files changed, 29 insertions(+), 8

[Qemu-devel] [PATCH/FIX FOR 1.6] block: Repair the throttling code.

2013-07-26 Thread Benoît Canet
The throttling code was segfaulting since commit 02ffb504485f0920cfc75a0982a602f824a9a4f4 because some qemu_co_queue_next caller does not run in a coroutine. qemu_co_queue_do_restart assume that the caller is a coroutinne. As suggested by Stefan fix this by entering the coroutine directly. Also

[Qemu-devel] [PATCH V4/FIX FOR 1.6] repair throttling code

2013-07-26 Thread Benoît Canet
since v3: silence bogus checkpatch warning [abligh] Benoît Canet (1): block: Repair the throttling code. block.c |7 +++ include/block/coroutine.h |9 +++-- qemu-coroutine-lock.c | 20 ++-- 3 files changed, 28 insertions(+), 8 deletions

[Qemu-devel] [PATCH V4/FIX FOR 1.6] block: Repair the throttling code.

2013-07-26 Thread Benoît Canet
The throttling code was segfaulting since commit 02ffb504485f0920cfc75a0982a602f824a9a4f4 because some qemu_co_queue_next caller does not run in a coroutine. qemu_co_queue_do_restart assume that the caller is a coroutinne. As suggested by Stefan fix this by entering the coroutine directly. Also

Re: [Qemu-devel] [PATCH V3 for-1.6 3/5] block: Add support for throttling burst threshold in QMP and the command line.

2013-07-26 Thread Benoît Canet
Kevin's series renamed these to have a dash in the name, and also moved all the throttling parameters into a sub-struct. Does it make more sense to have just '*throttling' with that sub-struct containing 12 parameters, 6 for limits and 6 for thresholds, or would it be better to have

Re: [Qemu-devel] VFIO and scheduled SR-IOV cards

2013-07-28 Thread Benoît Canet
Confused. You have one VF accessing BAR of another VF? Why? The VFs are scheduled and the board have only one microcontroller responsible to give the read results for some memory mapped registers. So it could respond the value of the active VF when a bar of an inactive VF is read. I handed

Re: [Qemu-devel] QCOW2 cryptography and secure key handling

2013-07-29 Thread Benoît Canet
: On Tue, Jul 23, 2013 at 05:38:00PM +0200, Kevin Wolf wrote: Am 23.07.2013 um 17:22 hat Stefan Hajnoczi geschrieben: On Tue, Jul 23, 2013 at 04:40:34PM +0200, Benoît Canet wrote: More generally, QCow2's current encryption support is woefully inadequate from a design POV. If we

Re: [Qemu-devel] QCOW2 cryptography and secure key handling

2013-07-31 Thread Benoît Canet
Crypto should be done by trained professionals[*]. I agree I feel uneasy to write cryptographic code. Best regards Benoît

Re: [Qemu-devel] QCOW2 cryptography and secure key handling

2013-07-31 Thread Benoît Canet
For example, current qcow2 encryption is vulnerable to a watermarking attack. http://en.wikipedia.org/wiki/Disk_encryption_theory#Cipher-block_chaining_.28CBC.29 void qcow2_encrypt_sectors(BDRVQcowState *s, int64_t sector_num, uint8_t *out_buf, const uint8_t *in_buf,

[Qemu-devel] [RFC V3 1/2] throttle: Add a new throttling API implementing continuus leaky bucket.

2013-08-02 Thread Benoît Canet
throttling infrastructure + * + * Copyright (C) Nodalink, SARL. 2013 + * + * Author: + * Benoît Canet benoit.ca...@irqsave.net + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License as + * published by the Free Software

[Qemu-devel] [RFC V3 2/2] block: Enable the new throttling code in the block layer.

2013-08-02 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c | 316 +++-- block/qapi.c | 21 ++- blockdev.c| 115 + include/block/block.h |1 - include/block/block_int.h | 33 +

[Qemu-devel] [RFC V3 0/2] continuous leaky bucket throttling

2013-08-02 Thread Benoît Canet
strategies to avoid this: two timer, two throttled request queues or even a different algorithm using a priority queue. The problem is still the same in every version of the code: reads and writes operation seems entangled. Benoît Canet (2): throttle: Add a new throttling API implementing continuus leaky

Re: [Qemu-devel] [PATCH] KVM: always use MADV_DONTFORK

2013-08-06 Thread Benoît Canet
Le Thursday 25 Jul 2013 à 12:11:15 (+0200), Andrea Arcangeli a écrit : MADV_DONTFORK prevents fork to fail with -ENOMEM if the default overcommit heuristics decides there's too much anonymous virtual memory allocated. If the KVM secondary MMU is synchronized with MMU notifiers or not, doesn't

Re: [Qemu-devel] [RFC V3 0/2] continuous leaky bucket throttling

2013-08-07 Thread Benoît Canet
I saw more discussion on IRC. Does this mean you will send another revision to address outstanding issues? Just wanted to check if you are waiting for code review or if you are still developing the next patch revision. I am currently finishing to write unit tests for the next patch

Re: [Qemu-devel] [PATCH V3 for-1.6 3/5] block: Add support for throttling burst threshold in QMP and the command line.

2013-08-08 Thread Benoît Canet
Kevin's series renamed these to have a dash in the name, and also moved all the throttling parameters into a sub-struct. Does it make more sense to have just '*throttling' with that sub-struct containing 12 parameters, 6 for limits and 6 for thresholds, or would it be better to have

[Qemu-devel] [PATCH V4 0/5] Continuous Leaky Bucket Throttling

2013-08-08 Thread Benoît Canet
: wrap qemu-option.hx declararation [Eric] continuus - continuous [Fam] unit test [Paolo] Benoît Canet (5): throttle: Add a new throttling API implementing continuous leaky bucket. throttle: Add units tests block: Enable the new throttling code in the block layer. block: Add support

[Qemu-devel] [PATCH V4 1/5] throttle: Add a new throttling API implementing continuous leaky bucket.

2013-08-08 Thread Benoît Canet
throttling infrastructure + * + * Copyright (C) Nodalink, SARL. 2013 + * + * Author: + * Benoît Canet benoit.ca...@irqsave.net + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License as + * published by the Free Software

[Qemu-devel] [PATCH V4 3/5] block: Enable the new throttling code in the block layer.

2013-08-08 Thread Benoît Canet
tip: Do not ever use the cfg scheduler in the guest with this code. It gives incorrect throttling. Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c | 351 ++--- block/qapi.c | 21 ++- blockdev.c

[Qemu-devel] [PATCH V4 2/5] throttle: Add units tests

2013-08-08 Thread Benoît Canet
file mode 100644 index 000..c687f5f --- /dev/null +++ b/tests/test-throttle.c @@ -0,0 +1,494 @@ +/* + * Throttle infrastructure tests + * + * Copyright Nodalink, SARL. 2013 + * + * Authors: + * Benoît Canet benoit.ca...@irqsave.net + * + * This work is licensed under the terms of the GNU

[Qemu-devel] [PATCH V4 4/5] block: Add support for throttling burst max in QMP and the command line.

2013-08-08 Thread Benoît Canet
The max parameter of the leaky bycket throttling algoritm can be used to allow the guest to do bursts. The max value is a pool of I/O that the guest can use without being throttled at all. Throttling is triggered once this pool is empty. Signed-off-by: Benoit Canet ben...@irqsave.net ---

[Qemu-devel] [PATCH V4 5/5] block: Add iops_sector_count to do the iops accounting for a given io size.

2013-08-08 Thread Benoît Canet
This feature can be used in case where users are avoiding the iops limit by doing jumbo I/Os hammering the storage backend. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qapi.c |3 +++ blockdev.c | 22 +++--- hmp.c|8 ++--

Re: [Qemu-devel] [PATCH V4 3/5] block: Enable the new throttling code in the block layer.o

2013-08-08 Thread Benoît Canet
This is not really accurate; the cfq scheduler reorders reads and writes to have longer bursts, and these sometimes exceed the rate you set. I understood this is mostly for separate rd/wr settings, or did you reproduce it with total rates too? I'll drop the comment. It's only with

Re: [Qemu-devel] [PATCH V4 3/5] block: Enable the new throttling code in the block layer.

2013-08-08 Thread Benoît Canet
Also, it would be better to have a workaround for this. Perhaps we could simply make the default value of max nonzero? In the old throttling code the slice time is 0.1s, so perhaps we could see what happens with max=0.1*avg. This gives correct results with cfg with some little auto

Re: [Qemu-devel] [PATCH V4 0/5] Continuous Leaky Bucket Throttling*

2013-08-09 Thread Benoît Canet
It fail with the following error message at exit and I don't know why yet. qemu-system-x86_64: block.c:1489: bdrv_drain_all: Assertion `((bs-tracked_requests)-lh_first == ((void *)0))' failed. I solved this issue: bdrv_drain_all was bogus. Best regards Benoît block.c |

Re: [Qemu-devel] [PATCH] qemu-iotests: Filter out actual image size in 067

2013-11-02 Thread Benoît Canet
Le Saturday 02 Nov 2013 à 14:52:11 (+0100), Max Reitz a écrit : The actual size of the image file may differ depending on the Linux kernel currently running on the host. Filtering out this value makes this test pass in such cases. Signed-off-by: Max Reitz mre...@redhat.com ---

Re: [Qemu-devel] [PATCH v2] block: per caller dirty bitmap

2013-11-04 Thread Benoît Canet
Le Monday 04 Nov 2013 à 17:30:10 (+0800), Fam Zheng a écrit : Previously a BlockDriverState has only one dirty bitmap, so only one caller (e.g. a block job) can keep track of writing. This changes the dirty bitmap to a list and creates a BdrvDirtyBitmap for each caller, the lifecycle is

Re: [Qemu-devel] [PATCH] block: Save errno before error_setg_errno

2013-11-05 Thread Benoît Canet
Le Tuesday 05 Nov 2013 à 20:03:33 (+0100), Max Reitz a écrit : error_setg_errno() may overwrite errno; therefore, its value should be read before calling that function and not afterwards. Signed-off-by: Max Reitz mre...@redhat.com --- block.c | 2 +- 1 file changed, 1 insertion(+), 1

Re: [Qemu-devel] [PATCH] block: Round up total_sectors

2013-11-06 Thread Benoît Canet
Le Wednesday 06 Nov 2013 à 19:48:06 (+0800), Fam Zheng a écrit : Since b94a2610, bdrv_getlength() is omitted when probing image. VMDK monolithicFlat is broken by that because a file 512 bytes can't be read with its total_sectors truncated to 0. This patch round up the size to

Re: [Qemu-devel] [PATCH] block: Print its file name if backing file opening failed

2013-11-07 Thread Benoît Canet
Le Thursday 07 Nov 2013 à 15:34:52 (+0800), Fam Zheng a écrit : If backing file doesn't exist, the error message is confusing and misleading: $ qemu /tmp/a.qcow2 qemu: could not open disk image /tmp/a.qcow2: Could not open file: No such file or directory But... $ ls

[Qemu-devel] [PATCH 0/2] Giving names to graph's BlockDriverState

2013-11-07 Thread Benoît Canet
it was not named. After this patchset is merged I would like to take care of presenting the graph to the management. (HMP /|| QMP) Eric: Do you have some ideas on this topic ? Best regards Benoît Benoît Canet (2): block: Add bs-node_name to hold the name of a bs node of the bs graph. block

[Qemu-devel] [PATCH 2/2] block: Allow the user to define node-name option.

2013-11-07 Thread Benoît Canet
As node-name is a separate name space as device-name we can enable it's definition right now: nobody will use it so no harm involved. Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/block.c b/block.c index

[Qemu-devel] [PATCH 1/2] block: Add bs-node_name to hold the name of a bs node of the bs graph.

2013-11-07 Thread Benoît Canet
Add the minimum of code to prepare the followings patches. If no node_name is provided to bdrv_new the bs-node_name is set to undefined. This will allow to have some default string to communicate in QMP and HMP. This also make undefined a reserved string for bs-node_name. Signed-off-by: Benoit

Re: [Qemu-devel] How to introduce bs-node_name ?

2013-11-07 Thread Benoît Canet
Le Monday 04 Nov 2013 à 19:33:21 (+0800), Fam Zheng a écrit : On 11/04/2013 07:06 PM, Kevin Wolf wrote: Am 04.11.2013 um 10:48 hat Fam Zheng geschrieben: On 11/04/2013 05:31 PM, Stefan Hajnoczi wrote: On Wed, Oct 30, 2013 at 02:49:32PM +0100, Markus Armbruster wrote: The first proposal is

Re: [Qemu-devel] [PATCH] qapi-schema: Update description for NewImageMode

2013-11-07 Thread Benoît Canet
Le Thursday 07 Nov 2013 à 19:47:48 (+0100), Max Reitz a écrit : If the NewImageMode is absolute-paths but no backing file is available (e.g., when mirroring a device with an unbacked image), the target image will not be backed either. This patch updates the documentation in qapi-schema.json

Re: [Qemu-devel] [PATCH] util/error: Save errno from clobbering

2013-11-07 Thread Benoît Canet
Le Thursday 07 Nov 2013 à 20:10:29 (+0100), Max Reitz a écrit : There may be calls to error_setg() and especially error_setg_errno() which blindly (and until now wrongly) assume these functions not to clobber errno (e.g., they pass errno to error_setg_errno() and return -errno afterwards).

Re: [Qemu-devel] [PATCH 1/2] block: Add bs-node_name to hold the name of a bs node of the bs graph.

2013-11-07 Thread Benoît Canet
Le Thursday 07 Nov 2013 à 13:23:43 (-0700), Eric Blake a écrit : On 11/07/2013 08:01 AM, Benoît Canet wrote: Add the minimum of code to prepare the followings patches. If no node_name is provided to bdrv_new the bs-node_name is set to undefined. This will allow to have some default

Re: [Qemu-devel] [PATCH 1/2] block: Add bs-node_name to hold the name of a bs node of the bs graph.

2013-11-07 Thread Benoît Canet
Le Thursday 07 Nov 2013 à 15:54:09 (-0500), Jeff Cody a écrit : On Thu, Nov 07, 2013 at 04:01:42PM +0100, Benoît Canet wrote: Add the minimum of code to prepare the followings patches. If no node_name is provided to bdrv_new the bs-node_name is set to undefined. This will allow

[Qemu-devel] [RFC V2 0/3] Giving names to BlockDriverState graph nodes

2013-11-12 Thread Benoît Canet
] s/see/sees/ [Eric] prevent duplicate node_name [Eric] drop undefined and use [Eric, Kevin, Jeff] remove from graph list on bdrv_make_anon [Jeff] comment the two chains [Fam] Add new command stub to retrieve the graph from libvirt [Benoît] Benoît Canet (3): block: Add bs

[Qemu-devel] [RFC V2 2/3] block: Allow the user to define node-name option.

2013-11-12 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c | 20 ++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/block.c b/block.c index c397ee9..6690e3d 100644 --- a/block.c +++ b/block.c @@ -872,6 +872,7 @@ int bdrv_file_open(BlockDriverState **pbs, const

[Qemu-devel] [RFC V2 1/3] block: Add bs-node_name to hold the name of a bs node of the bs graph.

2013-11-12 Thread Benoît Canet
Add the minimum of code to prepare the followings patches. Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c | 72 ++- block/blkverify.c | 2 +- block/iscsi.c | 2 +- block/vmdk.c | 2 +-

[Qemu-devel] [RFC V2 3/3] qapi: Add skeletton of command to query a drive bs graph.

2013-11-12 Thread Benoît Canet
--- blockdev.c | 8 qapi-schema.json | 32 2 files changed, 40 insertions(+) diff --git a/blockdev.c b/blockdev.c index 911ee7e..bfaeda0 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1938,6 +1938,14 @@ void qmp_drive_backup(const char *device,

Re: [Qemu-devel] [PATCH for 1.7] target-i386: do not override nr_cores for -cpu host

2013-11-19 Thread Benoît Canet
Le Tuesday 19 Nov 2013 à 17:49:46 (+0100), Paolo Bonzini a écrit : Commit 787aaf5 (target-i386: forward CPUID cache leaves when -cpu host is used, 2013-09-02) brings bits 31..26 of CPUID leaf 04h out of sync with the APIC IDs that QEMU reserves for each package. This number must come from

Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-07-02 Thread Benoît Canet
+QCOW2 can use one or more instance of a metadata journal. s/instance/instances/ Is there a reason to use multiple journals rather than a single journal for all entry types? The single journal area avoids seeks. Here are the main reason for this: For the deduplication some patterns

Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-07-02 Thread Benoît Canet
2. Byte-granularity means that read-modify-write is necessary to append entries to the journal. Therefore a failure could destroy previously committed entries. Any ideas how existing journals handle this? You commit only whole blocks. So in this case we can consider a block

Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-07-03 Thread Benoît Canet
Care to explain that in more detail? Why shouldn't it work on spinning disks? Hash are random they introduce random read access. With a QCOW2 cluster size of 4KB the deduplication code when writting duplicated data will do one random read per 4KB block to deduplicate. A server grade hardisk

Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-07-03 Thread Benoît Canet
Does this mean the journal forms the first-stage data structure for deduplication? Dedup records will accumulate in the journal until it becomes time to convert them in bulk into a more compact representation? The journal is mainly used to persist the last inserted dedup metadata across QEMU

Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-07-03 Thread Benoît Canet
By the way, I don't know much about journalling techniques. So I'm asking you these questions so that either you can answer them straight away or because they might warrant a look at existing journal implementations like: I tried to so something simple and performing for the deduplication

Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-07-04 Thread Benoît Canet
Simple is good. Even for deduplication alone, I think data integrity is critical - otherwise we risk stale dedup metadata pointing to clusters that are unallocated or do not contain the right data. So the journal will probably need to follow techniques for commits/checksums. I agree that

[Qemu-devel] Alternative mac address setting

2013-07-16 Thread Benoît Canet
Hello, I want to implement an alternative mac address setting to allow a guest using virtio-net to change it's mac address by itself. The main use case is high availability setups where a slave machine take the lead when the master is failing. (heartbeat) The thing that an alternate mac

Re: [Qemu-devel] Alternative mac address setting

2013-07-16 Thread Benoît Canet
Sorry for the gibberish I misread the code. Please ignore this mail. Best regards Benoît Le Tuesday 16 Jul 2013 à 13:36:49 (+0200), Benoît Canet a écrit : Hello, I want to implement an alternative mac address setting to allow a guest using virtio-net to change it's mac address by itself

Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-07-16 Thread Benoît Canet
Simple is good. Even for deduplication alone, I think data integrity is critical - otherwise we risk stale dedup metadata pointing to clusters that are unallocated or do not contain the right data. So the journal will probably need to follow techniques for commits/checksums. I'll add

[Qemu-devel] [PATCH 0/4] Leaky bucket throttling and features

2013-07-22 Thread Benoît Canet
to define the max size of an io when throttling by iops via iops_sector_count. Benoît Canet (4): block: Repair the throttling code. block: Modify the throttling code to implement the leaky bucket algorithm. block: Add support for throttling burst threshold in QMP and the command line

[Qemu-devel] [PATCH 1/4] block: Repair the throttling code.

2013-07-22 Thread Benoît Canet
The throttling code was segfaulting since commit 02ffb504485f0920cfc75a0982a602f824a9a4f4 because some qemu_co_queue_next caller does not run in a coroutine. qemu_co_queue_do_restart assume that the caller is a coroutinne. As sugested by Stefan fix this by entering the coroutine directly.

[Qemu-devel] [PATCH 2/4] block: Modify the throttling code to implement the leaky bucket algorithm.

2013-07-22 Thread Benoît Canet
This patch replace the previous algorithm by the well described leaky bucket algorithm: A bucket is filled by the incoming IOs and a periodic timer decrement the counter to make the bucket leak. When a given threshold is reached the bucket is full and the IOs are hold. In this patch the threshold

[Qemu-devel] [PATCH 3/4] block: Add support for throttling burst threshold in QMP and the command line.

2013-07-22 Thread Benoît Canet
The thresholds of the leaky bucket algorithm can be used to allow some burstiness. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qapi.c | 24 + blockdev.c | 105 +++--- hmp.c| 32 +++--

[Qemu-devel] [PATCH 4/4] block: Add iops_sector_count to do the iops accounting for a given io size.

2013-07-22 Thread Benoît Canet
This feature can be used in case where users are avoiding the iops limit by doing jumbo I/Os hammering the storage backend. Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c |8 +++- block/qapi.c |4 blockdev.c| 22

[Qemu-devel] [PATCH for-1.6 4/4] block: Add iops_sector_count to do the iops accounting for a given io size.

2013-07-22 Thread Benoît Canet
This feature can be used in case where users are avoiding the iops limit by doing jumbo I/Os hammering the storage backend. Signed-off-by: Benoit Canet ben...@irqsave.net --- block.c |8 +++- block/qapi.c |4 blockdev.c| 22

[Qemu-devel] [PATCH for-1.6 1/4] block: Repair the throttling code.

2013-07-22 Thread Benoît Canet
The throttling code was segfaulting since commit 02ffb504485f0920cfc75a0982a602f824a9a4f4 because some qemu_co_queue_next caller does not run in a coroutine. qemu_co_queue_do_restart assume that the caller is a coroutinne. As sugested by Stefan fix this by entering the coroutine directly.

[Qemu-devel] [PATCH for-1.6 3/4] block: Add support for throttling burst threshold in QMP and the command line.

2013-07-22 Thread Benoît Canet
The thresholds of the leaky bucket algorithm can be used to allow some burstiness. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qapi.c | 24 + blockdev.c | 105 +++--- hmp.c| 32 +++--

[Qemu-devel] [PATCH for-1.6 0/4] Leaky bucket throttling and features

2013-07-22 Thread Benoît Canet
to define the max size of an io when throttling by iops via iops_sector_count to avoid vm users cheating on the iops limit. Benoît Canet (4): block: Repair the throttling code. block: Modify the throttling code to implement the leaky bucket algorithm. block: Add support for throttling burst

[Qemu-devel] [PATCH for-1.6 2/4] block: Modify the throttling code to implement the leaky bucket algorithm.

2013-07-22 Thread Benoît Canet
This patch replace the previous algorithm by the well described leaky bucket algorithm: A bucket is filled by the incoming IOs and a periodic timer decrement the counter to make the bucket leak. When a given threshold is reached the bucket is full and the IOs are hold. In this patch the threshold

[Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

2013-06-20 Thread Benoît Canet
--- docs/specs/qcow2.txt | 42 ++ 1 file changed, 42 insertions(+) diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt index 36a559d..a4ffc85 100644 --- a/docs/specs/qcow2.txt +++ b/docs/specs/qcow2.txt @@ -350,3 +350,45 @@ Snapshot table entry:

[Qemu-devel] [RFC V8 00/24] QCOW2 deduplication core functionality

2013-06-20 Thread Benoît Canet
structure are comming. Best regards Benoît Benoît Canet (24): qcow2: Add journal specification. qcow2: Add deduplication structures and fields. qcow2: Add journal. qcow2: Create the log store. qcow2: Add the hash store. qcow2: Add the deduplication store. qcow2: Add

[Qemu-devel] [RFC V8 04/24] qcow2: Create the log store.

2013-06-20 Thread Benoît Canet
) Nodalink, SARL. 2013 + * + * Author: + * Benoît Canet benoit.ca...@irqsave.net + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the Software), to deal + * in the Software without restriction, including without

[Qemu-devel] [RFC V8 07/24] qcow2: Add qcow2_dedup_read_missing_and_concatenate

2013-06-20 Thread Benoît Canet
/qcow2-dedup.c b/block/qcow2-dedup.c new file mode 100644 index 000..bc6e2c2 --- /dev/null +++ b/block/qcow2-dedup.c @@ -0,0 +1,121 @@ +/* + * Deduplication for the QCOW2 format + * + * Copyright (C) Nodalink, SARL. 2012-2013 + * + * Author: + * Benoît Canet benoit.ca...@irqsave.net

[Qemu-devel] [RFC V8 09/24] qcow2: Make qcow2_update_cluster_refcount public.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-refcount.c | 17 ++--- block/qcow2.h |3 +++ 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index b32738f..3bd8f37 100644 ---

[Qemu-devel] [RFC V8 03/24] qcow2: Add journal.

2013-06-20 Thread Benoît Canet
+1,587 @@ +/* + * QCOW2 journal + * + * Copyright (C) Nodalink, SARL. 2013 + * + * Author: + * Benoît Canet benoit.ca...@irqsave.net + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the Software), to deal

[Qemu-devel] [RFC V8 08/24] qcow2: Create a way to link to l2 tables when deduplicating.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-cluster.c |6 -- block/qcow2.h |6 ++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index c71470a..d6db0b9 100644 --- a/block/qcow2-cluster.c +++

[Qemu-devel] [RFC V8 05/24] qcow2: Add the hash store.

2013-06-20 Thread Benoît Canet
--git a/block/qcow2-hash-store.c b/block/qcow2-hash-store.c new file mode 100644 index 000..5284740 --- /dev/null +++ b/block/qcow2-hash-store.c @@ -0,0 +1,802 @@ +/* + * QCOW2 hash store + * + * Copyright (C) Nodalink, SARL. 2013 + * + * Author: + * Benoît Canet benoit.ca...@irqsave.net

[Qemu-devel] [RFC V8 11/24] qcow2: Add qcow2_dedup_store_new_hashes.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-dedup.c | 97 ++- block/qcow2.h |6 +++- 2 files changed, 101 insertions(+), 2 deletions(-) diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c index 0daf77e..ffbf866

[Qemu-devel] [RFC V8 06/24] qcow2: Add the deduplication store.

2013-06-20 Thread Benoît Canet
..8e3fad5 --- /dev/null +++ b/block/qcow2-store.c @@ -0,0 +1,771 @@ +/* + * QCOW2 key value store for SSD storage + * + * Copyright (C) Nodalink, SARL. 2013 + * + * Author: + * Benoît Canet benoit.ca...@irqsave.net + * + * Permission is hereby granted, free of charge, to any person obtaining a copy

[Qemu-devel] [RFC V8 16/24] block: Add qcow2_dedup format and image creation code.

2013-06-20 Thread Benoît Canet
Also modify qemu-io-test. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2.c| 176 -- include/block/block_int.h|1 + tests/qemu-iotests/common.rc |3 +- 3 files changed, 173 insertions(+), 7 deletions(-) diff

[Qemu-devel] [RFC V8 13/24] qcow2: Implement qcow2_compute_cluster_hash.

2013-06-20 Thread Benoît Canet
Also factorize detection of libgnutls with vnc tls. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-dedup.c | 17 +- configure | 86 +-- 2 files changed, 79 insertions(+), 24 deletions(-) diff --git

[Qemu-devel] [RFC V8 15/24] qcow2: Extract qcow2_set_incompat_feature and qcow2_clear_incompat_feature.

2013-06-20 Thread Benoît Canet
Also change callers. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-cluster.c |2 +- block/qcow2.c | 43 ++- block/qcow2.h |7 --- 3 files changed, 27 insertions(+), 25 deletions(-) diff --git

[Qemu-devel] [RFC V8 21/24] qcow2: Integrate SKEIN hash algorithm in deduplication.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-dedup.c | 15 +++ block/qcow2.c |5 + configure | 35 +++ 3 files changed, 55 insertions(+) diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c index

[Qemu-devel] [RFC V8 23/24] qcow2: Enable the deduplication feature.

2013-06-20 Thread Benoît Canet
--- block/qcow2.c | 17 + 1 file changed, 17 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index f7b94dd..bb7bf74 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -569,6 +569,13 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags) goto

[Qemu-devel] [RFC V8 12/24] qcow2: Do allocate on rewrite on the dedup case.

2013-06-20 Thread Benoît Canet
This patch does allocate on rewrite when deduplication is on. This get rid of the need of removing the old hash of the lookup structure when a cluster get rewritten. The old data is left in place and will be collected/deleted when it's cluster will reach 0. Signed-off-by: Benoit Canet

[Qemu-devel] [RFC V8 10/24] qcow2: Add qcow2_dedup and related functions

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-dedup.c | 415 +++ block/qcow2.h |5 + 2 files changed, 420 insertions(+) diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c index bc6e2c2..0daf77e 100644 ---

[Qemu-devel] [RFC V8 14/24] qcow2: Load and save deduplication table header extension.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2.c | 53 + 1 file changed, 53 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index 34b2a87..3cd1051 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -54,9 +54,19 @@

[Qemu-devel] [RFC V8 17/24] qcow2: Drop hash for a given cluster when dedup makes refcount 2^16/2.

2013-06-20 Thread Benoît Canet
A new physical cluster with the same hash value will be used for further occurrence of this hash. Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-dedup.c | 20 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/block/qcow2-dedup.c

[Qemu-devel] [RFC V8 19/24] qcow2: Integrate deduplication in qcow2_co_writev loop.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2.c | 88 +++-- 1 file changed, 86 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index e1265a2..8eb63f1 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@

[Qemu-devel] [RFC V8 22/24] qcow2: Add qcow2_dedup_init and qcow2_dedup_close.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-dedup.c | 60 +++ block/qcow2.c |2 +- block/qcow2.h |2 ++ 3 files changed, 63 insertions(+), 1 deletion(-) diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c

[Qemu-devel] [RFC V8 24/24] qcow2: Enable deduplication tests

2013-06-20 Thread Benoît Canet
--- tests/qemu-iotests/common |6 ++ 1 file changed, 6 insertions(+) diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common index 6826ea7..2483742 100644 --- a/tests/qemu-iotests/common +++ b/tests/qemu-iotests/common @@ -130,6 +130,7 @@ check options -cow

[Qemu-devel] [RFC V8 02/24] qcow2: Add deduplication structures and fields.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2.h | 203 - 1 file changed, 201 insertions(+), 2 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index 9421843..953edfe 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@

[Qemu-devel] [RFC V8 18/24] qcow2: Remove hash when cluster is deleted.

2013-06-20 Thread Benoît Canet
Signed-off-by: Benoit Canet ben...@irqsave.net --- block/qcow2-dedup.c| 45 + block/qcow2-refcount.c |3 +++ block/qcow2.h |2 ++ 3 files changed, 50 insertions(+) diff --git a/block/qcow2-dedup.c b/block/qcow2-dedup.c index

<    1   2   3   4   5   6   7   8   9   10   >