Re: [Qemu-block] [PATCH v5 01/10] qapi: Add transaction support to block-dirty-bitmap operations

2015-06-05 Thread Stefan Hajnoczi
On Thu, Jun 04, 2015 at 05:46:03PM -0400, John Snow wrote: This adds two qmp commands to transactions. block-dirty-bitmap-add allows you to create a bitmap simultaneously alongside a new full backup to accomplish a clean synchronization point. block-dirty-bitmap-clear allows you to reset

Re: [Qemu-block] [PATCH v3 02/38] blockdev: Allow creation of BDS trees without BB

2015-06-05 Thread Alberto Garcia
On Wed 03 Jun 2015 09:43:43 PM CEST, Max Reitz wrote: If the id field is missing from the options given to blockdev-add, just omit the BlockBackend and create the BlockDriverState tree alone. However, if id is missing, node-name must be specified; otherwise, the BDS tree would no longer be

Re: [Qemu-block] [PATCH v5 06/10] block: add delayed bitmap successor cleanup

2015-06-05 Thread Stefan Hajnoczi
On Thu, Jun 04, 2015 at 05:46:08PM -0400, John Snow wrote: @@ -3190,19 +3193,13 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, * we may wish to re-join the parent and child/successor. * The merged parent will be un-frozen, but not explicitly re-enabled. */

Re: [Qemu-block] [PATCH v3 14/38] block: Move BlockAcctStats into BlockBackend

2015-06-05 Thread Eric Blake
On 06/05/2015 04:47 AM, Alberto Garcia wrote: On Wed 03 Jun 2015 10:52:34 PM CEST, Eric Blake wrote: As the comment above bdrv_get_stats() says, BlockAcctStats is something which belongs to the device instead of each BlockDriverState. This patch therefore moves it into the BlockBackend.

Re: [Qemu-block] [PATCH v5 05/10] block: add transactional callbacks feature

2015-06-05 Thread Stefan Hajnoczi
On Thu, Jun 04, 2015 at 05:46:07PM -0400, John Snow wrote: The goal here is to add a new method to transactions that allows developers to specify a callback that will get invoked only once all jobs spawned by a transaction are completed, allowing developers the chance to perform actions

Re: [Qemu-block] I/O accounting overhaul

2015-06-05 Thread Alberto Garcia
On Wed 03 Jun 2015 04:18:45 PM CEST, Eric Blake wrote: The accounting stats are stored in the BlockDriverState, but they're actually from the device backed by the BDS, so they could probably be moved there. For the interface we could extend BlockDeviceStats and add the new fields, but

Re: [Qemu-block] [PATCH v5 04/10] block: re-add BlkTransactionState

2015-06-05 Thread Max Reitz
On 04.06.2015 23:46, John Snow wrote: Now that the structure formerly known as BlkTransactionState has been renamed to something sensible (BlkActionState), re-introduce an actual BlkTransactionState that actually manages state for the entire Transaction. In the process, convert the old QSIMPLEQ

Re: [Qemu-block] [PATCH v5 06/10] block: add delayed bitmap successor cleanup

2015-06-05 Thread John Snow
On 06/05/2015 08:56 AM, Stefan Hajnoczi wrote: On Thu, Jun 04, 2015 at 05:46:08PM -0400, John Snow wrote: @@ -3190,19 +3193,13 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs, * we may wish to re-join the parent and child/successor. * The merged parent will be un-frozen,

[Qemu-block] [PATCH v6 2/4] qcow2: add option to clean unused cache entries after some time

2015-06-05 Thread Alberto Garcia
This adds a new 'cache-clean-interval' option that cleans all qcow2 cache entries that haven't been used in a certain interval, given in seconds. This allows setting a large L2 cache size so it can handle scenarios with lots of I/O and at the same time use little memory during periods of

[Qemu-block] [PATCH v6 3/4] docs: document how to configure the qcow2 L2/refcount caches

2015-06-05 Thread Alberto Garcia
QEMU has options to configure the size of the L2 and refcount caches for the qcow2 format. However, choosing the right sizes for a particular disk image is not a straightforward operation since the ratio between the cache size and the allocated disk space is not obvious and depends on the size of

Re: [Qemu-block] [PATCH 3/4] docs: document how to configure the qcow2 L2/refcount caches

2015-06-05 Thread Max Reitz
On 02.06.2015 14:22, Alberto Garcia wrote: QEMU has options to configure the size of the L2 and refcount caches for the qcow2 format. However, choosing the right sizes for a particular disk image is not a straightforward operation since the ratio between the cache size and the allocated disk

Re: [Qemu-block] [PATCH 2/2] vmdk: Use vmdk_find_index_in_cluster everywhere

2015-06-05 Thread Max Reitz
On 04.06.2015 08:02, Fam Zheng wrote: Signed-off-by: Fam Zheng f...@redhat.com --- block/vmdk.c | 10 ++ 1 file changed, 2 insertions(+), 8 deletions(-) Reviewed-by: Max Reitz mre...@redhat.com

Re: [Qemu-block] [PATCH v6 3/4] docs: document how to configure the qcow2 L2/refcount caches

2015-06-05 Thread Max Reitz
On 05.06.2015 18:27, Alberto Garcia wrote: QEMU has options to configure the size of the L2 and refcount caches for the qcow2 format. However, choosing the right sizes for a particular disk image is not a straightforward operation since the ratio between the cache size and the allocated disk

Re: [Qemu-block] [PATCH 3/4] docs: document how to configure the qcow2 L2/refcount caches

2015-06-05 Thread Alberto Garcia
On Fri 05 Jun 2015 05:52:29 PM CEST, Max Reitz wrote: +Reducing the memory usage +- +It is possible to clean unused cache entries in order to reduce the +memory usage during periods of low I/O activity. + +The parameter cache-clean-interval defines an interval (in

Re: [Qemu-block] [PATCH 3/4] docs: document how to configure the qcow2 L2/refcount caches

2015-06-05 Thread Max Reitz
On 05.06.2015 17:56, Alberto Garcia wrote: On Fri 05 Jun 2015 05:52:29 PM CEST, Max Reitz wrote: +Reducing the memory usage +- +It is possible to clean unused cache entries in order to reduce the +memory usage during periods of low I/O activity. + +The parameter

Re: [Qemu-block] [PATCH 1/2] vmdk: Fix index_in_cluster calculation in vmdk_co_get_block_status

2015-06-05 Thread Max Reitz
On 04.06.2015 08:02, Fam Zheng wrote: It has the similar issue with b1649fae49a8. Since the calculation is repeated for a few times already, introduce a function so it can be reused. Signed-off-by: Fam Zheng f...@redhat.com --- block/vmdk.c | 13 - 1 file changed, 12

[Qemu-block] [PATCH v6 4/4] qcow2: reorder fields in Qcow2CachedTable to reduce padding

2015-06-05 Thread Alberto Garcia
Changing the current ordering saves 8 bytes per cache entry in x86_64. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com Reviewed-by: Kevin Wolf kw...@redhat.com --- block/qcow2-cache.c | 2 +- 1 file changed, 1

[Qemu-block] [PATCH v6 1/4] qcow2: mark the memory as no longer needed after qcow2_cache_empty()

2015-06-05 Thread Alberto Garcia
After having emptied the cache, the data in the cache tables is no longer useful, so we can tell the kernel that we are done with it. In Linux this frees the resources associated with it. The effect of this can be seen in the HMP commit operation: it moves data from the top to the base image (and

Re: [Qemu-block] [PATCH v3 07/38] block/quorum: Implement bdrv_is_inserted()

2015-06-05 Thread Max Reitz
On 04.06.2015 14:37, Alberto Garcia wrote: On Wed 03 Jun 2015 09:43:48 PM CEST, Max Reitz wrote: bdrv_is_inserted() should be invoked recursively on the children of quorum. Signed-off-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- +static bool

Re: [Qemu-block] [PATCH v3 20/38] block: Add blk_insert_bs()

2015-06-05 Thread Alberto Garcia
On Wed 03 Jun 2015 09:44:01 PM CEST, Max Reitz wrote: This function associates the given BlockDriverState with the given BlockBackend. Signed-off-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com Reviewed-by: Alberto Garcia be...@igalia.com Berto

[Qemu-block] [PATCH v6 09/10] iotests: 124 - transactional failure test

2015-06-05 Thread John Snow
Use a transaction to request an incremental backup across two drives. Coerce one of the jobs to fail, and then re-run the transaction. Verify that no bitmap data was lost due to the partial transaction failure. Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com

[Qemu-block] [PATCH v6 00/10] block: incremental backup transactions

2015-06-05 Thread John Snow
This series adds incremental backup related commands to the QMP transaction subsystem in order to accomplish some slightly more sophisticated bitmap and backup management tasks that require atomic actions. Patch 1 adds basic support for add and clear transactions. Patch 2 tests this basic

[Qemu-block] [PATCH v6 01/10] qapi: Add transaction support to block-dirty-bitmap operations

2015-06-05 Thread John Snow
This adds two qmp commands to transactions. block-dirty-bitmap-add allows you to create a bitmap simultaneously alongside a new full backup to accomplish a clean synchronization point. block-dirty-bitmap-clear allows you to reset a bitmap back to as-if it were new, which can also be used

[Qemu-block] [PATCH v6 04/10] block: re-add BlkTransactionState

2015-06-05 Thread John Snow
Now that the structure formerly known as BlkTransactionState has been renamed to something sensible (BlkActionState), re-introduce an actual BlkTransactionState that actually manages state for the entire Transaction. In the process, convert the old QSIMPLEQ list of actions into a QTAILQ, to let

[Qemu-block] [PATCH v6 02/10] iotests: add transactional incremental backup test

2015-06-05 Thread John Snow
Test simple usage cases for using transactions to create and synchronize incremental backups. Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com --- tests/qemu-iotests/124 | 54

[Qemu-block] [PATCH v6 03/10] block: rename BlkTransactionState and BdrvActionOps

2015-06-05 Thread John Snow
These structures are misnomers, somewhat. (1) BlockTransactionState is not state for a transaction, but is rather state for a single transaction action. Rename it BlkActionState to be more accurate. (2) The BdrvActionOps describes operations for the BlkActionState, above. This name

[Qemu-block] [PATCH v6 06/10] block: add delayed bitmap successor cleanup

2015-06-05 Thread John Snow
Allow bitmap successors to carry reference counts. We can in a later patch use this ability to clean up the dirty bitmap according to both the individual job's success and the success of all jobs in the transaction group. As a consequence of moving the bitmap successor cleanup actions behind the

[Qemu-block] [PATCH v6 08/10] block: drive_backup transaction callback support

2015-06-05 Thread John Snow
This patch actually implements the transactional callback system for the drive_backup action. (1) We create a functional closure that envelops the original drive_backup callback, to be able to intercept the completion status and return code for the job. (2) We inform the BlockJob layer

[Qemu-block] [PATCH v6 05/10] block: add transactional callbacks feature

2015-06-05 Thread John Snow
The goal here is to add a new method to transactions that allows developers to specify a callback that will get invoked only once all jobs spawned by a transaction are completed, allowing developers the chance to perform actions conditionally pending complete success, partial failure, or complete

[Qemu-block] [PATCH v6 10/10] qmp-commands.hx: Update the supported 'transaction' operations

2015-06-05 Thread John Snow
From: Kashyap Chamarthy kcham...@redhat.com Although the canonical source of reference for QMP commands is qapi-schema.json, for consistency's sake, update qmp-commands.hx to state the list of supported transactionable operations, namely: drive-backup blockdev-backup

Re: [Qemu-block] [PATCH v3 26/38] blockdev: Add blockdev-open-tray

2015-06-05 Thread Max Reitz
On 03.06.2015 23:24, Eric Blake wrote: On 06/03/2015 01:44 PM, Max Reitz wrote: Signed-off-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- blockdev.c | 48 qapi/block-core.json | 23

Re: [Qemu-block] [PATCH v3 38/38] iotests: Add test for change-related QMP commands

2015-06-05 Thread Max Reitz
On 03.06.2015 23:46, Eric Blake wrote: On 06/03/2015 01:44 PM, Max Reitz wrote: Signed-off-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/118 | 638 + tests/qemu-iotests/118.out | 5 + tests/qemu-iotests/group | 1 + 3 files

Re: [Qemu-block] [PATCH v3 10/38] hw/usb-storage: Check whether BB is inserted

2015-06-05 Thread Max Reitz
On 03.06.2015 22:37, Eric Blake wrote: On 06/03/2015 01:43 PM, Max Reitz wrote: Only call bdrv_add_key() on the BlockDriverState if it is not NULL. Signed-off-by: Max Reitz mre...@redhat.com --- hw/usb/dev-storage.c | 30 -- 1 file changed, 16 insertions(+), 14