Re: [libvirt] [PATCH 00/21] Support NBD for tunnelled migration

Ping. May I have your attention guys? Pavel On Wed, Nov 18, 2015 at 8:12 PM, Pavel Boldin <pbol...@mirantis.com> wrote: > The provided patchset implements NBD disk migration over a tunnelled > connection provided by libvirt. > > The migration source instructs QEMU to

[libvirt] [PATCH 08/21] qemu: migration: src: add nbd tunnel socket data

Add local NBD tunnel socket info to the qemuMigrationSpec structure. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 5 + 1 file changed, 5 insertions(+) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 3eee3a5..fb2a216

[libvirt] [PATCH 05/21] domain: add virDomainMigrateOpenTunnel

Add auxiliary private function that calls the apropriate driver's domainMigrateOpenTunnel function. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- docs/apibuild.py | 1 + docs/hvsupport.pl| 1 + src/libvirt-domain.c

[libvirt] [PATCH 13/21] qemu: migration: src: qemuNBDTunnelAcceptAndPipe

the source QEMU's NBD. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 134 +- 1 file changed, 132 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 0682fd8..0

[libvirt] [PATCH 06/21] domain: add virDomainMigrateTunnelFlags

Add virDomainMigrateTunnelFlags enum. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- include/libvirt/libvirt-domain.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index a1ea6a5..444deee 100644 --- a/i

[libvirt] [PATCH 10/21] qemu: migration: src: qemu `drive-mirror` to UNIX

Make qemuMigrationDriveMirror able to instruct QEMU to connect to a local UNIX socket used for tunnelling. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 45 ++--- 1 file changed, 30 insertions(+), 15 deletions(-)

[libvirt] [PATCH 12/21] qemu: migration: src: add NBD unixSock to iothread

Pass UNIX socket used as a local NBD server destination to the migration iothread. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 46 ++ 1 file changed, 38 insertions(+), 8 deletions(-) diff --git a/sr

[libvirt] [PATCH 03/21] remote_driver: introduce virRemoteClientNew

Move common code to a function. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/remote/remote_driver.c | 48 +++--- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c

[libvirt] [PATCH 19/21] qemu: migration: dest: qemuMigrationOpenNBDTunnel

Add qemuMigrationOpenNBDTunnel that connects a remote stream to the local NBD UNIX socket. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 29 + 1 file changed, 29 insertions(+) diff --git a/src/qemu/qemu_migration.c b/sr

[libvirt] [PATCH 21/21] apparmor: fix tunnelmigrate permissions

Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/security/virt-aa-helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/security/virt-aa-helper.c b/src/security/virt-aa-helper.c index 5de56e5..87af98f 100644 --- a/src/security/virt-aa-helper.c +++

[libvirt] [PATCH 11/21] qemu: migration: src: qemuSock for running thread

it later. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 93 ++- 1 file changed, 59 insertions(+), 34 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index d95cd66..61e78c5

[libvirt] [PATCH 09/21] qemu: migration: src: nbdtunnel unix socket

Create a UNIX socket that will be a target for outgoing NBD connection from the QEMU side. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 24 +++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.c

[libvirt] [PATCH 00/21] Support NBD for tunnelled migration

that does the real job. * Finally, the code blocking NBD migration for tunnelled migration is removed. Pavel Boldin (21): rpc: add DOMAIN_MIGRATE_OPEN_TUNNEL proc driver: add virDrvDomainMigrateOpenTunnel remote_driver: introduce virRemoteClientNew remote_driver: add

[libvirt] [PATCH 20/21] qemu: migration: allow NBD tunneling migration

Now that all the pieces are in their places finally allow NBD in tunnelled migration. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 16 ++-- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/src/qemu/qemu_migration.c b/sr

[libvirt] [PATCH 02/21] driver: add virDrvDomainMigrateOpenTunnel

Add virDrvDomainMigrateOpenTunnel call that is to be implemented by the drivers in order to provide a way to open tunnel during migration. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/driver-hypervisor.h | 8 1 file changed, 8 insertions(+) diff --git a/src/

[libvirt] [PATCH 01/21] rpc: add DOMAIN_MIGRATE_OPEN_TUNNEL proc

Add REMOTE_PROC_DOMAIN_MIGRATE_OPEN_TUNNEL remote call, args and rets. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- daemon/remote.c | 12 src/remote/remote_protocol.x | 19 ++- src/remote_protocol-structs | 8 3 files chang

[libvirt] [PATCH 17/21] qemu: migration: dest: qemuMigrationOpenTunnel

Introduce an auxiliary handler domainMigrateOpenTunnel for QEMU. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 19 +++ src/qemu/qemu_migration.h | 6 ++ 2 files changed, 25 insertions(+) diff --git a/src/qemu/qemu_migration.c b/sr

[libvirt] [PATCH 07/21] remote: impl remoteDispatchDomainMigrateOpenTunnel

Implement remoteDispatchDomainMigrateOpenTunnel. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- daemon/remote.c | 50 -- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/daemon/remote.c b/daemon/remote.c index 237124d..3

[libvirt] [PATCH 14/21] qemu: migration: src: stream piping

Add and use qemuMigrationPipeEvent piped streams' event handler. It sets the appropriate event flags for each of the stream and pumps the pipe using qemuMigrationPipeIO whenever there is a data at any end. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.

[libvirt] [PATCH 04/21] remote_driver: add remoteDomainMigrateOpenTunnel

Add remoteDomainMigrateOpenTunnel that ties passed stream to the network stream and then makes the appropriate remote call. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/remote/remote_driver.c | 43 +++ 1 file changed, 43 insertions(+)

[libvirt] [PATCH 15/21] qemu: monitor: add qemuMonitorNBDServerStartUnix

Add qemuMonitorNBDServerStartUnix used to instruct QEMU to connect to a UNIX socket as a NBD drive mirror destination. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_monitor.c | 12 src/qemu/qemu_monitor.h | 2 ++ src/qemu/qemu_monitor_json.

[libvirt] [PATCH 18/21] qemu: driver: add qemuDomainMigrateOpenTunnel

Add domainMigrateOpenTunnel handler for QEMU driver. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_driver.c | 24 1 file changed, 24 insertions(+) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 92a9961..ad9a6a0 100644 ---

[libvirt] [PATCH 16/21] qemu: migration: dest: nbd-server to UNIX sock

Modify qemuMigrationStartNBDServer so it can instruct QEMU to start NBD server binded to a local UNIX socket. Signed-off-by: Pavel Boldin <pbol...@mirantis.com> --- src/qemu/qemu_migration.c | 18 +++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/sr

Re: [libvirt] [RFC] Design: Tunnelled NBD block disk migration

Michal, Please see my comments intermixed below. 1. `Pipe`s used to interconnect to the QEMU on the both sides are obviously to be replaced by the UNIX sockets since the pipes cannot support bidirectional output due to the design. This is to be made *one for each* block device,

[libvirt] [RFC] Design: Tunnelled NBD block disk migration

Dear All, I continue my work on the tunnelled selective block disks migration and the next step is to implement Tunnelled NBD block disk migration. As far as I see libvirt uses the following algorithm for tunnelling the QEMU migration stream that is unidirectional: 1. The

[libvirt] [PATCH libvirt-python] virPyDictToTypedParams: packing lists of values

Pack a list or a tuple of values passed to a Python method to the multi-value parameter. --- libvirt-override.c | 228 ++--- 1 file changed, 129 insertions(+), 99 deletions(-) diff --git a/libvirt-override.c b/libvirt-override.c index

[libvirt] [PATCH v4 1/9] virDomainDiskGetSource: Mark passed disk as 'const'

From: Michal Privoznik mpriv...@redhat.com The disk is not changed anywhere in the function. Mark this fact in the function header too. Signed-off-by: Michal Privoznik mpriv...@redhat.com --- src/conf/domain_conf.c | 2 +- src/conf/domain_conf.h | 2 +- 2 files changed, 2 insertions(+), 2

[libvirt] [PATCH v4 5/9] util: multi-value parameters in virTypedParamsAdd*

Allow multi-value parameters to be build using virTypedParamsAdd* functions by removing check for duplicates. Signed-off-by: Pavel Boldin pbol...@mirantis.com Signed-off-by: Michal Privoznik mpriv...@redhat.com --- src/util/virtypedparam.c | 16 1 file changed, 16 deletions

[libvirt] [PATCH v4 0/9] Selective block device migration implementation

header indentation qemuMigrationDriveMirror: Force raw format for NBD Pavel Boldin (6): util: multi-value virTypedParameter util: multi-value parameters in virTypedParamsAdd* util: virTypedParams{Filter,GetAllStrings} util: add virTypedParamsAddStringList qemu: migration: selective block

[libvirt] [PATCH v4 8/9] qemu: migration: selective block device migration

Implement a `migrate_disks' parameters for the QEMU driver. This multi- value parameter can be used to explicitly specify what block devices are to be migrated using the NBD server. Tunnelled migration using NBD is to be done. Signed-off-by: Pavel Boldin pbol...@mirantis.com Signed-off-by: Michal

[libvirt] [PATCH v4 6/9] util: virTypedParams{Filter, GetAllStrings}

Add multikey API: * virTypedParamsFilter that filters all the parameters with specified name. * virTypedParamsGetAllStrings that returns a list with all the values for specified name and string type. Signed-off-by: Pavel Boldin pbol...@mirantis.com Signed-off-by: Michal Privoznik mpriv

[libvirt] [PATCH v4 2/9] qemuMigrationBeginPhase: Fix function header indentation

From: Michal Privoznik mpriv...@redhat.com This function is returning a string (domain XML). Since d3ce7363 when it was first introduced, it was indented incorrectly: static char *qemuMigrationBeginPhase(..) Signed-off-by: Michal Privoznik mpriv...@redhat.com --- src/qemu/qemu_migration.c | 16

[libvirt] [PATCH v4 4/9] util: multi-value virTypedParameter

The `virTypedParamsValidate' function now can be instructed to allow multiple entries for some of the keys. For this flag the type with the `VIR_TYPED_PARAM_MULTIPLE' flag. Add unit tests for this new behaviour. Signed-off-by: Pavel Boldin pbol...@mirantis.com Signed-off-by: Michal Privoznik

[libvirt] [PATCH v4 7/9] util: add virTypedParamsAddStringList

The `virTypedParamsAddStringList' function provides interface to add a NULL-terminated array of string values as a multi-value to the params. Signed-off-by: Pavel Boldin pbol...@mirantis.com Signed-off-by: Michal Privoznik mpriv...@redhat.com --- include/libvirt/libvirt-host.h | 6 ++ src

[libvirt] [PATCH v4 3/9] qemuMigrationDriveMirror: Force raw format for NBD

Signed-off-by: Michal Privoznik mpriv...@redhat.com Signed-off-by: Pavel Boldin pbol...@mirantis.com --- src/qemu/qemu_migration.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 6c94052..1fa5e5f 100644 --- a/src/qemu

[libvirt] [PATCH v4 9/9] virsh: selective block device migration

Add `virsh migrate' option `--migratedisks' that allows CLI user to explicitly specify block devices to migrate. Signed-off-by: Pavel Boldin pbol...@mirantis.com Signed-off-by: Michal Privoznik mpriv...@redhat.com --- tools/virsh-domain.c | 23 +++ tools/virsh.pod | 21

Re: [libvirt] [PATCH v3 9/9] virsh: selective block device migration

Michal, Should I fix these or will you do it? Pavel On Fri, May 29, 2015 at 6:37 PM, John Ferlan jfer...@redhat.com wrote: On 05/26/2015 09:01 AM, Michal Privoznik wrote: From: Pavel Boldin pbol...@mirantis.com Add `virsh migrate' option `--migratedisks' that allows CLI user

Re: [libvirt] [PATCHv2 0/6] Selective block device migration implementation

Thank you. I will go through your comments just in case I'm going to push any other changes. Sorry for the inconvenience from my patches :-/. Pavel On Mon, May 25, 2015 at 5:58 PM, Michal Privoznik mpriv...@redhat.com wrote: On 21.05.2015 13:07, Pavel Boldin wrote: The patchset represented

[libvirt] [PATCH v2 3/6] util: virTypedParams{Filter,PickStrings}

Add multikey API: * virTypedParamsFilter that returns all the parameters with specified name. * virTypedParamsPickStrings that returns a NULL-terminated `const char**' list with all the values for specified name and string type. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- include

[libvirt] [PATCH v2 6/6] virsh: selective block device migration

Add `virsh migrate' option `--migratedisks' that allows CLI user to explicitly specify block devices to migrate. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- tools/virsh-domain.c | 23 +++ tools/virsh.pod | 5 - 2 files changed, 27 insertions(+), 1 deletion

[libvirt] [PATCH v2 1/6] util: multi-value virTypedParameter

The `virTypedParamsValidate' function now can be instructed to allow multiple entries for some of the keys. For this flag the type with the `VIR_TYPED_PARAM_MULTIPLE' flag. Add unit tests for this new behaviour. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- src/util/virtypedparam.c

[libvirt] [PATCH v2 5/6] qemu: migration: selective block device migration

Implement a `migrate_disks' parameters for the QEMU driver. This multi- value parameter can be used to explicitly specify what block devices are to be migrated using the NBD server. Tunnelled migration using NBD is to be done. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- include/libvirt

[libvirt] [PATCH v2 4/6] util: add virTypedParamsAddStringList

The `virTypedParamsAddStringList' function provides interface to add a NULL-terminated array of string values as a multi-value to the params. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- include/libvirt/libvirt-host.h | 6 ++ src/libvirt_public.syms| 1 + src/util

[libvirt] [PATCHv2 0/6] Selective block device migration implementation

*' * reimplemented virTypedParamsPick as virTypedParamsFilter * renamed virTypedParamsPackStrings to virTypedParamsAddStringList [1] https://bugzilla.redhat.com/show_bug.cgi?id=1203032 Pavel Boldin (6): util: multi-value virTypedParameter util: multi-value parameters in virTypedParamsAdd* util

[libvirt] [PATCH v2 2/6] util: multi-value parameters in virTypedParamsAdd*

Allow multi-value parameters to be build using virTypedParamsAdd* functions by removing check for duplicates. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- src/util/virtypedparam.c | 16 1 file changed, 16 deletions(-) diff --git a/src/util/virtypedparam.c b/src/util

[libvirt] [PATCH 2/5] util: virTypedParamsPick* multikey API

Add multikey APIs for virTypedParams*: * virTypedParamsPick that returns all the parameters with the specified name and type. * virTypedParamsPickStrings that returns a NULL-terminated `const char**' list with all the values for specified name and string type. Signed-off-by: Pavel Boldin

[libvirt] [PATCH 5/5] virsh: selective block device migration

Add `virsh migrate' option `--migratedisks' that allows CLI user to explicitly specify block devices to migrate. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- tools/virsh-domain.c | 43 +++ 1 file changed, 43 insertions(+) diff --git a/tools/virsh

[libvirt] [PATCH 1/5] util: multi-value virTypedParameter

The `virTypedParamsValidate' function now can be instructed to allow multiple entries for some of the keys. For this flag the type with the `VIR_TYPED_PARAM_MULTIPLE' flag. Add unit tests for this new behaviour. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- include/libvirt/libvirt-host.h

[libvirt] [PATCH 3/5] util: add virTypedParamsPackStrings

The `virTypedParamsPackStrings' function provides interface to pack multiple string values under the same key to the `virTypedParameter' array. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- include/libvirt/libvirt-host.h | 6 +++ src/libvirt_public.syms| 1 + src/util

[libvirt] [PATCH 0/5] Selective block device migration implementation

to specify a comma-separated list of the block device names that are to be migrated. The implemented Python bindings patch is to be presented. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1203032 Pavel Boldin (5): util: multi-value virTypedParameter util: virTypedParamsPick* multikey API util

[libvirt] [PATCH 4/5] qemu: migration: selective block device migration

Implement a `migrate_disks' parameters for the QEMU driver. This multi- value parameter can be used to explicitly specify what block devices are to be migrated using the NBD server. Tunnelled migration using NBD is to be done. Signed-off-by: Pavel Boldin pbol...@mirantis.com --- include/libvirt

Re: [libvirt] [BUG?] EAGAIN not triggering error and 'events' gets cleared

. This deadlocks the libvirt. Pavel On Tue, Apr 28, 2015 at 6:29 AM, Pavel Boldin pbol...@mirantis.com wrote: Dear Libvirt Developers, There seems to be a bug or at least a bad behavior in `src/qemu/qemu_monitor.c' lines 683-689 function `qemuMonitorIO': if (qemuMonitorIOWrite(mon

Re: [libvirt] [BUG?] EAGAIN not triggering error and 'events' gets cleared

28, 2015 at 11:24:37 +0200, Michal Privoznik wrote: On 28.04.2015 11:06, Pavel Boldin wrote: Well, actually that seems to be quite a different bug in there. I will start a new thread. In short: migration seems to be broken by commit 1a92c719101e5bfa6fe2b78006ad04c7f075ea28

[libvirt] [BUG?] EAGAIN not triggering error and 'events' gets cleared

Dear Libvirt Developers, There seems to be a bug or at least a bad behavior in `src/qemu/qemu_monitor.c' lines 683-689 function `qemuMonitorIO': if (qemuMonitorIOWrite(mon) 0) { error = true; if (errno == ECONNRESET) hangup = true;

Re: [libvirt] Selective block device migration implementation

That's why I've chosen to work purely with disk target at NBD level. We have strong rules what characters can occur there. Moreover, it's fairly easy to derive qemu disk ID from the target. Oh, and we require targets to be unique throughout the domain. So I think it's the best option for the

Re: [libvirt] Selective block device migration implementation

Michal, On Wed, Apr 15, 2015 at 10:54 AM, Michal Privoznik mpriv...@redhat.com wrote: On 26.03.2015 15:48, Pavel Boldin wrote: Dear Libvirt Developers, I'm working to implement feature request [1]. The feature request proposes to enhance `libvirt' code so the API caller can specify

Re: [libvirt] Selective block device migration implementation

On Wed, Apr 15, 2015 at 6:43 PM, Michal Privoznik mpriv...@redhat.com wrote: On 15.04.2015 16:38, Pavel Boldin wrote: Michal, On Wed, Apr 15, 2015 at 10:54 AM, Michal Privoznik mpriv...@redhat.com wrote: On 26.03.2015 15:48, Pavel Boldin wrote: Dear Libvirt Developers, I'm

[libvirt] Selective block device migration implementation

Dear Libvirt Developers, I'm working to implement feature request [1]. The feature request proposes to enhance `libvirt' code so the API caller can specify which block devices are to be migrated using e.g. parameters in the `virDomainMigrateToURI3' call. There is the following issues: 1. It