[pve-devel] [PATCH v2 qemu-server 1/4] add qemu_drive_mirror_monitor completion modes

2020-03-17 Thread Mira Limbeck
With Qemu 4.2 we encountered a problem with unix sockets and SSH socket forwarding for drive-mirror. It seems the socket gets reopened again and again after it closes for some reason. This can be worked around by specifying 'block-job-cancel' instead of 'block-job-complete' when we're not

[pve-devel] [PATCH v2 qemu-server 2/4] add NBD server unix socket support in vm_start

2020-03-17 Thread Mira Limbeck
As the NBD server spawned by qemu can only listen on a single socket, we're dependent on a version being passed to vm_start that indicates which protocol can be used, TCP or Unix, by the source node. The change in socket type (TCP to Unix) comes with a different URI. For unix sockets it has the

[pve-devel] [PATCH v2 qemu-server 4/4] add unix socket support for NBD storage migration

2020-03-17 Thread Mira Limbeck
The reuse of the tunnel, which we're opening to communicate with the target node and to forward the unix socket for the state migration, for the NBD unix socket requires adding support for an array of sockets to forward, not just a single one. We also have to change the $sock_addr variable to an

[pve-devel] [PATCH v2 qemu-server 3/4] parse nbd_protocol_version if available

2020-03-17 Thread Mira Limbeck
For secure live migration with local disks via NBD over a unix socket, we have to somehow communicate from the source node to the target node if it supports it. This is because there can only be one NBD server with exactly one socket bound. The source node passes that information via STDIN.

[pve-devel] [PATCH v2 qemu-server 0/4] fix secure live migration with local

2020-03-17 Thread Mira Limbeck
Currently NBD storage migration always uses unencrypted TCP. The following 4 patches add support for unix sockets that are forwarded over SSH. For backwards compatibility this requires some kind of communication from the source node to the target node, because the NBD server can only be started

Re: [pve-devel] [PATCH v3 qemu-server] vzdump: move include logic for volumes to method

2020-03-17 Thread Fabian Grünbichler
On March 16, 2020 4:44 pm, Aaron Lauterer wrote: > Move the logic which volumes are included in the backup job to its own > method and adapt the VZDump code accordingly. This makes it possible to > develop other features around backup jobs. > > Signed-off-by: Aaron Lauterer > --- > > v2 -> v3:

Re: [pve-devel] [PATCH v3 manager 1/1] backup: move logic to include guests into method

2020-03-17 Thread Fabian Grünbichler
On March 16, 2020 4:44 pm, Aaron Lauterer wrote: > This extracts the logic which guests are to be included in a backup job > into its own method 'get_included_guests'. This makes it possible to > develop other features around backup jobs. > > Logic which was spread out accross the API2/VZDump.pm

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 2:40 PM, Thomas Lamprecht wrote: On 3/17/20 2:33 PM, Wolfgang Bumiller wrote: On 3/17/20 2:24 PM, Thomas Lamprecht wrote: On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, 

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 2:33 PM, Wolfgang Bumiller wrote: > On 3/17/20 2:24 PM, Thomas Lamprecht wrote: >> On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: >>> On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: > On 3/17/20 7:35 AM, Thomas Lamprecht wrote: >>

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 2:24 PM, Thomas Lamprecht wrote: On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 2:10 PM, Wolfgang Bumiller wrote: > On 3/17/20 12:31 PM, Thomas Lamprecht wrote: >> On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: >>> On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, 

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 12:31 PM, Thomas Lamprecht wrote: On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, even if it means doing stupid unnecessary things, as else 

[pve-devel] [PATCH V3] Revision of the pvesr documentation

2020-03-17 Thread Wolfgang Link
Improvement of grammar and punctuation. Clarify the HA limitations. Remove future tense in some sentences. It is not good to use it in technical/scientific papers. Rewrite some sentences to improve understanding. --- pvesr.adoc | 112 ++--- 1 file

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Fabian Grünbichler
On March 17, 2020 12:40 pm, Thomas Lamprecht wrote: > On 3/17/20 11:21 AM, Stefan Reiter wrote: >>> +$local_volumes->{$opt} = $conf->{${opt}}; >> >> Does $conf->{${opt}} have too many brackets or is this another arcane perl  >> syntax I've yet to discover? (iow. why not just 

[pve-devel] [PATCH qemu-server] Add unused description to drivedesc_hash

2020-03-17 Thread Fabian Ebner
Moved code so that initialization of drivedesc_hash stays a single block. Avoid auto-vivication in parse_drive. Signed-off-by: Fabian Ebner --- Follow-up suggested by Fabian G. I checked the call-sites for is_valid_drivename and in most cases it won't benefit from including unused drives,

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Stefan Reiter
On 17/03/2020 12:40, Thomas Lamprecht wrote: On 3/17/20 11:21 AM, Stefan Reiter wrote: +$local_volumes->{$opt} = $conf->{${opt}}; Does $conf->{${opt}} have too many brackets or is this another arcane perl syntax  I've yet to discover? (iow. why not just $conf->{$opt} ?) It's not

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 11:21 AM, Stefan Reiter wrote: >> +$local_volumes->{$opt} = $conf->{${opt}}; > > Does $conf->{${opt}} have too many brackets or is this another arcane perl  > syntax I've yet to discover? (iow. why not just $conf->{$opt} ?) It's not that arcane, you surely used it

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: > On 3/17/20 7:35 AM, Thomas Lamprecht wrote: >> CONTAINER_INTERFACE[0] is omething systemd people call their API and >> we need to adapt to it a bit, even if it means doing stupid >> unnecessary things, as else systemd decides to regress and suddenly

Re: [pve-devel] [PATCH container 1/2] update_lxc_config: set in-CT network devices to up

2020-03-17 Thread Thomas Lamprecht
On 3/17/20 11:48 AM, Wolfgang Bumiller wrote: > On 3/17/20 7:35 AM, Thomas Lamprecht wrote: >> Else some newer system do not see the interface as up and refuse to >> manage it.. > > Where do you run into that actually? I can get my arch container to behave > normally with just adding

[pve-devel] [PATCH v2 0/2] Support all 8 corosync3 links in GUI

2020-03-17 Thread Stefan Reiter
Since it's been a while, here's a rebased version the two still missing patches of my previous RFC[0]. Except for a slight rewording of the message in patch 2 it's exactly the same, so consider this a RESEND more than anything. I think the series might have gotten lost a bit after parts where

[pve-devel] [PATCH v2 manager 2/2] gui/cluster: add structured peerLinks to join info

2020-03-17 Thread Stefan Reiter
Instead of the old 'ring_addr' property (which is kept for compatibility), we also encode the link numbers into the new peerLinks structure. This allows us to display which IP is assigned to which link on the cluster in the join dialog, helping a user identify which link should receive which

[pve-devel] [PATCH v2 manager 1/2] gui/cluster: add CorosyncLinkEdit component to support up to 8 links

2020-03-17 Thread Stefan Reiter
CorosyncLinkEdit is a Panel that contains between one and 8 CorosyncLinkSelectors. These can be added or removed with according buttons. Values submitted to the API are calculated by each ProxmoxNetworkSelector itself. This works because ExtJS searches recursively through all child components for

Re: [pve-devel] [PATCH container 1/2] update_lxc_config: set in-CT network devices to up

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 7:35 AM, Thomas Lamprecht wrote: Else some newer system do not see the interface as up and refuse to manage it.. Where do you run into that actually? I can get my arch container to behave normally with just adding `lxc.mount.auto = sys:mixed` to /etc/pve/lxc/$vmid.conf atm.

Re: [pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Stefan Reiter
Casually looked over the patches, looking good to me so far - I'll give them some testing later. Two things inline. On 17/03/2020 08:55, Fabian Grünbichler wrote: with incremental drive-mirror and dirty-bitmap tracking. 1.) get replicated disks that are currently referenced by running VM 2.)

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 10:27 AM, Wolfgang Bumiller wrote: On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, even if it means doing stupid unnecessary things, as else systemd decides to regress and suddenly break

Re: [pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Wolfgang Bumiller
On 3/17/20 7:35 AM, Thomas Lamprecht wrote: CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, even if it means doing stupid unnecessary things, as else systemd decides to regress and suddenly break network stack in CT after an upgrade[1]. This

[pve-devel] [PATCH v2 qemu-server 1/3] drive-mirror: add support for incremental sync

2020-03-17 Thread Fabian Grünbichler
by re-using a dirty bitmap that represents changes since the divergence of source and target volume. requires a qemu that supports incremental drive-mirroring, and will die otherwise. Signed-off-by: Fabian Grünbichler --- Notes: v1-v2: - use newer Qemu patches picked up by me and

[pve-devel] [PATCH v2 qemu/qemu-server 0/4] live-migration with replicated disks

2020-03-17 Thread Fabian Grünbichler
I recently picked up and finished some work-in-progress patches for adding bitmap support to drive-mirror (it got added to backup block jobs in 4.0, with plenty of fixes in 4.1 and 4.2) and submitted them upstream. IMHO this is in a shape now where we can include it, but I'd also be fine with

[pve-devel] [PATCH v2 qemu-server 2/3] migrate: add replication info to disk overview

2020-03-17 Thread Fabian Grünbichler
to make migration logs a bit easier to grasp with a quick glance. Signed-off-by: Fabian Grünbichler --- Notes: unchanged since v1 PVE/QemuMigrate.pm | 11 +++ 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/PVE/QemuMigrate.pm b/PVE/QemuMigrate.pm index

[pve-devel] [PATCH v2 qemu-server 3/3] migrate: add live-migration of replicated disks

2020-03-17 Thread Fabian Grünbichler
with incremental drive-mirror and dirty-bitmap tracking. 1.) get replicated disks that are currently referenced by running VM 2.) add a block-dirty-bitmap to each of them 3.) replicate ALL replicated disks 4.) pass bitmaps from 2) to drive-mirror for disks from 1) 5.) skip replicated disks when

Re: [pve-devel] lvm2: global_filter not ',' separated

2020-03-17 Thread Thomas Lamprecht
Hi, On 3/16/20 2:34 PM, Roland Kammerer wrote: > Hi all, > > Commit[1] extended the filter, but if you look closely, there is no ',' > between the second and third rule. Quite frankly, I don't know what LVM > makes out of that, most likely it even considers them as 2 separate > rules. Or it

[pve-devel] [PATCH container 1/2] update_lxc_config: set in-CT network devices to up

2020-03-17 Thread Thomas Lamprecht
Else some newer system do not see the interface as up and refuse to manage it.. Signed-off-by: Thomas Lamprecht --- src/PVE/LXC.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm index 0742a53..f811550 100644 --- a/src/PVE/LXC.pm +++ b/src/PVE/LXC.pm @@

[pve-devel] [PATCH container 2/2] update_lxc_config: mount /sys read-only for CONTAINER_INTERFACE comapt

2020-03-17 Thread Thomas Lamprecht
CONTAINER_INTERFACE[0] is omething systemd people call their API and we need to adapt to it a bit, even if it means doing stupid unnecessary things, as else systemd decides to regress and suddenly break network stack in CT after an upgrade[1]. This mounts the parent /sys as ro, child mounts can