Re: [pve-devel] [PATCH qemu-server v2] Fix #2412: Missing VMs in pools

On 10/14/19 1:10 PM, Dominic Jäger wrote: > Between calling vm_destroy and removing the ID from user.cfg > (remove_vm_access) > creating a new VM with this ID was possible. VMs could go missing from pools > as > a consequence. > > Adding a lock solves this for clones from the same node.

[pve-devel] applied: [PATCH v2 manager] ui: migrate: add checkbox to enable offline migration with local resources

On 10/14/19 11:58 AM, Tim Marx wrote: > Added to make use of [0] and because it does make sense for non HA vm's > as well, in accordance with #2241. > > [0] pve-ha-manager: 6e8b0c225405da9472f56fe5c94c94b204259caa > Signed-off-by: Tim Marx > --- > changes v2: > * changed naming

Re: [pve-devel] [PATCH v4 qemu-server 10/12] Rework get_cpu_options and allow custom CPU models

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > If a cputype is custom (check via prefix), try to load options from the > custom CPU model config, and set values accordingly. > > While at it, extract currently hardcoded values into seperate sub and add > reasonings. > > Since the new flag

Re: [pve-devel] [PATCH v4 qemu-server 08/12] Verify VM-specific CPU configs seperately

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > $cpu_fmt is being reused for custom CPUs as well as VM-specific CPU > settings. The "pve-vm-cpu-conf" format is introduced to verify a config > specifically for use as VM-specific settings. > > Signed-off-by: Stefan Reiter > --- > > v3 -> v4: >

Re: [pve-devel] [PATCH v4 qemu-server 09/12] Add helpers to better structure CPU option handling

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > To avoid hardcoding even more CPU-flag related things for custom CPU > models, introduce a dynamic approach to resolving flags. > > resolve_cpu_flags takes a list of hashes (as documented in the > comment) and resolves them to a valid "-cpu"

Re: [pve-devel] [PATCH v4 qemu-server 07/12] Add overrides and convenience functions to CPUConfig

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > Add two overrides to avoid writing redundant information to the config > file. > > get_model_by_name is used to return a cpu config with default values > filled out. > > Signed-off-by: Stefan Reiter > --- > > v3 -> v4: > * add is_custom_model

Re: [pve-devel] [PATCH v4 qemu-server 06/12] Adapt CPUConfig to handle custom models

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > Turn CPUConfig into a SectionConfig with parsing/writing support for > custom CPU models. IO is handled using cfs. > > Namespacing will be provided using "custom-" prefix for custom model > names (in VM config only, cpu-models.conf will contain

Re: [pve-devel] [PATCH v4 qemu-server 05/12] Add CPUConfig file and migrate some helpers

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > The package will be used for custom CPU models as a SectionConfig, hence > the name. For now we simply move some CPU related helper functions and > declarations over from QemuServer to reduce clutter there. > > qemu_machine_feature_enabled is

[pve-devel] [PATCH qemu-server v2] Fix #2412: Missing VMs in pools

Between calling vm_destroy and removing the ID from user.cfg (remove_vm_access) creating a new VM with this ID was possible. VMs could go missing from pools as a consequence. Adding a lock solves this for clones from the same node. Additionally, unlinking must happen at the very end of the

[pve-devel] [common 7/9] implement the setup function for DNS challenge

--- src/PVE/ACME/ACME_sh.pm | 23 +++ 1 file changed, 23 insertions(+) diff --git a/src/PVE/ACME/ACME_sh.pm b/src/PVE/ACME/ACME_sh.pm index 40be772..e60fd93 100644 --- a/src/PVE/ACME/ACME_sh.pm +++ b/src/PVE/ACME/ACME_sh.pm @@ -5,6 +5,9 @@ use warnings; use HTTP::Daemon;

[pve-devel] [common 5/9] dynamically search for the DNS API credential file.

This function search for credential files. The file must end with .cred and named as the API plugin. The presumption is that every cluster uses the same DNS API credential file. Also, only one credential file exists in the directory. --- src/PVE/ACME/ACME_sh.pm | 22 ++ 1

[pve-devel] [manager 4/6] allow acme.sh as plugin

--- PVE/NodeConfig.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PVE/NodeConfig.pm b/PVE/NodeConfig.pm index a807c710..7817bd1e 100644 --- a/PVE/NodeConfig.pm +++ b/PVE/NodeConfig.pm @@ -73,7 +73,7 @@ my $acmedesc = { account =>

[pve-devel] [manager 6/6] Add dns_api_config.

The DNS challenge uses this parameter for API authentication. --- PVE/NodeConfig.pm | 6 ++ 1 file changed, 6 insertions(+) diff --git a/PVE/NodeConfig.pm b/PVE/NodeConfig.pm index b84590ac..f605c7ee 100644 --- a/PVE/NodeConfig.pm +++ b/PVE/NodeConfig.pm @@ -78,6 +78,12 @@ my $acmedesc = {

[pve-devel] [common 9/9] implement the validating_url function for DNS challenge

--- src/PVE/ACME/ACME_sh.pm | 33 + 1 file changed, 33 insertions(+) diff --git a/src/PVE/ACME/ACME_sh.pm b/src/PVE/ACME/ACME_sh.pm index fc2881c..319a614 100644 --- a/src/PVE/ACME/ACME_sh.pm +++ b/src/PVE/ACME/ACME_sh.pm @@ -60,6 +60,39 @@ my $compose_cmd = sub {

[pve-devel] [common 4/9] refactor extract_callenge for code reuse.

--- src/PVE/ACME.pm| 16 src/PVE/ACME/StandAlone.pm | 9 + 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/src/PVE/ACME.pm b/src/PVE/ACME.pm index d6b6e99..173af69 100644 --- a/src/PVE/ACME.pm +++ b/src/PVE/ACME.pm @@ -76,6 +76,22 @@ sub

[pve-devel] [manager 5/6] add alias parameter for ACME DNS Challenge.

This parameter allows to use an alternative Domain for setup the DNS record. This can be useful for security reasons or if the real domain has none Ability to automatically set up a DNS record. --- PVE/NodeConfig.pm | 7 +++ 1 file changed, 7 insertions(+) diff --git a/PVE/NodeConfig.pm

[pve-devel] [common 2/9] refactor validating_url

--- src/PVE/ACME.pm| 12 src/PVE/ACME/Challenge.pm | 6 ++ src/PVE/ACME/StandAlone.pm | 32 +++- 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/src/PVE/ACME.pm b/src/PVE/ACME.pm index da4cbcc..c82f297 100644 ---

[pve-devel] [common 3/9] Add ACME_sh plugin

--- src/Makefile| 1 + src/PVE/ACME.pm | 2 ++ src/PVE/ACME/ACME_sh.pm | 24 3 files changed, 27 insertions(+) create mode 100644 src/PVE/ACME/ACME_sh.pm diff --git a/src/Makefile b/src/Makefile index 02f1f56..fcccac8 100644 --- a/src/Makefile +++

[pve-devel] [common 6/9] command composer for acme.sh

This composer supports two different operations. pve-setup: this operation adds the DNS TXT record. pve-teardown: this operation removes the DNS TXT record --- src/PVE/ACME/ACME_sh.pm | 16 1 file changed, 16 insertions(+) diff --git a/src/PVE/ACME/ACME_sh.pm

[pve-devel] [common 8/9] implement the teardown function for DNS challenge

--- src/PVE/ACME/ACME_sh.pm | 7 +++ 1 file changed, 7 insertions(+) diff --git a/src/PVE/ACME/ACME_sh.pm b/src/PVE/ACME/ACME_sh.pm index e60fd93..fc2881c 100644 --- a/src/PVE/ACME/ACME_sh.pm +++ b/src/PVE/ACME/ACME_sh.pm @@ -82,4 +82,11 @@ sub setup { return ($url, $digest, $domain);

[pve-devel] [manager 3/6] refactor validating_url

The validating have to be done in the Plugin to be generic. --- PVE/API2/ACME.pm | 27 +-- 1 file changed, 1 insertion(+), 26 deletions(-) diff --git a/PVE/API2/ACME.pm b/PVE/API2/ACME.pm index 2c05db9b..db94586d 100644 --- a/PVE/API2/ACME.pm +++ b/PVE/API2/ACME.pm @@

[pve-devel] [manager 1/6] Add plugin property for choosing different ACME modes.

With this property, different methods for ACME challenges are possible. At the moment we will only support HTTP-01 through StandAlone and DNS-01 through ACME_sh Plugin. --- PVE/NodeConfig.pm | 7 +++ 1 file changed, 7 insertions(+) diff --git a/PVE/NodeConfig.pm b/PVE/NodeConfig.pm index

[pve-devel] RFC for ACME DNS Challenge

This series also includes a new GIT repository on proxdev. The path is staff/wl/pve-acme. The acme_sh project is used as DNS API plugin systems. So we can reuse the already defiend plugins. I deside to install the complett acme_sh scrips so somwone could use it for alternative use. I'm not sure

[pve-devel] [manager 2/6] Pass parameters for use in plugin.

The node config is required to determine the plugin and possibly to pass plugin specific parameters. --- PVE/API2/ACME.pm | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/PVE/API2/ACME.pm b/PVE/API2/ACME.pm index b1bb6261..2c05db9b 100644 --- a/PVE/API2/ACME.pm +++

[pve-devel] [common 1/9] Add dynamic plugin lookup for ACME

The dynamic approach reduces failure if new plugins will included. --- src/PVE/ACME.pm| 4 src/PVE/ACME/Challenge.pm | 8 src/PVE/ACME/StandAlone.pm | 4 3 files changed, 16 insertions(+) diff --git a/src/PVE/ACME.pm b/src/PVE/ACME.pm index 38a14a5..da4cbcc 100644

Re: [pve-devel] [PATCH v4 manager 01/12] Broadcast supported CPU flags

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > pvestatd will check if the KVM version has changed using > kvm_user_version (which automatically clears its cache if QEMU/KVM > updates), and if it has, query supported CPU flags and broadcast them as > key-value pairs to the cluster. > > If

Re: [pve-devel] [PATCH v4 qemu 02/12] Write understood CPU flags into static file

On October 7, 2019 2:47 pm, Stefan Reiter wrote: > located at /usr/share/kvm/cpu-flags-understood-$arch > > This file can be read by qemu-server's "query_understood_cpu_flags" > function, avoiding a more expensive call to QEMU. > > For now, only x86_64 is implemented, since aarch64 doesn't print

Re: [pve-devel] [PATCH qemu-server v2] Fix #2171: VM statefile was not activated

On 10/14/19 11:59 AM, Alwin Antreich wrote: > On Mon, Oct 14, 2019 at 11:44:59AM +0200, Thomas Lamprecht wrote: >> On 10/11/19 1:45 PM, Alwin Antreich wrote: >>> On Fri, Oct 11, 2019 at 12:17:28PM +0200, Thomas Lamprecht wrote: On 10/11/19 12:02 PM, Alwin Antreich wrote: > On Fri, Oct 11,

Re: [pve-devel] [PATCH qemu-server v2] Fix #2171: VM statefile was not activated

On Mon, Oct 14, 2019 at 11:44:59AM +0200, Thomas Lamprecht wrote: > On 10/11/19 1:45 PM, Alwin Antreich wrote: > > On Fri, Oct 11, 2019 at 12:17:28PM +0200, Thomas Lamprecht wrote: > >> On 10/11/19 12:02 PM, Alwin Antreich wrote: > >>> On Fri, Oct 11, 2019 at 07:10:53AM +0200, Thomas Lamprecht

[pve-devel] [PATCH v2 manager] ui: migrate: add checkbox to enable offline migration with local resources

Added to make use of [0] and because it does make sense for non HA vm's as well, in accordance with #2241. [0] pve-ha-manager: 6e8b0c225405da9472f56fe5c94c94b204259caa Signed-off-by: Tim Marx --- changes v2: * changed naming overwriteLocalResourceCheck, resetMigrationPossible, Force * added

Re: [pve-devel] [PATCH qemu-server v2] Fix #2171: VM statefile was not activated

On 10/11/19 1:45 PM, Alwin Antreich wrote: > On Fri, Oct 11, 2019 at 12:17:28PM +0200, Thomas Lamprecht wrote: >> On 10/11/19 12:02 PM, Alwin Antreich wrote: >>> On Fri, Oct 11, 2019 at 07:10:53AM +0200, Thomas Lamprecht wrote: On 10/10/19 3:58 PM, Alwin Antreich wrote: > Machine states

[pve-devel] [PATCH v3 container 16/19] allow to check for in-use volumes in pending section

Signed-off-by: Oguz Bektas --- src/PVE/LXC/Config.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm index da5ef03..f73da1e 100644 --- a/src/PVE/LXC/Config.pm +++ b/src/PVE/LXC/Config.pm @@ -1326,9 +1326,10 @@ sub

[pve-devel] [PATCH v3 container 11/19] add 'pct pending'

same as 'qm pending', the code is shared via GuestHelpers Signed-off-by: Oguz Bektas --- src/PVE/CLI/pct.pm | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm index 35ad72f..476f44d 100755 --- a/src/PVE/CLI/pct.pm +++ b/src/PVE/CLI/pct.pm @@ -7,6 +7,7

[pve-devel] [PATCH v3 container 12/19] api: config: use shared guesthelpers in GET call

since containers can also have pending changes now, we need a method to get the current applied config as well as the one with the pending changes inside. this makes the GET config api more consistent with qemu-server's by reusing load_current_config and load_snapshot_config from AbstractConfig.

[pve-devel] [PATCH v3 container 13/19] skip pending changes while cloning

Signed-off-by: Oguz Bektas --- src/PVE/API2/LXC.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm index 6832f83..71eb110 100644 --- a/src/PVE/API2/LXC.pm +++ b/src/PVE/API2/LXC.pm @@ -1441,6 +1441,7 @@ __PACKAGE__->register_method({

[pve-devel] [PATCH v3 container 14/19] skip pending changes while taking backup

we can only clone the current state of container (without pending changes), as otherwise the on-disk state might not match the configuration. this also makes it more consistent to qemu-server behavior. Signed-off-by: Oguz Bektas --- src/PVE/VZDump/LXC.pm | 1 + 1 file changed, 1 insertion(+)

[pve-devel] [PATCH v3 container 15/19] prepend underscores for is_volume_in_use helper

this helper was defined twice, once as 'my $is_volume_in_use' sub and second as a helper sub. as our other helpers with a similar structure, it is better to prepend the variable sub with two underscores. Signed-off-by: Oguz Bektas --- src/PVE/LXC/Config.pm | 8 1 file changed, 4

[pve-devel] [PATCH v3 container 19/19] implement pending changes

previous behaviour directly applied the possible config changes, and died when there was something which can't be applied while CT is running. instead, we now write all the changes directly into the config pending section, and then apply or hotplug the changes depending on whether CT is running.

[pve-devel] [PATCH v3 container 10/19] add lxc/pending API path

Signed-off-by: Oguz Bektas --- src/PVE/API2/LXC.pm | 56 + 1 file changed, 56 insertions(+) diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm index 28c9047..6832f83 100644 --- a/src/PVE/API2/LXC.pm +++ b/src/PVE/API2/LXC.pm @@ -515,6 +515,7 @@

[pve-devel] [PATCH v3 container 18/19] apply pending changes during container start

Signed-off-by: Oguz Bektas --- src/PVE/LXC.pm | 7 +++ 1 file changed, 7 insertions(+) diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm index fa66aaf..6b5bd44 100644 --- a/src/PVE/LXC.pm +++ b/src/PVE/LXC.pm @@ -1934,6 +1934,13 @@ sub userns_command { sub vm_start { my ($vmid, $conf,

[pve-devel] [PATCH v3 container 17/19] add vmconfig_hotplug_pending and vmconfig_apply_pending

vmconfig_hotplug_pending is responsible for checking if a key/value pair in the pending section can be hotpugged, if yes; perform a generic replace, or perform specific actions for hotplugging the special cases. vmconfig_apply_pending is only supposed to be called when ct isn't live.

[pve-devel] [PATCH v3 qemu-server 06/19] use new config helpers from guest-common for pending changes

most of the pending changes related code has been moved into AbstractConfig, so we have to call them as class methods from QemuConfig instead. Signed-off-by: Oguz Bektas --- PVE/API2/Qemu.pm | 14 - PVE/QemuServer.pm | 79 +-- 2 files

[pve-devel] [PATCH v3 container 09/19] adapt CT config parser for pending changes

config parser can now read/write [pve:pending] section. this was named such, instead of [PENDING], after on- and offline discussion regarding namespacing the pending section and snapshots. Signed-off-by: Oguz Bektas --- src/PVE/LXC/Config.pm | 23 +-- 1 file changed, 21

[pve-devel] [PATCH v3 qemu-server 08/19] api: use guesthelper method for vm_pending path

we can use the shared conf_table_with_pending guesthelper to produce the config table with the extra delete and pending columns. Signed-off-by: Oguz Bektas --- PVE/API2/Qemu.pm | 44 1 file changed, 4 insertions(+), 40 deletions(-) diff --git

[pve-devel] [PATCH v3 qemu-server 07/19] cli: use guesthelper for pending

use the shared format_pending method from guesthelpers Signed-off-by: Oguz Bektas --- PVE/CLI/qm.pm | 28 +--- 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/PVE/CLI/qm.pm b/PVE/CLI/qm.pm index 17935d0..2f1969a 100755 --- a/PVE/CLI/qm.pm +++

[pve-devel] [PATCH v3 guest-common 01/19] abstractconfig: add pending changes related helpers

use a better naming scheme for methods from qemuserver: split_flagged_list -> parse_pending_delete join_flagged_list -> print_pending_delete vmconfig_delete_pending_option -> add_to_pending_delete vmconfig_undelete_pending_option -> remove_from_pending_delete vmconfig_cleanup_pending ->

[pve-devel] [PATCH v3 guest-common 02/19] abstractconfig: add load_current_config and load_snapshot_config

this code is already used by qemu-server's GET config API call. it is however better to split this into to methods and decide what to run in the API call. this general implementation uses some $class helpers which allow to abstract away the difference in the child classes. this will be used for

[pve-devel] [PATCH v3 00/19] lxc pending changes

this patch series makes it possible to add/delete/revert pending changes in the backend for containers. v2 -> v3: * reorder and squash/split commits * split load_current_config and load_snapshot_config * use perl hash representation for the pending delete hash * remove semi-related stuff (i.e.

[pve-devel] [PATCH v3 qemu-server 05/19] api: use shared methods in config GET

in config GET call, we can now use the new shared methods from guest-common, namely load_current_config and load_snapshot_config. the correct method is called depending on the parameters 'current' or 'snapshot' Signed-off-by: Oguz Bektas --- PVE/API2/Qemu.pm | 41

[pve-devel] [PATCH v3 guest-common 03/19] helpers: add pending-aware guest config printer from qemu-server

Signed-off-by: Oguz Bektas --- PVE/GuestHelpers.pm | 26 ++ 1 file changed, 26 insertions(+) diff --git a/PVE/GuestHelpers.pm b/PVE/GuestHelpers.pm index 35059e6..a36b9bc 100644 --- a/PVE/GuestHelpers.pm +++ b/PVE/GuestHelpers.pm @@ -116,4 +116,30 @@ sub

[pve-devel] [PATCH v3 guest-common 04/19] helpers: add method to represent config as a table

in vm_pending API, this method is used to represent the configuration as a table with current, pending and delete columns. by adding it as a guesthelper, we can also use it for container pending changes. Signed-off-by: Oguz Bektas --- PVE/GuestHelpers.pm | 35

Re: [pve-devel] [PATCH pve-network 0/8] generic sdn controller plugins + improvments

>>OK, thanks for the write up. I'll try to give this series a look the next >>days. >>If something holds you up on continuing ping me and I'll try to give it more >>priority. Thanks Thomas! (No Problem, I can wait. I'm going to do more tests on loadbalancing for now) - Mail original -

Re: [pve-devel] [PATCH pve-network 0/8] generic sdn controller plugins + improvments

On 10/11/19 1:13 PM, Alexandre DERUMIER wrote: >>> Sorry, I did not had time to give this a proper check and it looks to start >>> getting serious so I did not wanted to wave this through. If then something >>> was off it would be much more work to fix/change it later on.. > > No problem, It

Re: [pve-devel] [PATCH qemu-server] Fix #2412: Missing VMs in pools

On 10/11/19 1:13 PM, Fabian Grünbichler wrote: > the current situation is especially bad, since we vdisk_free unused > disks AFTER unlinking the config file, so the delay between unlinking > the config file, and removing from user.cfg/removing the firewall config > can be quite long.. that's a