Re: [pve-devel] [PATCH manager] pve5to6: Add warning for some Gluster versions
On August 28, 2019 1:25 pm, Dominic Jäger wrote: > After Fabian had sent his patch [0] we actually thought that this patch > for pve5to6 would be obsolete. Upgrades as well as fresh installs worked > flawlessly in my tests. Consequently, I removed the hint from the > upgrade documentation [1] and marked the bug [2] as fixed. Before > writing you this, I tested everything again and, unfortunately, > something (or maybe multiple things) seems to be broken now. upstream split the packages up now, but did not bother to fix the package releationships with versioned provides/replaces/.. the end result is that you need to either uninstall and reinstall gluster completely (not possible since we have a dependency via pve-qemu) or do the upgrade with -o Dpkg::Options::="--force-overwrite" added to the apt upgrade command to ignore the breakage, at least until gluster upstream has actually fixed packages available.. https://lists.gluster.org/pipermail/gluster-users/2019-August/036941.html ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] applied: [RFC container] mountpoints: create parent dirs with correct owner
On August 26, 2019 1:30 pm, Thomas Lamprecht wrote: > On 24.07.19 13:37, Fabian Grünbichler wrote: >> otherwise unprivileged containers might end up with directories that >> they cannot modify since they are owned by the user root in the host >> namespace, instead of root inside the container. >> >> note: the problematic behaviour is only exhibited when an intermediate >> directory needs to be created, e.g. a mountpoint /test/mp gets mounted, >> and /test does not yet exist. >> >> Signed-off-by: Fabian Grünbichler >> --- >> Notes: >> requires fchownat support in PVE::Tools - see other patch and bump >> build-depends + depends accordingly after applying! >> >> I am not sure whether this is 100% correct w.r.t. error edge cases, >> since we >> potentially die after mkdirat without calling fchownat. it is for sure >> better >> than the status quo though ;) >> >> thank you Dietmar for noticing the buggy behaviour! >> >> src/PVE/LXC.pm| 27 --- >> src/PVE/VZDump/LXC.pm | 7 +-- >> src/lxc-pve-prestart-hook | 4 +++- >> 3 files changed, 24 insertions(+), 14 deletions(-) >> > > > applied, thanks! > > For existing CTs, where problematic directories outside the CT GID/UID range > got > already created, we could add a "repair-ids" command, which would reset > UIDs/GIDs > from (specified) paths to a CT (root) ID again, if they where outside the CTs > range, e.g.: > > # pct repair-file-permissions [] [--repair-with-id ] > > Which could roughly follow those rules: > * only "repair" files with IDs outside of the CTs file range, e.g., "0" from > PVE > host POV > * only check at given path, if set > * maybe add a "no-recursive" flag to really only do a single file? > > With that at least a possibility would be there to allow people fixing the > directory tree permissions so that CT-root can change/delete/... them again. > Just as an idea. > given that there were no complaints so far and we found this internally, I think we don't need to provide such a command. a simple pct mount + chown with mapped numerical uid/gid should work if anyone ever asks for it.. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server] fix #1934: add qemu fw_cfg variables via 'tags'
On August 23, 2019 2:03 pm, Dominik Csapak wrote:
> this add the 'tags' property to vms, which has the format:
why 'tags'? seems rather generic for what it does ;)
>
> key=value(;key=value)*
>
> each value will be set as
>
> -fw_cfg 'name=opt/com.proxmox/$key,string=$value'
> (qemu recommends using a unique rfqdn)
>
> this way, users can tag the vm with that information available inside
> e.g. under linux the value can be read under
>
> /sys/firmware/qemu_fw_cfg/by_name/opt/com.proxmox./$key/raw
>
> see the file docs/specs/fw_cfg.txt in the qemu repository for more
> details
if we introduce this, wouldn't it also make sense to allow to pass in
snippet files via this interface (via file=$path instead of string=$value)?
>
> maybe we can also use this in the future to show/set in the gui
> e.g. some grouping/ordering etc.
>
> Signed-off-by: Dominik Csapak
> ---
> PVE/QemuServer.pm | 37 +
> 1 file changed, 37 insertions(+)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 9f5bf56..d55a1ae 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -232,6 +232,29 @@ my $watchdog_fmt = {
> };
> PVE::JSONSchema::register_format('pve-qm-watchdog', $watchdog_fmt);
>
> +PVE::JSONSchema::register_format('pve-qm-tags', \&verify_tag_format);
> +sub verify_tag_format {
> +my ($tagstring, $noerr) = @_;
> +
> +if (!$tagstring) {
> + return '';
> +}
> +
> +# forbid all characters not in range 0x20-0x7E
> +for my $tag (split(';', $tagstring)) {
> + my ($key, $value) = ($tag =~ m/^(.*)=(.*)$/);
> + if ($key =~ m/[\x00-\x1F\x7F-\xFF]/) {
> + die "invalid character in tag key\n" if !$noerr;
> + return undef;
> + }
> + if ($value =~ m/[\x00-\x1F\x7F-\xFF]/) {
> + die "invalid character in tag value\n" if !$noerr;
> + return undef;
> + }
> +}
> +
> +return $tagstring;
> +}
> my $agent_fmt = {
> enabled => {
> description => "Enable/disable Qemu GuestAgent.",
> @@ -672,6 +695,13 @@ EODESCR
> description => "Configure a audio device, useful in combination with
> QXL/Spice.",
> optional => 1
> },
> +tags => {
> + description => "Specify key/value pairs to be added to qemu fw_cfg.",
> + type => 'string',
> + maxLength => 4096,
> + format => 'pve-qm-tags',
> + optional => 1,
> +},
> };
>
> my $cicustom_fmt = {
> @@ -4152,6 +4182,13 @@ sub config_to_command {
> push @$cmd, '-loadstate', $statepath;
> }
>
> +if ($conf->{tags}) {
> + for my $tag (split(';', $conf->{tags})) {
> + my ($key, $value) = ($tag =~ m/^(.*)=(.*)$/);
> + push @$cmd, '-fw_cfg', "name=opt/com.proxmox/$key,string=$value";
> + }
> +}
> +
> # add custom args
> if ($conf->{args}) {
> my $aa = PVE::Tools::split_args($conf->{args});
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] Gitlab-ci
On August 27, 2019 11:24 am, Alexandre DERUMIER wrote: > Hi, > > I think 1 benefit of gitlab,github,... is the tracking of merge request. IMHO this is a big downside of Github and Gitlab - cleanly checking out an older iteration of a PR/MR is very cumbersome (and wasn't even really possible for the longest time). The review comments reference (and even link to!) code that is no longer available directly, and it gets confusing quickly. The only thing that's really nice is the direct integration into various CI tools, also for in-progress change requests. with mail, v1 and v2 are simply two mail threads, including any review comments. this is much simpler to keep around, checking the archive, etc. > (To not lost some patches series send months ago). > > For code review it great too. (make comments directly in the merge) see above. I hate the review interface with a passion :-P see [1] for example - some inline comments are included in the discussion, but the full review with all comments is 404[2], even though it is linked from [1]. I am also following sourcehut's progress - it might be able to bridge the gap between both worlds (plain email + nice looking web interface). > I'm curious, do you use some kind of email scripts, special email client, to > manage current workflow ? > (Sometime I'm lost myself with my patches ;) most of the actively reviewing people use either (neo)mutt or some variant of notmuch, but there are some not-yet converted thunderbird users as well ;) 1: https://github.com/corosync/corosync/pull/450 2: https://github.com/corosync/corosync/pull/450/files/68e7b4cbe3a4687bb6168500d3275d5969b9b9c0 ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server] fix #1934: add qemu fw_cfg variables via 'tags'
On September 2, 2019 2:27 pm, Thomas Lamprecht wrote:
> On 9/2/19 2:14 PM, Fabian Grünbichler wrote:
>> On August 23, 2019 2:03 pm, Dominik Csapak wrote:
>>> this add the 'tags' property to vms, which has the format:
>>
>> why 'tags'? seems rather generic for what it does ;)
>
> Second, tags is a NAK from me. Maybe just "fw_cfg" (or a variation), the
> Webinterface
> can show a full name and link to docs anyway (if it'd be added there,
> someday), and for
> the CLI we have the CLI help, man pages and the fact that it's named the same
> as the QEMU
> option, which helps a lot to associated both together (even if the sematic
> how values are
> passed differs, which is IMO totally fine).
>
>>>
>>> key=value(;key=value)*
>>>
>>> each value will be set as
>>>
>>> -fw_cfg 'name=opt/com.proxmox/$key,string=$value'
>>> (qemu recommends using a unique rfqdn)
>>>
>>> this way, users can tag the vm with that information available inside
>>> e.g. under linux the value can be read under
>>>
>>> /sys/firmware/qemu_fw_cfg/by_name/opt/com.proxmox./$key/raw
>>>
>>> see the file docs/specs/fw_cfg.txt in the qemu repository for more
>>> details
>>
>> if we introduce this, wouldn't it also make sense to allow to pass in
>> snippet files via this interface (via file=$path instead of string=$value)?
>
> How do you want to differentiate between? Would need a special syntax to
> detect that "file" is not meant to be a key for fw_cfg but a hint for PVE..
yes, obviously that would require a different syntax (or maybe even an
indexed property string? I don't know how many such values users would
want to set for a single VM..)
>
>>
>>>
>>> maybe we can also use this in the future to show/set in the gui
>>> e.g. some grouping/ordering etc.
>>>
>>> Signed-off-by: Dominik Csapak
>>> ---
>>> PVE/QemuServer.pm | 37 +
>>> 1 file changed, 37 insertions(+)
>>>
>>> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
>>> index 9f5bf56..d55a1ae 100644
>>> --- a/PVE/QemuServer.pm
>>> +++ b/PVE/QemuServer.pm
>>> @@ -232,6 +232,29 @@ my $watchdog_fmt = {
>>> };
>>> PVE::JSONSchema::register_format('pve-qm-watchdog', $watchdog_fmt);
>>>
>>> +PVE::JSONSchema::register_format('pve-qm-tags', \&verify_tag_format);
>>> +sub verify_tag_format {
>>> +my ($tagstring, $noerr) = @_;
>>> +
>>> +if (!$tagstring) {
>>> + return '';
>>> +}
>>> +
>>> +# forbid all characters not in range 0x20-0x7E
>>> +for my $tag (split(';', $tagstring)) {
>>> + my ($key, $value) = ($tag =~ m/^(.*)=(.*)$/);
>>> + if ($key =~ m/[\x00-\x1F\x7F-\xFF]/) {
>>> + die "invalid character in tag key\n" if !$noerr;
>>> + return undef;
>>> + }
>>> + if ($value =~ m/[\x00-\x1F\x7F-\xFF]/) {
>>> + die "invalid character in tag value\n" if !$noerr;
>>> + return undef;
>>> + }
>>> +}
>>> +
>>> +return $tagstring;
>>> +}
>>> my $agent_fmt = {
>>> enabled => {
>>> description => "Enable/disable Qemu GuestAgent.",
>>> @@ -672,6 +695,13 @@ EODESCR
>>> description => "Configure a audio device, useful in combination with
>>> QXL/Spice.",
>>> optional => 1
>>> },
>>> +tags => {
>>> + description => "Specify key/value pairs to be added to qemu fw_cfg.",
>>> + type => 'string',
>>> + maxLength => 4096,
>>> + format => 'pve-qm-tags',
>>> + optional => 1,
>>> +},
>>> };
>>>
>>> my $cicustom_fmt = {
>>> @@ -4152,6 +4182,13 @@ sub config_to_command {
>>> push @$cmd, '-loadstate', $statepath;
>>> }
>>>
>>> +if ($conf->{tags}) {
>>> + for my $tag (split(';', $conf->{tags})) {
>>> + my ($key, $value) = ($tag =~ m/^(.*)=(.*)$/);
>
> what do I do when I need a ";" in a value or a "=" in a key?
>
>>> + push @$cmd, '-fw_cfg', "name=opt/com.proxmox/$key,string=$value";
>>> + }
>>> +}
>>> +
>>> # add custom args
>>> if ($conf->{args}) {
>>> my $aa = PVE::Tools::split_args($conf->{args});
>>> --
>>> 2.20.1
>>>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server] fix #1934: add qemu fw_cfg variables via 'tags'
On September 4, 2019 8:49 am, Dominik Csapak wrote: > On 9/2/19 2:14 PM, Fabian Grünbichler wrote: >> On August 23, 2019 2:03 pm, Dominik Csapak wrote: >>> this add the 'tags' property to vms, which has the format: >> >> why 'tags'? seems rather generic for what it does ;) > > because the request in the bugreport was a more general one > (users wanted to simply set some tags for the vm > which are not in the description) > > i just wanted to also use them besides simply having them in > the config ;) to me that sounds like rather unexpected behaviour. while there is some overlap between "tagging guest for categorization" and "exposing key-value pairs to guest", there are plenty of cases where the information from the former should not be exposed to the guest (e.g., an admin tagging annoying users with unflattering tag values ;)), and vice-versa, where having everything put into fw_cfg displayed as categorization tags is inconvenient/cluttering as well. I am not opposed to a general 'tagging' feature at all (even if just for search/filtering and display purposes), but I'd keep it distinct from the fw_cfg stuff. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [RFC qemu-server] rewrite description for vm_config
On September 4, 2019 3:59 pm, Oguz Bektas wrote:
> the description doesn't match the default behaviour, which is to replace
> the current values with pending ones in the returned config, unless the
> 'current' option is passed.
>
> Signed-off-by: Oguz Bektas
> ---
>
> i tried to come up with a reasonable description, but sending it as RFC
> just in case someone else comes up with something better
>
> PVE/API2/Qemu.pm | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index b30931d..36487ee 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -827,7 +827,8 @@ __PACKAGE__->register_method({
> path => '{vmid}/config',
> method => 'GET',
> proxyto => 'node',
> -description => "Get current virtual machine configuration. This does not
> include pending configuration changes (see 'pending' API).",
> +description => "Get virtual machine configuration. If the 'current'
> option is" .
> + " not passed, pending changes will replace the current
> ones.",
besides style, this is also incorrect since you can pass '0' for
'current' ;)
Get virtual machine configuration. If 'current' is set, any pending
configuration changes will not be included in the returned value.
Otherwise, the returned configuration values will represent the state
with all pending changes applied.
A bit verbose, but matching what it does. Alternatively, just
Get virtual machine configuration. See 'pending' API for current and
pending configuration retrieval.
and leave the rest to the parameter description?
current -> "Get current values only (ignoring pending changes)."
> permissions => {
> check => ['perm', '/vms/{vmid}', [ 'VM.Audit' ]],
> },
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v3 guest-common 1/2] fix #1291: implement remove_vmid_from_backup_jobs
On September 4, 2019 4:41 pm, Thomas Lamprecht wrote: > On 01.07.19 15:43, Christian Ebner wrote: >> remove_vmid_from_backup_jobs updates the vzdump.cron backup jobs, >> excluding the given vmid. >> >> Signed-off-by: Christian Ebner >> --- >> PVE/VZDump/Plugin.pm | 46 >> 1 file changed, 46 insertions(+) >> >> diff --git a/PVE/VZDump/Plugin.pm b/PVE/VZDump/Plugin.pm >> index 9933ef6..f415242 100644 >> --- a/PVE/VZDump/Plugin.pm >> +++ b/PVE/VZDump/Plugin.pm >> @@ -7,6 +7,8 @@ use POSIX qw(strftime); >> >> use PVE::Tools; >> use PVE::SafeSyslog; >> +use PVE::Cluster qw(cfs_read_file cfs_write_file cfs_lock_file); >> +use PVE::API2::Backup; >^ > above won't fly, that's a module from pve-manager and thus would create a > cyclic > build dependency.. And I'd rather reduce than increase them ;) > > The use is for the vzdump cron parser/writer which are cfs_registered in that > file.. So either we move that out to a module here (or even higher up) or do > something else, avoiding use of modules which are lower in the dependency > chain. > we want to use it from pve-container and qemu-server, so the only sane choice is to move the cfs_register part to guest-common IMHO. polluting pve-common or pve-cluster (or pve-access-control) is even worse, and there's nothing else that fits the bill that we can access from pve-container and qemu-server unless I am missing something.. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH common] add postinst hook to fix /etc/aliases whitespace error
thanks to all three involved people ;) On September 3, 2019 1:06 pm, Thomas Lamprecht wrote: > This was wrongly shipped by our ISO since quite a bit (AFAICT, at > least 4.x), so fix it up in a versioned postinst snippet. > > Do so by usind sed with the following pattern: > # sed -E -i -e 's/^www:(\w)/www: \1/' /etc/aliases > proposed by Musee Ullah[0]. It even catches a bit more than exactly > our misstep, may help if one copied this line, or added some other > addresses to this specific aliases entry. > > Do this here, in pve-common, as it makes it sligthly simpler to roll > the change out to both, PVE and PMG. > > [0]: https://pve.proxmox.com/pipermail/pve-user/2019-September/170998.html > > Reported-by: Uwe Sauter > Signed-off-by: Thomas Lamprecht > --- > > One could even add to the proposed solution to make it more general: > # sed -E -i -e 's/^([^\s:]+):(\w)/\1: \2/' /etc/aliases > > but I think that's not in our responsibility.. > Open for different place to add this, but shared between PMG and PVE > /is/ really nice, IMO. > > debian/postinst | 20 > 1 file changed, 20 insertions(+) > create mode 100644 debian/postinst > > diff --git a/debian/postinst b/debian/postinst > new file mode 100644 > index 000..5a19c69 > --- /dev/null > +++ b/debian/postinst > @@ -0,0 +1,20 @@ > +#!/bin/sh > + > +set -e > + > +#DEBHELPER# > + > +case "$1" in > + configure) > +if test -n "$2"; then > + > +# TODO: remove once PVE 7.0 is released > +if dpkg --compare-versions "$2" 'lt' '6.0-5'; then > +sed -E -i -e 's/^www:(\w)/www: \1/' /etc/aliases > + fi > +fi > +;; > + > +esac > + > +exit 0 > -- > 2.20.1 > > > ___ > pve-devel mailing list > pve-devel@pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH ha-manager 0/4] Add inital HW based fencing
On March 27, 2019 5:42 pm, Thomas Lamprecht wrote:
> Actually this'd be v8, but it's to old and to unreviewed for anybody do
> remember anyway, so best seen as new series.
>
> Thomas Lamprecht (4):
> allow LRM lock stealing for fenced nodes
> allow use of external fencing devices
> send also email on hardware fence failure
> add some infos about HW fencing to README
first off, sorry for the long delay..
low-level nits and questions as replies to individual patches, some
stuff I noticed while testing, mostly regarding the already merged
Fence/FenceConfig code:
1) missing 'use FenceConfig;' in PVE::HA::Config.pm
2) as discussed off-list, it would be great if the fork could be moved
into exec_fence_agent to get rid of the test/ha env distinction in
Fence.pm (possibly as follow-up, since it does not change the
semantics/functionality)
3) the simulator does not support fencing_mode 'both' (since it just
decides beside on whether a fence config exists)
4) I wonder whether there is a use-case for an actual 'both' mode (since
IMHO the current 'both' mode is more like an 'either' mode, we are just
waiting for hardware or regular fencing to go through, not both ;)):
something like a hardware fence device that is optional and helpful, but
alone is not enough to ensure that the node is actually fully fenced.
e.g., fence the network of the node via switch (to prevent further
client access), but we still need to wait for the regular fencing to go
through since all nodes have some non-network shared storage,
so the not-yet-stopped services on the fenced node can still write to
the shared disks until the node is fully fenced.
or some other hacky use cases, like marking OSDs as down & out via
fencing, but that is more abuse case than use case ;)
possible this could also be done via fence.cfg, by marking certain
devices as 'not enough to consider fenced'? obviously with fencing_mode
hardware you'd need at least one proper device ;)
5) the config parser/writer pair is broken:
- writer: does not re-quote the node args
- parser: both [args] are not actually optional (like the README says)
(the writer is not yet used anywhere)
$ cat /etc/pve/ha/fence.cfg
device pve_nina:1 fence_pve 'ip=192.168.15.38' 'username=fäöµ€'
'password=12"345'
connect pve_nina:1 node=node1 plug=500 x
connect pve_nina:1 node=node2 'username=fäöµ€' 'password=12"345'
$ perl -e 'use strict; use warnings; use Data::Dumper; use PVE::Cluster; use
PVE::HA::FenceConfig; use PVE::HA::Config; PVE::Cluster::cfs_update(); my $cfg
= PVE::HA::Config::read_fence_config(); print Dumper($cfg), "\n";
PVE::HA::Config::write_fence_config($cfg);'
$VAR1 = {
'pve_nina' => {
'priority' => 0,
'sub_devs' => {
'1' => {
'node_args' => {
'node2' => [
'username=fäöµ€',
'password=12"345'
],
'node1' => [
'plug=500',
'x'
]
},
'args' => [
'ip=192.168.15.38',
'username=fäöµ€',
'password=12"345'
],
'agent' => 'fence_pve'
}
}
}
};
quotes from node arguments were dropped by the writer:
$ cat /etc/pve/ha/fence.cfg
device pve_nina:1 fence_pve 'ip=192.168.15.38' 'username=fäöµ€'
'password=12"345'
connect pve_nina:1 node=node1 plug=500 x
connect pve_nina:1 node=node2 username=fäöµ€ password=12"345
$ perl -e 'use strict; use warnings; use Data::Dumper; use PVE::Cluster; use
PVE::HA::FenceConfig; use PVE::HA::Config; PVE::Cluster::cfs_update(); my $cfg
= PVE::HA::Config::read_fence_config(); print Dumper($cfg), "\n";
PVE::HA::Config::write_fence_config($cfg);'
$VAR1 = {
'pve_nina' => {
'sub_devs' => {
'1' => {
'agent' => 'fence_pve',
'node_args' => {
'node2' => [],
'node1' => [
'plug=500',
'x'
]
},
'args' => [
'ip=192.168.15.38',
'username=fäöµ€',
'password=12"345'
]
}
},
'priority' => 0
}
};
which makes the parser fail since the input is not properly quoted,
dropping the args altogether!?
$ cat /etc/pve/ha/fence.cfg
device pve_nina:1 fence_pve 'ip=192.168.15.38' 'username=fäöµ€'
'password=12"345'
connect pve_nina:1 node=node1 plug=500 x
connect pve_nina:1 node=node2
$ perl -e 'use strict; use warnings; use Data::Dumper; use PVE::Cluster; use
PVE::HA::FenceConfig; use PVE::HA::Config; PVE::Cluster::cfs_update(); my $cfg
= PVE::HA::Config::read_fence_config(); print Dumper($cfg), "\n";
PVE::HA::Config::write_fence_config($cfg);'
/etc/pve/ha/fence.cfg ignore line 4: connect pve_nina:1 node=node2
$VAR1 = {
'pve_nina' => {
'priority' => 0,
'sub_devs' => {
'1' => {
'args' => [
'ip=192.168.15.38',
'username=fäöµ€',
'password=12"345'
],
'node_args' => {
'node1' => [
'plug=500',
'x'
]
},
Re: [pve-devel] [PATCH ha-manager 4/4] add some infos about HW fencing to README
On March 27, 2019 5:42 pm, Thomas Lamprecht wrote:
missing S-O-B
missing information about the fencing mode semantics? there's just one
sentence in 'man datacenter.cfg', see patch #2/cover letter comments.
looks fine for git-level readme, would need more grammaer/style cleanup
for reference docs IMHO.
one nit inline, otherwise
Acked-by: Fabian Grünbichler
> ---
> README | 110 +
> 1 file changed, 110 insertions(+)
>
> diff --git a/README b/README
> index 1c5177f..dbf8d6a 100644
> --- a/README
> +++ b/README
> @@ -72,6 +72,116 @@ works using reliable HW fence devices.
>
> Above 'self fencing' algorithm does not work if you use this option!
>
> +== Hardware Fencing ==
> +
> +This is for the users who want to use external fence devices.
> +
> +While they may have some advantages like:
> +
> +- possible faster recovery as its normally faster than the lock timeout
> +
> +- soft watchdog can (!) be less reliable
> +
> +- maybe someone does not want to reset but rather cut the machine off for
> + investigation
> +
> +But also can introduce some disadvantages, like:
> +
> +- added complexity
> +
> +- depends on external device (which can also fail)
> +
> +=== Fence Agents Supported ===
> +
> +We package the fence-agents collection from ClusterLabs:
> +https://github.com/ClusterLabs/fence-agents
nit: no longer true (atm?)
> +apt install fence-agents
> +
> +This has a wide variety of supported fence agents and should cover all needs.
> +
> +If you must use special hardware you can write your own agent, simply follow
> +the specs from the ClusterLabs agent so that your and their agents are
> interface
> +compatible.
> +
> +At least they have to support the following parameters:
> +- node
> +- plug
> +- action
> +
> +
> +=== Fence Device Configuration ===
> +
> +We use the configuration schema from dlm.conf (see man dlm.conf),
> +this allows us to use a quite easily and simple configuration schema
> +while not restricting an user, as complex setups are still possible.
> +
> +The basic format looks like:
> +
> + device dev_name agent [args]
> + connect dev_name node=nodeid [args]
> + connect dev_name node=nodeid [args]
> + connect dev_name node=nodeid [args]
> +
> +Example:
> +
> + # simple devices
> + device first_dev fence_pve ip="192.168.XX.XX" password="12345" action=off
> + connect first_dev node=node1 plug=100
> + connect first_dev node=node2 plug=101
> + connect first_dev node=node3 plug=102
> +
> +But the schema is able to do more, e.g., parallel devices or even multiple
> +devices can be configured per node. They will be executed from top to bottom
> +until the first fence device (set) succeeds.
> +
> +NOTE: use the longopts where possible, e.g.: use username=foo instead of
> l=foo
> + if a option has only a short option and its boolean then you may use
> it.
> +
> +=== Internal Configuration Representation ===
> +
> +We use an hash with a structure like this:
> +
> +#my $test_fence_device = {
> +#foo_apc => { # <-- device name
> +#priority => 0,
> +#sub_devs => {
> +#1 => { # two parallel devices
> +#agent => "fence_apc",
> +#args => ['ip=192.168.1.50', 'username=fencing',
> 'password=12345', 'x', 'o=off'],
> +#node_args => {
> +#uno => [n=1', 's=1'],
> +#due => [n=2', 's=2'],
> +#tre => [n=3', 's=1'],
> +#},
> +#},
> +#2 => {
> +#agent => "fence_apc",
> +#args => ['ip=192.168.1.51', 'username=fencing',
> 'password="12345 asd"', 'x', 'o=off'],
> +#node_args => {
> +#uno => [n=1', 's=1'],
> +#due => [n=2', 's=2'],
> +#tre => [n=3', 's=1'],
> +#},
> +#},
> +#}
> +#},
> +#bar_device => { # second fence device
> +#priority => 1,
> +#sub_devs => {
> +#1 => {
> +#agent => "fence_pve",
> +#args => ['ip=192.168.1.18', 'password="12345 asd"', 'o=off'],
> +#node_args => {
> +#uno => [n=100', 's=1'],
> +#due => [n=101', 's=1'],
> +#tre => [n=102', 's=1'],
> +#},
> +#},
> +#}
> +#},
> +#};
> +
> +
> == Testing requirements ==
>
> We want to be able to simulate HA cluster, using a GUI. This makes it easier
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH ha-manager 3/4] send also email on hardware fence failure
On March 27, 2019 5:42 pm, Thomas Lamprecht wrote:
> We introduced sending an email when Fencing fails/succeeds in
> general some time ago, send now also an email if a HW fence
> fails
missing S-O-B
> ---
> src/PVE/HA/NodeStatus.pm | 7 ---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/src/PVE/HA/NodeStatus.pm b/src/PVE/HA/NodeStatus.pm
> index ca13f2f..c4d147c 100644
> --- a/src/PVE/HA/NodeStatus.pm
> +++ b/src/PVE/HA/NodeStatus.pm
> @@ -2,6 +2,7 @@ package PVE::HA::NodeStatus;
>
> use strict;
> use warnings;
> +
> use PVE::HA::Fence;
>
> use JSON;
> @@ -204,7 +205,6 @@ EOF
> $haenv->sendmail($mail_subject, $mail_text);
> };
>
> -
> # start fencing
> sub fence_node {
> my ($self, $node) = @_;
> @@ -230,8 +230,9 @@ sub fence_node {
>
> # bad fence.cfg or no devices and only hardware fencing configured
> if ($hw_fence_success < 0 && $fencing_mode eq 'hardware') {
> - $haenv->log('err', "Fencing of node '$node' failed and needs " .
> - "manual intervention!");
> + my $msg = "Fencing of node '$node' failed and needs manual
> intervention!";
> + $haenv->log('err', $msg);
> + &$send_fence_state_email($self, 'FAILED', $msg, $node);
seems like a good idea in general, but see patch #2 - this part gets
called over and over again if hardware fencing fails? e.g.,
./ha-tester.pl test-hw-fence1 --nodiff | grep emai
emai160node1/crm: FENCE: Try to fence node 'node3'
emai160node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai160node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai180node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai200node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai220node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai240node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai260node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai280node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai300node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai320node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai340node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai360node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai380node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai400node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai420node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai440node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai460node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai480node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai500node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai520node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai540node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai560node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai580node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai600node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai620node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai640node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai660node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai680node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
emai700node1/crm: FAILED: Fencing of node 'node3' failed and needs
manual intervention!
if I hardcode $hw_success to -1
or is this just an artifact of some simulator limitation?
> return 0;
> }
>
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH ha-manager 2/4] allow use of external fencing devices
On March 27, 2019 5:42 pm, Thomas Lamprecht wrote:
> A node now can be fenced with the use of external hardware fence
> devices.
> Those device can be configured at /etc/pve/ha/fence.cfg
> also the fencing option in the datacenter configuration file must
> be set to either 'hardware' or 'both', else configured devices
> will *not* be used.
>
> If hardware is selected as mode a valid device config *must* be
> made, the fencing will not be marked as successful even if the CRM
> could theoretical acquire the lock from the failed node!
> This is done as some setups may require a HW fence agent to cut the
> node off and where a watchdog which resets the node may be
> dangerous.
this is worded a big confusingly, but the gist is:
mode => hardware means the fence device must be configured correctly and
work for fencing to succeed, mode => both means a fallback to regular
CRM fencing (which means that 'both' is more like
'fallback_to_watchdog' or 'either') ?
>
> We always *must* acquire the log before we can mark the failed node
> as fenced and place it into the 'unknown' state and recover its
> services.
>
> The CRM bails out in case of an lost manager lock event where
> $manager->cleanup() gets called.
> There we kill all remaining open fence processes, if any,
> and reset the fence status.
>
> The currents masters manager class processes the running fencing
> jobs, this means picking up finished fence workers and evaluating
> their result.
>
> Now regressions test with faked virtual HW fence devices are also
> possible.
> The current virtual devices succeed always, this will be changed
> in a future patch to allow testing of more (dangerous) corner cases.
>
> Device can be configured in the testdir/fence.cfg file and follow
> the exactly same format as the real ones (see man dlm.conf)
some nits and questions inline
missing S-O-B
> ---
> src/PVE/HA/Manager.pm | 4 +-
> src/PVE/HA/NodeStatus.pm| 54 +++-
> src/test/test-hw-fence1/README | 1 +
> src/test/test-hw-fence1/cmdlist | 4 +
> src/test/test-hw-fence1/fence.cfg | 6 ++
> src/test/test-hw-fence1/hardware_status | 5 ++
> src/test/test-hw-fence1/log.expect | 53
> src/test/test-hw-fence1/manager_status | 1 +
> src/test/test-hw-fence1/service_config | 5 ++
> src/test/test-hw-fence2/README | 3 +
> src/test/test-hw-fence2/cmdlist | 5 ++
> src/test/test-hw-fence2/fence.cfg | 8 ++
> src/test/test-hw-fence2/hardware_status | 7 ++
> src/test/test-hw-fence2/log.expect | 110
> src/test/test-hw-fence2/manager_status | 1 +
> src/test/test-hw-fence2/service_config | 10 +++
> src/test/test-hw-fence3/README | 5 ++
> src/test/test-hw-fence3/cmdlist | 4 +
> src/test/test-hw-fence3/fence.cfg | 17
> src/test/test-hw-fence3/hardware_status | 5 ++
> src/test/test-hw-fence3/log.expect | 57
> src/test/test-hw-fence3/manager_status | 1 +
> src/test/test-hw-fence3/service_config | 5 ++
> 23 files changed, 369 insertions(+), 2 deletions(-)
> create mode 100644 src/test/test-hw-fence1/README
> create mode 100644 src/test/test-hw-fence1/cmdlist
> create mode 100644 src/test/test-hw-fence1/fence.cfg
> create mode 100644 src/test/test-hw-fence1/hardware_status
> create mode 100644 src/test/test-hw-fence1/log.expect
> create mode 100644 src/test/test-hw-fence1/manager_status
> create mode 100644 src/test/test-hw-fence1/service_config
> create mode 100644 src/test/test-hw-fence2/README
> create mode 100644 src/test/test-hw-fence2/cmdlist
> create mode 100644 src/test/test-hw-fence2/fence.cfg
> create mode 100644 src/test/test-hw-fence2/hardware_status
> create mode 100644 src/test/test-hw-fence2/log.expect
> create mode 100644 src/test/test-hw-fence2/manager_status
> create mode 100644 src/test/test-hw-fence2/service_config
> create mode 100644 src/test/test-hw-fence3/README
> create mode 100644 src/test/test-hw-fence3/cmdlist
> create mode 100644 src/test/test-hw-fence3/fence.cfg
> create mode 100644 src/test/test-hw-fence3/hardware_status
> create mode 100644 src/test/test-hw-fence3/log.expect
> create mode 100644 src/test/test-hw-fence3/manager_status
> create mode 100644 src/test/test-hw-fence3/service_config
>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index a6c9b8e..177dfc2 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -7,6 +7,7 @@ use Digest::MD5 qw(md5_base64);
> use PVE::Tools;
> use PVE::HA::Tools ':exit_codes';
> use PVE::HA::NodeStatus;
> +use PVE::HA::Fence;
>
> sub new {
> my ($this, $haenv) = @_;
> @@ -32,7 +33,8 @@ sub new {
> sub cleanup {
> my ($self) = @_;
>
> -# todo: ?
> +# reset pending fence jobs and node states
> +$self->{ns}->cleanup();
> }
>
> sub flush_master_status {
> diff --git a/src/PVE/HA/NodeStatus.pm b
Re: [pve-devel] [PATCH ha-manager 1/4] allow LRM lock stealing for fenced nodes
On March 27, 2019 5:42 pm, Thomas Lamprecht wrote:
> We are only allowed to recover (=steal) a service when we have its
> LRMs lock, as this guarantees us that even if said LRM comes up
> again during the steal operation the LRM cannot start the services
> when the service config still belongs to it for a short time.
>
> This is important, else we have a possible race for the resource
> which can result in a service started on the old (restarted) node
> and the node where the service was recovered too, which is really
> bad!
two very long sentences ;)
In order to recover (=steal) a service from a fenced node, we need to
own its LRM's lock. Otherwise, the fenced node could start the service
during the short timespan where it still owns the service config file.
This requirement prevents a race between the stealing node and the
fenced node - we never want to start a service on two nodes!
small nits inline, otherwise
Acked-by: Fabian Grünbichler
> ---
> src/PVE/HA/Env.pm | 4 ++--
> src/PVE/HA/Env/PVE2.pm | 4 ++--
> src/PVE/HA/Sim/Env.pm | 16 ++--
> 3 files changed, 14 insertions(+), 10 deletions(-)
>
> diff --git a/src/PVE/HA/Env.pm b/src/PVE/HA/Env.pm
> index bb37486..6b76496 100644
> --- a/src/PVE/HA/Env.pm
> +++ b/src/PVE/HA/Env.pm
> @@ -171,9 +171,9 @@ sub get_ha_agent_lock {
> # this should only get called if the nodes LRM gracefully shuts down with
> # all services already cleanly stopped!
> sub release_ha_agent_lock {
> -my ($self) = @_;
> +my ($self, $node) = @_;
>
> -return $self->{plug}->release_ha_agent_lock();
> +return $self->{plug}->release_ha_agent_lock($node);
> }
>
> # return true when cluster is quorate
> diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
> index 796acd9..4428ed4 100644
> --- a/src/PVE/HA/Env/PVE2.pm
> +++ b/src/PVE/HA/Env/PVE2.pm
> @@ -309,9 +309,9 @@ sub get_ha_agent_lock {
> # this should only get called if the nodes LRM gracefully shuts down with
> # all services already cleanly stopped!
nit: this comment is now no longer true? we also call it after HW fencing,
which is possibly at the other end of the forceful vs graceful spectrum ;)
nit: I'd actually prefer splitting the release and steal into two separate
subs, to make it obvious what is happening also at first glance at the
call site(s). they are short and trivial enough that the duplication
should be harmless, but they can of course also be wrappers around the
code below.
> sub release_ha_agent_lock {
> -my ($self) = @_;
> +my ($self, $node) = @_;
>
> -my $node = $self->nodename();
> +$node = $node || $self->nodename();
>
> return rmdir("$lockdir/ha_agent_${node}_lock");
> }
> diff --git a/src/PVE/HA/Sim/Env.pm b/src/PVE/HA/Sim/Env.pm
> index 22e13e6..171e486 100644
> --- a/src/PVE/HA/Sim/Env.pm
> +++ b/src/PVE/HA/Sim/Env.pm
> @@ -83,13 +83,17 @@ sub sim_get_lock {
> if (my $d = $data->{$lock_name}) {
> my $tdiff = $ctime - $d->{time};
>
> + my $manager_node = $data->{'ha_manager_lock'}->{node} || '';
> +
> + $res = 0;
> if ($tdiff > $self->{lock_timeout}) {
> $res = 1;
> - } elsif (($tdiff <= $self->{lock_timeout}) && ($d->{node} eq
> $nodename)) {
> - delete $data->{$lock_name};
> - $res = 1;
> } else {
> - $res = 0;
> + # if we aren't manager we may unlock only *our* lock
> + if ($d->{node} eq $nodename || $manager_node eq $nodename) {
> + delete $data->{$lock_name};
> + $res = 1;
> + }
> }
> }
>
> @@ -342,9 +346,9 @@ sub get_ha_agent_lock {
> # this should only get called if the nodes LRM gracefully shuts down with
> # all services already cleanly stopped!
> sub release_ha_agent_lock {
> -my ($self) = @_;
> +my ($self, $node) = @_;
>
> -my $node = $self->nodename();
> +$node = $node || $self->nodename();
>
> my $lock = $self->get_ha_agent_lock_name($node);
> return $self->sim_get_lock($lock, 1);
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu 1/7] Trigger pve-api-updates on update
On September 2, 2019 4:27 pm, Stefan Reiter wrote: > A QEMU update can change supported CPU flags, thus we need to restart > API services (especially pvestatd) to refresh their cached values. given the recent discussion about this exact change, we could also use PVE::QemuServer::kvm_user_version() to decide whether we want to reload the CPU flag information. it's one additional stat for each pvestatd update cycle, but given all the other stuff we do there, it's probably not that expensive ;) > Signed-off-by: Stefan Reiter > --- > debian/triggers | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 debian/triggers > > diff --git a/debian/triggers b/debian/triggers > new file mode 100644 > index 000..59dd688 > --- /dev/null > +++ b/debian/triggers > @@ -0,0 +1 @@ > +activate-noawait pve-api-updates > -- > 2.20.1 > > > ___ > pve-devel mailing list > pve-devel@pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH manager 2/7] Broadcast supported CPU flags
On September 2, 2019 4:27 pm, Stefan Reiter wrote:
> pvestatd will read supported CPU flags once on startup (since these
> never change during runtime, and QEMU updates trigger a service
> restart), then broadcasts them as a key-value pair to the cluster.
>
> Signed-off-by: Stefan Reiter
> ---
>
> Changes from RFC:
>
> * Cache joined value
> * Add eval around QEMU query
> * Detect value regression (e.g. "better" value available in cluster kv) and
> handle accordingly
>
>
> PVE/Service/pvestatd.pm | 28 ++--
> 1 file changed, 26 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> index e138b2e8..84258daf 100755
> --- a/PVE/Service/pvestatd.pm
> +++ b/PVE/Service/pvestatd.pm
> @@ -47,12 +47,15 @@ my $cmdline = [$0, @ARGV];
> my %daemon_options = (restart_on_error => 5, stop_wait_time => 5);
> my $daemon = __PACKAGE__->new('pvestatd', $cmdline, %daemon_options);
>
> +my $supported_cpuflags;
> +
> sub init {
> my ($self) = @_;
>
> $opt_debug = $self->{debug};
>
> PVE::Cluster::cfs_update();
> +update_supported_cpuflags();
but this only happens once, not on every status update cycle like all
the other update_foo subs? if we go the "cache based on
kvm_user_version" route, then this could actually move to update_status
> }
>
> sub shutdown {
> @@ -72,6 +75,25 @@ sub hup {
> $restart_request = 1;
> }
>
> +sub update_supported_cpuflags {
at least if this is reworked:
1.) get kvm_user_version
2.) check if cached kvm_user_version is different
3.) yes -> query_supported_cpu_flags, cache results + kvm_user_version
4.) yes -> handle regressions
5.) broadcast supported flags
> +eval {
> + $supported_cpuflags = join(" ",
> @{PVE::QemuServer::query_supported_cpu_flags()});
> +};
> +warn $@ if $@;
> +
> +# detect regression
> +if (!$supported_cpuflags || $supported_cpuflags eq '') {
> + my $prev_cpuflags = PVE::Cluster::get_node_kv('cpuflags',
> $nodename)->{$nodename};
> + if ($prev_cpuflags && $prev_cpuflags ne '') {
> + $supported_cpuflags = $prev_cpuflags;
> + warn "CPU flag detection failed, using old values\n";
> + } else {
> + $supported_cpuflags = '';
> + warn "CPU flag detection failed and no previous values found\n";
> + }
> +}
> +}
> +
> my $generate_rrd_string = sub {
> my ($data) = @_;
>
> @@ -91,7 +113,9 @@ sub update_node_status {
>
> my $cpuinfo = PVE::ProcFSTools::read_cpuinfo();
>
> -my $maxcpu = $cpuinfo->{cpus};
> +my $maxcpu = $cpuinfo->{cpus};
> +
> +PVE::Cluster::broadcast_node_kv('cpuflags', $supported_cpuflags);
this can then be replaced with a call to update_supported_cpuflags
>
> my $subinfo = PVE::INotify::read_file('subscription');
> my $sublevel = $subinfo->{level} || '';
> @@ -104,7 +128,7 @@ sub update_node_status {
> $netin += $netdev->{$dev}->{receive};
> $netout += $netdev->{$dev}->{transmit};
> }
> -
> +
> my $meminfo = PVE::ProcFSTools::read_meminfo();
>
> my $dinfo = df('/', 1); # output is bytes
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server 3/7] Add QEMU CPU flag querying helpers
On September 2, 2019 4:27 pm, Stefan Reiter wrote:
> * query_understood_cpu_flags returns all flags that QEMU/KVM knows about
> * query_supported_cpu_flags returns all flags that QEMU/KVM can use on
> this particular host.
>
> To get supported flags, a temporary VM is started with QEMU, so we can
> issue the "query-cpu-model-expansion" QMP command. This is how libvirt
> queries supported flags for its "host-passthrough" CPU type.
>
> query_supported_cpu_flags is thus rather slow and shouldn't be called
> unnecessarily.
>
> Signed-off-by: Stefan Reiter
> ---
>
> Changes from RFC:
>
> * Clearer regexes
> * Use existing QMP infrastructure
> * Add locking for temporary VM start
> * Add comments
>
>
> PVE/QemuServer.pm | 84 +++
> 1 file changed, 84 insertions(+)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 8c519b5..97fa955 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -3500,6 +3500,90 @@ sub get_command_for_arch($) {
> return $cmd;
> }
>
> +# The format of the flags returned here uses dashes '-' as seperators,
nit: s/seperator/separator/a
> +# this is neither the default for 'query_supported_cpu_flags' below, nor for
> +# /proc/cpuinfo.
> +#
> +# To compare (or array_intersect) flags, it's a good idea to convert them all
> +# to a common format first (e.g. map s/\.|-/_/g).
> +sub query_understood_cpu_flags {
not used anywhere in this series? also, confusingly, this returns less
values then the other helper (maybe because of some aliases?), and it's
not a strict subset :-/
I guess this is in preparation of exposing it via the API? some sort of
comment how _understood and _supported interact/are supposed to interact
would be nice. either as comment or part of the commit message
> +my $flags = [];
> +my $flag_section = 0;
> +
> +run_command(
> + [get_command_for_arch('x86_64'), '-cpu', 'help'],
why not get_host_arch()?
> + noerr => 1,
> + quiet => 1,
> + outfunc => sub {
> + my ($line) = @_;
> +
> + if ($flag_section) {
> + return if $line =~ m/^\s*$/;
> + $line =~ s/^\s*|\s*$//g;
> + push @$flags, split(/\s+/, $line);
> + } elsif ($line =~ m/^\s*Recognized CPUID flags:\s*$/) {
> + $flag_section = 1;
> + }
> + }
> +);
> +
> +return $flags;
> +}
> +
> +# This function needs to start a temporary VM and is therefore rather
> expensive.
> +# If you need the value returned from this, you can get it much cheaper from
> +# pvestatd using PVE::Cluster::get_node_kv('cpuflags').
s/pvestatd/pmxcfs/
pvestatd is just the one putting it there ;)
> +#
> +# See also note to query_understood_cpu_flags above.
> +sub query_supported_cpu_flags {
> +my $flags = [];
> +
> +my $vmid = -1;
> +my $pidfile = pidfile_name($vmid);
> +
> +PVE::QemuConfig->lock_config($vmid, sub {
> + # We start a temporary (frozen) VM with vmid -1 to allow us to send a
> QMP command
> + my $rc = run_command([
> + get_command_for_arch('x86_64'),
again, why not get_host_arch()?
> + '-cpu', 'kvm64',
> + '-display', 'none',
> + '-chardev',
> "socket,id=qmp,path=/var/run/qemu-server/$vmid.qmp,server,nowait",
> + '-mon', 'chardev=qmp,mode=control',
> + '-pidfile', $pidfile,
> + '-S', '-daemonize'
> + ], noerr => 1, quiet => 1);
> + return if $rc;
> +
> + my $cmd_result = vm_mon_cmd_nocheck(
> + $vmid,
> + 'query-cpu-model-expansion',
> + type => 'full',
> + model => { name => 'host' }
> + );
> +
> + my $props = $cmd_result->{model}->{props};
> + if (%$props) {
> + foreach my $prop (keys %$props) {
> + push @$flags, $prop if "$props->{$prop}" eq '1';
> + }
> + }
> +
> + # force stop with 10 sec timeout and 'nocheck'
> + vm_stop(undef, $vmid, 1, 1, 10, 0, 1);
> +});
> +
> +# QEMU returns some flags multiple times, with '_', '.' or '-' as
> seperator
> +# (e.g. lahf_lm and lahf-lm; sse4.2, sse4-2 and sse4_2; ...).
> +# We only keep those with underscores, since they match the ones from
> +# /proc/cpuinfo (they do the same thing, but we get rid of duplicates).
> +@$flags = grep {
> + my $replaced = (my $underscore = $_) =~ s/\.|-/_/g;
> + !($replaced && grep { $_ eq $underscore } @$flags)
> +} @$flags;
> +
> +return $flags;
> +}
> +
> sub get_cpu_options {
> my ($conf, $arch, $kvm, $machine_type, $kvm_off, $kvmver, $winversion,
> $gpu_passthrough) = @_;
>
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server 4/7] Add CustomCPUConfig for storing/parsing custom CPU models
On September 2, 2019 4:27 pm, Stefan Reiter wrote:
> Inherits from SectionConfig to provide base parsing infrastructure.
>
> Use with helper functions:
> * config_from_file gives bless'd config
> * get_model_by_name returns a "formatted" hash for a single CPU model
> * config_to_file writes changes back
>
> File reads are cached in a local hash.
high-level:
use cfs_register/write/read_file please (it's our mechanism for handling
config files on pmxcfs after all ;))
is there a reason you need a class-like interface here? we usually use
that if we want to have multiple implementations of a certain interface
with a common shared code base, but that is not the case here..
>
> Signed-off-by: Stefan Reiter
> ---
>
> This will definitely require some sort of versioning mechanism, otherwise CPU
> definitions could be changed after starting a VM, thus breaking live-migration
> by starting the migration-target with different parameters.
>
> Hints, ideas, recommendations?
versioning is one possibility (cumbersome with SectionConfig though).
another would be to retrieve the CPU model from the running Qemu
instance, and convert that back to a '-cpu ...' string for starting the
target instance. if nothing hot-pluggable can change the generated -cpu
string, we could also record that somewhere (or take it from
/proc/$pid/cmdline) and override the target instance with that.
> PVE/QemuServer/CustomCPUConfig.pm | 129 ++
> PVE/QemuServer/Makefile | 1 +
> 2 files changed, 130 insertions(+)
> create mode 100644 PVE/QemuServer/CustomCPUConfig.pm
>
> diff --git a/PVE/QemuServer/CustomCPUConfig.pm
> b/PVE/QemuServer/CustomCPUConfig.pm
> new file mode 100644
> index 000..87ba9e6
> --- /dev/null
> +++ b/PVE/QemuServer/CustomCPUConfig.pm
> @@ -0,0 +1,129 @@
> +package PVE::QemuServer::CustomCPUConfig;
> +
> +use strict;
> +use warnings;
> +use PVE::Tools qw(file_get_contents file_set_contents);
> +
> +use base qw(PVE::SectionConfig);
> +
> +my $defaultData = {
> +propertyList => {
> + basemodel => {
> + description => "Emulated CPU type to inherit defaults from.",
> + type => 'string',
> + format_description => 'string',
> + default => 'kvm64',
should have the same pattern/format as QemuServer's $cpu_fmt->{cputype}.
in other words, it probably makes sense to extract and re-use that ;)
> + },
> + flags => {
> + description => "List of additional CPU flags separated by ';'."
> + . " Use '+FLAG' to enable, '-FLAG' to disable a flag."
> + . " Supports all flags supported by QEMU/KVM.",
> + format_description => '+FLAG[;-FLAG...]',
> + type => 'string',
> + pattern => qr/[+-][a-zA-Z0-9\-_\.]+(;[+-][a-zA-Z0-9\-_\.]+)*/,
same here (different description, please just use a single definition
for both instances)
> + optional => 1,
> + },
> + 'phys-bits' => {
> + type => 'integer',
> + minimum => 1,
> + maximum => 64,
> + optional => 1,
> + description => "The physical memory bits that are reported to the
> guest OS. Must be smaller or equal to the host.",
this one might make sense to also allow in the regular, per-VM 'cpu'
option (and then we could again re-use the definition)
> + },
> + 'host-phys-bits' => {
> + type => 'boolean',
> + default => 0,
> + description => "Whether to report the host's physical memory bits.
> Overrides 'phys-bits' when set.",
> + },
same
> + vendor => {
> + type => 'string',
> + enum => [qw(default AuthenticAMD GenuineIntel)],
> + default => 'default',
> + description => "The CPU vendor to report. 'default' uses the
> host's.",
> + },
> +}
this probably only makes sense for the custom types. the default value
means that you never inherit the vendor from your basemodel - is that
intended?
there are two more from the regular one that we might want to include in
custom ones as well:
hidden
hv-vendor-id
I am not sure whether just moving some format definitions, or moving
all/most CPU related stuff to a new module (e.g.,
QemuServer/CPUConfig.pm instead of QemuServer/CustomCPUConfig.pm) is
more clean / feasible. the latter probably would require moving some
helpers from QemuServer.pm to another new module as well, as they'd be
used by the CPU module and QemuServer.pm
I think we all agree that QemuServer.pm is way too big, so a patch
(series) that adds new features and reduces that bloat would be more
than welcome :D but if you think it's too much for this series, we can
also do it as a follow-up since it's mostly about moving code and format
definitions.
> +};
> +
> +sub private {
> +return $defaultData;
> +}
> +
> +sub options {
> +return {
> + basemodel => { optional => 1 },
> + flags => { optional => 1 },
> + 'phys-bits' => { optional => 1 },
> +
Re: [pve-devel] [PATCH qemu-server 6/7] Handle CPU flags defined in custom CPU type
On September 2, 2019 4:27 pm, Stefan Reiter wrote:
> Special care is taken not to overwrite any special flags, or ones
> manually set on the VM by the user. We warn if a flag is overruled.
hmm. I am unsure whether I like that behaviour or not. it's a bit
strange that VM specific flags get applied earlier, and then we skip
those from the models instead of just switching the order around.
maybe we need some other intermediate data structure, or some helper
methods?
the following seems more logical to me:
1.) add flags from custom model
2.) add flags from VM config
3.) (selectively) add auto-generated flags (based on other settings, OS, ..)
at each step we want to remove conflicting flags from before (this would
be where a different datastructure or a helper would come into play).
maybe some of the auto-generated flags should not override explicitly
set ones, but be skipped instead (at least for explicit negative
flags, to allow opt-out).
>
> Signed-off-by: Stefan Reiter
> ---
> PVE/QemuServer.pm | 26 +-
> 1 file changed, 25 insertions(+), 1 deletion(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 417bea8..7cc1674 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -3597,6 +3597,7 @@ sub get_cpu_options {
> }
> my $hv_vendor_id;
> my $custom_cpu_config;
> +my @vm_specific_flags;
> if (my $cputype = $conf->{cpu}) {
> my $cpuconf = PVE::JSONSchema::parse_property_string($cpu_fmt, $cputype)
> or die "Cannot parse cpu description: $cputype\n";
> @@ -3624,7 +3625,8 @@ sub get_cpu_options {
> $hv_vendor_id = $cpuconf->{'hv-vendor-id'};
>
> if (defined(my $flags = $cpuconf->{flags})) {
> - push @$cpuFlags, split(";", $flags);
> + @vm_specific_flags = split(";", $flags);
> + push @$cpuFlags, @vm_specific_flags;
> }
> }
>
> @@ -3649,6 +3651,28 @@ sub get_cpu_options {
>
> push @$cpuFlags, 'kvm=off' if $kvm_off;
>
> +if (defined($custom_cpu_config) && defined($custom_cpu_config->{flags}))
> {
> + my @custom_flags = split(/;/, $custom_cpu_config->{flags});
> + foreach my $flag (@custom_flags) {
> + # find index of $flag in $cpuFlags while ignoring prefix [+-=]
> + $flag =~ m/^[+-=]?(.*)$/;
> + my $match_flag = $1;
> + my @match_index = grep { $cpuFlags->[$_] =~ m/[+-=]?\Q$match_flag/ }
> + 0..scalar(@$cpuFlags)-1;
> +
> + if (@match_index) {
> + my $other_flag = $cpuFlags->[$match_index[0]];
> + # warn only if prefix differs and flag is not vm specific
> + warn "warning: custom CPU model flag '$flag' overruled by
> '$other_flag' due to either default handling of basemodel or selected OS.\n"
I don't understand that message, so probably users won't either ;) but I
think the messages would be more clear if we apply the auto-generated
flags last:
"unsetting flag '$other_flag', not compatible with " (OS foo, CPU bar,
...)
"not setting flag '$flag', explicit '$other_flag' found." (explicit
opt-out for auto-generated flags via VM or CPU model config)
> + if (substr $other_flag, 0, 1) ne (substr $flag, 0, 1) &&
> + (!@vm_specific_flags ||
> + !grep(m/[+-=]?\Q$match_flag/, @vm_specific_flags));
> + } else {
> + push(@$cpuFlags, $flag);
> + }
> + }
> +}
> +
> if (defined($custom_cpu_config) && (my $custom_vendor =
> $custom_cpu_config->{vendor})) {
> push @$cpuFlags, "vendor=${custom_vendor}"
> if $custom_vendor ne 'default';
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server 5/7] Support custom CPU types in get_cpu_options
On September 2, 2019 4:27 pm, Stefan Reiter wrote:
> Supports custom basemodels (model shown to QEMU, i.e. must be a default
nit: s/shown/known/ ?
high-level: if we allow basing custom models on other custom models,
wouldn't we need to compute an effective model first, and then use that?
e.g., if I define the following:
cpu-model: leaf
basemodel: intermediate
vendor: intel
flags: +aes,+pcid
host-phys-bits: 1
cpu-model: intermediate1
vendor: amd
basemodel: intermediate2
flags: -aes;-ssbd
phys-bits: 32
cpu-model: intermediate2
vendor: amd
basemodel: phenom
flags: +ssbd;+md-clear
phys-bits: 48
I'd expect the end result to be:
vendor: intel
flags: phenom +aes +pcid -ssbd +md-clear
phys-bits: host (overriding the one from intermediate)
(ignore that those flags don't make much sense ;))
this patch (series) only looks at the leaf and the last basemodel in the
chain, and ignores everything inbetween. vendor also can't be
'inherited' from basemodel, because it is always set.
we can either go full-blown 'inherit properties across multiple
basemodels', or we limit ourselves to 'basemodel must be a regular cpu
model'. the former is of course more flexible (allowing stuff like
re-using a custom model and just adding one flag, without duplication),
but the latter is a lot more simple (one additional level to check
instead of recursive walk, loop and conflict detection, no nested
versioning needed, etc.pp.). something inbetween like the current patch
does is probably confusing.
> model), vendors and (host-)phys-bits for VMs with large amounts of RAM
> (see bug #2318).
>
> Signed-off-by: Stefan Reiter
> ---
> PVE/QemuServer.pm | 32 +++-
> 1 file changed, 31 insertions(+), 1 deletion(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 97fa955..417bea8 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -37,6 +37,7 @@ use PVE::QemuServer::PCI qw(print_pci_addr print_pcie_addr);
> use PVE::QemuServer::Memory;
> use PVE::QemuServer::USB qw(parse_usb_device);
> use PVE::QemuServer::Cloudinit;
> +use PVE::QemuServer::CustomCPUConfig;
> use PVE::SysFSTools;
> use PVE::Systemd;
> use Time::HiRes qw(gettimeofday);
> @@ -3595,10 +3596,30 @@ sub get_cpu_options {
> $cpu = 'cortex-a57';
> }
> my $hv_vendor_id;
> +my $custom_cpu_config;
> if (my $cputype = $conf->{cpu}) {
> my $cpuconf = PVE::JSONSchema::parse_property_string($cpu_fmt, $cputype)
> or die "Cannot parse cpu description: $cputype\n";
> $cpu = $cpuconf->{cputype};
> +
> + if (!defined($cpu_vendor_list->{$cpu})) {
> + # not a default CPU model, read config and see if $cpu is name of
> + # a custom model
> + my $cpu_models =
> PVE::QemuServer::CustomCPUConfig::config_from_file();
> + if ($cpu_models && ($custom_cpu_config =
> $cpu_models->get_model_by_name($cpu))) {
> + $cpu = $custom_cpu_config->{basemodel};
> +
> + # 'basemodel' could be custom as well
> + while (!defined($cpu_vendor_list->{$cpu})) {
> + my $custom_base = $cpu_models->get_model_by_name($cpu);
> + die "unknown CPU basemodel: $cpu\n" if !$custom_base;
> + $cpu = $custom_base->{basemodel};
> + }
possible endless loop
> + } else {
> + die "unknown CPU model (neither default nor custom): $cpu\n";
> + }
> + }
> +
> $kvm_off = 1 if $cpuconf->{hidden};
> $hv_vendor_id = $cpuconf->{'hv-vendor-id'};
>
> @@ -3628,13 +3649,22 @@ sub get_cpu_options {
>
> push @$cpuFlags, 'kvm=off' if $kvm_off;
>
> -if (my $cpu_vendor = $cpu_vendor_list->{$cpu}) {
> +if (defined($custom_cpu_config) && (my $custom_vendor =
> $custom_cpu_config->{vendor})) {
> + push @$cpuFlags, "vendor=${custom_vendor}"
> + if $custom_vendor ne 'default';
> +} elsif (my $cpu_vendor = $cpu_vendor_list->{$cpu}) {
> push @$cpuFlags, "vendor=${cpu_vendor}"
> if $cpu_vendor ne 'default';
> } elsif ($arch ne 'aarch64') {
> die "internal error"; # should not happen
> }
>
> +$cpu .= ",host-phys-bits=true"
> + if defined($custom_cpu_config) &&
> $custom_cpu_config->{'host-phys-bits'};
> +$cpu .= ",phys-bits=$custom_cpu_config->{'phys-bits'}"
> + if defined($custom_cpu_config) && $custom_cpu_config->{'phys-bits'}
> + && !$custom_cpu_config->{'host-phys-bits'};
> +
> $cpu .= "," . join(',', @$cpuFlags) if scalar(@$cpuFlags);
>
> return ('-cpu', $cpu);
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server 7/7] Allow custom CPU types in API
On September 2, 2019 4:27 pm, Stefan Reiter wrote:
> Custom CPU types can be specified via the API, but to prevent arbitrary
> ones we have to manually check if the given model exists (as default or
> custom).
>
> Signed-off-by: Stefan Reiter
> ---
> PVE/QemuServer.pm | 31 +--
> 1 file changed, 29 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 7cc1674..5bdd729 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -187,7 +187,8 @@ my $cpu_fmt = {
> cputype => {
> description => "Emulated CPU type.",
> type => 'string',
> - enum => [ sort { "\L$a" cmp "\L$b" } keys %$cpu_vendor_list ],
> + format_description => 'string',
> + pattern => qr/[0-9a-zA-Z\-_]+/,
this removes information from our API schema dumps (e.g., API viewer,
'man qm', ...). we could at least put it into format_description with
some additional sentence referencing custom types)
> default => 'kvm64',
> default_key => 1,
> },
> @@ -215,6 +216,32 @@ my $cpu_fmt = {
> },
> };
>
> +# we need to verify the "cpu" property of the VM config manually, since a
> +# syntactially valid, but non-existant CPU model might be given
> +PVE::JSONSchema::register_format('pve-cpu-conf', \&verify_cpu);
> +sub verify_cpu {
> +my ($cpu, $noerr) = @_;
> +
> +my $conf;
> +eval {
> + $conf = PVE::JSONSchema::parse_property_string($cpu_fmt, $cpu);
> +};
> +if ($@) {
> + die $@ if !$noerr;
> + return undef;
> +}
> +
> +my $cputype = $conf->{cputype};
> +
> +return $cpu if defined($cpu_vendor_list->{$cputype});
> +
> +my $cpu_models = PVE::QemuServer::CustomCPUConfig::config_from_file();
> +return $cpu if $cpu_models && $cpu_models->get_model_by_name($cputype);
and here I just realized something that affects the whole patch series -
we probably need to namespace the custom cpu models somehow when
referencing them in the config. otherwise a future QEMU version might
add an additional CPU model that conflicts with a defined custom one.
we can either add a flag to 'cpu', just prefix the names in the VM
config, or prefix the names both in the VM config and in the CPU model
config. all approaches have pros and cons - maybe there is some other
clever way to avoid this problem?
> +
> +die "cputype '$cputype' not found\n" if !$noerr;
> +return undef;
> +}
> +
> my $watchdog_fmt = {
> model => {
> default_key => 1,
> @@ -587,7 +614,7 @@ EODESCR
> optional => 1,
> description => "Emulated CPU type.",
> type => 'string',
> - format => $cpu_fmt,
> + format => 'pve-cpu-conf',
> },
> parent => get_standard_option('pve-snapshot-name', {
> optional => 1,
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH 0/7] Add basics for custom CPU models
On September 2, 2019 4:27 pm, Stefan Reiter wrote: > Based on the RFC and following on- and off-list discussion about custom CPU > models [0]. > > In essence, this revised patch allows a user to specify custom CPU models in > /etc/pve/cpu-models.conf (section-config style [1]), where VMs using that CPU > model inherit details from the definition. This removes any fragile > "auto-magical" CPU flag detection, while still giving the user a way to create > VMs with the best possible subset of CPU features maintaining live-migration > compatibility. > > Includes the infrastructure for broadcasting supported CPU flags for each > cluster-node via the key-value store - this is not necessary for the > custom-cpu feature in particular, but I think could prove useful for > implementing the GUI part (e.g. show the user which flags are supported on > which > nodes). > > I intentionally wanted to send this series before starting any GUI or new API > work, to get some feedback if this approach works better than the cluster-cpu > one. > > [0]: https://pve.proxmox.com/pipermail/pve-devel/2019-July/038268.html > [1]: e.g.: > cpu-model: custom-cpu-name > host-phys-bits 1 > flags +aes;+avx;+avx2 > basemodel kvm64 some more detailed feedback on individual patches, here a summary of open questions from my side: - nested models, or just custom models derived from qemu-provided base model - namespacing of custom models? where, how, ...? - versioning/live-migration and some things for the 'future' part: - permissions? who can create a custom model? - is there a security risk associated with any of the flags (besides making the VM itself easier to attack)? do we want to have some global whitelist of flags that we have closely looked at and deemed unproblematic? - do we want to make the whole CRUD of models root/admin only (probably need to, unless we want to create permission paths for CPU models?) I like the direction this is going, although a lot of the magic will be in how to make it straight-forward to generate a new model using the available information, which is not yet part of this series ;) > > > qemu: Stefan Reiter (1): > Trigger pve-api-updates on update > > debian/triggers | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 debian/triggers > > manager: Stefan Reiter (1): > Broadcast supported CPU flags > > PVE/Service/pvestatd.pm | 28 ++-- > 1 file changed, 26 insertions(+), 2 deletions(-) > > qemu-server: Stefan Reiter (5): > Add QEMU CPU flag querying helpers > Add CustomCPUConfig for storing/parsing custom CPU models > Support custom CPU types in get_cpu_options > Handle CPU flags defined in custom CPU type > Allow custom CPU types in API > > PVE/QemuServer.pm | 173 +- > PVE/QemuServer/CustomCPUConfig.pm | 129 ++ > PVE/QemuServer/Makefile | 1 + > 3 files changed, 299 insertions(+), 4 deletions(-) > create mode 100644 PVE/QemuServer/CustomCPUConfig.pm > > -- > 2.20.1 > > ___ > pve-devel mailing list > pve-devel@pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server 3/7] Add QEMU CPU flag querying helpers
On September 9, 2019 2:56 pm, Stefan Reiter wrote:
> On 9/6/19 1:42 PM, Fabian Grünbichler wrote:
>>> +# this is neither the default for 'query_supported_cpu_flags' below, nor
>>> for
>>> +# /proc/cpuinfo.
>>> +#
>>> +# To compare (or array_intersect) flags, it's a good idea to convert them
>>> all
>>> +# to a common format first (e.g. map s/\.|-/_/g).
>>> +sub query_understood_cpu_flags {
>>
>> not used anywhere in this series? also, confusingly, this returns less
>> values then the other helper (maybe because of some aliases?), and it's
>> not a strict subset :-/
>>
>
> Yes, this is for exposing via the API (for the GUI to create/edit custom
> models I have in mind). I just thought it fits with this patch well,
> since the comments refer to each other.
>
> Also, query_understood_cpu_flags returns 188 flags for me, while
> query_supported_cpu_flags only returns 104...
yes, a double check confirms this - seems I had the output files mixed
up.. sorry for the noise
>
> The discrepancies generally arise because
> a) query_understood_cpu_flags returns flags the host cannot use and
> b) query_supported_cpu_flags (rather the QMP call) doesn't actually
> return CPU flags, but CPU settings - with most of them being flags.
> Those settings (and some flags, curiously) cannot be specified as a
> "-cpu" argument however.
that makes sense - and would make a good starting point for an
informative comment ;)
>
> The intersection of those provides only flags that can be passed to
> "-cpu" without issues in my testing. Since query_understood_cpu_flags is
> not host specific though, it doesn't need broadcasting.
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container] fix issue where ttys aren't correctly set after restore
NAK, see inline
On September 10, 2019 3:27 pm, Oguz Bektas wrote:
> restore from unpriv to priv causes a problem with the log-in, since the
> /etc/securetty file isn't modified after a restore to reflect the change
> (/dev/lxc/tty1 and so on).
>
> template_fixup is normally called in post_create_hook, but we have no
> $password
> or $ssh_keys to call the hook with during restore. instead we call
> template_fixup by
> itself to fix the ttys on some distributions.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/API2/LXC.pm | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm
> index 26c4f88..738556b 100644
> --- a/src/PVE/API2/LXC.pm
> +++ b/src/PVE/API2/LXC.pm
> @@ -413,10 +413,11 @@ __PACKAGE__->register_method({
> $bwlimit = PVE::Storage::get_bandwidth_limit('restore',
> [keys %used_storages], $bwlimit);
> PVE::LXC::Create::restore_archive($archive, $rootdir,
> $conf, $ignore_unpack_errors, $bwlimit);
>
> + my $lxc_setup = PVE::LXC::Setup->new($conf, $rootdir); #
> detect OS
this makes decisions based on $conf (e.g., explicit 'ostype: unmanaged')
> if ($restore) {
> PVE::LXC::Create::restore_configuration($vmid,
> $rootdir, $conf, !$is_root, $unique, $skip_fw_config_restore);
when restoring, the full $conf is only available here.
> + $lxc_setup->template_fixup($conf);
> } else {
> - my $lxc_setup = PVE::LXC::Setup->new($conf, $rootdir);
> # detect OS
> PVE::LXC::Config->write_config($vmid, $conf); # safe
> config (after OS detection)
> $lxc_setup->post_create_hook($password, $ssh_keys);
> }
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH guest-common 1/2] move pending changes related functions into AbstractConfig
meta: please send v2 together with v2 of pve-container (and future) patches.
On September 4, 2019 6:00 pm, Oguz Bektas wrote:
> some functions from Qemu w.r.t. pending changes can be moved to
> AbstractConfig, in order to make them available for both QemuConfig and
> LXC::Config.
>
> Signed-off-by: Oguz Bektas
> ---
> PVE/AbstractConfig.pm | 79 +++
> 1 file changed, 79 insertions(+)
>
> diff --git a/PVE/AbstractConfig.pm b/PVE/AbstractConfig.pm
> index e0d0f10..18522b9 100644
> --- a/PVE/AbstractConfig.pm
> +++ b/PVE/AbstractConfig.pm
> @@ -68,6 +68,85 @@ sub write_config {
> PVE::Cluster::cfs_write_file($cfspath, $conf);
> }
>
> +## Pending changes related
> +
> +sub split_flagged_list {
> +my ($class, $text) = @_;
> +$text ||= '';
> +$text =~ s/[,;]/ /g;
> +$text =~ s/^\s+//;
> +return { map { /^(!?)(.*)$/ && ($2, $1) } ($text =~ /\S+/g) };
> +}
> +
> +sub join_flagged_list {
> +my ($class, $how, $lst) = @_;
> +join $how, map { $lst->{$_} . $_ } keys %$lst;
> +}
these two are really only used for the pending delete hash, so instead
of just moving them with this strange naming scheme, why not simplify
and rename them to
parse_pending_delete($class, $raw)
print_pending_delete($class, $pending_delete_hash)
we need versioned breaks/depends anyway.. while we are at it, instead of
$hash->{foo} being either '!' or '' (which is just how the $force
property is encoded in the 'flagged list', but not a good representation
in a perl hash), it would be nicer to have
$hash->{foo}->{force} being 0/1 IMHO.
> +
> +sub vmconfig_delete_pending_option {
> +my ($class, $conf, $key, $force) = @_;
> +
> +delete $conf->{pending}->{$key};
> +my $pending_delete_hash =
> $class->split_flagged_list($conf->{pending}->{delete});
> +$pending_delete_hash->{$key} = $force ? '!' : '';
> +$conf->{pending}->{delete} = $class->join_flagged_list(',',
> $pending_delete_hash);
> +}
> +
> +sub vmconfig_undelete_pending_option {
> +my ($class, $conf, $key) = @_;
> +
> +my $pending_delete_hash =
> $class->split_flagged_list($conf->{pending}->{delete});
> +delete $pending_delete_hash->{$key};
> +
> +if (%$pending_delete_hash) {
> + $conf->{pending}->{delete} = $class->join_flagged_list(',',
> $pending_delete_hash);
> +} else {
> + delete $conf->{pending}->{delete};
> +}
> +}
ugh, those two are also not very well named. not your fault, but IMHO
also an opportunity to think about a better name if we refactor this.
there are only 9 existing callers after all..
they don't delete or undelete a pending option (whatever that would
actually do ;)), they queue an option for deletion via the pending
delete hash/list..
add_to_pending_delete
remove_from_pending_delete
? better proposals welcome, like always ;)
> +
> +sub vmconfig_cleanup_pending {
I'd drop the 'vmconfig_' from the name here
> +my ($class, $conf) = @_;
> +
> +# remove pending changes when nothing changed
> +my $changes;
> +foreach my $opt (keys %{$conf->{pending}}) {
> + if (defined($conf->{$opt}) && ($conf->{pending}->{$opt} eq
> $conf->{$opt})) {
> + $changes = 1;
> + delete $conf->{pending}->{$opt};
> + }
> +}
> +
> +my $current_delete_hash =
> $class->split_flagged_list($conf->{pending}->{delete});
> +my $pending_delete_hash = {};
> +while (my ($opt, $force) = each %$current_delete_hash) {
> + if (defined($conf->{$opt})) {
> + $pending_delete_hash->{$opt} = $force;
> + } else {
> + $changes = 1;
> + }
> +}
> +
> +if (%$pending_delete_hash) {
> + $conf->{pending}->{delete} = $class->join_flagged_list(',',
> $pending_delete_hash);
> +} else {
> + delete $conf->{pending}->{delete};
> +}
> +
> +return $changes;
> +}
> +
> +sub vmconfig_hotplug_pending {
same - although I am not sure whether I'd not opt for this sub to stay
in QemuServer.pm altogether
the interdependency between QemuServer and QemuConfig is already way too
messy for my taste, and moving this more than doubles the calls from
QemuConfig to QemuServer:
old:
QemuServer->QemuConfig: 70 (these are fine!)
QemuConfig->QemuServer: 29 (these are only there because we did not want
to move 60% of QemuServer.pm including the $confdesc to QemuConfig)
new:
QemuServer->QemuConfig: 59 (-11)
QemuConfig->QemuServer: 69 (+40)
the -11 are actually all load/write_config..
moving this sub also adds another module dependency cycle:
QemuServer->QemuConfig->QemuServer::CloudInit->QemuServer
> +my ($class, $vmid, $conf, $storecfg, $selection, $errors) = @_;
> +die "implement me - abstract method\n";
> +}
> +
> +sub vmconfig_apply_pending {
same (name if moved, but maybe not move at all?)
> +my ($class, $vmid, $conf, $storecfg) = @_;
> +die "implement me - abstract method\n";
> +}
> +
> +
> # Lock config file using flock, run $code with @param, unlock conf
Re: [pve-devel] [PATCH container 2/9] adapt config GET call for taking pending changes
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> the default behaviour is the same as in Qemu, so without the 'current'
> flag set, current values will be replaced with their respective pending
> counterparts.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/API2/LXC/Config.pm | 23 +++
> 1 file changed, 23 insertions(+)
>
> diff --git a/src/PVE/API2/LXC/Config.pm b/src/PVE/API2/LXC/Config.pm
> index 769fc3b..6e67186 100644
> --- a/src/PVE/API2/LXC/Config.pm
> +++ b/src/PVE/API2/LXC/Config.pm
> @@ -53,6 +53,12 @@ __PACKAGE__->register_method({
> items => { type => 'array', items => { type => 'string' }},
> optional => 1,
> },
> + current => {
> + description => "Get current values (instead of pending
> values).",
> + optional => 1,
> + default => 0,
> + type => 'boolean',
> + },
did you test this? the new parameter is in the return property instead
of the parameters property..
> digest => {
> type => 'string',
> description => 'SHA1 digest of configuration file. This can be
> used to prevent concurrent modifications.',
> @@ -73,11 +79,28 @@ __PACKAGE__->register_method({
> $conf = $snapshot;
> }
>
> + # take pending changes in
> + if (!$param->{current}) {
> + foreach my $opt (keys %{$conf->{pending}}) {
> + next if $opt eq 'delete';
> + my $value = $conf->{pending}->{$opt};
> + next if ref($value); # just to be sure
> + $conf->{$opt} = $value;
> + }
> + my $pending_delete_hash =
> PVE::LXC::Config->split_flagged_list($conf->{pending}->{delete});
> + foreach my $opt (keys %$pending_delete_hash) {
> + delete $conf->{$opt} if $conf->{$opt};
> + }
> + }
> +
> +
> delete $conf->{snapshots};
> + delete $conf->{pending};
the whole thing is now identical (modulo comments and line ordering) to
the same Qemu API path, except that Qemu also masks the 'cipassword'
option if it is set.
we could instead move the whole 'delete snapshots, conditionally apply
or delete pending changes' into a new AbstractConfig method
(e.g., load_current_config($apply_pending)), or add two parameters to
load_config to get this reduced config instead of the full one.
diff --git a/config_ct b/config_vm
index 04dbfe6..7372ba3 100644
--- a/config_ct
+++ b/config_vm
@@ -1,18 +1,19 @@
code => sub {
my ($param) = @_;
- my $conf = PVE::LXC::Config->load_config($param->{vmid});
+ my $conf = PVE::QemuConfig->load_config($param->{vmid});
if (my $snapname = $param->{snapshot}) {
my $snapshot = $conf->{snapshots}->{$snapname};
die "snapshot '$snapname' does not exist\n" if !defined($snapshot);
- # we need the digest of the file
- $snapshot->{digest} = $conf->{digest};
+ $snapshot->{digest} = $conf->{digest}; # keep file digest for API
+
$conf = $snapshot;
}
- # take pending changes in
+ delete $conf->{snapshots};
+
if (!$param->{current}) {
foreach my $opt (keys %{$conf->{pending}}) {
next if $opt eq 'delete';
@@ -20,15 +21,18 @@
next if ref($value); # just to be sure
$conf->{$opt} = $value;
}
- my $pending_delete_hash =
PVE::LXC::Config->split_flagged_list($conf->{pending}->{delete});
+ my $pending_delete_hash =
PVE::QemuConfig->split_flagged_list($conf->{pending}->{delete});
foreach my $opt (keys %$pending_delete_hash) {
delete $conf->{$opt} if $conf->{$opt};
}
}
-
- delete $conf->{snapshots};
delete $conf->{pending};
+ # hide cloudinit password
+ if ($conf->{cipassword}) {
+ $conf->{cipassword} = '**';
+ }
+
return $conf;
}});
>
> return $conf;
> }});
>
> +
> my $vm_config_perm_list = [
> 'VM.Config.Disk',
> 'VM.Config.CPU',
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container 1/9] add pending section to lxc config parser/writer
NAK, this breaks existing configs with a snapshot called pending (which
is an allowed snapshot name, and not such an uncommon word that we can
confidently say that noone would be affected). we could do _PENDING_ or
__PENDING__ (similar to what we do with __base__ and __replicate__ and
__migration__ storage snapshots).
in general, this would align the VM and CT config parser/writer pairs
more closely, so it's a step in the right direction.
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> allow parsing and writing of the pending changes section in CTID.conf
> files.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/LXC/Config.pm | 35 ++-
> 1 file changed, 30 insertions(+), 5 deletions(-)
>
> diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
> index 9790345..08b958f 100644
> --- a/src/PVE/LXC/Config.pm
> +++ b/src/PVE/LXC/Config.pm
> @@ -751,6 +751,7 @@ sub parse_pct_config {
> my $res = {
> digest => Digest::SHA::sha1_hex($raw),
> snapshots => {},
> + pending => {},
> };
>
> $filename =~ m|/lxc/(\d+).conf$|
> @@ -766,7 +767,13 @@ sub parse_pct_config {
> foreach my $line (@lines) {
> next if $line =~ m/^\s*$/;
>
> - if ($line =~ m/^\[([a-z][a-z0-9_\-]+)\]\s*$/i) {
> + if ($line =~ m/^\[PENDING\]\s*$/i) {
> + $section = 'pending';
> + $conf->{description} = $descr if $descr;
> + $descr = '';
> + $conf = $res->{$section} = {};
> + next;
> + } elsif ($line =~ m/^\[([a-z][a-z0-9_\-]+)\]\s*$/i) {
> $section = $1;
> $conf->{description} = $descr if $descr;
> $descr = '';
> @@ -794,6 +801,13 @@ sub parse_pct_config {
> $descr .= PVE::Tools::decode_text($2);
> } elsif ($line =~ m/snapstate:\s*(prepare|delete)\s*$/) {
> $conf->{snapstate} = $1;
> + } elsif ($line =~ m/^delete:\s*(.*\S)\s*$/) {
> + my $value = $1;
> + if ($section eq 'pending') {
> + $conf->{delete} = $value;
> + } else {
> + warn "vm $vmid - property 'delete' is only allowed in
> [PENDING]\n";
you copied this without the typo, but did not fix it in the original ;)
> + }
> } elsif ($line =~ m/^([a-z][a-z_]*\d*):\s*(\S.*)\s*$/) {
> my $key = $1;
> my $value = $2;
> @@ -832,14 +846,19 @@ sub write_pct_config {
> }
>
> my $generate_raw_config = sub {
> - my ($conf) = @_;
> + my ($conf, $pending) = @_;
>
> my $raw = '';
>
> # add description as comment to top of file
> - my $descr = $conf->{description} || '';
> - foreach my $cl (split(/\n/, $descr)) {
> - $raw .= '#' . PVE::Tools::encode_text($cl) . "\n";
> + if (defined(my $descr = $conf->{description})) {
> + if ($descr) {
> + foreach my $cl (split(/\n/, $descr)) {
> + $raw .= '#' . PVE::Tools::encode_text($cl) . "\n";
> + }
> + } else {
> + $raw .= "#\n" if $pending;
> + }
> }
>
> foreach my $key (sort keys %$conf) {
> @@ -864,7 +883,13 @@ sub write_pct_config {
>
> my $raw = &$generate_raw_config($conf);
>
> +if (scalar(keys %{$conf->{pending}})){
> + $raw .= "\n[PENDING]\n";
> + $raw .= &$generate_raw_config($conf->{pending}, 1);
> +}
> +
> foreach my $snapname (sort keys %{$conf->{snapshots}}) {
> + die "internal error" if $snapname eq 'pending';
> $raw .= "\n[$snapname]\n";
> $raw .= &$generate_raw_config($conf->{snapshots}->{$snapname});
> }
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container 3/9] adapt config PUT to use 'revert' and 'force' parameters
the commit message is misleading - this just adds two parameters, but no
code that handles them? the code/change that adds handling only comes in
patch #8. either merge these two, or put this one after the other
please.
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/API2/LXC/Config.pm | 13 +
> 1 file changed, 13 insertions(+)
>
> diff --git a/src/PVE/API2/LXC/Config.pm b/src/PVE/API2/LXC/Config.pm
> index 6e67186..405a079 100644
> --- a/src/PVE/API2/LXC/Config.pm
> +++ b/src/PVE/API2/LXC/Config.pm
> @@ -20,6 +20,8 @@ use PVE::JSONSchema qw(get_standard_option);
>
> use base qw(PVE::RESTHandler);
>
> +my $opt_force_description = "Force physical removal. Without this, we simply
> remove the disk from the config file and create an additional configuration
> entry called 'unused[n]', which contains the volume ID. Unlink of unused[n]
> always results in physical removal.";
> +
> __PACKAGE__->register_method({
> name => 'vm_config',
> path => '',
> @@ -131,6 +133,17 @@ __PACKAGE__->register_method({
> description => "A list of settings you want to delete.",
> optional => 1,
> },
> + revert => {
> + type => 'string', format => 'pve-configid-list',
> + description => "Revert a pending change.",
> + optional => 1,
> + },
> + force => {
> + type => 'boolean',
> + description => $opt_force_description,
> + optional => 1,
> + requires => 'delete',
> + },
> digest => {
> type => 'string',
> description => 'Prevent changes if current configuration
> file has different SHA1 digest. This can be used to prevent concurrent
> modifications.',
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container 5/9] add 'pending' API method to LXC
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> analog to Qemu, it returns an array of elements, which shows the
> current value, pending value, and delete requests.
and again, this is completely identical to the Qemu API path (modulo
comments, one superfluous next statement) except for the cipassword
handling.
we could just move this into AbstractConfig, and let Qemu cleanup the
cipassword after calling the shared code.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/API2/LXC.pm | 88 +
> 1 file changed, 88 insertions(+)
>
> diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm
> index 9ddaf58..b885e90 100644
> --- a/src/PVE/API2/LXC.pm
> +++ b/src/PVE/API2/LXC.pm
> @@ -513,6 +513,7 @@ __PACKAGE__->register_method({
>
> my $res = [
> { subdir => 'config' },
> + { subdir => 'pending' },
> { subdir => 'status' },
> { subdir => 'vncproxy' },
> { subdir => 'termproxy' },
> @@ -1863,4 +1864,91 @@ __PACKAGE__->register_method({
> return $task;
>}});
>
> +__PACKAGE__->register_method({
> +name => 'vm_pending',
> +path => '{vmid}/pending',
> +method => 'GET',
> +proxyto => 'node',
> +description => 'Get container configuration, including pending changes.',
> +permissions => {
> + check => ['perm', '/vms/{vmid}', [ 'VM.Audit' ]],
> +},
> +parameters => {
> + additionalProperties => 0,
> + properties => {
> + node => get_standard_option('pve-node'),
> + vmid => get_standard_option('pve-vmid', { completion =>
> \&PVE::LXC::complete_ctid }),
> + },
> +},
> +returns => {
> + type => "array",
> + items => {
> + type => "object",
> + properties => {
> + key => {
> + description => 'Configuration option name.',
> + type => 'string',
> + },
> + value => {
> + description => 'Current value.',
> + type => 'string',
> + optional => 1,
> + },
> + pending => {
> + description => 'Pending value.',
> + type => 'string',
> + optional => 1,
> + },
> + delete => {
> + description => "Indicates a pending delete request if
> present and not 0. " .
> + "The value 2 indicates a force-delete request.",
> + type => 'integer',
> + minimum => 0,
> + maximum => 2,
> + optional => 1,
> + },
> + },
> + },
> +},
> +code => sub {
> + my ($param) = @_;
> +
> + my $conf = PVE::LXC::Config->load_config($param->{vmid});
> +
> + my $pending_delete_hash =
> PVE::LXC::Config->split_flagged_list($conf->{pending}->{delete});
> +
> + my $res = [];
> +
> + foreach my $opt (keys %$conf) {
> + next if ref($conf->{$opt});
> + next if $opt eq 'pending';
> + my $item = { key => $opt } ;
> + $item->{value} = $conf->{$opt} if defined($conf->{$opt});
> + $item->{pending} = $conf->{pending}->{$opt} if
> defined($conf->{pending}->{$opt});
> + $item->{delete} = ($pending_delete_hash->{$opt} ? 2 : 1) if exists
> $pending_delete_hash->{$opt};
> +
> + push @$res, $item;
> + }
> +
> + foreach my $opt (keys %{$conf->{pending}}) {
> + next if $opt eq 'delete';
> + next if ref($conf->{pending}->{$opt});
> + next if defined($conf->{$opt});
> + my $item = { key => $opt };
> + $item->{pending} = $conf->{pending}->{$opt};
> +
> + push @$res, $item;
> + }
> +
> + while (my ($opt, $force) = each %$pending_delete_hash) {
> + next if $conf->{pending}->{$opt};
> + next if $conf->{$opt};
> + my $item = { key => $opt, delete => ($force ? 2 : 1)};
> + push @$res, $item;
> + }
> +
> + return $res;
> +
> +}});
> +
> 1;
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container 6/9] add 'pct pending' command
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> analog to 'qm pending', it shows a list of keys and values defined in
> configuration.
copied verbatim from qm.pm, maybe we could move this code to
GuestHelpers.pm (format_pending)?
>
> cur: current change
> new: pending change
> del: pending delete
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/CLI/pct.pm | 27 +++
> 1 file changed, 27 insertions(+)
>
> diff --git a/src/PVE/CLI/pct.pm b/src/PVE/CLI/pct.pm
> index 35ad72f..78cb406 100755
> --- a/src/PVE/CLI/pct.pm
> +++ b/src/PVE/CLI/pct.pm
> @@ -821,6 +821,33 @@ our $cmddef = {
> }
> }
> }],
> +
> +pending => [ "PVE::API2::LXC", "vm_pending", ['vmid'],
> + { node => $nodename }, sub {
> + my $data = shift;
> + foreach my $item (sort { $a->{key} cmp $b->{key}} @$data) {
> + my $k = $item->{key};
> + next if $k eq 'digest';
> + my $v = $item->{value};
> + my $p = $item->{pending};
> + if ($k eq 'description') {
> + $v = PVE::Tools::encode_text($v) if defined($v);
> + $p = PVE::Tools::encode_text($p) if defined($p);
> + }
> + if (defined($v)) {
> + if ($item->{delete}) {
> + print "del $k: $v\n";
> + } elsif (defined($p)) {
> + print "cur $k: $v\n";
> + print "new $k: $p\n";
> + } else {
> + print "cur $k: $v\n";
> + }
> + } elsif (defined($p)) {
> + print "new $k: $p\n";
> + }
> + }
> + }],
> set => [ 'PVE::API2::LXC::Config', 'update_vm', ['vmid'], { node =>
> $nodename }],
>
> resize => [ "PVE::API2::LXC", 'resize_vm', ['vmid', 'disk', 'size'], {
> node => $nodename } ],
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container 7/9] add vmconfig_hotplug_pending and vmconfig_apply_pending
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> vmconfig_hotplug_pending is responsible for checking if a key in the
> pending section is hotpluggable, if yes; perform a generic config value
> replace or perform specific actions if a special case.
>
> vmconfig_apply_pending is only supposed to be called when ct isn't live.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/LXC/Config.pm | 186 +-
> 1 file changed, 184 insertions(+), 2 deletions(-)
>
> diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
> index 08b958f..26c694f 100644
> --- a/src/PVE/LXC/Config.pm
> +++ b/src/PVE/LXC/Config.pm
> @@ -7,11 +7,13 @@ use PVE::AbstractConfig;
> use PVE::Cluster qw(cfs_register_file);
> use PVE::GuestHelpers;
> use PVE::INotify;
> +use PVE::Exception qw(raise_param_exc);
> use PVE::JSONSchema qw(get_standard_option);
> -use PVE::Tools;
> +use PVE::Tools qw(extract_param);
>
> use base qw(PVE::AbstractConfig);
>
> +my $confdesc;
see other comment - I'd prefer not making vmconfig_hotplug_pending and
vmconfig_apply_pending methods in AbstractConfig, which means they can
be moved below the existing $confdesc.
> my $nodename = PVE::INotify::nodename();
> my $lock_handles = {};
> my $lockdir = "/run/lock/lxc";
> @@ -76,6 +78,186 @@ sub has_feature {
> return $err ? 0 : 1;
> }
>
> +my $LXC_FASTPLUG_OPTIONS= {
> +'description' => 1,
> +'onboot' => 1,
> +'startup' => 1,
> +'protection' => 1,
> +'hostname' => 1,
hostname is reflected inside the container though? you'd need a trip
through Setup.pm or something to actually hotplug it..
> +'hookscript' => 1,
> +'cores' => 1,
> +};
> +
> +sub vmconfig_hotplug_pending {
> +my ($class, $vmid, $conf, $storecfg, $selection, $errors) = @_;
> +
> +my $pid = PVE::LXC::find_lxc_pid($vmid);
> +my $rootdir = "/proc/$pid/root";
> +
> +my $add_error = sub {
> +my ($opt, $msg) = @_;
> +$errors->{$opt} = "hotplug problem - $msg";
> +};
> +
> +my $changes;
> +foreach my $opt (keys %{$conf->{pending}}) { # add/change
> +if ($LXC_FASTPLUG_OPTIONS->{$opt}) {
> +$conf->{$opt} = delete $conf->{pending}->{$opt};
> +$changes = 1;
> +}
ignores $selection?
> +}
> +
> +if ($changes) {
> + $class->write_config($vmid, $conf);
> +$conf = $class->load_config($vmid); # update/reload
should not be needed, this needs to be called while holding an flock
anyway..
> +}
> +
> +my $pending_delete_hash =
> $class->split_flagged_list($conf->{pending}->{delete});
> +while (my ($opt, $force) = each %$pending_delete_hash) {
> + next if $selection && !$selection->{$opt};
> + eval {
> + if ($LXC_FASTPLUG_OPTIONS->{$opt}) {
> + # pass
> + } elsif ($opt eq 'swap') { # TODO FIXME: unlimited swap instead of
> no swap
wouldn't this just be able to re-use the hotplug_memory code from below?
> + PVE::LXC::write_cgroup_value("memory", $vmid,
> + "memory.memsw.limit_in_bytes", -1);
> + } elsif ($opt eq 'cpulimit') {
> + PVE::LXC::write_cgroup_value("cpu", $vmid, "cpu.cfs_quota_us",
> -1);
> + } elsif ($opt eq 'cpuunits') {
> + PVE::LXC::write_cgroup_value("cpu", $vmid, "cpu.shared",
> $confdesc->{cpuunits}->{default});
> + } elsif ($opt =~ m/^net(\d)$/) {
> + my $netid = $1;
> + PVE::Network::veth_delete("veth${vmid}i$netid");
> + } else {
> + die "skip\n"; # skip non-hotpluggable opts
> + }
> + };
> + if (my $err = $@) {
> + $add_error->($opt, $err) if $err ne "skip\n";
> + } else {
> + delete $conf->{$opt};
> + $class->vmconfig_undelete_pending_option($conf, $opt);
> + $class->write_config($vmid, $conf);
> + $conf = $class->load_config($vmid); # update/reload
reload shouldn't be needed
> + }
> +}
> +
> +# There's no separate swap size to configure, there's memory and "total"
> +# memory (iow. memory+swap). This means we have to change them together.
> +my $hotplug_memory_done;
> +my $hotplug_memory = sub {
> + my ($wanted_memory, $wanted_swap) = @_;
> + my $old_memory = ($conf->{memory} || 512);
> + my $old_swap = ($conf->{swap} || 0);
$confdesc says swap also defaults to 512? which one is wrong?
why not use the defaults here, instead of hard-coding them?
> +
> + $wanted_memory //= $old_memory;
> + $wanted_swap //= $old_swap;
> +
> + my $total = $wanted_memory + $wanted_swap;
> + my $old_total = $old_memory + $old_swap;
> +
> +
superfluous blank line
> + if ($total > $old_total) {
does this fail if the order is wrong? if so, wouldn't it be better to
read the current value instead of guessing it based on $conf? still
racy, but that can probably not be avoided..
> + PVE::LXC::write_cgroup_
Re: [pve-devel] [PATCH container 9/9] apply pending changes when container is started
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/LXC.pm | 7 +++
> 1 file changed, 7 insertions(+)
>
> diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
> index 475d9be..9dd5bc9 100644
> --- a/src/PVE/LXC.pm
> +++ b/src/PVE/LXC.pm
> @@ -1939,6 +1939,13 @@ sub vm_start {
> close($fh);
> }
>
> +# apply pending changes while starting
> +my $storecfg = PVE::Storage::config();
> +if (scalar(keys %{$conf->{pending}})) {
> + PVE::LXC::Config->vmconfig_apply_pending($vmid, $conf, $storecfg);
> + $conf = PVE::LXC::Config->load_config($vmid); # update/reload
> +}
> +
this should probably happen before the call to update_lxc_config a few
lines above..
> my $cmd = ['systemctl', 'start', "pve-container\@$vmid"];
>
> PVE::GuestHelpers::exec_hookscript($conf, $vmid, 'pre-start', 1);
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container 0/9] lxc pending changes
On September 5, 2019 4:11 pm, Oguz Bektas wrote: > this series makes it possible to add/revert/delete pending changes in > the backend. > > it depends on my previous patch for pve-guest-common, for enabling > inheritance of pending changes related methods into PVE::LXC::Config thanks for these patches! this functionality is a long-time overdue.. detailed comments/questions on individual patches (it's a lot, but I am also on vacation next week so you have plenty of time to prepare a v2 ;)). the big picture already looks mostly good (there are some refactoring choices that I would do differently), but with such functionality the devil is always in the details/edge cases. please test your v2 thoroughly, in particular regarding - mountpoint handling - backup/restore - custom lxc.foo options - clones - migration - various ways to restart containers (from within, stop/start, systemctl, ..) also, please think about which parts can be unit-tested and write tests for them! if you find stuff about the interface or semantics from Qemu that is sub-optimal / unclear, don't just copy it, but try to understand and improve it.. duplicate or very similar code is already bad enough, two or three forks of BAD code is even worse ;) I haven't checked for everything whether particular lines are originally from you, or based on qemu-server - some / a lot of my comments might apply to qemu-server's implementation as well.. > > some notes or to-be-fixed-soon-after-review points: > [1]. gui patches are coming soon > [2]. right now pending changes are applied __only__ at vm_start. > [3]. --delete $opt --force is not implemented (is in schema, but has no > effect atm) unless you plan to implement this in this series, I'd drop it from the schema/API at least. > [4]. clones will __keep__ pending changes in config (for now) > [5]. there are a couple TODO/FIXME's inside, since they were extra > features or unrelated bugs (which i think should be fixed in separate > commits later since they don't directly affect pending changes functionality) for small, well-understood stuff it often makes sense to fix it in separate commits up front. who knows when the next person takes a closer look at that particular code - a fix done now is done, a FIXME added now might be ignored forever. > > comments: > > [2]: i thought it was best to keep it like this for the first version, > until we decide what's the best way to go about it. > > [3]: it seemed quite tricky to implement live force-delete of mounpoints > because of a few reasons, like unmount not being allowed or mp being on > network storage etc. > > [4]: will fix this in v2 after the first review > > [5]: one bug, where if swap option is deleted while ct is running, > memory.memsw.limit_in_bytes is set to infinite instead of zero. the > other one is the --force implementation. backup is also missing pending changes skipping. some more stuff in the review itself.. > Oguz Bektas (9): > add pending section to lxc config parser/writer > adapt config GET call for taking pending changes > adapt config PUT to use 'revert' and 'force' parameters > remove trailing whitespace > add 'pending' API method to LXC > add 'pct pending' command > add vmconfig_hotplug_pending and vmconfig_apply_pending > rework update_pct_config to write and apply pending changes > apply pending changes when container is started > > src/PVE/API2/LXC.pm| 90 +- > src/PVE/API2/LXC/Config.pm | 88 +++--- > src/PVE/CLI/pct.pm | 27 ++ > src/PVE/LXC.pm | 7 + > src/PVE/LXC/Config.pm | 549 - > 5 files changed, 520 insertions(+), 241 deletions(-) > > -- > 2.20.1 > > ___ > pve-devel mailing list > pve-devel@pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container 8/9] rework update_pct_config to write and apply pending changes
On September 5, 2019 4:11 pm, Oguz Bektas wrote:
> use vmconfig_hotplug_pending (when ct up) and vmconfig_apply_pending
> (when ct down) to apply or write pending changes.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/API2/LXC/Config.pm | 52 +-
> src/PVE/LXC/Config.pm | 328 +
> 2 files changed, 147 insertions(+), 233 deletions(-)
>
> diff --git a/src/PVE/API2/LXC/Config.pm b/src/PVE/API2/LXC/Config.pm
> index 405a079..a31ddd5 100644
> --- a/src/PVE/API2/LXC/Config.pm
> +++ b/src/PVE/API2/LXC/Config.pm
> @@ -156,58 +156,10 @@ __PACKAGE__->register_method({
> code => sub {
> my ($param) = @_;
>
> - my $rpcenv = PVE::RPCEnvironment::get();
> - my $authuser = $rpcenv->get_user();
> -
> my $node = extract_param($param, 'node');
> my $vmid = extract_param($param, 'vmid');
> -
> my $digest = extract_param($param, 'digest');
>
> - die "no options specified\n" if !scalar(keys %$param);
> -
> - my $delete_str = extract_param($param, 'delete');
> - my @delete = PVE::Tools::split_list($delete_str);
> -
> - PVE::LXC::check_ct_modify_config_perm($rpcenv, $authuser, $vmid, undef,
> {}, [@delete]);
> -
> - foreach my $opt (@delete) {
> - raise_param_exc({ delete => "you can't use '-$opt' and -delete
> $opt' at the same time" })
> - if defined($param->{$opt});
> -
> - if (!PVE::LXC::Config->option_exists($opt)) {
> - raise_param_exc({ delete => "unknown option '$opt'" });
> - }
> - }
> -
> - PVE::LXC::check_ct_modify_config_perm($rpcenv, $authuser, $vmid, undef,
> $param, []);
> -
> - my $storage_cfg = cfs_read_file("storage.cfg");
> -
> - my $repl_conf = PVE::ReplicationConfig->new();
> - my $is_replicated = $repl_conf->check_for_existing_jobs($vmid, 1);
> - if ($is_replicated) {
> - PVE::LXC::Config->foreach_mountpoint_full($param, 0, sub {
> - my ($opt, $mountpoint) = @_;
> - my $volid = $mountpoint->{volume};
> - return if !$volid || !($mountpoint->{replicate}//1);
> - if ($mountpoint->{type} eq 'volume') {
> - my ($storeid, $format);
> - if ($volid =~ $PVE::LXC::NEW_DISK_RE) {
> - $storeid = $1;
> - $format = $mountpoint->{format} ||
> PVE::Storage::storage_default_format($storage_cfg, $storeid);
> - } else {
> - ($storeid, undef) =
> PVE::Storage::parse_volume_id($volid, 1);
> - $format = (PVE::Storage::parse_volname($storage_cfg,
> $volid))[6];
> - }
> - return if PVE::Storage::storage_can_replicate($storage_cfg,
> $storeid, $format);
> - my $scfg = PVE::Storage::storage_config($storage_cfg,
> $storeid);
> - return if $scfg->{shared};
> - }
> - die "cannot add non-replicatable volume to a replicated VM\n";
> - });
> - }
> -
> my $code = sub {
>
> my $conf = PVE::LXC::Config->load_config($vmid);
> @@ -217,10 +169,8 @@ __PACKAGE__->register_method({
>
> my $running = PVE::LXC::check_running($vmid);
>
> - PVE::LXC::Config->update_pct_config($vmid, $conf, $running, $param,
> \@delete);
> + PVE::LXC::Config->update_pct_config($vmid, $conf, $running, $param);
>
> - PVE::LXC::Config->write_config($vmid, $conf);
> - PVE::LXC::update_lxc_config($vmid, $conf);
this call to update_lxc_config gets dropped without any comment - is it
not longer needed? was the removal accidental?
> };
>
> PVE::LXC::Config->lock_config($vmid, $code);
> diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
> index 26c694f..6bec9b7 100644
> --- a/src/PVE/LXC/Config.pm
> +++ b/src/PVE/LXC/Config.pm
> @@ -1080,129 +1080,132 @@ sub write_pct_config {
> }
>
> sub update_pct_config {
> -my ($class, $vmid, $conf, $running, $param, $delete) = @_;
> +my ($class, $vmid, $conf, $running, $param) = @_;
>
> -my @nohotplug;
> +my $rpcenv = PVE::RPCEnvironment::get();
missing use statement for PVE::RPCEnvironment
> +my $authuser = $rpcenv->get_user();
>
> -my $new_disks = 0;
> -my @deleted_volumes;
> +my $delete_str = extract_param($param, 'delete');
> +my @delete = PVE::Tools::split_list($delete_str);
> +my $revert_str = extract_param($param, 'revert');
> +my $revert = {};
why this completely different handling of revert and delete?
> +my $force = extract_param($param, 'force');
>
> -my $rootdir;
> -if ($running) {
> - my $pid = PVE::LXC::find_lxc_pid($vmid);
> - $rootdir = "/proc/$pid/root";
> +die "no options specified\n" if !$delete_str && !$revert_str &&
> !scalar(keys %$param);
this also gets called for container creation/restore, which triggers
this die in case of an in-place restore.
> +
> +PVE::LXC::check_ct_
Re: [pve-devel] [PATCH container 1/9] add pending section to lxc config parser/writer
On September 11, 2019 2:02 pm, Thomas Lamprecht wrote: > On 11.09.19 09:39, Fabian Grünbichler wrote: >> NAK, this breaks existing configs with a snapshot called pending (which >> is an allowed snapshot name, and not such an uncommon word that we can >> confidently say that noone would be affected). we could do _PENDING_ or > > naming it pending for VMs was possible and broke things in obvious ways, > nevertheless we never had a single report about it, AFAICT, even after > pending being released ~4.5 years ago. So while yes, the chance is there > I'd guess that's highly unlikely - not that that really helps us. > >> __PENDING__ (similar to what we do with __base__ and __replicate__ and >> __migration__ storage snapshots). > > confuses things more, IMO, the other two are special snapshots, this is > completely something different, so mixing those two into having a similar > syntax may make people think that "__PENDING__" is a pending snapshot. > > IMO, the snapshots should have been prefixed with a marker from the > beginning, but as the time machine isn't there yet and retro actively > re-writing snapshot sections as "[snap:]" or the like isn't a to > easy task, I'd maybe rather add a prefix for non-snapshot sections with > a snapshot incompatible syntax. [_special:PENDING] or the like. > > Another possible way could be to really to the > [snap:] for snapshots, rewriting old ones once the config is written > out for other changes anyway, and for names where we are not sure like > "pending" check if's there's a "snaptime" property in the section? > As else it cannot be a valid snapshot made through the API. we had a bit of offline discussion yesterday about this, IMHO namespacing the sections would be the best way to go forward. either special: or pve: for stuff like 'pending', snap: or snapshot: for snapshots. since we don't allow ':' in section names currently, this could be done fairly backwards-compatible (plain 'pending' would be handled differently for qemu-server and pve-container).. vzdump filters both pending and snapshots from the stored config, replication is frequent enough, so we don't really have to worry about long-term compat, just for one release cycle.. the hash representation can stay liek it is, so no big code changes needed either, besides the par ser/writer and qemu-server's vzdump (which has its own filtering config reader). ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH ceph] build: use dgit for download target
since it has all the necessary wrapping to download and verify dscs from
arbitraty repositories, without being affected by the system's APT state
and configuration.
Signed-off-by: Fabian Grünbichler
---
Makefile | 13 +++--
upstream-key.asc | 29 +
2 files changed, 36 insertions(+), 6 deletions(-)
create mode 100644 upstream-key.asc
diff --git a/Makefile b/Makefile
index f885c1283..0219426df 100644
--- a/Makefile
+++ b/Makefile
@@ -99,15 +99,16 @@ ${DSC}: ${BUILDSRC}
cd ${BUILDSRC}; dpkg-buildpackage -S -uc -us -d -nc
@echo ${DSC}
+# NOTE: always downloads latest version!
.PHONY: download
download:
- # FIXME: better verification (download dsc and use dscverify with
ceph.com release key?)
- dget --allow-unauthenticated --download-only
'https://download.ceph.com/debian-nautilus/pool/main/c/ceph/ceph_${VER}.orig.tar.gz'
rm -rf ${SRCDIR}.tmp ${SRCDIR}
- mkdir ${SRCDIR}.tmp
- tar -C ${SRCDIR}.tmp --strip-components=1 -xf ceph_${VER}.orig.tar.gz
- # needed because boost and zstd builds fail otherwise
- find ${SRCDIR}.tmp -type f -name ".gitignore" -delete
+ dgit -cdgit-distro.ceph.archive-query=aptget:
-cdgit-distro.ceph.mirror=http://download.ceph.com/debian-nautilus
-cdgit-distro.ceph.git-check=false
--apt-get:--option=Dir::Etc::Trusted=${CURDIR}/upstream-key.asc -d ceph clone
ceph xenial ./${SRCDIR}.tmp
+ @echo "WARNING"
+ @echo "Check output above for verification errors!"
+ @echo "WARNING"
+ rm -rf ${SRCDIR}.tmp/.git
+ find ${SRCDIR}.tmp/ -type f -name '.gitignore' -delete
mv ${SRCDIR}.tmp/debian/changelog ${SRCDIR}.tmp/changelog.upstream
mv ${SRCDIR}.tmp ${SRCDIR}
diff --git a/upstream-key.asc b/upstream-key.asc
new file mode 100644
index 0..d2961c52e
--- /dev/null
+++ b/upstream-key.asc
@@ -0,0 +1,29 @@
+-BEGIN PGP PUBLIC KEY BLOCK-
+Version: GnuPG v1
+
+mQINBFX4hgkBEADLqn6O+UFp+ZuwccNldwvh5PzEwKUPlXKPLjQfXlQRig1flpCH
+E0HJ5wgGlCtYd3Ol9f9+qU24kDNzfbs5bud58BeE7zFaZ4s0JMOMuVm7p8JhsvkU
+C/Lo/7NFh25e4kgJpjvnwua7c2YrA44ggRb1QT19ueOZLK5wCQ1mR+0GdrcHRCLr
+7Sdw1d7aLxMT+5nvqfzsmbDullsWOD6RnMdcqhOxZZvpay8OeuK+yb8FVQ4sOIzB
+FiNi5cNOFFHg+8dZQoDrK3BpwNxYdGHsYIwU9u6DWWqXybBnB9jd2pve9PlzQUbO
+eHEa4Z+jPqxY829f4ldaql7ig8e6BaInTfs2wPnHJ+606g2UH86QUmrVAjVzlLCm
+nqoGymoAPGA4ObHu9X3kO8viMBId9FzooVqR8a9En7ZE0Dm9O7puzXR7A1f5sHoz
+JdYHnr32I+B8iOixhDUtxIY4GA8biGATNaPd8XR2Ca1hPuZRVuIiGG9HDqUEtXhV
+fY5qjTjaThIVKtYgEkWMT+Wet3DPPiWT3ftNOE907e6EWEBCHgsEuuZnAbku1GgD
+LBH4/a/yo9bNvGZKRaTUM/1TXhM5XgVKjd07B4cChgKypAVHvef3HKfCG2U/DkyA
+LjteHt/V807MtSlQyYaXUTGtDCrQPSlMK5TjmqUnDwy6Qdq8dtWN3DtBWQARAQAB
+tCpDZXBoLmNvbSAocmVsZWFzZSBrZXkpIDxzZWN1cml0eUBjZXBoLmNvbT6JAjgE
+EwECACIFAlX4hgkCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEOhKwsBG
+DzmUXdIQAI8YPcZMBWdv489q8CzxlfRIRZ3Gv/G/8CH+EOExcmkVZ89mVHngCdAP
+DOYCl8twWXC1lwJuLDBtkUOHXNuR5+Jcl5zFOUyldq1Hv8u03vjnGT7lLJkJoqpG
+l9QD8nBqRvBU7EM+CU7kP8+09b+088pULil+8x46PwgXkvOQwfVKSOr740Q4J4nm
+/nUOyTNtToYntmt2fAVWDTIuyPpAqA6jcqSOC7Xoz9cYxkVWnYMLBUySXmSS0uxl
+3p+wK0lMG0my/gb+alke5PAQjcE5dtXYzCn+8Lj0uSfCk8Gy0ZOK2oiUjaCGYN6D
+u72qDRFBnR3jaoFqi03bGBIMnglGuAPyBZiI7LJgzuT9xumjKTJW3kN4YJxMNYu1
+FzmIyFZpyvZ7930vB2UpCOiIaRdZiX4Z6ZN2frD3a/vBxBNqiNh/BO+Dex+PDfI4
+TqwF8zlcjt4XZ2teQ8nNMR/D8oiYTUW8hwR4laEmDy7ASxe0p5aijmUApWq5UTsF
++s/QbwugccU0iR5orksM5u9MZH4J/mFGKzOltfGXNLYI6D5Mtwrnyi0BsF5eY0u6
+vkdivtdqrq2DXY+ftuqLOQ7b+t1RctbcMHGPptlxFuN9ufP5TiTWSpfqDwmHCLsT
+k2vFiMwcHdLpQ1IH8ORVRgPPsiBnBOJ/kIiXG2SxPUTjjEGOVgeA
+=/Tod
+-END PGP PUBLIC KEY BLOCK-
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH container] don't duplicate lxc.idmap entries during restore
On September 24, 2019 4:04 pm, Oguz Bektas wrote:
> merging $conf->{lxc} causes lxc.idmap entries to be duplicated in the
> restored configuration. instead, we can overwrite the contents from the
> extracted configuration file. this way we don't duplicate these entries.
> (having duplicate idmap entries causes container to crash during start)
>
> Co-developed-by: Stefan Reiter
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/LXC/Create.pm | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/src/PVE/LXC/Create.pm b/src/PVE/LXC/Create.pm
> index a46a50c..4add540 100644
> --- a/src/PVE/LXC/Create.pm
> +++ b/src/PVE/LXC/Create.pm
> @@ -186,7 +186,9 @@ sub restore_configuration {
> }
> warn "\n";
> } else {
> - @{$conf->{$key}} = (@$lxc_list, @{$conf->{$key}});
> + # $conf->{lxc} can only have lxc.idmap
> + # so we can overwrite the current $conf from $oldconf
> + @{$conf->{$key}} = @$lxc_list;
why not $conf->{$key} = $lxc_list
?
> }
> next;
> }
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container] don't duplicate lxc.idmap entries during restore
On September 25, 2019 1:30 pm, Oguz Bektas wrote:
> merging $conf->{lxc} causes lxc.idmap entries to be duplicated in the
> restored configuration. instead, we can overwrite the contents from the
> extracted configuration file. this way we don't duplicate these entries.
> (having duplicate idmap entries causes container to crash during start)
>
> Co-developed-by: Stefan Reiter
> Signed-off-by: Oguz Bektas
What about the following instead (note, generated with -w for
readability)? seems simpler, and with less potential for future breakage
;)
From 8e0679858748be369d5ddc5695376b12504daa50 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fabian=20Gr=C3=BCnbichler?=
Date: Wed, 25 Sep 2019 14:35:04 +0200
Subject: [PATCH container] restore lxc.* entries once
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
either via recover_config, OR via restore_configuration. non-root behaviour
stays the same.
Signed-off-by: Fabian Grünbichler
---
src/PVE/API2/LXC.pm | 4 ++--
src/PVE/LXC/Create.pm | 6 +-
2 files changed, 3 insertions(+), 7 deletions(-)
diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm
index 07280fb..28c9047 100644
--- a/src/PVE/API2/LXC.pm
+++ b/src/PVE/API2/LXC.pm
@@ -353,11 +353,11 @@ __PACKAGE__->register_method({
my $orig_conf;
($orig_conf, $orig_mp_param) =
PVE::LXC::Create::recover_config($archive);
$was_template = delete $orig_conf->{template};
- # When we're root call 'restore_configuration' with
ristricted=0,
+ # When we're root call 'restore_configuration' with
restricted=0,
# causing it to restore the raw lxc entries, among
which there may be
# 'lxc.idmap' entries. We need to make sure that the
extracted contents
# of the container match up with the restored
configuration afterwards:
- $conf->{lxc} = [grep { $_->[0] eq 'lxc.idmap' }
@{$orig_conf->{lxc}}];
+ $conf->{lxc} = $orig_conf->{lxc};
}
}
if ($storage_only_mode) {
diff --git a/src/PVE/LXC/Create.pm b/src/PVE/LXC/Create.pm
index a46a50c..241ca88 100644
--- a/src/PVE/LXC/Create.pm
+++ b/src/PVE/LXC/Create.pm
@@ -176,18 +176,14 @@ sub restore_configuration {
# storage supports creating a template there
next if $key =~ /^template$/;
- if ($key eq 'lxc') {
+ if ($key eq 'lxc' && $restricted) {
my $lxc_list = $oldconf->{'lxc'};
- if ($restricted) {
warn "skipping custom lxc options, restore manually as root:\n";
warn "\n";
foreach my $lxc_opt (@$lxc_list) {
warn "$lxc_opt->[0]: $lxc_opt->[1]\n"
}
warn "\n";
- } else {
- @{$conf->{$key}} = (@$lxc_list, @{$conf->{$key}});
- }
next;
}
--
2.20.1
> ---
> src/PVE/LXC/Create.pm | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/src/PVE/LXC/Create.pm b/src/PVE/LXC/Create.pm
> index a46a50c..8b2cfc9 100644
> --- a/src/PVE/LXC/Create.pm
> +++ b/src/PVE/LXC/Create.pm
> @@ -186,7 +186,9 @@ sub restore_configuration {
> }
> warn "\n";
> } else {
> - @{$conf->{$key}} = (@$lxc_list, @{$conf->{$key}});
> + # $conf->{lxc} can only have lxc.idmap
> + # so we can overwrite the current $conf from $oldconf
> + $conf->{$key} = $lxc_list;
> }
> next;
> }
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] libknet1 update don't restart corosync
On September 27, 2019 8:53 am, Alexandre DERUMIER wrote:
> Hi,
>
> I have noticed that when you upgrade libknet1 (and fix the crash of corosync),
>
> corosync is not auto restarted.
yes, just like any other library except libc6 in Debian. there is
tooling to handle this in general ('needrestart' is the most widespread
one), but they require some tuning to not restart services that need to
keep running, avoid false positives, etc.pp.
> Maybe should we bump the corosync package too to force a restart ?
would be one possibility. the other would be to add a trigger from knet
to corosync, if we want to always have this behaviour for that set of
packages.
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH ha-manager 09/13] Add crm command 'stop'
one comment inline
On September 26, 2019 1:38 pm, Fabian Ebner wrote:
> Not every command parameter is 'target' anymore, so
> it was necessary to modify the parsing of $sd->{cmd}.
>
> Just changing the state to request_stop is not enough,
> we need to actually update the service configuration as well.
>
> Signed-off-by: Fabian Ebner
> ---
> src/PVE/HA/Manager.pm | 27 +++
> 1 file changed, 23 insertions(+), 4 deletions(-)
>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index 0b90de4..768baa7 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -349,6 +349,14 @@ sub update_crm_commands {
> $haenv->log('err', "crm command error - no such service: $cmd");
> }
>
> + } elsif ($cmd =~ m/^stop\s+(\S+)\s+(\S+)$/) {
> + my ($sid, $timeout) = ($1, $2);
> + if (my $sd = $ss->{$sid}) {
> + $haenv->log('info', "got crm command: $cmd");
> + $ss->{$sid}->{cmd} = [ 'stop', $timeout ];
> + } else {
> + $haenv->log('err', "crm command error - no such service: $cmd");
> + }
> } else {
> $haenv->log('err', "unable to parse crm command: $cmd");
> }
> @@ -562,10 +570,10 @@ sub next_state_stopped {
> }
>
> if ($sd->{cmd}) {
> - my ($cmd, $target) = @{$sd->{cmd}};
> - delete $sd->{cmd};
> + my $cmd = shift @{$sd->{cmd}};
>
> if ($cmd eq 'migrate' || $cmd eq 'relocate') {
> + my $target = shift @{$sd->{cmd}};
> if (!$ns->node_is_online($target)) {
> $haenv->log('err', "ignore service '$sid' $cmd request - node
> '$target' not online");
> } elsif ($sd->{node} eq $target) {
> @@ -575,9 +583,12 @@ sub next_state_stopped {
> target => $target);
> return;
> }
> + } elsif ($cmd eq 'stop') {
> + $haenv->log('info', "ignore service '$sid' $cmd request -
> service already stopped");
> } else {
> $haenv->log('err', "unknown command '$cmd' for service '$sid'");
> }
> + delete $sd->{cmd};
> }
>
> if ($cd->{state} eq 'disabled') {
> @@ -639,10 +650,10 @@ sub next_state_started {
> if ($cd->{state} eq 'started') {
>
> if ($sd->{cmd}) {
> - my ($cmd, $target) = @{$sd->{cmd}};
> - delete $sd->{cmd};
> + my $cmd = shift @{$sd->{cmd}};
>
> if ($cmd eq 'migrate' || $cmd eq 'relocate') {
> + my $target = shift @{$sd->{cmd}};
> if (!$ns->node_is_online($target)) {
> $haenv->log('err', "ignore service '$sid' $cmd request -
> node '$target' not online");
> } elsif ($sd->{node} eq $target) {
> @@ -651,9 +662,17 @@ sub next_state_started {
> $haenv->log('info', "$cmd service '$sid' to node
> '$target'");
> &$change_service_state($self, $sid, $cmd, node =>
> $sd->{node}, target => $target);
> }
> + } elsif ($cmd eq 'stop') {
> + my $timeout = shift @{$sd->{cmd}};
> + $haenv->log('info', "$cmd service with timeout '$timeout'");
> + &$change_service_state($self, $sid, 'request_stop', timeout =>
> $timeout);
> + $haenv->update_service_config(0, 0, $sid, {'state' =>
> 'stopped'});
this uses the methods introduced in patch #2/3, but is missing the
locking that the other call site (and the other read/modify/write blocks
for that config file) have.
either move the locking to PVE::HA::Config::update_resources_config in
#2, or add locking here, or re-evaluate whether there is an approach
that can skip modifying the config file ;). there are other calls to
lock_ha_domain in the CRM command handling flow, so it's probably not
that bad to have another few instances here..
> } else {
> $haenv->log('err', "unknown command '$cmd' for service '$sid'");
> }
> +
> + delete $sd->{cmd};
> +
> } else {
>
> my $try_next = 0;
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH kernel] backport new FPU register copy helpers
small nit inline, otherwise
Acked-by: Fabian Grünbichler
On September 27, 2019 12:49 pm, Thomas Lamprecht wrote:
> This allows us to fix the ZFS SIMD patch for 5.0 kernel way easier,
> as we can use the same code path as used for 5.2 and newer kernels
> there.
>
> The helper itself do not do anything, just exposes them for modules
> to use.
>
> Signed-off-by: Thomas Lamprecht
> ---
> ...kport-copy_kernel_to_XYZ_err-helpers.patch | 102 ++
> 1 file changed, 102 insertions(+)
> create mode 100644
> patches/kernel/0007-x86-fpu-backport-copy_kernel_to_XYZ_err-helpers.patch
>
> diff --git
> a/patches/kernel/0007-x86-fpu-backport-copy_kernel_to_XYZ_err-helpers.patch
> b/patches/kernel/0007-x86-fpu-backport-copy_kernel_to_XYZ_err-helpers.patch
> new file mode 100644
> index ..94fc630a0ca9
> --- /dev/null
> +++
> b/patches/kernel/0007-x86-fpu-backport-copy_kernel_to_XYZ_err-helpers.patch
> @@ -0,0 +1,102 @@
> +From Mon Sep 17 00:00:00 2001
> +From: Sebastian Andrzej Siewior
I'd move this down below, and put your name here ;)
> +Date: Wed, 3 Apr 2019 18:41:50 +0200
> +Subject: [PATCH] x86/fpu: backport copy_kernel_to_XYZ_err helpers
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +partial cherry-pick from upstream 5.2 "86/fpu: Restore from kernel
> +memory on the 64-bit path too"
> +commit 926b21f37b072ae4c117052de45a975c6d468fec
> +
> +Namely, only backport the added helpers, none of the semantic changes.
> +
> +relevant parts of the original commit message:
> +> In order to avoid that mess, copy the FPU state from userland, validate
> +> it and then load it. The copy_kernel_…() helpers are basically just
> +> like the old helpers except that they operate on kernel memory and the
> +> fault handler just sets the error value and the caller handles it.
(as discussed offline, there is definitely still an issue with upstream
ZFS code here, as there is no error handling there whatsoever? but that
is something to continue on GH I guess.)
> +
> +Link: https://lkml.kernel.org/r/20190403164156.19645-22-bige...@linutronix.de
> +(partial cherry picked from commit 926b21f37b072ae4c117052de45a975c6d468fec)
> +Signed-off-by: Thomas Lamprecht
> +---
> + arch/x86/include/asm/fpu/internal.h | 43 +
> + 1 file changed, 43 insertions(+)
> +
> +diff --git a/arch/x86/include/asm/fpu/internal.h
> b/arch/x86/include/asm/fpu/internal.h
> +index fa2c93cb42a2..f3193ab0a2fb 100644
> +--- a/arch/x86/include/asm/fpu/internal.h
> b/arch/x86/include/asm/fpu/internal.h
> +@@ -122,6 +122,21 @@ extern void fpstate_sanitize_xstate(struct fpu *fpu);
> + err;\
> + })
> +
> ++#define kernel_insn_err(insn, output, input...)
> \
> ++({ \
> ++int err;\
> ++asm volatile("1:" #insn "\n\t" \
> ++ "2:\n" \
> ++ ".section .fixup,\"ax\"\n" \
> ++ "3: movl $-1,%[err]\n"\
> ++ "jmp 2b\n"\
> ++ ".previous\n" \
> ++ _ASM_EXTABLE(1b, 3b) \
> ++ : [err] "=r" (err), output \
> ++ : "0"(0), input); \
> ++err;\
> ++})
> ++
> + #define kernel_insn(insn, output, input...) \
> + asm volatile("1:" #insn "\n\t" \
> + "2:\n" \
> +@@ -158,6 +173,14 @@ static inline void copy_kernel_to_fxregs(struct
> fxregs_state *fx)
> + }
> + }
> +
> ++static inline int copy_kernel_to_fxregs_err(struct fxregs_state *fx)
> ++{
> ++if (IS_ENABLED(CONFIG_X86_32))
> ++return kernel_insn_err(fxrstor %[fx], "=m" (*fx), [fx] "m"
> (*fx));
> ++else
> ++return kernel_insn_err(fxrstorq %[fx], "=m" (*fx), [fx] "m"
> (*fx));
> ++}
> ++
> + static inline int copy_user_to_fxregs(struct fxregs_sta
Re: [pve-devel] [PATCH zfsonlinux] [SIMD]: FPU register save/restore is also required on 5.0 kernels
Acked-by: Fabian Grünbichler
On September 27, 2019 12:49 pm, Thomas Lamprecht wrote:
> Do a very minimal fix, effectively just making the code path for 5.0
> to 5.1 and 5.2 to 5.3 the same one, which can be done by backporting
> the new copy_kernel helpers into the kernel, as the existing ones are
> indirectly exported as GPL symbol
>
> Signed-off-by: Thomas Lamprecht
> ---
> ...r-save-restore-is-also-required-on-5.patch | 57 +++
> debian/patches/series | 1 +
> 2 files changed, 58 insertions(+)
> create mode 100644
> debian/patches/0010-SIMD-FPU-register-save-restore-is-also-required-on-5.patch
>
> diff --git
> a/debian/patches/0010-SIMD-FPU-register-save-restore-is-also-required-on-5.patch
>
> b/debian/patches/0010-SIMD-FPU-register-save-restore-is-also-required-on-5.patch
> new file mode 100644
> index 000..87c8af1
> --- /dev/null
> +++
> b/debian/patches/0010-SIMD-FPU-register-save-restore-is-also-required-on-5.patch
> @@ -0,0 +1,57 @@
> +From Mon Sep 17 00:00:00 2001
> +From: Thomas Lamprecht
> +Date: Wed, 25 Sep 2019 10:48:48 +0200
> +Subject: [PATCH] [SIMD]: FPU register save/restore is also required on 5.0
> + kernels
> +
> +NOTE: the kernel needs to have the copy_kernel_to_xregs_err,
> +copy_kernel_to_fxregs_err and copy_kernel_to_fregs_err funcitons
> +backported for this to work.
> +
> +Signed-off-by: Thomas Lamprecht
> +---
> + include/linux/simd_x86.h | 11 +--
> + 1 file changed, 1 insertion(+), 10 deletions(-)
> +
> +diff --git a/include/linux/simd_x86.h b/include/linux/simd_x86.h
> +index 5f243e0cc..aac63d964 100644
> +--- a/include/linux/simd_x86.h
> b/include/linux/simd_x86.h
> +@@ -179,7 +179,6 @@ kfpu_begin(void)
> + preempt_disable();
> + local_irq_disable();
> +
> +-#if defined(HAVE_KERNEL_TIF_NEED_FPU_LOAD)
> + /*
> + * The current FPU registers need to be preserved by kfpu_begin()
> + * and restored by kfpu_end(). This is required because we can
> +@@ -188,20 +187,13 @@ kfpu_begin(void)
> + * context switch.
> + */
> + copy_fpregs_to_fpstate(¤t->thread.fpu);
> +-#elif defined(HAVE_KERNEL_FPU_INITIALIZED)
> +-/*
> +- * There is no need to preserve and restore the FPU registers.
> +- * They will always be restored from the task's stored FPU state
> +- * when switching contexts.
> +- */
> ++
> + WARN_ON_ONCE(current->thread.fpu.initialized == 0);
> +-#endif
> + }
> +
> + static inline void
> + kfpu_end(void)
> + {
> +-#if defined(HAVE_KERNEL_TIF_NEED_FPU_LOAD)
> + union fpregs_state *state = ¤t->thread.fpu.state;
> + int error;
> +
> +@@ -213,7 +205,6 @@ kfpu_end(void)
> + error = copy_kernel_to_fregs_err(&state->fsave);
> + }
> + WARN_ON_ONCE(error);
> +-#endif
> +
> + local_irq_enable();
> + preempt_enable();
> diff --git a/debian/patches/series b/debian/patches/series
> index 9b0d7fb..c65e410 100644
> --- a/debian/patches/series
> +++ b/debian/patches/series
> @@ -7,3 +7,4 @@
> 0007-Fix-race-in-parallel-mount-s-thread-dispatching-algo.patch
> 0008-Linux-5.0-compat-SIMD-compatibility.patch
> 0009-Fix-CONFIG_X86_DEBUG_FPU-build-failure.patch
> +0010-SIMD-FPU-register-save-restore-is-also-required-on-5.patch
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] applied: [PATCH RESEND container] restore lxc.* entries once
either via recover_config, OR via restore_configuration. non-root behaviour
stays the same.
Tested-by: Oguz Bektas
Signed-off-by: Fabian Grünbichler
---
Note: added Tested-by
src/PVE/API2/LXC.pm | 4 ++--
src/PVE/LXC/Create.pm | 16 ++--
2 files changed, 8 insertions(+), 12 deletions(-)
diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm
index 07280fb..28c9047 100644
--- a/src/PVE/API2/LXC.pm
+++ b/src/PVE/API2/LXC.pm
@@ -353,11 +353,11 @@ __PACKAGE__->register_method({
my $orig_conf;
($orig_conf, $orig_mp_param) =
PVE::LXC::Create::recover_config($archive);
$was_template = delete $orig_conf->{template};
- # When we're root call 'restore_configuration' with
ristricted=0,
+ # When we're root call 'restore_configuration' with
restricted=0,
# causing it to restore the raw lxc entries, among
which there may be
# 'lxc.idmap' entries. We need to make sure that the
extracted contents
# of the container match up with the restored
configuration afterwards:
- $conf->{lxc} = [grep { $_->[0] eq 'lxc.idmap' }
@{$orig_conf->{lxc}}];
+ $conf->{lxc} = $orig_conf->{lxc};
}
}
if ($storage_only_mode) {
diff --git a/src/PVE/LXC/Create.pm b/src/PVE/LXC/Create.pm
index a46a50c..241ca88 100644
--- a/src/PVE/LXC/Create.pm
+++ b/src/PVE/LXC/Create.pm
@@ -176,18 +176,14 @@ sub restore_configuration {
# storage supports creating a template there
next if $key =~ /^template$/;
- if ($key eq 'lxc') {
+ if ($key eq 'lxc' && $restricted) {
my $lxc_list = $oldconf->{'lxc'};
- if ($restricted) {
- warn "skipping custom lxc options, restore manually as
root:\n";
- warn "\n";
- foreach my $lxc_opt (@$lxc_list) {
- warn "$lxc_opt->[0]: $lxc_opt->[1]\n"
- }
- warn "\n";
- } else {
- @{$conf->{$key}} = (@$lxc_list, @{$conf->{$key}});
+ warn "skipping custom lxc options, restore manually as root:\n";
+ warn "\n";
+ foreach my $lxc_opt (@$lxc_list) {
+ warn "$lxc_opt->[0]: $lxc_opt->[1]\n"
}
+ warn "\n";
next;
}
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH pve-zsync] Allow detecting a syncing instance of a job
On September 30, 2019 12:55 pm, Fabian Ebner wrote:
> Before, the check whether a syncing instance of the same job is already
> present
> was inside the locked section. This caused cron to continuously spawn new
> instances of pve-zsync on syncs (or rather groups of syncs) taking longer
> than 15 minutes, see [0] in the forum. This patch introduces a new locked
> section for checking the current status and a new 'waiting' status.
> The 'waiting' status is needed to mark jobs which are currently waiting
> for the lock for syncing. So if job A is syncing and job B is waiting for
> the lock then all new instances of job B will see that one instance is
> already scheduled to sync.
>
> [0]:
> https://forum.proxmox.com/threads/pve-zsync-bug-spawns-endless-cron-processes.58087/
>
> Signed-off-by: Fabian Ebner
> ---
> pve-zsync | 25 -
> 1 file changed, 20 insertions(+), 5 deletions(-)
>
> diff --git a/pve-zsync b/pve-zsync
> index 425ffa2..90c1bb3 100755
> --- a/pve-zsync
> +++ b/pve-zsync
> @@ -19,6 +19,7 @@ my $PVE_DIR = "/etc/pve/local";
> my $QEMU_CONF = "${PVE_DIR}/qemu-server";
> my $LXC_CONF = "${PVE_DIR}/lxc";
> my $LOCKFILE = "$CONFIG_PATH/${PROGNAME}.lock";
> +my $LOCKFILE_STATUS_CHECK = "$CONFIG_PATH/${PROGNAME}_status_check.lock";
> my $PROG_PATH = "$PATH/${PROGNAME}";
> my $INTERVAL = 15;
> my $DEBUG;
> @@ -578,20 +579,34 @@ sub destroy_job {
> sub sync {
> my ($param) = @_;
>
> +my $lock_status_check_fh = IO::File->new("> $LOCKFILE_STATUS_CHECK");
> +die "Can't open Lock File: $LOCKFILE_STATUS_CHECK $!\n" if
> !$lock_status_check_fh;
> +lock($lock_status_check_fh);
> +
> +my $job;
> +eval {
> + $job = get_job($param);
> +};
> +
> +if ($job && defined($job->{state}) && ($job->{state} eq "syncing" ||
> $job->{state} eq "waiting")) {
> + unlock($lock_status_check_fh);
> + die "Job --source $param->{source} --name $param->{name} is already
> scheduled to sync\n";
> +}
> +
> +$job->{state} = "waiting";
> +update_state($job);
> +unlock($lock_status_check_fh);
I don't think we need a (new) lock here - it would only protect against
other processes entering sync(), but they could at worst do the same
change (again)? update_state() already has locking to prevent races
around the modification itself, and this new lock does not protect
against other read-modify-write cycles (like destroy_job, enable_job,
disable_job, but also init which just does write ;)).
if we introduce a new lock, it should be to guard all state r-m-w cycles
against races with eachother (e.g., by implementing Thomas locked()
suggestion, and pulling the lock from update_state there and updating
all call sites). the other locks (sync, update_cron) can also re-use
this locked mechanism, by passing in some lock identifier/filename
(you can also have wrappers around locked, e.g. lock_state, lock_cron
and lock_sync).
> +
> my $lock_fh = IO::File->new("> $LOCKFILE");
> die "Can't open Lock File: $LOCKFILE $!\n" if !$lock_fh;
> lock($lock_fh);
>
> +#job might've changed while we waited for the lock, but we can be sure
> it's not syncing
> my $date = get_date();
> -my $job;
> eval {
> $job = get_job($param);
> };
>
> -if ($job && defined($job->{state}) && $job->{state} eq "syncing") {
> - die "Job --source $param->{source} --name $param->{name} is syncing at
> the moment";
> -}
> -
> my $dest = parse_target($param->{dest});
> my $source = parse_target($param->{source});
>
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH storage] Use bigger timeouts for zfs operations
On October 1, 2019 12:17 pm, Fabian Ebner wrote:
> Seems like 'zfs destroy' can take longer than 5 seconds, see [0].
> I changed the timeout to 15 seconds and also changed the default
> timeout to 10 instead of 5 seconds, to be on the safe side
> for other commands like 'zfs create'.
>
> [0]:
> https://forum.proxmox.com/threads/timeout-beim-l%C3%B6schen-des-entfernen-replikats-bei-entfernung-der-replikation.58467/
NAK, we have a 30s timeout for synchronous API requests that call this,
and they might do more than 2 zfs_requests. we'd rather timeout and have
time to do error handling, than finish successfully sometimes, and die
without cleanup other times.
the real solution for this is to convert the remaining synchronous API
calls that trigger storage operations to async ones, and switch the GUI
over to use the new variants. we had previous discussions about this
issue already. to implement it really nice, we'd need to things:
- tasks with structured return value (to implement async content listing
and similar operations, and make conversion of sync to async API calls
easier)
- light-weight / ephemeral tasks (not included in regular task
lists/log, cleaned up after final poll / 1 day after finishing / ...)
in case of the mentioned report, please investigate whether this call in
'pvesr' context is wrongly treated as sync API call (i.e., is_worker
should maybe return true for pvesr calls?)
>
> Signed-off-by: Fabian Ebner
> ---
> PVE/Storage/ZFSPoolPlugin.pm | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/Storage/ZFSPoolPlugin.pm b/PVE/Storage/ZFSPoolPlugin.pm
> index f66b277..3ce06be 100644
> --- a/PVE/Storage/ZFSPoolPlugin.pm
> +++ b/PVE/Storage/ZFSPoolPlugin.pm
> @@ -182,7 +182,7 @@ sub zfs_request {
> my $msg = '';
> my $output = sub { $msg .= "$_[0]\n" };
>
> -$timeout = PVE::RPCEnvironment->is_worker() ? 60*60 : 5 if !$timeout;
> +$timeout = PVE::RPCEnvironment->is_worker() ? 60*60 : 10 if !$timeout;
>
> run_command($cmd, errmsg => "zfs error", outfunc => $output, timeout =>
> $timeout);
>
> @@ -346,7 +346,7 @@ sub zfs_delete_zvol {
>
> for (my $i = 0; $i < 6; $i++) {
>
> - eval { $class->zfs_request($scfg, undef, 'destroy', '-r',
> "$scfg->{pool}/$zvol"); };
> + eval { $class->zfs_request($scfg, 15, 'destroy', '-r',
> "$scfg->{pool}/$zvol"); };
> if ($err = $@) {
> if ($err =~ m/^zfs error:(.*): dataset is busy.*/) {
> sleep(1);
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 guest-common 01/18] refactor pending changes related config code into AbstractConfig
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> also use a better naming scheme for methods:
>
> split_flagged_list -> parse_pending_delete
> join_flagged_list -> print_pending_delete
> vmconfig_delete_pending_option -> add_to_pending_delete
> vmconfig_undelete_pending_option -> remove_from_pending_delete
> vmconfig_cleanup_pending -> cleanup_pending
>
> Signed-off-by: Oguz Bektas
> ---
> PVE/AbstractConfig.pm | 68 +++
> 1 file changed, 68 insertions(+)
>
> diff --git a/PVE/AbstractConfig.pm b/PVE/AbstractConfig.pm
> index e0d0f10..910ca86 100644
> --- a/PVE/AbstractConfig.pm
> +++ b/PVE/AbstractConfig.pm
> @@ -68,6 +68,74 @@ sub write_config {
> PVE::Cluster::cfs_write_file($cfspath, $conf);
> }
>
> +# Pending changes related
> +
> +sub parse_pending_delete {
> +my ($class, $text) = @_;
conventionally, $text is called $data in our parse_ subs
> +$text ||= '';
> +$text =~ s/[,;]/ /g;
> +$text =~ s/^\s+//;
> +return { map { /^(!?)(.*)$/ && ($2, $1) } ($text =~ /\S+/g) };
like I said in v1 of this part, why not use the opportunity to give this
a sane perl representation?
e.g.,
{
'opt1' => { 'force' => 0 },
'opt2' => { 'force' => 1 },
'opt3' => { 'force' => 0 },
}
then you could still iterate over all pending deletions, and where you
need that info, you can check if $pending_delete_hash->{$opt}->{force}?
right now, if you accidentally do
if ($pending_delete_hash->{$opt})
instead of
if (defined($pending_delete_hash->{$opt}))
or
if (exists($pending_delete_hash->{$opt}))
you probably do the opposite of what you want ;) there is only a single
such caller atm, and that one is correct, but it's very easy to get
wrong IMHO.
> +}
> +
> +sub print_pending_delete {
> +my ($class, $how, $lst) = @_;
> +join $how, map { $lst->{$_} . $_ } keys %$lst;
we always want to join with ',', so why make it configurable? also, this
ain't no l(i)st anymore ;) if we change the $pending_delete_hash
structure, this obviously needs to be adapted to encode the
'forcefulness' correctly.
> +}
> +
> +sub add_to_pending_delete {
> +my ($class, $conf, $key, $force) = @_;
> +
> +delete $conf->{pending}->{$key};
> +my $pending_delete_hash =
> $class->parse_pending_delete($conf->{pending}->{delete});
> +$pending_delete_hash->{$key} = $force ? '!' : '';
> +$conf->{pending}->{delete} = $class->print_pending_delete(',',
> $pending_delete_hash);
> +}
also would need adaption if we don't store '!' in the hash.
> +
> +sub remove_from_pending_delete {
> +my ($class, $conf, $key) = @_;
> +
> +my $pending_delete_hash =
> $class->parse_pending_delete($conf->{pending}->{delete});
> +delete $pending_delete_hash->{$key};
> +
> +if (%$pending_delete_hash) {
> + $conf->{pending}->{delete} = $class->print_pending_delete(',',
> $pending_delete_hash);
> +} else {
> + delete $conf->{pending}->{delete};
> +}
> +}
> +
> +sub cleanup_pending {
> +my ($class, $conf) = @_;
> +
> +# remove pending changes when nothing changed
> +my $changes;
> +foreach my $opt (keys %{$conf->{pending}}) {
> + if (defined($conf->{$opt}) && ($conf->{pending}->{$opt} eq
> $conf->{$opt})) {
> + $changes = 1;
> + delete $conf->{pending}->{$opt};
> + }
> +}
> +
> +my $current_delete_hash =
> $class->parse_pending_delete($conf->{pending}->{delete});
> +my $pending_delete_hash = {};
> +while (my ($opt, $force) = each %$current_delete_hash) {
> + if (defined($conf->{$opt})) {
> + $pending_delete_hash->{$opt} = $force;
> + } else {
> + $changes = 1;
> + }
> +}
> +
> +if (%$pending_delete_hash) {
> + $conf->{pending}->{delete} = $class->print_pending_delete(',',
> $pending_delete_hash);
> +} else {
> + delete $conf->{pending}->{delete};
> +}
> +
> +return $changes;
> +}
> +
> # Lock config file using flock, run $code with @param, unlock config file.
> # $timeout is the maximum time to aquire the flock
> sub lock_config_full {
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 guest-common 02/18] refactor method used by config GET calls into AbstractConfig
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> since this method will be both used by qemu and lxc config GET calls, it
> makes sense to move it into AbstractConfig. only difference is that qemu
> also hides the cipassword when it's set. this can be handled by having
> qemu overwrite the method and add the cipassword code.
>
> Signed-off-by: Oguz Bektas
> ---
> PVE/AbstractConfig.pm | 35 +++
> 1 file changed, 35 insertions(+)
>
> diff --git a/PVE/AbstractConfig.pm b/PVE/AbstractConfig.pm
> index 910ca86..6d3f169 100644
> --- a/PVE/AbstractConfig.pm
> +++ b/PVE/AbstractConfig.pm
> @@ -136,6 +136,41 @@ sub cleanup_pending {
> return $changes;
> }
>
> +sub load_current_config {
like I said when we discussed this offline, that name is at most a
working title :-P the method either returns
A the top level config, ignoring any pending changes (called 'current'
in the API schema)
B the top level config, with any pending changes pseudo-applied (the
opposite of 'current')
C the config of a specific snapshot + digest of whole file (also not very
'current').
we can either split this up into
sub load_config_snapshot => returns C
sub load_ ? => returns A or B, depending on parameter
or make a
sub load_config_filtered {
my ($class, $vmid, $filter) = @_;
where $filter is a hash, currently with
{ 'snapshot' => $snapname }
or
{ 'current' => 1 }
both at once don't make sense, and should probably lead to death.
I'd prefer the split, since the first half for case C, and the second
half for cases A and B don't have much in common anyway. if we have some
other filters in mind that we might add in the future, then the $filter
variant might be more attractive.
> +my ($class, $vmid, $snapname, $current) = @_;
> +
> +my $conf = $class->load_config($vmid);
> +
> +if ($snapname) {
> + my $snapshot = $conf->{snapshots}->{$snapname};
> + die "snapshot '$snapname' does not exist\n" if !defined($snapshot);
> +
> + # we need the digest of the file
> + $snapshot->{digest} = $conf->{digest};
> + $conf = $snapshot;
> +}
> +
> +# take pending changes in
> +if (!$current) {
> + foreach my $opt (keys %{$conf->{pending}}) {
> + next if $opt eq 'delete';
> + my $value = $conf->{pending}->{$opt};
> + next if ref($value); # just to be sure
> + $conf->{$opt} = $value;
> + }
> + my $pending_delete_hash =
> $class->parse_pending_delete($conf->{pending}->{delete});
> + foreach my $opt (keys %$pending_delete_hash) {
> + delete $conf->{$opt} if $conf->{$opt};
> + }
> +}
> +
> +delete $conf->{snapshots};
> +delete $conf->{pending};
> +
> +return $conf;
> +}
> +
> +
> # Lock config file using flock, run $code with @param, unlock config file.
> # $timeout is the maximum time to aquire the flock
> sub lock_config_full {
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container 08/18] add lxc/pending API path
you completely ignored my comments for v1 of this patch? the whole code
is identical to qemu-server's, except for cipassword handling. pending
changes are also encoded identically for both pve-container and
qemu-server, so it makes sense to move this to AbstractConfig or
GuestHelpers.pm with an override for cipassword in QemuConfig.pm/here..
this basically just converts $conf into a different representation for
easier client consumption..
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/API2/LXC.pm | 88 +
> 1 file changed, 88 insertions(+)
>
> diff --git a/src/PVE/API2/LXC.pm b/src/PVE/API2/LXC.pm
> index 07280fb..9c040d1 100644
> --- a/src/PVE/API2/LXC.pm
> +++ b/src/PVE/API2/LXC.pm
> @@ -515,6 +515,7 @@ __PACKAGE__->register_method({
>
> my $res = [
> { subdir => 'config' },
> + { subdir => 'pending' },
> { subdir => 'status' },
> { subdir => 'vncproxy' },
> { subdir => 'termproxy' },
> @@ -1865,4 +1866,91 @@ __PACKAGE__->register_method({
> return $task;
>}});
>
> +__PACKAGE__->register_method({
> +name => 'vm_pending',
> +path => '{vmid}/pending',
> +method => 'GET',
> +proxyto => 'node',
> +description => 'Get container configuration, including pending changes.',
> +permissions => {
> + check => ['perm', '/vms/{vmid}', [ 'VM.Audit' ]],
> +},
> +parameters => {
> + additionalProperties => 0,
> + properties => {
> + node => get_standard_option('pve-node'),
> + vmid => get_standard_option('pve-vmid', { completion =>
> \&PVE::LXC::complete_ctid }),
> + },
> +},
> +returns => {
> + type => "array",
> + items => {
> + type => "object",
> + properties => {
> + key => {
> + description => 'Configuration option name.',
> + type => 'string',
> + },
> + value => {
> + description => 'Current value.',
> + type => 'string',
> + optional => 1,
> + },
> + pending => {
> + description => 'Pending value.',
> + type => 'string',
> + optional => 1,
> + },
> + delete => {
> + description => "Indicates a pending delete request if
> present and not 0.",
> + type => 'integer',
> + minimum => 0,
> + maximum => 1,
> + optional => 1,
> + },
> + },
> + },
> +},
> +code => sub {
> + my ($param) = @_;
> +
> + my $conf = PVE::LXC::Config->load_config($param->{vmid});
> +
> + my $pending_delete_hash =
> PVE::LXC::Config->parse_pending_delete($conf->{pending}->{delete});
> +
> + my $res = [];
> +
> + foreach my $opt (keys %$conf) {
> + next if ref($conf->{$opt});
> + next if $opt eq 'pending';
> + my $item = { key => $opt } ;
> + $item->{value} = $conf->{$opt} if defined($conf->{$opt});
> + $item->{pending} = $conf->{pending}->{$opt} if
> defined($conf->{pending}->{$opt});
> + $item->{delete} = ($pending_delete_hash->{$opt} ? 2 : 1) if exists
> $pending_delete_hash->{$opt};
> +
> + push @$res, $item;
> + }
> +
> + foreach my $opt (keys %{$conf->{pending}}) {
> + next if $opt eq 'delete';
> + next if ref($conf->{pending}->{$opt});
> + next if defined($conf->{$opt});
> + my $item = { key => $opt };
> + $item->{pending} = $conf->{pending}->{$opt};
> +
> + push @$res, $item;
> + }
> +
> + # FIXME: $force delete is not implemented for CTs
> + while (my ($opt, undef) = each %$pending_delete_hash) {
> + next if $conf->{pending}->{$opt};
> + next if $conf->{$opt};
> + my $item = { key => $opt, delete => 1 };
> + push @$res, $item;
> + }
> +
> + return $res;
> +
> +}});
> +
> 1;
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container 10/18] adapt CT config parser for pending changes
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> config parser can now read/write [pve:pending] section. this was named
> such, instead of [PENDING], after on- and offline discussion regarding
> namespacing the pending section and snapshots.
>
> this also adds an optional non-capturing regex group into the parser for
> [snap: snapname] entries which can be supported in PVE 7.0
like Thomas said, please split these two (the latter is not even related
to this series at all! while I agree it's good to do it now, together
with the same change on qemu-server's part, let's not make this series
any bigger than it needs to be ;))
one question inline
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/LXC/Config.pm | 37 -
> 1 file changed, 32 insertions(+), 5 deletions(-)
>
> diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
> index 9790345..47bd4bb 100644
> --- a/src/PVE/LXC/Config.pm
> +++ b/src/PVE/LXC/Config.pm
> @@ -751,6 +751,7 @@ sub parse_pct_config {
> my $res = {
> digest => Digest::SHA::sha1_hex($raw),
> snapshots => {},
> + pending => {},
> };
>
> $filename =~ m|/lxc/(\d+).conf$|
> @@ -766,7 +767,14 @@ sub parse_pct_config {
> foreach my $line (@lines) {
> next if $line =~ m/^\s*$/;
>
> - if ($line =~ m/^\[([a-z][a-z0-9_\-]+)\]\s*$/i) {
> + if ($line =~ m/^\[pve:pending\]\s*$/i) {
> + $section = 'pending';
> + $conf->{description} = $descr if $descr;
> + $descr = '';
> + $conf = $res->{$section} = {};
> + next;
> + } elsif ($line =~ m/^\[(?:snap:)?([a-z][a-z0-9_\-]+)\]\s*$/i) {
> + # extended regex for namespacing snapshots in PVE 7.0
> $section = $1;
> $conf->{description} = $descr if $descr;
> $descr = '';
> @@ -794,6 +802,13 @@ sub parse_pct_config {
> $descr .= PVE::Tools::decode_text($2);
> } elsif ($line =~ m/snapstate:\s*(prepare|delete)\s*$/) {
> $conf->{snapstate} = $1;
> + } elsif ($line =~ m/^delete:\s*(.*\S)\s*$/) {
> + my $value = $1;
> + if ($section eq 'pending') {
> + $conf->{delete} = $value;
> + } else {
> + warn "vm $vmid - property 'delete' is only allowed in
> [pve:pending]\n";
> + }
> } elsif ($line =~ m/^([a-z][a-z_]*\d*):\s*(\S.*)\s*$/) {
> my $key = $1;
> my $value = $2;
> @@ -832,14 +847,19 @@ sub write_pct_config {
> }
>
> my $generate_raw_config = sub {
> - my ($conf) = @_;
> + my ($conf, $pending) = @_;
>
> my $raw = '';
>
> # add description as comment to top of file
> - my $descr = $conf->{description} || '';
> - foreach my $cl (split(/\n/, $descr)) {
> - $raw .= '#' . PVE::Tools::encode_text($cl) . "\n";
> + if (defined(my $descr = $conf->{description})) {
> + if ($descr) {
> + foreach my $cl (split(/\n/, $descr)) {
> + $raw .= '#' . PVE::Tools::encode_text($cl) . "\n";
> + }
> + } else {
> + $raw .= "#\n" if $pending;
> + }
that seems wrong.. why handle a pending description like that?
> }
>
> foreach my $key (sort keys %$conf) {
> @@ -864,7 +884,14 @@ sub write_pct_config {
>
> my $raw = &$generate_raw_config($conf);
>
> +if (scalar(keys %{$conf->{pending}})){
> + $raw .= "\n[pve:pending]\n";
> + $raw .= &$generate_raw_config($conf->{pending}, 1);
> +}
> +
> foreach my $snapname (sort keys %{$conf->{snapshots}}) {
> + # TODO: namespace snapshots for PVE 7.0
> + #$raw .= "\n[snap:$snapname]\n";
> $raw .= "\n[$snapname]\n";
> $raw .= &$generate_raw_config($conf->{snapshots}->{$snapname});
> }
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container 10/18] adapt CT config parser for pending changes
On October 2, 2019 12:22 pm, Thomas Lamprecht wrote:
> On 9/30/19 2:44 PM, Oguz Bektas wrote:
>> config parser can now read/write [pve:pending] section. this was named
>> such, instead of [PENDING], after on- and offline discussion regarding
>> namespacing the pending section and snapshots.
>>
>> this also adds an optional non-capturing regex group into the parser for
>> [snap: snapname] entries which can be supported in PVE 7.0
>
> 1. completely nothing to do with pending changes itself -> separate patch
>
> 2. they could be supported in 6.x already? PVE cluster host need to be on the
> same version of software, we guarantee only that old -> new and new <-> new
> works, so doing this now is no issue... Else we never could add any new
> feature in a stable release..
>
> Also this is missing from qemu-server?
the problem is that this is not just a case of new -> old does not work,
but it's a case of new -> old silently drops parts (/most) of your guest
config.. since it's just nice to have, but not blocking anything
important, we should postpone it to 7.0 IMHO (where we can assume that
the 'old' 6.x parser already supports it, so the fallout is basically
non-existent at that point).
>
>>
>> Signed-off-by: Oguz Bektas
>> ---
>> src/PVE/LXC/Config.pm | 37 -
>> 1 file changed, 32 insertions(+), 5 deletions(-)
>>
>> diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
>> index 9790345..47bd4bb 100644
>> --- a/src/PVE/LXC/Config.pm
>> +++ b/src/PVE/LXC/Config.pm
>> @@ -751,6 +751,7 @@ sub parse_pct_config {
>> my $res = {
>> digest => Digest::SHA::sha1_hex($raw),
>> snapshots => {},
>> +pending => {},
>> };
>>
>> $filename =~ m|/lxc/(\d+).conf$|
>> @@ -766,7 +767,14 @@ sub parse_pct_config {
>> foreach my $line (@lines) {
>> next if $line =~ m/^\s*$/;
>>
>> -if ($line =~ m/^\[([a-z][a-z0-9_\-]+)\]\s*$/i) {
>> +if ($line =~ m/^\[pve:pending\]\s*$/i) {
>
> why not add the new stuff to the end of the elsif block? less churn..
for qemu-server, we need to add it up-front to parse it earlier then a
snapshot called pending, this would keep the two parsers (that we
hopefully unify some day) closer together. OTOH, we'd drop all of this
if we unify it anyway, so..
>
>> +$section = 'pending';
>> +$conf->{description} = $descr if $descr;
>> +$descr = '';
>> +$conf = $res->{$section} = {};
>> +next;
>> +} elsif ($line =~ m/^\[(?:snap:)?([a-z][a-z0-9_\-]+)\]\s*$/i) {
>> +# extended regex for namespacing snapshots in PVE 7.0
>> $section = $1;
>> $conf->{description} = $descr if $descr;
>> $descr = '';
>> @@ -794,6 +802,13 @@ sub parse_pct_config {
>> $descr .= PVE::Tools::decode_text($2);
>> } elsif ($line =~ m/snapstate:\s*(prepare|delete)\s*$/) {
>> $conf->{snapstate} = $1;
>> +} elsif ($line =~ m/^delete:\s*(.*\S)\s*$/) {
>> +my $value = $1;
>> +if ($section eq 'pending') {
>> +$conf->{delete} = $value;
>> +} else {
>> +warn "vm $vmid - property 'delete' is only allowed in
>> [pve:pending]\n";
>> +}
>> } elsif ($line =~ m/^([a-z][a-z_]*\d*):\s*(\S.*)\s*$/) {
>> my $key = $1;
>> my $value = $2;
>> @@ -832,14 +847,19 @@ sub write_pct_config {
>> }
>>
>> my $generate_raw_config = sub {
>> -my ($conf) = @_;
>> +my ($conf, $pending) = @_;
>>
>> my $raw = '';
>>
>> # add description as comment to top of file
>> -my $descr = $conf->{description} || '';
>> -foreach my $cl (split(/\n/, $descr)) {
>> -$raw .= '#' . PVE::Tools::encode_text($cl) . "\n";
>> +if (defined(my $descr = $conf->{description})) {
>> +if ($descr) {
>> +foreach my $cl (split(/\n/, $descr)) {
>> +$raw .= '#' . PVE::Tools::encode_text($cl) . "\n";
>> +}
>> +} else {
>> +$raw .= "#\n" if $pending;
>> +}
>> }
>>
>> foreach my $key (sort keys %$conf) {
>> @@ -864,7 +884,14 @@ sub write_pct_config {
>>
>> my $raw = &$generate_raw_config($conf);
>>
>> +if (scalar(keys %{$conf->{pending}})){
>> +$raw .= "\n[pve:pending]\n";
>> +$raw .= &$generate_raw_config($conf->{pending}, 1);
>> +}
>> +
>> foreach my $snapname (sort keys %{$conf->{snapshots}}) {
>> +# TODO: namespace snapshots for PVE 7.0
>> +#$raw .= "\n[snap:$snapname]\n";
>> $raw .= "\n[$snapname]\n";
>> $raw .= &$generate_raw_config($conf->{snapshots}->{$snapname});
>> }
>>
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container 16/18] adapt config PUT method for the new update_pct_config
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> we don't need to extract 'delete' here, instead we pass it all as $param
> and extract 'delete', 'revert' and most other things in
> update_pct_config
I already asked that in v1 - wouldn't it make sense to keep the
parameter checks in the API call, and pass the already "verified"
$delete and $revert to update_pct_config? this would avoid the need of
adding new module dependencies from PVE::LXC::Config to
PVE::RPCEnvironment and PVE::Exception in #17..
if you see a good reason for moving all that to update_pct_config,
please say so (either in the changelog of the patch, or as reply to the
original review) and don't silently ignore open questions ;)
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/API2/LXC/Config.pm | 55 --
> 1 file changed, 5 insertions(+), 50 deletions(-)
>
> diff --git a/src/PVE/API2/LXC/Config.pm b/src/PVE/API2/LXC/Config.pm
> index 2c036f5..46d8e2f 100644
> --- a/src/PVE/API2/LXC/Config.pm
> +++ b/src/PVE/API2/LXC/Config.pm
> @@ -119,58 +119,10 @@ __PACKAGE__->register_method({
> code => sub {
> my ($param) = @_;
>
> - my $rpcenv = PVE::RPCEnvironment::get();
> - my $authuser = $rpcenv->get_user();
> -
> my $node = extract_param($param, 'node');
> my $vmid = extract_param($param, 'vmid');
> -
> my $digest = extract_param($param, 'digest');
>
> - die "no options specified\n" if !scalar(keys %$param);
> -
> - my $delete_str = extract_param($param, 'delete');
> - my @delete = PVE::Tools::split_list($delete_str);
> -
> - PVE::LXC::check_ct_modify_config_perm($rpcenv, $authuser, $vmid, undef,
> {}, [@delete]);
> -
> - foreach my $opt (@delete) {
> - raise_param_exc({ delete => "you can't use '-$opt' and -delete
> $opt' at the same time" })
> - if defined($param->{$opt});
> -
> - if (!PVE::LXC::Config->option_exists($opt)) {
> - raise_param_exc({ delete => "unknown option '$opt'" });
> - }
> - }
> -
> - PVE::LXC::check_ct_modify_config_perm($rpcenv, $authuser, $vmid, undef,
> $param, []);
> -
> - my $storage_cfg = cfs_read_file("storage.cfg");
> -
> - my $repl_conf = PVE::ReplicationConfig->new();
> - my $is_replicated = $repl_conf->check_for_existing_jobs($vmid, 1);
> - if ($is_replicated) {
> - PVE::LXC::Config->foreach_mountpoint_full($param, 0, sub {
> - my ($opt, $mountpoint) = @_;
> - my $volid = $mountpoint->{volume};
> - return if !$volid || !($mountpoint->{replicate}//1);
> - if ($mountpoint->{type} eq 'volume') {
> - my ($storeid, $format);
> - if ($volid =~ $PVE::LXC::NEW_DISK_RE) {
> - $storeid = $1;
> - $format = $mountpoint->{format} ||
> PVE::Storage::storage_default_format($storage_cfg, $storeid);
> - } else {
> - ($storeid, undef) =
> PVE::Storage::parse_volume_id($volid, 1);
> - $format = (PVE::Storage::parse_volname($storage_cfg,
> $volid))[6];
> - }
> - return if PVE::Storage::storage_can_replicate($storage_cfg,
> $storeid, $format);
> - my $scfg = PVE::Storage::storage_config($storage_cfg,
> $storeid);
> - return if $scfg->{shared};
> - }
> - die "cannot add non-replicatable volume to a replicated VM\n";
> - });
> - }
> -
> my $code = sub {
>
> my $conf = PVE::LXC::Config->load_config($vmid);
> @@ -180,10 +132,13 @@ __PACKAGE__->register_method({
>
> my $running = PVE::LXC::check_running($vmid);
>
> - PVE::LXC::Config->update_pct_config($vmid, $conf, $running, $param,
> \@delete);
> + die "no options specified\n" if !scalar(keys %$param);
> +
> + PVE::LXC::Config->update_pct_config($vmid, $conf, $running, $param);
> + $conf = PVE::LXC::Config->load_config($vmid);
>
> - PVE::LXC::Config->write_config($vmid, $conf);
> PVE::LXC::update_lxc_config($vmid, $conf);
> +
> };
>
> PVE::LXC::Config->lock_config($vmid, $code);
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container 17/18] rework update_pct_config to write into pending section
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> use vmconfig_hotplug_pending or vmconfig_apply_pending to apply the
> pending changes, depending on whether the CT is on- or offline.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/LXC/Config.pm | 334 ++
> 1 file changed, 106 insertions(+), 228 deletions(-)
>
> diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
> index 47bd4bb..14c26bc 100644
> --- a/src/PVE/LXC/Config.pm
> +++ b/src/PVE/LXC/Config.pm
> @@ -4,11 +4,13 @@ use strict;
> use warnings;
>
> use PVE::AbstractConfig;
> +use PVE::RPCEnvironment;
> use PVE::Cluster qw(cfs_register_file);
> use PVE::GuestHelpers;
> use PVE::INotify;
> +use PVE::Exception qw(raise_param_exc);
> use PVE::JSONSchema qw(get_standard_option);
> -use PVE::Tools;
> +use PVE::Tools qw(extract_param);
>
> use base qw(PVE::AbstractConfig);
>
> @@ -900,267 +902,143 @@ sub write_pct_config {
> }
>
> sub update_pct_config {
> -my ($class, $vmid, $conf, $running, $param, $delete) = @_;
> +my ($class, $vmid, $conf, $running, $param) = @_;
>
> -my @nohotplug;
> +my $rpcenv = PVE::RPCEnvironment::get();
> +my $authuser = $rpcenv->get_user();
>
> -my $new_disks = 0;
> -my @deleted_volumes;
> +my $delete_str = extract_param($param, 'delete');
> +my @delete = PVE::Tools::split_list($delete_str);
> +my $revert_str = extract_param($param, 'revert');
> +my @revert = PVE::Tools::split_list($revert_str);
I am still not convinced that hashes for delete and revert would not be
more readable down below.
>
> -my $rootdir;
> -if ($running) {
> - my $pid = PVE::LXC::find_lxc_pid($vmid);
> - $rootdir = "/proc/$pid/root";
> +PVE::LXC::check_ct_modify_config_perm($rpcenv, $authuser, $vmid, undef,
> {}, [@delete]);
still no permission checks for reverting?
> +
> +foreach my $opt (PVE::Tools::split_list($revert_str)) {
should be @revert..
> + raise_param_exc({ revert => "unknown option '$opt'" })
> + if !$class->option_exists($opt);
> +
> + raise_param_exc({ delete => "you can't use '-$opt' and " .
wrong parameter to raise the exception for
> + "'-revert $opt' at the same time" })
> + if defined($param->{$opt});
> +
> + push @revert, $opt;
> }
@revert now contains each reverted option twice ;)
>
> -my $hotplug_error = sub {
> - if ($running) {
> - push @nohotplug, @_;
> - return 1;
> - } else {
> - return 0;
> - }
> -};
> +foreach my $opt (@revert) {
> + delete $conf->{pending}->{$opt};
> + # FIXME: maybe check before doing this?
check what?
> + $class->remove_from_pending_delete($conf, $opt); # remove from deletion
> queue
> +}
> +$class->write_config($vmid, $conf);
do we really need to write_config here? this was just a logical change,
and if we die afterwards the API call has failed and the changes so
far where side-effect free anyway..
>
> -if (defined($delete)) {
> - foreach my $opt (@$delete) {
> - if (!exists($conf->{$opt})) {
> - # silently ignore
> - next;
> - }
> +foreach my $opt (@delete) {
> + raise_param_exc({ delete => "you can't use '-$opt' and -delete $opt' at
> the same time" })
> + if defined($param->{$opt});
>
> - if ($opt eq 'memory' || $opt eq 'rootfs') {
> - die "unable to delete required option '$opt'\n";
> - } elsif ($opt eq 'hostname') {
> - delete $conf->{$opt};
> - } elsif ($opt eq 'swap') {
> - delete $conf->{$opt};
> - PVE::LXC::write_cgroup_value("memory", $vmid,
> - "memory.memsw.limit_in_bytes", -1);
> - } elsif ($opt eq 'description' || $opt eq 'onboot' || $opt eq
> 'startup' || $opt eq 'hookscript') {
> - delete $conf->{$opt};
> - } elsif ($opt eq 'nameserver' || $opt eq 'searchdomain' ||
> - $opt eq 'tty' || $opt eq 'console' || $opt eq 'cmode') {
> - next if $hotplug_error->($opt);
> - delete $conf->{$opt};
> - } elsif ($opt eq 'cores') {
> - delete $conf->{$opt}; # rest is handled by pvestatd
> - } elsif ($opt eq 'cpulimit') {
> - PVE::LXC::write_cgroup_value("cpu", $vmid, "cpu.cfs_quota_us",
> -1);
> - delete $conf->{$opt};
> - } elsif ($opt eq 'cpuunits') {
> - PVE::LXC::write_cgroup_value("cpu", $vmid, "cpu.shares",
> $confdesc->{cpuunits}->{default});
> - delete $conf->{$opt};
> - } elsif ($opt =~ m/^net(\d)$/) {
> - delete $conf->{$opt};
> - next if !$running;
> - my $netid = $1;
> - PVE::Network::veth_delete("veth${vmid}i$netid");
> - } elsif ($opt eq 'protection') {
> - delete $conf->{$opt};
> - } elsif ($opt =
Re: [pve-devel] [PATCH v2 container 18/18] add vmconfig_hotplug_pending and vmconfig_apply_pending
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> vmconfig_hotplug_pending is responsible for checking if a key/value pair in
> the
> pending section can be hotplugged, if yes; perform a generic replace,
> or perform specific actions for hotplugging the special cases.
>
> vmconfig_apply_pending is only supposed to be called when ct isn't live.
>
> Signed-off-by: Oguz Bektas
> ---
> src/PVE/LXC.pm| 14 +--
> src/PVE/LXC/Config.pm | 199 --
> 2 files changed, 203 insertions(+), 10 deletions(-)
>
> diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
> index 65c41f5..f91e27d 100644
> --- a/src/PVE/LXC.pm
> +++ b/src/PVE/LXC.pm
> @@ -1632,7 +1632,7 @@ sub alloc_disk {
>
> our $NEW_DISK_RE = qr/^([^:\s]+):(\d+(\.\d+)?)$/;
> sub create_disks {
> -my ($storecfg, $vmid, $settings, $conf) = @_;
> +my ($storecfg, $vmid, $settings, $conf, $pending) = @_;
>
> my $vollist = [];
>
> @@ -1659,10 +1659,14 @@ sub create_disks {
> push @$vollist, $volid;
> $mountpoint->{volume} = $volid;
> $mountpoint->{size} = $size_kb * 1024;
> - $conf->{$ms} =
> PVE::LXC::Config->print_ct_mountpoint($mountpoint, $ms eq 'rootfs');
> + if ($pending) {
> + $conf->{pending}->{$ms} =
> PVE::LXC::Config->print_ct_mountpoint($mountpoint, $ms eq 'rootfs');
> + } else {
> + $conf->{$ms} =
> PVE::LXC::Config->print_ct_mountpoint($mountpoint, $ms eq 'rootfs');
> + }
> } else {
> -# use specified/existing volid/dir/device
> -$conf->{$ms} =
> PVE::LXC::Config->print_ct_mountpoint($mountpoint, $ms eq 'rootfs');
> + # use specified/existing volid/dir/device
> + $conf->{$ms} =
> PVE::LXC::Config->print_ct_mountpoint($mountpoint, $ms eq 'rootfs');
> }
> });
>
> @@ -1676,7 +1680,7 @@ sub create_disks {
> # free allocated images on error
> if (my $err = $@) {
> destroy_disks($storecfg, $vollist);
> -die $err;
> + die $err;
> }
> return $vollist;
> }
> diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
> index 14c26bc..10dfc75 100644
> --- a/src/PVE/LXC/Config.pm
> +++ b/src/PVE/LXC/Config.pm
> @@ -1177,6 +1177,194 @@ sub option_exists {
> }
> # END JSON config code
>
> +my $LXC_FASTPLUG_OPTIONS= {
> +'description' => 1,
> +'onboot' => 1,
> +'startup' => 1,
> +'protection' => 1,
> +'hostname' => 1,
> +'hookscript' => 1,
> +'cores' => 1,
> +'tags' => 1,
> +};
> +
> +sub vmconfig_hotplug_pending {
> +my ($class, $vmid, $conf, $storecfg, $selection, $errors) = @_;
> +
> +my $pid = PVE::LXC::find_lxc_pid($vmid);
> +my $rootdir = "/proc/$pid/root";
> +
> +my $add_error = sub {
> + my ($opt, $msg) = @_;
> + $errors->{$opt} = "hotplug problem - $msg";
> +};
> +
> +my $changes;
> +foreach my $opt (keys %{$conf->{pending}}) { # add/change
> + next if $selection && !$selection->{$opt};
> + if ($LXC_FASTPLUG_OPTIONS->{$opt}) {
> + $conf->{$opt} = delete $conf->{pending}->{$opt};
> + $changes = 1;
> + }
> +}
> +
> +if ($changes) {
> + $class->write_config($vmid, $conf);
> +}
> +
> +# There's no separate swap size to configure, there's memory and "total"
> +# memory (iow. memory+swap). This means we have to change them together.
> +my $hotplug_memory_done;
> +my $hotplug_memory = sub {
> + my ($wanted_memory, $wanted_swap) = @_;
> + my $old_memory = ($conf->{memory} || $confdesc->{memory}->{default});
> + my $old_swap = ($conf->{swap} || $confdesc->{swap}->{default});
> +
> + $wanted_memory //= $old_memory;
> + $wanted_swap //= $old_swap;
> +
> + my $total = $wanted_memory + $wanted_swap;
> + my $old_total = $old_memory + $old_swap;
> +
> + if ($total > $old_total) {
> + PVE::LXC::write_cgroup_value("memory", $vmid,
> + "memory.memsw.limit_in_bytes",
> + int($total*1024*1024));
> + PVE::LXC::write_cgroup_value("memory", $vmid,
> + "memory.limit_in_bytes",
> + int($wanted_memory*1024*1024));
> + } else {
> + PVE::LXC::write_cgroup_value("memory", $vmid,
> + "memory.limit_in_bytes",
> + int($wanted_memory*1024*1024));
> + PVE::LXC::write_cgroup_value("memory", $vmid,
> + "memory.memsw.limit_in_bytes",
> + int($total*1024*1024));
> + }
> + $hotplug_memory_done = 1;
> +};
> +
> +my $pending_delete_hash =
> $class->parse_pending_delete($conf->{pending}->{delete});
> +# FIXME: $force deletion is not implemented for CTs
> +while (my ($opt, unde
Re: [pve-devel] [PATCH v2 00/18] lxc pending changes
On September 30, 2019 2:44 pm, Oguz Bektas wrote:
> this series makes it possible to add/delete/revert pending changes in
> the backend for containers.
>
> this v2 took longer than expected, mainly because there were small bugs
> popping up everywhere, everytime i tried to change anything :)
>
> big thanks to fabian for the extensive review on v1, and for putting up
> with me :D
thanks for the v2! some detailed feedback on invidivual patches. except
for the naming/order/splitting of commits (see below) and some open
questions, this already looks quite good!
I'll give v3 a more detailed test run again, since some stuff will
probably be moved/refactored/rewritten again.
>
>
> v1 -> v2:
> * better refactoring into guest-common, as suggested by fabian and
> thomas
> * fixed up some bugs (probably added some too):
> * backup/restore
> * cloning
> * unlimited swap bug
> * mountpoint handling (more on that)
> * other stuff that i can't remember right now :D
> * small changes with style
> * in v1, mountpoints were a special-special case, since they were
> handled earlier in code before going into the hotplug/apply pending
> routines. after much discussion, they're now created/deleted in these
> phases instead of update_pct_config. unused disks can still be removed
> directly.
> * some other small changes around helper functions, mainly adding them
> support for handling $conf->{pending}
>
> pve-container:
>
> Oguz Bektas (11):
> add lxc/pending API path
> add 'pct pending'
> adapt CT config parser for pending changes
> use load_current_config for config GET call
> skip pending changes while cloning
> skip pending changes while taking backup
> apply pending changes during container start
> add revert parameter to config PUT
> adapt config PUT method for the new update_pct_config
> rework update_pct_config to write into pending section
> add vmconfig_hotplug_pending and vmconfig_apply_pending
the order here is a bit wrong IMHO
#12 and #13 can go further up (they just make the later commits safe,
and don't change anything without the later commits - good for bisecting
;))
#14 relies on a method only introduced in #18
#15 adds an API parameter, but no code to handle it (I already told you
this in v1 ;))
#16 reworks that API call further to adapt for changes that only come in
patch #17
#17 relies on methods only introduced in #18
so, #18 needs to come first (if nobody calls those methods yet, it is
clear that that change alone cannot break anything!)
#14 can come after, since it only has an effect if pending changes can
exist (which, barring manual editing, is impossible with just the
patches up to this point)
#15-17 are a single logical change (switching from "directly apply what
is possible, die otherwise" to "write to pending, and either apply or
hotplug depending on whether the CT is running"). no part of that works
without the other, so just squash them into a single commit unless you
find a *meaningful* way to split them up. sometimes commits are big, and
that is okay as long as they are not big because lots of un- or
semi-related stuff got thrown together.
>
> src/PVE/API2/LXC.pm| 89 ++
> src/PVE/API2/LXC/Config.pm | 82 ++
> src/PVE/CLI/pct.pm | 3 +
> src/PVE/LXC.pm | 21 +-
> src/PVE/LXC/Config.pm | 566 +
> src/PVE/VZDump/LXC.pm | 1 +
> 6 files changed, 457 insertions(+), 305 deletions(-)
>
> qemu-server:
>
> Oguz Bektas (4):
> overwrite load_current_config from AbstractConfig
> use load_current_config for config GET call
> refactor pending changes related code
> use format_pending from GuestHelpers for 'qm pending' command
those could almost be one commit, since they all do the same:
code moved to AbstractConfig/GuestHelpers, drop own copy and use shared
one. at least the first two should be squashed for sure.
>
> PVE/API2/Qemu.pm | 49 +
> PVE/CLI/qm.pm | 28 +
> PVE/QemuConfig.pm | 14 +
> PVE/QemuServer.pm | 79 +--
> 4 files changed, 31 insertions(+), 139 deletions(-)
>
> pve-guest-common:
> Oguz Bektas (3):
> refactor pending changes related config code into AbstractConfig
> refactor method used by config GET calls into AbstractConfig
> refactor code from qm/pct 'pending' call into AbstractConfig
>
> PVE/AbstractConfig.pm | 103 ++
> PVE/GuestHelpers.pm | 26 +++
> 2 files changed, 129 insertions(+)
>
> --
> 2.20.1
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH access-control] parse_user_cfg: correctly parse group names in ACLs
usernames are allowed to start with '@', so adding a user '@test@pve'
and adding it to an ACL should work, instead of ignoring that part of
the ACL entry.
note: there is no potential for user and group to be confused, since a
username must end with '@REALM', and a group reference in an ACL can
only contain one '@' (as first character).
Signed-off-by: Fabian Grünbichler
---
Notes:
alternatively, we could also disallow usernames starting with '@', but those
are currently working as long as they just have ACLs via groups, and not
directly..
PVE/AccessControl.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index 44f4a01..6ea0b85 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -974,8 +974,9 @@ sub parse_user_config {
}
foreach my $ug (split_list($uglist)) {
- if ($ug =~ m/^@(\S+)$/) {
- my $group = $1;
+ my ($group) = $ug =~ m/^@(\S+)$/;
+
+ if ($group && verify_groupname($group, 1)) {
if ($cfg->{groups}->{$group}) { # group exists
$cfg->{acl}->{$path}->{groups}->{$group}->{$role} = $propagate;
} else {
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v2 container 16/18] adapt config PUT method for the new update_pct_config
On October 3, 2019 4:32 pm, Oguz Bektas wrote:
> On Wed, Oct 02, 2019 at 01:52:58PM +0200, Fabian Grünbichler wrote:
>> On September 30, 2019 2:44 pm, Oguz Bektas wrote:
>> > we don't need to extract 'delete' here, instead we pass it all as $param
>> > and extract 'delete', 'revert' and most other things in
>> > update_pct_config
>>
>> I already asked that in v1 - wouldn't it make sense to keep the
>> parameter checks in the API call, and pass the already "verified"
>> $delete and $revert to update_pct_config? this would avoid the need of
>> adding new module dependencies from PVE::LXC::Config to
>> PVE::RPCEnvironment and PVE::Exception in #17..
>>
>> if you see a good reason for moving all that to update_pct_config,
>> please say so (either in the changelog of the patch, or as reply to the
>> original review) and don't silently ignore open questions ;)
>
> i actually wanted to move even more. in qemu api it's like:
>
> ---
>
> ---
> __PACKAGE__->register_method({
> name => 'update_vm_async',
> path => '{vmid}/config',
> method => 'POST',
> ...
>
> returns => {
> type => 'string',
> optional => 1,
> },
> code => $update_vm_api,
> });
> ---
>
> and
>
> ---
> __PACKAGE__->register_method({
> name => 'update_vm',
> path => '{vmid}/config',
> method => 'PUT',
> protected => 1,
> proxyto => 'node',
> description => "Set virtual machine options (synchrounous API) - You
> should consider using the POST method instead for any actions involving
> hotplug or storage allocation.",
> ...
> returns => { type => 'null' },
> code => sub {
> my ($param) = @_;
> &$update_vm_api($param, 1);
> return undef;
> }
> });
> ---
>
> which i find more readable than the current situation in container code, and
> the checks are in the $update_vm_api sub.
yes, but in qemu-server ALL of that is in the API file, not in
QemuServer.pm ;) you can try to see how moving update_pct_config to the
API feels, all callers are in the API anyway.
>
> now i don't know which is better in terms of implementation, so if you
> believe that the checks should stay in api, then we can do it like that.
>
>>
>> >
>> > Signed-off-by: Oguz Bektas
>> > ---
>> > src/PVE/API2/LXC/Config.pm | 55 --
>> > 1 file changed, 5 insertions(+), 50 deletions(-)
>> >
>> > diff --git a/src/PVE/API2/LXC/Config.pm b/src/PVE/API2/LXC/Config.pm
>> > index 2c036f5..46d8e2f 100644
>> > --- a/src/PVE/API2/LXC/Config.pm
>> > +++ b/src/PVE/API2/LXC/Config.pm
>> > @@ -119,58 +119,10 @@ __PACKAGE__->register_method({
>> > code => sub {
>> >my ($param) = @_;
>> >
>> > - my $rpcenv = PVE::RPCEnvironment::get();
>> > - my $authuser = $rpcenv->get_user();
>> > -
>> >my $node = extract_param($param, 'node');
>> >my $vmid = extract_param($param, 'vmid');
>> > -
>> >my $digest = extract_param($param, 'digest');
>> >
>> > - die "no options specified\n" if !scalar(keys %$param);
>> > -
>> > - my $delete_str = extract_param($param, 'delete');
>> > - my @delete = PVE::Tools::split_list($delete_str);
>> > -
>> > - PVE::LXC::check_ct_modify_config_perm($rpcenv, $authuser, $vmid, undef,
>> > {}, [@delete]);
>> > -
>> > - foreach my $opt (@delete) {
>> > - raise_param_exc({ delete => "you can't use '-$opt' and -delete
>> > $opt' at the same time" })
>> > - if defined($param->{$opt});
>> > -
>> > - if (!PVE::LXC::Config->option_exists($opt)) {
>> > - raise_param_exc({ delete => "unknown option '$opt'" });
>> > - }
>> > - }
>> > -
>> > - PVE::LXC::check_ct_modify_config_perm($rpcenv, $authuser, $vmid, undef,
>> > $param, []);
>> > -
>> > - my $storage_cfg = cfs_read_file("storage.cfg");
>> > -
>> > - my
Re: [pve-devel] [PATCH qemu-server] fix #2395: check for iscsi on efidisk creation
On October 4, 2019 10:55 am, Dominik Csapak wrote:
> otherwise qemu-img uses its default intiator id which may not have access
isn't this also missing from
PVE/QemuServer/Cloudinit.pm
PVE/QemuServer/ImportDisk.pm
and isn't the 'cache=none' addition for ZFS missing from all three?
maybe it would make sense to have a 'convert volume to qemu-img options'
helper that does all of that correctly in one place?
>
> Signed-off-by: Dominik Csapak
> ---
> PVE/QemuServer.pm | 12 +++-
> 1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 8376260..2267a3a 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -7277,8 +7277,18 @@ sub create_efidisk($) {
> PVE::Storage::activate_volumes($storecfg, [$volid]);
>
> my $path = PVE::Storage::path($storecfg, $volid);
> +
> +my $targetopts = ['-O', $fmt];
> +if ($path =~ m|^iscsi://|) {
> + $targetopts = ['--target-image-opts'];
> + $path = convert_iscsi_path($path);
> +}
> +
> +my $cmd = ['/usr/bin/qemu-img', 'convert', '-n', '-f', 'raw' ];
> +push @$cmd, @$targetopts, $ovmf_vars, $path;
> +
> eval {
> - run_command(['/usr/bin/qemu-img', 'convert', '-n', '-f', 'raw', '-O',
> $fmt, $ovmf_vars, $path]);
> + run_command($cmd);
> };
> die "Copying EFI vars image failed: $@" if $@;
>
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server] Fix #2171: VM statefile was not activated
On October 7, 2019 2:41 pm, Alwin Antreich wrote:
> Machine states that were created on snapshots with memory could not be
> restored on rollback. The state volume was not activated so KVM couldn't
> load the state.
>
> This patch moves the path generation into vm_start and de-/activates the
> state volume.
alternatively, the following could also work and re-use more code so
that we don't miss the next special handling of some corner case.
rolling back from a snapshot with state is just like resuming, but we
want to keep the statefile instead of deleting it.
(untested):
diff --git a/PVE/QemuConfig.pm b/PVE/QemuConfig.pm
index edbf1a7..b70c276 100644
--- a/PVE/QemuConfig.pm
+++ b/PVE/QemuConfig.pm
@@ -358,9 +358,7 @@ sub __snapshot_rollback_vm_stop {
sub __snapshot_rollback_vm_start {
my ($class, $vmid, $vmstate, $data) = @_;
-my $storecfg = PVE::Storage::config();
-my $statefile = PVE::Storage::path($storecfg, $vmstate);
-PVE::QemuServer::vm_start($storecfg, $vmid, $statefile, undef, undef,
undef, $data->{forcemachine});
+PVE::QemuServer::vm_start($storecfg, $vmid, $vmstate, undef, undef, undef,
$data->{forcemachine});
}
sub __snapshot_rollback_get_unused {
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 8376260..f2d19e1 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -5418,6 +5418,11 @@ sub vm_start {
print "Resuming suspended VM\n";
}
+ if ($statefile && $statefile ne 'tcp' && $statefile ne 'unix') {
+ # re-use resume code
+ $conf->{vmstate} = $statefile;
+ }
+
my ($cmd, $vollist, $spice_port) = config_to_command($storecfg, $vmid,
$conf, $defaults, $forcemachine);
my $migrate_port = 0;
@@ -5465,8 +5470,6 @@ sub vm_start {
push @$cmd, '-incoming', $migrate_uri;
push @$cmd, '-S';
- } else {
- push @$cmd, '-loadstate', $statefile;
}
} elsif ($paused) {
push @$cmd, '-S';
@@ -5616,11 +5619,16 @@ sub vm_start {
property => "guest-stats-polling-interval",
value => 2) if (!defined($conf->{balloon}) ||
$conf->{balloon});
- if ($is_suspended && (my $vmstate = $conf->{vmstate})) {
- print "Resumed VM, removing state\n";
- delete $conf->@{qw(lock vmstate runningmachine)};
+ if (my $vmstate = $conf->{vmstate}) {
PVE::Storage::deactivate_volumes($storecfg, [$vmstate]);
- PVE::Storage::vdisk_free($storecfg, $vmstate);
+ delete $conf->{vmstate};
+
+ if ($is_suspended) {
+ print "Resumed VM, removing state\n";
+ delete $conf->@{qw(lock runningmachine)};
+ PVE::Storage::vdisk_free($storecfg, $vmstate);
+ }
+
PVE::QemuConfig->write_config($vmid, $conf);
}
>
> Signed-off-by: Alwin Antreich
> ---
> PVE/QemuConfig.pm | 3 +--
> PVE/QemuServer.pm | 10 +-
> 2 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/PVE/QemuConfig.pm b/PVE/QemuConfig.pm
> index edbf1a7..e9796a3 100644
> --- a/PVE/QemuConfig.pm
> +++ b/PVE/QemuConfig.pm
> @@ -359,8 +359,7 @@ sub __snapshot_rollback_vm_start {
> my ($class, $vmid, $vmstate, $data) = @_;
>
> my $storecfg = PVE::Storage::config();
> -my $statefile = PVE::Storage::path($storecfg, $vmstate);
> -PVE::QemuServer::vm_start($storecfg, $vmid, $statefile, undef, undef,
> undef, $data->{forcemachine});
> +PVE::QemuServer::vm_start($storecfg, $vmid, $vmstate, undef, undef,
> undef, $data->{forcemachine});
> }
>
> sub __snapshot_rollback_get_unused {
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 8376260..39315b3 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -5420,6 +5420,7 @@ sub vm_start {
>
> my ($cmd, $vollist, $spice_port) = config_to_command($storecfg, $vmid,
> $conf, $defaults, $forcemachine);
>
> +
> my $migrate_port = 0;
> my $migrate_uri;
> if ($statefile) {
> @@ -5466,7 +5467,12 @@ sub vm_start {
> push @$cmd, '-S';
>
> } else {
> - push @$cmd, '-loadstate', $statefile;
> + my $sfile = $statefile;
> + if (!-e $statefile) {
> + PVE::Storage::activate_volumes($storecfg, [$statefile]);
> + $sfile = PVE::Storage::path($storecfg, $statefile);
> + }
why only activate if the file does not exist? $statefile should be a
full volume ID including storage at this point, it does not make sense
to check for existence?
> + push @$cmd, '-loadstate', $sfile;
> }
> } elsif ($paused) {
> push @$cmd, '-S';
> @@ -5622,6 +5628,8 @@ sub vm_start {
> PVE::Storage::deactivate_volumes($storecfg, [$vmstate]);
> PVE::Storage::vdisk_free($storecfg, $vmstate);
> PVE::QemuConfig->write_config($vmid, $conf);
> + } elsif (
Re: [pve-devel] [PATCH qemu-server] Fix #2171: VM statefile was not activated
On October 8, 2019 11:25 am, Alwin Antreich wrote:
> On Tue, Oct 08, 2019 at 08:36:57AM +0200, Fabian Grünbichler wrote:
>> On October 7, 2019 2:41 pm, Alwin Antreich wrote:
>> > Machine states that were created on snapshots with memory could not be
>> > restored on rollback. The state volume was not activated so KVM couldn't
>> > load the state.
>> >
>> > This patch moves the path generation into vm_start and de-/activates the
>> > state volume.
>>
>> alternatively, the following could also work and re-use more code so
>> that we don't miss the next special handling of some corner case.
>> rolling back from a snapshot with state is just like resuming, but we
>> want to keep the statefile instead of deleting it.
> I will send another version with your alternative.
>
>>
>> (untested):
>>
>> diff --git a/PVE/QemuConfig.pm b/PVE/QemuConfig.pm
>> index edbf1a7..b70c276 100644
>> --- a/PVE/QemuConfig.pm
>> +++ b/PVE/QemuConfig.pm
>> @@ -358,9 +358,7 @@ sub __snapshot_rollback_vm_stop {
>> sub __snapshot_rollback_vm_start {
>> my ($class, $vmid, $vmstate, $data) = @_;
>>
>> -my $storecfg = PVE::Storage::config();
>> -my $statefile = PVE::Storage::path($storecfg, $vmstate);
>> -PVE::QemuServer::vm_start($storecfg, $vmid, $statefile, undef, undef,
>> undef, $data->{forcemachine});
>> +PVE::QemuServer::vm_start($storecfg, $vmid, $vmstate, undef, undef,
>> undef, $data->{forcemachine});
>> }
>>
>> sub __snapshot_rollback_get_unused {
>> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
>> index 8376260..f2d19e1 100644
>> --- a/PVE/QemuServer.pm
>> +++ b/PVE/QemuServer.pm
>> @@ -5418,6 +5418,11 @@ sub vm_start {
>> print "Resuming suspended VM\n";
>> }
>>
>> +if ($statefile && $statefile ne 'tcp' && $statefile ne 'unix') {
>> +# re-use resume code
>> +$conf->{vmstate} = $statefile;
>> +}
>> +
>> my ($cmd, $vollist, $spice_port) = config_to_command($storecfg, $vmid,
>> $conf, $defaults, $forcemachine);
>>
>> my $migrate_port = 0;
>> @@ -5465,8 +5470,6 @@ sub vm_start {
>> push @$cmd, '-incoming', $migrate_uri;
>> push @$cmd, '-S';
>>
>> -} else {
>> -push @$cmd, '-loadstate', $statefile;
>> }
>> } elsif ($paused) {
>> push @$cmd, '-S';
>> @@ -5616,11 +5619,16 @@ sub vm_start {
>> property => "guest-stats-polling-interval",
>> value => 2) if (!defined($conf->{balloon}) ||
>> $conf->{balloon});
>>
>> -if ($is_suspended && (my $vmstate = $conf->{vmstate})) {
>> -print "Resumed VM, removing state\n";
>> -delete $conf->@{qw(lock vmstate runningmachine)};
>> +if (my $vmstate = $conf->{vmstate}) {
>> PVE::Storage::deactivate_volumes($storecfg, [$vmstate]);
>> -PVE::Storage::vdisk_free($storecfg, $vmstate);
>> +delete $conf->{vmstate};
>> +
>> +if ($is_suspended) {
>> +print "Resumed VM, removing state\n";
>> +delete $conf->@{qw(lock runningmachine)};
>> +PVE::Storage::vdisk_free($storecfg, $vmstate);
>> +}
>> +
>> PVE::QemuConfig->write_config($vmid, $conf);
>> }
>>
>>
>> >
>> > Signed-off-by: Alwin Antreich
>> > ---
>> > PVE/QemuConfig.pm | 3 +--
>> > PVE/QemuServer.pm | 10 +-
>> > 2 files changed, 10 insertions(+), 3 deletions(-)
>> >
>> > diff --git a/PVE/QemuConfig.pm b/PVE/QemuConfig.pm
>> > index edbf1a7..e9796a3 100644
>> > --- a/PVE/QemuConfig.pm
>> > +++ b/PVE/QemuConfig.pm
>> > @@ -359,8 +359,7 @@ sub __snapshot_rollback_vm_start {
>> > my ($class, $vmid, $vmstate, $data) = @_;
>> >
>> > my $storecfg = PVE::Storage::config();
>> > -my $statefile = PVE::Storage::path($storecfg, $vmstate);
>> > -PVE::QemuServer::vm_start($storecfg, $vmid, $statefile, undef, undef,
>> > undef, $data->{forcemachine});
>> > +PVE::QemuServer::vm_start($storecfg, $vmid, $vmstate, undef, undef,
>> > undef, $data->{forcemachine});
>> > }
>> >
>> > sub __snapshot_rollback_get_unused {
>> &g
Re: [pve-devel] [PATCH storage] Use bigger timeouts for zfs operations
On October 10, 2019 8:55 am, Fabian Ebner wrote:
> On 10/1/19 12:28 PM, Fabian Grünbichler wrote:
>> On October 1, 2019 12:17 pm, Fabian Ebner wrote:
>>> Seems like 'zfs destroy' can take longer than 5 seconds, see [0].
>>> I changed the timeout to 15 seconds and also changed the default
>>> timeout to 10 instead of 5 seconds, to be on the safe side
>>> for other commands like 'zfs create'.
>>>
>>> [0]:
>>> https://forum.proxmox.com/threads/timeout-beim-l%C3%B6schen-des-entfernen-replikats-bei-entfernung-der-replikation.58467/
>> NAK, we have a 30s timeout for synchronous API requests that call this,
>> and they might do more than 2 zfs_requests. we'd rather timeout and have
>> time to do error handling, than finish successfully sometimes, and die
>> without cleanup other times.
>>
>> the real solution for this is to convert the remaining synchronous API
>> calls that trigger storage operations to async ones, and switch the GUI
>> over to use the new variants. we had previous discussions about this
>> issue already. to implement it really nice, we'd need to things:
>> - tasks with structured return value (to implement async content listing
>>and similar operations, and make conversion of sync to async API calls
>>easier)
>> - light-weight / ephemeral tasks (not included in regular task
>>lists/log, cleaned up after final poll / 1 day after finishing / ...)
>>
>> in case of the mentioned report, please investigate whether this call in
>> 'pvesr' context is wrongly treated as sync API call (i.e., is_worker
>> should maybe return true for pvesr calls?)
> I looked at setting the worker flag for the (whole) pvesr environment,
> but it seems a
> bit much. E.g. in prepare we need to delete stale snapshots before
> starting the replication
> and we probably don't want to wait all too long for that to complete,
> i.e. not
> have the worker flag when calling volume_snapshot_delete.
>
> The situation is different with the vdisk_free call, since there we
> don't need to
> sit and wait for the result, so there it would make sense to be treated
> as a worker.
> So we could set the worker flag locally before that call and use our own
> timeout,
> but it feels like a bad workaround. Also we can't use run_with_timeout
> to set our own
> timeout, since run_command is called inside vdisk_free, which resets the
> alarm.
but we could use run_fork_with_timeout ;)
> Since we have a mechanism to spawn a worker in vdisk_free already
> (currently only
> used by LVM with saferemove), wouldn't it make sense to use that for zfs
> as well? So
> having free_image in the zfs plugin return a subroutine instead of
> executing zfs destroy
> directly. Also other callers of vdisk_free should be fine with such a
> change, since it already
> can happen that vdisk_free spawns a worker (but I haven't looked in detail).
but that is the non-default behaviour, not the default one. I'd rather
not switch all ZFS vdisk removals to forking a task if there are other
solutions.
AFAICT, this vdisk_free happens only for previously, but no-longer
replicated volumes? a simple solution might be to re-factor 139ff of
pvesr.pm to fork a worker if any vdisks need to be removed, and remove
them all asynchronously in a single task?
OTOH, if vdisk_free runs into this problem, it's only a question of load
for other zfs_requests (like listing volumes, snapshots, creating
snapshots, removing snapshots) to also run into the low timeouts.. so
the big question still remains - do we want to increase those timeouts
in general for pvesr, and if we do, how do we do it?
> It would introduce a bit of noise since it would create a task on every
> vdisk_free for a zfs volume.
> There the light-weight tasks you mentioned would be ideal, but for now
> we don't have those.
> And maybe the timeout is hit very rarely and so it's not worth the
> change, what do you think?
>
>>> Signed-off-by: Fabian Ebner
>>> ---
>>> PVE/Storage/ZFSPoolPlugin.pm | 4 ++--
>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/PVE/Storage/ZFSPoolPlugin.pm b/PVE/Storage/ZFSPoolPlugin.pm
>>> index f66b277..3ce06be 100644
>>> --- a/PVE/Storage/ZFSPoolPlugin.pm
>>> +++ b/PVE/Storage/ZFSPoolPlugin.pm
>>> @@ -182,7 +182,7 @@ sub zfs_request {
>>> my $msg = '';
>>> my $output = sub { $msg .= "$_[0]\n" };
>>>
>>> -$timeout = PVE::RPCEnvironment->is_worker() ? 60*60 : 5 if !$timeout;
>>> +$timeout = PVE
Re: [pve-devel] [PATCH qemu-server] Fix #2412: Missing VMs in pools
On October 11, 2019 11:55 am, Dominic Jäger wrote:
> Between destroying a VM (unlink config file) and removing it from user.cfg
> creating a new VM with the ID that is still in use in user.cfg was possible.
> VMs could go missing as a consequence.
>
> Adding a lock solves this. This lock does not interfere with the one in
> vm_destroy as they are held by the same process.
>
> Signed-off-by: Dominic Jäger
> ---
> PVE/API2/Qemu.pm | 8 +---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index 267a08e..7dd83a9 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -1492,9 +1492,11 @@ __PACKAGE__->register_method({
> my $upid = shift;
>
> syslog('info', "destroy VM $vmid: $upid\n");
> - PVE::QemuServer::vm_destroy($storecfg, $vmid, $skiplock);
> - PVE::AccessControl::remove_vm_access($vmid);
> -PVE::Firewall::remove_vmfw_conf($vmid);
> + PVE::QemuConfig->lock_config($vmid, sub {
> + PVE::QemuServer::vm_destroy($storecfg, $vmid, $skiplock);
> + PVE::AccessControl::remove_vm_access($vmid);
> + PVE::Firewall::remove_vmfw_conf($vmid);
> + });
this is not enough - as soon as PVE::QemuServer::destroy_vm unlinks the
VM config file, the lock_config becomes worthless in a clustered setup.
you need to move the VM config unlinking to become the last step, e.g.
by exposing destroy_vm's $keep_empty_config via vm_destroy as well,
and setting it here.
the current situation is especially bad, since we vdisk_free unused
disks AFTER unlinking the config file, so the delay between unlinking
the config file, and removing from user.cfg/removing the firewall config
can be quite long..
> };
>
> return $rpcenv->fork_worker('qmdestroy', $vmid, $authuser, $realcmd);
> --
> 2.20.1
>
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v4 qemu 02/12] Write understood CPU flags into static file
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> located at /usr/share/kvm/cpu-flags-understood-$arch
>
> This file can be read by qemu-server's "query_understood_cpu_flags"
> function, avoiding a more expensive call to QEMU.
>
> For now, only x86_64 is implemented, since aarch64 doesn't print any flags
> when
> called this way.
>
> Signed-off-by: Stefan Reiter
> ---
> debian/rules | 9 +
> 1 file changed, 9 insertions(+)
>
> diff --git a/debian/rules b/debian/rules
> index 8f428c7..7cf73fb 100755
> --- a/debian/rules
> +++ b/debian/rules
> @@ -128,6 +128,15 @@ install: build
> rm -Rf $(destdir)/usr/include
> rm -Rf $(destdir)/usr/lib*
>
> + # write files for understood CPU flags, since these are static for every
> + # version (saves a QEMU call in qemu-server at runtime)
> + $(destdir)/usr/bin/qemu-system-x86_64 -cpu help \
> + | perl -0777 -pe ' \
> + s/^.*Recognized CPUID flags://s; # remove up to flags \
> + s/\n{2,}.*$$//s; # remove any trailing text \
> + s/\s{2,}|\n/\s/g; # remove unnecessary whitespace \
> + ' > $(destdir)/usr/share/kvm/cpu-flags-understood-x86_64
might make sense to add a check that the file is non-empty afterwards,
to catch breakage early?
> +
> # Build architecture-independent files here.
> binary-indep: build install
> # We have nothing to do by default.
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v4 manager 01/12] Broadcast supported CPU flags
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> pvestatd will check if the KVM version has changed using
> kvm_user_version (which automatically clears its cache if QEMU/KVM
> updates), and if it has, query supported CPU flags and broadcast them as
> key-value pairs to the cluster.
>
> If detection fails, we clear the kv-store and set up a delay (120s), to not
> try again too quickly.
>
> Signed-off-by: Stefan Reiter
> ---
>
> v2 -> v3:
> * broadcast tcg and kvm flags if available
> * clear kv-store on error, wait a bit until retry
>
> v1 -> v2:
> * broadcast directly in update_supported_cpuflags
> * use kvm_user_version to determine when to re-query CPU flags
> * don't broadcast flags when unchanged or empty
>
>
> PVE/Service/pvestatd.pm | 46 +++--
> 1 file changed, 44 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> index bad1b73d..f99036c4 100755
> --- a/PVE/Service/pvestatd.pm
> +++ b/PVE/Service/pvestatd.pm
> @@ -78,6 +78,46 @@ sub hup {
> $restart_request = 1;
> }
>
> +my $cached_kvm_version = '';
> +my $next_flag_update_time;
> +my $failed_flag_update_delay_sec = 120;
> +
> +sub update_supported_cpuflags {
> +my $kvm_version = PVE::QemuServer::kvm_user_version();
> +
> +# only update when QEMU/KVM version has changed, as that is the only
> reason
> +# why flags could change without restarting pvestatd
> +return if $cached_kvm_version && $cached_kvm_version eq $kvm_version;
> +
> +if ($next_flag_update_time && $next_flag_update_time > time()) {
> + return;
> +}
> +$next_flag_update_time = 0;
> +
> +my $supported_cpuflags = eval {
> PVE::QemuServer::query_supported_cpu_flags() };
shouldn't this always return at least
{ tcg => undef, kvm => undef }
unless arch is aarch64 (which means it always fails anyway) or locking
the fake VMID fails?
> +warn $@ if $@;
> +
> +if (!$supported_cpuflags) {
which makes this check kind of wrong?
> + # something went wrong, clear broadcast flags and set try-again delay
> + warn "CPU flag detection failed, will try again after delay\n";
> + $next_flag_update_time = time() + $failed_flag_update_delay_sec;
> +
> + $supported_cpuflags = {};
> +} else {
we enter the else branch
> + # only set cached version if there's actually something to braodcast
> + $cached_kvm_version = $kvm_version if $supported_cpuflags;
and set this, even though we don't have anything to broadcast. the
post-fix if is redundant btw, since it's the negation of the outer if
condition.
> +}
> +
> +for my $accel ("tcg", "kvm") {
> + if ($supported_cpuflags->{$accel}) {
> + PVE::Cluster::broadcast_node_kv("cpuflags-$accel", join(' ',
> @{$supported_cpuflags->{$accel}}));
> + } else {
> + # clear potentially invalid data
> + PVE::Cluster::broadcast_node_kv("cpuflags-$accel", '');
at least we end up here and clear both values ;)
> + }
> +}
> +}
> +
> my $generate_rrd_string = sub {
> my ($data) = @_;
>
> @@ -97,7 +137,9 @@ sub update_node_status {
>
> my $cpuinfo = PVE::ProcFSTools::read_cpuinfo();
>
> -my $maxcpu = $cpuinfo->{cpus};
> +my $maxcpu = $cpuinfo->{cpus};
> +
> +update_supported_cpuflags();
>
> my $subinfo = PVE::INotify::read_file('subscription');
> my $sublevel = $subinfo->{level} || '';
> @@ -110,7 +152,7 @@ sub update_node_status {
> $netin += $netdev->{$dev}->{receive};
> $netout += $netdev->{$dev}->{transmit};
> }
> -
> +
> my $meminfo = PVE::ProcFSTools::read_meminfo();
>
> my $dinfo = df('/', 1); # output is bytes
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v4 qemu-server 05/12] Add CPUConfig file and migrate some helpers
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> The package will be used for custom CPU models as a SectionConfig, hence
> the name. For now we simply move some CPU related helper functions and
> declarations over from QemuServer to reduce clutter there.
>
> qemu_machine_feature_enabled is moved to avoid a cyclic module dependency.
>
> Signed-off-by: Stefan Reiter
> ---
>
> v3:
> * mention qemu_machine_feature_enabled in commit msg
>
>
> PVE/QemuServer.pm | 245 +-
> PVE/QemuServer/CPUConfig.pm | 254
> PVE/QemuServer/Makefile | 1 +
> 3 files changed, 259 insertions(+), 241 deletions(-)
> create mode 100644 PVE/QemuServer/CPUConfig.pm
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 4447afc..58e2944 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -33,6 +33,7 @@ use PVE::QemuConfig;
> use PVE::QMPClient;
> use PVE::RPCEnvironment;
> use PVE::GuestHelpers;
> +use PVE::QemuServer::CPUConfig qw(qemu_machine_feature_enabled
> print_cpu_device get_cpu_options);
> use PVE::QemuServer::PCI qw(print_pci_addr print_pcie_addr
> print_pcie_root_port);
> use PVE::QemuServer::Memory;
> use PVE::QemuServer::USB qw(parse_usb_device);
> @@ -116,108 +117,6 @@ mkdir $var_run_tmpdir;
> my $lock_dir = "/var/lock/qemu-server";
> mkdir $lock_dir;
>
> -my $cpu_vendor_list = {
> -# Intel CPUs
> -486 => 'GenuineIntel',
> -pentium => 'GenuineIntel',
> -pentium2 => 'GenuineIntel',
> -pentium3 => 'GenuineIntel',
> -coreduo => 'GenuineIntel',
> -core2duo => 'GenuineIntel',
> -Conroe => 'GenuineIntel',
> -Penryn => 'GenuineIntel',
> -Nehalem => 'GenuineIntel',
> -'Nehalem-IBRS' => 'GenuineIntel',
> -Westmere => 'GenuineIntel',
> -'Westmere-IBRS' => 'GenuineIntel',
> -SandyBridge => 'GenuineIntel',
> -'SandyBridge-IBRS' => 'GenuineIntel',
> -IvyBridge => 'GenuineIntel',
> -'IvyBridge-IBRS' => 'GenuineIntel',
> -Haswell => 'GenuineIntel',
> -'Haswell-IBRS' => 'GenuineIntel',
> -'Haswell-noTSX' => 'GenuineIntel',
> -'Haswell-noTSX-IBRS' => 'GenuineIntel',
> -Broadwell => 'GenuineIntel',
> -'Broadwell-IBRS' => 'GenuineIntel',
> -'Broadwell-noTSX' => 'GenuineIntel',
> -'Broadwell-noTSX-IBRS' => 'GenuineIntel',
> -'Skylake-Client' => 'GenuineIntel',
> -'Skylake-Client-IBRS' => 'GenuineIntel',
> -'Skylake-Server' => 'GenuineIntel',
> -'Skylake-Server-IBRS' => 'GenuineIntel',
> -'Cascadelake-Server' => 'GenuineIntel',
> -KnightsMill => 'GenuineIntel',
> -
> -
> -# AMD CPUs
> -athlon => 'AuthenticAMD',
> -phenom => 'AuthenticAMD',
> -Opteron_G1 => 'AuthenticAMD',
> -Opteron_G2 => 'AuthenticAMD',
> -Opteron_G3 => 'AuthenticAMD',
> -Opteron_G4 => 'AuthenticAMD',
> -Opteron_G5 => 'AuthenticAMD',
> -EPYC => 'AuthenticAMD',
> -'EPYC-IBPB' => 'AuthenticAMD',
> -
> -# generic types, use vendor from host node
> -host => 'default',
> -kvm32 => 'default',
> -kvm64 => 'default',
> -qemu32 => 'default',
> -qemu64 => 'default',
> -max => 'default',
> -};
> -
> -my @supported_cpu_flags = (
> -'pcid',
> -'spec-ctrl',
> -'ibpb',
> -'ssbd',
> -'virt-ssbd',
> -'amd-ssbd',
> -'amd-no-ssb',
> -'pdpe1gb',
> -'md-clear',
> -'hv-tlbflush',
> -'hv-evmcs',
> -'aes'
> -);
> -my $cpu_flag = qr/[+-](@{[join('|', @supported_cpu_flags)]})/;
> -
> -my $cpu_fmt = {
> -cputype => {
> - description => "Emulated CPU type.",
> - type => 'string',
> - enum => [ sort { "\L$a" cmp "\L$b" } keys %$cpu_vendor_list ],
> - default => 'kvm64',
> - default_key => 1,
> -},
> -hidden => {
> - description => "Do not identify as a KVM virtual machine.",
> - type => 'boolean',
> - optional => 1,
> - default => 0
> -},
> -'hv-vendor-id' => {
> - type => 'string',
> - pattern => qr/[a-zA-Z0-9]{1,12}/,
> - format_description => 'vendor-id',
> - description => 'The Hyper-V vendor ID. Some drivers or programs inside
> Windows guests need a specific ID.',
> - optional => 1,
> -},
> -flags => {
> - description => "List of additional CPU flags separated by ';'."
> - . " Use '+FLAG' to enable, '-FLAG' to disable a flag."
> - . " Currently supported flags: @{[join(', ',
> @supported_cpu_flags)]}.",
> - format_description => '+FLAG[;-FLAG...]',
> - type => 'string',
> - pattern => qr/$cpu_flag(;$cpu_flag)*/,
> - optional => 1,
> -},
> -};
> -
> my $watchdog_fmt = {
> model => {
> default_key => 1,
> @@ -606,7 +505,7 @@ EODESCR
> optional => 1,
> description => "Emulated CPU type.",
> type => 'string',
> - format => $cpu_fmt,
> + format => $PVE::QemuServer::CPUConfig::cpu_fmt,
> },
> parent => get_standard_option('pve-snapshot-name',
Re: [pve-devel] [PATCH v4 qemu-server 07/12] Add overrides and convenience functions to CPUConfig
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> Add two overrides to avoid writing redundant information to the config
> file.
>
> get_model_by_name is used to return a cpu config with default values
> filled out.
>
> Signed-off-by: Stefan Reiter
> ---
>
> v3 -> v4:
> * add is_custom_model
>
> v2 -> v3:
> * add validity checks to write_config
>
>
> PVE/QemuServer/CPUConfig.pm | 65 +
> 1 file changed, 65 insertions(+)
>
> diff --git a/PVE/QemuServer/CPUConfig.pm b/PVE/QemuServer/CPUConfig.pm
> index 6ea5811..9876757 100644
> --- a/PVE/QemuServer/CPUConfig.pm
> +++ b/PVE/QemuServer/CPUConfig.pm
> @@ -151,6 +151,71 @@ sub type {
> return 'cpu-model';
> }
>
> +sub parse_section_header {
> +my ($class, $line) = @_;
> +
> +my ($type, $sectionId, $errmsg, $config) =
> + $class->SUPER::parse_section_header($line);
> +
> +return undef if !$type;
> +return ($type, $sectionId, $errmsg, {
> + # to avoid duplicate model name in config file, parse id as cputype
> + # models parsed from file are by definition always custom
> + cputype => "custom-$sectionId",
> +});
> +}
> +
> +sub write_config {
> +my ($class, $filename, $cfg) = @_;
> +
> +for my $model (keys %{$cfg->{ids}}) {
> + my $model_conf = $cfg->{ids}->{$model};
> +
> + die "internal error: tried saving built-in CPU model (or missing
> prefix)\n"
> + if !is_custom_model($model_conf->{cputype});
> +
> + die "internal error: tried saving custom cpumodel with cputype
> (ignoring prefix) not equal to \$cfg->ids entry\n"
> + if "custom-$model" ne $model_conf->{cputype};
> +
> + # saved in section header
> + delete $model_conf->{cputype};
> +}
> +
> +$class->SUPER::write_config($filename, $cfg);
> +}
> +
> +sub is_custom_model {
> +my ($cputype) = @_;
> +return $cputype =~ m/^custom-/;
> +}
> +
> +# Use this to get a single model in the format described by $cpu_fmt.
> +# Fills in defaults from schema if value is not set in config.
> +# Returns undef for unknown $name, allows names with and without custom-
> prefix.
> +sub get_model_by_name {
> +my ($conf, $name) = @_;
see latter patches as well.
I'd rename this to get_custom_model, and inline the call to
load_custom_model_conf since there is only one call-site which does the
load right before just to call this, and we are only ever interested in
a single model anyway..
adding $noerr
> +
> +$name =~ s/^custom-//;
> +
> +my $entry = $conf->{ids}->{$name};
> +return undef if !defined($entry);
and a 'die' here would also be a good idea I think.
> +
> +my $propList = $defaultData->{propertyList};
> +
> +my $model = {};
> +for my $property (keys %$propList) {
> + next if $property eq 'type';
> +
> + if (my $value = $entry->{$property}) {
> + $model->{$property} = $value;
> + } elsif (my $default = $propList->{$property}->{default}) {
> + $model->{$property} = $default;
> + }
does this really make sense here? for 'hidden' it's problematic, for
'host-phys-bits' it has no effect anyway since the default is 0,
reported-model is only accessed once so we could move the default
application there, and cputype is non-optional for all intents and
purposes, so we should never need to apply the default here?
> +}
> +
> +return $model;
> +}
> +
> # Print a QEMU device node for a given VM configuration for hotplugging CPUs
> sub print_cpu_device {
> my ($conf, $id) = @_;
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v4 qemu-server 06/12] Adapt CPUConfig to handle custom models
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> Turn CPUConfig into a SectionConfig with parsing/writing support for
> custom CPU models. IO is handled using cfs.
>
> Namespacing will be provided using "custom-" prefix for custom model
> names (in VM config only, cpu-models.conf will contain unprefixed
> names).
>
> Signed-off-by: Stefan Reiter
> ---
>
> v4: Use "custom-" prefix instead of distinct property for namespacing as
> suggested by Thomas. Left the v3 message below, but this fixes that issue.
>
> v3: I changed the "custom" property to "built-in" and negated its uses, as
> Thomas suggested. I'm not sure I like this too much compared to "custom" (with
> "built-in", you have to explicitly set a value in the config to 0, which looks
> weird - basically "unsetting" something to use a feature), but I'm very much
> fine with either.
>
>
> PVE/QemuServer/CPUConfig.pm | 46 +++--
> 1 file changed, 44 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/QemuServer/CPUConfig.pm b/PVE/QemuServer/CPUConfig.pm
> index c55128f..6ea5811 100644
> --- a/PVE/QemuServer/CPUConfig.pm
> +++ b/PVE/QemuServer/CPUConfig.pm
> @@ -4,6 +4,8 @@ use strict;
> use warnings;
>
> use PVE::JSONSchema;
> +use PVE::Cluster qw(cfs_register_file cfs_read_file);
> +use base qw(PVE::SectionConfig);
>
> use base 'Exporter';
while this seems to work, IMHO it's would be better to merge both 'use
base' into a single statement..
> our @EXPORT_OK = qw(
> @@ -12,6 +14,15 @@ get_cpu_options
> qemu_machine_feature_enabled
> );
>
> +my $default_filename = "cpu-models.conf";
> +cfs_register_file($default_filename,
> + sub { PVE::QemuServer::CPUConfig->parse_config(@_); },
> + sub { PVE::QemuServer::CPUConfig->write_config(@_); });
> +
> +sub load_custom_model_conf {
> +return cfs_read_file($default_filename);
> +}
> +
> my $cpu_vendor_list = {
> # Intel CPUs
> 486 => 'GenuineIntel',
> @@ -83,11 +94,20 @@ my $cpu_flag = qr/[+-](@{[join('|',
> @supported_cpu_flags)]})/;
>
> our $cpu_fmt = {
> cputype => {
> - description => "Emulated CPU type.",
> + description => "Emulated CPU type. Can be default or custom name
> (custom model names must be prefixed with 'custom-').",
> type => 'string',
> - enum => [ sort { "\L$a" cmp "\L$b" } keys %$cpu_vendor_list ],
> + format_description => 'string',
> default => 'kvm64',
> default_key => 1,
> + optional => 1,
> +},
> +'reported-model' => {
> + description => "CPU model and vendor to report to the guest. Must be a
> QEMU/KVM supported model."
> + . " Only valid for custom CPU model definitions, default
> models will always report themselves to the guest OS.",
> + type => 'string',
> + enum => [ sort { lc("$a") cmp lc("$b") } keys %$cpu_vendor_list ],
> + default => 'kvm64',
> + optional => 1,
> },
> hidden => {
> description => "Do not identify as a KVM virtual machine.",
> @@ -113,6 +133,24 @@ our $cpu_fmt = {
> },
> };
>
> +# Section config settings
> +my $defaultData = {
> +# shallow copy, since SectionConfig modifies propertyList internally
> +propertyList => { %$cpu_fmt },
> +};
> +
> +sub private {
> +return $defaultData;
> +}
> +
> +sub options {
> +return { %$cpu_fmt };
> +}
> +
> +sub type {
> +return 'cpu-model';
> +}
> +
> # Print a QEMU device node for a given VM configuration for hotplugging CPUs
> sub print_cpu_device {
> my ($conf, $id) = @_;
> @@ -251,4 +289,8 @@ sub qemu_machine_feature_enabled {
>$current_minor >= $version_minor);
> }
>
> +
> +__PACKAGE__->register();
> +__PACKAGE__->init();
> +
> 1;
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v4 qemu-server 08/12] Verify VM-specific CPU configs seperately
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> $cpu_fmt is being reused for custom CPUs as well as VM-specific CPU
> settings. The "pve-vm-cpu-conf" format is introduced to verify a config
> specifically for use as VM-specific settings.
>
> Signed-off-by: Stefan Reiter
> ---
>
> v3 -> v4:
> * use is_custom_model
>
> v2 -> v3:
> * move $cpu_fmt->{flags} changes here, to avoid having broken checks between
> patches
>
>
> PVE/QemuServer.pm | 2 +-
> PVE/QemuServer/CPUConfig.pm | 70 ++---
> 2 files changed, 67 insertions(+), 5 deletions(-)
>
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 58e2944..04ab1c2 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -505,7 +505,7 @@ EODESCR
> optional => 1,
> description => "Emulated CPU type.",
> type => 'string',
> - format => $PVE::QemuServer::CPUConfig::cpu_fmt,
> + format => 'pve-vm-cpu-conf',
> },
> parent => get_standard_option('pve-snapshot-name', {
> optional => 1,
> diff --git a/PVE/QemuServer/CPUConfig.pm b/PVE/QemuServer/CPUConfig.pm
> index 9876757..fb8e852 100644
> --- a/PVE/QemuServer/CPUConfig.pm
> +++ b/PVE/QemuServer/CPUConfig.pm
> @@ -90,9 +90,9 @@ my @supported_cpu_flags = (
> 'hv-evmcs',
> 'aes'
> );
> -my $cpu_flag = qr/[+-](@{[join('|', @supported_cpu_flags)]})/;
> +my $cpu_flag_re = qr/([+-])(@{[join('|', @supported_cpu_flags)]})/;
maybe it makes sense to pull out both REs?
>
> -our $cpu_fmt = {
> +my $cpu_fmt = {
> cputype => {
> description => "Emulated CPU type. Can be default or custom name
> (custom model names must be prefixed with 'custom-').",
> type => 'string',
> @@ -125,14 +125,76 @@ our $cpu_fmt = {
> flags => {
> description => "List of additional CPU flags separated by ';'."
>. " Use '+FLAG' to enable, '-FLAG' to disable a flag."
> - . " Currently supported flags: @{[join(', ',
> @supported_cpu_flags)]}.",
> + . " Custom CPU models can specify any flag supported by"
> + . " QEMU/KVM, VM-specific flags must be from the following"
> + . " set for security reasons: @{[join(', ',
> @supported_cpu_flags)]}.",
> format_description => '+FLAG[;-FLAG...]',
> type => 'string',
> - pattern => qr/$cpu_flag(;$cpu_flag)*/,
> + pattern => qr/[+-][a-zA-Z0-9\-_\.]+(;[+-][a-zA-Z0-9\-_\.]+)*/,
> optional => 1,
> },
> };
>
> +# $cpu_fmt describes both the CPU config passed as part of a VM config, as
> well
> +# as the definition of a custom CPU model. There are some slight differences
> +# though, which we catch in the custom verification function below.
> +sub parse_cpu_conf_basic {
> +my ($cpu_str, $noerr) = @_;
> +
> +my $cpu = eval { PVE::JSONSchema::parse_property_string($cpu_fmt,
> $cpu_str) };
> +if ($@) {
> +die $@ if !$noerr;
> +return undef;
> +}
> +
> +# required, but can't be made "optional => 0" since it's not included as
> a
> +# seperate property in config file
required, but only encoded in section header?
> +if (!$cpu->{cputype}) {
> + die "CPU is missing cputype\n" if !$noerr;
> + return undef;
> +}
> +
> +return $cpu;
> +}
> +
> +PVE::JSONSchema::register_format('pve-vm-cpu-conf', \&verify_vm_cpu_conf);
> +sub verify_vm_cpu_conf {
> +my ($cpu_str, $noerr) = @_;
> +
> +my $cpu = parse_cpu_conf_basic($cpu_str, $noerr);
> +return undef if !$cpu;
> +
> +my $cputype = $cpu->{cputype};
> +
> +if (is_custom_model($cputype)) {
> + # Custom model only has to exist, all props are allowed
> + my $config = load_custom_model_conf();
> + my $cputype_id = $cputype;
> + $cputype_id =~ s/^custom-//;
> + return $cpu if $config && defined($config->{ids}->{$cputype_id});
this is very similar to get_model_by_name, see my other comments there
and think about re-using it here..
> +
> + die "Custom cputype '$cputype_id' not found\n" if !$noerr;
> + return undef;
> +}
> +
> +# Only built-in models get here
> +if (!defined($cpu_vendor_list->{$cputype})) {
> + die "Default cputype '$cputype' not found\n" if !$noerr;
s/Default/Built-in/
or even
"(Built-in) cputype '$cputype' is not defined (missing 'custom-' prefix?)\n"
> + return undef;
> +}
> +
> +if ($cpu->{flags} && $cpu->{flags} !~ m/$cpu_flag_re(;$cpu_flag_re)*/) {
> + die "VM-specific CPU flags must be a subset of: @{[join(', ',
> @supported_cpu_flags)]}\n"
> + if !$noerr;
> + return undef;
> +}
> +
> +die "Property 'reported-model' not allowed in VM-specific CPU config.\n"
> + if defined($cpu->{'reported-model'});
> +
> +return $cpu;
> +}
> +
> # Section config settings
> my $defaultData = {
> # shallow copy, since SectionConfig modifies propertyList internally
> --
> 2.20.1
>
>
> ___
>
Re: [pve-devel] [PATCH v4 qemu-server 09/12] Add helpers to better structure CPU option handling
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> To avoid hardcoding even more CPU-flag related things for custom CPU
> models, introduce a dynamic approach to resolving flags.
>
> resolve_cpu_flags takes a list of hashes (as documented in the
> comment) and resolves them to a valid "-cpu" argument without
> duplicates. This also helps by providing a reason why specific CPU flags
> have been added, and thus allows for useful warning messages should a
> flag be overwritten by another.
>
> Signed-off-by: Stefan Reiter
> ---
> PVE/QemuServer/CPUConfig.pm | 67 +
> 1 file changed, 67 insertions(+)
>
> diff --git a/PVE/QemuServer/CPUConfig.pm b/PVE/QemuServer/CPUConfig.pm
> index fb8e852..405ad86 100644
> --- a/PVE/QemuServer/CPUConfig.pm
> +++ b/PVE/QemuServer/CPUConfig.pm
> @@ -298,6 +298,73 @@ sub print_cpu_device {
> return
> "$cpu-x86_64-cpu,id=cpu$id,socket-id=$current_socket,core-id=$current_core,thread-id=0";
> }
>
> +# Resolves multiple arrays of hashes representing CPU flags with metadata to
> a
> +# single string in QEMU "-cpu" compatible format. Later arrays have higher
> +# priority.
> +#
> +# Hashes take the following format:
> +# {
> +# aes => {
> +# op => "+", # defaults to "" if undefined
> +# reason => "to support AES acceleration", # for override warnings
> +# value => "" # needed for kvm=off (value: off) etc...
> +# },
> +# ...
> +# }
> +sub resolve_cpu_flags {
> +my $flag_hashes = \@_;
why? it's only used once two lines below, where you immediately
derefence it again.
> +
> +my $flags = {};
> +
> +for my $hash (@$flag_hashes) {
> + for my $flag_name (keys %$hash) {
> + my $flag = $hash->{$flag_name};
> + my $old_flag = $flags->{$flag_name};
> +
> + $flag->{op} //= "";
> +
> + if ($old_flag) {
> + my $value_changed = defined($flag->{value}) !=
> defined($old_flag->{value});
> + if (!$value_changed && defined($flag->{value})) {
> + $value_changed = $flag->{value} eq $old_flag->{value};
> + }
eq should be ne, right? either the definedness changed, or it was and is
defined and the value changed?
AFAICT, this is in fact the following nested conditional:
my $value_changed = (defined($flag->{value}) != defined($old_flag->{value})) ||
(defined($flag->{value}) && $flag->{value} ne
$old_flag->{value});
which is more readable IMHO, but you could also expand it more with
additional variables.
> +
> + if ($old_flag->{op} eq $flag->{op} && !$value_changed) {
> + $flags->{$flag_name}->{reason} .= " & $flag->{reason}";
> + next;
> + }
> +
> + my $old = print_cpuflag_hash($flag_name, $flags->{$flag_name});
> + my $new = print_cpuflag_hash($flag_name, $flag);
> + warn "warning: CPU flag/setting $new overwrites $old\n";
> + }
> +
> + $flags->{$flag_name} = $flag;
> + }
> +}
> +
> +my $flag_str = '';
> +# sort for command line stability
> +for my $flag_name (sort keys %$flags) {
> + $flag_str .= ',';
> + $flag_str .= $flags->{$flag_name}->{op};
> + $flag_str .= $flag_name;
> + $flag_str .= "=$flags->{$flag_name}->{value}"
> + if $flags->{$flag_name}->{value};
> +}
> +
> +return $flag_str;
> +}
> +
> +sub print_cpuflag_hash {
> +my ($flag_name, $flag) = @_;
> +my $formatted = "'$flag->{op}$flag_name";
> +$formatted .= "=$flag->{value}" if defined($flag->{value});
> +$formatted .= "'";
> +$formatted .= " ($flag->{reason})" if defined($flag->{reason});
> +return $formatted;
> +}
> +
> # Calculate QEMU's '-cpu' argument from a given VM configuration
> sub get_cpu_options {
> my ($conf, $arch, $kvm, $machine_type, $kvm_off, $kvmver, $winversion,
> $gpu_passthrough) = @_;
> --
> 2.20.1
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v4 qemu-server 10/12] Rework get_cpu_options and allow custom CPU models
On October 7, 2019 2:47 pm, Stefan Reiter wrote:
> If a cputype is custom (check via prefix), try to load options from the
> custom CPU model config, and set values accordingly.
>
> While at it, extract currently hardcoded values into seperate sub and add
> reasonings.
>
> Since the new flag resolving outputs flags in sorted order for
> consistency, adapt the test cases to not break. Only the order is
> changed, not which flags are present.
>
> Signed-off-by: Stefan Reiter
> ---
>
> v3 -> v4:
> * use is_custom_model
>
> v3: Since it's just a few lines, I included the test changes into this patch
> directly. This way all patches in the series should be buildable without
> problems.
>
> v2: It was quite interesting to dig through old commit messages/mail archives
> to
> find the actual reasons some of the hardcoded flags are included. I do feel
> like
> the "reason" field is quite useful though, both for future developers and
> users.
>
> PVE/QemuServer/CPUConfig.pm| 185 +++--
> test/cfg2cmd/i440fx-win10-hostpci.conf.cmd | 2 +-
> test/cfg2cmd/minimal-defaults.conf.cmd | 2 +-
> test/cfg2cmd/q35-linux-hostpci.conf.cmd| 2 +-
> test/cfg2cmd/q35-win10-hostpci.conf.cmd| 2 +-
> test/cfg2cmd/simple1.conf.cmd | 2 +-
> test/cfg2cmd/spice-usb3.conf.cmd | 2 +-
> 7 files changed, 144 insertions(+), 53 deletions(-)
>
> diff --git a/PVE/QemuServer/CPUConfig.pm b/PVE/QemuServer/CPUConfig.pm
> index 405ad86..0948f67 100644
> --- a/PVE/QemuServer/CPUConfig.pm
> +++ b/PVE/QemuServer/CPUConfig.pm
> @@ -369,96 +369,187 @@ sub print_cpuflag_hash {
> sub get_cpu_options {
> my ($conf, $arch, $kvm, $machine_type, $kvm_off, $kvmver, $winversion,
> $gpu_passthrough) = @_;
>
> -my $cpuFlags = [];
> -my $ostype = $conf->{ostype};
> -
> -my $cpu = $kvm ? "kvm64" : "qemu64";
> +my $cputype = $kvm ? "kvm64" : "qemu64";
> if ($arch eq 'aarch64') {
> - $cpu = 'cortex-a57';
> + $cputype = 'cortex-a57';
> }
> +
> +my $cpuconf = {};
> +my $custom_cpuconf;
bad naming:
$cpu (further below)
$cpuconf
$custom_cpuconf
$custom_conf (a bit below)
maybe
$cpu -> $cpu_str (string that is generated to be passed via -cpu)
$cpuconf -> $cpu
$custom_cpuconf -> $custom_cpu
$custom_conf -> inline, maybe even into get_model_by_name ?
> my $hv_vendor_id;
> -if (my $cputype = $conf->{cpu}) {
> - my $cpuconf = PVE::JSONSchema::parse_property_string($cpu_fmt, $cputype)
> - or die "Cannot parse cpu description: $cputype\n";
> - $cpu = $cpuconf->{cputype};
> - $kvm_off = 1 if $cpuconf->{hidden};
> - $hv_vendor_id = $cpuconf->{'hv-vendor-id'};
> +if (my $cpu_prop_str = $conf->{cpu}) {
> + $cpuconf = verify_vm_cpu_conf($cpu_prop_str)
> + or die "Cannot parse cpu description: $cpu_prop_str\n";
s/verify_vm_cpu_conf/parse_cpu_conf_basic ?
>
> - if (defined(my $flags = $cpuconf->{flags})) {
> - push @$cpuFlags, split(";", $flags);
> - }
> -}
> + $cputype = $cpuconf->{cputype};
>
> -push @$cpuFlags , '+lahf_lm' if $cpu eq 'kvm64' && $arch eq 'x86_64';
> + if (is_custom_model($cputype)) {
> + my $custom_conf = load_custom_model_conf();
> + $custom_cpuconf = get_model_by_name($custom_conf, $cputype)
> + or die "invalid custom model definition for '$cputype'\n";
>
> -push @$cpuFlags , '-x2apic'
> - if $conf->{ostype} && $conf->{ostype} eq 'solaris';
> + $cputype = $custom_cpuconf->{'reported-model'};
> + $kvm_off = $custom_cpuconf->{hidden};
this is a bit tricky - the schema default is 0, get_model_by_name
applies this default, and now we override the passed-in explicit value
with the default value from the custom CPU schema.
maybe drop the default and add an 'if defined' here?
> + $hv_vendor_id = $custom_cpuconf->{'hv-vendor-id'};
> + }
>
> -push @$cpuFlags, '+sep' if $cpu eq 'kvm64' || $cpu eq 'kvm32';
> + # VM-specific settings override custom CPU config
> + $kvm_off = $cpuconf->{hidden}
> + if defined($cpuconf->{hidden});
> + $hv_vendor_id = $cpuconf->{'hv-vendor-id'}
> + if defined($cpuconf->{'hv-vendor-id'});
> +}
>
> -push @$cpuFlags, '-rdtscp' if $cpu =~ m/^Opteron/;
> +my $pve_flags = get_pve_cpu_flags($conf, $kvm, $cputype, $arch,
> + $machine_type, $kvmver);
>
> -if (qemu_machine_feature_enabled ($machine_type, $kvmver, 2, 3) && $arch
> eq 'x86_64') {
> +my $hv_flags = get_hyperv_enlightenments($winversion, $machine_type,
> $kvmver,
> + $conf->{bios}, $gpu_passthrough, $hv_vendor_id) if $kvm;
>
> - push @$cpuFlags , '+kvm_pv_unhalt' if $kvm;
> - push @$cpuFlags , '+kvm_pv_eoi' if $kvm;
> +my $custom_cputype_flags = {};
> +if ($custom_cpuconf && defined($custom_cpuconf->{flags})) {
> + foreach my $flag (split(";", $custom_cpuconf->{flags})) {
Re: [pve-devel] [PATCH kernel-meta] fix #2403: exclude initrd entries from /proc/cmdline
On October 15, 2019 3:54 pm, Thomas Lamprecht wrote:
> On 10/15/19 3:22 PM, Oguz Bektas wrote:
>> if we fallback to /proc/cmdline, it can include the booted initrd.
>>
>> to avoid loader entries with (sometimes even multiple) initrd lines,
>> we have to parse them out.
>>
>> Signed-off-by: Oguz Bektas
>> ---
>> efiboot/zz-pve-efiboot | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/efiboot/zz-pve-efiboot b/efiboot/zz-pve-efiboot
>> index 4756555..5a84af2 100755
>> --- a/efiboot/zz-pve-efiboot
>> +++ b/efiboot/zz-pve-efiboot
>> @@ -50,7 +50,8 @@ update_esps() {
>> CMDLINE="$(cat /etc/kernel/cmdline)"
>> else
>> warn "No /etc/kernel/cmdline found - falling back to
>> /proc/cmdline"
>> -CMDLINE="$(cat /proc/cmdline)"
>> +# remove initrd entries
>> +CMDLINE="$(cat /proc/cmdline | awk
>> '{gsub(/initrd=([0-9a-zA-Z\\.-])*\s/,x)}1')"
>
> could pontentially match a field which ends with initrd, e.g.,
> noinitrd=foo
>
> while just an example that still should be caught..
>
> also, this is a wrong use of cat, you could just do:
> "$(awk '...' /proc/cmdline)"
>
> not point for the pipe..
>
> now if the initrd= is _always_ the first one we could also do:
> "$(cut -d ' ' -f2- /proc/cmdline)"
there's potentially multiple initrd stanzas, and they can appear
wherever in the cmdline..
>
> but no hard feelings.
>
>> fi
>>
>> loop_esp_list update_esp_func
>>
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] applied: [PATCH v3 kernel-meta] fix #2403: exclude initrd entries from /proc/cmdline
On October 17, 2019 7:45 am, Thomas Lamprecht wrote:
> On 10/16/19 1:17 PM, Oguz Bektas wrote:
>> if we fallback to /proc/cmdline, it can include the booted initrd.
>>
>> to avoid loader entries with initrd 'options' lines, we have to parse
>> them out.
>>
>> Signed-off-by: Oguz Bektas
>> ---
>>
>> v2->v3:
>> * match forward slashes
>> * match underscore
>> * match zero or more whitespace at the end
>>
>> efiboot/zz-pve-efiboot | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/efiboot/zz-pve-efiboot b/efiboot/zz-pve-efiboot
>> index 4756555..8771da9 100755
>> --- a/efiboot/zz-pve-efiboot
>> +++ b/efiboot/zz-pve-efiboot
>> @@ -50,7 +50,8 @@ update_esps() {
>> CMDLINE="$(cat /etc/kernel/cmdline)"
>> else
>> warn "No /etc/kernel/cmdline found - falling back to
>> /proc/cmdline"
>> -CMDLINE="$(cat /proc/cmdline)"
>> +# remove initrd entries
>> +CMDLINE="$(awk '{gsub(/\yinitrd=([0-9a-zA-Z\/\\._-])*\s*/,x)}1'
>> /proc/cmdline)"
>
> applied, but fixed up a few things, mostly style-wise so IMO no need for a v4:
> * add some spaces for separation, increasing readability
> * do not use the non-existent variable x as replacement, but an actual
> empty string ""
> * don't use the "truth-y action" at end to make awk print the line ($0)
> but explicitly print $0 after the gsub, makes it easier to get for
> people with not much awk background ;)
>
> thanks!
a bit late to the party unfortunately, but AFAICT it's also valid to
have quoted parameter values with spaces inside - maybe it's worth a
follow-up? not relevant for our generated ones, but maybe for manually
built/added kernels..
also not sure whether plain ascii alpha-numeric is the proper choice for
characters. we can also leave it as is until somebody reports actually
running into either issue though ;)
>
>> fi
>>
>> loop_esp_list update_esp_func
>>
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 02/23] add missing 'use PVE::Auth::Plugin'
Signed-off-by: Fabian Grünbichler --- PVE/API2/AccessControl.pm | 1 + 1 file changed, 1 insertion(+) diff --git a/PVE/API2/AccessControl.pm b/PVE/API2/AccessControl.pm index dfbdfc6..9d2da8d 100644 --- a/PVE/API2/AccessControl.pm +++ b/PVE/API2/AccessControl.pm @@ -19,6 +19,7 @@ use PVE::API2::User; use PVE::API2::Group; use PVE::API2::Role; use PVE::API2::ACL; +use PVE::Auth::Plugin; use PVE::OTP; use PVE::Tools; -- 2.20.1 ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 1/23] API schema: add 'notoken' property
to mark API methods which should not be available to clients authenticated
using an API token
Signed-off-by: Fabian Grünbichler
---
Notes:
if applied, any users of this need corresponding versioned depends.
src/PVE/JSONSchema.pm | 5 +
1 file changed, 5 insertions(+)
diff --git a/src/PVE/JSONSchema.pm b/src/PVE/JSONSchema.pm
index db38d44..a42d814 100644
--- a/src/PVE/JSONSchema.pm
+++ b/src/PVE/JSONSchema.pm
@@ -1263,6 +1263,11 @@ my $method_schema = {
description => "Method needs special privileges - only pvedaemon
can execute it",
optional => 1,
},
+ notoken => {
+ type => 'boolean',
+ description => "Method is not available for clients authenticated
using an API token.",
+ optional => 1,
+ },
download => {
type => 'boolean',
description => "Method downloads the file content (filename is the
return value of the method).",
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 04/23] user.cfg: sort ACL members
Signed-off-by: Fabian Grünbichler
---
PVE/AccessControl.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index 3e52c5f..aff9137 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -1135,11 +1135,11 @@ sub write_user_config {
}
foreach my $rolelist (sort keys %{$ra->{0}}) {
- my $uglist = join (',', keys %{$ra->{0}->{$rolelist}});
+ my $uglist = join (',', sort keys %{$ra->{0}->{$rolelist}});
$data .= "acl:0:$path:$uglist:$rolelist:\n";
}
foreach my $rolelist (sort keys %{$ra->{1}}) {
- my $uglist = join (',', keys %{$ra->{1}->{$rolelist}});
+ my $uglist = join (',', sort keys %{$ra->{1}->{$rolelist}});
$data .= "acl:1:$path:$uglist:$rolelist:\n";
}
}
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 03/23] user.cfg: sort entries alphabetically in each section
it's not required for dependencies (since those are only ever between
sections, and not within), but makes for easier diffing.
Signed-off-by: Fabian Grünbichler
---
PVE/AccessControl.pm | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index 6ea0b85..3e52c5f 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -1049,7 +1049,7 @@ sub write_user_config {
my $data = '';
-foreach my $user (keys %{$cfg->{users}}) {
+foreach my $user (sort keys %{$cfg->{users}}) {
my $d = $cfg->{users}->{$user};
my $firstname = $d->{firstname} ?
PVE::Tools::encode_text($d->{firstname}) : '';
my $lastname = $d->{lastname} ? PVE::Tools::encode_text($d->{lastname})
: '';
@@ -1063,7 +1063,7 @@ sub write_user_config {
$data .= "\n";
-foreach my $group (keys %{$cfg->{groups}}) {
+foreach my $group (sort keys %{$cfg->{groups}}) {
my $d = $cfg->{groups}->{$group};
my $list = join (',', keys %{$d->{users}});
my $comment = $d->{comment} ? PVE::Tools::encode_text($d->{comment}) :
'';
@@ -1072,7 +1072,7 @@ sub write_user_config {
$data .= "\n";
-foreach my $pool (keys %{$cfg->{pools}}) {
+foreach my $pool (sort keys %{$cfg->{pools}}) {
my $d = $cfg->{pools}->{$pool};
my $vmlist = join (',', keys %{$d->{vms}});
my $storelist = join (',', keys %{$d->{storage}});
@@ -1082,7 +1082,7 @@ sub write_user_config {
$data .= "\n";
-foreach my $role (keys %{$cfg->{roles}}) {
+foreach my $role (sort keys %{$cfg->{roles}}) {
next if $special_roles->{$role};
my $d = $cfg->{roles}->{$role};
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 07/23] auth: pull username REs into variables
for reusage in API token ID format/verification
Signed-off-by: Fabian Grünbichler
---
PVE/Auth/Plugin.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/PVE/Auth/Plugin.pm b/PVE/Auth/Plugin.pm
index 5c11991..6d59b72 100755
--- a/PVE/Auth/Plugin.pm
+++ b/PVE/Auth/Plugin.pm
@@ -27,7 +27,8 @@ sub lock_domain_config {
}
}
-my $realm_regex = qr/[A-Za-z][A-Za-z0-9\.\-_]+/;
+our $realm_regex = qr/[A-Za-z][A-Za-z0-9\.\-_]+/;
+our $user_regex = qr![^\s:/]+!;
PVE::JSONSchema::register_format('pve-realm', \&pve_verify_realm);
sub pve_verify_realm {
@@ -66,7 +67,7 @@ sub verify_username {
# colon separated lists)!
# slash is not allowed because it is used as pve API delimiter
# also see "man useradd"
-if ($username =~ m!^([^\s:/]+)\@(${realm_regex})$!) {
+if ($username =~ m!^(${user_regex})\@(${realm_regex})$!) {
return wantarray ? ($username, $1, $2) : $username;
}
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 09/23] API token: add REs, helpers, parsing + writing
token definitions/references in user.cfg always use the full form of the
token id, consisting of:
USER@REALM!TOKENID
token definitions are represented by their own lines prefixed with
'token', which need to come after the corresponding user definition, but
before any ACLs referencing them.
parsed representation in a user config hash is inside a new 'tokens'
element of the corresponding user object, using the unique-per-user
token id as key.
only token metadata is stored inside user.cfg / accessible via the
parsed user config hash. the actual token values will be stored
root-readable only in a separate (shadow) file.
'enable', 'comment' and 'expire' have the same semantics as for users.
'privsep' determines whether an API token gets the full privileges of
the corresponding user, or just the intersection of privileges of the
corresponding user and those of the API token itself.
Signed-off-by: Fabian Grünbichler
---
Notes:
I am a bit unsure how to differentiate in a clean way between:
A full userid/tokenid (username@realm OR username@real!token)
B user (username@realm)
C tokenid (username@realm!token)
D token/tokensubid/tokenid-per-user (just the part after !)
I am not sure whether it makes much sense to replace all the existing naming
where B becomes A with the introduction of tokens. it might make sense to
have
some specific variable naming for those few places where we explicitly
handle
the difference (A goes in, we check if it's B or C and do different stuff in
either case), as well as for cleanly separating between C and D. applies to
patches after this as well..
recommendations/input welcome ;)
PVE/AccessControl.pm | 88
1 file changed, 88 insertions(+)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index 48c9930..a43aab2 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -211,6 +211,45 @@ sub rotate_authkey {
die $@ if $@;
}
+our $token_subid_regex = $PVE::Auth::Plugin::realm_regex;
+
+# username@realm username realm tokenid
+our $token_full_regex =
qr/((${PVE::Auth::Plugin::user_regex})\@(${PVE::Auth::Plugin::realm_regex}))!(${token_subid_regex})/;
+
+sub split_tokenid {
+my ($tokenid, $noerr) = @_;
+
+if ($tokenid =~ /^${token_full_regex}$/) {
+ return ($1, $4);
+}
+
+die "'$tokenid' is not a valid token ID - not able to split into user and
token parts\n" if !$noerr;
+
+return undef;
+}
+
+sub join_tokenid {
+my ($username, $tokensubid) = @_;
+
+my $joined = "${username}!${tokensubid}";
+
+return pve_verify_tokenid($joined);
+}
+
+PVE::JSONSchema::register_format('pve-tokenid', \&pve_verify_tokenid);
+sub pve_verify_tokenid {
+my ($tokenid, $noerr) = @_;
+
+if ($tokenid =~ /^${token_full_regex}$/) {
+ return wantarray ? ($tokenid, $2, $3, $4) : $tokenid;
+}
+
+die "value '$tokenid' does not look like a valid token ID\n" if !$noerr;
+
+return undef;
+}
+
+
my $csrf_prevention_secret;
my $csrf_prevention_secret_legacy;
my $get_csrfr_secret = sub {
@@ -988,6 +1027,12 @@ sub parse_user_config {
} else {
warn "user config - ignore invalid acl member
'$ug'\n";
}
+ } elsif (my ($user, $token) = split_tokenid($ug, 1)) {
+ if ($cfg->{users}->{$user}->{tokens}->{$token}) { #
token exists
+ $cfg->{acl}->{$path}->{tokens}->{$ug}->{$role}
= $propagate;
+ } else {
+ warn "user config - ignore invalid acl token
'$ug'\n";
+ }
} else {
warn "user config - invalid user/group '$ug' in
acl\n";
}
@@ -1034,6 +1079,36 @@ sub parse_user_config {
}
$cfg->{pools}->{$pool}->{storage}->{$storeid} = 1;
}
+ } elsif ($et eq 'token') {
+ my ($tokenid, $enable, $expire, $privsep, $comment) = @data;
+
+ my ($user, $token) = split_tokenid($tokenid, 1);
+ if (!($user && $token)) {
+ warn "user config - ignore invalid tokenid '$tokenid'\n";
+ next;
+ }
+
+ $enable = $enable ? 1 : 0;
+ $privsep = $privsep ? 1 : 0;
+
+ $expire = 0 if !$expire;
+
+ if ($expire !~ m/^\d+$/) {
+ warn "user config - ignore token '$tokenid' - (illegal
characters in expire '$expire')\n";
+ next;
+ }
+ $expire = int($expire);
+
[pve-devel] [PATCH 16/23] proxy_request: drop duplicate, unused parameter
which was mistakenly added back when this was still in pve-manager.
Signed-off-by: Fabian Grünbichler
---
PVE/APIServer/AnyEvent.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/PVE/APIServer/AnyEvent.pm b/PVE/APIServer/AnyEvent.pm
index c1dade8..9aba27d 100644
--- a/PVE/APIServer/AnyEvent.pm
+++ b/PVE/APIServer/AnyEvent.pm
@@ -747,7 +747,7 @@ sub handle_api2_request {
$res->{proxy_params}->{tmpfilename} = $reqstate->{tmpfilename} if
$upload_state;
$self->proxy_request($reqstate, $clientip, $host,
$res->{proxynode}, $method,
-$r->uri, $auth->{ticket}, $auth->{token},
$res->{proxy_params}, $res->{proxynode});
+$r->uri, $auth->{ticket}, $auth->{token},
$res->{proxy_params});
return;
} elsif ($upgrade && ($method eq 'GET') && ($path =~ m|websocket$|)) {
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 10/23] API token: add API helpers
the _exist/_enabled are modelled after the corresponding user methods.
the 'tokenid' option goes into PVE::AccessControl, since we need it in
multiple API modules.
Signed-off-by: Fabian Grünbichler
---
PVE/AccessControl.pm | 32
1 file changed, 32 insertions(+)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index a43aab2..432ccc0 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -211,6 +211,12 @@ sub rotate_authkey {
die $@ if $@;
}
+PVE::JSONSchema::register_standard_option('tokenid', {
+description => "API token identifier.",
+type => "string",
+format => "pve-tokenid",
+});
+
our $token_subid_regex = $PVE::Auth::Plugin::realm_regex;
# username@realm username realm tokenid
@@ -531,6 +537,32 @@ sub check_user_enabled {
return undef;
}
+sub check_token_exist {
+my ($usercfg, $username, $tokenid, $noerr) = @_;
+
+my $user = check_user_exist($usercfg, $username, $noerr);
+return undef if !$user;
+
+return $user->{tokens}->{$tokenid} if $user->{tokens}->{$tokenid};
+
+die "no such token ('$tokenid') for user ('$username')\n" if !$noerr;
+
+return undef;
+}
+
+sub check_token_enabled {
+my ($usercfg, $username, $tokenid, $noerr) = @_;
+
+my $data = check_token_exist($usercfg, $username, $tokenid, $noerr);
+return undef if !$data;
+
+return 1 if $data->{enable};
+
+die "token '$tokenid' for user '$username' is disabled\n" if !$noerr;
+
+return undef;
+}
+
sub verify_one_time_pw {
my ($type, $username, $keys, $tfa_cfg, $otp) = @_;
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 08/23] refactor acl transformation code
pull it into helper sub, since we need this one more time for token ACL
members.
Signed-off-by: Fabian Grünbichler
---
PVE/AccessControl.pm | 61 +++-
1 file changed, 26 insertions(+), 35 deletions(-)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index a84173e..48c9930 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -1092,47 +1092,38 @@ sub write_user_config {
$data .= "\n";
+my $collect_ra = sub {
+ my ($acl_members, $ra, $prefix, $exclude) = @_;
+
+ foreach my $member (keys %$acl_members) {
+ next if $exclude && $member eq $exclude;
+
+ my $l0 = '';
+ my $l1 = '';
+ foreach my $role (sort keys %{$acl_members->{$member}}) {
+ my $propagate = $acl_members->{$member}->{$role};
+ if ($propagate) {
+ $l1 .= ',' if $l1;
+ $l1 .= $role;
+ } else {
+ $l0 .= ',' if $l0;
+ $l0 .= $role;
+ }
+ }
+ $ra->{0}->{$l0}->{"${prefix}${member}"} = 1 if $l0;
+ $ra->{1}->{$l1}->{"${prefix}${member}"} = 1 if $l1;
+ }
+};
+
foreach my $path (sort keys %{$cfg->{acl}}) {
my $d = $cfg->{acl}->{$path};
my $ra = {};
- foreach my $group (keys %{$d->{groups}}) {
- my $l0 = '';
- my $l1 = '';
- foreach my $role (sort keys %{$d->{groups}->{$group}}) {
- my $propagate = $d->{groups}->{$group}->{$role};
- if ($propagate) {
- $l1 .= ',' if $l1;
- $l1 .= $role;
- } else {
- $l0 .= ',' if $l0;
- $l0 .= $role;
- }
- }
- $ra->{0}->{$l0}->{"\@$group"} = 1 if $l0;
- $ra->{1}->{$l1}->{"\@$group"} = 1 if $l1;
- }
+ $collect_ra->($d->{'groups'}, $ra, '@');
- foreach my $user (keys %{$d->{users}}) {
- # no need to save, because root is always 'Administrator'
- next if $user eq 'root@pam';
-
- my $l0 = '';
- my $l1 = '';
- foreach my $role (sort keys %{$d->{users}->{$user}}) {
- my $propagate = $d->{users}->{$user}->{$role};
- if ($propagate) {
- $l1 .= ',' if $l1;
- $l1 .= $role;
- } else {
- $l0 .= ',' if $l0;
- $l0 .= $role;
- }
- }
- $ra->{0}->{$l0}->{$user} = 1 if $l0;
- $ra->{1}->{$l1}->{$user} = 1 if $l1;
- }
+ # no need to save 'root@pam', it is always 'Administrator'
+ $collect_ra->($d->{'users'}, $ra, '', 'root@pam');
foreach my $rolelist (sort keys %{$ra->{0}}) {
my $uglist = join (',', sort keys %{$ra->{0}->{$rolelist}});
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 12/23] API: add API token API endpoints
and integration for user API endpoints.
Signed-off-by: Fabian Grünbichler
---
Notes:
pveum integration will come in a future version, but
pveum token add/modify/delete [OPTIONS]
or
pveum user token add/modify/delete [OPTIONS]
seem like likely candidates.
PVE/API2/User.pm | 287 ++-
1 file changed, 284 insertions(+), 3 deletions(-)
diff --git a/PVE/API2/User.pm b/PVE/API2/User.pm
index fb5b22a..1646418 100644
--- a/PVE/API2/User.pm
+++ b/PVE/API2/User.pm
@@ -2,9 +2,9 @@ package PVE::API2::User;
use strict;
use warnings;
-use PVE::Exception qw(raise raise_perm_exc);
+use PVE::Exception qw(raise raise_perm_exc raise_param_exc);
use PVE::Cluster qw (cfs_read_file cfs_write_file);
-use PVE::Tools qw(split_list);
+use PVE::Tools qw(split_list extract_param);
use PVE::AccessControl;
use PVE::JSONSchema qw(get_standard_option register_standard_option);
@@ -40,6 +40,39 @@ register_standard_option('group-list', {
optional => 1,
completion => \&PVE::AccessControl::complete_group,
});
+register_standard_option('token-subid', {
+type => 'string',
+pattern => $PVE::AccessControl::token_subid_regex,
+description => 'User-specific token identifier.',
+});
+register_standard_option('token-enable', {
+description => "Enable the API token (default). You can set this to '0' to
disable this specific token.",
+type => 'boolean',
+optional => 1,
+default => 1,
+});
+register_standard_option('token-expire', {
+description => "API token expiration date (seconds since epoch). '0' means
no expiration date.",
+type => 'integer',
+minimum => 0,
+optional => 1,
+});
+register_standard_option('token-privsep', {
+description => "Restrict API token privileges with separate ACLs
(default), or give full privileges of corresponding user.",
+type => 'boolean',
+optional => 1,
+default => 1,
+});
+register_standard_option('token-comment', { type => 'string', optional => 1 });
+register_standard_option('token-info', {
+type => 'object',
+properties => {
+ enable => get_standard_option('token-enable'),
+ expire => get_standard_option('token-expire'),
+ privsep => get_standard_option('token-privsep'),
+ comment => get_standard_option('token-comment'),
+}
+});
my $extract_user_data = sub {
my ($data, $full) = @_;
@@ -53,6 +86,7 @@ my $extract_user_data = sub {
return $res if !$full;
$res->{groups} = $data->{groups} ? [ keys %{$data->{groups}} ] : [];
+$res->{tokens} = $data->{tokens} ? [ keys %{$data->{tokens}} ] : [];
return $res;
};
@@ -228,7 +262,17 @@ __PACKAGE__->register_method ({
email => get_standard_option('user-email'),
comment => get_standard_option('user-comment'),
keys => get_standard_option('user-keys'),
- groups => { type => 'array' },
+ groups => {
+ type => 'array',
+ items => {
+ type => 'string',
+ format => 'pve-groupid',
+ },
+ },
+ tokens => {
+ type => 'array',
+ items => get_standard_option('token-subid'),
+ },
},
type => "object"
},
@@ -428,4 +472,241 @@ __PACKAGE__->register_method ({
return $res;
}});
+my $token_info_extend = sub {
+my ($props) = @_;
+
+my $obj = get_standard_option('token-info');
+my $base_props = $obj->{properties};
+$obj->{properties} = {};
+
+foreach my $prop (keys %$base_props) {
+ $obj->{properties}->{$prop} = $base_props->{$prop};
+}
+
+foreach my $add_prop (keys %$props) {
+ $obj->{properties}->{$add_prop} = $props->{$add_prop};
+}
+
+return $obj;
+};
+
+__PACKAGE__->register_method ({
+name => 'token_index',
+path => '{userid}/token',
+method => 'GET',
+description => "Get user API tokens.",
+permissions => {
+ check => ['userid-param', 'self'],
+},
+parameters => {
+ additionalProperties => 0,
+ properties => {
+ userid => get_standard_option('userid-completed'),
+ },
+},
+returns => {
+ type => "array",
+ items => $token_info_extend->({
+ tokenid =&
[pve-devel] [RFC 06/23] rpcenv: drop unused roles()
it doesn't really serve a purpose, and it's not called anywhere in the codebase.
Signed-off-by: Fabian Grünbichler
---
Notes:
alternatively, we can give this the same semantics w.r.t. tokens as
PVE::AccessControl::roles, but with pool roles mixed in via
$compile_acl_path->()
PVE/AccessControl.pm | 2 +-
PVE/RPCEnvironment.pm | 23 ---
2 files changed, 1 insertion(+), 24 deletions(-)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index f246c85..a84173e 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -1209,7 +1209,7 @@ sub roles {
my ($cfg, $user, $path) = @_;
# NOTE: we do not consider pools here.
-# You need to use $rpcenv->roles() instead if you want that.
+# Use $rpcenv->permission() for any actual permission checks!
return 'Administrator' if $user eq 'root@pam'; # root can do anything
diff --git a/PVE/RPCEnvironment.pm b/PVE/RPCEnvironment.pm
index 95d3029..7e0af70 100644
--- a/PVE/RPCEnvironment.pm
+++ b/PVE/RPCEnvironment.pm
@@ -81,29 +81,6 @@ my $compile_acl_path = sub {
return $privs;
};
-sub roles {
- my ($self, $user, $path) = @_;
-
- if ($user eq 'root@pam') { # root can do anything
- return ('Administrator');
- }
-
- $user = PVE::AccessControl::verify_username($user, 1);
- return () if !$user;
-
- my $cache = $self->{aclcache};
- $cache->{$user} = {} if !$cache->{$user};
-
- my $acl = $cache->{$user};
-
- my $roles = $acl->{roles}->{$path};
- return @$roles if $roles;
-
- &$compile_acl_path($self, $user, $path);
- $roles = $acl->{roles}->{$path} || [];
- return @$roles;
-}
-
sub permissions {
my ($self, $user, $path) = @_;
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 11/23] DO NOT APPLY: API token stubs for token value handling
two proposed implementation sites so far:
- pmxcfs (accessible via IPCC)
- stand-alone root daemon/setuid binary
two proposed token formats so far:
- plain UUID
- hash/crypt of UUID(+salt)
in both cases the UUID would be provided as token to the user, the
latter format would provide a bit of protection against brute-forcing in
case the shadow file leaks, but makes verification more expensive.
I'd like to focus on the big picture/API structure/API integration for
this initial RFC v1 (hence just the stubs), but feedback on these
variants is of course already welcome as well ;)
Signed-off-by: Fabian Grünbichler
---
PVE/AccessControl.pm | 47
1 file changed, 47 insertions(+)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index 432ccc0..b5dfe4b 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -397,6 +397,39 @@ sub verify_ticket {
return wantarray ? ($username, $age, $tfa_info) : $username;
}
+# API Tokens
+sub verify_token {
+my ($api_token) = @_;
+
+die "no API token specified\n" if !$api_token;
+
+my ($tokenid, $value);
+if ($api_token =~ /^($token_full_regex)=(.*)$/) {
+ $tokenid = $1;
+ $value = $2;
+} else {
+ die "no tokenid specified\n";
+}
+
+my ($username, $token) = split_tokenid($tokenid);
+
+my $usercfg = cfs_read_file('user.cfg');
+check_user_enabled($usercfg, $username);
+check_token_enabled($usercfg, $username, $token);
+
+my $ctime = time();
+
+my $user = $usercfg->{users}->{$username};
+die "account expired\n" if $user->{expire} && ($user->{expire} < $ctime);
+
+my $token_info = $user->{tokens}->{$token};
+die "token expired\n" if $token_info->{expire} && ($token_info->{expire} <
$ctime);
+
+# FIXME: actually implement token verification here
+return wantarray ? ($tokenid) : $tokenid;
+}
+
+
# VNC tickets
# - they do not contain the username in plain text
# - they are restricted to a specific resource path (example: '/vms/100')
@@ -1578,6 +1611,20 @@ sub user_get_tfa {
}
}
+# FIXME: actually implement token generation here
+sub generate_token {
+my ($tokenid) = @_;
+
+return "${tokenid}_VALUE";
+}
+
+# FIXME: actually implement token deletion here
+sub delete_token {
+my ($tokenid) = @_;
+
+return;
+}
+
# bash completion helpers
register_standard_option('userid-completed',
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 17/23] allow ticket in auth header as fallback
based on idea & RFC by Tim Marx, incorporating feedback by Thomas
Lamprecht. this will be extended to support API tokens in the
Authorization header as well, so make it generic.
Signed-off-by: Fabian Grünbichler
---
Notes:
semi-independent, could also leave extract_auth_cookie as alias/wrapper to
avoid a change in PMG. but since we need to change other method signatures
anyway for the token part, we could change this as well.
as-is, needs a versioned breaks/depends on pve-manager and some part of PMG?
PVE/APIServer/AnyEvent.pm | 5 -
PVE/APIServer/Formatter.pm | 12 ++--
2 files changed, 10 insertions(+), 7 deletions(-)
diff --git a/PVE/APIServer/AnyEvent.pm b/PVE/APIServer/AnyEvent.pm
index 9aba27d..c0b90ab 100644
--- a/PVE/APIServer/AnyEvent.pm
+++ b/PVE/APIServer/AnyEvent.pm
@@ -1232,7 +1232,10 @@ sub unshift_read_header {
} elsif ($path =~ m/^\Q$base_uri\E/) {
my $token = $r->header('CSRFPreventionToken');
my $cookie = $r->header('Cookie');
- my $ticket =
PVE::APIServer::Formatter::extract_auth_cookie($cookie, $self->{cookie_name});
+ my $auth_header = $r->header('Authorization');
+ my $ticket =
PVE::APIServer::Formatter::extract_auth_value($cookie, $self->{cookie_name});
+ $ticket =
PVE::APIServer::Formatter::extract_auth_value($auth_header,
$self->{cookie_name})
+ if !$ticket;
my ($rel_uri, $format) = &$split_abs_uri($path,
$self->{base_uri});
if (!$format) {
diff --git a/PVE/APIServer/Formatter.pm b/PVE/APIServer/Formatter.pm
index 0c459bd..def1932 100644
--- a/PVE/APIServer/Formatter.pm
+++ b/PVE/APIServer/Formatter.pm
@@ -75,16 +75,16 @@ sub get_login_formatter {
# some helper functions
-sub extract_auth_cookie {
-my ($cookie, $cookie_name) = @_;
+sub extract_auth_value {
+my ($header, $key) = @_;
-return undef if !$cookie;
+return undef if !$header;
-my $ticket = ($cookie =~ /(?:^|\s)\Q$cookie_name\E=([^;]*)/)[0];
+my $value = ($header =~ /(?:^|\s)\Q$key\E(?:=| )([^;]*)/)[0];
-$ticket = uri_unescape($ticket) if $ticket;
+$value = uri_unescape($value) if $value;
-return $ticket;
+return $value;
}
sub create_auth_cookie {
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC/PATCH 0/23] API Tokens
This is a draft of a patch series which aims to introduce API tokens into PVE. The basic idea is to allow users to generate API token values that - are attributed to this users - easily revokable - possibly less privileged than the user itself - allow direct API calls without round-trips to create/refresh a ticket Token information is stored in user.cfg together with the other access-control information. The actual token values will be stored in a 'shadow' file under /etc/pve/priv (see patch #12 for some ideas/comments). Still missing: - thorough review ;) - adaption/extension of user.cfg tests (especially w.r.t. #6/7) - PVE::APIClient support - token value generation, deletion & verification (#12) - pveum integration - GUI integration (just the token API, not using tokens for login..) - documentation - PMG adaption (at least for the changed method signatures in pve-http-server) - checking API endpoints for 'notoken'-ification I tried to order independent clean-ups etc. up front with-in each repo, but some of them require versioned breaks/depends so it might make sense to wait for the full series for those. pve-common: Fabian Grünbichler (1): API schema: add 'notoken' property src/PVE/JSONSchema.pm | 5 + 1 file changed, 5 insertions(+) pve-access-control: Fabian Grünbichler (14): add missing 'use PVE::Auth::Plugin' user.cfg: sort entries alphabetically in each section user.cfg: sort ACL members access-control: remove check_permissions/permission rpcenv: drop unused roles() auth: pull username REs into variables refactor acl transformation code API token: add REs, helpers, parsing + writing API token: add API helpers DO NOT APPLY: API token stubs for token value handling API: add API token API endpoints API: include API tokens in ACL API endpoints API token: implement permission checks api: mark some paths notoken PVE/API2/ACL.pm | 30 +++- PVE/API2/AccessControl.pm | 5 + PVE/API2/User.pm | 287 ++- PVE/AccessControl.pm | 307 -- PVE/Auth/Plugin.pm| 5 +- PVE/RPCEnvironment.pm | 50 +++ 6 files changed, 568 insertions(+), 116 deletions(-) pve-http-server: Fabian Grünbichler (2): proxy_request: drop duplicate, unused parameter allow ticket in auth header as fallback api-server: extract, set and handle API token header PVE/APIServer/AnyEvent.pm| 27 +++ PVE/APIServer/Formatter.pm | 21 +++-- PVE/APIServer/Formatter/Bootstrap.pm | 1 + 3 files changed, 35 insertions(+), 14 deletions(-) pve-manager: Fabian Grünbichler (5): subscription: use rpcenv for permission check auth_handler: handle API tokens rest_handler: implement 'notoken' API endpoints pveproxy: use new cookie extraction method api/tasks: attribute token tasks to user PVE/API2/Cluster.pm | 3 ++ PVE/API2/Subscription.pm | 2 +- PVE/API2/Tasks.pm| 15 ++ PVE/HTTPServer.pm| 60 +++- PVE/Service/pveproxy.pm | 2 +- 5 files changed, 54 insertions(+), 28 deletions(-) -- 2.20.1 ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 15/23] api: mark some paths notoken
Signed-off-by: Fabian Grünbichler
---
Notes:
requires versioned dependency on libpve-common-perl
PVE/API2/AccessControl.pm | 4
1 file changed, 4 insertions(+)
diff --git a/PVE/API2/AccessControl.pm b/PVE/API2/AccessControl.pm
index 9d2da8d..2a52105 100644
--- a/PVE/API2/AccessControl.pm
+++ b/PVE/API2/AccessControl.pm
@@ -234,6 +234,7 @@ __PACKAGE__->register_method ({
user => 'world'
},
protected => 1, # else we can't access shadow files
+notoken => 1, # we don't want tokens to create tickets
description => "Create or verify authentication ticket.",
parameters => {
additionalProperties => 0,
@@ -347,6 +348,7 @@ __PACKAGE__->register_method ({
],
},
protected => 1, # else we can't access shadow files
+notoken => 1, # we don't want tokens to change the regular user password
description => "Change user password.",
parameters => {
additionalProperties => 0,
@@ -479,6 +481,7 @@ __PACKAGE__->register_method ({
],
},
protected => 1, # else we can't access shadow files
+notoken => 1, # we don't want tokens to change the regular user's TFA
settings
description => "Change user u2f authentication.",
parameters => {
additionalProperties => 0,
@@ -605,6 +608,7 @@ __PACKAGE__->register_method({
method => 'POST',
permissions => { user => 'all' },
protected => 1, # else we can't access shadow files
+notoken => 1, # we don't want tokens to access TFA information
description => 'Finish a u2f challenge.',
parameters => {
additionalProperties => 0,
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 13/23] API: include API tokens in ACL API endpoints
Signed-off-by: Fabian Grünbichler
---
PVE/API2/ACL.pm | 30 ++
1 file changed, 22 insertions(+), 8 deletions(-)
diff --git a/PVE/API2/ACL.pm b/PVE/API2/ACL.pm
index 3e42ac0..c340267 100644
--- a/PVE/API2/ACL.pm
+++ b/PVE/API2/ACL.pm
@@ -46,7 +46,7 @@ __PACKAGE__->register_method ({
properties => {
propagate => get_standard_option('acl-propagate'),
path => get_standard_option('acl-path'),
- type => { type => 'string', enum => ['user', 'group'] },
+ type => { type => 'string', enum => ['user', 'group', 'token']
},
ugid => { type => 'string' },
roleid => { type => 'string' },
},
@@ -68,8 +68,8 @@ __PACKAGE__->register_method ({
my $acl = $usercfg->{acl};
foreach my $path (keys %$acl) {
- foreach my $type (qw(users groups)) {
- my $d = $acl->{$path}->{$type};
+ foreach my $type (qw(user group token)) {
+ my $d = $acl->{$path}->{"${type}s"};
next if !$d;
next if !($audit || $rpcenv->check_perm_modify($authuser,
$path, 1));
foreach my $id (keys %$d) {
@@ -77,7 +77,7 @@ __PACKAGE__->register_method ({
my $propagate = $d->{$id}->{$role};
push @$res, {
path => $path,
- type => $type eq 'groups' ? 'group' : 'user',
+ type => $type,
ugid => $id,
roleid => $role,
propagate => $propagate,
@@ -114,6 +114,11 @@ __PACKAGE__->register_method ({
type => 'string', format => 'pve-groupid-list',
optional => 1,
},
+ tokens => {
+ description => "List of API tokens.",
+ type => 'string', format => 'pve-tokenid-list',
+ optional => 1,
+ },
roles => {
description => "List of roles.",
type => 'string', format => 'pve-roleid-list',
@@ -129,10 +134,8 @@ __PACKAGE__->register_method ({
code => sub {
my ($param) = @_;
- if (!($param->{users} || $param->{groups})) {
- raise_param_exc({
- users => "either 'users' or 'groups' is required.",
- groups => "either 'users' or 'groups' is required." });
+ if (!($param->{users} || $param->{groups} || $param->{tokens})) {
+ raise_param_exc({ map { $_ => "either 'users', 'groups' or 'tokens'
is required." } qw(users groups tokens) });
}
my $path = PVE::AccessControl::normalize_path($param->{path});
@@ -177,6 +180,17 @@ __PACKAGE__->register_method ({
$cfg->{acl}->{$path}->{users}->{$username}->{$role}
= $propagate;
}
}
+
+ foreach my $tokenid (split_list($param->{tokens})) {
+ my ($username, $token) =
PVE::AccessControl::split_tokenid($tokenid);
+ PVE::AccessControl::check_token_exist($cfg, $username,
$token);
+
+ if ($param->{delete}) {
+ delete
$cfg->{acl}->{$path}->{tokens}->{$tokenid}->{$role};
+ } else {
+ $cfg->{acl}->{$path}->{tokens}->{$tokenid}->{$role}
= $propagate;
+ }
+ }
}
cfs_write_file("user.cfg", $cfg);
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 05/23] access-control: remove check_permissions/permission
they have been handled by PVE::RPCEnvironment for quite some time
already, and the versions there are the complete ones that should be
actually used.
Signed-off-by: Fabian Grünbichler
---
Notes:
this requires the corresponding patch for and a versioned breaks on
pve-manager, where a call to check_permissions has been introduced in
2018(!?).
PVE/AccessControl.pm | 37 -
1 file changed, 37 deletions(-)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index aff9137..f246c85 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -1273,43 +1273,6 @@ sub roles {
return @ra;
}
-sub permission {
-my ($cfg, $user, $path) = @_;
-
-$user = PVE::Auth::Plugin::verify_username($user, 1);
-return {} if !$user;
-
-my @ra = roles($cfg, $user, $path);
-
-my $privs = {};
-
-foreach my $role (@ra) {
- if (my $privset = $cfg->{roles}->{$role}) {
- foreach my $p (keys %$privset) {
- $privs->{$p} = 1;
- }
- }
-}
-
-#print "priviledges $user $path = " . Dumper ($privs);
-
-return $privs;
-}
-
-sub check_permissions {
-my ($username, $path, $privlist) = @_;
-
-$path = normalize_path($path);
-my $usercfg = cfs_read_file('user.cfg');
-my $perm = permission($usercfg, $username, $path);
-
-foreach my $priv (split_list($privlist)) {
- return undef if !$perm->{$priv};
-};
-
-return 1;
-}
-
sub remove_vm_access {
my ($vmid) = @_;
my $delVMaccessFn = sub {
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 21/23] rest_handler: implement 'notoken' API endpoints
that are not available with API tokens for security reasons, such as access
control related endpoints.
Signed-off-by: Fabian Grünbichler
---
Notes:
pairs with patch in pve-common that adds this to the schema-schema. any
modules
setting that flag need a corresponding versioned depends on
libpve-common-perl..
PVE/HTTPServer.pm | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/PVE/HTTPServer.pm b/PVE/HTTPServer.pm
index c11b4813..b45c3262 100755
--- a/PVE/HTTPServer.pm
+++ b/PVE/HTTPServer.pm
@@ -7,7 +7,7 @@ use PVE::SafeSyslog;
use PVE::INotify;
use PVE::Tools;
use PVE::APIServer::AnyEvent;
-use PVE::Exception qw(raise_param_exc raise);
+use PVE::Exception qw(raise_param_exc raise_perm_exc raise);
use PVE::RPCEnvironment;
use PVE::AccessControl;
@@ -147,6 +147,9 @@ sub rest_handler {
$uri_param->{$p} = $params->{$p};
}
+ raise_perm_exc("URI '$rel_uri' not available with API token, need
proper ticket.\n")
+ if $auth->{api_token} && $info->{notoken};
+
# check access permissions
$rpcenv->check_api2_permissions($info->{permissions}, $auth->{userid},
$uri_param);
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [PATCH 19/23] subscription: use rpcenv for permission check
this is the last caller of PVE::AccessControl::check_permissions(),
which is the last caller of PVE::AccessControl::permission(). both can
thus be dropped altogether.
Signed-off-by: Fabian Grünbichler
---
Notes:
this is semi-independent, but if applied we need to remember that the
corresponding patch in pve-access-control needs to break on pve-manager with
the right version
PVE/API2/Subscription.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/PVE/API2/Subscription.pm b/PVE/API2/Subscription.pm
index 2c0c4e84..959c2c1b 100644
--- a/PVE/API2/Subscription.pm
+++ b/PVE/API2/Subscription.pm
@@ -108,7 +108,7 @@ __PACKAGE__->register_method ({
my $rpcenv = PVE::RPCEnvironment::get();
my $authuser = $rpcenv->get_user();
- my $has_permission = PVE::AccessControl::check_permissions($authuser,
"/nodes/$node", 'Sys.Audit');
+ my $has_permission = $rpcenv->check($authuser, "/nodes/$node",
['Sys.Audit'], 1);
my $server_id = PVE::API2Tools::get_hwaddress();
my $url = "https://www.proxmox.com/proxmox-ve/pricing";;
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 14/23] API token: implement permission checks
non-privsep tokens will always return the roles/permissions of their
associated users. privsep tokens will return unfiltered roles, but
filtered permissions.
Signed-off-by: Fabian Grünbichler
---
PVE/AccessControl.pm | 30 ++
PVE/RPCEnvironment.pm | 27 +--
2 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/PVE/AccessControl.pm b/PVE/AccessControl.pm
index b5dfe4b..0e7f1f7 100644
--- a/PVE/AccessControl.pm
+++ b/PVE/AccessControl.pm
@@ -1353,10 +1353,25 @@ sub roles {
my ($cfg, $user, $path) = @_;
# NOTE: we do not consider pools here.
+# NOTE: for privsep tokens, this does not filter roles by those that the
+# corresponding user has.
# Use $rpcenv->permission() for any actual permission checks!
return 'Administrator' if $user eq 'root@pam'; # root can do anything
+if (pve_verify_tokenid($user, 1)) {
+ my $tokenid = $user;
+ my ($username, $token) = split_tokenid($tokenid);
+
+ my $token_info = $cfg->{users}->{$username}->{tokens}->{$token};
+ return () if !$token_info;
+
+ my @user_roles = roles($cfg, $username, $path);
+
+ # return full user privileges
+ return @user_roles if !$token_info->{privsep};
+}
+
my $perm = {};
foreach my $p (sort keys %{$cfg->{acl}}) {
@@ -1368,6 +1383,21 @@ sub roles {
#print "CHECKACL $path $p\n";
#print "ACL $path = " . Dumper ($acl);
+ if (my $ri = $acl->{tokens}->{$user}) {
+ my $new;
+ foreach my $role (keys %$ri) {
+ my $propagate = $ri->{$role};
+ if ($final || $propagate) {
+ #print "APPLY ROLE $p $user $role\n";
+ $new = {} if !$new;
+ $new->{$role} = 1;
+ }
+ }
+ if ($new) {
+ $perm = $new; # overwrite previous settings
+ next;
+ }
+ }
if (my $ri = $acl->{users}->{$user}) {
my $new;
diff --git a/PVE/RPCEnvironment.pm b/PVE/RPCEnvironment.pm
index 7e0af70..f378522 100644
--- a/PVE/RPCEnvironment.pm
+++ b/PVE/RPCEnvironment.pm
@@ -30,6 +30,9 @@ my $compile_acl_path = sub {
$cache->{$user} = {} if !$cache->{$user};
my $data = $cache->{$user};
+my ($username, undef) = PVE::AccessControl::split_tokenid($user, 1);
+die "internal error" if $username && !defined($cache->{$username});
+
if (!$data->{poolroles}) {
$data->{poolroles} = {};
@@ -76,6 +79,13 @@ my $compile_acl_path = sub {
}
}
}
+
+if ($username) {
+ # intersect user and token permissions
+ my $user_privs = $cache->{$username}->{privs}->{$path};
+ $privs = { map { $_ => 1 } grep { $user_privs->{$_} } keys %$privs };
+}
+
$data->{privs}->{$path} = $privs;
return $privs;
@@ -89,8 +99,21 @@ sub permissions {
return $cfg->{roles}->{'Administrator'};
}
-$user = PVE::AccessControl::verify_username($user, 1);
-return {} if !$user;
+if (PVE::AccessControl::pve_verify_tokenid($user, 1)) {
+ my ($username, $token) = PVE::AccessControl::split_tokenid($user);
+ my $cfg = $self->{user_cfg};
+ my $token_info = $cfg->{users}->{$username}->{tokens}->{$token};
+ return {} if !$token_info;
+
+ # ensure cache for user is populated
+ my $user_perms = $self->permissions($username, $path);
+
+ # return user privs for non-privsep tokens
+ return $user_perms if !$token_info->{privsep};
+} else {
+ $user = PVE::AccessControl::verify_username($user, 1);
+ return {} if !$user;
+}
my $cache = $self->{aclcache};
$cache->{$user} = {} if !$cache->{$user};
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 18/23] api-server: extract, set and handle API token header
Signed-off-by: Fabian Grünbichler
---
Notes:
versioned breaks/depends with pve-manager and part of PMG?
PVE/APIServer/AnyEvent.pm| 22 +++---
PVE/APIServer/Formatter.pm | 9 +
PVE/APIServer/Formatter/Bootstrap.pm | 1 +
3 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/PVE/APIServer/AnyEvent.pm b/PVE/APIServer/AnyEvent.pm
index c0b90ab..1197857 100644
--- a/PVE/APIServer/AnyEvent.pm
+++ b/PVE/APIServer/AnyEvent.pm
@@ -544,7 +544,7 @@ sub websocket_proxy {
}
sub proxy_request {
-my ($self, $reqstate, $clientip, $host, $node, $method, $uri, $ticket,
$token, $params) = @_;
+my ($self, $reqstate, $clientip, $host, $node, $method, $uri, $auth,
$params) = @_;
eval {
my $target;
@@ -564,8 +564,12 @@ sub proxy_request {
PVEClientIP => $clientip,
};
- $headers->{'cookie'} =
PVE::APIServer::Formatter::create_auth_cookie($ticket, $self->{cookie_name}) if
$ticket;
- $headers->{'CSRFPreventionToken'} = $token if $token;
+ $headers->{'cookie'} =
PVE::APIServer::Formatter::create_auth_cookie($auth->{ticket},
$self->{cookie_name})
+ if $auth->{ticket};
+ $headers->{'Authorization'} =
PVE::APIServer::Formatter::create_auth_header($auth->{api_token},
$self->{apitoken_name})
+ if $auth->{api_token};
+ $headers->{'CSRFPreventionToken'} = $auth->{token}
+ if $auth->{token};
$headers->{'Accept-Encoding'} = 'gzip' if ($reqstate->{accept_gzip} &&
$self->{compression});
if (defined(my $host = $reqstate->{request}->header('Host'))) {
@@ -747,7 +751,7 @@ sub handle_api2_request {
$res->{proxy_params}->{tmpfilename} = $reqstate->{tmpfilename} if
$upload_state;
$self->proxy_request($reqstate, $clientip, $host,
$res->{proxynode}, $method,
-$r->uri, $auth->{ticket}, $auth->{token},
$res->{proxy_params});
+$r->uri, $auth, $res->{proxy_params});
return;
} elsif ($upgrade && ($method eq 'GET') && ($path =~ m|websocket$|)) {
@@ -1236,6 +1240,8 @@ sub unshift_read_header {
my $ticket =
PVE::APIServer::Formatter::extract_auth_value($cookie, $self->{cookie_name});
$ticket =
PVE::APIServer::Formatter::extract_auth_value($auth_header,
$self->{cookie_name})
if !$ticket;
+ my $api_token =
PVE::APIServer::Formatter::extract_auth_value($auth_header,
$self->{apitoken_name})
+ if !$ticket;
my ($rel_uri, $format) = &$split_abs_uri($path,
$self->{base_uri});
if (!$format) {
@@ -1244,7 +1250,7 @@ sub unshift_read_header {
}
eval {
- $auth = $self->auth_handler($method, $rel_uri, $ticket,
$token,
+ $auth = $self->auth_handler($method, $rel_uri, $ticket,
$token, $api_token,
$reqstate->{peer_host});
};
if (my $err = $@) {
@@ -1638,6 +1644,7 @@ sub new {
my $self = bless { %args }, $class;
$self->{cookie_name} //= 'PVEAuthCookie';
+$self->{apitoken_name} //= 'PVEAPIToken';
$self->{base_uri} //= "/api2";
$self->{dirs} //= {};
$self->{title} //= 'API Inspector';
@@ -1645,7 +1652,7 @@ sub new {
# formatter_config: we pass some configuration values to the Formatter
$self->{formatter_config} = {};
-foreach my $p (qw(cookie_name base_uri title)) {
+foreach my $p (qw(apitoken_name cookie_name base_uri title)) {
$self->{formatter_config}->{$p} = $self->{$p};
}
$self->{formatter_config}->{csrfgen_func} =
@@ -1780,7 +1787,7 @@ sub generate_csrf_prevention_token {
}
sub auth_handler {
-my ($self, $method, $rel_uri, $ticket, $token, $peer_host) = @_;
+my ($self, $method, $rel_uri, $ticket, $token, $api_token, $peer_host) =
@_;
die "implement me";
@@ -1790,6 +1797,7 @@ sub auth_handler {
#userid => $username,
#age => $age,
#isUpload => $isUpload,
+#api_token => $api_token,
#};
}
diff --git a/PVE/APIServer/Formatter.pm b/PVE/APIServer/Formatter.pm
index def1932..20455a0 100644
--- a/PVE/APIServer/Formatter.pm
+++ b/PVE/APIServer/Formatter.pm
@@ -95,4 +95,13 @@ sub create_auth_cookie {
return "${cookie_name}=$encticket; path=/; secure;";
}
+sub create_auth_header {
+my ($value, $key) = @_;
+
+return undef if !$key;
+
+my $encoded = uri_escape($
[pve-devel] [RFC 20/23] auth_handler: handle API tokens
Signed-off-by: Fabian Grünbichler
---
Notes:
versioned breaks/depends between pve-manager and libpve-http-server-perl!
versioned depends on libpve-access-control
PVE/HTTPServer.pm | 55 ++-
1 file changed, 30 insertions(+), 25 deletions(-)
diff --git a/PVE/HTTPServer.pm b/PVE/HTTPServer.pm
index ce895725..c11b4813 100755
--- a/PVE/HTTPServer.pm
+++ b/PVE/HTTPServer.pm
@@ -52,7 +52,7 @@ sub generate_csrf_prevention_token {
}
sub auth_handler {
-my ($self, $method, $rel_uri, $ticket, $token, $peer_host) = @_;
+my ($self, $method, $rel_uri, $ticket, $token, $api_token, $peer_host) =
@_;
my $rpcenv = $self->{rpcenv};
@@ -77,36 +77,40 @@ sub auth_handler {
my $isUpload = 0;
if ($require_auth) {
+ if ($api_token) {
+ $username = PVE::AccessControl::verify_token($api_token);
+ $rpcenv->set_user($username); #actually tokenid in this case
+ } else {
+ die "No ticket\n" if !$ticket;
- die "No ticket\n" if !$ticket;
+ ($username, $age, my $tfa_info) =
PVE::AccessControl::verify_ticket($ticket);
- ($username, $age, my $tfa_info) =
PVE::AccessControl::verify_ticket($ticket);
-
- if (defined($tfa_info)) {
- if (defined(my $challenge = $tfa_info->{challenge})) {
- $rpcenv->set_u2f_challenge($challenge);
+ if (defined($tfa_info)) {
+ if (defined(my $challenge = $tfa_info->{challenge})) {
+ $rpcenv->set_u2f_challenge($challenge);
+ }
+ die "No ticket\n"
+ if ($rel_uri ne '/access/tfa' || $method ne 'POST');
}
- die "No ticket\n"
- if ($rel_uri ne '/access/tfa' || $method ne 'POST');
- }
- $rpcenv->set_user($username);
+ $rpcenv->set_user($username);
- if ($method eq 'POST' && $rel_uri =~
m|^/nodes/([^/]+)/storage/([^/]+)/upload$|) {
- my ($node, $storeid) = ($1, $2);
- # we disable CSRF checks if $isUpload is set,
- # to improve security we check user upload permission here
- my $perm = { check => ['perm', "/storage/$storeid",
['Datastore.AllocateTemplate']] };
- $rpcenv->check_api2_permissions($perm, $username, {});
- $isUpload = 1;
- }
+ if ($method eq 'POST' && $rel_uri =~
m|^/nodes/([^/]+)/storage/([^/]+)/upload$|) {
+ my ($node, $storeid) = ($1, $2);
+ # we disable CSRF checks if $isUpload is set,
+ # to improve security we check user upload permission here
+ my $perm = { check => ['perm', "/storage/$storeid",
['Datastore.AllocateTemplate']] };
+ $rpcenv->check_api2_permissions($perm, $username, {});
+ $isUpload = 1;
+ }
- # we skip CSRF check for file upload, because it is
- # difficult to pass CSRF HTTP headers with native html forms,
- # and it should not be necessary at all.
- my $euid = $>;
- PVE::AccessControl::verify_csrf_prevention_token($username, $token)
- if !$isUpload && ($euid != 0) && ($method ne 'GET');
+ # we skip CSRF check for file upload, because it is
+ # difficult to pass CSRF HTTP headers with native html forms,
+ # and it should not be necessary at all.
+ my $euid = $>;
+ PVE::AccessControl::verify_csrf_prevention_token($username, $token)
+ if !$isUpload && ($euid != 0) && ($method ne 'GET');
+ }
}
return {
@@ -115,6 +119,7 @@ sub auth_handler {
userid => $username,
age => $age,
isUpload => $isUpload,
+ api_token => $api_token,
};
}
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 23/23] api/tasks: attribute token tasks to user
and store token ID in separate, currently unused member.
Signed-off-by: Fabian Grünbichler
---
Notes:
versioned depends on libpve-access-control
alternatively, we could also change the fork_worker signature and encode
this
inside the task information on disk, but that would be a more invasive
solution..
PVE/API2/Cluster.pm | 3 +++
PVE/API2/Tasks.pm | 15 +++
2 files changed, 18 insertions(+)
diff --git a/PVE/API2/Cluster.pm b/PVE/API2/Cluster.pm
index 382a2c12..00f811cd 100644
--- a/PVE/API2/Cluster.pm
+++ b/PVE/API2/Cluster.pm
@@ -460,6 +460,9 @@ __PACKAGE__->register_method({
my $all = $rpcenv->check($authuser, "/", [ 'Sys.Audit' ], 1);
foreach my $task (@$tlist) {
+ if (PVE::AccessControl::pve_verify_tokenid($task->{user}, 1)) {
+ ($task->{user}, $task->{tokenid}) =
PVE::AccessControl::split_tokenid($task->{user});
+ }
push @$res, $task if $all || ($task->{user} eq $authuser);
}
diff --git a/PVE/API2/Tasks.pm b/PVE/API2/Tasks.pm
index 73e398eb..5dc6a4ad 100644
--- a/PVE/API2/Tasks.pm
+++ b/PVE/API2/Tasks.pm
@@ -16,6 +16,14 @@ use PVE::AccessControl;
use base qw(PVE::RESTHandler);
+my $convert_token_task = sub {
+my ($task) = @_;
+
+if (PVE::AccessControl::pve_verify_tokenid($task->{user}, 1)) {
+ ($task->{user}, $task->{tokenid}) =
PVE::AccessControl::split_tokenid($task->{user});
+}
+};
+
__PACKAGE__->register_method({
name => 'node_tasks',
path => '',
@@ -142,6 +150,7 @@ __PACKAGE__->register_method({
$task->{endtime} = hex($endtime) if $endtime;
$task->{status} = $status if $status;
+ $convert_token_task->($task);
if (!$filter_task->($task)) {
push @$res, $task;
$limit--;
@@ -242,6 +251,8 @@ __PACKAGE__->register_method({
my $user = $rpcenv->get_user();
my $node = $param->{node};
+ $convert_token_task->($task);
+
if ($user ne $task->{user}) {
$rpcenv->check($user, "/nodes/$node", [ 'Sys.Modify' ]);
}
@@ -309,6 +320,8 @@ __PACKAGE__->register_method({
my $start = $param->{start} // 0;
my $limit = $param->{limit} // 50;
+ $convert_token_task->($task);
+
if ($user ne $task->{user}) {
$rpcenv->check($user, "/nodes/$node", [ 'Sys.Audit' ]);
}
@@ -365,6 +378,8 @@ __PACKAGE__->register_method({
my $user = $rpcenv->get_user();
my $node = $param->{node};
+ $convert_token_task->($task);
+
if ($user ne $task->{user}) {
$rpcenv->check($user, "/nodes/$node", [ 'Sys.Audit' ]);
}
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
[pve-devel] [RFC 22/23] pveproxy: use new cookie extraction method
we only care about the regular cookie case for the index.
Signed-off-by: Fabian Grünbichler
---
Notes:
versioned breaks/depends on libpve-http-perl!
PVE/Service/pveproxy.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/PVE/Service/pveproxy.pm b/PVE/Service/pveproxy.pm
index dd123dd8..786545f2 100755
--- a/PVE/Service/pveproxy.pm
+++ b/PVE/Service/pveproxy.pm
@@ -183,7 +183,7 @@ sub get_index {
$lang = $newlang;
}
}
- my $ticket = PVE::APIServer::Formatter::extract_auth_cookie($cookie,
$server->{cookie_name});
+ my $ticket = PVE::APIServer::Formatter::extract_auth_value($cookie,
$server->{cookie_name});
if (($username = PVE::AccessControl::verify_ticket($ticket, 1))) {
$token =
PVE::AccessControl::assemble_csrf_prevention_token($username);
}
--
2.20.1
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [RFC/PATCH 0/23] API Tokens
and I just realized that I dropped the per-repo subject-prefix from all patches instead of just the cover-letter.. #1 is pve-common #2-15 are pve-access-control #16-18 are pve-http-server #19-#23 are pve-manager if you want a resend, just shout.. On October 17, 2019 3:13 pm, Fabian Grünbichler wrote: > This is a draft of a patch series which aims to introduce API tokens > into PVE. > > The basic idea is to allow users to generate API token values that > - are attributed to this users > - easily revokable > - possibly less privileged than the user itself > - allow direct API calls without round-trips to create/refresh a ticket > > Token information is stored in user.cfg together with the other > access-control information. The actual token values will be stored in a > 'shadow' file under /etc/pve/priv (see patch #12 for some > ideas/comments). > > Still missing: > - thorough review ;) > - adaption/extension of user.cfg tests (especially w.r.t. #6/7) > - PVE::APIClient support > - token value generation, deletion & verification (#12) > - pveum integration > - GUI integration (just the token API, not using tokens for login..) > - documentation > - PMG adaption (at least for the changed method signatures in > pve-http-server) > - checking API endpoints for 'notoken'-ification > > I tried to order independent clean-ups etc. up front with-in each repo, > but some of them require versioned breaks/depends so it might make sense > to wait for the full series for those. > > pve-common: > > Fabian Grünbichler (1): > API schema: add 'notoken' property > > src/PVE/JSONSchema.pm | 5 + > 1 file changed, 5 insertions(+) > > pve-access-control: > > Fabian Grünbichler (14): > add missing 'use PVE::Auth::Plugin' > user.cfg: sort entries alphabetically in each section > user.cfg: sort ACL members > access-control: remove check_permissions/permission > rpcenv: drop unused roles() > auth: pull username REs into variables > refactor acl transformation code > API token: add REs, helpers, parsing + writing > API token: add API helpers > DO NOT APPLY: API token stubs for token value handling > API: add API token API endpoints > API: include API tokens in ACL API endpoints > API token: implement permission checks > api: mark some paths notoken > > PVE/API2/ACL.pm | 30 +++- > PVE/API2/AccessControl.pm | 5 + > PVE/API2/User.pm | 287 ++- > PVE/AccessControl.pm | 307 -- > PVE/Auth/Plugin.pm| 5 +- > PVE/RPCEnvironment.pm | 50 +++ > 6 files changed, 568 insertions(+), 116 deletions(-) > > pve-http-server: > > Fabian Grünbichler (2): > proxy_request: drop duplicate, unused parameter > allow ticket in auth header as fallback > api-server: extract, set and handle API token header > > PVE/APIServer/AnyEvent.pm| 27 +++ > PVE/APIServer/Formatter.pm | 21 +++-- > PVE/APIServer/Formatter/Bootstrap.pm | 1 + > 3 files changed, 35 insertions(+), 14 deletions(-) > > pve-manager: > > Fabian Grünbichler (5): > subscription: use rpcenv for permission check > auth_handler: handle API tokens > rest_handler: implement 'notoken' API endpoints > pveproxy: use new cookie extraction method > api/tasks: attribute token tasks to user > > PVE/API2/Cluster.pm | 3 ++ > PVE/API2/Subscription.pm | 2 +- > PVE/API2/Tasks.pm| 15 ++ > PVE/HTTPServer.pm| 60 +++- > PVE/Service/pveproxy.pm | 2 +- > 5 files changed, 54 insertions(+), 28 deletions(-) > > -- > 2.20.1 > > > ___ > pve-devel mailing list > pve-devel@pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [RFC 1/23] API schema: add 'notoken' property
On October 17, 2019 4:46 pm, Thomas Lamprecht wrote:
> On 10/17/19 3:13 PM, Fabian Grünbichler wrote:
>> to mark API methods which should not be available to clients authenticated
>> using an API token
>
> please wrap lines, for commit messages it's common to use << 70 as limit,
> at least <= 80 would be really great.
indeed. that sometimes happens to slip through when I compose the
message directly with -m "..." -m "..." ;)
>
>>
>> Signed-off-by: Fabian Grünbichler
>> ---
>>
>> Notes:
>> if applied, any users of this need corresponding versioned depends.
>>
>> src/PVE/JSONSchema.pm | 5 +
>> 1 file changed, 5 insertions(+)
>>
>> diff --git a/src/PVE/JSONSchema.pm b/src/PVE/JSONSchema.pm
>> index db38d44..a42d814 100644
>> --- a/src/PVE/JSONSchema.pm
>> +++ b/src/PVE/JSONSchema.pm
>> @@ -1263,6 +1263,11 @@ my $method_schema = {
>> description => "Method needs special privileges - only pvedaemon
>> can execute it",
>> optional => 1,
>> },
>> +notoken => {
>
> IMO slightly to general. Also, negative named properties are not ideal, even
> if I sometimes use them too, Dominik always notices it in the web ui ;)
>
> The question is, do we want to have certain API endpoints which we straight
> forbid, or do we want some which can only be called by sending along the full
> user credentials.
>
> The latter would allow to only use an API token for a mobile app (for
> example),
> but still have a password change dialog there, with allows to change the
> password of the user which the token belongs, but only if the current password
> is sent along and without some hacks required by the app where it temporarily
> requests a "full user ticket". (how TFA would play into something like this is
> another question...)
>
> I mean, just an example, but we could then resort to name it something like
> "require-fullauth"
>
> or we reverse the default and name it:
> "token-auth-sufficient"
>
> anyway, bike-shedding details, but for an RFC OK I hope ;P
sure - I don't care about the name at all, and a positively-named option
with a default of 1 is also okay :)
I am not sure whether we even really need such a schema option - we
could also just have a helper in PVE::RPCEnvironment or
PVE::AccessControl and call that directly at the start of the API
methods. in the end, it probably comes down to how many such API methods
there are.
>
>> +type => 'boolean',
>> +description => "Method is not available for clients authenticated
>> using an API token.",
>> +optional => 1,
>> +},
>> download => {
>> type => 'boolean',
>> description => "Method downloads the file content (filename is the
>> return value of the method).",
>>
>
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v3 guest-common 04/19] helpers: add method to represent config as a table
On October 17, 2019 7:48 pm, Thomas Lamprecht wrote:
> On 10/14/19 10:28 AM, Oguz Bektas wrote:
>> in vm_pending API, this method is used to represent the configuration as
>> a table with current, pending and delete columns.
>>
>> by adding it as a guesthelper, we can also use it for container pending
>> changes.
>>
>> Signed-off-by: Oguz Bektas
>> ---
>> PVE/GuestHelpers.pm | 35 +++
>> 1 file changed, 35 insertions(+)
>>
>> diff --git a/PVE/GuestHelpers.pm b/PVE/GuestHelpers.pm
>> index a36b9bc..a1ec76f 100644
>> --- a/PVE/GuestHelpers.pm
>> +++ b/PVE/GuestHelpers.pm
>> @@ -142,4 +142,39 @@ sub format_pending {
>> }
>> }
>>
>> +sub conf_table_with_pending {
>
> nothing to do with table itself, it just returns a array of
> hashes with current and pending value..
>
> I'd rather like to not bind method names to something their caller
> may do with them, but to something they actually do their self.
>
> Albeit a good name is not to easy here... Maybe:
> get_current_config_with_pending
>
it does not really 'get' the config either though ;)
I don't really know how to call
hash1 hash2 => list of items derived from hash1 and hash2
except that it's some kind of 'transformation'. it's not a mapping
operation (it could be seen as sequence of three mappings), it's not a
flattening, it's not folding.
there is in fact no information added to the input, it's just presented
in a different structure.
'transform_to_list_with_pending' was about as far as we got off-list,
and Oguz didn't like that at all :-P
>> +my ($conf, $pending_delete_hash) = @_;
>> +
>> +my $res = [];
>> +
>> +foreach my $opt (keys %$conf) {
>> +next if ref($conf->{$opt});
>> +my $item = { key => $opt };
>> +$item->{value} = $conf->{$opt} if defined($conf->{$opt});
>> +$item->{pending} = $conf->{pending}->{$opt} if
>> defined($conf->{pending}->{$opt});
>> +$item->{delete} = ($pending_delete_hash->{$opt}->{force} ? 2 : 1) if
>> exists $pending_delete_hash->{$opt};
>> +
>> +push @$res, $item;
>> +}
>> +
>> +foreach my $opt (keys %{$conf->{pending}}) {
>> +next if $opt eq 'delete';
>> +next if ref($conf->{pending}->{$opt}); # just to be sure
>> +next if defined($conf->{$opt});
>> +my $item = { key => $opt };
>> +$item->{pending} = $conf->{pending}->{$opt};
>> +
>> +push @$res, $item;
>> +}
>> +
>> +while (my ($opt, $force) = each %$pending_delete_hash) {
>> +next if $conf->{pending}->{$opt}; # just to be sure
>> +next if $conf->{$opt};
>> +my $item = { key => $opt, delete => ($force ? 2 : 1)};
>> +push @$res, $item;
>> +}
>> +
>> +return $res;
>> +}
>> +
>> 1;
>>
>
>
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH qemu-server v3] Fix #2412: Missing VMs in pools
On October 15, 2019 12:17 pm, Dominic Jäger wrote:
> Between calling vm_destroy and removing the ID from user.cfg
> (remove_vm_access)
> creating a new VM with this ID was possible. VMs could go missing from pools
> as
> a consequence.
>
> Adding a lock solves this for clones from the same node. Additionally,
> unlinking must happen at the very end of the deletion process to avoid that
> other nodes use the ID in the meanwhile.
>
> Co-developed-by: Fabian Grünbichler
> Signed-off-by: Dominic Jäger
> ---
> v1->v2: Move unlink to the end of the deletion process
> v2->v3: Use skiplock parameter and add error handling for unlink
>
> I'll send a separate patch for the discussed clean up.
and for pve-container? ;)
> PVE/API2/Qemu.pm | 12 +---
> 1 file changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index 267a08e..7e1d314 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -1492,9 +1492,15 @@ __PACKAGE__->register_method({
> my $upid = shift;
>
> syslog('info', "destroy VM $vmid: $upid\n");
> - PVE::QemuServer::vm_destroy($storecfg, $vmid, $skiplock);
> - PVE::AccessControl::remove_vm_access($vmid);
> -PVE::Firewall::remove_vmfw_conf($vmid);
> + PVE::QemuConfig->lock_config($vmid, sub {
> + die "VM $vmid is running - destroy failed\n"
> + if (PVE::QemuServer::check_running($vmid));
> + PVE::QemuServer::destroy_vm($storecfg, $vmid, 1, $skiplock);
> + PVE::AccessControl::remove_vm_access($vmid);
> + PVE::Firewall::remove_vmfw_conf($vmid);
> + unlink PVE::QemuConfig->config_file($vmid)
> + or die "Removal of VM $vmid config file failed: $!\n";
> + });
> };
>
> return $rpcenv->fork_worker('qmdestroy', $vmid, $authuser, $realcmd);
> --
> 2.20.1
>
>
>
___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] [PATCH v4 0/7] add purge option for VM/CT destroy
On October 17, 2019 6:40 pm, Thomas Lamprecht wrote: > On 10/15/19 1:00 PM, Christian Ebner wrote: >> When destroying a VM/CT, we intentionally did not remove all related configs >> such >> as backup or replication jobs. >> The intention of this flag is to allow the removal of such configs on >> destroy. >> >> This patch series implements this functionality and additionally moves the >> cfs register code for vzdump.cron to pve-guest-common to avoid a cyclic >> dependency. > > Looks OK for me, after a quick look. > > If there are no high-level objections to the move of VZDump config/schema > stuff I'd give it a deeper look + test and apply it, if all seems well. > > Fabian, Dietmar? no objections from me (see discussion of previous version ;)) from a quick glance: the commit IDs in commit messages are probably not a good idea, and this conflicts with the "fix locking in vm_destroy API path" patch by Dominic[1] so it might need a rebase. [1]: applied for qemu-server, still missing for pve-container - potentially the latter could be fixed together with this series? ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
