date:20190515

Re: [pve-devel] qemu : add disk option for physical/logical block (could improve windows guest performance wit ceph)

2019-05-15 Thread Dietmar Maurer

> could we add this as disk option ?  (not defined by default).

Sounds like a good idea. Maybe we can even set a better default (512e)?

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] qemu : add disk option for physical/logical block (could improve windows guest performance wit ceph)

2019-05-15 Thread Alexandre DERUMIER

Hi,

A guy have reported on ceph mailing, a feature which is helping windows guest
with ceph

https://review.opendev.org/#/c/658283/

"Add support for configuring the physical_block_size and
logical_block_size hints of Cinder RBD volumes using the new
'disk_geometry' configuration option.

Ceph performs much better when I/O is 4k-aligned otherwise it has to
read the 4k from disk, modify and write it again. Linux guests generally
submit 4k-aligned I/O, however Windows guests generally submit
512b-aligned I/O. When hinted with physical_block_size=4096 Windows
guests will switch to submitting most I/O as 4k-aligned based on both
testing and information from Microsoft KB 2510009.
...

"

the qemu command line options are

physical_block_size
logical_block_size

-device
virtio-blk-pci,id=data1,drive=data_image1,bus=pci.0,physical_block_size=4096,logical_block_size=512

in openstack,they propose to make a combo option "disk_geometry".

https://review.opendev.org/#/c/658283/2/releasenotes/notes/rbd-disk-geometry-7753e2e3bc44cc3d.yaml

cfg.StrOpt('disk_geometry', default='512e', choices=['512', '512e', '4k'],
help='Configure the reported geometry (physical, logical) '
'block size as 512 native (512/512), '
'512e emulated (4096/512) or 4k native (4096/4096). '
'This hints to the operating system the desired size '
'of I/O updates submitted to the disk. Windows systems '
'send 512b-aligned I/O by default (unlike Linux which '
'typically submits 4k-aligned I/O). Setting this option '
'to 512e may allow the operating system to submit a 4k '
'operation instead which will prevent Ceph needing to do '
'an expensive read-modify-write of the smaller operation. '
'512e is the recommended setting as 4k native can cause '
'compatability issues with some software and operating '
'system versions according to Microsoft KB 2510009.'),

could we add this as disk option ? (not defined by default).

I'm going to do some benchmark next week.

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] pve-firewall: default loglevel behaviour change

2019-05-15 Thread Alexandre DERUMIER

Thanks !

Also,
about new log ratelimit
https://git.proxmox.com/?p=pve-firewall.git;a=blobdiff;f=src/PVE/Firewall.pm;h=48e630004bf424c2db578aad3e60eb61db42f8e6;hp=ba1cb5f5924af094fe647f98d40e2703eca96ff4;hb=cc37e0005d0617e405883fd12b80faf65cedb153;hpb=95ed90a9a8418173ab7cac2d0c1f6ddd04c376cc

it's seem that the default is now
+my $global_log_ratelimit = '--limit 1/sec';


(If don't have the option "log_ratelimit" in cluster.fw)


 
+my $set_global_log_ratelimit = sub {
+my $cluster_opts = shift;
+
+$global_log_ratelimit = '--limit 1/sec';
+if (defined(my $log_rlimit = $cluster_opts->{log_ratelimit})) {
+   my $ll_format = $cluster_option_properties->{log_ratelimit}->{format};
+   my $limit = PVE::JSONSchema::parse_property_string($ll_format, 
$log_rlimit);
+
+   if ($limit->{enable}) {
+   if (my $rate = $limit->{rate}) {
+   $global_log_ratelimit = "--limit $rate";
+   }
+   if (my $burst = $limit->{burst}) {
+   $global_log_ratelimit .= " --limit-burst $burst";
+   }
+   } else {
+   $global_log_ratelimit = undef;
+   }
+}
+};


but it's correctly disabled with
"log_ratelimit: enable=0" in cluster.fw

I think the behaviour should be also the same when option is not defined.


- Mail original -
De: "Christian Ebner" 
À: "pve-devel" , "aderumier" 
Envoyé: Mercredi 15 Mai 2019 17:03:13
Objet: Re: [pve-devel] pve-firewall: default loglevel behaviour change

Hi Alexandre, 

you are right, logging the packets passing the firewall bridge seems a bit 
overkill and redundant. 

Will send a patch to fix this. 

> On May 15, 2019 at 4:08 PM Alexandre DERUMIER  wrote: 
> 
> 
> Hi, 
> 
> since this commit 
> 
> 
> https://git.proxmox.com/?p=pve-firewall.git;a=blobdiff;f=src/PVE/Firewall.pm;h=ccc5d7ffb62f1ef4c4d59e363192f512c88742c9;hp=6ac303831a62f630d437ef0e0508decd2b72f5ac;hb=3489f8a2b9597201fe8e42fa5832507f96169619;hpb=33efd363ab32d3f8f6a9c49e481cb62a8da40b5d
>  
> 
> @@ -2341,10 +2348,10 @@ sub generate_tap_rules_direction { 
> # plug the tap chain to bridge chain 
> if ($direction eq 'IN') { 
> ruleset_addrule($ruleset, "PVEFW-FWBR-IN", 
> - "-m physdev --physdev-is-bridged --physdev-out $iface", "-j $tapchain"); 
> + "-m physdev --physdev-is-bridged --physdev-out $iface", "-j $tapchain", 
> $loglevel, 'FWBR-IN: ', $vmid); 
> } else { 
> ruleset_addrule($ruleset, "PVEFW-FWBR-OUT", 
> - "-m physdev --physdev-is-bridged --physdev-in $iface", "-j $tapchain"); 
> + "-m physdev --physdev-is-bridged --physdev-in $iface", "-j $tapchain", 
> $loglevel, 'FWBR-OUT: ', $vmid); 
> } 
> } 
> 
> 
> 
> The default loglevel of a vm is logging all packets comming to the tap chain 
> ex: 
> -A PVEFW-FWBR-IN -m physdev --physdev-out tap135i0 --physdev-is-bridged -m 
> limit --limit 1/sec -j NFLOG --nflog-prefix ":135:6:PVEFW-FWBR-IN: FWBR-IN: " 
> 
> Previously, it was only for the final DROP/REJECT/ACCEPT action 
> 
> 
> Here a example with a connection to port 53, with default action input/output 
> policy = reject, loglevelin:info, and no rule to allow port 
> [OPTIONS] 
> 
> log_level_out: info 
> macfilter: 1 
> policy_out: REJECT 
> enable: 1 
> policy_in: REJECT 
> dhcp: 0 
> log_level_in: info 
> 
> [RULES] 
> 
> 
> 
> 
> 
> 135 6 PVEFW-FWBR-IN 15/May/2019:15:53:23 +0200 FWBR-IN: IN=fwbr135i0 
> OUT=fwbr135i0 PHYSIN=fwln135i0 PHYSOUT=tap135i0 
> MAC=56:af:a2:0d:53:9b:58:49:3b:80:fb:24:08:00 SRC=10.11.53.33 DST=10.3.95.29 
> LEN=283 TOS=0x00 PREC=0x00 TTL=127 ID=20498 PROTO=UDP SPT=54689 DPT=389 
> LEN=263 
> 135 6 tap135i0-IN 15/May/2019:15:53:23 +0200 policy REJECT: IN=fwbr135i0 
> OUT=fwbr135i0 PHYSIN=fwln135i0 PHYSOUT=tap135i0 
> MAC=56:af:a2:0d:53:9b:58:49:3b:80:fb:24:08:00 SRC=10.11.53.33 DST=10.3.95.29 
> LEN=283 TOS=0x00 PREC=0x00 TTL=127 ID=20498 PROTO=UDP SPT=54689 DPT=389 
> LEN=263 
> 
> 
> As you see, it's logged twice. (once when coming to tap chain, one when final 
> action is reject) 
> 
> Same if you make a rule in the vm, activating log on the rule, it's loggued 
> twice. 
> 
> I would like to be able to remove this logging in FWBR-IN, and only log on 
> rules or default input/output policy action. 
> But currently, if I disable the loglevel=nolog, it's disabling too the log 
> for default action. 
> 
> 
> I'm not sure, but do we really need this log on FWBR-IN ? (I mean, we don't 
> have any info if it's drop/accept/reject, so it's pretty useless, we only now 
> that a packet is coming). 
> Or maybe could we have a different loglevel option for default input/output 
> policy ? 
> 
> ___ 
> pve-devel mailing list 
> pve-devel@pve.proxmox.com 
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH container 1/2] fix #1451: add more mount options for containers

2019-05-15 Thread Oguz Bektas

added atime-related options (noatime, nodiratime, strictatime,
relatime), and some other security-related mount options (noexec,
nosuid, nobarrier, nodev) for container mountpoints (and rootfs).

Signed-off-by: Oguz Bektas 
---
 src/PVE/LXC.pm| 19 +--
 src/PVE/LXC/Config.pm | 26 ++
 2 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/src/PVE/LXC.pm b/src/PVE/LXC.pm
index 62b6b8c..81c 100644
--- a/src/PVE/LXC.pm
+++ b/src/PVE/LXC.pm
@@ -1415,11 +1415,26 @@ sub mountpoint_mount {
 
 die "unknown snapshot path for '$volid'" if !$storage && 
defined($snapname);
 
-my $optstring = '';
+my @mount_options = qw(nosuid noexec nodev nobarrier);
+my $optlist = [];
+foreach my $opt (@mount_options) {
+   if (defined $mountpoint->{$opt}) {
+   push @$optlist, $opt;
+   }
+}
+
 my $acl = $mountpoint->{acl};
 if (defined($acl)) {
-   $optstring .= ($acl ? 'acl' : 'noacl');
+   push @$optlist, ($acl ? 'acl' : 'noacl');
 }
+
+my $atime = $mountpoint->{atime};
+if (defined($atime)) {
+   push @$optlist, $atime;
+}
+
+my $optstring = '';
+$optstring = join(',', @$optlist);
 my $readonly = $mountpoint->{ro};
 
 my @extra_opts;
diff --git a/src/PVE/LXC/Config.pm b/src/PVE/LXC/Config.pm
index 8dcd73c..d796a45 100644
--- a/src/PVE/LXC/Config.pm
+++ b/src/PVE/LXC/Config.pm
@@ -236,6 +236,32 @@ my $rootfs_desc = {
description => 'Explicitly enable or disable ACL support.',
optional => 1,
 },
+atime => {
+   type => 'string',
+   description => 'Option for atime',
+   optional => 1,
+   enum => [qw(noatime nodiratime relatime strictatime)],
+},
+nosuid => {
+   type => 'boolean',
+   description => 'Disable suid.',
+   optional => 1,
+},
+noexec => {
+   type => 'boolean',
+   description => 'Disable ability to execute.',
+   optional => 1,
+},
+nodev => {
+   type => 'boolean',
+   description => 'Disable devices.',
+   optional => 1,
+},
+nobarrier => {
+   type => 'boolean',
+   description => 'Disable write barriers.',
+   optional => 1,
+},
 ro => {
type => 'boolean',
description => 'Read-only mount point',
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH 0/2] fix #1451: extra mount options for cts

2019-05-15 Thread Oguz Bektas

pve-container:

Oguz Bektas (1):
  fix #1451: add more mount options for containers

 src/PVE/LXC.pm| 19 +--
 src/PVE/LXC/Config.pm | 26 ++
 2 files changed, 43 insertions(+), 2 deletions(-)

pve-manager:

Oguz Bektas (1):
  fix #1451: add more mount options to containers in gui

  www/manager6/lxc/MPEdit.js | 75 --
  1 file changed, 59 insertions(+), 16 deletions(-)

-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 2/2] fix #1451: add more mount options to containers in gui

2019-05-15 Thread Oguz Bektas

adds atime related options (noatime, nodiratime, relatime,
strictatime) and other extra mount options for performance (nobarrier)  and
security (noexec, nosuid, nodev).

Signed-off-by: Oguz Bektas 
---

i tried to group the options by security or performance relation. the
placement is open to change, if we can find a better way to put all
this on the mp create/edit window.

 www/manager6/lxc/MPEdit.js | 75 --
 1 file changed, 59 insertions(+), 16 deletions(-)

diff --git a/www/manager6/lxc/MPEdit.js b/www/manager6/lxc/MPEdit.js
index c7c3870a..3416d7eb 100644
--- a/www/manager6/lxc/MPEdit.js
+++ b/www/manager6/lxc/MPEdit.js
@@ -233,20 +233,6 @@ Ext.define('PVE.lxc.MountPointInputPanel', {
 advancedColumn1: [
{
xtype: 'proxmoxcheckbox',
-   name: 'quota',
-   defaultValue: 0,
-   bind: {
-   disabled: '{!quota}'
-   },
-   fieldLabel: gettext('Enable quota'),
-   listeners: {
-   disable: function() {
-   this.reset();
-   }
-   }
-   },
-   {
-   xtype: 'proxmoxcheckbox',
name: 'ro',
defaultValue: 0,
bind: {
@@ -254,7 +240,27 @@ Ext.define('PVE.lxc.MountPointInputPanel', {
disabled: '{isRoot}'
},
fieldLabel: gettext('Read-only')
-   }
+   },
+   {
+   xtype: 'proxmoxcheckbox',
+   name: 'noexec',
+   defaultValue: 0,
+   fieldLabel: gettext('noexec')
+   },
+   {
+   xtype: 'proxmoxcheckbox',
+   name: 'nodev',
+   defaultValue: 0,
+   fieldLabel: gettext('nodev')
+   },
+   {
+   xtype: 'proxmoxcheckbox',
+   name: 'nosuid',
+   defaultValue: 0,
+   fieldLabel: gettext('nosuid')
+   },
+
+
 ],
 
 advancedColumn2: [
@@ -275,11 +281,48 @@ Ext.define('PVE.lxc.MountPointInputPanel', {
allowBlank: true
},
{
+   xtype: 'proxmoxKVComboBox',
+   name: 'atime',
+   fieldLabel: 'atime',
+   deleteEmpty: false,
+   comboItems: [
+   ['__default__', Proxmox.Utils.defaultText],
+   ['noatime', 'noatime'],
+   ['nodiratime', 'nodiratime'],
+   ['relatime', 'relatime'],
+   ['strictatime', 'strictatime'],
+   ],
+   value: '__default__',
+   allowBlank: true
+   },
+   {
+   xtype: 'proxmoxcheckbox',
+   name: 'quota',
+   defaultValue: 0,
+   bind: {
+   disabled: '{!quota}'
+   },
+   fieldLabel: gettext('Enable quota'),
+   listeners: {
+   disable: function() {
+   this.reset();
+   }
+   }
+   },
+   {
xtype: 'proxmoxcheckbox',
inputValue: '0', // reverses the logic
name: 'replicate',
fieldLabel: gettext('Skip replication')
-   }
+   },
+   {
+   xtype: 'proxmoxcheckbox',
+   name: 'nobarrier',
+   defaultValue: 0,
+   fieldLabel: gettext('nobarrier')
+   },
+
+
 ]
 });
 
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage 0/1] Fix for ZFS over iSCSI on Solaris 11

2019-05-15 Thread David Hedbor

I'm testing out ZFS over iSCSI with a Oracle Solaris 11.3 server
(Sparc based specifically). It almost works - what doesn't work is
when I try to act on the disk in any way (such as start
KVM). Specifically it is unable to find the LUN.

Looking at the source, it seems like Proxmox runs: 

stmfadm list-view -l 

to get the LUN. However on Solaris 11.3 (at least my installation),
this returns the following:

View Entry: 0
Host group   : All
Target Group : All
LUN  : Auto

There's no LUN listed here. You can find ones assigned if you use the -v flag:

$ sudo stmfadm list-view -l 600144f2a8fee54fcb4fb8583c6d11dd -v 
View Entry: 0
Host group   : All
Target Group : All
LUN  : Auto
Lun  : 1for iqn.1993-08.org.debian:01:e06b5e9eb438
Lun  : 1for iqn.1993-08.org.debian:01:7c967782bc4
Lun  : 1for iqn.1993-08.org.debian:01:dc213f1eceab
Lun  : 1for iqn.1993-08.org.debian:01:8ee9e0466ec5

I modified Comstar.pm to handle this, which I assume is a change from
Solaris 10 (OpenSolaris) to Solaris 11. I'm not happy with the copy of
get_initiator_name - it's also used by QemuServer.pm. I'm not familar
enough to know where to best place it. 

David Hedbor (1):
  storage: iscsi: Fix LunCmd for Comstar on Oracle Solaris 11

 PVE/Storage/LunCmd/Comstar.pm | 31 ++-
 1 file changed, 26 insertions(+), 5 deletions(-)

-- 
2.11.0

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage 1/1] storage: iscsi: Fix LunCmd for Comstar on Oracle Solaris 11

2019-05-15 Thread David Hedbor

Fix #2213: ZFS over iSCSI doesn't work with Solaris 11.3

Signed-off-by: David Hedbor 
---
 PVE/Storage/LunCmd/Comstar.pm | 31 ++-
 1 file changed, 26 insertions(+), 5 deletions(-)

diff --git a/PVE/Storage/LunCmd/Comstar.pm b/PVE/Storage/LunCmd/Comstar.pm
index 527e4ba..ca5938e 100644
--- a/PVE/Storage/LunCmd/Comstar.pm
+++ b/PVE/Storage/LunCmd/Comstar.pm
@@ -1,4 +1,5 @@
 package PVE::Storage::LunCmd::Comstar;
+use PVE::SafeSyslog;
 
 use strict;
 use warnings;
@@ -31,13 +32,26 @@ my $get_lun_cmd_map = sub {
 return $cmdmap->{$method};
 };
 
+sub get_initiator_name {
+my $initiator;
+
+my $fh = IO::File->new('/etc/iscsi/initiatorname.iscsi') || return undef;
+while (defined(my $line = <$fh>)) {
+   next if $line !~ m/^\s*InitiatorName\s*=\s*([\.\-:\w]+)/;
+   $initiator = $1;
+   last;
+}
+$fh->close();
+
+return $initiator;
+}
+
 sub get_base {
 return '/dev/zvol/rdsk';
 }
 
 sub run_lun_command {
 my ($scfg, $timeout, $method, @params) = @_;
-
 my $msg = '';
 my $luncmd;
 my $target;
@@ -62,7 +76,7 @@ sub run_lun_command {
 } elsif ($method eq 'modify_lu') {
 @params = ('-s', @params);
 } elsif ($method eq 'list_view') {
-@params = ('-l', @params);
+@params = ('-l',  @params, '-v');
 } elsif ($method eq 'list_lu') {
 $guid = $params[0];
 @params = undef;
@@ -90,8 +104,16 @@ sub run_lun_command {
 if ($method eq 'list_view') {
 my @lines = split /\n/, $msg;
 $msg = undef;
-foreach my $line (@lines) {
-if ($line =~ /^\s*LUN\s*:\s*(\d+)$/) {
+   my $initiator = undef;
+   foreach my $line (@lines) {
+   if(defined $initiator) {
+   if ($line =~ /^\s*Lun\s*:\s*(\d+)\s*for\s*\Q${initiator}/) {
+   $msg = $1;
+   last;
+   }
+   } elsif ($line =~ /^\s*LUN\s*:\s*Auto$/) {
+   $initiator = get_initiator_name();
+   } elsif ($line =~ /^\s*LUN\s*:\s*(\d+)$/) {
 $msg = $1;
 last;
 }
@@ -112,4 +134,3 @@ sub run_lun_command {
 
 return $msg;
 }
-
-- 
2.11.0

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH firewall] Remove redundant logging of packets passing the tap chain.

2019-05-15 Thread Christian Ebner

Incomming and outgoing packets passing the firewall bridge were unneccessarily
logged, leading to double entries.
The first log entry occurred when passing the bridge, the second when the 
packets
fate was decided (ACCEPT/DROP/REJECT).

Signed-off-by: Christian Ebner 
---
 src/PVE/Firewall.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/PVE/Firewall.pm b/src/PVE/Firewall.pm
index f84e6d9..abcc1e8 100644
--- a/src/PVE/Firewall.pm
+++ b/src/PVE/Firewall.pm
@@ -2387,10 +2387,10 @@ sub generate_tap_rules_direction {
 # plug the tap chain to bridge chain
 if ($direction eq 'IN') {
ruleset_addrule($ruleset, "PVEFW-FWBR-IN",
-   "-m physdev --physdev-is-bridged --physdev-out $iface", 
"-j $tapchain", $loglevel, 'FWBR-IN: ', $vmid);
+   "-m physdev --physdev-is-bridged --physdev-out $iface", 
"-j $tapchain");
 } else {
ruleset_addrule($ruleset, "PVEFW-FWBR-OUT",
-   "-m physdev --physdev-is-bridged --physdev-in $iface", 
"-j $tapchain", $loglevel, 'FWBR-OUT: ', $vmid);
+   "-m physdev --physdev-is-bridged --physdev-in $iface", 
"-j $tapchain");
 }
 }
 
-- 
2.11.0

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] pve-firewall: default loglevel behaviour change

2019-05-15 Thread Christian Ebner

Hi Alexandre,

you are right, logging the packets passing the firewall bridge seems a bit 
overkill and redundant.

Will send a patch to fix this.

> On May 15, 2019 at 4:08 PM Alexandre DERUMIER  wrote:
> 
> 
> Hi,
> 
> since this commit
> 
> 
> https://git.proxmox.com/?p=pve-firewall.git;a=blobdiff;f=src/PVE/Firewall.pm;h=ccc5d7ffb62f1ef4c4d59e363192f512c88742c9;hp=6ac303831a62f630d437ef0e0508decd2b72f5ac;hb=3489f8a2b9597201fe8e42fa5832507f96169619;hpb=33efd363ab32d3f8f6a9c49e481cb62a8da40b5d
> 
> @@ -2341,10 +2348,10 @@ sub generate_tap_rules_direction {
>  # plug the tap chain to bridge chain
>  if ($direction eq 'IN') {
> ruleset_addrule($ruleset, "PVEFW-FWBR-IN",
> -   "-m physdev --physdev-is-bridged --physdev-out 
> $iface", "-j $tapchain");
> +   "-m physdev --physdev-is-bridged --physdev-out 
> $iface", "-j $tapchain", $loglevel, 'FWBR-IN: ', $vmid);
>  } else {
> ruleset_addrule($ruleset, "PVEFW-FWBR-OUT",
> -   "-m physdev --physdev-is-bridged --physdev-in 
> $iface", "-j $tapchain");
> +   "-m physdev --physdev-is-bridged --physdev-in 
> $iface", "-j $tapchain", $loglevel, 'FWBR-OUT: ', $vmid);
>  }
>  }
> 
> 
> 
> The default loglevel of a vm is logging all packets comming to the tap chain
> ex:
> -A PVEFW-FWBR-IN -m physdev --physdev-out tap135i0 --physdev-is-bridged -m 
> limit --limit 1/sec -j NFLOG --nflog-prefix  ":135:6:PVEFW-FWBR-IN: FWBR-IN: "
> 
> Previously, it was only for the final DROP/REJECT/ACCEPT action
> 
> 
> Here a example with a connection to port 53, with default action input/output 
> policy = reject, loglevelin:info, and no rule to allow port
> [OPTIONS]
> 
> log_level_out: info
> macfilter: 1
> policy_out: REJECT
> enable: 1
> policy_in: REJECT
> dhcp: 0
> log_level_in: info
> 
> [RULES]
> 
> 
> 
> 
> 
> 135 6 PVEFW-FWBR-IN 15/May/2019:15:53:23 +0200 FWBR-IN: IN=fwbr135i0 
> OUT=fwbr135i0 PHYSIN=fwln135i0 PHYSOUT=tap135i0 
> MAC=56:af:a2:0d:53:9b:58:49:3b:80:fb:24:08:00 SRC=10.11.53.33 DST=10.3.95.29 
> LEN=283 TOS=0x00 PREC=0x00 TTL=127 ID=20498 PROTO=UDP SPT=54689 DPT=389 
> LEN=263
> 135 6 tap135i0-IN 15/May/2019:15:53:23 +0200 policy REJECT: IN=fwbr135i0 
> OUT=fwbr135i0 PHYSIN=fwln135i0 PHYSOUT=tap135i0 
> MAC=56:af:a2:0d:53:9b:58:49:3b:80:fb:24:08:00 SRC=10.11.53.33 DST=10.3.95.29 
> LEN=283 TOS=0x00 PREC=0x00 TTL=127 ID=20498 PROTO=UDP SPT=54689 DPT=389 
> LEN=263
> 
> 
> As you see, it's logged twice. (once when coming to tap chain, one when final 
> action is reject)
> 
> Same if you make a rule in the vm, activating log on the rule, it's loggued 
> twice.
> 
> I would like to be able to remove this logging in FWBR-IN, and only log on 
> rules or default input/output policy action.
> But currently, if I disable the loglevel=nolog, it's disabling too the log 
> for default action.
> 
> 
> I'm not sure, but do we really need this log on FWBR-IN ? (I mean, we don't 
> have any info if it's drop/accept/reject, so it's pretty useless, we only now 
> that a packet is coming).
> Or maybe could we have a different loglevel option for default input/output 
> policy ?
> 
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH qemu-server] fix ceph volume with krbd not mapped for file_size_info

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 12:53 PM, Mira Limbeck wrote:
> file_size_info can't find the file if it is not mapped beforehand when
> using krbd and returns 0 which we interpret as the disk not existing.
> vdisk_alloc is called but the disk already exists. with this patch we call
> activate_volume before trying file_size_info. on ceph with krbd it maps the
> volume.
> if the disk does not exist and is created with vdisk_alloc we still require an
> additional call to activate_volume for the new disk.
> 
> Signed-off-by: Mira Limbeck 
> ---
> tested with ceph, ceph + krbd, dir, lvm, lvm-thin, zfs
> 
>  PVE/QemuServer/Cloudinit.pm | 7 +--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 

applied, but rewrote commit message, as this is not a "map issue with ceph"
but may be also needed for other storage, like LVM, and not the map but as 
your code does, the activate is the right thing, like Dominik explained
off-list.

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager] fix #2185: add option to change nfs version on gui

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 3:21 PM, Oguz Bektas wrote:
> this enables us to specify an nfs version while editing/creating an nfs
> mount. it used to default to vers=3 without the ability to change it in
> gui. now it supports: 3, 4, 4.1 and 4.2
> 
> it should also be possible to add further options in the future (rsize,
> wsize, timeo, etc.) on this screen.
> 
> Co-Authored-by: Dominik Csapak 
> Signed-off-by: Oguz Bektas 
> ---
> 
> this patch depends on the patches from dominik[0] and thomas[1] in
> pve-common and pve-storage and shouldn't be applied without them.
> 
> [0]: 
> https://git.proxmox.com/?p=pve-common.git;a=commit;h=d0ba18e17e96e9a2db54d050c4bd86b6d2d2dd03
> [1]: 
> https://git.proxmox.com/?p=pve-storage.git;a=commit;h=4273e3ace94a22c39d98385d9dfb72249eac09aa
> 
> 

applied, thanks!

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] pve-firewall: default loglevel behaviour change

2019-05-15 Thread Alexandre DERUMIER

Hi,

since this commit


https://git.proxmox.com/?p=pve-firewall.git;a=blobdiff;f=src/PVE/Firewall.pm;h=ccc5d7ffb62f1ef4c4d59e363192f512c88742c9;hp=6ac303831a62f630d437ef0e0508decd2b72f5ac;hb=3489f8a2b9597201fe8e42fa5832507f96169619;hpb=33efd363ab32d3f8f6a9c49e481cb62a8da40b5d

@@ -2341,10 +2348,10 @@ sub generate_tap_rules_direction {
 # plug the tap chain to bridge chain
 if ($direction eq 'IN') {
ruleset_addrule($ruleset, "PVEFW-FWBR-IN",
-   "-m physdev --physdev-is-bridged --physdev-out $iface", 
"-j $tapchain");
+   "-m physdev --physdev-is-bridged --physdev-out $iface", 
"-j $tapchain", $loglevel, 'FWBR-IN: ', $vmid);
 } else {
ruleset_addrule($ruleset, "PVEFW-FWBR-OUT",
-   "-m physdev --physdev-is-bridged --physdev-in $iface", 
"-j $tapchain");
+   "-m physdev --physdev-is-bridged --physdev-in $iface", 
"-j $tapchain", $loglevel, 'FWBR-OUT: ', $vmid);
 }
 }



The default loglevel of a vm is logging all packets comming to the tap chain
ex:
-A PVEFW-FWBR-IN -m physdev --physdev-out tap135i0 --physdev-is-bridged -m 
limit --limit 1/sec -j NFLOG --nflog-prefix  ":135:6:PVEFW-FWBR-IN: FWBR-IN: "

Previously, it was only for the final DROP/REJECT/ACCEPT action


Here a example with a connection to port 53, with default action input/output 
policy = reject, loglevelin:info, and no rule to allow port
[OPTIONS]

log_level_out: info
macfilter: 1
policy_out: REJECT
enable: 1
policy_in: REJECT
dhcp: 0
log_level_in: info

[RULES]





135 6 PVEFW-FWBR-IN 15/May/2019:15:53:23 +0200 FWBR-IN: IN=fwbr135i0 
OUT=fwbr135i0 PHYSIN=fwln135i0 PHYSOUT=tap135i0 
MAC=56:af:a2:0d:53:9b:58:49:3b:80:fb:24:08:00 SRC=10.11.53.33 DST=10.3.95.29 
LEN=283 TOS=0x00 PREC=0x00 TTL=127 ID=20498 PROTO=UDP SPT=54689 DPT=389 LEN=263
135 6 tap135i0-IN 15/May/2019:15:53:23 +0200 policy REJECT: IN=fwbr135i0 
OUT=fwbr135i0 PHYSIN=fwln135i0 PHYSOUT=tap135i0 
MAC=56:af:a2:0d:53:9b:58:49:3b:80:fb:24:08:00 SRC=10.11.53.33 DST=10.3.95.29 
LEN=283 TOS=0x00 PREC=0x00 TTL=127 ID=20498 PROTO=UDP SPT=54689 DPT=389 LEN=263


As you see, it's logged twice. (once when coming to tap chain, one when final 
action is reject)

Same if you make a rule in the vm, activating log on the rule, it's loggued 
twice.

I would like to be able to remove this logging in FWBR-IN, and only log on 
rules or default input/output policy action.
But currently, if I disable the loglevel=nolog, it's disabling too the log for 
default action.


I'm not sure, but do we really need this log on FWBR-IN ? (I mean, we don't 
have any info if it's drop/accept/reject, so it's pretty useless, we only now 
that a packet is coming).
Or maybe could we have a different loglevel option for default input/output 
policy ?

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] fix #2185: add option to change nfs version on gui

2019-05-15 Thread Oguz Bektas

hi,

should have been v3, forgot to add changelog as well..

v2 -> v3:
* only send values["delete"] when editing an existing storage
i.e. if !me.isCreate
* add deleteEmpty: false in column


On Wed, May 15, 2019 at 03:21:56PM +0200, Oguz Bektas wrote:
> this enables us to specify an nfs version while editing/creating an nfs
> mount. it used to default to vers=3 without the ability to change it in
> gui. now it supports: 3, 4, 4.1 and 4.2
> 
> it should also be possible to add further options in the future (rsize,
> wsize, timeo, etc.) on this screen.
> 
> Co-Authored-by: Dominik Csapak 
> Signed-off-by: Oguz Bektas 
> ---
> 
> this patch depends on the patches from dominik[0] and thomas[1] in
> pve-common and pve-storage and shouldn't be applied without them.
> 
> [0]: 
> https://git.proxmox.com/?p=pve-common.git;a=commit;h=d0ba18e17e96e9a2db54d050c4bd86b6d2d2dd03
> [1]: 
> https://git.proxmox.com/?p=pve-storage.git;a=commit;h=4273e3ace94a22c39d98385d9dfb72249eac09aa
> 
> 
>  www/manager6/storage/NFSEdit.js | 56 
> ++---
>  1 file changed, 52 insertions(+), 4 deletions(-)
> 
> diff --git a/www/manager6/storage/NFSEdit.js b/www/manager6/storage/NFSEdit.js
> index 9eaa8bc5..bc512e3b 100644
> --- a/www/manager6/storage/NFSEdit.js
> +++ b/www/manager6/storage/NFSEdit.js
> @@ -62,18 +62,49 @@ Ext.define('PVE.storage.NFSInputPanel', {
>  
>  onlineHelp: 'storage_nfs',
>  
> +options : [],
> +
>  onGetValues: function(values) {
>   var me = this;
>  
> - if (me.isCreate) {
> - // hack: for now we always create nvf v3
> - // fixme: make this configurable
> - values.options = 'vers=3';
> + var i;
> + var res = [];
> + for (i = 0; i < me.options.length; i++) {
> + var item = me.options[i];
> + if (!item.match(/^vers=(.*)$/)) {
> + res.push(item);
> + }
> + }
> + if (values.nfsversion && values.nfsversion !== '__default__') {
> + res.push('vers=' + values.nfsversion);
> + }
> + delete values.nfsversion;
> + values.options = res.join(',');
> + if (values.options === '') {
> + delete values.options;
> + if (!me.isCreate) {
> + values["delete"] = "options";
> + }
>   }
>  
>   return me.callParent([values]);
>  },
>  
> +setValues: function(values) {
> + var me = this;
> + if (values.options) {
> + var res = values.options;
> + me.options = values.options.split(',');
> + me.options.forEach(function(item) {
> + var match;
> + if (match = item.match(/^vers=(.*)$/)) {
> + values.nfsversion = match[1];
> + }
> + });
> + }
> + return me.callParent([values]);
> +},
> +
>  initComponent : function() {
>   var me = this;
>  
> @@ -126,6 +157,23 @@ Ext.define('PVE.storage.NFSInputPanel', {
>   }
>   ];
>  
> + me.advancedColumn1 = [
> + {
> + xtype: 'proxmoxKVComboBox',
> + fieldLabel: gettext('NFS Version'),
> + name: 'nfsversion',
> + value: '__default__',
> + deleteEmpty: false,
> + comboItems: [
> + ['__default__', Proxmox.Utils.defaultText],
> + ['3', '3'],
> + ['4', '4'],
> + ['4.1', '4.1'],
> + ['4.2', '4.2']
> + ],
> + }
> + ];
> +
>   me.callParent();
>  }
>  });
> -- 
> 2.11.0
> 
> 

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager] fix #2185: add option to change nfs version on gui

2019-05-15 Thread Oguz Bektas

this enables us to specify an nfs version while editing/creating an nfs
mount. it used to default to vers=3 without the ability to change it in
gui. now it supports: 3, 4, 4.1 and 4.2

it should also be possible to add further options in the future (rsize,
wsize, timeo, etc.) on this screen.

Co-Authored-by: Dominik Csapak 
Signed-off-by: Oguz Bektas 
---

this patch depends on the patches from dominik[0] and thomas[1] in
pve-common and pve-storage and shouldn't be applied without them.

[0]: 
https://git.proxmox.com/?p=pve-common.git;a=commit;h=d0ba18e17e96e9a2db54d050c4bd86b6d2d2dd03
[1]: 
https://git.proxmox.com/?p=pve-storage.git;a=commit;h=4273e3ace94a22c39d98385d9dfb72249eac09aa


 www/manager6/storage/NFSEdit.js | 56 ++---
 1 file changed, 52 insertions(+), 4 deletions(-)

diff --git a/www/manager6/storage/NFSEdit.js b/www/manager6/storage/NFSEdit.js
index 9eaa8bc5..bc512e3b 100644
--- a/www/manager6/storage/NFSEdit.js
+++ b/www/manager6/storage/NFSEdit.js
@@ -62,18 +62,49 @@ Ext.define('PVE.storage.NFSInputPanel', {
 
 onlineHelp: 'storage_nfs',
 
+options : [],
+
 onGetValues: function(values) {
var me = this;
 
-   if (me.isCreate) {
-   // hack: for now we always create nvf v3
-   // fixme: make this configurable
-   values.options = 'vers=3';
+   var i;
+   var res = [];
+   for (i = 0; i < me.options.length; i++) {
+   var item = me.options[i];
+   if (!item.match(/^vers=(.*)$/)) {
+   res.push(item);
+   }
+   }
+   if (values.nfsversion && values.nfsversion !== '__default__') {
+   res.push('vers=' + values.nfsversion);
+   }
+   delete values.nfsversion;
+   values.options = res.join(',');
+   if (values.options === '') {
+   delete values.options;
+   if (!me.isCreate) {
+   values["delete"] = "options";
+   }
}
 
return me.callParent([values]);
 },
 
+setValues: function(values) {
+   var me = this;
+   if (values.options) {
+   var res = values.options;
+   me.options = values.options.split(',');
+   me.options.forEach(function(item) {
+   var match;
+   if (match = item.match(/^vers=(.*)$/)) {
+   values.nfsversion = match[1];
+   }
+   });
+   }
+   return me.callParent([values]);
+},
+
 initComponent : function() {
var me = this;
 
@@ -126,6 +157,23 @@ Ext.define('PVE.storage.NFSInputPanel', {
}
];
 
+   me.advancedColumn1 = [
+   {
+   xtype: 'proxmoxKVComboBox',
+   fieldLabel: gettext('NFS Version'),
+   name: 'nfsversion',
+   value: '__default__',
+   deleteEmpty: false,
+   comboItems: [
+   ['__default__', Proxmox.Utils.defaultText],
+   ['3', '3'],
+   ['4', '4'],
+   ['4.1', '4.1'],
+   ['4.2', '4.2']
+   ],
+   }
+   ];
+
me.callParent();
 }
 });
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH common] JSONSchema: sort keys when dumping config

2019-05-15 Thread Thomas Lamprecht

On 4/17/19 8:56 AM, Thomas Lamprecht wrote:
> Signed-off-by: Thomas Lamprecht 
> ---
>  src/PVE/JSONSchema.pm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/PVE/JSONSchema.pm b/src/PVE/JSONSchema.pm
> index 51ef3ab..f0192c2 100644
> --- a/src/PVE/JSONSchema.pm
> +++ b/src/PVE/JSONSchema.pm
> @@ -1590,7 +1590,7 @@ sub dump_config {
>  
>  my $data = '';
>  
> -foreach my $k (keys %$cfg) {
> +foreach my $k (sort keys %$cfg) {
>   $data .= "$k: $cfg->{$k}\n";
>  }
>  
> 

applied

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager v2 2/2] gui: node: change syslog to JournalView and journal api

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 11:15 AM, Dominik Csapak wrote:
> Signed-off-by: Dominik Csapak 
> ---
>  www/manager6/node/Config.js | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/www/manager6/node/Config.js b/www/manager6/node/Config.js
> index 831d2e02..700ca611 100644
> --- a/www/manager6/node/Config.js
> +++ b/www/manager6/node/Config.js
> @@ -217,9 +217,8 @@ Ext.define('PVE.node.Config', {
>   groups: ['services'],
>   disabled: !caps.nodes['Sys.Syslog'],
>   itemId: 'syslog',
> - xtype: 'proxmoxLogView',
> - url: "/api2/extjs/nodes/" + nodename + "/syslog",
> - log_select_timespan: 1
> + xtype: 'proxmoxJournalView',
> + url: "/api2/extjs/nodes/" + nodename + "/journal"
>   });
>  
>   if (caps.nodes['Sys.Modify']) {
> 

applied, thanks!

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager v2 1/2] node: add journal api

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 11:15 AM, Dominik Csapak wrote:
> this uses the new journalreader instead of journalctl, which is a bit
> faster and can read from/to cursor and returns a start/end cursor
> 
> also you can give an unix epoch as time parameters
> 
> Signed-off-by: Dominik Csapak 
> ---
>  PVE/API2/Nodes.pm | 71 
> +++
>  debian/control|  1 +
>  2 files changed, 72 insertions(+)
> 

applied, thanks!

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH widget-toolkit v2 1/1] add panel/JournalView

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 11:15 AM, Dominik Csapak wrote:
> similar to LogView, but expects the result from the /node/journal api call,
> which is an array of strings, with start/endcursor instead of
> an array of objects with line numbers
> 
> the new api call also does not accept start line numbers and limit,
> so it is necessary to handle it differently
> 
> Signed-off-by: Dominik Csapak 
> ---
>  Makefile |   1 +
>  panel/JournalView.js | 335 
> +++
>  2 files changed, 336 insertions(+)
>  create mode 100644 panel/JournalView.js
> 


applied, with follow up fixing the enormous whitespace between the datepickers
and it labels..

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server] fix ceph volume with krbd not mapped for file_size_info

2019-05-15 Thread Mira Limbeck

file_size_info can't find the file if it is not mapped beforehand when
using krbd and returns 0 which we interpret as the disk not existing.
vdisk_alloc is called but the disk already exists. with this patch we call
activate_volume before trying file_size_info. on ceph with krbd it maps the
volume.
if the disk does not exist and is created with vdisk_alloc we still require an
additional call to activate_volume for the new disk.

Signed-off-by: Mira Limbeck 
---
tested with ceph, ceph + krbd, dir, lvm, lvm-thin, zfs

 PVE/QemuServer/Cloudinit.pm | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/PVE/QemuServer/Cloudinit.pm b/PVE/QemuServer/Cloudinit.pm
index f46f7fd..fbd71ec 100644
--- a/PVE/QemuServer/Cloudinit.pm
+++ b/PVE/QemuServer/Cloudinit.pm
@@ -32,6 +32,10 @@ sub commit_cloudinit_disk {
 my $scfg = PVE::Storage::storage_config($storecfg, $storeid);
 my $format = PVE::QemuServer::qemu_img_format($scfg, $volname);
 
+# required before file_size_info for ceph + krbd as it gets mapped too 
late otherwise
+my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+eval { $plugin->activate_volume($storeid, $scfg, $volname) };
+
 my $size = eval { PVE::Storage::file_size_info($iso_path) };
 if ($size <= 0) {
$volname =~ m/(vm-$vmid-cloudinit(.\Q$format\E)?)/;
@@ -39,10 +43,9 @@ sub commit_cloudinit_disk {
$size = 4 * 1024;
PVE::Storage::vdisk_alloc($storecfg, $storeid, $vmid, $format, $name, 
$size);
$size *= 1024; # vdisk alloc takes KB, qemu-img dd's osize takes byte
+   $plugin->activate_volume($storeid, $scfg, $volname);
 }
 
-my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
-$plugin->activate_volume($storeid, $scfg, $volname);
 
 eval {
run_command([['genisoimage', '-R', '-V', $label, $path],
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH common 1/1] Fix #545: Allow descriptive names for network bridges and bonds

2019-05-15 Thread Dominic Jäger

Allowing more descriptive names for network bridges and bonds makes
their identification easier.

Signed-off-by: Dominic Jäger 
---
 src/PVE/INotify.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/PVE/INotify.pm b/src/PVE/INotify.pm
index b660041..75e5010 100644
--- a/src/PVE/INotify.pm
+++ b/src/PVE/INotify.pm
@@ -1027,7 +1027,7 @@ sub __read_etc_network_interfaces {
 
 foreach my $iface (keys %$ifaces) {
my $d = $ifaces->{$iface};
-   if ($iface =~ m/^bond\d+$/) {
+   if ($iface =~ m/^bond\w+$/) {
if (!$d->{ovs_type}) {
$d->{type} = 'bond';
} elsif ($d->{ovs_type} eq 'OVSBond') {
@@ -1049,7 +1049,7 @@ sub __read_etc_network_interfaces {
} else {
$d->{type} = 'unknown';
}
-   } elsif ($iface =~ m/^vmbr\d+$/) {
+   } elsif ($iface =~ m/^vmbr\w+$/) {
if (!$d->{ovs_type}) {
$d->{type} = 'bridge';
 
-- 
2.11.0

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH docu] Fix #545: Update allowed network device names

2019-05-15 Thread Dominic Jäger

Signed-off-by: Dominic Jäger 
---
 pve-network.adoc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/pve-network.adoc b/pve-network.adoc
index 9e0c439..2a78dd1 100644
--- a/pve-network.adoc
+++ b/pve-network.adoc
@@ -32,12 +32,12 @@ We currently use the following naming conventions for 
device names:
 scheme is used for {pve} hosts which were installed before the 5.0
 release. When upgrading to 5.0, the names are kept as-is.
 
-* Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (`vmbr0` - `vmbr4094`)
+* Bridge names: vmbr[w], where w is a "word" character (`vmbr0`, `vmbr123`, 
`vmbr_test`, ...)
 
-* Bonds: bond[N], where 0 ≤ N (`bond0`, `bond1`, ...)
+* Bonds: bond[w], where w is a "word" character (`bond0`, `bond123`, 
`bond_test`, ...)
 
 * VLANs: Simply add the VLAN number to the device name,
-  separated by a period (`eno1.50`, `bond1.30`)
+  separated by a period (`eno1.50`, `bond1.30`, ...)
 
 This makes it easier to debug networks problems, because the device
 name implies the device type.
-- 
2.11.0

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH pmg-api v2 1/1] node: add journal api

2019-05-15 Thread Dominik Csapak


disregard this for now, copying was not enough

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager v2 1/2] node: add journal api

2019-05-15 Thread Dominik Csapak

this uses the new journalreader instead of journalctl, which is a bit
faster and can read from/to cursor and returns a start/end cursor

also you can give an unix epoch as time parameters

Signed-off-by: Dominik Csapak 
---
 PVE/API2/Nodes.pm | 71 +++
 debian/control|  1 +
 2 files changed, 72 insertions(+)

diff --git a/PVE/API2/Nodes.pm b/PVE/API2/Nodes.pm
index df47be1f..426c026e 100644
--- a/PVE/API2/Nodes.pm
+++ b/PVE/API2/Nodes.pm
@@ -699,6 +699,77 @@ __PACKAGE__->register_method({
return $lines;
 }});
 
+__PACKAGE__->register_method({
+name => 'journal',
+path => 'journal',
+method => 'GET',
+description => "Read Journal",
+proxyto => 'node',
+permissions => {
+   check => ['perm', '/nodes/{node}', [ 'Sys.Syslog' ]],
+},
+protected => 1,
+parameters => {
+   additionalProperties => 0,
+   properties => {
+   node => get_standard_option('pve-node'),
+   since => {
+   type=> 'number',
+   description => "Display all log since this UNIX epoch.",
+   optional => 1,
+   },
+   until => {
+   type=> 'number',
+   description => "Display all log until this UNIX epoch.",
+   optional => 1,
+   },
+   lastentries => {
+   description => "Limit to the last X lines.",
+   type => 'integer',
+   optional => 1,
+   },
+   startcursor => {
+   description => "Start after the given Cursor.",
+   type => 'string',
+   optional => 1,
+   },
+   endcursor => {
+   description => "End before the given Cursor.",
+   type => 'string',
+   optional => 1,
+   },
+   },
+},
+returns => {
+   type => 'array',
+   items => {
+   type => "string",
+   }
+},
+code => sub {
+   my ($param) = @_;
+
+   my $rpcenv = PVE::RPCEnvironment::get();
+   my $user = $rpcenv->get_user();
+
+   my $lines = [];
+
+   my $parser = sub {
+   push @$lines, shift;
+   };
+
+   my $cmd = ["/usr/bin/mini-journalreader"];
+   push @$cmd, '-n', $param->{lastentries} if $param->{lastentries};
+   push @$cmd, '-b', $param->{since} if $param->{since};
+   push @$cmd, '-e', $param->{until} if $param->{until};
+   push @$cmd, '-f', $param->{startcursor} if $param->{startcursor};
+   push @$cmd, '-t', $param->{endcursor} if $param->{endcursor};
+
+   PVE::Tools::run_command($cmd, outfunc => $parser);
+
+   return $lines;
+}});
+
 my $sslcert;
 
 my $shell_cmd_map = {
diff --git a/debian/control b/debian/control
index eb39e1e9..1a200c0a 100644
--- a/debian/control
+++ b/debian/control
@@ -60,6 +60,7 @@ Depends: apt-transport-https,
  pciutils,
  perl (>= 5.10.0-19),
  postfix | mail-transport-agent,
+ proxmox-mini-journalreader,
  proxmox-widget-toolkit (>= 1.0-26),
  pve-cluster (>= 5.0-27),
  pve-container (>= 2.0-21),
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager v2 2/2] gui: node: change syslog to JournalView and journal api

2019-05-15 Thread Dominik Csapak

Signed-off-by: Dominik Csapak 
---
 www/manager6/node/Config.js | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/www/manager6/node/Config.js b/www/manager6/node/Config.js
index 831d2e02..700ca611 100644
--- a/www/manager6/node/Config.js
+++ b/www/manager6/node/Config.js
@@ -217,9 +217,8 @@ Ext.define('PVE.node.Config', {
groups: ['services'],
disabled: !caps.nodes['Sys.Syslog'],
itemId: 'syslog',
-   xtype: 'proxmoxLogView',
-   url: "/api2/extjs/nodes/" + nodename + "/syslog",
-   log_select_timespan: 1
+   xtype: 'proxmoxJournalView',
+   url: "/api2/extjs/nodes/" + nodename + "/journal"
});
 
if (caps.nodes['Sys.Modify']) {
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH widget-toolkit/manager/pmg-api v2] improve syslog tab

2019-05-15 Thread Dominik Csapak

this series aims to improve the nodes syslog tab, by changing
journalctl to a slimmer, slightly faster binary called journalreader

the big improvement is that it returns/accepts cursors, making it
possible to get only the entries we need

the name journalreader is nothing i am particularly fond of, so if
someone has a better idea, please tell :)

also the gui panel (JournalView) is very similar to the already existing
LogView, but i found no neat way to combine the two

another thing is the api:
originally i did not want to have a new api call, but i wanted to get
rid of the weird result structure, but found no good way to keep
backwards compatibility

changes from v1:
* journalreader is already applied (with a different name) so drop that
  and adapt the name
* move the run_command into the api call so we do not update pve-common
  anymore (it is small enough)
* fix last -> lastentries
* add the same api call to pmg-api (i will send the pmg-gui patch later)
* add dependency of pve-manager/pmg-api

proxmox-widget-toolkit:

Dominik Csapak (1):
  add panel/JournalView

 Makefile |   1 +
 panel/JournalView.js | 335 +++
 2 files changed, 336 insertions(+)
 create mode 100644 panel/JournalView.js

pve-manager:

Dominik Csapak (2):
  node: add journal api
  gui: node: change syslog to JournalView and journal api

 PVE/API2/Nodes.pm   | 71 +
 debian/control  |  1 +
 www/manager6/node/Config.js |  5 ++--
 3 files changed, 74 insertions(+), 3 deletions(-)

pmg-api:

Dominik Csapak (1):
  node: add journal api

 PMG/API2/Nodes.pm | 71 +++
 debian/control|  1 +
 2 files changed, 72 insertions(+)

-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH pmg-api v2 1/1] node: add journal api

2019-05-15 Thread Dominik Csapak

this uses the new journalreader instead of journalctl, which is a bit
faster and can read from/to cursor and returns a start/end cursor

also you can give an unix epoch as time parameters

Signed-off-by: Dominik Csapak 
---
 PMG/API2/Nodes.pm | 71 +++
 debian/control|  1 +
 2 files changed, 72 insertions(+)

diff --git a/PMG/API2/Nodes.pm b/PMG/API2/Nodes.pm
index 7734977..f836af8 100644
--- a/PMG/API2/Nodes.pm
+++ b/PMG/API2/Nodes.pm
@@ -260,6 +260,77 @@ __PACKAGE__->register_method({
return $lines;
 }});
 
+__PACKAGE__->register_method({
+name => 'journal',
+path => 'journal',
+method => 'GET',
+description => "Read Journal",
+proxyto => 'node',
+permissions => {
+   check => ['perm', '/nodes/{node}', [ 'Sys.Syslog' ]],
+},
+protected => 1,
+parameters => {
+   additionalProperties => 0,
+   properties => {
+   node => get_standard_option('pve-node'),
+   since => {
+   type=> 'number',
+   description => "Display all log since this UNIX epoch.",
+   optional => 1,
+   },
+   until => {
+   type=> 'number',
+   description => "Display all log until this UNIX epoch.",
+   optional => 1,
+   },
+   lastentries => {
+   description => "Limit to the last X lines.",
+   type => 'integer',
+   optional => 1,
+   },
+   startcursor => {
+   description => "Start after the given Cursor.",
+   type => 'string',
+   optional => 1,
+   },
+   endcursor => {
+   description => "End before the given Cursor.",
+   type => 'string',
+   optional => 1,
+   },
+   },
+},
+returns => {
+   type => 'array',
+   items => {
+   type => "string",
+   }
+},
+code => sub {
+   my ($param) = @_;
+
+   my $rpcenv = PVE::RPCEnvironment::get();
+   my $user = $rpcenv->get_user();
+
+   my $lines = [];
+
+   my $parser = sub {
+   push @$lines, shift;
+   };
+
+   my $cmd = ["/usr/bin/mini-journalreader"];
+   push @$cmd, '-n', $param->{lastentries} if $param->{lastentries};
+   push @$cmd, '-b', $param->{since} if $param->{since};
+   push @$cmd, '-e', $param->{until} if $param->{until};
+   push @$cmd, '-f', $param->{startcursor} if $param->{startcursor};
+   push @$cmd, '-t', $param->{endcursor} if $param->{endcursor};
+
+   PVE::Tools::run_command($cmd, outfunc => $parser);
+
+   return $lines;
+}});
+
 
 __PACKAGE__->register_method ({
 name => 'termproxy',
diff --git a/debian/control b/debian/control
index 49861fa..e0b038c 100644
--- a/debian/control
+++ b/debian/control
@@ -65,6 +65,7 @@ Depends: apt,
  pmg-log-tracker,
  postfix (>= 2.5.5),
  postgresql-9.6,
+ proxmox-mini-journalreader,
  proxmox-spamassassin,
  pve-xtermjs (>= 1.0-1),
  rrdcached,
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH widget-toolkit v2 1/1] add panel/JournalView

2019-05-15 Thread Dominik Csapak

similar to LogView, but expects the result from the /node/journal api call,
which is an array of strings, with start/endcursor instead of
an array of objects with line numbers

the new api call also does not accept start line numbers and limit,
so it is necessary to handle it differently

Signed-off-by: Dominik Csapak 
---
 Makefile |   1 +
 panel/JournalView.js | 335 +++
 2 files changed, 336 insertions(+)
 create mode 100644 panel/JournalView.js

diff --git a/Makefile b/Makefile
index d2b905c..049a1fb 100644
--- a/Makefile
+++ b/Makefile
@@ -40,6 +40,7 @@ JSSRC=\
grid/PendingObjectGrid.js   \
panel/InputPanel.js \
panel/LogView.js\
+   panel/JournalView.js\
panel/RRDChart.js   \
panel/GaugeWidget.js\
window/Edit.js  \
diff --git a/panel/JournalView.js b/panel/JournalView.js
new file mode 100644
index 000..79edaeb
--- /dev/null
+++ b/panel/JournalView.js
@@ -0,0 +1,335 @@
+/*
+ * Display log entries in a panel with scrollbar
+ * The log entries are automatically refreshed via a background task,
+ * with newest entries comming at the bottom
+ */
+Ext.define('Proxmox.panel.JournalView', {
+extend: 'Ext.panel.Panel',
+xtype: 'proxmoxJournalView',
+
+numEntries: 500,
+lineHeight: 16,
+
+scrollToEnd: true,
+
+controller: {
+   xclass: 'Ext.app.ViewController',
+
+   updateParams: function() {
+   var me = this;
+   var viewModel = me.getViewModel();
+   var since = viewModel.get('since');
+   var until = viewModel.get('until');
+
+   since.setHours(0, 0, 0, 0);
+   until.setHours(0, 0, 0, 0);
+   until.setDate(until.getDate()+1);
+
+   me.getView().loadTask.delay(200, undefined, undefined, [
+   false,
+   false,
+   Ext.Date.format(since, "U"),
+   Ext.Date.format(until, "U")
+   ]);
+   },
+
+   scrollPosBottom: function() {
+   var view = this.getView();
+   var pos = view.getScrollY();
+   var maxPos = view.getScrollable().getMaxPosition().y;
+   return maxPos - pos;
+   },
+
+   scrollPosTop: function() {
+   var view = this.getView();
+   return view.getScrollY();
+   },
+
+   updateScroll: function(livemode, num, scrollPos, scrollPosTop) {
+   var me = this;
+   var view = me.getView();
+
+   if (!livemode) {
+   setTimeout(function() { view.scrollTo(0, 0); }, 10);
+   } else if (view.scrollToEnd && scrollPos <= 0) {
+   setTimeout(function() { view.scrollTo(0, Infinity); }, 10);
+   } else if (!view.scrollToEnd && scrollPosTop < 20*view.lineHeight) {
+   setTimeout(function() { view.scrollTo(0, num*view.lineHeight + 
scrollPosTop); }, 10);
+   }
+   },
+
+   updateView: function(lines, livemode, top) {
+   var me = this;
+   var view = me.getView();
+   var viewmodel = me.getViewModel();
+   if (viewmodel.get('livemode') !== livemode) {
+   return; // we switched mode, do not update the content
+   }
+   var contentEl = me.lookup('content');
+
+   // save old scrollpositions
+   var scrollPos = me.scrollPosBottom();
+   var scrollPosTop = me.scrollPosTop();
+
+   var newend = lines.shift();
+   var newstart = lines.pop();
+
+   var num = lines.length;
+   var text = lines.map(Ext.htmlEncode).join('');
+
+   if (!livemode) {
+   view.content = num ? text : 'no content';
+   } else {
+   // update content
+   if (top && num) {
+   view.content = view.content ? text + '' + view.content 
: text;
+   } else if (!top && num) {
+   view.content = view.content ? view.content + '' + text 
: text;
+   }
+
+   // update cursors
+   if (!top || !view.startcursor) {
+   view.startcursor = newstart;
+   }
+
+   if (top || !view.endcursor) {
+   view.endcursor = newend;
+   }
+   }
+
+   contentEl.update(view.content);
+
+   me.updateScroll(livemode, num, scrollPos, scrollPosTop);
+   },
+
+   doLoad: function(livemode, top, since, until) {
+   var me = this;
+   if (me.running) {
+   me.requested = true;
+   return;
+   }
+   me.running = true;
+   var view = me.getView();
+   var params = {
+   lastentries: view.numEntries || 500,
+   };
+   if (livemode) {
+   if (!top && view.startcursor) {
+

[pve-devel] applied: [PATCH storage] pvesm set: handle deletion of properties

2019-05-15 Thread Thomas Lamprecht

On 5/14/19 4:24 PM, Thomas Lamprecht wrote:
> the delete parameter get's injected by the SectionConfigs
> updateSchem, but we need to handle it ourself in the code
> This makes the following possible:
> 
> pvesm set STORAGEID --delete property
> 
> Also the API equivalent is now possible. Adapted from the HA
> managers Resource update API call.
> 
> Signed-off-by: Thomas Lamprecht 
> ---
>  PVE/API2/Storage/Config.pm | 15 +++
>  1 file changed, 15 insertions(+)
> 


applied

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH storage] pvesm set: handle deletion of properties

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 10:14 AM, Dominik Csapak wrote:
> works like expected, there is some issue in the context though (comment 
> inline)
> 
> On 5/14/19 4:24 PM, Thomas Lamprecht wrote:
>> the delete parameter get's injected by the SectionConfigs
>> updateSchem, but we need to handle it ourself in the code
>> This makes the following possible:
>>
>> pvesm set STORAGEID --delete property
>>
>> Also the API equivalent is now possible. Adapted from the HA
>> managers Resource update API call.
>>
>> Signed-off-by: Thomas Lamprecht 
>> ---
>>   PVE/API2/Storage/Config.pm | 15 +++
>>   1 file changed, 15 insertions(+)
>>
>> diff --git a/PVE/API2/Storage/Config.pm b/PVE/API2/Storage/Config.pm
>> index c114ddc..f5e1560 100755
>> --- a/PVE/API2/Storage/Config.pm
>> +++ b/PVE/API2/Storage/Config.pm
>> @@ -195,6 +195,7 @@ __PACKAGE__->register_method ({
>>     my $storeid = extract_param($param, 'storage');
>>   my $digest = extract_param($param, 'digest');
>> +    my $delete = extract_param($param, 'delete');
>>     PVE::Storage::lock_storage_config(
>>    sub {
>> @@ -208,6 +209,20 @@ __PACKAGE__->register_method ({
>>   my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
>>   my $opts = $plugin->check_config($storeid, $param, 0, 1);
>>   +    if ($delete) {
>> +    my $options = $plugin->private()->{options}->{$scfg->{type}};
>> +    foreach my $k (PVE::Tools::split_list($delete)) {
>> +    my $d = $options->{$k} || die "no such option '$k'\n";
>> +    die "unable to delete required option '$k'\n" if 
>> !$d->{optional};
>> +    die "unable to delete fixed option '$k'\n" if $d->{fixed};
>> +    die "cannot set and delete property '$k' at the same time!\n"
>> +    if defined($opts->{$k});
>> +
>> +    delete $scfg->{$k};
>> +    }
>> +    }
>> +
>> +
>>   foreach my $k (%$opts) {
> 
> here we ought to do (keys %$opts), else we use
> even the values as keys...
> 
> shall i send a patch which is based on yours? can you do it as a fixup?

let's save the overhead, I'll fix it up with Suggested-by tag for you.

Thanks for review, and noticing the issue in existing context!

> 
>>   $scfg->{$k} = $opts->{$k};
>>   }
>>


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH common] prevent autovivification of sectionconfig options

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 10:03 AM, Dominik Csapak wrote:
> If, somehow, someone passes a config to check_config with keys set
> that are not in the options for that type, this fixed check
> lead to autovivification, meaning that any future calls to the same
> worker had an additional option for that type which is not optional
> 
> this lead to a wrongfully deleting of entries when updating an entry of
> a different type, since all entries of the original types suddenly
> did not satisfy their required options and would not get parsed
> by read_file anymore (thus missing when a successful write_file was done)
> 
> Signed-off-by: Dominik Csapak 
> ---
>  src/PVE/SectionConfig.pm | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/src/PVE/SectionConfig.pm b/src/PVE/SectionConfig.pm
> index cc03aea..09ab5c2 100644
> --- a/src/PVE/SectionConfig.pm
> +++ b/src/PVE/SectionConfig.pm
> @@ -379,10 +379,10 @@ sub check_config {
>  
>  foreach my $k (keys %$config) {
>   my $value = $config->{$k};
> - 
> +
>   die "can't change value of fixed parameter '$k'\n"
> - if !$create && $opts->{$k}->{fixed};
> - 
> + if !$create && $opts->{$k} && $opts->{$k}->{fixed};
> +
>   if (defined($value)) {
>   my $tmp = $class->check_value($type, $k, $value, $sectionId, 
> $skipSchemaCheck);
>   $settings->{$k} = $class->decode_value($type, $k, $tmp);
> 

applied, with followup to use "defined" here, maybe fully correct could be
a ref(..) eq 'HASH', but we die on scalar as hash access anyway as we use
strict mode.

Thanks for finding this!

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH storage] pvesm set: handle deletion of properties

2019-05-15 Thread Dominik Csapak

works like expected, there is some issue in the context though (comment 
inline)


On 5/14/19 4:24 PM, Thomas Lamprecht wrote:

the delete parameter get's injected by the SectionConfigs
updateSchem, but we need to handle it ourself in the code
This makes the following possible:

pvesm set STORAGEID --delete property

Also the API equivalent is now possible. Adapted from the HA
managers Resource update API call.

Signed-off-by: Thomas Lamprecht 
---
  PVE/API2/Storage/Config.pm | 15 +++
  1 file changed, 15 insertions(+)

diff --git a/PVE/API2/Storage/Config.pm b/PVE/API2/Storage/Config.pm
index c114ddc..f5e1560 100755
--- a/PVE/API2/Storage/Config.pm
+++ b/PVE/API2/Storage/Config.pm
@@ -195,6 +195,7 @@ __PACKAGE__->register_method ({
  
  	my $storeid = extract_param($param, 'storage');

my $digest = extract_param($param, 'digest');
+   my $delete = extract_param($param, 'delete');
  
  PVE::Storage::lock_storage_config(

 sub {
@@ -208,6 +209,20 @@ __PACKAGE__->register_method ({
my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
my $opts = $plugin->check_config($storeid, $param, 0, 1);
  
+	if ($delete) {

+   my $options = $plugin->private()->{options}->{$scfg->{type}};
+   foreach my $k (PVE::Tools::split_list($delete)) {
+   my $d = $options->{$k} || die "no such option '$k'\n";
+   die "unable to delete required option '$k'\n" if 
!$d->{optional};
+   die "unable to delete fixed option '$k'\n" if $d->{fixed};
+   die "cannot set and delete property '$k' at the same 
time!\n"
+   if defined($opts->{$k});
+
+   delete $scfg->{$k};
+   }
+   }
+
+
foreach my $k (%$opts) {


here we ought to do (keys %$opts), else we use
even the values as keys...

shall i send a patch which is based on yours? can you do it as a fixup?


$scfg->{$k} = $opts->{$k};
}




___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH common] prevent autovivification of sectionconfig options

2019-05-15 Thread Dominik Csapak

If, somehow, someone passes a config to check_config with keys set
that are not in the options for that type, this fixed check
lead to autovivification, meaning that any future calls to the same
worker had an additional option for that type which is not optional

this lead to a wrongfully deleting of entries when updating an entry of
a different type, since all entries of the original types suddenly
did not satisfy their required options and would not get parsed
by read_file anymore (thus missing when a successful write_file was done)

Signed-off-by: Dominik Csapak 
---
 src/PVE/SectionConfig.pm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/PVE/SectionConfig.pm b/src/PVE/SectionConfig.pm
index cc03aea..09ab5c2 100644
--- a/src/PVE/SectionConfig.pm
+++ b/src/PVE/SectionConfig.pm
@@ -379,10 +379,10 @@ sub check_config {
 
 foreach my $k (keys %$config) {
my $value = $config->{$k};
-   
+
die "can't change value of fixed parameter '$k'\n"
-   if !$create && $opts->{$k}->{fixed};
-   
+   if !$create && $opts->{$k} && $opts->{$k}->{fixed};
+
if (defined($value)) {
my $tmp = $class->check_value($type, $k, $value, $sectionId, 
$skipSchemaCheck);
$settings->{$k} = $class->decode_value($type, $k, $tmp);
-- 
2.11.0


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Bug in Firewall for KVM

2019-05-15 Thread Detlef Bracker

Correction: The IP-Filter blocks only traffic from inside to outside and 
not as I have written from outside to inside!


Am 15.05.2019 um 09:07 schrieb Detlef Bracker:

Dear,

their are 2 bugs (or thats features?) in the firewall with VM with 
Cloud-Init!


(

a) When in VM -> Cloud-Init -> Network-Interface:  firewall set on

AND

b) VM -> Firewall -> Options -> IP-Filter: set on

)

AND

c) VM -> Firewall -> Options -> Firewall: set off / set on - is equal 
will been ignored for IP-Filter !!!


The 1st is that traffic from outside will been blocked, why the 
settings for cloud-init will not been used for the IP-Filter?!
The 2nd is that IP-Filter will not been ignored, when the Firewall is 
generally off in the options! I understand, the filter is one thing of 
the firewall and when the firewall is off, then all filters must been 
ignored?! Thats from the understanding the same as when firewall 
switch off by the network interface!


Regards

Detlef


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

--

ACHTUNG: Ihr Anfragetext befindet sich unter unserem Absender!
P.S. ePrivacy in Europa - lesen Sie mehr - read more 
 



Mit freundlichen Gruessen
1awww.com - Internet-Service-Provider

Detlef Bracker
Camino Velilla 1, E 18690 Almunecar, Tel.: +34.6 343 232 61 * EU-VAT-ID: 
ESX4516542D


This email and any files transmitted are confidential and intended only 
or the person(s) directly addressed. If you are not the intended 
recipient, any use, copying, transmission, distribution, or other forms 
of dissemination is strictly prohibited. If you have received this email 
in error, please notify the sender immediately and permanently delete 
this email with any files that may be attached.


Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, 
contiene información de carácter confidencial exclusivamente dirigida a 
su destinatario o destinatarios. Queda prohibida su divulgación, copia o 
distribución a terceros sin la previa autorización escrita de Detlef 
Bracker. En caso de no ser usted la persona a la que fuera dirigido este 
mensaje y a pesar de ello está continúa leyéndolo, ponemos en su 
conocimiento que está cometiendo un acto ilícito en virtud de la 
legislación vigente en la actualidad, por lo que deberá dejarlo de leer 
automáticamente.


Detlef Bracker no es responsable de su integridad, exactitud, o de lo 
que acontezca cuando el correo electrónico circula por las 
infraestructuras de comunicaciones electrónicas públicas. En el caso de 
haber recibido este correo electrónico por error, se ruega notificar 
inmediatamente esta circunstancia mediante reenvío a la dirección 
electrónica del remitente.


El correo electrónico vía Internet no permite asegurar la 
confidencialidad de los mensajes que se transmiten ni su integridad o 
correcta recepción, por lo que Detlef Bracker no asume ninguna 
responsabilidad que pueda derivarse de este hecho.


No imprima este correo si no es necesario. Ahorrar papel protege el 
medio ambiente.


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] Bug in Firewall for KVM

2019-05-15 Thread Detlef Bracker


Dear,

their are 2 bugs (or thats features?) in the firewall with VM with 
Cloud-Init!


(

a) When in VM -> Cloud-Init -> Network-Interface:  firewall set on

AND

b) VM -> Firewall -> Options -> IP-Filter: set on

)

AND

c) VM -> Firewall -> Options -> Firewall: set off / set on - is equal 
will been ignored for IP-Filter !!!


The 1st is that traffic from outside will been blocked, why the settings 
for cloud-init will not been used for the IP-Filter?!
The 2nd is that IP-Filter will not been ignored, when the Firewall is 
generally off in the options! I understand, the filter is one thing of 
the firewall and when the firewall is off, then all filters must been 
ignored?! Thats from the understanding the same as when firewall switch 
off by the network interface!


Regards

Detlef


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH cluster v3] add generic data broadcast interface

2019-05-15 Thread Thomas Lamprecht

On 5/14/19 11:15 AM, Dominik Csapak wrote:
> similar to how we handle the cluster wide tasklist and rrd data,
> have an interface that can sync data across the cluster
> 
> this data is only transient and will not be written to disk
> 
> we can use this for a number of things, e.g. listing ceph services
> across the cluster, etc.
> 
> Signed-off-by: Dominik Csapak 
> ---
> changes from v2:
> * user proper sub ($sub->() vs $get_node_data->())
> * encode data only once
> * add comment about maximum pmxcfs size
> * die instead of eval/warn/syslog (caller must handle failure)
> * remove post if variable declaration
> 

implementation looks OK, but like we talked offline I'd have two changes in
behaviour:

* ability to "delete" a key - optimal, real deletion from the hashtable, else
  at leas a way to set it to '' or the like.
* "intelligent" data encoding, simplified and untested, something like:

$data = encode_json($data) if (ref($data));
die "to big" if length($data) > 32 * 1024;
...

 

>  data/PVE/Cluster.pm | 44 
>  1 file changed, 44 insertions(+)
> 
> diff --git a/data/PVE/Cluster.pm b/data/PVE/Cluster.pm
> index 5af11e6..4c719eb 100644
> --- a/data/PVE/Cluster.pm
> +++ b/data/PVE/Cluster.pm
> @@ -540,6 +540,50 @@ sub get_nodelist {
>  return [ keys %$nodelist ];
>  }
>  
> +# best effort data store for cluster
> +# this data is gone if the pmxcfs is restarted, but only the local data,
> +# so we should not use this for very important data
> +sub broadcast_node_kv {
> +my ($key, $data) = @_;
> +
> +my $enc_data = encode_json($data);
> +my $size = length($enc_data);
> +# pmxcfs has an upper bound of 32k for each entry
> +if ($size >= (32 * 1024)) {
> + warn "data for '$key' too big\n";
> + return;
> +}
> +
> +eval {
> + $ipcc_update_status->("kv/$key", $enc_data);
> +};
> +
> +warn $@ if $@;
> +}
> +
> +sub get_node_kv {
> +my ($key, $nodename) = @_;
> +
> +my $res = {};
> +my $get_node_data = sub {
> + my ($node) = @_;
> + my $raw = $ipcc_get_status->("kv/$key", $node);
> + $res->{$node} = decode_json($raw) if $raw;
> +};
> +
> +if ($nodename) {
> + $get_node_data->($nodename);
> +} else {
> + my $nodelist = get_nodelist();
> +
> + foreach my $node (@$nodelist) {
> + $get_node_data->($node);
> + }
> +}
> +
> +return $res;
> +}
> +
>  # $data must be a chronological descending ordered array of tasks
>  sub broadcast_tasklist {
>  my ($data) = @_;
> 


___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH] cherry pick MDS fixes from stable 4.14.119

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 8:24 AM, Thomas Lamprecht wrote:
> On 5/15/19 7:57 AM, Thomas Lamprecht wrote:
>> With some manual merging, most of it straight forward, cherry-pick
>> the all but the two PowerPC and S390 patches from the 4.14.119
>> released by Greg KH[0]. It mainly comes with some mitigation for
>> MDS[1][3][4][5], for best result a microupdate of the CPU is
>> required, else the kernel falls back to some "best effort
>> mitigation", trying to clear the CPU buffers on kernel/userspace,
>> hypervisor/guest and C-state (idle) transitions.
>>
>> With this applied you will have a new file in sysfs to get the
>> mitigation state of the server regarding MDS:
>>  $ cat /sys/devices/system/cpu/vulnerabilities/mds
>>
>> Microcode updates should come available in stretch with
>> 3.20190514.1~deb9u1 [2] version currently only tagged[2], but not yet
>> released.
>>
>> [0]: https://lwn.net/ml/linux-kernel/20190514180538.ga13...@kroah.com/
>> [1]: 
>> https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html#mitigation-strategy
>> [2]: 
>> https://salsa.debian.org/hmh/intel-microcode/commits/debian/3.20190514.1_deb9u1
>> [3]: https://mdsattacks.com/
>> [4]: https://cpu.fail/
>> [5]: 
>> https://www.intel.com/content/www/us/en/architecture-and-technology/mds.html
>>
>> Signed-off-by: Thomas Lamprecht 
>> ---
> 
> See also the 5.1.2 stable release announcement[0] for some other links and a 
> little
> more details from Linux perspective. As stated, this is probably not final 
> and may
> break some things, that said, my build here worked well without issues in a 
> physical
> cluster with VMs, CTs and ceph, so at least this isn't broken in a obvious 
> way.
> 
> A look over this (@Fabian ;-) would still be great.

OK, scratch that, let's just use Ubuntu-4.15.0-50.54 [0] which has this too, 
while
it was commited > 8days ago I swear that I did not see it yesterday evening 
before
starting the backport work, maybe it just was to late...

[0]: 
https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/tag/?h=Ubuntu-4.15.0-50.54

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH] cherry pick MDS fixes from stable 4.14.119

2019-05-15 Thread Thomas Lamprecht

On 5/15/19 7:57 AM, Thomas Lamprecht wrote:
> With some manual merging, most of it straight forward, cherry-pick
> the all but the two PowerPC and S390 patches from the 4.14.119
> released by Greg KH[0]. It mainly comes with some mitigation for
> MDS[1][3][4][5], for best result a microupdate of the CPU is
> required, else the kernel falls back to some "best effort
> mitigation", trying to clear the CPU buffers on kernel/userspace,
> hypervisor/guest and C-state (idle) transitions.
> 
> With this applied you will have a new file in sysfs to get the
> mitigation state of the server regarding MDS:
>  $ cat /sys/devices/system/cpu/vulnerabilities/mds
> 
> Microcode updates should come available in stretch with
> 3.20190514.1~deb9u1 [2] version currently only tagged[2], but not yet
> released.
> 
> [0]: https://lwn.net/ml/linux-kernel/20190514180538.ga13...@kroah.com/
> [1]: 
> https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html#mitigation-strategy
> [2]: 
> https://salsa.debian.org/hmh/intel-microcode/commits/debian/3.20190514.1_deb9u1
> [3]: https://mdsattacks.com/
> [4]: https://cpu.fail/
> [5]: 
> https://www.intel.com/content/www/us/en/architecture-and-technology/mds.html
> 
> Signed-off-by: Thomas Lamprecht 
> ---

See also the 5.1.2 stable release announcement[0] for some other links and a 
little
more details from Linux perspective. As stated, this is probably not final and 
may
break some things, that said, my build here worked well without issues in a 
physical
cluster with VMs, CTs and ceph, so at least this isn't broken in a obvious way.

A look over this (@Fabian ;-) would still be great.

[0]: https://www.spinics.net/lists/stable/msg302862.html

___
pve-devel mailing list
pve-devel@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel