Re: [PVE-User] PVE 6, wireless and regulatory database...
Mandi! Alwin Antreich In chel di` si favelave... > > I've installed the buster package... > You will need the package from the backports. Sorry for the late answer, but even at home i've needed to define with my stakeholder a mainenance windows for a cluster reboot. ;-) I confirm, work as expected. Jun 3 23:51:57 ino kernel: [7.866523] cfg80211: Loading compiled-in X.509 certificates for regulatory database Jun 3 23:51:57 ino kernel: [7.878636] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' Jun 3 23:51:57 ino kernel: [8.070696] ath: EEPROM regdomain: 0x809c Jun 3 23:51:57 ino kernel: [8.070698] ath: EEPROM indicates we should expect a country code Jun 3 23:51:57 ino kernel: [8.070698] ath: doing EEPROM country->regdmn map search Jun 3 23:51:57 ino kernel: [8.070699] ath: country maps to regdmn code: 0x52 Jun 3 23:51:57 ino kernel: [8.070700] ath: Country alpha2 being used: CN Jun 3 23:51:57 ino kernel: [8.070700] ath: Regpair used: 0x52 Jun 3 23:51:57 ino kernel: [8.072132] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht' Jun 3 23:51:57 ino kernel: [8.072647] ieee80211 phy0: Atheros AR9287 Rev:2 mem=0x9d084dbf, irq=16 Jun 3 23:51:57 ino kernel: [8.080354] ath9k :10:00.0 wls1: renamed from wlan0 Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE 6, wireless and regulatory database...
Mandi! Alwin Antreich In chel di` si favelave... > > root@ino:~# dpkg -l | grep wireless-regdb > > ii wireless-regdb 2016.06.10-1 > > all wireless regulatory database > You need the Debian package for the alternatives to work. IIRC, you > finally installed the ubuntu package. https://packages.debian.org/buster/wireless-regdb I've installed the buster package... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE 6, wireless and regulatory database...
Mandi! Alwin Antreich In chel di` si favelave... > It is not an issue with the package. I forgot about the alternatives in > Debian (thanks Thomas). Once you set the alternative (tool: > update-alternatives) to the regulatory.db-upstream it will be loaded > without complaining. > I suppose Debian can't switch to the signed database yet, since their > shipped kernel version might not yet know how to handle the signed > regulatory.db. root@ino:~# dpkg -l | grep wireless-regdb ii wireless-regdb 2016.06.10-1 all wireless regulatory database root@ino:~# update-alternatives --display wireless-regdb update-alternatives: error: no alternatives for wireless-regdb But: root@ino:~# update-alternatives --display regulatory.db update-alternatives: warning: alternative /lib/firmware/regulatory.db-debian (part of link group regulatory.db) doesn't exist; removing from list of alternatives update-alternatives: warning: alternative /lib/firmware/regulatory.db-upstream (part of link group regulatory.db) doesn't exist; removing from list of alternatives regulatory.db - auto mode link best version not available link currently points to /lib/firmware/regulatory.db-debian link regulatory.db is /lib/firmware/regulatory.db slave regulatory.db.p7s is /lib/firmware/regulatory.db.p7s but anyway: root@ino:~# update-alternatives --config regulatory.db update-alternatives: error: no alternatives for regulatory.db (boh, seems also /lib/firmware/regulatory.db-debian is missing... strange...) I need to download manually the regulatory DB? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE 6, wireless and regulatory database...
> it make sense to fire up a bug in PVE BTS, to have this package added > in repository? Anyway, done: https://bugzilla.proxmox.com/show_bug.cgi?id=2753 -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE 6, wireless and regulatory database...
Mandi! Alwin Antreich In chel di` si favelave... > Debian uses a different file name for the signature file then ubuntu. A-HA! > You can always download the latest wireless-regdb and replace the db + > signature files. > https://wireless.wiki.kernel.org/en/developers/regulatory/wireless-regdb I've downloaded and installed by hand ubuntu package (wireless-regdb_2018.05.09-0ubuntu1_all.deb) and installed flawlessy. Now regulatory DB load: May 20 22:35:03 ino kernel: [7.795925] cfg80211: Loading compiled-in X.509 certificates for regulatory database May 20 22:35:03 ino kernel: [7.816343] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' May 20 22:35:03 ino kernel: [7.961683] ath: EEPROM regdomain: 0x809c May 20 22:35:03 ino kernel: [7.961684] ath: EEPROM indicates we should expect a country code May 20 22:35:03 ino kernel: [7.961685] ath: doing EEPROM country->regdmn map search May 20 22:35:03 ino kernel: [7.961686] ath: country maps to regdmn code: 0x52 May 20 22:35:03 ino kernel: [7.961686] ath: Country alpha2 being used: CN May 20 22:35:03 ino kernel: [7.961687] ath: Regpair used: 0x52 May 20 22:35:03 ino kernel: [7.963108] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht' May 20 22:35:03 ino kernel: [7.964097] ieee80211 phy0: Atheros AR9287 Rev:2 mem=0xa51c0d30, irq=16 May 20 22:35:03 ino kernel: [7.981453] ath9k :10:00.0 wls1: renamed from wlan0 it make sense to fire up a bug in PVE BTS, to have this package added in repository? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE 6, wireless and regulatory database...
Mandi! Martin Maurer In chel di` si favelave... > use the buster-backports - > https://packages.debian.org/buster-backports/wireless-regdb > (see also https://backports.debian.org/) Seems is not sufficient: May 18 23:55:24 ino kernel: [7.751088] cfg80211: Loading compiled-in X.509 certificates for regulatory database May 18 23:55:24 ino kernel: [7.757944] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' May 18 23:55:24 ino kernel: [7.764861] cfg80211: loaded regulatory.db is malformed or signature is missing/invalid May 18 23:55:24 ino kernel: [7.905519] ath: EEPROM regdomain: 0x809c May 18 23:55:24 ino kernel: [7.905520] ath: EEPROM indicates we should expect a country code May 18 23:55:24 ino kernel: [7.905521] ath: doing EEPROM country->regdmn map search May 18 23:55:24 ino kernel: [7.905521] ath: country maps to regdmn code: 0x52 May 18 23:55:24 ino kernel: [7.905522] ath: Country alpha2 being used: CN May 18 23:55:24 ino kernel: [7.905523] ath: Regpair used: 0x52 May 18 23:55:24 ino kernel: [7.909904] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht' May 18 23:55:24 ino kernel: [7.916935] ieee80211 phy0: Atheros AR9287 Rev:2 mem=0xb2508cdd, irq=16 May 18 23:55:24 ino kernel: [7.922095] ath9k :10:00.0 wls1: renamed from wlan0 slightly different error, but still regulatory.db is not loaded. This seems a bit strange; kernel 5.4 is the focal fossa kernel, and in focal fossa the 'wireless-regdb' package is dated '2018': https://packages.ubuntu.com/search?suite=focal=names=wireless-regdb while debian backport is 2019. Package have to be recompiled against the kernel to have key match? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] PVE 6, wireless and regulatory database...
OK, i admit that running a wireless card on a virtualization host server is a bit strange, but... it is my home server! ;-) After upgrading to 6.2 (but the same was on 6.1, i've simply missed that...) i've seen: May 17 23:47:55 ino kernel: [7.839970] cfg80211: Loading compiled-in X.509 certificates for regulatory database May 17 23:47:55 ino kernel: [7.848525] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2 May 17 23:47:55 ino kernel: [7.848527] cfg80211: failed to load regulatory.db AFAI've understood by google, the current wireless-regdb buster package (2016.06.10-1) is old for the PVE kernel. Can i ask to 'backport' it as for other hardware-support package? I know is nothing more then a warning, but... thanks. PS: i've not tried to install simply from debian backport, package seems available: https://tracker.debian.org/pkg/wireless-regdb -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Unprivileged container and fstrim...
Mandi! Tom Weber In chel di` si favelave... > pct fstrim > on the host? AAARRGHHH! Totaly missed that. Sorry to the list, and thanks. ;-) -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Unprivileged container and fstrim...
I was abused to do a 'fstrim /' weekly run on my privileged containers, and seems that the command was effective: root@vdcsv1:~# fstrim -v / /: 18,6 GiB (19946131456 bytes) trimmed but after convert many of them to unprivileged ones: root@vmw:~# fstrim -v / fstrim: /: FITRIM ioctl failed: Operazione non permessa Clearly, underlying storage is 'trimmable'. Two question. 1) trim on container generally works, or simply on privileged ones does not throw errors but do nothing? ;-) Container space change slowly, so could be that trim was simply ignored... 2) there's another way to 'trim' container disk space? Eg, from the host? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Debian buster, systemd, container and nesting=1
Mandi! Stoiko Ivanov
In chel di` si favelave...
> AFAICU one robust (although not very performant way) to run a AD DC with
> NTACLs on a unprivileged container would be to use the xattr_tdb module
> (not actively tested though):
> https://wiki.samba.org/index.php/Using_the_xattr_tdb_VFS_Module
Specifically asked in samba ML; xattr_tdb i a test module, broken, that
HAVE NOT to be used in production.
The only ''supported'' way to run Samba AD DC is via filesystem XATTR.
Also, seems the same 'troubles' hit BSD Jails:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220844
https://bugzilla.samba.org/show_bug.cgi?id=12912
see:
https://lists.samba.org/archive/samba/2020-February/228653.html
> They are independent - a good explanation of what nesting does can be
> found in our source:
> https://git.proxmox.com/?p=pve-container.git;a=blob;f=src/PVE/LXC.pm;h=34ca2a357294f63e8b49d965bd54c24905642e17;hb=HEAD#l581
> (it allows among other things to mount /proc, and /sys, which is
> problematic for privileged containers
>
> The issue with apache('s systemd-unit) in the privileged container, is
> that the mount is denied by apparmor (the apparmor rules are stricter for
> privileged containers, than for unprivileged, because if someone breaks
> out of an unprivileged container they are only a regular user on the host)
>
> I hope this explains it.
Ahem, no. ;-)
But indeed is my fault that i know very little about systemd, apparmor
and all those new wizardry... ;-)
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] do_IRQ: No irq handler for vector
Mandi! Alex Chekholko via pve-user In chel di` si favelave... > I have seen this on various kernel versions (CentOS / Debian) and on > various hardware; my understanding is that it is safe to ignore. OK. Thanks! -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] do_IRQ: No irq handler for vector
I've setup an home server, using an old HP ProLiant ML110G6 server, with PVE6, kernel 5.3.18-2-pve. I reveive, spuriously, logs like: root@ino:~# grep do_IRQ /var/log/kern.log Mar 1 05:10:51 ino kernel: [197698.269626] do_IRQ: 3.40 No irq handler for vector Mar 1 08:12:52 ino kernel: [208619.012511] do_IRQ: 2.41 No irq handler for vector Mar 1 11:59:10 ino kernel: [222196.730017] do_IRQ: 3.40 No irq handler for vector Mar 1 17:09:00 ino kernel: [240787.084289] do_IRQ: 2.41 No irq handler for vector Mar 1 17:19:10 ino kernel: [241396.356441] do_IRQ: 0.41 No irq handler for vector Mar 1 23:33:00 ino kernel: [263826.632318] do_IRQ: 1.40 No irq handler for vector Mar 2 00:08:51 ino kernel: [265976.985582] do_IRQ: 3.41 No irq handler for vector Mar 2 05:40:21 ino kernel: [285866.764386] do_IRQ: 1.41 No irq handler for vector Mar 2 08:48:00 ino kernel: [297125.421168] do_IRQ: 2.40 No irq handler for vector Mar 2 08:52:01 ino kernel: [297366.060392] do_IRQ: 2.41 No irq handler for vector Mar 2 11:32:01 ino kernel: [306965.868404] do_IRQ: 0.39 No irq handler for vector Server work as expected, without any trouble, crash, ... AFAI've understood, first numer is the core (i have 4), second is the interrupt vector. Googling around i've hit: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1781016 and effectively i've many 'serial' USB interfaces connected (2 USB serial port, an APC UPS, a GSM dongle). I've tried to, using the irq vecrot, to understand the peripheral, but the command proposed: grep 35 /sys/kernel/debug/irq/irqs/* does not work, there's no more '/sys/kernel/debug/irq/irqs/*' interface in sysfs. Someone have some hint? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Debian buster, systemd, container and nesting=1
Mandi! Stoiko Ivanov In chel di` si favelave... > > i can convert this container to an unprivileged ones, but other no, for > > examples some containers are samba domain controller, that need a > > privileged container. > not sure - but why would a samba need to be privileged? https://lists.samba.org/archive/samba/2019-December/227626.html samba, as AD Domain Controller, not as general 'share service', need the use of 'SYSTEM' namespace, that in containers is reserved by root. Indeed, if there's some 'caps' to relax that permit to use system namespace with unprivileged containers, they are welcomed! > > There's another/better way to make systemd work on containers? > I guess my preferred actions in order: > * setup new unprivileged container and migrate the workload/services from > the old one (optionally enabling nesting if needed) > * try backup/restore to get a privileged container to an unprivileged one > * keep the privileged container with nesting off > * migrate the setup into a qemu-guest > * edit the unit files of the affected services (e.g. apache) - usually > it's the PrivateTmp option which causes this (it wants to mount --rbind > -o rw /) - and drop the PrivateTmp option (see [0]) > * consider making an apparmor override for this particular mount > combination+container (which also can potentially be a security hole > (some apparmor rules are bound to absolute paths and using rbind you can > change the path) > * turn on nesting for a privileged container (keep in mind that you then > open it up quite a bit for breakouts) > of course probably not all of those options can be applied in your > environment. > [0]https://forum.proxmox.com/threads/apache2-service-failed-to-set-up-mount-namespacing-permission-denied.56871/ Mmmh... i'm a bit confused. Firstly, it is not clear to me if nesting is needed because the container is privileged, or privileged/unprivileged and nesting/non nesting are property totally indipendent. Second, in a PVE6 installation i've creared a debian buster container (unprivileged, without nesting), installed apache and run correctly, without tackling systemd units: root@vbaculalpb:~# systemctl status apache2 ● apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-02-26 11:35:29 CET; 15min ago Docs: https://httpd.apache.org/docs/2.4/ Main PID: 1992 (apache2) Tasks: 54 (limit: 4915) Memory: 6.7M CGroup: /system.slice/apache2.service ├─1992 /usr/sbin/apache2 -k start ├─1994 /usr/sbin/apache2 -k start └─1995 /usr/sbin/apache2 -k start feb 26 11:35:29 vbaculalpb systemd[1]: Starting The Apache HTTP Server... feb 26 11:35:29 vbaculalpb systemd[1]: Started The Apache HTTP Server. root@vbaculalpb:~# systemctl show apache2 | grep PrivateTmp PrivateTmp=yes This could lead to the answer to first question (nesting is needed only for privileged containers), but also could lead to the fact that container management could be diffierent between PVE5 (the original request) and PVE6 (this test). So, thanks for the answer but i hope in some more clue. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] How to restart ceph-mon?
Mandi! Alwin Antreich
In chel di` si favelave...
> > Anyway, nothing changed. On a rebooted node:
> > root@capitanmarvel:~# ps aux | grep ceph[-]mon
> > ceph2725 0.5 0.2 54 98428 ?Ssl feb18 21:14
> > /usr/bin/ceph-mon -i capitanmarvel --pid-file
> > /var/run/ceph/mon.capitanmarvel.pid -c /etc/ceph/ceph.conf --cluster ceph
> > --setuser ceph --setgroup ceph
> > on a node when i do a 'systemctl restart ceph-mgr@.service':
> > root@hulk:~# ps aux | grep ceph[-]mon
> > ceph 4166380 0.8 0.1 466648 55676 ?Ssl 15:19 0:03
> > /usr/bin/ceph-mon -f --cluster ceph --id hulk --setuser ceph --setgroup ceph
> I don't see this in the systemd unit files for Ceph. Also my test
> systems do not have the pid file either. Maybe this is something from
> an previous upgrade?
Could be. This cluster (all, indeed) was upgraded from 4.4.
> systemctl cat ceph-mon@.service
> You can check with the above command how each Ceph service or target
> should be started.
root@capitanmarvel:~# systemctl cat ceph-mon@capitanmarvel.service
# /lib/systemd/system/ceph-mon@.service
[Unit]
Description=Ceph cluster monitor daemon
# According to:
# http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget
# these can be removed once ceph-mon will dynamically change network
# configuration.
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target
PartOf=ceph-mon.target
[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
EnvironmentFile=-/etc/default/ceph
Environment=CLUSTER=ceph
ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph
--setgroup ceph
ExecReload=/bin/kill -HUP $MAINPID
PrivateDevices=yes
ProtectHome=true
ProtectSystem=full
PrivateTmp=true
TasksMax=infinity
Restart=on-failure
StartLimitInterval=30min
StartLimitBurst=5
RestartSec=10
[Install]
WantedBy=ceph-mon.target
# /lib/systemd/system/ceph-mon@.service.d/ceph-after-pve-cluster.conf
[Unit]
After=pve-cluster.service
root@hulk:~# systemctl cat ceph-mon@hulk.service
# /lib/systemd/system/ceph-mon@.service
[Unit]
Description=Ceph cluster monitor daemon
# According to:
# http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget
# these can be removed once ceph-mon will dynamically change network
# configuration.
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target
PartOf=ceph-mon.target
[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
EnvironmentFile=-/etc/default/ceph
Environment=CLUSTER=ceph
ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph
--setgroup ceph
ExecReload=/bin/kill -HUP $MAINPID
PrivateDevices=yes
ProtectHome=true
ProtectSystem=full
PrivateTmp=true
TasksMax=infinity
Restart=on-failure
StartLimitInterval=30min
StartLimitBurst=5
RestartSec=10
[Install]
WantedBy=ceph-mon.target
# /lib/systemd/system/ceph-mon@.service.d/ceph-after-pve-cluster.conf
[Unit]
After=pve-cluster.service
seems identical to me...
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] How to restart ceph-mon?
Mandi! Alwin Antreich In chel di` si favelave... > Yes, that looks strange. But as said before, it is deprecated to use > IDs. Best destroy and re-create the MON one-by-one. The default command > will create them with the hostname as ID. Then this phenomenon should > disappear as well. Done, via web interface, with a little glitch. I've stopped and dropped the monitor, but these don't stop (and drop) the manager, and so creating a new mon va webinterface lead to: Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@hulk.service -> /lib/systemd/system/ceph-mon@.service. INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing' INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-osd/ceph.keyring INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-rgw/ceph.keyring INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-mds/ceph.keyring INFO:ceph-create-keys:Talking to monitor... TASK ERROR: ceph manager directory '/var/lib/ceph/mgr/ceph-hulk' already exists probably because the task try also to fire up a mgr, that was just created. Anyway, nothing changed. On a rebooted node: root@capitanmarvel:~# ps aux | grep ceph[-]mon ceph2725 0.5 0.2 54 98428 ?Ssl feb18 21:14 /usr/bin/ceph-mon -i capitanmarvel --pid-file /var/run/ceph/mon.capitanmarvel.pid -c /etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph on a node when i do a 'systemctl restart ceph-mgr@.service': root@hulk:~# ps aux | grep ceph[-]mon ceph 4166380 0.8 0.1 466648 55676 ?Ssl 15:19 0:03 /usr/bin/ceph-mon -f --cluster ceph --id hulk --setuser ceph --setgroup ceph All cluster is healthy and works as expected, anyway: root@hulk:~# ceph -s cluster: id: 8794c124-c2ec-4e81-8631-742992159bd6 health: HEALTH_OK services: mon: 5 daemons, quorum blackpanther,capitanmarvel,deadpool,hulk,thor mgr: blackpanther(active), standbys: capitanmarvel, deadpool, thor, hulk osd: 12 osds: 12 up, 12 in -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] How to restart ceph-mon?
Mandi! Alwin Antreich In chel di` si favelave... > > it is time to kill it? > I suppose you did that already. Did it work? No, i've done just now. But yes, a 'kill' worked. Monitor restarted. Only a little note. On boot, monitor run with this cmdline: root@deadpool:~# ps aux | grep ceph-[m]on ceph2402 0.6 2.1 808428 356540 ? Ssl feb18 16:43 /usr/bin/ceph-mon -i 4 --pid-file /var/run/ceph/mon.4.pid -c /etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph on 'systemctl start ceph-mon@.service', monitor run with: root@hulk:~# ps aux | grep ceph-[m]on ceph 3276772 36.0 0.7 768580 357484 ? Ssl 15:05 0:04 /usr/bin/ceph-mon -f --cluster ceph --id 2 --setuser ceph --setgroup ceph eg, without '--pid-file /var/run/ceph/mon.4.pid -c /etc/ceph/ceph.conf' and with a different cmdline. FYI. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] How to restart ceph-mon?
Mandi! Alwin Antreich
In chel di` si favelave...
> What does the status of the service show?
> systemctl status ceph-mon@3.service
Uh, never minded about that, damn me!
root@thor:~# systemctl status ceph-mon@3.service
● ceph-mon@3.service - Ceph cluster monitor daemon
Loaded: loaded (/lib/systemd/system/ceph-mon@.service; disabled; vendor
preset: enabled)
Drop-In: /lib/systemd/system/ceph-mon@.service.d
└─ceph-after-pve-cluster.conf
Active: failed (Result: exit-code) since Wed 2020-02-19 11:31:47 CET; 29min
ago
Process: 3434884 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id 3
--setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Main PID: 3434884 (code=exited, status=1/FAILURE)
Feb 19 11:31:37 thor systemd[1]: ceph-mon@3.service: Failed with result
'exit-code'.
Feb 19 11:31:47 thor systemd[1]: ceph-mon@3.service: Service hold-off time
over, scheduling restart.
Feb 19 11:31:47 thor systemd[1]: Stopped Ceph cluster monitor daemon.
Feb 19 11:31:47 thor systemd[1]: ceph-mon@3.service: Start request repeated
too quickly.
Feb 19 11:31:47 thor systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 19 11:31:47 thor systemd[1]: ceph-mon@3.service: Unit entered failed state.
Feb 19 11:31:47 thor systemd[1]: ceph-mon@3.service: Failed with result
'exit-code'.
I've tried:
systemctl stop ceph-mon@3.service
but old daemon is still alive:
root@thor:~# systemctl stop ceph-mon@3.service
root@thor:~# ps aux | grep ceph-[m]on
ceph2469 0.7 0.1 539704 67408 ?Ssl 2019 918:08
/usr/bin/ceph-mon -i 3 --pid-file /var/run/ceph/mon.3.pid -c
/etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph
it is time to kill it?
> To add, the numeric ID for MONs is an old concept and already depricated
> for some time now. Best recreate them, the default uses already the
> hostname for its ID.
OK, thanks.
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] How to restart ceph-mon?
I've upgraded ceph, PVE5, minor upgrade from 12.2.12 to 12.2.13. OSD nodes get rebooted, but i have also two nodes that are only monitors, and host some VM/LXC so i've tried to simply restart ceph-mon. But seems isineffective: root@thor:~# ps aux | grep ceph-[m]on ceph2469 0.7 0.1 539852 67720 ?Ssl 2019 917:55 /usr/bin/ceph-mon -i 3 --pid-file /var/run/ceph/mon.3.pid -c /etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph root@thor:~# systemctl restart ceph-mon@3.service root@thor:~# ps aux | grep ceph-[m]on ceph2469 0.7 0.1 539852 67720 ?Ssl 2019 917:55 /usr/bin/ceph-mon -i 3 --pid-file /var/run/ceph/mon.3.pid -c /etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph I've tried to see in pve wiki if there's some know procedure to do ceph 'minor upgrades' but found nothing. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] PVE 6: postfix in a debian buster container, 'satellite' does not work.
I'm not sue this is a debian/posfix or a consequences of packaging it in a container, so i try to ask here. I've setup a new container 'debian buster', and configure inside postfix a 'satellite system' to send all email to my internal SMTP server. A local inject work: echo prova | mail -s test root Feb 17 16:35:29 vbaculalpb postfix/pickup[4047]: 318E18A4A: uid=0 from= Feb 17 16:35:29 vbaculalpb postfix/cleanup[4104]: 318E18A4A: message-id=<20200217153529.318E18A4A@vbaculalpb.localdomain> Feb 17 16:35:29 vbaculalpb postfix/qmgr[4048]: 318E18A4A: from=, size=408, nrcpt=1 (queue active) Feb 17 16:35:29 vbaculalpb postfix/smtp[4106]: 318E18A4A: to=, orig_to=, relay=mail.lilliput.linux.it[192.168.1.1]:25, delay=0.21, delays=0.02/0.01/0.01/0.16, dsn=2.0.0, status=sent (250 OK id=1j3iQj-0003Yc-7t) Feb 17 16:35:29 vbaculalpb postfix/qmgr[4048]: 318E18A4A: removed bu some services, that use 'localhost' as SMTP server, no: Feb 17 16:32:18 vbaculalpb postfix/master[383]: warning: process /usr/lib/postfix/sbin/smtpd pid 3687 exit status 1 Feb 17 16:32:18 vbaculalpb postfix/master[383]: warning: /usr/lib/postfix/sbin/smtpd: bad command startup -- throttling Feb 17 16:33:18 vbaculalpb postfix/smtpd[3688]: fatal: in parameter smtpd_relay_restrictions or smtpd_recipient_restrictions, specify at least one working instance of: reject_unauth_destination, defer_unauth_destination, reject, defer, defer_if_permit or check_relay_domains googling a bit lead me to: postconf compatibility_level=2 postfix reload and after that 'satellite system' work as usual. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Debian buster, systemd, container and nesting=1
I'm still on PVE 5.4. I've upgraded a (privileged) LXC container to debian buster, that was originally installed as debian jessie, then upgraded to stretch, but still without systemd. Upgrading to buster trigger systemd installation. After installation, most of the services, not all, does not start, eg apache: root@vnc:~# systemctl status apache2.service ● apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2020-02-18 16:06:35 CET; 44s ago Docs: https://httpd.apache.org/docs/2.4/ Process: 120 ExecStart=/usr/sbin/apachectl start (code=exited, status=226/NAMESPACE) feb 18 16:06:35 vnc systemd[1]: Starting The Apache HTTP Server... feb 18 16:06:35 vnc systemd[120]: apache2.service: Failed to set up mount namespacing: Permission denied feb 18 16:06:35 vnc systemd[120]: apache2.service: Failed at step NAMESPACE spawning /usr/sbin/apachectl: Permission denied feb 18 16:06:35 vnc systemd[1]: apache2.service: Control process exited, code=exited, status=226/NAMESPACE feb 18 16:06:35 vnc systemd[1]: apache2.service: Failed with result 'exit-code'. feb 18 16:06:35 vnc systemd[1]: Failed to start The Apache HTTP Server. google say me to add 'nesting=1' to 'features', that works, but looking at: https://pve.proxmox.com/wiki/Linux_Container i read: nesting= (default = 0) Allow nesting. Best used with unprivileged containers with additional id mapping. Note that this will expose procfs and sysfs contents of the host to the guest. i can convert this container to an unprivileged ones, but other no, for examples some containers are samba domain controller, that need a privileged container. There's another/better way to make systemd work on containers? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Network interfaces renaming strangeness...
Mandi! Alexandre DERUMIER In chel di` si favelave... > can you try to edit: > /usr/lib/systemd/network/99-default.link Mmmh.. it is not a good practice edit systemd config file, but create an override in /etc/systemd... Anyway, looking at systemd bygreport you post later, i've done: root@ino:~# cat /etc/systemd/network/10-e1000e-quirks.link [Match] Driver=e1000e [Link] NamePolicy=path rebuild initrd, rebot and now my network card is called 'enp16s0' (yes, in the meantime i've moved the card in another pcie slot ;). I don't know if this is worth a note in pve wiki, anyway, thanks to all. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Network interfaces renaming strangeness...
Mandi! Alexandre DERUMIER
In chel di` si favelave...
> is it a fresh proxmox6 install ? or upgraded from proxmox5?
Fresh proxmox 6, from scratch. Upgraded daily via APT.
> any /etc/udev/rules.d/70-persistent-net.rules file somewhere ? (should be
> removed)
No:
root@ino:~# ls -la /etc/udev/rules.d/70-persistent-net.rules
ls: cannot access '/etc/udev/rules.d/70-persistent-net.rules': No such
file or directory
> no special grub option ? (net.ifnames, ...)
No:
root@ino:~# cat /etc/default/grub /etc/default/grub.d/init-select.cfg |
egrep -v '^[[:space:]]*#'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet nmi_watchdog=0 intel_iommu=on"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"
GRUB_DISABLE_OS_PROBER=true
GRUB_DISABLE_RECOVERY="true"
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Network interfaces renaming strangeness...
I've setup a little 'home' PVE server: proxmox6 (debian buster, kernel 5.3.18-1-pve). Bultin network card get detected and renamed: root@ino:~# grep tg3 /var/log/kern.log Feb 11 09:16:42 ino kernel: [3.449190] tg3.c:v3.137 (May 11, 2014) Feb 11 09:16:42 ino kernel: [3.468877] tg3 :1e:00.0 eth0: Tigon3 [partno(BCM95723) rev 5784100] (PCI Express) MAC address 9c:8e:99:7b:86:d9 Feb 11 09:16:42 ino kernel: [3.468879] tg3 :1e:00.0 eth0: attached PHY is 5784 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[0]) Feb 11 09:16:42 ino kernel: [3.468881] tg3 :1e:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Feb 11 09:16:42 ino kernel: [3.468883] tg3 :1e:00.0 eth0: dma_rwctrl[7618] dma_mask[64-bit] Feb 11 09:16:42 ino kernel: [3.485088] tg3 :1e:00.0 ens1: renamed from eth0 Feb 11 09:16:44 ino kernel: [ 101.292206] tg3 :1e:00.0 ens1: Link is up at 100 Mbps, full duplex Feb 11 09:16:44 ino kernel: [ 101.292208] tg3 :1e:00.0 ens1: Flow control is on for TX and on for RX After that, i've addedd a second nic, pcie slot, that get detected but 'strangely' renamed: root@ino:~# grep e1000e /var/log/kern.log Feb 11 09:16:42 ino kernel: [3.451250] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k Feb 11 09:16:42 ino kernel: [3.451251] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. Feb 11 09:16:42 ino kernel: [3.451430] e1000e :20:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode Feb 11 09:16:42 ino kernel: [3.502028] e1000e :20:00.0 :20:00.0 (uninitialized): registered PHC clock Feb 11 09:16:42 ino kernel: [3.556137] e1000e :20:00.0 eth0: (PCI Express:2.5GT/s:Width x1) 2c:27:d7:14:9b:67 Feb 11 09:16:42 ino kernel: [3.556139] e1000e :20:00.0 eth0: Intel(R) PRO/1000 Network Connection Feb 11 09:16:42 ino kernel: [3.556182] e1000e :20:00.0 eth0: MAC: 3, PHY: 8, PBA No: G17305-003 Feb 11 09:16:42 ino kernel: [3.557393] e1000e :20:00.0 rename3: renamed from eth0 Looking at: https://wiki.debian.org/NetworkInterfaceNames Strangeness remain, look at: root@ino:~# udevadm test-builtin net_id /sys/class/net/ens1 Load module index Parsed configuration file /usr/lib/systemd/network/99-default.link Created link configuration context. Using default interface naming scheme 'v240'. ID_NET_NAMING_SCHEME=v240 ID_NET_NAME_MAC=enx9c8e997b86d9 ID_OUI_FROM_DATABASE=Hewlett Packard ID_NET_NAME_PATH=enp30s0 ID_NET_NAME_SLOT=ens1 Unload module index Unloaded link configuration context. root@ino:~# udevadm test-builtin net_id /sys/class/net/rename3 Load module index Parsed configuration file /usr/lib/systemd/network/99-default.link Created link configuration context. Using default interface naming scheme 'v240'. ID_NET_NAMING_SCHEME=v240 ID_NET_NAME_MAC=enx2c27d7149b67 ID_OUI_FROM_DATABASE=Hewlett Packard ID_NET_NAME_PATH=enp32s0 ID_NET_NAME_SLOT=ens1 Unload module index Unloaded link configuration context. AFAI've understood network names would have to be 'enp30s0' and 'enp32s0' respectively, not 'ens1' and 'rename3'... Why?! Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] bmc-watchdog curiosity
Mandi! Dietmar Maurer In chel di` si favelave... > In theory, a HW watchdog is considered more reliable than softdog. Just we are here... 'pve-ha-manager' is an alternative to 'watchdog', right? Looking at debian package seems so, to me: root@thor:~# apt-cache show pve-ha-manager Package: pve-ha-manager Architecture: amd64 Version: 2.0-9 Priority: optional Section: perl Maintainer: Proxmox Support Team Installed-Size: 227 Depends: libjson-perl, libpve-common-perl, pve-cluster (>= 3.0-17), systemd, init-system-helpers (>= 1.18~), perl, libc6 (>= 2.7) Conflicts: watchdog [...] but i've not seens this reported on documentation... wiki or manuals. Also, 'watchdog' deaemon do other things, like reboot if load go over a theresold and so on, all things that probably are BAD in a virtualized environment. But probably sysadmin are used to configure it, so... i think it worth a note. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE6 and PCI(e) Passthrough...
Mandi! Dmitry Petuhov In chel di` si favelave... > It may be disabled in BIOS/UEFI. Bingo! It was disabled in BIOS! root@ino:~# find /sys/kernel/iommu_groups/ -type l /sys/kernel/iommu_groups/7/devices/:00:1c.2 /sys/kernel/iommu_groups/15/devices/:1e:00.0 /sys/kernel/iommu_groups/5/devices/:00:1c.0 /sys/kernel/iommu_groups/13/devices/:11:08.0 /sys/kernel/iommu_groups/13/devices/:10:00.0 /sys/kernel/iommu_groups/3/devices/:00:10.0 /sys/kernel/iommu_groups/3/devices/:00:10.1 /sys/kernel/iommu_groups/11/devices/:00:1e.0 /sys/kernel/iommu_groups/1/devices/:00:03.0 /sys/kernel/iommu_groups/8/devices/:00:1c.3 /sys/kernel/iommu_groups/6/devices/:00:1c.1 /sys/kernel/iommu_groups/14/devices/:1c:00.0 /sys/kernel/iommu_groups/4/devices/:00:1a.0 /sys/kernel/iommu_groups/12/devices/:00:1f.2 /sys/kernel/iommu_groups/12/devices/:00:1f.0 /sys/kernel/iommu_groups/12/devices/:00:1f.3 /sys/kernel/iommu_groups/2/devices/:00:08.0 /sys/kernel/iommu_groups/2/devices/:00:08.3 /sys/kernel/iommu_groups/2/devices/:00:08.1 /sys/kernel/iommu_groups/2/devices/:00:08.2 /sys/kernel/iommu_groups/10/devices/:00:1d.0 /sys/kernel/iommu_groups/0/devices/:00:00.0 /sys/kernel/iommu_groups/9/devices/:00:1c.4 Thanks! -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE6 and PCI(e) Passthrough...
Mandi! leesteken--- via pve-user In chel di` si favelave... > > I'm missing something? Or simply my hardware is not supported? > Can you post the output from the following command? > find /sys/kernel/iommu_groups/ -type l root@ino:~# find /sys/kernel/iommu_groups/ -type l root@ino:~# none. So seems it is not supported... right? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] PVE6 and PCI(e) Passthrough...
I've done a fresh installation of PVE6 (6.1) on an old HP ProLiant ML110 G6 (CPU Intel(R) Xeon(R) CPU X3430@2.40GHz), and following: https://pve.proxmox.com/wiki/PCI(e)_Passthrough I've tried to enable PCI passthrough, that seems work: root@ino:~# grep -i iommu /var/log/kern.log Feb 2 14:27:38 ino kernel: [0.00] Command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-5.3.13-2-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on Feb 2 14:27:38 ino kernel: [0.075004] Kernel command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-5.3.13-2-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on Feb 2 14:27:38 ino kernel: [0.075119] DMAR: IOMMU enabled but still PVE web interface refuse me to put in passthrough a device (say: iommu not enabled) and the box reboot spontaneously under load (i'm copying files...) every half an hour, roughly. I'm missing something? Or simply my hardware is not supported? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] LVM autoactivation failed with multipath over iSCSI
Mandi! Stefan M. Radman via pve-user In chel di` si favelave... > My own workaround for the issue at hand is to delay the start of lvm2-pvscan > by a few seconds (see below). > That gives multipathd more than enough time to complete its job. > It solves the problem for me (at least until the next upgrade) but there > might be a smarter solution. Would be better to make 'lvm2-pvscan' service depends on 'multipathd'? Something like (NOT TESTED!) adding in: /etc/systemd/system/lvm2-pvscan.service.d/wait-multipath.conf the rows: [Unit] After=multipathd.service Wants=multipathd.service -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] VZdump: No such disk, but the disk is there!
Mandi! Gilberto Nunes In chel di` si favelave... > Any advice? Happen 'spot' also here; i'm convinced that, under some specific circumstances, eg, high load on the SAN, backup 'timeout' and the error reported is that, a bit misleading indeed. FYI. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] LVM autoactivation failed with multipath over iSCSI
Mandi! nada In chel di` si favelave... > root@mox11:~# grep santest /var/log/syslog > Jan 14 15:36:09 mox11 blkdeactivate[19162]: [LVM]: deactivating Volume > Group santest... skipping > Jan 14 15:39:36 mox11 lvm[2086]: Cannot activate LVs in VG santest while > PVs appear on duplicate devices. > Jan 14 15:39:36 mox11 lvm[2086]: Cannot activate LVs in VG santest while > PVs appear on duplicate devices. > Jan 14 15:39:36 mox11 lvm[2086]: 0 logical volume(s) in volume group > "santest" now active > Jan 14 15:39:36 mox11 lvm[2086]: santest: autoactivation failed. Mmm... seems to me that LVM get activated before multipath, and so they see multiple PVs (as effectively is). Never happened before, sorry... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] LVM autoactivation failed with multipath over iSCSI
Mandi! nada In chel di` si favelave... > the LVMThin VG and relevant LVs are NOT autoactivated > what am i missing ? Usually you need: node.startup = automatic in /etc/iscsi/iscsid.conf on *every* server of the pool (and do initrd recreation, of course). -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Watchdog in containers?
I've a container that, suddenly, consume all the RAM available, start to swap and load climb to 10+ . Clearly, i've to find the guilty process (probably it is Samba) but in the meantime... there's some sort of 'watchdog' for containers? I can safely install 'watchdog' on containers, disable /dev/watchdog and configure to do a reboot if load go too high? eg: max-load-1 = 24 max-load-5 = 18 max-load-15= 12 ? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] ZFS rpool grub rescue boot...
Mandi! Gilberto Nunes In chel di` si favelave... > I just installed Proxmox 6 in an HPE server, which has HP Smart Array P420i > and, unfortunately, this not so Smart Array doesn't give me the options to > make non-raid or HBA/IT Mode... > The Proxmox installer ran smoothly, but when try to boot, get this error I use that controller, but not with ZFS. Anyway seems that IS possible tu put the controller in HBA mode, see: https://www.youtube.com/watch?v=JuaezJd4C3I Probably you have the same effect: a) upgrading to the latest bios b) using hpssacli from a temp-installed linux distro, eg a USB key or an usb disk. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Container restore with pct, --rootfs syntiax?
Mandi! Fabian Grünbichler In chel di` si favelave... > > What are the correct syntiax? Thanks. > --rootfs STORAGE:SIZE_IN_GB > e.g., > --rootfs local:4 > see 'Storage Backed Mount Points' in > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pct_settings OK, i think i've had to read: NoteThe special option syntax STORAGE_ID:SIZE_IN_GB for storage backed mount point volumes will automatically allocate a volume of the specified size on the specified storage. E.g., calling pct set 100 -mp0 thin1:10,mp=/path/in/container will allocate a 10GB volume on the storage thin1 and replace the volume ID place holder 10 with the allocated volume ID. so the doc (and manpage) explain the configuration file format, not the pct commandline, right? Also, just i'm here: for VMs i can 'detach' additianal volumes, to prevent backup/restore to destroy them; why in LXC it is not possible, eg a restore 'destroy' all container volumes (and there's no way to detach it)? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Container restore with pct, --rootfs syntiax?
I need to 'resize' (shrink) a container, so i've done a backup, and following some gogle-fu and 'pct' manpage i've done: root@tma-18:~# pct restore 130 /mnt/pve/backup/dump/vzdump-lxc-130-2019_11_05-17_39_08.tar.lzo --rootfs volume=local,size=4G --force unable to parse volume ID 'local' finally i've done: root@tma-18:~# pct restore 130 /mnt/pve/backup/dump/vzdump-lxc-130-2019_11_05-17_39_08.tar.lzo --rootfs volume=local:4,size=4G --force Formatting '/var/lib/vz/images/130/vm-130-disk-1.raw', fmt=raw size=4294967296 [...] and work as expected, but seems very very strange to me... What are the correct syntiax? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] P2V of an XP box, VM ask for fwcfg driver...
Ok, it is a bit late to P2V a XP box, but... P2V done (on PVE 5.4), now the VMs ask at every boot to install the 'fwcfg' driver (qemufwcfg), but virtio-win CD 0.1.171 have only versions for win7+ OS. I can safely install driver for newer OS? Or where i can find (old?) driver for XP? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Strange behavior vzdump
Mandi! Gilberto Nunes In chel di` si favelave... > I am thing my problems comes after install > https://github.com/ayufan/pve-patches no, i've never used that patchset... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Strange behavior vzdump
Mandi! Gilberto Nunes In chel di` si favelave... > I have notice that vzdump options, maxfiles doesn't work properly. > I set --maxfiles to 10, but still it's hold old files... > For now, I add --remove 1, to the /etc/vzdump.conf, but, according to the > vzdump man page, default --remove set is 1, i.e., enable! > Why vzdump do not remove old backup, just when set maxfiles?? > Or even worst, if --remove 1 is the default options, why vzdump doesn't > work?? > Proxmox VE version 5.4 This make some noise on my ear... two clusters, one with ''traditional'' iSCSI SAN storage, one with Ceph. On Ceph one: root@hulk:~# ls /srv/pve/dump/ | grep \.lzo | cut -d '-' -f 1-3 | sort | uniq -c 1 vzdump-lxc-103 1 vzdump-lxc-105 1 vzdump-lxc-106 1 vzdump-lxc-109 3 vzdump-lxc-111 50 vzdump-lxc-114 49 vzdump-lxc-117 1 vzdump-qemu-104 3 vzdump-qemu-108 3 vzdump-qemu-113 1 vzdump-qemu-115 49 vzdump-qemu-116 My backup stategy is: + for some VM/LXC, daily backup (114, 116, 117 are 'daily') all day's week apart saturday. + for all VM/LXC, on saturday bacula pre-script that run the backup, and then bacula put on tape. the bacula 'pre' script do: /usr/bin/vzdump 117 -storage Backup -maxfiles 1 -remove -compress lzo -mode suspend -quiet -mailto c...@sv.lnf.it -mailnotification failure for every LXC/VM, and as you can see, delete old backup only for some VM/LXC, not all. 'backup' storage is defined as: nfs: Backup export /srv/pve path /mnt/pve/Backup server 10.27.251.11 content vztmpl,images,iso,backup,rootdir maxfiles 0 options vers=3,soft,intr Clearly, no error in logs. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Migrating 4->5, from hammer to luminous: some shortcut?
Mandi! Eneko Lacunza In chel di` si favelave... > Don't waste time :-) You are absolutely right. Cluster migrated. All went well! -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Migrating 4->5, from hammer to luminous: some shortcut?
> Why an intermediate passage via 'jessie ceph' packages is needed? Sorry, i totally misread my sketches i've done in july, before going to holiday. ;-( Reset. Restart. I've just upgraded, flawlessy, from hammer to jewel, using the Ceph debian repository. Now, next step (from jewel to luminous) suggest to modify repository and use PVE (NOT Ceph) jessie luminous repository, upgrade, and then upgrade PVE to 5. I suppose because, in a production environment, this minimize downtime and errors. But, because i can move most of the data to a temporary storage, how are the drawbacks of simply upgrading to PVE 5/luminous? The path 'jessie/4.4+jessie-Ceph/jewel -> jessie/4.4+jessie-PVE/luminous -> stretch/5+stretch-PVE/luminous' is a requisite or a suggestion? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Migrating 4->5, from hammer to luminous: some shortcut?
I've stared today my cluster migration, currently migrating from ceph hammer to jewel following: https://pve.proxmox.com/wiki/Ceph_Hammer_to_Jewel I've found a temporary storage device (a NAS), and so i've migrated (or stopped) roughly all VMs to that storage. Next step would be migrating to luminous using official debian packages from ceph: https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous Considering that i can move out (or stop) all Ceph traffic, it is needed to upgrade ceph and then upgrade PVE? Or i can have some shortcut, and upgrade directly to luminous using official PVE 5 repository? Why an intermediate passage via 'jessie ceph' packages is needed? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Again trouble, but this time with ext4/trim...
Ah, forgot to say. > 3) in december i've upgraded the cluster to PVE 5; filesystem > corruption desappear. > 4) some weeks ago i was forced ot reboot the cluster (to change an > UPS), and FS corruption reappeared. 5) a (unused, indeed) W2016 server that was installed month before, start to complain about corrupted 'EDB' (internal database) files. After removing the 'discard' options on disk (again, on main/OS disk; there's a second 'DATA' disk with discard enabled), problem desappear. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Again trouble, but this time with ext4/trim...
Mandi! Fabian Grünbichler In chel di` si favelave... > are you sure this storage box supports fstrim at all? thin provisioning > and trim/discard often go together, but they don't have to.. Whai i can say is: 1) In the past, i've done 'direct iSCSI assignment' to some VM, eg the VM mount the iSCSI volume, and i can trim them flawlessy. 2) in autumn, i've created a new set ov VMs, where i've abandoned direct iSCSI mount and i use PVE volumes instead. Clearly, i've done some test for trim/discard, verifying that space get correctly reclaimed. After some weeks, i've suffered ext4 corruption as now. 3) in december i've upgraded the cluster to PVE 5; filesystem corruption desappear. 4) some weeks ago i was forced ot reboot the cluster (to change an UPS), and FS corruption reappeared. For now, i've removed the 'discard=on' options from volumes that have ext4 FS. The crazy things is that volumes formatted in XFS, residing on the same SAN, seems does not suffer at all of FS corruption. If i can do some test, please, say me. > quick googling only showed some vmware specific integration to 'reclaim > no longer used space of thin provisioned volumes', but no explicit > mention of discard support. Ahem, but trim/discard are not the same things? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Again trouble, but this time with ext4/trim...
> In that servers i've also some other FS, but the ext4 ones are low > varying, mounted RO or noatime, but also some FS in XFS fs, that seems > does not suffer. I've disabled 'discard' for /dev/sda in both server, and keeped it for /dev/sdb (squid spool, ext4) and /dev/sdc (XFS filesystem). I've also added a 'randomization' for the fstrim task, but still: Jul 29 01:14:10 vdmsv1 systemd[1]: Starting Discard unused blocks... Jul 29 01:14:14 vdmsv1 kernel: [455671.833357] EXT4-fs error (device sdb1): ext4_validate_block_bitmap:386: comm fstrim: bg 97: bad block bitmap checksum Jul 29 01:14:14 vdmsv1 kernel: [455671.859823] EXT4-fs warning (device sdb1): ext4_trim_all_free:5130: Error -74 loading buddy information for 97 Jul 29 01:14:14 vdmsv1 fstrim[23986]: fstrim: /var/spool/squid: FITRIM ioctl failed: Messaggio errato Jul 29 01:15:02 vdmsv1 CRON[24072]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Main process exited, code=exited, status=64/n/a Jul 29 01:15:29 vdmsv1 systemd[1]: Failed to start Discard unused blocks. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.timer: Adding 2h 8min 41.708255s random time. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Unit entered failed state. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.timer: Adding 3h 56min 32.453953s random time. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Failed with result 'exit-code'. So seems that i've some trouble with 'discard', at least with ext4 FS... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Cluster does not start, corosync timeout...
ying to acquire cfs lock
'file-replication_cfg' ...
Jul 3 20:00:09 pvecn1 pvesr[4068]: error with cfs lock 'file-replication_cfg':
no quorum!
Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Start operation timed out.
Terminating.
Jul 3 20:00:09 pvecn1 systemd[1]: pvesr.service: Main process exited,
code=exited, status=13/n/a
Jul 3 20:00:09 pvecn1 systemd[1]: Failed to start Proxmox VE replication
runner.
Jul 3 20:00:09 pvecn1 systemd[1]: pvesr.service: Unit entered failed state.
Jul 3 20:00:09 pvecn1 systemd[1]: pvesr.service: Failed with result
'exit-code'.
Jul 3 20:00:09 pvecn1 systemd[1]: Failed to start Corosync Cluster Engine.
Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Unit entered failed state.
Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Failed with result
'timeout'.
Jul 3 20:00:09 pvecn1 systemd[1]: Starting PVE API Daemon...
Note that i'm not using pvesr, so all the warning about it can be
safely ignored.
Also, 'Backup' storage is a NFS storage in one of the node, probably
still booting...
> > But... some host in the cluster missed from /etc/hosts: this suffices
> > to have corosync not to start correctly?
> depends on the config, as you stated yourself with multicast it normally
> won't be an issue, but maybe the switch had some issues with multicast
> initially
> after the power outage, as a guess.
I've tried to check multipath with 'omping' now (and i'm sure i've had
checked setting up the claster), and works.
So i'm not sure how multicast can 'not work initially', and subsequent
start to work...
> can you please post your corosync.conf ?
Sure!
root@pvecn1:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: pvecn2
nodeid: 2
quorum_votes: 1
ring0_addr: pvecn2
}
node {
name: pvecn1
nodeid: 1
quorum_votes: 1
ring0_addr: pvecn1
}
node {
name: pvecn3
nodeid: 3
quorum_votes: 1
ring0_addr: pvecn3
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: CONEGLIANO
config_version: 3
ip_version: ipv4
secauth: on
version: 2
interface {
bindnetaddr: 10.10.1.50
ringnumber: 0
}
}
Thanks.
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Cluster does not start, corosync timeout...
We had a major power outgage here, and our cluster have some trouble on
restart. The worster was:
Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync Cluster Engine
('2.4.4-dirty'): started and ready to provide service.
Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync built-in features:
dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie relro
bindnow
Jul 3 19:58:40 pvecn1 corosync[3443]: notice [MAIN ] Corosync Cluster
Engine ('2.4.4-dirty'): started and ready to provide service.
Jul 3 19:58:40 pvecn1 corosync[3443]: info[MAIN ] Corosync built-in
features: dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie
relro bindnow
Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Start operation timed
out. Terminating.
Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Unit entered failed state.
Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Failed with result
'timeout'.
after fiddling a bit, we solved all the stuff and cluster goes back as
normal.
But... some host in the cluster missed from /etc/hosts: this suffices
to have corosync not to start correctly?
Looking at docs (https://pve.proxmox.com/pve-docs/pve-admin-guide.html):
While it’s often common use to reference all other nodenames in /etc/hosts
with their IP this is not strictly necessary for a cluster, which normally uses
multicast, to work. It maybe useful as you then can connect from one node to
the other with SSH through the easier to remember node name.
this mean i've not multicast correctly working? I was sure i had...
Thanks.
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Trouble with Win2016 server...
> but nothing seems have changed, still i got errors on > %windir%\security\database\secedit.sdb. Ok, still this is a bit mysterious, but at least have some more info. I've simply disabled 'discard' on the bootdisk, now the VM are: agent: 1 boot: dc bootdisk: scsi0 cores: 2 ide0: none,media=cdrom memory: 8192 name: vwssv1 net0: virtio=DA:CA:FA:50:8C:01,bridge=vmbr0 numa: 0 ostype: win10 scsi0: VM1:vm-124-disk-0,size=100G scsi1: DATA2:vm-124-disk-0,backup=0,discard=on,size=500G scsihw: virtio-scsi-pci smbios1: uuid=169d7e42-8a69-4f96-aeb1-df604af44bd1 sockets: 1 and windows recreate correctly '%windir%\security\database\secedit.sdb' file and stop complain at all. Still i had some more DB corrupted (for 'Administrator' user) but i've removed the profile and let windows recreate it from scratch, and clearly also this error deasppeard. So, seems there's some trouble with 'discard', at least in my setup... Still i've the 'scratch' VM i used for test, so if can be useful, i can provide some more info... Say me... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Again trouble, but this time with ext4/trim...
Trying to fix my trouble with Win2016, last week i've reboot completely my PVE 5 cluster. Cluster use a SAN (HP MSA 1040), thin storage mode, used as LVM in PVE, eg: lvm: VM0 vgname VM0 content images,rootdir shared 1 Because the underlying storage are thin, i've do some test in the past and enable 'discard' on PVE volumes, verifying that storage space get effectively freed. After that reboot, at every execution of 'fstrim' (via systemd timer) in two different VMs i got: Jul 1 00:00:03 vdmsv2 kernel: [542165.662171] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 647, block bitmap and bg descriptor inconsistent: 21279 vs 32768 free clusters Jul 1 00:00:03 vdmsv2 kernel: [542165.695216] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 648, block bitmap and bg descriptor inconsistent: 12011 vs 32768 free clusters Jul 1 00:00:03 vdmsv2 kernel: [542165.705122] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 649, block bitmap and bg descriptor inconsistent: 14971 vs 32768 free clusters Jul 1 00:00:03 vdmsv2 kernel: [542165.711123] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 650, block bitmap and bg descriptor inconsistent: 12194 vs 32768 free clusters Jul 1 00:00:06 vdmsv2 kernel: [542169.243447] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. Jul 1 00:00:06 vdmsv2 kernel: [542169.244995] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. (sda6 is /var) Jul 1 00:01:05 vdmsv1 fstrim[3806]: fstrim: /var/spool/squid: FITRIM ioctl failed: Messaggio errato Jul 1 00:01:05 vdmsv1 kernel: [9519779.612188] EXT4-fs error (device sdb1): ext4_validate_block_bitmap:386: comm fstrim: bg 506: bad block bitmap checksum Jul 1 00:01:05 vdmsv1 kernel: [9519779.636314] EXT4-fs warning (device sdb1): ext4_trim_all_free:5130: Error -74 loading buddy information for 506 Jul 1 00:01:08 vdmsv1 kernel: [9519782.441786] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 181, block bitmap and bg descriptor inconsistent: 32767 vs 32768 free clusters Jul 1 00:01:08 vdmsv1 kernel: [9519782.448571] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. Jul 1 00:01:09 vdmsv1 kernel: [9519782.973496] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 278, block bitmap and bg descriptor inconsistent: 32749 vs 32768 free clusters Jul 1 00:01:09 vdmsv1 kernel: [9519783.034358] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 325, block bitmap and bg descriptor inconsistent: 32740 vs 32768 free clusters Jul 1 00:01:09 vdmsv1 kernel: [9519783.238781] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. (sdb1 is /var/spool/squid and sda6 is /var) In that servers i've also some other FS, but the ext4 ones are low varying, mounted RO or noatime, but also some FS in XFS fs, that seems does not suffer. Every time i run 'fstrim -av' by hand i've never catch an error, so seems to me that trouble happen whene the system are under high IO (as when run many weekly tasks). For now, i disable fstrim timer. But seeking some help... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Trouble with Win2016 server...
Mandi! Marco Gaiarin
In chel di` si favelave...
> Things seems get worster.
Yesterday server do another spontaneous reboot:
/var/log/syslog.1:Jun 27 17:38:10 vwssv1
microsoft-windows-windowsupdateclient[info] 44 NT AUTHORITY\SYSTEM Windows
Update ha avviato il download di un aggiornamento.
/var/log/syslog.1:Jun 27 17:38:10 vwssv1
microsoft-windows-windowsupdateclient[info] 44 NT AUTHORITY\SYSTEM Windows
Update ha avviato il download di un aggiornamento.
/var/log/syslog.1:Jun 27 17:42:48 vwssv1 eventlog[error] 6008 Precedente
arresto del sistema inatteso a 17:38:53 su?27/?06/?2019.
/var/log/syslog.1:Jun 27 17:42:48 vwssv1 eventlog[info] 6009 Microsoft (R)
Windows (R) 10.00. 14393Multiprocessor Free.
/var/log/syslog.1:Jun 27 17:42:48 vwssv1 eventlog[info] 6005 Il servizio
Registro eventi stato avviato.
/var/log/syslog.1:Jun 27 17:42:48 vwssv1 eventlog[info] 6013 Il periodo di
disponibilit del sistema 40 secondi.
/var/log/syslog.1:Jun 27 17:42:48 vwssv1 bugcheck[error] 1005 Impossibile
creare un file di dump ridotto dal file di dumpcompleto.
/var/log/syslog.1:Jun 27 17:42:48 vwssv1 bugcheck[error] 1001 Il computer
stato riavviato da un controllo errori.Controllo errori:0x001a
(0x003f, 0x967b, 0x5980f5d6,
0x967b). I dettagli sono stati salvati in: C:\Windows\MEMORY.DMP.ID
segnalazione: ----.
/var/log/syslog.1:Jun 27 17:42:07 vwssv1
microsoft-windows-kernel-general[info] 12 NT AUTHORITY\SYSTEM Codice di accesso
non valido.
/var/log/syslog.1:Jun 27 17:42:07 vwssv1 microsoft-windows-kernel-boot[info]
153 NT AUTHORITY\SYSTEM Elenco DosMuxSemWait non corretto.
/var/log/syslog.1:Jun 27 17:42:07 vwssv1 microsoft-windows-kernel-boot[info]
20 NT AUTHORITY\SYSTEM Impossibile trovare il dispositivo specificato.
/var/log/syslog.1:Jun 27 17:42:07 vwssv1 microsoft-windows-kernel-boot[info]
27 NT AUTHORITY\SYSTEM L'unit non riesce trovare il settore richiesto.
/var/log/syslog.1:Jun 27 17:42:07 vwssv1 microsoft-windows-kernel-boot[info]
25 NT AUTHORITY\SYSTEM L'unit non riesce a trovare un'area o una traccia
deldisco specifica.
/var/log/syslog.1:Jun 27 17:42:07 vwssv1 microsoft-windows-kernel-boot[info]
18 NT AUTHORITY\SYSTEM File esauriti.
/var/log/syslog.1:Jun 27 17:42:07 vwssv1 microsoft-windows-kernel-boot[info]
32 NT AUTHORITY\SYSTEM Impossibile accedere al file. Il file utilizzato da
unaltro processo.
/var/log/syslog.1:Jun 27 17:42:08 vwssv1 microsoft-windows-filtermanager[info]
6 NT AUTHORITY\SYSTEM Caricamento del filtro del file system 'Wof'
(10.0,2017-10-09T03:58:20.0Z) e registrazione conGestione filtri
riusciti.
/var/log/syslog.1:Jun 27 17:42:08 vwssv1 microsoft-windows-filtermanager[info]
6 NT AUTHORITY\SYSTEM Caricamento del filtro del file system 'WdFilter'
(10.0,2050-11-17T22:31:54.0Z) e registrazione conGestione filtri
riusciti.
/var/log/syslog.1:Jun 27 17:42:54 vwssv1 winrm[info] 10148 Servizio Gestione
remota Windows: in corso l'ascoltodelle richieste WS-Management. Azione
utente Per visualizzare gli IP specifici su cui in ascolto ilservizio
Gestione remota Windows, utilizzare il comandoseguente: winrm enumerate
winrm/config/listener
/var/log/syslog.1:Jun 27 17:42:10 vwssv1 microsoft-windows-ntfs[info] 98 NT
AUTHORITY\SYSTEM C: \Device\HarddiskVolume2 0
/var/log/syslog.1:Jun 27 17:42:33 vwssv1 microsoft-windows-filtermanager[info]
6 NT AUTHORITY\SYSTEM Caricamento del filtro del file system 'FileCrypt'
(10.0,2018-08-30T22:44:27.0Z) e registrazione conGestione filtri
riusciti.
/var/log/syslog.1:Jun 27 17:42:33 vwssv1 microsoft-windows-filtermanager[info]
6 NT AUTHORITY\SYSTEM Caricamento del filtro del file system 'npsvctrig'
(10.0,2016-07-16T04:28:33.0Z) e registrazione conGestione filtri
riusciti.
/var/log/syslog.1:Jun 27 17:42:33 vwssv1
microsoft-windows-kernel-power[success] 41 NT AUTHORITY\SYSTEM 26 0x3f 0x967b
0x5980f5d6 0x967b 0 0 0 0 false 0 0
/var/log/syslog.1:Jun 27 17:42:33 vwssv1 microsoft-windows-kernel-power[info]
172 NT AUTHORITY\SYSTEM 2 6
/var/log/syslog.1:Jun 27 17:42:34 vwssv1 microsoft-windows-ntfs[info] 98 NT
AUTHORITY\SYSTEM F: \Device\HarddiskVolume3 0
/var/log/syslog.1:Jun 27 17:42:34 vwssv1 microsoft-windows-ntfs[info] 98 NT
AUTHORITY\SYSTEM \\?\Volume{b077600e----1000}
\Device\HarddiskVolume1 0
/var/log/syslog.1:Jun 27 17:42:35 vwssv1
microsoft-windows-kernel-general[info] 16 NT AUTHORITY\SYSTEM Impossibile
rimuovere la directory.
/var/log/syslog.1:Jun 27 17:42:35 vwssv1
microsoft-windows-kernel-general[info] 16 NT AUTHORITY\SYSTEM Impossibile
rimuovere la directory.
/var/log/syslog.1:Jun 27 17:42:35 vwssv1
microsoft-windows-kernel-general[info] 16 NT AUTHORITY\SYSTEM Impossibile
rimuovere la directory.
/var/log/syslog.1:Jun 27 17:42:38 vwssv1
microsoft-windows-kernel-general[info] 15 NT AUTHORITY\SYSTEM Impossibile
trovare l'u
Re: [PVE-User] Trouble with Win2016 server...
Mandi! Gianni Milo In chel di` si favelave... > Could you try applying KB4503294 > as it appears to have fixed some of the issues in KB4503267 (if they are > related to yours of course)... > https://support.microsoft.com/en-ca/help/4503294/windows-10-update-kb4503294 Things seems get worster. Yesterday some other updates get installed: Jun 20 13:00:29 vwssv2 microsoft-windows-windowsupdateclient[info] 43 NT AUTHORITY\SYSTEM Installazione avviata. Windows ha avviato l'installazionedel seguente aggiornamento:2019-06 Aggiornamento cumulativo per Windows Server 2016 per sistemi basati su x64 (KB4503267) Jun 20 15:35:25 vwssv2 microsoft-windows-windowsupdateclient[info] 43 NT AUTHORITY\SYSTEM Installazione avviata. Windows ha avviato l'installazionedel seguente aggiornamento:Aggiornamento per la piattaforma antimalware di Windows Defender Antivirus - KB4052623 (versione 4.18.1905.4) Jun 20 16:07:57 vwssv2 microsoft-windows-windowsupdateclient[info] 43 NT AUTHORITY\SYSTEM Installazione avviata. Windows ha avviato l'installazionedel seguente aggiornamento:2019-06 Aggiornamento cumulativo per Windows Server 2016 per sistemi basati su x64 (KB4503294) as you can see, also KB4503294. After that, machine was rebooted. After some hour, exactly the same error of the first server popups: Jun 20 20:08:38 vwssv2 esent[error] 477 services (544) Non stato possibile leggere l'intervallodi log dal file "C:\WINDOWS\Security\Database\edb.log"all'offset 4096 (0x1000) per 4096 (0x1000)byte a causa di una mancata corrispondenza del checksumdell'intervallo. Checksum previsto:170325518057664 (0x9ae900c0). Checksum effettivo:4545124189 (0x10ee9175d). L'operazione di lettura non verreffettuata con errore -501 (0xfe0b). Se tale condizionepersiste, ripristinare il file di log da un backupprecedente. Jun 20 20:08:38 vwssv2 esent[error] 465 services (544) Sono stati rilevati dati danneggiati duranteil recupero dati del file di logC:\WINDOWS\Security\Database\edb.log. Il record che original'errore nel checksum si trova nella seguente posizione:END. I primi dati non corrispondenti al criterio dicompilazione del file di log sono comparsi nel settoreisec 1 reason 6. Il file di log danneggiato e non utilizzabile. Jun 20 20:08:38 vwssv2 esent[error] 477 services (544) Non stato possibile leggere l'intervallodi log dal file "C:\WINDOWS\Security\Database\edb.log"all'offset 4096 (0x1000) per 4096 (0x1000)byte a causa di una mancata corrispondenza del checksumdell'intervallo. Checksum previsto:5968412998358794349 (0x52d4106d). Checksumeffettivo: 7609752797 (0x1c5939cdd). L'operazione dilettura non verr effettuata con errore -501 (0xfe0b).Se tale condizione persiste, ripristinare il file di log daun backup precedente. Jun 20 20:08:38 vwssv2 esent[error] 465 services (544) Sono stati rilevati dati danneggiati duranteil recupero dati del file di logC:\WINDOWS\Security\Database\edb.log. Il record che original'errore nel checksum si trova nella seguente posizione:END. I primi dati non corrispondenti al criterio dicompilazione del file di log sono comparsi nel settoreisec 1 reason 6. Il file di log danneggiato e non utilizzabile. Jun 20 20:08:38 vwssv2 scecli[warning] 1202 Criteri di sicurezza propagati con avviso.0x10d9 : Impossibile leggere o scrivere nel database. possibile trovare assistenza avanzata per questo problema nel sito http://support.microsoft.com. Eseguire una ricerca del codice evento 1202. Jun 20 20:08:38 vwssv2 scecli[error] 1005 Alcuni database JET sono danneggiati. Eseguire esentutl /gper verificare l'integrit del database di sicurezza%windir%\security\Database\secedit.sdb. Se il database danneggiato, tentare un recupero a caldo eseguendo esentutl/r nella directory %windir%\security. Se il recupero acaldo non riesce, provare a riparare il database tramiteesentutl /p in %windir%\security\Database\secedit.sdb,quindi eliminare i file di registro in %windir%\security.Errore durante l'apertura di alcuni database di sicurezza, tra cui %windir%\security\database\secedit.sdb. So seems that one of KB4503267 and/or KB4503294 can be the cause... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Trouble with Win2016 server...
Mandi! Mike Ryan In chel di` si favelave... > KB4503267 is known to have issues with Event Viewer, BitLocker secured > devices and Dell iSCSI logons. > https://www.zdnet.com/article/windows-latest-patches-crash-event-viewer-but-microsoft-has-temporary-fix/ > > https://www.reddit.com/r/HyperV/comments/bzv3gc/kb4503267_201906_hyperv_wbitlocker_still_broken/ > > https://www.dell.com/community/EqualLogic/EQL-iSCSI-login-failures-after-MS-patches-KB4497934-KB4499177/td-p/7314811 > Interesting. But none of theese seems related... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Trouble with Win2016 server...
Mandi! Gianni Milo In chel di` si favelave... > Can you please post the VM config file ? Sure! agent: 1 boot: dcn bootdisk: scsi1 cores: 2 ide0: none,media=cdrom memory: 8192 name: vwssv2 net0: virtio=E6:DE:2C:88:BB:4D,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsi1: VM1:vm-100-disk-0,discard=on,size=100G scsihw: virtio-scsi-pci smbios1: uuid=6582aaa8-761a-4013-90fb-7b6667719d70 sockets: 1 unused0: DATA2:vm-100-disk-0 vmgenid: 784ef60a-b5ee-40c1-a5aa-7bcadb0091f4 > Also, which type of storage are you using for storing the VM disk(s)? On, working, 4.4: ceph. On, troubled, 5.4: iSCSI/LVM > Sounds strange that the event log security database gets corrupted, perhaps > RAM or disk issues? The machine spotted an id '41' 'Kernel-Power' event, and a: Nome registro: System Origine: Microsoft-Windows-WER-SystemErrorReporting Data: 19/06/2019 14:34:40 ID evento: 1001 Categoria attività:Nessuna Livello: Errore Parole chiave: Classico Utente:N/D Computer: VWSSV2 Descrizione: Il computer è stato riavviato da un controllo errori. Controllo errori: 0x001a (0x003f, 0x421f, 0x78ac0f51, 0x421f). I dettagli sono stati salvati in: C:\Windows\MEMORY.DMP. ID segnalazione: ----. XML evento: http://schemas.microsoft.com/win/2004/08/events/event;> 1001 0 2 0 0 0x80 3080 System VWSSV2 0x001a (0x003f, 0x421f, 0x78ac0f51, 0x421f) C:\Windows\MEMORY.DMP ---- > Have you tried switching to IDE disk mode temporarily and see if the issue > is reproducible there as well ? No. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Trouble with Win2016 server...
I manage two sites, one with a PVE 4.4 cluster and one with a, fresh updated, 5.4 cluster. In both sites i've instaleld a Win2016 Server VMs, both site with the same configuration (both have to manage a Kaspersky Security Center). VMs are configured in the same way, following the wiki: VirtIO drivers (stable version, 0.1.141) for disk and net, agent, ballon. In 4.4 site, VM are rock solid. In 5.4 site, VM, after installing the KB4503267 (latest Win2016 cumulative upgrade) the VMs crash, mostly of the time rollback the upgrade but in 3-5 tries finally the update go online. But in this way, i've got a DB corruption on the VM that i've not found a way to fix (sorry for italian, but blame microsoft ;): Jun 20 07:07:56 vwssv1 esent[error] 477 services (552) Non stato possibile leggere l'intervallodi log dal file "C:\WINDOWS\Security\Database\edb.log"all'offset 4096 (0x1000) per 4096 (0x1000)byte a causa di una mancata corrispondenza del checksumdell'intervallo. Checksum previsto:15481368538579046 (0x37003900610066). Checksum effettivo:6282815003 (0x1767c261b). L'operazione di lettura non verreffettuata con errore -501 (0xfe0b). Se tale condizionepersiste, ripristinare il file di log da un backupprecedente. Jun 20 07:07:56 vwssv1 esent[error] 465 services (552) Sono stati rilevati dati danneggiati duranteil recupero dati del file di logC:\WINDOWS\Security\Database\edb.log. Il record che original'errore nel checksum si trova nella seguente posizione:END. I primi dati non corrispondenti al criterio dicompilazione del file di log sono comparsi nel settoreisec 1 reason 6. Il file di log danneggiato e non utilizzabile. Jun 20 07:07:56 vwssv1 esent[error] 477 services (552) Non stato possibile leggere l'intervallodi log dal file "C:\WINDOWS\Security\Database\edb.log"all'offset 4096 (0x1000) per 4096 (0x1000)byte a causa di una mancata corrispondenza del checksumdell'intervallo. Checksum previsto:4854704636100944 (0x113f5401530150). Checksum effettivo:922407576148658820 (0xccd0ccdaa784a84). L'operazione dilettura non verr effettuata con errore -501 (0xfe0b).Se tale condizione persiste, ripristinare il file di log daun backup precedente. Jun 20 07:07:56 vwssv1 esent[error] 465 services (552) Sono stati rilevati dati danneggiati duranteil recupero dati del file di logC:\WINDOWS\Security\Database\edb.log. Il record che original'errore nel checksum si trova nella seguente posizione:END. I primi dati non corrispondenti al criterio dicompilazione del file di log sono comparsi nel settoreisec 1 reason 6. Il file di log danneggiato e non utilizzabile. Jun 20 07:07:56 vwssv1 scecli[error] 1005 Alcuni database JET sono danneggiati. Eseguire esentutl /gper verificare l'integrit del database di sicurezza%windir%\security\Database\secedit.sdb. Se il database danneggiato, tentare un recupero a caldo eseguendo esentutl/r nella directory %windir%\security. Se il recupero acaldo non riesce, provare a riparare il database tramiteesentutl /p in %windir%\security\Database\secedit.sdb,quindi eliminare i file di registro in %windir%\security.Errore durante l'apertura di alcuni database di sicurezza, tra cui %windir%\security\database\secedit.sdb. After that, i've tried to install a second test VM, configured exactly as the first, following verbatim the proxmox wiki, and as the other i've got the same result: machine crash after installation of KB4503267. I've not got DB corruption as the first, but, for example, if i go to the windows update history, it is empty (while i've installed at least KB4103720 and KB4503537). Probably it is not a proxmox trouble (qemu? kvm? VirtIO?), but really i don't know where to start a search. A simple 'KB4503267 VirtIO' google search lead to nothing... Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Backup fail with 'no disk', but manually works!
> A backup of a VM fail, every week, with: I come back to this thread, with more info. No, seems was not a partucular VM trouble, but a general one. And was not my backup script, but the 'plain' VM backup configured by web interface. Simply seems that VM backup, when run within the 'in-VM' file backup (bacula), load too much the cluster and so the spotted failure. I want to try to limit the bandwidth used in backup, eg '-bwlimit' option of 'vzdump', but seems that in web interface there's no way to set a bandwidth limit in backup tasks. I need to modify /etc/vzdump.conf? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Backup fail with 'no disk', but manually works!
I've setup a script, for a bacula backup setup, that weekly dump VMs/LXCs, cleanup the backup dir anche then put VMs on tape. A backup of a VM fail, every week, with: 108: 2019-02-08 20:19:04 INFO: Starting Backup of VM 108 (qemu) 108: 2019-02-08 20:19:04 INFO: status = running 108: 2019-02-08 20:19:28 INFO: update VM 108: -lock backup 108: 2019-02-08 20:19:28 INFO: VM Name: dixie 108: 2019-02-08 20:19:28 INFO: include disk 'virtio0' 'VM0:vm-108-disk-1' 100G 108: 2019-02-08 20:20:12 ERROR: Backup of VM 108 failed - no such volume 'VM0:vm-108-disk-1' note that: a) disk exist; disk reside on the same SAN with all my other VMs, that the same script backup without trouble. b) i've also a ''normal'' PVE backup tasks that do a dump of that VMs every day, and works. c) if i try a manual backup (from PVE web interface), work. The script do simply: vzdump 108 -storage Backup -maxfiles 1 -remove -compress lzo -mode snapshot -quiet -mailto ced -mailnotification failure -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Backup stalls, on server where it is not needed to run...
> I've tried to restart nfs-common in a client, and nfs-kernel-server in > the server but seems does not change. Ok, STOPping and the STARTing nfs-kernel-server seems have resolved the issue, or at least after some hours all accumulated tasks go away. On a node corosinc get crazy: Jan 23 22:29:38 thor pmxcfs[30679]: [status] notice: cpg_send_message retried 59 times Jan 23 22:29:39 thor pmxcfs[30679]: [status] notice: cpg_send_message retry 10 but a restart of 'pve-cluster' fixed that. FYI. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Backup stalls, on server where it is not needed to run...
I'v a 5 node cluster of proxmox 4.4. I've setup a backup task that do some daily backup of some VM. The backup task use a storage that is phisically on one of the server, mounted to all other server via NFS. In the server where the task actually have the NFS share, the task run without trouble. In all the other 4 servers, stalls: root@thor:~# ps aux | grep vzdump root 3748 0.0 0.0 4340 728 ?Ss Jan20 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 3749 0.0 0.1 236772 65620 ?DJan20 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 6596 0.0 0.0 4340 764 ?Ss Jan15 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 6597 0.0 0.1 236852 65644 ?DJan15 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 7240 0.0 0.0 12736 2120 pts/5S+ 18:44 0:00 grep vzdump root 15055 0.0 0.0 4340 764 ?Ss Jan21 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 15056 0.0 0.1 236780 65532 ?DJan21 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 18046 0.0 0.0 4340 724 ?Ss Jan16 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 18047 0.0 0.1 236804 65504 ?DJan16 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 25072 0.0 0.0 4340 724 ?Ss Jan22 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 25073 0.0 0.1 236716 65632 ?DJan22 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 29237 0.0 0.0 4340 760 ?Ss Jan17 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 29238 0.0 0.1 236812 65540 ?DJan17 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 57286 0.0 0.0 4340 816 ?Ss Jan18 0:00 /bin/sh -c vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 root 57287 0.0 0.1 236772 65716 ?DJan18 0:00 /usr/bin/perl -T /usr/bin/vzdump 106 107 108 113 111 114 116 --mailnotification failure --mode snapshot --mailto c...@pp.lnf.it --compress lzo --storage Backup --quiet 1 I've tried to restart nfs-common in a client, and nfs-kernel-server in the server but seems does not change. Processes are unkillable, even with 'kill -9'. Boh... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
I come back to this thread. > OK, with 16GB of ram 5 days passed without FS errors. I've upgraded SAN firmware to the latest available, then upgrade the cluster from PVE 4.4 to latest 5 version. Today is 2 weeks without filesystem corruption happens. Previous interval was lesser then 5 days, so i suppose that i've fixed that trouble. FYI. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE 4 -> 5, multipath differences?
Mandi! Sten Aus
In chel di` si favelave...
> As this is my third storage for not critical data, I don't worry much, but I
> haven't checked if two paths (prio=50) get more load than other two (prio=10).
Apart from the prio, the differences between jessie multipath is that
in jessie all path are active (and hwhandler is '0'), while in stretch
hwhandler is 'ALUA' and only two path are active.
Looking at SCSI path seems al correct: the path active are exactly the
path where the volume in the SAN reside.
Looking at code (git master branch) of multipath-tools seems that
hardware get correctly identified.
So, probably is really the right thing.
But also, i've noted that for some strange mean the blacklist in multipath
does not work as expected. I've:
blacklist {
wwid .*
}
but still i've log full of:
Jan 7 09:55:20 ashpool multipathd[470]: sda: spurious uevent, path not
found
where SDA is:
root@ashpool:~# ls -la /dev/disk/by-id/ | grep sda
lrwxrwxrwx 1 root root9 gen 7 10:25
scsi-0HP_LOGICAL_VOLUME_ -> ../../sda
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-0HP_LOGICAL_VOLUME_-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-0HP_LOGICAL_VOLUME_-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-0HP_LOGICAL_VOLUME_-part3 -> ../../sda3
lrwxrwxrwx 1 root root9 gen 7 10:25
scsi-3600508b1001c4972bd1d4906bb1dc75d -> ../../sda
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-3600508b1001c4972bd1d4906bb1dc75d-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-3600508b1001c4972bd1d4906bb1dc75d-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-3600508b1001c4972bd1d4906bb1dc75d-part3 -> ../../sda3
lrwxrwxrwx 1 root root9 gen 7 10:25
scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F -> ../../sda
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 gen 7 10:25
scsi-SHP_LOGICAL_VOLUME_PDNNK0BRH9Z27F-part3 -> ../../sda3
lrwxrwxrwx 1 root root9 gen 7 10:25
wwn-0x600508b1001c4972bd1d4906bb1dc75d -> ../../sda
lrwxrwxrwx 1 root root 10 gen 7 10:25
wwn-0x600508b1001c4972bd1d4906bb1dc75d-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 gen 7 10:25
wwn-0x600508b1001c4972bd1d4906bb1dc75d-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 gen 7 10:25
wwn-0x600508b1001c4972bd1d4906bb1dc75d-part3 -> ../../sda3
the boot disk, in a hw raid controller.
Boh...
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] PVE 4 -> 5, multipath differences?
I've just upgraded my cluster frm 4.4 to latest 5. Before that, i've also do a firmware upgrade of the SAN (HP MSA 1040), but seems not related. In old PVE 4.4/jessie i got: dixie:~# multipath -ll mpath0 (3600c0ff00026ed11a7cb56570100) dm-1 HP,MSA 1040 SAN size=1.4T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 2:0:0:0 sda 8:0 active ready running |- 3:0:0:0 sdb 8:16 active ready running |- 4:0:0:0 sdc 8:32 active ready running `- 5:0:0:0 sdd 8:48 active ready running in new PVE 5/stretch i got: root@ashpool:~# multipath -ll mpath2 (3600c0ff00026ed11475d215a0100) dm-0 HP,MSA 1040 SAN size=1.8T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 12:0:0:4 sdg 8:96 active ready running | `- 15:0:0:4 sdr 65:16 active ready running `-+- policy='round-robin 0' prio=10 status=enabled |- 13:0:0:4 sdj 8:144 active ready running `- 14:0:0:4 sds 65:32 active ready running config file is the same. Reading around seems that my SAN is a dual controller Active/Passive one, so probably it is correct to have two path active and two ready, but... i seek some feedback. ;-) Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Remove a MON from a cluster...
Seems sufficiently clear, but i prefere to ask here. I need to remove a node from a PVE 4.4 cluster; the node was only a MON and OSDs node. I have to: 0) remove every single OSD, remove the host from the crushmap (done). 1) remove the node from the MONs (via web interface, or via: pveceph destroymon ) 2) edit by hand (seems there's no way in 4.4) /etc/pve/storage.cfg, adding/removing in 'monhost'; Q: some 'restart' are needed? 3) shut down the node, and apply: https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Remove_a_cluster_node Right? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
> This night another 4GB of RAM, another stop and start, ... OK, with 16GB of ram 5 days passed without FS errors. Also, the other VM, same stretch kernel, roughly same configuration, start to expose same errors: Nov 18 10:12:21 vdmsv2 kernel: [584252.496880] EXT4-fs error (device sda6): ext4_mb_generate_buddy:758: group 104, block bitmap and bg descriptor inconsistent: 2048 vs 32768 free clusters Nov 18 10:12:21 vdmsv2 kernel: [584252.590564] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. Note that this VM was built *AFTER* my SAN glitches happens. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
Mandi! Daniel Berteaud In chel di` si favelave... > Then probably the issue is somewhere on the underlying block on your > SAN. You should destroy and recreate the image. OK. Because the disks that expose the trouble is: 1) the one that contain / 2) the one that contain /var/cache/squid, and so is 'disposable'. Can i simply stop, backup the VM and recreate it back? Or can be risky, i can take with me some FS corruption? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
Mandi! Daniel Berteaud In chel di` si favelave... > In my experience, XFS has been more reliable, and robust. But anyway, > 99.9% of the time, FS corruption is caused by one of the underlying layers ...but the 'underlying layers' is the same of half a dozen other VM/LXC, that have to trouble at all... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
Mandi! Gerald Brandt In chel di` si favelave... > I've only had filesystem corruption when using XFS in a VM. The same VM have two XFS filesystem, that never get corrupted. ;( -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
Mandi! Daniel Berteaud In chel di` si favelave... > If at one time, the storage pool went out of space, then the FS is most > likely corrupted. Fixing the space issue will prevent further > corruption, but won't fix the already corrupted FS. You said But *I* fix every day FS corruption! Every night i reboot the VMs that have: fsck.mode=force as grub boot parameters. In logs, i can se that FS get fixed. Nov 13 23:44:20 vdmsv1 kernel: [0.00] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-8-amd64 root=UUID=587fe965-e914-4c0b-a497-a0c71c7e0301 ro quiet fsck.mode=force Nov 13 23:44:20 vdmsv1 systemd-fsck[644]: /dev/sda6: 15062/8495104 files (3.0% non-contiguous), 1687411/33949952 blocks Nov 13 23:44:20 vdmsv1 systemd-fsck[647]: /dev/sdb1: 113267/6553600 files (1.9% non-contiguous), 1590050/26214144 blocks > Anyway, If errors come back after reformating the volume, then you still > have something not fixed. Reading the Ubuntu, Debian and RH bugs in my initial posts, seems to me that this is not the case. The trouble seems exactly the same: same errors, same partial fix incrementing the available RAM to the VM. > Please tell us how are things configured, what > kind of storage it's using, which layers are involved etc... (thin prov, > iSCSI, LVM on top etc...) HS MSA 1040 SAN, exporting iSCSI volumes via LVM. The 'thin' part is on the SAN side, eg no thin-LVM, no ZFS on top of it, ... Another error popup now: Nov 15 13:44:44 vdmsv1 kernel: [136834.664486] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 957, block bitmap and bg descriptor inconsistent: 32747 vs 32768 free clusters Nov 15 13:44:44 vdmsv1 kernel: [136834.671565] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 958, block bitmap and bg descriptor inconsistent: 32765 vs 32768 free clusters Nov 15 13:44:44 vdmsv1 kernel: [136834.813465] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. increasing the VM ram from 8 to 12 GB lead to a 1,5 day interval between errors, while before errors was every 'less than a day'. This night another 4GB of RAM, another stop and start, ... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
Mandi! Daniel Berteaud In chel di` si favelave... > Not that strange. It's expected to have FS corruption if they resides on > a thin provisionned volume, which itself has no space left. Lucky you > only had one FS corrupted. ...but currently space is OK (really: space on VM images pool was never on shortage, was the 'DATA' pool...), and i've many time done 'e2fsck' on filesystem (as stated, i've also reformatted one...) and errors pop up back again... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
Mandi! Mark Schouten In chel di` si favelave... > Obviously, a misbehaving SAN is a much better explanation for > filesystemcorruption.. Sure, but: a) errors start a bit befose the SAN trouble b) this is the only VM/LXC that have troubles c) i've tried to unmount, reformat and remount a disk/partition (was the squid spool) and errors come back again. It is really strange... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
Mandi! Luis G. Coralle In chel di` si favelave... > Hi, I have a lot of VM ( debian 8 and debian 9 ) with 512 MB of RAM on PVE > 4.4-24 version and have not problem. ...i have a second cluster, but with ceph storage, not iSCSI/SAN, with simlar VM, but no troubles at all. True. > Have you enough free space on the storage? Now, yes. As just stated, i've had a temporary fill of SAN space (something on my trim tasks, or on the SAN, goes wrong) but now all are back as normal. > How much ram memory do you have on PVE? Nodes have 64GB of RAM, 52% full. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Filesystem corruption on a VM?
I come back on this: > In a PVE 4.4 cluster i continue to get FS errors like: > Oct 22 20:51:10 vdmsv1 kernel: [268329.890910] EXT4-fs error (device sda6): > ext4_mb_generate_buddy:758: group 932, block bitmap and bg descriptor > inconsistent: 30722 vs 32768 free clusters > and > Oct 23 09:43:16 vdmsv1 kernel: [314655.032561] EXT4-fs error (device sdb1): > ext4_validate_block_bitmap:384: comm kworker/u8:2: bg 12: bad block bitmap > checksum > Oct 23 09:43:16 vdmsv1 kernel: [314655.034265] EXT4-fs (sdb1): Delayed block > allocation failed for inode 2632026 at logical offset 2048 with max blocks > 1640 with error 74 > Oct 23 09:43:16 vdmsv1 kernel: [314655.034335] EXT4-fs (sdb1): This should > not happen!! Data will be lost > Host run 4.4.134-1-pve kernel, and guest is a debian stretch > (4.9.0-8-amd64), and in the same cluster, but also in other clusters, i > have other stretch VMs running in the same host kernel, without > troubles. > Googling around lead me to old jessie bugs (kernels 3.16): > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1423672 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=818502#22 Seems the bug is really this. I've increased the RAM of the problematic VM, and FS corruption deasppeared. Effectively, all other VMs have plently of free ram, this was a bit full. I know that PVE 4.4 is EOL, but still i'm seeking feedback. For example, is a 'host' kernel bug, or a 'guest' kernel bug? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] SAN and correctly propagate 'discard'...
> Someone can give me some clue? Thanks. Only for the sake of google: trim correctly propagate between virtual disks to the SAN. After rebooting the SAN controllers(s), all ''numbers'' go OK. Sorry. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Filesystem corruption on a VM?
In a PVE 4.4 cluster i continue to get FS errors like: Oct 22 20:51:10 vdmsv1 kernel: [268329.890910] EXT4-fs error (device sda6): ext4_mb_generate_buddy:758: group 932, block bitmap and bg descriptor inconsistent: 30722 vs 32768 free clusters and Oct 23 09:43:16 vdmsv1 kernel: [314655.032561] EXT4-fs error (device sdb1): ext4_validate_block_bitmap:384: comm kworker/u8:2: bg 12: bad block bitmap checksum Oct 23 09:43:16 vdmsv1 kernel: [314655.034265] EXT4-fs (sdb1): Delayed block allocation failed for inode 2632026 at logical offset 2048 with max blocks 1640 with error 74 Oct 23 09:43:16 vdmsv1 kernel: [314655.034335] EXT4-fs (sdb1): This should not happen!! Data will be lost Host run 4.4.134-1-pve kernel, and guest is a debian stretch (4.9.0-8-amd64), and in the same cluster, but also in other clusters, i have other stretch VMs running in the same host kernel, without troubles. Googling around lead me to old jessie bugs (kernels 3.16): https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1423672 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=818502#22 or things i make it hard to correlate with: https://access.redhat.com/solutions/155873 Someone have some hints?! Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] SAN and correctly propagate 'discard'...
Mandi! Alexandre DERUMIER In chel di` si favelave... > It's just not possible, sorry. OK. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] SAN and correctly propagate 'discard'...
My SAN (HP MSA 1040) have 'virtual volume' enabled, eg discard. I've had to use directly some LUN in some VM, via iSCSI, and 'fstrim' works as expected. Now i'm using for some disks in my VMs, as storage of type 'LVM'. But if i look at disk space on SAN, it is coherent with the real disk space occupied. While, in Proxmox, the storage volume are full or near full. Example, VM: root@vdmsv1:~# df -h | grep sdc /dev/sdc1 800G 519G282G 65% /home /dev/sdc2 1000G 152G848G 16% /srv Proxmox node: root@tessier:~# pvesm status | grep DATA0 DATA0 lvm 1 1953120256 188743680065683456 97.14% root@tessier:~# pvesm list DATA0 DATA0:vm-120-disk-1 raw 1932735283200 120 and SAN report 730GB allocated. Clearly disks of VM have 'discard' enabled, but seems that discard/trim does not ''traverse'' all the storage chains (partition in the VM, LV/PV/VG in proxmox node, iSCSI in the SAN). Clearly i cannot use 'LVM Thin', because is not shared. https://pve.proxmox.com/wiki/Storage:_LVM_Thin Someone can give me some clue? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Upgrade containers and host OS/Kernel version...
I've done some google lookup, but i've not found an answer... a) generally there's some trouble to have too much differences in kernel/systemd/OS/... version in containers? EG, if i run a 'stretch' container in proxmox 4.4 (jessie based) i can expect troubles? b) containers can be upgraded as normal VMs? EG, if i have a 'jessie' container, can i upgrade it to 'stretch' via standard debian upgrade procedure, or better to fire up a new container with the new OS revision and move over services/configuration within? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Proxmox upgrade 4 - 5
Mandi! John Crisp In chel di` si favelave... > Perhaps a note on the wiki to say check for space before upgrading older > systems would be helpful (assuming you have an older system with this) ...'im not involved in Proxmox, but... i think this is a 'debian' thing and not a proxmox thing... and the wiki page (i suppose this: https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0 ) say explicitly to read the debian documentation: https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0#External_links where: https://www.debian.org/releases/stretch/amd64/release-notes/ch-upgrading.en.html#sufficient-space Sayed that... welcome to the «damned me! why i've crated a so small /boot (or /) partition!» sysadmin (very large ;) group. ;-) -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] In-place upgrade and clusters...
Mandi! Mark Schouten In chel di` si favelave... > Yes. That works for me. OK. Thanks. > But beware that you need to upgrade Ceph first > (if used) and not to reboot in between. Because otherwise you end up > with mon's and osd's of different versions in the same cluster, which > is unsupported. ...I've also a Ceph cluster, but the upgrade of that is scheduled a bit later... ;-))) For now, i've to upgrade the non-ceph cluster. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] In-place upgrade and clusters...
Mandi! Mark Schouten In chel di` si favelave... > I’ve been able to migrate VM’s between upgraded and ‘old’ nodes. Not all VM’s > survived it without crashing, but I’m not sure what caused it (might be Ceph > client stuff as well). So seems that, in upgrading a cluster: a) moving VMs/CTs from 4.X to 5.X is supported, as documented in the wiki b) moving VMs/CTs from 5.X and 4.X works with some troubles. So, probably the best upgrade path will be: 1) move all the VMs/CTs out of a node (or, if possible, stop them). 2) upgrade the node from 4.X to 5.X, reboot (repeat as needed) 3) move all the VMs/CTs in a node to the upgraded node(s), repeat step 2). Right? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] In-place upgrade and clusters...
I've upgraded some standalone PVE installation with 'In-place upgrade' method: https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0#In-place_upgrade without a fuss. And in this page the note: If you run a PVE 4 cluster it's tested and supported to add a PVE 5 node and migrate your guests to the new host. seem confident to me that the answer to my answer is 'yes'... but... Upgrading a cluster, can i upgrade node by node, migrating VMs/CTs to other nodes, without troubles if the source node and the target node run different PVE versions (latest 4.X and latest 5.X)? Or i've to shutdown all the VMs/CTs and upgrade all the cluster at the same time? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Snapshot rollback slow
Mandi! Marcus Haarmann In chel di` si favelave... > However, the rollback of snapshot took > 1 hour and during this timeframe, > the whole cluster > was reacting very slow. AFAIK this is a 'feature' of ceph (again AFAIK fixed/better handled in recent ceph version and on bluestore): a deletion (of a volume or a snapshot rollback) trigger a 'write amplification'. In older ceph version there's some workarounds... Sorry for be not so precise, look at ceph mailing list archive for more precise info... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] ZFS, grub cannot load second stage...
Mandi! Marco Gaiarin In chel di` si favelave... > zpool import -f -d /dev/ -R /mnt rpool Sorry, a little notes. I was forced to use -d /dev because if i omit it, zpool found only two disks out of 4 (does not found the two disks added after the first install). This seems strange. As just stated, the server was rebooted many time, so, anyway, it worked. Again, thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] ZFS, grub cannot load second stage...
I've installed a little server vith PVE 5.2, using ZFS, as just done before. The only differences is that server have a raid-1 zpool with 4 disks (2 1TB and 2 4TB disks) all in a single 5TB rpool. The server have a plain AHCI controller, no RAID hardware. After configuring the server, configuring some VM/LXC inside, probably a user inadvertitely powered off it. The server never gone back, and print in console: error: no such device: aeX. error: unknown filesystem. Entering rescue mode... grub rescue> in rescue mode, i can 'ls' all the disks and seems OK (number of disks and partition match), but i cannot 'ls' inside (filesystem not found or similar error). I've googled and found: https://forum.proxmox.com/threads/crashes-with-zfs-root-and-stuck-on-grub-rescue-prompt.34172/ and i was able to run the install CD (rescue mode does nothing, say there's no rpool) and with: zpool import -f -d /dev/ -R /mnt rpool i've mounted the pool, chmod-ed on it and tried to reinstall grub and rebuild grub confing and initrd. Data seems OK. But after a reboot, throw the same error. How can fix that? The hex code in 'error: no such device: aeX.' is relative to the 'ROOT' zfs volume? How can i determine if match correctly? Please, help me. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] ZFS/PVE5 and fstrim...
I'm used, on PVE 4.4, to enable fstrim also for containers. I've setup a simple script: root@vdctms1:~# cat /etc/cron.weekly/dofstrim #!/bin/ sh for mount in /; do fstrim $mount done but: root@vdctms1:~# fstrim -v / fstrim: /: the discard operation is not supported it is PVE5 or ZFS that have no support for fstrim on containers? Seems that on VM fstrim works (same phisical server): root@vdmtms1:~# fstrim -v / /: 5,9 MiB (6168576 bytes) trimmed Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE 5.2, time sync, container...
Mandi! Marco Baldini - H.S. Amiata In chel di` si favelave... > Just install a little VM with a NTP server (i use chrony) and sync hosts > with that. I went this route about two years ago and had no problem since > then After googling a bit better then yesterday, i've done: root@clerk:~# lxc-info -n 100 -c lxc.cap.drop lxc.cap.drop = mac_admin mac_override sys_time sys_module sys_rawio and so i've added to /etc/pve/lxc/100.conf: lxc.cap.drop: lxc.cap.drop: mac_admin mac_override sys_module sys_rawio and now: root@clerk:~# lxc-info -n 100 -c lxc.cap.drop lxc.cap.drop = mac_admin mac_override sys_module sys_rawio So now i can run a NTP on my container, providing probably that is the only container with cap 'sys_time' enabled. Clearly i can run ntp on the hostm, or in a VM, but an AD DC mandate the presence of a ntp server, and probably having an NTP server that, if needed, cannot write to HWclock, could be a source of troubles... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE and Active Directory...
Mandi! Dmitry Petuhov In chel di` si favelave... > To be able to use LDAPS, your client machine (PVE) must trust server's > certificate. Sign LDAP server's SSL certificate with some CA (private CA is > ok) and place that CA certificate to /usr/local/share/ca-certificates in PEM > format with .crt extension on PVE and run `update-ca-certificates` to make > system trust it. Good point, libldap. I've added to /etc/ldap/ldap.conf: TLS_REQCERT never but nothing changed, still does not work. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] PVE and Active Directory...
I've tried to setup my AD domain (with samba!) as authentication source. It works, but i was not able to setup SSL, and i was forced to disable 'sign or seal' in samba conf, eg: ldap server require strong auth = no In 'Authentication' i've put: Realm: LNFFVG Domain: AD.FVG.LNF.IT Server: Fallback Server: Port: empty SSL: is not editable TFA: empty/none if i (un)set Port:, eg keep the default, AND i put 'ldap server require strong auth = no' in smb.conf auth work. If i set Port: 636, does not work. Someone have some hint?! Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] PVE 5.2, time sync, container...
I'm setting up a set of PVE hosts, all with a samba AD DC container, that came also with a NTP server (ntpd). Most of that servers get installed on place where there's just a NTP ''network'' (based on phisical or P2V machines), and i'm using mostly PVE 4.4, so probably i've never hitted that trouble. Now i've added a new node/site, where i've put a single PVE 5.2 node, and my samba AD DC container that act as NTP server for that site. I've found, so, that container take time for the host (and is reasonable) but seems that an NTP server run on a container cannot modify local time, and sooner or later will crash. Apart the ''bootstrap'' problem (a host that sync time to a container, that have the same time of the host; i need to revise my NTP network... ;-), looking on google i've found that relative to old containers: https://forum.proxmox.com/threads/ntp-and-containers.10258/ How can i enable NTP ''safely'' on a cantainer? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Interfaces startup and ip-up.d scripts...
Mandi! Josh Knight In chel di` si favelave... > Interesting, I couldn't reproduce the problem on my server. I'm not a very large user case: i've many PVE system, but they are 4.4 and not ''firewalled'', this is a 5.2 and a case ''per se''... > I set > verbose=yes, I created a test script that simply did echo $IFACE and after > 3 reboots it seems to execute each time. After boot I just did journalctl > -b | grep ifup and I was able to see the interface names printed. Boh... > in journalctl -b, are you seeing anything related to run-parts? Or does > ifup not print anything at all? In a ''falied'' boot i can se the logs for interfaces 'lo' and '--all' (why '--all'?). Logs report, for every interface: /bin/ip link set dev up and then the run of the 'run-parts': run-parts: executing /etc/network/if-up.d/0sysctl and the the single runs of the scripts: run-parts: executing /etc/network/if-up.d/bridgevlan In a good boot, i can se the logs for interfaces 'lo', 'vmbr0', 'vmbr1' and '--all''. Same logs. > Is your firewall script using anything interface specific? If you put it > in that directory, it will be executed for each interface. My script are parametrizied, and get runned only on particular interfaces. > I'm curious if > you add a post-up line to your /etc/network/interfaces file, it would be > called only once when your mgt interface comes up. I was short on time, and so i was forced to put that server in production, in a non too easy reachable place. So, now i cannot play with networking. ;-) -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Interfaces startup and ip-up.d scripts...
Mandi! Josh Knight In chel di` si favelave... > Are you using the script to assign an IP address manually, or are you using > it to set firewall rules? I'm setting firewall rules; my script are coded carefully, and exit always with 0 status. They works in my debisn server/firewall without a trouble (but, indeed, with standard debian kernel, and still in jessie). > I'm trying to determine what you're trying to > do. Having the IP defined in /etc/network/interfaces should be enough for > it to come up correctly without any custom scripts. Is this not the case? Tha strange things is exactly that. With simple stanzas like that: > > auto vmbr0 > > iface vmbr0 inet static > > address 10.99.25.254 > > netmask 255.255.252.0 > > bridge_ports enp2s0f0 > > bridge_stp off > > bridge_fd 0 > > bridge_vlan_aware yes > > > > auto vmbr1 > > iface vmbr1 inet static > > address 10.5.2.230 > > netmask 255.255.0.0 > > gateway 10.5.1.254 > > bridge_ports enp2s0f1 > > bridge_stp off > > bridge_fd 0 interfaces get correctly brought up (eg, and 'ip address show' list interfaces correctly), but looking at 'journalctl -b' seems that if-up.d and if-down.d scripts get never executed. *ALL* scripts, of course, not only mine... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Interfaces startup and ip-up.d scripts...
Mandi! dORSY In chel di` si favelave... > I suggest using post-up / pre-down hooks in interfaces. It makes sure that > the interfaces are actually up before the commands get executed. From 'interfeces' manpage: Please note that as post-up and pre-down are aliases, no files in the corresponding directories are processed. Please use if-up.d and if-down.d directories instead. so seems that put script in if-up.d or as post-up stanzas is the same. Also, seems to me that the trouble came from the fact that network scripts does not run at all fro vmbr0 and vmbr1 interfaces! -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Interfaces startup and ip-up.d scripts...
I've to setup a little PVE server in a private but hostile network, and i've only an IP available, so i was forced to assign the IP to the phisical server, running latest proxmox, and i've setup a firewall using my hand-made scripts. I've put the script, as usual with debian, in /etc/network/if-up.d/ and if-down.d/, but i've found that not at every boot they get started. So, i've enabled networking debug (eg, set VERBOSE=yes in /etc/default/networking) and found that at every boot scripts get called with 'lo' interface, but only roughly 1 out of 10 times the other bridge interfaces get started. So, i got: May 15 10:18:25 clerk ifup[2958]: /bin/ip link set dev lo up May 15 10:18:25 clerk ifup[2958]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d but then: May 15 10:18:26 clerk ifup[2958]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d without interface name. With some more debug i discovered that is the '--all' interface. The strange things is that interfaces vmbr0 and vmbr1 are up, simply the scripts get not called. Some boot, instead: May 15 10:18:27 clerk ifup[4043]: /bin/ip link set dev vmbr0 up May 15 10:18:27 clerk ifup[4043]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d May 15 10:18:28 clerk ifup[4043]: /bin/ip addr add 10.5.2.230/255.255.0.0 broadcast 10.5.255.255 dev vmbr1 label vmbr1 May 15 10:18:28 clerk ifup[4043]: /bin/ip link set dev vmbr1 up May 15 10:18:28 clerk ifup[4043]: /bin/ip route add default via 10.5.1.254 dev vmbr1 onlink May 15 10:18:28 clerk ifup[4043]: /bin/run-parts --exit-on-error --verbose /etc/network/if-up.d and clearly firewalling script works. For now, i've put 'systemctl restart networking' in /etc/rc.local. My /etc/network/interfaces is rather simple: auto lo iface lo inet loopback iface enp2s0f0 inet manual iface enp2s0f1 inet manual ethernet-autoneg on link-speed 100 link-duplex full hardware-dma-ring-tx 18 offload-tso off offload-gso off offload-gro off auto vmbr0 iface vmbr0 inet static address 10.99.25.254 netmask 255.255.252.0 bridge_ports enp2s0f0 bridge_stp off bridge_fd 0 bridge_vlan_aware yes auto vmbr1 iface vmbr1 inet static address 10.5.2.230 netmask 255.255.0.0 gateway 10.5.1.254 bridge_ports enp2s0f1 bridge_stp off bridge_fd 0 There's something i can do to fix this? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] LXC Container 'disk trim'?
(I'm speaking of PVE 4.4, i don't know if in 5.0 things are better...) I know that, if the underlying storage is a 'thin' storage, i can use in VM 'VirtIO SCSI' and enable trim to reclaim space: https://pve.proxmox.com/wiki/Qemu_trim/discard_and_virtio_scsi But on Containers?! I've tried: root@baculav:~# fstrim -v / /: 21,1 GiB (22610677760 bytes) trimmed and so seems to work, but i've not found info on the wiki, so i'm asking here. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Containers, stretch and php...
> I've build up a LXC container based on debian 9 (stretch), but after > installing PHP i've started to have in logs in the container: I've upgraded to stretch 9.1 (the container) and upgraded pve-container to 1.0-101, but nothing changed. FYI. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Creating a template from an existing container...
I've googled a bit finding old result (relative to old container system, not LXC). Having a just setup container, there's some quick way to 'template-ize' it? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Containers, stretch and php...
> Why? My first stretch bug. ;-) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=869182 -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Containers, stretch and php...
(PVE 4.4, upgraded to latest patches) I've build up a LXC container based on debian 9 (stretch), but after installing PHP i've started to have in logs in the container: Jul 20 16:09:14 vglpi systemd[1]: phpsessionclean.service: Failed to reset devices.list: Operation not permitted Jul 20 16:09:14 vglpi systemd[6345]: phpsessionclean.service: Failed at step NETWORK spawning /usr/lib/php/sessionclean: Permission denied Jul 20 16:09:14 vglpi systemd[1]: phpsessionclean.service: Main process exited, code=exited, status=225/NETWORK Jul 20 16:09:14 vglpi systemd[1]: Failed to start Clean php session files. Jul 20 16:09:14 vglpi systemd[1]: phpsessionclean.service: Unit entered failed state. Jul 20 16:09:14 vglpi systemd[1]: phpsessionclean.service: Failed with result 'exit-code'. Jul 20 16:39:14 vglpi systemd[1]: phpsessionclean.service: Failed to reset devices.list: Operation not permitted Jul 20 16:39:14 vglpi systemd[6364]: phpsessionclean.service: Failed at step NETWORK spawning /usr/lib/php/sessionclean: Permission denied Jul 20 16:39:14 vglpi systemd[1]: phpsessionclean.service: Main process exited, code=exited, status=225/NETWORK Jul 20 16:39:14 vglpi systemd[1]: Failed to start Clean php session files. Jul 20 16:39:14 vglpi systemd[1]: phpsessionclean.service: Unit entered failed state. Jul 20 16:39:14 vglpi systemd[1]: phpsessionclean.service: Failed with result 'exit-code'. and on the host: Jul 20 16:09:14 tessier kernel: [22451057.039944] audit: type=1400 audit(1500559754.627:239): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=10038 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Jul 20 16:09:14 tessier kernel: [22451057.039949] audit: type=1400 audit(1500559754.627:240): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=10038 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Jul 20 16:09:14 tessier kernel: [22451057.039953] audit: type=1400 audit(1500559754.627:241): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=10038 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Jul 20 16:09:14 tessier kernel: [22451057.039956] audit: type=1400 audit(1500559754.627:242): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=10038 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Jul 20 16:39:14 tessier kernel: [22452857.015429] audit: type=1400 audit(1500561554.627:243): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=12677 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Jul 20 16:39:14 tessier kernel: [22452857.015434] audit: type=1400 audit(1500561554.627:244): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=12677 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Jul 20 16:39:14 tessier kernel: [22452857.015438] audit: type=1400 audit(1500561554.627:245): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=12677 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Jul 20 16:39:14 tessier kernel: [22452857.015441] audit: type=1400 audit(1500561554.627:246): apparmor="DENIED" operation="file_lock" profile="lxc-container-default-cgns" pid=12677 comm="(ionclean)" family="unix" sock_type="dgram" protocol=0 addr=none Why? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Unattended installation....
Mandi! Elias Werberich In chel di` si favelave... > You may not be able to use the nice GUI but the PVE ISO simply does the same > in the background. AFAIK, minus the ZFS filesystem. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE, Ceph, OSD in stop/out state: how to restart from commandline?
Mandi! Fabian Grünbichler In chel di` si favelave... > Hammer is not using per-service-instance systemd units, but a single > init script. You can simply use that init script (directly or via the > "service" wrapper) to start single service instances. Do you mean: /etc/init.d/ceph restart osd ?! Seems to me that also on hammer the systemd approach work, at least now i've all the OSD listed and the 'ceph.service' main service: root@vedovanera:~# systemctl restart ceph ceph-mon.1.1496583375.326412133.service ceph-osd.3.1496585083.378765026.service ceph-osd.5.1496584949.775435241.service ceph-osd.2.1496585013.322315859.service ceph-osd.4.1496583466.847479891.service ceph.service > My guess is that > your cluster is overloaded when cold-booting, and thus not all OSDs > start within their timeout. Coud be. 'vedovanera' server need to be replaced soon... Again, thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] PVE, Ceph, OSD in stop/out state: how to restart from commandline?
.2 4002 crush map has features
1107558400, adjusting msgr requires for clients
2017-06-04 16:03:43.034150 7feecc646880 0 osd.2 4002 crush map has features
1107558400 was 8705, adjusting msgr requires for mons
2017-06-04 16:03:43.034162 7feecc646880 0 osd.2 4002 crush map has features
1107558400, adjusting msgr requires for osds
2017-06-04 16:03:43.034173 7feecc646880 0 osd.2 4002 load_pgs
2017-06-04 16:04:15.169521 7feecc646880 0 osd.2 4002 load_pgs opened 253 pgs
2017-06-04 16:04:15.178051 7feecc646880 -1 osd.2 4002 log_to_monitors
{default=true}
2017-06-04 16:04:15.201447 7feeb6454700 0 osd.2 4002 ignoring osdmap until we
have initialized
I hope can be useful. Thanks.
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] PVE, Ceph, OSD in stop/out state: how to restart from commandline?
Again my Ceph cluster suffer a main power outgage. ;-( The cluster went down well, but after that the power come back a bit intermittently, so servers boot and shutdown some times... When power come back, all server run, cluster works as expected bu i've had 5 OSD (out of 12) down/out. I was away, so i've connected via SSH, but i've found there's no way to restart OSD, because systemd does not have the ''stanza'', eg in a server where i've had 2 OSD down out of 4, i was able to do: systemctl start ceph-osd. and i've seen the two running one, not the other. I've tried other systemd command (enable, restart, ...) but there's no stanza for the faulty osd. I was forced to do some ssh port forwarding, connect to the web interface and restart the faulty OSD with the 'start' button on the node ceph->osd page. Why?! Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
