[CentOS-virt] centos-virt CPU microcode updates?
Hi, is there any guide for CPU microcode updates on CentOS6, Xen 4.10, kernel 4.9 ? Thanks Karel ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] libvirtd hang on CentOS6 after latest updates
Bump. Folks, any ideas? Cheers Karel On 22.5.2018 11:33, Karel Hendrych wrote: Hi, I am seeing frequent libvirtd hangs (clients not responding) after last CentOS6-Xen update : libvirt-libs-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64 libgcc-4.4.7-18.el6_9.2.x86_64 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64 libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-common-8-4.el6.x86_64 xen-licenses-4.6.6-12.el6.x86_64 xen-libs-4.6.6-12.el6.x86_64 libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64 libstdc++-4.4.7-18.el6_9.2.x86_64 libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64 libvirt-client-4.1.0-2.xen46.el6.x86_64 linux-firmware-20171215-82.git2451bb22.el6.noarch 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64 libvirt-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-46-8-4.el6.x86_64 10:centos-release-xen-44-8-4.el6.x86_64 tzdata-2018e-3.el6.noarch libgomp-4.4.7-18.el6_9.2.x86_64 kernel-4.9.86-30.el6.x86_64 xen-hypervisor-4.6.6-12.el6.x86_64 xen-runtime-4.6.6-12.el6.x86_64 xen-4.6.6-12.el6.x86_64 libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64 Remedy is to kill -9 libvirtd and start again. Issue can be replicated within few domU starts. Usually libvirtd hangs when domU is bringing up xen drivers or something around udev, like: xen_netfront: Initialising Xen virtual ethernet driver I've been looking into libvirtd strace and debug logs, so far most suspicious in libvirtd debug log is this: libvirtd.log:2018-05-22 08:32:44.760+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7' libvirtd.log:2018-05-22 08:32:44.761+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6' libvirtd.log:2018-05-22 08:32:44.761+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4' libvirtd.log:2018-05-22 08:32:44.762+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5' libvirtd.log:2018-05-22 08:32:44.763+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2' libvirtd.log:2018-05-22 08:32:44.764+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3' libvirtd.log:2018-05-22 08:32:44.765+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6' libvirtd.log:2018-05-22 08:32:44.766+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5' libvirtd.log:2018-05-22 08:32:44.767+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4' libvirtd.log:2018-05-22 08:32:44.767+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7' libvirtd.log:2018-05-22 08:32:44.768+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2' libvirtd.log:2018-05-22 08:32:44.769+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3' I could not get rid of this by reducing amount of driver queues (not sure if that applies to PV) Is someone out there seeing similar issues? Anyone perhaps interested in reviewing full debug log / strace ? Cheers Karel ___ CentOS-virt mailing list CentOS-virt@centos.org https
[CentOS-virt] libvirtd hang on CentOS6 after latest updates
Hi, I am seeing frequent libvirtd hangs (clients not responding) after last CentOS6-Xen update : libvirt-libs-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64 libgcc-4.4.7-18.el6_9.2.x86_64 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64 libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-common-8-4.el6.x86_64 xen-licenses-4.6.6-12.el6.x86_64 xen-libs-4.6.6-12.el6.x86_64 libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64 libstdc++-4.4.7-18.el6_9.2.x86_64 libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64 libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64 libvirt-client-4.1.0-2.xen46.el6.x86_64 linux-firmware-20171215-82.git2451bb22.el6.noarch 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64 libvirt-4.1.0-2.xen46.el6.x86_64 10:centos-release-xen-46-8-4.el6.x86_64 10:centos-release-xen-44-8-4.el6.x86_64 tzdata-2018e-3.el6.noarch libgomp-4.4.7-18.el6_9.2.x86_64 kernel-4.9.86-30.el6.x86_64 xen-hypervisor-4.6.6-12.el6.x86_64 xen-runtime-4.6.6-12.el6.x86_64 xen-4.6.6-12.el6.x86_64 libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64 Remedy is to kill -9 libvirtd and start again. Issue can be replicated within few domU starts. Usually libvirtd hangs when domU is bringing up xen drivers or something around udev, like: xen_netfront: Initialising Xen virtual ethernet driver I've been looking into libvirtd strace and debug logs, so far most suspicious in libvirtd debug log is this: libvirtd.log:2018-05-22 08:32:44.760+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7' libvirtd.log:2018-05-22 08:32:44.761+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6' libvirtd.log:2018-05-22 08:32:44.761+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4' libvirtd.log:2018-05-22 08:32:44.762+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5' libvirtd.log:2018-05-22 08:32:44.763+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2' libvirtd.log:2018-05-22 08:32:44.764+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3' libvirtd.log:2018-05-22 08:32:44.765+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6' libvirtd.log:2018-05-22 08:32:44.766+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5' libvirtd.log:2018-05-22 08:32:44.767+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4' libvirtd.log:2018-05-22 08:32:44.767+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7' libvirtd.log:2018-05-22 08:32:44.768+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2' libvirtd.log:2018-05-22 08:32:44.769+: 25455: debug : udevRemoveOneDevice:1289 : Failed to find device to remove that has udev name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3' I could not get rid of this by reducing amount of driver queues (not sure if that applies to PV) Is someone out there seeing similar issues? Anyone perhaps interested in reviewing full debug log / strace ? Cheers Karel ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] NIC Stability Problems Under Xen 4.4 / CentOS 6 / Linux 3.18
Have you tried to eliminate all power management features all over? Are the devices connected to the same network infrastructure? There has to be something common. I've been using Intel NICs with Xen/CentOS for ages with no issues. Karel On 27.1.2017 02:57, Kevin Stange wrote: On 01/26/2017 02:08 PM, Kevin Stange wrote: On 01/26/2017 09:35 AM, Johnny Hughes wrote: On 01/26/2017 09:32 AM, Johnny Hughes wrote: On 01/25/2017 11:49 AM, Kevin Stange wrote: On 01/24/2017 11:16 AM, Kevin Stange wrote: On 01/24/2017 09:10 AM, Konrad Rzeszutek Wilk wrote: On Tue, Jan 24, 2017 at 09:29:39PM +0800, -=X.L.O.R.D=- wrote: Kevin Stange, It can be either kernel or update the NIC driver or firmware of the NIC card. Hope that helps! Xlord -Original Message- From: CentOS-virt [mailto:centos-virt-boun...@centos.org] On Behalf Of Kevin Stange Sent: Tuesday, January 24, 2017 1:04 AM To: centos-virt@centos.org Subject: [CentOS-virt] NIC Stability Problems Under Xen 4.4 / CentOS 6 / Linux 3.18 Has anyone experienced similar issues with this configuration, and if so, does anyone have tips on how to resolve the issues? Honeslty I would email Intel and see if they can help. This looks like the NIC decides something is wrong, throws off an PCIe error and then resets itself. This happens for several different NICs. Is there a good contact at Intel for this kind of thing, or should I just try to reach them through their web site? It could also be an error in the Linux stack which would "eat" an interrupt when migrating interrupts (which was fixed upstream, see below). Are you running irqbalance? Could you try turning it off? irqbalance is enabled on these servers. I'll try disabling it. I had stopped irqbalance yesterday afternoon, but had a hypervisor's NICs fail anyway in early morning this morning, so I'm pretty sure this is not the right tree to bark up. Here is a set of drivers/fireware from Intel for those NICs: https://downloadcenter.intel.com/download/15817/Intel-Network-Adapter-Driver-for-PCI-E-Gigabit-Network-Connections-under-Linux- I will see if I can get a CentOS-6 build of the latest version of that from our older SRPM: http://vault.centos.org/6.7/xen4/Source/SPackages/e1000e-2.5.4-3.10.68.2.el6.centos.alt.src.rpm I am currently very busy with several c5, c6, c7 updates and the i686 altarch c7 tree .. but I have this on my list. In the meantime, maybe someone else could also see if those drivers help you (or you could try to compile / install it). Do you have another machine that you can use to see if you can duplicate the issue NOT running the xen.gz hypervisor boot, but just the straight kernel? I can't actually reproduce this problem reliably. It happens randomly when the servers are up and running anywhere between a few hours and a month or more, and I haven't been able to isolate any specific way to cause it to happen. As a result I can't really test different solutions on different servers to see what helps. I was hoping other people were seeing it so that I could get some direction. If I can reproduce it, it won't take me very long to identify what the cause is. Right now if I do upgrade the drivers on the systems I won't really know if it's fixed until I don't see another issue for several months. Actually .. I think this is the driver for you: https://downloadcenter.intel.com/download/13663 And this explains how to make it work: http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/05767.html The different combinations of NICs overlap both the e1000e and igb drivers, but the most egregious issues have been with the igb ones. I'll try to give this a shot and report back if I still see issues with a server after doing so, but it might be a week or two before I find out. So the NICs giving issues in most cases were igb drivers. I've tried replacing the drivers on some HVs with the version you suggested, but it doesn't seem to have helped with stability. Any other ideas? ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Updated centos-release-xen packages (fixes missing initrd issue on upgrade)
Thanks! Upgrade to 3.18.25-18.el6/4.4.3-10.el6 went fine. No need to manualy adjust grub anymore. BR -- Karel On 20.1.2016 19:33, George Dunlap wrote: I've pushed an update to the centos-release-xen package, centos-release-xen-7-12.el6, to centos-virt-xen-testing, which should fix the "missing initrd" issue people have been seeing. Please test and see if it fixes your issue (and that it makes no other issues). If everything works well, I'll get it pushed to centos-extras. Thanks, -George ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] 3.18.21-16 upgrade, kernel panic, unable to mount root fs
Hi, just a heads-up: 3.18.21 didn't boot up on HP ML310e Gen8 v2, SATA drives in AHCI mode, / on software raid 1, no LVM. It ended up in kernel panic with unable to mount root fs. Attached. No difference in grub kernel/xen settings. I didn't investigate the things deeper so far. 4.4.3-3/3.18.17 is booting fine. Any similar experience? BR -- Karel ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] 3.18.21-16 upgrade, kernel panic, unable to mount root fs
Indeed, a good catch. Module initramfs line was missing at 3.18.21 grub config. Thanks! -- Karel On 1.11.2015 21:46, Sarah Newman wrote: On 11/01/2015 12:07 PM, Karel Hendrych wrote: Hi, just a heads-up: 3.18.21 didn't boot up on HP ML310e Gen8 v2, SATA drives in AHCI mode, / on software raid 1, no LVM. It ended up in kernel panic with unable to mount root fs. Attached. No difference in grub kernel/xen settings. I didn't investigate the things deeper so far. 4.4.3-3/3.18.17 is booting fine. Any similar experience? Check the module line for the initramfs is present in grub.conf. It's gone missing in all our recent upgrades, but I wasn't sure if it was an upstream problem or our own. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] libvirt, xen PV, qemu-system-i386, root user
Good test, non-buffered dom0 dd write speed is similar with tap2. I'll likely stay with the QEMU backend. Are there any best practises regarding security, at least if QEMU can operate under non-root account? Cheers -- Karel On 12.9.2015 10:51, Pasi Kärkkäinen wrote: On Sat, Sep 12, 2015 at 01:35:48AM +0200, Karel Hendrych wrote: Comparing simple dd bs=1M count=1 on dom0 vs domU. Qemu driver is achieving pretty much the same like dom0. So you're measuring buffered speed. Try measuring non-buffered (iflag=direct or oflag=direct, depending if you're reading or writing). -- Pasi Thanks -- Karel On 7.9.2015 21:45, Pasi Kärkkäinen wrote: On Mon, Sep 07, 2015 at 05:47:39PM +0200, Karel Hendrych wrote: ... changing from: to: makes the domain start without QEMU. However I see much better performance with QEMU (close to dom0, tested using simple dd writes) than with tap2 driver. Is that expected? How did you measure it? buffered or direct io? -- Pasi What's best practise to file based storage on latest CentOS6-xen (Kernel 3.18.17, Xen 4.4.2-7) Are there any guides around running QEMU on CentOS6-xen as non-root user? Cheers -- Karel On 7.9.2015 17:42, Karel Hendrych wrote: Hi, spot on! On 6.9.2015 12:56, Pasi Kärkkäinen wrote: On Sun, Sep 06, 2015 at 09:08:50AM +0200, Karel Hendrych wrote: Hi, after migrating to libvirt/libxl according to: Hi, https://wiki.centos.org/HowTos/Xen/Xen4QuickStart/Xen4Libvirt I've noticed that my Xen PV domains are being launched by qemu-system-i386 running under root privileges. I am wondering why is this? Previously no qemu process was used. If qemu is needed for some reason, are there any guidelines for non-root operation? In general qemu is used for the following purposes: - for certain domU disk backend types (image files), and/or if there's no blktap driver in dom0 kernel. - domU graphical console (PVFB) VNC server, if it's enabled for the domU. -- Karel Hendrych -- Pasi ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] libvirt, xen PV, qemu-system-i386, root user
Hi, spot on! On 6.9.2015 12:56, Pasi Kärkkäinen wrote: On Sun, Sep 06, 2015 at 09:08:50AM +0200, Karel Hendrych wrote: Hi, after migrating to libvirt/libxl according to: Hi, https://wiki.centos.org/HowTos/Xen/Xen4QuickStart/Xen4Libvirt I've noticed that my Xen PV domains are being launched by qemu-system-i386 running under root privileges. I am wondering why is this? Previously no qemu process was used. If qemu is needed for some reason, are there any guidelines for non-root operation? In general qemu is used for the following purposes: - for certain domU disk backend types (image files), and/or if there's no blktap driver in dom0 kernel. - domU graphical console (PVFB) VNC server, if it's enabled for the domU. -- Karel Hendrych -- Pasi ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] libvirt, xen PV, qemu-system-i386, root user
Hi, after migrating to libvirt/libxl according to: https://wiki.centos.org/HowTos/Xen/Xen4QuickStart/Xen4Libvirt I've noticed that my Xen PV domains are being launched by qemu-system-i386 running under root privileges. I am wondering why is this? Previously no qemu process was used. If qemu is needed for some reason, are there any guidelines for non-root operation? Thanks -- Karel Hendrych ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
