Re: Kernel Panic after updating Kernel
Le 18/06/2020 à 21:38, Jason A. Donenfeld a écrit : > Remove all wireguard packages. Update to the latest kernel. Run `find > /lib/modules -name '*wireguard*' -print -delete`. Then install > kmod-wireguard. Sorry for late reply. I can confirm that removing wireguard module and reinstalling (from elrepo) worked. Thanks, -- Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527 signature.asc Description: OpenPGP digital signature
Re: Kernel Panic after updating Kernel
For me it was. Elrepo work's fine for me I had the same issue before when I used the old separate dkms build from the WG repo. 21 Jun 2020 14:45:44 Jean-Dens Girard : > Le 18/06/2020 à 10:11, Jason A. Donenfeld a écrit : >> On Thu, Jun 18, 2020 at 2:10 PM Jean-Denis Girard >> wrote: >>> I cannot reboot now, I will let you know how it goes later. > > Reboot with newer kernel and latest wireguard still hangs. > >> Oh, in your case, you appear to be using the dkms package instead of >> the elrepo package. > > Is that a problem? > > Thanks, > > -- > > Jean-Denis Girard > > SysNux Systèmes Linux en Polynésie française > > https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527 >
Re: Kernel Panic after updating Kernel
On Fri, Jun 19, 2020 at 12:58 AM Jean-Dens Girard wrote: > > Le 18/06/2020 à 10:11, Jason A. Donenfeld a écrit : > > On Thu, Jun 18, 2020 at 2:10 PM Jean-Denis Girard > > wrote: > >> I cannot reboot now, I will let you know how it goes later. > > Reboot with newer kernel and latest wireguard still hangs. > > > Oh, in your case, you appear to be using the dkms package instead of > > the elrepo package. > > Is that a problem? Remove all wireguard packages. Update to the latest kernel. Run `find /lib/modules -name '*wireguard*' -print -delete`. Then install kmod-wireguard. For further help, you're probably best off asking on IRC. From debugging this with the original reporter there, this turned out to be a mixing and matching of versions issue due to weird yum things.
Re: Kernel Panic after updating Kernel
Le 18/06/2020 à 10:11, Jason A. Donenfeld a écrit : On Thu, Jun 18, 2020 at 2:10 PM Jean-Denis Girard wrote: I cannot reboot now, I will let you know how it goes later. Reboot with newer kernel and latest wireguard still hangs. Oh, in your case, you appear to be using the dkms package instead of the elrepo package. Is that a problem? Thanks, -- Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527
Re: Kernel Panic after updating Kernel
On Thu, Jun 18, 2020 at 2:10 PM Jean-Denis Girard wrote: > > Le 18/06/2020 à 09:48, Jason A. Donenfeld a écrit : > > I am unable to reproduce this issue with vmxnet3. However, as I noted > > earlier, your wireguard version seems old. Try updating everything at > > once, and then see. > > yum updated to wireguard-dkms.noarch 1:1.0.20200611-1.el7 > > By the way, yum complains : > Error! Could not locate dkms.conf file. > File: /var/lib/dkms/wireguard/0.0.20181218/source/dkms.conf does not exist. > > I cannot reboot now, I will let you know how it goes later. Oh, in your case, you appear to be using the dkms package instead of the elrepo package.
Re: Kernel Panic after updating Kernel
Le 18/06/2020 à 09:48, Jason A. Donenfeld a écrit : > I am unable to reproduce this issue with vmxnet3. However, as I noted > earlier, your wireguard version seems old. Try updating everything at > once, and then see. yum updated to wireguard-dkms.noarch 1:1.0.20200611-1.el7 By the way, yum complains : Error! Could not locate dkms.conf file. File: /var/lib/dkms/wireguard/0.0.20181218/source/dkms.conf does not exist. I cannot reboot now, I will let you know how it goes later. Thanks, -- Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527 signature.asc Description: OpenPGP digital signature
Re: Kernel Panic after updating Kernel
On 18/06/2020 05:31, dx...@xirihosting.com wrote: 6) Yum operations trigger a lot of exclutions for elrepo, but nothing seems wireguard related: Not related to this bug, so for information only. The following is caused by a difference in the way CentOS compose their repositories over RHEL: https://bugs.centos.org/view.php?id=15476 The solution is to enable the CentOS vault repo which will allow CentOS to more closely match RHEL behaviour and prevent the exclusions notified below. This is documented in /usr/share/doc/yum-plugin-elrepo-7.5.1/README Loaded plugins: changelog, elrepo, fastestmirror, priorities, tsflags, universal-hooks Loading mirror speeds from cached hostfile * EA4: 208.100.0.204 * cpanel-addons-production-feed: 208.100.0.204 * cpanel-plugins: 208.100.0.204 * elrepo: elrepo.0m3n.net * epel: mirror.csis.ysu.edu [elrepo]: excluding package: kmod-3c59x-0.0-3.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-8188eu-4.1.4_6773.20130222-4.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-8188eu-4.1.4_6773.20130222-5.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-8188eu-5.2.2.4-1.20190907git.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-a2818-1.20-1.el7.elrepo.x86_64 [elrepo]: excluding package: kmod-a3818-1.6.0-1.el7.elrepo.x86_64 [elrepo]: excluding package: kmod-a3818-1.6.2-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-aacraid-1.2.1-5.el7.elrepo.x86_64 [elrepo]: excluding package: kmod-aic7xxx-7.0-3.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-ar5523-0.0-8.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-ar5523-0.0-9.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-ath5k-0.0-12.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-cassini-1.6-2.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-cciss-3.6.26-5.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-cciss-3.6.26-6.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-cciss-3.6.26-7.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-drbd84-8.4.11-1.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-drbd84-8.4.11-1.1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-drbd90-9.0.14-1.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-drbd90-9.0.16-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-drbd90-9.0.20-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-e100-3.5.24-3.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-ecryptfs-0.1-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-forcedeth-0.64-3.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-fpga-mgr-0.0-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-hfs-0.0-4.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-hfsplus-0.0-5.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-i2c-i801-0.0-4.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-i2c-i801-0.0-5.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-i2c-i801-0.0-6.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-ixgb-1.0.135-4.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-ixgbe-5.5.5-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-ixgbe-5.6.3-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-ixgbe-5.6.3-2.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-joydev-0.0-4.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-mt7601u-4.14.108-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-mt7601u-4.14.108-2.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-nct6775-0.0-4.20180327git.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-nct6775-0.0-5.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-ne2k-pci-1.03-4.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-netatop-0.3-4.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-netatop-2.0-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-niu-1.1-2.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-nvidia-440.44-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: nvidia-x11-drv-libs-440.44-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: nvidia-x11-drv-libs-440.44-1.el7_7.elrepo.i686 [elrepo]: excluding package: nvidia-x11-drv-440.44-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-nvidia-440.59-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: nvidia-x11-drv-libs-440.59-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: nvidia-x11-drv-440.59-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: nvidia-x11-drv-libs-440.59-1.el7_7.elrepo.i686 [elrepo]: excluding package: kmod-nvidia-440.64-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: nvidia-x11-drv-libs-440.64-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: nvidia-x11-drv-libs-440.64-1.el7_7.elrepo.i686 [elrepo]: excluding package: nvidia-x11-drv-440.64-1.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-nvidia-340xx-340.107-2.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-nvidia-340xx-340.107-3.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-nvidia-390xx-390.116-1.el7_6.elrepo.x86_64 [elrepo]:
Re: Kernel Panic after updating Kernel
On Thu, Jun 18, 2020 at 10:48 AM Jean-Denis Girard wrote: > For what it's worth, I seem to have the same problem on a CentOS-7 > virtual machine hosted on VMware with vmxnet3. It has been working fine > since installed in 2017, but does lock up after upgrading to > kernel-3.10.0-1127.8.2.el7.x86_64 or kernel-3.10.0-1127.10.1.el7.x86_64. > The VM is now running fine on kernel-3.10.0-1062.18.1.el7.x86_64. I am unable to reproduce this issue with vmxnet3. However, as I noted earlier, your wireguard version seems old. Try updating everything at once, and then see.
Re: Kernel Panic after updating Kernel
On Thu, Jun 18, 2020 at 10:48 AM Jean-Denis Girard wrote: > [ 17.886512] wireguard: WireGuard 1.0.20200520 loaded. See > www.wireguard.com for information. 20200520 is old. Have you tried the newer version yet?
Re: Kernel Panic after updating Kernel
Hi list, Le 17/06/2020 à 19:53, Jason A. Donenfeld a écrit : > Hmm, still not able to reproduce. > > Are you sure you're running the latest up to date module? Try > uninstalling kmod-wireguard and reinstalling? > > What driver is your ethernet NIC using? > For what it's worth, I seem to have the same problem on a CentOS-7 virtual machine hosted on VMware with vmxnet3. It has been working fine since installed in 2017, but does lock up after upgrading to kernel-3.10.0-1127.8.2.el7.x86_64 or kernel-3.10.0-1127.10.1.el7.x86_64. The VM is now running fine on kernel-3.10.0-1062.18.1.el7.x86_64. [4.751926] NET: Registered protocol family 40 [5.008840] vmxnet3 :03:00.0 ens160: intr type 3, mode 0, 3 vectors allocated [5.009298] vmxnet3 :03:00.0 ens160: NIC Link is Up 1 Mbps [9.148571] vmxnet3 :13:00.0 ens224: intr type 3, mode 0, 3 vectors allocated [9.149062] vmxnet3 :13:00.0 ens224: NIC Link is Up 1 Mbps [ 13.318360] vmxnet3 :1b:00.0 ens256: intr type 3, mode 0, 3 vectors allocated [ 13.318908] vmxnet3 :1b:00.0 ens256: NIC Link is Up 1 Mbps [ 17.704052] FS-Cache: Loaded [ 17.823986] FS-Cache: Netfs 'nfs' registered for caching [ 17.837062] Key type dns_resolver registered [ 17.867211] NFS: Registering the id_resolver key type [ 17.867218] Key type id_resolver registered [ 17.867220] Key type id_legacy registered [ 17.879846] wireguard: module verification failed: signature and/or required key missing - tainting kernel [ 17.886512] wireguard: WireGuard 1.0.20200520 loaded. See www.wireguard.com for information. [ 17.886514] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld . All Rights Reserved. [ 564.297446] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) Thanks, -- Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527 signature.asc Description: OpenPGP digital signature
Re: Kernel Panic after updating Kernel
Hmm, still not able to reproduce. Are you sure you're running the latest up to date module? Try uninstalling kmod-wireguard and reinstalling? What driver is your ethernet NIC using?
Re: Kernel Panic after updating Kernel
-Original Message- From: "Jason A. Donenfeld" Sent: Wednesday, June 17, 2020 4:32am To: "dx...@xirihosting.com" Cc: "WireGuard mailing list" , "ElRepo" Subject: Re: Kernel Panic after updating Kernel Hi Diego, On Wed, Jun 17, 2020 at 2:01 AM dx...@xirihosting.com wrote: > > Posted this on IRC but got no response, probably this will be a better place: > > I updated my Centos7 server yesterday and wireguard is causing a kernel > panic, wanted to know if this is a known issue? > > Using kernel 3.10.0-1127.10.1.el7.x86_64 > > I Tried with 2 different repos (elrepo and Copr repo for wireguard owned by > jdoss) and I have the same issue. > > I took a screenshot of The kernel panic and uploaded at > https://imgur.com/a/Ojxeor0 > > Another interesting bit of info is that as long as I don't move traffic > trough wg0 vnic, no panic happens. I can easily trigger the panic by just > doing a ping to the other VPN endpoint and I am able to reproduce this every > single time. > > # lsmod | grep -i wire > wireguard 200896 0 > ip6_udp_tunnel 12755 1 wireguard > udp_tunnel 14423 1 wireguard > > Thanks for the help! > Diego Huh, that's funny -- I'm unable to reproduce the bug at all. Does running this script crash for you? https://salsa.debian.org/debian/wireguard-linux-compat/-/raw/debian/master/debian/tests/netns-mini If not, could you describe your setup more and maybe some repro steps for me? Thanks, Jason Hi Jason, Tried your script, here is the result (spoiler...no crash): root@box [4542 22:04:00 /etc/wireguard]# bash netns-mini-test.sh [+] ip netns add wg-test-36633-0 [+] ip netns add wg-test-36633-1 [+] ip netns add wg-test-36633-2 [+] NS0: ip link set up dev lo [+] NS0: ip link add dev wg0 type wireguard [+] NS0: ip link set wg0 netns wg-test-36633-1 [+] NS0: ip link add dev wg0 type wireguard [+] NS0: ip link set wg0 netns wg-test-36633-2 [+] NS1: ip addr add 192.168.241.1/24 dev wg0 [+] NS2: ip addr add 192.168.241.2/24 dev wg0 [+] wg genkey [+] wg genkey [+] wg pubkey [+] wg pubkey [+] NS1: wg set wg0 private-key /dev/fd/63 listen-port 1 peer 0zUBih0nFOHRDzl6mBxcHaCfwX+s9sE6rLgK4f8LdiU= allowed-ips 192.168.241.2/32 [+] NS2: wg set wg0 private-key /dev/fd/63 listen-port 2 peer jBLy+DQDc21/f5z9Yfz287FB/Z9PhaXvNdMw7WDoWFM= allowed-ips 192.168.241.1/32 [+] NS1: ip link set up dev wg0 [+] NS2: ip link set up dev wg0 [+] NS2: wg set wg0 peer jBLy+DQDc21/f5z9Yfz287FB/Z9PhaXvNdMw7WDoWFM= endpoint 127.0.0.1:1 [+] NS2: ping -c 10 -f -W 1 192.168.241.1 PING 192.168.241.1 (192.168.241.1) 56(84) bytes of data. --- 192.168.241.1 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 1ms rtt min/avg/max/mdev = 0.054/0.159/0.884/0.243 ms, ipg/ewma 0.209/0.316 ms [+] NS0: ip link del dev wg0 [+] NS1: ip link del dev wg0 [+] NS2: ip link del dev wg0 [+] ip netns del wg-test-36633-1 [+] ip netns del wg-test-36633-2 [+] ip netns del wg-test-36633-0 About my setup: 1) KVM hosted VM 2) Using wg-quick, followed this tutorial: https://www.stavros.io/posts/how-to-configure-wireguard/ 3) CPanel v88.0.10 (as far as I know, CPanel does NOT modify stock Centos 7 kernel) 4) root@box [4545 22:07:54 /etc/wireguard]# free -m totalusedfree shared buff/cache available Mem: 23631373 174 12 815 793 Swap: 19991637 362 5) root@box [4547 22:10:37 /etc/wireguard]# cat wg0.conf [Interface] Address = 192.168.100.101/28 PrivateKey = 000xxxpjdlkkljkljalkjlkjl= ListenPort = 11555 [Peer] PublicKey = djkjadlkjlkjkldjlkjaslkjadlk= AllowedIPs = 192.168.100.100/32 Endpoint = 1.1.1.1:11555 6) Yum operations trigger a lot of exclutions for elrepo, but nothing seems wireguard related: Loaded plugins: changelog, elrepo, fastestmirror, priorities, tsflags, universal-hooks Loading mirror speeds from cached hostfile * EA4: 208.100.0.204 * cpanel-addons-production-feed: 208.100.0.204 * cpanel-plugins: 208.100.0.204 * elrepo: elrepo.0m3n.net * epel: mirror.csis.ysu.edu [elrepo]: excluding package: kmod-3c59x-0.0-3.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-8188eu-4.1.4_6773.20130222-4.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-8188eu-4.1.4_6773.20130222-5.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-8188eu-5.2.2.4-1.20190907git.el7_7.elrepo.x86_64 [elrepo]: excluding package: kmod-a2818-1.20-1.el7.elrepo.x86_64 [elrepo]: excluding package: kmod-a3818-1.6.0-1.el7.elrepo.x86_64 [elrepo]: excluding package: kmod-a3818-1.6.2-1.el7_6.elrepo.x86_64 [elrepo]: excluding package: kmod-aacraid-1.2.1-5.el7.elrepo.x86_64 [elrepo]: excluding package: kmod-aic7xxx-7.0-3.el7_5.elrepo.x86_64 [elrepo]: excluding package: kmod-ar5523-0.0-8.el7_6.elrepo.x86_64 [elrepo]: excluding pac
Re: Kernel Panic after updating Kernel
Hi Diego, On Wed, Jun 17, 2020 at 2:01 AM dx...@xirihosting.com wrote: > > Posted this on IRC but got no response, probably this will be a better place: > > I updated my Centos7 server yesterday and wireguard is causing a kernel > panic, wanted to know if this is a known issue? > > Using kernel 3.10.0-1127.10.1.el7.x86_64 > > I Tried with 2 different repos (elrepo and Copr repo for wireguard owned by > jdoss) and I have the same issue. > > I took a screenshot of The kernel panic and uploaded at > https://imgur.com/a/Ojxeor0 > > Another interesting bit of info is that as long as I don't move traffic > trough wg0 vnic, no panic happens. I can easily trigger the panic by just > doing a ping to the other VPN endpoint and I am able to reproduce this every > single time. > > # lsmod | grep -i wire > wireguard 200896 0 > ip6_udp_tunnel 12755 1 wireguard > udp_tunnel 14423 1 wireguard > > Thanks for the help! > Diego Huh, that's funny -- I'm unable to reproduce the bug at all. Does running this script crash for you? https://salsa.debian.org/debian/wireguard-linux-compat/-/raw/debian/master/debian/tests/netns-mini If not, could you describe your setup more and maybe some repro steps for me? Thanks, Jason
