On Thursday, October 15, 2015 05:46:07 PM hw wrote: > J. Roeleveld wrote: > > On Thursday, October 15, 2015 03:30:01 PM hw wrote: > >> Hi, > >> > >> I have a xen host with some HV guests which becomes unreachable via > >> the network after apparently random amount of times. I have already > >> switched the network card to see if that would make a difference, > >> and with the card currently installed, it worked fine for over 20 days > >> until it become unreachable again. Before switching the network card, > >> it would run a week or two before becoming unreachable. The previous > >> card was the on-board BCM5764M which uses the tg3 driver. > >> > >> There are messages like this in the log file: > >> > >> > >> Oct 14 20:58:02 moonflo kernel: ------------[ cut here ]------------ > >> Oct 14 20:58:02 moonflo kernel: WARNING: CPU: 10 PID: 0 at > >> net/sched/sch_generic.c:303 dev_watchdog+0x259/0x270() Oct 14 20:58:02 > >> moonflo kernel: NETDEV WATCHDOG: enp55s4 (r8169): transmit queue 0 timed > >> out Oct 14 20:58:02 moonflo kernel: Modules linked in: arc4 ecb md4 hmac > >> nls_utf8 cifs fscache xt_physdev br_netfilter iptable_filter ip_tables > >> xen_pciback xen_gntalloc xen_gntdev bridge stp llc zfs(PO) nouveau > >> snd_hda_codec_realtek snd_hda_codec_generic zunicode(PO) zavl(PO) > >> zcommon(PO) znvpair(PO) spl(O) zlib_deflate video backlight > >> drm_kms_helper > >> ttm snd_hda_intel snd_hda_controller snd_hda_codec snd_pcm snd_timer snd > >> soundcore r8169 mii xts aesni_intel glue_helper lrw gf128mul ablk_helper > >> cryptd aes_x86_64 sha256_generic hid_generic usbhid uhci_hcd usb_storage > >> ehci_pci ehci_hcd usbcore usb_common Oct 14 20:58:02 moonflo kernel: CPU: > >> 10 PID: 0 Comm: swapper/10 Tainted: P O 4.0.5-gentoo #3 Oct > >> 14 > >> 20:58:02 moonflo kernel: Hardware name: Hewlett-Packard HP Z800 > >> Workstation/0AECh, BIOS 786G5 v03.57 07/15/2013 Oct 14 20:58:02 moonflo > >> kernel: ffffffff8175a77d ffff880124d43d98 ffffffff814da8d8 > >> 0000000000000001 Oct 14 20:58:02 moonflo kernel: ffff880124d43de8 > >> ffff880124d43dd8 ffffffff81088850 ffff880124d43dd8 Oct 14 20:58:02 > >> moonflo > >> kernel: 0000000000000000 ffff8800d45f2000 0000000000000001 > >> ffff8800d5294880 Oct 14 20:58:02 moonflo kernel: Call Trace: > >> Oct 14 20:58:02 moonflo kernel: <IRQ> [<ffffffff814da8d8>] > >> dump_stack+0x45/0x57 Oct 14 20:58:02 moonflo kernel: > >> [<ffffffff81088850>] > >> warn_slowpath_common+0x80/0xc0 Oct 14 20:58:02 moonflo kernel: > >> [<ffffffff810888d1>] warn_slowpath_fmt+0x41/0x50 Oct 14 20:58:02 moonflo > >> kernel: [<ffffffff812b31c5>] ? add_interrupt_randomness+0x35/0x1e0 Oct > >> 14 > >> 20:58:02 moonflo kernel: [<ffffffff8145b819>] dev_watchdog+0x259/0x270 > >> Oct > >> 14 20:58:02 moonflo kernel: [<ffffffff8145b5c0>] ? > >> dev_graft_qdisc+0x80/0x80 Oct 14 20:58:02 moonflo kernel: > >> [<ffffffff8145b5c0>] ? dev_graft_qdisc+0x80/0x80 Oct 14 20:58:02 moonflo > >> kernel: [<ffffffff810d4047>] call_timer_fn.isra.30+0x17/0x70 Oct 14 > >> 20:58:02 moonflo kernel: [<ffffffff810d42a6>] > >> run_timer_softirq+0x176/0x2b0 Oct 14 20:58:02 moonflo kernel: > >> [<ffffffff8108bd0a>] __do_softirq+0xda/0x1f0 Oct 14 20:58:02 moonflo > >> kernel: [<ffffffff8108c04e>] irq_exit+0x7e/0xa0 Oct 14 20:58:02 moonflo > >> kernel: [<ffffffff8130e075>] xen_evtchn_do_upcall+0x35/0x50 Oct 14 > >> 20:58:02 moonflo kernel: [<ffffffff814e1e8e>] > >> xen_do_hypervisor_callback+0x1e/0x40 Oct 14 20:58:02 moonflo kernel: > >> <EOI> > >> > >> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 Oct 14 20:58:02 > >> > >> moonflo kernel: [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20 > >> Oct > >> 14 20:58:02 moonflo kernel: [<ffffffff810459e0>] ? > >> xen_safe_halt+0x10/0x20 > >> Oct 14 20:58:02 moonflo kernel: [<ffffffff81053979>] ? > >> default_idle+0x9/0x10 Oct 14 20:58:02 moonflo kernel: > >> [<ffffffff810542da>] > >> ? arch_cpu_idle+0xa/0x10 Oct 14 20:58:02 moonflo kernel: > >> [<ffffffff810bd170>] ? cpu_startup_entry+0x190/0x2f0 Oct 14 20:58:02 > >> moonflo kernel: [<ffffffff81047cd5>] ? cpu_bringup_and_idle+0x25/0x40 > >> Oct > >> 14 20:58:02 moonflo kernel: ---[ end trace 98d961bae351244d ]--- Oct 14 > >> 20:58:02 moonflo kernel: r8169 0000:37:04.0 enp55s4: link up > >> > >> > >> After that, there are lots of messages about the link being up, one > >> message > >> every 12 seconds. When you unplug the network cable, you get a message > >> that the link is down, and no message when you plug it in again. > >> > >> I was hoping that switching the network card (to one that uses a > >> different > >> driver) might solve the problem, and it did not. Now I can only guess > >> that > >> the network card goes to sleep and sometimes cannot be woken up again. > >> > >> I tried to reduce the connection speed to 100Mbit and found that > >> accessing > >> the VMs (via RDP) becomes too slow to use them. So I disabled the power > >> management of the network card (through sysfs) and will have to see if > >> the > >> problem persists. > >> > >> We'll be getting decent network cards in a couple days, but since the > >> problem doesn't seem to be related to a particular > >> card/model/manufacturer, > >> that might not fix it, either. > >> > >> This problem seems to only occur on machines that operate as a xen > >> server. > >> Other machines, identical Z800s, not running xen, run just fine. > >> > >> What would you suggest? > > > > More info required: > > > > - Which version of Xen > > 4.5.1 > > Installed versions: 4.5.1^t(02:44:35 PM 07/14/2015)(-custom-cflags -debug > -efi -flask -xsm)
Ok, recent one. > > - Does this only occur with HVM guests? > > The host has been running only HVM guests every time it happend. > It was running a PV guest in between (which I had to shut down > because other VMs were migrated, requiring the RAM). The PV didn't have any issues? > > - Which network-driver are you using inside the guest > > r8169, compiled as a module > > Same happened with the tg3 driver when the on-board cards were used. > The tg3 driver is completely disabled in the kernel config, i. e. > not even compiled as a module. You have network cards assigned to the guests? > > - Can you connect to the "local" console of the guest? > > Yes, the host seems to be running fine except for having no network > connectivity. There's a keyboard and monitor physically connected to > it with which you can log in and do stuff. The HOST loses network connectivity? > You get no answer when you ping the host while it is unreachable. > > > - If yes, does it still have no connectivity? > > It has been restarted this morning when it was found to be unreachable. > > > I saw the same on my lab machine, which was related to: > > - Not using correct drivers inside HVM guests > > There are Windoze 7 guests running that have PV drivers installed. > One of those has formerly been running on a VMware host and was > migrated on Tuesday. I deinstalled the VMware tools from it. Which PV drivers? And did you ensure all VMWare related drivers were removed? I am not convinced uninstalling the VMWare tools is sufficient. > Since Monday, a HVM Linux system (a modified 32-bit Debian) has also > been migrated from the VMware host to this one. I don't know if it > has VMware tools installed (I guess it does because it could be shut > down via VMware) and how those might react now. It's working, and I > don't want to touch it. > > However, the problem already occured before this migration, when the > on-board cards were still used. > > > - Switch hardware not keeping the MAC/IP/Port lists long enough > > What might be the reason for the lists becoming too short? Too many > devices connected to the network? No network activity for a while. (clean installs, nothing running) Switch forgetting the MAC-address assigned to the VM. Connecting to the VM-console, I could ping www.google.com and then the connectivity re-appeared. > The host has been connected to two different switches and showed the > problem. Previously, that was an 8-port 1Gb switch, now it's a 24-port > 1Gb switch. However, the 8-port switch is also connected to the 24-port > switch the host is now connected to. (The 24-port switch connects it > "directly" to the rest of the network.) Assuming it's a managed switch, you could test this. Alternatively, check if you can access the VMs from the host. -- Joost
