[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
We updated our test instances to run on the latest hardware made available in AWS, I have not seen this reoccur in the xenial testing. closing bug. ** Changed in: ubuntu-kernel-tests Status: In Progress => Invalid ** Changed in: linux-aws (Ubuntu) Status: In Progress => Invalid ** Changed in: ubuntu-kernel-tests Assignee: Sean Feole (sfeole) => (unassigned) ** Changed in: linux-aws (Ubuntu) Assignee: Stefan Bader (smb) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
It might be fixed if AWS runs a Xen hypervisor which has the following patch included (this is from the development tree of upstream Xen, so will be part of Xen-4.12). commit be07023be115c94b7fbb51d2ef6f421ddd680de8 Author: Roger Pau Monné Date: Tue Jul 24 15:54:18 2018 +0200 x86/vhpet: add support for level triggered interrupts One can never say for sure what AWS runs, so whether its fixed or not can only be found out by trial and error. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
Been sorting through many of the ubuntu-kernel-tests bugs. This is one of the few that actually is being worked. Stefan, any update on this? Should this be/ Has it been fixed? I can revisit once i finish cleaning up the list ** Changed in: ubuntu-kernel-tests Status: New => In Progress ** Changed in: ubuntu-kernel-tests Assignee: (unassigned) => Sean Feole (sfeole) ** Changed in: ubuntu-kernel-tests Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
** Also affects: ubuntu-kernel-tests Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
Right now I do not think there is much choice to fix this (other than not touch /dev/hpet on AWS). The linux kernel deliberately wants to set a level triggered interrupt. The xen hypervisor has no support for that (there might be some addition done but certainly not in any released version of Xen). And as "error handling" forcefully crashes the domain. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
** Changed in: linux-aws (Ubuntu) Assignee: Colin Ian King (colin-king) => Stefan Bader (smb) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
Darn, ok I take everything back. Somehow the compiled reproducer was mangled in such a way it did maybe no longer do what it was intended to do. Anyhow, with freshly generated reproducers, even Xen 4.9 has the crash. :( -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
I was booting the same Xenial based HVM guest on the same host (but this time running Bionic / Xen 4.9). This combination does not crash the domain when opening HPET. Though the check and code that would do it is still there. I also found a bug report against xenserver which I believe is based on the same Xen version as we have in Xenial (4.6.5): https://bugs.xenserver.org/browse/XSO-809?focusedCommentId=16484=com.atlassian.jira.plugin.system.issuetabpanels %3Acomment-tabpanel This and the code say that the crash is done because HPET is set to use an unsupported interrupt method (edge/level). Since the Linux guest is the same in both cases, and also the test + crash code, either the hypervisor or maybe the seabios seem to use a different default. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
I was able to observe the crash on a Ubuntu Xenial Xen host which produced the following text on the host console: (XEN) domain_crash called from hpet.c:387 (XEN) Domain 2 (vcpu#1) crashed on cpu#4: (XEN) [ Xen-4.6.5 x86_64 debug=n Not tainted ] (XEN) CPU:4 (XEN) RIP:0010:[] (XEN) RFLAGS: 00010002 CONTEXT: hvm guest (d2v1) (XEN) rax: 0032 rbx: 880034e41a00 rcx: (XEN) rdx: 0001 rsi: 0032 rdi: 821fdfb0 (XEN) rbp: 8800e90afc10 rsp: 8800e90afbd8 r8: 0003 (XEN) r9: r10: 000a r11: (XEN) r12: 880107a4a8f8 r13: 821fdfb0 r14: c9002140 (XEN) r15: 81a7c600 cr0: 80050033 cr4: 00360670 (XEN) cr3: 3492c000 cr2: 7fcc1156a030 (XEN) ds: es: fs: gs: ss: 0018 cs: 0010 Will investigate further (whether this persists in newer xen versions) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
demsg of guest: [0.00] Linux version 4.16.0-rc2+ (cking@gloin) (gcc version 7.3.0 (Ubuntu 7.3.0-3ubuntu1)) #7 SMP Tue Feb 20 14:27:20 UTC 2018 [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-4.16.0-rc2+ root=UUID=b6adc449-5e3d-4331-ba6b-6e99a75fa48e ro console=tty1 console=ttyS0 nvme.io_timeout=4294967295 [0.00] KERNEL supported cpus: [0.00] Intel GenuineIntel [0.00] AMD AuthenticAMD [0.00] Centaur CentaurHauls [0.00] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] e820: BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009dfff] usable [0.00] BIOS-e820: [mem 0x0009e000-0x0009] reserved [0.00] BIOS-e820: [mem 0x000e-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0x3fff] usable [0.00] BIOS-e820: [mem 0xfc00-0x] reserved [0.00] NX (Execute Disable) protection: active [0.00] random: fast init done [0.00] SMBIOS 2.7 present. [0.00] DMI: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [0.00] Hypervisor detected: Xen HVM [0.00] Xen version 4.2. [0.00] Xen Platform PCI: I/O protocol version 1 [0.00] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs. [0.00] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks. You might have to change the root device from /dev/hd[a-d] to /dev/xvd[a-d] in your root= kernel command line option [0.00] HVMOP_pagetable_dying not supported [0.00] e820: update [mem 0x-0x0fff] usable ==> reserved [0.00] e820: remove [mem 0x000a-0x000f] usable [0.00] e820: last_pfn = 0x4 max_arch_pfn = 0x4 [0.00] MTRR default type: write-back [0.00] MTRR fixed ranges enabled: [0.00] 0-9 write-back [0.00] A-B write-combining [0.00] C-F write-back [0.00] MTRR variable ranges enabled: [0.00] 0 base F000 mask 3800 uncachable [0.00] 1 base F800 mask 3C00 uncachable [0.00] 2 disabled [0.00] 3 disabled [0.00] 4 disabled [0.00] 5 disabled [0.00] 6 disabled [0.00] 7 disabled [0.00] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [0.00] found SMP MP-table at [mem 0x000fbc20-0x000fbc2f] mapped at [ (ptrval)] [0.00] Scanning 1 areas for low memory corruption [0.00] Base memory trampoline at [(ptrval)] 98000 size 24576 [0.00] BRK [0x0ff42000, 0x0ff42fff] PGTABLE [0.00] BRK [0x0ff43000, 0x0ff43fff] PGTABLE [0.00] BRK [0x0ff44000, 0x0ff44fff] PGTABLE [0.00] BRK [0x0ff45000, 0x0ff45fff] PGTABLE [0.00] RAMDISK: [mem 0x346e2000-0x36368fff] [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x000EA020 24 (v02 Xen ) [0.00] ACPI: XSDT 0xFC00DDC0 54 (v01 XenHVM HVML ) [0.00] ACPI: FACP 0xFC00DA80 F4 (v04 XenHVM HVML ) [0.00] ACPI: DSDT 0xFC001CE0 00BD19 (v02 XenHVM INTL 20090123) [0.00] ACPI: FACS 0xFC001CA0 40 [0.00] ACPI: FACS 0xFC001CA0 40 [0.00] ACPI: APIC 0xFC00DB80 D8 (v02 XenHVM HVML ) [0.00] ACPI: HPET 0xFC00DCD0 38 (v01 XenHVM HVML ) [0.00] ACPI: WAET 0xFC00DD10 28 (v01 XenHVM HVML ) [0.00] ACPI: SSDT 0xFC00DD40 31 (v02 XenHVM INTL 20090123) [0.00] ACPI: SSDT 0xFC00DD80 31 (v02 XenHVM INTL 20090123) [0.00] ACPI: Local APIC address 0xfee0 [0.00] No NUMA configuration found [0.00] Faking a node at [mem 0x-0x3fff] [0.00] NODE_DATA(0) allocated [mem 0x3ffd5000-0x3fff] [0.00] tsc: Fast TSC calibration using PIT [0.00] Zone ranges: [0.00] DMA [mem 0x1000-0x00ff] [0.00] DMA32[mem 0x0100-0x3fff] [0.00] Normal empty [0.00] Device empty [0.00] Movable zone start for each
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
I can reproduce this with 4.16-rc2, I've debugged this down to: drivers/char/hpet.c, hpet_timer_set_irq(): if (irq < HPET_MAX_IRQ) { spin_lock_irq(_lock); v = readl(>hpet_config); v |= irq << Tn_INT_ROUTE_CNF_SHIFT; writel(v, >hpet_config); .. the writel to hpet_config causes the reboot. How to reproduce this issue: git clone git://kernel.ubuntu.com/cking/stress-ng cd stress-ng git revert 0124b250ec205ea3cd6d9d68fb96c03ac294d12f make sudo ./stress-ng --dev 1 .. wait a while and it will eventually get around to the /dev/hpet and opening this causes the hang. The minimal reproducer is: #include #include #include #include #include int main(void) { int fd; fd = open("/dev/hpet", O_RDONLY | O_NONBLOCK); if (fd > 0) close(fd); exit(0); } run this as root and it will cause the reboot. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
Tested with 4.4.0-109 lowlatency kernel, this dev test can pass now. I will leave this bug open as discussed on the IRC. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
Do you mind re-running the test to see if we get passed this stress test now? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
..and way back to v4.0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
issue occurs with v4.15-rc7 upstream kernel too -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
4.4.0-73 has the same issue, so it's not an aws specific kernel issue per se. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
This is locking up on opening a specific device. It is not a race condition as I originally suspected, but a lockup on a simple read open of a device on just AWS. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
** Changed in: linux-aws (Ubuntu) Importance: Undecided => High ** Changed in: linux-aws (Ubuntu) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1741409] Re: stress smoke test hang with dev test on AWS Xenial kernel
Manually tested with older kernel (4.4.0-1043-aws), this issue still can be reproduced. The node will get rebooted when bumping into this test. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1741409 Title: stress smoke test hang with dev test on AWS Xenial kernel To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1741409/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs