Re: interrupt storm on ahci
On Wed, Jun 11, 2014 at 09:18:17AM +, Steve Wills wrote: On Sat, Jun 07, 2014 at 02:53:05PM -0700, Neel Natu wrote: Hi Steve, On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote: Hi Steve, I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running FreeBSD CURRENT, r266947. I've gotten this message about 12 times since boot: interrupt storm detected on irq268:; throttling interrupt source vmstat -i shows: irq268: ahci1 236514222839 ahci1 is the second disk connected to the system: ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18 at device 4.0 on pci0 The VM itself runs poudriere and was building a bunch of packages. At the moment, the VM seems to be in a rather odd state. The poudriere jails are running, but not doing anything. Ideas? Is this an 8.* host ? I don't believe AHCI has MSI support on that version, and AHCI legacy interrupts haven't had a huge amount of testing under load. If it is 8.*, I'd recommend using virtio-blk for the block device until we can work out what's going wrong. The KTR trace would be useful to figure out what's happening. To do that you can compile the host kernel and vmm.ko with the following options: options KTR options KTR_MASK=(KTR_GEN) options KTR_ENTRIES=(4*1024*1024) And when you see the interrupt storm message in the guest you can execute: sudo ktrdump -cto /tmp/ktrdump.out This was added to the kernel config, the kernel was built and installed and the host was rebooted. I started the VM and almost as soon as the workload was put back, the messages started again. So, I ran the ktrdump command above. I'll mail you privately with the location of the output. Replying to myself just to note for the record that the interrupt storm was harmless, because the throttling only reduced performance (I thought another issue I was having at the same time was a result, but it was just a coincidence) and that the default threshold for interrupt storms of 1000 interrupts per second may be too low in some cases, such as this one. I disabled the throttling by setting the hw.intr_storm_threshold sysctl to 0 and haven't had any issues. Steve ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: interrupt storm on ahci
On Sat, Jun 07, 2014 at 02:53:05PM -0700, Neel Natu wrote: Hi Steve, On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote: Hi Steve, I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running FreeBSD CURRENT, r266947. I've gotten this message about 12 times since boot: interrupt storm detected on irq268:; throttling interrupt source vmstat -i shows: irq268: ahci1 236514222839 ahci1 is the second disk connected to the system: ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18 at device 4.0 on pci0 The VM itself runs poudriere and was building a bunch of packages. At the moment, the VM seems to be in a rather odd state. The poudriere jails are running, but not doing anything. Ideas? Is this an 8.* host ? I don't believe AHCI has MSI support on that version, and AHCI legacy interrupts haven't had a huge amount of testing under load. If it is 8.*, I'd recommend using virtio-blk for the block device until we can work out what's going wrong. The KTR trace would be useful to figure out what's happening. To do that you can compile the host kernel and vmm.ko with the following options: options KTR options KTR_MASK=(KTR_GEN) options KTR_ENTRIES=(4*1024*1024) And when you see the interrupt storm message in the guest you can execute: sudo ktrdump -cto /tmp/ktrdump.out This was added to the kernel config, the kernel was built and installed and the host was rebooted. I started the VM and almost as soon as the workload was put back, the messages started again. So, I ran the ktrdump command above. I'll mail you privately with the location of the output. Thanks, Steve ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: interrupt storm on ahci
Hi Steve, I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running FreeBSD CURRENT, r266947. I've gotten this message about 12 times since boot: interrupt storm detected on irq268:; throttling interrupt source vmstat -i shows: irq268: ahci1 236514222839 ahci1 is the second disk connected to the system: ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18 at device 4.0 on pci0 The VM itself runs poudriere and was building a bunch of packages. At the moment, the VM seems to be in a rather odd state. The poudriere jails are running, but not doing anything. Ideas? Is this an 8.* host ? I don't believe AHCI has MSI support on that version, and AHCI legacy interrupts haven't had a huge amount of testing under load. If it is 8.*, I'd recommend using virtio-blk for the block device until we can work out what's going wrong. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: interrupt storm on ahci
Hi Steve, On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote: Hi Steve, I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running FreeBSD CURRENT, r266947. I've gotten this message about 12 times since boot: interrupt storm detected on irq268:; throttling interrupt source vmstat -i shows: irq268: ahci1 236514222839 ahci1 is the second disk connected to the system: ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18 at device 4.0 on pci0 The VM itself runs poudriere and was building a bunch of packages. At the moment, the VM seems to be in a rather odd state. The poudriere jails are running, but not doing anything. Ideas? Is this an 8.* host ? I don't believe AHCI has MSI support on that version, and AHCI legacy interrupts haven't had a huge amount of testing under load. If it is 8.*, I'd recommend using virtio-blk for the block device until we can work out what's going wrong. The KTR trace would be useful to figure out what's happening. To do that you can compile the host kernel and vmm.ko with the following options: options KTR options KTR_MASK=(KTR_GEN) options KTR_ENTRIES=(4*1024*1024) And when you see the interrupt storm message in the guest you can execute: sudo ktrdump -cto /tmp/ktrdump.out best Neel later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: interrupt storm on ahci
On 2014-06-07 17:28, Peter Grehan wrote: Hi Steve, I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running FreeBSD CURRENT, r266947. I've gotten this message about 12 times since boot: interrupt storm detected on irq268:; throttling interrupt source vmstat -i shows: irq268: ahci1 236514222839 ahci1 is the second disk connected to the system: ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18 at device 4.0 on pci0 The VM itself runs poudriere and was building a bunch of packages. At the moment, the VM seems to be in a rather odd state. The poudriere jails are running, but not doing anything. Ideas? Is this an 8.* host ? I don't believe AHCI has MSI support on that version, and AHCI legacy interrupts haven't had a huge amount of testing under load. If it is 8.*, I'd recommend using virtio-blk for the block device until we can work out what's going wrong. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org He mentioned that both the host and guest are recent -CURRENT -- Allan Jude ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: interrupt storm on ahci
On Sat, Jun 07, 2014 at 02:53:05PM -0700, Neel Natu wrote: Hi Steve, On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote: Hi Steve, I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running FreeBSD CURRENT, r266947. I've gotten this message about 12 times since boot: interrupt storm detected on irq268:; throttling interrupt source vmstat -i shows: irq268: ahci1 236514222839 ahci1 is the second disk connected to the system: ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18 at device 4.0 on pci0 The VM itself runs poudriere and was building a bunch of packages. At the moment, the VM seems to be in a rather odd state. The poudriere jails are running, but not doing anything. Ideas? Is this an 8.* host ? I don't believe AHCI has MSI support on that version, and AHCI legacy interrupts haven't had a huge amount of testing under load. If it is 8.*, I'd recommend using virtio-blk for the block device until we can work out what's going wrong. Both the host and guest are running FreeBSD CURRENT r266947. The KTR trace would be useful to figure out what's happening. To do that you can compile the host kernel and vmm.ko with the following options: options KTR options KTR_MASK=(KTR_GEN) options KTR_ENTRIES=(4*1024*1024) And when you see the interrupt storm message in the guest you can execute: sudo ktrdump -cto /tmp/ktrdump.out Ok, I'll work on getting that added to the host. Thanks, Steve ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org