Re: interrupt storm on ahci

2014-06-12 Thread Steve Wills
On Wed, Jun 11, 2014 at 09:18:17AM +, Steve Wills wrote:
 On Sat, Jun 07, 2014 at 02:53:05PM -0700, Neel Natu wrote:
  Hi Steve,
  
  On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote:
   Hi Steve,
  
  
   I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
   FreeBSD CURRENT, r266947. I've gotten this message about 12 times since
   boot:
  
   interrupt storm detected on irq268:; throttling interrupt source
  
   vmstat -i shows:
  
   irq268: ahci1  236514222839
  
   ahci1 is the second disk connected to the system:
  
   ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18
   at device 4.0 on pci0
  
   The VM itself runs poudriere and was building a bunch of packages. At the
   moment, the VM seems to be in a rather odd state. The poudriere jails are
   running, but not doing anything. Ideas?
  
  
Is this an 8.* host ? I don't believe AHCI has MSI support on that 
   version,
   and AHCI legacy interrupts haven't had a huge amount of testing under 
   load.
  
If it is 8.*, I'd recommend using virtio-blk for the block device until 
   we
   can work out what's going wrong.
  
  
  The KTR trace would be useful to figure out what's happening.
  
  To do that you can compile the host kernel and vmm.ko with the
  following options:
  options KTR
  options KTR_MASK=(KTR_GEN)
  options KTR_ENTRIES=(4*1024*1024)
  
  And when you see the interrupt storm message in the guest you can execute:
  sudo ktrdump -cto /tmp/ktrdump.out
  
 
 This was added to the kernel config, the kernel was built and installed and 
 the 
 host was rebooted. I started the VM and almost as soon as the workload was put
 back, the messages started again. So, I ran the ktrdump command above. I'll
 mail you privately with the location of the output.

Replying to myself just to note for the record that the interrupt storm was
harmless, because the throttling only reduced performance (I thought another
issue I was having at the same time was a result, but it was just a
coincidence) and that the default threshold for interrupt storms of 1000
interrupts per second may be too low in some cases, such as this one. I
disabled the throttling by setting the hw.intr_storm_threshold sysctl to 0 and
haven't had any issues.

Steve
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org


Re: interrupt storm on ahci

2014-06-11 Thread Steve Wills
On Sat, Jun 07, 2014 at 02:53:05PM -0700, Neel Natu wrote:
 Hi Steve,
 
 On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote:
  Hi Steve,
 
 
  I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
  FreeBSD CURRENT, r266947. I've gotten this message about 12 times since
  boot:
 
  interrupt storm detected on irq268:; throttling interrupt source
 
  vmstat -i shows:
 
  irq268: ahci1  236514222839
 
  ahci1 is the second disk connected to the system:
 
  ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18
  at device 4.0 on pci0
 
  The VM itself runs poudriere and was building a bunch of packages. At the
  moment, the VM seems to be in a rather odd state. The poudriere jails are
  running, but not doing anything. Ideas?
 
 
   Is this an 8.* host ? I don't believe AHCI has MSI support on that version,
  and AHCI legacy interrupts haven't had a huge amount of testing under load.
 
   If it is 8.*, I'd recommend using virtio-blk for the block device until we
  can work out what's going wrong.
 
 
 The KTR trace would be useful to figure out what's happening.
 
 To do that you can compile the host kernel and vmm.ko with the
 following options:
 options KTR
 options KTR_MASK=(KTR_GEN)
 options KTR_ENTRIES=(4*1024*1024)
 
 And when you see the interrupt storm message in the guest you can execute:
 sudo ktrdump -cto /tmp/ktrdump.out
 

This was added to the kernel config, the kernel was built and installed and the 
host was rebooted. I started the VM and almost as soon as the workload was put
back, the messages started again. So, I ran the ktrdump command above. I'll
mail you privately with the location of the output.

Thanks,
Steve
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org


Re: interrupt storm on ahci

2014-06-07 Thread Peter Grehan

Hi Steve,


I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
FreeBSD CURRENT, r266947. I've gotten this message about 12 times since boot:

interrupt storm detected on irq268:; throttling interrupt source

vmstat -i shows:

irq268: ahci1  236514222839

ahci1 is the second disk connected to the system:

ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18 at 
device 4.0 on pci0

The VM itself runs poudriere and was building a bunch of packages. At the
moment, the VM seems to be in a rather odd state. The poudriere jails are
running, but not doing anything. Ideas?


 Is this an 8.* host ? I don't believe AHCI has MSI support on that 
version, and AHCI legacy interrupts haven't had a huge amount of testing 
under load.


 If it is 8.*, I'd recommend using virtio-blk for the block device 
until we can work out what's going wrong.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org


Re: interrupt storm on ahci

2014-06-07 Thread Neel Natu
Hi Steve,

On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote:
 Hi Steve,


 I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
 FreeBSD CURRENT, r266947. I've gotten this message about 12 times since
 boot:

 interrupt storm detected on irq268:; throttling interrupt source

 vmstat -i shows:

 irq268: ahci1  236514222839

 ahci1 is the second disk connected to the system:

 ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18
 at device 4.0 on pci0

 The VM itself runs poudriere and was building a bunch of packages. At the
 moment, the VM seems to be in a rather odd state. The poudriere jails are
 running, but not doing anything. Ideas?


  Is this an 8.* host ? I don't believe AHCI has MSI support on that version,
 and AHCI legacy interrupts haven't had a huge amount of testing under load.

  If it is 8.*, I'd recommend using virtio-blk for the block device until we
 can work out what's going wrong.


The KTR trace would be useful to figure out what's happening.

To do that you can compile the host kernel and vmm.ko with the
following options:
options KTR
options KTR_MASK=(KTR_GEN)
options KTR_ENTRIES=(4*1024*1024)

And when you see the interrupt storm message in the guest you can execute:
sudo ktrdump -cto /tmp/ktrdump.out

best
Neel

 later,

 Peter.


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org


Re: interrupt storm on ahci

2014-06-07 Thread Allan Jude
On 2014-06-07 17:28, Peter Grehan wrote:
 Hi Steve,
 
 I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
 FreeBSD CURRENT, r266947. I've gotten this message about 12 times
 since boot:

 interrupt storm detected on irq268:; throttling interrupt source

 vmstat -i shows:

 irq268: ahci1  236514222839

 ahci1 is the second disk connected to the system:

 ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq
 18 at device 4.0 on pci0

 The VM itself runs poudriere and was building a bunch of packages. At the
 moment, the VM seems to be in a rather odd state. The poudriere jails are
 running, but not doing anything. Ideas?
 
  Is this an 8.* host ? I don't believe AHCI has MSI support on that
 version, and AHCI legacy interrupts haven't had a huge amount of testing
 under load.
 
  If it is 8.*, I'd recommend using virtio-blk for the block device until
 we can work out what's going wrong.
 
 later,
 
 Peter.
 
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org

He mentioned that both the host and guest are recent -CURRENT

-- 
Allan Jude
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org


Re: interrupt storm on ahci

2014-06-07 Thread Steve Wills
On Sat, Jun 07, 2014 at 02:53:05PM -0700, Neel Natu wrote:
 Hi Steve,
 
 On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote:
  Hi Steve,
 
 
  I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
  FreeBSD CURRENT, r266947. I've gotten this message about 12 times since
  boot:
 
  interrupt storm detected on irq268:; throttling interrupt source
 
  vmstat -i shows:
 
  irq268: ahci1  236514222839
 
  ahci1 is the second disk connected to the system:
 
  ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18
  at device 4.0 on pci0
 
  The VM itself runs poudriere and was building a bunch of packages. At the
  moment, the VM seems to be in a rather odd state. The poudriere jails are
  running, but not doing anything. Ideas?
 
 
   Is this an 8.* host ? I don't believe AHCI has MSI support on that version,
  and AHCI legacy interrupts haven't had a huge amount of testing under load.
 
   If it is 8.*, I'd recommend using virtio-blk for the block device until we
  can work out what's going wrong.
 

Both the host and guest are running FreeBSD CURRENT r266947.

 The KTR trace would be useful to figure out what's happening.
 
 To do that you can compile the host kernel and vmm.ko with the
 following options:
 options KTR
 options KTR_MASK=(KTR_GEN)
 options KTR_ENTRIES=(4*1024*1024)
 
 And when you see the interrupt storm message in the guest you can execute:
 sudo ktrdump -cto /tmp/ktrdump.out

Ok, I'll work on getting that added to the host.

Thanks,
Steve
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org