I haven't looked at the code, but it wouldn't surprise me if multicast fanout sucked. Multicast traffic is generally considered by *many* nic drivers and IP stacks to be lower bandwidth than unicast (a *lot* lower), and often gets "slowpath" treatment. Its a little surprising to me that you have enough multicast traffic to justify running in polled mode.
Garrett D'Amore [email protected] On Jun 28, 2012, at 12:24 PM, Sašo Kiselkov wrote: > I don't think it's being caused by interrupts, because the driver > instance for igb3 (where I'm receiving all the multicasts) is in polling > mode (I can't see any interrupts on cpu0 or cpu31, which is where igb3 > is bound, in intrstat). So instead, the problem seems to be in the fact > that only a single thread is doing the polling. Dtracing which CPU's > running the ip_fanout_udp_multi_v4 routine, I can see that it's mostly > only on cpu1: > > # dtrace -n 'fbt::ip_fanout_udp_multi_v4:entry{@[cpu]=count();}' > 14 2 > 0 10544 > 1 1185177 > > The question is why are all multicasts handled only in a single soft > ring worker? Can it have something to do with igb only creating a single > rx ring group? As I said earlier, when I set rx_group_number>=2 in > igb.conf, igb panics at boot on a BAD TRAP as in the attached kernel > stack trace. > > Cheers, > -- > Saso > > On 06/27/2012 05:30 PM, Sašo Kiselkov wrote: >> On 06/27/2012 04:58 PM, Garrett D'Amore wrote: >>> Mostly receive, or mostly transmit, traffic? echo ::interrupts | mdb -k >>> (to see interrupt bindings) >> >> Right now it's purely Rx. Later on, in production, this is going to be >> roughly a 20:1 split between Tx:Rx. My interrupts are mapped out as follows: >> >> >>> ::interrupts >> IRQ Vect IPL Bus Trg Type CPU Share APIC/INT# ISR(s) >> 3 0xb1 12 ISA Edg Fixed 5 1 0x0/0x3 asyintr >> 4 0xb0 12 ISA Edg Fixed 4 1 0x0/0x4 asyintr >> 9 0x80 9 PCI Lvl Fixed 1 1 0x0/0x9 acpi_wrapper_isr >> 16 0x84 9 PCI Lvl Fixed 9 2 0x0/0x10 ohci_intr, ohci_intr >> 17 0x82 9 PCI Lvl Fixed 7 1 0x0/0x11 ehci_intr >> 18 0x85 9 PCI Lvl Fixed 10 2 0x0/0x12 ohci_intr, ohci_intr >> 19 0x83 9 PCI Lvl Fixed 8 1 0x0/0x13 ehci_intr >> 88 0x81 7 PCI Edg MSI 2 1 - pcieb_intr_handler >> 89 0x40 5 PCI Edg MSI-X 3 1 - mrsas_isr >> 90 0x30 4 PCI Edg MSI 6 1 - pcieb_intr_handler >> 91 0x86 7 PCI Edg MSI 11 1 - pcieb_intr_handler >> 92 0x31 4 PCI Edg MSI 12 1 - pcieb_intr_handler >> 93 0x60 6 PCI Edg MSI-X 13 1 - igb_intr_tx_other >> 94 0x61 6 PCI Edg MSI-X 14 1 - igb_intr_rx >> 95 0x62 6 PCI Edg MSI-X 15 1 - igb_intr_tx_other >> 96 0x63 6 PCI Edg MSI-X 16 1 - igb_intr_rx >> 97 0x64 6 PCI Edg MSI-X 31 1 - igb_intr_tx_other >> 98 0x65 6 PCI Edg MSI-X 0 1 - igb_intr_rx >> 99 0x41 5 PCI Edg MSI 19 1 - mptsas_intr >> 100 0x42 5 PCI Edg MSI 20 1 - mptsas_intr >> 101 0x66 6 PCI Edg MSI-X 23 1 - igb_intr_tx_other >> 102 0x67 6 PCI Edg MSI-X 24 1 - igb_intr_rx >> 160 0xa0 0 Edg IPI all 0 - poke_cpu >> 208 0xd0 14 Edg IPI all 1 - kcpc_hw_overflow_intr >> 209 0xd1 14 Edg IPI all 1 - cbe_fire >> 210 0xd3 14 Edg IPI all 1 - cbe_fire >> 240 0xe0 15 Edg IPI all 1 - xc_serv >> 241 0xe1 15 Edg IPI all 1 - apic_error_intr >> >> The CPU being thrashed currently is cpu1 (after I did a "dladm >> set-linkprop cpus=0-31 igb3"). >> >> Cheers, >> -- >> Saso > > <igb_crash.txt> ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
