I've been repeatedly loading and unloading the latest igb driver on a few
boxes in my fleet today and was finally able to replicate the issue and get
a register dump. The switch did not report any problem, the port was
enabled and up.
Here is the register dump from ethtool:
0x00000: CTRL (Device control register) 0x181C0241
Invert Loss-Of-Signal: no
Receive flow control: enabled
Transmit flow control: enabled
VLAN mode: disabled
Set link up: 1
D3COLD WakeUp capability advertisement: enabled
Auto speed detect: disabled
Speed select: 1000Mb/s
Force speed: no
Force duplex: no
0x00008: STATUS (Device status register) 0x00280783
Duplex: full
Link up: link config
Transmission: on
DMA clock gating: disabled
TBI mode: disabled
Link speed: 1000Mb/s
Bus type: PCI Express
0x00100: RCTL (Receive control register) 0x04048022
Receiver: enabled
Store bad packets: disabled
Unicast promiscuous: disabled
Multicast promiscuous: disabled
Long packet: enabled
Descriptor minimum threshold size: 1/2
Broadcast accept mode: accept
VLAN filter: enabled
Cononical form indicator: disabled
Discard pause frames: filtered
Pass MAC control frames: don't pass
Loopback mode: normal
Receive buffer size: 2048
0x02808: RDLEN (Receive desc length) 0x00001000
0x02810: RDH (Receive desc head) 0x000000FF
0x02818: RDT (Receive desc tail) 0x000000FF
0x00400: TCTL (Transmit ctrl register) 0xA50400FA
Transmitter: enabled
Pad short packets: enabled
Software XOFF Transmission: disabled
Re-transmit on late collision: enabled
0x03808: TDLEN (Transmit desc length) 0x00001000
0x03810: TDH (Transmit desc head) 0x000000EA
0x03818: TDT (Transmit desc tail) 0x000000EA
0x00018: CTRL_EXT (Extended device control) 0x101400C0
0x00018: MDIC (MDI control) 0x180A3C00
0x00024: SCTL (SERDES ANA) 0x00000000
0x00034: CONNSW (Copper/Fiber switch control) 0x00000400
0x00038: VET (VLAN Ether type) 0x00008100
0x00E00: LEDCTL (LED control) 0x07068302
0x01000: PBA (Packet buffer allocation) 0x00000000
0x01008: PBS (Packet buffer size) 0x00000000
0x01048: FRTIMER (Free running timer) 0x00AD979F
0x0104C: TCPTIMER (TCP timer) 0x00000000
0x00010: EEC (EEPROM/FLASH control) 0x00083F1E
0x01580: EICR (Extended interrupt cause) 0x80000001
0x01520: EICS (Extended interrupt cause set) 0x80000001
0x01524: EIMS (Extended interrup set/read) 0x80000000
0x01528: EIMC (Extended interrupt mask clear) 0x80000000
0x0152C: EIAC (Extended interrupt auto clear) 0x00000000
0x01530: EIAM (Extended interrupt auto mask) 0x00000000
0x01500: ICR (Interrupt cause read) 0x800000D5
0x01504: ICS (Interrupt cause set) 0x800000D5
0x01508: IMS (Interrupt mask set/read) 0x4000009D
0x0150C: IMC (Interrupt mask clear) 0x4000009D
0x04100: IAC (Interrupt assertion count) 0x00000000
0x01510: IAM (Interr acknowledge auto-mask) 0x4000009D
0x05AC0: IMIRVP (Immed interr rx VLAN priority) 0x00000000
0x00028: FCAL (Flow control address low) 0x00C28001
0x0002C: FCAH (Flow control address high) 0x00000100
0x00170: FCTTV (Flow control tx timer value) 0x0000FFFF
0x02160: FCRTL (Flow control rx threshold low) 0x80010320
0x02168: FCRTH (Flow control rx threshold high) 0x00010330
0x02460: FCRTV (Flow control refresh threshold) 0x00000000
0x05000: RXCSUM (Receive checksum control) 0x00002F00
0x05004: RLPML (Receive long packet max length) 0x000005F2
0x05008: RFCTL (Receive filter control) 0x00000000
0x05818: MRQC (Multiple rx queues command) 0x00370002
0x0581C: VMD_CTL (VMDq control) 0x00000000
0x00404: TCTL_EXT (Transmit control extended) 0x0000FC40
0x00410: TIPG (Transmit IPG) 0x00601008
0x03590: DTXCTL (DMA tx control) 0x00000080
0x05800: WUC (Wake up control) 0x00000000
0x05808: WUFC (Wake up filter control) 0x00000000
0x05810: WUS (Wake up status) 0x00000000
0x05838: IPAV (IP address valid) 0x00000000
0x05900: WUPL (Wake up packet length) 0x61400C00
0x04200: PCS_CFG (PCS configuration 0) 0x00000008
0x04208: PCS_LCTL (PCS link control) 0x0210000C
0x0420C: PCS_LSTS (PCS link status) 0x0040000D
0x04218: PCS_ANADV (AN advertisement) 0x000001A0
0x0421C: PCS_LPAB (Link partner ability) 0x00000000
0x04220: PCS_NPTX (Next Page transmit) 0x00000000
0x04224: PCS_LPABNP (Link partner ability Next Page) 0x00000000
0x04000: CRCERRS (CRC error count) 0x00000000
0x04004: ALGNERRC (Alignment error count) 0x00000000
0x04008: SYMERRS (Symbol error count) 0x00000000
0x0400C: RXERRC (RX error count) 0x00000000
0x04010: MPC (Missed packets count) 0x0000BC53
0x04014: SCC (Single collision count) 0x00000000
0x04018: ECOL (Excessive collisions count) 0x00000000
0x0401C: MCC (Multiple collision count) 0x00000000
0x04020: LATECOL (Late collisions count) 0x00000000
0x04028: COLC (Collision count) 0x00000000
0x04030: DC (Defer count) 0x00000000
0x04034: TNCRS (Transmit with no CRS) 0x00000000
0x04038: SEC (Sequence error count) 0x00000000
0x0403C: HTDPMC (Host tx discrd pkts MAC count) 0x00000000
0x04040: RLEC (Receive length error count) 0x00000000
0x04048: XONRXC (XON received count) 0x00000000
0x0404C: XONTXC (XON transmitted count) 0x00106088
0x04050: XOFFRXC (XOFF received count) 0x00000000
0x04054: XOFFTXC (XOFF transmitted count) 0x0010C20C
0x04058: FCRUC (FC received unsupported count) 0x00000000
0x0405C: PRC64 (Packets rx (64 B) count) 0x28F63B22
0x04060: PRC127 (Packets rx (65-127 B) count) 0xFED2D250
0x04064: PRC255 (Packets rx (128-255 B) count) 0xCB84AA45
0x04068: PRC511 (Packets rx (256-511 B) count) 0x19680DBD
0x0406C: PRC1023 (Packets rx (512-1023 B) count) 0x1544D787
0x04070: PRC1522 (Packets rx (1024-max B) count) 0xDD436F09
0x04074: GPRC (Good packets received count) 0xFF3E074B
0x04078: BPRC (Broadcast packets rx count) 0x116363E9
0x0407C: MPRC (Multicast packets rx count) 0x01F0D30E
0x04080: GPTC (Good packets tx count) 0x885B7E0D
0x04088: GORCL (Good octets rx count lower) 0xBBCD3E03
0x0408C: GORCH (Good octets rx count upper) 0x00000000
0x04090: GOTCL (Good octets tx count lower) 0xE3FB2867
0x04094: GOTCH (Good octets tx count upper) 0x00000000
0x040A0: RNBC (Receive no buffers count) 0x00224303
0x040A4: RUC (Receive undersize count) 0x00000000
0x040A8: RFC (Receive fragment count) 0x00000000
0x040AC: ROC (Receive oversize count) 0x00000000
0x040B0: RJC (Receive jabber count) 0x00000000
0x040B4: MGPRC (Management packets rx count) 0x00000000
0x040B8: MGPDC (Management pkts dropped count) 0x00000000
0x040BC: MGPTC (Management packets tx count) 0x00000000
0x040C0: TORL (Total octets received lower) 0x00000D36
0x040C4: TORH (Total octets received upper) 0x00000000
0x040C8: TOTL (Total octets transmitted lower) 0x00001DB9
0x040CC: TOTH (Total octets transmitted upper) 0x00000000
0x040D0: TPR (Total packets received) 0x0089FF8E
0x040D4: TPT (Total packets transmitted) 0x887CAC52
0x040D8: PTC64 (Packets tx (64 B) count) 0x42F682FC
0x040DC: PTC127 (Packets tx (65-127 B) count) 0xFAC12DDB
0x040E0: PTC255 (Packets tx (128-255 B) count) 0x3F97A509
0x040E4: PTC511 (Packets tx (256-511 B) count) 0x4ADC83CB
0x040E8: PTC1023 (Packets tx (512-1023 B) count) 0x35B56FE5
0x040EC: PTC1522 (Packets tx (> 1024 B) count) 0x8A7A3CC8
0x040F0: MPTC (Multicast packets tx count) 0x0010598B
0x040F4: BPTC (Broadcast packets tx count) 0x0000163F
0x040F8: TSCTC (TCP segment context tx count) 0x0099940A
0x04100: IAC (Interrupt assertion count) 0xC71B6C92
0x04104: RPTHC (Rx packets to host count) 0x00000000
0x04118: HGPTC (Host good packets tx count) 0x00000000
0x04128: HGORCL (Host good octets rx cnt lower) 0x00000000
0x0412C: HGORCH (Host good octets rx cnt upper) 0x00000000
0x04130: HGOTCL (Host good octets tx cnt lower) 0x00000000
0x04134: HGOTCH (Host good octets tx cnt upper) 0x00000000
0x04138: LENNERS (Length error count) 0x00000000
0x04228: SCVPC (SerDes/SGMII code viol pkt cnt) 0x00000000
0x0A018: HRMPC (Header redir missed pkt count) 0x00000000
0x0280C: SRRCTL0 (Split and replic rx ctl0) 0x42000402
0x0290C: SRRCTL1 (Split and replic rx ctl1) 0x80000400
0x02A0C: SRRCTL2 (Split and replic rx ctl2) 0x80000400
0x02B0C: SRRCTL3 (Split and replic rx ctl3) 0x80000400
0x05480: PSRTYPE0 (Packet split receive type0) 0x0007FFFE
0x05484: PSRTYPE1 (Packet split receive type1) 0x0007FFFE
0x05488: PSRTYPE2 (Packet split receive type2) 0x0007FFFE
0x0548C: PSRTYPE3 (Packet split receive type3) 0x0007FFFE
0x02800: RDBAL0 (Rx desc base addr low0) 0x0EAA0000
0x02900: RDBAL1 (Rx desc base addr low1) 0x00000000
0x02A00: RDBAL2 (Rx desc base addr low2) 0x00000000
0x02B00: RDBAL3 (Rx desc base addr low3) 0x00000000
0x02804: RDBAH0 (Rx desc base addr high0) 0x00000010
0x02904: RDBAH1 (Rx desc base addr high1) 0x00000000
0x02A04: RDBAH2 (Rx desc base addr high2) 0x00000000
0x02B04: RDBAH3 (Rx desc base addr high3) 0x00000000
0x02808: RDLEN0 (Rx descriptor length0) 0x00001000
0x02908: RDLEN1 (Rx descriptor length1) 0x00000000
0x02A08: RDLEN2 (Rx descriptor length2) 0x00000000
0x02B08: RDLEN3 (Rx descriptor length3) 0x00000000
0x02810: RDH0 (Rx descriptor head0) 0x000000FF
0x02910: RDH1 (Rx descriptor head1) 0x00000000
0x02A10: RDH2 (Rx descriptor head2) 0x00000000
0x02B10: RDH3 (Rx descriptor head3) 0x00000000
0x02818: RDT0 (Rx descriptor tail0) 0x000000FF
0x02918: RDT1 (Rx descriptor tail1) 0x00000000
0x02A18: RDT2 (Rx descriptor tail2) 0x00000000
0x02B18: RDT3 (Rx descriptor tail3) 0x00000000
0x02828: RXDCTL0 (Rx descriptor control0) 0x02040808
0x02928: RXDCTL1 (Rx descriptor control1) 0x00010A0C
0x02A28: RXDCTL2 (Rx descriptor control2) 0x00010A0C
0x02B28: RXDCTL3 (Rx descriptor control3) 0x00010A0C
0x01680: EITR0 (Interrupt throttle0) 0x00C000C4
0x01684: EITR1 (Interrupt throttle1) 0x00000000
0x01688: EITR2 (Interrupt throttle2) 0x00000000
0x0168C: EITR3 (Interrupt throttle3) 0x00000000
0x01690: EITR4 (Interrupt throttle4) 0x00000000
0x01694: EITR5 (Interrupt throttle5) 0x00000000
0x01698: EITR6 (Interrupt throttle6) 0x00000000
0x0169C: EITR7 (Interrupt throttle7) 0x00000000
0x016A0: EITR8 (Interrupt throttle8) 0x00000000
0x016A4: EITR9 (Interrupt throttle9) 0x00000000
0x05A80: IMIR0 (Immediate interrupt Rx0) 0x00000000
0x05A84: IMIR1 (Immediate interrupt Rx1) 0x00000000
0x05A88: IMIR2 (Immediate interrupt Rx2) 0x00000000
0x05A8C: IMIR3 (Immediate interrupt Rx3) 0x00000000
0x05A90: IMIR4 (Immediate interrupt Rx4) 0x00000000
0x05A94: IMIR5 (Immediate interrupt Rx5) 0x00000000
0x05A98: IMIR6 (Immediate interrupt Rx6) 0x00000000
0x05A9C: IMIR7 (Immediate interrupt Rx7) 0x00000000
0x05AA0: IMIREXT0 (Immediate interr Rx extended0) 0x00000000
0x05AA4: IMIREXT1 (Immediate interr Rx extended1) 0x00000000
0x05AA8: IMIREXT2 (Immediate interr Rx extended2) 0x00000000
0x05AAC: IMIREXT3 (Immediate interr Rx extended3) 0x00000000
0x05AB0: IMIREXT4 (Immediate interr Rx extended4) 0x00000000
0x05AB4: IMIREXT5 (Immediate interr Rx extended5) 0x00000000
0x05AB8: IMIREXT6 (Immediate interr Rx extended6) 0x00000000
0x05ABC: IMIREXT7 (Immediate interr Rx extended7) 0x00000000
0x05400: RAL00 (Receive address low00) 0x359F36A0
0x05408: RAL01 (Receive address low01) 0x00000000
0x05410: RAL02 (Receive address low02) 0x00000000
0x05418: RAL03 (Receive address low03) 0x00000000
0x05420: RAL04 (Receive address low04) 0x00000000
0x05428: RAL05 (Receive address low05) 0x00000000
0x05430: RAL06 (Receive address low06) 0x00000000
0x05438: RAL07 (Receive address low07) 0x00000000
0x05440: RAL08 (Receive address low08) 0x00000000
0x05448: RAL09 (Receive address low09) 0x00000000
0x05450: RAL10 (Receive address low10) 0x00000000
0x05458: RAL11 (Receive address low11) 0x00000000
0x05460: RAL12 (Receive address low12) 0x00000000
0x05468: RAL13 (Receive address low13) 0x00000000
0x05470: RAL14 (Receive address low14) 0x00000000
0x05478: RAL15 (Receive address low15) 0x00000000
0x05404: RAH00 (Receive address high00) 0x800460CC
0x0540C: RAH01 (Receive address high01) 0x00000000
0x05414: RAH02 (Receive address high02) 0x00000000
0x0541C: RAH03 (Receive address high03) 0x00000000
0x05424: RAH04 (Receive address high04) 0x00000000
0x0542C: RAH05 (Receive address high05) 0x00000000
0x05434: RAH06 (Receive address high06) 0x00000000
0x0543C: RAH07 (Receive address high07) 0x00000000
0x05444: RAH08 (Receive address high08) 0x00000000
0x0544C: RAH09 (Receive address high09) 0x00000000
0x05454: RAH10 (Receive address high10) 0x00000000
0x0545C: RAH11 (Receive address high11) 0x00000000
0x05464: RAH12 (Receive address high12) 0x00000000
0x0546C: RAH13 (Receive address high13) 0x00000000
0x05474: RAH14 (Receive address high14) 0x00000000
0x0547C: RAH15 (Receive address high15) 0x00000000
0x03800: TDBAL0 (Tx desc base address low0) 0x0E884000
0x03900: TDBAL1 (Tx desc base address low1) 0x00000000
0x03A00: TDBAL2 (Tx desc base address low2) 0x00000000
0x03B00: TDBAL3 (Tx desc base address low3) 0x00000000
0x03804: TDBAH0 (Tx desc base address high0) 0x00000010
0x03904: TDBAH1 (Tx desc base address high1) 0x00000000
0x03A04: TDBAH2 (Tx desc base address high2) 0x00000000
0x03B04: TDBAH3 (Tx desc base address high3) 0x00000000
0x03808: TDLEN0 (Tx descriptor length0) 0x00001000
0x03908: TDLEN1 (Tx descriptor length1) 0x00000000
0x03A08: TDLEN2 (Tx descriptor length2) 0x00000000
0x03B08: TDLEN3 (Tx descriptor length3) 0x00000000
0x03810: TDH0 (Transmit descriptor head0) 0x000000EA
0x03910: TDH1 (Transmit descriptor head1) 0x00000000
0x03A10: TDH2 (Transmit descriptor head2) 0x00000000
0x03B10: TDH3 (Transmit descriptor head3) 0x00000000
0x03818: TDT0 (Transmit descriptor tail0) 0x000000EA
0x03918: TDT1 (Transmit descriptor tail1) 0x00000000
0x03A18: TDT2 (Transmit descriptor tail2) 0x00000000
0x03B18: TDT3 (Transmit descriptor tail3) 0x00000000
0x03828: TXDCTL0 (Transmit descriptor control0) 0x02100108
0x03928: TXDCTL1 (Transmit descriptor control1) 0x00000000
0x03A28: TXDCTL2 (Transmit descriptor control2) 0x00000000
0x03B28: TXDCTL3 (Transmit descriptor control3) 0x00000000
0x03838: TDWBAL0 (Tx desc complete wb addr low0) 0x00000000
0x03938: TDWBAL1 (Tx desc complete wb addr low1) 0x00000000
0x03A38: TDWBAL2 (Tx desc complete wb addr low2) 0x00000000
0x03B38: TDWBAL3 (Tx desc complete wb addr low3) 0x00000000
0x0383C: TDWBAH0 (Tx desc complete wb addr hi0) 0x00000000
0x0393C: TDWBAH1 (Tx desc complete wb addr hi1) 0x00000000
0x03A3C: TDWBAH2 (Tx desc complete wb addr hi2) 0x00000000
0x03B3C: TDWBAH3 (Tx desc complete wb addr hi3) 0x00000000
0x03814: DCA_TXCTRL0 (Tx DCA control0) 0x00002A00
0x03914: DCA_TXCTRL1 (Tx DCA control1) 0x00002A00
0x03A14: DCA_TXCTRL2 (Tx DCA control2) 0x00002A00
0x03B14: DCA_TXCTRL3 (Tx DCA control3) 0x00002A00
0x05840: IP4AT0 (IPv4 address table0) 0x00000000
0x05848: IP4AT1 (IPv4 address table1) 0x00000000
0x05850: IP4AT2 (IPv4 address table2) 0x00000000
0x05858: IP4AT3 (IPv4 address table3) 0x00000000
0x05880: IP6AT0 (IPv6 address table0) 0x00000000
0x05884: IP6AT1 (IPv6 address table1) 0x00000000
0x05888: IP6AT2 (IPv6 address table2) 0x00000000
0x0588C: IP6AT3 (IPv6 address table3) 0x00000000
0x05A00: WUPM00 (Wake up packet memory00) 0x359F36A0
0x05A04: WUPM01 (Wake up packet memory01) 0x120060CC
0x05A08: WUPM02 (Wake up packet memory02) 0x00B891F2
0x05A0C: WUPM03 (Wake up packet memory03) 0x00450008
0x05A10: WUPM04 (Wake up packet memory04) 0xFE8A6005
0x05A14: WUPM05 (Wake up packet memory05) 0x06350040
0x05A18: WUPM06 (Wake up packet memory06) 0xC4424420
0x05A1C: WUPM07 (Wake up packet memory07) 0xBF6B5B78
0x05A20: WUPM08 (Wake up packet memory08) 0xBA13776E
0x05A24: WUPM09 (Wake up packet memory09) 0x70CD5EC0
0x05A28: WUPM10 (Wake up packet memory10) 0xA33E5E1B
0x05A2C: WUPM11 (Wake up packet memory11) 0x10502541
0x05A30: WUPM12 (Wake up packet memory12) 0xF20B9E20
0x05A34: WUPM13 (Wake up packet memory13) 0x4D590000
0x05A38: WUPM14 (Wake up packet memory14) 0x13004753
0x05A3C: WUPM15 (Wake up packet memory15) 0xC3000000
0x05A40: WUPM16 (Wake up packet memory16) 0x0000F000
0x05A44: WUPM17 (Wake up packet memory17) 0x4A000100
0x05A48: WUPM18 (Wake up packet memory18) 0x303347EB
0x05A4C: WUPM19 (Wake up packet memory19) 0x3380C032
0x05A50: WUPM20 (Wake up packet memory20) 0x80C03531
0x05A54: WUPM21 (Wake up packet memory21) 0xC0303033
0x05A58: WUPM22 (Wake up packet memory22) 0x35313380
0x05A5C: WUPM23 (Wake up packet memory23) 0xC03780C0
0x05A60: WUPM24 (Wake up packet memory24) 0x63657280
0x05A64: WUPM25 (Wake up packet memory25) 0x69706163
0x05A68: WUPM26 (Wake up packet memory26) 0xC0736570
0x05A6C: WUPM27 (Wake up packet memory27) 0xC0303180
0x05A70: WUPM28 (Wake up packet memory28) 0x80C03080
0x05A74: WUPM29 (Wake up packet memory29) 0x80C03331
0x05A78: WUPM30 (Wake up packet memory30) 0x3180C031
0x05A7C: WUPM31 (Wake up packet memory31) 0x80C03239
0x09000: FFMT000 (Flexible filter mask table000) 0x00000000
0x09008: FFMT001 (Flexible filter mask table001) 0x00000000
0x09010: FFMT002 (Flexible filter mask table002) 0x00000000
0x09018: FFMT003 (Flexible filter mask table003) 0x00000000
0x09020: FFMT004 (Flexible filter mask table004) 0x00000000
0x09028: FFMT005 (Flexible filter mask table005) 0x00000000
0x09030: FFMT006 (Flexible filter mask table006) 0x00000000
0x09038: FFMT007 (Flexible filter mask table007) 0x00000000
0x09040: FFMT008 (Flexible filter mask table008) 0x00000000
0x09048: FFMT009 (Flexible filter mask table009) 0x00000000
0x09050: FFMT010 (Flexible filter mask table010) 0x00000000
0x09058: FFMT011 (Flexible filter mask table011) 0x00000000
0x09060: FFMT012 (Flexible filter mask table012) 0x00000000
0x09068: FFMT013 (Flexible filter mask table013) 0x00000000
0x09070: FFMT014 (Flexible filter mask table014) 0x00000000
0x09078: FFMT015 (Flexible filter mask table015) 0x00000000
0x09080: FFMT016 (Flexible filter mask table016) 0x00000000
0x09088: FFMT017 (Flexible filter mask table017) 0x00000000
0x09090: FFMT018 (Flexible filter mask table018) 0x00000000
0x09098: FFMT019 (Flexible filter mask table019) 0x00000000
0x090A0: FFMT020 (Flexible filter mask table020) 0x00000000
0x090A8: FFMT021 (Flexible filter mask table021) 0x00000000
0x090B0: FFMT022 (Flexible filter mask table022) 0x00000000
0x090B8: FFMT023 (Flexible filter mask table023) 0x00000000
0x090C0: FFMT024 (Flexible filter mask table024) 0x00000000
0x090C8: FFMT025 (Flexible filter mask table025) 0x00000000
0x090D0: FFMT026 (Flexible filter mask table026) 0x00000000
0x090D8: FFMT027 (Flexible filter mask table027) 0x00000000
0x090E0: FFMT028 (Flexible filter mask table028) 0x00000000
0x090E8: FFMT029 (Flexible filter mask table029) 0x00000000
0x090F0: FFMT030 (Flexible filter mask table030) 0x00000000
0x090F8: FFMT031 (Flexible filter mask table031) 0x00000000
0x09100: FFMT032 (Flexible filter mask table032) 0x00000000
0x09108: FFMT033 (Flexible filter mask table033) 0x00000000
0x09110: FFMT034 (Flexible filter mask table034) 0x00000000
0x09118: FFMT035 (Flexible filter mask table035) 0x00000000
0x09120: FFMT036 (Flexible filter mask table036) 0x00000000
0x09128: FFMT037 (Flexible filter mask table037) 0x00000000
0x09130: FFMT038 (Flexible filter mask table038) 0x00000000
0x09138: FFMT039 (Flexible filter mask table039) 0x00000000
0x09140: FFMT040 (Flexible filter mask table040) 0x00000000
0x09148: FFMT041 (Flexible filter mask table041) 0x00000000
0x09150: FFMT042 (Flexible filter mask table042) 0x00000000
0x09158: FFMT043 (Flexible filter mask table043) 0x00000000
0x09160: FFMT044 (Flexible filter mask table044) 0x00000000
0x09168: FFMT045 (Flexible filter mask table045) 0x00000000
0x09170: FFMT046 (Flexible filter mask table046) 0x00000000
0x09178: FFMT047 (Flexible filter mask table047) 0x00000000
0x09180: FFMT048 (Flexible filter mask table048) 0x00000000
0x09188: FFMT049 (Flexible filter mask table049) 0x00000000
0x09190: FFMT050 (Flexible filter mask table050) 0x00000000
0x09198: FFMT051 (Flexible filter mask table051) 0x00000000
0x091A0: FFMT052 (Flexible filter mask table052) 0x00000000
0x091A8: FFMT053 (Flexible filter mask table053) 0x00000000
0x091B0: FFMT054 (Flexible filter mask table054) 0x00000000
0x091B8: FFMT055 (Flexible filter mask table055) 0x00000000
0x091C0: FFMT056 (Flexible filter mask table056) 0x00000000
0x091C8: FFMT057 (Flexible filter mask table057) 0x00000000
0x091D0: FFMT058 (Flexible filter mask table058) 0x00000000
0x091D8: FFMT059 (Flexible filter mask table059) 0x00000000
0x091E0: FFMT060 (Flexible filter mask table060) 0x00000000
0x091E8: FFMT061 (Flexible filter mask table061) 0x00000000
0x091F0: FFMT062 (Flexible filter mask table062) 0x00000000
0x091F8: FFMT063 (Flexible filter mask table063) 0x00000000
0x09200: FFMT064 (Flexible filter mask table064) 0x00000000
0x09208: FFMT065 (Flexible filter mask table065) 0x00000000
0x09210: FFMT066 (Flexible filter mask table066) 0x00000000
0x09218: FFMT067 (Flexible filter mask table067) 0x00000000
0x09220: FFMT068 (Flexible filter mask table068) 0x00000000
0x09228: FFMT069 (Flexible filter mask table069) 0x00000000
0x09230: FFMT070 (Flexible filter mask table070) 0x00000000
0x09238: FFMT071 (Flexible filter mask table071) 0x00000000
0x09240: FFMT072 (Flexible filter mask table072) 0x00000000
0x09248: FFMT073 (Flexible filter mask table073) 0x00000000
0x09250: FFMT074 (Flexible filter mask table074) 0x00000000
0x09258: FFMT075 (Flexible filter mask table075) 0x00000000
0x09260: FFMT076 (Flexible filter mask table076) 0x00000000
0x09268: FFMT077 (Flexible filter mask table077) 0x00000000
0x09270: FFMT078 (Flexible filter mask table078) 0x00000000
0x09278: FFMT079 (Flexible filter mask table079) 0x00000000
0x09280: FFMT080 (Flexible filter mask table080) 0x00000000
0x09288: FFMT081 (Flexible filter mask table081) 0x00000000
0x09290: FFMT082 (Flexible filter mask table082) 0x00000000
0x09298: FFMT083 (Flexible filter mask table083) 0x00000000
0x092A0: FFMT084 (Flexible filter mask table084) 0x00000000
0x092A8: FFMT085 (Flexible filter mask table085) 0x00000000
0x092B0: FFMT086 (Flexible filter mask table086) 0x00000000
0x092B8: FFMT087 (Flexible filter mask table087) 0x00000000
0x092C0: FFMT088 (Flexible filter mask table088) 0x00000000
0x092C8: FFMT089 (Flexible filter mask table089) 0x00000000
0x092D0: FFMT090 (Flexible filter mask table090) 0x00000000
0x092D8: FFMT091 (Flexible filter mask table091) 0x00000000
0x092E0: FFMT092 (Flexible filter mask table092) 0x00000000
0x092E8: FFMT093 (Flexible filter mask table093) 0x00000000
0x092F0: FFMT094 (Flexible filter mask table094) 0x00000000
0x092F8: FFMT095 (Flexible filter mask table095) 0x00000000
0x09300: FFMT096 (Flexible filter mask table096) 0x00000000
0x09308: FFMT097 (Flexible filter mask table097) 0x00000000
0x09310: FFMT098 (Flexible filter mask table098) 0x00000000
0x09318: FFMT099 (Flexible filter mask table099) 0x00000000
0x09320: FFMT100 (Flexible filter mask table100) 0x00000000
0x09328: FFMT101 (Flexible filter mask table101) 0x00000000
0x09330: FFMT102 (Flexible filter mask table102) 0x00000000
0x09338: FFMT103 (Flexible filter mask table103) 0x00000000
0x09340: FFMT104 (Flexible filter mask table104) 0x00000000
0x09348: FFMT105 (Flexible filter mask table105) 0x00000000
0x09350: FFMT106 (Flexible filter mask table106) 0x00000000
0x09358: FFMT107 (Flexible filter mask table107) 0x00000000
0x09360: FFMT108 (Flexible filter mask table108) 0x00000000
0x09368: FFMT109 (Flexible filter mask table109) 0x00000000
0x09370: FFMT110 (Flexible filter mask table110) 0x00000000
0x09378: FFMT111 (Flexible filter mask table111) 0x00000000
0x09380: FFMT112 (Flexible filter mask table112) 0x00000000
0x09388: FFMT113 (Flexible filter mask table113) 0x00000000
0x09390: FFMT114 (Flexible filter mask table114) 0x00000000
0x09398: FFMT115 (Flexible filter mask table115) 0x00000000
0x093A0: FFMT116 (Flexible filter mask table116) 0x00000000
0x093A8: FFMT117 (Flexible filter mask table117) 0x00000000
0x093B0: FFMT118 (Flexible filter mask table118) 0x00000000
0x093B8: FFMT119 (Flexible filter mask table119) 0x00000000
0x093C0: FFMT120 (Flexible filter mask table120) 0x00000000
0x093C8: FFMT121 (Flexible filter mask table121) 0x00000000
0x093D0: FFMT122 (Flexible filter mask table122) 0x00000000
0x093D8: FFMT123 (Flexible filter mask table123) 0x00000000
0x093E0: FFMT124 (Flexible filter mask table124) 0x00000000
0x093E8: FFMT125 (Flexible filter mask table125) 0x00000000
0x093F0: FFMT126 (Flexible filter mask table126) 0x00000000
0x093F8: FFMT127 (Flexible filter mask table127) 0x00000000
0x09800: FFVT000 (Flexible filter value table000) 0x00000000
0x09808: FFVT001 (Flexible filter value table001) 0x00000000
0x09810: FFVT002 (Flexible filter value table002) 0x00000000
0x09818: FFVT003 (Flexible filter value table003) 0x00000000
0x09820: FFVT004 (Flexible filter value table004) 0x00000000
0x09828: FFVT005 (Flexible filter value table005) 0x00000000
0x09830: FFVT006 (Flexible filter value table006) 0x00000000
0x09838: FFVT007 (Flexible filter value table007) 0x00000000
0x09840: FFVT008 (Flexible filter value table008) 0x00000000
0x09848: FFVT009 (Flexible filter value table009) 0x00000000
0x09850: FFVT010 (Flexible filter value table010) 0x00000000
0x09858: FFVT011 (Flexible filter value table011) 0x00000000
0x09860: FFVT012 (Flexible filter value table012) 0x00000000
0x09868: FFVT013 (Flexible filter value table013) 0x00000000
0x09870: FFVT014 (Flexible filter value table014) 0x00000000
0x09878: FFVT015 (Flexible filter value table015) 0x00000000
0x09880: FFVT016 (Flexible filter value table016) 0x00000000
0x09888: FFVT017 (Flexible filter value table017) 0x00000000
0x09890: FFVT018 (Flexible filter value table018) 0x00000000
0x09898: FFVT019 (Flexible filter value table019) 0x00000000
0x098A0: FFVT020 (Flexible filter value table020) 0x00000000
0x098A8: FFVT021 (Flexible filter value table021) 0x00000000
0x098B0: FFVT022 (Flexible filter value table022) 0x00000000
0x098B8: FFVT023 (Flexible filter value table023) 0x00000000
0x098C0: FFVT024 (Flexible filter value table024) 0x00000000
0x098C8: FFVT025 (Flexible filter value table025) 0x00000000
0x098D0: FFVT026 (Flexible filter value table026) 0x00000000
0x098D8: FFVT027 (Flexible filter value table027) 0x00000000
0x098E0: FFVT028 (Flexible filter value table028) 0x00000000
0x098E8: FFVT029 (Flexible filter value table029) 0x00000000
0x098F0: FFVT030 (Flexible filter value table030) 0x00000000
0x098F8: FFVT031 (Flexible filter value table031) 0x00000000
0x09900: FFVT032 (Flexible filter value table032) 0x00000000
0x09908: FFVT033 (Flexible filter value table033) 0x00000000
0x09910: FFVT034 (Flexible filter value table034) 0x00000000
0x09918: FFVT035 (Flexible filter value table035) 0x00000000
0x09920: FFVT036 (Flexible filter value table036) 0x00000000
0x09928: FFVT037 (Flexible filter value table037) 0x00000000
0x09930: FFVT038 (Flexible filter value table038) 0x00000000
0x09938: FFVT039 (Flexible filter value table039) 0x00000000
0x09940: FFVT040 (Flexible filter value table040) 0x00000000
0x09948: FFVT041 (Flexible filter value table041) 0x00000000
0x09950: FFVT042 (Flexible filter value table042) 0x00000000
0x09958: FFVT043 (Flexible filter value table043) 0x00000000
0x09960: FFVT044 (Flexible filter value table044) 0x00000000
0x09968: FFVT045 (Flexible filter value table045) 0x00000000
0x09970: FFVT046 (Flexible filter value table046) 0x00000000
0x09978: FFVT047 (Flexible filter value table047) 0x00000000
0x09980: FFVT048 (Flexible filter value table048) 0x00000000
0x09988: FFVT049 (Flexible filter value table049) 0x00000000
0x09990: FFVT050 (Flexible filter value table050) 0x00000000
0x09998: FFVT051 (Flexible filter value table051) 0x00000000
0x099A0: FFVT052 (Flexible filter value table052) 0x00000000
0x099A8: FFVT053 (Flexible filter value table053) 0x00000000
0x099B0: FFVT054 (Flexible filter value table054) 0x00000000
0x099B8: FFVT055 (Flexible filter value table055) 0x00000000
0x099C0: FFVT056 (Flexible filter value table056) 0x00000000
0x099C8: FFVT057 (Flexible filter value table057) 0x00000000
0x099D0: FFVT058 (Flexible filter value table058) 0x00000000
0x099D8: FFVT059 (Flexible filter value table059) 0x00000000
0x099E0: FFVT060 (Flexible filter value table060) 0x00000000
0x099E8: FFVT061 (Flexible filter value table061) 0x00000000
0x099F0: FFVT062 (Flexible filter value table062) 0x00000000
0x099F8: FFVT063 (Flexible filter value table063) 0x00000000
0x09A00: FFVT064 (Flexible filter value table064) 0x00000000
0x09A08: FFVT065 (Flexible filter value table065) 0x00000000
0x09A10: FFVT066 (Flexible filter value table066) 0x00000000
0x09A18: FFVT067 (Flexible filter value table067) 0x00000000
0x09A20: FFVT068 (Flexible filter value table068) 0x00000000
0x09A28: FFVT069 (Flexible filter value table069) 0x00000000
0x09A30: FFVT070 (Flexible filter value table070) 0x00000000
0x09A38: FFVT071 (Flexible filter value table071) 0x00000000
0x09A40: FFVT072 (Flexible filter value table072) 0x00000000
0x09A48: FFVT073 (Flexible filter value table073) 0x00000000
0x09A50: FFVT074 (Flexible filter value table074) 0x00000000
0x09A58: FFVT075 (Flexible filter value table075) 0x00000000
0x09A60: FFVT076 (Flexible filter value table076) 0x00000000
0x09A68: FFVT077 (Flexible filter value table077) 0x00000000
0x09A70: FFVT078 (Flexible filter value table078) 0x00000000
0x09A78: FFVT079 (Flexible filter value table079) 0x00000000
0x09A80: FFVT080 (Flexible filter value table080) 0x00000000
0x09A88: FFVT081 (Flexible filter value table081) 0x00000000
0x09A90: FFVT082 (Flexible filter value table082) 0x00000000
0x09A98: FFVT083 (Flexible filter value table083) 0x00000000
0x09AA0: FFVT084 (Flexible filter value table084) 0x00000000
0x09AA8: FFVT085 (Flexible filter value table085) 0x00000000
0x09AB0: FFVT086 (Flexible filter value table086) 0x00000000
0x09AB8: FFVT087 (Flexible filter value table087) 0x00000000
0x09AC0: FFVT088 (Flexible filter value table088) 0x00000000
0x09AC8: FFVT089 (Flexible filter value table089) 0x00000000
0x09AD0: FFVT090 (Flexible filter value table090) 0x00000000
0x09AD8: FFVT091 (Flexible filter value table091) 0x00000000
0x09AE0: FFVT092 (Flexible filter value table092) 0x00000000
0x09AE8: FFVT093 (Flexible filter value table093) 0x00000000
0x09AF0: FFVT094 (Flexible filter value table094) 0x00000000
0x09AF8: FFVT095 (Flexible filter value table095) 0x00000000
0x09B00: FFVT096 (Flexible filter value table096) 0x00000000
0x09B08: FFVT097 (Flexible filter value table097) 0x00000000
0x09B10: FFVT098 (Flexible filter value table098) 0x00000000
0x09B18: FFVT099 (Flexible filter value table099) 0x00000000
0x09B20: FFVT100 (Flexible filter value table100) 0x00000000
0x09B28: FFVT101 (Flexible filter value table101) 0x00000000
0x09B30: FFVT102 (Flexible filter value table102) 0x00000000
0x09B38: FFVT103 (Flexible filter value table103) 0x00000000
0x09B40: FFVT104 (Flexible filter value table104) 0x00000000
0x09B48: FFVT105 (Flexible filter value table105) 0x00000000
0x09B50: FFVT106 (Flexible filter value table106) 0x00000000
0x09B58: FFVT107 (Flexible filter value table107) 0x00000000
0x09B60: FFVT108 (Flexible filter value table108) 0x00000000
0x09B68: FFVT109 (Flexible filter value table109) 0x00000000
0x09B70: FFVT110 (Flexible filter value table110) 0x00000000
0x09B78: FFVT111 (Flexible filter value table111) 0x00000000
0x09B80: FFVT112 (Flexible filter value table112) 0x00000000
0x09B88: FFVT113 (Flexible filter value table113) 0x00000000
0x09B90: FFVT114 (Flexible filter value table114) 0x00000000
0x09B98: FFVT115 (Flexible filter value table115) 0x00000000
0x09BA0: FFVT116 (Flexible filter value table116) 0x00000000
0x09BA8: FFVT117 (Flexible filter value table117) 0x00000000
0x09BB0: FFVT118 (Flexible filter value table118) 0x00000000
0x09BB8: FFVT119 (Flexible filter value table119) 0x00000000
0x09BC0: FFVT120 (Flexible filter value table120) 0x00000000
0x09BC8: FFVT121 (Flexible filter value table121) 0x00000000
0x09BD0: FFVT122 (Flexible filter value table122) 0x00000000
0x09BD8: FFVT123 (Flexible filter value table123) 0x00000000
0x09BE0: FFVT124 (Flexible filter value table124) 0x00000000
0x09BE8: FFVT125 (Flexible filter value table125) 0x00000000
0x09BF0: FFVT126 (Flexible filter value table126) 0x00000000
0x09BF8: FFVT127 (Flexible filter value table127) 0x00000000
0x05F00: FFLT0 (Flexible filter length table0) 0x00000000
0x05F08: FFLT1 (Flexible filter length table1) 0x00000000
0x05F10: FFLT2 (Flexible filter length table2) 0x00000000
0x05F18: FFLT3 (Flexible filter length table3) 0x00000000
0x03410: TDFH (Tx data FIFO head) 0x00000604
0x03418: TDFT (Tx data FIFO tail) 0x00000604
0x03420: TDFHS (Tx data FIFO head saved) 0x00000604
0x03430: TDFPC (Tx data FIFO packet count) 0x00000000
Here is the kernel log from this latest event:
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067444] ------------[ cut here
]------------
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067452] WARNING: at
net/sched/sch_generic.c:267 dev_watchdog+0x26b/0x280() (Tainted: G
--------------- T)
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067455] Hardware name: X9DRL-3F/iF
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067456] NETDEV WATCHDOG: eth0
(igb): transmit queue 0 timed out
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067458] Modules linked in:
xt_connmark xt_CONNMARK ip6table_raw mpt3sas mpt2sas raid_class mptctl
mptbase xt_hashlimit xt_policy ip6t_rt ipt_addrtype ctr twofish_x86_64
twofish_common camellia serpent blowfish cast5 des_generic cbc
crypto_null aesni_intel ablk_helper cryptd lrw gf128mul glue_helper
aes_x86_64 aes_generic xcbc rmd160 sha512_generic vzethdev pio_nfs
pio_direct pfmt_raw pfmt_ploop1 ploop simfs vziolimit vzdquota ip6t_REJECT
xfrm6_mode_tunnel xfrm4_mode_tunnel nf_conntrack_netbios_ns nf_
conntrack_broadcast nf_conntrack_netlink xt_comment nfsd ip6_tunnel ip_vs
ipip xt_NFQUEUE xt_pkttype ecryptfs(T) ip_gre ip_tunnel ipt_MASQUERADE
nf_nat_irc xt_helper nf_conntrack_irc nf_conntrack_ipv6 nf_defrag_ipv6
xt_conntrack ip6t_LOG xt_connlimit xt_recent pppoatm atm
vzrst vzcpt nfs lockd fscache auth_rpcgss nfs_acl sunrpc
xfrm4_mode_transport xfrm6_mode_transport ccm authenc esp6 ah6 cnic uio
xfrm4_tunnel tunnel4 ipcomp6 xfrm6_tunnel tunnel6 ipcomp xfrm_ipcomp esp4
ah4 af
Jun 9 16:35:58 nyccvz2 kernel: _key arc4 ecb ppp_mppe ppp_deflate
zlib_deflate ppp_async ppp_generic slhc crc_ccitt fuse tun xt_MARK xt_mark
vzevent kcare(U) autofs4 vznetdev vzmon vzdev ipt_REDIRECT xt_owner
nf_nat_ftp nf_conntrack_ftp iptable_nat nf_nat xt_state xt_leng
th xt_hl xt_tcpmss xt_TCPMSS xt_multiport xt_limit nf_conntrack_ipv4
nf_conntrack nf_defrag_ipv4 ipt_LOG xt_DSCP xt_dscp ipt_REJECT
iptable_mangle xt_set iptable_filter iptable_raw ip_tables ip6table_mangle
ip6table_filter ip6_tables ipv6 ip_set_hash_ip ip_set nfnetlink iT
CO_wdt iTCO_vendor_support ipmi_devintf ipmi_si ipmi_msghandler e1000e(U)
ses enclosure sg igb(U) dca i2c_algo_bit ptp pps_core acpi_pad sb_edac
edac_core i2c_i801 i2c_core lpc_ich mfd_core shpchp tcp_htcp ext4 jbd2
mbcache sd_mod crc_t10dif isci libsas scsi_transport_sas
megaraid_sas ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last
unloaded: scsi_wait_scan]
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067568] Pid: 0, comm: swapper
veid: 0 Tainted: G --------------- T 2.6.32-042stab106.4 #1
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067571] Call Trace:
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067573] <IRQ>
[<ffffffff8107b7e7>] ? warn_slowpath_common+0x87/0xc0
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067583] [<ffffffff8107b8d6>] ?
warn_slowpath_fmt+0x46/0x50
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067587] [<ffffffff8148f9bb>] ?
dev_watchdog+0x26b/0x280
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067592] [<ffffffff81015009>] ?
sched_clock+0x9/0x10
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067599] [<ffffffff8108f68c>] ?
run_timer_softirq+0x1bc/0x380
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067602] [<ffffffff8148f750>] ?
dev_watchdog+0x0/0x280
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067606] [<ffffffff81034ddd>] ?
lapic_next_event+0x1d/0x30
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067610] [<ffffffff81084c3d>] ?
__do_softirq+0x10d/0x250
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067617] [<ffffffff8100c48c>] ?
call_softirq+0x1c/0x30
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067619] [<ffffffff810102b5>] ?
do_softirq+0x65/0xa0
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067622] [<ffffffff81084a5d>] ?
irq_exit+0xcd/0xd0
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067627] [<ffffffff8153f1ba>] ?
smp_apic_timer_interrupt+0x4a/0x60
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067630] [<ffffffff8100bc93>] ?
apic_timer_interrupt+0x13/0x20
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067631] <EOI>
[<ffffffff812f9f7e>] ? intel_idle+0xde/0x170
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067638] [<ffffffff812f9f61>] ?
intel_idle+0xc1/0x170
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067645] [<ffffffff81435a17>] ?
cpuidle_idle_call+0xa7/0x140
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067649] [<ffffffff8100a026>] ?
cpu_idle+0xb6/0x110
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067653] [<ffffffff8152dc74>] ?
start_secondary+0x2be/0x301
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067659] ---[ end trace
bcae5b3d43e8bb25 ]---
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067660] Tainting kernel with flag
0x9
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067662] Pid: 0, comm: swapper
veid: 0 Tainted: G --------------- T 2.6.32-042stab106.4 #1
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067664] Call Trace:
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067665] <IRQ>
[<ffffffff8107b671>] ? add_taint+0x71/0x80
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067670] [<ffffffff8107b7f4>] ?
warn_slowpath_common+0x94/0xc0
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067674] [<ffffffff8107b8d6>] ?
warn_slowpath_fmt+0x46/0x50
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067677] [<ffffffff8148f9bb>] ?
dev_watchdog+0x26b/0x280
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067681] [<ffffffff81015009>] ?
sched_clock+0x9/0x10
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067686] [<ffffffff8108f68c>] ?
run_timer_softirq+0x1bc/0x380
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067689] [<ffffffff8148f750>] ?
dev_watchdog+0x0/0x280
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067692] [<ffffffff81034ddd>] ?
lapic_next_event+0x1d/0x30
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067695] [<ffffffff81084c3d>] ?
__do_softirq+0x10d/0x250
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067700] [<ffffffff8100c48c>] ?
call_softirq+0x1c/0x30
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067703] [<ffffffff810102b5>] ?
do_softirq+0x65/0xa0
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067706] [<ffffffff81084a5d>] ?
irq_exit+0xcd/0xd0
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067709] [<ffffffff8153f1ba>] ?
smp_apic_timer_interrupt+0x4a/0x60
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067711] [<ffffffff8100bc93>] ?
apic_timer_interrupt+0x13/0x20
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067713] <EOI>
[<ffffffff812f9f7e>] ? intel_idle+0xde/0x170
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067718] [<ffffffff812f9f61>] ?
intel_idle+0xc1/0x170
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067723] [<ffffffff81435a17>] ?
cpuidle_idle_call+0xa7/0x140
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067727] [<ffffffff8100a026>] ?
cpu_idle+0xb6/0x110
Jun 9 16:35:58 nyccvz2 kernel: [5282279.067729] [<ffffffff8152dc74>] ?
start_secondary+0x2be/0x301
Jun 9 16:36:02 nyccvz2 kernel: [5282283.069323] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:36:18 nyccvz2 kernel: [5282299.063252] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:36:32 nyccvz2 kernel: [5282313.108165] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:36:46 nyccvz2 kernel: [5282327.110353] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:37:00 nyccvz2 kernel: [5282341.078752] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:37:14 nyccvz2 kernel: [5282355.078001] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:37:30 nyccvz2 kernel: [5282371.104739] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:37:44 nyccvz2 kernel: [5282385.092118] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:37:58 nyccvz2 kernel: [5282399.139912] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:38:12 nyccvz2 kernel: [5282413.151028] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:38:26 nyccvz2 kernel: [5282427.159257] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:38:42 nyccvz2 kernel: [5282443.162101] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:38:56 nyccvz2 kernel: [5282457.121562] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 9 16:39:12 nyccvz2 kernel: [5282473.164253] igb 0000:03:00.0: eth0:
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
On Mon, Jun 8, 2015 at 6:27 PM Brandon Whaley <[email protected]> wrote:
> Thanks for the input everyone. I went over notes from previous outages of
> this kind and we have checked that the port was not disabled in the past,
> but I'll make sure we check again next time to be sure.
>
> I'm unable to run ethregs as root due to what I assume is a security
> related kernel config (mmap - try rebooting with iomem=relaxed: Operation
> not permitted), but ethtool -d works on the interface in question so I'll
> gather that information as well when this occurs next time.
>
> Thanks again for your time.
>
> On Mon, Jun 8, 2015 at 5:56 PM Ronciak, John <[email protected]>
> wrote:
>
>> We think that maybe you are having link issues since the problem is
>> intermittent and moving from machine to machine. If it more repeatable on
>> a since system we could do more. Please check the link at the switch port
>> to see if it think it has link on that failing system.
>>
>> Thanks.
>>
>> Cheers,
>> John
>>
>> > -----Original Message-----
>> > From: Fujinaka, Todd [mailto:[email protected]]
>> > Sent: Monday, June 8, 2015 2:53 PM
>> > To: Fujinaka, Todd; Brandon Whaley; [email protected]
>> > Subject: Re: [E1000-devel] igb driver sometimes stops responding after
>> dkms
>> > build
>> >
>> > ethregs would be my choice. ethtool -d also gives some information.
>> >
>> > Todd Fujinaka
>> > Software Application Engineer
>> > Networking Division (ND)
>> > Intel Corporation
>> > [email protected]
>> > (503) 712-4565
>> >
>> > -----Original Message-----
>> > From: Fujinaka, Todd [mailto:[email protected]]
>> > Sent: Monday, June 08, 2015 2:42 PM
>> > To: Brandon Whaley; [email protected]
>> > Subject: Re: [E1000-devel] igb driver sometimes stops responding after
>> dkms
>> > build
>> >
>> > It's difficult to troubleshoot a problem that's erratic and hard to
>> replicate. A
>> > dump of the registers and status from the link partner would be a good
>> idea. I
>> > wonder if it isn't something triggered on the other side (is it a
>> switch?) that
>> > sees something odd and disables the port, and a reboot just takes long
>> enough
>> > that the port comes back?
>> >
>> > Todd Fujinaka
>> > Software Application Engineer
>> > Networking Division (ND)
>> > Intel Corporation
>> > [email protected]
>> > (503) 712-4565
>> >
>> > -----Original Message-----
>> > From: Brandon Whaley [mailto:[email protected]]
>> > Sent: Monday, June 08, 2015 10:27 AM
>> > To: [email protected]
>> > Subject: [E1000-devel] igb driver sometimes stops responding after dkms
>> build
>> >
>> > I use dkms to build the igb driver after new kernel installs on my
>> fleet of
>> > servers using the following commands after every yum update:
>> >
>> > dkms build -m igb -v 5.2.18
>> > dkms install -m igb -v 5.2.18
>> >
>> > About once a month, one of my boxes (a different one each time) will
>> stop
>> > responding after this. Nothing I do is able to recover network
>> connectivity short
>> > of a reboot (not loading/unloading the driver, restarting networking,
>> etc.) and
>> > since these are production machines, it causes some downtime for us.
>> Below is
>> > what you see in the syslog when the event occurs:
>> >
>> > Jun 8 12:29:19 localhost kernel: [147419.444969] ------------[ cut here
>> > ]------------
>> > Jun 8 12:29:19 localhost kernel: [147419.444978] WARNING: at
>> > net/sched/sch_generic.c:267 dev_watchdog+0x26b/0x280() (Tainted: G
>> > --------------- T)
>> > Jun 8 12:29:19 localhost kernel: [147419.444981] Hardware name:
>> X9DRL-3F/iF
>> > Jun 8 12:29:19 localhost kernel: [147419.444982] NETDEV WATCHDOG: eth0
>> > (igb): transmit queue 0 timed out
>> > Jun 8 12:29:19 localhost kernel: [147419.444984] Modules linked in:
>> > mpt3sas mpt2sas raid_class mptctl mptbase ip6t_rt ipt_addrtype xt_policy
>> > aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64
>> > aes_generic cbc kcare(U) vzethdev pio_kaio pio_nfs pio_direct pfmt_raw
>> > pfmt_ploop1 ploop simfs vziolimit vzdquota ip6t_REJECT xfrm6_mode_tunnel
>> > xfrm4_mode_tunnel nf_conntrack_netbios_ns nf_conntrack_broadcast
>> > nf_conntrack_netlink xt_comment nfsd ip6_tunnel ip_vs ipip xt_NFQUEUE
>> > xt_pkttype ecryptfs(T) ip_gre ip_tunnel ipt_MASQUERADE nf_nat_irc
>> xt_helper
>> > nf_conntrack_irc nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6t_LOG
>> > xt_connlimit xt_recent pppoatm atm vzrst vzcpt nfs lockd fscache
>> auth_rpcgss
>> > nfs_acl sunrpc xfrm4_mode_transport xfrm6_mode_transport ccm authenc
>> > esp6 ah6 cnic uio xfrm4_tunnel tunnel4 ipcomp6 xfrm6_tunnel tunnel6
>> ipcomp
>> > xfrm_ipcomp esp4 ah4 af_key arc4 ecb ppp_mppe ppp_deflate zlib_deflate
>> > ppp_async ppp_generic slhc crc_ccitt fuse tun xt_MARK xt_mark vzevent
>> > autofs4 vznetdev vzmon vzdev ipt Jun 8 12:29:19 localhost kernel:
>> _REDIRECT
>> > xt_owner nf_nat_ftp nf_conntrack_ftp iptable_nat nf_nat xt_state
>> xt_length
>> > xt_hl xt_tcpmss xt_TCPMSS xt_multiport xt_limit nf_conntrack_ipv4
>> > nf_conntrack
>> > nf_defrag_ipv4 ipt_LOG xt_DSCP xt_dscp ipt_REJECT iptable_mangle xt_set
>> > iptable_filter iptable_raw ip_tables ip6table_mangle ip6table_filter
>> > ip6table_raw ip6_tables ipv6 ip_set_hash_ip ip_set nfnetlink iTCO_wdt
>> > iTCO_vendor_support ipmi_devintf ipmi_si ipmi_msghandler acpi_pad
>> > e1000e(U) ses enclosure sg igb(U) dca i2c_algo_bit ptp pps_core sb_edac
>> > edac_core
>> > i2c_i801 i2c_core lpc_ich mfd_core shpchp tcp_htcp ext4 jbd2 mbcache
>> > sd_mod crc_t10dif isci libsas scsi_transport_sas ahci megaraid_sas wmi
>> > dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
>> > Jun 8 12:29:19 localhost kernel: [147419.445094] Pid: 0, comm: swapper
>> > veid: 0 Tainted: G --------------- T 2.6.32-042stab108.2 #1
>> > Jun 8 12:29:19 localhost kernel: [147419.445096] Call Trace:
>> > Jun 8 12:29:19 localhost kernel: [147419.445098] <IRQ>
>> [<ffffffff8107b827>] ?
>> > warn_slowpath_common+0x87/0xc0 Jun 8 12:29:19 localhost kernel:
>> > [147419.445109] [<ffffffff8107b916>] ?
>> > warn_slowpath_fmt+0x46/0x50
>> > Jun 8 12:29:19 localhost kernel: [147419.445112] [<ffffffff8148fccb>]
>> ?
>> > dev_watchdog+0x26b/0x280
>> > Jun 8 12:29:19 localhost kernel: [147419.445117] [<ffffffff81015009>]
>> ?
>> > sched_clock+0x9/0x10
>> > Jun 8 12:29:19 localhost kernel: [147419.445123] [<ffffffff8108f6cc>]
>> ?
>> > run_timer_softirq+0x1bc/0x380
>> > Jun 8 12:29:19 localhost kernel: [147419.445126] [<ffffffff8148fa60>]
>> ?
>> > dev_watchdog+0x0/0x280
>> > Jun 8 12:29:19 localhost kernel: [147419.445130] [<ffffffff81034ddd>]
>> ?
>> > lapic_next_event+0x1d/0x30
>> > Jun 8 12:29:19 localhost kernel: [147419.445134] [<ffffffff81084c7d>]
>> ?
>> > __do_softirq+0x10d/0x250
>> > Jun 8 12:29:19 localhost kernel: [147419.445139] [<ffffffff8100c48c>]
>> ?
>> > call_softirq+0x1c/0x30
>> > Jun 8 12:29:19 localhost kernel: [147419.445142] [<ffffffff810102b5>]
>> ?
>> > do_softirq+0x65/0xa0
>> > Jun 8 12:29:19 localhost kernel: [147419.445145] [<ffffffff81084a9d>]
>> ?
>> > irq_exit+0xcd/0xd0
>> > Jun 8 12:29:19 localhost kernel: [147419.445149] [<ffffffff8153f44a>]
>> ?
>> > smp_apic_timer_interrupt+0x4a/0x60
>> > Jun 8 12:29:19 localhost kernel: [147419.445152] [<ffffffff8100bc93>]
>> ?
>> > apic_timer_interrupt+0x13/0x20
>> > Jun 8 12:29:19 localhost kernel: [147419.445154] <EOI>
>> [<ffffffff812fa20e>] ?
>> > intel_idle+0xde/0x170 Jun 8 12:29:19 localhost kernel: [147419.445160]
>> > [<ffffffff812fa1f1>] ?
>> > intel_idle+0xc1/0x170
>> > Jun 8 12:29:19 localhost kernel: [147419.445166] [<ffffffff81435d27>]
>> ?
>> > cpuidle_idle_call+0xa7/0x140
>> > Jun 8 12:29:19 localhost kernel: [147419.445170] [<ffffffff8100a026>]
>> ?
>> > cpu_idle+0xb6/0x110
>> > Jun 8 12:29:19 localhost kernel: [147419.445174] [<ffffffff8152df04>]
>> ?
>> > start_secondary+0x2be/0x301
>> > Jun 8 12:29:19 localhost kernel: [147419.445179] ---[ end trace
>> > 2179b48f00e92658 ]---
>> > Jun 8 12:29:19 localhost kernel: [147419.445180] Tainting kernel with
>> flag
>> > 0x9
>> > Jun 8 12:29:19 localhost kernel: [147419.445182] Pid: 0, comm: swapper
>> > veid: 0 Tainted: G --------------- T 2.6.32-042stab108.2 #1
>> > Jun 8 12:29:19 localhost kernel: [147419.445184] Call Trace:
>> > Jun 8 12:29:19 localhost kernel: [147419.445185] <IRQ>
>> [<ffffffff8107b6b1>] ?
>> > add_taint+0x71/0x80 Jun 8 12:29:19 localhost kernel: [147419.445190]
>> > [<ffffffff8107b834>] ?
>> > warn_slowpath_common+0x94/0xc0
>> > Jun 8 12:29:19 localhost kernel: [147419.445193] [<ffffffff8107b916>]
>> ?
>> > warn_slowpath_fmt+0x46/0x50
>> > Jun 8 12:29:19 localhost kernel: [147419.445196] [<ffffffff8148fccb>]
>> ?
>> > dev_watchdog+0x26b/0x280
>> > Jun 8 12:29:19 localhost kernel: [147419.445199] [<ffffffff81015009>]
>> ?
>> > sched_clock+0x9/0x10
>> > Jun 8 12:29:19 localhost kernel: [147419.445204] [<ffffffff8108f6cc>]
>> ?
>> > run_timer_softirq+0x1bc/0x380
>> > Jun 8 12:29:19 localhost kernel: [147419.445206] [<ffffffff8148fa60>]
>> ?
>> > dev_watchdog+0x0/0x280
>> > Jun 8 12:29:19 localhost kernel: [147419.445209] [<ffffffff81034ddd>]
>> ?
>> > lapic_next_event+0x1d/0x30
>> > Jun 8 12:29:19 localhost kernel: [147419.445213] [<ffffffff81084c7d>]
>> ?
>> > __do_softirq+0x10d/0x250
>> > Jun 8 12:29:19 localhost kernel: [147419.445217] [<ffffffff8100c48c>]
>> ?
>> > call_softirq+0x1c/0x30
>> > Jun 8 12:29:19 localhost kernel: [147419.445219] [<ffffffff810102b5>]
>> ?
>> > do_softirq+0x65/0xa0
>> > Jun 8 12:29:19 localhost kernel: [147419.445222] [<ffffffff81084a9d>]
>> ?
>> > irq_exit+0xcd/0xd0
>> > Jun 8 12:29:19 localhost kernel: [147419.445225] [<ffffffff8153f44a>]
>> ?
>> > smp_apic_timer_interrupt+0x4a/0x60
>> > Jun 8 12:29:19 localhost kernel: [147419.445227] [<ffffffff8100bc93>]
>> ?
>> > apic_timer_interrupt+0x13/0x20
>> > Jun 8 12:29:19 localhost kernel: [147419.445229] <EOI>
>> [<ffffffff812fa20e>] ?
>> > intel_idle+0xde/0x170 Jun 8 12:29:19 localhost kernel: [147419.445233]
>> > [<ffffffff812fa1f1>] ?
>> > intel_idle+0xc1/0x170
>> > Jun 8 12:29:19 localhost kernel: [147419.445238] [<ffffffff81435d27>]
>> ?
>> > cpuidle_idle_call+0xa7/0x140
>> > Jun 8 12:29:19 localhost kernel: [147419.445241] [<ffffffff8100a026>]
>> ?
>> > cpu_idle+0xb6/0x110
>> > Jun 8 12:29:19 localhost kernel: [147419.445243] [<ffffffff8152df04>]
>> ?
>> > start_secondary+0x2be/0x301
>> > Jun 8 12:29:27 localhost kernel: [147427.448304] igb 0000:03:00.0:
>> eth0:
>> > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun
>> 8
>> > 12:29:45 localhost kernel: [147445.478853] igb 0000:03:00.0: eth0:
>> > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun
>> 8
>> > 12:29:59 localhost kernel: [147459.496009] igb 0000:03:00.0: eth0:
>> > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun
>> 8
>> > 12:30:13 localhost kernel: [147473.459481] igb 0000:03:00.0: eth0:
>> > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun
>> 8
>> > 12:30:27 localhost kernel: [147487.460716] igb 0000:03:00.0: eth0:
>> > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun
>> 8
>> > 12:30:45 localhost kernel: [147505.498234] igb 0000:03:00.0: eth0:
>> > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun
>> 8
>> > 12:31:03 localhost kernel: [147523.513873] igb 0000:03:00.0: eth0:
>> > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
>> >
>> > Is there any more information I can provide the next time this
>> happens? If
>> > trends continue I should see it again in 3-5 weeks and will be able to
>> collect
>> > necessary info then. Unfortunately I've yet to find a way to replicate
>> this on
>> > demand.
>> >
>> ------------------------------------------------------------------------------
>> > _______________________________________________
>> > E1000-devel mailing list
>> > [email protected]
>> > https://lists.sourceforge.net/lists/listinfo/e1000-devel
>> > To learn more about Intel® Ethernet, visit
>> > http://communities.intel.com/community/wired
>> >
>> >
>> ------------------------------------------------------------------------------
>> > _______________________________________________
>> > E1000-devel mailing list
>> > [email protected]
>> > https://lists.sourceforge.net/lists/listinfo/e1000-devel
>> > To learn more about Intel® Ethernet, visit
>> > http://communities.intel.com/community/wired
>>
>
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
