We’ll need more info, like what the switch config is on the other end, etc, and reading this in email is pretty painful.
Can you open a bug on e1000.sourceforge.com and submit the logs as attachments? I’d like to see the output of “lspci –vvv”, the whole of /var/log/messages, your “uname –a”, and all of dmesg. “ethtool –d” for the working case would be nice, and “ethtool –i” It seems very odd that “auto speed detect” is “disbled”. You may also want to show us what you actually did to load and unload the driver. Todd Fujinaka Software Application Engineer Networking Division (ND) Intel Corporation todd.fujin...@intel.com (503) 712-4565 From: Brandon Whaley [mailto:redkr...@gmail.com] Sent: Tuesday, June 09, 2015 2:09 PM To: Ronciak, John; Fujinaka, Todd; e1000-devel@lists.sourceforge.net Subject: Re: [E1000-devel] igb driver sometimes stops responding after dkms build I've been repeatedly loading and unloading the latest igb driver on a few boxes in my fleet today and was finally able to replicate the issue and get a register dump. The switch did not report any problem, the port was enabled and up. Here is the register dump from ethtool: 0x00000: CTRL (Device control register) 0x181C0241 Invert Loss-Of-Signal: no Receive flow control: enabled Transmit flow control: enabled VLAN mode: disabled Set link up: 1 D3COLD WakeUp capability advertisement: enabled Auto speed detect: disabled Speed select: 1000Mb/s Force speed: no Force duplex: no 0x00008: STATUS (Device status register) 0x00280783 Duplex: full Link up: link config Transmission: on DMA clock gating: disabled TBI mode: disabled Link speed: 1000Mb/s Bus type: PCI Express 0x00100: RCTL (Receive control register) 0x04048022 Receiver: enabled Store bad packets: disabled Unicast promiscuous: disabled Multicast promiscuous: disabled Long packet: enabled Descriptor minimum threshold size: 1/2 Broadcast accept mode: accept VLAN filter: enabled Cononical form indicator: disabled Discard pause frames: filtered Pass MAC control frames: don't pass Loopback mode: normal Receive buffer size: 2048 0x02808: RDLEN (Receive desc length) 0x00001000 0x02810: RDH (Receive desc head) 0x000000FF 0x02818: RDT (Receive desc tail) 0x000000FF 0x00400: TCTL (Transmit ctrl register) 0xA50400FA Transmitter: enabled Pad short packets: enabled Software XOFF Transmission: disabled Re-transmit on late collision: enabled 0x03808: TDLEN (Transmit desc length) 0x00001000 0x03810: TDH (Transmit desc head) 0x000000EA 0x03818: TDT (Transmit desc tail) 0x000000EA 0x00018: CTRL_EXT (Extended device control) 0x101400C0 0x00018: MDIC (MDI control) 0x180A3C00 0x00024: SCTL (SERDES ANA) 0x00000000 0x00034: CONNSW (Copper/Fiber switch control) 0x00000400 0x00038: VET (VLAN Ether type) 0x00008100 0x00E00: LEDCTL (LED control) 0x07068302 0x01000: PBA (Packet buffer allocation) 0x00000000 0x01008: PBS (Packet buffer size) 0x00000000 0x01048: FRTIMER (Free running timer) 0x00AD979F 0x0104C: TCPTIMER (TCP timer) 0x00000000 0x00010: EEC (EEPROM/FLASH control) 0x00083F1E 0x01580: EICR (Extended interrupt cause) 0x80000001 0x01520: EICS (Extended interrupt cause set) 0x80000001 0x01524: EIMS (Extended interrup set/read) 0x80000000 0x01528: EIMC (Extended interrupt mask clear) 0x80000000 0x0152C: EIAC (Extended interrupt auto clear) 0x00000000 0x01530: EIAM (Extended interrupt auto mask) 0x00000000 0x01500: ICR (Interrupt cause read) 0x800000D5 0x01504: ICS (Interrupt cause set) 0x800000D5 0x01508: IMS (Interrupt mask set/read) 0x4000009D 0x0150C: IMC (Interrupt mask clear) 0x4000009D 0x04100: IAC (Interrupt assertion count) 0x00000000 0x01510: IAM (Interr acknowledge auto-mask) 0x4000009D 0x05AC0: IMIRVP (Immed interr rx VLAN priority) 0x00000000 0x00028: FCAL (Flow control address low) 0x00C28001 0x0002C: FCAH (Flow control address high) 0x00000100 0x00170: FCTTV (Flow control tx timer value) 0x0000FFFF 0x02160: FCRTL (Flow control rx threshold low) 0x80010320 0x02168: FCRTH (Flow control rx threshold high) 0x00010330 0x02460: FCRTV (Flow control refresh threshold) 0x00000000 0x05000: RXCSUM (Receive checksum control) 0x00002F00 0x05004: RLPML (Receive long packet max length) 0x000005F2 0x05008: RFCTL (Receive filter control) 0x00000000 0x05818: MRQC (Multiple rx queues command) 0x00370002 0x0581C: VMD_CTL (VMDq control) 0x00000000 0x00404: TCTL_EXT (Transmit control extended) 0x0000FC40 0x00410: TIPG (Transmit IPG) 0x00601008 0x03590: DTXCTL (DMA tx control) 0x00000080 0x05800: WUC (Wake up control) 0x00000000 0x05808: WUFC (Wake up filter control) 0x00000000 0x05810: WUS (Wake up status) 0x00000000 0x05838: IPAV (IP address valid) 0x00000000 0x05900: WUPL (Wake up packet length) 0x61400C00 0x04200: PCS_CFG (PCS configuration 0) 0x00000008 0x04208: PCS_LCTL (PCS link control) 0x0210000C 0x0420C: PCS_LSTS (PCS link status) 0x0040000D 0x04218: PCS_ANADV (AN advertisement) 0x000001A0 0x0421C: PCS_LPAB (Link partner ability) 0x00000000 0x04220: PCS_NPTX (Next Page transmit) 0x00000000 0x04224: PCS_LPABNP (Link partner ability Next Page) 0x00000000 0x04000: CRCERRS (CRC error count) 0x00000000 0x04004: ALGNERRC (Alignment error count) 0x00000000 0x04008: SYMERRS (Symbol error count) 0x00000000 0x0400C: RXERRC (RX error count) 0x00000000 0x04010: MPC (Missed packets count) 0x0000BC53 0x04014: SCC (Single collision count) 0x00000000 0x04018: ECOL (Excessive collisions count) 0x00000000 0x0401C: MCC (Multiple collision count) 0x00000000 0x04020: LATECOL (Late collisions count) 0x00000000 0x04028: COLC (Collision count) 0x00000000 0x04030: DC (Defer count) 0x00000000 0x04034: TNCRS (Transmit with no CRS) 0x00000000 0x04038: SEC (Sequence error count) 0x00000000 0x0403C: HTDPMC (Host tx discrd pkts MAC count) 0x00000000 0x04040: RLEC (Receive length error count) 0x00000000 0x04048: XONRXC (XON received count) 0x00000000 0x0404C: XONTXC (XON transmitted count) 0x00106088 0x04050: XOFFRXC (XOFF received count) 0x00000000 0x04054: XOFFTXC (XOFF transmitted count) 0x0010C20C 0x04058: FCRUC (FC received unsupported count) 0x00000000 0x0405C: PRC64 (Packets rx (64 B) count) 0x28F63B22 0x04060: PRC127 (Packets rx (65-127 B) count) 0xFED2D250 0x04064: PRC255 (Packets rx (128-255 B) count) 0xCB84AA45 0x04068: PRC511 (Packets rx (256-511 B) count) 0x19680DBD 0x0406C: PRC1023 (Packets rx (512-1023 B) count) 0x1544D787 0x04070: PRC1522 (Packets rx (1024-max B) count) 0xDD436F09 0x04074: GPRC (Good packets received count) 0xFF3E074B 0x04078: BPRC (Broadcast packets rx count) 0x116363E9 0x0407C: MPRC (Multicast packets rx count) 0x01F0D30E 0x04080: GPTC (Good packets tx count) 0x885B7E0D 0x04088: GORCL (Good octets rx count lower) 0xBBCD3E03 0x0408C: GORCH (Good octets rx count upper) 0x00000000 0x04090: GOTCL (Good octets tx count lower) 0xE3FB2867 0x04094: GOTCH (Good octets tx count upper) 0x00000000 0x040A0: RNBC (Receive no buffers count) 0x00224303 0x040A4: RUC (Receive undersize count) 0x00000000 0x040A8: RFC (Receive fragment count) 0x00000000 0x040AC: ROC (Receive oversize count) 0x00000000 0x040B0: RJC (Receive jabber count) 0x00000000 0x040B4: MGPRC (Management packets rx count) 0x00000000 0x040B8: MGPDC (Management pkts dropped count) 0x00000000 0x040BC: MGPTC (Management packets tx count) 0x00000000 0x040C0: TORL (Total octets received lower) 0x00000D36 0x040C4: TORH (Total octets received upper) 0x00000000 0x040C8: TOTL (Total octets transmitted lower) 0x00001DB9 0x040CC: TOTH (Total octets transmitted upper) 0x00000000 0x040D0: TPR (Total packets received) 0x0089FF8E 0x040D4: TPT (Total packets transmitted) 0x887CAC52 0x040D8: PTC64 (Packets tx (64 B) count) 0x42F682FC 0x040DC: PTC127 (Packets tx (65-127 B) count) 0xFAC12DDB 0x040E0: PTC255 (Packets tx (128-255 B) count) 0x3F97A509 0x040E4: PTC511 (Packets tx (256-511 B) count) 0x4ADC83CB 0x040E8: PTC1023 (Packets tx (512-1023 B) count) 0x35B56FE5 0x040EC: PTC1522 (Packets tx (> 1024 B) count) 0x8A7A3CC8 0x040F0: MPTC (Multicast packets tx count) 0x0010598B 0x040F4: BPTC (Broadcast packets tx count) 0x0000163F 0x040F8: TSCTC (TCP segment context tx count) 0x0099940A 0x04100: IAC (Interrupt assertion count) 0xC71B6C92 0x04104: RPTHC (Rx packets to host count) 0x00000000 0x04118: HGPTC (Host good packets tx count) 0x00000000 0x04128: HGORCL (Host good octets rx cnt lower) 0x00000000 0x0412C: HGORCH (Host good octets rx cnt upper) 0x00000000 0x04130: HGOTCL (Host good octets tx cnt lower) 0x00000000 0x04134: HGOTCH (Host good octets tx cnt upper) 0x00000000 0x04138: LENNERS (Length error count) 0x00000000 0x04228: SCVPC (SerDes/SGMII code viol pkt cnt) 0x00000000 0x0A018: HRMPC (Header redir missed pkt count) 0x00000000 0x0280C: SRRCTL0 (Split and replic rx ctl0) 0x42000402 0x0290C: SRRCTL1 (Split and replic rx ctl1) 0x80000400 0x02A0C: SRRCTL2 (Split and replic rx ctl2) 0x80000400 0x02B0C: SRRCTL3 (Split and replic rx ctl3) 0x80000400 0x05480: PSRTYPE0 (Packet split receive type0) 0x0007FFFE 0x05484: PSRTYPE1 (Packet split receive type1) 0x0007FFFE 0x05488: PSRTYPE2 (Packet split receive type2) 0x0007FFFE 0x0548C: PSRTYPE3 (Packet split receive type3) 0x0007FFFE 0x02800: RDBAL0 (Rx desc base addr low0) 0x0EAA0000 0x02900: RDBAL1 (Rx desc base addr low1) 0x00000000 0x02A00: RDBAL2 (Rx desc base addr low2) 0x00000000 0x02B00: RDBAL3 (Rx desc base addr low3) 0x00000000 0x02804: RDBAH0 (Rx desc base addr high0) 0x00000010 0x02904: RDBAH1 (Rx desc base addr high1) 0x00000000 0x02A04: RDBAH2 (Rx desc base addr high2) 0x00000000 0x02B04: RDBAH3 (Rx desc base addr high3) 0x00000000 0x02808: RDLEN0 (Rx descriptor length0) 0x00001000 0x02908: RDLEN1 (Rx descriptor length1) 0x00000000 0x02A08: RDLEN2 (Rx descriptor length2) 0x00000000 0x02B08: RDLEN3 (Rx descriptor length3) 0x00000000 0x02810: RDH0 (Rx descriptor head0) 0x000000FF 0x02910: RDH1 (Rx descriptor head1) 0x00000000 0x02A10: RDH2 (Rx descriptor head2) 0x00000000 0x02B10: RDH3 (Rx descriptor head3) 0x00000000 0x02818: RDT0 (Rx descriptor tail0) 0x000000FF 0x02918: RDT1 (Rx descriptor tail1) 0x00000000 0x02A18: RDT2 (Rx descriptor tail2) 0x00000000 0x02B18: RDT3 (Rx descriptor tail3) 0x00000000 0x02828: RXDCTL0 (Rx descriptor control0) 0x02040808 0x02928: RXDCTL1 (Rx descriptor control1) 0x00010A0C 0x02A28: RXDCTL2 (Rx descriptor control2) 0x00010A0C 0x02B28: RXDCTL3 (Rx descriptor control3) 0x00010A0C 0x01680: EITR0 (Interrupt throttle0) 0x00C000C4 0x01684: EITR1 (Interrupt throttle1) 0x00000000 0x01688: EITR2 (Interrupt throttle2) 0x00000000 0x0168C: EITR3 (Interrupt throttle3) 0x00000000 0x01690: EITR4 (Interrupt throttle4) 0x00000000 0x01694: EITR5 (Interrupt throttle5) 0x00000000 0x01698: EITR6 (Interrupt throttle6) 0x00000000 0x0169C: EITR7 (Interrupt throttle7) 0x00000000 0x016A0: EITR8 (Interrupt throttle8) 0x00000000 0x016A4: EITR9 (Interrupt throttle9) 0x00000000 0x05A80: IMIR0 (Immediate interrupt Rx0) 0x00000000 0x05A84: IMIR1 (Immediate interrupt Rx1) 0x00000000 0x05A88: IMIR2 (Immediate interrupt Rx2) 0x00000000 0x05A8C: IMIR3 (Immediate interrupt Rx3) 0x00000000 0x05A90: IMIR4 (Immediate interrupt Rx4) 0x00000000 0x05A94: IMIR5 (Immediate interrupt Rx5) 0x00000000 0x05A98: IMIR6 (Immediate interrupt Rx6) 0x00000000 0x05A9C: IMIR7 (Immediate interrupt Rx7) 0x00000000 0x05AA0: IMIREXT0 (Immediate interr Rx extended0) 0x00000000 0x05AA4: IMIREXT1 (Immediate interr Rx extended1) 0x00000000 0x05AA8: IMIREXT2 (Immediate interr Rx extended2) 0x00000000 0x05AAC: IMIREXT3 (Immediate interr Rx extended3) 0x00000000 0x05AB0: IMIREXT4 (Immediate interr Rx extended4) 0x00000000 0x05AB4: IMIREXT5 (Immediate interr Rx extended5) 0x00000000 0x05AB8: IMIREXT6 (Immediate interr Rx extended6) 0x00000000 0x05ABC: IMIREXT7 (Immediate interr Rx extended7) 0x00000000 0x05400: RAL00 (Receive address low00) 0x359F36A0 0x05408: RAL01 (Receive address low01) 0x00000000 0x05410: RAL02 (Receive address low02) 0x00000000 0x05418: RAL03 (Receive address low03) 0x00000000 0x05420: RAL04 (Receive address low04) 0x00000000 0x05428: RAL05 (Receive address low05) 0x00000000 0x05430: RAL06 (Receive address low06) 0x00000000 0x05438: RAL07 (Receive address low07) 0x00000000 0x05440: RAL08 (Receive address low08) 0x00000000 0x05448: RAL09 (Receive address low09) 0x00000000 0x05450: RAL10 (Receive address low10) 0x00000000 0x05458: RAL11 (Receive address low11) 0x00000000 0x05460: RAL12 (Receive address low12) 0x00000000 0x05468: RAL13 (Receive address low13) 0x00000000 0x05470: RAL14 (Receive address low14) 0x00000000 0x05478: RAL15 (Receive address low15) 0x00000000 0x05404: RAH00 (Receive address high00) 0x800460CC 0x0540C: RAH01 (Receive address high01) 0x00000000 0x05414: RAH02 (Receive address high02) 0x00000000 0x0541C: RAH03 (Receive address high03) 0x00000000 0x05424: RAH04 (Receive address high04) 0x00000000 0x0542C: RAH05 (Receive address high05) 0x00000000 0x05434: RAH06 (Receive address high06) 0x00000000 0x0543C: RAH07 (Receive address high07) 0x00000000 0x05444: RAH08 (Receive address high08) 0x00000000 0x0544C: RAH09 (Receive address high09) 0x00000000 0x05454: RAH10 (Receive address high10) 0x00000000 0x0545C: RAH11 (Receive address high11) 0x00000000 0x05464: RAH12 (Receive address high12) 0x00000000 0x0546C: RAH13 (Receive address high13) 0x00000000 0x05474: RAH14 (Receive address high14) 0x00000000 0x0547C: RAH15 (Receive address high15) 0x00000000 0x03800: TDBAL0 (Tx desc base address low0) 0x0E884000 0x03900: TDBAL1 (Tx desc base address low1) 0x00000000 0x03A00: TDBAL2 (Tx desc base address low2) 0x00000000 0x03B00: TDBAL3 (Tx desc base address low3) 0x00000000 0x03804: TDBAH0 (Tx desc base address high0) 0x00000010 0x03904: TDBAH1 (Tx desc base address high1) 0x00000000 0x03A04: TDBAH2 (Tx desc base address high2) 0x00000000 0x03B04: TDBAH3 (Tx desc base address high3) 0x00000000 0x03808: TDLEN0 (Tx descriptor length0) 0x00001000 0x03908: TDLEN1 (Tx descriptor length1) 0x00000000 0x03A08: TDLEN2 (Tx descriptor length2) 0x00000000 0x03B08: TDLEN3 (Tx descriptor length3) 0x00000000 0x03810: TDH0 (Transmit descriptor head0) 0x000000EA 0x03910: TDH1 (Transmit descriptor head1) 0x00000000 0x03A10: TDH2 (Transmit descriptor head2) 0x00000000 0x03B10: TDH3 (Transmit descriptor head3) 0x00000000 0x03818: TDT0 (Transmit descriptor tail0) 0x000000EA 0x03918: TDT1 (Transmit descriptor tail1) 0x00000000 0x03A18: TDT2 (Transmit descriptor tail2) 0x00000000 0x03B18: TDT3 (Transmit descriptor tail3) 0x00000000 0x03828: TXDCTL0 (Transmit descriptor control0) 0x02100108 0x03928: TXDCTL1 (Transmit descriptor control1) 0x00000000 0x03A28: TXDCTL2 (Transmit descriptor control2) 0x00000000 0x03B28: TXDCTL3 (Transmit descriptor control3) 0x00000000 0x03838: TDWBAL0 (Tx desc complete wb addr low0) 0x00000000 0x03938: TDWBAL1 (Tx desc complete wb addr low1) 0x00000000 0x03A38: TDWBAL2 (Tx desc complete wb addr low2) 0x00000000 0x03B38: TDWBAL3 (Tx desc complete wb addr low3) 0x00000000 0x0383C: TDWBAH0 (Tx desc complete wb addr hi0) 0x00000000 0x0393C: TDWBAH1 (Tx desc complete wb addr hi1) 0x00000000 0x03A3C: TDWBAH2 (Tx desc complete wb addr hi2) 0x00000000 0x03B3C: TDWBAH3 (Tx desc complete wb addr hi3) 0x00000000 0x03814: DCA_TXCTRL0 (Tx DCA control0) 0x00002A00 0x03914: DCA_TXCTRL1 (Tx DCA control1) 0x00002A00 0x03A14: DCA_TXCTRL2 (Tx DCA control2) 0x00002A00 0x03B14: DCA_TXCTRL3 (Tx DCA control3) 0x00002A00 0x05840: IP4AT0 (IPv4 address table0) 0x00000000 0x05848: IP4AT1 (IPv4 address table1) 0x00000000 0x05850: IP4AT2 (IPv4 address table2) 0x00000000 0x05858: IP4AT3 (IPv4 address table3) 0x00000000 0x05880: IP6AT0 (IPv6 address table0) 0x00000000 0x05884: IP6AT1 (IPv6 address table1) 0x00000000 0x05888: IP6AT2 (IPv6 address table2) 0x00000000 0x0588C: IP6AT3 (IPv6 address table3) 0x00000000 0x05A00: WUPM00 (Wake up packet memory00) 0x359F36A0 0x05A04: WUPM01 (Wake up packet memory01) 0x120060CC 0x05A08: WUPM02 (Wake up packet memory02) 0x00B891F2 0x05A0C: WUPM03 (Wake up packet memory03) 0x00450008 0x05A10: WUPM04 (Wake up packet memory04) 0xFE8A6005 0x05A14: WUPM05 (Wake up packet memory05) 0x06350040 0x05A18: WUPM06 (Wake up packet memory06) 0xC4424420 0x05A1C: WUPM07 (Wake up packet memory07) 0xBF6B5B78 0x05A20: WUPM08 (Wake up packet memory08) 0xBA13776E 0x05A24: WUPM09 (Wake up packet memory09) 0x70CD5EC0 0x05A28: WUPM10 (Wake up packet memory10) 0xA33E5E1B 0x05A2C: WUPM11 (Wake up packet memory11) 0x10502541 0x05A30: WUPM12 (Wake up packet memory12) 0xF20B9E20 0x05A34: WUPM13 (Wake up packet memory13) 0x4D590000 0x05A38: WUPM14 (Wake up packet memory14) 0x13004753 0x05A3C: WUPM15 (Wake up packet memory15) 0xC3000000 0x05A40: WUPM16 (Wake up packet memory16) 0x0000F000 0x05A44: WUPM17 (Wake up packet memory17) 0x4A000100 0x05A48: WUPM18 (Wake up packet memory18) 0x303347EB 0x05A4C: WUPM19 (Wake up packet memory19) 0x3380C032 0x05A50: WUPM20 (Wake up packet memory20) 0x80C03531 0x05A54: WUPM21 (Wake up packet memory21) 0xC0303033 0x05A58: WUPM22 (Wake up packet memory22) 0x35313380 0x05A5C: WUPM23 (Wake up packet memory23) 0xC03780C0 0x05A60: WUPM24 (Wake up packet memory24) 0x63657280 0x05A64: WUPM25 (Wake up packet memory25) 0x69706163 0x05A68: WUPM26 (Wake up packet memory26) 0xC0736570 0x05A6C: WUPM27 (Wake up packet memory27) 0xC0303180 0x05A70: WUPM28 (Wake up packet memory28) 0x80C03080 0x05A74: WUPM29 (Wake up packet memory29) 0x80C03331 0x05A78: WUPM30 (Wake up packet memory30) 0x3180C031 0x05A7C: WUPM31 (Wake up packet memory31) 0x80C03239 0x09000: FFMT000 (Flexible filter mask table000) 0x00000000 0x09008: FFMT001 (Flexible filter mask table001) 0x00000000 0x09010: FFMT002 (Flexible filter mask table002) 0x00000000 0x09018: FFMT003 (Flexible filter mask table003) 0x00000000 0x09020: FFMT004 (Flexible filter mask table004) 0x00000000 0x09028: FFMT005 (Flexible filter mask table005) 0x00000000 0x09030: FFMT006 (Flexible filter mask table006) 0x00000000 0x09038: FFMT007 (Flexible filter mask table007) 0x00000000 0x09040: FFMT008 (Flexible filter mask table008) 0x00000000 0x09048: FFMT009 (Flexible filter mask table009) 0x00000000 0x09050: FFMT010 (Flexible filter mask table010) 0x00000000 0x09058: FFMT011 (Flexible filter mask table011) 0x00000000 0x09060: FFMT012 (Flexible filter mask table012) 0x00000000 0x09068: FFMT013 (Flexible filter mask table013) 0x00000000 0x09070: FFMT014 (Flexible filter mask table014) 0x00000000 0x09078: FFMT015 (Flexible filter mask table015) 0x00000000 0x09080: FFMT016 (Flexible filter mask table016) 0x00000000 0x09088: FFMT017 (Flexible filter mask table017) 0x00000000 0x09090: FFMT018 (Flexible filter mask table018) 0x00000000 0x09098: FFMT019 (Flexible filter mask table019) 0x00000000 0x090A0: FFMT020 (Flexible filter mask table020) 0x00000000 0x090A8: FFMT021 (Flexible filter mask table021) 0x00000000 0x090B0: FFMT022 (Flexible filter mask table022) 0x00000000 0x090B8: FFMT023 (Flexible filter mask table023) 0x00000000 0x090C0: FFMT024 (Flexible filter mask table024) 0x00000000 0x090C8: FFMT025 (Flexible filter mask table025) 0x00000000 0x090D0: FFMT026 (Flexible filter mask table026) 0x00000000 0x090D8: FFMT027 (Flexible filter mask table027) 0x00000000 0x090E0: FFMT028 (Flexible filter mask table028) 0x00000000 0x090E8: FFMT029 (Flexible filter mask table029) 0x00000000 0x090F0: FFMT030 (Flexible filter mask table030) 0x00000000 0x090F8: FFMT031 (Flexible filter mask table031) 0x00000000 0x09100: FFMT032 (Flexible filter mask table032) 0x00000000 0x09108: FFMT033 (Flexible filter mask table033) 0x00000000 0x09110: FFMT034 (Flexible filter mask table034) 0x00000000 0x09118: FFMT035 (Flexible filter mask table035) 0x00000000 0x09120: FFMT036 (Flexible filter mask table036) 0x00000000 0x09128: FFMT037 (Flexible filter mask table037) 0x00000000 0x09130: FFMT038 (Flexible filter mask table038) 0x00000000 0x09138: FFMT039 (Flexible filter mask table039) 0x00000000 0x09140: FFMT040 (Flexible filter mask table040) 0x00000000 0x09148: FFMT041 (Flexible filter mask table041) 0x00000000 0x09150: FFMT042 (Flexible filter mask table042) 0x00000000 0x09158: FFMT043 (Flexible filter mask table043) 0x00000000 0x09160: FFMT044 (Flexible filter mask table044) 0x00000000 0x09168: FFMT045 (Flexible filter mask table045) 0x00000000 0x09170: FFMT046 (Flexible filter mask table046) 0x00000000 0x09178: FFMT047 (Flexible filter mask table047) 0x00000000 0x09180: FFMT048 (Flexible filter mask table048) 0x00000000 0x09188: FFMT049 (Flexible filter mask table049) 0x00000000 0x09190: FFMT050 (Flexible filter mask table050) 0x00000000 0x09198: FFMT051 (Flexible filter mask table051) 0x00000000 0x091A0: FFMT052 (Flexible filter mask table052) 0x00000000 0x091A8: FFMT053 (Flexible filter mask table053) 0x00000000 0x091B0: FFMT054 (Flexible filter mask table054) 0x00000000 0x091B8: FFMT055 (Flexible filter mask table055) 0x00000000 0x091C0: FFMT056 (Flexible filter mask table056) 0x00000000 0x091C8: FFMT057 (Flexible filter mask table057) 0x00000000 0x091D0: FFMT058 (Flexible filter mask table058) 0x00000000 0x091D8: FFMT059 (Flexible filter mask table059) 0x00000000 0x091E0: FFMT060 (Flexible filter mask table060) 0x00000000 0x091E8: FFMT061 (Flexible filter mask table061) 0x00000000 0x091F0: FFMT062 (Flexible filter mask table062) 0x00000000 0x091F8: FFMT063 (Flexible filter mask table063) 0x00000000 0x09200: FFMT064 (Flexible filter mask table064) 0x00000000 0x09208: FFMT065 (Flexible filter mask table065) 0x00000000 0x09210: FFMT066 (Flexible filter mask table066) 0x00000000 0x09218: FFMT067 (Flexible filter mask table067) 0x00000000 0x09220: FFMT068 (Flexible filter mask table068) 0x00000000 0x09228: FFMT069 (Flexible filter mask table069) 0x00000000 0x09230: FFMT070 (Flexible filter mask table070) 0x00000000 0x09238: FFMT071 (Flexible filter mask table071) 0x00000000 0x09240: FFMT072 (Flexible filter mask table072) 0x00000000 0x09248: FFMT073 (Flexible filter mask table073) 0x00000000 0x09250: FFMT074 (Flexible filter mask table074) 0x00000000 0x09258: FFMT075 (Flexible filter mask table075) 0x00000000 0x09260: FFMT076 (Flexible filter mask table076) 0x00000000 0x09268: FFMT077 (Flexible filter mask table077) 0x00000000 0x09270: FFMT078 (Flexible filter mask table078) 0x00000000 0x09278: FFMT079 (Flexible filter mask table079) 0x00000000 0x09280: FFMT080 (Flexible filter mask table080) 0x00000000 0x09288: FFMT081 (Flexible filter mask table081) 0x00000000 0x09290: FFMT082 (Flexible filter mask table082) 0x00000000 0x09298: FFMT083 (Flexible filter mask table083) 0x00000000 0x092A0: FFMT084 (Flexible filter mask table084) 0x00000000 0x092A8: FFMT085 (Flexible filter mask table085) 0x00000000 0x092B0: FFMT086 (Flexible filter mask table086) 0x00000000 0x092B8: FFMT087 (Flexible filter mask table087) 0x00000000 0x092C0: FFMT088 (Flexible filter mask table088) 0x00000000 0x092C8: FFMT089 (Flexible filter mask table089) 0x00000000 0x092D0: FFMT090 (Flexible filter mask table090) 0x00000000 0x092D8: FFMT091 (Flexible filter mask table091) 0x00000000 0x092E0: FFMT092 (Flexible filter mask table092) 0x00000000 0x092E8: FFMT093 (Flexible filter mask table093) 0x00000000 0x092F0: FFMT094 (Flexible filter mask table094) 0x00000000 0x092F8: FFMT095 (Flexible filter mask table095) 0x00000000 0x09300: FFMT096 (Flexible filter mask table096) 0x00000000 0x09308: FFMT097 (Flexible filter mask table097) 0x00000000 0x09310: FFMT098 (Flexible filter mask table098) 0x00000000 0x09318: FFMT099 (Flexible filter mask table099) 0x00000000 0x09320: FFMT100 (Flexible filter mask table100) 0x00000000 0x09328: FFMT101 (Flexible filter mask table101) 0x00000000 0x09330: FFMT102 (Flexible filter mask table102) 0x00000000 0x09338: FFMT103 (Flexible filter mask table103) 0x00000000 0x09340: FFMT104 (Flexible filter mask table104) 0x00000000 0x09348: FFMT105 (Flexible filter mask table105) 0x00000000 0x09350: FFMT106 (Flexible filter mask table106) 0x00000000 0x09358: FFMT107 (Flexible filter mask table107) 0x00000000 0x09360: FFMT108 (Flexible filter mask table108) 0x00000000 0x09368: FFMT109 (Flexible filter mask table109) 0x00000000 0x09370: FFMT110 (Flexible filter mask table110) 0x00000000 0x09378: FFMT111 (Flexible filter mask table111) 0x00000000 0x09380: FFMT112 (Flexible filter mask table112) 0x00000000 0x09388: FFMT113 (Flexible filter mask table113) 0x00000000 0x09390: FFMT114 (Flexible filter mask table114) 0x00000000 0x09398: FFMT115 (Flexible filter mask table115) 0x00000000 0x093A0: FFMT116 (Flexible filter mask table116) 0x00000000 0x093A8: FFMT117 (Flexible filter mask table117) 0x00000000 0x093B0: FFMT118 (Flexible filter mask table118) 0x00000000 0x093B8: FFMT119 (Flexible filter mask table119) 0x00000000 0x093C0: FFMT120 (Flexible filter mask table120) 0x00000000 0x093C8: FFMT121 (Flexible filter mask table121) 0x00000000 0x093D0: FFMT122 (Flexible filter mask table122) 0x00000000 0x093D8: FFMT123 (Flexible filter mask table123) 0x00000000 0x093E0: FFMT124 (Flexible filter mask table124) 0x00000000 0x093E8: FFMT125 (Flexible filter mask table125) 0x00000000 0x093F0: FFMT126 (Flexible filter mask table126) 0x00000000 0x093F8: FFMT127 (Flexible filter mask table127) 0x00000000 0x09800: FFVT000 (Flexible filter value table000) 0x00000000 0x09808: FFVT001 (Flexible filter value table001) 0x00000000 0x09810: FFVT002 (Flexible filter value table002) 0x00000000 0x09818: FFVT003 (Flexible filter value table003) 0x00000000 0x09820: FFVT004 (Flexible filter value table004) 0x00000000 0x09828: FFVT005 (Flexible filter value table005) 0x00000000 0x09830: FFVT006 (Flexible filter value table006) 0x00000000 0x09838: FFVT007 (Flexible filter value table007) 0x00000000 0x09840: FFVT008 (Flexible filter value table008) 0x00000000 0x09848: FFVT009 (Flexible filter value table009) 0x00000000 0x09850: FFVT010 (Flexible filter value table010) 0x00000000 0x09858: FFVT011 (Flexible filter value table011) 0x00000000 0x09860: FFVT012 (Flexible filter value table012) 0x00000000 0x09868: FFVT013 (Flexible filter value table013) 0x00000000 0x09870: FFVT014 (Flexible filter value table014) 0x00000000 0x09878: FFVT015 (Flexible filter value table015) 0x00000000 0x09880: FFVT016 (Flexible filter value table016) 0x00000000 0x09888: FFVT017 (Flexible filter value table017) 0x00000000 0x09890: FFVT018 (Flexible filter value table018) 0x00000000 0x09898: FFVT019 (Flexible filter value table019) 0x00000000 0x098A0: FFVT020 (Flexible filter value table020) 0x00000000 0x098A8: FFVT021 (Flexible filter value table021) 0x00000000 0x098B0: FFVT022 (Flexible filter value table022) 0x00000000 0x098B8: FFVT023 (Flexible filter value table023) 0x00000000 0x098C0: FFVT024 (Flexible filter value table024) 0x00000000 0x098C8: FFVT025 (Flexible filter value table025) 0x00000000 0x098D0: FFVT026 (Flexible filter value table026) 0x00000000 0x098D8: FFVT027 (Flexible filter value table027) 0x00000000 0x098E0: FFVT028 (Flexible filter value table028) 0x00000000 0x098E8: FFVT029 (Flexible filter value table029) 0x00000000 0x098F0: FFVT030 (Flexible filter value table030) 0x00000000 0x098F8: FFVT031 (Flexible filter value table031) 0x00000000 0x09900: FFVT032 (Flexible filter value table032) 0x00000000 0x09908: FFVT033 (Flexible filter value table033) 0x00000000 0x09910: FFVT034 (Flexible filter value table034) 0x00000000 0x09918: FFVT035 (Flexible filter value table035) 0x00000000 0x09920: FFVT036 (Flexible filter value table036) 0x00000000 0x09928: FFVT037 (Flexible filter value table037) 0x00000000 0x09930: FFVT038 (Flexible filter value table038) 0x00000000 0x09938: FFVT039 (Flexible filter value table039) 0x00000000 0x09940: FFVT040 (Flexible filter value table040) 0x00000000 0x09948: FFVT041 (Flexible filter value table041) 0x00000000 0x09950: FFVT042 (Flexible filter value table042) 0x00000000 0x09958: FFVT043 (Flexible filter value table043) 0x00000000 0x09960: FFVT044 (Flexible filter value table044) 0x00000000 0x09968: FFVT045 (Flexible filter value table045) 0x00000000 0x09970: FFVT046 (Flexible filter value table046) 0x00000000 0x09978: FFVT047 (Flexible filter value table047) 0x00000000 0x09980: FFVT048 (Flexible filter value table048) 0x00000000 0x09988: FFVT049 (Flexible filter value table049) 0x00000000 0x09990: FFVT050 (Flexible filter value table050) 0x00000000 0x09998: FFVT051 (Flexible filter value table051) 0x00000000 0x099A0: FFVT052 (Flexible filter value table052) 0x00000000 0x099A8: FFVT053 (Flexible filter value table053) 0x00000000 0x099B0: FFVT054 (Flexible filter value table054) 0x00000000 0x099B8: FFVT055 (Flexible filter value table055) 0x00000000 0x099C0: FFVT056 (Flexible filter value table056) 0x00000000 0x099C8: FFVT057 (Flexible filter value table057) 0x00000000 0x099D0: FFVT058 (Flexible filter value table058) 0x00000000 0x099D8: FFVT059 (Flexible filter value table059) 0x00000000 0x099E0: FFVT060 (Flexible filter value table060) 0x00000000 0x099E8: FFVT061 (Flexible filter value table061) 0x00000000 0x099F0: FFVT062 (Flexible filter value table062) 0x00000000 0x099F8: FFVT063 (Flexible filter value table063) 0x00000000 0x09A00: FFVT064 (Flexible filter value table064) 0x00000000 0x09A08: FFVT065 (Flexible filter value table065) 0x00000000 0x09A10: FFVT066 (Flexible filter value table066) 0x00000000 0x09A18: FFVT067 (Flexible filter value table067) 0x00000000 0x09A20: FFVT068 (Flexible filter value table068) 0x00000000 0x09A28: FFVT069 (Flexible filter value table069) 0x00000000 0x09A30: FFVT070 (Flexible filter value table070) 0x00000000 0x09A38: FFVT071 (Flexible filter value table071) 0x00000000 0x09A40: FFVT072 (Flexible filter value table072) 0x00000000 0x09A48: FFVT073 (Flexible filter value table073) 0x00000000 0x09A50: FFVT074 (Flexible filter value table074) 0x00000000 0x09A58: FFVT075 (Flexible filter value table075) 0x00000000 0x09A60: FFVT076 (Flexible filter value table076) 0x00000000 0x09A68: FFVT077 (Flexible filter value table077) 0x00000000 0x09A70: FFVT078 (Flexible filter value table078) 0x00000000 0x09A78: FFVT079 (Flexible filter value table079) 0x00000000 0x09A80: FFVT080 (Flexible filter value table080) 0x00000000 0x09A88: FFVT081 (Flexible filter value table081) 0x00000000 0x09A90: FFVT082 (Flexible filter value table082) 0x00000000 0x09A98: FFVT083 (Flexible filter value table083) 0x00000000 0x09AA0: FFVT084 (Flexible filter value table084) 0x00000000 0x09AA8: FFVT085 (Flexible filter value table085) 0x00000000 0x09AB0: FFVT086 (Flexible filter value table086) 0x00000000 0x09AB8: FFVT087 (Flexible filter value table087) 0x00000000 0x09AC0: FFVT088 (Flexible filter value table088) 0x00000000 0x09AC8: FFVT089 (Flexible filter value table089) 0x00000000 0x09AD0: FFVT090 (Flexible filter value table090) 0x00000000 0x09AD8: FFVT091 (Flexible filter value table091) 0x00000000 0x09AE0: FFVT092 (Flexible filter value table092) 0x00000000 0x09AE8: FFVT093 (Flexible filter value table093) 0x00000000 0x09AF0: FFVT094 (Flexible filter value table094) 0x00000000 0x09AF8: FFVT095 (Flexible filter value table095) 0x00000000 0x09B00: FFVT096 (Flexible filter value table096) 0x00000000 0x09B08: FFVT097 (Flexible filter value table097) 0x00000000 0x09B10: FFVT098 (Flexible filter value table098) 0x00000000 0x09B18: FFVT099 (Flexible filter value table099) 0x00000000 0x09B20: FFVT100 (Flexible filter value table100) 0x00000000 0x09B28: FFVT101 (Flexible filter value table101) 0x00000000 0x09B30: FFVT102 (Flexible filter value table102) 0x00000000 0x09B38: FFVT103 (Flexible filter value table103) 0x00000000 0x09B40: FFVT104 (Flexible filter value table104) 0x00000000 0x09B48: FFVT105 (Flexible filter value table105) 0x00000000 0x09B50: FFVT106 (Flexible filter value table106) 0x00000000 0x09B58: FFVT107 (Flexible filter value table107) 0x00000000 0x09B60: FFVT108 (Flexible filter value table108) 0x00000000 0x09B68: FFVT109 (Flexible filter value table109) 0x00000000 0x09B70: FFVT110 (Flexible filter value table110) 0x00000000 0x09B78: FFVT111 (Flexible filter value table111) 0x00000000 0x09B80: FFVT112 (Flexible filter value table112) 0x00000000 0x09B88: FFVT113 (Flexible filter value table113) 0x00000000 0x09B90: FFVT114 (Flexible filter value table114) 0x00000000 0x09B98: FFVT115 (Flexible filter value table115) 0x00000000 0x09BA0: FFVT116 (Flexible filter value table116) 0x00000000 0x09BA8: FFVT117 (Flexible filter value table117) 0x00000000 0x09BB0: FFVT118 (Flexible filter value table118) 0x00000000 0x09BB8: FFVT119 (Flexible filter value table119) 0x00000000 0x09BC0: FFVT120 (Flexible filter value table120) 0x00000000 0x09BC8: FFVT121 (Flexible filter value table121) 0x00000000 0x09BD0: FFVT122 (Flexible filter value table122) 0x00000000 0x09BD8: FFVT123 (Flexible filter value table123) 0x00000000 0x09BE0: FFVT124 (Flexible filter value table124) 0x00000000 0x09BE8: FFVT125 (Flexible filter value table125) 0x00000000 0x09BF0: FFVT126 (Flexible filter value table126) 0x00000000 0x09BF8: FFVT127 (Flexible filter value table127) 0x00000000 0x05F00: FFLT0 (Flexible filter length table0) 0x00000000 0x05F08: FFLT1 (Flexible filter length table1) 0x00000000 0x05F10: FFLT2 (Flexible filter length table2) 0x00000000 0x05F18: FFLT3 (Flexible filter length table3) 0x00000000 0x03410: TDFH (Tx data FIFO head) 0x00000604 0x03418: TDFT (Tx data FIFO tail) 0x00000604 0x03420: TDFHS (Tx data FIFO head saved) 0x00000604 0x03430: TDFPC (Tx data FIFO packet count) 0x00000000 Here is the kernel log from this latest event: Jun 9 16:35:58 nyccvz2 kernel: [5282279.067444] ------------[ cut here ]------------ Jun 9 16:35:58 nyccvz2 kernel: [5282279.067452] WARNING: at net/sched/sch_generic.c:267 dev_watchdog+0x26b/0x280() (Tainted: G --------------- T) Jun 9 16:35:58 nyccvz2 kernel: [5282279.067455] Hardware name: X9DRL-3F/iF Jun 9 16:35:58 nyccvz2 kernel: [5282279.067456] NETDEV WATCHDOG: eth0 (igb): transmit queue 0 timed out Jun 9 16:35:58 nyccvz2 kernel: [5282279.067458] Modules linked in: xt_connmark xt_CONNMARK ip6table_raw mpt3sas mpt2sas raid_class mptctl mptbase xt_hashlimit xt_policy ip6t_rt ipt_addrtype ctr twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic cbc crypto_null aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 aes_generic xcbc rmd160 sha512_generic vzethdev pio_nfs pio_direct pfmt_raw pfmt_ploop1 ploop simfs vziolimit vzdquota ip6t_REJECT xfrm6_mode_tunnel xfrm4_mode_tunnel nf_conntrack_netbios_ns nf_ conntrack_broadcast nf_conntrack_netlink xt_comment nfsd ip6_tunnel ip_vs ipip xt_NFQUEUE xt_pkttype ecryptfs(T) ip_gre ip_tunnel ipt_MASQUERADE nf_nat_irc xt_helper nf_conntrack_irc nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6t_LOG xt_connlimit xt_recent pppoatm atm vzrst vzcpt nfs lockd fscache auth_rpcgss nfs_acl sunrpc xfrm4_mode_transport xfrm6_mode_transport ccm authenc esp6 ah6 cnic uio xfrm4_tunnel tunnel4 ipcomp6 xfrm6_tunnel tunnel6 ipcomp xfrm_ipcomp esp4 ah4 af Jun 9 16:35:58 nyccvz2 kernel: _key arc4 ecb ppp_mppe ppp_deflate zlib_deflate ppp_async ppp_generic slhc crc_ccitt fuse tun xt_MARK xt_mark vzevent kcare(U) autofs4 vznetdev vzmon vzdev ipt_REDIRECT xt_owner nf_nat_ftp nf_conntrack_ftp iptable_nat nf_nat xt_state xt_leng th xt_hl xt_tcpmss xt_TCPMSS xt_multiport xt_limit nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ipt_LOG xt_DSCP xt_dscp ipt_REJECT iptable_mangle xt_set iptable_filter iptable_raw ip_tables ip6table_mangle ip6table_filter ip6_tables ipv6 ip_set_hash_ip ip_set nfnetlink iT CO_wdt iTCO_vendor_support ipmi_devintf ipmi_si ipmi_msghandler e1000e(U) ses enclosure sg igb(U) dca i2c_algo_bit ptp pps_core acpi_pad sb_edac edac_core i2c_i801 i2c_core lpc_ich mfd_core shpchp tcp_htcp ext4 jbd2 mbcache sd_mod crc_t10dif isci libsas scsi_transport_sas megaraid_sas ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Jun 9 16:35:58 nyccvz2 kernel: [5282279.067568] Pid: 0, comm: swapper veid: 0 Tainted: G --------------- T 2.6.32-042stab106.4 #1 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067571] Call Trace: Jun 9 16:35:58 nyccvz2 kernel: [5282279.067573] <IRQ> [<ffffffff8107b7e7>] ? warn_slowpath_common+0x87/0xc0 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067583] [<ffffffff8107b8d6>] ? warn_slowpath_fmt+0x46/0x50 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067587] [<ffffffff8148f9bb>] ? dev_watchdog+0x26b/0x280 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067592] [<ffffffff81015009>] ? sched_clock+0x9/0x10 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067599] [<ffffffff8108f68c>] ? run_timer_softirq+0x1bc/0x380 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067602] [<ffffffff8148f750>] ? dev_watchdog+0x0/0x280 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067606] [<ffffffff81034ddd>] ? lapic_next_event+0x1d/0x30 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067610] [<ffffffff81084c3d>] ? __do_softirq+0x10d/0x250 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067617] [<ffffffff8100c48c>] ? call_softirq+0x1c/0x30 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067619] [<ffffffff810102b5>] ? do_softirq+0x65/0xa0 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067622] [<ffffffff81084a5d>] ? irq_exit+0xcd/0xd0 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067627] [<ffffffff8153f1ba>] ? smp_apic_timer_interrupt+0x4a/0x60 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067630] [<ffffffff8100bc93>] ? apic_timer_interrupt+0x13/0x20 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067631] <EOI> [<ffffffff812f9f7e>] ? intel_idle+0xde/0x170 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067638] [<ffffffff812f9f61>] ? intel_idle+0xc1/0x170 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067645] [<ffffffff81435a17>] ? cpuidle_idle_call+0xa7/0x140 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067649] [<ffffffff8100a026>] ? cpu_idle+0xb6/0x110 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067653] [<ffffffff8152dc74>] ? start_secondary+0x2be/0x301 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067659] ---[ end trace bcae5b3d43e8bb25 ]--- Jun 9 16:35:58 nyccvz2 kernel: [5282279.067660] Tainting kernel with flag 0x9 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067662] Pid: 0, comm: swapper veid: 0 Tainted: G --------------- T 2.6.32-042stab106.4 #1 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067664] Call Trace: Jun 9 16:35:58 nyccvz2 kernel: [5282279.067665] <IRQ> [<ffffffff8107b671>] ? add_taint+0x71/0x80 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067670] [<ffffffff8107b7f4>] ? warn_slowpath_common+0x94/0xc0 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067674] [<ffffffff8107b8d6>] ? warn_slowpath_fmt+0x46/0x50 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067677] [<ffffffff8148f9bb>] ? dev_watchdog+0x26b/0x280 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067681] [<ffffffff81015009>] ? sched_clock+0x9/0x10 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067686] [<ffffffff8108f68c>] ? run_timer_softirq+0x1bc/0x380 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067689] [<ffffffff8148f750>] ? dev_watchdog+0x0/0x280 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067692] [<ffffffff81034ddd>] ? lapic_next_event+0x1d/0x30 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067695] [<ffffffff81084c3d>] ? __do_softirq+0x10d/0x250 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067700] [<ffffffff8100c48c>] ? call_softirq+0x1c/0x30 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067703] [<ffffffff810102b5>] ? do_softirq+0x65/0xa0 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067706] [<ffffffff81084a5d>] ? irq_exit+0xcd/0xd0 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067709] [<ffffffff8153f1ba>] ? smp_apic_timer_interrupt+0x4a/0x60 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067711] [<ffffffff8100bc93>] ? apic_timer_interrupt+0x13/0x20 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067713] <EOI> [<ffffffff812f9f7e>] ? intel_idle+0xde/0x170 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067718] [<ffffffff812f9f61>] ? intel_idle+0xc1/0x170 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067723] [<ffffffff81435a17>] ? cpuidle_idle_call+0xa7/0x140 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067727] [<ffffffff8100a026>] ? cpu_idle+0xb6/0x110 Jun 9 16:35:58 nyccvz2 kernel: [5282279.067729] [<ffffffff8152dc74>] ? start_secondary+0x2be/0x301 Jun 9 16:36:02 nyccvz2 kernel: [5282283.069323] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:36:18 nyccvz2 kernel: [5282299.063252] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:36:32 nyccvz2 kernel: [5282313.108165] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:36:46 nyccvz2 kernel: [5282327.110353] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:37:00 nyccvz2 kernel: [5282341.078752] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:37:14 nyccvz2 kernel: [5282355.078001] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:37:30 nyccvz2 kernel: [5282371.104739] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:37:44 nyccvz2 kernel: [5282385.092118] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:37:58 nyccvz2 kernel: [5282399.139912] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:38:12 nyccvz2 kernel: [5282413.151028] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:38:26 nyccvz2 kernel: [5282427.159257] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:38:42 nyccvz2 kernel: [5282443.162101] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:38:56 nyccvz2 kernel: [5282457.121562] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Jun 9 16:39:12 nyccvz2 kernel: [5282473.164253] igb 0000:03:00.0: eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX On Mon, Jun 8, 2015 at 6:27 PM Brandon Whaley <redkr...@gmail.com<mailto:redkr...@gmail.com>> wrote: Thanks for the input everyone. I went over notes from previous outages of this kind and we have checked that the port was not disabled in the past, but I'll make sure we check again next time to be sure. I'm unable to run ethregs as root due to what I assume is a security related kernel config (mmap - try rebooting with iomem=relaxed: Operation not permitted), but ethtool -d works on the interface in question so I'll gather that information as well when this occurs next time. Thanks again for your time. On Mon, Jun 8, 2015 at 5:56 PM Ronciak, John <john.ronc...@intel.com<mailto:john.ronc...@intel.com>> wrote: We think that maybe you are having link issues since the problem is intermittent and moving from machine to machine. If it more repeatable on a since system we could do more. Please check the link at the switch port to see if it think it has link on that failing system. Thanks. Cheers, John > -----Original Message----- > From: Fujinaka, Todd > [mailto:todd.fujin...@intel.com<mailto:todd.fujin...@intel.com>] > Sent: Monday, June 8, 2015 2:53 PM > To: Fujinaka, Todd; Brandon Whaley; > e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> > Subject: Re: [E1000-devel] igb driver sometimes stops responding after dkms > build > > ethregs would be my choice. ethtool -d also gives some information. > > Todd Fujinaka > Software Application Engineer > Networking Division (ND) > Intel Corporation > todd.fujin...@intel.com<mailto:todd.fujin...@intel.com> > (503) 712-4565 > > -----Original Message----- > From: Fujinaka, Todd > [mailto:todd.fujin...@intel.com<mailto:todd.fujin...@intel.com>] > Sent: Monday, June 08, 2015 2:42 PM > To: Brandon Whaley; > e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> > Subject: Re: [E1000-devel] igb driver sometimes stops responding after dkms > build > > It's difficult to troubleshoot a problem that's erratic and hard to > replicate. A > dump of the registers and status from the link partner would be a good idea. I > wonder if it isn't something triggered on the other side (is it a switch?) > that > sees something odd and disables the port, and a reboot just takes long enough > that the port comes back? > > Todd Fujinaka > Software Application Engineer > Networking Division (ND) > Intel Corporation > todd.fujin...@intel.com<mailto:todd.fujin...@intel.com> > (503) 712-4565 > > -----Original Message----- > From: Brandon Whaley [mailto:redkr...@gmail.com<mailto:redkr...@gmail.com>] > Sent: Monday, June 08, 2015 10:27 AM > To: > e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> > Subject: [E1000-devel] igb driver sometimes stops responding after dkms build > > I use dkms to build the igb driver after new kernel installs on my fleet of > servers using the following commands after every yum update: > > dkms build -m igb -v 5.2.18 > dkms install -m igb -v 5.2.18 > > About once a month, one of my boxes (a different one each time) will stop > responding after this. Nothing I do is able to recover network connectivity > short > of a reboot (not loading/unloading the driver, restarting networking, etc.) > and > since these are production machines, it causes some downtime for us. Below is > what you see in the syslog when the event occurs: > > Jun 8 12:29:19 localhost kernel: [147419.444969] ------------[ cut here > ]------------ > Jun 8 12:29:19 localhost kernel: [147419.444978] WARNING: at > net/sched/sch_generic.c:267 dev_watchdog+0x26b/0x280() (Tainted: G > --------------- T) > Jun 8 12:29:19 localhost kernel: [147419.444981] Hardware name: X9DRL-3F/iF > Jun 8 12:29:19 localhost kernel: [147419.444982] NETDEV WATCHDOG: eth0 > (igb): transmit queue 0 timed out > Jun 8 12:29:19 localhost kernel: [147419.444984] Modules linked in: > mpt3sas mpt2sas raid_class mptctl mptbase ip6t_rt ipt_addrtype xt_policy > aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 > aes_generic cbc kcare(U) vzethdev pio_kaio pio_nfs pio_direct pfmt_raw > pfmt_ploop1 ploop simfs vziolimit vzdquota ip6t_REJECT xfrm6_mode_tunnel > xfrm4_mode_tunnel nf_conntrack_netbios_ns nf_conntrack_broadcast > nf_conntrack_netlink xt_comment nfsd ip6_tunnel ip_vs ipip xt_NFQUEUE > xt_pkttype ecryptfs(T) ip_gre ip_tunnel ipt_MASQUERADE nf_nat_irc xt_helper > nf_conntrack_irc nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6t_LOG > xt_connlimit xt_recent pppoatm atm vzrst vzcpt nfs lockd fscache auth_rpcgss > nfs_acl sunrpc xfrm4_mode_transport xfrm6_mode_transport ccm authenc > esp6 ah6 cnic uio xfrm4_tunnel tunnel4 ipcomp6 xfrm6_tunnel tunnel6 ipcomp > xfrm_ipcomp esp4 ah4 af_key arc4 ecb ppp_mppe ppp_deflate zlib_deflate > ppp_async ppp_generic slhc crc_ccitt fuse tun xt_MARK xt_mark vzevent > autofs4 vznetdev vzmon vzdev ipt Jun 8 12:29:19 localhost kernel: _REDIRECT > xt_owner nf_nat_ftp nf_conntrack_ftp iptable_nat nf_nat xt_state xt_length > xt_hl xt_tcpmss xt_TCPMSS xt_multiport xt_limit nf_conntrack_ipv4 > nf_conntrack > nf_defrag_ipv4 ipt_LOG xt_DSCP xt_dscp ipt_REJECT iptable_mangle xt_set > iptable_filter iptable_raw ip_tables ip6table_mangle ip6table_filter > ip6table_raw ip6_tables ipv6 ip_set_hash_ip ip_set nfnetlink iTCO_wdt > iTCO_vendor_support ipmi_devintf ipmi_si ipmi_msghandler acpi_pad > e1000e(U) ses enclosure sg igb(U) dca i2c_algo_bit ptp pps_core sb_edac > edac_core > i2c_i801 i2c_core lpc_ich mfd_core shpchp tcp_htcp ext4 jbd2 mbcache > sd_mod crc_t10dif isci libsas scsi_transport_sas ahci megaraid_sas wmi > dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] > Jun 8 12:29:19 localhost kernel: [147419.445094] Pid: 0, comm: swapper > veid: 0 Tainted: G --------------- T 2.6.32-042stab108.2 #1 > Jun 8 12:29:19 localhost kernel: [147419.445096] Call Trace: > Jun 8 12:29:19 localhost kernel: [147419.445098] <IRQ> > [<ffffffff8107b827>] ? > warn_slowpath_common+0x87/0xc0 Jun 8 12:29:19 localhost kernel: > [147419.445109] [<ffffffff8107b916>] ? > warn_slowpath_fmt+0x46/0x50 > Jun 8 12:29:19 localhost kernel: [147419.445112] [<ffffffff8148fccb>] ? > dev_watchdog+0x26b/0x280 > Jun 8 12:29:19 localhost kernel: [147419.445117] [<ffffffff81015009>] ? > sched_clock+0x9/0x10 > Jun 8 12:29:19 localhost kernel: [147419.445123] [<ffffffff8108f6cc>] ? > run_timer_softirq+0x1bc/0x380 > Jun 8 12:29:19 localhost kernel: [147419.445126] [<ffffffff8148fa60>] ? > dev_watchdog+0x0/0x280 > Jun 8 12:29:19 localhost kernel: [147419.445130] [<ffffffff81034ddd>] ? > lapic_next_event+0x1d/0x30 > Jun 8 12:29:19 localhost kernel: [147419.445134] [<ffffffff81084c7d>] ? > __do_softirq+0x10d/0x250 > Jun 8 12:29:19 localhost kernel: [147419.445139] [<ffffffff8100c48c>] ? > call_softirq+0x1c/0x30 > Jun 8 12:29:19 localhost kernel: [147419.445142] [<ffffffff810102b5>] ? > do_softirq+0x65/0xa0 > Jun 8 12:29:19 localhost kernel: [147419.445145] [<ffffffff81084a9d>] ? > irq_exit+0xcd/0xd0 > Jun 8 12:29:19 localhost kernel: [147419.445149] [<ffffffff8153f44a>] ? > smp_apic_timer_interrupt+0x4a/0x60 > Jun 8 12:29:19 localhost kernel: [147419.445152] [<ffffffff8100bc93>] ? > apic_timer_interrupt+0x13/0x20 > Jun 8 12:29:19 localhost kernel: [147419.445154] <EOI> > [<ffffffff812fa20e>] ? > intel_idle+0xde/0x170 Jun 8 12:29:19 localhost kernel: [147419.445160] > [<ffffffff812fa1f1>] ? > intel_idle+0xc1/0x170 > Jun 8 12:29:19 localhost kernel: [147419.445166] [<ffffffff81435d27>] ? > cpuidle_idle_call+0xa7/0x140 > Jun 8 12:29:19 localhost kernel: [147419.445170] [<ffffffff8100a026>] ? > cpu_idle+0xb6/0x110 > Jun 8 12:29:19 localhost kernel: [147419.445174] [<ffffffff8152df04>] ? > start_secondary+0x2be/0x301 > Jun 8 12:29:19 localhost kernel: [147419.445179] ---[ end trace > 2179b48f00e92658 ]--- > Jun 8 12:29:19 localhost kernel: [147419.445180] Tainting kernel with flag > 0x9 > Jun 8 12:29:19 localhost kernel: [147419.445182] Pid: 0, comm: swapper > veid: 0 Tainted: G --------------- T 2.6.32-042stab108.2 #1 > Jun 8 12:29:19 localhost kernel: [147419.445184] Call Trace: > Jun 8 12:29:19 localhost kernel: [147419.445185] <IRQ> > [<ffffffff8107b6b1>] ? > add_taint+0x71/0x80 Jun 8 12:29:19 localhost kernel: [147419.445190] > [<ffffffff8107b834>] ? > warn_slowpath_common+0x94/0xc0 > Jun 8 12:29:19 localhost kernel: [147419.445193] [<ffffffff8107b916>] ? > warn_slowpath_fmt+0x46/0x50 > Jun 8 12:29:19 localhost kernel: [147419.445196] [<ffffffff8148fccb>] ? > dev_watchdog+0x26b/0x280 > Jun 8 12:29:19 localhost kernel: [147419.445199] [<ffffffff81015009>] ? > sched_clock+0x9/0x10 > Jun 8 12:29:19 localhost kernel: [147419.445204] [<ffffffff8108f6cc>] ? > run_timer_softirq+0x1bc/0x380 > Jun 8 12:29:19 localhost kernel: [147419.445206] [<ffffffff8148fa60>] ? > dev_watchdog+0x0/0x280 > Jun 8 12:29:19 localhost kernel: [147419.445209] [<ffffffff81034ddd>] ? > lapic_next_event+0x1d/0x30 > Jun 8 12:29:19 localhost kernel: [147419.445213] [<ffffffff81084c7d>] ? > __do_softirq+0x10d/0x250 > Jun 8 12:29:19 localhost kernel: [147419.445217] [<ffffffff8100c48c>] ? > call_softirq+0x1c/0x30 > Jun 8 12:29:19 localhost kernel: [147419.445219] [<ffffffff810102b5>] ? > do_softirq+0x65/0xa0 > Jun 8 12:29:19 localhost kernel: [147419.445222] [<ffffffff81084a9d>] ? > irq_exit+0xcd/0xd0 > Jun 8 12:29:19 localhost kernel: [147419.445225] [<ffffffff8153f44a>] ? > smp_apic_timer_interrupt+0x4a/0x60 > Jun 8 12:29:19 localhost kernel: [147419.445227] [<ffffffff8100bc93>] ? > apic_timer_interrupt+0x13/0x20 > Jun 8 12:29:19 localhost kernel: [147419.445229] <EOI> > [<ffffffff812fa20e>] ? > intel_idle+0xde/0x170 Jun 8 12:29:19 localhost kernel: [147419.445233] > [<ffffffff812fa1f1>] ? > intel_idle+0xc1/0x170 > Jun 8 12:29:19 localhost kernel: [147419.445238] [<ffffffff81435d27>] ? > cpuidle_idle_call+0xa7/0x140 > Jun 8 12:29:19 localhost kernel: [147419.445241] [<ffffffff8100a026>] ? > cpu_idle+0xb6/0x110 > Jun 8 12:29:19 localhost kernel: [147419.445243] [<ffffffff8152df04>] ? > start_secondary+0x2be/0x301 > Jun 8 12:29:27 localhost kernel: [147427.448304] igb 0000:03:00.0: eth0: > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun 8 > 12:29:45 localhost kernel: [147445.478853] igb 0000:03:00.0: eth0: > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun 8 > 12:29:59 localhost kernel: [147459.496009] igb 0000:03:00.0: eth0: > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun 8 > 12:30:13 localhost kernel: [147473.459481] igb 0000:03:00.0: eth0: > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun 8 > 12:30:27 localhost kernel: [147487.460716] igb 0000:03:00.0: eth0: > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun 8 > 12:30:45 localhost kernel: [147505.498234] igb 0000:03:00.0: eth0: > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Jun 8 > 12:31:03 localhost kernel: [147523.513873] igb 0000:03:00.0: eth0: > igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None > > Is there any more information I can provide the next time this happens? If > trends continue I should see it again in 3-5 weeks and will be able to collect > necessary info then. Unfortunately I've yet to find a way to replicate this > on > demand. > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net<mailto:E1000-devel@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net<mailto:E1000-devel@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
