BTW – I believe this is a link to the supported PHY types. Maybe one of these would work for you?
http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005528.html?wapkw=compatible+sfp Thanks, -Don <donald.c.skidm...@intel.com> From: Skidmore, Donald C Sent: Thursday, January 07, 2016 8:33 AM To: 'Peter Volkov' Cc: e1000-devel@lists.sourceforge.net Subject: RE: [E1000-devel] ixgbe: link flapping on two directly connected servers with 82599ES adapter Hey Peter, As you are well award the driver isn’t set up to configure any of those PHY’s and I’ve never seen them before myself. If you want to enable them I would start with the PHY specs and see what they expect as far as initialization. If you don’t have the specs you could see what the PHY is identifying itself as and modify the driver to just treat it as a zero init. I don’t expect anything bad (hurting your HW) would happen but I can’t say for sure either, so proceed at your own risk. Most likely all that would happen is you wouldn’t get link. If you look at the code your notice we have to do something for almost every PHY we do support. If it was me I would start from the PHY specs. Best of luck, -Don <donald.c.skidm...@intel.com<mailto:donald.c.skidm...@intel.com>> From: Peter Volkov [mailto:peter.vol...@gmail.com] Sent: Wednesday, January 06, 2016 6:21 PM To: Skidmore, Donald C Cc: e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> Subject: Re: [E1000-devel] ixgbe: link flapping on two directly connected servers with 82599ES adapter Donald, thank you for answer. On Thu, Jan 7, 2016 at 3:11 AM, Skidmore, Donald C <donald.c.skidm...@intel.com<mailto:donald.c.skidm...@intel.com>> wrote: Sorry our hardware doesn't support LX4 modules, which is what I'm assuming you mean by WDM SFP+. The unsupported module parameter is only used for modules that we recognize but haven't validated. This way we at least know the module won't hurt the hardware even if we can't vouch for their performance. Attempting to use any unknown SFP+ module that was plugged in (assuming zero init) could potently harm the HW. Well, I'm not sure this is LX4 module. According to vendor this module is: - 10GBASE-BX at 10.3125 Gb/s - 10GBASE-BX at 9.953 Gb/s - 1000 Base-LX Ethernet So this is not standard module for sure. Still vendor suggests that this module is supposed work with most hardware and it really works with cisco/juniper hardware. The question is, are there any hints that may help to make module work with intel card? -- Peter. > -----Original Message----- > From: Peter Volkov > [mailto:peter.vol...@gmail.com<mailto:peter.vol...@gmail.com>] > Sent: Tuesday, January 05, 2016 8:54 PM > To: > e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net> > Subject: [E1000-devel] ixgbe: link flapping on two directly connected servers > with 82599ES adapter > > Good time. > > We experince link flapping while connecting two supermicro servers with > 82599ES adapters: > > [Wed Jan 6 07:10:14 2016] ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Down > [Wed Jan 6 07:10:15 2016] ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Up 10 > Gbps, Flow Control: RX/TX [Wed Jan 6 07:10:15 2016] ixgbe 0000:81:00.1 > enp129s0f1: NIC Link is Down [Wed Jan 6 07:10:16 2016] ixgbe 0000:81:00.1 > enp129s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX [Wed Jan 6 > 07:10:16 2016] ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Down [Wed Jan 6 > 07:10:16 2016] ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Up 10 Gbps, Flow > Control: RX/TX [Wed Jan 6 07:10:16 2016] ixgbe 0000:81:00.1 enp129s0f1: > NIC Link is Down [Wed Jan 6 07:10:16 2016] ixgbe 0000:81:00.1 enp129s0f1: > NIC Link is Up 10 Gbps, Flow Control: RX/TX [Wed Jan 6 07:10:17 2016] > ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Down [Wed Jan 6 07:10:17 2016] > ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Up 10 Gbps, Flow Control: RX/TX > > So we are trying to connect two supermicro servers with E10G42BTDA > X520-DA2 controller > in each server: > 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > Network Connection (rev 01) > > In one controller there is following SFP module: > > # ethtool --module-info enp129s0f1 > Identifier : 0x03 (SFP) > Extended identifier : 0x04 (GBIC/SFP defined by > 2-wire interface ID) > Connector : 0x07 (LC) > Transceiver codes : 0x20 0x00 0x00 0x00 0x00 0x00 > 0x00 0x00 > Transceiver type : 10G Ethernet: 10G Base-LR > Encoding : 0x06 (64B/66B) > BR, Nominal : 10300MBd > Rate identifier : 0x00 (unspecified) > Length (SMF,km) : 20km > Length (SMF) : 20000m > Length (50um) : 0m > Length (62.5um) : 0m > Length (Copper) : 0m > Length (OM3) : 0m > Laser wavelength : 1330nm > Vendor name : GIGALINK > Vendor OUI : 00:90:65 > Vendor PN : GL-OT-ST12LC1-13 > Vendor rev : A > Optical diagnostics support : Yes > Laser bias current : 37.020 mA > Laser output power : 0.9654 mW / -0.15 dBm > Receiver signal average optical power : 0.4904 mW / -3.09 dBm > Module temperature : 44.23 degrees C / 111.61 > degrees F > Module voltage : 3.2836 V > Alarm/warning flags implemented : Yes > Laser bias current high alarm : Off > Laser bias current low alarm : Off > Laser bias current high warning : Off > Laser bias current low warning : Off > Laser output power high alarm : Off > Laser output power low alarm : Off > Laser output power high warning : Off > Laser output power low warning : Off > Module temperature high alarm : Off > Module temperature low alarm : Off > Module temperature high warning : Off > Module temperature low warning : Off > Module voltage high alarm : Off > Module voltage low alarm : Off > Module voltage high warning : Off > Module voltage low warning : Off > Laser rx power high alarm : Off > Laser rx power low alarm : Off > Laser rx power high warning : Off > Laser rx power low warning : Off > Laser bias current high alarm threshold : 85.000 mA > Laser bias current low alarm threshold : 10.000 mA > Laser bias current high warning threshold : 80.000 mA Laser bias current low > warning threshold : 12.000 mA > Laser output power high alarm threshold : 3.1623 mW / 5.00 dBm > Laser output power low alarm threshold : 0.3162 mW / -5.00 dBm > Laser output power high warning threshold : 2.5119 mW / 4.00 dBm Laser > output power low warning threshold : 0.3981 mW / -4.00 dBm > Module temperature high alarm threshold : 85.00 degrees C / 185.00 > degrees F > Module temperature low alarm threshold : -10.00 degrees C / 14.00 > degrees F > Module temperature high warning threshold : 80.00 degrees C / 176.00 > degrees F Module temperature low warning threshold : -5.00 degrees C / > 23.00 degrees F > Module voltage high alarm threshold : 3.7000 V > Module voltage low alarm threshold : 2.9000 V > Module voltage high warning threshold : 3.6000 V > Module voltage low warning threshold : 3.0000 V > Laser rx power high alarm threshold : 1.0000 mW / 0.00 dBm > Laser rx power low alarm threshold : 0.0200 mW / -16.99 dBm > Laser rx power high warning threshold : 0.7943 mW / -1.00 dBm > Laser rx power low warning threshold : 0.0251 mW / -16.00 dBm > > In another controller: > > # ethtool --module-info enp129s0f1 > Identifier : 0x03 (SFP) > Extended identifier : 0x04 (GBIC/SFP defined by > 2-wire interface ID) > Connector : 0x07 (LC) > Transceiver codes : 0x20 0x00 0x00 0x00 0x00 0x00 > 0x00 0x00 > Transceiver type : 10G Ethernet: 10G Base-LR > Encoding : 0x06 (64B/66B) > BR, Nominal : 10300MBd > Rate identifier : 0x00 (unspecified) > Length (SMF,km) : 20km > Length (SMF) : 20000m > Length (50um) : 0m > Length (62.5um) : 0m > Length (Copper) : 0m > Length (OM3) : 0m > Laser wavelength : 1270nm > Vendor name : GIGALINK > Vendor OUI : 00:90:65 > Vendor PN : GL-OT-ST12LC1-12 > Vendor rev : A > Option values : 0x00 0x1a > Option : RX_LOS implemented > Option : TX_FAULT implemented > Option : TX_DISABLE implemented > BR margin, max : 0% > BR margin, min : 0% > Vendor SN : G201511300616 > Date code : 151118 > Optical diagnostics support : Yes > Laser bias current : 35.120 mA > Laser output power : 0.7842 mW / -1.06 dBm > Receiver signal average optical power : 0.5695 mW / -2.45 dBm > Module temperature : 45.68 degrees C / 114.22 > degrees F > Module voltage : 3.2440 V > Alarm/warning flags implemented : Yes > Laser bias current high alarm : Off > Laser bias current low alarm : Off > Laser bias current high warning : Off > Laser bias current low warning : Off > Laser output power high alarm : Off > Laser output power low alarm : Off > Laser output power high warning : Off > Laser output power low warning : Off > Module temperature high alarm : Off > Module temperature low alarm : Off > Module temperature high warning : Off > Module temperature low warning : Off > Module voltage high alarm : Off > Module voltage low alarm : Off > Module voltage high warning : Off > Module voltage low warning : Off > Laser rx power high alarm : Off > Laser rx power low alarm : Off > Laser rx power high warning : Off > Laser rx power low warning : Off > Laser bias current high alarm threshold : 85.000 mA > Laser bias current low alarm threshold : 10.000 mA > Laser bias current high warning threshold : 80.000 mA Laser bias current low > warning threshold : 12.000 mA > Laser output power high alarm threshold : 3.1623 mW / 5.00 dBm > Laser output power low alarm threshold : 0.3162 mW / -5.00 dBm > Laser output power high warning threshold : 2.5119 mW / 4.00 dBm Laser > output power low warning threshold : 0.3981 mW / -4.00 dBm > Module temperature high alarm threshold : 85.00 degrees C / 185.00 > degrees F > Module temperature low alarm threshold : -10.00 degrees C / 14.00 > degrees F > Module temperature high warning threshold : 80.00 degrees C / 176.00 > degrees F Module temperature low warning threshold : -5.00 degrees C / > 23.00 degrees F > Module voltage high alarm threshold : 3.7000 V > Module voltage low alarm threshold : 2.9000 V > Module voltage high warning threshold : 3.6000 V > Module voltage low warning threshold : 3.0000 V > Laser rx power high alarm threshold : 1.0000 mW / 0.00 dBm > Laser rx power low alarm threshold : 0.0200 mW / -16.99 dBm > Laser rx power high warning threshold : 0.7943 mW / -1.00 dBm > Laser rx power low warning threshold : 0.0251 mW / -16.00 dBm > > > We've checked both SFP+ modules and patchcord in other hardware and link > was working fine there. As for module version we've tried both 4.1.5 and > 4.3.13 and we experience problem with both versions. I've built v4.3.13 > module with printk enabled, and I found following messages in dmesg: > > [Wed Jan 6 06:04:02 2016] ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Up 10 > Gbps, Flow Control: RX/TX [Wed Jan 6 06:04:02 2016] > ixgbe_get_media_type_82599 [Wed Jan 6 06:04:02 2016] > ixgbe_check_mac_link_generic [Wed Jan 6 06:04:02 2016] > ixgbe_get_media_type_82599 [Wed Jan 6 06:04:02 2016] ixgbe > 0000:81:00.1 enp129s0f1: NIC Link is Down [Wed Jan 6 06:04:02 2016] > ixgbe_check_mac_link_genericixgbe_fc_enable_genericixgbe_fc_autonegixg > be_check_mac_link_generic > [Wed Jan 6 06:04:02 2016] ixgbe 0000:81:00.1 enp129s0f1: NIC Link is Up 10 > Gbps, Flow Control: RX/TX [Wed Jan 6 06:04:02 2016] > ixgbe_get_media_type_82599 [Wed Jan 6 06:04:02 2016] > ixgbe_check_mac_link_generic [Wed Jan 6 06:04:02 2016] > ixgbe_get_media_type_82599 [Wed Jan 6 06:04:02 2016] ixgbe > 0000:81:00.1 enp129s0f1: NIC Link is Down > > Does this helps to understand reasons for such behaviour? > > What makes me wonder. In Documentation/networking/ixgbe.txt I found > following information > > *82599-BASED ADAPTERS* > > *NOTES: If your 82599-based Intel(R) Network Adapter came with Intel > optics, or* *is an Intel(R) Ethernet Server Adapter X520-2, then it only > supports Intel* *optics and/or the direct attach cables listed below.* > > *When 82599-based SFP+ devices are connected back to back, they should > be set to* *the same Speed setting via ethtool. Results may vary if you mix > speed > settings.* > *82598-based adapters support all passive direct attach cables that comply* > *with SFF-8431 v4.1 and SFF-8472 v10.4 specifications. Active direct attach* > *cables are not supported.* > > Yet any attempt to set speed fails: > # ethtool -s enp129s0f1 speed 10000 > Cannot set new settings: Invalid argument > not setting speed > > Why is that? That said I found suggestion to set advertising mode: > > ethtool -s enp129s0f1 advertise 0x1000 > > and this worked, but link flapping continues. > > Yes, I realize that these modules are unsupported and to make them working > I had to load module with allow_unsupported_sfp=1,1 but the main problem > we have is that I failed to find any intel suggested WDM SFP+ module. We > need to use only single fiber cable since in our final configuration we need > to > connect both servers to our provider and they highly recomend WDM > connections. If these modules are not going to work does there exist any > WDM module that will? > > # ethtool -i enp129s0f1 > driver: ixgbe > version: 4.1.5 > firmware-version: 0x61c10001 > bus-info: 0000:81:00.1 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: no > > # ethtool enp129s0f1 > Settings for enp129s0f1: > Supported ports: [ FIBRE ] > Supported link modes: 10000baseT/Full > Supported pause frame use: No > Supports auto-negotiation: No > Advertised link modes: 10000baseT/Full > Advertised pause frame use: Symmetric > Advertised auto-negotiation: No > Speed: 10000Mb/s > Duplex: Full > Port: FIBRE > PHYAD: 0 > Transceiver: external > Auto-negotiation: off > Supports Wake-on: d > Wake-on: d > Current message level: 0x00000007 (7) > drv probe link > Link detected: yes > > We've struggle with this problem for few days already so any suggestions are > more then wellcome. > > > TIA, > -- > Peter.
------------------------------------------------------------------------------
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
