Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
Not to hijack the thread, but I wanted to add -- Just because the fiber jumpers are new, does not mean they are clean. I had a 40 gig link that started taking errors. Moreso when it was under load. I personally cleaned everything. Still had issues. Replaced the optics, no change. New, cleaned jumper. No change. Eventually had our fiber techs look at it. When they scoped the jumpers, they were awful. They cleaned them (one-click, and wipe style cleaner), still bad. After a serious wet clean they finally pronounced them good. Circuit has been fibe ever since. On Mon, Feb 12, 2024 at 3:24 AM Saku Ytti via cisco-nsp < cisco-nsp@puck.nether.net> wrote: > On Mon, 12 Feb 2024 at 09:44, james list wrote: > > > I'd like to test with LACP slow, then can see if physical interface > still flaps... > > I don't think that's good idea, like what would we know? Would we have > to wait 30 times longer, so month-3months, to hit what ever it is, > before we have confidence? > > I would suggest > - turn on debugging, to see cisco emitting LACP PDU, and juniper > receiving LACP PDU > - do packet capture, if at all reasonable, ideally tap, but in > absence of tap mirror > - turn off LACP distributed handling on junos > - ping on the link, ideally 0.2-0.5s interval, to record how ping > stops in relation to first syslog emitted about LACP going down > - wait for 4days > > > -- > ++ytti > ___ > cisco-nsp mailing list cisco-nsp@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp > archive at http://puck.nether.net/pipermail/cisco-nsp/ > ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
On Mon, 12 Feb 2024 at 09:44, james list wrote: > I'd like to test with LACP slow, then can see if physical interface still > flaps... I don't think that's good idea, like what would we know? Would we have to wait 30 times longer, so month-3months, to hit what ever it is, before we have confidence? I would suggest - turn on debugging, to see cisco emitting LACP PDU, and juniper receiving LACP PDU - do packet capture, if at all reasonable, ideally tap, but in absence of tap mirror - turn off LACP distributed handling on junos - ping on the link, ideally 0.2-0.5s interval, to record how ping stops in relation to first syslog emitted about LACP going down - wait for 4days -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
hi I'd like to test with LACP slow, then can see if physical interface still flaps... Thanks for your support Il giorno dom 11 feb 2024 alle ore 18:02 Saku Ytti ha scritto: > On Sun, 11 Feb 2024 at 17:52, james list wrote: > > > - why physical interface flaps in DC1 if it is related to lacp ? > > 16:39:35.813 Juniper reports LACP timeout (so problem started at > 16:39:32, (was traffic passing at 32, 33, 34 seconds?)) > 16:39:36.xxx Cisco reports interface down, long after problem has > already started > > Why Cisco reports physical interface down, I'm not sure. But clearly > the problem was already happening before interface down, and first log > entry is LACP timeout, which occurs 3s after the problem starts. > Perhaps Juniper asserts for some reason RFI? Perhaps Cisco resets the > physical interface once removed from LACP? > > > - why the same setup in DC2 do not report issues ? > > If this is is LACP related software issue, could be difference not > identified. You need to gather more information, like how does ping > look throughout this event, particularly before syslog entries. And if > ping still works up-until syslog, you almost certainly have software > issue with LACP inject at Cisco, or more likely LACP punt at Juniper. > > -- > ++ytti > ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
On Sun, 11 Feb 2024 at 17:52, james list wrote: > - why physical interface flaps in DC1 if it is related to lacp ? 16:39:35.813 Juniper reports LACP timeout (so problem started at 16:39:32, (was traffic passing at 32, 33, 34 seconds?)) 16:39:36.xxx Cisco reports interface down, long after problem has already started Why Cisco reports physical interface down, I'm not sure. But clearly the problem was already happening before interface down, and first log entry is LACP timeout, which occurs 3s after the problem starts. Perhaps Juniper asserts for some reason RFI? Perhaps Cisco resets the physical interface once removed from LACP? > - why the same setup in DC2 do not report issues ? If this is is LACP related software issue, could be difference not identified. You need to gather more information, like how does ping look throughout this event, particularly before syslog entries. And if ping still works up-until syslog, you almost certainly have software issue with LACP inject at Cisco, or more likely LACP punt at Juniper. -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
Hi I have a couple of points to ask related to your idea: - why physical interface flaps in DC1 if it is related to lacp ? - why the same setup in DC2 do not report issues ? NEXUS01# sh logging | in Initia | last 15 2024 Jan 17 22:37:49 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 18 23:54:25 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 19 00:58:13 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 19 07:15:04 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 22 16:03:13 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 25 21:32:29 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 26 18:41:12 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 28 05:07:20 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 29 04:06:52 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Jan 30 03:09:44 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Feb 5 18:13:20 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Feb 6 02:17:25 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Feb 6 22:00:24 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Feb 9 09:29:36 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) 2024 Feb 9 16:39:36 NEXUS01 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/44 is down (Initializing) Il giorno dom 11 feb 2024 alle ore 14:36 Saku Ytti ha scritto: > On Sun, 11 Feb 2024 at 15:24, james list wrote: > > > While on Juniper when the issue happens I always see: > > > > show log messages | last 440 | match LACPD_TIMEOUT > > Jan 25 21:32:27.948 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: > lacp current while timer expired current Receive State: CURRENT > > > Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: > lacp current while timer expired current Receive State: CURRENT > > Ok so problem always starts by Juniper seeing 3seconds without LACP > PDU, i.e. missing 3 consecutive LACP PDU. It would be good to ping > while this problem is happening, to see if ping stops at 3s before the > syslog lines, or at the same time as syslog lines. > If ping stops 3s before, it's link problem from cisco to juniper. > If ping stops at syslog time (my guess), it's software problem. > > There is unfortunately log of bug surface here, both on inject and on > punt path. You could be hitting PR1541056 on the Juniper end. You > could test for this by removing distributed LACP handling with 'set > routing-options ppm no-delegate-processing' > You could also do packet capture for LACP on both ends, to try to see > if LACP was sent by Cisco and received by capture, but not by system. > > > -- > ++ytti > ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
On Sun, 11 Feb 2024 at 15:24, james list wrote: > While on Juniper when the issue happens I always see: > > show log messages | last 440 | match LACPD_TIMEOUT > Jan 25 21:32:27.948 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp > current while timer expired current Receive State: CURRENT > Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp > current while timer expired current Receive State: CURRENT Ok so problem always starts by Juniper seeing 3seconds without LACP PDU, i.e. missing 3 consecutive LACP PDU. It would be good to ping while this problem is happening, to see if ping stops at 3s before the syslog lines, or at the same time as syslog lines. If ping stops 3s before, it's link problem from cisco to juniper. If ping stops at syslog time (my guess), it's software problem. There is unfortunately log of bug surface here, both on inject and on punt path. You could be hitting PR1541056 on the Juniper end. You could test for this by removing distributed LACP handling with 'set routing-options ppm no-delegate-processing' You could also do packet capture for LACP on both ends, to try to see if LACP was sent by Cisco and received by capture, but not by system. -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
On Cisco I see physical goes down (initializing), what does that mean? While on Juniper when the issue happens I always see: show log messages | last 440 | match LACPD_TIMEOUT Jan 25 21:32:27.948 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Jan 26 18:41:12.514 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Jan 28 05:07:20.283 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Jan 29 04:06:51.768 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Jan 30 03:09:43.923 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Feb 5 18:13:20.158 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Feb 6 02:17:23.703 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Feb 6 22:00:23.758 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Feb 9 09:29:35.728 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Il giorno dom 11 feb 2024 alle ore 14:10 Saku Ytti ha scritto: > Hey James, > > You shared this off-list, I think it's sufficiently material to share. > > 2024 Feb 9 16:39:36 NEXUS1 > %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface > port-channel101 is down (No operational members) > 2024 Feb 9 16:39:36 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN: > port-channel101: Ethernet1/44 is down > Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: > lacp current while timer expired current Receive State: CURRENT > Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACP_INTF_DOWN: ae49: > Interface marked down due to lacp timeout on member et-0/1/5 > > We can't know the order of events here, due to no subsecond precision > enabled on Cisco end. > > But if failure would start from interface down, it would take 3seconds > for Juniper to realise LACP failure. However we can see that it > happens in less than 1s, so we can determine the interface was not > down first, the first problem was Juniper not receiving 3 consecutive > LACP PDUs, 1s apart, prior to noticing any type of interface state > related problems. > > Is this always the order of events? Does it always happen with Juniper > noticing problems receiving LACP PDU first? > > > On Sun, 11 Feb 2024 at 14:55, james list via juniper-nsp > wrote: > > > > Hi > > > > 1) cable has been replaced with a brand new one, they said that to check > an > > MPO 100 Gbs cable is not that easy > > > > 3) no errors reported on both side > > > > 2) here the output of cisco and juniper > > > > NEXUS1# sh interface eth1/44 transceiver details > > Ethernet1/44 > > transceiver is present > > type is QSFP-100G-SR4 > > name is CISCO-INNOLIGHT > > part number is TR-FC85S-NC3 > > revision is 2C > > serial number is INL27050TVT > > nominal bitrate is 25500 MBit/sec > > Link length supported for 50/125um OM3 fiber is 70 m > > cisco id is 17 > > cisco extended id number is 220 > > cisco part number is 10-3142-03 > > cisco product id is QSFP-100G-SR4-S > > cisco version id is V03 > > > > Lane Number:1 Network Lane > >SFP Detail Diagnostics Information (internal calibration) > > > > > > > Current Alarms Warnings > > Measurement HighLow High Low > > > > > > > Temperature 30.51 C75.00 C -5.00 C 70.00 C > 0.00 C > > Voltage3.28 V 3.63 V 2.97 V 3.46 V > 3.13 V > > Current6.40 mA 12.45 mA 3.25 mA12.45 mA > 3.25 > > mA > > Tx Power 0.98 dBm 5.39 dBm -12.44 dBm2.39 dBm > -8.41 > > dBm > > Rx Power -1.60 dBm 5.39 dBm -14.31 dBm2.39 dBm > -10.31 > > dBm > > Transmit Fault Count = 0 > > > > > > > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning > > > > Lane Number:2 Network Lane > >SFP Detail Diagnostics Information (internal calibration) > > > > > > > Current Alarms Warnings > > Measurement HighLow High Low > > > > >
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
Hey James, You shared this off-list, I think it's sufficiently material to share. 2024 Feb 9 16:39:36 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (No operational members) 2024 Feb 9 16:39:36 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel101: Ethernet1/44 is down Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACPD_TIMEOUT: et-0/1/5: lacp current while timer expired current Receive State: CURRENT Feb 9 16:39:35.813 2024 MX1 lacpd[31632]: LACP_INTF_DOWN: ae49: Interface marked down due to lacp timeout on member et-0/1/5 We can't know the order of events here, due to no subsecond precision enabled on Cisco end. But if failure would start from interface down, it would take 3seconds for Juniper to realise LACP failure. However we can see that it happens in less than 1s, so we can determine the interface was not down first, the first problem was Juniper not receiving 3 consecutive LACP PDUs, 1s apart, prior to noticing any type of interface state related problems. Is this always the order of events? Does it always happen with Juniper noticing problems receiving LACP PDU first? On Sun, 11 Feb 2024 at 14:55, james list via juniper-nsp wrote: > > Hi > > 1) cable has been replaced with a brand new one, they said that to check an > MPO 100 Gbs cable is not that easy > > 3) no errors reported on both side > > 2) here the output of cisco and juniper > > NEXUS1# sh interface eth1/44 transceiver details > Ethernet1/44 > transceiver is present > type is QSFP-100G-SR4 > name is CISCO-INNOLIGHT > part number is TR-FC85S-NC3 > revision is 2C > serial number is INL27050TVT > nominal bitrate is 25500 MBit/sec > Link length supported for 50/125um OM3 fiber is 70 m > cisco id is 17 > cisco extended id number is 220 > cisco part number is 10-3142-03 > cisco product id is QSFP-100G-SR4-S > cisco version id is V03 > > Lane Number:1 Network Lane >SFP Detail Diagnostics Information (internal calibration) > > > Current Alarms Warnings > Measurement HighLow High Low > > > Temperature 30.51 C75.00 C -5.00 C 70.00 C0.00 C > Voltage3.28 V 3.63 V 2.97 V 3.46 V3.13 V > Current6.40 mA 12.45 mA 3.25 mA12.45 mA 3.25 > mA > Tx Power 0.98 dBm 5.39 dBm -12.44 dBm2.39 dBm -8.41 > dBm > Rx Power -1.60 dBm 5.39 dBm -14.31 dBm2.39 dBm-10.31 > dBm > Transmit Fault Count = 0 > > > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning > > Lane Number:2 Network Lane >SFP Detail Diagnostics Information (internal calibration) > > > Current Alarms Warnings > Measurement HighLow High Low > > > Temperature 30.51 C75.00 C -5.00 C 70.00 C0.00 C > Voltage3.28 V 3.63 V 2.97 V 3.46 V3.13 V > Current6.40 mA 12.45 mA 3.25 mA12.45 mA 3.25 > mA > Tx Power 0.62 dBm 5.39 dBm -12.44 dBm2.39 dBm -8.41 > dBm > Rx Power -1.18 dBm 5.39 dBm -14.31 dBm2.39 dBm-10.31 > dBm > Transmit Fault Count = 0 > > > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning > > Lane Number:3 Network Lane >SFP Detail Diagnostics Information (internal calibration) > > > Current Alarms Warnings > Measurement HighLow High Low > > > Temperature 30.51 C75.00 C -5.00 C 70.00 C0.00 C > Voltage3.28 V 3.63 V 2.97 V 3.46 V3.13 V > Current6.40 mA 12.45 mA 3.25 mA12.45 mA 3.25 > mA > Tx Power 0.87 dBm 5.39 dBm -12.44 dBm2.39 dBm -8.41 > dBm > Rx Power 0.01 dBm 5.39 dBm -14.31 dBm2.39 dBm-10.31 > dBm > Transmit Fault Count = 0 > > > Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning > > Lane Number:4 Network Lane >SFP Detail Diagnostics
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
Hi 1) cable has been replaced with a brand new one, they said that to check an MPO 100 Gbs cable is not that easy 3) no errors reported on both side 2) here the output of cisco and juniper NEXUS1# sh interface eth1/44 transceiver details Ethernet1/44 transceiver is present type is QSFP-100G-SR4 name is CISCO-INNOLIGHT part number is TR-FC85S-NC3 revision is 2C serial number is INL27050TVT nominal bitrate is 25500 MBit/sec Link length supported for 50/125um OM3 fiber is 70 m cisco id is 17 cisco extended id number is 220 cisco part number is 10-3142-03 cisco product id is QSFP-100G-SR4-S cisco version id is V03 Lane Number:1 Network Lane SFP Detail Diagnostics Information (internal calibration) Current Alarms Warnings Measurement HighLow High Low Temperature 30.51 C75.00 C -5.00 C 70.00 C0.00 C Voltage3.28 V 3.63 V 2.97 V 3.46 V3.13 V Current6.40 mA 12.45 mA 3.25 mA12.45 mA 3.25 mA Tx Power 0.98 dBm 5.39 dBm -12.44 dBm2.39 dBm -8.41 dBm Rx Power -1.60 dBm 5.39 dBm -14.31 dBm2.39 dBm-10.31 dBm Transmit Fault Count = 0 Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning Lane Number:2 Network Lane SFP Detail Diagnostics Information (internal calibration) Current Alarms Warnings Measurement HighLow High Low Temperature 30.51 C75.00 C -5.00 C 70.00 C0.00 C Voltage3.28 V 3.63 V 2.97 V 3.46 V3.13 V Current6.40 mA 12.45 mA 3.25 mA12.45 mA 3.25 mA Tx Power 0.62 dBm 5.39 dBm -12.44 dBm2.39 dBm -8.41 dBm Rx Power -1.18 dBm 5.39 dBm -14.31 dBm2.39 dBm-10.31 dBm Transmit Fault Count = 0 Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning Lane Number:3 Network Lane SFP Detail Diagnostics Information (internal calibration) Current Alarms Warnings Measurement HighLow High Low Temperature 30.51 C75.00 C -5.00 C 70.00 C0.00 C Voltage3.28 V 3.63 V 2.97 V 3.46 V3.13 V Current6.40 mA 12.45 mA 3.25 mA12.45 mA 3.25 mA Tx Power 0.87 dBm 5.39 dBm -12.44 dBm2.39 dBm -8.41 dBm Rx Power 0.01 dBm 5.39 dBm -14.31 dBm2.39 dBm-10.31 dBm Transmit Fault Count = 0 Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning Lane Number:4 Network Lane SFP Detail Diagnostics Information (internal calibration) Current Alarms Warnings Measurement HighLow High Low Temperature 30.51 C75.00 C -5.00 C 70.00 C0.00 C Voltage3.28 V 3.63 V 2.97 V 3.46 V3.13 V Current6.40 mA 12.45 mA 3.25 mA12.45 mA 3.25 mA Tx Power 0.67 dBm 5.39 dBm -12.44 dBm2.39 dBm -8.41 dBm Rx Power 0.11 dBm 5.39 dBm -14.31 dBm2.39 dBm-10.31 dBm Transmit Fault Count = 0 Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning MX1> show interfaces diagnostics optics et-1/0/5 Physical interface: et-1/0/5 Module temperature: 38 degrees C / 100 degrees F Module voltage: 3.2740 V Module temperature high alarm : Off Module temperature low alarm : Off Module temperature high warning : Off Module temperature low warning: Off Module voltage high alarm
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
I want to clarify, I meant this in the context of the original question.
That is, if you have a BGP specific problem, and no FCS errors, then
you can't have link problems.
But in this case, the problem is not BGP specific, in fact it has
nothing to do with BGP, since the problem begins on observing link
flap.
On Sun, 11 Feb 2024 at 14:14, Saku Ytti wrote:
>
> I don't think any of these matter. You'd see FCS failure on any
> link-related issue causing the BGP packet to drop.
>
> If you're not seeing FCS failures, you can ignore all link related
> problems in this case.
>
>
> On Sun, 11 Feb 2024 at 14:13, Havard Eidnes via juniper-nsp
> wrote:
> >
> > > DC technicians states cable are the same in both DCs and
> > > direct, no patch panel
> >
> > Things I would look at:
> >
> > * Has all the connectors been verified clean via microscope?
> >
> > * Optical levels relative to threshold values (may relate to the
> >first).
> >
> > * Any end seeing any input errors? (May relate to the above
> >two.) On the Juniper you can see some of this via PCS
> >("Physical Coding Sublayer") unexpected events independently
> >of whether you have payload traffic, not sure you can do the
> >same on the Nexus boxes.
> >
> > Regards,
> >
> > - Håvard
> > ___
> > juniper-nsp mailing list juniper-...@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/juniper-nsp
>
>
>
> --
> ++ytti
--
++ytti
___
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
I don't think any of these matter. You'd see FCS failure on any
link-related issue causing the BGP packet to drop.
If you're not seeing FCS failures, you can ignore all link related
problems in this case.
On Sun, 11 Feb 2024 at 14:13, Havard Eidnes via juniper-nsp
wrote:
>
> > DC technicians states cable are the same in both DCs and
> > direct, no patch panel
>
> Things I would look at:
>
> * Has all the connectors been verified clean via microscope?
>
> * Optical levels relative to threshold values (may relate to the
>first).
>
> * Any end seeing any input errors? (May relate to the above
>two.) On the Juniper you can see some of this via PCS
>("Physical Coding Sublayer") unexpected events independently
>of whether you have payload traffic, not sure you can do the
>same on the Nexus boxes.
>
> Regards,
>
> - Håvard
> ___
> juniper-nsp mailing list juniper-...@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
--
++ytti
___
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
> DC technicians states cable are the same in both DCs and
> direct, no patch panel
Things I would look at:
* Has all the connectors been verified clean via microscope?
* Optical levels relative to threshold values (may relate to the
first).
* Any end seeing any input errors? (May relate to the above
two.) On the Juniper you can see some of this via PCS
("Physical Coding Sublayer") unexpected events independently
of whether you have payload traffic, not sure you can do the
same on the Nexus boxes.
Regards,
- Håvard
___
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
On Sun, 11 Feb 2024 at 13:51, james list via juniper-nsp wrote: > One think I've omit to say is that BGP is over a LACP with currently just > one interface 100 Gbs. > > I see that the issue is triggered on Cisco when eth interface seems to go > in Initializing state: Ok, so we can forget BGP entirely. And focus on why the LACP is going down. Is the LACP single port, eth1/44? When the LACP fails, does Juniper end emit any syslog? Does Juniper see the interface facing eth1/44 flapping? -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] [j-nsp] Stange issue on 100 Gbs interconnection Juniper - Cisco
Open JTAC and CTAC cases. The amount of information provided is wildly insufficient. 'BGP flaps' what does that mean, is it always the same direction? If so, which direction thinks it's not seeing keepalives? Do you also observe loss in 'ping' between the links during the period? Purely stabbing in the dark, I'd say you always observe it in a single direction, because in that direction you are losing reliably every nTh keepalive, and statistically it takes 1-3 days to lose 3 in a row, with the probability you're seeing. Now why exactly is this, is one end not sending to wire or is one end not receiving from wire. Again stabbing in the dark, more likely that problem is in the punt path, rather than inject path, so I would focus my investigation on the party who is tearing down the session, due to lack of keepalive, on thesis this device has problem in punt path and is for some reason dropping at reliable probability BGP packets from the wire. On Sun, 11 Feb 2024 at 12:09, james list via juniper-nsp wrote: > > Dear experts > we have a couple of BGP peers over a 100 Gbs interconnection between > Juniper (MX10003) and Cisco (Nexus N9K-C9364C) in two different datacenters > like this: > > DC1 > MX1 -- bgp -- NEXUS1 > MX2 -- bgp -- NEXUS2 > > DC2 > MX3 -- bgp -- NEXUS3 > MX4 -- bgp -- NEXUS4 > > The issue we see is that sporadically (ie every 1 to 3 days) we notice BGP > flaps only in DC1 on both interconnections (not at the same time), there is > still no traffic since once noticed the flaps we have blocked deploy on > production. > > We've already changed SPF (we moved the ones from DC2 to DC1 and viceversa) > and cables on both the interconnetion at DC1 without any solution. > > SFP we use in both DCs: > > Juniper - QSFP-100G-SR4-T2 > Cisco - QSFP-100G-SR4 > > over MPO cable OM4. > > Distance is DC1 70 mt and DC2 80 mt, hence is less where we see the issue. > > Any idea or suggestion what to check or to do ? > > Thanks in advance > Cheers > James > ___ > juniper-nsp mailing list juniper-...@puck.nether.net > https://puck.nether.net/mailman/listinfo/juniper-nsp -- ++ytti ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
