Re: Bell Canada outage?
On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. -- Darius Jahandarie
Re: Bell Canada outage?
On Wed, Aug 8, 2012 at 12:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Outages mailing list is reporting that Tata is having problems in Montreal affecting 'many routers'...maybe this is related? Chris -- Chris Stone AxisInternet, Inc. www.axint.net
Re: Bell Canada outage?
That's what it looked like. We are connected (@ McGill University) to RISQ and a lot of our routes started to go via RISQ-Bell instead of RISQ-CANET/Canarie On Wed, Aug 8, 2012 at 2:35 PM, Chris Stone axi...@gmail.com wrote: On Wed, Aug 8, 2012 at 12:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Outages mailing list is reporting that Tata is having problems in Montreal affecting 'many routers'...maybe this is related? Chris -- Chris Stone AxisInternet, Inc. www.axint.net
Re: Bell Canada outage?
On Wed, Aug 8, 2012 at 2:35 PM, Chris Stone axi...@gmail.com wrote: Outages mailing list is reporting that Tata is having problems in Montreal affecting 'many routers'...maybe this is related? I am a transit customer of both TATA and Bell Canada. We saw route churn and heavy packet loss via both Bell and TATA beginning at approximately 13:25 ET. It took us some time to assess the situation and deactivate both our TATA and Bell BGP sessions. It also took over 10 minutes for my BGP withdraws to propagate from Bell to their neighbors, including Level3. I would guess Bell Canada's routers all have very busy CPU. No information from either NOC yet. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Bell Canada outage?
On 8/8/2012 2:50 PM, Jeff Wheeler wrote: It also took over 10 minutes for my BGP withdraws to propagate from Bell to their neighbors, including Level3. I would guess Bell Canada's routers all have very busy CPU. I saw the same sort of behaviour from TATA. I shut my peer with them, yet TATA was still advertising to Level3 a good 15min after !??! route-views traceroute 199.212.134.1 Type escape sequence to abort. Tracing the route to 199.212.134.1 1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 280 msec 364 msec 280 msec 2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 252 msec 272 msec 272 msec 3 eugn-car1-gw.nero.net (207.98.68.177) [AS 3701] 272 msec 268 msec 244 msec 4 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 260 msec 272 msec 280 msec 5 ge-6-21.car1.Sacramento1.Level3.net (4.53.200.1) [AS 3356] 296 msec 260 msec 312 msec 6 ae-11-11.car2.Sacramento1.Level3.net (4.69.132.150) [AS 3356] [MPLS: Label 83 Exp 0] 304 msec 328 msec 376 msec 7 ae-4-4.ebr2.SanJose1.Level3.net (4.69.132.158) [AS 3356] [MPLS: Label 1123 Exp 0] 328 msec 252 msec 308 msec 8 ae-3-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] [MPLS: Label 1191 Exp 0] 332 msec 240 msec 232 msec 9 ae-1-100.ebr2.Denver1.Level3.net (4.69.151.182) [AS 3356] [MPLS: Label 1189 Exp 0] 252 msec 56 msec 40 msec 10 ae-3-3.ebr1.Chicago2.Level3.net (4.69.132.62) [AS 3356] [MPLS: Label 1186 Exp 0] 72 msec 72 msec 248 msec 11 ae-1-51.edge3.Chicago3.Level3.net (4.69.138.136) [AS 3356] 200 msec 344 msec 200 msec 12 tata-level3.chicago3.level3.net (4.68.110.194) [AS 3356] 204 msec 200 msec 216 msec 13 * * * 14 if-4-2.tcore1.MTT-Montreal.as6453.net (64.86.31.17) [AS 6453] 80 msec if-13-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.32.2) [AS 6453] [MPLS: Label 3208 Exp 0] 84 msec 228 msec 15 * * * 16 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 244 msec 236 msec 17 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 272 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 272 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 260 msec 18 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 208 msec 204 msec 19 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 344 msec 300 msec 20 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 256 msec if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 240 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 268 msec 21 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 248 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 256 msec 22 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec 240 msec 240 msec 23 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 244 msec * * 24 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 284 msec 208 msec 220 msec 25 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 204 msec 256 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 204 msec 26 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 424 msec 208 msec 27 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 220 msec * * 28 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 136 msec 128 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 116 msec 29 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 128 msec 124 msec 116 msec 30 * 152 msec 120 msec route-views -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
Re: Bell Canada outage?
This is what we saw via our upstream provider, it happened twice: [image: Inline image 1] On Wed, Aug 8, 2012 at 2:58 PM, Mike Tancsa m...@sentex.net wrote: On 8/8/2012 2:50 PM, Jeff Wheeler wrote: It also took over 10 minutes for my BGP withdraws to propagate from Bell to their neighbors, including Level3. I would guess Bell Canada's routers all have very busy CPU. I saw the same sort of behaviour from TATA. I shut my peer with them, yet TATA was still advertising to Level3 a good 15min after !??! route-views traceroute 199.212.134.1 Type escape sequence to abort. Tracing the route to 199.212.134.1 1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 280 msec 364 msec 280 msec 2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 252 msec 272 msec 272 msec 3 eugn-car1-gw.nero.net (207.98.68.177) [AS 3701] 272 msec 268 msec 244 msec 4 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 260 msec 272 msec 280 msec 5 ge-6-21.car1.Sacramento1.Level3.net (4.53.200.1) [AS 3356] 296 msec 260 msec 312 msec 6 ae-11-11.car2.Sacramento1.Level3.net (4.69.132.150) [AS 3356] [MPLS: Label 83 Exp 0] 304 msec 328 msec 376 msec 7 ae-4-4.ebr2.SanJose1.Level3.net (4.69.132.158) [AS 3356] [MPLS: Label 1123 Exp 0] 328 msec 252 msec 308 msec 8 ae-3-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] [MPLS: Label 1191 Exp 0] 332 msec 240 msec 232 msec 9 ae-1-100.ebr2.Denver1.Level3.net (4.69.151.182) [AS 3356] [MPLS: Label 1189 Exp 0] 252 msec 56 msec 40 msec 10 ae-3-3.ebr1.Chicago2.Level3.net (4.69.132.62) [AS 3356] [MPLS: Label 1186 Exp 0] 72 msec 72 msec 248 msec 11 ae-1-51.edge3.Chicago3.Level3.net (4.69.138.136) [AS 3356] 200 msec 344 msec 200 msec 12 tata-level3.chicago3.level3.net (4.68.110.194) [AS 3356] 204 msec 200 msec 216 msec 13 * * * 14 if-4-2.tcore1.MTT-Montreal.as6453.net (64.86.31.17) [AS 6453] 80 msec if-13-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.32.2) [AS 6453] [MPLS: Label 3208 Exp 0] 84 msec 228 msec 15 * * * 16 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 244 msec 236 msec 17 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 272 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 272 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 260 msec 18 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 208 msec 204 msec 19 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 260 msec 344 msec 300 msec 20 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 256 msec if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 240 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 268 msec 21 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 248 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 256 msec 22 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 232 msec 240 msec 240 msec 23 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 244 msec * * 24 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 284 msec 208 msec 220 msec 25 if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 204 msec 256 msec if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 204 msec 26 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 224 msec 424 msec 208 msec 27 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 220 msec * * 28 if-5-0-0.mcore3.TTT-Scarborough.as6453.net (64.86.31.22) [AS 6453] [MPLS: Label 1678 Exp 0] 136 msec 128 msec if-1-0-0.core4.TNK-Toronto.as6453.net (63.243.172.1) [AS 6453] 116 msec 29 if-14-0-0.mcore3.TTT-Scarborough.as6453.net (63.243.172.2) [AS 6453] [MPLS: Label 3208 Exp 0] 128 msec 124 msec 116 msec 30 * 152 msec 120 msec route-views -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ image.png
Re: Bell Canada outage?
Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
On Aug 8, 2012, at 3:50 PM, Andree Toonk wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Taking a cursory look at the data processed by my now (very old) leak auditing tool, this should give you an idea of the size of the impact... bgp=# select count(*) from leakinfo where aprox_time::date='2012-08-08'; count --- 21212 bgp=# select count(*) from leakinfo where aprox_time::date='2012-08-07'; count --- 3681 bgp=# select count(*) from leakinfo where aprox_time::date='2012-08-06'; count --- 2428 You can see details of this at the leak info page here: http://puck.nether.net/bgp/leakinfo.cgi - Jared
RE: Bell Canada outage?
Could this be causing issue with XO in east coast ? -Original Message- From: Andree Toonk [mailto:andree+na...@toonk.nl] Sent: Wednesday, August 08, 2012 12:50 PM To: nanog@nanog.org Subject: Re: Bell Canada outage? Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
Thanks for the info, looks like Bell needs to put some filtering on their customer links! Things seem to have calmed down now but I'm still watching my prefixes received from my upstream provider: while true; do snmpget -v2c -c community router 1.3.6.1.4.1.9.9.187.1.2.4.1.1.ip of bgp peer.1.128; sleep 1; done I also monitor all my prefixes and bgp messages in Cacti On Wed, Aug 8, 2012 at 3:50 PM, Andree Toonk andree+na...@toonk.nl wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
On 8 August 2012 16:10, Zachary McGibbon zachary.mcgibbon+na...@gmail.comwrote: Thanks for the info, looks like Bell needs to put some filtering on their customer links! I remember when AS577 had those... ;) -- Harald
Re: Bell Canada outage?
CPU's were pegged for a customer of mine in California. tracked it down to 2 events that went down at that time with a large message volume. 1) Peering between GLBX and Level3 dopped somewhere, causing many prefixes to shift away from L3 paths. 2) Some IPv6 prefixes were aggressively bouncing from HE starting at 11:25. Can't trace it any further upstream for HE unfortunately. This cause lots of CPU churn on one of my customers networks May or may not be related. -- Steve
Re: Bell Canada outage?
Further analysis shows that there were actually 107,409 prefixes affected of 14,391 unique origin ASn's. Interested if your prefixes was affected? I've uploaded a list of prefixes and ASn's that were leaked here: http://www.bgpmon.net/bell-leak.txt Cheers, Andree .-- My secret spy satellite informs me that at 12-08-08 12:50 PM Andree Toonk wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
Ouch!! That's a lot! What do you think the outcome of this will be? What do you think that Bell did/will do (hopefully) to fix this so it doesn't happen again? On Wed, Aug 8, 2012 at 4:31 PM, Andree Toonk andree+na...@toonk.nl wrote: Further analysis shows that there were actually 107,409 prefixes affected of 14,391 unique origin ASn's. Interested if your prefixes was affected? I've uploaded a list of prefixes and ASn's that were leaked here: http://www.bgpmon.net/bell-leak.txt Cheers, Andree .-- My secret spy satellite informs me that at 12-08-08 12:50 PM Andree Toonk wrote: Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Cheers, Andree
Re: Bell Canada outage?
On 8/8/2012 4:14 PM, Steve Dalberg wrote: CPU's were pegged for a customer of mine in California. tracked it down to 2 events that went down at that time with a large message volume. 1) Peering between GLBX and Level3 dopped somewhere, causing many prefixes to shift away from L3 paths. FYI: We saw significant route churn on GBLX (AS3549) as well as on Tata. 2) Some IPv6 prefixes were aggressively bouncing from HE starting at 11:25. Can't trace it any further upstream for HE unfortunately. This cause lots of CPU churn on one of my customers networks May or may not be related.
Re: Bell Canada outage?
We have been advised that TATA/6453 is back to normal, and re-activated our BGP to them. Everything seems okay on this front. No update from Bell Canada yet. On Wed, Aug 8, 2012 at 4:11 PM, Harald Koch c...@pobox.com wrote: On 8 August 2012 16:10, Zachary McGibbon Thanks for the info, looks like Bell needs to put some filtering on their customer links! I remember when AS577 had those... ;) We actually have asked Bell Canada not to filter routes from us, and use prefix-limit only, because they were not able to build a prefix-list for us if we have any downstream customer ASNs, which we do. :-/ If someone at Bell Canada is reading and cared to contact me off-list, I would sure love to get my own route filtering fixed. I have had little success through the normal channels. -- Jeff S Wheeler j...@inconcepts.biz Sr Network Operator / Innovative Network Concepts
Re: Bell Canada outage?
Hi, .-- My secret spy satellite informs me that at 12-08-08 11:35 AM Darius Jahandarie wrote: On Wed, Aug 8, 2012 at 2:31 PM, Zachary McGibbon zachary.mcgibbon+na...@gmail.com wrote: Anyone at Bell Canada / Sympatico can tell us what's going on? Our routing table is going nuts with Bell advertising a lot of routes they shouldn't be Bell leaked a full table. To add to the fun, it seems that TATA took the full table and releaked it. A quick analysis leads met to believe AS46618 ( Dery Telecom Inc) is the cause of this. AS46618 is dual homed to VIDEOTRON and Bell. What seems to have happened is that they leaked routes learned from VIDEOTRON to Bell. Based on BGP data I see that at 17:27 UTC AS46618 ( Dery Telecom Inc) started to leak a 'full table', or at least a significant chunk of it to its provider Bell AS577. Bell propagated that to it's peers. Tata was one of the ones that accepted all of that. I can see that Bell propagated at least 74,109 prefixes learned from AS46618 to Tata. Tata selected 70,160 of those routes. Interesting. I have a server hosted on Bell Canada's network and I saw an outage of about 30 minutes today, but it ONLY affected connections from Verizon's network. This includes my own FIOS connection. I still could connect to the server through Comcast, Level 3 and XO with no problems. Traceroutes from my Verizon IP only got 2 hops, stopping at a philly router and traceroutes back to the same IP from that server got as far as NYC.
