Re: AS4788 Telecom Malaysia major route leak?
From: Rafael Possamai raf...@gav.ufsc.br ... I assume they would have to, granted the issue lasted for a couple hours. Now, it depends on how they define the outage A L3 outage is something you manage to open a ticket for, if you don't then it didn't happen (been there before, one of their DC lost power, transit down for around 7h, nothing) With the world trying to call them on this you can't get through so no SLA payout for you brandon
RE: [SPAM]Re: AS4788 Telecom Malaysia major route leak?
Absolutely on point. Let's solve the problem, not the blame. ERM Evan R Moore Network Engineer and Bitwrangler Sovernet Communications -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Randy Bush Sent: Sunday, June 14, 2015 9:02 PM To: North American Network Operators' Group Subject: [SPAM]Re: AS4788 Telecom Malaysia major route leak? what i have yet to understand (probably my fault) is how L(3) propagated the disease or, more correctly, what has happened over there that they did not stop the propagation? the crew that went there from mci ran a very tight ship and L(3) has always had pretty rigid filters. what happened? and i mean that in the sense of how can i not make a similar mistake? randy
Re: AS4788 Telecom Malaysia major route leak?
On 15/Jun/15 03:01, Randy Bush wrote: what i have yet to understand (probably my fault) is how L(3) propagated the disease or, more correctly, what has happened over there that they did not stop the propagation? the crew that went there from mci ran a very tight ship and L(3) has always had pretty rigid filters. what happened? and i mean that in the sense of how can i not make a similar mistake? Given that TM were leaking into 3549, one may infer that Level(3)'s tight screws have not yet completely filtered down to the GBLX network of old. Conjecture on my part... It is no secret that Level(3)'s IRR client is broken, and as others have mentioned before, it's reasonably common to give them a call and get the spanner rammed over its head for the thing to work. Whether they are using the same for the GBLX network, or if the GBLX network is the nasty cousin we don't care about until he leaves the house, is an exercise left to all of us. Mark.
Re: AS4788 Telecom Malaysia major route leak?
On Fri, Jun 12, 2015 at 08:25:40PM +, Jürgen Jaritsch wrote: This is the official [level3] feedback: [ ... ] For completeness sake: here is what Telekom Malaysia published about the issue: Telekom Malaysia Berhad (TM) wishes to update on the service related issue detected yesterday, 12 June 2015 affecting a number of our Internet services customers that caused a deterioration in connection performance. We identified the root cause and our network team immediately took steps to optimise traffic flows, while we worked to restore connectivity to its expected level of performance. The services were restored at 6.30pm on the same day. We would like to clarify that during a network reconfiguration exercise, we had unintentionally updated traffic routing information which caused congestion and packet loss to our international connectivity. This had affected the internet traffic flow for some of our customers and some international traffic routes. We apologise for any inconvenience caused by the service disruption and would like to assure customers that we are undertaking all the necessary measures to ensure customers continue to experience uninterrupted services. Meanwhile, customers who have any enquiry or require further assistance can email us at h...@tm.com.my or tweet to us via @tmconnects on Twitter. source: https://www.tm.com.my/OnlineHelp/Announcement/Pages/INTERNET-SERVICES-DISRUPTION-12-June-2015.aspx Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
Raymond, They provided a simple sorry: We apologise for any inconvenience caused by the service disruption. It doesn't get much more simple than that. -mel beckman On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 20:27 heeft Job Snijders j...@instituut.net het volgende geschreven: On Fri, Jun 12, 2015 at 08:25:40PM +, Jürgen Jaritsch wrote: This is the official [level3] feedback: [ ... ] For completeness sake: here is what Telekom Malaysia published about the issue: Telekom Malaysia Berhad (TM) wishes to update on the service related issue detected yesterday, 12 June 2015 affecting a number of our Internet services customers that caused a deterioration in connection performance. We identified the root cause and our network team immediately took steps to optimise traffic flows, while we worked to restore connectivity to its expected level of performance. The services were restored at 6.30pm on the same day. We would like to clarify that during a network reconfiguration exercise, we had unintentionally updated traffic routing information which caused congestion and packet loss to our international connectivity. This had affected the internet traffic flow for some of our customers and some international traffic routes. We apologise for any inconvenience caused by the service disruption and would like to assure customers that we are undertaking all the necessary measures to ensure customers continue to experience uninterrupted services. Meanwhile, customers who have any enquiry or require further assistance can email us at h...@tm.com.my or tweet to us via @tmconnects on Twitter. source: https://www.tm.com.my/OnlineHelp/Announcement/Pages/INTERNET-SERVICES-DISRUPTION-12-June-2015.aspx Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
Hello Mel, Must just be me then. I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:27 heeft Mel Beckman m...@beckman.org het volgende geschreven: Raymond, They provided a simple sorry: We apologise for any inconvenience caused by the service disruption. It doesn't get much more simple than that. -mel beckman On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
RE: AS4788 Telecom Malaysia major route leak?
They should verify the GBLX customer ports as well ... Jürgen Jaritsch Head of Network Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: j...@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Re: AS4788 Telecom Malaysia major route leak?
SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a best effort network, with zero guarantees. -mel beckman On Jun 14, 2015, at 4:06 PM, Rafael Possamai raf...@gav.ufsc.brmailto:raf...@gav.ufsc.br wrote: Does anyone know if there's an official ruling as to who gets to pay for the SLA breaches? On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman m...@beckman.orgmailto:m...@beckman.org wrote: Raymond, But you said A simple 'sorry' would have done. Now you're asking for lots more detail. Why the change? -mel beckman On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn raym...@prolocation.netmailto:raym...@prolocation.net wrote: Hello Mel, Must just be me then. I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:27 heeft Mel Beckman m...@beckman.orgmailto:m...@beckman.org het volgende geschreven: Raymond, They provided a simple sorry: We apologise for any inconvenience caused by the service disruption. It doesn't get much more simple than that. -mel beckman On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn raym...@prolocation.netmailto:raym...@prolocation.net wrote: Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mumailto:mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
Does anyone know if there's an official ruling as to who gets to pay for the SLA breaches? On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman m...@beckman.org wrote: Raymond, But you said A simple 'sorry' would have done. Now you're asking for lots more detail. Why the change? -mel beckman On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hello Mel, Must just be me then. I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:27 heeft Mel Beckman m...@beckman.org het volgende geschreven: Raymond, They provided a simple sorry: We apologise for any inconvenience caused by the service disruption. It doesn't get much more simple than that. -mel beckman On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
Well, I was wondering the same. I am guessing it depends on the SLA contract since they are all very unique and specific. I assume they would have to, granted the issue lasted for a couple hours. Now, it depends on how they define the outage. A fiber cut that yields a customer's service unusable would be an easy SLA breach. Their legal team most likely removed any liability due to someone else's negligence, although you could argue they were negligent as well. So in this case they can claim the whole best effort thing and get away with it. I am not a L3 customer, so was just wondering out of curiosity. On Sun, Jun 14, 2015 at 8:07 PM, Aftab Siddiqui aftab.siddi...@gmail.com wrote: Hi Rafael, I get that much, just wondering if Level3 would have to pay an SLA breach to its customers given the mess started with TM (even though it could have been avoided). And I am guessing if they do, they wouldn't be able to recover anything from TM. I doubt if L3 has to pay anything to its customers in terms of SLA breach, its best effort. Are you aware of any such agreement which suggest otherwise? that would be interesting.
Re: AS4788 Telecom Malaysia major route leak?
I get that much, just wondering if Level3 would have to pay an SLA breach to its customers given the mess started with TM (even though it could have been avoided). And I am guessing if they do, they wouldn't be able to recover anything from TM. On Sun, Jun 14, 2015 at 7:07 PM, Mel Beckman m...@beckman.org wrote: SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a best effort network, with zero guarantees. -mel beckman On Jun 14, 2015, at 4:06 PM, Rafael Possamai raf...@gav.ufsc.br wrote: Does anyone know if there's an official ruling as to who gets to pay for the SLA breaches? On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman m...@beckman.org wrote: Raymond, But you said A simple 'sorry' would have done. Now you're asking for lots more detail. Why the change? -mel beckman On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hello Mel, Must just be me then. I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:27 heeft Mel Beckman m...@beckman.org het volgende geschreven: Raymond, They provided a simple sorry: We apologise for any inconvenience caused by the service disruption. It doesn't get much more simple than that. -mel beckman On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
what i have yet to understand (probably my fault) is how L(3) propagated the disease or, more correctly, what has happened over there that they did not stop the propagation? the crew that went there from mci ran a very tight ship and L(3) has always had pretty rigid filters. what happened? and i mean that in the sense of how can i not make a similar mistake? randy
Re: AS4788 Telecom Malaysia major route leak?
Raymond, But you said A simple 'sorry' would have done. Now you're asking for lots more detail. Why the change? -mel beckman On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hello Mel, Must just be me then. I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:27 heeft Mel Beckman m...@beckman.org het volgende geschreven: Raymond, They provided a simple sorry: We apologise for any inconvenience caused by the service disruption. It doesn't get much more simple than that. -mel beckman On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn raym...@prolocation.net wrote: Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
On Sun, Jun 14, 2015 at 9:07 PM, Mel Beckman m...@beckman.org wrote: SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a best effort network, with zero guarantees. -mel beckman Ok, I'll bite: my $dayjob is a Level 3 client that was directly affected by lack of availability due to recovery attempt Level 3 tried in our region. Where $dayjob can collect $ for this incident ? Rubens
Re: AS4788 Telecom Malaysia major route leak?
keep in mind their target audience with that message is probably local malaysian customers, not the world. On Sun, Jun 14, 2015 at 5:09 PM Mel Beckman m...@beckman.org wrote: SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a best effort network, with zero guarantees. -mel beckman On Jun 14, 2015, at 4:06 PM, Rafael Possamai raf...@gav.ufsc.brmailto: raf...@gav.ufsc.br wrote: Does anyone know if there's an official ruling as to who gets to pay for the SLA breaches? On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman m...@beckman.orgmailto: m...@beckman.org wrote: Raymond, But you said A simple 'sorry' would have done. Now you're asking for lots more detail. Why the change? -mel beckman On Jun 14, 2015, at 2:32 PM, Raymond Dijkxhoorn raym...@prolocation.net mailto:raym...@prolocation.net wrote: Hello Mel, Must just be me then. I was most likely expecting a more in depth report. Strange things happened. Perhaps they could post a 'what exactly happened' since this wasnt a average route leak. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:27 heeft Mel Beckman m...@beckman.orgmailto: m...@beckman.org het volgende geschreven: Raymond, They provided a simple sorry: We apologise for any inconvenience caused by the service disruption. It doesn't get much more simple than that. -mel beckman On Jun 14, 2015, at 2:21 PM, Raymond Dijkxhoorn raym...@prolocation.netmailto:raym...@prolocation.net wrote: Hai! Mark, mistakes and oopses happen. No problem at all. I understand that completely. There is human faillure and this happenes. A simple 'sorry' would have done. Yet their whole message tells 'they did ok' In my very limited view they did NOT ok. Did i misread? I am also very much looking how level3 is going to prevent things like this. But out of own experience they will not. We have seen before that they implemented filtering based on customer lists. But not a per customer filter. They did this globally. So any l3 customer can announce routes of another l3 customer. While this can be changed this outage tells there is certainly room for improvements. I hope people will learn from what happened and implement proper filtering. Thats even more important then a message from a operator that didnt even understand fully what they caused to the internet globally. Thanks, Raymond Dijkxhoorn Op 14 jun. 2015 om 23:04 heeft Mark Tinka mark.ti...@seacom.mu mailto:mark.ti...@seacom.mu het volgende geschreven: On 14/Jun/15 22:55, Raymond Dijkxhoorn wrote: Hai! Wouw! This is what they came up with?! Hopefully Level3 will take appropriate measures. Its amazing. Really. 'Some internationally routes' Have they any idea what they did at all? Its amazing that with parties like that the internet still works as is tm ... I wouldn't be as hard. Stuff happens - and as they said, during a maintenance activity, they boo-boo'ed. Are Level(3) going to own up and say they should have had filters in place? I certainly hope they do. But more importantly, are Level(3) going to implement the filters against TM's circuit? Are they going to run around the network looking for any additional customer circuits that need plugging? That's my concern... Mark.
Re: AS4788 Telecom Malaysia major route leak?
In addition to that, losing face in SE Asia is not done. On Mon, Jun 15, 2015 at 12:14:43AM +, ryanL wrote: keep in mind their target audience with that message is probably local malaysian customers, not the world. On Sun, Jun 14, 2015 at 5:09 PM Mel Beckman m...@beckman.org wrote: SLAs are part of a contract, and thus only apply to the parties of the contract. There are no payments due to other parties. The Internet is a best effort network, with zero guarantees. -mel beckman On Jun 14, 2015, at 4:06 PM, Rafael Possamai raf...@gav.ufsc.brmailto: raf...@gav.ufsc.br wrote: Does anyone know if there's an official ruling as to who gets to pay for the SLA breaches? On Sun, Jun 14, 2015 at 5:56 PM, Mel Beckman m...@beckman.orgmailto: m...@beckman.org wrote: Raymond, But you said A simple 'sorry' would have done. Now you're asking for lots more detail. Why the change? -mel beckman
Re: AS4788 Telecom Malaysia major route leak?
Hi Rafael, I get that much, just wondering if Level3 would have to pay an SLA breach to its customers given the mess started with TM (even though it could have been avoided). And I am guessing if they do, they wouldn't be able to recover anything from TM. I doubt if L3 has to pay anything to its customers in terms of SLA breach, its best effort. Are you aware of any such agreement which suggest otherwise? that would be interesting.
Re: AS4788 Telecom Malaysia major route leak?
On Sun, Jun 14, 2015 at 9:20 PM, Rafael Possamai raf...@gav.ufsc.br wrote: Well, I was wondering the same. I am guessing it depends on the SLA contract since they are all very unique and specific. I'm going to bet that aside from a few one-off cases the SLA in question talks about maintaining reachability inside L3's network, or maybe even 'is your link up and can you ping the L3 gateway router you connect to?' SLA's aren't meant to actually get paid out...
Re: AS4788 Telecom Malaysia major route leak?
On 6/13/15 3:39 AM, Mark Tinka wrote: On 12/Jun/15 22:25, Jürgen Jaritsch wrote: This is the official feedback: Level 3's network, alongside some other ISP's, experienced service disruptions affecting customers in Europe, Asia and multiple other markets. IP, Voice and Content Delivery Network (CDN) services were affected for Level 3. The root cause of the issue was isolated to a third party Internet Service Provider in Asia that leaked internet routes resulting in traffic being sent to a destination that could not route them, which affected IP, Voice and CDN services in multiple markets. The issue has been resolved, but the provider continues working to determine the specific root cause of the incident. At this time, customer services are restored with the exception of any that pose any possible risk to the Level 3 network. Maintaining a reliable, high-performing network for our customers is our top priority. Level 3 will continue to work with the provider to prevent a recurrence. While I agree that TM needs to look into its operational procedures, I think Level(3) needs to shoulder more of the blame, and not simply pass the buck to TM. if you localpref your customer up, you should probably not be willing to accept the whole internet from them. TM has several more upstreams other than Level(3). Assuming their issue affected all their border routers, we did not see an issue via their other upstreams other than Level(3) - although this is conjecture on my part. they also have ~ 180 ASNs in their downstream cone who presumably get a full table have the export policy that did the business in this case applied all the time. Level(3) should have filtered at the time they were turning up TM. Simple as that. We all know we should never trust customers. So... Mark. signature.asc Description: OpenPGP digital signature
Re: AS4788 Telecom Malaysia major route leak?
On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson t...@fud.no wrote a message of 10 lines which said: I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 I, validation-state: valid Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
AS4788 Telecom Malaysia major route leak?
I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 I, validation-state: valid to 64.210.69.85 via xe-1/1/0.0 Tore
Re: AS4788 Telecom Malaysia major route leak?
* Tore Anderson t...@fud.no [2015-06-12 11:12]: I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 I, validation-state: valid to 64.210.69.85 via xe-1/1/0.0 I confirm, something is going on: http://www.karotte.org/pics/bgp-stability.png Regards Sebastian -- GPG Key: 0x93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
Re: AS4788 Telecom Malaysia major route leak?
On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson wrote: I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 I, validation-state: valid to 64.210.69.85 via xe-1/1/0.0 It appears that AS3549 propagated the (almost?) full routing table leak to its peers, where in lots of instances max prefix kicked in. This has global impact, lots of alerts on the SQA collector page http://sqa.ring.nlnog.net/ Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
On 12 Jun 2015, at 16:16, Job Snijders wrote: This has global impact, lots of alerts on the SQA collector page http://sqa.ring.nlnog.net/ I'm reaching out to them now. --- Roland Dobbins rdobb...@arbor.net
Re: AS4788 Telecom Malaysia major route leak?
It *looks* like GBLX stopped accepting the leak. Regards, Marty Strong -- CloudFlare - AS13335 Network Engineer ma...@cloudflare.com +44 20 3514 6970 UK (Office) +44 7584 906 055 UK (Mobile) +1 888 993 5273 US (Office) smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335 On 12 Jun 2015, at 10:27, Roland Dobbins rdobb...@arbor.net wrote: On 12 Jun 2015, at 16:16, Job Snijders wrote: This has global impact, lots of alerts on the SQA collector page http://sqa.ring.nlnog.net/ I'm reaching out to them now. --- Roland Dobbins rdobb...@arbor.net
Re: AS4788 Telecom Malaysia major route leak?
* Marty Strong via NANOG nanog@nanog.org It *looks* like GBLX stopped accepting the leak. If so, it's a partial fix at best, I still see plenty of leaked routes, both via 3356 and 3549, e.g.: tore@cr1-osl3 show route 195.24.168.98 all Jun 12 12:03:54 +0200 inet.0: 544405 destinations, 1591203 routes (543086 active, 3 holddown, 526626 hidden) + = Active Route, - = Last Active, * = Both 195.24.160.0/19*[BGP/170] 00:03:59, MED 2000, localpref 50, from 87.238.63.5 AS path: 3356 3549 4788 6939 39648 I, validation-state: unverified to 87.238.63.56 via ae0.0 [BGP/170] 00:05:24, MED 0, localpref 50, from 87.238.63.2 AS path: 3356 3549 4788 6939 39648 I, validation-state: unverified to 87.238.63.56 via ae0.0 [BGP ] 01:16:00, MED 25245, localpref 100 AS path: 3549 4788 6939 39648 I, validation-state: unverified to 64.210.69.85 via xe-1/1/0.0 It seems to have started around 08:47 UTC, that's when I got my first alarm from ring-sqa at least. Tore
Re: AS4788 Telecom Malaysia major route leak?
Yes, you’re right, I was too trigger happy :( Regards, Marty Strong -- CloudFlare - AS13335 Network Engineer ma...@cloudflare.com +44 20 3514 6970 UK (Office) +44 7584 906 055 UK (Mobile) +1 888 993 5273 US (Office) smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335 On 12 Jun 2015, at 11:18, Job Snijders j...@instituut.net wrote: On Fri, Jun 12, 2015 at 10:43:09AM +0100, Marty Strong via NANOG wrote: It *looks* like GBLX stopped accepting the leak. I disagree. Since 08:44 UTC up until now (10:15) the DFZ has been a radio-active wasteland with hordes of unwelcome announcements. Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
On 12 Jun 2015, at 17:46, Job Snijders wrote: OK, as of now (~ 10:40) UTC things look normalised. Just got off the phone, I think things may be in hand, now. --- Roland Dobbins rdobb...@arbor.net
Re: AS4788 Telecom Malaysia major route leak?
On Fri, Jun 12, 2015 at 01:21:14PM +0200, Sebastian Wiesinger wrote: * Roland Dobbins rdobb...@arbor.net [2015-06-12 12:57]: On 12 Jun 2015, at 17:46, Job Snijders wrote: OK, as of now (~ 10:40) UTC things look normalised. Just got off the phone, I think things may be in hand, now. Still seeing a lot more updates than usual: http://www.karotte.org/pics/bgp-stability-2.png Is this just folks turning up their sessions again? Looks a bit much... Yes, I suspect tons of 3356 / 3549 customers shut down their BGP sessions waiting for the storm to blow over. I expect more churn then usual the next 6 ~ 12 hours, due to customers slowly turning session back on. Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
These aren't just leaks - they're more specifics of what's normally advertised, but keeping the proper origin. Hard to see how that could be accidental... Having looked further - the examples of these I was looking at (advertisements from AS34556 AS17709) were being advertised before the leak, but only with limited visibility. The leak caused them to be (intermittently) globally visible. Tin foil hat off - can all just be accidental. Chris On Fri, 12 Jun 2015, Stephane Bortzmeyer wrote: On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson t...@fud.no wrote a message of 10 lines which said: I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 I, validation-state: valid Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
Re: AS4788 Telecom Malaysia major route leak?
On 06/12/2015 10:43 AM, Marty Strong via NANOG wrote: It *looks* like GBLX stopped accepting the leak I think you just saw it flapping. :-) That's what I've been seeing since ~ 0845 UTC :-( -- rrbone UG (haftungsbeschraenkt) - Leibnizstr. 8a - 44147 Dortmund HR B 23168 Amtsgericht Dortmund - Geschaeftsfuehrer: Dominik Bay
Re: AS4788 Telecom Malaysia major route leak?
First news: http://www.xgn.nl/nieuws/69593/grote-internetstoring-in-europa-problemen-door-route-leak -- Alessandro Martins +55 11 94715-4700 On Fri, Jun 12, 2015 at 8:27 AM, Job Snijders j...@instituut.net wrote: On Fri, Jun 12, 2015 at 01:21:14PM +0200, Sebastian Wiesinger wrote: * Roland Dobbins rdobb...@arbor.net [2015-06-12 12:57]: On 12 Jun 2015, at 17:46, Job Snijders wrote: OK, as of now (~ 10:40) UTC things look normalised. Just got off the phone, I think things may be in hand, now. Still seeing a lot more updates than usual: http://www.karotte.org/pics/bgp-stability-2.png Is this just folks turning up their sessions again? Looks a bit much... Yes, I suspect tons of 3356 / 3549 customers shut down their BGP sessions waiting for the storm to blow over. I expect more churn then usual the next 6 ~ 12 hours, due to customers slowly turning session back on. Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
* Job Snijders j...@instituut.net [2015-06-12 13:30]: Yes, I suspect tons of 3356 / 3549 customers shut down their BGP sessions waiting for the storm to blow over. I expect more churn then usual the next 6 ~ 12 hours, due to customers slowly turning session back on. Yes. It's nice and stable now. http://www.karotte.org/pics/bgp-stability-3.png So after this interesting morning let's hope for a boring weekend. :) Let's wait and see what explanation will be given for this hiccup. Regards Sebastian -- GPG Key: 0x93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
Re: AS4788 Telecom Malaysia major route leak?
These aren't just leaks - they're more specifics of what's normally advertised, but keeping the proper origin. Hard to see how that could be accidental... Chris On Fri, 12 Jun 2015, Stephane Bortzmeyer wrote: On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson t...@fud.no wrote a message of 10 lines which said: I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 I, validation-state: valid Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
Re: AS4788 Telecom Malaysia major route leak?
Hi Marty, Noted. We are still checking this issue. Regards, LEE BON SHENG | leebonsh...@tm.com.my | ipmc_ipc...@tm.com.my NOC2 IPCORE, ISP Network Management, Telekom Malaysia, AS4788 TOLLFREE: 1-800-88-2646 (Opt 4) / International: +603-22466646 (Opt 4) We're committed to perform. We strive to excel. We deliver THE BEST! Regards, Marty Strong -- CloudFlare - AS13335 Network Engineer ma...@cloudflare.com +44 20 3514 6970 UK (Office) +44 7584 906 055 UK (Mobile) +1 888 993 5273 US (Office) smartflare (Skype) http://www.peeringdb.com/view.php?asn=13335 On 12 Jun 2015, at 10:41, Stephane Bortzmeyer bortzme...@nic.fr wrote: On Fri, Jun 12, 2015 at 11:09:34AM +0200, Tore Anderson t...@fud.no wrote a message of 10 lines which said: I see tons of bogus routes show up with AS4788 in the path, and at least AS3549 is acceping them. E.g. for the RIPE NCC (193.0.0.0/21): [BGP/170] 00:20:29, MED 1000, localpref 150 AS path: 3549 4788 12859 I, validation-state: valid Unlike most BGP leaks, they kept the proper origin, so validation by ROA was useless :-(
Re: AS4788 Telecom Malaysia major route leak?
On Fri, 2015-06-12 at 10:43 +0100, Marty Strong via NANOG wrote: It *looks* like GBLX stopped accepting the leak. Nope. Churn is ongoing, nothing has been fixed. Global outage began 08:44 UTC and is still ongoing. It's been so long people have now had time to come up with things like 33.333%. Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :) /M signature.asc Description: This is a digitally signed message part
RE: AS4788 Telecom Malaysia major route leak?
Still on hold with Level3, but some of my sites are clearing up. Chris -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Martin Millnert Sent: Friday, June 12, 2015 3:24 AM To: Marty Strong Cc: NANOG Subject: Re: AS4788 Telecom Malaysia major route leak? On Fri, 2015-06-12 at 10:43 +0100, Marty Strong via NANOG wrote: It *looks* like GBLX stopped accepting the leak. Nope. Churn is ongoing, nothing has been fixed. Global outage began 08:44 UTC and is still ongoing. It's been so long people have now had time to come up with things like 33.333%. Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :) /M
Re: AS4788 Telecom Malaysia major route leak?
On Fri, Jun 12, 2015 at 10:43:09AM +0100, Marty Strong via NANOG wrote: It *looks* like GBLX stopped accepting the leak. I disagree. Since 08:44 UTC up until now (10:15) the DFZ has been a radio-active wasteland with hordes of unwelcome announcements. Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
On Fri, Jun 12, 2015 at 12:18:38PM +0200, Job Snijders wrote: On Fri, Jun 12, 2015 at 10:43:09AM +0100, Marty Strong via NANOG wrote: It *looks* like GBLX stopped accepting the leak. I disagree. Since 08:44 UTC up until now (10:15) the DFZ has been a radio-active wasteland with hordes of unwelcome announcements. OK, as of now (~ 10:40) UTC things look normalised. Kind regards, Job
Re: AS4788 Telecom Malaysia major route leak?
This is the official feedback: Level 3's network, alongside some other ISP's, experienced service disruptions affecting customers in Europe, Asia and multiple other markets. IP, Voice and Content Delivery Network (CDN) services were affected for Level 3. The root cause of the issue was isolated to a third party Internet Service Provider in Asia that leaked internet routes resulting in traffic being sent to a destination that could not route them, which affected IP, Voice and CDN services in multiple markets. The issue has been resolved, but the provider continues working to determine the specific root cause of the incident. At this time, customer services are restored with the exception of any that pose any possible risk to the Level 3 network. Maintaining a reliable, high-performing network for our customers is our top priority. Level 3 will continue to work with the provider to prevent a recurrence. Jürgen Jaritsch Head of Network Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: j...@anexia.atmailto:j...@anexia.at Web: http://www.anexia.athttp://www.anexia.at/ Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Re: AS4788 Telecom Malaysia major route leak?
* milln...@gmail.com (Martin Millnert) [Fri 12 Jun 2015, 12:54 CEST]: Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :) https://scontent-sea1-1.xx.fbcdn.net/hphotos-xat1/t31.0-8/10914977_10152809997716851_748171875526832420_o.jpg Is that tweet for real? How is that company (not TM) still in business? -- Niels.
Re: AS4788 Telecom Malaysia major route leak?
* Roland Dobbins rdobb...@arbor.net [2015-06-12 12:57]: On 12 Jun 2015, at 17:46, Job Snijders wrote: OK, as of now (~ 10:40) UTC things look normalised. Just got off the phone, I think things may be in hand, now. Still seeing a lot more updates than usual: http://www.karotte.org/pics/bgp-stability-2.png Is this just folks turning up their sessions again? Looks a bit much... Regards Sebastian -- GPG Key: 0x93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
Re: AS4788 Telecom Malaysia major route leak?
http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/ Jürgen Jaritsch Head of Network Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: j...@anexia.atmailto:j...@anexia.at Web: http://www.anexia.athttp://www.anexia.at/ Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Re: AS4788 Telecom Malaysia major route leak?
Does anyone at Level3 care to comment here about this event, and if there are any plans to push BGP prefix security? 2015-06-12 8:25 GMT-05:00 Jürgen Jaritsch j...@anexia.at: http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/ Jürgen Jaritsch Head of Network Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: j...@anexia.atmailto:j...@anexia.at Web: http://www.anexia.athttp://www.anexia.at/ Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Re: AS4788 Telecom Malaysia major route leak?
On Fri, Jun 12, 2015 at 09:58:55AM -0500, Charles van Niman char...@phukish.com wrote a message of 25 lines which said: Does anyone at Level3 care to comment here about this event, https://twitter.com/Level3/status/609353696787496960
Re: AS4788 Telecom Malaysia major route leak?
Looks to be edited from their original tweet. On Fri, Jun 12, 2015 at 9:07 AM, niels=na...@bakker.net wrote: * milln...@gmail.com (Martin Millnert) [Fri 12 Jun 2015, 12:54 CEST]: Also, possible explanation for why nobody's fixing it: https://twitter.com/TMCorp/status/609167065300271104 :) https://scontent-sea1-1.xx.fbcdn.net/hphotos-xat1/t31.0-8/10914977_10152809997716851_748171875526832420_o.jpg Is that tweet for real? How is that company (not TM) still in business? -- Niels.
