Re: [outages] COX outage?

[Andree Toonk via Outages]

We are seeing major packet loss on Level3 at various of our west coast 
north America locations, also Denver.

Guessing it's related...

Cheers
 Andree



Sam Thomas via Outages wrote on 2020-04-21 8:35 AM:
A number of our remote users and some of our remote offices on COX are 
having some extreme difficulties with their VPN connections. One of 
the users called COX and was informed that there is a "major outage". 
Could someone shed some light on the scope of this "major outage"? 
(e.g. geographic, peering, backbone, etc)


TIA,
Sam


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] FW: [Twilio status] Investigating : Call Failures

[Andree Toonk via Outages]

There's also an AWS US east issue affecting folks with direct connects
there.. may be related to the east coast issues.. That or memcache..

https://status.aws.amazon.com/

7:29 AM PST We are investigating increased packet loss possibly
impacting some AWS Direct Connect customers in the US-EAST-1 Region.
8:03 AM PST We continue to work towards resolving the increased packet
loss impacting AWS Direct Connect connectivity to the US-EAST-1 Region.
Direct Connect connections from the Equinix DC1 - DC6 & DC10 - DC12,
Ashburn, VA and CoreSite VA1 & VA2, Reston, VA locations are affected by
this issue.




My secret spy satellite informs me that Russell Zen via Outages wrote On
2018-03-02, 9:09 AM:
> Twilio's on Akamai no? I am now wondering if this is related: 
> 
> https://www.wired.com/story/github-ddos-memcached/
> 
> https://githubengineering.com/ddos-incident-report/
> 
> 
> 
> 
> 
> 
> 
> *From:* Outages  on behalf of Joe Adam via
> Outages 
> *Sent:* Friday, March 2, 2018 9:51 AM
> *To:* National Outages List
> *Subject:* [outages] FW: [Twilio status] Investigating : Call Failures
>  
> 
> Notice from Twilio, this may have some impact on the VCC sites.
> 
>  
> 
> *From:* status-h...@statuspage.twilio.com
> [mailto:status-h...@statuspage.twilio.com]
> *Sent:* Friday, March 02, 2018 11:40 AM
> *To:* te...@answernet.com
> *Subject:* [Twilio status] Investigating : Call Failures
> 
>  
> 
> *Call Failures  *
> 
> Incident Report for Twilio
> 
> *Investigating *
> 
> We are experience increased packet loss on outbound and inbound calls
> related to networking failures on the east coast. Team is working to
> adjust traffic to fully restore service
> 
> Mar 2, 08:40 PST
> 
> *Previous Updates *
> 
> *Monitoring *
> 
> We are seeing recovery in Ireland and Germany. We are continuing to monitor
> 
> Mar 2, 08:26 PST
> 
> *Investigating *
> 
> We are observing call failures in Ireland and Germany datacenters. On
> call team is responding
> 
> Mar 2, 08:25 PST
> 
> This incident affects: Programmable Voice (PSTN) and Elastic SIP Trunking.
> 
> View status page  or manage your subscription
>  to these emails.
> Powered by Statuspage
> 
> 
> 
>  
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Level 3 Outage in Western US

[Andree Toonk via Outages]

Hi Brandon,

My secret spy satellite informs me that Brandon Bianchi via Outages
wrote On 2017-08-17, 7:49 PM:
> I have Level 3 transit services that connect in Vegas and Los Angeles
> that seem to be experiencing loss and increased latency to the point
> where the circuits are not useable. Anyone else experiencing these issues?

Yup, we saw the same. i.e. Loss from our level3 circuit in LAX towards
several other Level3 services we have on the west coast (SJC, PAO, SEA)
as well as to our level3 circuits in Chi and Toronto. It started at
around 1:34 UTC till 3:01 UTC

Also see the visual here: http://i.imgur.com/mYDi4Fa.png

Cheers
 Andree
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Level3 huge packet loss around AMS

[Andree Toonk via Outages]

Yup lots of packet loss (up to 100%) between US and EU over level3.
First packet loss detected at 21:04 UTC, things got better at around
21:55 UTC


.-- My secret spy satellite informs me that at 2015-09-04 2:52 PM  Luca
Salvatore via Outages wrote:
> Anyway else seeing this?
> Huge loss in Amsterdam on level3
> 
> 
>   1. 10.15.252.1   0.0%500.4   0.3   0.2   0.6   0.1
>   2. 69.55.63.193  0.0%500.6   2.4   0.4  11.0   3.2
>   3. 198.211.111.780.0%500.6   0.6   0.4   1.3   0.2
>   4. nyk-b6-link.telia.net  0.0%  
>  500.8   1.0   0.7   9.7   1.3
>   5. nyk-b5-link.telia.net  0.0%  
>  500.8   1.1   0.7   9.8   1.4
>   6. ???  100.0500.0   0.0   0.0   0.0   0.0
>   7. ae-237-3613.edge6.Amsterdam1.Level3.net
>  88.0%50  142.2
> 141.2 139.3 142.2   1.2
>   8. te5-8.bg1.am3.eu.equinix.net 
> 88.0%50  133.8 137.3 133.8 140.1   2.6
>   9. v116.ed1.am3.eu.equinix.net 
>  90.0%50  138.7 136.7 131.1 139.1   3.4
>  10. 5.175.83.162 90.0%50  137.2 138.4 137.2 139.2   0.9
>  11. 5.175.83.163 90.0%50  133.4 134.5 133.2 135.4   1.1
> 
> Luca
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Level3 DECIX/Frankfurt Packet Loss

[Andree Toonk via Outages]

Also see here for more details:
http://www.bgpmon.net/massive-route-leak-cause-internet-slowdown/


.-- My secret spy satellite informs me that at 2015-06-12 6:17 AM  Hoyle
Anderson (AM) via Outages wrote:
 Possibly related: http://seclists.org/nanog/2015/Jun/586 and
 http://seclists.org/nanog/2015/Jun/607
 
  
 
  
 
  
 
 *From:*Outages [mailto:outages-boun...@outages.org] *On Behalf Of
 *Plamen Stoev via Outages
 *Sent:* Friday, June 12, 2015 6:40
 *To:* Max Mühlbronner
 *Cc:* outages@outages.org
 *Subject:* Re: [outages] Level3 DECIX/Frankfurt Packet Loss
 
  
 
  
 
 Hi,
 
  
 
 Level3 raised an Event Ticket ID 9340550  with following information:
 
  
 
 06/12/2015 10:06 AM GMT
 
 The IP NOC is investigating routing issues in Multiple Markets across
 the network globally that are impacting IP services. Troubleshooting
 efforts are currently ongoing with no estimated time to restore services
 available at this point. Please be advised that updates for this event
 will be relayed every half an hour unless otherwise noted.
 
  
 
 ​Best Regards,
 
 ​Plamen​
 
  
 
  
 
 On Fri, Jun 12, 2015 at 11:52 AM, Max Mühlbronner outages@outages.org
 mailto:outages@outages.org wrote:
 
 Hi,
 
 i don't think so, nothing shown at my level3.
 
 
 
 
 On 12.06.2015 11:48, James Baldwin wrote:
 
 Max, do you know if there is a master L3 ticket yet?
 
  
 
 On Fri, Jun 12, 2015 at 4:48 AM, James Baldwin
 jbald...@antinode.net mailto:jbald...@antinode.net wrote:
 
 We are seeing packetloss across all of our EU sites backed by L3.
 
  
 
 Possibly related to the AS4788 Telecom Malaysia major route leak
 thread on NANOG?
 
  
 
 On Fri, Jun 12, 2015 at 4:41 AM, Max Mühlbronner
 outages@outages.org mailto:outages@outages.org wrote:
 
 Hi,
 
 indeed, we were also having serious issues and confirmed the
 packetloss/connectivity-loss (worldwide) via RIPE Atlas queries. We
 had to switch it off. Ticket opened @level3.
 
 
 Best Regards
 
 Max M.
 
 
 
 On 12.06.2015 11:29, Harald Haas (hahaas) via Outages wrote:
 
 Hey All,
 
  
 
 we are currently experiencing very high packet loss over
 DECIX/Frankfurt with two different routings.
 
  
 
 The hops which are dropping the packets seem to be:
 
  
 
  4.69.154.135 (ae-3-80.edge3.Frankfurt1.Level3.net
 http://ae-3-80.edge3.Frankfurt1.Level3.net)
 
  62.4.65.106 (62.4.65.106.not.updated.fra.de.eunx.net
 http://62.4.65.106.not.updated.fra.de.eunx.net)
 
  
 
 Someone else has recognized similar problems?
 
 
 
 ___
 
 Outages mailing list
 
 Outages@outages.org mailto:Outages@outages.org
 
 https://puck.nether.net/mailman/listinfo/outages
 
  
 
 
 ___
 Outages mailing list
 Outages@outages.org mailto:Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages
 
  
 
  
 
 
 
 itevomcid
 
 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] GTT Peering issues

[Andree Toonk via Outages]

Seeing similar issues between a few US, Asia and EU sites. What I'm
seeing is what appears to be a congested peering connection between GTT
and NTT in Seattle as that is what all traceroutes have in common.

Our graphs indicate this started Jan 1st, around ~ 15:00 utc.

Example traceroute from Seattle (ntt) to Vancouver (gtt):


HOST: rtr1-re0.seaLoss%   Snt   Last   Avg  Best  Wrst StDev
  1. xe-0-0-0-34.r04.sttlwa01.us.  0.0%604.1   8.3   0.8  23.9   6.5
  2. ae3.sea22.ip4.gtt.net20.0%60   44.6  41.9  33.6  60.3   6.0
  3. xe-1-2-0.van10.ip4.gtt.net   18.3%60   48.9  51.6  38.0 129.1  17.3
  4. opendns-gw.ip4.gtt.net   20.0%60   46.6  46.6  38.3  65.1   6.9
  5. rtr1.yvr.opendns.com 28.3%60   44.8  47.5  38.6  74.2   8.9

Cheers,
 Andree



.-- My secret spy satellite informs me that at 2015-01-05 3:04 PM  Hugh
Smallwood via Outages wrote:
 Seeing awful peering issues with GTT today.  Specifically between Level3 in 
 LA and their own network between Washington and Phoenix.  
 
 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages
 
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] widespread level3 issue amsterdam/western europe?

[Andree Toonk via Outages]

.-- My secret spy satellite informs me that at 2014-09-08 10:31 PM  joel
jaeggli via Outages wrote:
 is anyone else seeing routing implosion on l3 or first symtoms were in
 western europe.
 
 I had packets bound for l3 ips in ams making lazy circles. started circa
 0430 utc

Same here, 100% loss over Level3 between San Jose and LA as of 4:50 UTC.
Also seeing issues transatlantic over level3, Amsterdam being one of them.

Andree

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages