Re: [outages] Twelve99 / AWS usw2 significant loss

I see the AWS->me path has just changed to Cogent. me->AWS path still appears to be Twelve99. Loss and latency has now subsided. Thanks > On 26 Jan 2024, at 08:19, Phil Lavin wrote: > > Hi Folks, > > Cross posting to outages and nanog. > > Can anybody from Twelve99 / AWS investigate high

Re: [outages] Twelve99 / AWS usw2 significant loss

Thanks, ytti. I have raised a case with AWS but I expect it to be as unproductive as usual. In this type of issue, there is often a friendly Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. Hoping that to be the case this time, also. > On 26 Jan 2024, at 08:40, Saku

[outages] Twelve99 / AWS usw2 significant loss

Hi Folks, Cross posting to outages and nanog. Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS? Traces below: 88.99.88.67 to 216.147.3.209: My traceroute [v0.94] phil (10.88.10.10) -> 216.147.3.209

Re: [outages] AS37468 (Angola Cables) - Route Leak?

Return path is via NTT and looks OK: My traceroute [v0.94] http-lb-01 (88.99.88.69) -> 216.147.4.201 2023-05-25T12:34:28+ Keys: Help Display mode Restart statistics Order of fields quit

[outages] AS37468 (Angola Cables) - Route Leak?

Hey Folks, Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69). Traffic was transiting via AS37468 (Angola Cables) in Coresite IX. Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX. I e-mailed Hetzner

Re: [outages] anyone from charter ?

That was raised by me, if I recall, in job—. It was related to an ACL change they did which caused fragmented UDP packets (the fragment not containing UDP headers) to get dropped. Does the issue sound similar? -- Phil Lavin | Engineering Director, Voice APIs T: +44 2033 710587

Re: [outages] More AWS Trouble 12/15?

> Did anyone else see some AWS-hosted services go wonky Yeh. AWS have confirmed "Internet Connectivity” issues. It looks to have stabilised somewhat but still seeing some unusual errors Phil ___ Outages mailing list Outages@outages.org

Re: [outages] Major Fiber cut Baltimore area

> Major fiber outage in Baltimore area, level3 confirmed field techs on site > investigating. Approximately what time did it happen? We saw high RTT and some packet loss on an AWS link from Boston to Virginia starting at approx. 15:17 UTC. RTT is still fractionally higher (~3ms) than normal,

Re: [outages] CenturyLink Outages this morning

> Facebook is blowing up with 911 outages and claiming it's CenturyLink to blame For what it's worth, here is a list of PSAPs which were reported as having problems, presumably due to the CenturyLink outages. Can't be sure if it's complete or not: Malden Police Department, MA Seekonk Police

[outages] AWS eu-west-1b issues?

We lost connectivity to eu-west-1b over direct connect circuits around 20 mins ago. Is anybody else seeing issues? ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Re: [outages] AWS East

> At 12:38 EDT we experienced severe routing issues within AWS East. Can anyone > else confirm? US East (Virginia)? If so, nothing seen here ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outage in the midwest?

BGP session to Comcast has dropped for us on the East Coast. We were unaffected until that happened. I suspect other East Coast providers have seen similar, which is the cause of the routing change. From: Outages On Behalf Of Dovid Bender via Outages Sent: 29 June 2018 19:16 To: Rob Repp Cc:

Re: [outages] Comcast data issues

We saw a BGP session to them in Boston MA flap > On 7 Jun 2018, at 20:03, Chuck Anderson via Outages > wrote: > > Has anyone noticed any Comcast data issues? I could't ping 8.8.8.8 a few > minutes ago although that came back. Also issues with Google Docs. My IPv6 > connectivity was also

Re: [outages] Google 8.8.8.8 Resolution of Route53 domains

...@outages.org>> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 13:04 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Looks more specific to AWS than it does to Google+AWS. Can’t resolve against some of AW

Re: [outages] Google 8.8.8.8 Resolution of Route53 domains

a fair bit of AWS Route53. If you (or your DNS resolver) are a HE transit customer you will be impacted the most. Cheers, Joseph On Tue, Apr 24, 2018, at 9:50 PM, Phil Lavin via Outages wrote: This doesn’t feel right, though I’ll admit I’ve never checked before. Our only route to ns-163.aws

Re: [outages] Google 8.8.8.8 Resolution of Route53 domains

In fact, 3 of 4 nameservers on our main domain are advertised as different /24s by AS 10297. None of these can be queried. The other is advertised by AS 16509 (AWS) – this works. From: Outages <outages-boun...@outages.org> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018

Re: [outages] Google 8.8.8.8 Resolution of Route53 domains

;; connection timed out; no servers could be reached From: Outages <outages-boun...@outages.org> On Behalf Of Phil Lavin via Outages Sent: 24 April 2018 12:56 To: outages@outages.org Subject: Re: [outages] Google 8.8.8.8 Resolution of Route53 domains Yeh. Still digging into it. From: Outage

Re: [outages] Google 8.8.8.8 Resolution of Route53 domains

Yeh. Still digging into it. From: Outages On Behalf Of Zach Hanna via Outages Sent: 24 April 2018 12:54 To: outages@outages.org Subject: [outages] Google 8.8.8.8 Resolution of Route53 domains Anyone else seeing SERVFAIL for route53-hosted domains trying to resolve

[outages] Telehouse Vodafone Circuit

Just had a BGP session pop on a Vodafone transit circuit in Telehouse North. Session came back but routes are coming through at a rate of about 100 per minute. Get dead air when calling the CSC Helpdesk. Anyone else seeing issues? ___ Outages mailing

Re: [outages] Slack outage

I think it’s clear by now that it’s having an outage. Not sure we need any more “me too” responses. From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Ivan Kovacevic via Outages Sent: 09 January 2018 19:22 To: Josh Luthman Cc: outages@outages.org

Re: [outages] Bandwidth Voice Issues?

Which POPs are you connected into? Not seeing any issues to New York or Atlanta POPs. ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Vodafone Outage?

Transit and VoIP OK here from London Docklands From: Outages [mailto:outages-boun...@outages.org] On Behalf Of support via Outages Sent: 01 November 2017 17:04 To: outages@outages.org Subject: [outages] Vodafone Outage? Is anyone experiencing issues with Vodafone? We have multiple services

Re: [outages] NTT Down - London Docklands

. Last time this happened, ISTR it was a problem with the fibre feeding the ASR 9000 nV satellite exetender things that they use for aggregating 1G services. Edward Dore Freethought Internet From: Outages <outages-boun...@outages.org<mailto:outages-boun...@outages.org>> on behalf of P

Re: [outages] NTT Down - London Docklands

rom: Outages <outages-boun...@outages.org<mailto:outages-boun...@outages.org>> on behalf of Phil Lavin via Outages <outages@outages.org<mailto:outages@outages.org>> Reply-To: Phil Lavin <phil.la...@cloudcall.com<mailto:phil.la...@cloudcall.com>> Date: Tuesday, 31

[outages] NTT Down - London Docklands

We've just lost light from NTT on circuits in London Docklands. Anyone else seen issues? Cheers Phil ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

[outages] Godaddy

Hello all, Sorry for off topic. Does anyone have a tech contact at Godaddy? Getting nowhere with 1st line support. Phil ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Re: [outages] XO SIP call quality poor in Chicago

SIP calls over XO seem OK out of Boston. Phil From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Judah Sameth via Outages Sent: 01 June 2017 15:59 To: outages@outages.org Subject: [outages] XO SIP call quality poor in Chicago Having issues starting at 8:31AM CST with call quality

Re: [outages] Level3 voice outage?

Looks OK here. Is this SIP? Which destinations are failing what response code are you getting on the failed calls? -Original Message- From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Julian Y Koh via Outages Sent: 16 May 2017 16:47 To: outages@outages.org Subject:

Re: [outages] Amazon S3 down

Also seeing issues with EC2 Auto Scaling Groups and Lambda functions. Getting fun errors like "Status Reason: Insufficient capacity. Launching EC2 instance failed". From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Peter Kristolaitis via Outages Sent: 28 February 2017 18:23 To:

[outages] Level 3 Packet Loss - Boston Area

Is anybody else experiencing issues in this area? Seeing loss between us and a couple of customers. MTR on both sides shows loss around Boston inside Level3. Phil ___ Outages mailing list Outages@outages.org

[outages] Bandwidth - Packet Loss

Hi all, Between 15:07 and 15:58 UTC we observed high packet loss on all inbound calls from the Bandwidth SBC at 67.231.13.113. We did not see loss from other carriers/customers, nor from other Bandwidth SBCs. Did anybody else see packet loss? Bandwidth say they are not aware of any problems,

[outages] Telehouse North - Major Problems

We've just had 3 links drop simultaneously to (different) equipment in Telehouse North. Fibre link to Vodafone - port is down BGP peering to GTT is dropped Copper link to BT - port is down Anyone else seeing anything? We spoke to BT and they have confirmed a "major national problem". Phil