Re: [outages] Optus/Akamai

[Jared Mauch via Outages]

Disclaimer: I am employed by Akamai

I’m not aware of any changes that could have triggered this, if someone has 
information they can share you can reach out to me directly or to our NOCC.  I 
can be reached at ja...@akamai.com if you have something private to share and 
my mobile number is easily findable as well.

- Jared

> On Nov 8, 2023, at 9:20 AM, Jay Ashworth via Outages  
> wrote:
> 
> Off of Facebook of all places, I hear this report that Optus had a major 
> outage in Australia last night, which is believed to have been a BGP screw up 
> involving Akamai:
> 
> "Nope: it was Akamai increasing their advertised prefix count from their cdn 
> nodes inside the Optus network, and Optus’s configured max-prefix settings on 
> their core bgp sessions from their route reflectors weren’t quite high enough 
> and their whole network went down.  Someone didn’t quite think through the 
> value for internal max-prefix rules.."
> 
> Anybody confirm, deny, got hit by it?
> 
> Cheers,
> -- jra
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Backlog cleared

[Jared Mauch via Outages]

I wanted to give everyone a quick update (reply-to set to just me, so send your 
gripes/complaints/whatnot there)

The mail backlog is cleared.

There were approximately 60k e-mails backlogged to go out to people which is 
now caught up.  I learned about a script called qshape which can tell you what 
domains are backlogged.  It was largely who you would expect.  I cranked some 
options on postfix to allow more concurrent connections for mail delivery per 
message and have adjusted mailman to split into groups as well - these should 
hopefully be the right balance to keep mail flowing quickly.  

smtp_destination_concurrency_limit has been set much larger (500) which also 
helped clear the backlog faster.  I’ll keep monitoring things to figure out 
what else needs to be adjusted.

Apologies for not meeting expectations.  I will refund you a full month service 
fee :-)

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Facebook back up

[Jared Mauch via Outages]

> On Oct 4, 2021, at 5:54 PM, Ryan McGinnis via Outages  
> wrote:
> 
> Productivity back down
> 
> Due to the outages list backlog, you will probably receive this sometime 
> after Thanksgiving

“I’m not even supposed to be here today” - outages sysadmin staff 

:-)

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Meta: outages outage

[Jared Mauch via Outages]

> On Oct 5, 2021, at 5:34 AM, Jürgen Botz via Outages  
> wrote:
> 
> OK, aside from the Facebook outage, and no doubt because of it, we 
> effectively had an outages outage... messages to the lsit were massively
> delayed, causing lots of duplicate reports and general failure to have
> any meaningful confirmations, etc.
> 
> Why?  What happened?  Well, outages is hosted on a server
> (puck.nether.net) with a couple dozen other lists, and some of these lists 
> probably have members with email addresses at facebook domains,
> and the SMTP server kept trying to look those up and having wait for
> timeouts?  Something like that, except that since there were no routes
> to facebook's nameservers there shouldn't have been any need to wait
> for timeouts.  Anyone here form nether.net actually take a look?
> 
> It's more than idle curiosity, I'd like make my mail servers more
> resilient to this kind of situation, and the outages list of all
> probably should be, too.


Sure, I’ve gone and tweaked a few more things, there’s a balancing act here 
between having a single system send out lots of concurrent mail to a server, eg:

60756540E50*   22760 Tue Oct  5 01:28:00  outages-boun...@outages.org
(host itchy.cerento.com[199.190.154.20] said: 451 Only one recipient at a time 
(in reply to RCPT TO command))
 redac...@cerento.com

Some systems are explicitly configured to not be optimized (see above, then 
imagine how many people in parallel @gmail might get a message)

And there are many people with old subscriptions or domains that don’t exist 
anymore which gets to be exciting when everything is going on at once:

  (connect to mail.bestii.com[172.96.180.81]:25: Connection timed out)
 redac...@bestii.com
 (connect to canadawebhosting.com[74.201.58.138]:25: Connection timed out)
 redac...@canadawebhosting.com
(connect to muhpanel.ironusmaidenus.com[51.15.246.204]:25: Connection refused)
 redac...@ironusmaidenus.com
(connect to mail.mailhost4.com[198.50.245.163]:25: Connection refused)
 redac...@mailhost4.com
   (connect to mail.etsms.com[50.232.238.69]:25: Connection timed out)
 redac...@etsms.com
  (connect to keyedupmedia.com[204.11.56.48]:25: Connection timed out)
 redac...@keyedupmedia.com
  (connect to metacloud.com[72.52.10.14]:25: Connection timed out)
 redac...@metacloud.com
 (connect to smtp.naturalwireless.com[52.119.91.194]:25: Connection timed out)
 redac...@naturalwireless.com
 redac...@naturalwireless.com
(host s1.mail.pciwest.net[2604:2400:a::425] said: 450 4.7.1 Bad Attachment .com 
(in reply to end of DATA command))
 redac...@presys.com
(delivery temporarily suspended: connect to 
mail.talueee.com[159.69.230.243]:25: No route to host)
 redac...@talueee.com
(connect to mail-1.meridian-enviro.com[198.58.69.47]:25: Connection timed out)
 redac...@meridian-enviro.com
  (reason unavailable)

(All of these are from the same message btw)

I was away for personal reasons yesterday so missed out on all the fun.  I’ll 
check how this message goes out and see if the tweaks helped.

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Spectrum/LTWC Ohio/KY

[Jared Mauch via Outages]

Sorry just saw this. I think this may have been related to some unknown BGP 
attributes today and I expect their BGP was flapping as a result.

Sent from my TI-99/4a

> On Sep 7, 2021, at 6:24 PM, Ben Groves via Outages  
> wrote:
> 
> 
> Per Spectrum NOC, the issue has been resolved.
> "The Spectrum Enterprise NOC confirmed that a routing issue was found by 
> Spectrum Enterprise Engineering team once corrected the severe packet loss 
> and intermittent connectivity that customers were experiencing. This will 
> serve as the final notification for this event. "
> 
>> On Tue, Sep 7, 2021, 1:14 PM Josh Luthman via Outages  
>> wrote:
>> Looks like there is an issue with the coax network.
>> > /tool traceroute 1.1.1.1
>>  # ADDRESS  LOSS SENTLAST AVGBEST   WORST
>>  1 142.254.145.13   9.8%  143 timeout13.5 7.9   100.7
>>  2 24.29.6.105  9.1%  143  24.3ms42.413.4   226.8
>>  3 65.29.36.238 10..  143  14.1ms19.7 9.8   179.4
>>  4 65.29.1.4614%  14325ms27.216.795.6
>>  5 66.109.6.66  10..  143  34.1ms30.723.8   141.6
>>  6 66.109.3.24  13..  143  26.6ms35.424.5 202
>>  7 209.18.43.59 8.4%  143  28.3ms30.724.395.1
>>  8 173.245.63.5612..  143  42.2ms39.327.5   135.5
>>  9 172.70.44.4  11..  142  43.5ms35.124.2   161.6
>> 10 1.1.1.1  6.3%  142  44.1ms33.424.1   149.8
>> 11 1.1.1.10%1  26.1ms26.126.126.1
>> 
>> 
>> Fiber DIA seems just fine to me.
>> 
>>  > /tool traceroute 1.1.1.1
>>  # ADDRESS  LOSS SENTLAST AVGBEST   WORST
>>  1 10.10.10.1 0%  171   0.3ms 0.3 0.3 3.6
>>  2 192.149.254.1  0%  171   0.4ms 0.4 0.3 1.3
>>  3 74.218.88.129  0%  171 2ms 1.4 0.631.9
>>  4 65.189.190.189 0%  171   8.7ms10.4 0.8   143.4
>>  5 65.29.38.112   0%  171  14.2ms12.5 8.316.9
>>  6 65.29.1.46 0%  170  11.4ms 9.6 8.214.9
>>  7 66.109.6.120%  170  17.5ms16.516.1  20
>>  8 66.109.3.240%  170  16.9ms17.116.542.2
>>  9 209.18.43.59   0%  170  26.2ms20.216.264.7
>> 10 24.30.200.73   0%  170  20.4ms31.116.789.4
>> 11 172.70.44.20%  170  17.2ms19.516.672.3
>> 12 1.1.1.10%  170  16.2ms16.416.231.5
>> 
>> Josh Luthman
>> 24/7 Help Desk: 937-552-2340
>> Direct: 937-552-2343
>> 1100 Wayne St
>> Suite 1337
>> Troy, OH 45373
>> 
>> 
>>> On Tue, Sep 7, 2021 at 1:09 PM Savarrah Goss via Outages 
>>>  wrote:
>>> https://app.fing.com/internet/outage/DROP:US-Ohio--Spectrum%20Business@2021-09-07-1650-0
>>> 
>>>  
>>> 
>>> Savarrah Goss | Hamilton Relay
>>> O: 229.338.7086 | C: 229.854.8872 | savarrah.g...@hamiltonrelay.com | 
>>> HamiltonRelay.com
>>> 
>>>   Hamilton Relay  |   @HamiltonRelay
>>>  
>>> 
>>> From: Outages  On Behalf Of Mike Markwood via 
>>> Outages
>>> Sent: Tuesday, September 7, 2021 1:00 PM
>>> To: 'outages@outages.org' ; Neader, Brent 
>>> 
>>> Subject: Re: [outages] Spectrum/LTWC Ohio/KY
>>> 
>>>  
>>> 
>>> **This email originated from outside of the Hamilton network.  Please use 
>>> extra caution with links and attachments. **
>>> 
>>> We're seeing our Spectrum cable modem customers across Northeast Ohio 
>>> bouncing as well as Columbus, OH and a few customers in TN. 
>>> 
>>>  
>>> 
>>> Mike Markwood | Director of Network Operations
>>> P: 866.986.1066  E: mmarkw...@geostarllc.net
>>> 
>>> 1000 S. Cleveland Massillon Road, Suite 3
>>> Fairlawn, OH 44333
>>> 
>>> 
>>> 
>>>  
>>> 
>>> From: Outages  on behalf of Neader, Brent via 
>>> Outages 
>>> Sent: Tuesday, September 7, 2021 12:52 PM
>>> To: 'outages@outages.org' 
>>> Subject: [outages] Spectrum/LTWC Ohio/KY
>>> 
>>>  
>>> 
>>> Seeing enterprise DIA sites drop off and come back repeatedly in these 
>>> areas, Dayton, Cinci, Cleveland, Middletown, Louisville.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> CONFIDENTIALITY NOTE: This email and any attachments are confidential and 
>>> may be protected by legal privilege. If you have received this email in 
>>> error, please immediately notify the sender by return email and delete all 
>>> copies of this email.
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> 

[outages] Testing - Delete please

[Jared Mauch via Outages]

I've been tweaking a bunch of the backend settings to try and speed up
mail delivery for these e-mails.  I am hoping that I got it this time™

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Akamai DNS outage?

[Jared Mauch via Outages]

On Thu, Jul 22, 2021 at 09:59:15PM -0400, chris wrote:
> https://youtu.be/cWYwUUQJzsw
> 
> On Thu, Jul 22, 2021, 9:48 PM Jared Mauch via Outages 
> wrote:
> 
> > Outages isn't on Akamai dns.


Yes, but the outages.org DNS and infrastructure is not hosted
via Akamai resources.  Machine is still colocated with my old employer
(thank you!).

Apparently I need to tweak the behavior of some of the anti-spam
stuff to ensure messages flow faster.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Akamai DNS outage?

[Jared Mauch via Outages]

Outages isn't on Akamai dns. 

Sent from my TI-99/4a

> On Jul 22, 2021, at 6:46 PM, Seth Mattinen via Outages  
> wrote:
> 
> 
> 
>> On 7/22/21 9:45 AM, Andy Ringsmuth via Outages wrote:
>> Surprised I haven’t see any mention of it here yet.
> 
> 
> I see delayed messages here; was outages on Akamai DNS?
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Some kind of big issue on east coast?

[Jared Mauch via Outages]

Looks like there was a lot of BGP instability around the 7018 and 6939 networks 
for that hour.  

- jared


> On Jun 15, 2021, at 11:38 AM, Charles Sprickman via Outages 
>  wrote:
> 
> Traceroutes, pings, anything would be really helpful when reporting something 
> like this.
> 
> It at least shows the hundreds of people that get a slight eye twitch when 
> this list wakes up that some basic troubleshooting has happened.
> 
> Charles
> 
> (in NJ, on fios, and no issues, fwiw)
> 
>> On Jun 15, 2021, at 10:38 AM, Douglas Haber via Outages 
>>  wrote:
>> 
>> I have clients on multiple different ISP's all reporting connectivity issues 
>> and packet loss starting in the last half hour.
>> 
>> Optimum/Altice and Verizon in NJ  and AT in NC
>> 
>>  -- 
>> 
>> Douglas Haber
>> Managing Member
>> Garden State Computing, LLC
>> Tel: (973) 636-7350
>> www.gardenstatecomputing.com
>> supp...@gardenstatecomputing.com
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Akamai Geolocation Concerns

[Jared Mauch via Outages]

If you contact me off-list I can help :-)

We don't want you going to europe when you're in the US for
example, but there may be reasons/cases that happens as well.

- Jared

On Fri, Jan 17, 2020 at 01:35:00PM -0600, Dan White wrote:
> Thanks Jared. We operate our own DNS resolvers, which are seeing the same
> behavior. I wanted to test against a couple of public servers to,
> presumably, rule out a geolocation database problem.
> 
> Josh,
> 
> All looks good with regards to the data provided at
> http://thebrotherswisp.com/index.php/geo-and-vpn with the exception of
> https://www.ipligence.com/geolocation which seems to be junk serving junk
> data.
> 
> So far, Akamai is the only provider we are seeing issues with.
> 
> Thanks for the feedback.
> 
> On 01/17/20 14:24 -0500, Jared Mauch wrote:
> > I recommend against using 1.1.1.1 as that may direct you to far away 
> > locations.
> > I can help you off-list further.
> > 
> > - Jared
> > 
> > > On Jan 17, 2020, at 2:20 PM, Josh Luthman via Outages 
> > >  wrote:
> > > 
> > > The IPs you're coming from, look what they say at the various geolocation 
> > > services here:
> > > 
> > > http://thebrotherswisp.com/index.php/geo-and-vpn/
> > > 
> > > Also, have you tried a local DNS server and not Google/Cloudflare Anycast 
> > > ones?
> > > 
> > > Josh Luthman
> > > 
> > > On Fri, Jan 17, 2020 at 2:13 PM Dan White via Outages 
> > >  wrote:
> > > We are seeing some geolocation oddities with a couple of websites hosted 
> > > at, or resolved by, Akamai.
> > > 
> > > Two hostnames I'm looking at include:
> > > 
> > > business.officedepot.com
> > > www.teacherspayteachers.com
> > > 
> > > 'A' records for both have a TTL of 20 seconds, and randomly resolve to 
> > > IPs in the US, and Europe (we are in the US/Oklahoma).
> > > 
> > > For example, business.officedepot.com is resolving to both 96.17.32.230 
> > > and
> > > 88.221.37.87 (at different times). I've verified the same inconsistent
> > > behavior against 8.8.8.8 and 1.1.1.1.
> > > 
> > > This is causing issues for an educational customer of ours. Is anyone 
> > > aware
> > > of issues within Akamai that may be causing them to source content from
> > > Europe?
> 
> -- 
> Dan White
> BTC Broadband
> Network Admin Lead
> Ph  918.366.0248 (direct)   main: (918)366-8000
> Fax 918.366.6610email: dwh...@mybtc.com
> http://www.btcbroadband.com

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Akamai Geolocation Concerns

[Jared Mauch via Outages]

I recommend against using 1.1.1.1 as that may direct you to far away locations.

I can help you off-list further.

- Jared

> On Jan 17, 2020, at 2:20 PM, Josh Luthman via Outages  
> wrote:
> 
> The IPs you're coming from, look what they say at the various geolocation 
> services here:
> 
> http://thebrotherswisp.com/index.php/geo-and-vpn/ 
> 
> Also, have you tried a local DNS server and not Google/Cloudflare Anycast 
> ones? 
> 
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> 
> On Fri, Jan 17, 2020 at 2:13 PM Dan White via Outages  
> wrote:
> We are seeing some geolocation oddities with a couple of websites hosted at, 
> or resolved by, Akamai.
> 
> Two hostnames I'm looking at include:
> 
> business.officedepot.com
> www.teacherspayteachers.com
> 
> 'A' records for both have a TTL of 20 seconds, and randomly resolve to IPs in 
> the US, and Europe (we are in the US/Oklahoma).
> 
> For example, business.officedepot.com is resolving to both 96.17.32.230 and
> 88.221.37.87 (at different times). I've verified the same inconsistent
> behavior against 8.8.8.8 and 1.1.1.1.
> 
> This is causing issues for an educational customer of ours. Is anyone aware
> of issues within Akamai that may be causing them to source content from
> Europe?
> 
> -- 
> Dan White
> BTC Broadband
> Network Admin Lead
> Ph  918.366.0248 (direct)   main: (918)366-8000
> Fax 918.366.6610email: dwh...@mybtc.com
> http://www.btcbroadband.com
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Fast.com down

[Jared Mauch via Outages]

Do you have data on the Akamai issue?  If so, I’d like to look into it.  I 
would be surprised if we had issues.  I see fast.com is on Neustar for DNS:

;; Received 763 bytes from 192.5.6.30#53(a.gtld-servers.net) in 70 ms

fast.com.   1800IN  NS  pdns154.ultradns.net.
fast.com.   1800IN  NS  pdns154.ultradns.com.
fast.com.   1800IN  NS  pdns154.ultradns.biz.
fast.com.   1800IN  NS  pdns154.ultradns.org.

- Jared

> On Oct 28, 2019, at 3:33 PM, Bob Strecansky via Outages  
> wrote:
> 
> This seems to be related to an Akamai change that was made at some point 
> today.  Anything that is zone apex mapped at Akamai has been having problems 
> for the last hour or so.
> 
> On Mon, Oct 28, 2019 at 3:31 PM Matt Hoppes via Outages  
> wrote:
> Fast.com appears to be down from multiple providers in Pennsylvania. 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] sales critter Ben Reynolds

[Jared Mauch via Outages]

This person spammed all sorts of weird addresses for projects that I run (eg: 
Open Resolver) so clearly has no clue.

That domain is currently banned on my mail server so it’s unlikely they can get 
anything else through.

- Jared

> On Oct 19, 2019, at 12:37 PM, PE via Outages  wrote:
> 
> 
> I'm sure many of us got email from this critter, since he seems to be
> scraping posts to get email addrs to annoy with spam.
> 
> List moderator: can you please yank them?
> 
> List: got another company/critter to add to the "never do business with
> ever list"...
> 
> --
> <1.eml>___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Ticketing system

[Jared Mauch via Outages]

Please don’t do that. I just unsubscribed it. 

Sent from my iCar
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] [Admin] Testing 1, 2, 3...

[Jared Mauch via Outages]

Sigh. Unstuck. Not sure why it did not start on reboot. It’s there...

Sent from my iCar

> On Jul 2, 2019, at 4:00 PM, Jay Ashworth  wrote:
> 
> We had a report there has been traffic and it did not make the list. This is 
> a test.
> -- jra
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Google Mail rejecting IPv6 originated mails

[Jared Mauch via Outages]

I since figured out that it’s people whose personal domain is forwarding to 
google and their server must be setup poorly, but they aren’t seeing the bounce.

- Jared

> On May 23, 2019, at 7:57 AM, Marlen Caemmerer  wrote:
> 
> Hey,
> 
> as far as I remember this issue comes up once in a while since several years.
> There are postfix configs out there to adress this issue by sending via IPv4 
> to Gmail.
> 
> https://tanguy.ortolo.eu/blog/article109/google-ipv6-smtp-restrictions
> or
> https://blog.hqcodeshop.fi/archives/122-Fixing-Googles-new-IPv6-mail-policy-with-Postfix.html
> 
> Hth
> 
> On Thu, 23 May 2019, Jared Mauch via Outages wrote:
> 
>> Date: Thu, 23 May 2019 13:31:28
>> From: Jared Mauch via Outages 
>> Reply-To: Jared Mauch 
>> To: Robbie Trencheny via Outages 
>> Subject: [outages] Google Mail rejecting IPv6 originated mails
>> Anyone else having this problem?  Their system is periodically rejecting 
>> mails and reporting it as invalid IPv6 PTR but there have been no changes in 
>> my systems, and the help pages at google do not actually provide any 
>> additional guidance.
>> 
>> - Jared
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>> 

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Google Mail rejecting IPv6 originated mails

[Jared Mauch via Outages]

Anyone else having this problem?  Their system is periodically rejecting mails 
and reporting it as invalid IPv6 PTR but there have been no changes in my 
systems, and the help pages at google do not actually provide any additional 
guidance.

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Facebook and Instagram are down for some users

[Jared Mauch via Outages]

I need to figure out why rc.local doesn’t run at boot :-(
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Akamai significant increase in traffic

[Jared Mauch via Outages]

Did this correspond with the Fortnite 8 release?  (My teenager says he’s done 
with it, but I think it dropped today).

- Jared

> On Feb 28, 2019, at 9:41 AM, Erich Kaiser via Outages  
> wrote:
> 
> We have seen a significant increase in Akamai traffic as of a few hours ago, 
> anyone else seeing this and have any insight?  Most of the traffic is coming 
> out of Atlanta.
> 
> 
> 
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] TLS related failures

[Jared Mauch via Outages]

Apologies.  I adjusted the mail server to do more TLS and mailman had issues 
with that.

Should be better now.

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] usno.navy.mil disappeared?

[Jared Mauch via Outages]

Looks like either a DNSSEC problem or their servers are down/unreachable.

http://dnsviz.net/d/usno.navy.mil/dnssec/

- Jared

> On Dec 21, 2018, at 9:48 AM, Steven Surdock via Outages  
> wrote:
> 
> Anyone know what is going on with The US Naval Observatory?   Usno.navy.mil 
> appears to be completely unresolvable.  Affecting access to tick and tock 
> time servers.
> 
> $ nslookup -type=NS usno.navy.mil 8.8.8.8
> *** google-public-dns-a.google.com can't find usno.navy.mil: Server failed
> Server:  google-public-dns-a.google.com
> Address:  8.8.8.8
> 
> -Steve S.
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Slack outage

[Jared Mauch via Outages]

I’m starting to see things reconnect.


> On Jan 9, 2018, at 2:23 PM, Laurent Dumont via Outages  
> wrote:
> 
> Their status page might be down too - or just extremely slow.
> 
> On Tue, Jan 9, 2018 at 2:21 PM, Carlos Alvarez via Outages 
>  wrote:
> Same here, from two ISPs in Phoenix, AZ.
> 
> 
> On Tue, Jan 9, 2018 at 12:19 PM, Josh Luthman via Outages 
>  wrote:
> The community says "Server Error" and links the status page, which doesn't 
> report anything at this time.
> 
> https://status.slack.com
> 
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> 
> 
> 
> -- 
> Carlos Alvarez
> 602-368-6403
> 
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> 
> 
> 
> -- 
> Laurent Dumont
> Spécialiste réseau / Network specialist
> Fibrenoire - www.fibrenoire.ca
> A: 550​, avenue Beaumont, ​bureau 320, Montréal (Québec) H3N 1V1
> T. 514 907-3002 x132
> C.438-394-9603
> ldum...@fibrenoire.ca
> Twitter: Twitter Fibrenoire
> 
> --Note de confidentialité--
> 
> Ce message électronique et toutes pièces jointes contiennent des informations 
> privilégiées et confidentielles. Si vous n'êtes pas le destinataire prévu, il 
> est strictement interdit d'utiliser, de copier ou divulguer les informations 
> ainsi transmises; merci d'en aviser l'expéditeur et de les supprimer de votre 
> ordinateur.
> 
> --Confidentiality Note--
> 
> This email message and any attachments contain privileged and confidential 
> information. If you are not the intended recipient, your use, copying or 
> disclosure of this information is prohibited; please contact the sender and 
> delete the material from your computer.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Akamai DNS issues

[Jared Mauch via Outages]

They seem to be back, but I’m curious as to what is going on.

- Jared

> On Jan 24, 2017, at 9:32 PM, frnkblk--- via Outages  
> wrote:
> 
> Thanks, now feeling relieved that I’m not going crazy.  We saw the first 
> event at 1:07 pm Pacific.
>  
> Just saw another wave again (www.java.com, www.oracle.com, wireless.att.com, 
> www.comcast.net, www.apple.com, www.att.com, www.microsoft.com, etc).
>  
> Frank
>  
> From: Ca By [mailto:cb.li...@gmail.com] 
> Sent: Tuesday, January 24, 2017 8:04 PM
> To: frnk...@iname.com; outages@outages.org
> Subject: Re: [outages] Akamai DNS issues
>  
> Yes, i have also seen akamai removing  records today 
>  
> It started at 1:15 pacific and is still going on. 
>  
> CB
>  
> On Tue, Jan 24, 2017 at 5:31 PM frnkblk--- via Outages  
> wrote:
>> Starting this afternoon our monitoring system recorded several 5 to 15
>> 
>> minute outages accessing some IPv6-enabled Akamai-delivered sites.  In each
>> 
>> case, where I could catch the problem while it happened, I could not resolve
>> 
>> the  to an IPv6 address.
>> 
>> 
>> 
>> We and our upstream provider have over 10 servers of different flavors, and
>> 
>> they are all exhibiting this issue, though not Google or OpenDNS' servers.
>> 
>> 
>> 
>> Just two examples are below, but in each case, I get a CNAME back, but it
>> 
>> doesn't show an IPv6 address.  And also five from
>> 
>> https://www.digwebinterface.com/?type=SOA=on=8.8.4.4
>> 
>> =all  And also here:
>> 
>> https://www.whatsmydns.net/#/www.apple.com and here
>> 
>> http://www.preshweb.co.uk/cgi-bin/dns-propagation-tracker.pl?domain=www.appl
>> 
>> e.com==random==
>> 
>> 
>> 
>> Anyone else seeing the same?
>> 
>> 
>> 
>> I emailed Akamai's tix address, but have not gotten anything back other than
>> 
>> the auto-responder.
>> 
>> 
>> 
>> Frank
>> 
>> 
>> 
>> 
>> 
>> ; <<>> DiG 9.7.3 <<>>  www.apple.com @10.20.0.10
>> 
>> ;; global options: +cmd
>> 
>> ;; Got answer:
>> 
>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9110
>> 
>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 0
>> 
>> 
>> 
>> ;; QUESTION SECTION:
>> 
>> ;www.apple.com. IN  
>> 
>> 
>> 
>> ;; ANSWER SECTION:
>> 
>> www.apple.com.  483 IN  CNAME   www.apple.com.edgekey.net.
>> 
>> www.apple.com.edgekey.net. 3193 IN  CNAME
>> 
>> www.apple.com.edgekey.net.globalredir.akadns.net.
>> 
>> www.apple.com.edgekey.net.globalredir.akadns.net. 19 IN CNAME
>> 
>> e6858.dsce9.akamaiedge.net.
>> 
>> 
>> 
>> ;; AUTHORITY SECTION:
>> 
>> dsce9.akamaiedge.net.   107 IN  SOA n0dsce9.akamaiedge.net.
>> 
>> hostmaster.akamai.com. 1485306230 1000 1000 1000 1800
>> 
>> 
>> 
>> ;; Query time: 0 msec
>> 
>> ;; SERVER: 10.20.0.10#53(10.20.0.10)
>> 
>> ;; WHEN: Tue Jan 24 19:18:43 2017
>> 
>> ;; MSG SIZE  rcvd: 228
>> 
>> 
>> 
>> ; <<>> DiG 9.7.3 <<>>  www.apple.com @96.31.0.32
>> 
>> ;; global options: +cmd
>> 
>> ;; Got answer:
>> 
>> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58259
>> 
>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 0
>> 
>> 
>> 
>> ;; QUESTION SECTION:
>> 
>> ;www.apple.com. IN  
>> 
>> 
>> 
>> ;; ANSWER SECTION:
>> 
>> www.apple.com.  1073IN  CNAME   www.apple.com.edgekey.net.
>> 
>> www.apple.com.edgekey.net. 3584 IN  CNAME
>> 
>> www.apple.com.edgekey.net.globalredir.akadns.net.
>> 
>> www.apple.com.edgekey.net.globalredir.akadns.net. 446 IN CNAME
>> 
>> e6858.dsce9.akamaiedge.net.
>> 
>> 
>> 
>> ;; AUTHORITY SECTION:
>> 
>> dsce9.akamaiedge.net.   257 IN  SOA n0dsce9.akamaiedge.net.
>> 
>> hostmaster.akamai.com. 1485306380 1000 1000 1000 1800
>> 
>> 
>> 
>> ;; Query time: 0 msec
>> 
>> ;; SERVER: 96.31.0.32#53(96.31.0.32)
>> 
>> ;; WHEN: Tue Jan 24 19:18:43 2017
>> 
>> ;; MSG SIZE  rcvd: 228
>> 
>> 
>> 
>> 
>> 
>> ===
>> 
>> 
>> 
>> https://www.digwebinterface.com/?hostnames=www.apple.com=
>> 
>> nd=on=on=8.8.4.4=all=
>> 
>> 
>> 
>> www.apple.com@168.95.1.1 (HiNet (TW)):
>> 
>> dig  +noadditional +noquestion +nocomments +nocmd +nostats
>> 
>> www.apple.com. @168.95.1.1
>> 
>> www.apple.com.  1494IN  CNAME   www.apple.com.edgekey.net.
>> 
>> www.apple.com.edgekey.net. 6888 IN  CNAME
>> 
>> www.apple.com.edgekey.net.globalredir.akadns.net.
>> 
>> www.apple.com.edgekey.net.globalredir.akadns.net. 3289 IN CNAME
>> 
>> e6858.dsce9.akamaiedge.net.
>> 
>> dsce9.akamaiedge.net.   191 IN  SOA n0dsce9.akamaiedge.net.
>> 
>> hostmaster.akamai.com. 1485306682 1000 1000 1000 1800
>> 
>> 
>> 
>> www.apple.com@195.129.12.122 (UUNET (CH)):
>> 
>> dig  +noadditional +noquestion +nocomments +nocmd +nostats
>> 
>> www.apple.com. @195.129.12.122
>> 
>> www.apple.com.  387 IN  CNAME   www.apple.com.edgekey.net.
>> 
>> www.apple.com.edgekey.net. 12797 IN CNAME
>> 
>> 

Re: [outages] NTT to Level3 routing issue in NYC?

[Jared Mauch via Outages]

Yeah, so totally unreachable from any of the ATLAS probes, but measurement is 
still running.

https://atlas.ripe.net/measurements/6939080/#!probes

- Jared

> On Nov 23, 2016, at 2:08 PM, Doug Roberts  wrote:
> 
> Here's what I see from here:
> 
> https://puu.sh/ss76o/27b9de01e6.png
> 
> 
> 
> 
> 
> 
> 
> 
> --
> Doug Roberts
> d...@dougr.com
> 251-308-4040 (Mobile)
> 228-254-3035 (Biloxi)
> 850-361-3300 (Pensacola)
> 334-543-0222 (Montgomery)
> 251-308-1333 (FAX)
> 
> CONFIDENTIALITY NOTICE: The contents of this e-mail message and any 
> attachments are intended for the addressee(s) named in this message. This 
> communication may contain confidential and/or privileged information and may 
> be legally protected from disclosure. If you are not the intended recipient 
> of this message, or if this message has been addressed to you in error, 
> please immediately alert the sender by reply e-mail and then delete this 
> message and its attachments. Do not disclose the contents or take any action 
> in reliance upon the information contained in this communication or any 
> attachments. Do not deliver, distribute or copy this message and/or any 
> attachments and, if you are not the intended recipient, you are hereby 
> notified that any use, dissemination, copying, or storage of this message or 
> its attachments is strictly prohibited.
> 
> On Wed, Nov 23, 2016 at 1:03 PM, Jared Mauch  wrote:
> 
> > On Nov 23, 2016, at 1:33 PM, Doug Roberts via Outages  
> > wrote:
> >
> > NTT is indeed having issues. Pinging from ATL to us here in Alabama fails:
> >
> > Query Results:
> > Router: Atlanta, GA - US
> > Command: ping 45.31.x.x
> >
> > Sending 5, 100-byte ICMP Echos to 45.31.217.222, timeout is 2 seconds:
> > .
> > Success rate is 0 percent (0/5)
> >
> 
> Do you have details?
> 
> I’m showing atlas working fine measuring things, but others have asked me as 
> well about issues, so continuing to look.
> 
> https://atlas.ripe.net/measurements/6939072/#!probes
> 
> More details of brokenness would be helpful either to me or to the NTT NOC.
> 
> - Jared
> 
> 

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] NTT to Level3 routing issue in NYC?

[Jared Mauch via Outages]

I see that going to AT not Level3, let me keep looking, eg:

r02.atlnga05.us.bb-re1> traceroute 107.207.236.1
traceroute to 107.207.236.1 (107.207.236.1), 30 hops max, 40 byte packets
 1  ge-100-0-0-1.r04.atlnga05.us.bb.gin.ntt.net (129.250.4.88)  1.027 ms  0.807 
ms  0.760 ms
 2  192.205.36.157 (192.205.36.157)  4.377 ms  2.789 ms  3.655 ms
 3  cr2.attga.ip.att.net (12.122.117.122)  11.292 ms  27.985 ms  10.449 ms
 MPLS Label=24148 CoS=0 TTL=1 S=1
 4  12.122.163.69 (12.122.163.69)  10.736 ms  11.141 ms  12.007 ms
 5  *^C
{master}

- Jared

> On Nov 23, 2016, at 2:08 PM, Doug Roberts  wrote:
> 
> Here's what I see from here:
> 
> https://puu.sh/ss76o/27b9de01e6.png
> 
> 
> 
> 
> 
> 
> 
> 
> --
> Doug Roberts
> d...@dougr.com
> 251-308-4040 (Mobile)
> 228-254-3035 (Biloxi)
> 850-361-3300 (Pensacola)
> 334-543-0222 (Montgomery)
> 251-308-1333 (FAX)
> 
> CONFIDENTIALITY NOTICE: The contents of this e-mail message and any 
> attachments are intended for the addressee(s) named in this message. This 
> communication may contain confidential and/or privileged information and may 
> be legally protected from disclosure. If you are not the intended recipient 
> of this message, or if this message has been addressed to you in error, 
> please immediately alert the sender by reply e-mail and then delete this 
> message and its attachments. Do not disclose the contents or take any action 
> in reliance upon the information contained in this communication or any 
> attachments. Do not deliver, distribute or copy this message and/or any 
> attachments and, if you are not the intended recipient, you are hereby 
> notified that any use, dissemination, copying, or storage of this message or 
> its attachments is strictly prohibited.
> 
> On Wed, Nov 23, 2016 at 1:03 PM, Jared Mauch  wrote:
> 
> > On Nov 23, 2016, at 1:33 PM, Doug Roberts via Outages  
> > wrote:
> >
> > NTT is indeed having issues. Pinging from ATL to us here in Alabama fails:
> >
> > Query Results:
> > Router: Atlanta, GA - US
> > Command: ping 45.31.x.x
> >
> > Sending 5, 100-byte ICMP Echos to 45.31.217.222, timeout is 2 seconds:
> > .
> > Success rate is 0 percent (0/5)
> >
> 
> Do you have details?
> 
> I’m showing atlas working fine measuring things, but others have asked me as 
> well about issues, so continuing to look.
> 
> https://atlas.ripe.net/measurements/6939072/#!probes
> 
> More details of brokenness would be helpful either to me or to the NTT NOC.
> 
> - Jared
> 
> 

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] he.net Authoritative DNS Down

[Jared Mauch via Outages]

[disclosure further down]

With the recent other DNS related outages, I'd like to remind people
of the wonderful RFC 2182.

https://www.rfc-editor.org/rfc/rfc2182.txt

Basically: don't put all your eggs behind or in the same ASN,
prefix or company.

If you need off-net secondary there are many free services
out there.

Also: consider IPv6 as providng a secondary path of
resilience.  If IPv4 is down or under attack (eg: twitter
this week, IPv4 was down, but IPv6 routes were not impacted)
let the protocols work for you.

- jared

[i operate a free secondary dns service and welcome people to
 leverage it for your needs, https://puck.nether.net/dns/ ]
 
On Tue, Nov 08, 2016 at 08:39:42AM -0600, Matthew Beckwell via Outages wrote:
> For the most part, I'm not seeing responses from
> ns1.he.net
> ns2.he.net
> ns3.he.net
> ns4.he.net
> ns5.he.net
> 
> In Minneapolis. I did see a single response from ns5.he.net sneak in at
> 1482 msec
> 
> ~Matthew

> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages


-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] EDNS Issues with bell.ca ?

[Jared Mauch via Outages]

> On Oct 7, 2016, at 4:05 PM, Zachary McGibbon via Outages 
>  wrote:
> 
> Hi, we started noticing DNS issues today with the bell.ca domain.  From our 
> troubleshooting, it looks like Bell might have turned on EDNS?

What do you mean, the handling of EDNS 0, or EDNS > 0 ?

> Is anyone from Bell Canada on this list and can comment?
> 
> I did tests from our network here at McGill University and all my lookups 
> fail.  I also tried on my home connection (Videotron) and that works fine.
> 
> From our DNS server:
> 
> $ dig +trace www.bell.ca
> 
> ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-25.P1.el5_11.8 <<>> +trace www.bell.ca
^

While RedHat does a lot of back porting of fixes and makes their version 
numbers often meaningless as a result.  Do you know what that translates to for 
equivalency?  Can you use something a bit more modern?

Bind 9.3 went EOL in January 2009, so I would try with something else.

If you’re on RHEL5 support may have already ended unless you’re out to the 
March 2017 date.

Due to this, please try with something else perhaps?

also, consider posting on dns-operations list, you may find a few more insights 
there.

https://ednscomp.isc.org/ednscomp/8a3e3c1092

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Big fiber cut due to manhole(s) explosion(s) in NYC/Union Square

[Jared Mauch via Outages]

> On Sep 30, 2016, at 9:04 AM, Eric Dugas via Outages  
> wrote:
> 
> 5 days + outage on a land cable... that's a first for me.
> 


https://en.wikipedia.org/wiki/Howard_Street_Tunnel_fire#Impact_on_other_services

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] that's exciting!

[Jared Mauch via Outages]

I attempt to minimize the spam/malware to the list
using a variety of tools.  Apologies for the issue and thank
you for the reports.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] AWS issues

[Jared Mauch via Outages]

Basically if your provider took this path they need to learn basic BGP hygiene 
101.

Filter out private ASNs. 

https://twitter.com/jaredmauch/status/722179208135331840

Jared Mauch

> On Apr 22, 2016, at 7:33 PM, Nick Buraglio via Outages  
> wrote:
> 
> I suspect this was due to a large bgp event today.  
> http://www.bgpmon.net/large-hijack-affects-reachability-of-high-traffic-destinations/
> 
>> On Friday, April 22, 2016, Charles Sprickman via Outages 
>>  wrote:
>> I can reach east-1 from HE in NYC and VZ in NNJ.
>> 
>> I’m relatively new to FiOS, is this path normal?
>> 
>> frankentosh:~ spork$ traceroute !$
>> traceroute monitor.xxxs.com
>> traceroute to monitor.xxxs.com (54.85.105.220), 64 hops max, 52 byte packets
>>  1  lo0-100.nwrknj-vfttp-312.verizon-gni.net (108.53.194.1)  1.406 ms  1.233 
>> ms  1.904 ms
>>  2  t1-8-0-6.nwrknj-lcr-22.verizon-gni.net (100.41.220.233)  5.804 ms
>> t1-6-0-0.nwrknj-lcr-22.verizon-gni.net (100.41.220.228)  3.732 ms
>> t1-8-0-6.nwrknj-lcr-21.verizon-gni.net (100.41.220.235)  6.286 ms
>>  3  * * *
>>  4  0.ae6.br1.nyc1.alter.net (140.222.228.131)  4.642 ms
>> 0.ae5.br1.nyc1.alter.net (140.222.228.107)  4.304 ms
>> 0.ae6.br1.nyc1.alter.net (140.222.228.131)  4.582 ms
>>  5  pax-brdr-01.inet.qwest.net (63.235.40.53)  4.864 ms  2.956 ms  13.092 ms
>>  6  dca2-edge-01.inet.qwest.net (67.14.36.10)  9.539 ms  10.112 ms  11.018 ms
>>  7  72.165.86.74 (72.165.86.74)  9.879 ms
>> 65.120.78.82 (65.120.78.82)  9.580 ms
>> 67.133.224.206 (67.133.224.206)  8.967 ms
>>  8  * * *
>>  9  * * *
>> 10  54.239.110.245 (54.239.110.245)  32.092 ms
>> 54.239.110.235 (54.239.110.235)  11.426 ms
>> 54.239.110.233 (54.239.110.233)  40.425 ms
>> 
>> I would have assumed AS701 would peer directly with AWS, that has to 
>> represent at least a quarter of FiOS traffic.
>> 
>> Charles
>> 
>> -- 
>> Charles Sprickman
>> NetEng/SysAdmin
>> Bway.net - New York's Best Internet www.bway.net
>> sp...@bway.net - 212.982.9800
>> 
>> 
>> 
>>> On Apr 22, 2016, at 1:48 PM, Ben Burns via Outages  
>>> wrote:
>>> 
>>> Can confirm inbound connectivity still down for us-east-1.
>>> 
 On 16-04-22 11:42 AM, Sebastian J. Orsini II via Outages wrote:
 That correlates with the timing of some really weird DNS issues for us. 
 (ISP=Hargray)  
 
 Ie some webpages would load some wouldn't.   we lost 1/3 of our vpns and 
 about half our public ips were not pingable from some different isps.
 
 Then it just magically cleared up.   Felt like about 4   ,months 
 ago when level3 had DNS issues?
 
 I dunno.   Its all okay now..
 
 
 
> On Fri, Apr 22, 2016 at 1:29 PM, John Kinsella via Outages 
>  wrote:
> Back to normal as others have said. Lasted 10ish minutes? Normal trace 
> below. Not sure if HE issue or Amazon dropped an announcement maybe...
> 
>  Packets   
> Pings
>  Host  Loss%   Snt   Last   Avg  
> Best  Wrst StDev
>  1. 10.0.1.10.0% 11.1   1.1   
> 1.1   1.1   0.0
>  2. 10.0.1.49   0.0% 12.2   2.2   
> 2.2   2.2   0.0
>  3. 173.247.205.38  0.0% 13.7   3.7   
> 3.7   3.7   0.0
>  4. x.196.247.173.web-pass.com  0.0% 14.9   4.9   
> 4.9   4.9   0.0
>  5. x.196.247.173.web-pass.com  0.0% 13.7   3.7   
> 3.7   3.7   0.0
>  6. equinix01-sfo5.amazon.com   0.0% 15.6   5.6   
> 5.6   5.6   0.0
>  7. 54.240.242.112  0.0% 1   37.3  37.3  
> 37.3  37.3   0.0
>  8. 54.240.242.115  0.0% 1   28.6  28.6  
> 28.6  28.6   0.0
>  9. 205.251.229.189 0.0% 1   42.8  42.8  
> 42.8  42.8   0.0
> 10. 54.239.42.6 0.0% 1   25.2  25.2  
> 25.2  25.2   0.0
> 11. 205.251.232.130 0.0% 1   25.1  25.1  
> 25.1  25.1   0.0
> 12. 205.251.232.143 0.0% 1   25.0  25.0  
> 25.0  25.0   0.0
> 13. 54.239.48.179   0.0% 1   26.8  26.8  
> 26.8  26.8   0.0
> 14. ???
> 
> 
>> On Apr 22, 2016, at 10:22 AM, Hal Ponton  wrote:
>> 
>> I've also been seeing some issues to youtube, twitter, amazon and reddit 
>> from the UK but they just seem to have been kicked back into life.
>> -- 
>> --
>> Regards,
>> 
>> Hal Ponton
>> Senior Network Engineer
>> 
>> Buzcom / FibreWiFi
>> 
>> 
>> 
>> 
>>> John Kinsella via Outages 22 April 2016 at 

Re: [outages] he.net Authoritative DNS down

[Jared Mauch via Outages]

Some websites like dynstatus.com are still hosed it seems.

- Jared

> On Mar 21, 2016, at 1:12 PM, Scott Howard via Outages  
> wrote:
> 
> Looked to have come back up about 35 mins ago for static entries, and 
> sometime in the past 30 minutes for dynamic entries.
> 
> Their portal is still down - "undergoing some maintenance".
> 
>   Scott
> 
> 
> On Mon, Mar 21, 2016 at 9:26 AM, Reid Fishler via Outages 
>  wrote:
> Hurricane is aware of an issue, and we are addressing it.
> 
> Reid
> 
> On Mon, Mar 21, 2016 at 12:23 PM, Frank Bulk via Outages 
>  wrote:
> I called their NOC (voice and fax) and I get a “all circuits are busy now”.  
> I trust they are aware by now and scrambling.
> 
>  
> 
> Frank
> 
>  
> 
> From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Matthew 
> Beckwell via Outages
> Sent: Monday, March 21, 2016 11:17 AM
> To: Outages@outages.org
> Subject: [outages] he.net Authoritative DNS down
> 
>  
> 
> Currently getting empty results on queries to:
> 
>  
> 
> ns1.he.net
> 
> ns2.he.net
> 
> ns3.he.net
> 
> ns4.he.net
> 
> ns5.he.net
> 
>  
> 
> ~Matthew
> 
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> 
> 
> 
> -- 
> Reid Fishler
> Director
> Hurricane Electric
> +1-510-580-4178
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] he.net Authoritative DNS down

[Jared Mauch via Outages]

Have an off-carrier secondary like I provide here:

http://puck.nether.net/dns/

Note: Free

You can also use something like the ESGOB secondary DNS service which has an 
API front-end.  I need to add something similar.



> On Mar 21, 2016, at 12:18 PM, Daniel J. Grinkevich via Outages 
>  wrote:
> 
> This explains why I can't dns my domains.
> 
> On Mar 21, 2016 12:17, "Matthew Beckwell via Outages"  
> wrote:
> Currently getting empty results on queries to:
> 
> ns1.he.net
> ns2.he.net
> ns3.he.net
> ns4.he.net
> ns5.he.net
> 
> ~Matthew
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Junk Message Apology

[Jared Mauch via Outages]

Apologies for the spam overnight.  Rules are now in place to block these 
messages.

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] comcast chicago blackholing

[Jared Mauch via Outages]

This is both an issue over V6 and V4, eg:

Jareds-Mac-mini:~% traceroute puck
traceroute to puck.nether.net (204.42.254.5), 64 hops max, 52 byte packets
 1  10.0.0.1 (10.0.0.1)  0.603 ms  0.418 ms  0.403 ms
 2  173-167-0-110-michigan.hfc.comcastbusiness.net (173.167.0.110)  2.044 ms  
2.019 ms  2.179 ms
 3  96.120.40.141 (96.120.40.141)  17.399 ms  12.408 ms  9.639 ms
 4  68.85.85.37 (68.85.85.37)  14.371 ms  8.378 ms  15.355 ms
 5  te-0-8-0-6-ar02.pontiac.mi.michigan.comcast.net (68.87.191.153)  14.025 ms
te-0-8-0-8-ar02.pontiac.mi.michigan.comcast.net (69.139.255.69)  14.749 ms  
18.706 ms
 6  be-33668-cr02.350ecermak.il.ibone.comcast.net (68.86.90.45)  21.955 ms  
20.456 ms  19.259 ms
 7  he-0-16-0-0-pe03.350ecermak.il.ibone.comcast.net (68.86.88.146)  18.333 ms  
17.245 ms  16.952 ms
^C

Jareds-Mac-mini:~% traceroute6 puck
traceroute6 to puck.nether.net (2001:418:3f4::5) from 
2601:401:3:6a00:ad42:ce75:8a0:c370, 64 hops max, 12 byte packets
 1  2601:401:3:6a00:20f:f7ff:feab:1ec0  1.135 ms  0.965 ms  0.948 ms
 2  2601:401:3:6a00:4694:fcff:fecd:64bc  3.710 ms  3.544 ms  21.221 ms
 3  2001:558:6007:24::1  30.051 ms  10.909 ms  10.868 ms
 4  te8-2-ur02.wannarbor.mi.michigan.comcast.net  10.664 ms  10.451 ms  10.570 
ms
 5  xe-11-1-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  14.826 ms
te-0-8-0-5-ar02.taylor.mi.michigan.comcast.net  13.890 ms  13.179 ms
 6  te-0-7-0-13-ar01.pontiac.mi.michigan.comcast.net  18.917 ms  15.503 ms
te-0-8-0-11-ar01.pontiac.mi.michigan.comcast.net  13.686 ms
 7  be-33668-cr01.350ecermak.il.ibone.comcast.net  22.316 ms  22.014 ms  28.086 
ms
 8  hu-0-16-0-1-pe03.350ecermak.il.ibone.comcast.net  20.933 ms  20.740 ms  
20.831 ms
^C

- Jared


On Tue, Nov 17, 2015 at 10:02:58AM -0700, John Neiberger wrote:
> Can you give me some more details please? I'll take a look.
> 
> Thanks,
> John
> 
> On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
> <outages@outages.org> wrote:
> >
> > Anyone know what is going on here?  It seems to have started
> > around 1600 utc and is still ongoing.
> >
> > - Jared
> >
> > --
> > Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
> > clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
> > ___
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Outage of (puck.nether.net)?

[Jared Mauch via Outages]

On Mon, Nov 02, 2015 at 02:14:36PM -0800, Hugo Slabbert via Outages wrote:
> Doesn't appear to be just me:
> 
> hslabbert@sg-ws-hugol:~$ wget puck.nether.net
> --2015-11-02 14:11:25--  http://puck.nether.net/
> Resolving puck.nether.net (puck.nether.net)... 2001:418:3f4::5, 204.42.254.5
> Connecting to puck.nether.net (puck.nether.net)|2001:418:3f4::5|:80... 
> failed: Connection refused.
> Connecting to puck.nether.net (puck.nether.net)|204.42.254.5|:80... failed: 
> Connection refused.

Fixed.

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Outages.org spam!

[Jared Mauch via Outages]

Wonder how this might be related to the Joomla
stuff as many seem to be webhosting related.

http://arstechnica.com/security/2015/10/joomla-bug-puts-millions-of-websites-at-risk-of-remote-takeover-hacks/

- Jared

On Sat, Oct 24, 2015 at 06:57:34PM -0700, Grant Ridder via Outages wrote:
> Its not just outages.  NANOG and an IPv6 op email list are getting hit too
> 
> -Grant
> 
> On Sat, Oct 24, 2015 at 6:50 PM, Jim Witherell via Outages <
> outages@outages.org> wrote:
> 
> > Think the guys managing the listserv are on top of it?
> >
> > Sent from Yahoo Mail on Android
> > 
> >
> > ___
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >
> >

> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages


-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Outages.org spam!

[Jared Mauch via Outages]

BTW, these should be blocked now from here.

I’ll try to keep an eye out for the next hour if the pattern changes at all.

- Jared

> On Oct 24, 2015, at 9:50 PM, Jim Witherell via Outages  
> wrote:
> 
> Think the guys managing the listserv are on top of it? 
> 
> Sent from Yahoo Mail on Android
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] CenturyLink fiber cut between Modesto, CA and San Jose, CA this AM.. Start time 4:26AM PST

[Jared Mauch via Outages]

Seems to be...

Jared Mauch

 On Jun 30, 2015, at 3:56 PM, Hugo Slabbert via Outages outages@outages.org 
 wrote:
 
 Related?
 
 http://arstechnica.com/tech-policy/2015/06/fbi-baffled-over-wave-of-nighttime-fiber-optic-cable-vandalism/
 
 --
 Hugo
 
 
 
  Peter Kranz via Outages wrote 
 
 This appears to be part of a larger act of intentional sabotage:
 
 We have just been advised that crews have been removed from the area while
 law enforcement investigates the damage, which was caused by vandalism.
 Three enclosures were damaged impacting three different carriers. There is
 no time given for when they will be allowed to begin work again,
 subsequently, there is still no ETR.
 
 
 
 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages
 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Nation wide t-mobile inbound and outbound

[Jared Mauch via Outages]

 On Jun 25, 2015, at 7:49 AM, Mark Tinka via Outages outages@outages.org 
 wrote:
 
 I suppose one would say that since the outages list is part of the NANOG
 umbrella, one could infer that nationwide means the U.S. However, the
 NA in NANOG is North America... Canada and all, so...

The outages list is hosted by me at puck.nether.net on behalf
and at the request of Virendra Rode fyi.

BTW, not all americans are like this either:

http://www.finlandforthought.net/wp-content/simpsons_shock.jpg

- Jared
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] NTT instability in Dallas and Miami

[Jared Mauch via Outages]

Sounds like VNOC-1-1306038169

Working to isolate the hardware fault with the vendor. 

Jared Mauch

 On Feb 8, 2015, at 5:21 AM, Justin via Outages outages@outages.org wrote:
 
 Anyone experiencing NTT instability in Dallas or Miami?
 
 http://i.imgur.com/he7VL0J.png
 
 http://i.imgur.com/jjEDkN4.png
 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages