[outages] Level3 DNS Servers down?
Is anyone else seeing issues with Level3's DNS servers? We got an alert our mail servers were down but mail is flowing normally, if slowly. Trying to resolve our domain names against Google's 8.8.8.8 and OpenDNS's 208.67.222.222 servers seems fine but 4.2.2.2 and 4.2.2.1 is spotty at best, failing to resolve in most cases. Testing www.google.com http://www.google.com and www.amazon.com http://www.amazon.com yields the same results. _ Dann Schuler Network Administrator MCSA 2012, MCITP, MCTS, Security+, Server+ mailto:dschu...@ksrinc.com dschu...@ksrinc.com (315) 470-1350 x187 ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Level3 DNS Servers down?
Solid for me in the UK dig @4.2.2.2 www.google.com +short 173.194.40.177 173.194.40.179 173.194.40.176 173.194.40.178 173.194.40.180 dig @4.2.2.2 www.amazon.com +short 205.251.242.54 On 03/09/14 16:03, Dann Schuler via Outages wrote: Is anyone else seeing issues with Level3's DNS servers? We got an alert our mail servers were down but mail is flowing normally, if slowly. Trying to resolve our domain names against Google's 8.8.8.8 and OpenDNS's 208.67.222.222 servers seems fine but 4.2.2.2 and 4.2.2.1 is spotty at best, failing to resolve in most cases. Testing www.google.com http://www.google.com and www.amazon.com http://www.amazon.com yields the same results. *_ _* *___* *Dann Schuler* *Network Administrator* *MCSA 2012, MCITP, MCTS, Security+, Server+* dschu...@ksrinc.com mailto:dschu...@ksrinc.com (315) 470-1350 x187 signature.asc Description: OpenPGP digital signature ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Level3 DNS Servers down?
On 09/03/14 11:03 -0400, Dann Schuler via Outages wrote: Is anyone else seeing issues with Level3's DNS servers? We got an alert our mail servers were down but mail is flowing normally, if slowly. Trying to resolve our domain names against Google's 8.8.8.8 and OpenDNS's 208.67.222.222 servers seems fine but 4.2.2.2 and 4.2.2.1 is spotty at best, failing to resolve in most cases. I'm not seeing any problems with the Dallas-hosted 4.2.2.2: $ time dig +short @4.2.2.2 google.com 173.194.115.7 173.194.115.2 173.194.115.14 173.194.115.5 173.194.115.8 173.194.115.3 173.194.115.4 173.194.115.0 173.194.115.9 173.194.115.6 173.194.115.1 real0m0.023s user0m0.012s sys 0m0.000s (Repeated 10 times with similar results) $ time dig +short @4.2.2.1 google.com 173.194.46.4 173.194.46.0 173.194.46.14 173.194.46.3 173.194.46.8 173.194.46.6 173.194.46.2 173.194.46.9 173.194.46.5 173.194.46.1 173.194.46.7 real0m0.037s user0m0.008s sys 0m0.004s $ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 internal ip 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.719 ms 2.976 ms 3.211 ms 3 router.olp.net (67.217.151.97) 2.322 ms 2.388 ms 2.525 ms 4 olp-67-217-150-202.olp.net (67.217.150.202) 10.949 ms 11.083 ms 11.177 ms 5 vlan1125.car1.dallas1.level3.net (4.28.42.33) 9.912 ms 10.053 ms 10.291 ms 6 ae-2-70.edge2.dallas1.level3.net (4.69.145.75) 10.177 ms 8.414 ms 8.427 ms 7 b.resolvers.level3.net (4.2.2.2) 8.891 ms 8.665 ms 8.903 ms -- Dan White ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Level3 DNS Servers down?
Not sure about L3, but I have confirmation that GoDaddy Premium DNS services are all down at the moment. I spoke with a rep and he informed me that they don't have an estimate on return to service yet but that they have all hands on deck working on this. *Sean E. Smith* Director of Network Operations at Simpleview T 520.575.1151 x131 www.simpleviewinc.com On Wed, Sep 3, 2014 at 8:03 AM, Dann Schuler via Outages outages@outages.org wrote: Is anyone else seeing issues with Level3’s DNS servers? We got an alert our mail servers were down but mail is flowing normally, if slowly. Trying to resolve our domain names against Google’s 8.8.8.8 and OpenDNS’s 208.67.222.222 servers seems fine but 4.2.2.2 and 4.2.2.1 is spotty at best, failing to resolve in most cases. Testing www.google.com and www.amazon.com yields the same results. *_* *Dann Schuler* *Network Administrator* *MCSA 2012, MCITP, MCTS, Security+, Server+* dschu...@ksrinc.com (315) 470-1350 x187 ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Level3 DNS Servers down?
Pinging and resolving for me in Dayton. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Sep 3, 2014 11:18 AM, Dann Schuler via Outages outages@outages.org wrote: Is anyone else seeing issues with Level3’s DNS servers? We got an alert our mail servers were down but mail is flowing normally, if slowly. Trying to resolve our domain names against Google’s 8.8.8.8 and OpenDNS’s 208.67.222.222 servers seems fine but 4.2.2.2 and 4.2.2.1 is spotty at best, failing to resolve in most cases. Testing www.google.com and www.amazon.com yields the same results. *_* *Dann Schuler* *Network Administrator* *MCSA 2012, MCITP, MCTS, Security+, Server+* dschu...@ksrinc.com (315) 470-1350 x187 ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
[outages] GoDaddy DNS - Here We Go Again...
A few of our client domains just suddenly started failing to resolve. As it turns out, those domains are ones whose name service is handled via GoDaddy. There's a thread on Reddit that suggests that this is pretty widespread, but the exact scope isn't known yet, and GD has not announced the issue. http://www.reddit.com/r/sysadmin/comments/2fczdc/godaddy_premium_dns_down/ Dig queries to domains that are on PDNS01 or PDNS02 fail. There are probably others. ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Level3 DNS Servers down?
I'm seeing issues with domains using domaincontrol.com nameservers. Some nameservers are answering queries. dig [ommited] @pdns01.domaincontrol.com. [ommited] IN A ;; ANSWER SECTION: [ommited]3600IN A 66.209.212.92 ;; AUTHORITY SECTION: [ommited]. 3600IN NS pdns02.domaincontrol.com. [ommited] 3600IN NS pdns01.domaincontrol.com. ;; Query time: 28 msec ;; SERVER: 216.69.185.50#53(216.69.185.50) dig [ommited] @pdns02.domaincontrol.com. [ommited] IN A ;; Query time: 25 msec ;; SERVER: 208.109.255.50#53(208.109.255.50) Andres On Wed, Sep 3, 2014 at 10:14 AM, Dan White via Outages outages@outages.org wrote: On 09/03/14 11:03 -0400, Dann Schuler via Outages wrote: Is anyone else seeing issues with Level3's DNS servers? We got an alert our mail servers were down but mail is flowing normally, if slowly. Trying to resolve our domain names against Google's 8.8.8.8 and OpenDNS's 208.67.222.222 servers seems fine but 4.2.2.2 and 4.2.2.1 is spotty at best, failing to resolve in most cases. I'm not seeing any problems with the Dallas-hosted 4.2.2.2: $ time dig +short @4.2.2.2 google.com 173.194.115.7 173.194.115.2 173.194.115.14 173.194.115.5 173.194.115.8 173.194.115.3 173.194.115.4 173.194.115.0 173.194.115.9 173.194.115.6 173.194.115.1 real0m0.023s user0m0.012s sys 0m0.000s (Repeated 10 times with similar results) $ time dig +short @4.2.2.1 google.com 173.194.46.4 173.194.46.0 173.194.46.14 173.194.46.3 173.194.46.8 173.194.46.6 173.194.46.2 173.194.46.9 173.194.46.5 173.194.46.1 173.194.46.7 real0m0.037s user0m0.008s sys 0m0.004s $ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 internal ip 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.719 ms 2.976 ms 3.211 ms 3 router.olp.net (67.217.151.97) 2.322 ms 2.388 ms 2.525 ms 4 olp-67-217-150-202.olp.net (67.217.150.202) 10.949 ms 11.083 ms 11.177 ms 5 vlan1125.car1.dallas1.level3.net (4.28.42.33) 9.912 ms 10.053 ms 10.291 ms 6 ae-2-70.edge2.dallas1.level3.net (4.69.145.75) 10.177 ms 8.414 ms 8.427 ms 7 b.resolvers.level3.net (4.2.2.2) 8.891 ms 8.665 ms 8.903 ms -- Dan White ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] GoDaddy DNS - Here We Go Again...
We have several sites with DNS at the GD Premium DNS service, all services appeared to have returned to normal about ten minutes ago. *Sean E. Smith* Director of Network Operations at Simpleview T 520.575.1151 x131 www.simpleviewinc.com On Wed, Sep 3, 2014 at 8:12 AM, Mike Phipps via Outages outages@outages.org wrote: A few of our client domains just suddenly started failing to resolve. As it turns out, those domains are ones whose name service is handled via GoDaddy. There's a thread on Reddit that suggests that this is pretty widespread, but the exact scope isn't known yet, and GD has not announced the issue. http://www.reddit.com/r/sysadmin/comments/2fczdc/godaddy_premium_dns_down/ Dig queries to domains that are on PDNS01 or PDNS02 fail. There are probably others. ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Level3 DNS Servers down?
We’ve had major packet loss and routing issues with Cox at 9:48AM CST this morning in the Dallas area. Possibly unrelated but time correlates. From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Sean Smith via Outages Sent: Wednesday, September 03, 2014 10:16 AM To: Dann Schuler Cc: outages@outages.org Subject: Re: [outages] Level3 DNS Servers down? Not sure about L3, but I have confirmation that GoDaddy Premium DNS services are all down at the moment. I spoke with a rep and he informed me that they don't have an estimate on return to service yet but that they have all hands on deck working on this. Sean E. Smith Director of Network Operations at Simpleview T 520.575.1151 x131 www.simpleviewinc.comhttp://cp.mcafee.com/d/1jWVIi4xAq6jqb5PhOevujvpdTdLLLfe6zBdUSeud7dQQNMUS--YYUqekPhOrhK-CDsQsTd7b1Lo8hYNN1K27wHicGYKrQzaLbCX9IczHIfZvCkSm7DSeLsKDtdPC77SnT6umKDp5dmUVPBgY-F6lK1FJ4SMrLRQkhPz0WXVEVdTdw0GilmfYPbP4EmVsTlKuAvEn8lrxrW0GHsZ8_jw09IQsI6zCVI5-Aq83iTlKuAvEqnjh05vc-uOq80U631Ew6fyp-d3h0xkN0ArpudEKb6dtfZ On Wed, Sep 3, 2014 at 8:03 AM, Dann Schuler via Outages outages@outages.orgmailto:outages@outages.org wrote: Is anyone else seeing issues with Level3’s DNS servers? We got an alert our mail servers were down but mail is flowing normally, if slowly. Trying to resolve our domain names against Google’s 8.8.8.8 and OpenDNS’s 208.67.222.222 servers seems fine but 4.2.2.2 and 4.2.2.1 is spotty at best, failing to resolve in most cases. Testing www.google.comhttp://cp.mcafee.com/d/k-Kr6zqb5PhOevujvpdTdLLLfe6zBdUSeud7dQQNMUS--YYUqekPhOrhK-CDsQsTd7b1Lo8hYNN1K27wHicGYKrQzaLbCX9IczHIfZvCkSm7DSeLsKDtdPC77SnT6umKDp5dmUVPBgY-F6lK1FJcSMrLRQkhPz0WXVEVdTdw0ydqAj_VsTlKuAvEn8lrxrW0GHsZ8_jw09IQsI6zCVI5-Aq83iTlKuAvEqnjh05vc-uOq80U631Ew6fyp-d3h0xkN0ArpudIoEh and www.amazon.comhttp://cp.mcafee.com/d/k-Kr6zqb5PhOevujvpdTdLLLfe6zBdUSeud7dQQNMUS--YYUqekPhOrhK-CDsQsTd7b1Lo8hYNN1K27wHicGYKrQzaLbCX9IczHIfZvCkSm7DSeLsKDtdPC77SnT6umKDp5dmUVPBgY-F6lK1FJASMrLRQkhPz0WXVEVdTdw0JlQkH3BPtmVWh-xsxlK5LE2GJPQzZe00CPhOMqerCMnWhEwdbtmVWh-xFtd40lYPVX9Ew3woc6y0o-9DUQd425j42hJBUSPIJXXok yields the same results. _ Dann Schuler Network Administrator MCSA 2012, MCITP, MCTS, Security+, Server+ dschu...@ksrinc.commailto:dschu...@ksrinc.com (315) 470-1350 x187tel:%28315%29%20470-1350%20x187 ___ Outages mailing list Outages@outages.orgmailto:Outages@outages.org https://puck.nether.net/mailman/listinfo/outageshttp://cp.mcafee.com/d/5fHCN8i43qb5PhOevujvpdTdLLLfe6zBdUSeud7dQQNMUS--YYUqekPhOrhK-CDsQsTd7b1Lo8hYNN1K27wHicGYKrQzaLbCX9IczHIfZvCkSm7DSeLsKDtdPC77SnT6umKDp5dmUVPBgY-F6lK1FJwSMrLRQkhPz0WXVEVdTdAVPmEBCbdSaY3ivNU6U9GX33VkDa3JsGTfifQdRrDF7W5O5mUm-waGTfifQU02rd7b1EVKr1vF6y0QJRrDF7W6BQQg1nPfDICy0e1wMq81zUCvzgQg8lcg96SnzoXZuYFupQ6e4 If you are not the intended recipient of this email, please notify the sender, delete the email, and do not read, act upon, print, disclose, retain or distribute this email. To view the full terms that apply to this email, go to Agio Disclaimerhttp://agio.com/email-disclaimer/ ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] GoDaddy DNS - Here We Go Again...
On Wed, Sep 3, 2014 at 12:12 PM, Mike Phipps via Outages outages@outages.org wrote: A few of our client domains just suddenly started failing to resolve. As it turns out, those domains are ones whose name service is handled via GoDaddy. There's a thread on Reddit that suggests that this is pretty widespread, but the exact scope isn't known yet, and GD has not announced the issue. http://www.reddit.com/r/sysadmin/comments/2fczdc/godaddy_premium_dns_down/ Dig queries to domains that are on PDNS01 or PDNS02 fail. There are probably others. We are seeing queries to PDNS02 being refused depending on anycast location, but we can't pinpoint them exactly as hostname.bind queries are not answered. Some locations answer fine, some not, depending not only on country but also on transit provider. Rubens ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
[outages] Centurylink North Florida instabilty
The CL edge point at 199.44.253.1 started dropping large amounts of traffic at around 11:40 EST. It seems to have smoothed out for now, but it dropped all of our tunnels for about 15 minutes. Wed Sep 3 11:49:22 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings HostLoss% Snt Last Avg Best Wrst StDev 1. 10.100.0.40.0%189.1 9.9 2.6 70.4 16.2 2. 10.100.0.254 0.0%181.5 2.2 1.5 3.2 0.4 3. 63-166-22-2-18663440890. 0.0%182.5 2.6 1.9 4.9 0.7 4. 199.44.253.1 72.2%18 316.4 308.9 297.0 316.4 7.2 5. 223.15.252.1 61.1%18 39.0 43.6 38.5 70.3 11.8 6. 208-110-248-137.centurylink.net 58.8%18 304.9 301.0 288.0 307.9 7.6 7. bb-atlngamv-jx9-01-xe-10-1-0.cor 38.9%18 39.7 39.3 37.8 41.6 1.1 8. ??? 9. ??? 10. 208.46.16.20647.1%18 40.2 39.1 38.3 40.2 0.8 11. 72.14.239.10047.1%17 40.1 41.2 37.7 51.1 4.4 12. 66.249.94.20 75.0%17 293.0 295.7 293.0 297.5 1.9 13. 209.85.242.138 73.3%16 286.2 310.6 286.2 332.4 22.0 14. ??? 15. google-public-dns-a.google.com 57.1%14 40.0 39.7 38.7 41.4 1.0 -- David Kelly ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
Now getting this [image: Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
[outages] Facebook down
tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
Same for Washington DC, Atlanta, GA and Tallahassee, FL on all local incumbent, CLEC and MSO networks. On Wed, Sep 3, 2014 at 3:44 PM, Luke Rockwell via Outages outages@outages.org wrote: Now getting this [image: Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com http://t.senaldos.com/link?url=http%3A%2F%2Ffacebook.com%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=367d1953-ee2d-4126-8c7a-a8bbbefcec2c Tracing route to facebook.com http://t.senaldos.com/link?url=http%3A%2F%2Ffacebook.com%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=e0cee6cd-4d81-42a6-f8d0-89e0e85cb4eb [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net http://t.senaldos.com/link?url=http%3A%2F%2Fdc-sfo-agg-1--ucb-10ge.cenic.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=901e6294-8c4f-48f8-8706-3e42887c5826 [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net http://t.senaldos.com/link?url=http%3A%2F%2Foak-agg2--sfo-agg1-10g.cenic.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=fb450bb3-119d-4791-ae2e-b7e444b48bb8 [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net http://t.senaldos.com/link?url=http%3A%2F%2Fdc-paix-px1--oak-core1-ge.cenic.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=80d0d24f-e377-4219-cb3c-065db33b980f [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net http://t.senaldos.com/link?url=http%3A%2F%2Fpaix-px1--facebook-ge.cenic.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=a790bc13-e815-4603-a541-f68ad20e71a3 [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net http://t.senaldos.com/link?url=http%3A%2F%2Fae1.bb01.pao1.tfbnw.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=df6c02e3-8bee-4e6d-8691-56d9830dcea6 [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net http://t.senaldos.com/link?url=http%3A%2F%2Fae4.bb01.sjc1.tfbnw.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=0946c9f8-d49f-4797-c389-fc8791964071 [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net http://t.senaldos.com/link?url=http%3A%2F%2Fbe16.bb01.frc3.tfbnw.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=a89545d4-ccc2-4fb6-d39c-f5daa406d051 [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net http://t.senaldos.com/link?url=http%3A%2F%2Fae87.dr04.frc1.tfbnw.net%2Fukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=c5f8233c-a426-4c75-f224-b5074771ddc5 [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages http://t.senaldos.com/link?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutagesukey=agxzfnNpZ25hbHNjcnhyGAsSC1VzZXJQcm9maWxlGICAwPyn16YKDAk=6a5e97e2-ede0-453c-a53e-a1bacb37a11f On Wed, Sep 3, 2014 at 3:44 PM, Luke Rockwell via Outages outages@outages.org wrote: Now getting this [image: Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
Our HTTPv6 monitoring showed “Connection refused” from 2:39 to 2:56 pm Central. Frank From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Luke Rockwell via Outages Sent: Wednesday, September 03, 2014 2:45 PM To: Outages@outages.org Subject: Re: [outages] Facebook down Now getting this On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu mailto:luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com http://facebook.com Tracing route to facebook.com http://facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net http://dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net http://oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net http://dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net http://paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net http://ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net http://ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net http://be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net http://ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
Get back to work :) I'm getting the same 'Sorry, something went wrong' error too from NJ, and mobile is not working from ATT. -James On Wed, Sep 3, 2014 at 3:44 PM, Luke Rockwell via Outages outages@outages.org wrote: Now getting this [image: Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- *James Cornman* *Chief Technology Officer*jcorn...@atlanticmetro.net 212.792.9950 - ext 101 *Atlantic Metro Communications, Inc.* *4 Century Drive, Parsippany NJ 07054* *Data Centers • Managed Hosting • Internet Access • Cloud Computing* Follow @atlanticmetro http://www.twitter.com/atlanticmetro on Twitter to learn more about daily specials. www.atlanticmetro.net ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
It was only down for me in Denver on Comcast for a few minutes. It has been back up for me for around 20 minutes. I'm connecting via v6. On Wed, Sep 3, 2014 at 1:54 PM, James Cornman via Outages outages@outages.org wrote: Get back to work :) I'm getting the same 'Sorry, something went wrong' error too from NJ, and mobile is not working from ATT. -James On Wed, Sep 3, 2014 at 3:44 PM, Luke Rockwell via Outages outages@outages.org wrote: Now getting this [image: Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- *James Cornman* *Chief Technology Officer*jcorn...@atlanticmetro.net 212.792.9950 - ext 101 *Atlantic Metro Communications, Inc.* *4 Century Drive, Parsippany NJ 07054* *Data Centers • Managed Hosting • Internet Access • Cloud Computing* Follow @atlanticmetro http://www.twitter.com/atlanticmetro on Twitter to learn more about daily specials. www.atlanticmetro.net ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
No issue here 1.*** 0.0% 35.0 118.7 5.0 232.5 160.9 2. *** 0.0% 26.0 6.0 6.0 6.1 0.0 3. sanjbprj01-ae0.0.rd.sj.cox.net 0.0% 2 39.8 39.8 39.8 39.8 0.0 4. sv1.pr02.tfbnw.net 0.0% 2 39.8 39.9 39.8 40.0 0.2 5. ae2.bb02.sjc1.tfbnw.net 0.0% 2 40.2 40.1 40.0 40.2 0.1 6. be16.bb02.frc3.tfbnw.net 0.0% 2 91.9 91.7 91.5 91.9 0.3 7. ae88.dr03.frc1.tfbnw.net 0.0% 2 97.2 94.1 91.0 97.2 4.4 8. ??? 9. ??? 10. edge-star-shv-13-frc1.facebook.com 0.0% 2 92.3 92.3 92.3 92.3 0.0 From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Luke Rockwell via Outages Sent: Wednesday, September 03, 2014 12:45 PM To: Outages@outages.org Subject: Re: [outages] Facebook down Now getting this [Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edumailto:luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.comhttp://facebook.com Tracing route to facebook.comhttp://facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.nethttp://dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.nethttp://oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.nethttp://dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.nethttp://paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.nethttp://ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.nethttp://ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.nethttp://be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.nethttp://ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
- Original Message - From: Luke Rockwell via Outages outages@outages.org Subject: [outages] Facebook down tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: Facebook was being observed not to be able to refresh web UI newsfeeds here around 1600EDT, via Sprint LTE. It's now working again. Android client via the same connection seemed to be working, as were XMPP connections to FB chat. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274 ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
Here in Brazil (Ceará State) normal. We have some trouble at 03:00 p.m today. But this was solve in few minutes via Global Village Telecom. On Wed, Sep 3, 2014 at 5:21 PM, John Neiberger via Outages outages@outages.org wrote: It was only down for me in Denver on Comcast for a few minutes. It has been back up for me for around 20 minutes. I'm connecting via v6. On Wed, Sep 3, 2014 at 1:54 PM, James Cornman via Outages outages@outages.org wrote: Get back to work :) I'm getting the same 'Sorry, something went wrong' error too from NJ, and mobile is not working from ATT. -James On Wed, Sep 3, 2014 at 3:44 PM, Luke Rockwell via Outages outages@outages.org wrote: Now getting this [image: Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- *James Cornman* *Chief Technology Officer*jcorn...@atlanticmetro.net 212.792.9950 - ext 101 *Atlantic Metro Communications, Inc.* *4 Century Drive, Parsippany NJ 07054* *Data Centers • Managed Hosting • Internet Access • Cloud Computing* Follow @atlanticmetro http://www.twitter.com/atlanticmetro on Twitter to learn more about daily specials. www.atlanticmetro.net ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- Att, Osvaldo Filho. Analista de Suporte (Operações de Datacenter - Backup e Storage) - Synapsis Brasil. Sócio-fundador do i-TIC Digital Livre https://www.facebook.com/iticdigital - Informação Livre para todos! Twitter i-TIC Digital Livre: @iticdigital TUX-CE Member. Graduado em Redes de Computadores - Centro Universitário Estácio do Ceará. Técnico em Conectividade com Extensão em Desenvolvimento de Software - IFCE (Cefet). Graduando em Telemática - IFCE (Cefet). Novell Datacenter Technical Specialist. Linux Professional Institute Certified Level 1. Novell Certified Linux Administrator (CLA). Facebook: Osvaldo Filho Twitter: @osvaldofilho ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Facebook down
back from getting food and it is now online. On Wed, Sep 3, 2014 at 1:14 PM, Cal Leeming [iops.io] c...@iops.io wrote: Same from here, dies at the edge nodes; traceroute to facebook.com (173.252.110.27), 64 hops max, 52 byte packets 1 192.168.2.1 (192.168.2.1) 1.003 ms 0.746 ms 0.672 ms 2 * * * 3 31.55.187.209 (31.55.187.209) 18.480 ms 17.699 ms 17.574 ms 4 31.55.187.228 (31.55.187.228) 17.248 ms 17.279 ms 17.027 ms 5 195.99.127.52 (195.99.127.52) 18.935 ms 195.99.127.0 (195.99.127.0) 18.730 ms 23.040 ms 6 peer1-xe7-1-0.redbus.ukcore.bt.net (62.6.200.199) 19.057 ms 19.331 ms peer1-xe-8-1-1.redbus.ukcore.bt.net (213.121.193.89) 24.326 ms 7 ae13.pr01.lhr2.tfbnw.net (103.4.96.146) 19.574 ms 18.615 ms 22.839 ms 8 ae1.bb02.lhr2.tfbnw.net (31.13.30.198) 19.630 ms 19.438 ms 19.147 ms 9 be14.bb01.ewr2.tfbnw.net (74.119.78.142) 113.900 ms 111.550 ms 112.879 ms 10 be44.bb02.iad3.tfbnw.net (31.13.26.5) 113.668 ms be44.bb01.iad3.tfbnw.net (31.13.26.1) 110.996 ms 111.451 ms 11 be14.bb02.frc3.tfbnw.net (31.13.24.38) 110.351 ms ae12.bb04.frc3.tfbnw.net (31.13.24.95) 117.545 ms be18.bb02.frc3.tfbnw.net (31.13.24.46) 109.387 ms 12 ae90.dr04.frc1.tfbnw.net (173.252.65.115) 114.930 ms ae90.dr02.frc1.tfbnw.net (173.252.65.111) 111.628 ms ae89.dr04.frc1.tfbnw.net (173.252.65.99) 120.079 ms 13 * * * Cal On Wed, Sep 3, 2014 at 8:44 PM, Luke Rockwell via Outages outages@outages.org wrote: Now getting this [image: Inline image 1] On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell luke.rockw...@alumni.berkeley.edu wrote: tracert facebook.com Tracing route to facebook.com [173.252.110.27] over a maximum of 30 hops: 11 ms1 ms1 ms 21 ms1 ms1 ms 3 1 ms1 ms1 ms 4 1 ms 1 ms 1 ms dc-sfo-agg-1--ucb-10ge.cenic.net [137.164.50.16] 5 4 ms 2 ms 3 ms oak-agg2--sfo-agg1-10g.cenic.net [137.164.22.25] 6 3 ms 3 ms 3 ms dc-paix-px1--oak-core1-ge.cenic.net [137.164.47. 18] 7 3 ms 3 ms 3 ms paix-px1--facebook-ge.cenic.net [198.32.251.133] 8 4 ms 3 ms 4 ms ae1.bb01.pao1.tfbnw.net [74.119.76.134] 9 4 ms 4 ms12 ms ae4.bb01.sjc1.tfbnw.net [74.119.76.137] 1063 ms63 ms67 ms be16.bb01.frc3.tfbnw.net [31.13.24.28] 1167 ms68 ms66 ms ae87.dr04.frc1.tfbnw.net [173.252.64.53] 12 *** Request timed out. 13 ^C ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
