This info pertains to accounts.google.com: Errors started around noon MST 12-Nov.
We didn't start actively troubleshooting until the morning of the 14th. accounts.google.com was resolving to 216.58.219.237 from our DC and our corporate office. We were seeing constant curl timeouts (well over 50%) and mtr showed tons of packet loss at the last two hops (anywhere from 4%-60%). An offsite user found that they were resolving accounts.google.com to 216.58.219.205 and they were not seeing the same curl timeouts. Setup some host file hacks on a few hosts, and didn't see any more errors from those hosts. The 216.58.219.237 endpoint is still problematic today. ---------- Tim Korn Network Ninja On Tue, Nov 15, 2016 at 4:50 PM, Laurie Denness <[email protected]> wrote: > Note this affecting Gmail and adwords heavily too, if you're in the NYC > metro, see if you can reproduce: > > [box:~] $ time curl -4 -vv http://adwords.google.com > * About to connect() to adwords.google.com port 80 (#0) > * Trying 74.125.29.100... > <hang for 1 minute> > > > tcpdump: > 23:47:02.402713 IP (tos 0x0, ttl 64, id 6873, offset 0, flags [DF], proto > TCP (6), length 60) > box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum > 0x738d (incorrect -> 0x4660), seq 570037774, win 29200, options [mss > 1460,sackOK,TS val 1067194520 ecr 0,nop,wscale 9], length 0 > 23:47:06.410714 IP (tos 0x0, ttl 64, id 6874, offset 0, flags [DF], proto > TCP (6), length 60) > box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum > 0x738d (incorrect -> 0x36b8), seq 570037774, win 29200, options [mss > 1460,sackOK,TS val 1067198528 ecr 0,nop,wscale 9], length 0 > 23:47:14.426726 IP (tos 0x0, ttl 64, id 6875, offset 0, flags [DF], proto > TCP (6), length 60) > box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum > 0x738d (incorrect -> 0x1768), seq 570037774, win 29200, options [mss > 1460,sackOK,TS val 1067206544 ecr 0,nop,wscale 9], length 0 > > I'd think we were going nuts, except we're seeing from many independent > networks and looks like there's a flurry of NY based people reporting on > http://downdetector.com/status/google/map/ > > Definitely seems NYC based only, although I cannot replicate on Verizon > FiOS in Manhattan. > > On 15 November 2016 at 18:38, Tim Korn via Outages <[email protected]> > wrote: > >> We've been having issues reaching google since last Friday. We're trying >> from at least four different ISPs (Zayo, Cogent, TimeWarner, Verizon FiOS) >> and all show the same issues intermittently. >> >> We're having issues with the google apps suite, accounts.google.com, and >> www.google.com. >> >> >> From our corporate office: >> [bam01:~] $ time curl -4 -vv http://www.google.com >> * Rebuilt URL to: http://www.google.com/ >> * Trying 74.125.21.103... >> * connect to 74.125.21.103 port 80 failed: Connection refused >> * Trying 74.125.21.99... >> * Connected to www.google.com (74.125.21.99) port 80 (#0) >> > GET / HTTP/1.1 >> > Host: www.google.com >> > User-Agent: curl/7.43.0 >> > Accept: */* >> > >> < HTTP/1.1 200 OK >> <snip> >> *real 1m6.523s (1 minute connect time)* >> >> >> From Datacenter: >> [dingy01.ny2:~] $ time curl -4 -vv http://www.google.com >> * About to connect() to www.google.com port 80 (#0) >> * Trying 216.58.193.132... >> * Connected to www.google.com (216.58.193.132) port 80 (#0) >> > GET / HTTP/1.1 >> > User-Agent: curl/7.29.0 >> > Host: www.google.com >> > Accept: */* >> > >> < HTTP/1.1 200 OK >> <snip> >> *real 1m3.189s (took 1 minute to connect)* >> >> [ding02.ny5:~] $ curl -4 -k https://accounts.google.com >> *curl: (7) couldn't connect to host* >> [ding02.ny5:~] $ curl -4 -k https://accounts.google.com >> *curl: (7) couldn't connect to host* >> [ding02.ny5:~] $ curl -4 -k https://accounts.google.com >> <HTML> >> <HEAD> >> <TITLE>Moved Temporarily</TITLE> >> </HEAD> >> <BODY BGCOLOR="#FFFFFF" TEXT="#000000"> >> <H1>Moved Temporarily</H1> >> The document has moved <A HREF="https://accounts.google.com/ManageAccount >> ">here</A>. >> </BODY> >> </HTML> >> >> >> ---------- >> Tim Korn >> >> >> _______________________________________________ >> Outages mailing list >> [email protected] >> https://puck.nether.net/mailman/listinfo/outages >> >> >
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
