google troubles?
Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair
RE: google troubles?
I did have connectivity issues for mobile devices this AM between 8:00am-10:00am EST. I am in NE Ohio. Seems to be resolved now. -Original Message- From: Blair Trosper [mailto:blair.tros...@gmail.com] Sent: Wednesday, July 10, 2013 10:28 AM To: nanog@nanog.org Subject: google troubles? Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair
Re: google troubles?
We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.comwrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
Re: google troubles?
Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.comwrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
Re: google troubles?
Traceroutes worked fine for me during the outage. Seems to have been something at L4-L7. -- max On Wed, Jul 10, 2013 at 9:56 AM, Grant Ridder shortdudey...@gmail.comwrote: Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.com wrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
RE: google troubles?
We (were) peered with Google in Atlanta. We were unable to bring up web pages for google.com or gmail.com. Traceroute worked, though. I shut off the peering my outbound route switched to Cogent, which works now. I'll try peering again tonight. Maybe they'll have fixed it by then. -Original Message- From: N. Max Pierson [mailto:nmaxpier...@gmail.com] Sent: Wednesday, July 10, 2013 11:00 AM To: Grant Ridder Cc: nanog@nanog.org Subject: Re: google troubles? Traceroutes worked fine for me during the outage. Seems to have been something at L4-L7. -- max On Wed, Jul 10, 2013 at 9:56 AM, Grant Ridder shortdudey...@gmail.comwrote: Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.com wrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
Re: google troubles?
tcptraceroutes working fine too? On Wed, Jul 10, 2013 at 8:16 AM, Milt Aitken m...@net2atlanta.com wrote: We (were) peered with Google in Atlanta. We were unable to bring up web pages for google.com or gmail.com. Traceroute worked, though. I shut off the peering my outbound route switched to Cogent, which works now. I'll try peering again tonight. Maybe they'll have fixed it by then. -Original Message- From: N. Max Pierson [mailto:nmaxpier...@gmail.com] Sent: Wednesday, July 10, 2013 11:00 AM To: Grant Ridder Cc: nanog@nanog.org Subject: Re: google troubles? Traceroutes worked fine for me during the outage. Seems to have been something at L4-L7. -- max On Wed, Jul 10, 2013 at 9:56 AM, Grant Ridder shortdudey...@gmail.comwrote: Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.com wrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
google troubles?
I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers. AS701 = TOR2-CORE-R1#traceroute www.google.com Translating www.google.com...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) === *Query:* *tr 64.71.249.45* Type escape sequence to abort. Tracing the route to 64.71.249.45 1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Cheers, - MJ -Original Message- From: Grant Ridder [mailto:shortdudey...@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles? Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.comwrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
Re: google troubles?
I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8) davinci#traceroute www.google.com Translating www.google.com...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29) 1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1# Thanks, - Mike On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson m...@routed.ca wrote: I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers. AS701 = TOR2-CORE-R1#traceroute www.google.com Translating www.google.com...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) === *Query:* *tr 64.71.249.45* Type escape sequence to abort. Tracing the route to 64.71.249.45 1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Cheers, - MJ -Original Message- From: Grant Ridder [mailto:shortdudey...@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles? Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.comwrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
Re: google troubles?
On Wed, Jul 10, 2013 at 12:20 PM, Mike Jackson m...@routed.ca wrote: I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8) so: 1) rogers is hijacking traffic to 8.8.8.8 2) the copy of 8.8.8.8 in rogers-land is replying with incorrect information for google properties (at least). err... any idea if it's lying about other things too? :) davinci#traceroute www.google.com Translating www.google.com...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29) 1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1# Thanks, - Mike On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson m...@routed.ca wrote: I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers. AS701 = TOR2-CORE-R1#traceroute www.google.com Translating www.google.com...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) === *Query:* *tr 64.71.249.45* Type escape sequence to abort. Tracing the route to 64.71.249.45 1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Cheers, - MJ -Original Message- From: Grant Ridder [mailto:shortdudey...@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles? Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.comwrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh
Re: google troubles?
On Wed, Jul 10, 2013 at 2:18 PM, Christopher Morrow morrowc.li...@gmail.com wrote: On Wed, Jul 10, 2013 at 12:20 PM, Mike Jackson m...@routed.ca wrote: I just realized that it's not Google IP space (74.125.0.0/16), Rogers is hijacking the DNS and resolving www.google.com to space within 64.71.240.0/20 which is Rogers IP space! (note the name server set as 8.8.8.8) so: 1) rogers is hijacking traffic to 8.8.8.8 2) the copy of 8.8.8.8 in rogers-land is replying with incorrect information for google properties (at least). err... any idea if it's lying about other things too? :) oops, so a polite caller noted that google often ships people at a local 'google global cache' node if they have one... it seems like that might be what's going on here with 8.8.8.8 sending you to 64.71 and/or 66.185 addresses. so maybe rogers isn't doing something untoward after all! -chris (thanks for the headslap polite caller) davinci#traceroute www.google.com Translating www.google.com...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (66.185.85.29) 1 x.x.x.x [AS 812] 4 msec 4 msec 4 msec 2 so-4-0-2.gw02.ym.phub.net.cable.rogers.com (66.185.82.129) [AS 812] 8 msec 8 msec 4 msec 3 69.63.252.222 [AS 812] 4 msec 0 msec 8 msec 4 69.63.250.162 [AS 812] 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * TOR2-CORE-R1#show ip bgp 66.185.85.29 BGP routing table entry for 66.185.80.0/20, version 13095115 Paths: (1 available, best #1, table default) Advertised to update-groups: 14 701 6461 812 205.205.23.121 from 205.205.23.121 (137.39.8.42) Origin IGP, localpref 100, valid, external, best TOR2-CORE-R1# Thanks, - Mike On Wed, Jul 10, 2013 at 11:28 AM, Mike Jackson m...@routed.ca wrote: I can see the Google IP space (64.71.240.0/20) from Verizon/AS701, but not from Rogers/AS812 in Toronto. I've done a few other test traceroutes through Rogers to verify that they didn't filter UDP/ICMP. At this point nothing would surprise me from Rogers. AS701 = TOR2-CORE-R1#traceroute www.google.com Translating www.google.com...domain server (8.8.8.8) [OK] Type escape sequence to abort. Tracing the route to www.google.com (74.125.26.99) VRF info: (vrf in name/id, vrf out name/id) 1 x.x.x.x [AS701] 4 msec 0 msec 0 msec 2 0.ge-11-0-0.XT4.TOR2.ALTER.NET (152.63.133.78) [AS 701] 4 msec 0 msec 4 msec 3 0.so-4-0-3.XT2.NYC4.ALTER.NET (152.63.0.73) [AS 701] 16 msec 16 msec 16 msec 4 TenGigE0-7-1-0.GW8.NYC4.ALTER.NET (152.63.21.125) [AS 701] 20 msec TenGigE0-5-1-0.GW8.NYC4.ALTER.NET (152.63.21.73) [AS 701] 16 msec TenGigE0-5-4-0.GW8.NYC4.ALTER.NET (152.63.18.206) [AS 701] 20 msec 5 72.14.238.232 [AS 15169] 16 msec 16 msec 20 msec 6 72.14.236.208 [AS 15169] [MPLS: Label 680976 Exp 4] 16 msec 16 msec 72.14.236.206 [AS 15169] [MPLS: Label 533197 Exp 4] 20 msec 7 209.85.249.11 [AS 15169] [MPLS: Label 16668 Exp 4] 24 msec 72.14.239.93 [AS 15169] [MPLS: Label 14644 Exp 4] 24 msec 209.85.249.11 [AS 15169] [MPLS: Label 13978 Exp 4] 24 msec 8 209.85.243.114 [AS 15169] [MPLS: Label 568789 Exp 4] 32 msec 209.85.241.222 [AS 15169] [MPLS: Label 632535 Exp 4] 32 msec 32 msec 9 216.239.48.159 [AS 15169] 36 msec 32 msec 216.239.48.59 [AS 15169] 32 msec 10 www.google.com (74.125.26.147) [AS 15169] 32 msec 32 msec 32 msec AS812 (Rogers Looking Glass https://supernoc.rogerstelecom.net/lg/) === *Query:* *tr 64.71.249.45* Type escape sequence to abort. Tracing the route to 64.71.249.45 1 64.71.255.62 0 msec 0 msec 0 msec 2 ge-4-3-0.gw02.ym.phub.net.cable.ROGERS.com (66.185.82.237) 4 msec 4 msec 4 msec 3 69.63.250.189 4 msec 4 msec 4 msec 4 69.63.250.174 4 msec 4 msec 4 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Cheers, - MJ -Original Message- From: Grant Ridder [mailto:shortdudey...@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles? Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.comwrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair
Re: google troubles?
On Wed, Jul 10, 2013 at 2:55 PM, Mike Jackson m...@routed.ca wrote: That makes sense, I figured it was some type of a CDN caching environment. I forget that we do this at times... :) I've found that youtube.com is in the same boat: Translating www.youtube.com...domain server (8.8.8.8) [OK] yup, so this makes sense as well... good.
Re: google troubles?
On Wed, Jul 10, 2013 at 2:41 PM, Mike Jackson m...@routed.ca wrote: Hey Chris, long time! From what I can tell, it's only Google Services (that I've found so far; other things appear to resolve correctly). I'm wondering if they're bouncing Google based traffic through some type of caching / accelerator? Or maybe it's an NSA/DPI box ;) in .CA? :) don't you mean the meat-helmet-wearing CSE folks? :) I've tested Maps, Gmail, Translate as well as www.google.com, www.google.ca and they're all hijacked replies yup... well, not hijacked in a bad sense... it's actually supposed to be making things better. Translating www.google.com...domain server (8.8.8.8) [OK] (www.google.com) Type escape sequence to abort. Tracing the route to www.google.com (66.185.84.44) (www.google.ca) Type escape sequence to abort. Tracing the route to www.google.ca (66.185.95.44) (gmail) Type escape sequence to abort. Tracing the route to www3.l.google.com (66.185.85.39) (maps) Type escape sequence to abort. Tracing the route to maps.l.google.com (64.71.249.114) (translate) Type escape sequence to abort. Tracing the route to www3.l.google.com (66.185.84.30) those all seem properly done, yes. -chris
Re: google troubles?
* * * Cheers, - MJ -Original Message- From: Grant Ridder [mailto:shortdudey...@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles? Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.comwrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
Re: google troubles?
[mailto:shortdudey...@gmail.com] Sent: July-10-13 10:57 AM To: John York Cc: nanog@nanog.org Subject: Re: google troubles? Does anyone have traceroutes showing where the issues are? -Grant On Wed, Jul 10, 2013 at 7:45 AM, John York jo...@griffintechnology.comwrote: We saw the same thing, but seems to be cleared up now. All our providers that routed to Google addresses in ATL had the issue. We have one provider that lands on Google addresses in DFW, and it was working. ...And now I see that it isn't completely resolved. Some Google apps are still inaccessible via the Atlanta routes. On Wed, Jul 10, 2013 at 9:28 AM, Blair Trosper blair.tros...@gmail.com wrote: Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
