On Thu, Nov 11, 2010 at 05:11:47PM -0500, Srikanth Sundaresan wrote:
Here are the traceroutes (without the first 3 hops)
(Note: NANOG is not really the right place to troubleshoot everyone's
home connectivity, I'm mostly just posting this as an educational
example of how to do inter-network troubleshooting... though in
retrospect this may not be the worlds best example :P).
From ADSL:
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
4 12.81.16.32 30.196 ms 32.292 ms 35.161 ms
5 12.81.16.25 37.774 ms 40.627 ms 44.209 ms
6 74.175.192.78 48.008 ms 50.841 ms 53.946 ms
7 12.122.140.186 59.278 ms 61.510 ms 61.824 ms
8 12.123.22.129 61.111 ms 59.803 ms 59.382 ms
9 12.88.97.6 116.059 ms 115.757 ms 116.331 ms
10 72.14.233.54 59.856 ms 60.354 ms 61.088 ms
11 72.14.232.213 61.312 ms 78.592 ms 209.85.254.243 60.396 ms
12 209.85.253.137 105.800 ms 100.558 ms 209.85.253.141 96.095 ms
13 8.8.8.8 96.571 ms 98.721 ms 98.514 ms
From UVerse:
4 76.201.204.10 24.020 ms 24.321 ms 24.250 ms
5 76.201.208.22 25.754 ms 25.701 ms 25.633 ms
6 76.201.208.8 25.558 ms 25.230 ms *
7 70.159.177.248 24.910 ms 22.452 ms 23.436 ms
8 12.81.16.2 24.478 ms 24.420 ms 24.514 ms
9 12.81.16.21 128.798 ms 127.685 ms 126.821 ms
10 74.175.192.90 22.999 ms 21.932 ms 23.057 ms
11 12.122.140.186 24.397 ms 12.122.141.186 24.647 ms 24.594 ms
12 12.123.22.5 32.763 ms 12.123.22.129 22.016 ms 12.123.22.5 26.850 ms
13 * * *
14 72.14.233.54 40.287 ms 72.14.233.56 40.716 ms 40.660 ms
15 209.85.254.241 41.964 ms 41.909 ms 41.842 ms
16 209.85.253.137 51.698 ms 209.85.253.133 44.534 ms 209.85.253.145
39.621 ms
17 8.8.8.8 41.278 ms 42.124 ms 42.718 ms
Both the homes are in the same city. The entry point to Google is the
same: 72.14.233.54 (from whois).
Actually the entry point to Google is probably the hop before that,
12.88.97.6. In all likelihood this is the /30 between the two networks,
where .5 is the ATT side and .6 is the Google side. The IP space of the
demarc point belongs to ATT of course, but this is what you'd expect in
a provider-customer relationship. :) In an ordinary network you would
be able to confirm this with DNS and/or some traceroutes to the routers,
but both ATT and Google have intentionally obfuscated the hell out of
their networks from the outside world (no dns, blocking traceroutes
directly to router IPs, etc), so that won't help you much. There is also
no Google looking glass (at least that I can find), nor do they support
record-route, so you're probably SOL on the reverse path too.
From ADSL, latency to that google router is about 10ms:
rtt min/avg/max/mdev = 9.461/13.137/59.856/7.841 ms
from UVerse, it's about 40ms.
rtt min/avg/max/mdev = 38.923/44.503/70.535/7.162 ms
There isn't enough jitter to justify this difference. And it's not
just to Google. i tested to another server (where ATT hands off to
Qwest), and it's the same. It can't be congestion/location, because if
it were, the ADSL gateway should see it too. Reverse path effects,
perhaps.
Well we can start by eliminating the possibility that the 8.8.8.8 node
you're hitting is a significant distance away once you hit Google's
network. What little bit of DNS ATT does have working shows that this
is coming out of Atlanta, which could also be confirmed with a few
traceroutes from route-server.ip.att.net. From there, it's trivial to
find a network with a looking glass and direct Google connectivity in
Atlanta, and match up the exact same path:
2 72.14.233.54 (72.14.233.54) 0.944 ms 0.902 ms
72.14.233.56 (72.14.233.56) 0.720 ms
3 209.85.254.241 (209.85.254.241) 1.005 ms
209.85.254.243 (209.85.254.243) 16.214 ms
72.14.232.215 (72.14.232.215) 1.264 ms
4 209.85.253.141 (209.85.253.141) 1.797 ms
209.85.253.133 (209.85.253.133) 1.937 ms
209.85.253.137 (209.85.253.137) 1.408 ms
5 google-public-dns-a.google.com (8.8.8.8) 1.413 ms 1.539 ms 1.481 ms
Honestly I've got to question the measurement that you're taking above,
since in your first (DSL) traceroute it looks like you're actually
seeing higher latency than you are on the second (Uverse) path. Without
being able to actually repeat the traceroute multiple times and verify
that the reading was accurate it's obviously hard to say for certain,
but your numbers look VERY consistent, showing a clear progression with
very little jitter from ~30ms at the first visible hop, to ~60ms at the
Google handoff. If there was really a measurement artifact, you would
expect at least a healthy percentage of those numbers to be
significantly different.
As for the ~17ms jump between Uverse and Google in the second
traceroute, I can't tell for certain without full IPs, but my gut says
that the reverse path might be going back via Ashburn once it hits the
Google side. Remember ATT is actually composed of classic ATT,
