Re: Extreme Slowness

2006-10-27 Thread Michael . Dillon 

 Which begs the same question I've asked in the recent past: then
 what *is* a good diagnostic tool?  If ICMP is not the best way to
 test, then what is?  What other globally-implemented layer 3 or
 below protocols do we have available for troubleshooting?
 
 Sure, UDP-based traceroute still relies on ICMP TTL exceeded
 responses to work.  I've no idea what TCP traceroute relies on,
 as I haven't looked at it.

I love it when people answer their own questions
and tell us that they are lazy, to boot.

For the record, TCP traceroute and similar TCP based
tools rely on the fact that if you send a TCP SYN 
packet to a host it will respond with either a
TCP RST (if the port is NOT listening) or a TCP
SYN/ACK. The round trip time of this provides useful
information which is unaffected by any ICMP chicanery
on the part of routers or firewalls. A polite application
such as TCP traceroute will reply to the SYN/ACK with
an RST packet so it is reasonably safe to use this tool
with live services.

Of course, even TCP packets can be blocked or dropped
for various reasons so this is not a 100% solution.
However, if you want to avoid ICMP filtering or low
precedence, then TCP traceroute will help.

--Michael Dillon

Re: Extreme Slowness

2006-10-27 Thread Mikael Abrahamsson 


On Fri, 27 Oct 2006, [EMAIL PROTECTED] wrote:


For the record, TCP traceroute and similar TCP based
tools rely on the fact that if you send a TCP SYN
packet to a host it will respond with either a
TCP RST (if the port is NOT listening) or a TCP
SYN/ACK. The round trip time of this provides useful
information which is unaffected by any ICMP chicanery
on the part of routers or firewalls. A polite application
such as TCP traceroute will reply to the SYN/ACK with
an RST packet so it is reasonably safe to use this tool
with live services.


Intermediate nodes are still discovered by ICMP TTL Exceeded in transit 
just like UDP based traceroute, ie the outgoing TCP SYN packet has a low 
TTL.


So yes, tcptraceroute is good for getting thru firewalls in the forward 
direction, but intermediate routers are discovered in the same way by you 
getting an ICMP back because the TTL ran out.


--
Mikael Abrahamssonemail: [EMAIL PROTECTED]

Re: Extreme Slowness

2006-10-27 Thread Elijah Savage 


Adam,

Because of contractual issues it makes it very hard for me to  
participate on this list hence the vague original post. I was just  
asking a general question to see if anyone else was having issues. I  
have peering points with Broadwing(now level3), Sprint, ATT and MCI 
(now Verizon) that I can test for throughput from. This was not just  
about home cable connectivity though when frontline starts to get  
calls I often use wget (very low overhead) to test throughput between  
my sites or to home my home box often times simulating the same sort  
of connectivity that a customer may have. There were customers that  
could not even get to level3.net yesterday which is their home page,  
but it is always nice to get the refresher course on ICMP though :).


As for html posted messages truly my mistake I know better and thank  
you for mentioning it. The new duo core 2 mac mail client which I am  
still trying to get use to under preferences says it is set to plain  
text hmmm something I need to look into.


Thank you

On Oct 27, 2006, at 12:22 AM, Adam Rothschild wrote:



Elijah,

On 2006-10-26-16:34:18, Elijah Savage [EMAIL PROTECTED] wrote:
[HTML mail stripped]

It seems anything traversing level3 has very high latency along with
what seems overloaded capacity as if they are running in a degraded
mode I have connections with Time Warner, ATT, and MCI [...]


On 2006-10-26-16:48:15, Elijah Savage [EMAIL PROTECTED] wrote:
[HTML mail stripped]

Say like this traceroute. This is from TW to a Broadwing DS3.

5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  153.267 ms
207.125 ms
tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  218.920 ms
6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923
ms  57.770 ms
7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)  254.145 ms
ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)  258.522 ms
ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)  227.223 ms
8  broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10)   
231.451 ms

9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568
ms  22.511 ms


Your postings appear to be missing two key pieces of information which
would help with the community diagnosis requested: source and
destination IP addresses.  From the information you did provide, one
can deduce that you're behind a TW/RoadRunner cable modem:

  13.216.78.4.IN-ADDR.ARPA domain name pointer  
tenge-3-2.car1.Cincinnati1.Level3.net
  14.216.78.4.IN-ADDR.ARPA domain name pointer  
ROADRUNNER.car1.Cincinnati1.Level3.net
  9.216.78.4.IN-ADDR.ARPA domain name pointer  
tenge-3-1.car1.Cincinnati1.Level3.net
  10.216.78.4.IN-ADDR.ARPA domain name pointer  
ROADRUNNER.car1.Cincinnati1.Level3.net


Now, the jitter and high latency you're seeing could be a result of
one or more factors, including but not limited to RF/plant issues, TWC
running their transport and/or Level(3) transit hot (which seems to be
a common occurrence these days), ECMP across two circuits of uneven
loading, or your neighbor might be jacking wifi and downloading a
bunch of torrents -- we, the readers, just don't know.

Of note when performing armchair troubleshooting across Level(3)'s
network: the 'ebr's (PTR record of ebr*.{pop}.level3.net == Force10
E1200; Experimental Backbone Router?) tend to drop a lot of diagnostic
traffic (such as, say, 'ping' and 'traceroute') as a part of overly
aggressive control-plane policers.  This loss is, of course, strictly
cosmetic, and has no bearing on end-to-end performance.  Hence, the
old to it, not through it rule applies.

smokeping[1] and iperf[2] (to end hosts) are your friends.

As an aside, I've noticed your string of postings today were all
HTML-tagged.  While not expressly forbidden (or even discouraged) by
the current Mailing List AUP, this is generally regarded as bad form;
you might wish to reconfigure your mail client accordingly...

Hope this helps,
-a

[1] http://oss.oetiker.ch/smokeping/
[2] http://dast.nlanr.net/Projects/Iperf/

Re: Extreme Slowness

2006-10-27 Thread W. Kevin Hunt 


We peer with UUnet and Telcove (now L3 and being assimulated)
Latency across Telcove has been terrible (not just routers on the path 
with higher than norm latency, but latency all the way to the endpoint)
I have personal opinions as to why the latency is so bad, but until I 
can prove something I'd rather not say anything in public.
Some examples : 72.30.33.194 is 60.8 ms away via uunet, it is 109ms away 
via Telcove.   www.level3.com via uunet is 30ms away, via Telcove it is 
55ms away.


Trace to level3.com via UUNet

Hostname 
%Loss  Rcv  Snt  Last Best  Avg  Worst
 1. ndcr3-52.datasync.net 
0%66 000  0
 2. ndcr6-ndcr3.datasync.net 
0%55 000  0
 3. POS1-2.GW4.NOL1.ALTER.NET 
0%55 101  1
 4. 501.at-0-0-0.XL2.NOL1.ALTER.NET 
0%55 111  1
 5. 0.so-6-2-0.XT1.DFW9.ALTER.NET 
0%5514   14   15 15
 6. 0.so-6-0-0.BR6.DFW9.ALTER.NET 
0%5515   14   14 15
 7. so-1-0-0.edge1.Dallas1.Level3.net 
0%5516   15   16 17
 8. so-1-2-0.bbr1.Dallas1.Level3.net 
0%5516   15   16 16
 9. ae-0-0.bbr2.Denver1.Level3.net 
0%5583   29   41 83
10. ge-6-1.hsa1.Denver1.Level3.net 
   0%5535   29   31 35
11. 4.68.94.1 
   0%5530   30   30 32
12. www.Level3.com 
   0%5531   29   30 31



Via Telcove

 1. ndcr3-52.datasync.net 
0%55 000  0
 2. 64.66.101.89 
0%55 777  7
 3. 24.56.107.229 
0%4420   20   20 20

 4. ???
 5. 24.56.107.94 
0%4420   20   20 20
 6. ge-6-23.car1.Atlanta1.Level3.net 
0%44   143   20   51143
 7. ae-1-51.bbr1.Atlanta1.Level3.net 
0%4420   20   30 59
 8. as-0-0.bbr1.Denver1.Level3.net 
0%4454   53   54 54
 9. ge-9-0.hsa1.Denver1.Level3.net 
0%4455   54   54 55
10. 4.68.94.1 
   0%4455   54   55 55
11. www.Level3.com 
   0%4455   55   55 55



--
W. Kevin Hunt
CCIE #11841
Linux+ SME

There are 10 kinds of people in this world, those that understand 
binary and those that do not.


Elijah Savage wrote:


Adam,

Because of contractual issues it makes it very hard for me to 
participate on this list hence the vague original post. I was just 
asking a general question to see if anyone else was having issues. I 
have peering points with Broadwing(now level3), Sprint, ATT and MCI(now 
Verizon) that I can test for throughput from.

Re: Extreme Slowness

2006-10-27 Thread Adam Rothschild 

On 2006-10-27-07:37:37, Elijah Savage [EMAIL PROTECTED] wrote:
 Because of contractual issues it makes it very hard for me to
 participate on this list hence the vague original post.

I can understand you might have various NDAs in place limiting what
you can and can't disclose.  

Unfortunately, without full information, it is difficult to provide a
full and proper diagnosis.

 I was just asking a general question to see if anyone else was
 having issues.

See, therein lies the problem.

As pointed out in recent congressional testimony by the esteemed
Senator from Alaska, the internets are comprised of very many
tangled-up tubes.  At any given time, something just isn't working.
Without source and destination IP addresses, it's difficult to
determine whether a problem is global in scope (entirely appropriate
for this list), or an end-user issue (inappropriate for this list,
though some folk may beg to differ :-), as suggested by your snippet
of 'traceroute' output -- and ultimately take corrective action.

 I have peering points with Broadwing(now level3), Sprint, ATT and
 MCI (now Verizon) that I can test for throughput from.

This phraseology is also a bit confusing, though sadly, all too common
these days.  Unless you're settlement-free, a better idea might be to
word this as I buy transit from... or perhaps more appropriately,
My cable MSO buys transit from...

 This was not just about home cable connectivity though when
 frontline starts to get calls I often use wget (very low overhead)
 to test throughput between my sites or to home my home box often
 times simulating the same sort of connectivity that a customer may
 have. There were customers that could not even get to level3.net
 yesterday which is their home page

Be that as it may, a little information would have helped greatly.
Had you said this sooner, and backed it up with some supporting data
such as IP addresses and perhaps 'wget' output, chances are we
wouldn't be having this discussion.

On the other hand, if you can't trust us, perhaps a better course of
action would be to open trouble tickets with your provider(s)...

-a

Re: Extreme Slowness

2006-10-26 Thread Brandon Galbraith 
Can you be more specific?-brandonOn 10/26/06, Elijah Savage [EMAIL PROTECTED] wrote:
Looks like level3 is having issues. Anyone know what is going on?
 
--Elijah Savage  |AOL IM:layer3rules
Senior Network Engineer |When it has to be switched or routed.
http://www.digitalrage.org|The Information Technology News Center- 
http://www.digitalrage.org/?page_id=46 for pgp public key 
-- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost

Re: Extreme Slowness

2006-10-26 Thread Elijah Savage 
It seems anything traversing level3 has very high latency along with what seems overloaded capacity as if they are running in a degraded mode I have connections with Time Warner, ATT, and MCI. Though I know it is not concrete it seems as if something is going on according to this http://www.internetpulse.net/ -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage [EMAIL PROTECTED] wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Re: Extreme Slowness

2006-10-26 Thread Elijah Savage 
Say like this traceroute. This is from TW to a Broadwing DS3.5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  153.267 ms  207.125 ms    tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  218.920 ms 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923 ms  57.770 ms 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)  254.145 ms    ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)  258.522 ms    ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)  227.223 ms 8  broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10)  231.451 ms   9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568 ms  22.511 ms10  216.140.14.17 (216.140.14.17)  34.751 ms  39.008 ms  46.644 ms11  p5-0-0.e0.cncn.broadwing.net (216.140.15.78)  32.065 ms  60.797 ms  54.766 ms12  67.98.17.122 (67.98.17.122)  44.772 ms  27.631 ms  30.655 ms13  * * * -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage [EMAIL PROTECTED] wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Re: Extreme Slowness

2006-10-26 Thread Elijah Savage 
Here is one from that browdwing ds3 to MCI well Verizon now. 5  tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  157.795 ms  179.050 ms    tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  205.087 ms 6  * * ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  50.134 ms 7  * ae-1-100.ebr1.Chicago1.Level3.net (4.69.132.41)  45.873 ms * 8  ae-2.ebr2.NewYork1.Level3.net (4.69.132.66)  66.346 ms  72.509 ms * -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage [EMAIL PROTECTED] wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Re: Extreme Slowness

2006-10-26 Thread Elijah Savage 
Seems to be all cleared up now. I had a couple of my customers even try to pull up their home site and could not get to it. For FYI :) I realize that ICMP is not the best way to test and it is not a true indication of slowness or the presence of a problem. On Oct 26, 2006, at 5:14 PM, Elijah Savage wrote:Here is one from that browdwing ds3 to MCI well Verizon now. 5  tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  157.795 ms  179.050 ms    tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  205.087 ms 6  * * ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  50.134 ms 7  * ae-1-100.ebr1.Chicago1.Level3.net (4.69.132.41)  45.873 ms * 8  ae-2.ebr2.NewYork1.Level3.net (4.69.132.66)  66.346 ms  72.509 ms * -- Elijah Savage               |  AOL IM:layer3rules Senior Network Engineer     |  When it has to be switched or routed. http://www.digitalrage.org  |  The Information Technology News Center- http://www.digitalrage.org/?page_id=46 for pgp public key On Oct 26, 2006, at 4:30 PM, Brandon Galbraith wrote:Can you be more specific?-brandonOn 10/26/06, Elijah Savage [EMAIL PROTECTED] wrote: Looks like level3 is having issues. Anyone know what is going on?  -- Elijah Savage               |  AOL IM:layer3rules  Senior Network Engineer     |  When it has to be switched or routed.  http://www.digitalrage.org  |  The Information Technology News Center-  http://www.digitalrage.org/?page_id=46 for pgp public key  -- Brandon GalbraithEmail: [EMAIL PROTECTED]AIM: brandong00Voice: 630.400.6992"A true pirate starts drinking before the sun hits the yard-arm. Ya. --thelost"

Re: Extreme Slowness

2006-10-26 Thread Aaron Glenn 


On 10/26/06, Elijah Savage [EMAIL PROTECTED] wrote:

Say like this traceroute. This is from TW to a Broadwing DS3.

5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)
153.267 ms  207.125 ms
tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)
218.920 ms
 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923 ms
57.770 ms
 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)
254.145 ms
ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)
258.522 ms
ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)
227.223 ms
 8  broadwing-level3-oc12.Chicago1.Level3.net
(209.0.225.10)  231.451 ms
 9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568 ms
22.511 ms
10  216.140.14.17 (216.140.14.17)  34.751 ms  39.008 ms  46.644 ms
11  p5-0-0.e0.cncn.broadwing.net (216.140.15.78)  32.065 ms  60.797 ms
54.766 ms
12  67.98.17.122 (67.98.17.122)  44.772 ms  27.631 ms  30.655 ms
13  * * *


Uhh, you do realize the end to end latency there (to hop 12, at least)
is ~30ms...not the 250ms+ you see on intermediate hops, right?

Re: Extreme Slowness

2006-10-26 Thread Elijah Savage 
Yes sir I did. This is now resolved. But thank you for noticing.  On Oct 26, 2006, at 7:11 PM, Aaron Glenn wrote:On 10/26/06, Elijah Savage [EMAIL PROTECTED] wrote: Say like this traceroute. This is from TW to a Broadwing DS3.5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)153.267 ms  207.125 ms    tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)218.920 ms 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923 ms57.770 ms 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)254.145 ms    ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)258.522 ms    ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)227.223 ms 8  broadwing-level3-oc12.Chicago1.Level3.net(209.0.225.10)  231.451 ms 9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568 ms22.511 ms10  216.140.14.17 (216.140.14.17)  34.751 ms  39.008 ms  46.644 ms11  p5-0-0.e0.cncn.broadwing.net (216.140.15.78)  32.065 ms  60.797 ms54.766 ms12  67.98.17.122 (67.98.17.122)  44.772 ms  27.631 ms  30.655 ms13  * * * Uhh, you do realize the end to end latency there (to hop 12, at least)is ~30ms...not the 250ms+ you see on intermediate hops, right?

Re: Extreme Slowness

2006-10-26 Thread Jeremy Chadwick 

On Thu, Oct 26, 2006 at 06:01:43PM -0400, Elijah Savage wrote:
 For FYI :) I realize that ICMP is not the best way to test and it is  
 not a true indication of slowness or the presence of a problem.

Which begs the same question I've asked in the recent past: then
what *is* a good diagnostic tool?  If ICMP is not the best way to
test, then what is?  What other globally-implemented layer 3 or
below protocols do we have available for troubleshooting?

Sure, UDP-based traceroute still relies on ICMP TTL exceeded
responses to work.  I've no idea what TCP traceroute relies on,
as I haven't looked at it.

-- 
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networkinghttp://www.parodius.com/ |
| UNIX Systems Administrator   Mountain View, CA, USA |
| Making life hard for others since 1977.   PGP: 4BD6C0CB |

ICMP PathMTU (was: Re: Extreme Slowness)

2006-10-26 Thread Jim Popovitch 

On Thu, 2006-10-26 at 18:01 -0400, Elijah Savage wrote:
 For FYI :) I realize that ICMP is not the best way to test and it is
 not a true indication of slowness or the presence of a problem.

Two questions for everybody...(any and all responses appreciated, even
if the reply mentions botnets or hammers ;-) )

1) What value is ICMP if everybody pretty much considers it's accuracy
suspect?

2) How does ICMP's suspect nature affect Path MTU?



-Jim P.

Re: ICMP PathMTU (was: Re: Extreme Slowness)

2006-10-26 Thread Randy Bush 

 1) What value is ICMP if everybody pretty much considers it's accuracy
suspect?

because for some uses, narrow precision is not needed.  like is it
pingable?  what is the current path?

my eyes are not highly accurate at measuring distance, color, size,
motion, ...  accurately.  but i'll keep them, thanks.

 2) How does ICMP's suspect nature affect Path MTU?

pmtu is hosed for other sicker reasons

randy

---

on precision
guy is at mummy exhibit in british museum
asks guard how old mummy is
guard says 2007 years
guy asks how he knows 2007
guard replies that he's been here seven years and mummy was 2000
years old when he got here

Re: Extreme Slowness

2006-10-26 Thread Adam Rothschild 

Elijah,

On 2006-10-26-16:34:18, Elijah Savage [EMAIL PROTECTED] wrote:
[HTML mail stripped]
 It seems anything traversing level3 has very high latency along with
 what seems overloaded capacity as if they are running in a degraded
 mode I have connections with Time Warner, ATT, and MCI [...]

On 2006-10-26-16:48:15, Elijah Savage [EMAIL PROTECTED] wrote:
[HTML mail stripped]
 Say like this traceroute. This is from TW to a Broadwing DS3.
 
 5  tenge-3-2.car1.Cincinnati1.Level3.net (4.78.216.13)  153.267 ms   
 207.125 ms
 tenge-3-1.car1.Cincinnati1.Level3.net (4.78.216.9)  218.920 ms
 6  ae-5-5.ebr2.Chicago1.Level3.net (4.69.132.206)  36.976 ms  26.923  
 ms  57.770 ms
 7  ge-11-0.core2.Chicago1.Level3.net (4.68.101.37)  254.145 ms
 ge-11-1.core2.Chicago1.Level3.net (4.68.101.101)  258.522 ms
 ge-11-2.core2.Chicago1.Level3.net (4.68.101.165)  227.223 ms
 8  broadwing-level3-oc12.Chicago1.Level3.net (209.0.225.10)  231.451 ms
 9  so-1-1-0.c1.gnwd.broadwing.net (216.140.15.1)  53.269 ms  35.568  
 ms  22.511 ms

Your postings appear to be missing two key pieces of information which
would help with the community diagnosis requested: source and
destination IP addresses.  From the information you did provide, one
can deduce that you're behind a TW/RoadRunner cable modem:

  13.216.78.4.IN-ADDR.ARPA domain name pointer 
tenge-3-2.car1.Cincinnati1.Level3.net
  14.216.78.4.IN-ADDR.ARPA domain name pointer 
ROADRUNNER.car1.Cincinnati1.Level3.net
  9.216.78.4.IN-ADDR.ARPA domain name pointer 
tenge-3-1.car1.Cincinnati1.Level3.net
  10.216.78.4.IN-ADDR.ARPA domain name pointer 
ROADRUNNER.car1.Cincinnati1.Level3.net

Now, the jitter and high latency you're seeing could be a result of
one or more factors, including but not limited to RF/plant issues, TWC
running their transport and/or Level(3) transit hot (which seems to be
a common occurrence these days), ECMP across two circuits of uneven
loading, or your neighbor might be jacking wifi and downloading a
bunch of torrents -- we, the readers, just don't know.

Of note when performing armchair troubleshooting across Level(3)'s
network: the 'ebr's (PTR record of ebr*.{pop}.level3.net == Force10
E1200; Experimental Backbone Router?) tend to drop a lot of diagnostic
traffic (such as, say, 'ping' and 'traceroute') as a part of overly
aggressive control-plane policers.  This loss is, of course, strictly
cosmetic, and has no bearing on end-to-end performance.  Hence, the
old to it, not through it rule applies.

smokeping[1] and iperf[2] (to end hosts) are your friends.

As an aside, I've noticed your string of postings today were all
HTML-tagged.  While not expressly forbidden (or even discouraged) by
the current Mailing List AUP, this is generally regarded as bad form;
you might wish to reconfigure your mail client accordingly...

Hope this helps,
-a

[1] http://oss.oetiker.ch/smokeping/
[2] http://dast.nlanr.net/Projects/Iperf/