Re: packet loss question

2016-07-12 Thread cpolish 
On 2016-07-12 03:25, Sean Donelan wrote:
> RFC791 was written during the internet's anti-standard era.
> 
> We reject: kings, presidents and voting. We believe in: rough consensus and
> running code

Hi Sean,

Lovely quote and all, but... do you mean that when RFC791 was
drafted the IETF didn't issue 'standards'? RFC791 was written by
Jon Postel for DARPA and AFAIKT is foundational. It's referenced
by more than 420 RFCs. It begins, "This document specifies the
DoD Standard Internet Protocol." Seems about as official as
times permitted. 

Point was, 576 bytes is the minimum MTU for transporting IP
datagrams. Also see RFC1122/3.3.2, which references
RFC791 of course.

Best regards,
-- 
Charles Polisher
"Pedentic, I?"

Re: packet loss question

2016-07-12 Thread Sean Donelan 

On Mon, 11 Jul 2016, cpol...@surewest.net wrote:

Thanks for identifying the source, I wish more people did this.
My nitpick is that RFC791 doesn't label MTU=68 as "standard";
it says (section 3.2, p.25):


RFC791 was written during the internet's anti-standard era.

We reject: kings, presidents and voting. We believe in: rough consensus 
and running code

Re: packet loss question

2016-07-11 Thread cpolish 
On 2016-07-11 11:26, Mark Andrews wrote:
> 
> In message <25577fe1-6366-4d6d-b82e-a779193cb...@beckman.org>, Mel Beckman 
> writ
> The Internet Standard MTU's are 68 octets for IPv4 (RFC 791) and
> 1280 octets for IPv6 (RFC 2460).
>
> Every size greater than those is subject to negotiation.  Now most
> paths pass packets greater than those values.  Ethernet is very
> common and passes 1500.

Thanks for identifying the source, I wish more people did this. 
My nitpick is that RFC791 doesn't label MTU=68 as "standard"; 
it says (section 3.2, p.25):

Every internet module must be able to forward a datagram of 68
octets without further fragmentation.  

More to your point, RFC791 also says (section 3.1, p. 13):

All hosts must be prepared to accept datagrams of up to 576
octets (whether they arrive whole or in fragments).  It is
recommended that hosts only send datagrams larger than 576
octets if they have assurance that the destination is prepared
to accept the larger datagrams.
 
> Encapsulated / translate traffic is also very common and has MTUs
> < 1500 and affects BOTH IPv4 and IPv6 data streams and will become
> more so as we move from dual stack to IPv6 only where IPv4 is a
> service running on top of IPv6.

-- 
Charles Polisher

Re: packet loss question

2016-07-11 Thread James Greig 
That's the one:) 

Kind regards

James Greig 

> On 11 Jul 2016, at 01:26, Mel Beckman  wrote:
> 
> James,
> 
> You may be thinking of this presentation: 
> 
> https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf
> 
> -mel beckman
> 
>> On Jul 10, 2016, at 4:49 PM, James Greig  wrote:
>> 
>> There was a useful nanog presentation somewhere that explained this really 
>> well in particular reading traceroutes correctly
>> 
>> Kind regards
>> 
>> James Greig 
>> 
>>> On 7 Jul 2016, at 20:17, Phillip Lynn  wrote:
>>> 
>>> Hi all,
>>> 
>>> I am writing because I do not understand what is happening.  I ran mtr 
>>> against our email server and www.teco.comand below are the results.  I am 
>>> not a network engineer so I am at a loss.  I think what I am seeing is 
>>> maybe a hand off issue, between Frontier and Level3Miami2. If I am correct 
>>> then what can I do?
>>> 
>>> My system is running Centos 6.5 Linux.
>>> 
>>> Thanks,
>>> 
>>> Phillip
>>> 
>>> 
>>> 
>>> (! 1011)-> sudo mtr -r netwolves.securence.com
>>> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>>> 1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>>> 2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%103.2 2.0 1.0   4.3   1.2
>>> 3. 172.99.44.214 0.0%104.0 4.9 2.3   6.9   1.5
>>> 4. ae8---0.scr02.mias.fl.fronti  0.0%109.3 9.1 7.5   9.8   1.0
>>> 5. ae1---0.cbr01.mias.fl.fronti  0.0%108.9   9.1   7.6 9.7 0.7
>>> 6. lag-101.ear3.Miami2.Level3.n 80.0%109.0   8.9   8.8 9.0 0.1
>>> 7. 10ge9-14.core1.mia1.he.net0.0%10   14.3 13.0 7.6  18.1   4.3
>>> 8. 10ge1-1.core1.atl1.he.net 0.0%10   25.6  33.2 22.4  99.7  23.6
>>> 9. 10ge10-4.core1.chi1.he.net0.0%10   45.6  51.8 45.5  82.7  12.5
>>> 10. 100ge14-2.core1.msp1.he.net   0.0%10   53.6  63.9 53.6 125.2  21.8
>>> 11. t4-2-usi-cr02-mpls-usinterne  0.0%10   53.2  73.1 53.2 225.6  54.0
>>> 12. v102.usi-cr04-mtka.usinterne  0.0%10   53.2  53.9 53.2  55.3   0.6
>>> 13. netwolves.securence.com   0.0%10   53.4  53.9 53.4  55.4   0.7
>>> 
>>> (! 1014)-> sudo mtr -r www.teco.com
>>> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>>> 1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>>> 2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%10  104.8 81.4 1.1 113.2  43.2
>>> 3. 172.99.47.198 0.0%10  115.0 77.8 2.9 115.0  40.2
>>> 4. ae7---0.scr01.mias.fl.fronti  0.0%10  111.1 80.2 8.5 113.5  41.3
>>> 5. ae0---0.cbr01.mias.fl.fronti  0.0%10  105.9  82.2   7.6 115.4 33.8
>>> 6. lag-101.ear3.Miami2.Level3.n 70.0%10  116.1  80.2   8.5 116.1 62.0
>>> 7. NTT-level3-80G.Miami.Level3.  0.0%10  110.0 81.5 9.0 120.3  41.9
>>> 8. ae-3.r20.miamfl02.us.bb.gin.  0.0%10  119.8  84.0 10.0 119.8  38.5
>>> 9. ae-4.r23.asbnva02.us.bb.gin. 10.0%10  137.4 107.6 30.1 142.7  45.7
>>> 10. ae-2.r05.asbnva02.us.bb.gin.  0.0%10  135.0 109.9 36.6 140.0  39.1
>>> 11. xe-0-9-0-8.r05.asbnva02.us.c  0.0%10  147.5 125.6 49.4 165.5  41.1
>>> 12. 24.52.112.21  0.0%10  158.6 124.0 49.6 161.3  41.5
>>> 13. 24.52.112.42  0.0%10  151.0 127.7 52.2 159.0  41.2
>>> 14. ???  100.0100.0 0.0 0.0   0.0   0.0
>>> 
>>> -- 
>>> Phillip Lynn
>>> Software Engineer III
>>> NetWolves
>>> Phone:813-579-3214
>>> Fax:813-882-0209
>>> Email: phillip.l...@netwolves.com
>>> www.netwolves.com
>>

Re: packet loss question

2016-07-10 Thread Mark Andrews 

In message <25577fe1-6366-4d6d-b82e-a779193cb...@beckman.org>, Mel Beckman writ
es:
> Philip,
>
> Quite often slow Web page loading and email transport -- termed an
> application-layer problem because basic transport seems unaffected -- is
> due to DNS problems, particularly reverse DNS for the IP addresses
> originating your Web queries. If you have non-existent or intermittent
> IN-ADDR entries for those IPs, the remote Web servers can be timing out
> if they have older configurations that, for example, do DNS lookups in
> order to log HTTP requests and block on completion, resulting in
> timeouts. Use "nslookup x.x.x.x" command line queries (nslookup is on
> Windows, Mac and UNIX/Linux) to see if you can resolve the public IP
> addresses your users original queries from. You can find those addresses
> by visiting http://whatismyip.com from a problem desktop.
>
> A second common cause of app-specific throughput problems, particularly
> where email is involved, is failed MTU discovery. The standard Internet
> MTU is 1500 bytes, but sometimes a router misconfiguration or change in
> encapsulation type along the path through your ISP lowers that to, say,
> 1492 or 1486 bytes (MTU is in increments of 8). The result is that
> whenever your web or email client sends a maximum MTU packet, the packet
> is dropped, resulting in connection impairment. Most HTTP and Email
> packets are not max-MTU in size, so you get very uneven performance
> simulating network congestion.

The Internet Standard MTU's are 68 octets for IPv4 (RFC 791) and
1280 octets for IPv6 (RFC 2460).

Every size greater than those is subject to negotiation.  Now most
paths pass packets greater than those values.  Ethernet is very
common and passes 1500.

Encapsulated / translate traffic is also very common and has MTUs
< 1500 and affects BOTH IPv4 and IPv6 data streams and will become
more so as we move from dual stack to IPv6 only where IPv4 is a
service running on top of IPv6.

> You can force the MTU to a lower number at your border to test this. You
> typically do this at your firewall; it's a setting on the WAN interface
> config. Temporarily lower that value dramatically to something like 1440
> and see if your problem goes away. If it does, you may need to
> permanently reduce MTU, so you should try other divisible-by-8 values --
> 1492, 1486, 1478, etc -- until you find the largest one that works. I
> commonly see this when a customer switches ISPs from DSL to Cable. Cable
> providers are fond of stealing 8 or 16 bytes for their CMT headers in a
> way that breaks MTU discovery.
>
> A third frequent application-layer throughout debillitator is IPv6
> misconfiguration. If you support IPv6 for your end users, they may be
> getting directed to IPv6 web or mail servers (which are generally
> preferred via DNS) but thwarted by IPv6 transport issues, which could be
> as simple as routing or MTU, or as complex as an invisible 6-over-4 NAT
> somewhere (such as a your upstream ISP). These problems generally require
> an IPv6-competent network engineer to resolve, but you can test by
> disabling IPv6 on your network (which also requires an IPv6-competent
> network engineer :)
>
> I'm always amazed at how often these three causes are at the root of
> performance problems. So it's worth investigating each.
>
>  -mel beckman
>
> > On Jul 8, 2016, at 6:02 AM, Phillip Lynn 
> wrote:
> >
> >> On 07/07/2016 03:52 PM, Ken Chase wrote:
> >> No offence, but i swear that mtr should come with a license to use it.
> I get more
> >> questions from people accusing us of network issues with mtr in hand...
> >>
> >> You shoudlnt care that there's 80% packet loss in the middle of your
> route, unless
> >> you have actual traffic to lag-101.ear3.miami2.level3.net. I suspect
> you dont.
> >> (If you did, you'd have mtr'd to it directly of course.)
> >>
> >> As for your second trace, the sudden jump from 0% on 2nd last hop to
> 100% last
> >> hop packetloss seems like firewalling to me. (long discussion about the
> >> probabilities of getting 5 0%pl hops in a row and 100% on an
> unfirewalled
> >> endpoint elided. TL;DR: use more packets in your test -i 0.1 -c 100
> thanks).
> >>
> >> If you have 0% packetloss to your target endpoint, is there an issue
> here?
> >> What caused you to mtr?  0% pl is pretty good. You could play quake 1.0
> >> through that pl and ping time. The +20ms ATL<>CHI jump in the route
> you'd have
> >> to take up with einstein/bill nye/$deity.
> >>
> >> For the 2nd trace, the 1st hop is your latency issue (plus the big
> jump from
> >> miami<>ashburn, again the limit is c.)
> >>
> >> ICMP is allowed to be dropped by intervening routers. Someone will
> quote an RFC
> >> at us shortly.
> >>
> >> Mtr without a return route is not that useful in figuring out
> packetloss
> >> because pl requires the packet make it there and back. Pl could be
> anywhere on
> >> the return route, which is probably not symmetrical. The

Re: packet loss question

2016-07-10 Thread Mel Beckman 
James,

You may be thinking of this presentation: 

https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf

 -mel beckman

> On Jul 10, 2016, at 4:49 PM, James Greig  wrote:
> 
> There was a useful nanog presentation somewhere that explained this really 
> well in particular reading traceroutes correctly
> 
> Kind regards
> 
> James Greig 
> 
>> On 7 Jul 2016, at 20:17, Phillip Lynn  wrote:
>> 
>> Hi all,
>> 
>> I am writing because I do not understand what is happening.  I ran mtr 
>> against our email server and www.teco.comand below are the results.  I am 
>> not a network engineer so I am at a loss.  I think what I am seeing is maybe 
>> a hand off issue, between Frontier and Level3Miami2. If I am correct then 
>> what can I do?
>> 
>> My system is running Centos 6.5 Linux.
>> 
>> Thanks,
>> 
>> Phillip
>> 
>> 
>> 
>> (! 1011)-> sudo mtr -r netwolves.securence.com
>> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>> 1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>> 2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%103.2 2.0 1.0   4.3   1.2
>> 3. 172.99.44.214 0.0%104.0 4.9 2.3   6.9   1.5
>> 4. ae8---0.scr02.mias.fl.fronti  0.0%109.3 9.1 7.5   9.8   1.0
>> 5. ae1---0.cbr01.mias.fl.fronti  0.0%108.9   9.1   7.6 9.7 0.7
>> 6. lag-101.ear3.Miami2.Level3.n 80.0%109.0   8.9   8.8 9.0 0.1
>> 7. 10ge9-14.core1.mia1.he.net0.0%10   14.3 13.0 7.6  18.1   4.3
>> 8. 10ge1-1.core1.atl1.he.net 0.0%10   25.6  33.2 22.4  99.7  23.6
>> 9. 10ge10-4.core1.chi1.he.net0.0%10   45.6  51.8 45.5  82.7  12.5
>> 10. 100ge14-2.core1.msp1.he.net   0.0%10   53.6  63.9 53.6 125.2  21.8
>> 11. t4-2-usi-cr02-mpls-usinterne  0.0%10   53.2  73.1 53.2 225.6  54.0
>> 12. v102.usi-cr04-mtka.usinterne  0.0%10   53.2  53.9 53.2  55.3   0.6
>> 13. netwolves.securence.com   0.0%10   53.4  53.9 53.4  55.4   0.7
>> 
>> (! 1014)-> sudo mtr -r www.teco.com
>> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>> 1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>> 2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%10  104.8 81.4 1.1 113.2  43.2
>> 3. 172.99.47.198 0.0%10  115.0 77.8 2.9 115.0  40.2
>> 4. ae7---0.scr01.mias.fl.fronti  0.0%10  111.1 80.2 8.5 113.5  41.3
>> 5. ae0---0.cbr01.mias.fl.fronti  0.0%10  105.9  82.2   7.6 115.4 33.8
>> 6. lag-101.ear3.Miami2.Level3.n 70.0%10  116.1  80.2   8.5 116.1 62.0
>> 7. NTT-level3-80G.Miami.Level3.  0.0%10  110.0 81.5 9.0 120.3  41.9
>> 8. ae-3.r20.miamfl02.us.bb.gin.  0.0%10  119.8  84.0 10.0 119.8  38.5
>> 9. ae-4.r23.asbnva02.us.bb.gin. 10.0%10  137.4 107.6 30.1 142.7  45.7
>> 10. ae-2.r05.asbnva02.us.bb.gin.  0.0%10  135.0 109.9 36.6 140.0  39.1
>> 11. xe-0-9-0-8.r05.asbnva02.us.c  0.0%10  147.5 125.6 49.4 165.5  41.1
>> 12. 24.52.112.21  0.0%10  158.6 124.0 49.6 161.3  41.5
>> 13. 24.52.112.42  0.0%10  151.0 127.7 52.2 159.0  41.2
>> 14. ???  100.0100.0 0.0 0.0   0.0   0.0
>> 
>> -- 
>> Phillip Lynn
>> Software Engineer III
>> NetWolves
>> Phone:813-579-3214
>> Fax:813-882-0209
>> Email: phillip.l...@netwolves.com
>> www.netwolves.com
>> 
>

Re: packet loss question

2016-07-10 Thread James Greig 
There was a useful nanog presentation somewhere that explained this really well 
in particular reading traceroutes correctly

Kind regards

James Greig 

> On 7 Jul 2016, at 20:17, Phillip Lynn  wrote:
> 
> Hi all,
> 
>  I am writing because I do not understand what is happening.  I ran mtr 
> against our email server and www.teco.comand below are the results.  I am not 
> a network engineer so I am at a loss.  I think what I am seeing is maybe a 
> hand off issue, between Frontier and Level3Miami2. If I am correct then what 
> can I do?
> 
>  My system is running Centos 6.5 Linux.
> 
> Thanks,
> 
> Phillip
> 
> 
> 
> (! 1011)-> sudo mtr -r netwolves.securence.com
> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%103.2 2.0 1.0   4.3   1.2
>  3. 172.99.44.214 0.0%104.0 4.9 2.3   6.9   1.5
>  4. ae8---0.scr02.mias.fl.fronti  0.0%109.3 9.1 7.5   9.8   1.0
>  5. ae1---0.cbr01.mias.fl.fronti  0.0%108.9   9.1   7.6 9.7 0.7
>  6. lag-101.ear3.Miami2.Level3.n 80.0%109.0   8.9   8.8 9.0 0.1
>  7. 10ge9-14.core1.mia1.he.net0.0%10   14.3 13.0 7.6  18.1   4.3
>  8. 10ge1-1.core1.atl1.he.net 0.0%10   25.6  33.2 22.4  99.7  23.6
>  9. 10ge10-4.core1.chi1.he.net0.0%10   45.6  51.8 45.5  82.7  12.5
> 10. 100ge14-2.core1.msp1.he.net   0.0%10   53.6  63.9 53.6 125.2  21.8
> 11. t4-2-usi-cr02-mpls-usinterne  0.0%10   53.2  73.1 53.2 225.6  54.0
> 12. v102.usi-cr04-mtka.usinterne  0.0%10   53.2  53.9 53.2  55.3   0.6
> 13. netwolves.securence.com   0.0%10   53.4  53.9 53.4  55.4   0.7
> 
> (! 1014)-> sudo mtr -r www.teco.com
> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%10  104.8 81.4 1.1 113.2  43.2
>  3. 172.99.47.198 0.0%10  115.0 77.8 2.9 115.0  40.2
>  4. ae7---0.scr01.mias.fl.fronti  0.0%10  111.1 80.2 8.5 113.5  41.3
>  5. ae0---0.cbr01.mias.fl.fronti  0.0%10  105.9  82.2   7.6 115.4 33.8
>  6. lag-101.ear3.Miami2.Level3.n 70.0%10  116.1  80.2   8.5 116.1 62.0
>  7. NTT-level3-80G.Miami.Level3.  0.0%10  110.0 81.5 9.0 120.3  41.9
>  8. ae-3.r20.miamfl02.us.bb.gin.  0.0%10  119.8  84.0 10.0 119.8  38.5
>  9. ae-4.r23.asbnva02.us.bb.gin. 10.0%10  137.4 107.6 30.1 142.7  45.7
> 10. ae-2.r05.asbnva02.us.bb.gin.  0.0%10  135.0 109.9 36.6 140.0  39.1
> 11. xe-0-9-0-8.r05.asbnva02.us.c  0.0%10  147.5 125.6 49.4 165.5  41.1
> 12. 24.52.112.21  0.0%10  158.6 124.0 49.6 161.3  41.5
> 13. 24.52.112.42  0.0%10  151.0 127.7 52.2 159.0  41.2
> 14. ???  100.0100.0 0.0 0.0   0.0   0.0
> 
> -- 
> Phillip Lynn
> Software Engineer III
> NetWolves
> Phone:813-579-3214
> Fax:813-882-0209
> Email: phillip.l...@netwolves.com
> www.netwolves.com
>

Re: packet loss question

2016-07-08 Thread jmkeller 

On 2016-07-07 11:53 PM, William Herrin wrote:

On Thu, Jul 7, 2016 at 3:52 PM, Ken Chase  wrote:
ICMP is allowed to be dropped by intervening routers. Someone will 
quote an RFC

at us shortly.


Hi Ken,

That's not correct. Routers might not generate an ICMP time-exceeded
packet for every packet whose TTL reaches zero, but that's not the
same thing. Routers dropping ICMP packets in transit would be bad.
Protocols like TCP depend on path MTU discovery and path MTU discovery
critically depends on ICMP.

Regards,
Bill Herrin



All we are seeing here is control plane filtering by intermediate 
routers.   Unless packet loss numbers start at a router and hops past it 
show the same or higher losses it's not an actual issue with the 
transport path at that hop.   Outside of your own domain of 
administrative control, you can't rely on intermediate routers 
responding to ICMP (either filtered completely or rate limited 
responses).


--
James

Re: packet loss question

2016-07-08 Thread William Herrin 
On Fri, Jul 8, 2016 at 9:09 AM, Ken Chase  wrote:
> And we know the whole internet observes handling
> mtu discovery properly
> and doesnt just firewall all ICMP because 'hackers'.
>
> (OP's issue may well be MTU discovery, esp if he's on
> broadband. Don't have
> enough details. I just solved this exact problem a couple
> weeks ago for a
> client with an UBNT ERX by turning on it's MTU hacking
> feature. Sites that
> engaged in ICMP mtu blocking included cnn.com.)

Heh. And because life isn't interesting enough, Amazon AWS has started
defaulting their larger VMs to a 9001 byte interface MTU.


> I meant routers are allowed to drop ICMP request
> packets to themselves, not
> the packets to be transitted. I wasnt clear.

I think you really meant echo-request packets to themselves, right? ;)

Regards,
Bill Herrin


-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Owner, Dirtside Systems . Web:

Re: packet loss question

2016-07-08 Thread Mel Beckman 
Philip,

Quite often slow Web page loading and email transport -- termed an 
application-layer problem because basic transport seems unaffected -- is due to 
DNS problems, particularly reverse DNS for the IP addresses originating your 
Web queries. If you have non-existent or intermittent IN-ADDR entries for those 
IPs, the remote Web servers can be timing out if they have older configurations 
that, for example, do DNS lookups in order to log HTTP requests and block on 
completion, resulting in timeouts. Use "nslookup x.x.x.x" command line queries 
(nslookup is on Windows, Mac and UNIX/Linux) to see if you can resolve the 
public IP addresses your users original queries from. You can find those 
addresses by visiting http://whatismyip.com from a problem desktop.

A second common cause of app-specific throughput problems, particularly where 
email is involved, is failed MTU discovery. The standard Internet MTU is 1500 
bytes, but sometimes a router misconfiguration or change in encapsulation type 
along the path through your ISP lowers that to, say, 1492 or 1486 bytes (MTU is 
in increments of 8). The result is that whenever your web or email client sends 
a maximum MTU packet, the packet is dropped, resulting in connection 
impairment. Most HTTP and Email packets are not max-MTU in size, so you get 
very uneven performance simulating network congestion.

You can force the MTU to a lower number at your border to test this. You 
typically do this at your firewall; it's a setting on the WAN interface config. 
Temporarily lower that value dramatically to something like 1440 and see if 
your problem goes away. If it does, you may need to permanently reduce MTU, so 
you should try other divisible-by-8 values -- 1492, 1486, 1478, etc -- until 
you find the largest one that works. I commonly see this when a customer 
switches ISPs from DSL to Cable. Cable providers are fond of stealing 8 or 16 
bytes for their CMT headers in a way that breaks MTU discovery. 

A third frequent application-layer throughout debillitator is IPv6 
misconfiguration. If you support IPv6 for your end users, they may be getting 
directed to IPv6 web or mail servers (which are generally preferred via DNS) 
but thwarted by IPv6 transport issues, which could be as simple as routing or 
MTU, or as complex as an invisible 6-over-4 NAT somewhere (such as a your 
upstream ISP). These problems generally require an IPv6-competent network 
engineer to resolve, but you can test by disabling IPv6 on your network (which 
also requires an IPv6-competent network engineer :)

I'm always amazed at how often these three causes are at the root of 
performance problems. So it's worth investigating each.

 -mel beckman

> On Jul 8, 2016, at 6:02 AM, Phillip Lynn  wrote:
> 
>> On 07/07/2016 03:52 PM, Ken Chase wrote:
>> No offence, but i swear that mtr should come with a license to use it. I get 
>> more
>> questions from people accusing us of network issues with mtr in hand...
>> 
>> You shoudlnt care that there's 80% packet loss in the middle of your route, 
>> unless
>> you have actual traffic to lag-101.ear3.miami2.level3.net. I suspect you 
>> dont.
>> (If you did, you'd have mtr'd to it directly of course.)
>> 
>> As for your second trace, the sudden jump from 0% on 2nd last hop to 100% 
>> last
>> hop packetloss seems like firewalling to me. (long discussion about the
>> probabilities of getting 5 0%pl hops in a row and 100% on an unfirewalled
>> endpoint elided. TL;DR: use more packets in your test -i 0.1 -c 100 thanks).
>> 
>> If you have 0% packetloss to your target endpoint, is there an issue here?
>> What caused you to mtr?  0% pl is pretty good. You could play quake 1.0
>> through that pl and ping time. The +20ms ATL<>CHI jump in the route you'd 
>> have
>> to take up with einstein/bill nye/$deity.
>> 
>> For the 2nd trace, the 1st hop is your latency issue (plus the big jump from
>> miami<>ashburn, again the limit is c.)
>> 
>> ICMP is allowed to be dropped by intervening routers. Someone will quote an 
>> RFC
>> at us shortly.
>> 
>> Mtr without a return route is not that useful in figuring out packetloss
>> because pl requires the packet make it there and back. Pl could be anywhere 
>> on
>> the return route, which is probably not symmetrical. The internet stopped
>> being symetrical about 20+ years ago (if it ever even loosely was), so get a 
>> friend
>> to send you an mtr to your ip from the farside.
>> 
>> (I remember a project long ago, some cgi-bin (yeah that long ago) that was
>> basically a full-path forward+reverse traceroute you could hit on a selected
>> server at the provider.  Rather handy. not sure if its still a thing, or 
>> what it was
>> called.)
>> 
>> /kc
>> 
>> 
>> On Thu, Jul 07, 2016 at 03:17:40PM -0400, Phillip Lynn said:
>>   >Hi all,
>>   >
>>   >  I am writing because I do not understand what is happening.  I ran mtr
>>   >against our email server and www.teco.comand below are the results.

Re: packet loss question

2016-07-08 Thread Ken Chase 
And we know the whole internet observes handling mtu discovery properly
and doesnt just firewall all ICMP because 'hackers'. 

(OP's issue may well be MTU discovery, esp if he's on broadband. Don't have
enough details. I just solved this exact problem a couple weeks ago for a
client with an UBNT ERX by turning on it's MTU hacking feature. Sites that
engaged in ICMP mtu blocking included cnn.com.)

I meant routers are allowed to drop ICMP request packets to themselves, not
the packets to be transitted. I wasnt clear.

/kc


On Thu, Jul 07, 2016 at 11:53:38PM -0400, William Herrin said:
  >On Thu, Jul 7, 2016 at 3:52 PM, Ken Chase  wrote:
  >> ICMP is allowed to be dropped by intervening routers. Someone will quote 
an RFC
  >> at us shortly.
  >
  >Hi Ken,
  >
  >That's not correct. Routers might not generate an ICMP time-exceeded
  >packet for every packet whose TTL reaches zero, but that's not the
  >same thing. Routers dropping ICMP packets in transit would be bad.
  >Protocols like TCP depend on path MTU discovery and path MTU discovery
  >critically depends on ICMP.
  >
  >Regards,
  >Bill Herrin
  >
  >
  >-- 
  >William Herrin  her...@dirtside.com  b...@herrin.us
  >Owner, Dirtside Systems . Web: 

-- 
Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto 
Canada
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front 
St. W.

Re: packet loss question

2016-07-08 Thread Phillip Lynn 

On 07/07/2016 03:52 PM, Ken Chase wrote:

No offence, but i swear that mtr should come with a license to use it. I get 
more
questions from people accusing us of network issues with mtr in hand...

You shoudlnt care that there's 80% packet loss in the middle of your route, 
unless
you have actual traffic to lag-101.ear3.miami2.level3.net. I suspect you dont.
(If you did, you'd have mtr'd to it directly of course.)

As for your second trace, the sudden jump from 0% on 2nd last hop to 100% last
hop packetloss seems like firewalling to me. (long discussion about the
probabilities of getting 5 0%pl hops in a row and 100% on an unfirewalled
endpoint elided. TL;DR: use more packets in your test -i 0.1 -c 100 thanks).

If you have 0% packetloss to your target endpoint, is there an issue here?
What caused you to mtr?  0% pl is pretty good. You could play quake 1.0
through that pl and ping time. The +20ms ATL<>CHI jump in the route you'd have
to take up with einstein/bill nye/$deity.

For the 2nd trace, the 1st hop is your latency issue (plus the big jump from
miami<>ashburn, again the limit is c.)

ICMP is allowed to be dropped by intervening routers. Someone will quote an RFC
at us shortly.

Mtr without a return route is not that useful in figuring out packetloss
because pl requires the packet make it there and back. Pl could be anywhere on
the return route, which is probably not symmetrical. The internet stopped
being symetrical about 20+ years ago (if it ever even loosely was), so get a 
friend
to send you an mtr to your ip from the farside.

(I remember a project long ago, some cgi-bin (yeah that long ago) that was
basically a full-path forward+reverse traceroute you could hit on a selected
server at the provider.  Rather handy. not sure if its still a thing, or what 
it was
called.)

/kc


On Thu, Jul 07, 2016 at 03:17:40PM -0400, Phillip Lynn said:
   >Hi all,
   >
   >  I am writing because I do not understand what is happening.  I ran mtr
   >against our email server and www.teco.comand below are the results.  I am
   >not a network engineer so I am at a loss.  I think what I am seeing is
   >maybe a hand off issue, between Frontier and Level3Miami2. If I am correct
   >then what can I do?
   >
   >  My system is running Centos 6.5 Linux.
   >
   >Thanks,
   >
   >Phillip
   >
   >
   >
   >(! 1011)-> sudo mtr -r netwolves.securence.com
   >HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
   >  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
   >  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%103.2 2.0 1.0   4.3   1.2
   >  3. 172.99.44.214 0.0%104.0 4.9 2.3   6.9   1.5
   >  4. ae8---0.scr02.mias.fl.fronti  0.0%109.3 9.1 7.5   9.8   1.0
   >  5. ae1---0.cbr01.mias.fl.fronti  0.0%108.9   9.1   7.6 9.7 0.7
   >  6. lag-101.ear3.Miami2.Level3.n 80.0%109.0   8.9   8.8 9.0 0.1
   >  7. 10ge9-14.core1.mia1.he.net0.0%10   14.3 13.0 7.6  18.1   4.3
   >  8. 10ge1-1.core1.atl1.he.net 0.0%10   25.6  33.2 22.4  99.7  23.6
   >  9. 10ge10-4.core1.chi1.he.net0.0%10   45.6  51.8 45.5  82.7  12.5
   > 10. 100ge14-2.core1.msp1.he.net   0.0%10   53.6  63.9 53.6 125.2  21.8
   > 11. t4-2-usi-cr02-mpls-usinterne  0.0%10   53.2  73.1 53.2 225.6  54.0
   > 12. v102.usi-cr04-mtka.usinterne  0.0%10   53.2  53.9 53.2  55.3   0.6
   > 13. netwolves.securence.com   0.0%10   53.4  53.9 53.4  55.4   0.7
   >
   >(! 1014)-> sudo mtr -r www.teco.com
   >HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
   >  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
   >  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%10  104.8 81.4 1.1 113.2  43.2
   >  3. 172.99.47.198 0.0%10  115.0 77.8 2.9 115.0  40.2
   >  4. ae7---0.scr01.mias.fl.fronti  0.0%10  111.1 80.2 8.5 113.5  41.3
   >  5. ae0---0.cbr01.mias.fl.fronti  0.0%10  105.9  82.2   7.6 115.4 33.8
   >  6. lag-101.ear3.Miami2.Level3.n 70.0%10  116.1  80.2   8.5 116.1 62.0
   >  7. NTT-level3-80G.Miami.Level3.  0.0%10  110.0 81.5 9.0 120.3  41.9
   >  8. ae-3.r20.miamfl02.us.bb.gin.  0.0%10  119.8  84.0 10.0 119.8  38.5
   >  9. ae-4.r23.asbnva02.us.bb.gin. 10.0%10  137.4 107.6 30.1 142.7  45.7
   > 10. ae-2.r05.asbnva02.us.bb.gin.  0.0%10  135.0 109.9 36.6 140.0  39.1
   > 11. xe-0-9-0-8.r05.asbnva02.us.c  0.0%10  147.5 125.6 49.4 165.5  41.1
   > 12. 24.52.112.21  0.0%10  158.6 124.0 49.6 161.3  41.5
   > 13. 24.52.112.42  0.0%10  151.0 127.7 52.2 159.0  41.2
   > 14. ???  100.0100.0 0.0 0.0   0.0   0.0
   >
   >--
   >Phillip Lynn
   >Software Engineer III
   >NetWolves
   >Phone:813-579-3214
   >Fax:813-882-0209
   >Email: phillip.l...@netwolves.com
   >www.netwolves.com
   >



None taken,

  We are having issues with our email and loading some web pages. I 
used mtr to try and find if there is a

Re: packet loss question

2016-07-07 Thread William Herrin 
On Thu, Jul 7, 2016 at 3:52 PM, Ken Chase  wrote:
> ICMP is allowed to be dropped by intervening routers. Someone will quote an 
> RFC
> at us shortly.

Hi Ken,

That's not correct. Routers might not generate an ICMP time-exceeded
packet for every packet whose TTL reaches zero, but that's not the
same thing. Routers dropping ICMP packets in transit would be bad.
Protocols like TCP depend on path MTU discovery and path MTU discovery
critically depends on ICMP.

Regards,
Bill Herrin


-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Owner, Dirtside Systems . Web:

Re: packet loss question

2016-07-07 Thread Brielle Bruns 

On 7/7/16 3:50 PM, Mel Beckman wrote:

Ken,

I should have made clear I wasn't replying to you. I was replying to Brielle's 
comment:


>  Is it bad that the first thing that came to mind is "Oh FFS, another troll"?
I'd never say I was always knowledgeable, but after the thread the other 
day, and just general stress lately, some reactions are hard not to give 
into :)


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: packet loss question

2016-07-07 Thread Mel Beckman 
Ken,

I should have made clear I wasn't replying to you. I was replying to Brielle's 
comment:

>  Is it bad that the first thing that came to mind is "Oh FFS, another troll"?

 -mel beckman

> On Jul 7, 2016, at 2:35 PM, Ken Chase  wrote:
> 
> On Thu, Jul 07, 2016 at 08:32:19PM +, Mel Beckman said:
>> Yes. It indicates that there was never a time when you did not know 
>> everything :)
>> 
>> -mel beckman
> 
> The issue isnt knowing everything, it's making accusations of issues while 
> you still
> dont know how much you dont know. (~D. Rumsfeld) -- My customers in a nutshell
> (they pay to be able to yell about random stuff I guess, and I provide that 
> service!).
> 
> The OP didnt make any accusations however, and just asked what was going on 
> (sorry
> if I sounded harsh in reply). Once, Google having a 8.8.8.8 failure locally on
> its (anycast?) dns servers resulted in dozens of calls to us "your server
> hosting our site must be down!! Our website isnt working! People are calling 
> us!".
> 
> Most of my work is with these situations is spent proving it's not our fault.
> Mtr makes it very hard because it's a very subtle tool, and only gives partial
> information. (I still think mtr is a killer app though!)
> 
> consider this (fake, example) trace:
> 
>  6. 100ge13-1.core1.chi1.he.net0.0%10 
>  7. 100ge14-1.core2.chi1.he.net0.0%10 
>  8. 100ge3-1.core1.sjc2.he.net30.0%10 
>  9. ???
> 10. UNKNOWN-216-115-101-X.yahoo.com   10.0%10 
> 11. routerer-ext.ysv.freebsd.org  20.0%10 
> 12. wfe0.ysv.freebsd.org  30.0%10 
> 
> First off, the OP may have asked "who's fault is hop 9, yahoo or HE?" and 
> seen it
> as an issue. Ignoring that for now, the rest of the packetloss is an issue --
> where is the problem though?
> 
> This is very tricky - it looks like hop 8 is at fault of course - or is it
> just dropping ICMP as it's allowed to? How did hop 10 get only 10% loss then 
> if
> 8 has 30? Is 8 then dropping ~20% (not statistically correct..) of ICMP just 
> cuz
> it can, and then having a 'real' 10% loss on top of that?
> 
> Or it's hop 11? But hop 12 has more PL, perhaps hop 12 is the issue
> all along and 8 10 and 11 are just dropping ICMP? Or it's 8, 11 and 12 doing
> ~10% each? (not statistically correct.)
> 
> Can't say for sure - it's a probabilities game - and being completely correct
> about it, hop 6 isn't blameless either (just very unlikely to be at fault
> statistically, though not impossible with only 10 pings per hop - a 
> statistician
> can calculate it for us).
> 
> This is why more pings are required to be sure of the situation - I like to do
> -i 0.1 -c 100 so it's completed quickly before conditions change.  Then you
> can make a statistically valid pronouncement of where the problem MIGHT BE
> within a useful confidence interval - however, without the return route we're
> still largely in the dark as to the actual location of the issue. You cant be
> '100% sure' with this stuff - technically speaking, it's all 'luck of the 
> draw'.
> 
> (Beware: this one time, at band camp, some etherchannel or equiv at HE was
> showing PL only for specific ips in any target subnet -- because they were 
> xor'ing
> the source & target IP to load balance and one channel was wonky. Fun times
> debugging that one: "WFM from here, what's your issue?")
> 
> /kc
> -- 
> Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto 
> Canada
> Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 
> Front St. W.

Re: packet loss question

2016-07-07 Thread Ken Chase 
On Thu, Jul 07, 2016 at 08:32:19PM +, Mel Beckman said:
  >Yes. It indicates that there was never a time when you did not know 
everything :)
  >
  > -mel beckman

The issue isnt knowing everything, it's making accusations of issues while you 
still
dont know how much you dont know. (~D. Rumsfeld) -- My customers in a nutshell
(they pay to be able to yell about random stuff I guess, and I provide that 
service!).

The OP didnt make any accusations however, and just asked what was going on 
(sorry
if I sounded harsh in reply). Once, Google having a 8.8.8.8 failure locally on
its (anycast?) dns servers resulted in dozens of calls to us "your server
hosting our site must be down!! Our website isnt working! People are calling 
us!".

Most of my work is with these situations is spent proving it's not our fault.
Mtr makes it very hard because it's a very subtle tool, and only gives partial
information. (I still think mtr is a killer app though!)

consider this (fake, example) trace:

  6. 100ge13-1.core1.chi1.he.net0.0%10 
  7. 100ge14-1.core2.chi1.he.net0.0%10 
  8. 100ge3-1.core1.sjc2.he.net30.0%10 
  9. ???
 10. UNKNOWN-216-115-101-X.yahoo.com   10.0%10 
 11. routerer-ext.ysv.freebsd.org  20.0%10 
 12. wfe0.ysv.freebsd.org  30.0%10 

First off, the OP may have asked "who's fault is hop 9, yahoo or HE?" and seen 
it
as an issue. Ignoring that for now, the rest of the packetloss is an issue --
where is the problem though?

This is very tricky - it looks like hop 8 is at fault of course - or is it
just dropping ICMP as it's allowed to? How did hop 10 get only 10% loss then if
8 has 30? Is 8 then dropping ~20% (not statistically correct..) of ICMP just cuz
it can, and then having a 'real' 10% loss on top of that?

Or it's hop 11? But hop 12 has more PL, perhaps hop 12 is the issue
all along and 8 10 and 11 are just dropping ICMP? Or it's 8, 11 and 12 doing
~10% each? (not statistically correct.)

Can't say for sure - it's a probabilities game - and being completely correct
about it, hop 6 isn't blameless either (just very unlikely to be at fault
statistically, though not impossible with only 10 pings per hop - a statistician
can calculate it for us).

This is why more pings are required to be sure of the situation - I like to do
-i 0.1 -c 100 so it's completed quickly before conditions change.  Then you
can make a statistically valid pronouncement of where the problem MIGHT BE
within a useful confidence interval - however, without the return route we're
still largely in the dark as to the actual location of the issue. You cant be
'100% sure' with this stuff - technically speaking, it's all 'luck of the draw'.

(Beware: this one time, at band camp, some etherchannel or equiv at HE was
showing PL only for specific ips in any target subnet -- because they were 
xor'ing
the source & target IP to load balance and one channel was wonky. Fun times
debugging that one: "WFM from here, what's your issue?")

/kc
-- 
Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto 
Canada
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front 
St. W.

Re: packet loss question

2016-07-07 Thread Mel Beckman 
Yes. It indicates that there was never a time when you did not know everything 
:)

 -mel beckman

> On Jul 7, 2016, at 1:28 PM, Brielle Bruns  wrote:
> 
>> On 7/7/16 1:17 PM, Phillip Lynn wrote:
>> Hi all,
>> 
>>  I am writing because I do not understand what is happening.  I ran mtr
>> against our email server and www.teco.comand below are the results.  I
>> am not a network engineer so I am at a loss.  I think what I am seeing
>> is maybe a hand off issue, between Frontier and Level3Miami2. If I am
>> correct then what can I do?
>> 
>>  My system is running Centos 6.5 Linux.
> 
> 
> Is it bad that the first thing that came to mind is "Oh FFS, another troll"?
> 
> 
> -- 
> Brielle Bruns
> The Summit Open Source Development Group
> http://www.sosdg.org/ http://www.ahbl.org

Re: packet loss question

2016-07-07 Thread Brielle Bruns 

On 7/7/16 1:17 PM, Phillip Lynn wrote:

Hi all,

  I am writing because I do not understand what is happening.  I ran mtr
against our email server and www.teco.comand below are the results.  I
am not a network engineer so I am at a loss.  I think what I am seeing
is maybe a hand off issue, between Frontier and Level3Miami2. If I am
correct then what can I do?

  My system is running Centos 6.5 Linux.



Is it bad that the first thing that came to mind is "Oh FFS, another troll"?


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: packet loss question

2016-07-07 Thread James R Cutler 
> On Jul 7, 2016, at 3:17 PM, Phillip Lynn  wrote:
> 
> Hi all,
> 
>  I am writing because I do not understand what is happening.  I ran mtr 
> against our email server and www.teco.comand below are the results.  I am not 
> a network engineer so I am at a loss.  I think what I am seeing is maybe a 
> hand off issue, between Frontier and Level3Miami2. If I am correct then what 
> can I do?
> 
>  My system is running Centos 6.5 Linux.
> 
> Thanks,
> 
> Phillip
> 
> 
> 
> (! 1011)-> sudo mtr -r netwolves.securence.com
> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%103.2 2.0 1.0   4.3   1.2
>  3. 172.99.44.214 0.0%104.0 4.9 2.3   6.9   1.5
>  4. ae8---0.scr02.mias.fl.fronti  0.0%109.3 9.1 7.5   9.8   1.0
>  5. ae1---0.cbr01.mias.fl.fronti  0.0%108.9   9.1   7.6 9.7 0.7
>  6. lag-101.ear3.Miami2.Level3.n 80.0%109.0   8.9   8.8 9.0 0.1
>  7. 10ge9-14.core1.mia1.he.net0.0%10   14.3 13.0 7.6  18.1   4.3
>  8. 10ge1-1.core1.atl1.he.net 0.0%10   25.6  33.2 22.4  99.7  23.6
>  9. 10ge10-4.core1.chi1.he.net0.0%10   45.6  51.8 45.5  82.7  12.5
> 10. 100ge14-2.core1.msp1.he.net   0.0%10   53.6  63.9 53.6 125.2  21.8
> 11. t4-2-usi-cr02-mpls-usinterne  0.0%10   53.2  73.1 53.2 225.6  54.0
> 12. v102.usi-cr04-mtka.usinterne  0.0%10   53.2  53.9 53.2  55.3   0.6
> 13. netwolves.securence.com   0.0%10   53.4  53.9 53.4  55.4   0.7
> 
> (! 1014)-> sudo mtr -r www.teco.com
> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%10  104.8 81.4 1.1 113.2  43.2
>  3. 172.99.47.198 0.0%10  115.0 77.8 2.9 115.0  40.2
>  4. ae7---0.scr01.mias.fl.fronti  0.0%10  111.1 80.2 8.5 113.5  41.3
>  5. ae0---0.cbr01.mias.fl.fronti  0.0%10  105.9  82.2   7.6 115.4 33.8
>  6. lag-101.ear3.Miami2.Level3.n 70.0%10  116.1  80.2   8.5 116.1 62.0
>  7. NTT-level3-80G.Miami.Level3.  0.0%10  110.0 81.5 9.0 120.3  41.9
>  8. ae-3.r20.miamfl02.us.bb.gin.  0.0%10  119.8  84.0 10.0 119.8  38.5
>  9. ae-4.r23.asbnva02.us.bb.gin. 10.0%10  137.4 107.6 30.1 142.7  45.7
> 10. ae-2.r05.asbnva02.us.bb.gin.  0.0%10  135.0 109.9 36.6 140.0  39.1
> 11. xe-0-9-0-8.r05.asbnva02.us.c  0.0%10  147.5 125.6 49.4 165.5  41.1
> 12. 24.52.112.21  0.0%10  158.6 124.0 49.6 161.3  41.5
> 13. 24.52.112.42  0.0%10  151.0 127.7 52.2 159.0  41.2
> 14. ???  100.0100.0 0.0 0.0   0.0   0.0
> 
> --
> Phillip Lynn
> Software Engineer III
> NetWolves
> Phone:813-579-3214
> Fax:813-882-0209
> Email: phillip.l...@netwolves.com
> www.netwolves.com
> 
Phillip,

The data for netwolves.securence.com shows 0% loss between HOST and 
netwolves.securence.com. This is most certainly good. The 80% Loss in line 4 
simply indicates that that particular router was too busy to respond in a 
timely manner to an ECHO request because it was busy forwarding data traffic. 
There is no problem to solve for this connection.

The data for www.teco.com  has a couple of busy hops. 
However, for as far as the trace succeeds (24.52.112.42) there is no effective 
loss end to end.  The ??? response, similar to *** from traceroute, indicates 
that there is probably no route to the destination from that point. (Or there 
is a firewall blocking SNMP ECHO requests at that point.) Diagnosis may require 
contacting the operator of www.teco.com  to confirm the 
system is actually on line and operational. Contact information for tech.com is 
in whois.

Auxiliary information - traceroute data from a system in 
plymouth.mi.michigan.comcast.net  
shows similar results for both targets.

Are there other hosts difficult to reach?

James R. Cutler
james.cut...@consultant.com
PGP keys at http://pgp.mit.edu



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: packet loss question

2016-07-07 Thread Ken Chase 
No offence, but i swear that mtr should come with a license to use it. I get 
more
questions from people accusing us of network issues with mtr in hand...

You shoudlnt care that there's 80% packet loss in the middle of your route, 
unless
you have actual traffic to lag-101.ear3.miami2.level3.net. I suspect you dont.
(If you did, you'd have mtr'd to it directly of course.)

As for your second trace, the sudden jump from 0% on 2nd last hop to 100% last
hop packetloss seems like firewalling to me. (long discussion about the
probabilities of getting 5 0%pl hops in a row and 100% on an unfirewalled
endpoint elided. TL;DR: use more packets in your test -i 0.1 -c 100 thanks).

If you have 0% packetloss to your target endpoint, is there an issue here?
What caused you to mtr?  0% pl is pretty good. You could play quake 1.0
through that pl and ping time. The +20ms ATL<>CHI jump in the route you'd have
to take up with einstein/bill nye/$deity. 

For the 2nd trace, the 1st hop is your latency issue (plus the big jump from
miami<>ashburn, again the limit is c.)

ICMP is allowed to be dropped by intervening routers. Someone will quote an RFC
at us shortly.

Mtr without a return route is not that useful in figuring out packetloss
because pl requires the packet make it there and back. Pl could be anywhere on
the return route, which is probably not symmetrical. The internet stopped
being symetrical about 20+ years ago (if it ever even loosely was), so get a 
friend
to send you an mtr to your ip from the farside.

(I remember a project long ago, some cgi-bin (yeah that long ago) that was
basically a full-path forward+reverse traceroute you could hit on a selected
server at the provider.  Rather handy. not sure if its still a thing, or what 
it was
called.)

/kc


On Thu, Jul 07, 2016 at 03:17:40PM -0400, Phillip Lynn said:
  >Hi all,
  >
  >  I am writing because I do not understand what is happening.  I ran mtr
  >against our email server and www.teco.comand below are the results.  I am
  >not a network engineer so I am at a loss.  I think what I am seeing is
  >maybe a hand off issue, between Frontier and Level3Miami2. If I am correct
  >then what can I do?
  >
  >  My system is running Centos 6.5 Linux.
  >
  >Thanks,
  >
  >Phillip
  >
  >
  >
  >(! 1011)-> sudo mtr -r netwolves.securence.com
  >HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
  >  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
  >  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%103.2 2.0 1.0   4.3   1.2
  >  3. 172.99.44.214 0.0%104.0 4.9 2.3   6.9   1.5
  >  4. ae8---0.scr02.mias.fl.fronti  0.0%109.3 9.1 7.5   9.8   1.0
  >  5. ae1---0.cbr01.mias.fl.fronti  0.0%108.9   9.1   7.6 9.7 0.7
  >  6. lag-101.ear3.Miami2.Level3.n 80.0%109.0   8.9   8.8 9.0 0.1
  >  7. 10ge9-14.core1.mia1.he.net0.0%10   14.3 13.0 7.6  18.1   4.3
  >  8. 10ge1-1.core1.atl1.he.net 0.0%10   25.6  33.2 22.4  99.7  23.6
  >  9. 10ge10-4.core1.chi1.he.net0.0%10   45.6  51.8 45.5  82.7  12.5
  > 10. 100ge14-2.core1.msp1.he.net   0.0%10   53.6  63.9 53.6 125.2  21.8
  > 11. t4-2-usi-cr02-mpls-usinterne  0.0%10   53.2  73.1 53.2 225.6  54.0
  > 12. v102.usi-cr04-mtka.usinterne  0.0%10   53.2  53.9 53.2  55.3   0.6
  > 13. netwolves.securence.com   0.0%10   53.4  53.9 53.4  55.4   0.7
  >
  >(! 1014)-> sudo mtr -r www.teco.com
  >HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
  >  1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
  >  2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%10  104.8 81.4 1.1 113.2  43.2
  >  3. 172.99.47.198 0.0%10  115.0 77.8 2.9 115.0  40.2
  >  4. ae7---0.scr01.mias.fl.fronti  0.0%10  111.1 80.2 8.5 113.5  41.3
  >  5. ae0---0.cbr01.mias.fl.fronti  0.0%10  105.9  82.2   7.6 115.4 33.8
  >  6. lag-101.ear3.Miami2.Level3.n 70.0%10  116.1  80.2   8.5 116.1 62.0
  >  7. NTT-level3-80G.Miami.Level3.  0.0%10  110.0 81.5 9.0 120.3  41.9
  >  8. ae-3.r20.miamfl02.us.bb.gin.  0.0%10  119.8  84.0 10.0 119.8  38.5
  >  9. ae-4.r23.asbnva02.us.bb.gin. 10.0%10  137.4 107.6 30.1 142.7  45.7
  > 10. ae-2.r05.asbnva02.us.bb.gin.  0.0%10  135.0 109.9 36.6 140.0  39.1
  > 11. xe-0-9-0-8.r05.asbnva02.us.c  0.0%10  147.5 125.6 49.4 165.5  41.1
  > 12. 24.52.112.21  0.0%10  158.6 124.0 49.6 161.3  41.5
  > 13. 24.52.112.42  0.0%10  151.0 127.7 52.2 159.0  41.2
  > 14. ???  100.0100.0 0.0 0.0   0.0   0.0
  >
  >-- 
  >Phillip Lynn
  >Software Engineer III
  >NetWolves
  >Phone:813-579-3214
  >Fax:813-882-0209
  >Email: phillip.l...@netwolves.com
  >www.netwolves.com
  >

-- 
Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto 
Canada
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front 
St. W.

Re: packet loss question

2016-07-07 Thread Job Snijders 
Hi Philip,

I can't address your immediate concern, but I do have some hints
regarding traceroute:

1/ Please review the excellent presentation from RA{T,S}:

https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf
https://www.youtube.com/watch?v=a1IaRAVGPEE

2/ When you run mtr, make sure to run it with the '-w' argument for
easier to read reverse DNS entries.

Kind regards,

Job

On Thu, Jul 07, 2016 at 03:17:40PM -0400, Phillip Lynn wrote:
>   I am writing because I do not understand what is happening.  I ran mtr
> against our email server and www.teco.comand below are the results.  I am
> not a network engineer so I am at a loss.  I think what I am seeing is maybe
> a hand off issue, between Frontier and Level3Miami2. If I am correct then
> what can I do?
> 
>   My system is running Centos 6.5 Linux.
> 
> Thanks,
> 
> Phillip
> 
> 
> 
> (! 1011)-> sudo mtr -r netwolves.securence.com
> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>   1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>   2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%103.2 2.0 1.0   4.3   1.2
>   3. 172.99.44.214 0.0%104.0 4.9 2.3   6.9   1.5
>   4. ae8---0.scr02.mias.fl.fronti  0.0%109.3 9.1 7.5   9.8   1.0
>   5. ae1---0.cbr01.mias.fl.fronti  0.0%108.9   9.1   7.6 9.7 0.7
>   6. lag-101.ear3.Miami2.Level3.n 80.0%109.0   8.9   8.8 9.0 0.1
>   7. 10ge9-14.core1.mia1.he.net0.0%10   14.3 13.0 7.6  18.1   4.3
>   8. 10ge1-1.core1.atl1.he.net 0.0%10   25.6  33.2 22.4  99.7  23.6
>   9. 10ge10-4.core1.chi1.he.net0.0%10   45.6  51.8 45.5  82.7  12.5
>  10. 100ge14-2.core1.msp1.he.net   0.0%10   53.6  63.9 53.6 125.2  21.8
>  11. t4-2-usi-cr02-mpls-usinterne  0.0%10   53.2  73.1 53.2 225.6  54.0
>  12. v102.usi-cr04-mtka.usinterne  0.0%10   53.2  53.9 53.2  55.3   0.6
>  13. netwolves.securence.com   0.0%10   53.4  53.9 53.4  55.4   0.7
> 
> (! 1014)-> sudo mtr -r www.teco.com
> HOST: x@netwolves.comLoss%   Snt   Last   Avg Best  Wrst StDev
>   1. 172.24.109.1  0.0%100.6 0.6 0.6   0.7   0.0
>   2. lo0-100.TAMPFL-VFTTP-322.gni  0.0%10  104.8 81.4 1.1 113.2  43.2
>   3. 172.99.47.198 0.0%10  115.0 77.8 2.9 115.0  40.2
>   4. ae7---0.scr01.mias.fl.fronti  0.0%10  111.1 80.2 8.5 113.5  41.3
>   5. ae0---0.cbr01.mias.fl.fronti  0.0%10  105.9  82.2   7.6 115.4 33.8
>   6. lag-101.ear3.Miami2.Level3.n 70.0%10  116.1  80.2   8.5 116.1 62.0
>   7. NTT-level3-80G.Miami.Level3.  0.0%10  110.0 81.5 9.0 120.3  41.9
>   8. ae-3.r20.miamfl02.us.bb.gin.  0.0%10  119.8  84.0 10.0 119.8  38.5
>   9. ae-4.r23.asbnva02.us.bb.gin. 10.0%10  137.4 107.6 30.1 142.7  45.7
>  10. ae-2.r05.asbnva02.us.bb.gin.  0.0%10  135.0 109.9 36.6 140.0  39.1
>  11. xe-0-9-0-8.r05.asbnva02.us.c  0.0%10  147.5 125.6 49.4 165.5  41.1
>  12. 24.52.112.21  0.0%10  158.6 124.0 49.6 161.3  41.5
>  13. 24.52.112.42  0.0%10  151.0 127.7 52.2 159.0  41.2
>  14. ???  100.0100.0 0.0 0.0   0.0   0.0
> 
> -- 
> Phillip Lynn
> Software Engineer III
> NetWolves
> Phone:813-579-3214
> Fax:813-882-0209
> Email: phillip.l...@netwolves.com
> www.netwolves.com
>