Re: Symmetry, DSL, and all that

[Steve Clark]

On 03/02/2015 02:19 PM, Naslund, Steve wrote:

The backend is still symmetric. It's still something like 1.25 gigs up and 2.5 
gigs down. You can only beat that going to AE.



Truth is, once the user is achieving what they consider to be acceptable 
performance they don't care if it is symmetric or not.



Not a very informative discussion.

Points of fact...


>From Verizon's January filings regarding 2014Q4:

1. Verizon has about eight million FIOS customers.
2. "Fifty-nine percent of FiOS consumer Internet customers subscribed to data speeds 
of at least 50Mbps, up from 46 percent one year earlier."


Eight million FIOS customers does not even come close to representing the bulk of users 
out there.  In fact, it does not even represent the majority of "high speed" 
customers out there.

>From a Verizon press release last summer, all FIOS speeds are now symmetric.

And no one cares.  I don't even see Verizon commercials crowing about how great 
it is to have symmetry.  If customers loved it that much don't you think they 
would market that way?

Hi Steve,

I live in the Tampa Bay area and I see Verizon commercial all the time where 
other ISP customers are complaining that theirs ISP take so long to upload 
pictures, backups, etc.
Plus there are commercial with people on an escalator where the up escalator is 
much slower than the down escalator and people are complaining up should be as 
fast as down.

Regards,
Steve



http://arstechnica.com/business/2014/07/verizon-fios-finally-symmetrical-upload-speeds-boosted-to-match-download/

ADSL development proceeded the development of the consumer Internet. The 
original patent was filed in 1988. DSL was designed originally to deliver video 
in an ISDN/ATM world. For that reason, it was asymmetric.

ADSL did not proceed the development of the consumer Internet in the commercial 
world.  If it did, we would never have gone with dial-up modems.  Patent dates 
have very little to do with commercial availability at all.  Please give me an 
example of a purchasable service using ADSL prior to its use in Internet 
delivery.  The number one reason ADSL succeeded and SDSL did not.you could 
put an ADSL signal on the phone line you already had in your house, SDSL 
required a new loop to be ordered.  Faster to provision and it can be done 
without a truck roll.

Steven Naslund
Chicago IL







--
Stephen Clark
*NetWolves Managed Services, LLC.*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Verizon Policy Statement on Net Neutrality

[Steve Clark]

On 02/27/2015 04:11 PM, Scott Helms wrote:

Daniel,


"50MB/s might be tough to fill, but even at home I can get good use out of
the odd 25MB/s upstream burst for a few minutes."

Which would you choose, 50/50 or 75/25?  My point is not that upstream
speed isn't valuable, but merely that demand for it isn't symmetrical and
unless the market changes won't be in the near term.  Downstream demand is
growing, in most markets I can see, much faster than upstream demand.

Scott,

Who can foresee what APPs might come about if uplinks speeds weren't so low. I 
liken it to
whoever said no one will ever need more than 640KB of memory.

Regards,
Steve


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms





--
Stephen Clark
*NetWolves Managed Services, LLC.*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Verizon Policy Statement on Net Neutrality

[Steve Clark]

Scott,

Maybe if it the upstream bandwidth was there would be more applications to use 
it. I know it is a real
pain to upload pics to Facebook, etc on my 1mbs uplink, or move things to work 
across my VPN.

Steve

On 02/27/2015 02:30 PM, Scott Helms wrote:

Daniel,

Well, I wouldn't call using the mean a "myth", after all understanding most
customer behavior is what we all have to build our business cases around.
If we throw out what customers use today and simply take a build it and
they will come approach then I suspect there would fewer of us in this
business.

Even when we look at anomalous users we don't see symmetrical usage, ie top
10% of uploaders.  We also see less contended seconds on their upstream
than we do on the downstream.  These observations are based on ~500k
residential and business subscribers across North America using FTTH
(mostly GPON), DOCSIS cable modems, and various flavors of DSL.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms


On Fri, Feb 27, 2015 at 2:21 PM, Daniel Taylor  wrote:


But by this you are buying into the myth of the mean.

It isn't that most, or even many, people would take advantage of equal
upstream bandwidth, but that the few who would need to take extra measures
unrelated to the generation of that content to be able to do so.

Given symmetrical provisioning, no extra measures need to be taken when
that 10 year old down the street turns out to be a master musician.

On 02/27/2015 11:59 AM, Scott Helms wrote:


This is true in our measurements today, even when subscribers are given
symmetrical connections.  It might change at some point in the future,
especially when widespread IPv6 lets us get rid of NAT as a de facto
deployment reality.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms


On Fri, Feb 27, 2015 at 12:48 PM, Naslund, Steve 
wrote:

  How about this?  Show me 10 users in the average neighborhood creating

content at 5 mbpsPeriod.  Only realistic app I see is home
surveillance
but I don't think you want everyone accessing that anyway.  The truth is
that the average user does not create content that anyone needs to see.
This has not changed throughout the ages, the ratio of authors to
readers,
artists to art lovers, musicians to music lovers, YouTube cat video
creator
to cat video lovers, has never been a many to many relationship.

On 2015-02-27 12:13, valdis.kletni...@vt.edu wrote:


Consider a group of 10 users, who all create new content.  If each one
creates at a constant rate of 5 mbits, they need 5 up.  But to
download all the new content from the other 9, they need close to 50


down.


And when you expand to several billion people creating new content,
you need a *huge* pipe down.


Steven Naslund
Chicago IL




--
Daniel Taylor  VP OperationsVocal Laboratories, Inc.
dtay...@vocalabs.com   http://www.vocalabs.com/(612)235-5711





--
Stephen Clark
*NetWolves Managed Services, LLC.*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: US patent 5473599

[Steve Clark]

On 04/22/2014 01:30 PM, Paul WALL wrote:

On Tuesday, April 22, 2014, Henning Brauer  wrote:


I won't waste time on your uninformed ramblings, you have the facts
plain wrong. There is enough material on the net for everybody to read
up on what happened.

"carp causing outages" however is nothing short of a lie. carp
announces itself as vrrp version 3. anything trying to parse it as
vrrp2 without checking the version number in the header is buggy as
hell and that is ITS fault, not carps. affected cisco 3600, afair.


I wasn't talking about CARP's announcements causing outages due to
bugs in VRRP implementations, I was talking about CARP's intentional
use of another organization's OUI and deliberately constructing its
bits in the host section to conflict with established practice for
VRRP.  That was childish, and causes outages.  This design choice
should be documented in CARP's man page.  It is not.

In response to Ryan Shea, here's the way it breaks down:

Both CARP and VRRP use virtual router MAC addresses that start with
00:00:5e.  This organizational unique identifier (OUI) is assigned to
IANA, not OpenBSD or a related project.  The CARP authors could have
gotten their own from IEEE.  OUIs are not free but the cost is quite
reasonable (and was even more reasonable years ago when this
unfortunate decision was made).

The next two octets for IPv4 VRRP are 00:01.  Highly coincidentally,
the CARP folks *also* decided to use 00:01 after they got upset at the
IETF for dissing their slide deck.

If either of these decisions had not been made, we would not be having
this discussion today.

The last octet is the VRID for both CARP and VRRP.  If you don't have
the same VRID configured, the protocols should peacefully coexist,
assuming no bugs in the software listening to the multicast
advertisements (which Henning mentioned above - a legitimate concern
to be sure, but not the big one).

So yes, the problem only exists if you are running VRRP and CARP on
the same subnet (say, a pair of routers speaking VRRP and a pair of
firewalls backing each other with CARP and pfsync, which is actually
fairly common) and happen to have a host identifier conflict.  In a
completely random world the likelihood of this is 1 in 256.
Unfortunately, human nature and a plethora of examples on the
intarwebs makes "interface GigabitEthernet 0/3 // vrrp 1 ip blah"
reasonably likely.  The same human nature causes the out of the box
configuration on many products, e.g. pfSense, to be "ifconfig carp0
vhid 1".  Presto - outage for everyone on the subnet.

Soapbox time.  There are some people who decide that they will only
run FOSS software because of how they feel about software patents.  In
my case, I will not run CARP because of how I feel about folks
deliberately violating the interoperability maxim ("be conservative in
what you send and liberal in what you accept") and causing *me* to be
the collateral damage.  I think we all have enough on our plates
dealing with legitimate software bugs without having rogue protocols
deliberately interfering with our networks because of a grudge.  Is
CARP malware?  A trojan?  I'm not sure I'd go that far, but it seems
to meet some of the definitions.

Nothing personal Henning (and I like what you did with OpenBGPd and
OpenNTPd) but you'd gain a lot of respect in my eyes, as well as a
bunch of other people's, if you publicly admitted the CARP OUI
decision was a huge mistake.  If your lawyers have advised you not to
apologize because of liability concerns (despite that "no warranty"
bit in the BSD license) it's OK - I completely understand.

Drive Slow,
Paul


I think there is also the issue it uses the same protocol number - 112.

From /etc/protocolsvrrp112 VRRP# Virtual Router Redundancy 
Protocol


--
Stephen Clark
*NetWolves Managed Services, LLC.*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Fwd: Serious bug in ubiquitous OpenSSL library: "Heartbleed"

[Steve Clark]

According to the changelog it cvs is fixed now.

$ rpm -qa|grep openssl
openssl-1.0.1e-16.el6_5.7.x86_64
openssl-devel-1.0.1e-16.el6_5.7.x86_64
Tue Apr  8 12:17:25 EDT 2014
Z643357:~
$ rpm -q --changelog openssl | less
* Mon Apr 07 2014 Tomás( Mráz  1.0.1e-16.7
- fix CVE-2014-0160 - information disclosure in TLS heartbeat extension

On 04/08/2014 12:11 PM, Jonathan Lassoff wrote:

For testing, I've had good luck with
https://github.com/titanous/heartbleeder and
https://gist.github.com/takeshixx/10107280

Both are mostly platform-independent, so they should be able to work even
if you don't have a modern OpenSSL to test with.

Cheers and good luck (you're going to need it),
jof

On Tue, Apr 8, 2014 at 5:03 PM, Michael Thomas  wrote:


Just as a data point, I checked the servers I run and it's a good thing I
didn't reflexively update them first.
On Centos 6.0, the default openssl is 1.0.0 which supposedly doesn't have
the vulnerability, but the
ones queued up for update do. I assume that redhat will get the patched
version soon but be careful!

Mike


On 04/07/2014 10:06 PM, Paul Ferguson wrote:


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

I'm really surprised no one has mentioned this here yet...

FYI,

- - ferg



Begin forwarded message:

  From: Rich Kulawiec  Subject: Serious bug in

ubiquitous OpenSSL library: "Heartbleed" Date: April 7, 2014 at
9:27:40 PM EDT

This reaches across many versions of Linux and BSD and, I'd
presume, into some versions of operating systems based on them.
OpenSSL is used in web servers, mail servers, VPNs, and many other
places.

Writeup: Heartbleed: Serious OpenSSL zero day vulnerability
revealed
http://www.zdnet.com/heartbleed-serious-openssl-zero-day-vulnerability-
revealed-728166/

   Technical details: Heartbleed Bug http://heartbleed.com/

OpenSSL versions affected (from link just above):  OpenSSL 1.0.1
through 1.0.1f (inclusive) are vulnerable OpenSSL 1.0.1g is NOT
vulnerable (released today, April 7, 2014) OpenSSL 1.0.0 branch is
NOT vulnerable OpenSSL 0.9.8 branch is NOT vulnerable



- -- Paul Ferguson
VP Threat Intelligence, IID
PGP Public Key ID: 0x54DC85B2
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlNDg9gACgkQKJasdVTchbIrAAD9HzKaElH1Tk0oIomAOoSOvfJf
3Dvt4QB54os4/yewQQ8A/0dhFZ/YuEdA81dkNfR9KIf1ZF72CyslSPxPvkDcTz5e
=aAzE
-END PGP SIGNATURE-







--
Stephen Clark
*NetWolves Managed Services, LLC.*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: random dns queries with random sources

[Steve Clark]

On 02/20/2014 08:57 AM, Pavel Zeleny wrote:

Masataka Ohta  necom830.hpcl.titech.ac.jp> writes:


Joe Maimon wrote:


What is the purpose of this?

...

Masataka Ohta


Hi guys,
for a second, have you any clue how to block this traffic on DNS server
side? As our company operates recursive resolvers for our customers, we can
see this weird traffic concentrated in our logs. It started Feb 3 about
16:30 (GMT/UTC+1). Very large amount of DNS A queries are sent from source
IP addresses of our customers, and they always looks like
[randomjunk].SLD.com. We have seen 143 this SLD's so far, and we had to
block it manually one by one.
We suspect some kind of botnet, because attack wave with new SLD's starts at
the same time, coming from broad range of valid non-spoofed source IP
addresses. Content of UDP packets belonging to this traffic doesn't seem to
have any identical pattern.

Any ideas are highly appreciated.
Thank you!

Pavel Zeleny



iptables -A INPUT -p udp --dport 53 -m hashlimit \
   --hashlimit-name DNS --hashlimit-above 20/second --hashlimit-mode srcip \
   --hashlimit-burst 100 --hashlimit-srcmask 28 -j DROP

So, every prefix (length 28) can send 20 r/s with allowed bursts of
100. This requires a Netfilter >= 1.4 (recent options of module hashlimit).


--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: "Programmers can't get IPv6 thus that is why they do not have IPv6 in their applications"....

[Steve Clark]

On 11/30/2012 09:45 AM, Ray Soucy wrote:

I'll see your disagree and raise you another ;-)

I would say you almost never want to store addresses as character data
unless the only thing you're using them for is logging (even then it's
questionable).  I run into people who do this all the time and it's a
nightmare.

It's easy to store a v6 address as a string, but when you want to select a
range of IPv6 addresses from a database, not having them represented as
integers means you can't do efficient numerical comparisons in your SQL
statements, it also makes indexing your table slower; to put it simply, it
doesn't scale well.

So as a general rule, if you need to do any comparison or calculation on a
v6 address, please don't store it as a string.

>From an efficiency standpoint, you want to store it in chunks of the
largest integer your DBMS supports.  If a DBMS supports 128-bit integers
and has optimized operations for them, then go for it.  Most only support
64-, or even 32-bit.  I say 64-bit because that's what the majority of
current systems actually support and I don't see anyone coming out with a
128-bit architecture ;(

For convenience I would very much love to see MySQL include inet6_aton and
inet6_ntoa, along with a 128-bit data structure that would
be implemented as either a pair of 64-bit or 4x 32-bit values depending on
the architecture.  But from a performance standpoint, I really don't want
my DBMS doing that calculation; I want the application server doing it
(because it's much easier to scale and distribute the application side than
the storage side).

Postgresql has an inet data type that handles both ipv4 and ipv6 addresses with 
a slew of
functions to manipulate the data type.

http://www.postgresql.org/docs/8.4/static/functions-net.html


Note that I'm talking about more from a database storage perspective than
an internal application perspective.

By all means, you should use the standard data structure for v6.  As
mentioned below a lot of the internal structures use 8-bit unsigned
integers (or char); but that's mainly a hold-over from when we had the
reality of 8-bit and 16-bit platforms (for compatibility).  With unions,
these structs are treated as a collection of 8, 16, 32, 64 or a single
128-bit variable which makes it something the developer doesn't need to
worry about once the libraries are written.




On Thu, Nov 29, 2012 at 9:55 AM, William Herrin  wrote:


On Thu, Nov 29, 2012 at 9:01 AM, Ray Soucy  wrote:

You should store IPv6 as a pair of 64-bit integers.  While PHP lacks
the function set to do this on its own, it's not very difficult to do.

Hi Ray,

I have to disagree. In your SQL database you should store addresses as
a fixed length character string containing a zero-padded hexadecimal
representation of the IPv4 or IPv6 address with A through F forced to
the consistent case of your choice. Expand :: and optionally strip the
colons entirely. If you want to store a block of addresses, store it
as two character strings: start and end of the range.

Bytes are cheap and query simplicity is important. Multi-element
indexes are messy and the code to manage an array of integers is
messier than managing a character string in most programming
languages. memcmp() that integer array for less or greater than? Not
on a little endian machine!



Here are a set of functions I wrote a while back to do just that
(though I admit I should spend some time to try and make it more
elegant and I'm not sure it's completely up to date compared to my
local copy ... I would love some eyes on it to make some
improvements).

http://soucy.org/project/inet6/

If we're plugging our code, give my public domain libeasyv6 a try. It
eases entry into dual stack programming for anyone used to doing
gethostbyname followed by a blocking connect(). Just do a
connectbyname() with the hostname or textual IP address, the port, a
timeout and null options. The library takes care of finding a working
IPv4 or IPv6 address for the host and connecting to it in a timely
manner.

http://bill.herrin.us/freebies/

Currently Linux only but if you're willing to lose timeout control on
the DNS lookup you can replace getaddrinfo_a() with standard
getaddrinfo() and the code should run anywhere.

Regards,
Bill Herrin


--
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004







--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: IPv6 /64 links (was Re: ipv6 book recommendations?)

[Steve Clark]

On 06/06/2012 03:05 PM, Owen DeLong wrote:

It is because of IEEE EUI-64 standard.

It was believed at the time of IPv6 development that EUI-48 would run out of
numbers and IEEE had proposed going to EUI-64. While IEEE still hasn't
quite made that change (though Firewire does appear to use EUI-64 already),
it will likely occur prior to the EOL for IPv6.

There is a simple algorithm used by IEEE for mapping EUI-48 onto the EUI-64
space.

The 0x02 bit of the first octet of an EUI-64 address is an L-Flag, indicating 
that
the address was locally generated (if it is a 1) vs. IEEE/vendor assigned (if 
it is a 0).

The mapping process takes the EUI-48 address XX:YY:ZZ:RR:SS:TT and maps
it as follows:

let AA = XX xor 0x02.

AAYY:ZZff:feRR:SSTT

ff:fe above is literal.

IPv6 was originally going to be a 32-bit address space, but, the developers

did you mean "originally going to be a 64-bit address space"...

and proponent of SLAAC convinced IETF to add 64 more bits to the IPv6
address for this purpose. Since bits are free when designing a new protocol,
there really was no reason to impose such limitations.

You really don't gain anything by going to /80 at this point. There are more
than enough addresses available in IPv6 for any foreseeable future even
with /64 subnets.

Owen




On Jun 6, 2012, at 7:58 AM, Chuck Church wrote:


Does anyone know the reason /64 was proposed as the size for all L2 domains?
I've looked for this answer before, never found a good one.  I thought I
read there are some L2 technologies that use a 64 bit hardware address,
might have been Bluetooth.  Guaranteeing that ALL possible hosts could live
together in the same L2 domain seems like overkill, even for this group.
/80 would make more sense, it does match up with Ethernet MACs.  Not as easy
to compute, for humans nor processors that like things in 32 or 64 bit
chunks however.  Anyone have a definite answer?

Thanks,

Chuck

-Original Message-
From: jean-francois.tremblay...@videotron.com
[mailto:jean-francois.tremblay...@videotron.com]
Sent: Wednesday, June 06, 2012 10:36 AM
To: an...@huge.geek.nz
Cc: NANOG list
Subject: IPv6 /64 links (was Re: ipv6 book recommendations?)

Anton Smith  a écrit sur 06/06/2012 09:53:02 AM :


Potentially silly question but, as Bill points out a LAN always
occupies a /64.

Does this imply that we would have large L2 segments with a large
number of hosts on them? What about the age old discussion about
keeping broadcast segments small?

The /64 only removes the limitation on the number of *addresses* on the L2
domain. Limitations still apply for the amount of ARP and ND noise. A
maximum number of hosts is reached when that noise floor represents a
significant portion of the link bandwidth. If ARP/ND proxying is used, the
limiting factor may instead be the CPU on the gateway.

The ND noise generated is arguably higher than ARP because of DAD, but I
don't remember seeing actual numbers on this (anybody?).
I've seen links with up to 15k devices where ARP represented a significant
part of the link usage, but most weren't (yet) IPv6.

/JF









--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Host scanning in IPv6 Networks

[Steve Clark]

On 04/20/2012 08:17 AM, Tei wrote:

It would be a very fast dictionary attack :D

accede
bade
dad
decade
face
axed
babe
deaf
bed
Abe
bee
Decca
exec
fade
bead
bedded
deed
exceed
Abba
deface
efface
feed


On 20 April 2012 09:08, Fernando Gont  wrote:

FYI

 Original Message 
Subject: IPv6 host scanning in IPv6
Date: Fri, 20 Apr 2012 03:57:48 -0300
From: Fernando Gont
Organization: SI6 Networks
To: IPv6 Hackers Mailing List

Folks,

We've just published an IETF internet-draft about IPv6 host scanning
attacks.

The aforementioned document is available at:


The Abstract of the document is:
 cut here 
   IPv6 offers a much larger address space than that of its IPv4
   counterpart.  The standard /64 IPv6 subnets can (in theory)
   accommodate approximately 1.844 * 10^19 hosts, thus resulting in a
   much lower host density (#hosts/#addresses) than their IPv4
   counterparts.  As a result, it is widely assumed that it would take a
   tremendous effort to perform host scanning attacks against IPv6
   networks, and therefore IPv6 host scanning attacks have long been
   considered unfeasible.  This document analyzes the IPv6 address
   configuration policies implemented in most popular IPv6 stacks, and
   identifies a number of patterns in the resulting addresses lead to a
   tremendous reduction in the host address search space, thus
   dismantling the myth that IPv6 host scanning attacks are unfeasible.
 cut here 

Any comments will be very welcome (note: this is a drafty initial
version, with lots of stuff still to be added... but hopefully a good
starting point, and a nice reading ;-) ).

Thanks!

Best regards,





exec ?
exceed ?


--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Common operational misconceptions

[Steve Clark]

I agree with this 100%.

Having worked with many people over the last 40 years, the good trouble 
shooters understood how things
were suppose to work. This helps immeasurably in determining where to start 
looking.

On 02/17/2012 10:12 AM, Mario Eirea wrote:

Well, I will argue this. I think the important factor in any troubleshooting is 
having a real understanding of how the system works. That is, how different 
things interact with each others to achieve a specific goal. The biggest 
problem I see is that many people understand understand the individual parts 
but when it comes to understanding the system as a whole they fall miserably 
short.

A short example, probably not the best but the one that comes to mind right now:

Someone replaces a device on the network with a new one. They give it the same IP address 
as the old device. They don't understand why the router cant communicate with it at first 
and then starts working. The people "understand" ARP, but cant correlate one 
event with another.

I guess if your 35 you have seen this at least once and can fix it. But what 
happens if you have never seen this problem or a related one? At this point 
your going to have to really troubleshoot, not just go on experience.

Mario Eirea

From: -Hammer- [bhmc...@gmail.com]
Sent: Friday, February 17, 2012 9:52 AM
To: nanog@nanog.org
Subject: Re: Common operational misconceptions

Let me simplify that. If you are over 35 you know how to troubleshoot.

Yes, I'm going to get flamed. Yes, there are exceptions in both directions.

-Hammer-

"I was a normal American nerd"
-Jack Herer



On 2/17/2012 8:29 AM, Leo Bicknell wrote:

In a message written on Thu, Feb 16, 2012 at 08:50:11PM -1000, Paul Graydon 
wrote:

At the same time, it's shocking how many network people I come across
with no real grasp of even what OSI means by each layer, even if it's
only in theory.  Just having a grasp of that makes all the world of
difference when it comes to troubleshooting.  Start at layer 1 and work
upwards (unless you're able to make appropriate intuitive leaps.) Is it
physically connected? Are the link lights flashing? Can traffic route to
it, etc. etc.

I wouldn't call it a "misconception", but I want to echo Paul's
comment.  I would venture over 90% of the engineers I work with
have no idea how to troubleshoot properly.  Thinking back to my own
education, I don't recall anyone in highschool or college attempting
to teach troubleshooting skills.  Most classes teach you how to
build things, not deal with them when they are broken.

The basic skills are probably obvious to someone who might design
course material if they sat down and thought about how to teach
troubleshooting.  However, there is one area that may not be obvious.
There's also a group management problem.  Many times troubleshooting
is done with multiple folks on the phone (say, customer, ISP and
vendor).  Not only do you have to know how to troubleshoot, but how
to get everyone on the same page so every possible cause isn't
tested 3 times.

I think all college level courses should include a "break/fix"
exercise/module after learning how to build something, and much of that
should be done in a group enviornment.







--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Polling Bandwidth as an Aggregate

[Steve Clark]

On 01/20/2012 10:53 AM, Chris Adams wrote:

Once upon a time, Leo Bicknell  said:

To suggest Netflow is more accurate than rrdtool seems rather strange
to me.   It can be as accurate, but is not the way most people
deploy it.

Comparing Netflow to RRDTool is comparing apples to cabinets; one is a
source of information and one is a way of storing information.


RRDTool pulls the SNMP counters from an interface and records them to a
file.

No, RRDTool stores data given to it by a front end such as MRTG,
Cricket, Cacti, etc.  That front end can fetch data from any number of
sources, including (but not limited to) SNMP.  RRDTool then stores
information in its database.


With no aggregation, and assuming your device has accurate SNMP,
this should be 100% accurate.  While you are right that the defaults for
RRDTOOL aggregate data (after a day, week, and month, approximately)
those aggregates can be disabled keeping the raw data.

RRDTool does not store the raw data.  Even for 5-minute intervals, it
adjusts the data vs. the timestamp to fit the desired interval.  Since
you don't read every counter at the exact time of your interval, RRDTool
is always manipulating the numbers to fit.  The only numbers that are
not changed before storing are the timestamp and value for the most
recent update (which get overwritten at each update); everything else is
adjusted to fit.


I suggest reading
http://oss.oetiker.ch/rrdtool/tut/rrd-beginners.en.html


--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: IPV6 issue (occaid.net)

[Steve Clark]

On 12/20/2011 12:12 PM, Michael Sinatra wrote:

On 12/20/11 06:33, Jeroen Massar wrote:

On 2011-12-20 15:17 , Steve Clark wrote:

Hello,

I have a SIXXS ipv6 tunnel that terminates in Ashburn, Va.
I have two HE ipv6 tunnels, one terminates in Dallas the other
terminate in Ashburn. I can ping each endpoint of the tunnels that
terminate
in Ashburn, but I can't ping between the SIXXS and HE with the HE
termination in Dallas.

Using Looking Glass at HE I can traceroute to my SIXXS tunnel from
Chicago but
not from Dallas.

Any ideas on how to get this fixed.

Contact the providers involved directly?

Sending a mail to i...@he.net + i...@sixxs.net should get you what you
need, given that you actually provide IP addresses and other such useful
diagnostics like interface configuration, routing tables etc etc etc.
The above mail is far from useful and nobody would be able to help you
in anyway except to state the above.

Actually, I was about to send a message about this.  I believe the
problem is in occaid.net, particularly their router in Atlanta.

SIXXS uses a variety of providers at various PoPs to provide their
tunnel connectivity and occaid.net is the provider at Ashburn (I have a
SIXXS tunnel there as well).  Tracerouting from the West Coast or Texas
goes through occaid.net's router in Atlanta and dies there with 'network
unreachable':

traceroute6 to burnttofu.net (2001:4830:1600:3bf::2) from
2001:470:1f05:17a6:219:d1ff:fe26:5246, 64 hops max, 12 byte packets
   1  2001:470:1f05:17a6::1  0.316 ms  0.321 ms  0.321 ms
   2  10-1.tunnel.tserv3.fmt2.ipv6.he.net  28.000 ms  22.402 ms
26.169 ms
   3  gige-g5-19.core1.fmt2.he.net  16.697 ms  18.046 ms  15.891 ms
   4  10gigabitethernet6-4.core1.lax1.he.net  23.735 ms  25.327 ms  25.711 ms
   5  10gigabitethernet1-3.core1.lax2.he.net  25.708 ms  24.923 ms  25.793 ms
   6  2001:504:13::8  25.713 ms  23.731 ms  25.705 ms
   7  bbr01-v441.atln01.occaid.net  80.617 ms !N  88.252 ms !N  79.369 ms !N

Tracerouting from the East Coast is fine:
traceroute6 to burnttofu.net (2001:4830:1600:3bf::2) from
2001:470:30:80:e076:63ff:fe88:2d62, 64 hops max, 12 byte packets
   1  2001:470:30:80::2  21.739 ms  1.938 ms  2.474 ms
   2  gige-g3-3.core1.nyc4.he.net  8.678 ms  2.710 ms  2.596 ms
   3  10gigabitethernet2-3.core1.ash1.he.net  7.488 ms  7.168 ms  8.449 ms
   4  ibr01-ve96.asbn01.occaid.net  7.211 ms  7.272 ms  7.177 ms
   5  equi6ix.dc.hotnic.net  9.789 ms  8.597 ms  8.610 ms
   6  sixxs-asbnva-gw.customer.occaid.net  8.782 ms  8.100 ms  9.522 ms
   7  cl-960.qas-01.us.sixxs.net  22.621 ms  20.880 ms  21.072 ms

Attempts to get a response from n...@occaid.net regarding this issue over
the past 36 hours have failed.  If there is anyone here from occaid.net
or knows a clueful person there, can you please point them to this thread.

I still think it's a good idea to contact i...@sixxs.net, so they know
what's going on, but I don't think it's actually their problem.

michael


I did and now it appears to be resolved. Thanks HE and SixXS.

--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

IPV6 issue

[Steve Clark]

Hello,

I have a SIXXS ipv6 tunnel that terminates in Ashburn, Va.
I have two HE ipv6 tunnels, one terminates in Dallas the other
terminate in Ashburn. I can ping each endpoint of the tunnels that terminate
in Ashburn, but I can't ping between the SIXXS and HE with the HE termination 
in Dallas.

Using Looking Glass at HE I can traceroute to my SIXXS tunnel from Chicago but
not from Dallas.

Any ideas on how to get this fixed.

This problem only started occurring within the last week or so.

Thanks for your indulgence,
--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Comcast IPv6 Update

[Steve Clark]

On 11/09/2011 11:40 AM, Jeroen Massar wrote:

On 2011-11-09 17:32 , Brzozowski, John wrote:

Update from http://www.comcast6.net
IPv6 Pilot Market Deployment Begins
Wednesday, November 9, 2011

Comcast has started our first pilot market deployment of IPv6...

Congrats! One step closer to full deployment!

Greets,
  Jeroen


Sort of interesting that you are only handing out a single address and not a 
prefix  - which seems contradictory
to all recommended practices.

Will this be a continued effort for ISP's to charge for extra routable 
addresses?

My $.02




--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: lots of latency on qwest to google?

[Steve Clark]

On 09/20/2011 03:06 PM, Chris Brookes wrote:

Anyone else seeing a lot of latency to google via qwest?

..

11 2 ms 2 ms 2 ms  min-edge-12.inet.qwest.net [207.225.128.1]
1215 ms13 ms12 ms  chx-edge-03.inet.qwest.net [67.14.38.5]
1312 ms21 ms13 ms  72.14.214.78
1413 ms13 ms13 ms  72.14.236.178
1561 ms61 ms61 ms  216.239.43.80
1672 ms61 ms62 ms  66.249.94.200
17   152 ms   145 ms   144 ms  216.239.43.213
18   148 ms   149 ms   150 ms  64.233.175.2
19   149 ms   150 ms   149 ms  66.249.94.34
20   212 ms   221 ms   212 ms  66.249.94.105
21   244 ms   244 ms   245 ms  66.249.94.75
22   244 ms   244 ms   244 ms  209.85.241.33
23   244 ms   243 ms   243 ms  74.125.236.52


We are seeing a routing loop at Qwest at one of our sites.

5  ge-5-2-0-0.ATL01-BB-RTR1.verizon-gni.net (130.81.17.115)  16.142 ms  16.093 
ms  16.101 ms
 6  0.xe-7-1-0.BR3.ATL4.ALTER.NET (152.63.80.73)  16.682 ms  16.254 ms  16.232 
ms
 7  204.255.168.222 (204.255.168.222)  16.412 ms  22.460 ms  21.343 ms
 8  eug-core-02.inet.qwest.net (67.14.32.33)  100.977 ms  99.921 ms  101.427 ms
 9  eug-edge-04.inet.qwest.net (205.171.150.38)  99.565 ms  98.840 ms  100.322 
ms
10  207.109.242.6 (207.109.242.6)  112.195 ms  110.977 ms  112.466 ms
11  eug-edge-04.inet.qwest.net (207.109.242.5)  110.768 ms  111.701 ms  111.362 
ms
12  207.109.242.6 (207.109.242.6)  117.494 ms  113.060 ms  113.308 ms
13  eug-edge-04.inet.qwest.net (207.109.242.5)  120.939 ms  120.411 ms  119.971 
ms
14  207.109.242.6 (207.109.242.6)  125.842 ms  122.599 ms  122.036 ms
15  eug-edge-04.inet.qwest.net (207.109.242.5)  120.446 ms  118.881 ms  119.204 
ms
16  207.109.242.6 (207.109.242.6)  125.540 ms  125.478 ms  138.716 ms
17  eug-edge-04.inet.qwest.net (207.109.242.5)  138.225 ms  132.476 ms  131.683 
ms
18  207.109.242.6 (207.109.242.6)  141.288 ms  142.909 ms  150.655 ms
19  eug-edge-04.inet.qwest.net (207.109.242.5)  148.538 ms  148.713 ms  148.130 
ms
20  207.109.242.6 (207.109.242.6)  156.247 ms  152.812 ms  155.129 ms
21  eug-edge-04.inet.qwest.net (207.109.242.5)  156.888 ms  158.048 ms  156.072 
ms
22  207.109.242.6 (207.109.242.6)  165.790 ms  165.101 ms  168.350 ms
23  eug-edge-04.inet.qwest.net (207.109.242.5)  166.783 ms  167.106 ms  165.928 
ms
24  207.109.242.6 (207.109.242.6)  175.051 ms  176.857 ms  175.693 ms
25  eug-edge-04.inet.qwest.net (207.109.242.5)  176.788 ms  176.379 ms  175.867 
ms
26  207.109.242.6 (207.109.242.6)  184.702 ms  184.590 ms  186.183 ms
27  eug-edge-04.inet.qwest.net (207.109.242.5)  186.509 ms  187.398 ms  185.913 
ms
28  207.109.242.6 (207.109.242.6)  194.984 ms  196.161 ms  195.821 ms
29  eug-edge-04.inet.qwest.net (207.109.242.5)  196.193 ms  195.687 ms  196.331 
ms
30  207.109.242.6 (207.109.242.6)  205.271 ms  205.732 ms  205.718 ms



--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: The stupidity of trying to "fix" DHCPv6

[Steve Clark]

On 06/10/2011 09:37 AM, Ray Soucy wrote:

You really didn't just write an entire post saying that RA is bad
because if a moron of a network engineer plugs an incorrectly
configured device into a production network it may cause problems, did
you?



You are the moron - this stuff happens and wishing it didn't doesn't stop it. 
Get a clue!


Honestly.  This whole argument is getting ridiculous.

On Fri, Jun 10, 2011 at 9:32 AM, Leo Bicknell  wrote:

In a message written on Fri, Jun 10, 2011 at 01:03:01PM +, Bjoern A. Zeeb 
wrote:

On Jun 10, 2011, at 10:10 AM, sth...@nethelp.no wrote:

Several large operators have said, repeatedly, that they want to use
DHCPv6 without RA. I disagree that this is stupid.

I wonder if it's just a "violation" of rule #1: stop thinking legacy!

People are used to what they have done for a decade or two.  It's hard to
see the change and results in "why is this all so different and complicated?".
It's hard to open ones mind for the new, but it is essential to do with new
technology.

The problem in this case is that the failure modes are significantly
different.  Some folks have learned this the hard way.

It's a very easy scenario to reconstruct.  Consider the "branch
office router" in a typical corporate enviornment.  We're talking
a device with one WAN port, and one LAN port.  Configure it for
dual stack, speaking IPv4, and in IPv4 configure it the typical
corporate way with a "DHCP Helper" forwarding requests over the WAN
to a central DHCP server.  In IPv6, configure it with RA's, the
supposed "better" way.

Now, take the 100% working branch router and have it sent back to
corporate.  Maybe they got a bigger router, maybe the office closed.
A network engineer gets the router and is tasked with making it
ready to redeploy.

The network engineer plugs it into the switch on his desktop, plugs in a
serial cable, turns it on and steps out to get a coffee while it boots.
He's planning to erase the configuration and then load new software over
the network.

As soon as the router boots the IPv6 network fails for all the users on
his subnet.  IPv4 keeps working fine.

Oops.

What happened?  Well, the router sent IPv6 RA's as soon as it came
up, and every workstation instantly started using them.  In IPv4,
the router received DHCPv4 requests and forwarded them per the
helper address, except that its WAN port is down, and thus it in
fact didn't send them anywhere.

The important points:

- IPv4 "failed safe" with the DHCP config.  This "rogue device" will
  never disrupt the IPv4 configuration.  DHCP snooping isn't even needed
  in your switches, since it never returns a response.

- IPv6 "failed immediately" with the RA configuration.  What's worse is
  if you simply turn the device off after you realized you took down the
  entire network devices will continue to be broken for 2-4 hours until
  the RA's time out.  The only method to mitigate is to deploy RA guard
  on all of your switches, which probably means replacing 100% of your
  hardware with new stuff that can do that, and then deploying new
  features.

The fact of the matter is that the failure modes of these two
protocols are vastly different operationally.  The DHCP failure
semantics are not only better understood, but cause less disruption
to the network.  Even a properly rouge DHCP server will only damage
_new_ clients coming up on a network, existing folks will work just
fine.  Contrast with RA's which instantly break 100% of the users.

Even more annoying is that if I use RA's for the default gateway,
I still have to run DHCPv6 anyway.  If I don't my boxes don't have
DNS servers, NTP servers, know where to tftpboot, etc.  It's not a
choice of one or the other, it's I always run DHCPv6, do I need
RA's or not.

Given the failure modes I would much prefer to run with RA's turned off
completely, and have DHCPv6 able to provide a default gateway just as it
works in IPv4.

My opinion comes not from "thinking legacy", indeed my employer has been
fully dual stacked since 2003.  My opinion comes from the fact that in
the 8 years of operational experience we have RA's are significantly
more fragile, and IMHO not ready for widespread IPv6 deployment.

--
   Leo Bicknell - bickn...@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/







--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Cogent & HE

[Steve Clark]

On 06/09/2011 06:21 PM, Richard A Steenbergen wrote:

On Thu, Jun 09, 2011 at 12:55:44AM -0700, Owen DeLong wrote:

Respectfully, RAS, I disagree. I think there's a big difference
between being utterly unwilling to resolve the situation by peering
and merely refusing to purchase transit to a network that appears to
offer little or no value to the purchaser or their customers.

Owen, can you please name me one single instance in the history of the
Internet where a peering dispute which lead to network partitioning did
NOT involve one side saying "hey, we're willing to peer" and the other
side saying "no thanks"? Being the one who wants to peer means
absolutely NOTHING here, the real question is which side is causing the
partitioning, and in this case the answer is very clearly HE.

HE wants to peer with Cogent, Cogent doesn't want to peer with HE, and
thus you have an impass and there will be no peering. HE has no problem
using transit to reach Cogent for IPv4 (I see HE reaching Cogent via
1299/Telia, and Cogent reaching HE via 3549/Global Crossing, both very
clearly HE transit providers and Cogent peers), but HE has chosen not to
use transit for the IPv6 traffic. Quite simply, HE feels that they are
entitled to peer with Cogent for the IPv6 traffic, and has deliberately
chosen to create this partition to try and force the issue. These are
*PRECISELY* the same motivations and actions as EVERY OTHER NETWORK who
has ever created a network partition in pursuit of peering that the
other party doesn't want to give them, period.

Again, this isn't necessarily a bad thing if HE thinks it can work to
their long term advantage, but to try and claim that this is anything
else is completely disingenuous. I understand that you have a PR
position to take, and you may even have done a good job convincing the
weak minded who don't understand how peering works that HE is the
victim, but please don't try to feed a load of bullshit to the rest of
us. :)


From reading everything you have said my impression is YOU either work for 
Cogent or have an axe to
grind with HE.

Otherwise I can see no reason for your obvious bias against HE.



--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Yahoo and IPv6

[Steve Clark]

On 05/31/2011 05:31 PM, Voll, Toivo wrote:

Going to http://help.yahoo.com/l/us/yahoo/ipv6/ and hitting "Start IPv6 Test" I 
get:
"Your system will continue to work for you on World IPv6 day. However, we found that 
your server only supports IPv4 at this time. You'll simply continue to use IPv4 to reach 
your favorite web sites."

Netalyzr (http://n3.netalyzr.icsi.berkeley.edu/analysis) finds no issues with 
my IPv6 status, but alerts me to the fact (since confirmed by switching to IE) 
that Google Chrome defaults to IPv4 rather than IPv6, and consequently a lot of 
the testing tools claim that my IPv6 is broken.

Toivo Voll
Network Administrator
Information Technology Communications
University of South Florida

-Original Message-
From: Brandon Ross [mailto:br...@pobox.com]
Sent: Monday, May 09, 2011 12:25
To: Arie Vayner
Cc: nanog@nanog.org
Subject: Re: Yahoo and IPv6

Even more disturbing than that is that when I run a test from here it says
that I have broken v6.  But I don't have broken v6 and test-v6.com proves
it with a 10/10.  This Yahoo tool doesn't seem to even give a hint as to
what it thinks is broken.

Can anyone from Yahoo shed some light on what this tool is doing and how
to get it to tell us what it thinks is broken?


Interesting - must be a windows issue, Google Chrome on Linux works fine at
http://help.yahoo.com/l/us/yahoo/ipv6/


--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: Yahoo and IPv6

[Steve Clark]

On 05/17/2011 08:56 AM, Paul Vixie wrote:

Date: Tue, 17 May 2011 11:07:17 +0200
From: Mans Nilsson


... It's not like you can even reach anything at home now, let alone
reach it by name.

that must and will change.  let's be the generation who makes it possible.

I'd like to respond to this by stating that I support this fully, but
I'm busy making sure I can reach my machines at home from the IPv6
Internet. By name. ;-)

:-).

to be clear, the old pre-web T1 era internet did not have much content
but what content there was, was not lopsided.  other than slip and ppp
there weren't a lot of networks one would call "access" and a smaller
number of networks one would call "content".  i am not wishing for that,
i like the web, i like content, i know there will be specialized networks
for access and content.  but i also think (as jim gettys does) that we
ought to be able to get useful work done without being able to reach the
whole internet all the time.  that's going to mean being able to reach
other mostly-access networks in our same neighborhoods and multitenant
buildings and towns and cities, directly, and by name.  it does not mean
being able to start facebook 2.0 out of somebody's basement, but it does
mean being able to run a personal smtp or web server in one's basement
and have it mostly work for the whole internet and work best for accessors
who are close by and still work even when the "upstream" path for the
neighborhood is down.


This is all very confusing to me. How are meaningful names going to assigned 
automatically?
Right now I see something like ool-6038bdcc.static.optonline.net for one of our 
servers, how does this
mean anything to anyone else?


--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com

Re: 365x24x7

[Steve Clark]

On 04/15/2011 09:25 AM, Charles Mills wrote:

I've had it done in places where I work where you'll have 3 rotations
working 12 hour shifts.

In a 2 week pay period they get their 80 hours in a blend 36 one week
and 44 the next.  It gives some nice consecutive days off time which
also doubles as a retention tool for some employees.  You might have
to get creative to have all the days work out but it can be done.

On Fri, Apr 15, 2011 at 9:14 AM, harbor235  wrote:

If I were going to provide a 365x24x7 NOC, how many teams of personnel do I
need
to fully cover operations? I assume minimally you need 3 teams to cover the
required
24 hr coverage, but there is off time and schedule rotation?

thoughts, experience?

Mike





I worked once in a production plant. They had 4 crews a,b,c,d and worked 6 days 
on 2 off. It was rotating shifts. As an example crew A worked 6
days 8-4 was off 2 days then started the next shift on 4-midnight, then 
midnight to 8. So at any one day 3 of the 4 crews were working and the other
crew was off.

--
Stephen Clark
*NetWolves*
Sr. Software Engineer III
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.cl...@netwolves.com
http://www.netwolves.com