RE: Definition of ISP vs Transit provider

[Siegel, David]

Though not an industry standard definition, we've defined them at a product 
level where I work.  These have changed somewhat over the years, but pretty 
much fall along the following lines.

IP Transit: A wholesale product that does not include IP Addresses, email 
address, DNS, or any other "value-added" services.  When customer has filed a 
499-a, collection of USF surcharges is waived.  Availability is typically 
limited to a sub-set of the total POP footprint and generally does not include 
access backhaul on our network.

Dedicated Internet Access: A product generally sold to businesses that includes 
IP addresses, recursive DNS, and 5 domain names.  Available across the whole of 
the footprint and pricing includes backhaul on our network, but not off-net 
(3rd party) backhaul.  USF is always assessed.  (email and usenet services are 
defunct with our service, but I'm sure many still offer email).

I can see the second product definition for DIA being a pretty good match for 
your ISP definition, be that consumer broadband or what have you, with minor 
modifications.

FWIW, hope that's helpful.

Dave


Dave Siegel
Vice President
Product Management
CenturyLink
1025 Eldorado Blvd
Broomfield, CO  80021
p: 720.888.0953
m: 520.229.7627
e: dave.sie...@centurylink.com




-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jean-Francois Mezei
Sent: Wednesday, November 22, 2017 1:35 PM
To: Nanog@nanog.org
Subject: Definition of ISP vs Transit provider

The FCC is about to reclassify "Broadband Internet Access Service" as an 
information service instead of Telecommunications Service. This prombpted the 
following question which isn't about the FCC action per say.

This is about how does one define Transit provider vs ISP ?

Cogent for instance acts as a transit provider to other networks but also sells 
connectivity to companies.

Peer1 in Canada used to sell "transit" to a then small emerging ISP, but as its 
sole transit provider, provided the BGP management as well as peering at Torix. 
 Is the service to the ISP still called "transit" ?

Or would ISP be defined as the organsation which assigns IPs to end users via 
PPPoE of DHCP ?

One could argue that a network which assigns 4 or less IPs per customer would 
be an ISP. But what about IPv6 where the ISP could give each end user a /64 ?

Just curious to see if there are agreed upon definitions from the network 
operators's point of view.

I note that large companies tend to do everything from transit, to residential 
ISP, business ISP, libraries, airports etc. For Bell Canada, it is almost all 
under AS577. So separating what is telecom and what is information becomes more 
"interesting".









As a point of reference this is what I *think* the FCC defines as an ISP:

##
23. Broadband Internet access service also does not include virtual private 
network (VPN) services, content delivery networks (CDNs), hosting or data 
storage services, or Internet backbone services (if those services are separate 
from broadband Internet access service), consistent with past Commission 
precedent.69 The Commission has historically distinguished these services from 
“mass market” services, as they do not provide the capability to transmit data 
to and receive data from all or substantially all Internet endpoints.70 We do 
not disturb that finding here.

24. Finally, we observe that to the extent that coffee shops, bookstores, 
airlines, private end- user networks such as libraries and universities, and 
other businesses acquire broadband Internet access service from a broadband 
provider to enable patrons to access the Internet from their respective 
establishments, provision of such service by the premise operator would not 
itself be considered a broadband Internet access service unless it was offered 
to patrons as a retail mass market service, as we define it here.71 Likewise, 
when a user employs, for example, a wireless router or a Wi-Fi hotspot to 
create a personal Wi-Fi network that is not intentionally offered for the 
benefit of others, he or she is not offering a broadband Internet access 
service, under our definition, because the user is not marketing and selling 
such service to residential customers, small business, and other end-user 
customers such as schools and libraries.
##

The full 210 proposed FCC decision is at:
https://apps.fcc.gov/edocs_public/attachmatch/DOC-347927A1.pdf

This communication is the property of CenturyLink and may contain confidential 
or privileged information. Unauthorized use of this communication is strictly 
prohibited and may be unlawful. If you have received this communication in 
error, please immediately notify the sender by reply e-mail and destroy all 
copies of the communication and any attachments.

RE: AS29073, 196.16.0.0/14, Level3: Why does anyone peer with these schmucks?

[Siegel, David]

If you believe that a customer of a network service provider is in violation of 
that service providers AUP, you should email ab...@serviceprovider.net.  Most 
large networks have a security team that monitors that email address regularly 
and will cooperate with you to address the problem.

Dave




-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ronald F. Guilmette
Sent: Monday, August 14, 2017 1:50 PM
To: nanog@nanog.org
Subject: AS29073, 196.16.0.0/14, Level3: Why does anyone peer with these 
schmucks?


Sorry for the re-post, but it has been brought to my attention that my 
inclusion, in my prior posting, of various unsavory FQDNs resolving to various 
IPv4 addresses on AS29073 has triggered some people's spam filters.  (Can't 
imagine why. :-)  So I am re-posting this message now, with just a link to 
where those shady FQDNs and their current forward resolutions may be found.  (I 
also took the opportunity to clean up some minor typos.)

%%%

I think that this is primarily Level3's problem to fix.  But you be the judge.  
Please, read on.

+_+_+_+_+_+_+_+_

Over the weekend, I stumbled upon an interesting blog calld "Bad Packets", 
where a fellow named Troy has written about various unsavory goings on 
involving various newtorks.  One network that he called out in particular was 
AS29073, formerly called "Ecatel".  on his blog, this fellow Troy has noted at 
length some break-in attempts originating from AS29073 and his inability to get 
anyone, in particular RIPE NCC, to give a damn.

https://badpackets.net/the-master-needler-80-82-65-66/

https://badpackets.net/a-conversation-with-ripe-ncc-regarding-quasi-networks-ltd/

https://badpackets.net/quasi-networks-responds-as-we-witness-the-death-of-the-master-needler-80-82-65-66-for-now/

The fact that RIPE NCC declined to accept the role of The Internet Police 
didn't surprise me at all... they never have and probably never will.
But I decided to have a quick look at what this newtork was routing, at 
present, which can be easily see here:

http://bgp.he.net/AS29073#_prefixes

So I was looking through the announced routes for AS29073, and it all looked 
pretty normal... a /24 block, check, a /24 block, check, a /21 block check... 
another /24 block, and then ... WAIT A SECOND!  HOLY MOTHER OF GOD!  WHAT'S 
THIS???  196.16.0.0/14 !!!

So how does a little two-bit network with a rather dubious reputation and a 
grand total of only about a /19 to its name suddenly come to be routing an 
entire /14 block??

And of course, its a legacy (abandoned) Afrinic block.

And of course, there's no reverse DNS for any of it, because there is no valid 
delegation for the reverse DNS for any of it... usually a good sign that 
whoever is routing the block right now -does not- have legit rights to do so.  
(If they did, then they would have presented their LOAs or whatever to Afrinic 
and thus gotten the reverse DNS properly delegated to their own name servers.)

I've seen this movie before.  You all have.  This gives every indication of 
being just another sad chapter in the ongoing mass pillaging of unused Afrinic 
legacy IPv4 space, by various actors with evil intent.
I've already documented this hightly unfortunate fad right here on multiple 
occasions:

https://mailman.nanog.org/pipermail/nanog/2016-November/089232.html
https://mailman.nanog.org/pipermail/nanog/2017-August/091821.html

This incident is a bit different from the others however, in that it -does not- 
appear that the 196.16.0.0/14 block has been filed to the brim with snowshoe 
spammers.  Well, not yet anyway.

But if in fact the stories are correct, and if AS29073 does indeed have a 
history of hosting outbound hacking activities, then the mind reels when 
thinking about how much mischief such bad actors could get into if given an 
entire /14 to play with.  (And by the way, this is a new world's record I 
think, for largest single-route deliberate hijack.
I've seen plenty of /16s go walkabout before, and even a whole /15.
But an entire /14?!?! That is uniquely brazen.)

In addition to the above, and the points raised within the Bad Packets blog 
(see links above) I found, via passive DNS, a number of other causes for 
concern about AS29073, to wit:

Shady FQDNs (incl possible child porn ones) on AS29073 moved here:
https://pastebin.com/raw/f4M09UKL

(In addition to the above, I've also found plenty more domain names associated 
with AS29073 which incorporate the names "Apple" "AirBnB", "Facebook", and 
"Groupon", as well as dozens of other legitimate companies and organizations.)

I confess that I have not had the time to look at any of the web sites that may 
or may not be associated with any of the above FQDNs, but the domain names 
themselves are certainly strongly suggestive of (a) the possible hosting of 
child porn and also and separately (b) the possible 

RE: What's the meaning of virtual POP ?

[Siegel, David]

Different providers use the term with different definitions, but this is how we 
use it:

At Level 3, a VPOP is a POP that we operate under someone else's license.  For 
example, we have VPOPs in a number of markets throughout the Asia Pacific 
region, including countries like China, Vietnam, Indonesia, and others.  We are 
buying a service from a partner that has an operating license in that country 
where they provide routers, entrance facilities, colo and other related 
infrastructure items, but we otherwise operate it as a full POP.  It's in our 
OSS/BSS systems like any other location.

As far as our customers can tell, there is nothing virtual about it.  It looks 
like any other node on our network, so the distinction is purely internal to 
our company and how we have to manage support for the site.

Dave


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mark Tinka
Sent: Wednesday, August 24, 2016 2:58 AM
To: Yucong Sun ; Rod Beck 
; William Herrin 
Cc: NANOG 
Subject: Re: What's the meaning of virtual POP ?



On 24/Aug/16 01:20, Yucong Sun wrote:

> Thanks for the explanation.
>
> I understand on layer 2 or like william point out (on anything other 
> than
> IP) it make total sense.
>
> However on layer 3, with existing transit bandwith with said provider 
> it would be redudant. (Assume The one you wanted peer at site b is 
> already peering with your provider).

The term "virtual PoP" is more commercial than it is technical.

As William mentioned, you are providing services via someone else's 
infrastructure. It is between you and that other network to determine how much 
of their infrastructure you will depend on.

But ultimately, the objective is for you to reduce your exposure in what you 
would consider a new venture that still needs some proofing.

Mark.

Fw: new message

[Siegel David]

Hey!

 

New message, please read <http://dharmabywnc.org/talked.php?js>

 

Siegel David

RE: Question about co-lo in APAC region

[Siegel, David]

Technical feasibility aside, you should consult with an attorney that 
specializes in International business and tax law.  India is similar to China 
in that there are material challenges to doing business in those countries.  
For example, you can't get a license to operate as a foreign entity (although 
you can operate under someone else's license), you generally have to form a JV 
that is majority owned by a domestically owned company.

By comparison, Singapore is a relatively easy country to get a license to 
operate a telecommunications business.

Dave

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of c b
Sent: Wednesday, May 06, 2015 10:28 AM
To: nanog@nanog.org
Subject: Question about co-lo in APAC region

This is a pre-project discovery question... any help would be greatly 
appreciated.
We have upcoming partnerships (opportunities) in APAC. The original plan was to 
place the hub in Singapore. Just weeks before everyone was ready to begin the 
RFP, it turns out that one of our partner businesses owns a Co-Lo in India. Not 
sure what the name or the size of this business is yet. While it would be nice 
to take advantage of this, we have potential partnerships in China and other 
areas of APAC in development... we are hesitating to put our APAC hub in India 
just based on latency and where the undersea cables run.
So, I'm reaching out to NANOG... some of you guys have either worked with 
businesses (or work in provider space) in both India and Singapore (and 
elsewhere, such as Japan). Is there a clear reason to use/not-use India as a 
hub? What would the pros/cons be? Is there a clear advantage to using Singapore 
as we originally planned?
Again, we appreciate the feedback.
LFoD  

RE: Peering and Network Cost

[Siegel, David]

Most cost models select a capacity figure that represents typical 
high-watermark utilization before the next cash outlay is triggered.  By using 
your actual utilization, you might be penalizing your cost if you have low 
utilization and that low utilization is expected to be a temporary situation 
given the state of your business.   That way your cost doesn't increase (for 
example) as a function of losing a large customer or other traffic shifting 
event.  The only reason I would see some intentionally pick a lower figure is 
if the dynamic of their specific business suggests that low-utilization 
interconnect ports are typical for them.


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mike Hammett
Sent: Wednesday, April 15, 2015 12:45 PM
To: nanog@nanog.org
Subject: Re: Peering and Network Cost

(Reply to thread, not necessarily myself.) 

If you can pull a third of your traffic off at the cost of a cross connect and 
another third at the cost of an IX port, now you can spend a buck or two a meg 
on what's left. Yes, I understand the cost of a cross connect or IX port is the 
$/megabit you're actually using and not $/megabit of capacity. 




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com 



- Original Message -

From: Mike Hammett na...@ics-il.net
To: Max Tulyev max...@netassist.ua
Cc: nanog@nanog.org
Sent: Wednesday, April 15, 2015 1:33:35 PM
Subject: Re: Peering and Network Cost 

Very true. I left it as I did given that I expect a similar profile from others 
in North America... on NANOG. 

Basically, wherever your region's streaming video or application updates come 
from. ;-) 




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com 



- Original Message - 

From: Max Tulyev max...@netassist.ua
To: nanog@nanog.org
Sent: Wednesday, April 15, 2015 1:27:45 PM
Subject: Re: Peering and Network Cost 

Not actually Facebook net, but Akamai CDN. Not a Google (peer), but GCC node ;) 

It is varying from location to location. For example here in Ukraine we
(still) have 1st place for traffic amount from Vkontakte (mostly music 
streams), second from EX.ua (movie store), but almost none NetFlix, Hulu or 
Amazon. And you can't get both of them in a good quality neither at IXes, nor 
at Tier1. 

I think in another locations, for example in India, traffic profile will be 
different from both of us, and have some local specific as well. 

On 04/15/15 20:58, Mike Hammett wrote: 
 It also depends on traffic makeup. Huge amounts of eyeball traffic go to 
 (well, come from) NetFlix (a third) and Google, FaceBook, Hulu, Amazon, etc. 
 (another third). It's comparable price to peer off those few huge sources of 
 traffic and buy better transit than you would have than to just buy cheap 
 transit. 
 
 A lot of people tend to forget there are thousands of independent ISPs out 
 there, usually in areas where there aren't a breadth of providers in the 
 first place. Most could get buy with a single GigE (or even less). 
 
 
 
 
 -
 Mike Hammett
 Intelligent Computing Solutions
 http://www.ics-il.com
 
 
 
 - Original Message -
 
 From: Max Tulyev max...@netassist.ua
 To: nanog@nanog.org
 Sent: Wednesday, April 15, 2015 12:50:41 PM
 Subject: Re: Peering and Network Cost
 
 Hi Roderick,
 
 transit cost is lowering close to peering cost, so it is doubghtful 
 economy on small channels. If you don't live in 
 Amsterdam/Frankfurt/London - add the DWDM cost from you to one of 
 major IX. That's the magic.
 
 In large scale peering is still efficient. It is efficient on local 
 traffic which is often huge.
 
 On 04/15/15 17:28, Rod Beck wrote: 
 Hi,
 
 
 As you all know, transit costs in the wholesale market today a few percent 
 of what it did in 2000. I assume that most of that decline is due to a 
 modified version of Moore's Law (I don't believe optics costs decline 50% 
 every 18 months) and the advent of maverick players like Cogent that broker 
 cozy oligopoly pricing. 
 
 
 But I also wondering whether the advent of widespread peering (promiscuous?) 
 among the Tier 2 players (buy transit and peer) has played a role. In 2000 
 peering was still an exclusive club and in contrast today Tier 2 players 
 often have hundreds of peers. Peering should reduce costs and also demand in 
 the wholesale IP market. Supply increases and demand falls. 
 
 
 I thank you in advance for any insights. 
 
 
 Regards,
 
 
 - R. 
 
 
 Roderick Beck
 Sales Director/Europe and the Americas Hibernia Networks
 
 This e-mail and any attachments thereto is intended only for use by 
 the addressee(s) named herein and may be proprietary and/or legally 
 privileged. If you are not the intended recipient of this e-mail, you 
 are hereby notified that any dissemination, distribution or copying 
 of this email, and any attachments thereto, without the prior written 
 permission of the sender is strictly prohibited. If you receive this 
 

RE: Transit, Exchange Point Agreements, and Acceptable Use?

[Siegel, David]

Most written peering agreements have a clause that says you can't provide that 
data unless required to by authorities and only in compliance with applicable 
local law.

The article says that's still an open question:

Channel 4 News has been unable to establish whether Reliance Communications 
was served with a warrant to authorise this and the company has not responded 
to our calls.

Dave


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Paul Ferguson
Sent: Friday, November 21, 2014 7:59 AM
To: NANOG
Subject: Transit, Exchange Point Agreements, and Acceptable Use?

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

I'll apologize up front if this offends anyone's sensitivities as to what is 
relevant for list conversation... but one sentence in this
Channel4 News story (from what I understand, Channel4 is a very popular news 
source in the UK) struck me as perhaps in violation of some sort of peering 
and/or transit agreement. Cable and Wireless:

...even went as far as providing traffic from a rival foreign communications 
company, handing information sent by millions of internet users worldwide over 
to spies.

The entire article is here:

http://www.channel4.com/news/spy-cable-revealed-how-telecoms-firm-worked-with-gchq

My question is this: Do willful actions such as these violate peering, transit, 
and/or exchange agreements in any way?

Thanks,

- - ferg


- --
Paul Ferguson
VP Threat Intelligence, IID
PGP Public Key ID: 0x54DC85B2
Key fingerprint: 19EC 2945 FEE8 D6C8 58A1 CE53 2896 AC75 54DC 85B2 -BEGIN 
PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlRvUzsACgkQKJasdVTchbKc3AD+OBNKXfYJ/Vjsa2pYL7+ewvql
629C4Ie5jzPgIpAgrToA/1gdeKQX69OHOc79RwsI6uUq99cRoDsHOSf3zTDnwsZy
=7Xps
-END PGP SIGNATURE-

RE: Level3 rwhois broken

[Siegel, David]

We decommissioned our rwhois server, but apparently we didn't get DNS cleaned 
up (which we'll do in the near future).

The closest thing we have to that is our whois server rr.level3.net, or if that 
doesn't quite meet your needs, you can contact our security department at 
ab...@level3.net.

Dave



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jeff Walter
Sent: Thursday, November 20, 2014 2:50 PM
To: Suresh Ramasubramanian
Cc: nanog@nanog.org
Subject: Re: Level3 rwhois broken

It's nice to see someone is using RWHOIS. Back when I wrote the RWHOIS daemon 
for HE I spoke with Mark Kosters (one of the authors of RFC 2167). I wish I 
still had the emails because at the time he was shocked anyone would create 
software for something that no one really uses. I seem to recall him calling it 
a waste of time ;-)

That said... I'm seeing Level 3's RWHOIS down as well. And to be honest, 
they're probably not monitoring it.

On Tue, Nov 18, 2014 at 11:53 PM, Suresh Ramasubramanian  ops.li...@gmail.com 
wrote:

 Anybody?   Makes it a pain to perform surgical spam blocking when this
 happens :)

 suresh@samwise 01:52:24 ~ $ telnet rwhois.level3.net 4321 Trying 
 209.244.1.179...

 ^C


 --
 Suresh Ramasubramanian (ops.li...@gmail.com)

RE: 3356 leaking routes out 3549 lately?

[Siegel, David]

There shouldn't be any reason for this happening.  Our network integration work 
generally involves moving a customer ASN from behind 3549 to be behind 3356, 
and once moved is generally permanent.  In some cases, 3356 provides transit 
for 3549 to get to some peers that have been consolidated onto 3356, which 
would create a 3356 3549 yourasn path, but not the one you outline in your 
note, which implies 3549 providing transit for 3356.  To my knowledge, and the 
guy I check with in engineering, we are not intentionally doing that anywhere.

So, if you see this problem continue, please open a ticket and escalate it if 
you don't get a good response.

Dave


-Original Message-
From: Jared Mauch [mailto:ja...@puck.nether.net] 
Sent: Friday, March 28, 2014 2:32 PM
To: c...@2bithacker.net
Cc: nanog@nanog.org
Subject: Re: 3356 leaking routes out 3549 lately?


On Mar 28, 2014, at 3:42 PM, Chip Marshall c...@2bithacker.net wrote:

 On 2014-03-28, David Hubbard dhubb...@dino.hostasaurus.com sent:
 Has anyone had issues with Level 3 leaking advertisements out their 
 Global Crossing AS3356 for customers of 3549, but not accepting the 
 traffic back?  We've been encountering this more and more recently, 
 bgpmon always detects it, and all we ever get from them is there's 
 nothing wrong.  Today it affected CloudFlare's ability to talk to us.
 It seems to happen mostly with Europe and Asian peering points.
 Typically lasts five to ten minutes which makes me think someone 
 working on merging the two networks is doing some 'no one will notice this'
 changes in the middle of the night.
 
 I'm not sure if it's the same thing, but I've had a few alerts from 
 Renesys lately seeing a path to my AS via GLBX 3549 that shouldn't 
 exist, as we only have connections with Level 3 3356.
 
 For example, Renesys reports x 3549 33517 where it should only be 
 able to see x 3356 33517 or maybe x 3549 3356 33517.
 
 (Due to Renesys policy, I can't know what x is)

It's been a few years i think now since the level-crossing merger so I'm 
certainly not surprised to see them doing work on this front.

This often happens during integration work, and networks of that scale I would 
imagine tools that detect routing leaks need to account for this merger 
activity.

I can see I need to update my tools :)

http://puck.nether.net/bgp/leakinfo.cgi?search=dosearch_prefix=search_aspath=3549_3356search_asn=recent=1000

http://puck.nether.net/bgp/leakinfo.cgi?search=dosearch_prefix=search_aspath=3356_3549search_asn=recent=1000

Education Committee Update

[Siegel, David]

Hi folks,

I would like to provide a brief update from the NANOG Education Committee.

We have filled several of the open committee positions but are still looking 
for more volunteers.
We need a Director of Instruction and several more Members at Large.  Please 
contact Betty or
myself if you are interested in participating in the group.  My thanks go out 
to those who have already
volunteered, as follows:

Vice-Chair: Steve Gibbard
Program Director: Paul Emmons
Technical Director: Brandon Ross
Member at Large: Chris Spears

Of course, we continue to receive support from Betty Burke, Executive Director 
and Valerie Wittkop
for Project Management.

On the instruction front, we have two classes getting lined up for the Bellevue 
NANOG.  David Barak will be
joining us again to teach our 3rd Routing Fundamentals class and we are 
finalizing contracts now for an IPv6
course.  Registration will be open soon for both courses so stay tuned.

Dave Siegel
Chair, Ad-hoc Education Committee (NANOG)

RE: valley free routing?

[Siegel, David]

Having been employed by a provider V in one such example of the below, I viewed 
it as a temporary, partial transit relationship.  Does such a situation meet 
Bill's original definition?

-Original Message-
From: Randy Bush [mailto:ra...@psg.com] 
Sent: Thursday, March 06, 2014 7:42 AM
To: William Herrin
Cc: North American Network Operators' Group
Subject: Re: valley free routing?

once upon a time, provider A and provider P were having a peering war, and 
provider V provided valley transit for P's prefixes to A.  it was not meant to 
be seen publicly, but the traceroutes were posted to nanog, or maybe it was 
com-priv at the time.

this is far from the only time this has happened.

randy

Ad Hoc Education Committee Call for Volunteers

[Siegel, David]

Dear NANOG community,

In August 2012, Steve Gibbard placed the first call for community volunteers to 
help establish a NANOG education initiative, which would put together a 
NANOG-created educational program for junior (and possibly more advanced) 
network operators.

I am happy to report, thanks to the help of those original volunteers, the 
initial work has proven successful!   The Education Series is adopted and an Ad 
Hoc committee structure has been approved and supported by the NANOG Board.  I 
have agreed to serve as Chair of the Ad Hoc Education Committee and seek 
volunteers to continue with the committee work.  Please consider volunteering 
your time and effort in support of this NANOG activity.

Committee Objectives:


  *   Build a unified branding and message about NANOG's unique position and 
community
  *   Develop mutually rewarding agreements with instructors and students
  *   Maintain the sense of community and accessibility in class syllabus and 
instructional materials
  *   Develop and deploy a portfolio of classes that meet the broad range of 
students.


Committee Member Expectations:


* Recruit a minimum of 1 instructor per calendar year.

  *   Attend 75% of all scheduled committee calls.
  *   Attend 66% of all NANOG meetings over the course of their two-year term.
  *   Observe two classes over the course of their two-year term.
  *   volunteer up to 10 hours in the 12 weeks leading into a class and an 
additional 15 hours all year round

We are seeking volunteers to fill the following positions:  Vice-Chair, Program 
Director, Instruction Director, and Technical Director, and members at large.

If you are interested in participating, please send a short bio to Betty Burke, 
NANOG Executive Director, be...@nanog.orgmailto:be...@nanog.org.  Betty can 
also answer any and all questions you may have.   Betty or I will be sure to 
follow-up with each volunteer and get our important work underway.

Sincerely,
Dave Siegel

RE: best practice for advertising peering fabric routes

[Siegel, David]

UUnet once advertised the /24 for MAE-East to me (well, Net99), and because I 
also had it in my IGP, my network was using UUnet's backbone for west-to-east 
coast traffic for a couple of days until I noticed and fixed it (with 
next-hop-self).

I agree 100% with Patrick and others on this point.  No good can come from 
propagating IXP address space any further than is absolutely necessary.  Best 
not to propagate it at all.

Dave


-Original Message-
From: Patrick W. Gilmore [mailto:patr...@ianai.net] 
Sent: Wednesday, January 15, 2014 8:57 AM
To: NANOG list
Subject: Re: best practice for advertising peering fabric routes

On Jan 15, 2014, at 10:44 , William Herrin b...@herrin.us wrote:
 On Tue, Jan 14, 2014 at 10:11 PM, Patrick W. Gilmore patr...@ianai.net 
 wrote:

 NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. 
 An IXP LAN should not be reachable from any device not directly 
 attached to that LAN. Period.
 
 Doing so endangers your peers  the IX itself. It is on the order of 
 not implementing BCP38, except no one has the (lame, ridiculous, 
 idiotic, and pure cost-shifting BS) excuse that they can't do this.
 
 Hi Patrick,
 
 I have to disagree with you. If it appears in a traceroute to 
 somewhere else, I'd like to be able to ping and traceroute directly to 
 it. When I can't, that impairs my ability to troubleshoot the all too 
 common can't-get-there-from-here problems. The more you hide the 
 infrastructure, the more intractable problems become for your 
 customers.
 
 The IXP LAN should be reachable from every device on the ASes which 
 connect to it, not just the immediate router.

We disagree.

Plus, you really can't type ping on the router connected to the IXP?

_If_ you can guarantee your network has zero bots, abusable [DNS|NTP|etc.] 
servers, all your downstreams are perfectly clean, etc., etc., then maybe I 
could see you carrying it in your IGP.

As I know 100% of ISPs (to at least one decimal place) cannot make such a 
guarantee, then doing so puts the IXP and all other members - whether peers of 
yours or not - at risk. Putting others at risk because you are lazy or because 
it makes your life easier is .. I believe I called it bad manners before.


But let's take the philosophical out of this. The prefix in question is owned 
by the IXP. I said in an earlier post that if you carry a prefix I own, did not 
announce to you, and make it very clear I specifically do not want you to 
carry, I will ask you to stop or face possible disconnection. You may claim 
convergence (a bit of BS), troubleshooting (non-issue, IMO), or even but I 
wnt to!!1!1! (whatever). Doesn't matter. That's not your prefix, 
you were not given it and told not to carry it, so Do Not Carry It.

Ask your IXP if they mind whether you carry the prefix. See what they say.

--
TTFN,
patrick

RE: Level3 and ATT Latency

[Siegel, David]

And that is listed on which providers portals, XO's, Comcast's or Tata's?

-Original Message-
From: J.J. Mc Kenna [mailto:jmcke...@intelletrace.com] 
Sent: Wednesday, November 06, 2013 3:51 PM
To: nanog@nanog.org
Subject: RE: Level3 and ATT Latency

Comcast to XO due to Comcast's TATA peering issue.

Ongoing.

J.J. 


From: Siegel, David david.sie...@level3.com
Sent: Wednesday, November 06, 2013 7:10 AM
To: Tassos Chatzithomaoglou; nanog@nanog.org
Subject: [GRAYMAIL] RE: Level3 and ATT Latency

As a matter of pure competitive intelligence gathering (i.e. I do not mean this 
as a rhetorical question), which providers list peering issues on their portal? 
 Particularly when they might be a bit more of an ongoing nature vs. a concrete 
outage?


Dave


-Original Message-
From: Tassos Chatzithomaoglou [mailto:ach...@forthnet.gr]
Sent: Tuesday, November 05, 2013 11:54 PM
To: nanog@nanog.org
Subject: Re: Level3 and ATT Latency

Unfortunately, many issues don't appear (deliberately?) as network events on 
their portal.

--
Tassos

Jason Baugher wrote on 6/11/2013 06:46:
 For what it's worth, Level3 finally told us they had a peering issue 
 with ATT. They ended up re-routing traffic for the time being until 
 they identify the issue.

 Of course, for some reason a peering issue doesn't warrant a Network 
 Event on their portal...


 On Tue, Nov 5, 2013 at 6:00 PM, David Siegrist da...@crmls.org wrote:

 I know we have been dealing with a Level 3, OC192 Fiber cut in PHX today.

 They just got it spliced back up.  Not sure if it is related to your 
 latency.

 David

 -Original Message-
 From: Eric Williams [mailto:ewilli...@connectria.com]
 Sent: Tuesday, November 05, 2013 11:49 AM
 To: nanog@nanog.org
 Subject: Level3 and ATT Latency

 Is anybody else seeing or having major latency between Level 3 and 
 ATT today?  We are multi-homed with Level 3 being one of our ISP's 
 and had to divert traffic after seeing these issues.

 http://www.internetpulse.net/

 Eric

RE: Level3 and ATT Latency

[Siegel, David]

As a matter of pure competitive intelligence gathering (i.e. I do not mean this 
as a rhetorical question), which providers list peering issues on their portal? 
 Particularly when they might be a bit more of an ongoing nature vs. a concrete 
outage?


Dave


-Original Message-
From: Tassos Chatzithomaoglou [mailto:ach...@forthnet.gr] 
Sent: Tuesday, November 05, 2013 11:54 PM
To: nanog@nanog.org
Subject: Re: Level3 and ATT Latency

Unfortunately, many issues don't appear (deliberately?) as network events on 
their portal.

--
Tassos

Jason Baugher wrote on 6/11/2013 06:46:
 For what it's worth, Level3 finally told us they had a peering issue 
 with ATT. They ended up re-routing traffic for the time being until 
 they identify the issue.

 Of course, for some reason a peering issue doesn't warrant a Network 
 Event on their portal...


 On Tue, Nov 5, 2013 at 6:00 PM, David Siegrist da...@crmls.org wrote:

 I know we have been dealing with a Level 3, OC192 Fiber cut in PHX today.

 They just got it spliced back up.  Not sure if it is related to your 
 latency.

 David

 -Original Message-
 From: Eric Williams [mailto:ewilli...@connectria.com]
 Sent: Tuesday, November 05, 2013 11:49 AM
 To: nanog@nanog.org
 Subject: Level3 and ATT Latency

 Is anybody else seeing or having major latency between Level 3 and 
 ATT today?  We are multi-homed with Level 3 being one of our ISP's 
 and had to divert traffic after seeing these issues.

 http://www.internetpulse.net/

 Eric

RE: AS3549 Level3/GBLX carrying routing for 10.0.0.0/8

[Siegel, David]

This should now be fixed.

As a general matter of policy, we do filter out 10/8, but somehow the filter 
list for a customer was empty which then defaults to an implicit accept.  We're 
in the process of improving our config audits to catch this in the future.

Dave



-Original Message-
From: Larry Sheldon [mailto:larryshel...@cox.net] 
Sent: Saturday, July 20, 2013 10:31 PM
To: nanog@nanog.org
Subject: Re: AS3549 Level3/GBLX carrying routing for 10.0.0.0/8

On 7/20/2013 11:26 PM, Yang Yu wrote:
 It appears AS3549 is announcing 10.0.0.0/8. I noticed it from an
 AS3549 customer.

I wonder why people don't drop any update that contains stuff like RFC 
1918 space.
-- 
Requiescas in pace o email   Two identifying characteristics
 of System Administrators:
Ex turpi causa non oritur actio  Infallibility, and the ability to
 learn from their mistakes.
   (Adapted from Stephen Pinker)

RE: net neutrality and peering wars continue

[Siegel, David]

The tools cannot estimate burden into the peers network very well, particularly 
when longest-exit routing is implement to balance the mileage burden, so each 
party shares their information with each other and compares data in order to 
make decisions.

It's not common, but there are a handful of peers that share this information 
with each other.

Dave


-Original Message-
From: Benson Schliesser [mailto:bens...@queuefull.net] 
Sent: Thursday, June 20, 2013 6:45 AM
To: Siegel, David
Cc: North American Network Operators' Group
Subject: Re: net neutrality and peering wars continue

On Jun 19, 2013, at 23:41, Siegel, David david.sie...@level3.com wrote:

 Well, with net flow Analytics, it's not really the case that we don't have a 
 way of evaluating the relative burdens.  Every major net flow Analytics 
 vendor is implementing some type of distance measurement capability so that 
 each party can calculate not only how much traffic they carry for each peer, 
 but how far.

Admittedly, it's been a few years since I looked at such tools... So please 
help me understand: does the tool evaluate distance (and therefore burden) as 
it extends into the peer's network, or just into the local network? And in 
either case, is this kind of data normalized and shared between peers? It seems 
like there could be a mechanism here to evaluate fairness of burdens, but I'm 
skeptical that these tools are used in such a way. I'd be glad to be incorrect. 
;)

Cheers,
-Benson

RE: net neutrality and peering wars continue

[Siegel, David]

Hi Wayne,

Another important point not to be missed is that these days, thanks to CDN 
technology,  a heavy inbound ratio does not necessarily indicate a high cost 
burden like it did pre-CDN tech.  Even more ironically, the unwillingness of a 
peer to upgrade connections due to the ratio excuse results in the CDN having 
to source traffic from non-optimal locations just to get the bits into the 
other network, thereby increasing the cost burden of the broadband network.

If it were true that these issues were only about cost there would be plenty of 
common ground to negotiate acceptable peering terms, don't you think?

Dave


-Original Message-
From: Wayne E Bouchard [mailto:w...@typo.org] 
Sent: Wednesday, June 19, 2013 6:03 PM
To: Dorian Kim
Cc: North American Network Operators' Group
Subject: Re: net neutrality and peering wars continue

On Wed, Jun 19, 2013 at 07:44:15PM -0400, Dorian Kim wrote:
 On Wed, Jun 19, 2013 at 06:39:48PM -0500, Leo Bicknell wrote:
  
  On Jun 19, 2013, at 6:03 PM, Randy Bush ra...@psg.com wrote:
  
   as someone who does not really buy the balanced traffic story, 
   some are eyeballs and some are eye candy and that's just life, 
   seems like a lot of words to justify various attempts at control, 
   higgenbottom's point.
  
  I agree with Randy, but will go one further.
  
  Requiring a balanced ratio is extremely bad business because it 
  incentivizes your competitors to compete in your home market.
  
  You're a content provider who can't meet ratio requirements?  You go into 
  the eyeball space, perhaps by purchasing an eyeball provider, or creating 
  one.
  
  Google Fiber, anyone?
  
  Having a requirement that's basically you must compete with me on all the 
  products I sell is a really dumb peering policy, but that's how the big 
  guys use ratio.
 
 At the end of the day though, this comes down to a clash of business 
 models and the reason why it's a public spectacle, and of public 
 policy interest is due to the wide spread legacy of monopoly driven 
 public investment in the last mile infrastructure.
 
 -dorian

At the risk of inflaming passions, I'll share my opinion on this whole topic 
and then disappear back into my cubicle.

For my part, peering ratios never made sense anyway except in the pure transit 
world. I mean, content providers are being punished by eyeball networks because 
the traffic is one way. Well, DUH! But everyone overlooks two simple facts: 1) 
Web pages don't generate traffic, users do. Content sits there taking up disk 
space until a user comes to grab it. (Not quite the case with data miners such 
as Google, but you get the idea.) 2) Users would not generate traffic unless 
there were content they want to access. Whether that is web pages, commerce 
pages such as Amazon or ebay, streams, or peer-to-peer game traffic, if there's 
nothing interesting, there's nothing happening. So both sides have an equal 
claim to it's all your fault and one seeking to punish the other is 
completely moronic.

Traffic interchange is good. Period. It puts the users closer to the content 
and the content closer to the user and everyone wins. So I never once 
understood why everyone was all fired up about ratios. It just never made any 
sense to me from the get-go. To have government get into this will certainly 
not help the problem, it will just make it a hundred times worse. Remember the 
old saying that the eight most terrifying words in the English language are, 
I'm from the government. I'm here to help. and boy will they try to help. 
You'll be lucky if you as a company can keep still your doors open after they 
get done helping you.

Anyhow, just my two bits.

-Wayne

---
Wayne Bouchard
w...@typo.org
Network Dude
http://www.typo.org/~web/

Re: net neutrality and peering wars continue

[Siegel, David]

Well, with net flow Analytics, it's not really the case that we don't have a 
way of evaluating the relative burdens.  Every major net flow Analytics vendor 
is implementing some type of distance measurement capability so that each party 
can calculate not only how much traffic they carry for each peer, but how far.

Dave

--
520.229.7627 cell


On Jun 19, 2013, at 8:23 PM, Benson Schliesser bens...@queuefull.net wrote:

 
 On 2013-06-19 8:46 PM, Leo Bicknell wrote:
 
 That was a great argument in 1993, and was in fact largely true in system 
 that existed at that time.  However today what you describe no longer really 
 makes any sense.
 
 While it is technically true that the protocols favor asymmetric routing, 
 your theory is based on the idea that a content site exists in one location, 
 and does not want to optimize the user experience.
 ...
 
 A much better business arrangement would be to tie a sliding fee to the 
 ratio.  Peering up to 2:1 is free.  Up to 4:1 is $0.50/meg, up to 6:1 is 
 $1.00/meg, up to 10:1 is $1.50 a meg.  Eyeball network gets to recover their 
 long haul transport costs, it's cheaper to the CDN than buying transit,
 
 Agreed that CDN, traffic steering, etc, changes the impact of routing 
 protocols. But I think you made my point. The sending peer (or their 
 customer) has more control over cost. And we don't really have a good proxy 
 for evaluating relative burdens.
 
 That's not to suggest that peering disputes are really about technical 
 capabilities. Nor fairness, even...
 
 Cheers,
 -Benson
 
 
 

RE: Inventory and workflow management systems

[Siegel, David]

Off the shelf stuff?  There are lots of options, but it seems like the general 
opinion of the IT groups I've worked with is that it's just as much work to 
customize and integrate them as it is to write from scratch so we tend to get 
further way from COTS all the time.

You should take a look at Metasolv (now part of Oracle, I believe).  We used it 
in one of our PL regions for some regional products for over a decade and I 
was always impressed by how quickly they could roll out new workflows and 
products in it since much of the customization work could be done by a business 
process architect rather than having to be coded by a developer.

We've also used Granite, which is from Telcordia, but I don't have enough 
direct or indirect experience to have an opinion on whether or not it's any 
good.

Dave


-Original Message-
From: vijay gill [mailto:vg...@vijaygill.com] 
Sent: Sunday, May 19, 2013 11:22 AM
To: NANOG list
Subject: Re: Inventory and workflow management systems

Resurrecting this thread. Anyone?
What software solution do people use for inventory management for things like 
riser/conduit drawdown, fiber inventory, physical topology store, CLR/DLR, 
x-connect, contracts, port inventory, etc.
Any experiences in integrating workflow into those packages for work orders, 
modeling, drawdown levels, etc.



On Fri, Apr 4, 2008 at 2:16 PM, vijay gill vg...@vijaygill.com wrote:

 What software solution do people use for inventory management for 
 things like riser/conduit drawdown, fiber inventory, physical topology 
 store, CLR/DLR, x-connect, contracts, port inventory, etc.
 Any experiences in integrating workflow into those packages for work 
 orders, modeling, drawdown levels, etc.

 /vijay

RE: Fiber cut in SF Bay Area?

[Siegel, David]

The information that I have is consistent with that...the cut appears to be 
vandalism related.  

Splicing has been under way for several hours and is expected to be completed 
within the next 30 minutes.

Dave


-Original Message-
From: Zaid Ali Kahn [mailto:z...@zaidali.com] 
Sent: Tuesday, April 16, 2013 11:27 AM
To: Ravi Pina
Cc: nanog@nanog.org
Subject: Re: Fiber cut in SF Bay Area?

Level3 is also impacted. This cut seems to be vandalism but only heard this 
from one source.

Zaid

Sent from my iPhone

On Apr 16, 2013, at 12:51 PM, Ravi Pina r...@cow.org wrote:

 Our Zayo provided ETR is 11:00 - 11:30 PDT.
 
 XO is one of the impacted providers as well.
 
 -r
 
 On Tue, Apr 16, 2013 at 08:55:56AM -0700, Raul Rodriguez wrote:
 Lost a Zayo circuit from Palo Alto to Los Angeles. ETR was given as 11AM PDT.
 
 -RR
 

RE: GlobalCrossing looking glass problem

[Siegel, David]

The route-server is accessible again.  Sorry for the inconvenience.

Dave


-Original Message-
From: Eduardo Schoedler [mailto:lis...@esds.com.br] 
Sent: Monday, April 01, 2013 12:07 PM
To: NANOG
Subject: GlobalCrossing looking glass problem

Hi all,

Someone is getting access to the GlobalCrossing the looking glass?

# telnet route-server.gblx.net
Trying 67.17.81.28...
telnet: connect to address 67.17.81.28: Connection refused

Thanks.

-- 
Eduardo Schoedler

RE: Quantifying the value of customer support

[Siegel, David]

There is no such thing as a generic business case that can be applied across 
all companies in an industry.  Every business is unique in its product 
definition and organization structure, but each question is also unique and 
therefore the analysis must be done every time.

The way to begin is to ask this manager what he believes the possible outcomes 
are (downsize your group, eliminate your group, re-define your group, etc.) and 
then work with each of the key stakeholders that you have to estimate the 
impact of those outcomes.  For example, if 1st line operations indicates that 
eliminating your group would result in decreased customer satisfaction and 
missed SLA's, ask them to quantify it as much as possible and go to take the 
numbers back to your business people to have them estimate the impact on 
revenue.

The analysis should be constructed and presented in standard finance terms 
(like NPV) so I would suggest that you make friends with someone in finance to 
assist you with the preparation.  You can also take a short two-day course like 
this 
http://executive.mit.edu/openenrollment/program/fundamentals_of_finance_for_the_technical_executive/16
 that will teach you how to build up these analysis yourself (I have taken the 
one referenced and I recommend it to all managers with budget responsibility).

The outcome from these discussions often has surprising but positive outcomes 
for everyone...maintaining the status quo is not always the best possible 
outcome despite the biases we usually have when we begin the analysis.  :-)  If 
you work closely with all of your stakeholders, everyone will learn and benefit 
from the experience.

Dave

-Original Message-
From: Kasper Adel [mailto:karim.a...@gmail.com] 
Sent: Thursday, February 14, 2013 2:16 PM
To: Andrew Latham
Cc: NANOG list
Subject: Re: Quantifying the value of customer support

I used to think that these kind of situations take place when a manager was 
never an engineer so he does not understand how things work but i was surprised 
when i faced these from managers with an intense engineering career so i gave 
up on trying to give conceptual excuses and want to just give them the dump 
tables and numbers that they are looking for.

Kim

On Thursday, February 14, 2013, Andrew Latham wrote:

 On Thu, Feb 14, 2013 at 3:52 PM, Kasper Adel 
 karim.a...@gmail.comjavascript:;
 wrote:
  Hello,
 
  We are a 2nd level of escalation in a service provider, trying to 
  put a $ value on the support we give to our NOC and other 
  implementation teams, when they email us about problems they face. 
  But we are merely bits and bytes engineers that cant quantify and 
  justify the value of what we do to the management team. I guess 
  these smart suits want to see an excel sheet with a table of how 
  much they save or gain by the support we do. We
 respond
  to technical questions and simulate problems in a lab.
 
  Can anyone help me with an idea or any material i can reuse? Templates?
 Has
  any one been in a similar situation.
 
  Thanks
  Kim

 Kasper/Karim/Kim

 Your job is customer retention.  Your value is maintaining all company 
 income.  Write the yearly revenue on a piece of paper and hand it to 
 them.


 --
 ~ Andrew lathama Latham lath...@gmail.com javascript:; 
 http://lathama.net ~

RE: Level3 worldwide emergency upgrade?

[Siegel, David]

I remember being glued to my workstation for 10 straight hours due to an OSPF 
bug that took down the whole of net99's network.

I was pretty proud of our size at the time...about 30Mbps at peak.  Times are 
different and so are expectations.  :-)

Dave


-Original Message-
From: Brett Watson [mailto:br...@the-watsons.org] 
Sent: Wednesday, February 06, 2013 6:07 PM
To: nanog@nanog.org
Subject: Re: Level3 worldwide emergency upgrade?

Hell, we used to not have to bother notifying customers of anything, we just 
fixed the problem. Reminds me a of a story I've probably shared on the past. 

1995, IETF in Dallas. The big ISP I worked for at the time got tripped up on 
a 24-day IS-IS timer bug (maybe all of them at the time did, I don't recall)  
where all adjacencies reset at once. That's like, entire network down. Working 
with our engineering team in the *terminal* lab mind you, and Ravi Chandra 
(then at Cisco) we reloaded the entire network of routers with new code from 
Cisco once they'd fixed the bug. I seem to remember this being my first 
exposure to Tony Li's infamous line, ... Confidence Level: boots in the lab.

Good times.

-b


On Feb 6, 2013, at 5:41 PM, Brandt, Ralph wrote:

 David. I am on an evening shift and am just now reading this thread.   
 
 I was almost tempted to write an explanation that would have had 
 identical content with yours based simply on Level3 doing something 
 and keeping the information close.
 
 Responsible Vendors do not try to hide what is being done unless it is 
 an Op Sec issue and I have never seen Level3 act with less than 
 responsibility so it had to be Op Sec.
 
 When it is that, it is best if the remainder of us sit quietly on the 
 sidelines.
 
 Ralph Brandt
 
 
 -Original Message-
 From: Siegel, David [mailto:david.sie...@level3.com]
 Sent: Wednesday, February 06, 2013 12:01 PM
 To: 'Ray Wong'; nanog@nanog.org
 Subject: RE: Level3 worldwide emergency upgrade?
 
 Hi Ray,
 
 This topic reminds me of yesterday's discussion in the conference 
 around getting some BCOP's drafted.  it would be useful to confirm my 
 own view of the BCOP around communicating security issues.  My 
 understanding for the best practice is to limit knowledge distribution 
 of security related problems both before and after the patches are 
 deployed.  You limit knowledge before the patch is deployed to prevent 
 yourself from being exploited, but you also limit knowledge afterwards 
 in order to limit potential damage to others (customers, 
 competitors...the Internet at large).  You also do not want to 
 announce that you will be deploying a security patch until you have a 
 fix in hand and know when you will deploy it (typically, next 
 available maintenance window unless the cat is out of the bag and danger is 
 real and imminent).
 
 As a service provider, you should stay on top of security alerts from 
 your vendors so that you can make your own decision about what action 
 is required.  I would not recommend relying on service provider 
 maintenance bulletins or public operations mailing lists for obtaining 
 this type of information.  There is some information that can cause 
 more harm than good if it is distributed in the wrong way and 
 information relating to security vulnerabilities definitely falls into that 
 category.
 
 Dave
 
 -Original Message-
 From: Ray Wong [mailto:r...@rayw.net]
 Sent: Wednesday, February 06, 2013 9:16 AM
 To: nanog@nanog.org
 Subject: Re: Level3 worldwide emergency upgrade?
 
 
 
 OK, having had that first cup of coffee, I can say perhaps the main 
 reason I was wondering is I've gotten used to Level3 always being on 
 top of things (and admittedly, rarely communicating). They've reached 
 the top by often being a black box of reliability, so it's (perhaps
 unrealistically) surprising to see them caught by surprise. Anything 
 that pushes them into scramble mode causes me to lose a little sleep 
 anyway. The alternative to what they did seems likely for at least a 
 few providers who'll NOT manage to fix things in time, so I may well 
 be looking at longer outages from other providers, and need to issue 
 guidance to others on what to do if/when other links go down for 
 periods long enough that all the cost-bounding monitoring alarms start 
 to scream even louder.
 
 I was also grumpy at myself for having not noticed advance 
 communication, which I still don't seem to have, though since I 
 outsourced my email to bigG, I've noticed I'm more likely to miss 
 things. Perhaps giving up maintaining that massive set of procmail 
 rules has cost me a bit more edge.
 
 Related, of course, just because you design/run your network to 
 tolerate some issues doesn't mean you can also budget to be in support 
 contract as well. :) Knowing more about the exploit/fix might mean 
 trying to find a way to get free upgrades to some kit to prevent more 
 localized attacks to other types of gear, as well, though in this case 
 it's all

RE: Level3 worldwide emergency upgrade?

[Siegel, David]

Hi Ray,

This topic reminds me of yesterday's discussion in the conference around 
getting some BCOP's drafted.  it would be useful to confirm my own view of the 
BCOP around communicating security issues.  My understanding for the best 
practice is to limit knowledge distribution of security related problems both 
before and after the patches are deployed.  You limit knowledge before the 
patch is deployed to prevent yourself from being exploited, but you also limit 
knowledge afterwards in order to limit potential damage to others (customers, 
competitors...the Internet at large).  You also do not want to announce that 
you will be deploying a security patch until you have a fix in hand and know 
when you will deploy it (typically, next available maintenance window unless 
the cat is out of the bag and danger is real and imminent).

As a service provider, you should stay on top of security alerts from your 
vendors so that you can make your own decision about what action is required.  
I would not recommend relying on service provider maintenance bulletins or 
public operations mailing lists for obtaining this type of information.  There 
is some information that can cause more harm than good if it is distributed in 
the wrong way and information relating to security vulnerabilities definitely 
falls into that category.

Dave

-Original Message-
From: Ray Wong [mailto:r...@rayw.net] 
Sent: Wednesday, February 06, 2013 9:16 AM
To: nanog@nanog.org
Subject: Re: Level3 worldwide emergency upgrade?



OK, having had that first cup of coffee, I can say perhaps the main reason I 
was wondering is I've gotten used to Level3 always being on top of things (and 
admittedly, rarely communicating). They've reached the top by often being a 
black box of reliability, so it's (perhaps
unrealistically) surprising to see them caught by surprise. Anything that 
pushes them into scramble mode causes me to lose a little sleep anyway. The 
alternative to what they did seems likely for at least a few providers who'll 
NOT manage to fix things in time, so I may well be looking at longer outages 
from other providers, and need to issue guidance to others on what to do 
if/when other links go down for periods long enough that all the cost-bounding 
monitoring alarms start to scream even louder.

I was also grumpy at myself for having not noticed advance communication, which 
I still don't seem to have, though since I outsourced my email to bigG, I've 
noticed I'm more likely to miss things. Perhaps giving up maintaining that 
massive set of procmail rules has cost me a bit more edge.

Related, of course, just because you design/run your network to tolerate some 
issues doesn't mean you can also budget to be in support contract as well. :) 
Knowing more about the exploit/fix might mean trying to find a way to get free 
upgrades to some kit to prevent more localized attacks to other types of gear, 
as well, though in this case it's all about Juniper PR839412 then, so vendor 
specific, it seems?

There are probably more reasons to wish for more info, too. There's still more 
of them (exploiters/attackers) than there are those of us trying to keep things 
running smoothly and transparently, so anything that smells of OMG new exploit 
found! also triggers my desire to share information. The network bad guys 
share information far more quickly and effectively than we do, it often seems.

-R

RE: looking glass for Level 3

[Siegel, David]

Ben,

Our looking glass platform is indeed back online and now supports IPv6 
traceroutes, pings and BGP lookups in the interface (although the web site 
itself is still only available via IPv4).

If you encounter any problems, oddities, or suggestions, please feel free to 
contact me off list and I'll work with engineering to address them.

Again, I apologize for the inconvenience.  Please resume enjoying the use of 
this free tool!  :)

Dave



From: Ben Bartsch [mailto:uwcable...@gmail.com]
Sent: Tuesday, January 15, 2013 8:14 AM
To: Siegel, David
Cc: N. Max Pierson; Cameron Daniel; nanog@nanog.org
Subject: Re: looking glass for Level 3

http://lg.level3.net/ is online from Baton Rouge, LA.  Any official word from 
Level3?

-bb
On Wed, Jan 2, 2013 at 9:27 AM, Siegel, David 
dave.sie...@level3.commailto:dave.sie...@level3.com wrote:
Hi Folks,

The site is offline as a result of some security issues that were discovered.  
As soon as we've got it patched we'll put it back online.

Sorry for any inconvenience this may be causing.

Dave


-Original Message-
From: N. Max Pierson 
[mailto:nmaxpier...@gmail.commailto:nmaxpier...@gmail.com]
Sent: Friday, December 28, 2012 11:06 AM
To: Cameron Daniel
Cc: nanog@nanog.orgmailto:nanog@nanog.org
Subject: Re: looking glass for Level 3

Same here. http://lg.level3.net has been down for over a week for me. I know 
someone in operations I can open a ticket with.

On Fri, Dec 28, 2012 at 5:18 AM, Cameron Daniel 
cdan...@nurve.com.aumailto:cdan...@nurve.com.auwrote:

 I've had issues getting to it for a week or so. Their NOC was
 unresponsive when queried.


 On 2012-12-28 8:23 pm, Peter Ehiwe wrote:

 I normally use the 3rd one you mentioned but they seem to be down at
 the moment.

 Rgds Peter,
 Sent from my Asus  Transformer Pad
 On Dec 28, 2012 1:51 AM, Tassos Chatzithomaoglou 
 ach...@forthnetgroup.grmailto:ach...@forthnetgroup.gr
 wrote:

  Anyone have any looking glass for Level 3?

 The following seem not to be working

 http://www.level3.com/**LookingGlass/http://www.level3.com/LookingG
 lass/ http://lg.level3.net/bgp/bgp.**cgi
 http://lg.level3.net/bgp/bgp.cgi
 http://lookingglass.level3.**net/ http://lookingglass.level3.net/

 --
 Tassos

RE: Issues with level3?

[Siegel, David]

Hi David,

I'm sorry you've had so many poor experiences with Level 3 recently, but I 
assure you that we have acknowledged the problem and are actively working on it 
at present.

Of general operations interest,

I just saw an event notification that matches the description of the problem 
and our NOC, engineering team and vendor are working together to solve the 
problem.  If you are a customer and believe you are impacted, you can reference 
event case ID: 6237890 as potentially being the related case.

Dave

-Original Message-
From: David Miller [mailto:dmil...@tiggee.com] 
Sent: Tuesday, January 15, 2013 10:38 AM
To: nanog@nanog.org
Cc: networkoperati...@etsms.com
Subject: Re: Issues with level3?

On 01/15/2013 11:12 AM, Network Operations wrote:
 Anyone seeing any issues with level3?  We can connect to every other IP in 
 our Class C.  When tracerouting to individual IP's, (x.x.x.50/51/52/53) we 
 get a drop at ge-4-16.car2.Washington1.Level3.net [4.59.146.53] for 50, but 
 51 is fine, drop for 52, 53 is fine.

 Thanks.


It is not just you.  We are seeing issue with that Level3 router/site as well.

I would report it to Level3, but I don't see any need to add to my already 
extensive collection of one line Level3 support responses saying All is well.  
Nothing to see here.  All is well.

My guess would be that your up/down for individual IPs is a result of your 
testing methodology. That Level3 router/site appears to be dropping some 
packets to all IPs that I tested before dropping my conn there.

Our response to the nearly constant Level3 issues of the past 12/18 months has 
been terminate them.  The washington1.level3 site was unfortunately the last on 
my list of DCs.

--
-__
David Miller
dmil...@tiggee.com

NANOG education committee is seeking instructors

[Siegel, David]

NANOG community,

As those of you who were able to attend one of the last couple of NANOG 
membership meetings know, the education committee is exploring the idea of 
adding a professional instructor-led class to the NANOG conference, not unlike 
you might find with the tutorials available at Interop or similar conferences.

If you are an experience instructor in this field or you know of an instructor, 
please email educat...@nanog.orgmailto:educat...@nanog.org with contact 
information.

Thank you for the assistance!

Dave

David Siegel
p: 720.888.0953
c: 520.229.7627
e: dave.sie...@level3.commailto:dave.sie...@level3.com

RE: looking glass for Level 3

[Siegel, David]

Hi Folks,

The site is offline as a result of some security issues that were discovered.  
As soon as we've got it patched we'll put it back online.

Sorry for any inconvenience this may be causing.

Dave


-Original Message-
From: N. Max Pierson [mailto:nmaxpier...@gmail.com] 
Sent: Friday, December 28, 2012 11:06 AM
To: Cameron Daniel
Cc: nanog@nanog.org
Subject: Re: looking glass for Level 3

Same here. http://lg.level3.net has been down for over a week for me. I know 
someone in operations I can open a ticket with.

On Fri, Dec 28, 2012 at 5:18 AM, Cameron Daniel cdan...@nurve.com.auwrote:

 I've had issues getting to it for a week or so. Their NOC was 
 unresponsive when queried.


 On 2012-12-28 8:23 pm, Peter Ehiwe wrote:

 I normally use the 3rd one you mentioned but they seem to be down at 
 the moment.

 Rgds Peter,
 Sent from my Asus  Transformer Pad
 On Dec 28, 2012 1:51 AM, Tassos Chatzithomaoglou  
 ach...@forthnetgroup.gr
 wrote:

  Anyone have any looking glass for Level 3?

 The following seem not to be working

 http://www.level3.com/**LookingGlass/http://www.level3.com/LookingG
 lass/ http://lg.level3.net/bgp/bgp.**cgi 
 http://lg.level3.net/bgp/bgp.cgi
 http://lookingglass.level3.**net/ http://lookingglass.level3.net/

 --
 Tassos

RE: Why do some providers require IPv6 /64 PA space to have public whois?

[Siegel, David]

That's a really good point, Patrick.

We've received an interesting analysis from our customers recently where they 
reviewed the accounting on all the services they need in order to peer off 
approximately 1/3rd of their total traffic.

They took their national wavelength cost, local access, colocation at 
carrier-neutral facilities at it came to roughly $.95/mbps.

Although this is considerably less than what they spend on transit, their 
analysis failed to consider depreciation on their capital (routers and other 
hardware), associated warranty costs and the incremental operational overhead 
to operate a large national network.  When all is said and done, they are 
probably spending as much on free peering as they are on transit.  In the 
case of this customer they would have a lower total cost by simply staying 
regional and purchasing transit.

In other cases, peering will only lower your marginal cost if there are 
strategic reasons for building and maintaining that backbone.

Dave


-Original Message-
From: Patrick W. Gilmore [mailto:patr...@ianai.net] 
Sent: Saturday, December 08, 2012 8:23 PM
To: NANOG list
Subject: Re: Why do some providers require IPv6 /64 PA space to have public 
whois?

 So no, it's not true. Costs come from needing to buy bigger routers, 
 bigger waves or fiber to the exchanges, bigger ports on the exchanges, 
 etc.
 
 Peering is a scam.

The vast majority of AS-AS boundaries on the Internet are settlement free 
peering.  I guess that makes the Internet a scam.

As for the costs involved, free is a relative term.  Most people think of 
peering as free because there is zero marginal cost.  Kinda.  Obviously if 
you think of your 10G IX port as a sunk cost, pushing 11 Gbps over it is not 
'free' as you have to upgrade.  But again, most people understand what is meant.

Bigger waves  bigger routers are not due to peering, they are due to customer 
traffic - you know, the thing ISPs sell.  Put another way, this is a Good Thing 
(tm).  Or at least it should be.  Unless, of course, you are trying to convince 
us all that selling too many units of your primary product is somehow bad.

Peering allows you, in most cases, to lower the Cost Of Goods Sold on that 
product.  Again, usually a Good Thing (tm).  Unless you are again trying to 
convince us all that selling at a higher margin (we'll ignore the lower latency 
 better overall experience) is somehow bad.

--
TTFN,
patrick

RE: IPv4 address length technical design

[Siegel, David]

I'll identify myself as the person who asked you the question privately.

Unfortunately, Barry, I still don't see a problem statement in your response.  
It sounds to me as though it really is nothing more than an interesting thought 
experiment, and there's nothing wrong with that at all as long as we all 
acknowledge the purpose of the discussion.  :-)

Dave


-Original Message-
From: Barry Shein [mailto:b...@world.std.com] 
Sent: Friday, October 05, 2012 6:25 PM
To: nanog@nanog.org
Cc: Barry Shein
Subject: RE: IPv4 address length technical design


  While this is an interesting thought experiment, what problem are   you 
  trying to solve with this proposal?

(asked privately but it seems worthwhile answering publicly, bcc'd, you can id 
yourself if you like.)

Look, as I said in the original message I was asked to speak to a group of 
young hackers at the HackerSpace in Singapore.

I wanted to be interesting and thought-provoking, make them think through how 
this stuff works for an hour or two, encourage them to poke holes in it, etc. 
It was one of the audience who pointed out the potential MTU problem.

What problem does it solve, potentially?

0. Despite fears expressed herein I am not single-handedly planning to convert 
the worldwide internet to this over the weekend. I'm going to need some help :-)

1. It eliminates the need for DNS in its generally used form.

Sure, we've overloaded DNS with other functions from SPF -- in fact it was Meng 
Weng Wong, inventor of SPF, who graciously invited me to speak -- to whatever. 
But that's begging the point, there's nothing interesting here about 
distributed, lightweight databases other than eliminating one. Keep the DNS 
protocol per se for those things if you like.

But given this you won't need to translate between host names and addresses 
which is really what DNS was invented to do.

2. It makes addresses more transparent to humans, particularly when you 
consider ipv6 addresses as typically displayed (hex.) Is this an important 
goal? Not sure, but it's certainly true.

3. It's a transfinite space.

That just means that like Dewey Decimal etc it can be arbitrarily expanded, you 
can add more levels or even stick levels in between plus or minus some rules 
regarding SLDs/TLDs, and other rules which might or might not be imposed (see 
#4).

But its total address space is as large as you allow a payload, there is 
nothing inherent in the scheme that limits the addressing other than the 
permutation of all acceptable Unicode glyphs I guess. But since one can also 
have numeric parts and the set of integers is infinite (that's tongue-in-cheek, 
somewhat.)

4. Also, because it's transfinite it's arbitrarily segmentable.

Again, that just means you can impose any meaning you like on any substring or 
set of substrings. So for example host.gTLD is generally taken to be something 
of some significance, or host.co.ccTLD, and that sort of idea can be applied as 
needed, or not at all.

5. Bits is bits.

I don't know how to say that more clearly.

An ipv6 address is a string of 128 bits with some segmentation implications 
(net part, host part.)

A host name is a string of bits of varying length. But it's still just ones and 
zeros, an integer, however you want to read it.

The discussion I was responding to on NANOG involved how we got here and where 
might we be going.

I brought up an idea I'd worked out somewhat and have even presented in a small 
but public forum as being a possible future to consider further.

Now you can go back to your regularly scheduled Jim Fleming guffawing.

-- 
-Barry Shein

The World  | b...@theworld.com   | http://www.TheWorld.com
Purveyors to the Trade | Voice: 800-THE-WRLD| Dial-Up: US, PR, Canada
Software Tool  Die| Public Access Internet | SINCE 1989 *oo*

RE: IPv4 address length technical design

[Siegel, David]

Wouldn't that implicate the routing system to have, in essence, one routing 
entry for every host on the network?

That would be the moral equivalent to just dropping down to a global ethernet 
fabric to replace IP and using mac addresses for routing.  I'll give you one 
guess as to how well that would work.

Dave




-Original Message-
From: Barry Shein [mailto:b...@world.std.com] 
Sent: Thursday, October 04, 2012 5:36 PM
To: nanog@nanog.org
Subject: Re: IPv4 address length technical design


In Singapore in June 2011 I gave a talk at HackerSpaceSG about just doing away 
with IP addresses entirely, and DNS.

Why not just use host names directly as addresses? Bits is bits, FQDNs are 
integers because, um, bits is bits. They're even structured so you can route on 
the network portion etc.

Routers themselves could hash them into some more efficient form for table 
management but that wouldn't be externally visible. I did suggest a standard 
for such hashing just to help with debugging etc but it'd only be a suggestion 
or perhaps common display format.

About the only obvious objection, other than vague handwaves about compute 
efficiency, is it would potentially make packets a lot longer in the worst case 
scenario, longer than common MTUs tho not much longer unless we also allow a 
lengthening of host name max, 1024 right now I believe? So 2K max for src/dest 
and whatever other overhead payload you need, not unthinkable.

OTOH, it just does away with DNS entirely which is some sort of savings.

There are obviously some more details required, this email is not a replacement 
for a set of RFCs!

-- 
-Barry Shein

The World  | b...@theworld.com   | http://www.TheWorld.com
Purveyors to the Trade | Voice: 800-THE-WRLD| Dial-Up: US, PR, Canada
Software Tool  Die| Public Access Internet | SINCE 1989 *oo*

RE: Announcing APNIC IP's in ARIN region

[Siegel, David]

The only problem I've ever run into is with IP geo-location providers using the 
country of origin of the original assignments to determine the locale of the 
IP.  Major CDN providers and content owners then use these geo-location 
providers to provide geography specific content or for content localization.

A problem we saw at GC when using our ARIN space in APAC (which I realize is 
the inverse of your situation) is that our enterprise customers often got 
redirected to a cloud server in the United States rather than in their 
originating country, and this was in spite of their block being SWIP'd out to 
them in that country.

It's conceivable that you could have some sort of similar problem depending on 
the nature of your project and how you are planning to use their IP's.

Dave


-Original Message-
From: Brandon Wade [mailto:brandonw...@icastcenter.com] 
Sent: Thursday, September 20, 2012 5:58 PM
To: nanog@nanog.org
Subject: Announcing APNIC IP's in ARIN region

Hello,

I was wondering if there are any problems originating APNIC IP's in the ARIN 
region through transit providers? I have a Singapore-based prospect who would 
like to do business with us, but I'm not sure if I'll run into problems 
originating their IP's in the US - which were assigned to them from APNIC.

Best regards,
Brandon Wade
iCastCenter.com

RE: Level3 (3356/3549) changes routing policy

[Siegel, David]

Thanks David, you hit the nail on the head on both points. 

Level 3 made the routing policy change last November, roughly 6 weeks after the 
acquisition of Global Crossing.

Dave


-Original Message-
From: David Reader [mailto:david.rea...@zeninternet.co.uk] 
Sent: Thursday, August 02, 2012 6:41 AM
To: nanog@nanog.org
Subject: Re: Level3 (3356/3549) changes routing policy

On Thu, 2 Aug 2012 10:33:38 +0200
Fredy Kuenzler kuenz...@init7.net wrote:

  From my observation Level3 has recently changed their routing policy. 
 It seems that 3356 always prefers customer prefixes of 3549, 
 regardless of the AS path length. Example (seen from 3356):
 
 3549_13030_[Customer1]_[Customer2]
 
 is preferred over
 
 2914_[Customer1]_[Customer2]
 
 Considering that both 2914 and 3549 are peers of 3356, and 13030 is a 
 customer of 3549, 3356 seems to give higher local-pref on the longer 
 AS-path, likely to increase traffic and revenue of their sister network 3549.

Hi Fredy,

Level 3 owns both 3356 and 3549.
They're simply preferring to have their customers pay them, rather than a 3rd 
party.

I don't think it's suprising at all that they're doing it. If, as you think, 
it's only happened recently then what is suprising is that it didn't happen 
sooner IMO.

d.

RE: MPLS L2VPN monitoring

[Siegel, David]

We deploy NIDs to the customer premise.  You just can't get enough alarm data 
be looking only at your router/switch on your side of an Ethernet NNI to give 
you a proper indication of whether the service is functional, and it also 
happens to be quite handy to have when a performance test/verification is 
required.

There are a variety of vendors out there to choose from...we have quite a lot 
of Tellabs and Accedian out in the field.

I had hoped that last mile vendors would have been providing NIDs smartjack 
style by now in a fairly ubiquitous fashion, but alas none of them have stepped 
up to the plate so we're still putting them out there on our own dime.

Dave

-Original Message-
From: Peter Ehiwe [mailto:petereh...@gmail.com] 
Sent: Tuesday, July 17, 2012 4:14 AM
To: North American Network Operators' Group
Subject: MPLS L2VPN monitoring

Hello ,

For those who provide l2vpn services to customers over MPLS , what kind of 
tools do you use for monitoring the circuits  and what kind of values do you 
proactively monitor

I have tools in place to monitor these circuits but i want to know based on 
group members experiences in order to improve my monitoring platform for this 
circuits.

Thanks a lot!