Re: Follow up to previous post regarding SAAVIS

[Leen Besselink]

Keith Medcalf wrote:
>> ... Dont know what web 2.0 is but the new portal is a web based 
>> object management system complete
>> with "recommended" changes and inconsistency lists.
>> We just added prefix allocation check with backend information
>> from PCH (prefix checker tool).
> 
> Web 2.0 is marketroid drivel-speak for a method of continuing to ensure that 
> Web Applications
> are uninspectable and unsecurable.  It is based on doing partial document 
> refreshes using code
> executing within the browser, usually in such a fashion that it modifies the 
> document tree
> directly through foreign (ie, from the net) code execution in the context of 
> the current
> user (or, because of the zillions of holes in those browsers supporting code 
> execution,
> with the priviledges of the OS itself).
> 
> It is highly insecure and cannot be secured by any products currently 
> available.  It is best
> to stay as far as possible from anything claiming that it is Web 2.0.  
> Hallmarks of Web 2.0
> are gratuitous javascript and java applications which cannot be disabled.  
> Enabling any type
> of even minimal security on any web site that is "Web 2.0" buzzword compliant 
> results in the
> display of completely blank pages.  Web 2.0 pages will indirect all 
> hyperlinks and navigation
> through javascript.  Not because it provides anything useful but rather in 
> order to force
> people to enable dangerous crap in their browsers (javascript, java, Flash 
> Virus, &c)
> 
> 

Their are people who do understand how to do these things right.

It's called progressive enhancement. [0] [1] Which means you don't need any 
fancy stuff to be
able to use it or read the content, but if you have support for it, it will add 
extra
convenience-features like search suggestions.

Also in certain ways things are starting to improve for example the HTML5 spec 
has a video-tag
[2] that's the only kinda of useful thing Flash is used for these days. Their 
is SVG and Canvas-
tag in the HTML5-spec as well, which means even less reason to use plugins.

The Chrome browser uses seperate processes with less priviledges to render the 
pages and run
scripts and plugins.

I'm just saying it's not all bad.

[0] http://en.wikipedia.org/wiki/Progressive_enhancement
[1] http://www.alistapart.com/articles/understandingprogressiveenhancement/
[2] Some may say, but their are no codecs specified, but the same is true for 
images, etc. and
I think images did pretty well
[3] http://en.wikipedia.org/wiki/Google_Chrome#Security

RE: Follow up to previous post regarding SAAVIS

[Keith Medcalf]

> ... Dont know what web 2.0 is but the new portal is a web based
> object management system complete
> with "recommended" changes and inconsistency lists.
> We just added prefix allocation check with backend information
> from PCH (prefix checker tool).

Web 2.0 is marketroid drivel-speak for a method of continuing to ensure that 
Web Applications are uninspectable and unsecurable.  It is based on doing 
partial document refreshes using code executing within the browser, usually in 
such a fashion that it modifies the document tree directly through foreign (ie, 
from the net) code execution in the context of the current user (or, because of 
the zillions of holes in those browsers supporting code execution, with the 
priviledges of the OS itself).

It is highly insecure and cannot be secured by any products currently 
available.  It is best to stay as far as possible from anything claiming that 
it is Web 2.0.  Hallmarks of Web 2.0 are gratuitous javascript and java 
applications which cannot be disabled.  Enabling any type of even minimal 
security on any web site that is "Web 2.0" buzzword compliant results in the 
display of completely blank pages.  Web 2.0 pages will indirect all hyperlinks 
and navigation through javascript.  Not because it provides anything useful but 
rather in order to force people to enable dangerous crap in their browsers 
(javascript, java, Flash Virus, &c)


--
()  ascii ribbon campaign against html e-mail
/\  www.asciiribbon.org

Re: Follow up to previous post regarding SAAVIS

[Richard A Steenbergen]

On Thu, Aug 13, 2009 at 12:03:24PM -0400, Joe Provo wrote:
> 
> Experience proves otherwise.  L3's filtergen is a great counter-example,
> where the customer-specific import policy dictates sources to believe
> regardless of what other stuff is in their local mirror.  It happily 
> drops prefixes not matching, so it does "make a real difference" WRT 

No, level3's filtergen follows the exact "search path" rules I described 
previously, which has no real impact for any reasonably sized isp. For 
example, say I describe my as-set and aut-num and routes in altdb, you 
can have level3 restrict the scope of the initial query to 
ALTDB::myasset, and you can have level3 restrict the search path to 
altdb, but what happens when I have a customer in my as-set who 
registered their prefixes in radb? Now you have to open up the scope 
there, and ripe, and arin, and level3, and ntt, and savvis, etc. Now say 
someone comes along and slips an unauthorized route with my origin: 
aut-num into one of these databases. You have no way to prevent that 
from happening, when you run the query on my as-set/aut-num you're going 
to get back the superset of my legit routes + any bogus ones. And this 
is a good thing, because it's a lot less destructive to have bogus 
routes in the system than it is to give someone the ability to override 
legitimate routes with a bogus entry on a "more trusted" db.

> customer filtering.  I'm not familiar with DTAG's tools, but would be 
> shocked if they were less featureful.  For querying other databases, see 
> IRRd's !s syntax, which specifies sources in order.  Also see Net:IRR's 
> $whois->sources() method.  For tools based on these, I would presume 
> it be up to your implementation or policy expression as to how you 
> decide the handling on multiple matches.  When mentioned, usually the 
> first which matches is specified as 'the one', which is why search 
> order matters.  What other purpose does specifying a search order serve?

This is the server side search path I talked about, it has nothing to do
with any specific client implementation nor is a client implementation
practical. See page 34 of:

http://www.irrd.net/irrd-user.pdf

Again you can restrict a global query, but this provides very little
practical benefit. You could dynamically restrict sources per query when
you go to do the !i or !g expansion, but there is no information on what
you should restrict it to, so again no practical benefit. The only thing
Level3 adds that isn't part of the stock query syntax is the top level
scope I mentioned above, ALTDB::AS-MYASSET. To support this recursively
you would have to run multiple full queries for the full records without
server side expansion, which is not practical for anyone with more than 
a few hundred routes.

> If I am running a tool to generate filter lists for my customers, I 
> want to believe my RR, the local RIR, some other RIR that is well run,
> and then maybe my upstream.  Specify that search order and believe 
> the first match.  Job done.  If you have highly clued downstreams, go
> the filtergen route and tune source-believability based on customer,
> or cook up another avenue.  There is nothing inherent in the system to
> prevent this.
...
> 
> Yes, reduced queries and the ability to ignore Bob's Bait and Tackle DB
> if I know it is part of the "piles of junk databases" you posit will 
> exist.  See above.

This doesn't work if your customers have customers. I'm also not aware
of anyone running any "bad" databases, or for that matter any databases
which are of lesser security/quality than the "big boys". Short of what
ripe implements because they are the RIR, there is no real security on
registrations here, so it doesn't much matter if the database is level3
or bob's bait and tackle. And even given what I consider to be an 
excessively large list of irr databases today, from the standpoint of 
keeping good records, I'd be hard pressed to name one on the list who's 
data I should trust any less than say level3's.

> Centralized scales better than distributed?  Quick, call the 80s - we 
> need HOSTS.TXT back.  

A silly argument. In this case, hosts.txt is equivalent to an ISP having
a human manually process an e-mail from a customer, add it to a
prefix-list on a router, and then manually e-mail their upstream or peer
ISPs to have them update the prefix-list, etc. In many cases centralized
(or at least, restricted to some reasonably sized set, obviously nobody
is proposing running the entire Internet on a single server run by a
single entity) has much better security properties. As far as scale
goes, you're talking about a pretty simple database of pretty simple
objects here. There is probably more overhead that goes into maintaining
the distributed nature of the db than there is actual work generating
prefix-lists. :)

> > There is no reason that this process needs to be
> > politicized, or cost anyone any money to use. 
> 
> Anytime you go down the road of advocati

Re: Follow up to previous post regarding SAAVIS

[Kanagaraj]

> The other problem is that when a SP has a customer who "can't figure it 
> out", a typical course of action is to just "register the route for 
> them" rather than try to explain it to them. Unfortunately, the same 
> thing as above happens here, you end up with a big pile of people who 
> register a big pile of routes in a big pile of random databases, often 
> times completely unnecessarily, and they'll never be removed either.
>
> The biggest problem with the entire system is the way that data gets 
> into it, and the lack of incentive for people to ever remove data from 
> it. But as a mechanism to allow the routes which need to be allowed, and 
> mostly prevent accidental leaks, it works.
>   
Agreed. Wish most providers take the extra mile effort to advise and
facilitate customers on the process. The redundant entries are annoying :-)

Re: Follow up to previous post regarding SAAVIS

[Christopher Morrow]

On Thu, Aug 13, 2009 at 11:27 PM, Andrew Euell wrote:
> "the pccw lesson, which is also the
> turk-telecom lesson"
>
> tangent here: what was the pccw and turk-telecom thing?
> is the turk telco thing the Youtube fiasco?

pccw + pktelecom == youtube incident

turk-telecom leaked covad + a bunch of other things ... 4 years back
at either the time of NANOG or a US Holiday (Christmas??)

Both incidents were: "Providers who didn't filter their customer(s)"

-chris

> On Wed, Aug 12, 2009 at 10:09 AM, Christopher Morrow
>  wrote:
>>
>> On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver
>> wrote:
>> > Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise
>> > routes for whatever IP addresses they want?
>>
>> sadly savvis didn't learn the pccw lesson, which is also the
>> turk-telecom lesson which is also the as7007 lesson which is... fairly
>> sad really in 2009.
>>
>> for the sake of $diety put a prefix-filter on your customer bgp
>> sessions, it ain't hard!
>>
>> -chris
>>
>> > route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768
>> >  2905 701 3561 13768
>> >  1221 4637 3561 13768
>> >  3549 3561 13768
>> >  3277 3267 174 3561 13768
>> >  6539 3561 13768
>> >  16150 3549 3561 13768
>> >  701 3561 13768
>> >  3267 174 3561 13768
>> >  6453 3561 13768
>> >  3582 3701 3356 3561 13768
>> >
>> > This is probably a fairly major problem...
>> >
>> > -Drew
>> >
>> >
>>
>
>
>
> --
> Andrew Euell
> andyzweb [at] gmail [dot] com
>

Re: Follow up to previous post regarding SAAVIS

[Andrew Euell]

"the pccw lesson, which is also the
turk-telecom lesson"

tangent here: what was the pccw and turk-telecom thing?
is the turk telco thing the Youtube fiasco?

On Wed, Aug 12, 2009 at 10:09 AM, Christopher Morrow <
morrowc.li...@gmail.com> wrote:

> On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver
> wrote:
> > Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise
> routes for whatever IP addresses they want?
>
> sadly savvis didn't learn the pccw lesson, which is also the
> turk-telecom lesson which is also the as7007 lesson which is... fairly
> sad really in 2009.
>
> for the sake of $diety put a prefix-filter on your customer bgp
> sessions, it ain't hard!
>
> -chris
>
> > route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768
> >  2905 701 3561 13768
> >  1221 4637 3561 13768
> >  3549 3561 13768
> >  3277 3267 174 3561 13768
> >  6539 3561 13768
> >  16150 3549 3561 13768
> >  701 3561 13768
> >  3267 174 3561 13768
> >  6453 3561 13768
> >  3582 3701 3356 3561 13768
> >
> > This is probably a fairly major problem...
> >
> > -Drew
> >
> >
>
>


-- 
Andrew Euell
andyzweb [at] gmail [dot] com

Re: Follow up to previous post regarding SAAVIS

[Joe Provo]

On Wed, Aug 12, 2009 at 10:03:23PM -0500, Richard A Steenbergen wrote:
> On Wed, Aug 12, 2009 at 10:06:49PM -0400, Joe Provo wrote:
> > On Wed, Aug 12, 2009 at 08:16:38PM -0500, Richard A Steenbergen wrote:
> > [snip]
> > > Unfortunately the distributed nature of the databases is one of the 
> > > biggest problems with the IRR system. Anyone can run an irrd, there is 
> > 
> > You misspelled "largest strength".  FOlks get to choose which registries
> > to believe in what order, not required to have a single [politicized]
> > entity.
> 
> Well, actually, no. I'm not aware of any mechanism under which you can
> effectively choose who to believe and in what order, nor do I think that
> it would make any real difference in the long run even if you could.

Experience proves otherwise.  L3's filtergen is a great counter-example,
where the customer-specific import policy dictates sources to believe
regardless of what other stuff is in their local mirror.  It happily 
drops prefixes not matching, so it does "make a real difference" WRT 
customer filtering.  I'm not familiar with DTAG's tools, but would be 
shocked if they were less featureful.  For querying other databases, see 
IRRd's !s syntax, which specifies sources in order.  Also see Net:IRR's 
$whois->sources() method.  For tools based on these, I would presume 
it be up to your implementation or policy expression as to how you 
decide the handling on multiple matches.  When mentioned, usually the 
first which matches is specified as 'the one', which is why search 
order matters.  What other purpose does specifying a search order serve?
 
> IRR database mirroring is like being a tier 1, you have to peer with
> every other database out there in order to obtain a full view. That

Funny, subject of the thread is filtering customers.  I don't need to
take the same point of view of the RADB ("RADB's mission is to mirror
all component databases so as to provide the most complete view possible 
of the entire IRR.") to do that, just a set of databases in which my 
customers can be/must be registered.  Once upon a time, 3561 had a 
highly automated and effective way of doing this based in part upon 
running their own DB and that being the default/requirement for 
"non-advanced" customers.

[snip]
> Basically the only thing you have control over are the list of IRR
> databases which are searched, but the results which are returned are a
> superset of all databases which you selected to search. You don't get to
> say "only listen to results from LEVEL3's db for this object unless they
> don't have results there, in which case you listen to SAVVIS" or
> anything like that. 

If I am running a tool to generate filter lists for my customers, I 
want to believe my RR, the local RIR, some other RIR that is well run,
and then maybe my upstream.  Specify that search order and believe 
the first match.  Job done.  If you have highly clued downstreams, go
the filtergen route and tune source-believability based on customer,
or cook up another avenue.  There is nothing inherent in the system to
prevent this.

[snip]
> from all the sources is always importing correctly, etc. And even after
> you do all this, what does being able to pick a data source order buy
> you anyways? Do you think you win something by preferring say RADB over
> LEVEL3 over SAVVIS over ARIN over RIPE over...? 

Yes, reduced queries and the ability to ignore Bob's Bait and Tackle DB
if I know it is part of the "piles of junk databases" you posit will 
exist.  See above.

> Where do you draw the line on who's data you look at, and why do we 
> need yet another system where people are left to make a judgement call 
> over who's data they should trust?

I'm not sure who has a better viewpoint on what my customers can/should
emit cross my network than me, so yeah I should make the call regarding 
what databases my customers need to be in for my business.  Obviously 
for multihomed or well-meshed customers you have to approach that sanely
or you'll get serious pushback from folks registered elsewhere. In the
earlier 3561 days, I had to get them to eat non-MCI sources for us so
I wouldn't be doubly & trebly registered.  

Assuming a perfect global datastore will fail.  It doesn't exist now, 
and migrating there to such a mythical beast is impossible.  Run your 
corner as cleanly as you can and apply as much error correction as 
possible to the outside world.  Since this topic is about running one's 
corner cleanly, taking the input from known-garbage is a bad plan. 

> Personally I'm of the belief that every ISP running their own IRR db is
> a very bad idea, which is why I have chosen not to run one myself. 
> To quote Vijay, it doesn't scale. The last thing the already very broken
> IRR system needs is more crap data by more random people spread out over
> more databases, and the majority of the current db's probably need to be
> shut down too.

Centralized scales better than distributed?  Quick, call the 80s - w

Re: Follow up to previous post regarding SAAVIS

[Joe Maimon]

Richard A Steenbergen wrote:

On Wed, Aug 12, 2009 at 04:57:07PM -0400, Jared Mauch wrote:
I've come to the conclusion that if someone put a nice web2.0+  
interface on creating and managing these objects it would be a lot  
easier.


Agreed, this is one of the projects I've been working on just haven't 
had the time to finish it. But please allow me to put in a shameless 
plug for IRR PowerTools, which many networks (including a couple tier 
1s) use to do their IRR:


http://sourceforge.net/projects/irrpt/

The highlights are:

* Automated retrieval of prefixes registered behind an IRR Object.
* Automatic exclusion of bogon or other configured undesirable routes.
* Tracking and long-term recording of prefix changes over time.
* Automatic aggregation to optimize data and reduce unnecessary changes.
* E-mail updates, letting users know that their change was processed.
* E-mail alerts to the ISP, letting them know of new routing changes.
* E-mail alerts to non-IRR using networks, with plain text changes.
* Router config generation, for easy automated config deployment.  


I'm also in the process of beta testing a new 2.0 version which will be
significantly rewritten for easier more scalable use, have a lot fewer
dependencies, integrate better with db backend systems to customer
prefix-list management, and fully support IPv6. Oh and there might just
be a web gui for managing and using it too, if I can find a decent web
developer who will do it for free. :) I'm hoping to have this out in the 
next couple of months.




The longer a network develops without using or maintaining IRR data, the 
more difficult the transition becomes. A truly awesome capability would 
be to have some way to slurp in current configuration and output IRR 
objects that can then generate a functionally identical configuration.


Even without that, I would happily settle for any improvements to irr 
tools, powertools or toolsets. I am sort of disappointed that there 
seemed to be far less then deserved support from those with a stake in 
this, the registries and vendors.


Joe

Re: Follow up to previous post regarding SAAVIS

[Nick Hilliard]

On 13/08/2009 04:03, Richard A Steenbergen wrote:

In fact this is one of
the reasons why querying data from RIPE is such a pain, their query
language lacks a recursive service side expansion mechanism so the
transaction latency turns querying a large AS-SET into a multi-hour or
day long operation.


I've brought this to RIPE's attention before, but the suggestion was met 
with a sharp inhalation of breath and a discussion about exactly how 
difficult that might be.  The ripe whoisd code-base is too complicated.


[Server side UTF8 support would be nice too, but for entirely different 
reasons - apparently there are some people in the world who need character 
sets outside ascii7.  Who knew?]


> Do you think you win something by preferring say RADB over

LEVEL3 over SAVVIS over ARIN over RIPE over...? You have no real idea
where your customer's are keeping their records, or their customers,
etc. Where do you draw the line on who's data you look at, and why do we
need yet another system where people are left to make a judgement call
over who's data they should trust?


Yes, this is a bit of a mess alright.


There is plenty of motivation to add data to IRR to make your
announcements work, but no motivation at all to remove data when it is
no longer needed. Nobody sees a problem with this until you step back
and realize that a lot of networks have IRR records so sloppy that they
list nearly every route on the Internet. Why bother filtering at all
then?


Data with "source: RIPE" is quite good from this point of view, as the 
mnt-lower: and mnt-routes: fields are delegated by the RIR function in 
RIPE.  There's no doubt that you can get all sorts of crap from non-RIR 
irrdbs, though.



I think if it was as simple as seeing a list of your routes (or
customers in your as-set, etc) and having a checkbox to delete old data,
people would be more reasonable about maintaining it. RPSL is scary and
confusing to a lot of people (and it should probably be scary to
everyone at any rate :P), there is no reason it needs to be like this.


RPSL is scary both from an end-user and a developer point of view.  From 
the developer point of view, if the requirement to support AS path regular 
expressions were removed, that would remove lots of scary code from 
irrtoolset.  The grammar is also very rich, which is just another word for 
complicated.


From the user point of view, as a means of maintaining full policy routing 
configuration information (which seemed to be its original goal), it fails 
quite badly: too complicated to be understood easily; to limited to fulfil 
its objective.


Like lots of technologies which didn't take off as intended (x.500, 
multicast, etc), it's found a stable niche, although there is no doubt that 
its prefix filtering capability is very undervalued.


Nick

Re: Follow up to previous post regarding SAAVIS

[Richard A Steenbergen]

On Wed, Aug 12, 2009 at 10:06:49PM -0400, Joe Provo wrote:
> On Wed, Aug 12, 2009 at 08:16:38PM -0500, Richard A Steenbergen wrote:
> [snip]
> > Unfortunately the distributed nature of the databases is one of the 
> > biggest problems with the IRR system. Anyone can run an irrd, there is 
> 
> You misspelled "largest strength".  FOlks get to choose which registries
> to believe in what order, not required to have a single [politicized]
> entity.

Well, actually, no. I'm not aware of any mechanism under which you can
effectively choose who to believe and in what order, nor do I think that
it would make any real difference in the long run even if you could.

IRR database mirroring is like being a tier 1, you have to peer with
every other database out there in order to obtain a full view. That
means there are two ways you can get access to all the data you need, 
you can either query someone else who maintains all those mirrors, or 
you can run your own irr db and do all the mirroring yourself. RADB is 
the de facto "main db" in the IRR system, it not only originates the 
vast majority of the routes but it maintains the most comprehensive 
mirroring of every other active IRR db. RADB currently tracks a total of 
32 databases:

http://www.radb.net/mirrorlist.html

At the end of the day the results are the same, whether the data is in
your local irrd or someone else's db (like RADB). You query them via the
same mechanism, which has extremely limited source selection control. 
Basically the only thing you have control over are the list of IRR
databases which are searched, but the results which are returned are a
superset of all databases which you selected to search. You don't get to
say "only listen to results from LEVEL3's db for this object unless they
don't have results there, in which case you listen to SAVVIS" or
anything like that. You could query the complete data for every
individual route yourself, but this would be a massively difficult
undertaking compared to the normal query operation. The normal query
operation is by no stretch of the imagination easy either, querying a
large ISP can take hours or more depending on the transaction latency
between yourself and the server you're querying. In fact this is one of
the reasons why querying data from RIPE is such a pain, their query
language lacks a recursive service side expansion mechanism so the
transaction latency turns querying a large AS-SET into a multi-hour or
day long operation. Their whois daemon also has an obnoxious "feature"
of forcefully closing the socket after 3 minutes, even if it's in the
middle of returning an answer. This is the only real advantage of
running your own local irrd, reducing the transaction latency, but it's
still a lot of work to maintain the mirror, verify that all the data
from all the sources is always importing correctly, etc. And even after
you do all this, what does being able to pick a data source order buy
you anyways? Do you think you win something by preferring say RADB over
LEVEL3 over SAVVIS over ARIN over RIPE over...? You have no real idea
where your customer's are keeping their records, or their customers,
etc. Where do you draw the line on who's data you look at, and why do we 
need yet another system where people are left to make a judgement call 
over who's data they should trust?

Personally I'm of the belief that every ISP running their own IRR db is
a very bad idea, which is why I have chosen not to run one myself. To
quote Vijay, it doesn't scale. The last thing the already very broken
IRR system needs is more crap data by more random people spread out over
more databases, and the majority of the current db's probably need to be
shut down too. There is no reason that this process needs to be
politicized, or cost anyone any money to use. Again, we've made a
horrible system here.

One reasonable solution is to have the server side run the complete 
query off its local database, and pass the complete results for a 
prefix-list back to the querier in a single transaction. This is how 
filtergen.level3.com works, though I personally find their system is be 
excessively slow. In IRRPT 2.0 development I'm writing a similar type of 
remote filter generator, which I hope will be useful to some people.

> If folks mistakenly believe there is a 1:1 correspendence between IRR
> data and BGP tables, they will lose.  The IRR data is more of a
> "flight plan", a set of what-is-possible per the originator of the
> data.
> 
> [snip]
> > people query and boom you're in the system. What tends to happen is 
> > someone puts a route into a database and then completely forgets about 
> > it, so there are a huge number of completely bogus routes out there 
> > which are never going to get cleaned up.
> 
> Lots of folks set up systems for provisioning without deprovisioning. 
> This is not an IRR problem, but a sloppy-human problem.  Folks that 
> get stuck with provisioning generally aren't incented to remove billable 
> 

Re: Follow up to previous post regarding SAAVIS

[Joe Provo]

On Wed, Aug 12, 2009 at 08:16:38PM -0500, Richard A Steenbergen wrote:
[snip]
> Unfortunately the distributed nature of the databases is one of the 
> biggest problems with the IRR system. Anyone can run an irrd, there is 

You misspelled "largest strength".  FOlks get to choose which registries
to believe in what order, not required to have a single [politicized]
entity.  If folks mistakenly believe there is a 1:1 correspendence 
between IRR data and BGP tables, they will lose.  The IRR data is more
of a "flight plan", a set of what-is-possible per the originator of the 
data.

[snip]
> people query and boom you're in the system. What tends to happen is 
> someone puts a route into a database and then completely forgets about 
> it, so there are a huge number of completely bogus routes out there 
> which are never going to get cleaned up.

Lots of folks set up systems for provisioning without deprovisioning. 
This is not an IRR problem, but a sloppy-human problem.  Folks that 
get stuck with provisioning generally aren't incented to remove billable 
resources.  CF good processes and management with backbone.

Cheers,

Joe

-- 
 RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

Re: Follow up to previous post regarding SAAVIS

[Joe Provo]

On Wed, Aug 12, 2009 at 05:55:10PM -0500, Richard A Steenbergen wrote:
> On Wed, Aug 12, 2009 at 05:41:03PM -0400, Joe Provo wrote:
> > Most ISPs don't have that level of management clue & willpower, as the 
> > same "but they will go to $competator who doesn't require it!" which 
> > has screwed up everything from domain registration to responsible BGP 
> > announcements fouls the customer interface as well. Account reps wanting
> > an exception 'just this once' are the norm.
> 
> I would make the opposite argument, my business would NEVER go to any
> network which didn't support IRR (and a bunch of other simple but
> important things, like a full set of non-secret BGP communities). 

You and I would agree, but there are far more edge ASNs than mine, 
and last I checked you weren't running an edge.  Just as with any
commoditization, the large number of buyers tends to win out. If 
the world we wished existied, 3561 wouldn't have lost their good
provider-hosted-IRR based filters.

Cheers,

Joe

-- 
 RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

Re: Follow up to previous post regarding SAAVIS

[Christopher Morrow]

On Wed, Aug 12, 2009 at 9:16 PM, Richard A Steenbergen wrote:
> On Wed, Aug 12, 2009 at 07:37:00PM -0500, Frank Bulk wrote:
>> Perhaps this is a stupid question, but does each SP need to run their own
>> physical RR?  Isn't this something that could be hosted?
>
> The data itself is stored on a distributed network of databases, so
> there is technically no reason any SP needs to run their own. However,
> they often do, because when a customer can't figure something out it
> gives them access to go in and tweak the customers' records for them.

Another reason one might choose to run their own is you never want to
answer a customer with: "Well, we query the remote systems when we
need to build prefix lists and apparently the remote system
blacklisted us because we queried too many times this
hour/minute/day/week.. sorry 'bout that!"

It gives you control over the data format, availability, freshness
(sorta) and 'security' of the data you'd update your network with. Not
everyone has that set of requirements, but if you ask L3  or other
folks that do IRR based 'auto filtering' I bet they have an answer
along these lines.

> Unfortunately the distributed nature of the databases is one of the
> biggest problems with the IRR system. Anyone can run an irrd, there is
> no inherient authentication of the data. In order to get your irrd
> "recognized" all you have to do is get mirrored by a database that other
> people query and boom you're in the system. What tends to happen is
> someone puts a route into a database and then completely forgets about
> it, so there are a huge number of completely bogus routes out there
> which are never going to get cleaned up.

drum, drum, drum.. rpki! (and hopefully having that tied back to a
bill you pay... see s...@ietf.org or wherever that list emanates from
these days)

> The other problem is that when a SP has a customer who "can't figure it
> out", a typical course of action is to just "register the route for
> them" rather than try to explain it to them. Unfortunately, the same
> thing as above happens here, you end up with a big pile of people who
> register a big pile of routes in a big pile of random databases, often
> times completely unnecessarily, and they'll never be removed either.
>
> The biggest problem with the entire system is the way that data gets
> into it, and the lack of incentive for people to ever remove data from
> it. But as a mechanism to allow the routes which need to be allowed, and
> mostly prevent accidental leaks, it works.

The above 2 paragraphs I think encapsulate what happened to ConEd in
~2005? (or someone-or-other-edison in newyork) Old/stale data which
cropped up in an unusual manner :(

> For more information, take a look at:
>
> http://www.nanog.org/meetings/nanog44/presentations/Tuesday/RAS_irrdata_N44.pdf

btw, +1 on irrtoolset.. good stuff.

-Chris

Re: Follow up to previous post regarding SAAVIS

[Richard A Steenbergen]

On Wed, Aug 12, 2009 at 07:37:00PM -0500, Frank Bulk wrote:
> Perhaps this is a stupid question, but does each SP need to run their own
> physical RR?  Isn't this something that could be hosted?

The data itself is stored on a distributed network of databases, so 
there is technically no reason any SP needs to run their own. However, 
they often do, because when a customer can't figure something out it 
gives them access to go in and tweak the customers' records for them. 

Unfortunately the distributed nature of the databases is one of the 
biggest problems with the IRR system. Anyone can run an irrd, there is 
no inherient authentication of the data. In order to get your irrd 
"recognized" all you have to do is get mirrored by a database that other 
people query and boom you're in the system. What tends to happen is 
someone puts a route into a database and then completely forgets about 
it, so there are a huge number of completely bogus routes out there 
which are never going to get cleaned up.

The other problem is that when a SP has a customer who "can't figure it 
out", a typical course of action is to just "register the route for 
them" rather than try to explain it to them. Unfortunately, the same 
thing as above happens here, you end up with a big pile of people who 
register a big pile of routes in a big pile of random databases, often 
times completely unnecessarily, and they'll never be removed either.

The biggest problem with the entire system is the way that data gets 
into it, and the lack of incentive for people to ever remove data from 
it. But as a mechanism to allow the routes which need to be allowed, and 
mostly prevent accidental leaks, it works.

For more information, take a look at:

http://www.nanog.org/meetings/nanog44/presentations/Tuesday/RAS_irrdata_N44.pdf

-- 
Richard A Steenbergenhttp://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Follow up to previous post regarding SAAVIS

[mkarir]

Hi Jared,

You should give the new RADB portal a try.  We were trying to do
pretty much what you describe.  Dont know what web2.0 is but the
new portal is a web based object management system complete
with "recommended" changes and inconsistency lists.
We just added prefix allocation check with backend information
from PCH (prefix checker tool).

Prefix/acl generation is coming soon too stay tuned.  Alert/ 
notification has
been tested and we are figuring out how to roll it out on a larger  
scale.


We are also starting to study whether the data itself
has actually gotten any better since we put this web gui in place.

-manish


On Aug 12, 2009, at 6:55 PM, nanog-requ...@nanog.org wrote:


Date: Wed, 12 Aug 2009 16:57:07 -0400
From: Jared Mauch 
Subject: Re: Follow up to previous post regarding SAAVIS
To: nanog-p...@rsuc.gweep.net
Cc: "nanog@nanog.org" 
Message-ID: 
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes


I've come to the conclusion that if someone put a nice web2.0+
interface on creating and managing these objects it would be a lot
easier.

If there were a customer portal where you could visit to say "update
my prefix-list/acl to include the following new prefix(es), and push
the change /now/" I presume that would drive customer utilization of
these services and allow people to manage things "better".

There are lots of leaks all the time, as can be evidenced by the leak
detection stuff I set up here:

http://puck.nether.net/bgp/leakinfo.cgi

- Jared

Re: Follow up to previous post regarding SAAVIS

[Steve Gibbard]

On Wed, 12 Aug 2009, Richard A Steenbergen wrote:


I would make the opposite argument, my business would NEVER go to any
network which didn't support IRR (and a bunch of other simple but
important things, like a full set of non-secret BGP communities). It's
amazing the number of networks that excludes in this day and age. And
not even because "omg IRR is good because someone told me so and we
should support it", but because I've seen FAR too much grief caused by
humans typoing prefix-lists, or taking days to process them. It is the
height of absurdity that this would ever be considered an acceptable
solution to the problem.


I'd be very hesitant to use an upstream that didn't use any filtering 
method.  But, as convenient as I find the IRR system to be (from the 
customer perspective, at least), I'm quite happy that a couple of our 
upstreams use other mechanisms instead.


I've had prefixes fall out of the IRR a couple of times, leading to 
outages of IRR-using transit providers.  I've had transit providers screw 
up manually maintained prefix-lists, or had mis-communications resulting 
in the wrong thing getting removed.  With multiple transit providers and 
multiple systems, they tend not to all have the same filtering problem at 
the same time.  That's a very good thing.


I hope the recommendation that comes out of this discussion is to filter 
somehow, rather than to use any particular filter-generation mechanism. 
Diversity and redundancy are good things, in processes as well as 
hardware.


-Steve

RE: Follow up to previous post regarding SAAVIS

[Frank Bulk]

Perhaps this is a stupid question, but does each SP need to run their own
physical RR?  Isn't this something that could be hosted?

Frank

-Original Message-
From: Richard A Steenbergen [mailto:r...@e-gerbil.net] 
Sent: Wednesday, August 12, 2009 5:55 PM
To: Joe Provo
Cc: nanog@nanog.org
Subject: Re: Follow up to previous post regarding SAAVIS

On Wed, Aug 12, 2009 at 05:41:03PM -0400, Joe Provo wrote:
> Most ISPs don't have that level of management clue & willpower, as the 
> same "but they will go to $competator who doesn't require it!" which 
> has screwed up everything from domain registration to responsible BGP 
> announcements fouls the customer interface as well. Account reps wanting
> an exception 'just this once' are the norm.

I would make the opposite argument, my business would NEVER go to any
network which didn't support IRR (and a bunch of other simple but
important things, like a full set of non-secret BGP communities). It's
amazing the number of networks that excludes in this day and age. And 
not even because "omg IRR is good because someone told me so and we 
should support it", but because I've seen FAR too much grief caused by 
humans typoing prefix-lists, or taking days to process them. It is the 
height of absurdity that this would ever be considered an acceptable 
solution to the problem.

But most of all I'm amazed that we as network operators have managed to 
take such a simple concept as a protocol for storing and recursively 
retrieving a list of prefixes in a database and turned it into such a 
sloppy mess. We really shot ourselves in the foot with the complexity of 
RPSL, which tries to be everything to everyone rather than actually 
provide a simple effective way to maintain prefix-lists.

-- 
Richard A Steenbergenhttp://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Follow up to previous post regarding SAAVIS

[Richard A Steenbergen]

On Wed, Aug 12, 2009 at 05:41:03PM -0400, Joe Provo wrote:
> Most ISPs don't have that level of management clue & willpower, as the 
> same "but they will go to $competator who doesn't require it!" which 
> has screwed up everything from domain registration to responsible BGP 
> announcements fouls the customer interface as well. Account reps wanting
> an exception 'just this once' are the norm.

I would make the opposite argument, my business would NEVER go to any
network which didn't support IRR (and a bunch of other simple but
important things, like a full set of non-secret BGP communities). It's
amazing the number of networks that excludes in this day and age. And 
not even because "omg IRR is good because someone told me so and we 
should support it", but because I've seen FAR too much grief caused by 
humans typoing prefix-lists, or taking days to process them. It is the 
height of absurdity that this would ever be considered an acceptable 
solution to the problem.

But most of all I'm amazed that we as network operators have managed to 
take such a simple concept as a protocol for storing and recursively 
retrieving a list of prefixes in a database and turned it into such a 
sloppy mess. We really shot ourselves in the foot with the complexity of 
RPSL, which tries to be everything to everyone rather than actually 
provide a simple effective way to maintain prefix-lists.

-- 
Richard A Steenbergenhttp://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Follow up to previous post regarding SAAVIS

[Ricky Beam]

On Wed, 12 Aug 2009 16:57:07 -0400, Jared Mauch   
wrote:
I've come to the conclusion that if someone put a nice web2.0+ interface  
on creating and managing these objects it would be a lot easier.


If there were a customer portal where you could visit to say "update my  
prefix-list/acl to include the following new prefix(es), and push the  
change /now/" I presume that would drive customer utilization of these  
services and allow people to manage things "better".


That's fine... until you learn the hard way 9 times out of 10, the person  
heading to such a thing is a clueless moron.  As much as it is a pain in  
the rear, having *people* proofing and editing the BGP configuration is  
far less work than creating the AI needed to keep idiots from doing  
idiotic things.


I've never worked for any of the tier-1 beasts, but I have had to manage  
dozens of customer BGP sessions. Over a decade, I never dealt with a  
clueful customer... we cannot announce address space that doesn't belong  
to you; you cannot announce *our* address space to other ISPs; you have  
one f'ing link, why do you need BGP? (note: never actually ask that  
question or mute your phone immediately after asking.)


How often do your prefixes change?  In my experience adding new netblocks  
and/or customers, taking a few weeks to get things setup wasn't a problem;  
it'd take that long to get their connection turned up. (and if they were  
talking about BGP, sales would be talking to us before the contract was  
signed.)


--Ricky

Re: Follow up to previous post regarding SAAVIS

[Joe Provo]

On Wed, Aug 12, 2009 at 02:30:38PM -0700, Kevin Oberman wrote:
[snip]
> While a web 2.0 app would be very nice, it's really not that hard to do
> now. You do need the IRRToolSet or something similar. the IRRToolSet has
> languished for a long time and was getting harder and harder to keep
> running, but the move to sourceforge and the massive number of updates
> and clean ups should soon result in a 5.0 release that builds cleanly on

[insert plug for the revitalization going on at irrtool...@lists.isc.org]

The internal piece (ras's code, etc) is the easy part.  The customer-
facing piece isn't particularly difficult either.  Getting it past the
inexplicably-powerful branding people in marketing and having a management
team with the iron will to dictate that the pointy-clicky form is not
just the standard vector but the ONLY vector for non-IRR clued users.
That was the support the cary team had at old 3561.

Most ISPs don't have that level of management clue & willpower, as the 
same "but they will go to $competator who doesn't require it!" which 
has screwed up everything from domain registration to responsible BGP 
announcements fouls the customer interface as well. Account reps wanting
an exception 'just this once' are the norm.

Cheers,

Joe


-- 
 RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

Re: Follow up to previous post regarding SAAVIS

[Kevin Oberman]

> Date: Wed, 12 Aug 2009 16:12:39 -0500
> From: Justin Shore 
> 
> Jared Mauch wrote:
> > I've come to the conclusion that if someone put a nice web2.0+ interface 
> > on creating and managing these objects it would be a lot easier.
> 
> I've looked into IRR several times, usually after events like PCCW. 
> Each time the amount of work to 1) figure out how to implement IRR and 
> 2) actually implementing it far outweighed the benefit of doing it for 
> my network.  I would love to implement it and looking towards the future 
> I will someday have to.  Until it becomes much easier to do so, I don't 
> foresee smaller SPs like myself allocating the necessary resources to an 
> IRR project until we're forced to because of our individual growth or an 
> idiot PCCW-type event that generates lots of bad PR.

While a web 2.0 app would be very nice, it's really not that hard to do
now. You do need the IRRToolSet or something similar. the IRRToolSet has
languished for a long time and was getting harder and harder to keep
running, but the move to sourceforge and the massive number of updates
and clean ups should soon result in a 5.0 release that builds cleanly on
most Unix/Linux platforms with a modern C++ compiler.

Once that happens, it's really pretty simple to use the IRR for this
sort of thing and, while the IRR data quality is pretty poor, it's a
vast improvement over crossing your fingers and sticking your head in
the sand.

Except for the ugliness of the Perl I wrote well over a decade ago (when
I was just learning Perl), I'd try to talk the powers that be into
allowing me to release it as an example, but the relevant code is
really, really trivial. (Not that government would let me without vast
quantities of paperwork.)
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: ober...@es.net  Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751

Re: Follow up to previous post regarding SAAVIS

[Richard A Steenbergen]

On Wed, Aug 12, 2009 at 04:57:07PM -0400, Jared Mauch wrote:
> 
> I've come to the conclusion that if someone put a nice web2.0+  
> interface on creating and managing these objects it would be a lot  
> easier.

Agreed, this is one of the projects I've been working on just haven't 
had the time to finish it. But please allow me to put in a shameless 
plug for IRR PowerTools, which many networks (including a couple tier 
1s) use to do their IRR:

http://sourceforge.net/projects/irrpt/

The highlights are:

* Automated retrieval of prefixes registered behind an IRR Object.
* Automatic exclusion of bogon or other configured undesirable routes.
* Tracking and long-term recording of prefix changes over time.
* Automatic aggregation to optimize data and reduce unnecessary changes.
* E-mail updates, letting users know that their change was processed.
* E-mail alerts to the ISP, letting them know of new routing changes.
* E-mail alerts to non-IRR using networks, with plain text changes.
* Router config generation, for easy automated config deployment.  

I'm also in the process of beta testing a new 2.0 version which will be
significantly rewritten for easier more scalable use, have a lot fewer
dependencies, integrate better with db backend systems to customer
prefix-list management, and fully support IPv6. Oh and there might just
be a web gui for managing and using it too, if I can find a decent web
developer who will do it for free. :) I'm hoping to have this out in the 
next couple of months.

-- 
Richard A Steenbergenhttp://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Re: Follow up to previous post regarding SAAVIS

[Justin Shore]

Jared Mauch wrote:
I've come to the conclusion that if someone put a nice web2.0+ interface 
on creating and managing these objects it would be a lot easier.


I've looked into IRR several times, usually after events like PCCW. 
Each time the amount of work to 1) figure out how to implement IRR and 
2) actually implementing it far outweighed the benefit of doing it for 
my network.  I would love to implement it and looking towards the future 
I will someday have to.  Until it becomes much easier to do so, I don't 
foresee smaller SPs like myself allocating the necessary resources to an 
IRR project until we're forced to because of our individual growth or an 
idiot PCCW-type event that generates lots of bad PR.


There are lots of leaks all the time, as can be evidenced by the leak 
detection stuff I set up here:


http://puck.nether.net/bgp/leakinfo.cgi


Crossing fingers hoping I'm not in that list...

Justin

Re: Follow up to previous post regarding SAAVIS

[Jared Mauch]

On Aug 12, 2009, at 4:37 PM, Joe Provo wrote:


On Wed, Aug 12, 2009 at 11:20:28AM -0700, goe...@anime.net wrote:

On Wed, 12 Aug 2009, Christopher Morrow wrote:
On Wed, Aug 12, 2009 at 9:57 AM, Drew  
Weaver wrote:
Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to  
advertise

routes for whatever IP addresses they want?

sadly savvis didn't learn the pccw lesson, which is also the
turk-telecom lesson which is also the as7007 lesson which is...  
fairly

sad really in 2009.
for the sake of $diety put a prefix-filter on your customer bgp
sessions, it ain't hard!


sounds too much like "work" to me. not interested.


The irony is that MCI had (and C&W maintained for quite some time)
a functional, highly automated IRR-based customer filtering system
[props to the Cary team].  Somewhere along the M&A highway, the
work to maintain it was substituted with the work to dismantle it.
Sad to see crap emitted with 3561 in the path.


I've come to the conclusion that if someone put a nice web2.0+  
interface on creating and managing these objects it would be a lot  
easier.


If there were a customer portal where you could visit to say "update  
my prefix-list/acl to include the following new prefix(es), and push  
the change /now/" I presume that would drive customer utilization of  
these services and allow people to manage things "better".


There are lots of leaks all the time, as can be evidenced by the leak  
detection stuff I set up here:


http://puck.nether.net/bgp/leakinfo.cgi

- Jared

Re: Follow up to previous post regarding SAAVIS

[Joe Provo]

On Wed, Aug 12, 2009 at 11:20:28AM -0700, goe...@anime.net wrote:
> On Wed, 12 Aug 2009, Christopher Morrow wrote:
> >On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver wrote:
> >>Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise 
> >>routes for whatever IP addresses they want?
> >sadly savvis didn't learn the pccw lesson, which is also the
> >turk-telecom lesson which is also the as7007 lesson which is... fairly
> >sad really in 2009.
> >for the sake of $diety put a prefix-filter on your customer bgp
> >sessions, it ain't hard!
> 
> sounds too much like "work" to me. not interested.

The irony is that MCI had (and C&W maintained for quite some time) 
a functional, highly automated IRR-based customer filtering system
[props to the Cary team].  Somewhere along the M&A highway, the
work to maintain it was substituted with the work to dismantle it.
Sad to see crap emitted with 3561 in the path.

-- 
 RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

Re: Follow up to previous post regarding SAAVIS

[Christopher Morrow]

On Wed, Aug 12, 2009 at 2:20 PM,  wrote:
> On Wed, 12 Aug 2009, Christopher Morrow wrote:
>>
>> On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver
>> wrote:
>>>
>>> Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise
>>> routes for whatever IP addresses they want?
>>
>> sadly savvis didn't learn the pccw lesson, which is also the
>> turk-telecom lesson which is also the as7007 lesson which is... fairly
>> sad really in 2009.
>> for the sake of $diety put a prefix-filter on your customer bgp
>> sessions, it ain't hard!
>
> sounds too much like "work" to me. not interested.

Re: Follow up to previous post regarding SAAVIS

[goemon]

On Wed, 12 Aug 2009, Christopher Morrow wrote:

On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver wrote:

Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes 
for whatever IP addresses they want?

sadly savvis didn't learn the pccw lesson, which is also the
turk-telecom lesson which is also the as7007 lesson which is... fairly
sad really in 2009.
for the sake of $diety put a prefix-filter on your customer bgp
sessions, it ain't hard!


sounds too much like "work" to me. not interested.

-Dan

Re: Follow up to previous post regarding SAAVIS

[Christopher Morrow]

On Wed, Aug 12, 2009 at 9:57 AM, Drew Weaver wrote:
> Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes 
> for whatever IP addresses they want?

sadly savvis didn't learn the pccw lesson, which is also the
turk-telecom lesson which is also the as7007 lesson which is... fairly
sad really in 2009.

for the sake of $diety put a prefix-filter on your customer bgp
sessions, it ain't hard!

-chris

> route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768
>  2905 701 3561 13768
>  1221 4637 3561 13768
>  3549 3561 13768
>  3277 3267 174 3561 13768
>  6539 3561 13768
>  16150 3549 3561 13768
>  701 3561 13768
>  3267 174 3561 13768
>  6453 3561 13768
>  3582 3701 3356 3561 13768
>
> This is probably a fairly major problem...
>
> -Drew
>
>

Follow up to previous post regarding SAAVIS

[Drew Weaver]

Anyone know why SAAVIS would be allowing PEER1 (AS 13768) to advertise routes 
for whatever IP addresses they want?

route-views.oregon-ix.net>sh ip bgp 173.45.110.0 | i 13768
  2905 701 3561 13768
  1221 4637 3561 13768
  3549 3561 13768
  3277 3267 174 3561 13768
  6539 3561 13768
  16150 3549 3561 13768
  701 3561 13768
  3267 174 3561 13768
  6453 3561 13768
  3582 3701 3356 3561 13768

This is probably a fairly major problem...

-Drew