Re: IP and Optical domains?

[Glen Kent]

Mikael,

Thanks. I was looking at a technical problem. I say this because you may
not have this problem when both are networks are being run by the same
vendor equipment, say Alcatel-Lucent (or Nokia now). What are the technical
problems because of which ISPs need to over-provision when there are IP and
optical domains involved. OR rather let me rephrase my question --  what is
the technical challenge involved in setting up an end to end path between
two IP domains that have an optical domain in between.

Thanks, Glen

On Sun, Jun 19, 2016 at 3:30 AM, Mikael Abrahamsson 
wrote:

> On Sun, 19 Jun 2016, Glen Kent wrote:
>
> Can somebody shed more light on what it means to say that the IP and
>> optical layers are run as independent kingdoms and why do ISPs need to
>> over-provision?
>>
>
> You have a group that runs fiber+dwdm+sonet(or SDH). You have another
> group that runs IP. When the IP guys ask "please tell us how the optical
> network is designed, and can we coordinate how they're built and btw, we
> want to put DWDM optics in our routers", the answer from the
> fiber+dwdm+sonet group is "no, but we can help you with transport using our
> transponders, please just order circuits, just give us addresses for each
> end and we'll take care of things, don't you worry your little IP engineer
> brain how things are transported long distance".
>
> I believe this is still the case at a lot of ISPs. Not all, hopefully not
> even most, but I'm sure there are some.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>

Re: 1GE L3 aggregation

[James Jun]

On Sat, Jun 18, 2016 at 01:04:49PM +0200, Mark Tinka wrote:
>
> Centralizing is just horrible, but that's just me. The goal is to make
> all these unreliable boxes work together to offer a reliable service to
> your customers, so making them too inter-dependent on each other has the
> potential to that away in the long run.
> 

One issue with pushing IP transit (L3-wise) with small boxes down to the
metro is that if a particular customer comes under attack, any DDoS in
excess of 10-30 Gbps is going to totally destroy the remote site down to
the floor and then some, until NOC intervenes to restore service.

A Big Expensive Router at head-end site fed with big pipes to your IP core
just needs a subscriber line rate policer configured on the customer EVC
off the NNI facing your metro transport network, largely protecting your
metro POP during an attack.

There are also issues with control-plane policing (or limited options
there of) with some of these low-end platforms.

> 
> We push IP/MPLS all the way into the Metro-E Access using a team of
> Cisco ASR920's and ME3600X's. The value of being able to instantiate an
> IP service or BGP session directly in the Metro-E Access simplifies
> network operations a great deal for us. Needless to say, not having to
> deal with eBGP Multi-Hop drama does not hurt.

BGP Selective Download has its own drawbacks too--in that, it's largely
meant to be used on single-tailed environment with FIB only having single
point of egress.

Consider a topology where an ASR920 in the metro is dual-homed to two 
peering sites using variably distant dark fiber (say 30km to Site A,
90km to Site B), with IGP costs configured to conform to fiber distances.

How will you guarantee that the best-path that ASR920 chooses for your
customer taking full table to be actually congruent with the box's
real forwarding path?  Your 920 may choose site A as best-path, only
to see FIB-programmed default route to force it out on site B.  If you're
doing active-standby on your fiber uplinks, then it would not be an issue;
or may be in metro environment where latency differences are minimal (<1ms),
you probably don't care in practice to be bothered with that.



Yes, there are some operational complexities and issues with L2vpn'ing
customers to head-end router -- such as, link-state propagation needs to
be properly validated; and now you're burning two ports instead of one,
one at the terminus, one at the access, doubling SPOF and maintenance
liabilities.

At the end of the day, it's lack of full-featured ports at reasonable cost
that drive centralization to head-ends.  Spamming Small Expensive Routers
(ASR9001/MX104) in every small metro site doesn't scale (btdt myself), but
neither is hacking up BGP to work on platforms that aren't really meant to
function as heavy L3 routers (e.g. ASR920, ME3600), IMHO.


James

AT Austin multiple 10Gs hard down, anyone seeing issues as well?

[Nick Crocker]

Looking at some sites and hearing chatter of a pretty wide scale AT
outage in the Austin and surrounding areas. I have two 10Gs one fed out of
Pflugerville, TX and one out of Austin, TX with PE's in Austin, TX and San
Antonio, TX. Both dropped at the same time and are on diverse fiber routes
and entrance facilities with power and equipment verified.

Regards,

Nick

Re: 1GE L3 aggregation

[Radu-Adrian Feurdean]

On Fri, Jun 17, 2016, at 12:43, Saku Ytti wrote:
> Last I checked you can't commit/replace configuration in VRP. Has this
> changed? Can you give it full new config and expect it to figure out
> how to apply the new config without breaking existing?

... later...

> Yeah it's best I've seen. 8-10k isn't anything special.

I suppose that's the reason I didn't see the Brocade CER-RT (some time
ago best-seller) listed. Probably the lack of need for full-table FIB
also counts.

Re: IP and Optical domains?

[Randy Bush]

> You have a group that runs fiber+dwdm+sonet(or SDH). You have another 
> group that runs IP. When the IP guys ask "please tell us how the optical 
> network is designed, and can we coordinate how they're built and btw, we 
> want to put DWDM optics in our routers", the answer from the 
> fiber+dwdm+sonet group is "no, but we can help you with transport using 
> our transponders, please just order circuits, just give us addresses for 
> each end and we'll take care of things, don't you worry your little IP 
> engineer brain how things are transported long distance".
> 
> I believe this is still the case at a lot of ISPs. Not all, hopefully not 
> even most, but I'm sure there are some.

you underestimate the extent of the dogged determination of circuitzilla
to hang on to the fiber with her/his fingernails.

randy

Re: IP and Optical domains?

[Mikael Abrahamsson]

On Sun, 19 Jun 2016, Glen Kent wrote:

Can somebody shed more light on what it means to say that the IP and 
optical layers are run as independent kingdoms and why do ISPs need to 
over-provision?


You have a group that runs fiber+dwdm+sonet(or SDH). You have another 
group that runs IP. When the IP guys ask "please tell us how the optical 
network is designed, and can we coordinate how they're built and btw, we 
want to put DWDM optics in our routers", the answer from the 
fiber+dwdm+sonet group is "no, but we can help you with transport using 
our transponders, please just order circuits, just give us addresses for 
each end and we'll take care of things, don't you worry your little IP 
engineer brain how things are transported long distance".


I believe this is still the case at a lot of ISPs. Not all, hopefully not 
even most, but I'm sure there are some.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

IP and Optical domains?

[Glen Kent]

HI,

I was reading the following article:
http://www.lightreading.com/optical/sedona-boasts-multilayer-network-orchestrator/d/d-id/714616

It says that "The IP layer and optical layer are run like two separate
kingdoms," Wellingstein says. "Two separate kings manage the IP and optical
networks. There is barely any resource alignment between them. The result
of this is that the networks are heavily underutilized," or, from an
alternative perspective, "they are heavily over-provisioned."

Can somebody shed more light on what it means to say that the IP and
optical layers are run as independent kingdoms and why do ISPs need to
over-provision?

Thanks, Glen

Re: Do people even read these? Re: BGP Update Report

[Geoff Huston]

> On 19 Jun 2016, at 6:05 AM, Mikael Abrahamsson  wrote:
> 
> On Fri, 17 Jun 2016, cidr-rep...@potaroo.net wrote:
> 
>> 
>> TOP 20 Unstable Prefixes
>> Rank Prefix Upds % Origin AS -- AS Name
>> 1 - 202.65.32.0/2128086  0.8%   AS10131 -- CKTELECOM-CK-AP Telecom Cook 
>> Islands, CK
>> 2 - 110.170.17.0/24   21868  0.7%   AS134438 -- AIRAAIFUL-AS-AP Aira & Aiful 
>> Public Company Limited, TH
>> 3 - 123.231.192.0/24  21562  0.7%   AS133841 -- IDOLA-BROADBAND-AS-ID 
>> INDONESIA BROADBAND ACCESS - ANYWHERE, ID
>> 4 - 93.181.192.0/19   20895  0.6%   AS13118 -- ASN-YARTELECOM Verhnevolzhsky 
>> branch, RU
>> 5 - 123.231.206.0/24  19170  0.6%   AS133841 -- IDOLA-BROADBAND-AS-ID 
>> INDONESIA BROADBAND ACCESS - ANYWHERE, ID
>> 6 - 123.231.193.0/24  19082  0.6%   AS133841 -- IDOLA-BROADBAND-AS-ID 
>> INDONESIA BROADBAND ACCESS - ANYWHERE, ID
>> 7 - 195.128.159.0/24  15455  0.5%   AS56636 -- ASVEDARU , RU
>> 8 - 192.254.88.0/24   15452  0.5%   AS21859 -- ZNET - Zenlayer Inc, US
>> 9 - 185.11.121.0/24   14957  0.5%   AS202105 -- DSP-AS , SA
> 
> Everyone of these prefixes have managed to average one update per 40 seconds 
> during a week, or worse. How is that even possible? Yes, I know we don't 
> generally have dampening anymore, but geez, that's a lot of updates.
> 

In the case of Cook Islands Telecom the problem is not directly with them - its 
their one-up upstream Spark NZ (AS4648) who appears to be flicking this route 
across a number of transit upstreams 
(http://bgpupdates.potaroo.net/cgi-bin/per-prefix?prefix=202.65.32.0.21)

Geoff

Re: Do people even read these? Re: BGP Update Report

[Larry Sheldon]

You did.

--
"Everybody is a genius.  But if you judge a fish by
its ability to climb a tree, it will live its whole
life believing that it is stupid."

--Albert Einstein

From Larry's Cox account.

Do people even read these? Re: BGP Update Report

[Mikael Abrahamsson]

On Fri, 17 Jun 2016, cidr-rep...@potaroo.net wrote:



TOP 20 Unstable Prefixes
Rank Prefix Upds % Origin AS -- AS Name
1 - 202.65.32.0/2128086  0.8%   AS10131 -- CKTELECOM-CK-AP Telecom Cook 
Islands, CK
2 - 110.170.17.0/24   21868  0.7%   AS134438 -- AIRAAIFUL-AS-AP Aira & Aiful 
Public Company Limited, TH
3 - 123.231.192.0/24  21562  0.7%   AS133841 -- IDOLA-BROADBAND-AS-ID INDONESIA 
BROADBAND ACCESS - ANYWHERE, ID
4 - 93.181.192.0/19   20895  0.6%   AS13118 -- ASN-YARTELECOM Verhnevolzhsky 
branch, RU
5 - 123.231.206.0/24  19170  0.6%   AS133841 -- IDOLA-BROADBAND-AS-ID INDONESIA 
BROADBAND ACCESS - ANYWHERE, ID
6 - 123.231.193.0/24  19082  0.6%   AS133841 -- IDOLA-BROADBAND-AS-ID INDONESIA 
BROADBAND ACCESS - ANYWHERE, ID
7 - 195.128.159.0/24  15455  0.5%   AS56636 -- ASVEDARU , RU
8 - 192.254.88.0/24   15452  0.5%   AS21859 -- ZNET - Zenlayer Inc, US
9 - 185.11.121.0/24   14957  0.5%   AS202105 -- DSP-AS , SA


Everyone of these prefixes have managed to average one update per 40 
seconds during a week, or worse. How is that even possible? Yes, I know we 
don't generally have dampening anymore, but geez, that's a lot of updates.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: 1GE L3 aggregation

[Baldur Norddahl]

On 18 June 2016 at 13:04, Mark Tinka  wrote:

> We push IP/MPLS all the way into the Metro-E Access using a team of
> Cisco ASR920's and ME3600X's. The value of being able to instantiate an
> IP service or BGP session directly in the Metro-E Access simplifies
> network operations a great deal for us. Needless to say, not having to
> deal with eBGP Multi-Hop drama does not hurt.
>

Just want to point out that there is no eBGP multi-hop involved. These are
L2 tunnels so the devices appear to be directly connected on the layer 3
level.

The advantage of using L2VPN is that you can connect the customer to
whatever can handle the requirements. You are not limited to what your
access edge devices can do. 99% of our customers are not BGP customers, so
it would be silly to spend cash on equipment that will support full table
BGP at each PoP.

The major downside is a) hops are invisible to traceroute, b) some traffic
might travel longer than necessary.

We are a residential ISP and we find that traffic between customers is
minimal. We choose to accept that traffic between two neighbors might be
backhauled to a central location and back instead of staying local.

Regards,

Baldur

Re: 1GE L3 aggregation

[Baldur Norddahl]

On 18 June 2016 at 13:07, Mark Tinka  wrote:

> > Our PoPs are connected in a ring topology (actually multiple rings). If a
> > link goes down somewhere, or an intermediate device crashes, the L2VPN
> will
> > reconfigure and find another path.
>
> Which is what would happen anyway with your IGP if the service were
> delivered in the Access, but with fewer moving parts and less
> inter-dependence if the problem went beyond just ring failure or device
> crash.
>

Is the claim about fewer moving parts actually true? Yes if you are
comparing to a plain native single-stack network with IPv4 (or IPv6)
directly on the wire. But we are doing MPLS, so in our case it is L2VPN vs
L3VPN. Both will reroute using the exact same mechanism, so no difference
here.

I found that I could remove large parts of the configuration on the access
edge devices when we went from L3VPN to L2VPN. Some people will find the
network easier to understand when all major configuration is in only two
devices, and those two devices are mostly a mirror of each other.

I agree that L3VPN is the better solution, at least in principle. That is
why we started by implementing L3VPN. But in practice the L2VPN solution we
have now is actually easier.

Regards,

Baldur

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Brandon Ross]

On Fri, 17 Jun 2016, Eric Kuhnke wrote:


What Randy just wrote is exactly the point I was trying to make in my last
email. Some real estate facility owners/managers have got into the mistaken
mindset that they can get the greatest value and the most monthly revenue
from the square-footage of their building by charging additional MRC XC
fees to the tenants of the building.


There are some VERY sucessful companies that would strongly disagree with 
you.



When in fact the opposite is true, and we need a concerted community effort
to lobby every IX real estate owner with this fact: Your real estate will
be MORE valuable and will attract a greater critical mass of carriers,
eyeball networks, CDNs, huge hosting providers/colo/VM, etc if you make the
crossconnects free.


But then why would we want to do that?  If you are correct and doing so 
would raise the value of the real esatate, doesn't that mean that the 
building managers would be able to charge operators a whole lot more than 
they are able to today, in aggregate?  Value based pricing is all the rage 
these days, which is why they charge you so much for cross connects.  Why 
do you think they wouldn't take advantage of higher value real estate by 
charging you more for that, instead?  After all, the free cross connect 
situation would be a great way for the owners to lock you into their real 
estate, then all they have to do is dramatically hike the rates when you 
can no longer leave.


--
Brandon Ross  Yahoo & AIM:  BrandonNRoss
+1-404-635-6667ICQ:  2269442
 Skype:  brandonross
Schedule a meeting:  http://www.doodle.com/bross

Re: RPKI implementation

[Randy Bush]

>> i am aware of that.  my point was that cache purge default might
>> better be set to cache refresh interval than 60 secs.
> 
> I would agree with (and in fact, prefer) this protocol.

i remembered wrongly

RFC6810

   A client SHOULD delete the data from a cache when it has been unable
   to refresh from that cache for a configurable timer value.  The
   default for that value is twice the polling period for that cache.

randy

Re: 1GE L3 aggregation

[Mark Tinka]

On 16/Jun/16 23:24, Baldur Norddahl wrote:

> The ZTE 5952E (routing switch) can do L3VPN including BGP. But it is
> limited to about 30k routes. It is usable if the customer wants a default
> route solution, but not if he wants the full default free zone.

Might be worthwhile to ask ZTE to develop their own implementation of
BGP Selective Download.

> Our PoPs are connected in a ring topology (actually multiple rings). If a
> link goes down somewhere, or an intermediate device crashes, the L2VPN will
> reconfigure and find another path.

Which is what would happen anyway with your IGP if the service were
delivered in the Access, but with fewer moving parts and less
inter-dependence if the problem went beyond just ring failure or device
crash.

> For a BGP customer I could offer two tunnels, one to each of our provider
> edge routers. But very few of our customers are BGP customers, they just
> want normal internet. For them we do VRRP between the two provider edge
> routers and have the one tunnel go to both.

If your BGP customer-count grows, while managing 2 eBGP sessions per
customer is not life-threatening, it's certainly won't go unnoticed from
an operational perspective, especially if you are doing this as a matter
of (redundancy) course, in lieu of a revenue-generating request by the
customer to increase their SLA.


>
> We actually moved away from a hybrid solution with L3 termination at the
> customer edge to simply backhauling everything in L2VPNs. We did this
> because the L2VPN tunnels are needed anyway for other reasons and it is
> easier to have one way to do things.

I've never been one to support the confluence of infrastructure tunnels
with customer service tunnels. That's why we avoid infrastructure
tunnels in general, e.g., creating a tunnel from a data centre to a
peering point over which you will run peering traffic because the device
at the data centre can't support peering, or running a tunnel between
two PoP's to handle intra-PoP traffic, e.t.c. When you have all these
tunnels running around, side-by-side with customer revenue-generating
tunnels for their own sake (like a site-to-site l2vpn you've sold to a
customer), it can get hairy at scale, I think. Too much
inter-dependence, too many lines coming together. But again, that's just me.

Mark.

Re: 1GE L3 aggregation

[Mark Tinka]

On 16/Jun/16 22:27, Saku Ytti wrote:

> I'm not saying it's bad solution, I know lot of people do it. But I
> think people only do it, because L3 at port isn't offered by vendors
> at lower rates.

A lot of people did it because because there really wasn't a cheap,
dense solution until about 2010. And even then, the traditional strategy
had become so entrenched that running IP all the way in the Access was
such a foreign concept which was most certainly a lot more expensive
than incumbent Layer 2-based Access models.

I feel this has since changed with the current offerings from Cisco,
Juniper and Brocade. The problem now is how to scale the low-speed port
density up, as well as add 10Gbps port density, without increasing the
cost or size of the platforms.

Mark.

Re: 1GE L3 aggregation

[Mark Tinka]

On 16/Jun/16 21:36, Baldur Norddahl wrote:

> Hi
>
> If I need to speak BGP with a customer that only has 1G I will simply make
> a MPLS L2VPN to one of my edge routers. We use the ZTE 5952E switch with
> 48x 1G plus 4x 10G for the L2VPN end point. If that is not enough the ZTE
> 8900 platform will provide a ton of ports that can do MPLS.
>
> The tunnel is automatically redundant and will promote link down events, so
> there is not really any downside to doing it this way on low bandwidth
> peers.

Personally (and at work), I stay away from such topologies. Centralizing
IP connectivity like this may seem sexy and cheap in the start, but it
has serious scaling and operational issues down the line, IMHO.

We push IP/MPLS all the way into the Metro-E Access using a team of
Cisco ASR920's and ME3600X's. The value of being able to instantiate an
IP service or BGP session directly in the Metro-E Access simplifies
network operations a great deal for us. Needless to say, not having to
deal with eBGP Multi-Hop drama does not hurt.

Centralizing is just horrible, but that's just me. The goal is to make
all these unreliable boxes work together to offer a reliable service to
your customers, so making them too inter-dependent on each other has the
potential to that away in the long run.

Mark.

Re: RPKI implementation

[Mark Tinka]

On 16/Jun/16 16:38, Randy Bush wrote:

> i am aware of that.  my point was that cache purge default might better
> be set to cache refresh interval than 60 secs.

I would agree with (and in fact, prefer) this protocol.

Mark.

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Pete Mundy]

Our DC (granted, not in the US!) charges a one-off fee of $75 to install the 
XC, which includes the cable too. Terminated to a 1U patch panel at TOR. Only 
one end pays the 1-off charge (the customer that requested the order). No 
ongoing charges. No one else rummages in the overheads other than the DC 
operator themselves.

I guess perhaps what we have is a bit of a luxury, but it's part of the reason 
we chose that DC over others available in the same vicinity, so it did give 
them a competitive advantage :)

Pete


> On 18/06/2016, at 5:31 am, Seth Mattinen  wrote:
> 
> I would expect some kind of MRC if it has any SLA, service, or support 
> attached. Or someone manages it and protects the infrastructure and enforces 
> the rules of the facility. Or the facility uses that money to maintain the 
> MMR. If it's a free for all where anyone can accidentally unplug it or cut it 
> or rummage around in the overhead causing damage then free is fine. What I 
> don't accept is variable pricing depending on what the xconnect is carrying 
> or what it's for. I don't buy into the thought process that an xconnect is 
> more expensive if it's carrying 10GbE vs. GigE when they're both SMF.
> 
> ~Seth