Re: Fiber Costs [Was: Re: SoCal FIOS outage(?) / static IP readdressing]

[Jared Mauch]

> On Jan 10, 2017, at 10:21 AM, Fletcher Kittredge  wrote:
> 
> Numbers for building fiber optic systems are out there if you do the
> research. Joining the FTTH Council is a good start. One thing to recognise
> is that the numbers vary widely based on what is being built and where it
> is being built. There are large regional, technology, and product
> variations. Verizon has economies of scale few can match.
> 
> Having said that, some of the numbers listed are unrecognizably low.

Labor can vary quite widely based on project, distance and environment.  What 
is $5/foot in a rural location can be $150 or more in an urban environment.  
It’s not uncommon to see pricing around $12/foot before engineering and other 
permitting work. 

If you’re in an environment that has favorable permitting process and can work 
with an existing insured contractor, $5/foot is attainable.  You are still up 
against 600-800 feet per day in favorable soil, which can easily turn to 200 
should there be a rock or complex utility work involved.

I’ll say depending on your project, you can start with the big-dreamer 
communities out there, or you go the other way and talk to folks that are doing 
it on the ground in your local area.  I’ve talked to people about pole attach 
as well as underground.  You can see costs as low as 10k per mile on poles, but 
that’s the really low end.

All numbers I’ve mentioned are for Michigan in the communities around my home 
as well as outlying areas.  If you’re around my area and want to talk costs and 
the projects, a private e-mail is welcome.

If you’re in Maine, that granite rock is really tough, the hemlocks rot and 
fall more often than the birch, etc.  Those risks make the situation tougher, 
and the population north of Bangor/Orono really thins out, but at least the 
speed limit is higher on 95 now.  :-)

- Jared

Re: Fiber Costs [Was: Re: SoCal FIOS outage(?) / static IP readdressing]

[Fletcher Kittredge]

Numbers for building fiber optic systems are out there if you do the
research. Joining the FTTH Council is a good start. One thing to recognise
is that the numbers vary widely based on what is being built and where it
is being built. There are large regional, technology, and product
variations. Verizon has economies of scale few can match.

Having said that, some of the numbers listed are unrecognizably low.



On Tue, Jan 10, 2017 at 8:32 AM, Leo Bicknell  wrote:

>
> I don't know about the rest of the list, but I find these numbers
> fascinating.  There's probably not that many people who are allowed
> to share them, but if more could I think that would be educational
> for a lot of folks.
>
> In a message written on Wed, Jan 04, 2017 at 08:37:19AM -0500, Jared Mauch
> wrote:
> > I’ve been thinking of the same in my underserved area.  Labor is $5/foot
> here and despite friends and colleagues telling me to move, it seems I have
> a sub-60 month ROI (and sub-year for some areas I’ve modeled with modest
> uptake rates of 15-20% where the other options are fixed wireless, Cellular
> data or dial).
>
> In a message written on Wed, Jan 04, 2017 at 01:50:48PM +, Luke
> Guillory wrote:
> > Our model is 15k a mile all in, this is  for aerial  not underground for
> our HFC/Coax builds. A partner of ours models their underground fiber
> builds at 30k a mile.
>
> In a message written on Wed, Jan 04, 2017 at 09:08:51AM -0500, Shawn L
> wrote:
> > Depending on the area and conditions (rock, etc).  We're seeing
> >
> > $4 /foot Aerial
> > $5-$7 /foot direct bury
> > $10 - $14 /foot directional bore
>
> --
> Leo Bicknell - bickn...@ufp.org
> PGP keys at http://www.ufp.org/~bicknell/
>



-- 
Fletcher Kittredge
GWI
207-602-1134
www.gwi.net

Re: Soliciting your opinions on Internet routing: A survey on BGP convergence

[Laurent Vanbever]

Hi Joel,

> On 10 Jan 2017, at 06:51, joel jaeggli  wrote:
> 
> On 1/9/17 2:56 PM, Laurent Vanbever wrote:
>> Hi NANOG,
>> 
>> We often read that the Internet (i.e. BGP) is "slow to converge". But how 
>> slow
>> is it really? Do you care anyway? And can we (researchers) do anything about 
>> it?
>> Please help us out to find out by answering our short anonymous survey 
>> (<10 minutes).
>> 
>> Survey URL: https://goo.gl/forms/JZd2CK0EFpCk0c272 
>> 
>> 
>> 
>> ** Background:
>> 
>> While existing fast-reroute mechanisms enable sub-second convergence upon 
>> local outages (planned or not), they do not apply to remote outages 
>> happening 
>> further away from your AS as their detection and protection mechanisms only 
>> work locally.
>> 
>> Remote outages therefore mandate a "BGP-only" convergence which tends to be
>> slow, as long streams of BGP UPDATEs (containing up to 100,000s of them) must
>> be propagated router-by-router. Our initial measurements indicate that it can
>> take state-of-the-art BGP routers dozens of seconds to process and propagate
>> these large streams of BGP UPDATEs. During this time, traffic for important
>> destinations can be lost.
> 
> One of the phenomena that is relatively easy to observe by withdrawing a
> prefix entirely is the convergence towards longer and longer AS paths
> until the route disappears entirely. that is providers that are further
> away will remain advertising the route and in the interim their
> neighbors  will ingest the available path will  until they too process
> the withdraw. it can take a comically long time (like 5 minutes)  to see
> the prefix ultimately disappear from the internet. When withdrawing a
> prefix from a peer with which you have a single adjacency this can
> easily happens in miniature.

Thanks! Yes, definitely. This relates to the issue Baldur was raising in which 
a less-preferred prefix (or not prefix at all in your case) has to take over a 
more preferred one. That case is definitely bad for BGP convergence. 

Our survey/study is more geared towards cases where there is diversity 
available (alternates paths are there and at least partially visible). We are 
especially interested in finding out whether, even when you take all the 
precautionary measures required by the book, long BGP convergence can still 
bite you and… whether we can do anything about it.


Laurent

PS: 

Thanks so much to the 21 operators who have answered already! If you haven’t so 
already, please help us out to find out about troublesome BGP convergence by 
answering our short anonymous survey  (<10 minutes): 
https://goo.gl/forms/JZd2CK0EFpCk0c272 

Re: Soliciting your opinions on Internet routing: A survey on BGP convergence

[Laurent Vanbever]

Dear Baldur,

> I find that the type of outage that affects our network the most is neither 
> of the two options you describe. As is probably typical for smaller networks, 
> we do not have redundant uplinks to all of our transits. If a transit link 
> goes, for example because we had to reboot a router, traffic is supposed to 
> reroute to the remaining transit links. Internally our network handles this 
> fairly fast for egress traffic.
> 
> However the problem is the ingress traffic - it can be 5 to 15 minutes before 
> everything has settled down. This is the time before everyone else on the 
> internet has processed that they will have to switch to your alternate 
> transit.

Thanks a lot for your input. Indeed, that case is a bit special. I’d say it is 
a kind of remote outage that remote ASes experience towards your prefix and, as 
such, requires a "BGP-only” convergence. I guess if your prefixes going via 
alternate transit are not visible at all prior to the switch (and I guess not), 
this is a kind of “extreme” convergence where routes have to be 
withdrawn/updated Internet-wide. This reminds me of the paper by Craig Labovitz 
et al. 
(http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-5-2.pdf 
) 
which I think classify these events as Tlong ("An active route with a short 
ASPath is implicitly replaced with a new route possessing a longer ASPath. This 
represents both a route failure and failover”). And indeed, these are the 
second slowest just before the withdraw of a prefix Internet-wide.

You’re right that our survey targets more the case in which large bursts of 
UPDATEs/WITHDRAWs are exchanged. I guess a parallel case to the one you mention 
could be that your prime transit performs a planned maintenance (or experiences 
a failure) that triggers the sending of WITHDRAWs for your prefixes out.

> The only solution I know of is to have redundant links to all transits. Going 
> forward I will make sure we have this because it is a huge disadvantage not 
> being able to take a router out of service without causing downtime for all 
> users. Not to mention that a router crash or link failure that should have 
> taken seconds at most to reroute, but instead causes at least 5 minutes of 
> unstable internet.

Maybe you could advertise better routes (i.e., with shorter AS-PATHs/longer 
prefixes) via the alternate transit prior to the take down? Ideally, if you 
could somehow make your primary transit switch to use an alternate transit 
prior to the maintenance (maybe with a special community?), you could 
completely avoid a disruption. This would go into the direction of minimizing 
the amount of WITHDRAWs in favor of UPDATEs. But, of course, this would only 
work in the case of planned maintenance.

We would definitely welcome more input on the convergence issue you face!

Best,
Laurent

Re: Soliciting your opinions on Internet routing: A survey on BGP convergence

[Mike Jones]

On 10 January 2017 at 19:58, Job Snijders  wrote:
> On Tue, Jan 10, 2017 at 03:51:04AM +0100, Baldur Norddahl wrote:
>> If a transit link goes, for example because we had to reboot a router,
>> traffic is supposed to reroute to the remaining transit links.
>> Internally our network handles this fairly fast for egress traffic.
>>
>> However the problem is the ingress traffic - it can be 5 to 15 minutes
>> before everything has settled down. This is the time before everyone
>> else on the internet has processed that they will have to switch to
>> your alternate transit.
>>
>> The only solution I know of is to have redundant links to all transits.
>
> Alternatively, if you reboot a router, perhaps you could first shutdown
> the eBGP sessions, then wait 5 to 10 minutes for the traffic to drain
> away (should be visible in your NMS stats), and then proceed with the
> maintenance?
>
> Of course this only works for planned reboots, not suprise reboots.
>
> Kind regards,
>
> Job

If I tear down my eBGP sessions the upstream router withdraws the
route and the traffic just stops. Are your upstreams propagating
withdraws without actually updating their own routing tables?

I believe the simple explanation of the problem can be seen by firing
up an inbound mtr from a distant network then withdrawing the route
from the path it is taking. It should show either destination
unreachable or a routing loop which "retreats" (under the right
circumstances I have observed it distinctly move 1 hop at a time)
until it finds an alternate path.

My observed convergence times for a single withdraw are however in the
sub-10 second range, to get all the networks in the original path
pointing at a new one. My view on the problem is that if you are
failing over frequently enough for a customer to notice and report it,
you have bigger problems than convergence times.

- Mike Jones

Re: Soliciting your opinions on Internet routing: A survey on BGP convergence

[Jared Mauch]

> On Jan 10, 2017, at 3:14 PM, Hugo Slabbert  wrote:
> 
> 
> On Tue 2017-Jan-10 20:58:02 +0100, Job Snijders  wrote:
> 
>> On Tue, Jan 10, 2017 at 03:51:04AM +0100, Baldur Norddahl wrote:
>>> If a transit link goes, for example because we had to reboot a router,
>>> traffic is supposed to reroute to the remaining transit links.
>>> Internally our network handles this fairly fast for egress traffic.
>>> 
>>> However the problem is the ingress traffic - it can be 5 to 15 minutes
>>> before everything has settled down. This is the time before everyone
>>> else on the internet has processed that they will have to switch to
>>> your alternate transit.
>>> 
>>> The only solution I know of is to have redundant links to all transits.
>> 
>> Alternatively, if you reboot a router, perhaps you could first shutdown
>> the eBGP sessions, then wait 5 to 10 minutes for the traffic to drain
>> away (should be visible in your NMS stats), and then proceed with the
>> maintenance?
>> 
>> Of course this only works for planned reboots, not suprise reboots.
> 
> ...or link failures.

One other comment:

there has been a long history of poorly behaving BGP stacks that would
take quite some time to hunt through the paths.  While this can still
occur with people with nearing ancient software and hardware still in-use,
many of the modern software/hardware options enable things like BGP-PIC
(in your survey) by default.

Many of these options you document as best practices like path mtu discovery
are well known fixes for networks, as well as using jumbo mtu internally to
obtain 9k+ mss for high performance TCP.  Vendors have not always chosen to
enable the TCP options by default like the protocols have, eg: BGP-PIC and
like Jakob’s response, tout other solutions vs fixing the TCP stack first.

Many of these performances were documented in 2002 and are considered best
practices by many networks, but due to their obscure knobs may not be
widely deployed as a result, or seen as risky to configure.  (We had a
vendor panic when we discovered a bug in their TCP-SACK code, they were
almost frozen in not fixing the code because touching TCP felt dangerous
and there was an inadequate testing culture around something seen as ‘stable’).

here’s the presentation from IETF 53, I don’t see it in the proceedings handily:

http://morse.colorado.edu/~epperson/courses/routing-protocols/handouts/bgp_scalability_IETF.ppt

- Jared

Re: Soliciting your opinions on Internet routing: A survey on BGP convergence

[Hugo Slabbert]

On Tue 2017-Jan-10 20:58:02 +0100, Job Snijders  wrote:


On Tue, Jan 10, 2017 at 03:51:04AM +0100, Baldur Norddahl wrote:

If a transit link goes, for example because we had to reboot a router,
traffic is supposed to reroute to the remaining transit links.
Internally our network handles this fairly fast for egress traffic.

However the problem is the ingress traffic - it can be 5 to 15 minutes
before everything has settled down. This is the time before everyone
else on the internet has processed that they will have to switch to
your alternate transit.

The only solution I know of is to have redundant links to all transits.


Alternatively, if you reboot a router, perhaps you could first shutdown
the eBGP sessions, then wait 5 to 10 minutes for the traffic to drain
away (should be visible in your NMS stats), and then proceed with the
maintenance?

Of course this only works for planned reboots, not suprise reboots.


...or link failures.



Kind regards,

Job


--
Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com
pgp key: B178313E   | also on Signal


signature.asc
Description: Digital signature

Re: Soliciting your opinions on Internet routing: A survey on BGP convergence

[Job Snijders]

On Tue, Jan 10, 2017 at 03:51:04AM +0100, Baldur Norddahl wrote:
> If a transit link goes, for example because we had to reboot a router,
> traffic is supposed to reroute to the remaining transit links.
> Internally our network handles this fairly fast for egress traffic.
>
> However the problem is the ingress traffic - it can be 5 to 15 minutes
> before everything has settled down. This is the time before everyone
> else on the internet has processed that they will have to switch to
> your alternate transit.
>
> The only solution I know of is to have redundant links to all transits.

Alternatively, if you reboot a router, perhaps you could first shutdown
the eBGP sessions, then wait 5 to 10 minutes for the traffic to drain
away (should be visible in your NMS stats), and then proceed with the
maintenance?

Of course this only works for planned reboots, not suprise reboots.

Kind regards,

Job

RE: Soliciting your opinions on Internet routing: A survey on BGP convergence

[Jakob Heitz (jheitz)]

Hi Baldur,

Have you tried graceful shutdown?
You need redundant links, but not to the same transit.
https://tools.ietf.org/html/draft-ietf-grow-bgp-gshut-06
This draft is expired, but it is actually implemented by several vendors.

I implemented this.
http://www.slideshare.net/bduvivie/bgp-graceful-shutdown-ios-xr
I added an option to configure AS-path prepends in case the gshut community was 
not supported by peers.

Thanks,
Jakob.


> Date: Tue, 10 Jan 2017 03:51:04 +0100
> From: Baldur Norddahl 
> 
> Hello
> 
> I find that the type of outage that affects our network the most is
> neither of the two options you describe. As is probably typical for
> smaller networks, we do not have redundant uplinks to all of our
> transits. If a transit link goes, for example because we had to reboot a
> router, traffic is supposed to reroute to the remaining transit links.
> Internally our network handles this fairly fast for egress traffic.
> 
> However the problem is the ingress traffic - it can be 5 to 15 minutes
> before everything has settled down. This is the time before everyone
> else on the internet has processed that they will have to switch to your
> alternate transit.
> 
> The only solution I know of is to have redundant links to all transits.
> Going forward I will make sure we have this because it is a huge
> disadvantage not being able to take a router out of service without
> causing downtime for all users. Not to mention that a router crash or
> link failure that should have taken seconds at most to reroute, but
> instead causes at least 5 minutes of unstable internet.
> 
> Regards,
> 
> Baldur

Fiber Costs [Was: Re: SoCal FIOS outage(?) / static IP readdressing]

[Leo Bicknell]

I don't know about the rest of the list, but I find these numbers
fascinating.  There's probably not that many people who are allowed
to share them, but if more could I think that would be educational
for a lot of folks.

In a message written on Wed, Jan 04, 2017 at 08:37:19AM -0500, Jared Mauch 
wrote:
> I’ve been thinking of the same in my underserved area.  Labor is $5/foot here 
> and despite friends and colleagues telling me to move, it seems I have a 
> sub-60 month ROI (and sub-year for some areas I’ve modeled with modest uptake 
> rates of 15-20% where the other options are fixed wireless, Cellular data or 
> dial).

In a message written on Wed, Jan 04, 2017 at 01:50:48PM +, Luke Guillory 
wrote:
> Our model is 15k a mile all in, this is  for aerial  not underground for our 
> HFC/Coax builds. A partner of ours models their underground fiber builds at 
> 30k a mile.

In a message written on Wed, Jan 04, 2017 at 09:08:51AM -0500, Shawn L wrote:
> Depending on the area and conditions (rock, etc).  We're seeing
>  
> $4 /foot Aerial
> $5-$7 /foot direct bury
> $10 - $14 /foot directional bore

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


signature.asc
Description: PGP signature

Re: Some standard of showing "this is how my network works"?

[Wolfgang Tremmel]

> On 9 Jan 2017, at 18:19, Mike Lund  wrote:
> 
> Is this such a thing anywhere?

Try RIPE-Stat:
https://stat.ripe.net

best regards
Wolfgang


-- 
Wolfgang Tremmel 

Phone +49 69 1730902 26 | Fax +49 69 4056 2716 | Mobile +49 171 8600 816 | 
wolfgang.trem...@de-cix.net
Geschaeftsfuehrer Harald A. Summa | Registergericht AG Köln HRB 51135
DE-CIX Management GmbH | Lindleystrasse 12 | 60314 Frankfurt am Main | Germany 
| www.de-cix.net

IETF RTGWG interim meeting - Existing problems for routing in the large Data Centers and potential solutions

[Jeff Tantsura]

/RTGWG chair hat on

 

Dear NANOG,

 

For those, who might be interested, on January 25 we (IETF Routing Area RTGWG) 
will be having an interim online meeting, dedicated to Existing problems for 
routing in the large Data Centers and potential solutions. Presenters will be 
describing (15 minutes per presentation) problems in the space, followed by 
potential solutions

 

Please join us.

 

Presenters:

Problem statements:

Peter Lapukhov - FB

Dmitry Afanasiev - YANDEX

Russ White/ Shawn Zandi - LinkedIn

 

Solutions proposed:

Keyur Patel, Arrcus  - Shortest Path Routing Extensions for BGP Protocol, 
draft-keyupate-idr-bgp-spf

Naiming Shen , Cisco  - IS-IS Routing for Spine-Leaf Topology, 
draft-shen-isis-spine-leaf-ext

Tony Przygienda, Juniper - RIFT: Routing in Fat Trees, draft-przygienda-rift  
(draft to be published later this week)

Bhumip Khasnabish, ZTE - Generic Fault-avoidance Routing Protocol for Data 
Center Networks, draft-sl-rtgwg-far-dcn

 

Cheers,

Jeff

 

 

From: Routing Area Working Group 
Reply-To: rtgwg-chairs 
Date: Tuesday, December 20, 2016 at 11:47
To: 
Subject: WebEx meeting invitation: Existing problems for routing in the large 
Data Centers and potentail solutions
Resent-From: 
Resent-To: , Chris Bowers 
Resent-Date: Tue, 20 Dec 2016 11:47:19 -0800 (PST)

 

Hello, 
Routing Area Working Group invites you to join this WebEx meeting. 

 

 

 

Existing problems for routing in the large Data Centers and potential solutions 
Wednesday, January 25, 2017 
9:00 am  |  Pacific Standard Time (San Francisco, GMT-08:00)  |  2 hrs 30 mins 

 

Meeting number (access code): 649 161 321 

 

Meeting password: px3Gk22M

 

 

 

Add to Calendar 
When it's time, join the meeting.

 

 

 

Join by phone
1-877-668-4493 Call-in toll free number (US/Canada)
1-650-479-3208 Call-in toll number (US/Canada)
Toll-free calling restrictions

 

 

 

Can't join the meeting? 

 

 

 

IMPORTANT NOTICE: Please note that this WebEx service allows audio and other 
information sent during the session to be recorded, which may be discoverable 
in a legal matter. By joining this session, you automatically consent to such 
recordings. If you do not consent to being recorded, discuss your concerns with 
the host or do not join the session.

 



WebEx_Meeting.ics
Description: Binary data