Re: DOs and DONTs for small ISP

[Baldur Norddahl]

For some reason our network looks nothing like that. He either needs to
define what a PoP is or define a module that does not have expensive gear
like two redundant routers at every location.

Regards

Baldur

tir. 4. jun. 2019 15.46 skrev Mehmet Akcin :

> This Gem is fantastic by the way,
>
>
> https://nsrc.org/workshops/2015/apricot2015/raw-attachment/wiki/Track1Agenda/01-ISP-Network-Design.pdf
>
> On Tue, Jun 4, 2019 at 5:57 AM Warren Kumari  wrote:
>
>> On Mon, Jun 3, 2019 at 11:34 PM Brandon Martin 
>> wrote:
>> >
>> > On 6/3/19 9:56 AM, Jon Lewis wrote:
>> > > 3) Don't advertise one transit provider's routes to another.  Each
>> should
>> > > be filtering your routes, but you never know.  Come up with, and
>> use
>> > > BGP communities to control route propagation.  As you grow, it
>> sucks
>> > > having to update prefix-list filters in multiple places every time
>> > > something changes...like a new customer with their own IPs.
>> >
>> > To reiterate all this, FILTER EVERYTHING.
>> >
>> > To start with, explicitly specify in a route-map or similar everything
>> > you want to advertise.  I usually create a separate route-map for each
>> > transit/peer and include what I want to advertise via prefix lists (for
>> > my IP space) and/or communities (for downstream BGP-speaking customers
>> > if anticipated).
>>
>> I think a related *principle* is: "Build everything as though you are
>> expecting to scale."
>>
>> This doesn't mean "spend lots of money to buy huge
>> [routers|servers|commercial software|], but rather "when you plan
>> your addressing structure and routing policies and monitoring and
>> device config generation and... keep the in mind the question "If this
>> suddenly takes off, and I hire N more people to run this, can I
>> explain to them how it works? Do I have documentation I can point them
>> at or is it stuck in my head / on the devices? If I need to add
>> another M customers in the next month, can I do that easily?".
>>
>> This is related to the FILTER EVERYTHING -- when you turn up a new
>> customer / peer / transit / whatever, you shouldn't be sitting around
>> trying to figure out how you will write their route-map /
>> policy-options -- this leads to weird one-offs, and quick hacks.
>> Instead you should have policies already largely designed and simply
>> plug in their prefixes (or, better yet, use bgpq3 or similar to build
>> and populate these). Obviously there will be some cases where a new
>> connection does require some special handling, but that *should* just
>> be a plugin/chain in an existing policy-statement. Related to this is
>> how you end up naming things -- I recently found 9 variants of
>> firewall-filters which basically do:
>>
>> filter ACCEPT {
>>term ACCEPT {
>> then accept;
>>   }
>> }
>> named things like: ACCEPT, ACEPT, Accept, Allow, Permit_all,
>> AcceptAll, dontdrop [0].
>>
>> Obviously, there is a tension in the "design for scale" - while it
>> would be great to design a complete automation system so that
>> everything from installing a new customer to a new sites is simply
>> typing 'make ' and having everything pull from a database, at
>> some point you will need to actually build a network, or you'll never
>> have customers :-) Just keep in mind that "Am I building myself into a
>> corner here?". E.g it only takes 10 or 15 minutes to install something
>> like NetBox to keep track of addresses (and prefixes and racks and
>> connections and ...) -- stuffing this in a spreadsheet might save you
>> a few minutes *now*, but will this scale? Can $new_person easily
>> figure it out?
>>
>>
>> W
>> [0]: My personal favorite is:
>> filter Accept_All {
>> term Accept {
>> then {
>> count dropped;
>> reject;
>> }
>> }
>> term filter_ {
>> from {
>> prefix-list {
>> ;
>>}
>> }
>> then accept;
>> }
>> term NEXT {
>> then log;
>> }
>> }
>>
>> Presumably this all made sense to 
>> when they stuck it in at 3AM to deal with some crazy issue, but...
>>
>>
>>
>> >
>> > When you turn on the session, check what you're squawking AND WHAT
>> > YOU'RE FILTERING.  You shouldn't be filtering anything you don't expect.
>> >   Belt + suspenders.
>> >
>> > The same goes for anything you accept.  Obviously for a blended full
>> > transit BGP edge router, you're probably going to accept almost
>> > everything.  But if you only want default + on-net, try to filter using
>> > communities from the peer, etc.  Again, right when you turn on the
>> > session, "sh ip bgp ... filtered" of whatever's equivalent on your
>> > platform.  If you're filtering something you don't expect to be
>> > receiving at all, figure out where the misunderstanding or
>> > misconfiguration lies.
>> >
>> > And of course it goes without saying that, if you've got BGP speaking
>> > customers, you filter the heck out of them.  Use ROAs and/or RPKI if 

Re: CenturyLink/Level3 feedback

[Dovid Bender]

If the FCC has their way the only place you will see the PSTN in history
books. I can only hope that the same happens to faxing.


On Wed, Jun 5, 2019 at 4:37 PM Mike Hammett  wrote:

> It's amazing how inconsistent the PSTN is.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
> 
> --
> *From: *"Dovid Bender" 
> *To: *"Larry Brower" 
> *Cc: *"nanog" 
> *Sent: *Wednesday, June 5, 2019 3:31:36 PM
> *Subject: *Re: CenturyLink/Level3 feedback
>
> For voice there are so many IP options I don't know why anyone even messes
> with the old school carriers. About 4 years ago we signed up for L3 VoIP.
> We sent calls to France and the callerID didn't make it. We opened a ticket
> we were told callerID wasn't guaranteed on international calls. That was
> the day we canceled our service and asked for a refund. I am sometimes
> amazed how some of these carriers still have customers signing up.
>
>
>
> On Wed, Jun 5, 2019 at 8:50 AM Brower, Larry <
> larry.bro...@aramcoservices.com> wrote:
>
>> Mehmet,
>>
>>
>>
>> Speaking strictly on their voice product, service has gone a bit downhill
>> since the merger.
>>
>>
>>
>> We never had problems with Level3 before the merger.
>>
>>
>>
>> After Centurylink took over we started experiencing problems.
>>
>>
>>
>> Just a couple of examples:
>>
>>
>>
>> We waited months just to turn up a simple PRI. The PRI was sent back to
>> design several times and then when it finally was turned up it isn’t
>> working properly. The CL techs who were formally L3 express nothing but
>> frustration with dealing with CL following the merger. Complaints to the
>> account manager are met with just apologies and delays.
>>
>>
>>
>> International call routing has become unreliable. In the last month alone
>> we have had to create several service requests related to call failures.
>> The result after anywhere from a couple hours to a day is just hey we
>> rerouted try again. Then it works for a couple days and back to call
>> failures and intercept messages.
>>
>>
>>
>> I’ve already been asked if we should drop CenturyLink as the carrier and
>> go back to using someone like AT
>>
>>
>>
>> Never had any of these issue when it was Level3.
>>
>>
>>
>> Regards,
>>
>>
>>
>> Larry Brower, CCNP Collaboration, SSCA, RHCSA, CCDA, CCNA
>>
>> Communications Technician | Unified Communications Group
>>
>>
>>
>> Aramco Services Company
>>
>> *Office:  713.432.4516 | Mobile:  832.570.5416 *
>>
>> *larry.bro...@aramcoservices.com* 
>>
>>
>> This email has been classified as: *General Use* by *Brower, Larry *on 
>> *Wednesday,
>> June 5, 2019*
>>
>>
>>
>> *From:* NANOG  *On Behalf Of *Mehmet Akcin
>> *Sent:* Tuesday, June 4, 2019 9:31 AM
>> *To:* nanog 
>> *Subject:* CenturyLink/Level3 feedback
>>
>>
>>
>> *EXTERNAL: This email came from the Internet. Report this message to
>> ascsuspiciousem...@aramcoservices.com
>>  as suspicious if it contains any
>> suspicious content.*
>>
>> hi there,
>>
>>
>>
>> Just a general high-level question about Centurylink/Level3 post-merger,
>> how is your overall experience with CenturyLink? if you could be sitting
>> with the CEO of the company what is one thing you would ask him to fix?
>>
>>
>>
>> please keep it high level and general. i intend to pass these to him and
>> his team in an upcoming meeting.
>>
>>
>>
>> Mehmet
>>
>
>

Re: CenturyLink/Level3 feedback

[Mike Hammett]

It's amazing how inconsistent the PSTN is. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Dovid Bender"  
To: "Larry Brower"  
Cc: "nanog"  
Sent: Wednesday, June 5, 2019 3:31:36 PM 
Subject: Re: CenturyLink/Level3 feedback 


For voice there are so many IP options I don't know why anyone even messes with 
the old school carriers. About 4 years ago we signed up for L3 VoIP. We sent 
calls to France and the callerID didn't make it. We opened a ticket we were 
told callerID wasn't guaranteed on international calls. That was the day we 
canceled our service and asked for a refund. I am sometimes amazed how some of 
these carriers still have customers signing up. 






On Wed, Jun 5, 2019 at 8:50 AM Brower, Larry < larry.bro...@aramcoservices.com 
> wrote: 





Mehmet, 

Speaking strictly on their voice product, service has gone a bit downhill since 
the merger. 

We never had problems with Level3 before the merger. 

After Centurylink took over we started experiencing problems. 

Just a couple of examples: 

We waited months just to turn up a simple PRI. The PRI was sent back to design 
several times and then when it finally was turned up it isn’t working properly. 
The CL techs who were formally L3 express nothing but frustration with dealing 
with CL following the merger. Complaints to the account manager are met with 
just apologies and delays. 

International call routing has become unreliable. In the last month alone we 
have had to create several service requests related to call failures. The 
result after anywhere from a couple hours to a day is just hey we rerouted try 
again. Then it works for a couple days and back to call failures and intercept 
messages. 

I’ve already been asked if we should drop CenturyLink as the carrier and go 
back to using someone like AT 

Never had any of these issue when it was Level3. 

Regards, 


Larry Brower, CCNP Collaboration, SSCA, RHCSA, CCDA, CCNA 
Communications Technician | Unified Communications Group 

Aramco Services Company 
Office: 713.432.4516 | Mobile: 832.570.5416 
larry.bro...@aramcoservices.com 

This email has been classified as: General Use by Brower, Larry on Wednesday, 
June 5, 2019 



From: NANOG < nanog-boun...@nanog.org > On Behalf Of Mehmet Akcin 
Sent: Tuesday, June 4, 2019 9:31 AM 
To: nanog < nanog@nanog.org > 
Subject: CenturyLink/Level3 feedback 

EXTERNAL: This email came from the Internet. Report this message to 
ascsuspiciousem...@aramcoservices.com as suspicious if it contains any 
suspicious content. 


hi there, 



Just a general high-level question about Centurylink/Level3 post-merger, how is 
your overall experience with CenturyLink? if you could be sitting with the CEO 
of the company what is one thing you would ask him to fix? 



please keep it high level and general. i intend to pass these to him and his 
team in an upcoming meeting. 



Mehmet 

Re: CenturyLink/Level3 feedback

[Dovid Bender]

For voice there are so many IP options I don't know why anyone even messes
with the old school carriers. About 4 years ago we signed up for L3 VoIP.
We sent calls to France and the callerID didn't make it. We opened a ticket
we were told callerID wasn't guaranteed on international calls. That was
the day we canceled our service and asked for a refund. I am sometimes
amazed how some of these carriers still have customers signing up.



On Wed, Jun 5, 2019 at 8:50 AM Brower, Larry <
larry.bro...@aramcoservices.com> wrote:

> Mehmet,
>
>
>
> Speaking strictly on their voice product, service has gone a bit downhill
> since the merger.
>
>
>
> We never had problems with Level3 before the merger.
>
>
>
> After Centurylink took over we started experiencing problems.
>
>
>
> Just a couple of examples:
>
>
>
> We waited months just to turn up a simple PRI. The PRI was sent back to
> design several times and then when it finally was turned up it isn’t
> working properly. The CL techs who were formally L3 express nothing but
> frustration with dealing with CL following the merger. Complaints to the
> account manager are met with just apologies and delays.
>
>
>
> International call routing has become unreliable. In the last month alone
> we have had to create several service requests related to call failures.
> The result after anywhere from a couple hours to a day is just hey we
> rerouted try again. Then it works for a couple days and back to call
> failures and intercept messages.
>
>
>
> I’ve already been asked if we should drop CenturyLink as the carrier and
> go back to using someone like AT
>
>
>
> Never had any of these issue when it was Level3.
>
>
>
> Regards,
>
>
>
> Larry Brower, CCNP Collaboration, SSCA, RHCSA, CCDA, CCNA
>
> Communications Technician | Unified Communications Group
>
>
>
> Aramco Services Company
>
> *Office:  713.432.4516 | Mobile:  832.570.5416 *
>
> *larry.bro...@aramcoservices.com* 
>
>
> This email has been classified as: *General Use* by *Brower, Larry *on 
> *Wednesday,
> June 5, 2019*
>
>
>
> *From:* NANOG  *On Behalf Of *Mehmet Akcin
> *Sent:* Tuesday, June 4, 2019 9:31 AM
> *To:* nanog 
> *Subject:* CenturyLink/Level3 feedback
>
>
>
> *EXTERNAL: This email came from the Internet. Report this message to
> ascsuspiciousem...@aramcoservices.com
>  as suspicious if it contains any
> suspicious content.*
>
> hi there,
>
>
>
> Just a general high-level question about Centurylink/Level3 post-merger,
> how is your overall experience with CenturyLink? if you could be sitting
> with the CEO of the company what is one thing you would ask him to fix?
>
>
>
> please keep it high level and general. i intend to pass these to him and
> his team in an upcoming meeting.
>
>
>
> Mehmet
>

Re: DOs and DONTs for small ISP

[William Herrin]

On Wed, Jun 5, 2019 at 5:44 AM William Waites  wrote:
> It's not enough to have monitoring and a ticket system. You need to pay
> attention to them, care for them and feed them. I can't count the number
> of ticket systems full of ancient and irrelevant things or monitoring
> systems that people have forgotten about or don't know how to add new
> stuff to. Even the cycle of,

Some points to consider when monitoring your network:

1. Beware early automation. If you write a generator to go and monitor all
your stuff without addressing how operators will change things one-off
(which is hard to design well) the other operators will find the monitoring
system unusable. Which means they won't update it when stuff is added and
changed. Making it quickly useless.

2. Careful aggregating alarms. That big green or red light is useless. The
operator has to be able to start with the alarm and immediately trace back
to exactly what tests and results bubbled up in to the aggregate and from
there to the malfunctioning component. If you lose this information during
the aggregation process, you're just producing noise.

2. Every alarm must be actionable. When the light goes red, what -exactly-
do you want the operator to do as a result? Don't create an alarm until you
can offer a detailed and specific answer, and link that answer to the alarm
so the operator doesn't have to hunt for it.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: CenturyLink/Level3 feedback

[Mike Hammett]

Anything more than a week for things not requiring last mile construction is 
ridiculous. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "JASON BOTHE via NANOG"  
To: "Mehmet Akcin"  
Cc: "nanog"  
Sent: Wednesday, June 5, 2019 9:56:14 AM 
Subject: Re: CenturyLink/Level3 feedback 

It’s taking over a year to get waves turned up in EU. I’m currently willing to 
wager on what comes up first, them or amazon peering (that’s taking just as 
long). After the merger, we have seen Level3 slide into the CL abyss becoming a 
pain to deal with. Pricing and ordering has been outsourced we’ve been told and 
decisions are no longer at a regional level. Frustrating at best. 

> On Jun 4, 2019, at 09:30, Mehmet Akcin  wrote: 
> 
> hi there, 
> 
> Just a general high-level question about Centurylink/Level3 post-merger, how 
> is your overall experience with CenturyLink? if you could be sitting with the 
> CEO of the company what is one thing you would ask him to fix? 
> 
> please keep it high level and general. i intend to pass these to him and his 
> team in an upcoming meeting. 
> 
> Mehmet 

Re: CenturyLink/Level3 feedback

[JASON BOTHE via NANOG]

It’s taking over a year to get waves turned up in EU. I’m currently willing to 
wager on what comes up first, them or amazon peering (that’s taking just as 
long). After the merger, we have seen Level3 slide into the CL abyss becoming a 
pain to deal with. Pricing and ordering has been outsourced we’ve been told and 
decisions are no longer at a regional level. Frustrating at best. 

> On Jun 4, 2019, at 09:30, Mehmet Akcin  wrote:
> 
> hi there,
> 
> Just a general high-level question about Centurylink/Level3 post-merger, how 
> is your overall experience with CenturyLink? if you could be sitting with the 
> CEO of the company what is one thing you would ask him to fix?
> 
> please keep it high level and general. i intend to pass these to him and his 
> team in an upcoming meeting.
> 
> Mehmet

RE: CenturyLink/Level3 feedback

[Brower, Larry]

Mehmet,

Speaking strictly on their voice product, service has gone a bit downhill since 
the merger.

We never had problems with Level3 before the merger.

After Centurylink took over we started experiencing problems.

Just a couple of examples:

We waited months just to turn up a simple PRI. The PRI was sent back to design 
several times and then when it finally was turned up it isn’t working properly. 
The CL techs who were formally L3 express nothing but frustration with dealing 
with CL following the merger. Complaints to the account manager are met with 
just apologies and delays.

International call routing has become unreliable. In the last month alone we 
have had to create several service requests related to call failures. The 
result after anywhere from a couple hours to a day is just hey we rerouted try 
again. Then it works for a couple days and back to call failures and intercept 
messages.

I’ve already been asked if we should drop CenturyLink as the carrier and go 
back to using someone like AT

Never had any of these issue when it was Level3.

Regards,

Larry Brower, CCNP Collaboration, SSCA, RHCSA, CCDA, CCNA
Communications Technician | Unified Communications Group

Aramco Services Company
Office:  713.432.4516 | Mobile:  832.570.5416
larry.bro...@aramcoservices.com

This email has been classified as: General Use by Brower, Larry on Wednesday, 
June 5, 2019

From: NANOG  On Behalf Of Mehmet Akcin
Sent: Tuesday, June 4, 2019 9:31 AM
To: nanog 
Subject: CenturyLink/Level3 feedback

EXTERNAL: This email came from the Internet. Report this message to 
ascsuspiciousem...@aramcoservices.com
 as suspicious if it contains any suspicious content.
hi there,

Just a general high-level question about Centurylink/Level3 post-merger, how is 
your overall experience with CenturyLink? if you could be sitting with the CEO 
of the company what is one thing you would ask him to fix?

please keep it high level and general. i intend to pass these to him and his 
team in an upcoming meeting.

Mehmet

Re: CenturyLink/Level3 feedback

[Danny Pinto via NANOG]

Adding couple of 10G ports in EU has taken 4 months ..   still 
waiting. Can start to imagine how support can be ..As telcos grow bigger with 
M they become slower. How can telcos sustain / install agility as they grow 
?  Could be interesting study on telco corp culture Danny  On Tue, 04 
Jun 2019 20:00:54 +0530  Mehmet Akcin wrote hi there,Just 
a general high-level question about Centurylink/Level3 post-merger, how is your 
overall experience with CenturyLink? if you could be sitting with the CEO of 
the company what is one thing you would ask him to fix?please keep it high 
level and general. i intend to pass these to him and his team in an upcoming 
meeting.Mehmet 

Re: DOs and DONTs for small ISP

[William Waites]

On 06/03, Mel Beckman wrote:
> I’m constantly amazed at the number of even medium-sized ISPs that have no
> network monitoring. An NMS should go in as the first software component —
> before billing starts and the provider is on the hook to deliver. 
> 
> The second lacking component is a ticket system, which is silly because
> turnkey cloud services are not expensive, and open source solutions abound
> for budget-limited operators.

It's not enough to have monitoring and a ticket system. You need to pay
attention to them, care for them and feed them. I can't count the number
of ticket systems full of ancient and irrelevant things or monitoring 
systems that people have forgotten about or don't know how to add new
stuff to. Even the cycle of,

  10 we need network monitoring
  20 stand up some monitoring system ...
  30 ... time passes, the person leaves etc ...
  40 ... the network monitoring system is forgotten ...
  50 GOTO 10

Cheers,
-w

Re: CenturyLink/Level3 feedback

[Mike Hammett]

Almost every M has been worse. The bulk of the times it hasn't been worse is 
when the alternative was liquidation. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mehmet Akcin"  
To: "Danny Pinto"  
Cc: "nanog"  
Sent: Wednesday, June 5, 2019 5:31:23 AM 
Subject: Re: CenturyLink/Level3 feedback 



In recent years at least i can not remember a single telco m which has 
resulted with better service and product. The question is how fast they can go 
back to the level of service they were providing prior, because during mergers 
lots of talent walk away, and often misalignments happen burning people 
out(depending who is buying who) 



On Wed, Jun 5, 2019 at 04:54 Danny Pinto < danny.pi...@zoho.com > wrote: 






Adding couple of 10G ports in EU has taken 4 months .. still waiting. Can start 
to imagine how support can be .. 



As telcos grow bigger with M they become slower. How can telcos sustain / 
install agility as they grow ? Could be interesting study on telco corp culture 
 


Danny 












 On Tue, 04 Jun 2019 20:00:54 +0530 Mehmet Akcin< meh...@akcin.net > wrote 
 







hi there, 



Just a general high-level question about Centurylink/Level3 post-merger, how is 
your overall experience with CenturyLink? if you could be sitting with the CEO 
of the company what is one thing you would ask him to fix? 


please keep it high level and general. i intend to pass these to him and his 
team in an upcoming meeting. 


Mehmet 




-- 

Mehmet 
+1-424-298-1903 

Re: CenturyLink/Level3 feedback

[Mark Tinka]

On 5/Jun/19 12:31, Mehmet Akcin wrote:

> In recent years at least i can not remember a single telco m which
> has resulted with better service and product. The question is how fast
> they can go back to the level of service they were providing prior,
> because during mergers lots of talent walk away, and often
> misalignments happen burning people out(depending who is buying who)

Well, IP Transit pricing isn't what it used to be either, so that will
have some kind of impact on how much more your get besides the x-connect
and a BGP session.

Mark.

Re: CenturyLink/Level3 feedback

[Mehmet Akcin]

In recent years at least i can not remember a single telco m which has
resulted with better service and product. The question is how fast they can
go back to the level of service they were providing prior, because during
mergers lots of talent walk away, and often misalignments happen burning
people out(depending who is buying who)

On Wed, Jun 5, 2019 at 04:54 Danny Pinto  wrote:

> Adding couple of 10G ports in EU has taken 4 months ..   still waiting.
> Can start to imagine how support can be ..
>
> As telcos grow bigger with M they become slower. How can telcos sustain
> / install agility as they grow ?  Could be interesting study on telco corp
> culture 
>
> Danny
>
>
>
>
>
>
>  On Tue, 04 Jun 2019 20:00:54 +0530 Mehmet Akcin
> wrote 
>
> hi there,
>
> Just a general high-level question about Centurylink/Level3 post-merger,
> how is your overall experience with CenturyLink? if you could be sitting
> with the CEO of the company what is one thing you would ask him to fix?
>
> please keep it high level and general. i intend to pass these to him and
> his team in an upcoming meeting.
>
> Mehmet
>
>
> --
Mehmet
+1-424-298-1903