Re: Nashville

[Robert DeVita]

AT Disaster Recovery Team is probably the best in the business. The resources 
they can bring to the table are unmatched. This would have been 100x worse if 
it hit a carrier neutral datacenter. They don’t have nearly the same resources 
to restore something like this. They usually do a road show (pre Covid). If you 
get a chance it’s definitely something you should go check out. Very impressive.

Robert DeVita
Founder & CEO
Mejeticks
c. 469-441-8864
e. radev...@mejeticks.com

From: NANOG  on behalf of Eric 
Kuhnke 
Sent: Tuesday, December 29, 2020 5:06:00 PM
To: Sean Donelan 
Cc: NANOG 
Subject: Re: Nashville

>From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in one 
>place has a long recovery time when it's physically damaged. Something to 
>think about for entities that own and operate traditional telco COs and their 
>plans for disaster recovery.


Nv1

Here is the latest update:  6:46AM 12/27:

Work continues restoring service to the CRS routers in the Nashville Central 
Office. One router remains out of service and the other is in service with some 
links remaining out of service.

The working bridge will reconvene at 08:00 CT with the following action plan:
Additional cabling added to the first portable generator to enable full load 
capabilities (08:00 CT)
Pigtails with camlocks installed for easy swap; investigate possibility to land 
generator on the emergency service board to give the site N+1 with a manual 
ability to choose anyone. (08:00 CT)
check small power plants on floors 4 and 6 (08:00 CT)
Investigate water damage on 1st floor and energize if safe (08:00 CT
Air handlers for floors 4,5 and 6 (09:00 CT)
complete all transport work
Turn up SS7
Turn up 911 service - Approximately noon or after)
Turn up switching service.
TDM Switching team will reconvene at 09:00 CT and the Signaling team will 
reconvene at 11:00 CT on 12/27/2020.
DMS equipment on the 1st floor will be assessed for water damage. Switching 
teams will monitor power and HVAC restoration and will begin switch restoration 
as soon as the go ahead is provided by the power team.

Recovery Priorities:
1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7 
isolation & Datakit needed for Local Switch restoration). Transport SMEs 
currently working to turn up transport equipment
2. 6th floor (ESINET Groomers)
3. 10th and 8th floors (N4E) – Trunks
4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS
5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks
6. 11th floor (DMS: 01T) – Trunks
7. 4th floor (STP and SCP with mates up in Donelson)

The next update will be issued at approximately 09:00 CT on December 27.



Nv2

As of 09:00 CT: Teams worked through the night to restore service and improve 
conditions at the Nashville 2nd Ave Central Office. Since the initial service 
impact, over 75% of the Out of Service Mobility Sites have been restored. 
Certain call flows may be limited and should improve as additional restoration 
activities complete.
The generator that is currently powering equipment on the 2nd and 3rd floor, 
was refueled and ran with no issues through the night. Overnight, the batteries 
connected to it, continued to charge. Teams have placed additional power 
cables, which once connected, will allow the working generator, to better 
handle the load in the building. In order to accomplish this, the generator 
will need to be shut down for 15-30 minutes this morning, so teams can connect 
the new cables to the system. The power team reports they are still on target 
to restore power and cooling to the 5th and 6th floor by approximately 12:00 
CT. Also, a portable chiller will be delivered this morning and strategically 
placed, in case it is needed to assist in cooling the office.
There is a Call Center at 333 Commerce, in Nashville that does not have network 
or phone services available. Corporate Real Estate (CRE) reports there is some 
damage to that office, but the extent of the damage will not be known until 
they can gain access to the site. Because of this, the impacted Call Center 
ceased operations until further notice.
DMS switching equipment on the 1st floor will be assessed for water damage. 
Switching teams will monitor power and HVAC restoration. Equipment power ups 
will begin, as soon as the go ahead is provided by the power team.
Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT 
Central Office providing critical communication for teams working restoration 
efforts. There are 17 assets deployed in the field- 15 are on air (the 2 at the 
CO and 13 supporting FN Customer Requests) and 2 are in hot-standby for FN 
Customers where macro service recently recovered. There is 1 asset staged at a 
deployment site in KY where macro service restored, and 8 additional assets are 
on route to Nashville today to fulfill pending FN Customer requests. Incoming 
requests continue to be triaged. The ones in areas where service looks to 

Re: Where do your 911 fees go and why does 911 fail

[Eric Kuhnke]

The massive 911 failure in WA state a few years ago was ultimately caused
by a failure in CenturyLink/legacy qwest transport equipment, where the
PSAP register was physically located in Colorado and inaccessible from the
point of view of network equipment in WA.


On Tue, Dec 29, 2020, 1:19 PM Matt Erculiani  wrote:

> This isn't the place where state governments are looking for feedback, so
> surely this will fall on deaf ears, but...
>
> Who runs 911 services on top of a single carrier solution? I wouldn't run
> a 10 seat mom and pop outfit without at least a cellular backup on a
> different carrier.
>
> 911 services are certainly not treated as critical as the public is led to
> believe. Not that anyone here is surprised by this, but hopefully positive
> change can come out of this otherwise horrible event.
>
> -Matt
>
> On Tue, Dec 29, 2020 at 1:30 PM Sean Donelan  wrote:
>
>> The FCC published its annual report on state 911 fees
>>
>> https://www.fcc.gov/document/fcc-issues-annual-report-state-911-fees-1
>>
>> The report finds that in 2019, states and territories collected more than
>> $3 billion in 911 fees, and more than $200 million of that funding was
>> diverted for uses other than 911.
>>
>>
>> You can look up your individual state's 911 report here
>>
>> https://www.fcc.gov/general/911-fee-reports
>>
>>
>> In case you are interested in Tennessee's 911 service resiliancy:
>>
>> [...]
>> The project, referred to as NG911, involves utilization of the State’s
>> secure, private, outsourced Multiprotocol Label Switching (“MPLS”)
>> network
>> called “NetTN,” provided by AT and managed by Strategic Technology
>> Solutions (“STS”) in the Tennessee Department of Finance and
>> Administration. The new network improves redundancy, reliability, and 911
>> call delivery. It enhances interoperability and increases the ease of
>> communication between ECDs, allowing immediate transfer of 911 calls,
>> caller information, and other data on a statewide level. NG911 will also
>> provide alternate paths to process emergency calls in the event of an
>> outage, providing lifesaving capabilities in the event of an emergency
>> that would have been unachievable on the outdated analog network.
>>
>> [...]
>> In fiscal year 2019, the TECB spent $11,224,726 million implementing and
>> maintaining the NG911 project: $6,974,790 to integrate with and adapt the
>> Net TN system for NG911 purposes; $780,966 for non-recurring start-up
>> costs of the statewide hosted controller or Call Handling as a Service
>> program; $3,451,369 to maintain the twenty-four hour network operations
>> center to assist PSAPs with technical issues; and $17,600 for Esri GIS
>> software licensing.
>> [...]
>>
>
>
> --
> Matt Erculiani
> ERCUL-ARIN
>

Re: Nashville

[Eric Kuhnke]

>From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in
one place has a long recovery time when it's physically damaged. Something
to think about for entities that own and operate traditional telco COs and
their plans for disaster recovery.


Nv1

Here is the latest update:  6:46AM 12/27:

Work continues restoring service to the CRS routers in the Nashville
Central Office. One router remains out of service and the other is in
service with some links remaining out of service.

The working bridge will reconvene at 08:00 CT with the following action
plan:
Additional cabling added to the first portable generator to enable full
load capabilities (08:00 CT)
Pigtails with camlocks installed for easy swap; investigate possibility to
land generator on the emergency service board to give the site N+1 with a
manual ability to choose anyone. (08:00 CT)
check small power plants on floors 4 and 6 (08:00 CT)
Investigate water damage on 1st floor and energize if safe (08:00 CT
Air handlers for floors 4,5 and 6 (09:00 CT)
complete all transport work
Turn up SS7
Turn up 911 service - Approximately noon or after)
Turn up switching service.
TDM Switching team will reconvene at 09:00 CT and the Signaling team will
reconvene at 11:00 CT on 12/27/2020.
DMS equipment on the 1st floor will be assessed for water damage. Switching
teams will monitor power and HVAC restoration and will begin switch
restoration as soon as the go ahead is provided by the power team.

Recovery Priorities:
1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7
isolation & Datakit needed for Local Switch restoration). Transport SMEs
currently working to turn up transport equipment
2. 6th floor (ESINET Groomers)
3. 10th and 8th floors (N4E) – Trunks
4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS
5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks
6. 11th floor (DMS: 01T) – Trunks
7. 4th floor (STP and SCP with mates up in Donelson)

The next update will be issued at approximately 09:00 CT on December 27.



Nv2

As of 09:00 CT: Teams worked through the night to restore service and
improve conditions at the Nashville 2nd Ave Central Office. Since the
initial service impact, over 75% of the Out of Service Mobility Sites have
been restored. Certain call flows may be limited and should improve as
additional restoration activities complete.
The generator that is currently powering equipment on the 2nd and 3rd
floor, was refueled and ran with no issues through the night. Overnight,
the batteries connected to it, continued to charge. Teams have placed
additional power cables, which once connected, will allow the working
generator, to better handle the load in the building. In order to
accomplish this, the generator will need to be shut down for 15-30 minutes
this morning, so teams can connect the new cables to the system. The power
team reports they are still on target to restore power and cooling to the
5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will
be delivered this morning and strategically placed, in case it is needed to
assist in cooling the office.

There is a Call Center at 333 Commerce, in Nashville that does not have
network or phone services available. Corporate Real Estate (CRE) reports
there is some damage to that office, but the extent of the damage will not
be known until they can gain access to the site. Because of this, the
impacted Call Center ceased operations until further notice.

DMS switching equipment on the 1st floor will be assessed for water damage.
Switching teams will monitor power and HVAC restoration. Equipment power
ups will begin, as soon as the go ahead is provided by the power team.

Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT
Central Office providing critical communication for teams working
restoration efforts. There are 17 assets deployed in the field- 15 are on
air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in
hot-standby for FN Customers where macro service recently recovered. There
is 1 asset staged at a deployment site in KY where macro service restored,
and 8 additional assets are on route to Nashville today to fulfill pending
FN Customer requests. Incoming requests continue to be triaged. The ones in
areas where service looks to have been restored, are being held, while the
others are being prioritized to be dispatched upon.

The next update will be issued at approximately 14:00 CT, unless there is a
significant change in status.



Nv3

AT Nashville update below, received at 3:35PM 12/27.

Since the initial service impact, over 95% of the Out of Service Mobility
Sites have been restored. Certain call flows may be limited and should
improve as additional restoration activities complete.

Electricians have installed the additional power cables from the generator,
to the emergency bus. These new cables will allow the generator to support
more of the load, of the building. The portable chiller requested, has

Re: Where do your 911 fees go and why does 911 fail

[Peter E . Fry]

--- Original message ---
Subject: Re: Where do your 911 fees go and why does 911 fail
From: Matt Erculiani 
To: Sean Donelan 
Cc: nanog@nanog.org list 
Date: Tuesday, 12/29/2020 15:19:00


This isn't the place where state governments are looking for feedback, 
so surely this will fall on deaf ears, but...
Who runs 911 services on top of a single carrier solution? I wouldn't 
run a 10 seat mom and pop outfit without at least a cellular backup on 
a different carrier.
911 services are certainly not treated as critical as the public is 
led to believe. Not that anyone here is surprised by this, but 
hopefully positive change can come out of this otherwise horrible 
event.


BellSouth and the states were very closely aligned.  I imagine it's a 
legacy deal.  Once upon a time, your LEC was your solution, and 
regulation forms a relationship.  (It's not always a bad thing, at 
least in practice.)



Peter E. Fry

Re: Where do your 911 fees go and why does 911 fail

[Matt Erculiani]

This isn't the place where state governments are looking for feedback, so
surely this will fall on deaf ears, but...

Who runs 911 services on top of a single carrier solution? I wouldn't run a
10 seat mom and pop outfit without at least a cellular backup on a
different carrier.

911 services are certainly not treated as critical as the public is led to
believe. Not that anyone here is surprised by this, but hopefully positive
change can come out of this otherwise horrible event.

-Matt

On Tue, Dec 29, 2020 at 1:30 PM Sean Donelan  wrote:

> The FCC published its annual report on state 911 fees
>
> https://www.fcc.gov/document/fcc-issues-annual-report-state-911-fees-1
>
> The report finds that in 2019, states and territories collected more than
> $3 billion in 911 fees, and more than $200 million of that funding was
> diverted for uses other than 911.
>
>
> You can look up your individual state's 911 report here
>
> https://www.fcc.gov/general/911-fee-reports
>
>
> In case you are interested in Tennessee's 911 service resiliancy:
>
> [...]
> The project, referred to as NG911, involves utilization of the State’s
> secure, private, outsourced Multiprotocol Label Switching (“MPLS”) network
> called “NetTN,” provided by AT and managed by Strategic Technology
> Solutions (“STS”) in the Tennessee Department of Finance and
> Administration. The new network improves redundancy, reliability, and 911
> call delivery. It enhances interoperability and increases the ease of
> communication between ECDs, allowing immediate transfer of 911 calls,
> caller information, and other data on a statewide level. NG911 will also
> provide alternate paths to process emergency calls in the event of an
> outage, providing lifesaving capabilities in the event of an emergency
> that would have been unachievable on the outdated analog network.
>
> [...]
> In fiscal year 2019, the TECB spent $11,224,726 million implementing and
> maintaining the NG911 project: $6,974,790 to integrate with and adapt the
> Net TN system for NG911 purposes; $780,966 for non-recurring start-up
> costs of the statewide hosted controller or Call Handling as a Service
> program; $3,451,369 to maintain the twenty-four hour network operations
> center to assist PSAPs with technical issues; and $17,600 for Esri GIS
> software licensing.
> [...]
>


-- 
Matt Erculiani
ERCUL-ARIN

Where do your 911 fees go and why does 911 fail

[Sean Donelan]

The FCC published its annual report on state 911 fees

https://www.fcc.gov/document/fcc-issues-annual-report-state-911-fees-1

The report finds that in 2019, states and territories collected more than 
$3 billion in 911 fees, and more than $200 million of that funding was 
diverted for uses other than 911.



You can look up your individual state's 911 report here

https://www.fcc.gov/general/911-fee-reports


In case you are interested in Tennessee's 911 service resiliancy:

[...]
The project, referred to as NG911, involves utilization of the State’s 
secure, private, outsourced Multiprotocol Label Switching (“MPLS”) network 
called “NetTN,” provided by AT and managed by Strategic Technology 
Solutions (“STS”) in the Tennessee Department of Finance and 
Administration. The new network improves redundancy, reliability, and 911 
call delivery. It enhances interoperability and increases the ease of
communication between ECDs, allowing immediate transfer of 911 calls, 
caller information, and other data on a statewide level. NG911 will also 
provide alternate paths to process emergency calls in the event of an 
outage, providing lifesaving capabilities in the event of an emergency 
that would have been unachievable on the outdated analog network.


[...]
In fiscal year 2019, the TECB spent $11,224,726 million implementing and 
maintaining the NG911 project: $6,974,790 to integrate with and adapt the 
Net TN system for NG911 purposes; $780,966 for non-recurring start-up 
costs of the statewide hosted controller or Call Handling as a Service 
program; $3,451,369 to maintain the twenty-four hour network operations 
center to assist PSAPs with technical issues; and $17,600 for Esri GIS 
software licensing.

[...]

Re: 10g residential CPE

[Owen DeLong]

> On Dec 25, 2020, at 09:45 , Bryan Fields  wrote:
> 
> On 12/25/20 4:52 AM, Mark Tinka wrote:
>> For the home, if you're looking at shipping 10Gbps-based CPE's for under 
>> US$200, I can't think of anything other than the Tik:
>> 
>> https://mikrotik.com/product/rb4011igs_rm
> 
> That has 1 10g port.  How can that be a 10g CPE?
> 
> 
>> They claim:
>> 
>> - 2.6Gbps forwarding for 64-byte packets.
>> - 7.8Gbps forwarding for 512-byte packets.
>> - 9.7Gbps forwarding for 1,518-byte packets.
> 
> so, not 10g :)
> 
> Add in some services and I bet it goes down from there.
> 
> The bigger question in all this if you're doing 10g to the residential user,
> what are they going to use for their home router/NAT device?  Even 60 ghz wifi
> routers top out at like 5 gbit/s, and NAT at this speed means a powerful CPU.

Sounds like a great reason not to do NAT… If you run IPv6, who needs NAT?

For the rest, there are relatively cheap 2x10G+rest 1G switches coming 
available and there’s this weirdness with 2G/5G RJ ports now appearing to try 
and eke out additional wifi performance over cat5e I guess.

Owen

Re: [External] Re: 10g residential CPE

[Matthew Petach]

On Mon, Dec 28, 2020 at 4:26 PM Niels Bakker  wrote:

> * mpet...@netflight.com (Matthew Petach) [Tue 29 Dec 2020, 01:08 CET]:
> >But as far as the physics goes, the conversion of biomatter into
> >petrochemicals in the ground is more "renewable" than the conversion
> >of hydrogen into helium in the sun.
>
> It's not. Where did Mr Metcalf think the energy comes from that is
> necessary for that process? You know, the energy that we can now
> extract by burning it?
>

The same place that provides the energy that gets
water back to the top of the mountains to make
hydroelectric energy "renewable".  The same place
that provides the energy that heats air masses to
different temperatures around the planet, creating
wind currents that move wind turbines to generate
"renewable" electricity.

It's just that water and wind energy cycles work on
shorter time cycles; those cycles are measured in
months and weeks, not in millenia the way the
absorption of solar energy by plants and then
eventual breakdown into petrochemicals underground
takes.

We have short-term renewables, like wind and
hydro; we have longer-term renewables like
oil and coal that take longer than the course
of human history to renew; and then we have
a completely consumable resource called the
sun which powers all the rest, but is itself on a
one-way trip to eventual extinction, albeit on a
much longer time scale.

I'm a huge fan of solar power, of wind power,
and pumped hydro energy storage.  But from
a long enough time horizon, it all depends on
a single, non-renewable energy source--the sun.

We just have the luxury of punting that concern
a few billion years down the road.   ;)

Coming back slightly more on topic--multiple
diverse power sources are always good to have,
but I'm mindful of the fried rodent incident at
Forsythe Hall from the mid-90s.  BARRnet
and SUNet were both impacted when the
datacenter there was taken completely offline
from a power perspective, in spite of having
two different off-campus power providers, plus
a local cogeneration plant and a generator out
in the parking lot.  One rodent in the heart of
the transfer switch made all the different power
feeds completely moot.  From a "single point of
failure" perspective, the transfer switch tends to
be the weakest link in the chain.  Has anyone
developed a distributed transfer switch, split
into different locations in a building, fed at different
entry points, that can withstand one portion of the
transfer system being knocked out?

Thanks!

Matt
(yes, Earth *is* a single point of failure...for now)

Re: 10g residential CPE

[James R Cutler]

On Dec 29, 2020, at 11:53 AM, Michael Thomas  wrote:
> 
> On 12/29/20 8:42 AM, Aaron Wendel wrote:
>> Oh, we still get calls about speed issues. It's always wonderful when 
>> someone puts their own 10 year old Linksys WRT54G and double NATs behind our 
>> CPE then sends in a speed test wondering why they're only getting 10Mbits on 
>> their Gbit line.  We get those ALL the time. :)
>> 
> Does your CPE not have wireless? If it's double NAT'ing it's at least a 
> router. If it doesn't have wireless, wouldn't it be cheaper to add it so you 
> don't get the support calls?
> 
> Mike
> 
Supplying any configurable residential CPE would not necessarily be cheaper. 
The tracking and accounting for the hardware and qualifying said hardware, not 
to mention truck rolls for hardware updates, could well be more costly than 
fielding support calls (which would likely not decrease anyway).

An intangible benefit of ‘free residential service’ is creation of good will 
far exceeding that received by many other ISPs.  
-
James R. Cutler
james.cut...@consultant.com
GPG keys: hkps://hkps.pool.sks-keyservers.net

Re: 10g residential CPE

[Michael Thomas]

On 12/29/20 10:36 AM, Aaron Wendel wrote:
It does have wireless.  That doesn't prevent people from trying to use 
their old equipment in addition. ("My dad's uncle's cousin's former 
roommate works in IT and told me I just needed to plug my old router 
into your new router.")




Yes, but does your CPE buffer bloat avoidance? Latency is still an issue 
when you have a big long packet queue...


Mike

Re: 10g residential CPE

[Aaron Wendel]

It does have wireless.  That doesn't prevent people from trying to use 
their old equipment in addition. ("My dad's uncle's cousin's former 
roommate works in IT and told me I just needed to plug my old router 
into your new router.")


On 12/29/2020 10:53 AM, Michael Thomas wrote:


On 12/29/20 8:42 AM, Aaron Wendel wrote:
Oh, we still get calls about speed issues. It's always wonderful when 
someone puts their own 10 year old Linksys WRT54G and double NATs 
behind our CPE then sends in a speed test wondering why they're only 
getting 10Mbits on their Gbit line.  We get those ALL the time. :)


Does your CPE not have wireless? If it's double NAT'ing it's at least 
a router. If it doesn't have wireless, wouldn't it be cheaper to add 
it so you don't get the support calls?


Mike



--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: 10g residential CPE

[Peter E . Fry]

From: Ben Cannon 
To: Darin Steffl 
[...]





Again, it seems nice to be able to do this but most companies don't 
have idle resources sitting around to give away things for free. We 
have zero extra time to work for free.
We’re a tiny company and I already have a department dedicated to 
giving [...]



It's all in resource allocation, particularly as applied to a small 
company.  I can't imagine 6x7 or Wholesale Internet have dedicated 
departments of bean counters (apparently no button-pushing monkeys as 
well, even if they work for bananas).  The disadvantage is that 
particular individual personnel tend to be even less replaceable than 
in larger companies.



Peter E. Fry

[Hm: 6by7.net web page? That's even a bit sadder than mine...]

Anyone from Draftkings.com on here?

[Matthew Crocker]

Hoping someone from Draftkings.com is on this list.  If so, please message me 
directly so I can get a subnet block cleared up.

Thanks

-Matt

Re: 10g residential CPE

[Michael Thomas]

On 12/29/20 9:00 AM, Mike Hammett wrote:
People love throwing their own router behind whatever Internet 
connection they have. It almost never fails to cause a problem.



Well *some* of us know what we're doing. And in my case, it's both 
because it doesn't deal with buffer bloat, but more importantly doesn't 
have wifi. I did get them to put it in bridge mode so it doesn't double nat.



Mike

Re: 10g residential CPE

[Mike Hammett]

People love throwing their own router behind whatever Internet connection they 
have. It almost never fails to cause a problem. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Michael Thomas"  
To: nanog@nanog.org 
Sent: Tuesday, December 29, 2020 10:53:39 AM 
Subject: Re: 10g residential CPE 


On 12/29/20 8:42 AM, Aaron Wendel wrote: 
> Oh, we still get calls about speed issues. It's always wonderful when 
> someone puts their own 10 year old Linksys WRT54G and double NATs 
> behind our CPE then sends in a speed test wondering why they're only 
> getting 10Mbits on their Gbit line. We get those ALL the time. :) 
> 
Does your CPE not have wireless? If it's double NAT'ing it's at least a 
router. If it doesn't have wireless, wouldn't it be cheaper to add it so 
you don't get the support calls? 

Mike 

Re: 10g residential CPE

[Michael Thomas]

On 12/29/20 8:42 AM, Aaron Wendel wrote:
Oh, we still get calls about speed issues. It's always wonderful when 
someone puts their own 10 year old Linksys WRT54G and double NATs 
behind our CPE then sends in a speed test wondering why they're only 
getting 10Mbits on their Gbit line.  We get those ALL the time. :)


Does your CPE not have wireless? If it's double NAT'ing it's at least a 
router. If it doesn't have wireless, wouldn't it be cheaper to add it so 
you don't get the support calls?


Mike

Re: 10g residential CPE

[Aaron Wendel]

The majority of our customers are still on Brocade MLXs.  We're in the 
process of upgrading all our equipment to Arista switches to accommodate 
the increased demand for 40G and 100G ports as well as implement 400G ports.


Aaron


On 12/29/2020 3:33 AM, Jonathon Exley wrote:

Hi Aaron,

Just out of interest, what switch gear are you using? You must have a 
pretty good cost per port.


Jonathon.

On 29/12/2020 9:38 AM, Aaron Wendel  wrote:
We prioritize calls based on severity.  If both Google and Grandma call
and say they have a cut then we have people to service both at the same
time.  If Google, Century Link, Verizon, AT and Grandma all call then
Grandma gets to wait a day.  That being the case, it's not dependent on
revenue. Emergency Services (911 and Police radio feeds) gets #1
priority even though they're non-paying.

But yes, in extreme situations the residential customers would be
delayed to service the paying customers.  We do have people cross
trained from other parts of our businesses so we can allocate internally
in emergencies.  In almost a decade though I can't think of a situation
where someone had to wait for service because we didn't have the
resources to service them.

Aaron


On 12/28/2020 2:02 PM, Mel Beckman wrote:
> Darin,
>
> Surely you at least give the paying customers priority over the
> non-paying? It’s one thing to say “I have to write paychecks no matter
> what”. It’s another to say “I’ll give away my support to free
> customers AND degrade support for paying customers as a result.” Your
> tech support guy “walking Grandma through getting her email” is
> necessarily not accessible for the duration to paying customers.
>
> This means your staffing must be large enough to never have any
> queuing, or you’re giving away your paying customers' time to
> non-paying customers. Neither approach is scalable in a competitive
> business environment, because SOMEBODY is paying for all those
> resources, and if it’s your customers, they will buy elsewhere. Your
> approach only work until you run out of other people’s money.
>
>   -mel
>
>> On Dec 28, 2020, at 11:50 AM, Baldur Norddahl
>> mailto:baldur.nordd...@gmail.com>> wrote:
>>
>> I applaud your commitment to helping your local community. Just want
>> to point out that this is a charity because it does not scale. Nobody
>> could build out a FTTH network and make it free as a business case.
>> But there are plenty of people that made a network for their
>> neighbors and provided that for free. Maybe a person had a commercial
>> fiber to his home and thought he could just as well share it. This
>> might be on a bigger scale but it is the same.
>>
>> Regards,
>>
>> Baldur
>>
>>
>> On Mon, Dec 28, 2020 at 8:27 PM Aaron Wendel
>> mailto:aa...@wholesaleinternet.net>> 
wrote:

>>
>> Darin,
>>
>> Our business support and residential support is the same
>> department.  I
>> have to pay those people to be in the office either way so it
>> doesn't
>> cost me any "more" to provide support for the residences. Yes,
>> walking
>> Grandma through getting her email can sometimes be a chore but that
>> person is on the payroll whether he/she is helping Grandma or
>> sitting
>> there chatting with his/her co-worker.  If we dumped all the
>> residential
>> customers we would still have the same cost structure we do now.
>>
>> Again, it's been free for the last 7 years at this point.  I've
>> never
>> been one to really do what I "should" anyway.
>>
>> Aaron
>>
>>
>> On 12/28/2020 11:48 AM, Darin Steffl wrote:
>> > Aaron,
>> >
>> > The "Free" service doesn't cover your cost of support which is
>> much
>> > higher for residential than any business customer. Our residential
>> > customers call at least 15x more often compared to business
>> customers
>> > compared on a 1:1 ratio.
>> >
>> > I honestly can't fathom providing free residential service
>> because we
>> > make enough money on the business side of things. You should be
>> > charging something, at least $20-30 per month.
>> >
>> > On Mon, Dec 28, 2020 at 11:15 AM Aaron Wendel
>> > > 
>> > >> wrote:
>> >
>> >     The $300 covers the equipment and the time to send someone
>> out to a
>> >     house to install it.  If $300 is too much you can pay in 12
>> >     installments
>> >     of $25.
>> >
>> >     The TIK alone costs us about $250.
>> >
>> >     Aaron
>> >
>> >
>> >     On 12/27/2020 5:04 AM, Mark Tinka wrote:
>> >     >
>> >     >
>> >     > On 12/26/20 20:48, Darin Steffl wrote:
>> >     >
>> >     >> Aaron,
>> >     >>
>> >     >> One simple question. Why on earth would you offer free
>> internet
>> >     >> service? How and why? Your site show 1 Gig symmetrical
>> for free
>> >     when
>> >     >> you should be a minimum of $65 per month to be competitive.
>> >     >
>> >     > They also ask for no monthly fee after a single payment
>> of US$300.
>> >     >
>> >     > Considering 

Re: 10g residential CPE

[Aaron Wendel]

Oh, we still get calls about speed issues. It's always wonderful when 
someone puts their own 10 year old Linksys WRT54G and double NATs behind 
our CPE then sends in a speed test wondering why they're only getting 
10Mbits on their Gbit line.  We get those ALL the time. :)


On 12/29/2020 1:28 AM, Mark Tinka wrote:



On 12/29/20 04:41, Keith Medcalf wrote:

Are you sure that is not related to "residential services" being of a 
generally lower quality than business services?  It has been my 
experience that shoddy service generates higher need for "support" 
than does "non-shoddy" service.  In this regard, the price for 
"business" services should be less than "residential service" by a 
couple of orders of magnitude since it costs orders of magnitude more 
money to "support" shoddy services than non-shoddy services.


Considering that Aaron said 98% of their residential customers are on 
the free plan, and that they use Active-E with every 1Gbps customer 
getting a proper switch port, I'd hazard the bulk of their support 
queries to be non-techie customers needing software support (grandma, 
et al), or fibres being cut.


It wouldn't seem like they'd be getting calls about "speed" issues, 
which are most annoying ones :-).


Mark.


--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: 10g residential CPE

[Ben Cannon]

> Again, it seems nice to be able to do this but most companies don't have idle 
> resources sitting around to give away things for free. We have zero extra 
> time to work for free. 

We’re a tiny company and I already have a department dedicated to giving - 
really we do have some often highly specific embarrassments of riches as 
telecom companies - and honestly reading between the lines here, Big Telco has 
already paid for the fiber and the trucks have rolled and the guys have half a 
day left, the entire spool’s paid for so why tf not...

It’s easy for the same activity to cost one entity 6 figures, and another, 
literally zero (or more realistically, some extra fuel and a switch and 48 
optics etc.).

Then again it can also cost us $1,000/ft to trench in some downtown metros.

Ms. Lady Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net
"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ

Sent from my iPhone via RFC1149.

> On Dec 29, 2020, at 5:42 AM, Darin Steffl  wrote:
> 
> 
> Oh they'll get plenty of support calls still, almost all about wifi issues. 
> They'll be connected to 2.4ghz on an old device, run a speedtest and only get 
> 30 mbps and complain they're not getting 950 mbps on their free connection.
> 
> WiFi issues will always cause support calls no matter what isp. The denser 
> the area, the more wifi interference that exists and will drive more calls. 
> 
> I understand wanting to offer free internet to a small number of entities and 
> residential areas, particularly hotspots. What I don't agree with is free 
> service for every residential home or apartment. It absolutely hurts your 
> business to do this. It's a charity, not a business then. You say it doesn't 
> take any additional resources to support but it absolutely does. You have way 
> more than $300 into an install. You'll also have to hire additional staff 
> sooner because of additional tech support calls from the res side. 
> 
>> On Tue, Dec 29, 2020, 1:28 AM Mark Tinka  wrote:
>> 
>> 
>> On 12/29/20 04:41, Keith Medcalf wrote:
>> 
>> > Are you sure that is not related to "residential services" being of a 
>> > generally lower quality than business services?  It has been my experience 
>> > that shoddy service generates higher need for "support" than does 
>> > "non-shoddy" service.  In this regard, the price for "business" services 
>> > should be less than "residential service" by a couple of orders of 
>> > magnitude since it costs orders of magnitude more money to "support" 
>> > shoddy services than non-shoddy services.
>> 
>> Considering that Aaron said 98% of their residential customers are on 
>> the free plan, and that they use Active-E with every 1Gbps customer 
>> getting a proper switch port, I'd hazard the bulk of their support 
>> queries to be non-techie customers needing software support (grandma, et 
>> al), or fibres being cut.
>> 
>> It wouldn't seem like they'd be getting calls about "speed" issues, 
>> which are most annoying ones :-).
>> 
>> Mark.

Re: 10g residential CPE

[Darin Steffl]

Oh they'll get plenty of support calls still, almost all about wifi issues.
They'll be connected to 2.4ghz on an old device, run a speedtest and only
get 30 mbps and complain they're not getting 950 mbps on their free
connection.

WiFi issues will always cause support calls no matter what isp. The denser
the area, the more wifi interference that exists and will drive more calls.

I understand wanting to offer free internet to a small number of entities
and residential areas, particularly hotspots. What I don't agree with is
free service for every residential home or apartment. It absolutely hurts
your business to do this. It's a charity, not a business then. You say it
doesn't take any additional resources to support but it absolutely does.
You have way more than $300 into an install. You'll also have to hire
additional staff sooner because of additional tech support calls from the
res side.

Again, it seems nice to be able to do this but most companies don't have
idle resources sitting around to give away things for free. We have zero
extra time to work for free.

On Tue, Dec 29, 2020, 1:28 AM Mark Tinka  wrote:

>
>
> On 12/29/20 04:41, Keith Medcalf wrote:
>
> > Are you sure that is not related to "residential services" being of a
> generally lower quality than business services?  It has been my experience
> that shoddy service generates higher need for "support" than does
> "non-shoddy" service.  In this regard, the price for "business" services
> should be less than "residential service" by a couple of orders of
> magnitude since it costs orders of magnitude more money to "support" shoddy
> services than non-shoddy services.
>
> Considering that Aaron said 98% of their residential customers are on
> the free plan, and that they use Active-E with every 1Gbps customer
> getting a proper switch port, I'd hazard the bulk of their support
> queries to be non-techie customers needing software support (grandma, et
> al), or fibres being cut.
>
> It wouldn't seem like they'd be getting calls about "speed" issues,
> which are most annoying ones :-).
>
> Mark.
>