Re: Comcast outages continue even in areas with PG power restored

[Brandon Martin]

On 10/16/19 4:04 PM, Michael Thomas wrote:
After some poking around, I found this gizmo. It says that it can use 
between 1-8 pairs to power it from the co. If there was already a home 
run to the co (which is almost certainly true in my case), it seems like 
that would be a cheaper option? Then you just have one diesel generator 
at the co that charges the batteries.


Yep, things like this are a great option if you're overbuilding existing 
copper plant with fiber.  The old copper gets relegated to duty as a 
power carrier, and the fiber moves the bits to the DSLAM.  As another 
poster said, you just keep pushing these out to keep loop lengths down 
and get the bandwidth available to the end user up.


G.FAST is the next iteration of this sort of thing.  You run fiber all 
the way to the ped at the curb or even into the building for 
multi-dwelling applications then re-use the existing drop to get into 
the customer prem.  4-8 ports is common on these types of things.  Many 
support either remote span power using the old copper plant or sometimes 
also reverse power from the customer prem which is really handy if your 
a pure-play fiber carrier re-using existing customer-owned copper 
infrastructure or if your copper plant has rotted to the point that 
you're loathe to put 190VDC on it for a few miles from the nearest 
powered RTU or CO.


The actual power that's needed per port is usually pretty small.  Maybe 
a dozen watts or so.  There's obviously a base load on the unit, so the 
more ports you have lit the lower that per-port number will go with 
diminishing returns.  It's low enough that, at 190VDC, you can feasibly 
power things over a mile or more with just a few pairs of existing 24AWG 
outside copper without the voltage drop or power loss and cable heating 
being too bad.

--
Brandon Martin

Re: Comcast outages continue even in areas with PG power restored

[Jeff Shultz]

We use 12 and 48 port VDSLAM's similar to that at some of our remote
locations, and we do generally line power those.

But before those came on the market we were putting out remote
cabinets that could support up to 144 subscribers fed off the same
sort of cards you would find in the CO.

I don't know our power budget per customer, but it's not unusual to
have 20 or more amps of capacity (probably overkill, likely because
that was the size available) at 48V in a cabinet. Because the CO cards
aren't hardened, the cabinet must be - and have some HVAC type
capabilities as well - at least fans.

We're now feeding line power out to some of the 12 and 48 port devices
like you linked to from some of those remote cabinets.

It's all about shrinking loop lengths until we can get both the time
and funds to put fiber in the ground everywhere.

On Wed, Oct 16, 2019 at 1:06 PM Michael Thomas  wrote:
>
> After some poking around, I found this gizmo. It says that it can use
> between 1-8 pairs to power it from the co. If there was already a home
> run to the co (which is almost certainly true in my case), it seems like
> that would be a cheaper option? Then you just have one diesel generator
> at the co that charges the batteries.
>
>
> https://portal.adtran.com/pub/Library/Data_Sheets/International_/I61179918F1-8_1148VXP.pdf
>
> Mike
>
> On 10/16/19 12:09 PM, Jeff Shultz wrote:
> > On Mon, Oct 14, 2019 at 4:26 PM Michael Thomas  wrote:
> >>
> >> On 10/14/19 4:16 PM, Sean Donelan wrote:
> >>> On Mon, 14 Oct 2019, Michael Thomas wrote:
>  Of course this is a lot of conjecture on my part... be glad to be
>  clued in by folks in know.
> >>> An old news story, but telco's usually have backup batteries in their
> >>> outside plant, cell towers, etc.  During power outages, they shuttle
> >>> small generators between outside cabinets to re-charge the batteries.
> >>> Remote Terminal Units (RTUs) use local power, i.e. look for the
> >>> utility meter nearby.  There is often a generator plug and battery
> >>> cabinet next to the RTU. They aren't powered from the central office.
> >> Interesting! And so primitive! So they go to all of the expense of
> >> laying fiber, but not power too?
> > Note: small local telco experience speaking below:
> >
> > Telco's tend to have experience with fiber, but probably not the
> > construction and transmission of the sort of power plant that would be
> > required to keep a bunch of  48V cabinets up and running reliably. We
> > certainly don't. Besides, an advantage of fiber is that hopefully the
> > copper thieves won't bother it.
> >
> >   By definition a remote terminal/cabinet is going to be... remote. Far
> > more simple to install commercial power, and then haul out a generator
> > if the battery string in the cabinet appears to be in danger of
> > dropping below about 46v.
> >
> > We do run some 360v DC at micro-amp levels out to equipment like ONT's
> > and remote 12 and 48 port remote VDSLAM's. But that's over existing
> > 24-26 ga. plant. Frequently using multiple pairs to avoid excessive
> > voltage drop over distances.
> >
> > Primitive is tested and works.
> >



-- 
Jeff Shultz
Central Office Technician

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

After some poking around, I found this gizmo. It says that it can use 
between 1-8 pairs to power it from the co. If there was already a home 
run to the co (which is almost certainly true in my case), it seems like 
that would be a cheaper option? Then you just have one diesel generator 
at the co that charges the batteries.



https://portal.adtran.com/pub/Library/Data_Sheets/International_/I61179918F1-8_1148VXP.pdf

Mike

On 10/16/19 12:09 PM, Jeff Shultz wrote:

On Mon, Oct 14, 2019 at 4:26 PM Michael Thomas  wrote:


On 10/14/19 4:16 PM, Sean Donelan wrote:

On Mon, 14 Oct 2019, Michael Thomas wrote:

Of course this is a lot of conjecture on my part... be glad to be
clued in by folks in know.

An old news story, but telco's usually have backup batteries in their
outside plant, cell towers, etc.  During power outages, they shuttle
small generators between outside cabinets to re-charge the batteries.
Remote Terminal Units (RTUs) use local power, i.e. look for the
utility meter nearby.  There is often a generator plug and battery
cabinet next to the RTU. They aren't powered from the central office.

Interesting! And so primitive! So they go to all of the expense of
laying fiber, but not power too?

Note: small local telco experience speaking below:

Telco's tend to have experience with fiber, but probably not the
construction and transmission of the sort of power plant that would be
required to keep a bunch of  48V cabinets up and running reliably. We
certainly don't. Besides, an advantage of fiber is that hopefully the
copper thieves won't bother it.

  By definition a remote terminal/cabinet is going to be... remote. Far
more simple to install commercial power, and then haul out a generator
if the battery string in the cabinet appears to be in danger of
dropping below about 46v.

We do run some 360v DC at micro-amp levels out to equipment like ONT's
and remote 12 and 48 port remote VDSLAM's. But that's over existing
24-26 ga. plant. Frequently using multiple pairs to avoid excessive
voltage drop over distances.

Primitive is tested and works.

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

On 10/16/19 12:09 PM, Jeff Shultz wrote:



Interesting! And so primitive! So they go to all of the expense of
laying fiber, but not power too?

Note: small local telco experience speaking below:

Telco's tend to have experience with fiber, but probably not the
construction and transmission of the sort of power plant that would be
required to keep a bunch of  48V cabinets up and running reliably. We
certainly don't. Besides, an advantage of fiber is that hopefully the
copper thieves won't bother it.

  By definition a remote terminal/cabinet is going to be... remote. Far
more simple to install commercial power, and then haul out a generator
if the battery string in the cabinet appears to be in danger of
dropping below about 46v.

We do run some 360v DC at micro-amp levels out to equipment like ONT's
and remote 12 and 48 port remote VDSLAM's. But that's over existing
24-26 ga. plant. Frequently using multiple pairs to avoid excessive
voltage drop over distances.

Primitive is tested and works.

This is all very interesting, and thanks to everybody for giving me an 
education. My provider is very small as well, and spread out over a 
pretty large area (i'm in amador county in the mother lode). I don't 
know how many remote terminals they have, but i would think that it 
would be a lot. And if they need to be recharged every 8 hours or so, 
you'd be talking about a lot of people out in the field just to keep the 
lights on, right? And of course it takes time to recharge a battery too, 
so that makes it even worse. It seems that would be a pretty significant 
recurring cost.


How many watts does a typical remote terminal draw per subscriber?

Mike

Re: Comcast outages continue even in areas with PG power restored

[Jeff Shultz]

On Mon, Oct 14, 2019 at 4:26 PM Michael Thomas  wrote:
>
>
> On 10/14/19 4:16 PM, Sean Donelan wrote:
> > On Mon, 14 Oct 2019, Michael Thomas wrote:
> >> Of course this is a lot of conjecture on my part... be glad to be
> >> clued in by folks in know.
> >
> > An old news story, but telco's usually have backup batteries in their
> > outside plant, cell towers, etc.  During power outages, they shuttle
> > small generators between outside cabinets to re-charge the batteries.
> > Remote Terminal Units (RTUs) use local power, i.e. look for the
> > utility meter nearby.  There is often a generator plug and battery
> > cabinet next to the RTU. They aren't powered from the central office.
>
> Interesting! And so primitive! So they go to all of the expense of
> laying fiber, but not power too?

Note: small local telco experience speaking below:

Telco's tend to have experience with fiber, but probably not the
construction and transmission of the sort of power plant that would be
required to keep a bunch of  48V cabinets up and running reliably. We
certainly don't. Besides, an advantage of fiber is that hopefully the
copper thieves won't bother it.

 By definition a remote terminal/cabinet is going to be... remote. Far
more simple to install commercial power, and then haul out a generator
if the battery string in the cabinet appears to be in danger of
dropping below about 46v.

We do run some 360v DC at micro-amp levels out to equipment like ONT's
and remote 12 and 48 port remote VDSLAM's. But that's over existing
24-26 ga. plant. Frequently using multiple pairs to avoid excessive
voltage drop over distances.

Primitive is tested and works.

-- 
Jeff Shultz
Central Office Technician

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: Comcast outages continue even in areas with PG power restored

[Brandon Martin]

On 10/15/19 11:28 AM, Matt Hoppes wrote:

Except I’m not talking about CPE. I agree that’s the customer’s job.


I was confused based on Masataka's focus on CPE power.


I’m talking about keeping the nodes up and running.


Yep.
--
Brandon Martin

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

On 10/14/19 5:58 PM, Brandon Martin wrote:

On 10/14/19 8:26 PM, Michael Thomas wrote:
So when we were working on this 20 years ago at Cisco, there was a 
tremendous amount of effort to deal with the issue of e911 and 
generally battery backup. I'm really surprised to hear that though we 
went through a lot of effort to deal with the CPE, that the cable 
plant was the actual problem. The cable companies should, imo, be 
held to the same standard as the telcos. Maybe even moreso these days 
since IP has taken over everything. The need for reliable e911 hasn't 
gone away just because the bits have turned into IP bit these days.


They get around it, at least in part, by selling it as a "VoIP" 
service rather than "phone service".


AT does the same with U-Verse voice.  You can still buy POTS from 
AT, but it's a separate product with a completely different pricing 
structure from the U-Verse voice product.


Voice over HFC networks is sometimes sold as a POTS-like service. I've 
only heard of this happening in places where the LEC and cable 
provider happen to end up being one-in-the-same.  In those cases, yeah 
uptime is a big deal.



That's what we were working on at Cisco... we partnered with Videotron 
up in Montreal and were trying to boil the entire telco ocean from the 
get-go with all of its features, guarantees, etc. The way it's turned 
out, nobody really cares about 90% of those features so we were wasting 
valuable time to get out an mvp instead. Hindsight, of course.


I never did hear whether all of the elaborate QoS schemes that John 
Chapman and the rest of the docsis folks were working on ended up 
getting used, or whether they really matter much anymore since voice is 
so low bandwidth.


Mike

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

On 10/14/19 6:11 PM, Sean Donelan wrote:

On Mon, 14 Oct 2019, Michael Thomas wrote:
deal with the CPE, that the cable plant was the actual problem. The 
cable companies should, imo, be held to the same standard as the 
telcos. Maybe even moreso these days since IP has taken over 
everything. The need for reliable e911 hasn't gone away just because 
the bits have turned into IP bit these days.



Oh, but they are equal.  The telco's went to the regulators and got 
the FCC and state PUCs to reduce or make backup power a 
customerresponsbility...

Just like the cable companies.

So now they are equal -- in the race to the bottom.

Service providers must "OFFER" customers an OPTION for 8/24-hour 
standby backup power.  The decision to puchase backup power is up to 
the customer.
I assume you read the fine print on the back of your bill or the order 
terms (subject to change at anytime, without notice).




Assuming that this power shutoff in california is the new normal (god 
help us), it would need to be a minimum of 3 days. The wind event itself 
usually lasts a day or two, but then they have to inspect -- completely 
manually from what i can tell -- the entire grid, which takes a day or 
two. Clearly more automation of the inspection is needed (ie, have cams, 
etc, deployed on the grid), and of course that would help with 
prevention which is where pge has fallen completely flat.


It seems to me that if we have to live with unburied power lines, 
cameras and other sensors with some AI-like grinding on the images could 
detect all manner of problems and relay it back to the ops folks for a 
closer look, and/or roll trucks. There are folks at texas a who are 
working on something like this (not sure about the image grinding 
though). Sure it might be expensive, but it's probably cheaper than 
undergrounding, or burning down cities and roasting people alive.


Mike

Re: Comcast outages continue even in areas with PG power restored

[Matt Hoppes]

Except I’m not talking about CPE. I agree that’s the customer’s job. 

I’m talking about keeping the nodes up and running. 

> On Oct 15, 2019, at 7:50 AM, Brandon Martin  wrote:
> 
>> On 10/15/2019 09:42, Matt Hoppes wrote:
>> I disagree with the statement that providers should not be required to 
>> backup their networks while I don't think it should be an FCC 
>> requirement, I do believe the providers have an obligation to do that.
>> That's one of the reasons we generally opt for larger node sizes. Yes, it 
>> does cut down on the total bandwidth you can push out to end users, but in a 
>> disaster its much faster to recover as well as provide backup power that is 
>> up 24x7 vs hundreds of little nodes.
> 
> 
> I'm generally OK with the provider pushing demarc CPE backup power off onto 
> the customer unless they're providing "POTS-like" service.  Aside from POTS, 
> I don't think there's ever been expectation of telecommunications services 
> working during a power outage unless the end user provides backup power.  I'm 
> not even sure residential deployments of ISDN came with power for 
> customer-prem ATAs, etc. unless you made special arrangements, but ISDN was 
> never popular here, so I'm not sure. If you have a T1 PRI, the LEC will 
> probably keep it up to your prem, but you're responsible for keeping whatever 
> you're plugging that T1 into up during a power outage.  Heck, they may not 
> even do that.  I've certainly seen T1 smartjacks hosting real, honest PRIs 
> that were line-powering repeaters without battery on them, so who knows even 
> in that case.
> 
> After all, the user has to keep the equipment they're using up, too. That's 
> certainly not the provider's responsibility.  Telcos aren't power utilities 
> (usually, and when they are it's generally a separate operation).  Yes, I 
> understand that many users have wireless/portable in-home equipment with 
> batteries of their own and are often using all-in-one demarc/routers provided 
> by their provider, nowadays...
> 
> I do think that providers should generally at least offer some basic guidance 
> on how their users might accomplish this even going so far as to provide, 
> install, and maintain such means if paid to do so, but I don't think it's 
> something to be expected when you're buying cable TV or public Internet 
> access service.  Again, POTS-like service is different.
> 
> I do feel that modern IP providers really should strive to keep their end of 
> the network up regardless.  That is, even during a reasonable power outage, 
> providers should be able to deliver usable signal to their customers absent 
> actual damage to plant.  Exceptions made for natural disasters, fuel 
> shortages, and widespread, extended power outages that just overcome the 
> ability to shuttle portable gensets around fast enough to keep batteries 
> charged.  However, unless you're otherwise subject to regulation due to 
> monopoly status, having received ample amounts of build-out subsidy, or 
> providing historically regulated telecommunications services, I'm not sure 
> that regulation is the best way to achieve this.
> 
> --
> Brandon Martin

Re: Comcast outages continue even in areas with PG power restored

[Brandon Martin]

On 10/15/2019 09:42, Matt Hoppes wrote:
I disagree with the statement that providers should not be required to 
backup their networks while I don't think it should be an FCC 
requirement, I do believe the providers have an obligation to do that.


That's one of the reasons we generally opt for larger node sizes. Yes, 
it does cut down on the total bandwidth you can push out to end users, 
but in a disaster its much faster to recover as well as provide backup 
power that is up 24x7 vs hundreds of little nodes.



I'm generally OK with the provider pushing demarc CPE backup power off 
onto the customer unless they're providing "POTS-like" service.  Aside 
from POTS, I don't think there's ever been expectation of 
telecommunications services working during a power outage unless the end 
user provides backup power.  I'm not even sure residential deployments 
of ISDN came with power for customer-prem ATAs, etc. unless you made 
special arrangements, but ISDN was never popular here, so I'm not sure. 
If you have a T1 PRI, the LEC will probably keep it up to your prem, but 
you're responsible for keeping whatever you're plugging that T1 into up 
during a power outage.  Heck, they may not even do that.  I've certainly 
seen T1 smartjacks hosting real, honest PRIs that were line-powering 
repeaters without battery on them, so who knows even in that case.


After all, the user has to keep the equipment they're using up, too. 
That's certainly not the provider's responsibility.  Telcos aren't power 
utilities (usually, and when they are it's generally a separate 
operation).  Yes, I understand that many users have wireless/portable 
in-home equipment with batteries of their own and are often using 
all-in-one demarc/routers provided by their provider, nowadays...


I do think that providers should generally at least offer some basic 
guidance on how their users might accomplish this even going so far as 
to provide, install, and maintain such means if paid to do so, but I 
don't think it's something to be expected when you're buying cable TV or 
public Internet access service.  Again, POTS-like service is different.


I do feel that modern IP providers really should strive to keep their 
end of the network up regardless.  That is, even during a reasonable 
power outage, providers should be able to deliver usable signal to their 
customers absent actual damage to plant.  Exceptions made for natural 
disasters, fuel shortages, and widespread, extended power outages that 
just overcome the ability to shuttle portable gensets around fast enough 
to keep batteries charged.  However, unless you're otherwise subject to 
regulation due to monopoly status, having received ample amounts of 
build-out subsidy, or providing historically regulated 
telecommunications services, I'm not sure that regulation is the best 
way to achieve this.


--
Brandon Martin

Re: Comcast outages continue even in areas with PG power restored

[Matt Hoppes]

I disagree with the statement that providers should not be required to  
backup their networks while I don't think it should be an FCC  
requirement, I do believe the providers have an obligation to do that.


That's one of the reasons we generally opt for larger node sizes.  Yes,  
it does cut down on the total bandwidth you can push out to end users,  
but in a disaster its much faster to recover as well as provide backup  
power that is up 24x7 vs hundreds of little nodes.

Re: Comcast outages continue even in areas with PG power restored

[Masataka Ohta]

Sean Donelan wrote:

Given that providers can't supply power to mobile phones,
that sending power over fiber is extremely eye unsafe
and that most CPEs are routers which themselves are
useless without end systems, it is reasonable that
providers are not required to supply power to home.

But,


The FCC is looking at standby power for cellular towers, but hasn't
been paying attention to wireline and cable systems outside plant
power.


why they are not equal?

Masataka Ohta

Re: Comcast outages continue even in areas with PG power restored

[Sean Donelan]

On Mon, 14 Oct 2019, Michael Thomas wrote:
deal with the CPE, that the cable plant was the actual problem. The cable 
companies should, imo, be held to the same standard as the telcos. Maybe even 
moreso these days since IP has taken over everything. The need for reliable 
e911 hasn't gone away just because the bits have turned into IP bit these 
days.



Oh, but they are equal.  The telco's went to the regulators and got the 
FCC and state PUCs to reduce or make backup power a customerresponsbility...

Just like the cable companies.

So now they are equal -- in the race to the bottom.

Service providers must "OFFER" customers an OPTION for 8/24-hour standby 
backup power.  The decision to puchase backup power is up to the customer.
I assume you read the fine print on the back of your bill or the order 
terms (subject to change at anytime, without notice).



The FCC is looking at standby power for cellular towers, but hasn't 
been paying attention to wireline and cable systems outside plant power.


As I mentioned a few postings ago, cellular/wireless systems have been 
getting more resiliant.  Wireline/cable systems have been getting less 
reliable.

Re: Comcast outages continue even in areas with PG power restored

[Brandon Martin]

On 10/14/19 8:26 PM, Michael Thomas wrote:
So when we were working on this 20 years ago at Cisco, there was a 
tremendous amount of effort to deal with the issue of e911 and generally 
battery backup. I'm really surprised to hear that though we went through 
a lot of effort to deal with the CPE, that the cable plant was the 
actual problem. The cable companies should, imo, be held to the same 
standard as the telcos. Maybe even moreso these days since IP has taken 
over everything. The need for reliable e911 hasn't gone away just 
because the bits have turned into IP bit these days.


They get around it, at least in part, by selling it as a "VoIP" service 
rather than "phone service".


AT does the same with U-Verse voice.  You can still buy POTS from 
AT, but it's a separate product with a completely different pricing 
structure from the U-Verse voice product.


Voice over HFC networks is sometimes sold as a POTS-like service.  I've 
only heard of this happening in places where the LEC and cable provider 
happen to end up being one-in-the-same.  In those cases, yeah uptime is 
a big deal.


I think what happens is that the standards get written and equipment 
designed with the assumption that everybody will be deploying all sorts 
of SLA'd, guaranteed services, then 99% of deployments end up being 
exclusively best-effort because it's so much easier and cheaper to deploy.


GPON seems to be an interesting case of this since it's commonly 
deployed by telcos rather than cable MSOs, and, in greenfield 
applications, is often deployed exclusive to copper plant at all.  It's 
pretty common to find GPON ONTs with inbuild UPS monitoring and 
communications as well as ATAs designed to deliver POTS-like service, 
but then a lot of SPs who are NOT the LEC of record just use that 
infrastructure to deliver VoIP-like services and push the UPS 
responsibility off onto the customer.

--
Brandon Martin

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

On 10/14/19 4:39 PM, Brandon Martin wrote:



All the conventional telcos are far more focused on keeping voice 
service alive since they get raked over the coals by the FCC if it 
drops due to lack of 911.  That includes wireless if they are both a 
wireline and wireless operator.  Interestingly, VoIP service delivered 
to the customer as such (even if there's an ATA built into the 
customer's "modem") often gets a pass on this since it's not 
considered POTS.  Same goes for SLA'd business services, too 
especially T1s since those may host regulated voice.


So when we were working on this 20 years ago at Cisco, there was a 
tremendous amount of effort to deal with the issue of e911 and generally 
battery backup. I'm really surprised to hear that though we went through 
a lot of effort to deal with the CPE, that the cable plant was the 
actual problem. The cable companies should, imo, be held to the same 
standard as the telcos. Maybe even moreso these days since IP has taken 
over everything. The need for reliable e911 hasn't gone away just 
because the bits have turned into IP bit these days.


Mike

Re: Comcast outages continue even in areas with PG power restored

[Lyle]

All true telco equipment is powered by batteries.  Commerical power or 
generators just recharge the batteries.  No switch over when commerical 
power is lost. Except when the generators(where equiped) switch over to 
recharge the batteries.



Comcast and telcos do not put batteries in all remote powered 
terminals.  I have an Enterprise grade Ethernet over coax connection.  
The headend it's distributed from doesn't have batteries.  If it loses 
power, doesn't matter if I have power or a generator or ups to take 
over.  This Internet connection goes down.



For telcos(when I worked there), they usually had batteries that would 
last 4 to 8 hrs at remote terminals with powered equipment. And a 
connection for a splice crew to come out and connect their generator to 
it for power in case of an extended outage.  Back then that was also how 
most cell phone towers were outfited.



I also have some knowledge of the commerical power grid in my local 
area.  It's not unheard of for the Comcast headend to lose power but my 
office doesn't.


Lyle Giese
LCR Computer Services, Inc.

On 10/14/19 17:38, Michael Thomas wrote:


On 10/14/19 3:06 PM, Sean Donelan wrote:


That is not why people are surprised.  When the house doesn't have 
power, and doesn't have home generator or UPS, (most) people are less 
surprised their DSL or Cable modem and VOIP doesn't work anymore.


The reasons I saw people angry on twitter was no Comcast service even 
when they had power at the house (utility, generator, UPS). Their 
Comcast service died quickly, even when the home had power but the 
Comcast outside plant didn't seem to have any backup power.


DSL modems also need power at the home, but the telco providers seem 
to have more backup power in the outside plant or central offices.  
That meant DSL worked as long as the house had power (or a home 
generator or UPS).


So it turns out that our local telco/isp does keep dsl running via the 
same mechanism as they keep pots power backed up (i'm guessing it's a 
diesel generator at the co, but am not sure). It seems that a lot of 
the pedestals terminating the local loop these days do the conversion 
to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get 
power to the pedestal, but these were all a home run to the co at one 
time so it probably wasn't hard to power them from the co. For all i 
know, that's how they're all powered all the time, with a transfer 
switch at the co, rather than tapping the local grid next to the 
pedestal.


Of course this is a lot of conjecture on my part... be glad to be 
clued in by folks in know.


Mike

Re: Comcast outages continue even in areas with PG power restored

[Brandon Martin]

On 10/14/19 6:38 PM, Michael Thomas wrote:


So it turns out that our local telco/isp does keep dsl running via the 
same mechanism as they keep pots power backed up (i'm guessing it's a 
diesel generator at the co, but am not sure). It seems that a lot of the 
pedestals terminating the local loop these days do the conversion to IP 
right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to 
the pedestal, but these were all a home run to the co at one time so it 
probably wasn't hard to power them from the co. For all i know, that's 
how they're all powered all the time, with a transfer switch at the co, 
rather than tapping the local grid next to the pedestal.


Of course this is a lot of conjecture on my part... be glad to be clued 
in by folks in know.


Legacy carriers with existing copper plant being overbuilt by fiber 
often use "span power" at 48V or ~190VDC from the nearest central 
facility (which may be an old-school CO or RTU with utility power) to 
power their distributed plant that's deep into the network near the 
customer edge.  There's usually then also some local battery as well but 
not much.  The assumption is that the legacy copper, now being used 
simply for power, follows essentially the same routes as the fiber and 
will drop at the same time.  In a cable cut situation this is often 
true, but in a "cable rot" situation it's obviously not, and fiber paths 
aren't always the same as power paths.


The CO and older RTUs from the POTS and early ADSL era will have utility 
power as primary with somewhat extensive battery facilities.  Old-school 
CO will have a lot more battery capacity than an RTU.  An RTU will 
usually end up with a portable genset being delivered during an extended 
power outage as mentioned.  These RTUs still host legacy POTS and TDM 
services that either have serious SLAs on them or regulatory uptime 
considerations whereas distributed peds are normally best-effort, 
non-regulated data services only in many cases precisely to keep the 
reliability requirements (and therefore cost) on them down.


SBC/AT's pedestals they built for Lightspeed (U-Verse) do usually have 
local utility with a few hours of backup.  I'm not sure why the went 
that route rather than span power.  Might be that the early VDSL DSLAMs 
just used too much power for that.  They seem to only have maybe a day 
or so of battery before they need a portable generator brought around 
which AT at least has procedures for (whether they are executed or not 
is another matter).  They're something of a hybrid between a 
conventional RTU and modern distributed pedestal.


All the conventional telcos are far more focused on keeping voice 
service alive since they get raked over the coals by the FCC if it drops 
due to lack of 911.  That includes wireless if they are both a wireline 
and wireless operator.  Interestingly, VoIP service delivered to the 
customer as such (even if there's an ATA built into the customer's 
"modem") often gets a pass on this since it's not considered POTS.  Same 
goes for SLA'd business services, too especially T1s since those may 
host regulated voice.



Coax operators have historically had less need for reliability as they 
were originally built purely for convenience services (cable television) 
and have been pressed into service for more modern data needs.  Those 
"Alpha boxes" you may see around providing line power do usually have 
some batteries in them, though they're often ill-maintained and only 
provide maybe an hour of hold-up at most.


Exclusively residential areas sometimes have zero hold-up ability at 
all.  They'll drop at least outside node based digital services (e.g. 
DOCSIS) as soon as power falls along with anything being distributed 
into the field on AM fiber carriers.  It's not unheard of for 
conventional linear TV to still be delivered into the field from a 
head-end at baseband on coax and sometimes that'll stay up longer 
depending on RF power/split budget as long as the local RF head end 
still has power.



Pure-play fiber carriers, especially PON-based, get to turn what is 
often a curse in terms of design into a blessing, here.  They usually 
have almost no active outside plant or, if they do, it's less 
distributed and can afford reasonable backup power infrastructure. 
While it's annoying to have no power available as you approach customer 
prem, that forces you to make choices such that there's no worry about 
backup until you can (usually) just make it the customer's 
responsibility right at the demarc.  Your CO/head-end needs backup, of 
course, but that's usually a facility that can afford it.

--
Brandon Martin

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

On 10/14/19 4:16 PM, Sean Donelan wrote:

On Mon, 14 Oct 2019, Michael Thomas wrote:
Of course this is a lot of conjecture on my part... be glad to be 
clued in by folks in know.


An old news story, but telco's usually have backup batteries in their 
outside plant, cell towers, etc.  During power outages, they shuttle 
small generators between outside cabinets to re-charge the batteries. 
Remote Terminal Units (RTUs) use local power, i.e. look for the 
utility meter nearby.  There is often a generator plug and battery 
cabinet next to the RTU. They aren't powered from the central office.


Interesting! And so primitive! So they go to all of the expense of 
laying fiber, but not power too?


Mike

Re: Comcast outages continue even in areas with PG power restored

[Sean Donelan]

On Mon, 14 Oct 2019, Michael Thomas wrote:
Of course this is a lot of conjecture on my part... be glad to be clued in by 
folks in know.


An old news story, but telco's usually have backup batteries in their 
outside plant, cell towers, etc.  During power outages, they shuttle small 
generators between outside cabinets to re-charge the batteries. Remote 
Terminal Units (RTUs) use local power, i.e. look for the utility meter 
nearby.  There is often a generator plug and battery cabinet next to the 
RTU. They aren't powered from the central office.


Some cable systems have battery and/or generator backup on their "I-Net" 
cable plant serving government and major businesses, but not on their 
residential cable plant.  I don't know Comcast's business practices.



Old news story:

https://www.multichannel.com/news/att-will-replace-batteries-after-fires-130936
ORIGINAL: JAN 18, 2008

AT Will Replace Batteries After Fires

City officials, long critical of the size and placement of powering 
cabinets needed to back up AT’s U-verse TV video service, now have 
concerns beyond aesthetics. Sometimes, the cabinets explode.


AT acknowledged the problem and said it would replace 17,000 lithium 
batteries in outdoor cabinets around the country.


[...]

The steel cabinets house controls and backup power supplies for the video 
network.


“They’ve been pretty cooperative,” Kesner said of AT “We’re in a 
holding pattern” regarding the video deployment, he said.


[...]

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

On 10/14/19 3:06 PM, Sean Donelan wrote:


That is not why people are surprised.  When the house doesn't have 
power, and doesn't have home generator or UPS, (most) people are less 
surprised their DSL or Cable modem and VOIP doesn't work anymore.


The reasons I saw people angry on twitter was no Comcast service even 
when they had power at the house (utility, generator, UPS). Their 
Comcast service died quickly, even when the home had power but the 
Comcast outside plant didn't seem to have any backup power.


DSL modems also need power at the home, but the telco providers seem 
to have more backup power in the outside plant or central offices.  
That meant DSL worked as long as the house had power (or a home 
generator or UPS).


So it turns out that our local telco/isp does keep dsl running via the 
same mechanism as they keep pots power backed up (i'm guessing it's a 
diesel generator at the co, but am not sure). It seems that a lot of the 
pedestals terminating the local loop these days do the conversion to IP 
right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to 
the pedestal, but these were all a home run to the co at one time so it 
probably wasn't hard to power them from the co. For all i know, that's 
how they're all powered all the time, with a transfer switch at the co, 
rather than tapping the local grid next to the pedestal.


Of course this is a lot of conjecture on my part... be glad to be clued 
in by folks in know.


Mike

Re: Comcast outages continue even in areas with PG power restored

[Sean Donelan]

On Fri, 11 Oct 2019, Ted Hatfield wrote:

First of all DSL is not pots.


[]


DSL is a data service that runs on the subscriber loop at the same time as 
the voice service.  This service is not required to be battery backed and 
will invariably stop working when power is cut at the customer end point.



That is not why people are surprised.  When the house doesn't have power, 
and doesn't have home generator or UPS, (most) people are less surprised 
their DSL or Cable modem and VOIP doesn't work anymore.


The reasons I saw people angry on twitter was no Comcast service even when 
they had power at the house (utility, generator, UPS). Their Comcast 
service died quickly, even when the home had power but the Comcast outside 
plant didn't seem to have any backup power.


DSL modems also need power at the home, but the telco providers seem to 
have more backup power in the outside plant or central offices.  That 
meant DSL worked as long as the house had power (or a home generator or 
UPS).



I know, rich people problems.  Rich people can afford backup generators 
and got upset when their Internet and TV didn't work.

Re: Comcast outages continue even in areas with PG power restored

[Ted Hatfield]

On Fri, 11 Oct 2019, Michael Thomas wrote:




On 10/11/19 4:31 PM, Sean Donelan wrote:
  The FCC asked a half-dozen carriers about their network resilience plans 
last month.  Comcast was not one of the
  service providers askedd about their plans.

  The FCC should have looked closer at Comcast in California. While it was 
expected many people would loose home
  Internet, voice, video service when their Customer Premise Equipment lost 
power.  The FCC no longer requires
  battery backup for CPE.  That is now a customer responsibility.

  It turns out, Comcast's outside plant was woefully unprepared to handle 
long, i.e. 24 hour, power outages.  And
  even when power is restored to people's homes, Comcast service is often 
still down.

So I knew that telcos are required to battery backup pots, but are isp's too? I 
have a dinky little provider who also provides
pots, but i have never been clear whether dsl stays up too in a blackout.

Mike



First of all DSL is not pots.

Traditional voice services run on a subscriber loop which is a pair 
of copper lines running from the central office to the customer end point.


This analog voice service is almost always backed up with a bank of 
batteries so that the service continues to run in the event of an 
emergency.


DSL is a data service that runs on the subscriber loop at the same time as 
the voice service.  This service is not required to be battery backed and 
will invariably stop working when power is cut at the customer end point.


Ted

Re: Comcast outages continue even in areas with PG power restored

[John Levine]

In article  you write:
>On 10/11/19 9:43 PM, Matt Hoppes wrote:
>How distributed is the power on a typical HFC system in practice?  I'm 
>sure I'm missing some of them, but having walked out most of a small-ish 
>(~2000 residences) city recently for a FTTx deployment, I think I only 
>saw 2-3 power nodes on Comcast's plant.

I spend too much time looking up at the power lines while walking the
dog and around here, Spectrum ex-Roadrunner, on just about every block
I see something on the cable plant with an electric meter.  I can't
tell how many are amplifiers and how many are fiber to coax adapters.
None have any evident batteries although I suppose there might be some
in the cabinets.

My current phone and Internet service is FTTH from the local RLEC.
The box they installed here is powered from a 12v UPS and I'm
reasonably sure there are no active components between here and the CO
since it's only 1/4 mile away, so I'll be interested to see how it
does when the power goes out.

Re: Comcast outages continue even in areas with PG power restored

[Jared Mauch]

> On Oct 11, 2019, at 9:43 PM, Matt Hoppes  
> wrote:
> 
> And this is why the distributed nature of small node’s is detrimental in an 
> extended power outage.
> 
> There is no practical way to back them up with power for an extended period 
> of time.

This is why I’m concerned about a future with micro-cell sites, we will have a 
few fortresses with cell service but the day-to-day service won’t look anything 
like the outage service and is likely to fail catastrophically as a result.

- jared

Re: Comcast outages continue even in areas with PG power restored

[Sean Donelan]

On Fri, 11 Oct 2019, Michael Thomas wrote:

So I knew that telcos are required to battery backup pots, but are isp's
too? I have a dinky little provider who also provides pots, but i have never
been clear whether dsl stays up too in a blackout.


Of course generalizing all service providers isn't fair... From my 
experience tracking telecommunication during disasters for the last 20-30 
years... Let me generalize (ignoring special goverment priority systems).



Generally, during natural (and man-made) disasters:

Telecommunication service providers historically failed in this order

1. VSAT/DTH/satellite (during weather events)
2. Cable
3. Cellular/wireless
4. Telco/Wireline
5. Broadcast radio/TV (less than 20% over-the-air stations operating)
6. Network backbone systems (inter-city and toll offices)

There are too few WISPs for reliable predictions.  I'd guess WISPs 
reliability is similar to cellular/wireless systems.



Restoration order is a bit different. Telecommunications network service 
historically recovers in this order, assuming customer premise isn't 
damaged:


1. VSAT/DTH/satellite (after weather clears)
2. Network backbone (inter-city and toll offices)
3. Cellular/wireless (COWs and COLTs deployed)
4. Broadcast radio/TV (20% over-the-air stations operating)
5. Telco/Wireline
5. Cable

Cable systems tend to be the first to fail, and the last to be restored.
Telco systems tend to fail later, but take a long time to be restored.

Network backbones can take a while to repair, but generally nothing else 
works until they are repaired, so they get repaired first or second.


Note: During even the worst catastrophes, there is almost always one or 
two broadcast radio stations still operating.  I set 20% radio/TV 
stations operating as an arbitrary minimum level.  Likewise, COWs and 
COLTs don't provide full cellular service, but do provide minimumal cell 
services.



In the last 10 years, cellular/wireless system resiliance has been 
improving while telco/wireline system resiliance has been getting 
noticablly worse.  I assume this a flywheel affect as telco companies 
have been shifting infrastructure investement to wireless networks and 
away from wireline networks for the last 20 years.

Re: Comcast outages continue even in areas with PG power restored

[Tom Beecher]

It's very difficult to properly build a resilient infrastructure when those
shareholders must get their value!

On Fri, Oct 11, 2019 at 7:33 PM Sean Donelan  wrote:

> The FCC asked a half-dozen carriers about their network resilience plans
> last month.  Comcast was not one of the service providers askedd about
> their plans.
>
> The FCC should have looked closer at Comcast in California. While it was
> expected many people would loose home Internet, voice, video service when
> their Customer Premise Equipment lost power.  The FCC no longer requires
> battery backup for CPE.  That is now a customer responsibility.
>
> It turns out, Comcast's outside plant was woefully unprepared to handle
> long, i.e. 24 hour, power outages.  And even when power is restored to
> people's homes, Comcast service is often still down.
>
>
> https://www.ktvu.com/news/still-no-relief-pge-shutoffs-disrupt-comcast-services
>
>
> Maybe the FCC needs to ask about cable system Outside Service Plant
> network reslience of Comcast.  Are other cable OSP resilience similar?
>

Re: Comcast outages continue even in areas with PG power restored

[Chris Adams]

Once upon a time, Sean Donelan  said:
> It turns out, Comcast's outside plant was woefully unprepared to
> handle long, i.e. 24 hour, power outages.  And even when power is
> restored to people's homes, Comcast service is often still down.

When I had Comcast in Huntsville, AL, there appeared to be no backup
power in their plant.  Any power blink and my Internet and TV both
dropped (my equipment is on UPS).
-- 
Chris Adams 

Re: Comcast outages continue even in areas with PG power restored

[Brandon Martin]

On 10/11/19 9:43 PM, Matt Hoppes wrote:

And this is why the distributed nature of small node’s is detrimental in an 
extended power outage.

There is no practical way to back them up with power for an extended period of 
time.


How distributed is the power on a typical HFC system in practice?  I'm 
sure I'm missing some of them, but having walked out most of a small-ish 
(~2000 residences) city recently for a FTTx deployment, I think I only 
saw 2-3 power nodes on Comcast's plant.  There were several times as 
many fiber-coax nodes being line-powered off the coax plant though still 
surprisingly few (the plant is ancient and hasn't seen a lot of fiber 
overbuild).


That's comparable to how many powered RTUs the LEC had in town and many 
fewer utility-powered field nodes than would have been present if it had 
been in AT Lightspeed territory.


Now, I have no idea what the backup line power is in practice on that 
Comcast plant.  I know that Bright House/Spectrum, in an another area 
I've supported, has very little backup on many residential-only parts of 
their plant.  I've observed that they have, in practice, maybe 15-30 
minutes of hold-up before DOCSIS nodes start dropping.

--
Brandon Martin

Re: Comcast outages continue even in areas with PG power restored

[Matt Hoppes]

And this is why the distributed nature of small node’s is detrimental in an 
extended power outage.

There is no practical way to back them up with power for an extended period of 
time.

> On Oct 11, 2019, at 8:44 PM, Sean Donelan  wrote:
> 
> 
> Why you don't have Comcast service during a power outage:
> 
> Throughout the state, Comcast equipment was knocked offline by PG’s power 
> shutdown, Hammel said. The cable company was “only using generators in very 
> discrete and specific cases where there’s a demonstrated need,” such as a 
> request from the Federal Emergency Management Agency, she said.

Re: Comcast outages continue even in areas with PG power restored

[Sean Donelan]

Why you don't have Comcast service during a power outage:

Throughout the state, Comcast equipment was knocked offline by PG’s 
power shutdown, Hammel said. The cable company was “only using generators 
in very discrete and specific cases where there’s a demonstrated need,” 
such as a request from the Federal Emergency Management Agency, she said.

Re: Comcast outages continue even in areas with PG power restored

[Michael Thomas]

On 10/11/19 4:31 PM, Sean Donelan wrote:
The FCC asked a half-dozen carriers about their network resilience 
plans last month.  Comcast was not one of the service providers askedd 
about their plans.


The FCC should have looked closer at Comcast in California. While it 
was expected many people would loose home Internet, voice, video 
service when their Customer Premise Equipment lost power. The FCC no 
longer requires battery backup for CPE.  That is now a customer 
responsibility.


It turns out, Comcast's outside plant was woefully unprepared to 
handle long, i.e. 24 hour, power outages.  And even when power is 
restored to people's homes, Comcast service is often still down.


So I knew that telcos are required to battery backup pots, but are isp's 
too? I have a dinky little provider who also provides pots, but i have 
never been clear whether dsl stays up too in a blackout.


Mike

Comcast outages continue even in areas with PG power restored

[Sean Donelan]

The FCC asked a half-dozen carriers about their network resilience plans 
last month.  Comcast was not one of the service providers askedd about 
their plans.


The FCC should have looked closer at Comcast in California. While it was 
expected many people would loose home Internet, voice, video service when 
their Customer Premise Equipment lost power.  The FCC no longer requires 
battery backup for CPE.  That is now a customer responsibility.


It turns out, Comcast's outside plant was woefully unprepared to handle 
long, i.e. 24 hour, power outages.  And even when power is restored to 
people's homes, Comcast service is often still down.


https://www.ktvu.com/news/still-no-relief-pge-shutoffs-disrupt-comcast-services


Maybe the FCC needs to ask about cable system Outside Service Plant 
network reslience of Comcast.  Are other cable OSP resilience similar?