Re: power to the internet

[Fred Baker]

> This time it’s PG all alone, but still fallout from back then. Too much 
> liability and they’ve not maintained the infrastructure and so they decided 
> that to reduce the liability costs it’s cheaper to blackout. Same story again 
> different colors. PG making a mint while people get screwed (PG was 
> mostly at the getting screwed end in 2000-2001)

PG has been the one in the news, but SCE appears to have been making the same 
choices with about the same effects. The Thomas Fire was briefly the largest 
wildfire in state history, and the source (well, with the rain) of the 
Montecito mud flow a few weeks later. We're told that SCE seems to figure in 
that one and several others before and since.

I go back and forth on who might be responsible. The electric utilities bear 
blame for their infrastructure; it should be underground, not strung from 
poles. I would put some to the state and the management of the various national 
forests and national parks in the area - one of the outcomes from a fire in 
2007 or thereabouts was that the ecology folks had been protecting foliage, and 
that foliage burned and clogged streams, with all sorts of results. Surprise! 
If you're worried about ecology, you should support management of it. In 
California, there are also laws holding home-owners responsible for "defensible 
space" around their homes.

https://www.google.com/search?q=california+brush+clearing+laws
https://en.m.wikipedia.org/wiki/Thomas_Fire 

https://www.washingtonpost.com/graphics/2018/national/montecito-before-after/ 
https://www.edhat.com/news/10-homes-destroyed-in-holiday-fire 
https://www.edhat.com/news/cave-fire-now-100-contained

Re: power to the internet

[Don Gould]

Hi Brandon,

I agree with lots of what you wrote, here's some more thoughts

tl;dr

Batteries from cars will fix your issues along with smart guys just 
wanting to get the job done.


WHO AM I?!

I am a coms tech.  I have been on this list for ~20 years.

My job is simply to look at the environment and make coms appear.

My job is not to blame others, question politics or argue ethics.


MY POWER, MY RULES, MY RIGHTS

I have two Nissan LEAF with the 24Kwh battery, both at 10 of 12 bars of 
original battery capacity, roughly 8.5 years old.  This gives each car 
roughly 110km (65 mile) of range between charges. Both are 6 months into 
a 5 year payment plan.


"My job is simply to look at the environment and make coms appear. "

So, looking at my actual environment, I know that I'm going to end up 
with two 15kwh batteries in about 8 years and two 15kwh generators (yes, 
the motors in those things put out about 15kwh under breaking).


Ok, so I'm not going to personally convert these into anything at 57 
(I'm 49 now), but some young up and coming (Mr 12, sitting in my living 
room, playing with his Christmas presents, perhaps) will.


MY POWER

In buying these two cars, I was wanting to address the 'demand side' of 
my energy question.  Now I have a decent amount of demand, I'm going to 
put PV on my roof this year.


My utility company will buy power back from me at about 15% of what I 
pay retail, so selling them power makes no sense.


My best friend has just completed a 7kwh in solar installation to power 
is 210 tank gold fish facility.  He's currently exporting power and now 
looking at storage solutions.


We're 'early adoptors', so you can look to us to see where the trend is 
going.


MY RULES

My other mate asked our utility for power to his new home and they said 
"Yes sir, half a years wages plus a monthly contribution to the up keep 
for the brand new plant you're paying for up front." I'm sure it comes 
as no surprise, he said "no thanks" and then built a whole off grid 
system including a 6kwh diesel generator. (He is also a coms tech, 20 
years older than I am, though not on this list).


MY RIGHTS

My country lets me do most of the electrical work on my own home up to 
the power board as long as I follow the rules.


I can generate power, I can export power, I can store power.

WHERE ARE WE GOING?

Just like the US, my community has people who will sit and blame 
government, will blame the power company, will blame their local elected 
members, will drag us all into court, will blame the boomers.


But also like the US my community also has people who are coms techs who 
just look at the space, see what's going on and build accordinly.


TRANSITION IS A PROBLEM TO SOLVE AT THE LOCAL LEVEL

As many of you are picking up, we're in a transitional time.  The US 
utilities know this and they're scared to invest, and for good reason, 
some have been very burnt in the past.


NOT ENOUGH COPPER  "COMS WILL DIE"

I remember ~20 years ago, reading posts about how there isn't enough 
copper to supply the growing demand in lines.  Now today I'm reading 
about how you have abundant abandoned copper as DSLAMs are moved closer 
to the edge.


Power is the same in my view.  We're going to see local edge generation, 
storage and change of use (I have all LED lighting, TV's that use a 
fraction of the power they used to, but two electric cars).


Your US power will stop browning out when you put storage in the network 
and can drop the peek load.  But you're going to have to drive Mrs Brown 
to make that personal investment, and she won't until it starts to hurt.


The utility will under ground lines when it can see a clearer picture 
for the future.  It will take coms with it (FTTH).  It won't until the 
situation smacks the political space so hard that regulation is sorted 
so it can be happy with investment (in NZ we had to get down this path - 
this is quite a good read, the incumbent - 
https://en.wikipedia.org/wiki/Spark_New_Zealand ).


RANDOM FYI

As I type this, I have over 8 layer 1 networks I can use at my house...

* ENABLED - FTTH
* VODAFONE - HFC
* CHORUS - FTTN/POTS
* VODAFONE - 4/5G
* SPARK - 4G
* 2DEGREES - 4G
* CCC - private council network
* YOURNET - my own private WISP network
* OTHER WISP - there's half a dozen that I can see with a quick scan 
from hills about 8km away


BUT THIS WAS ABOUT POWER FOR OUR EXISTING COMS, NOT THE WHOLE NEIGHBOURHOOD!

Many of you are thinking in a silo.  You live in a community and you're 
not acting like it.


Fix the power problems for the community rather than just trying to care 
only for your little nest.  As some of you have pointed out, you're 
lugging gas to generators, connecting trucks with inverters just to keep 
the coms going in case there's a 911 call required, without thinking 
that in the even of such, the emergency service won't be able to get to 
Mrs Brown because the streets are blocked by your trucks all connected 
to random 

Re: power to the internet

[Michael Thomas]

On 12/26/19 4:06 PM, John Levine wrote:

In article  you write:

run but are now showing their long term consequences, notably land use
that encourages sprawl and construction in ill-suited areas

If we stopped construction in all of the ill-suited areas, we'd stop
construction all together, and tear down much more. We have it all here:
earthquakes, floods, fires; often the trifecta.  We could certainly be
smarter, but the nature of the geography here is both a blessing and a
curse.

Among California's many problems is a bizarre terror of upzoning and
infill construction, hence the sprawl.  Here in my rustic bit of
upstate New York you can build a two-family anywhere you can build a
single family and the world has not come to an end.



NIMBYism. My previous state senator (Scott Wiener) has been trying all 
he can to make headway on that front. But NIMBY's are a strong force and 
don't cleave down party lines whatsoever.





PG is especially egregious as it has extremely high rates and
piss-poor maintenance. Where does all of that money go? Execs and
shareholders.

Evidently not since they've been through bankruptcy a few times.  I
think they're just institutionally incompetent as well as having an
unusually environmentally hostile territory to serve.  (Around here when
the power company screws up, the power fails but the county does not
catch fire.)
Well that was true here until about 10-20 years ago too. Fire seasons 
are about 2 months longer, iirc. From beginning of May into first part 
of December. We almost never had fires in June but now they're fairly 
common. That's true for a lot of western US now.



I don't know what the ultimate solution is, but
whatever it is cannot have those perverse incentives.

The LA DWP seems to do OK.


As does Sacramento's SMUD. Part of the problem is that they are just so 
large. San Jose and SF are thinking very seriously about splitting off. 
Which would probably create yet another death spiral for PG because it 
would leave all of the expensive distribution (= out in the boonies, 
etc) and allow cities to cherry pick the cheaper distribution areas when 
it makes sense. The entire thing is a shitshow.


Mike

Re: power to the internet

[John Levine]

In article  you write:
>> run but are now showing their long term consequences, notably land use
>> that encourages sprawl and construction in ill-suited areas
>
>If we stopped construction in all of the ill-suited areas, we'd stop 
>construction all together, and tear down much more. We have it all here: 
>earthquakes, floods, fires; often the trifecta.  We could certainly be 
>smarter, but the nature of the geography here is both a blessing and a 
>curse.

Among California's many problems is a bizarre terror of upzoning and
infill construction, hence the sprawl.  Here in my rustic bit of
upstate New York you can build a two-family anywhere you can build a
single family and the world has not come to an end.

>PG is especially egregious as it has extremely high rates and 
>piss-poor maintenance. Where does all of that money go? Execs and 
>shareholders.

Evidently not since they've been through bankruptcy a few times.  I
think they're just institutionally incompetent as well as having an
unusually environmentally hostile territory to serve.  (Around here when
the power company screws up, the power fails but the county does not
catch fire.)

>I don't know what the ultimate solution is, but 
>whatever it is cannot have those perverse incentives.

The LA DWP seems to do OK.

R's,
John

Re: power to the internet

[Brandon Butterworth]

On Thu Dec 26, 2019 at 11:20:01AM -0800, Michael Thomas wrote:
> I just looked up Telsa's battery packs and they seem to be between 
> 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 
> days. Admittedly we can turn off the hot tub, water heater, etc to 
> stretch it out. And of course, that means that you can't drive it... The 
> one thing that would be for everybody's good is using them during peak 
> hours. If you work normal hours, then that only gets part of the peak 
> load, unfortunately.

Many with a tesla (or three) are likely to get some local PV/wind
generation to charge them (more so if the outages become a regular
event). Then the base load doesn't matter and they can still drive
some of them while the others charge. Some may just use their EV as
a battery that has other uses when the power isn't out and not care
they can't use it for both. There are plenty of projects working on
this distributed storage/generation model.

> But of course this has nothing to do with the network power problem. I 
> assume they won't be parking a Tesla next to a CMTS headend.

Proliferation of local generation and storage may have a lot of impact
on network expectations. Last mile that assumes when the cab loses
power then the consumers have too will have to update their assumptions.
The consumers running on local power are able to carry on as normal
and expect network to carry on too.

Architectures with lots of distributed small active plant may be
harder hit trying to add generation than those with passive plant
and few active nodes.

brandon

Re: power to the internet

[Constantine A. Murenin]

On Thu, 26 Dec 2019 at 13:19, Stephen Satchell  wrote:

> Longer term, review your backhauls and interconnects.  Dark fiber would
> be preferred here, because you would be controlling backup power at both
> ends, and not depending on intermediate nodes.
>

What about the NSA taps?  Do they tap the dark fibre?  Would it go dark in
a power outage, or do they engage some passive sort of taps?

C.

RE: power to the internet

[Keith Medcalf]

>I just looked up Telsa's battery packs and they seem to be between
>60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3
>days. Admittedly we can turn off the hot tub, water heater, etc to
>stretch it out. And of course, that means that you can't drive it... The
>one thing that would be for everybody's good is using them during peak
>hours. If you work normal hours, then that only gets part of the peak
>load, unfortunately.

Just buy three of them.  Two to leave in the garage as a "mobile battery pack" 
and one to drive around.

All problems solved.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.

Re: power to the internet

[Michael Thomas]

On 12/26/19 11:18 AM, John Levine wrote:

In article  
you write:

To reanswer the question posed though, is still the same ; $$$. If network
operators take the position that the electric utility supply should be more
reliable than it is, then they need to start influencing and lobbying for
ways for that to happen. If not, they will have to increase investments
into local generation or storage capacity to bridge those gaps.

You seem to imply that regulation is inherently bad; however the scenario
that you describe (power failures impacting 911 service) is only a concern
to an operator if there is a legislatively define deterrent.

California suffers from an unusual combination of a dry climate that
is getting dryer and political decisions that made sense in the short
run but are now showing their long term consequences, notably land use
that encourages sprawl and construction in ill-suited areas


If we stopped construction in all of the ill-suited areas, we'd stop 
construction all together, and tear down much more. We have it all here: 
earthquakes, floods, fires; often the trifecta.  We could certainly be 
smarter, but the nature of the geography here is both a blessing and a 
curse.




, and a
regulator that keeps short term consumer prices down at the cost of
reliability and long term stability.  None of this should be a
surprise to anyone familiar with the situation.
PG is especially egregious as it has extremely high rates and 
piss-poor maintenance. Where does all of that money go? Execs and 
shareholders. And if some random nyc hedge fund gets its way it's going 
to get even worse. I don't know what the ultimate solution is, but 
whatever it is cannot have those perverse incentives.



Mike

Re: power to the internet

[Michael Thomas]

On 12/26/19 11:18 AM, Stephen Satchell wrote:

On 12/26/19 10:55 AM, Michael Thomas wrote:
Here in California, you're going to need a lot more than 8 hours. We 
had one that lasted 3 days, followed by about 8 hours of power, 
followed by 2 days of no power. If this is the new normal, and I'm 
afraid that it is, that's probably going to require some pretty hefty 
backup. Not to mention expensive.


The one "good" thing that PG did is expose all of these 
vulnerabilities. Every neighborhood probably knows whether their 
carrier is naughty or nice now.


Here in Nevada, specifically at Lake Tahoe, power is less reliable 
because of heavy snow and sliding trucks (the power equivalent to a 
backhoe disconnect).  One of the cell sites is on the top level of a 
casino parking garage.  I found out about this when the casino went 
bankrupt, the parking garage was blocked off, and I joined the 
security guard crew to protect the on-site gaming equipment.  Months 
into the project, the cell company in question begged the bankruptcy 
court for access -- to replace the empty propane cylinders in their 
shack.  That's right, no mains tap at all.  When the casino lost power 
because of bill non-payment, the cell site stayed up.


A network operator will need to look at the total cost, including 
labor, of backing up mains power. versus using local genertion 
exclusively -- or using mains power as the backup!  Factor in any 
upcoming fines for service outage, re 911.  (Try to avoid piped 
natural gas as the fuel for onsite generation.)


Longer term, review your backhauls and interconnects.  Dark fiber 
would be preferred here, because you would be controlling backup power 
at both ends, and not depending on intermediate nodes.



One of the interesting things I found out is that POTS termination out 
in the field can be powered by some of the pairs back to the CO they are 
making redundant. It's enough power that running the DSLAM isn't a 
problem either. I'm not sure that that could translate for anything 
else, but there is probably a lot of copper sitting idle these days.


Mike

Re: power to the internet

[Jason Wilson]

AT land line had service trucks parked at RT’s to power them. I talked
with one of the techs. He was on a 12 hour schedule and spent that time
between 3 sites charging the batteries to keep the copper plant running.
They plugged in to the truck inverter and ran the truck all day. He told me
they ran out of generators. Talk about a waste of manpower.

On Thu, Dec 26, 2019 at 10:56 AM Michael Thomas  wrote:

>
> On 12/26/19 10:41 AM, Ben Cannon wrote:
> > Exactly. And we will build it all.
> >
> > The power stuff is serious people.  We’ve gotten letters from the FCC
> > over it.  There is additional regulation coming down when people can’t
> > call 911!
> >
> > You need at minimum 8 hours (or your CRT response time with a
> > generator trailer, or a standby generator or two) of battery on your
> > telecom equipment. All of it. Everywhere.
> >
> > Comcast is the worst about this, they never replace and often don’t
> > even place batteries in their RTs at all - and they are going to get
> > fined over it mark my words.
> >
> >
> Here in California, you're going to need a lot more than 8 hours. We had
> one that lasted 3 days, followed by about 8 hours of power, followed by
> 2 days of no power. If this is the new normal, and I'm afraid that it
> is, that's probably going to require some pretty hefty backup. Not to
> mention expensive.
>
> The one "good" thing that PG did is expose all of these
> vulnerabilities. Every neighborhood probably knows whether their carrier
> is naughty or nice now.
>
> Mike
>
> --

Jason Wilson
Remotely Located
Providing High Speed Internet to out of the way places.
530-651-1736
530-748-9608 Cell
www.remotelylocated.com

Re: power to the internet

[Michael Thomas]

On 12/26/19 11:00 AM, Ben Cannon wrote:

How much generating capacity can you get out of a typical hybrid?


You’re joking right?  A lot… Enough to run an entire neighborhood…   
The Prius makes 50,000watts alone.


With the right circuitry, there is no need for power plants in the 
United States (save that they’re more efficient than internal 
combustion gas engines in the 76hp range) - existing hybrid car 
stock’s generating capacity exceeds the entire US supply.  And it’s 
entirely untapped.


Nissan just tested it for giggles, and found the Leaf (which has NO 
engine at all) can power a house for an entire week.  The batteries 
alone are a game changer, utterly transforming grids.


I just looked up Telsa's battery packs and they seem to be between 
60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 
days. Admittedly we can turn off the hot tub, water heater, etc to 
stretch it out. And of course, that means that you can't drive it... The 
one thing that would be for everybody's good is using them during peak 
hours. If you work normal hours, then that only gets part of the peak 
load, unfortunately.


But of course this has nothing to do with the network power problem. I 
assume they won't be parking a Tesla next to a CMTS headend.


Mike

Re: power to the internet

[John Levine]

In article  
you write:
>To reanswer the question posed though, is still the same ; $$$. If network
>operators take the position that the electric utility supply should be more
>reliable than it is, then they need to start influencing and lobbying for
>ways for that to happen. If not, they will have to increase investments
>into local generation or storage capacity to bridge those gaps.
>
>You seem to imply that regulation is inherently bad; however the scenario
>that you describe (power failures impacting 911 service) is only a concern
>to an operator if there is a legislatively define deterrent.

California suffers from an unusual combination of a dry climate that
is getting dryer and political decisions that made sense in the short
run but are now showing their long term consequences, notably land use
that encourages sprawl and construction in ill-suited areas, and a
regulator that keeps short term consumer prices down at the cost of
reliability and long term stability.  None of this should be a
surprise to anyone familiar with the situation.

Even well run US utilities are much less reliable than the norm in
Europe or Japan.  Where ISPs in the US are figuring out how to install
batteries and backup generators or private windmills or whatever,
their European peers pay somewhat higher utility bills and don't have
to worry about the other stuff.  You'll pay either way.  European
utilities aren't more reliable by accident; that's how they're
regulated.

Calfornia also offers an interesting natural experiment comparing
privately run utilities PG and SCE and the city owned Los Angeles
DWP.

Re: power to the internet

[Stephen Satchell]

On 12/26/19 10:55 AM, Michael Thomas wrote:
Here in California, you're going to need a lot more than 8 hours. We had 
one that lasted 3 days, followed by about 8 hours of power, followed by 
2 days of no power. If this is the new normal, and I'm afraid that it 
is, that's probably going to require some pretty hefty backup. Not to 
mention expensive.


The one "good" thing that PG did is expose all of these 
vulnerabilities. Every neighborhood probably knows whether their carrier 
is naughty or nice now.


Here in Nevada, specifically at Lake Tahoe, power is less reliable 
because of heavy snow and sliding trucks (the power equivalent to a 
backhoe disconnect).  One of the cell sites is on the top level of a 
casino parking garage.  I found out about this when the casino went 
bankrupt, the parking garage was blocked off, and I joined the security 
guard crew to protect the on-site gaming equipment.  Months into the 
project, the cell company in question begged the bankruptcy court for 
access -- to replace the empty propane cylinders in their shack.  That's 
right, no mains tap at all.  When the casino lost power because of bill 
non-payment, the cell site stayed up.


A network operator will need to look at the total cost, including labor, 
of backing up mains power. versus using local genertion exclusively -- 
or using mains power as the backup!  Factor in any upcoming fines for 
service outage, re 911.  (Try to avoid piped natural gas as the fuel for 
onsite generation.)


Longer term, review your backhauls and interconnects.  Dark fiber would 
be preferred here, because you would be controlling backup power at both 
ends, and not depending on intermediate nodes.

Re: power to the internet

[Ben Cannon]

> How much generating capacity can you get out of a typical hybrid?

You’re joking right?  A lot… Enough to run an entire neighborhood…   The Prius 
makes 50,000watts alone.

With the right circuitry, there is no need for power plants in the United 
States (save that they’re more efficient than internal combustion gas engines 
in the 76hp range) - existing hybrid car stock’s generating capacity exceeds 
the entire US supply.  And it’s entirely untapped. 

Nissan just tested it for giggles, and found the Leaf (which has NO engine at 
all) can power a house for an entire week.  The batteries alone are a game 
changer, utterly transforming grids.

> Self-isolating and re-tieing inverters. Economic household ATS systems. Do 
> those exist?

Yes I have a patent in one type of this this; it exists in numerous variants.

ATS-es for partial loads of up to 6,000watts are like $400.

You sound smart, but did you research any of this or just post?

-Ben.

-Ben Cannon
CEO 6x7 Networks & 6x7 Telecom, LLC 
b...@6by7.net 




> On Dec 26, 2019, at 2:31 AM, Joe Maimon  wrote:
> 
> Unless telecom infrastructure has been diligently changing out the lead acid 
> battery approach at all their remote terminals, powered gpon, hfc and 
> antennae plants will never last more than minutes. If at all.
> 
> A traditional car has between a 100-200amp alternator @12volts
> 
> How much generating capacity can you get out of a typical hybrid?
> 
> Self-isolating and re-tieing inverters. Economic household ATS systems. Do 
> those exist?
> 
> Enough independent distributed capacity and now comes the ability to create 
> grid islands. How might that look?
> 
> Electric grid shortage is likely coming to NYC, courtesy of folk of certain 
> political persuasion and their love of stone age era living. IP 
> decommissioning.
> 
> If you have CO loop copper, keep it.
> 
> Joe
> 
> Don Gould wrote:
>> This is a very short term problem.
>> 
>> The market is going to fill with battery storage sooner rather than later.
>> 
>> Solar is just exploding.
>> 
>> Your car will "house tie".
>> 
>> 6G will solve your data problem.
>> 
>> D
>> 
>> 
>> 
>> -- 
>> Don Gould
>> 5 Cargill Place
>> Richmond
>> Christchurch, New Zealand
>> Mobile/Telegram: + 64 21 114 0699
>> www. bowenvale.co.nz
>> 
>> 
>> 
>>  Original message 
>> From: Michael Thomas 
>> Date: 26/12/19 2:33 PM (GMT+12:00)
>> To: nanog@nanog.org
>> Subject: power to the internet
>> 
>> 
>> https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
>> 
>> 
>> This article details some of the issues with California's "new reality"
>> of planned blackouts. One of the big things that came to light with
>> these blackouts is that our network infrastructure's resilience is
>> pretty lacking. While I was (surprisingly to me) ok with my DSL
>> connection out in the boonies, lots and lots of people with cable
>> weren't so lucky. And I'm not sure how bad the situation is with
>> cellular infrastructure, but I assume it's not much better than cable.
>> And I wouldn't doubt that other DSL deployments go dark when power is
>> down. I have no clue with fiber.
>> 
>> So I guess what I'm wondering is what can we do about this? What should
>> we do about this? These days IP access is not just convenience, it's the
>> way we go about our lives, just like electricity itself. At base, it
>> seems to me that network operators should be required to keep the lights
>> on in blackouts just like POTS operators do now. If I have power to
>> light my modem or charge in my phone, I should be able to get onto the
>> net. That seems like table stakes.
>> 
>> One of the things we learned also is that the blackouts seem to last
>> between 2-3 days apiece. I happen to have a generator since I'm out in
>> the boonies and our power gets cut regularly because of snow, but not
>> everyone has that luxury. I kind of want to think that my router+modem
>> use about 20 watts, so powering it up would take about 1.5kwh for 3
>> days. a quick google look shows that I'd probably need to shell out $500
>> or so for a battery of that capacity, and that's doesn't include your
>> phones, laptops, tv's, etc power needs. What does that mean? That is a
>> major expense for a lot of people.
>> 
>> On the bright side, I hear that power generator companies stocks have
>> gone through the roof.
>> 
>> On the dark side, this is probably coming to a lot more states and
>> countries due to climate change. Australia. Sigh.
>> 
>> Mike
>> 
> 

Re: power to the internet

[Michael Thomas]

On 12/26/19 10:41 AM, Ben Cannon wrote:

Exactly. And we will build it all.

The power stuff is serious people.  We’ve gotten letters from the FCC 
over it.  There is additional regulation coming down when people can’t 
call 911!


You need at minimum 8 hours (or your CRT response time with a 
generator trailer, or a standby generator or two) of battery on your 
telecom equipment. All of it. Everywhere.


Comcast is the worst about this, they never replace and often don’t 
even place batteries in their RTs at all - and they are going to get 
fined over it mark my words.



Here in California, you're going to need a lot more than 8 hours. We had 
one that lasted 3 days, followed by about 8 hours of power, followed by 
2 days of no power. If this is the new normal, and I'm afraid that it 
is, that's probably going to require some pretty hefty backup. Not to 
mention expensive.


The one "good" thing that PG did is expose all of these 
vulnerabilities. Every neighborhood probably knows whether their carrier 
is naughty or nice now.


Mike

Re: power to the internet

[Michael Thomas]

On 12/26/19 10:26 AM, Tom Beecher wrote:
If that was a reference to my comments, it was certainly not my 
intention. I was striving to avoid it being seen as that, but 
apparently fell short.


Not directed at you at all.



To reanswer the question posed though, is still the same ; $$$. If 
network operators take the position that the electric utility supply 
should be more reliable than it is, then they need to start 
influencing and lobbying for ways for that to happen. If not, they 
will have to increase investments into local generation or storage 
capacity to bridge those gaps.


You seem to imply that regulation is inherently bad; however the 
scenario that you describe (power failures impacting 911 service) is 
only a concern to an operator if there is a legislatively define 
deterrent.



Not at all. I'm saying that this problem will be solved one way or the 
other. Frankly it's surprising that anybody offering telephony service 
has gotten away with not fulfilling the battery backup mandate. I guess 
there must have been some wiggle room that the carriers took advantage 
of. And if so, legislation to fix that will be immanent.


Mike

Re: power to the internet

[Ben Cannon]

Exactly. And we will build it all.

The power stuff is serious people.  We’ve gotten letters from the FCC over it.  
There is additional regulation coming down when people can’t call 911!  

You need at minimum 8 hours (or your CRT response time with a generator 
trailer, or a standby generator or two) of battery on your telecom equipment. 
All of it. Everywhere.  

Comcast is the worst about this, they never replace and often don’t even place 
batteries in their RTs at all - and they are going to get fined over it mark my 
words.

-Ben Cannon
CEO 6x7 Networks & 6x7 Telecom, LLC 
b...@6by7.net 




> On Dec 25, 2019, at 8:41 PM, Don Gould  wrote:
> 
> This is a very short term problem. 
> 
> The market is going to fill with battery storage sooner rather than later. 
> 
> Solar is just exploding. 
> 
> Your car will "house tie".
> 
> 6G will solve your data problem. 
> 
> D
> 
> 
> 
> -- 
> Don Gould
> 5 Cargill Place
> Richmond
> Christchurch, New Zealand
> Mobile/Telegram: + 64 21 114 0699
> www. bowenvale.co.nz
> 
> 
> 
>  Original message 
> From: Michael Thomas 
> Date: 26/12/19 2:33 PM (GMT+12:00)
> To: nanog@nanog.org
> Subject: power to the internet
> 
> 
> https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
> 
> 
> This article details some of the issues with California's "new reality" 
> of planned blackouts. One of the big things that came to light with 
> these blackouts is that our network infrastructure's resilience is 
> pretty lacking. While I was (surprisingly to me) ok with my DSL 
> connection out in the boonies, lots and lots of people with cable 
> weren't so lucky. And I'm not sure how bad the situation is with 
> cellular infrastructure, but I assume it's not much better than cable. 
> And I wouldn't doubt that other DSL deployments go dark when power is 
> down. I have no clue with fiber.
> 
> So I guess what I'm wondering is what can we do about this? What should 
> we do about this? These days IP access is not just convenience, it's the 
> way we go about our lives, just like electricity itself. At base, it 
> seems to me that network operators should be required to keep the lights 
> on in blackouts just like POTS operators do now. If I have power to 
> light my modem or charge in my phone, I should be able to get onto the 
> net. That seems like table stakes.
> 
> One of the things we learned also is that the blackouts seem to last 
> between 2-3 days apiece. I happen to have a generator since I'm out in 
> the boonies and our power gets cut regularly because of snow, but not 
> everyone has that luxury. I kind of want to think that my router+modem 
> use about 20 watts, so powering it up would take about 1.5kwh for 3 
> days. a quick google look shows that I'd probably need to shell out $500 
> or so for a battery of that capacity, and that's doesn't include your 
> phones, laptops, tv's, etc power needs. What does that mean? That is a 
> major expense for a lot of people.
> 
> On the bright side, I hear that power generator companies stocks have 
> gone through the roof.
> 
> On the dark side, this is probably coming to a lot more states and 
> countries due to climate change. Australia. Sigh.
> 
> Mike
> 

Re: power to the internet

[Michael Thomas]

On 12/26/19 10:19 AM, Jason Wilson wrote:
As a small WISP operator in Northern California and well into the 
urban interface we fell victim to the PSPS this year. Thousands was 
spent on upgrading battery plants that would normally hold during a 
short outage and generator purchases, whether it be small inverter 
style generators for small sites to permanent standby site generators 
for those sites that are larger or a PITA to get to. We still have 
more work to do and hope to be better prepared for next summers rounds 
of shut offs.  I am currently developing a portable trailer mounted 
solar/battery plant to replace the portable generators just for fuel 
cost savings since I spent just about $500/week in generator fuel 
alone for the largest outage.


Yeah, that's the biggest change: the outages are 2-3 days each. Maybe 
they'll get better but their stated goal is to inspect the entire 
distribution system before turning power back on, so that is going to 
take some time no matter what.


And the fuel cost is definitely a consideration. Unfortunately October 
and November is pretty lousy for solar. Wind might be better since that 
is the reason they're doing the PSPS in the first place :)


Mike

Re: power to the internet

[Tom Beecher]

If that was a reference to my comments, it was certainly not my intention.
I was striving to avoid it being seen as that, but apparently fell short.

To reanswer the question posed though, is still the same ; $$$. If network
operators take the position that the electric utility supply should be more
reliable than it is, then they need to start influencing and lobbying for
ways for that to happen. If not, they will have to increase investments
into local generation or storage capacity to bridge those gaps.

You seem to imply that regulation is inherently bad; however the scenario
that you describe (power failures impacting 911 service) is only a concern
to an operator if there is a legislatively define deterrent.

On Thu, Dec 26, 2019 at 13:00 Michael Thomas  wrote:

>
> On 12/26/19 7:51 AM, Mike Bolitho wrote:
> > I'm pretty sure political bickering is well beyond the scope of the
> > mailing list. Is anyone moderating this?
>
> It certainly wasn't my intent or desire to have this turn political, and
> shame on the person who did. This is a serious networking related issue
> for California *right* *now*. It may become a serious networking related
> issue for a lot of other places too -- California is hardly unique in
> its wildland - urban interface issues, and lots of places burn just like
> California. And definitely lots of places have a 100+ years of fire
> suppression which is a policy thing, not a political thing.
>
> The question is what are network operators going to do? If the answer is
> "nothing", don't be surprised to get legislation shoved down your
> throats. Don't expect the bay area of all places to passively put up
> with all of this. If your network fails because of power going out and I
> can't call 911, you've got a big, big problem.
>
> Mike
>
>
>

Re: power to the internet

[Jason Wilson]

As a small WISP operator in Northern California and well into the urban
interface we fell victim to the PSPS this year. Thousands was spent on
upgrading battery plants that would normally hold during a short outage and
generator purchases, whether it be small inverter style generators for
small sites to permanent standby site generators for those sites that are
larger or a PITA to get to. We still have more work to do and hope to be
better prepared for next summers rounds of shut offs.  I am currently
developing a portable trailer mounted solar/battery plant to replace the
portable generators just for fuel cost savings since I spent just about
$500/week in generator fuel alone for the largest outage.

On Thu, Dec 26, 2019 at 9:59 AM Michael Thomas  wrote:

>
> On 12/26/19 7:51 AM, Mike Bolitho wrote:
> > I'm pretty sure political bickering is well beyond the scope of the
> > mailing list. Is anyone moderating this?
>
> It certainly wasn't my intent or desire to have this turn political, and
> shame on the person who did. This is a serious networking related issue
> for California *right* *now*. It may become a serious networking related
> issue for a lot of other places too -- California is hardly unique in
> its wildland - urban interface issues, and lots of places burn just like
> California. And definitely lots of places have a 100+ years of fire
> suppression which is a policy thing, not a political thing.
>
> The question is what are network operators going to do? If the answer is
> "nothing", don't be surprised to get legislation shoved down your
> throats. Don't expect the bay area of all places to passively put up
> with all of this. If your network fails because of power going out and I
> can't call 911, you've got a big, big problem.
>
> Mike
>
>
> --

Jason Wilson
Remotely Located
Providing High Speed Internet to out of the way places.
530-651-1736
530-748-9608 Cell
www.remotelylocated.com

Re: power to the internet

[Michael Thomas]

On 12/26/19 7:51 AM, Mike Bolitho wrote:
I'm pretty sure political bickering is well beyond the scope of the 
mailing list. Is anyone moderating this?


It certainly wasn't my intent or desire to have this turn political, and 
shame on the person who did. This is a serious networking related issue 
for California *right* *now*. It may become a serious networking related 
issue for a lot of other places too -- California is hardly unique in 
its wildland - urban interface issues, and lots of places burn just like 
California. And definitely lots of places have a 100+ years of fire 
suppression which is a policy thing, not a political thing.


The question is what are network operators going to do? If the answer is 
"nothing", don't be surprised to get legislation shoved down your 
throats. Don't expect the bay area of all places to passively put up 
with all of this. If your network fails because of power going out and I 
can't call 911, you've got a big, big problem.


Mike

Re: power to the internet

[Mike Bolitho]

I'm pretty sure political bickering is well beyond the scope of the mailing
list. Is anyone moderating this?

- Mike Bolitho

On Thu, Dec 26, 2019, 7:20 AM Tom Beecher  wrote:

> Same story again different colors. PG making a mint while people get
>> screwed
>>
>
> I'm not quite sure that's an accurate statement.
>
> In 2000-2001, PG got screwed by Enron's market manipulation. ( Good job
> those who pushed so hard for deregulation of public utility services! )
>
> PG is currently in bankruptcy proceedings, largely as a result of
> liabilities from wildfires in 2017 and 2018. Under California's
> application of inverse condemnation, a power utility is responsible for any
> damage caused by a wildfire if it was determined that their equipment was
> part of the cause. This applies even if the utility was in 100% compliance
> with all laws and regulations.
>
> So you have a terrible combination where housing prices in the state are
> driving more and more people to build in wildfire prone areas, climate
> change is increasing the frequency of weather conditions favorable to
> wildfire ignition, and the utility company that is being held financially
> liable for damages while at the same time not being allowed by the PUC to
> raise capital for infrastructure changes to reduce the chances of
> electrical equipment starting such things.
>
> The answer is easy. Money. If people want a power grid that is safe and
> reliable, then the utility should be given the funds to do it via rates and
> appropriate tax revenues. They should not be expected to turn profits like
> private enterprise. The power grid is for the benefit of all, not just the
> financial benefit of those who have equity stakes.
>
> This situation is the logical extension of 40+ years of America's only
> real product ; financial engineering.
>
>
> On Wed, Dec 25, 2019 at 9:18 PM Michael Loftis  wrote:
>
>>
>>
>> On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin 
>> wrote:
>>
>>> On Wed, 25 Dec 2019 at 19:32, Michael Thomas  wrote:
>>>
 On the dark side, this is probably coming to a lot more states and
 countries due to climate change. Australia. Sigh.

>>>
>>> Do you have a source for this?  It would seem that these power issues
>>> are rather unique to California not because of some "climate change"
>>> bogeyman, but rather because of a failed public policy at the state level.
>>>
>>> It would also seem that these issues of rolling blackouts aren't even
>>> new to California, either, as, apparently, it's already been the norm
>>> during 2000/2001:
>>>
>>
>>
>> Having lived through the blackouts that was entirely different. 90% Enron
>> manipulating the markets. There was plenty of capacity both in transmission
>> and generation, but Enron manipulated prices and apparent supply to make
>> money and screwed the whole state over. There was just about 2x the
>> generating capacity, no real shortage.
>>
>> This time it’s PG all alone, but still fallout from back then. Too much
>> liability and they’ve not maintained the infrastructure and so they decided
>> that to reduce the liability costs it’s cheaper to blackout. Same story
>> again different colors. PG making a mint while people get screwed (PG
>> was mostly at the getting screwed end in 2000-2001)
>>
>>>
>>> * https://en.wikipedia.org/wiki/California_electricity_crisis
>>>
>>> C.
>>>
>> --
>>
>> "Genius might be described as a supreme capacity for getting its
>> possessors
>> into trouble of all kinds."
>> -- Samuel Butler
>>
>

Re: power to the internet

[Tom Beecher]

>
> Same story again different colors. PG making a mint while people get
> screwed
>

I'm not quite sure that's an accurate statement.

In 2000-2001, PG got screwed by Enron's market manipulation. ( Good job
those who pushed so hard for deregulation of public utility services! )

PG is currently in bankruptcy proceedings, largely as a result of
liabilities from wildfires in 2017 and 2018. Under California's
application of inverse condemnation, a power utility is responsible for any
damage caused by a wildfire if it was determined that their equipment was
part of the cause. This applies even if the utility was in 100% compliance
with all laws and regulations.

So you have a terrible combination where housing prices in the state are
driving more and more people to build in wildfire prone areas, climate
change is increasing the frequency of weather conditions favorable to
wildfire ignition, and the utility company that is being held financially
liable for damages while at the same time not being allowed by the PUC to
raise capital for infrastructure changes to reduce the chances of
electrical equipment starting such things.

The answer is easy. Money. If people want a power grid that is safe and
reliable, then the utility should be given the funds to do it via rates and
appropriate tax revenues. They should not be expected to turn profits like
private enterprise. The power grid is for the benefit of all, not just the
financial benefit of those who have equity stakes.

This situation is the logical extension of 40+ years of America's only real
product ; financial engineering.


On Wed, Dec 25, 2019 at 9:18 PM Michael Loftis  wrote:

>
>
> On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin 
> wrote:
>
>> On Wed, 25 Dec 2019 at 19:32, Michael Thomas  wrote:
>>
>>> On the dark side, this is probably coming to a lot more states and
>>> countries due to climate change. Australia. Sigh.
>>>
>>
>> Do you have a source for this?  It would seem that these power issues are
>> rather unique to California not because of some "climate change" bogeyman,
>> but rather because of a failed public policy at the state level.
>>
>> It would also seem that these issues of rolling blackouts aren't even new
>> to California, either, as, apparently, it's already been the norm during
>> 2000/2001:
>>
>
>
> Having lived through the blackouts that was entirely different. 90% Enron
> manipulating the markets. There was plenty of capacity both in transmission
> and generation, but Enron manipulated prices and apparent supply to make
> money and screwed the whole state over. There was just about 2x the
> generating capacity, no real shortage.
>
> This time it’s PG all alone, but still fallout from back then. Too much
> liability and they’ve not maintained the infrastructure and so they decided
> that to reduce the liability costs it’s cheaper to blackout. Same story
> again different colors. PG making a mint while people get screwed (PG
> was mostly at the getting screwed end in 2000-2001)
>
>>
>> * https://en.wikipedia.org/wiki/California_electricity_crisis
>>
>> C.
>>
> --
>
> "Genius might be described as a supreme capacity for getting its possessors
> into trouble of all kinds."
> -- Samuel Butler
>

Re: power to the internet

[Joe Maimon]

Unless telecom infrastructure has been diligently changing out the lead 
acid battery approach at all their remote terminals, powered gpon, hfc 
and antennae plants will never last more than minutes. If at all.


A traditional car has between a 100-200amp alternator @12volts

How much generating capacity can you get out of a typical hybrid?

Self-isolating and re-tieing inverters. Economic household ATS systems. 
Do those exist?


Enough independent distributed capacity and now comes the ability to 
create grid islands. How might that look?


Electric grid shortage is likely coming to NYC, courtesy of folk of 
certain political persuasion and their love of stone age era living. IP 
decommissioning.


If you have CO loop copper, keep it.

Joe

Don Gould wrote:

This is a very short term problem.

The market is going to fill with battery storage sooner rather than 
later.


Solar is just exploding.

Your car will "house tie".

6G will solve your data problem.

D



--
Don Gould
5 Cargill Place
Richmond
Christchurch, New Zealand
Mobile/Telegram: + 64 21 114 0699
www. bowenvale.co.nz



 Original message 
From: Michael Thomas 
Date: 26/12/19 2:33 PM (GMT+12:00)
To: nanog@nanog.org
Subject: power to the internet


https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678


This article details some of the issues with California's "new reality"
of planned blackouts. One of the big things that came to light with
these blackouts is that our network infrastructure's resilience is
pretty lacking. While I was (surprisingly to me) ok with my DSL
connection out in the boonies, lots and lots of people with cable
weren't so lucky. And I'm not sure how bad the situation is with
cellular infrastructure, but I assume it's not much better than cable.
And I wouldn't doubt that other DSL deployments go dark when power is
down. I have no clue with fiber.

So I guess what I'm wondering is what can we do about this? What should
we do about this? These days IP access is not just convenience, it's the
way we go about our lives, just like electricity itself. At base, it
seems to me that network operators should be required to keep the lights
on in blackouts just like POTS operators do now. If I have power to
light my modem or charge in my phone, I should be able to get onto the
net. That seems like table stakes.

One of the things we learned also is that the blackouts seem to last
between 2-3 days apiece. I happen to have a generator since I'm out in
the boonies and our power gets cut regularly because of snow, but not
everyone has that luxury. I kind of want to think that my router+modem
use about 20 watts, so powering it up would take about 1.5kwh for 3
days. a quick google look shows that I'd probably need to shell out $500
or so for a battery of that capacity, and that's doesn't include your
phones, laptops, tv's, etc power needs. What does that mean? That is a
major expense for a lot of people.

On the bright side, I hear that power generator companies stocks have
gone through the roof.

On the dark side, this is probably coming to a lot more states and
countries due to climate change. Australia. Sigh.

Mike