Re: power to the internet

[Mark Tinka]

On 4/Jan/20 12:54, Florian Weimer wrote:

>
> I guess most countries struggle to maintain basic infrastructure.

Then someone will ask, "What is basic, and is the Internet it?"

Mark.

Re: power to the internet

[Florian Weimer]

* John Levine:

> In article <87y2up1vc4@mid.deneb.enyo.de> you write:
>>I found the connection rather puzzling (that is, how switching off
>>power distribution prevents wildfires or at least reduces their risk).
>>I found some explanations here (downed lines, vegetation contact,
>>conductor slap, repetitive faults, apparatus failures):
>>
>>
>
> Oh, you're in Europe.  You wouldn't believe how cruddy US power
> distribution systems are.  California is particularly bad becuase the
> populist state regulator has keep retail prices low at the cost of
> reliability, safety, and everything else.
>
> Also keep in mind that California has conditions seen nowhere in
> Europe: bone dry forests with 40C temperaturees and 100Kph winds,
> and a power company too underfunded to keep up with tree trimming.

I think Greece also suffered a major wildfire in the 2018 that was
initiated by a faulty power line.  In Germany, we have some vegetation
issues on train lines partially due to insufficient maintenance, but
they fortunately don't trigger wildfires, only train outages.

I guess most countries struggle to maintain basic infrastructure.

Re: power to the internet

[t...@pelican.org]

On Thursday, 2 January, 2020 21:34, "Sabri Berisha"  
said:

> - On Jan 2, 2020, at 1:24 PM, John Levine jo...@iecc.com wrote:
> 
>> PS: You also wouldn't believe how cheap the power is.  California's
>> prices are high compared to most of the US, but it's still only about
>> €0.15 per KWh.
> 
> I don't know where you live, but I pay around 38 cents/KWh. Depending
> on your rate, that can go up to 53 cents/KWh during peak times.

Data point for comparison: I'm in the UK, paying ~0.18GBP/KWh (0.21EUR or 
0.24USD) - and that's on a tariff where I'm paying extra to ensure a certain 
amount of the power is sourced from renewables, I could get it cheaper.  We do 
have a standing charge though, irrespective of usage, 0.22GBP/day in my case - 
I don't know if you have that on US electricity bills typically.  It's a rare 
event and a cause of great annoyance if the power goes out.  (Most recent 
events - and I'm talking 2 or 3 in the last decade or more - have been down to 
idiots trying to steal copper from live substations, which doesn't end well for 
anyone).

The last time we had planned blackouts on any kind of scale was the 70s when 
due to the oil crisis and striking miners we couldn't source enough fossil-fuel 
to keep generating.

Obviously completely different climate, geography, and (as I understand it) 
industry set-up - we have the national distribution infrastructure, last mile 
infrastructure, generation, and consumer-facing commercial / billing all as 
distinct entities, which brings its own set of benefits and challenges!

Re: power to the internet

[Tom Beecher]

I'm familiar with the Sir Adam Beck plant, I grew up in and live in Niagara
County.

Not everything produced by the NYPA goes to munis. There is a lot sold
direct to businesses; last I checked roughly 5% of the generation from the
Niagara Power Project is allocated for businesses in WNY in a 30 mile ring.
( Although a sizeable chunk of that goes back to the wholesale markets
because there aren't enough qualified companies to take it. )

I can guarantee that some of that power ended up with you. Every commercial
supplier in NY buys from the wholesale market at some point, and a lot of
NYPA power ends up there.

On Thu, Jan 2, 2020 at 6:04 PM John Levine  wrote:

> In article  q...@mail.gmail.com> you write:
> >-=-=-=-=-=-
> >It helps that we have a 2.6GW pumped storage generation facility near
> >Niagara Falls. :)
>
> It does, but all that power goes to the munis, not the commercial
> company that supplies me.  We do import a lot of hydro power from
> Quebec.  There's another power plant the same size on the other side
> of the river that provides power for Toronto.
>
>
> >On Thu, Jan 2, 2020 at 5:05 PM Scott Weeks 
> wrote:
> >
> >>
> >> -
> >> > I don't know where you live, but I pay around 38 cents/KWh. Depending
> >> > on your rate, that can go up to 53 cents/KWh during peak times.
> >>
> >> I live in upstate New York where I pay about 8c/kwh and a fixed $15/mo
> >> connection charge.  We have day/night rates available but they're not
> very
> >> different for retail customers.  I get a slight discount due to credits
> >> from remote net metering at a nearby solar farm.
>

Re: power to the internet

[John Levine]

In article 
 you 
write:
>-=-=-=-=-=-
>It helps that we have a 2.6GW pumped storage generation facility near
>Niagara Falls. :)

It does, but all that power goes to the munis, not the commercial
company that supplies me.  We do import a lot of hydro power from
Quebec.  There's another power plant the same size on the other side
of the river that provides power for Toronto.


>On Thu, Jan 2, 2020 at 5:05 PM Scott Weeks  wrote:
>
>>
>> -
>> > I don't know where you live, but I pay around 38 cents/KWh. Depending
>> > on your rate, that can go up to 53 cents/KWh during peak times.
>>
>> I live in upstate New York where I pay about 8c/kwh and a fixed $15/mo
>> connection charge.  We have day/night rates available but they're not very
>> different for retail customers.  I get a slight discount due to credits
>> from remote net metering at a nearby solar farm.

Re: power to the internet

[Tom Beecher]

It helps that we have a 2.6GW pumped storage generation facility near
Niagara Falls. :)

On Thu, Jan 2, 2020 at 5:05 PM Scott Weeks  wrote:

>
> -
> > I don't know where you live, but I pay around 38 cents/KWh. Depending
> > on your rate, that can go up to 53 cents/KWh during peak times.
>
> I live in upstate New York where I pay about 8c/kwh and a fixed $15/mo
> connection charge.  We have day/night rates available but they're not very
> different for retail customers.  I get a slight discount due to credits
> from remote net metering at a nearby solar farm.
> --
>
>
> Damn, I'm jealous:
>
>
> https://www.hawaiianelectric.com/billing-and-payment/rates-and-regulations/average-price-of-electricity
>
> These're averaged...
>
> Rate Schedule  2018 Average Cents/kWh
>
> Oahu
> "R" Residential 31.18
> "G" Small Power Use Business32.58
> "J" Medium Power Use Business   27.44
> "P" Large Power Use Business25.17
> "DS" Large Power Use Business, Directly Served  24.04
>
> maui/molokai/lanai
> "R" Residential 34.21   40.16   40.14
> "G" Small Power Use Business38.40   47.17   45.09
> "J" Medium Power Use Business   33.53   38.98   42.73
> "P" Large Power Use Business30.80   33.05   38.78
>
> scott
>

Re: power to the internet

[Scott Weeks]

-
> I don't know where you live, but I pay around 38 cents/KWh. Depending
> on your rate, that can go up to 53 cents/KWh during peak times.

I live in upstate New York where I pay about 8c/kwh and a fixed $15/mo 
connection charge.  We have day/night rates available but they're not very 
different for retail customers.  I get a slight discount due to credits
from remote net metering at a nearby solar farm.
--


Damn, I'm jealous:

https://www.hawaiianelectric.com/billing-and-payment/rates-and-regulations/average-price-of-electricity

These're averaged...

Rate Schedule  2018 Average Cents/kWh

Oahu
"R" Residential 31.18
"G" Small Power Use Business32.58
"J" Medium Power Use Business   27.44
"P" Large Power Use Business25.17
"DS" Large Power Use Business, Directly Served  24.04

maui/molokai/lanai
"R" Residential 34.21   40.16   40.14
"G" Small Power Use Business38.40   47.17   45.09
"J" Medium Power Use Business   33.53   38.98   42.73
"P" Large Power Use Business30.80   33.05   38.78

scott

Re: power to the internet

[John R. Levine]

PS: You also wouldn't believe how cheap the power is.  California's
prices are high compared to most of the US, but it's still only about
€0.15 per KWh.


I don't know where you live, but I pay around 38 cents/KWh. Depending
on your rate, that can go up to 53 cents/KWh during peak times.


16x is supposed to be the average.

I live in upstate New York where I pay about 8c/kwh and a fixed $15/mo 
connection charge.  We have day/night rates available but they're not very 
different for retail customers.  I get a slight discount due to credits

from remote net metering at a nearby solar farm.

Regards,
John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

Re: power to the internet

[Michael Thomas]

On 1/2/20 1:34 PM, Sabri Berisha wrote:

- On Jan 2, 2020, at 1:24 PM, John Levine jo...@iecc.com wrote:


PS: You also wouldn't believe how cheap the power is.  California's
prices are high compared to most of the US, but it's still only about
€0.15 per KWh.

I don't know where you live, but I pay around 38 cents/KWh. Depending
on your rate, that can go up to 53 cents/KWh during peak times.

That's in Morgan Hill, CA, on PG power.

See here: https://www.pge.com/tariffs/Res_Inclu_TOU_Current.xlsx
Or the entire schedule: https://www.pge.com/tariffs/electric.shtml



Yeah, I looked it up too because that sounded just flat out wrong. And 
while the CPUC certainly has its share of the blame, PG is by far the 
most to blame for this.


Mike

Re: power to the internet

[Sabri Berisha]

- On Jan 2, 2020, at 1:24 PM, John Levine jo...@iecc.com wrote:

> PS: You also wouldn't believe how cheap the power is.  California's
> prices are high compared to most of the US, but it's still only about
> €0.15 per KWh.

I don't know where you live, but I pay around 38 cents/KWh. Depending
on your rate, that can go up to 53 cents/KWh during peak times.

That's in Morgan Hill, CA, on PG power.

See here: https://www.pge.com/tariffs/Res_Inclu_TOU_Current.xlsx
Or the entire schedule: https://www.pge.com/tariffs/electric.shtml

Thanks,

Sabri

Re: power to the internet

[John Levine]

In article <87y2up1vc4@mid.deneb.enyo.de> you write:
>I found the connection rather puzzling (that is, how switching off
>power distribution prevents wildfires or at least reduces their risk).
>I found some explanations here (downed lines, vegetation contact,
>conductor slap, repetitive faults, apparatus failures):
>
>

Oh, you're in Europe.  You wouldn't believe how cruddy US power
distribution systems are.  California is particularly bad becuase the
populist state regulator has keep retail prices low at the cost of
reliability, safety, and everything else.

Also keep in mind that California has conditions seen nowhere in
Europe: bone dry forests with 40C temperaturees and 100Kph winds,
and a power company too underfunded to keep up with tree trimming.

R's,
John

PS: You also wouldn't believe how cheap the power is.  California's
prices are high compared to most of the US, but it's still only about
€0.15 per KWh.

Re: power to the internet

[Florian Weimer]

* Jason Wilson:

> This is all in conjunction with the CPUC. I believe it is also a part of a
> court order. I’ll need to find that later
>
> https://www.cpuc.ca.gov/deenergization/

I found the connection rather puzzling (that is, how switching off
power distribution prevents wildfires or at least reduces their risk).
I found some explanations here (downed lines, vegetation contact,
conductor slap, repetitive faults, apparatus failures):

Re: power to the internet

[John Lightfoot]

That's exactly what Powerwalls are.

In Vermont, Green Mountain Power had a deal where they bought 2000 Powerwalls 
and gave them to customers throughout the state.  Customers could get up to 
two, paying only $1500 each for the installation and agreeing to let GMP manage 
them.  GMP now has ~2.7 gWh of stored capacity, distributed throughout the 
state to minimize transmission costs.  In times of high electricity spot market 
prices or outages, GMP draws down the Powerwalls, then refills them at night 
when prices are lower.  

The Powerwalls also act as a UPS for the whole house.  When bad weather is 
predicted in an area, GMP makes sure your Powerwall is full.  My town had a 4+ 
hour outage a few weeks back and I had power the whole time, the microwave 
clock didn't even reset.  I only have one Powerwall but could easily last 2+ 
days with it, and it's silent, unlike a generator.

--John Lightfoot

-Original Message-
From: NANOG  on behalf of Howard Leadmon 

Date: Monday, December 30, 2019 at 3:09 PM
To: "nanog@nanog.org" 
Subject: Re: power to the internet

   Isn't that what the Tesla Power Wall's are?   I thought that was the 
fill measure for when the solar panels aren't generating.   I have never 
gotten anything, but know when you look on their site for Solar, they 
try and pitch the batter power walls to run your house for days if needed..


---
Howard Leadmon
PBW Communications, LLC
http://www.pbwcomm.com

On 12/26/2019 2:08 PM, Keith Medcalf wrote:
>> I just looked up Telsa's battery packs and they seem to be between
>> 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3
>> days. Admittedly we can turn off the hot tub, water heater, etc to
>> stretch it out. And of course, that means that you can't drive it... The
>> one thing that would be for everybody's good is using them during peak
>> hours. If you work normal hours, then that only gets part of the peak
>> load, unfortunately.
> Just buy three of them.  Two to leave in the garage as a "mobile battery 
> pack" and one to drive around.
>
> All problems solved.
>

Re: power to the internet

[Howard Leadmon]

  Isn't that what the Tesla Power Wall's are?   I thought that was the 
fill measure for when the solar panels aren't generating.   I have never 
gotten anything, but know when you look on their site for Solar, they 
try and pitch the batter power walls to run your house for days if needed..



---
Howard Leadmon
PBW Communications, LLC
http://www.pbwcomm.com

On 12/26/2019 2:08 PM, Keith Medcalf wrote:

I just looked up Telsa's battery packs and they seem to be between
60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3
days. Admittedly we can turn off the hot tub, water heater, etc to
stretch it out. And of course, that means that you can't drive it... The
one thing that would be for everybody's good is using them during peak
hours. If you work normal hours, then that only gets part of the peak
load, unfortunately.

Just buy three of them.  Two to leave in the garage as a "mobile battery pack" 
and one to drive around.

All problems solved.

Re: power to the internet

[Joe Maimon]

They are quite lacking in longevity and density.

Which is fine if you have enough space and adequate maintenance to 
change them out and they are only a bridge for local generation.


Remote installations do not typically have these characteristics.

Pretty much every basement mux or other RT telco installation I have 
ever seen had the original battery setup still in place and lasted 
approximately 3 seconds when the lights went off. I try to put them into 
my batteries whenever possibly.


Further, many fios/cable customer NID devices do come with batteries, 
but they are usually lead acid, often the telco charges to replace the 
battery and they are typically only there to power the POTS lines on the 
NID.


Joe


Baldur Norddahl wrote:
What is wrong with lead acid battery backup? Seems to be exceedingly 
stable from my experience. We have all our equipment on -48V DC and 
have never had a power interruption at any site.


The requirements here are 48 hours of backup by law. Telecom is 
declared to be part of emergency and defense, so they put in a 
requirement for resilience.


Regards

Baldur


tor. 26. dec. 2019 11.33 skrev Joe Maimon >:


Unless telecom infrastructure has been diligently changing out the
lead
acid battery approach at all their remote terminals, powered gpon,
hfc
and antennae plants will never last more than minutes. If at all.

A traditional car has between a 100-200amp alternator @12volts

How much generating capacity can you get out of a typical hybrid?

Self-isolating and re-tieing inverters. Economic household ATS
systems.
Do those exist?

Enough independent distributed capacity and now comes the ability to
create grid islands. How might that look?

Electric grid shortage is likely coming to NYC, courtesy of folk of
certain political persuasion and their love of stone age era
living. IP
decommissioning.

If you have CO loop copper, keep it.

Joe

Don Gould wrote:
> This is a very short term problem.
>
> The market is going to fill with battery storage sooner rather than
> later.
>
> Solar is just exploding.
>
> Your car will "house tie".
>
> 6G will solve your data problem.
>
> D
>
>
>
> --
> Don Gould
> 5 Cargill Place
> Richmond
> Christchurch, New Zealand
> Mobile/Telegram: + 64 21 114 0699
> www. bowenvale.co.nz

>
>
>
>  Original message 
> From: Michael Thomas mailto:m...@mtcc.com>>
> Date: 26/12/19 2:33 PM (GMT+12:00)
> To: nanog@nanog.org 
> Subject: power to the internet
>
>
>
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
>
>
> This article details some of the issues with California's "new
reality"
> of planned blackouts. One of the big things that came to light with
> these blackouts is that our network infrastructure's resilience is
> pretty lacking. While I was (surprisingly to me) ok with my DSL
> connection out in the boonies, lots and lots of people with cable
> weren't so lucky. And I'm not sure how bad the situation is with
> cellular infrastructure, but I assume it's not much better than
cable.
> And I wouldn't doubt that other DSL deployments go dark when
power is
> down. I have no clue with fiber.
>
> So I guess what I'm wondering is what can we do about this? What
should
> we do about this? These days IP access is not just convenience,
it's the
> way we go about our lives, just like electricity itself. At base, it
> seems to me that network operators should be required to keep
the lights
> on in blackouts just like POTS operators do now. If I have power to
> light my modem or charge in my phone, I should be able to get
onto the
> net. That seems like table stakes.
>
> One of the things we learned also is that the blackouts seem to last
> between 2-3 days apiece. I happen to have a generator since I'm
out in
> the boonies and our power gets cut regularly because of snow,
but not
> everyone has that luxury. I kind of want to think that my
router+modem
> use about 20 watts, so powering it up would take about 1.5kwh for 3
> days. a quick google look shows that I'd probably need to shell
out $500
> or so for a battery of that capacity, and that's doesn't include
your
> phones, laptops, tv's, etc power needs. What does that mean?
That is a
> major expense for a lot of people.
>
> On the bright side, I hear that power generator companies stocks
have
> gone through the roof.
>
> On the dark side, this is probably coming to a lot more states and
> countries due to climate change. Australia. Sigh.
>
> Mike
   

Re: power to the internet

[Ben Cannon]

-48VDC is extremely reliable, we have also never had a power incursion on our 
DC plant. Any of them. 

I’m not sure I’d consider it cheap, but it’s not horrifying expensive and it 
*works* when you deploy enough of it in a 2 or 3N fashion.

-Ben

> On Dec 28, 2019, at 9:04 AM, Baldur Norddahl  
> wrote:
> 
> 
> What is wrong with lead acid battery backup? Seems to be exceedingly stable 
> from my experience. We have all our equipment on -48V DC and have never had a 
> power interruption at any site.
> 
> The requirements here are 48 hours of backup by law. Telecom is declared to 
> be part of emergency and defense, so they put in a requirement for 
> resilience. 
> 
> Regards 
> 
> Baldur 
> 
> 
> tor. 26. dec. 2019 11.33 skrev Joe Maimon :
>> Unless telecom infrastructure has been diligently changing out the lead 
>> acid battery approach at all their remote terminals, powered gpon, hfc 
>> and antennae plants will never last more than minutes. If at all.
>> 
>> A traditional car has between a 100-200amp alternator @12volts
>> 
>> How much generating capacity can you get out of a typical hybrid?
>> 
>> Self-isolating and re-tieing inverters. Economic household ATS systems. 
>> Do those exist?
>> 
>> Enough independent distributed capacity and now comes the ability to 
>> create grid islands. How might that look?
>> 
>> Electric grid shortage is likely coming to NYC, courtesy of folk of 
>> certain political persuasion and their love of stone age era living. IP 
>> decommissioning.
>> 
>> If you have CO loop copper, keep it.
>> 
>> Joe
>> 
>> Don Gould wrote:
>> > This is a very short term problem.
>> >
>> > The market is going to fill with battery storage sooner rather than 
>> > later.
>> >
>> > Solar is just exploding.
>> >
>> > Your car will "house tie".
>> >
>> > 6G will solve your data problem.
>> >
>> > D
>> >
>> >
>> >
>> > -- 
>> > Don Gould
>> > 5 Cargill Place
>> > Richmond
>> > Christchurch, New Zealand
>> > Mobile/Telegram: + 64 21 114 0699
>> > www. bowenvale.co.nz
>> >
>> >
>> >
>> >  Original message 
>> > From: Michael Thomas 
>> > Date: 26/12/19 2:33 PM (GMT+12:00)
>> > To: nanog@nanog.org
>> > Subject: power to the internet
>> >
>> >
>> > https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
>> >
>> >
>> > This article details some of the issues with California's "new reality"
>> > of planned blackouts. One of the big things that came to light with
>> > these blackouts is that our network infrastructure's resilience is
>> > pretty lacking. While I was (surprisingly to me) ok with my DSL
>> > connection out in the boonies, lots and lots of people with cable
>> > weren't so lucky. And I'm not sure how bad the situation is with
>> > cellular infrastructure, but I assume it's not much better than cable.
>> > And I wouldn't doubt that other DSL deployments go dark when power is
>> > down. I have no clue with fiber.
>> >
>> > So I guess what I'm wondering is what can we do about this? What should
>> > we do about this? These days IP access is not just convenience, it's the
>> > way we go about our lives, just like electricity itself. At base, it
>> > seems to me that network operators should be required to keep the lights
>> > on in blackouts just like POTS operators do now. If I have power to
>> > light my modem or charge in my phone, I should be able to get onto the
>> > net. That seems like table stakes.
>> >
>> > One of the things we learned also is that the blackouts seem to last
>> > between 2-3 days apiece. I happen to have a generator since I'm out in
>> > the boonies and our power gets cut regularly because of snow, but not
>> > everyone has that luxury. I kind of want to think that my router+modem
>> > use about 20 watts, so powering it up would take about 1.5kwh for 3
>> > days. a quick google look shows that I'd probably need to shell out $500
>> > or so for a battery of that capacity, and that's doesn't include your
>> > phones, laptops, tv's, etc power needs. What does that mean? That is a
>> > major expense for a lot of people.
>> >
>> > On the bright side, I hear that power generator companies stocks have
>> > gone through the roof.
>> >
>> > On the dark side, this is probably coming to a lot more states and
>> > countries due to climate change. Australia. Sigh.
>> >
>> > Mike
>> >
>> 

Re: power to the internet

[Tom Beecher]

To be hyper pedantic, lithium / li-on batteries are recyclable. It’s just
not being done today on a substantive scale today for mostly economic
reasons.

On Sat, Dec 28, 2019 at 14:51 Dan Hollis  wrote:

> Nothing.
>
> It is extremely cheap, extremely durable, and nearly 100% recyclable. All
> the things lithium is not.
>
> The only thing is lead acid is not power dense, but that is not generally
> a problem at sites.
>
> -Dan
>
> On Sat, 28 Dec 2019, Baldur Norddahl wrote:
>
> > What is wrong with lead acid battery backup? Seems to be exceedingly
> stable
> > from my experience. We have all our equipment on -48V DC and have never
> had
> > a power interruption at any site.
> >
> > The requirements here are 48 hours of backup by law. Telecom is declared
> to
> > be part of emergency and defense, so they put in a requirement for
> > resilience.
> >
> > Regards
> >
> > Baldur
> >
> >
> > tor. 26. dec. 2019 11.33 skrev Joe Maimon :
> >
> >> Unless telecom infrastructure has been diligently changing out the lead
> >> acid battery approach at all their remote terminals, powered gpon, hfc
> >> and antennae plants will never last more than minutes. If at all.
> >>
> >> A traditional car has between a 100-200amp alternator @12volts
> >>
> >> How much generating capacity can you get out of a typical hybrid?
> >>
> >> Self-isolating and re-tieing inverters. Economic household ATS systems.
> >> Do those exist?
> >>
> >> Enough independent distributed capacity and now comes the ability to
> >> create grid islands. How might that look?
> >>
> >> Electric grid shortage is likely coming to NYC, courtesy of folk of
> >> certain political persuasion and their love of stone age era living. IP
> >> decommissioning.
> >>
> >> If you have CO loop copper, keep it.
> >>
> >> Joe
> >>
> >> Don Gould wrote:
> >>> This is a very short term problem.
> >>>
> >>> The market is going to fill with battery storage sooner rather than
> >>> later.
> >>>
> >>> Solar is just exploding.
> >>>
> >>> Your car will "house tie".
> >>>
> >>> 6G will solve your data problem.
> >>>
> >>> D
> >>>
> >>>
> >>>
> >>> --
> >>> Don Gould
> >>> 5 Cargill Place
> >>> Richmond
> >>> Christchurch, New Zealand
> >>> Mobile/Telegram: + 64 21 114 0699
> >>> www. bowenvale.co.nz
> >>>
> >>>
> >>>
> >>>  Original message 
> >>> From: Michael Thomas 
> >>> Date: 26/12/19 2:33 PM (GMT+12:00)
> >>> To: nanog@nanog.org
> >>> Subject: power to the internet
> >>>
> >>>
> >>>
> >>
> https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
> >>>
> >>>
> >>> This article details some of the issues with California's "new reality"
> >>> of planned blackouts. One of the big things that came to light with
> >>> these blackouts is that our network infrastructure's resilience is
> >>> pretty lacking. While I was (surprisingly to me) ok with my DSL
> >>> connection out in the boonies, lots and lots of people with cable
> >>> weren't so lucky. And I'm not sure how bad the situation is with
> >>> cellular infrastructure, but I assume it's not much better than cable.
> >>> And I wouldn't doubt that other DSL deployments go dark when power is
> >>> down. I have no clue with fiber.
> >>>
> >>> So I guess what I'm wondering is what can we do about this? What should
> >>> we do about this? These days IP access is not just convenience, it's
> the
> >>> way we go about our lives, just like electricity itself. At base, it
> >>> seems to me that network operators should be required to keep the
> lights
> >>> on in blackouts just like POTS operators do now. If I have power to
> >>> light my modem or charge in my phone, I should be able to get onto the
> >>> net. That seems like table stakes.
> >>>
> >>> One of the things we learned also is that the blackouts seem to last
> >>> between 2-3 days apiece. I happen to have a generator since I'm out in
> >>> the boonies and our power gets cut regularly because of snow, but not
> >>> everyone has that luxury. I kind of want to think that my router+modem
> >>> use about 20 watts, so powering it up would take about 1.5kwh for 3
> >>> days. a quick google look shows that I'd probably need to shell out
> $500
> >>> or so for a battery of that capacity, and that's doesn't include your
> >>> phones, laptops, tv's, etc power needs. What does that mean? That is a
> >>> major expense for a lot of people.
> >>>
> >>> On the bright side, I hear that power generator companies stocks have
> >>> gone through the roof.
> >>>
> >>> On the dark side, this is probably coming to a lot more states and
> >>> countries due to climate change. Australia. Sigh.
> >>>
> >>> Mike
> >>>
> >>
> >>
> >
>

Re: power to the internet

[Dan Hollis]

Nothing.

It is extremely cheap, extremely durable, and nearly 100% recyclable. All 
the things lithium is not.


The only thing is lead acid is not power dense, but that is not generally 
a problem at sites.


-Dan

On Sat, 28 Dec 2019, Baldur Norddahl wrote:


What is wrong with lead acid battery backup? Seems to be exceedingly stable
from my experience. We have all our equipment on -48V DC and have never had
a power interruption at any site.

The requirements here are 48 hours of backup by law. Telecom is declared to
be part of emergency and defense, so they put in a requirement for
resilience.

Regards

Baldur


tor. 26. dec. 2019 11.33 skrev Joe Maimon :


Unless telecom infrastructure has been diligently changing out the lead
acid battery approach at all their remote terminals, powered gpon, hfc
and antennae plants will never last more than minutes. If at all.

A traditional car has between a 100-200amp alternator @12volts

How much generating capacity can you get out of a typical hybrid?

Self-isolating and re-tieing inverters. Economic household ATS systems.
Do those exist?

Enough independent distributed capacity and now comes the ability to
create grid islands. How might that look?

Electric grid shortage is likely coming to NYC, courtesy of folk of
certain political persuasion and their love of stone age era living. IP
decommissioning.

If you have CO loop copper, keep it.

Joe

Don Gould wrote:

This is a very short term problem.

The market is going to fill with battery storage sooner rather than
later.

Solar is just exploding.

Your car will "house tie".

6G will solve your data problem.

D



--
Don Gould
5 Cargill Place
Richmond
Christchurch, New Zealand
Mobile/Telegram: + 64 21 114 0699
www. bowenvale.co.nz



 Original message 
From: Michael Thomas 
Date: 26/12/19 2:33 PM (GMT+12:00)
To: nanog@nanog.org
Subject: power to the internet




https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678



This article details some of the issues with California's "new reality"
of planned blackouts. One of the big things that came to light with
these blackouts is that our network infrastructure's resilience is
pretty lacking. While I was (surprisingly to me) ok with my DSL
connection out in the boonies, lots and lots of people with cable
weren't so lucky. And I'm not sure how bad the situation is with
cellular infrastructure, but I assume it's not much better than cable.
And I wouldn't doubt that other DSL deployments go dark when power is
down. I have no clue with fiber.

So I guess what I'm wondering is what can we do about this? What should
we do about this? These days IP access is not just convenience, it's the
way we go about our lives, just like electricity itself. At base, it
seems to me that network operators should be required to keep the lights
on in blackouts just like POTS operators do now. If I have power to
light my modem or charge in my phone, I should be able to get onto the
net. That seems like table stakes.

One of the things we learned also is that the blackouts seem to last
between 2-3 days apiece. I happen to have a generator since I'm out in
the boonies and our power gets cut regularly because of snow, but not
everyone has that luxury. I kind of want to think that my router+modem
use about 20 watts, so powering it up would take about 1.5kwh for 3
days. a quick google look shows that I'd probably need to shell out $500
or so for a battery of that capacity, and that's doesn't include your
phones, laptops, tv's, etc power needs. What does that mean? That is a
major expense for a lot of people.

On the bright side, I hear that power generator companies stocks have
gone through the roof.

On the dark side, this is probably coming to a lot more states and
countries due to climate change. Australia. Sigh.

Mike

Re: power to the internet

[Baldur Norddahl]

What is wrong with lead acid battery backup? Seems to be exceedingly stable
from my experience. We have all our equipment on -48V DC and have never had
a power interruption at any site.

The requirements here are 48 hours of backup by law. Telecom is declared to
be part of emergency and defense, so they put in a requirement for
resilience.

Regards

Baldur


tor. 26. dec. 2019 11.33 skrev Joe Maimon :

> Unless telecom infrastructure has been diligently changing out the lead
> acid battery approach at all their remote terminals, powered gpon, hfc
> and antennae plants will never last more than minutes. If at all.
>
> A traditional car has between a 100-200amp alternator @12volts
>
> How much generating capacity can you get out of a typical hybrid?
>
> Self-isolating and re-tieing inverters. Economic household ATS systems.
> Do those exist?
>
> Enough independent distributed capacity and now comes the ability to
> create grid islands. How might that look?
>
> Electric grid shortage is likely coming to NYC, courtesy of folk of
> certain political persuasion and their love of stone age era living. IP
> decommissioning.
>
> If you have CO loop copper, keep it.
>
> Joe
>
> Don Gould wrote:
> > This is a very short term problem.
> >
> > The market is going to fill with battery storage sooner rather than
> > later.
> >
> > Solar is just exploding.
> >
> > Your car will "house tie".
> >
> > 6G will solve your data problem.
> >
> > D
> >
> >
> >
> > --
> > Don Gould
> > 5 Cargill Place
> > Richmond
> > Christchurch, New Zealand
> > Mobile/Telegram: + 64 21 114 0699
> > www. bowenvale.co.nz
> >
> >
> >
> >  Original message 
> > From: Michael Thomas 
> > Date: 26/12/19 2:33 PM (GMT+12:00)
> > To: nanog@nanog.org
> > Subject: power to the internet
> >
> >
> >
> https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
> >
> >
> > This article details some of the issues with California's "new reality"
> > of planned blackouts. One of the big things that came to light with
> > these blackouts is that our network infrastructure's resilience is
> > pretty lacking. While I was (surprisingly to me) ok with my DSL
> > connection out in the boonies, lots and lots of people with cable
> > weren't so lucky. And I'm not sure how bad the situation is with
> > cellular infrastructure, but I assume it's not much better than cable.
> > And I wouldn't doubt that other DSL deployments go dark when power is
> > down. I have no clue with fiber.
> >
> > So I guess what I'm wondering is what can we do about this? What should
> > we do about this? These days IP access is not just convenience, it's the
> > way we go about our lives, just like electricity itself. At base, it
> > seems to me that network operators should be required to keep the lights
> > on in blackouts just like POTS operators do now. If I have power to
> > light my modem or charge in my phone, I should be able to get onto the
> > net. That seems like table stakes.
> >
> > One of the things we learned also is that the blackouts seem to last
> > between 2-3 days apiece. I happen to have a generator since I'm out in
> > the boonies and our power gets cut regularly because of snow, but not
> > everyone has that luxury. I kind of want to think that my router+modem
> > use about 20 watts, so powering it up would take about 1.5kwh for 3
> > days. a quick google look shows that I'd probably need to shell out $500
> > or so for a battery of that capacity, and that's doesn't include your
> > phones, laptops, tv's, etc power needs. What does that mean? That is a
> > major expense for a lot of people.
> >
> > On the bright side, I hear that power generator companies stocks have
> > gone through the roof.
> >
> > On the dark side, this is probably coming to a lot more states and
> > countries due to climate change. Australia. Sigh.
> >
> > Mike
> >
>
>

Re: power to the internet

[Jason Wilson]

This is all in conjunction with the CPUC. I believe it is also a part of a
court order. I’ll need to find that later

https://www.cpuc.ca.gov/deenergization/


I don’t want to copy the whole thing but this is the bulk of it before it
goes Into when the outages were.

Wildfires are more destructive and deadlier than in the past, and the
threat of wildfires is more prevalent throughout the state and calendar
year. The overall pattern shows the emerging effects of climate change in
our daily lives. If you need information on disaster relief protections for
customers of affected areas during any state of emergency, please read our
blog .

Throughout the year, the CPUC works with CalFire and the Office of
Emergency Services to reduce the risk of utility infrastructure starting
wildfires, to strengthen utility preparedness for emergencies, and to
improve utility services during and after emergencies. Interagency
coordination, and cooperation from the utilities is essential when the
threat of wildfires is high.

The State's investor-owned electric utilities, notably Pacific Gas and
Electric Company (PG), Southern California Edison, and San Diego Gas &
Electric (SDG), may shut off electric power, referred to as
"de-energization" or Public Safety Power Shut-offs (PSPS), to protect
public safety under California law, specifically California Public
Utilities Code

(PU
Code) Sections 451 and 399.2(a).


CPUC Actions

On July 12, 2018, the CPUC adopted Resolution ESRB-8
to
strengthen customer notification requirements before de-energization events
and ordered utilities to engage local communities in developing
de-energization programs. Utilities must submit a report within 10 days
after each de-energization event, and after high-fire-threat events where
the utility provided notifications to local government, agencies, and
customers of possible de-energization though no de-energization occurred.
These reports are accessible below.

On December 13, 2018, the CPUC opened a new Rulemaking

 (R.18-12-005
)
to examine utilities' PSPS processes and practices, the impacts on
communities and access and functional needs populations, efforts to reduce
the need for de-energization, and mitigate measures to reduce the impacts
when implemented. The Rulemaking will also review and improve existing
reporting requirements.  On May 30, 2019, in its ongoing efforts to
expeditiously implement Senate Bill 901, the CPUC made its Phase I decision

in
the proceeding, making improvements to utility communication and
notification protocols to ensure that clear rules are in place as early as
possible to prepare for the 2019 fire season.

On August 14, 2019, the CPUC opened a second phase (Phase 2) in R.18-12-005
to address additional aspects of the utilities' PSPS processes and
practices and to expand upon the guidelines adopted in Phase 1. In Phase 2
the CPUC will consider, among other issues, identification and
communication with access and functional needs populations, communication
with customers while the power is turned off, communication during
reenergization, mitigation measures, coordination with emergency
responders, and transmission-level de-energization.

Proceeding documents are available on the Docket Card

.

Send your comments on the proceeding to public.advi...@cpuc.ca.gov and
refer to proceeding number R.18-12-005.

The CPUC is working with the Office of Emergency Services (Cal OES)
, CAL FIRE , and
first-responders to address potential impacts of utility de-energization
practices on emergency response activities, including evacuations. The CPUC
is also monitoring the development and will continuously assess
implementation of de-energization programs by utilities, including
performing a thorough review of de-energization events as they occur.

On October 18, 2019, the CPUC held an Emergency Meeting to hear from top
Pacific Gas and Electric Company (PG) executives to publicly address the
mistakes and operational gaps identified in the utility’s October 2019 PSPS
events and to provide lessons learned to ensure they are not repeated. More
information about the meeting is available under "October 2019 PSPS Events"
below.


On Fri, Dec 27, 2019 at 2:57 PM Michael Thomas  wrote:

>
> On 12/26/19 6:38 PM, Fred Baker wrote:
> >> This time it’s PG all alone, but still fallout from back then. Too
> >> much liability and 

Re: power to the internet

[Michael Thomas]

On 12/26/19 6:38 PM, Fred Baker wrote:
This time it’s PG all alone, but still fallout from back then. Too 
much liability and they’ve not maintained the infrastructure and so 
they decided that to reduce the liability costs it’s cheaper to 
blackout. Same story again different colors. PG making a mint while 
people get screwed (PG was mostly at the getting screwed end in 
2000-2001)


PG has been the one in the news, but SCE appears to have been making 
the same choices with about the same effects. The Thomas Fire was 
briefly the largest wildfire in state history, and the source (well, 
with the rain) of the Montecito mud flow a few weeks later. We're told 
that SCE seems to figure in that one and several others before and since.


I go back and forth on who might be responsible. The electric 
utilities bear blame for their infrastructure; it should be 
underground, not strung from poles. I would put some to the state and 
the management of the various national forests and national parks in 
the area - one of the outcomes from a fire in 2007 or thereabouts was 
that the ecology folks had been protecting foliage, and that foliage 
burned and clogged streams, with all sorts of results. Surprise! If 
you're worried about ecology, you should support management of it. In 
California, there are also laws holding home-owners responsible for 
"defensible space" around their homes.




When I lived in Socal, we certainly had hellacious brush fires when the 
Santa Ana winds blew. I don't remember any/many of them being attributed 
SCE though? Maybe I just wasn't paying attention? Do remember anything 
about that, Fred? We've forever had an urban-wildland interface problem 
-- I mean, how many times has Malibu burned down, it seemed like every 
other year.


Apparently San Diego Gas and Electric has been something of a pioneer 
after the horrible Cedar fire, and apparently it's made a difference.


Mike

Re: power to the internet

[Fred Baker]

> This time it’s PG all alone, but still fallout from back then. Too much 
> liability and they’ve not maintained the infrastructure and so they decided 
> that to reduce the liability costs it’s cheaper to blackout. Same story again 
> different colors. PG making a mint while people get screwed (PG was 
> mostly at the getting screwed end in 2000-2001)

PG has been the one in the news, but SCE appears to have been making the same 
choices with about the same effects. The Thomas Fire was briefly the largest 
wildfire in state history, and the source (well, with the rain) of the 
Montecito mud flow a few weeks later. We're told that SCE seems to figure in 
that one and several others before and since.

I go back and forth on who might be responsible. The electric utilities bear 
blame for their infrastructure; it should be underground, not strung from 
poles. I would put some to the state and the management of the various national 
forests and national parks in the area - one of the outcomes from a fire in 
2007 or thereabouts was that the ecology folks had been protecting foliage, and 
that foliage burned and clogged streams, with all sorts of results. Surprise! 
If you're worried about ecology, you should support management of it. In 
California, there are also laws holding home-owners responsible for "defensible 
space" around their homes.

https://www.google.com/search?q=california+brush+clearing+laws
https://en.m.wikipedia.org/wiki/Thomas_Fire 

https://www.washingtonpost.com/graphics/2018/national/montecito-before-after/ 
https://www.edhat.com/news/10-homes-destroyed-in-holiday-fire 
https://www.edhat.com/news/cave-fire-now-100-contained

Re: power to the internet

[Don Gould]

Hi Brandon,

I agree with lots of what you wrote, here's some more thoughts

tl;dr

Batteries from cars will fix your issues along with smart guys just 
wanting to get the job done.


WHO AM I?!

I am a coms tech.  I have been on this list for ~20 years.

My job is simply to look at the environment and make coms appear.

My job is not to blame others, question politics or argue ethics.


MY POWER, MY RULES, MY RIGHTS

I have two Nissan LEAF with the 24Kwh battery, both at 10 of 12 bars of 
original battery capacity, roughly 8.5 years old.  This gives each car 
roughly 110km (65 mile) of range between charges. Both are 6 months into 
a 5 year payment plan.


"My job is simply to look at the environment and make coms appear. "

So, looking at my actual environment, I know that I'm going to end up 
with two 15kwh batteries in about 8 years and two 15kwh generators (yes, 
the motors in those things put out about 15kwh under breaking).


Ok, so I'm not going to personally convert these into anything at 57 
(I'm 49 now), but some young up and coming (Mr 12, sitting in my living 
room, playing with his Christmas presents, perhaps) will.


MY POWER

In buying these two cars, I was wanting to address the 'demand side' of 
my energy question.  Now I have a decent amount of demand, I'm going to 
put PV on my roof this year.


My utility company will buy power back from me at about 15% of what I 
pay retail, so selling them power makes no sense.


My best friend has just completed a 7kwh in solar installation to power 
is 210 tank gold fish facility.  He's currently exporting power and now 
looking at storage solutions.


We're 'early adoptors', so you can look to us to see where the trend is 
going.


MY RULES

My other mate asked our utility for power to his new home and they said 
"Yes sir, half a years wages plus a monthly contribution to the up keep 
for the brand new plant you're paying for up front." I'm sure it comes 
as no surprise, he said "no thanks" and then built a whole off grid 
system including a 6kwh diesel generator. (He is also a coms tech, 20 
years older than I am, though not on this list).


MY RIGHTS

My country lets me do most of the electrical work on my own home up to 
the power board as long as I follow the rules.


I can generate power, I can export power, I can store power.

WHERE ARE WE GOING?

Just like the US, my community has people who will sit and blame 
government, will blame the power company, will blame their local elected 
members, will drag us all into court, will blame the boomers.


But also like the US my community also has people who are coms techs who 
just look at the space, see what's going on and build accordinly.


TRANSITION IS A PROBLEM TO SOLVE AT THE LOCAL LEVEL

As many of you are picking up, we're in a transitional time.  The US 
utilities know this and they're scared to invest, and for good reason, 
some have been very burnt in the past.


NOT ENOUGH COPPER  "COMS WILL DIE"

I remember ~20 years ago, reading posts about how there isn't enough 
copper to supply the growing demand in lines.  Now today I'm reading 
about how you have abundant abandoned copper as DSLAMs are moved closer 
to the edge.


Power is the same in my view.  We're going to see local edge generation, 
storage and change of use (I have all LED lighting, TV's that use a 
fraction of the power they used to, but two electric cars).


Your US power will stop browning out when you put storage in the network 
and can drop the peek load.  But you're going to have to drive Mrs Brown 
to make that personal investment, and she won't until it starts to hurt.


The utility will under ground lines when it can see a clearer picture 
for the future.  It will take coms with it (FTTH).  It won't until the 
situation smacks the political space so hard that regulation is sorted 
so it can be happy with investment (in NZ we had to get down this path - 
this is quite a good read, the incumbent - 
https://en.wikipedia.org/wiki/Spark_New_Zealand ).


RANDOM FYI

As I type this, I have over 8 layer 1 networks I can use at my house...

* ENABLED - FTTH
* VODAFONE - HFC
* CHORUS - FTTN/POTS
* VODAFONE - 4/5G
* SPARK - 4G
* 2DEGREES - 4G
* CCC - private council network
* YOURNET - my own private WISP network
* OTHER WISP - there's half a dozen that I can see with a quick scan 
from hills about 8km away


BUT THIS WAS ABOUT POWER FOR OUR EXISTING COMS, NOT THE WHOLE NEIGHBOURHOOD!

Many of you are thinking in a silo.  You live in a community and you're 
not acting like it.


Fix the power problems for the community rather than just trying to care 
only for your little nest.  As some of you have pointed out, you're 
lugging gas to generators, connecting trucks with inverters just to keep 
the coms going in case there's a 911 call required, without thinking 
that in the even of such, the emergency service won't be able to get to 
Mrs Brown because the streets are blocked by your trucks all connected 
to random 

Re: power to the internet

[Michael Thomas]

On 12/26/19 4:06 PM, John Levine wrote:

In article  you write:

run but are now showing their long term consequences, notably land use
that encourages sprawl and construction in ill-suited areas

If we stopped construction in all of the ill-suited areas, we'd stop
construction all together, and tear down much more. We have it all here:
earthquakes, floods, fires; often the trifecta.  We could certainly be
smarter, but the nature of the geography here is both a blessing and a
curse.

Among California's many problems is a bizarre terror of upzoning and
infill construction, hence the sprawl.  Here in my rustic bit of
upstate New York you can build a two-family anywhere you can build a
single family and the world has not come to an end.



NIMBYism. My previous state senator (Scott Wiener) has been trying all 
he can to make headway on that front. But NIMBY's are a strong force and 
don't cleave down party lines whatsoever.





PG is especially egregious as it has extremely high rates and
piss-poor maintenance. Where does all of that money go? Execs and
shareholders.

Evidently not since they've been through bankruptcy a few times.  I
think they're just institutionally incompetent as well as having an
unusually environmentally hostile territory to serve.  (Around here when
the power company screws up, the power fails but the county does not
catch fire.)
Well that was true here until about 10-20 years ago too. Fire seasons 
are about 2 months longer, iirc. From beginning of May into first part 
of December. We almost never had fires in June but now they're fairly 
common. That's true for a lot of western US now.



I don't know what the ultimate solution is, but
whatever it is cannot have those perverse incentives.

The LA DWP seems to do OK.


As does Sacramento's SMUD. Part of the problem is that they are just so 
large. San Jose and SF are thinking very seriously about splitting off. 
Which would probably create yet another death spiral for PG because it 
would leave all of the expensive distribution (= out in the boonies, 
etc) and allow cities to cherry pick the cheaper distribution areas when 
it makes sense. The entire thing is a shitshow.


Mike

Re: power to the internet

[John Levine]

In article  you write:
>> run but are now showing their long term consequences, notably land use
>> that encourages sprawl and construction in ill-suited areas
>
>If we stopped construction in all of the ill-suited areas, we'd stop 
>construction all together, and tear down much more. We have it all here: 
>earthquakes, floods, fires; often the trifecta.  We could certainly be 
>smarter, but the nature of the geography here is both a blessing and a 
>curse.

Among California's many problems is a bizarre terror of upzoning and
infill construction, hence the sprawl.  Here in my rustic bit of
upstate New York you can build a two-family anywhere you can build a
single family and the world has not come to an end.

>PG is especially egregious as it has extremely high rates and 
>piss-poor maintenance. Where does all of that money go? Execs and 
>shareholders.

Evidently not since they've been through bankruptcy a few times.  I
think they're just institutionally incompetent as well as having an
unusually environmentally hostile territory to serve.  (Around here when
the power company screws up, the power fails but the county does not
catch fire.)

>I don't know what the ultimate solution is, but 
>whatever it is cannot have those perverse incentives.

The LA DWP seems to do OK.

R's,
John

Re: power to the internet

[Brandon Butterworth]

On Thu Dec 26, 2019 at 11:20:01AM -0800, Michael Thomas wrote:
> I just looked up Telsa's battery packs and they seem to be between 
> 60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 
> days. Admittedly we can turn off the hot tub, water heater, etc to 
> stretch it out. And of course, that means that you can't drive it... The 
> one thing that would be for everybody's good is using them during peak 
> hours. If you work normal hours, then that only gets part of the peak 
> load, unfortunately.

Many with a tesla (or three) are likely to get some local PV/wind
generation to charge them (more so if the outages become a regular
event). Then the base load doesn't matter and they can still drive
some of them while the others charge. Some may just use their EV as
a battery that has other uses when the power isn't out and not care
they can't use it for both. There are plenty of projects working on
this distributed storage/generation model.

> But of course this has nothing to do with the network power problem. I 
> assume they won't be parking a Tesla next to a CMTS headend.

Proliferation of local generation and storage may have a lot of impact
on network expectations. Last mile that assumes when the cab loses
power then the consumers have too will have to update their assumptions.
The consumers running on local power are able to carry on as normal
and expect network to carry on too.

Architectures with lots of distributed small active plant may be
harder hit trying to add generation than those with passive plant
and few active nodes.

brandon

Re: power to the internet

[Constantine A. Murenin]

On Thu, 26 Dec 2019 at 13:19, Stephen Satchell  wrote:

> Longer term, review your backhauls and interconnects.  Dark fiber would
> be preferred here, because you would be controlling backup power at both
> ends, and not depending on intermediate nodes.
>

What about the NSA taps?  Do they tap the dark fibre?  Would it go dark in
a power outage, or do they engage some passive sort of taps?

C.

RE: power to the internet

[Keith Medcalf]

>I just looked up Telsa's battery packs and they seem to be between
>60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3
>days. Admittedly we can turn off the hot tub, water heater, etc to
>stretch it out. And of course, that means that you can't drive it... The
>one thing that would be for everybody's good is using them during peak
>hours. If you work normal hours, then that only gets part of the peak
>load, unfortunately.

Just buy three of them.  Two to leave in the garage as a "mobile battery pack" 
and one to drive around.

All problems solved.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.

Re: power to the internet

[Michael Thomas]

On 12/26/19 11:18 AM, John Levine wrote:

In article  
you write:

To reanswer the question posed though, is still the same ; $$$. If network
operators take the position that the electric utility supply should be more
reliable than it is, then they need to start influencing and lobbying for
ways for that to happen. If not, they will have to increase investments
into local generation or storage capacity to bridge those gaps.

You seem to imply that regulation is inherently bad; however the scenario
that you describe (power failures impacting 911 service) is only a concern
to an operator if there is a legislatively define deterrent.

California suffers from an unusual combination of a dry climate that
is getting dryer and political decisions that made sense in the short
run but are now showing their long term consequences, notably land use
that encourages sprawl and construction in ill-suited areas


If we stopped construction in all of the ill-suited areas, we'd stop 
construction all together, and tear down much more. We have it all here: 
earthquakes, floods, fires; often the trifecta.  We could certainly be 
smarter, but the nature of the geography here is both a blessing and a 
curse.




, and a
regulator that keeps short term consumer prices down at the cost of
reliability and long term stability.  None of this should be a
surprise to anyone familiar with the situation.
PG is especially egregious as it has extremely high rates and 
piss-poor maintenance. Where does all of that money go? Execs and 
shareholders. And if some random nyc hedge fund gets its way it's going 
to get even worse. I don't know what the ultimate solution is, but 
whatever it is cannot have those perverse incentives.



Mike

Re: power to the internet

[Michael Thomas]

On 12/26/19 11:18 AM, Stephen Satchell wrote:

On 12/26/19 10:55 AM, Michael Thomas wrote:
Here in California, you're going to need a lot more than 8 hours. We 
had one that lasted 3 days, followed by about 8 hours of power, 
followed by 2 days of no power. If this is the new normal, and I'm 
afraid that it is, that's probably going to require some pretty hefty 
backup. Not to mention expensive.


The one "good" thing that PG did is expose all of these 
vulnerabilities. Every neighborhood probably knows whether their 
carrier is naughty or nice now.


Here in Nevada, specifically at Lake Tahoe, power is less reliable 
because of heavy snow and sliding trucks (the power equivalent to a 
backhoe disconnect).  One of the cell sites is on the top level of a 
casino parking garage.  I found out about this when the casino went 
bankrupt, the parking garage was blocked off, and I joined the 
security guard crew to protect the on-site gaming equipment.  Months 
into the project, the cell company in question begged the bankruptcy 
court for access -- to replace the empty propane cylinders in their 
shack.  That's right, no mains tap at all.  When the casino lost power 
because of bill non-payment, the cell site stayed up.


A network operator will need to look at the total cost, including 
labor, of backing up mains power. versus using local genertion 
exclusively -- or using mains power as the backup!  Factor in any 
upcoming fines for service outage, re 911.  (Try to avoid piped 
natural gas as the fuel for onsite generation.)


Longer term, review your backhauls and interconnects.  Dark fiber 
would be preferred here, because you would be controlling backup power 
at both ends, and not depending on intermediate nodes.



One of the interesting things I found out is that POTS termination out 
in the field can be powered by some of the pairs back to the CO they are 
making redundant. It's enough power that running the DSLAM isn't a 
problem either. I'm not sure that that could translate for anything 
else, but there is probably a lot of copper sitting idle these days.


Mike

Re: power to the internet

[Jason Wilson]

AT land line had service trucks parked at RT’s to power them. I talked
with one of the techs. He was on a 12 hour schedule and spent that time
between 3 sites charging the batteries to keep the copper plant running.
They plugged in to the truck inverter and ran the truck all day. He told me
they ran out of generators. Talk about a waste of manpower.

On Thu, Dec 26, 2019 at 10:56 AM Michael Thomas  wrote:

>
> On 12/26/19 10:41 AM, Ben Cannon wrote:
> > Exactly. And we will build it all.
> >
> > The power stuff is serious people.  We’ve gotten letters from the FCC
> > over it.  There is additional regulation coming down when people can’t
> > call 911!
> >
> > You need at minimum 8 hours (or your CRT response time with a
> > generator trailer, or a standby generator or two) of battery on your
> > telecom equipment. All of it. Everywhere.
> >
> > Comcast is the worst about this, they never replace and often don’t
> > even place batteries in their RTs at all - and they are going to get
> > fined over it mark my words.
> >
> >
> Here in California, you're going to need a lot more than 8 hours. We had
> one that lasted 3 days, followed by about 8 hours of power, followed by
> 2 days of no power. If this is the new normal, and I'm afraid that it
> is, that's probably going to require some pretty hefty backup. Not to
> mention expensive.
>
> The one "good" thing that PG did is expose all of these
> vulnerabilities. Every neighborhood probably knows whether their carrier
> is naughty or nice now.
>
> Mike
>
> --

Jason Wilson
Remotely Located
Providing High Speed Internet to out of the way places.
530-651-1736
530-748-9608 Cell
www.remotelylocated.com

Re: power to the internet

[Michael Thomas]

On 12/26/19 11:00 AM, Ben Cannon wrote:

How much generating capacity can you get out of a typical hybrid?


You’re joking right?  A lot… Enough to run an entire neighborhood…   
The Prius makes 50,000watts alone.


With the right circuitry, there is no need for power plants in the 
United States (save that they’re more efficient than internal 
combustion gas engines in the 76hp range) - existing hybrid car 
stock’s generating capacity exceeds the entire US supply.  And it’s 
entirely untapped.


Nissan just tested it for giggles, and found the Leaf (which has NO 
engine at all) can power a house for an entire week.  The batteries 
alone are a game changer, utterly transforming grids.


I just looked up Telsa's battery packs and they seem to be between 
60-100kwh. Our daily use is about 30kwh in the fall, so it's only 2-3 
days. Admittedly we can turn off the hot tub, water heater, etc to 
stretch it out. And of course, that means that you can't drive it... The 
one thing that would be for everybody's good is using them during peak 
hours. If you work normal hours, then that only gets part of the peak 
load, unfortunately.


But of course this has nothing to do with the network power problem. I 
assume they won't be parking a Tesla next to a CMTS headend.


Mike

Re: power to the internet

[John Levine]

In article  
you write:
>To reanswer the question posed though, is still the same ; $$$. If network
>operators take the position that the electric utility supply should be more
>reliable than it is, then they need to start influencing and lobbying for
>ways for that to happen. If not, they will have to increase investments
>into local generation or storage capacity to bridge those gaps.
>
>You seem to imply that regulation is inherently bad; however the scenario
>that you describe (power failures impacting 911 service) is only a concern
>to an operator if there is a legislatively define deterrent.

California suffers from an unusual combination of a dry climate that
is getting dryer and political decisions that made sense in the short
run but are now showing their long term consequences, notably land use
that encourages sprawl and construction in ill-suited areas, and a
regulator that keeps short term consumer prices down at the cost of
reliability and long term stability.  None of this should be a
surprise to anyone familiar with the situation.

Even well run US utilities are much less reliable than the norm in
Europe or Japan.  Where ISPs in the US are figuring out how to install
batteries and backup generators or private windmills or whatever,
their European peers pay somewhat higher utility bills and don't have
to worry about the other stuff.  You'll pay either way.  European
utilities aren't more reliable by accident; that's how they're
regulated.

Calfornia also offers an interesting natural experiment comparing
privately run utilities PG and SCE and the city owned Los Angeles
DWP.

Re: power to the internet

[Stephen Satchell]

On 12/26/19 10:55 AM, Michael Thomas wrote:
Here in California, you're going to need a lot more than 8 hours. We had 
one that lasted 3 days, followed by about 8 hours of power, followed by 
2 days of no power. If this is the new normal, and I'm afraid that it 
is, that's probably going to require some pretty hefty backup. Not to 
mention expensive.


The one "good" thing that PG did is expose all of these 
vulnerabilities. Every neighborhood probably knows whether their carrier 
is naughty or nice now.


Here in Nevada, specifically at Lake Tahoe, power is less reliable 
because of heavy snow and sliding trucks (the power equivalent to a 
backhoe disconnect).  One of the cell sites is on the top level of a 
casino parking garage.  I found out about this when the casino went 
bankrupt, the parking garage was blocked off, and I joined the security 
guard crew to protect the on-site gaming equipment.  Months into the 
project, the cell company in question begged the bankruptcy court for 
access -- to replace the empty propane cylinders in their shack.  That's 
right, no mains tap at all.  When the casino lost power because of bill 
non-payment, the cell site stayed up.


A network operator will need to look at the total cost, including labor, 
of backing up mains power. versus using local genertion exclusively -- 
or using mains power as the backup!  Factor in any upcoming fines for 
service outage, re 911.  (Try to avoid piped natural gas as the fuel for 
onsite generation.)


Longer term, review your backhauls and interconnects.  Dark fiber would 
be preferred here, because you would be controlling backup power at both 
ends, and not depending on intermediate nodes.

Re: power to the internet

[Ben Cannon]

> How much generating capacity can you get out of a typical hybrid?

You’re joking right?  A lot… Enough to run an entire neighborhood…   The Prius 
makes 50,000watts alone.

With the right circuitry, there is no need for power plants in the United 
States (save that they’re more efficient than internal combustion gas engines 
in the 76hp range) - existing hybrid car stock’s generating capacity exceeds 
the entire US supply.  And it’s entirely untapped. 

Nissan just tested it for giggles, and found the Leaf (which has NO engine at 
all) can power a house for an entire week.  The batteries alone are a game 
changer, utterly transforming grids.

> Self-isolating and re-tieing inverters. Economic household ATS systems. Do 
> those exist?

Yes I have a patent in one type of this this; it exists in numerous variants.

ATS-es for partial loads of up to 6,000watts are like $400.

You sound smart, but did you research any of this or just post?

-Ben.

-Ben Cannon
CEO 6x7 Networks & 6x7 Telecom, LLC 
b...@6by7.net 




> On Dec 26, 2019, at 2:31 AM, Joe Maimon  wrote:
> 
> Unless telecom infrastructure has been diligently changing out the lead acid 
> battery approach at all their remote terminals, powered gpon, hfc and 
> antennae plants will never last more than minutes. If at all.
> 
> A traditional car has between a 100-200amp alternator @12volts
> 
> How much generating capacity can you get out of a typical hybrid?
> 
> Self-isolating and re-tieing inverters. Economic household ATS systems. Do 
> those exist?
> 
> Enough independent distributed capacity and now comes the ability to create 
> grid islands. How might that look?
> 
> Electric grid shortage is likely coming to NYC, courtesy of folk of certain 
> political persuasion and their love of stone age era living. IP 
> decommissioning.
> 
> If you have CO loop copper, keep it.
> 
> Joe
> 
> Don Gould wrote:
>> This is a very short term problem.
>> 
>> The market is going to fill with battery storage sooner rather than later.
>> 
>> Solar is just exploding.
>> 
>> Your car will "house tie".
>> 
>> 6G will solve your data problem.
>> 
>> D
>> 
>> 
>> 
>> -- 
>> Don Gould
>> 5 Cargill Place
>> Richmond
>> Christchurch, New Zealand
>> Mobile/Telegram: + 64 21 114 0699
>> www. bowenvale.co.nz
>> 
>> 
>> 
>>  Original message 
>> From: Michael Thomas 
>> Date: 26/12/19 2:33 PM (GMT+12:00)
>> To: nanog@nanog.org
>> Subject: power to the internet
>> 
>> 
>> https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
>> 
>> 
>> This article details some of the issues with California's "new reality"
>> of planned blackouts. One of the big things that came to light with
>> these blackouts is that our network infrastructure's resilience is
>> pretty lacking. While I was (surprisingly to me) ok with my DSL
>> connection out in the boonies, lots and lots of people with cable
>> weren't so lucky. And I'm not sure how bad the situation is with
>> cellular infrastructure, but I assume it's not much better than cable.
>> And I wouldn't doubt that other DSL deployments go dark when power is
>> down. I have no clue with fiber.
>> 
>> So I guess what I'm wondering is what can we do about this? What should
>> we do about this? These days IP access is not just convenience, it's the
>> way we go about our lives, just like electricity itself. At base, it
>> seems to me that network operators should be required to keep the lights
>> on in blackouts just like POTS operators do now. If I have power to
>> light my modem or charge in my phone, I should be able to get onto the
>> net. That seems like table stakes.
>> 
>> One of the things we learned also is that the blackouts seem to last
>> between 2-3 days apiece. I happen to have a generator since I'm out in
>> the boonies and our power gets cut regularly because of snow, but not
>> everyone has that luxury. I kind of want to think that my router+modem
>> use about 20 watts, so powering it up would take about 1.5kwh for 3
>> days. a quick google look shows that I'd probably need to shell out $500
>> or so for a battery of that capacity, and that's doesn't include your
>> phones, laptops, tv's, etc power needs. What does that mean? That is a
>> major expense for a lot of people.
>> 
>> On the bright side, I hear that power generator companies stocks have
>> gone through the roof.
>> 
>> On the dark side, this is probably coming to a lot more states and
>> countries due to climate change. Australia. Sigh.
>> 
>> Mike
>> 
> 

Re: power to the internet

[Michael Thomas]

On 12/26/19 10:41 AM, Ben Cannon wrote:

Exactly. And we will build it all.

The power stuff is serious people.  We’ve gotten letters from the FCC 
over it.  There is additional regulation coming down when people can’t 
call 911!


You need at minimum 8 hours (or your CRT response time with a 
generator trailer, or a standby generator or two) of battery on your 
telecom equipment. All of it. Everywhere.


Comcast is the worst about this, they never replace and often don’t 
even place batteries in their RTs at all - and they are going to get 
fined over it mark my words.



Here in California, you're going to need a lot more than 8 hours. We had 
one that lasted 3 days, followed by about 8 hours of power, followed by 
2 days of no power. If this is the new normal, and I'm afraid that it 
is, that's probably going to require some pretty hefty backup. Not to 
mention expensive.


The one "good" thing that PG did is expose all of these 
vulnerabilities. Every neighborhood probably knows whether their carrier 
is naughty or nice now.


Mike

Re: power to the internet

[Michael Thomas]

On 12/26/19 10:26 AM, Tom Beecher wrote:
If that was a reference to my comments, it was certainly not my 
intention. I was striving to avoid it being seen as that, but 
apparently fell short.


Not directed at you at all.



To reanswer the question posed though, is still the same ; $$$. If 
network operators take the position that the electric utility supply 
should be more reliable than it is, then they need to start 
influencing and lobbying for ways for that to happen. If not, they 
will have to increase investments into local generation or storage 
capacity to bridge those gaps.


You seem to imply that regulation is inherently bad; however the 
scenario that you describe (power failures impacting 911 service) is 
only a concern to an operator if there is a legislatively define 
deterrent.



Not at all. I'm saying that this problem will be solved one way or the 
other. Frankly it's surprising that anybody offering telephony service 
has gotten away with not fulfilling the battery backup mandate. I guess 
there must have been some wiggle room that the carriers took advantage 
of. And if so, legislation to fix that will be immanent.


Mike

Re: power to the internet

[Ben Cannon]

Exactly. And we will build it all.

The power stuff is serious people.  We’ve gotten letters from the FCC over it.  
There is additional regulation coming down when people can’t call 911!  

You need at minimum 8 hours (or your CRT response time with a generator 
trailer, or a standby generator or two) of battery on your telecom equipment. 
All of it. Everywhere.  

Comcast is the worst about this, they never replace and often don’t even place 
batteries in their RTs at all - and they are going to get fined over it mark my 
words.

-Ben Cannon
CEO 6x7 Networks & 6x7 Telecom, LLC 
b...@6by7.net 




> On Dec 25, 2019, at 8:41 PM, Don Gould  wrote:
> 
> This is a very short term problem. 
> 
> The market is going to fill with battery storage sooner rather than later. 
> 
> Solar is just exploding. 
> 
> Your car will "house tie".
> 
> 6G will solve your data problem. 
> 
> D
> 
> 
> 
> -- 
> Don Gould
> 5 Cargill Place
> Richmond
> Christchurch, New Zealand
> Mobile/Telegram: + 64 21 114 0699
> www. bowenvale.co.nz
> 
> 
> 
>  Original message 
> From: Michael Thomas 
> Date: 26/12/19 2:33 PM (GMT+12:00)
> To: nanog@nanog.org
> Subject: power to the internet
> 
> 
> https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678
> 
> 
> This article details some of the issues with California's "new reality" 
> of planned blackouts. One of the big things that came to light with 
> these blackouts is that our network infrastructure's resilience is 
> pretty lacking. While I was (surprisingly to me) ok with my DSL 
> connection out in the boonies, lots and lots of people with cable 
> weren't so lucky. And I'm not sure how bad the situation is with 
> cellular infrastructure, but I assume it's not much better than cable. 
> And I wouldn't doubt that other DSL deployments go dark when power is 
> down. I have no clue with fiber.
> 
> So I guess what I'm wondering is what can we do about this? What should 
> we do about this? These days IP access is not just convenience, it's the 
> way we go about our lives, just like electricity itself. At base, it 
> seems to me that network operators should be required to keep the lights 
> on in blackouts just like POTS operators do now. If I have power to 
> light my modem or charge in my phone, I should be able to get onto the 
> net. That seems like table stakes.
> 
> One of the things we learned also is that the blackouts seem to last 
> between 2-3 days apiece. I happen to have a generator since I'm out in 
> the boonies and our power gets cut regularly because of snow, but not 
> everyone has that luxury. I kind of want to think that my router+modem 
> use about 20 watts, so powering it up would take about 1.5kwh for 3 
> days. a quick google look shows that I'd probably need to shell out $500 
> or so for a battery of that capacity, and that's doesn't include your 
> phones, laptops, tv's, etc power needs. What does that mean? That is a 
> major expense for a lot of people.
> 
> On the bright side, I hear that power generator companies stocks have 
> gone through the roof.
> 
> On the dark side, this is probably coming to a lot more states and 
> countries due to climate change. Australia. Sigh.
> 
> Mike
> 

Re: power to the internet

[Michael Thomas]

On 12/26/19 10:19 AM, Jason Wilson wrote:
As a small WISP operator in Northern California and well into the 
urban interface we fell victim to the PSPS this year. Thousands was 
spent on upgrading battery plants that would normally hold during a 
short outage and generator purchases, whether it be small inverter 
style generators for small sites to permanent standby site generators 
for those sites that are larger or a PITA to get to. We still have 
more work to do and hope to be better prepared for next summers rounds 
of shut offs.  I am currently developing a portable trailer mounted 
solar/battery plant to replace the portable generators just for fuel 
cost savings since I spent just about $500/week in generator fuel 
alone for the largest outage.


Yeah, that's the biggest change: the outages are 2-3 days each. Maybe 
they'll get better but their stated goal is to inspect the entire 
distribution system before turning power back on, so that is going to 
take some time no matter what.


And the fuel cost is definitely a consideration. Unfortunately October 
and November is pretty lousy for solar. Wind might be better since that 
is the reason they're doing the PSPS in the first place :)


Mike

Re: power to the internet

[Tom Beecher]

If that was a reference to my comments, it was certainly not my intention.
I was striving to avoid it being seen as that, but apparently fell short.

To reanswer the question posed though, is still the same ; $$$. If network
operators take the position that the electric utility supply should be more
reliable than it is, then they need to start influencing and lobbying for
ways for that to happen. If not, they will have to increase investments
into local generation or storage capacity to bridge those gaps.

You seem to imply that regulation is inherently bad; however the scenario
that you describe (power failures impacting 911 service) is only a concern
to an operator if there is a legislatively define deterrent.

On Thu, Dec 26, 2019 at 13:00 Michael Thomas  wrote:

>
> On 12/26/19 7:51 AM, Mike Bolitho wrote:
> > I'm pretty sure political bickering is well beyond the scope of the
> > mailing list. Is anyone moderating this?
>
> It certainly wasn't my intent or desire to have this turn political, and
> shame on the person who did. This is a serious networking related issue
> for California *right* *now*. It may become a serious networking related
> issue for a lot of other places too -- California is hardly unique in
> its wildland - urban interface issues, and lots of places burn just like
> California. And definitely lots of places have a 100+ years of fire
> suppression which is a policy thing, not a political thing.
>
> The question is what are network operators going to do? If the answer is
> "nothing", don't be surprised to get legislation shoved down your
> throats. Don't expect the bay area of all places to passively put up
> with all of this. If your network fails because of power going out and I
> can't call 911, you've got a big, big problem.
>
> Mike
>
>
>

Re: power to the internet

[Jason Wilson]

As a small WISP operator in Northern California and well into the urban
interface we fell victim to the PSPS this year. Thousands was spent on
upgrading battery plants that would normally hold during a short outage and
generator purchases, whether it be small inverter style generators for
small sites to permanent standby site generators for those sites that are
larger or a PITA to get to. We still have more work to do and hope to be
better prepared for next summers rounds of shut offs.  I am currently
developing a portable trailer mounted solar/battery plant to replace the
portable generators just for fuel cost savings since I spent just about
$500/week in generator fuel alone for the largest outage.

On Thu, Dec 26, 2019 at 9:59 AM Michael Thomas  wrote:

>
> On 12/26/19 7:51 AM, Mike Bolitho wrote:
> > I'm pretty sure political bickering is well beyond the scope of the
> > mailing list. Is anyone moderating this?
>
> It certainly wasn't my intent or desire to have this turn political, and
> shame on the person who did. This is a serious networking related issue
> for California *right* *now*. It may become a serious networking related
> issue for a lot of other places too -- California is hardly unique in
> its wildland - urban interface issues, and lots of places burn just like
> California. And definitely lots of places have a 100+ years of fire
> suppression which is a policy thing, not a political thing.
>
> The question is what are network operators going to do? If the answer is
> "nothing", don't be surprised to get legislation shoved down your
> throats. Don't expect the bay area of all places to passively put up
> with all of this. If your network fails because of power going out and I
> can't call 911, you've got a big, big problem.
>
> Mike
>
>
> --

Jason Wilson
Remotely Located
Providing High Speed Internet to out of the way places.
530-651-1736
530-748-9608 Cell
www.remotelylocated.com

Re: power to the internet

[Michael Thomas]

On 12/26/19 7:51 AM, Mike Bolitho wrote:
I'm pretty sure political bickering is well beyond the scope of the 
mailing list. Is anyone moderating this?


It certainly wasn't my intent or desire to have this turn political, and 
shame on the person who did. This is a serious networking related issue 
for California *right* *now*. It may become a serious networking related 
issue for a lot of other places too -- California is hardly unique in 
its wildland - urban interface issues, and lots of places burn just like 
California. And definitely lots of places have a 100+ years of fire 
suppression which is a policy thing, not a political thing.


The question is what are network operators going to do? If the answer is 
"nothing", don't be surprised to get legislation shoved down your 
throats. Don't expect the bay area of all places to passively put up 
with all of this. If your network fails because of power going out and I 
can't call 911, you've got a big, big problem.


Mike

Re: power to the internet

[Mike Bolitho]

I'm pretty sure political bickering is well beyond the scope of the mailing
list. Is anyone moderating this?

- Mike Bolitho

On Thu, Dec 26, 2019, 7:20 AM Tom Beecher  wrote:

> Same story again different colors. PG making a mint while people get
>> screwed
>>
>
> I'm not quite sure that's an accurate statement.
>
> In 2000-2001, PG got screwed by Enron's market manipulation. ( Good job
> those who pushed so hard for deregulation of public utility services! )
>
> PG is currently in bankruptcy proceedings, largely as a result of
> liabilities from wildfires in 2017 and 2018. Under California's
> application of inverse condemnation, a power utility is responsible for any
> damage caused by a wildfire if it was determined that their equipment was
> part of the cause. This applies even if the utility was in 100% compliance
> with all laws and regulations.
>
> So you have a terrible combination where housing prices in the state are
> driving more and more people to build in wildfire prone areas, climate
> change is increasing the frequency of weather conditions favorable to
> wildfire ignition, and the utility company that is being held financially
> liable for damages while at the same time not being allowed by the PUC to
> raise capital for infrastructure changes to reduce the chances of
> electrical equipment starting such things.
>
> The answer is easy. Money. If people want a power grid that is safe and
> reliable, then the utility should be given the funds to do it via rates and
> appropriate tax revenues. They should not be expected to turn profits like
> private enterprise. The power grid is for the benefit of all, not just the
> financial benefit of those who have equity stakes.
>
> This situation is the logical extension of 40+ years of America's only
> real product ; financial engineering.
>
>
> On Wed, Dec 25, 2019 at 9:18 PM Michael Loftis  wrote:
>
>>
>>
>> On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin 
>> wrote:
>>
>>> On Wed, 25 Dec 2019 at 19:32, Michael Thomas  wrote:
>>>
 On the dark side, this is probably coming to a lot more states and
 countries due to climate change. Australia. Sigh.

>>>
>>> Do you have a source for this?  It would seem that these power issues
>>> are rather unique to California not because of some "climate change"
>>> bogeyman, but rather because of a failed public policy at the state level.
>>>
>>> It would also seem that these issues of rolling blackouts aren't even
>>> new to California, either, as, apparently, it's already been the norm
>>> during 2000/2001:
>>>
>>
>>
>> Having lived through the blackouts that was entirely different. 90% Enron
>> manipulating the markets. There was plenty of capacity both in transmission
>> and generation, but Enron manipulated prices and apparent supply to make
>> money and screwed the whole state over. There was just about 2x the
>> generating capacity, no real shortage.
>>
>> This time it’s PG all alone, but still fallout from back then. Too much
>> liability and they’ve not maintained the infrastructure and so they decided
>> that to reduce the liability costs it’s cheaper to blackout. Same story
>> again different colors. PG making a mint while people get screwed (PG
>> was mostly at the getting screwed end in 2000-2001)
>>
>>>
>>> * https://en.wikipedia.org/wiki/California_electricity_crisis
>>>
>>> C.
>>>
>> --
>>
>> "Genius might be described as a supreme capacity for getting its
>> possessors
>> into trouble of all kinds."
>> -- Samuel Butler
>>
>

Re: power to the internet

[Tom Beecher]

>
> Same story again different colors. PG making a mint while people get
> screwed
>

I'm not quite sure that's an accurate statement.

In 2000-2001, PG got screwed by Enron's market manipulation. ( Good job
those who pushed so hard for deregulation of public utility services! )

PG is currently in bankruptcy proceedings, largely as a result of
liabilities from wildfires in 2017 and 2018. Under California's
application of inverse condemnation, a power utility is responsible for any
damage caused by a wildfire if it was determined that their equipment was
part of the cause. This applies even if the utility was in 100% compliance
with all laws and regulations.

So you have a terrible combination where housing prices in the state are
driving more and more people to build in wildfire prone areas, climate
change is increasing the frequency of weather conditions favorable to
wildfire ignition, and the utility company that is being held financially
liable for damages while at the same time not being allowed by the PUC to
raise capital for infrastructure changes to reduce the chances of
electrical equipment starting such things.

The answer is easy. Money. If people want a power grid that is safe and
reliable, then the utility should be given the funds to do it via rates and
appropriate tax revenues. They should not be expected to turn profits like
private enterprise. The power grid is for the benefit of all, not just the
financial benefit of those who have equity stakes.

This situation is the logical extension of 40+ years of America's only real
product ; financial engineering.


On Wed, Dec 25, 2019 at 9:18 PM Michael Loftis  wrote:

>
>
> On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin 
> wrote:
>
>> On Wed, 25 Dec 2019 at 19:32, Michael Thomas  wrote:
>>
>>> On the dark side, this is probably coming to a lot more states and
>>> countries due to climate change. Australia. Sigh.
>>>
>>
>> Do you have a source for this?  It would seem that these power issues are
>> rather unique to California not because of some "climate change" bogeyman,
>> but rather because of a failed public policy at the state level.
>>
>> It would also seem that these issues of rolling blackouts aren't even new
>> to California, either, as, apparently, it's already been the norm during
>> 2000/2001:
>>
>
>
> Having lived through the blackouts that was entirely different. 90% Enron
> manipulating the markets. There was plenty of capacity both in transmission
> and generation, but Enron manipulated prices and apparent supply to make
> money and screwed the whole state over. There was just about 2x the
> generating capacity, no real shortage.
>
> This time it’s PG all alone, but still fallout from back then. Too much
> liability and they’ve not maintained the infrastructure and so they decided
> that to reduce the liability costs it’s cheaper to blackout. Same story
> again different colors. PG making a mint while people get screwed (PG
> was mostly at the getting screwed end in 2000-2001)
>
>>
>> * https://en.wikipedia.org/wiki/California_electricity_crisis
>>
>> C.
>>
> --
>
> "Genius might be described as a supreme capacity for getting its possessors
> into trouble of all kinds."
> -- Samuel Butler
>

Re: power to the internet

[Joe Maimon]

Unless telecom infrastructure has been diligently changing out the lead 
acid battery approach at all their remote terminals, powered gpon, hfc 
and antennae plants will never last more than minutes. If at all.


A traditional car has between a 100-200amp alternator @12volts

How much generating capacity can you get out of a typical hybrid?

Self-isolating and re-tieing inverters. Economic household ATS systems. 
Do those exist?


Enough independent distributed capacity and now comes the ability to 
create grid islands. How might that look?


Electric grid shortage is likely coming to NYC, courtesy of folk of 
certain political persuasion and their love of stone age era living. IP 
decommissioning.


If you have CO loop copper, keep it.

Joe

Don Gould wrote:

This is a very short term problem.

The market is going to fill with battery storage sooner rather than 
later.


Solar is just exploding.

Your car will "house tie".

6G will solve your data problem.

D



--
Don Gould
5 Cargill Place
Richmond
Christchurch, New Zealand
Mobile/Telegram: + 64 21 114 0699
www. bowenvale.co.nz



 Original message 
From: Michael Thomas 
Date: 26/12/19 2:33 PM (GMT+12:00)
To: nanog@nanog.org
Subject: power to the internet


https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678


This article details some of the issues with California's "new reality"
of planned blackouts. One of the big things that came to light with
these blackouts is that our network infrastructure's resilience is
pretty lacking. While I was (surprisingly to me) ok with my DSL
connection out in the boonies, lots and lots of people with cable
weren't so lucky. And I'm not sure how bad the situation is with
cellular infrastructure, but I assume it's not much better than cable.
And I wouldn't doubt that other DSL deployments go dark when power is
down. I have no clue with fiber.

So I guess what I'm wondering is what can we do about this? What should
we do about this? These days IP access is not just convenience, it's the
way we go about our lives, just like electricity itself. At base, it
seems to me that network operators should be required to keep the lights
on in blackouts just like POTS operators do now. If I have power to
light my modem or charge in my phone, I should be able to get onto the
net. That seems like table stakes.

One of the things we learned also is that the blackouts seem to last
between 2-3 days apiece. I happen to have a generator since I'm out in
the boonies and our power gets cut regularly because of snow, but not
everyone has that luxury. I kind of want to think that my router+modem
use about 20 watts, so powering it up would take about 1.5kwh for 3
days. a quick google look shows that I'd probably need to shell out $500
or so for a battery of that capacity, and that's doesn't include your
phones, laptops, tv's, etc power needs. What does that mean? That is a
major expense for a lot of people.

On the bright side, I hear that power generator companies stocks have
gone through the roof.

On the dark side, this is probably coming to a lot more states and
countries due to climate change. Australia. Sigh.

Mike

RE: power to the internet

[Don Gould]

This is a very short term problem. The market is going to fill with battery 
storage sooner rather than later. Solar is just exploding. Your car will "house 
tie".6G will solve your data problem. D-- Don Gould5 Cargill 
PlaceRichmondChristchurch, New ZealandMobile/Telegram: + 64 21 114 
0699www.bowenvale.co.nz
 Original message From: Michael Thomas  Date: 
26/12/19  2:33 PM  (GMT+12:00) To: nanog@nanog.org Subject: power to the 
internet 
https://www.politico.com/news/2019/12/25/california-power-shutoffs-089678This 
article details some of the issues with California's "new reality" of planned 
blackouts. One of the big things that came to light with these blackouts is 
that our network infrastructure's resilience is pretty lacking. While I was 
(surprisingly to me) ok with my DSL connection out in the boonies, lots and 
lots of people with cable weren't so lucky. And I'm not sure how bad the 
situation is with cellular infrastructure, but I assume it's not much better 
than cable. And I wouldn't doubt that other DSL deployments go dark when power 
is down. I have no clue with fiber.So I guess what I'm wondering is what can we 
do about this? What should we do about this? These days IP access is not just 
convenience, it's the way we go about our lives, just like electricity itself. 
At base, it seems to me that network operators should be required to keep the 
lights on in blackouts just like POTS operators do now. If I have power to 
light my modem or charge in my phone, I should be able to get onto the net. 
That seems like table stakes.One of the things we learned also is that the 
blackouts seem to last between 2-3 days apiece. I happen to have a generator 
since I'm out in the boonies and our power gets cut regularly because of snow, 
but not everyone has that luxury. I kind of want to think that my router+modem 
use about 20 watts, so powering it up would take about 1.5kwh for 3 days. a 
quick google look shows that I'd probably need to shell out $500 or so for a 
battery of that capacity, and that's doesn't include your phones, laptops, 
tv's, etc power needs. What does that mean? That is a major expense for a lot 
of people.On the bright side, I hear that power generator companies stocks have 
gone through the roof.On the dark side, this is probably coming to a lot more 
states and countries due to climate change. Australia. Sigh.Mike

Re: power to the internet

[Constantine A. Murenin]

On Wed, 25 Dec 2019 at 22:08, Michael Thomas  wrote:

>
> On 12/25/19 7:26 PM, Constantine A. Murenin wrote:
> >
> > I'm an ex-California resident myself here
>
> Good riddance. This has nothing to do with the climate change that is
> actually happening here.
>
> Mike
>

Well, enjoy your climate change!  Surely has nothing to do with the ban of
preemptive burning that works in adjacent states just fine!

C

Re: power to the internet

[Michael Thomas]

On 12/25/19 7:26 PM, Constantine A. Murenin wrote:


I'm an ex-California resident myself here


Good riddance. This has nothing to do with the climate change that is 
actually happening here.


Mike

Re: power to the internet

[Michael Thomas]

On 12/25/19 7:10 PM, Stephen Satchell wrote:

On 12/25/19 6:29 PM, Michael Thomas wrote:
Yes, this is exactly right. My point here isn't to assign blame, but 
to ask what the hell we're going to do about it. Trying to score 
political points is disgusting.


Do you live in California?



Yes.

Mike

Re: power to the internet

[Constantine A. Murenin]

On Wed, 25 Dec 2019 at 20:29, Michael Thomas  wrote:

>
> On 12/25/19 6:16 PM, Michael Loftis wrote:
>
>
>
> Having lived through the blackouts that was entirely different. 90% Enron
> manipulating the markets. There was plenty of capacity both in transmission
> and generation, but Enron manipulated prices and apparent supply to make
> money and screwed the whole state over. There was just about 2x the
> generating capacity, no real shortage.
>
> This time it’s PG all alone, but still fallout from back then. Too much
> liability and they’ve not maintained the infrastructure and so they decided
> that to reduce the liability costs it’s cheaper to blackout. Same story
> again different colors. PG making a mint while people get screwed (PG
> was mostly at the getting screwed end in 2000-2001)
>
>
> Yes, this is exactly right. My point here isn't to assign blame, but to
> ask what the hell we're going to do about it. Trying to score political
> points is disgusting.
>
> Mike
>


The same thing we've always done and recommended — Vote With Your Wallet.
Move to state that takes care of its infrastructure and doesn't have such a
gridlock.  Or remain in California if you think "climate deniers" (whatever
that term may mean) are "disgusting".

As a consumer and internet infrastructure operator, I don't particularly
see or care about the difference between PG getting screwed or doing the
screwing.It's the populace of the state that gets the resulting fallout
in terms of the rolling blackouts.  Which other state has had this in the
last 20 years?

I'm an ex-California resident myself here — voted with my wallet already.
Love the idea and implementation of an independent power grid of my new
home state.

C.

Re: power to the internet

[Stephen Satchell]

On 12/25/19 6:29 PM, Michael Thomas wrote:
Yes, this is exactly right. My point here isn't to assign blame, but to 
ask what the hell we're going to do about it. Trying to score political 
points is disgusting.


Do you live in California?  Do you have your business in California? 
Take a look at neighboring states.  I did.  California madness is why I 
now live in Nevada.


Our ecology doesn't have the Austrailian plant Eucalyptus.  We do have 
tumbleweeds which pose their own fire risk.  Place like Lake Tahoe is 
heavily forested -- the difference is that in Nevada there is active 
fuel control and controlled burns, so we have fewer burn-to-the-ground 
fires in populated areas.


I used to make a living as a freelance writer.  A GOOD living.  In 
Nevada I'm outside the reach of CA AB 5 should I want to give up 
$DAYJOB.  When I have to subcontract freelance work, I won't hire a 
California resident, because I don't want to be controlled by any 
"long-reach" laws.  Because the law and Calufornia court decisions are 
currently silent about C, S, and LLC writers, I avoid them until the 
climate becomes clearer.


Do I experience power outages?  Yes.  Longest duration?  Several hours, 
when high winds cause damage (but that damage doesn't start wildfires). 
One very nice thing is that where I am, we have a geothermal power plant 
close by (on the order of 15 miles).  No pre-emptive wide-area 
shutdowns, though.


NV Energy has photoelectric arrays that are part of the utility, not 
only on private roof-tops.

Re: power to the internet

[Michael Thomas]

On 12/25/19 6:16 PM, Michael Loftis wrote:



Having lived through the blackouts that was entirely different. 90% 
Enron manipulating the markets. There was plenty of capacity both in 
transmission and generation, but Enron manipulated prices and apparent 
supply to make money and screwed the whole state over. There was just 
about 2x the generating capacity, no real shortage.


This time it’s PG all alone, but still fallout from back then. Too 
much liability and they’ve not maintained the infrastructure and so 
they decided that to reduce the liability costs it’s cheaper to 
blackout. Same story again different colors. PG making a mint while 
people get screwed (PG was mostly at the getting screwed end in 
2000-2001)




Yes, this is exactly right. My point here isn't to assign blame, but to 
ask what the hell we're going to do about it. Trying to score political 
points is disgusting.


Mike

Re: power to the internet

[Michael Thomas]

On 12/25/19 5:59 PM, Constantine A. Murenin wrote:
On Wed, 25 Dec 2019 at 19:32, Michael Thomas > wrote:


On the dark side, this is probably coming to a lot more states and
countries due to climate change. Australia. Sigh.


Do you have a source for this?  It would seem that these power issues 
are rather unique to California not because of some "climate change" 
bogeyman, but rather because of a failed public policy at the state level.


There's no point in engaging climate deniers. This is what's happening 
to us right now regardless of what science you choose to call 
"bogeymen". We are living in fear of becoming the next Paradise.


Mike

Re: power to the internet

[Michael Loftis]

On Wed, Dec 25, 2019 at 19:00 Constantine A. Murenin 
wrote:

> On Wed, 25 Dec 2019 at 19:32, Michael Thomas  wrote:
>
>> On the dark side, this is probably coming to a lot more states and
>> countries due to climate change. Australia. Sigh.
>>
>
> Do you have a source for this?  It would seem that these power issues are
> rather unique to California not because of some "climate change" bogeyman,
> but rather because of a failed public policy at the state level.
>
> It would also seem that these issues of rolling blackouts aren't even new
> to California, either, as, apparently, it's already been the norm during
> 2000/2001:
>


Having lived through the blackouts that was entirely different. 90% Enron
manipulating the markets. There was plenty of capacity both in transmission
and generation, but Enron manipulated prices and apparent supply to make
money and screwed the whole state over. There was just about 2x the
generating capacity, no real shortage.

This time it’s PG all alone, but still fallout from back then. Too much
liability and they’ve not maintained the infrastructure and so they decided
that to reduce the liability costs it’s cheaper to blackout. Same story
again different colors. PG making a mint while people get screwed (PG
was mostly at the getting screwed end in 2000-2001)

>
> * https://en.wikipedia.org/wiki/California_electricity_crisis
>
> C.
>
-- 

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler

Re: power to the internet

[Constantine A. Murenin]

On Wed, 25 Dec 2019 at 19:32, Michael Thomas  wrote:

> On the dark side, this is probably coming to a lot more states and
> countries due to climate change. Australia. Sigh.
>

Do you have a source for this?  It would seem that these power issues are
rather unique to California not because of some "climate change" bogeyman,
but rather because of a failed public policy at the state level.

It would also seem that these issues of rolling blackouts aren't even new
to California, either, as, apparently, it's already been the norm during
2000/2001:

* https://en.wikipedia.org/wiki/California_electricity_crisis

C.